Vitalik Buterin is the founder of Ethereum, the world's second-largest cryptocurrency by market cap, currently valued at around $500 billion. But beyond revolutionizing blockchain technology, Vitalik has become one of the most thoughtful voices on AI safety and existential risk.
He's donated over $665 million to pandemic prevention and other causes, and has a 12% P(Doom) – putting him squarely in what I consider the "sane zone" for AI risk assessment. What makes Vitalik particularly interesting is that he's both a hardcore techno-optimist who built one of the most successful decentralized systems ever created, and someone willing to seriously consider AI regulation and coordination mechanisms.
Vitalik coined the term "d/acc" – defensive, decentralized, democratic, differential acceleration – as a middle path between uncritical AI acceleration and total pause scenarios. He argues we need to make the world more like Switzerland (defensible, decentralized) and less like the Eurasian steppes (vulnerable to conquest).
We dive deep into the tractability of AI alignment, whether current approaches like DAC can actually work when superintelligence arrives, and why he thinks a pluralistic world of competing AIs might be safer than a single aligned superintelligence. We also explore his vision for human-AI merger through brain-computer interfaces and uploading.
The crux of our disagreement is that I think we're heading for a "plants vs. animals" scenario where AI will simply operate on timescales we can't match, while Vitalik believes we can maintain agency through the right combination of defensive technologies and institutional design.
Finally, we tackle the discourse itself – I ask Vitalik to debunk the common ad hominem attacks against AI doomers, from "it's just a fringe position" to "no real builders believe in doom." His responses carry weight given his credibility as both a successful entrepreneur and someone who's maintained intellectual honesty throughout his career.
Timestamps
00:00:00 - Cold Open
00:00:37 - Introducing Vitalik Buterin
00:02:14 - Vitalik's altruism
00:04:36 - Rationalist community influence
00:06:30 - Opinion of Eliezer Yudkowsky and MIRI
00:09:00 - What’s Your P(Doom)™
00:24:42 - AI timelines
00:31:33 - AI consciousness
00:35:01 - Headroom above human intelligence
00:48:56 - Techno optimism discussion
00:58:38 - e/acc: Vibes-based ideology without deep arguments
01:02:49 - d/acc: Defensive, decentralized, democratic acceleration
01:11:37 - How plausible is d/acc?
01:20:53 - Why libertarian acceleration can paradoxically break decentralization
01:25:49 - Can we merge with AIs?
01:35:10 - Military AI concerns: How war accelerates dangerous development
01:42:26 - The intractability question
01:51:10 - Anthropic and tractability-washing the AI alignment problem
02:00:05 - The state of AI x-risk discourse
02:05:14 - Debunking ad hominem attacks against doomers
02:23:41 - Liron’s outro
Links
Vitalik’s website: https://vitalik.eth.limo
Vitalik’s Twitter: https://x.com/vitalikbuterin
Eliezer Yudkowsky’s explanation of p-Zombies: https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zombies
Transcript
Cold Open
Vitalik Buterin: I'm Vitalik Buterin and you're watching Doom Debates.
Liron Shapira: I'm excited to talk with Vitalik about d/acc - decentralized, democratic, defensive acceleration.
Vitalik: I would feel more confident in any ecosystem where there is some collection of different agents that are kind of aligned.
Liron: This is tractability washing the problem.
Vitalik: It's very possible that super intelligent AI alignment is intractable.
Liron: Vitalik Buterin, what's your P(Doom)?
Introduction and Vitalik's Background
Welcome to Doom Debates. My guest, Vitalik Buterin, is the creator of Ethereum, which is the number two cryptocurrency in the world after Bitcoin by market cap, currently valued at $500 billion. It's also a general purpose blockchain platform for decentralized applications.
Vitalik is a top philanthropist who supports many causes, including human longevity research, existential risk mitigation, especially AI X-risk, pandemic prevention and mitigation. He's donated a whopping $665 million worth of Shiba Inu tokens toward COVID relief programs in India.
Finally, he is a thought leader on a wide range of topics, including economics, governance, institution design, and AI. I'm excited to talk with Vitalik about P(Doom) and what can be done to lower it, especially his idea known as d/acc - decentralized, democratic, differential, defensive acceleration.
Vitalik Buterin, welcome to Doom Debates.
Vitalik: Thank you so much, Liron. It's great to be here.
Intellectual Journey and Rationalist Influences
Liron: You have many writings and talks focused on Ethereum blockchain systems, applied cryptography, institution design. What would you say is the central thread of your intellectual and productive career?
Vitalik: I think if I had to narrow down all of the goals of all of the different writings and projects that I've done into one sentence, I would have to say it's about a more free and open world, but particularly in a way that actually works for people and that isn't just about ideology for the sake of itself, but about something that actually makes the world better.
Liron: I want to tell the viewers, there's a couple things I really like about what you've been doing besides inventing Ethereum. It's this combination of theoretical and applied institution design work. And then it's also altruism - you're donating your money, you're encouraging other people to donate their money. You're setting a great example of how to use money to help people. That is quite a good mix.
You have a good example of one of the beneficiaries of this kind of funding.
Vitalik: The Balvi Fund - this is what came out of the dog money from 2021. Those dog coins that turned into way more money than I expected. It's ended up funding this global effort to try to support very full spectrum and fully open source biodefense - basically interrupting pandemics at every step.
The biggest grantee that we've had is OpenWater. It's basically making medical imaging personal scale. The idea is it's like a device that you can wear and then it uses ultrasound. The use case that we got excited for is being able to detect micro clots, which are a common cause of long COVID.
But then we realized that actually this is super general purpose technology. Actually it even has applications for BCI, interestingly enough, because it's the exact same kind of stuff - you just point it at your brain instead of pointing it at a couple of blood vessels.
I don't think something at this level of funding and sophistication in this industry has been open sourced before. And this year and next year it's actually - they have a product and it's starting to have customers.
Liron: The other thing that's salient to me, going back through your work, you're all about high quality discourse in a way that's pretty rare. You're open to dialogue. You're always truth seeking. You write in a very candid way.
And then in the crypto space where it's so easy to overhype and pump your bags, you never did that. You were always a very straight shooter. I think everybody knows you're low ego, which is pretty impressive.
And then just in general, it would be so easy for somebody with your following to build an echo chamber and pump your ego. That's kind of a natural tendency. But you have just remained very humble. You're coming on podcasts like this - I'm not gonna pull my punches if I disagree with you. And you're totally cool with it because you value high quality discourse as productive.
Vitalik: Absolutely. Actually the rationalist community, especially people like Eliezer Yudkowsky and Scott Alexander, that I know you're a fan of as well, has been a really important teenage influence of mine in that regard.
A lot of the things that they really held up as high virtues - things like intellectual honesty, this concept of meta level values. So basically trying to build a world in such a way that you're not making things horribly wrong, even if you're wrong about a lot of issues. Culturalism as well. And trying to see things from the bigger picture and not just be a soldier of a particular tribe.
Those kinds of things I grew up reading when I was in high school. And I think they set a moral example that I've always strived to follow.
Liron: So you basically first discovered LessWrong when you were in high school.
Vitalik: I think so, yeah.
Liron: How much do you play prediction markets?
Vitalik: Actually quite a bit. I do have a Polymarket portfolio. I'm one of these contrarian people. I basically find questions that people are just obviously overexcited about.
I made some money last year - there was a market on whether or not there would be a civil war in the UK in 2024. I think this started because Elon tweeted "there will definitely be a civil war." How many civil wars has the UK had in the last 10 decades? The base rate is definitely much less than once per decade. So I said, you know, I'll buy "no." Doing this kind of stuff is definitely one of my many side hobbies.
Liron: What's your overall opinion of Eliezer Yudkowsky?
Vitalik: I actually really highly respect Eliezer. I think a lot of the writing that he wrote in the Sequences made things clear for me in a lot of ways that all kinds of other writing did not.
Rationalism sometimes gets criticized in very strong ways. People point to Sam Bankman-Fried - they would say he defrauded people for $5 billion to fund their effective altruism charities. But then we have Eliezer and Scott on record writing lots of things that would directly contradict what Sam did. So I respect him for that.
MIRI did a lot of important foundational work for AI safety and I'm glad that the space is also now big enough where it's managed to be far beyond just them. So generally quite positive.
Liron: What do you think of Eliezer's thesis that if anyone builds it, everyone dies? Referring to super intelligent AI.
Vitalik: His worldview was definitely one of the worldviews that I consider very plausible. And it's definitely something that worries me.
Liron: I think it's really great that you're coming out and saying that you deeply respect Eliezer, because some people throw shade on him. Obviously not me, but I've seen some people in the tech community throwing shade and even going so far as to be like, oh, doomers, he's just trying to make money, this is his grift or whatever.
A lot of those same people actually deeply respect you and your decentralization ideals and your system design. So by the transitive property of respect, seeing that you respect Eliezer, I think that a lot of those people should take a second look.
Vitalik: I think they absolutely should. Though I think one of the challenges is that once you become part of a machine that has things like money, it has things like human beings that are dedicated to the organism and see it as being part of their identity, you definitely need more than just logic to convince people.
But at the same time, it's good to be optimistic and definitely hope that logic and reason can continue to be a common ground and both sides can continue being convincible of new things.
P(Doom) Discussion - Vitalik's 12% Estimate
Liron: So we're heading into the real meat of the discussion here. Let's kick it off with the number one question that everybody wants to know. You ready for this?
Vitalik: Ready.
Liron: Vitalik Buterin, what's your P(Doom)?
Vitalik: About 12% right now.
Liron: Oh, really? 12%. I think maybe it started at 10 and then it got lower for a while, so now it's back up.
Vitalik: Mm-hmm.
Liron: What happened?
Vitalik: I think geopolitics becoming just much worse and more stupid over the past year or so is definitely a big part of it. I think that's one of those variables that determines whether or not the world cooperates or even individual countries within the world even feel like cooperating regardless of what everyone else does. So that's been a big update for me.
I think another one is that some of the paths that would have led to much longer timelines ended up being closed off for me. So I think the existence of things like chain of thought - my timelines have definitely been compressed somewhat by that.
Most of my P(Doom) mass is in the pre-2050 period. Basically, the more you make my timelines go longer, the more my P(Doom) drops. And similarly, if they go shorter, then P(Doom) goes up.
It even went up even higher at the start of the year. I think at the start of the year, it was like 15 or 16%. It felt like the whole agent and chain of thought stuff could go much further and even faster. The political side of the world looked like it would break even more. But it does feel like both of those variables have calmed down somewhat.
Liron: I want to refine P(Doom) a little bit. So you said it's 12% and you said that if timelines were longer then it would be lower. Let me throw a pin in 2050. So what if I define doom as everybody is dead - humanity is extinguished by 2050. Is that roughly what your 12% refers to?
Vitalik: Let me think about this. I think what percent of that is doom pre-2050 - at least 9%.
Liron: When I talk about P(Doom), my position is I think by 2040 there's roughly a 50-50 chance that it's all over. So I'm like a 50% doomer, but I also think it's not like it's a hard 50. It's not like 50.000%. It's obviously very vague. When I say 50, what I'm really saying is I think it's kind of crazy for people to say 1% or 2% or 0.1%. And I also think it's kind of crazy to be like 99%.
So I feel confident that I'm sane in terms of having the right order of magnitude. I think 10% to 90%, as famously Jan Leike from OpenAI was saying, "my P(Doom) is 10% to 90%." To me, that captures the sane zone. It starts to get not sane if you're outside that zone.
Vitalik: I think there's definitely lots of smart people at all kinds of places. I mean, there's also smart people who are outside the sane zone in both directions whom I definitely respect. But I definitely have this general intuition that people who say probabilities of really complicated things in the world being very close to zero or very close to 100% are being overconfident about something.
AI Safety Statement and Mainstream Recognition
Liron: You signed the famous Center for AI Safety statement on AI risk. The one that says, "Mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemics and nuclear war." What was your reasoning there?
Vitalik: I do think that the risk of extinction from AI and even sub-extinction level, very bad things that could come from AI are something that's clearly very important to watch and pay attention to. This is clearly an incredibly powerful technology.
What I call the standard story of AI Doom - basically the combination of very rapid capability growth and the orthogonality thesis, instrumental convergence, just leading to a super powerful AI wanting to kill us. Not because it wants to, but because we're made of atoms that it could rearrange into something else. That makes enough sense.
And I've just never heard a super amazing knockdown argument against it. And I think if you pair that with the fact that the portion of society that's thinking about how to make this kind of thing less likely to be a risk or a problem is definitely quite a bit smaller than it should be given how fast capabilities are going. It's clearly something important that we should pay attention to and view with that level of care and concern.
Liron: So this statement came out in 2023, so a couple years ago now. Are you happy with how it built mutual knowledge that all of these luminaries all agree that it's risky?
Vitalik: It definitely did a good job of making the topic mainstream in a very positive way. I think what things like that statement did is it helps to show that actually worry about AI risk is a very mainstream thing within the field. And I think that did a lot to help bring the public around to view that this is very mainstream and not at all fringe to worry about. So it's been a valuable part of the discussion in that way.
Liron: I completely agree. I think a lot of the work of mitigating doom starts with moving the Overton window where people just realize that everybody else is also on the same page of "Hey, this super intelligent AI thing seems like a real concern," as opposed to totally ignoring it, which most people are still doing.
Vitalik: Absolutely.
AI Timelines and Definitions of AGI/ASI
Liron: Let's talk about AI timelines. I know you've got some thoughts there. When is AGI or ASI most likely coming?
Vitalik: I'll start with my definitions of AGI and ASI, because they're somewhat bespoke. My AGI is AI that is powerful and general enough that if you were to suddenly upload it into robot bodies and all human beings suddenly disappeared at the same time, it would be able to independently continue civilization.
That's very clearly the moment at which we're not just dealing with a tool, we're actually dealing with a species and where we're dealing with something where it's no longer unambiguous that humans are the top dog. You actually have two different entities, both of which could do their own thing absolutely on their own if they had to.
AGI. And then for ASI, my definition of ASI is AI that is powerful enough that basically human beings can no longer contribute, even in assistance, to performance for any task other than tasks where human beings are just explicitly valued in that task for being human.
One of the motivating analogies here is if you look at chess, then computers got better than average professionals at chess sometime around 1980. Computers got better than the world's best chess player - the famous Deep Blue versus Kasparov match in 1996. But then computers got so powerful that computer alone beat a human plus computer. So what was called a centaur - that happened in 2017.
So there was actually a 20-year regime during which human plus AI was the most optimal combination.
Liron: How did you know it was 2017? Because I actually looked this up and I couldn't find a definite year on that.
Vitalik: I remember there was an article by Tyler Cowen talking about centaurs in the early to mid-2010s that definitely talked about it as not a dead concept. And then I remember people very recently talking about centaurs as a dead concept. And at some point, I asked an AI and it just gave 2017 as the answer.
Liron: I mean, I have to believe by this point, I don't think a centaur helps just because it'd be kind of crazy if it did. But you could have a situation - remember AlphaGo kind of got hacked? Like somebody beat AlphaGo by doing something really unexpected. So I can imagine maybe the only role of the human centaur is still to be there and avoid somebody doing something way out of distribution.
Vitalik: Very possible. But if you look at the way human AI collaboration works today - even my own experience as a coder - before AI, coding was 90% bug fixing. Now coding is 95% bug fixing.
But my 95% confidence for AGI, I think when I was asked this about a year and a half ago, it was 2027 to 2200.
Liron: I mean, 95% is a huge confidence. So it makes sense that your interval would be wide. I wouldn't even ask you for a 95%.
Vitalik: I think I have two large estimates that overlap quite a bit. The distribution maybe looks two-peaked. But the biggest part is a single peak. And it's kind of in this zone where it could be 5 to 10 years away. It could be 30 years away.
And I look at this as there being two kind of stories that are describing AI progress. The first story is where basically right now we're on the train. We're on a train whose destination stop is being thrown off and immediately thrown onto a doom train right as it's already getting off the cliff.
But the train is that there's rapid growth in capabilities. There's rapid growth in investment. There's enough of a self-sustaining industry that every problem is quickly being solved. The dominating exponential chart that you would follow is this chart of how long are the activities that an AI can take in terms of the time it would take for a human to evaluate them. And this kind of evaluates independent agency and independent ability to do things without human assistance. And it's been doubling roughly once every seven months.
Liron: Just to clarify for the viewers, you're talking about the chart where it's saying how long of a task can you have a human versus an AI where the AIs are beating the humans? Like maybe they can beat the humans for a two-hour task, but then if it gets longer than the human takes the lead.
Vitalik: Right. If you follow that chart, then basically the time at which that time intersects with a full human career looks like it might be the early to mid-2030s. And then you could also see that even accelerating, because as it becomes more and more clear that actually AI is a really big deal, then people keep putting more and more resources into it. And you could easily see this being double digit percentages of GDP even within a couple of years.
And then AI gets to human level and even before that, we get AI researchers. AI researchers accelerate AI progress. And then it just very quickly fooms and then it blows right past the human level. And very quickly goes super intelligent.
The other one of my two stories is this story where basically the thing that we're doing this decade is doing a bit of a replay of what we did in the 1970s where in the 1970s basically people got really excited because AI was able to do things like math calculation. A lot of these rote tasks that could be formally specified and just blew past them at a very superhuman level.
And people's intuitions were like, oh, it's really hard for humans to calculate a hundred digits of pi in our heads. This computer can do it. And so how hard can it be for a computer to tell apart a cat from a dog? And at the time, there was a lot of this thought that maybe AGI is actually only 5 or 10 years away. And that ended up not being true.
What ended up being true was we ended up being very good at automating one subset of human capability. But then there were these other sets of human capability that were still very far away.
And the story is that maybe we're repeating the exact same thing except it's more subtle what the category divide is. And there's different ways to make that divide. You can talk about interpolation versus extrapolation, within domain versus making things that are very fundamentally new.
I think the best formalization of this is probably still François Chollet's ARC prize, but if the ARC Prize ends up falling this year, that doesn't invalidate the story. I think the central use case that I think will still be hard - this is annoyingly hard to measure - is I'm very confident that within two years AIs will not be able to independently start a new large scale online business.
Liron: That's a good milestone.
Vitalik: Yeah. It's just this very complete, very comprehensive task that involves interacting with the environment, solving technical problems, solving business problems, solving human coordination problems.
But the question is what if basically in the 2020s, we're seeing a boom of AI getting really good at these tasks that you might call within distribution work interpolation. And a lot of tasks fall into that category, but there's still this somewhat more illegible subset that humans are just naturally good at.
And then the story is well, we're actually gonna blow past those eventually. And the reason why I say 2050 is there was this other interesting analysis I remember from I think maybe about a couple of years ago where people looked at basically if you estimate what is the amount of compute that went into the process of evolution, and the amount of compute that goes into an individual human life, then the first one is bigger. And then you say, well, when is the amount of compute going into AI going to blow past that? And it looks like about mid-century is the answer.
And then the argument is life by itself created something as powerful as us once by basically just very naively stirring a big computational soup for about 10 to the 40 steps. And then we came out. And so maybe if we just fumble around and do roughly the same thing, then after 10 to the 40 steps, something like us will come out. So that's a plausible argument.
Basically the more we get to that, it'll be a combination of the two, but I could see AGI and then coming out of that and then ASI probably taking somewhat longer.
Liron: If you just had to guess which year AGI is coming by your definition of it could sustain its own civilization, what's your mainline year for that?
Vitalik: Probably 2030s.
Liron: I mean, I think that's the consensus of people who are looking at it. That does seem like a good mainline guess. I guess early 2030s would be the most popular answer.
Vitalik: And I think there's some chance it's that, there's some chance it's late 2030s, and then I think there's some chance that we get something that meets this definition of AGI, but then it actually takes quite a long time for it to get to the ASI level. And you know, say it turns out that the 21-year time for chess - it basically turns out that it actually takes that long or potentially even longer for AI to break through from the AGI level to ASI.
So I think a lot of those things are possible. And I want to work on things that are valuable in a large subset of those possibilities.
Current AI Limitations vs Future Capabilities
Liron: How would you describe what current AIs lack compared to true AGI? LLMs - people love to dis LLMs. Remember when they'd call them a stochastic parrot. I feel like that term is going out of favor as people realize okay, it's gotta be more than that. It's doing some pretty sophisticated functionality that involves deeply understanding the meanings of things besides just statistics.
So they're mostly not calling it a stochastic parrot, but what do you think is fundamentally still the limitation of LLM-based systems?
Vitalik: So the pattern I notice when I use them is I notice that AI is amazing at helping me navigate domains that are well-trodden by humanity, but where I'm a noob. So one example of this is Android development. I've just never really done Android development before. And then this year I decided to just make a couple of simple productivity apps for myself and it helps me through it. And I was able to actually make something basic in one to two hours, and that would've just been unimaginable before.
But I find AI as close to useless, sometimes absolutely useless in domains where I am a top 1000 domain expert. And especially these very new domains. Things like cryptography is probably one of the big examples and some of the crazier zero-knowledge SNARK type of stuff.
Basically I think one of the limitations is that to achieve a high level of quality, it's still dependent on a large number of training examples. And so this is where this kind of intuition that they're good at something that you might analogize to interpolation, but they're bad at things that you would analogize to extrapolation, comes out.
System one versus system two thinking is one of those analogies - they're better at things that are more like instinct than deep thought. And especially deep original thought.
Liron: But o3 is pretty impressive. The kind of chain of thought systems or the reasoning models, because it does think. And then take an action and then search the web, pull up a bunch of documents, analyze them, think about what to do next. So I feel like that's system two.
Vitalik: It does. But it definitely gives answers that score extremely highly on impressiveness, but often score much lower on utility to me. I give it hard problems - I have a new laptop that has a 4090 on it because I want to do local AI stuff and I have Linux and I have Linux compatibility issues. And it keeps coming up with these amazingly clever plans that seem like they show a deep understanding of Linux and what to do, but then I do them and they don't work. I do them again and they don't work.
The fact that the weakness that it has is of this type is very relevant from an AI safety perspective. Because one of the big arguments for risk is this kind of super intelligent foom issue, where basically the AI can do really amazing things in the context of our current world and our current science. And then it can create the next wave of science and then it can do even better things and create the next wave of science after that.
But then the AI as I've just described - the way that AI is today - then what would actually happen today is it does amazing things using the current wave of science. And then it invents the next wave and then it starts doing things with that. And then now that it's a little bit out of distribution, it just starts completely fumbling and then it screws up.
And so for both any kind of super progress to happen and for anything really risky to happen, it does really have to cross that gap.
Liron: I mean, that makes sense. Using LLMs today, they have a lot more trouble giving good answers outside of being similar to documents they've seen. I mean, I definitely know what you mean. It's just always weird to me because the next version comes out and they just somehow get better.
So it's like they're obviously not fully good, but I hesitate to make any confident predictions about the next year.
Vitalik: And I think to be clear, I am definitely very against these kinds of dismissive theories of AI that say AI is just X and human beings are something more profound. I think if you have some kind of reductive theory that says AI is just X, then I think there's one of three things that are true.
Thing one is actually AI already is not just X. Thing two, possibility two is AI will soon be not just X, and possibility three is well, actually humans are also just X.
There's that lovely internet meme that was "hey, don't mind the tiger. It's just something that's made out of atoms and it knows that you're made out of atoms and it's just trying to helpfully rearrange some of them for you."
Exactly. I'm very against these attempts to be dismissive of AI in a very categorical way. I think the weaknesses that I talk about, they're true of AI today. They definitely will not be true of AI at some point in the future. At some point in the future, AI will be able to do either all of the things that humans do or some of the things that humans do plus enough things that are a replacement of capabilities that maybe it never just needs to get, because there's even better paths to the same goal.
And the question is basically when, and in what format it'll reach that.
Liron: There's some people, famously Roger Penrose, but also a lot of people in my YouTube comments who think that consciousness is a firewall between current AIs and humans. Do you think that that is going to stop AIs from having human-like functionality if they're not conscious?
Vitalik: No, and my theory is that it's very likely that AI will eventually be conscious. I'm one of these algorithmically centered people. So I believe consciousness is a property of algorithms of a certain type. And so AIs are conscious, and simulations, if we get uploaded into computers, then the upload is gonna be conscious as well.
My argument for this is basically step one is I don't believe in P-zombies. So P-zombies are a fancy term for basically entities that act like people and have everything in common with people except for not being conscious. And the reason why I don't believe in them is actually, consciousness is not just purely a metaphysical property. Consciousness is a property that influences the physically observable world.
And the way that it influences the physical observable world is that there's lots of people that talk about consciousness - words that get written down that are collections of atoms that appear in a particular pattern in the universe. And so it feels extremely Occam's Razor unintuitive that you would have entities be able to make that - right, things that talk about consciousness without actually being conscious.
And so if you accept step one, then step two - if you believe that it's possible to simulate the laws of physics at all, if basically if you believe the laws of physics are laws, then you should be able to implement them in silicon. And then if you implement in silicon, then humans evolve in exactly the same way. And then they end up talking about consciousness inside of silicon.
And then if you accept those two things, then that implies that it is possible to have consciousness inside of silicon. Then the next question is, well, will AI be that type? And the basic argument is we evolved consciousness at some point in time because consciousness was useful to us, because it enables certain forms of agency to be more effective.
And AI is being increasingly built for agency. And also AI is being built by mimicking humans and getting lots of training data from humans. And so we get basically two vectors that select for consciousness. And so there's some chance that that'll actually lead to consciousness and some chance it won't.
Liron: I agree with everything you said. And that point you made about P-zombies - I think we're both reading the same source here. Eliezer has written very eloquently on this. I'll put it in the show notes - the LessWrong P-zombies sequence.
So it sounds like you don't have a hardcore principle. You don't have a firewall saying current LLMs can never break through this firewall. Some people say, oh, it's not truly reasoning. It's not truly agentic. It's not truly conscious. It sounds like you don't have a firewall. You're just open to having it extrapolate in a number of different ways. Correct.
Vitalik: I think so, yeah.
Headroom Above Human Intelligence
Liron: Talk about headroom above human intelligence, or in other words, how powerful do you expect ASI to eventually become, say, in a hundred years?
Vitalik: I think quite a bit above human intelligence. I mean, one of the arguments against this is that each IQ point only gives you 1% extra income. And then potentially it even levels off past 140 or 160. And then maybe actually if you go above 160, you start going as an inevitable byproduct, crazy in some counterproductive ways. And so maybe we're at the cap. That's the standard argument that people would give against.
The other argument against that I've heard is if you actually look, compare neurons to thermodynamic limits, I remember there was an article on this and I think that one gave the impression that we're still somewhere between three to eight orders of magnitude from being optimal.
The reason why the IQ stuff doesn't seem persuasive to me is because I think IQ type intelligence gains are not the only way that AI can be better than humans. The most natural other approach is just clock cycle time. If you just imagine an AI that has an IQ of 125, but that can think 10,000 times faster than we do. So basically in the time that it takes for me to say a word, it's gonna be able to finish saying something of the length of half of this podcast.
Then that's something that definitely can outthink humans and run circles around humans and lead to very fast economic doubling times and all kinds of things. I mean, it actually gets really spooky in interesting ways. One of the observations from Robin Hanson's book on the age of em that really stuck with me is how if you think a thousand times faster than humans, then the subjective speed of light drops to 300 kilometers a second. And then if you think a million times faster than humans, the subjective speed of light drops to 300 meters per second.
And so basically in some ways we sort of go back to ancient times when cities were far away and messages had to go between on horses. From the perspective of us, everything would just be zooming in crazy. Things would happen. So I definitely think quite a bit of headroom.
Liron: There's this very striking visual when you think about plants. Plants have a rich inner life - they bend toward the sun. They do kind of move and how they grow, it just happens over a really slow timescale. And when you put them head to head versus an animal, it's not exactly a fair fight.
Vitalik: Absolutely.
Liron: So I do think we're - I do honestly expect to see that kind of phenomenon. Actions per minute. It's like the AI is going to play the game of life with much higher actions per minute than you and I do.
Vitalik: That's something that many different forms of AI are doing already. I mean, I think unfortunately there's some even pretty dark things that create incentives in exactly that direction. I'm sure both of us have been following the rise of AI-assisted warfare. We're definitely not quite there yet. The theme this year has been fiber optic drones, which are basically human powered drones that literally have 20 kilometer cables sticking out the back. So they're still controlled by human operators, but 20 kilometer cables are freaking annoying, and there's just lots of pressure to remove the need for a human operator.
And for that kind of competitive game, once it gets to bot versus bot, then reducing reaction time becomes incredibly important. And the other big one is finance. You could imagine just there being incentives to push reaction times even further down. So I definitely think we're going to go there.
Liron: Okay. So what about the sci-fi stuff? Nanotech and some people give Eliezer a lot of flak for proposing this, but I'm on board with it. It's just the idea that the AI will do things that to us feel like science fiction, but they're basically accessible new types of technologies that we haven't gotten to yet. Like we haven't gotten to nanotech. But in principle, Eric Drexler's book about nanotech - in principle, it seems like the AI should be able to figure out how to do this kind of thing pretty quickly without necessarily many years of research. What do you think?
Vitalik: Nanotech in particular is one of those things that I have a big uncertainty about. Because in a lot of sense, nanotech is kind of a superset of everything. It's a superset of pandemics, it's a superset of drones. It's a superset of data centers. And it's theoretically extremely powerful.
But then on the other hand, I have this nagging instinct that if there were very low molecule count things that could have some of those properties, then life and evolution would've figured them out by now. And I definitely feel a little bit out of my depth in evaluating this kind of thing.
And then there's also just the kind of look at outside view things. And then it was just facts that this stuff was theorized as being a big deal in the 1990s. But then since then, we've somehow stopped making much progress on it.
So it could be the case that AI is the thing that actually discovers everything and discovers how to do it super well. Or it could be the case that there's fundamental thermodynamic instability of small molecule configuration type of reasons why that stuff is fundamentally limited. The distribution for me.
Liron: To your other point about observing that life hasn't discovered new types of nanotech, I guess besides the familiar types of stuff in cells, I would point out that life is very constrained. The tech tree, the build path of cells. It's like you have to work with proteins that are coming from amino acids. And Eliezer points out their bonds are very weak. They're using van der Waals forces, so they don't really operate with the full spectrum of ways you could put molecules together.
Vitalik: Okay. So here's one argument I would make, and if you've come up with an amazing rebuttal then I would enjoy being enlightened is basically, okay. So if something is more efficient than life, then chances are what needs to happen by being smaller than life.
The probability of abiogenesis, intuitively speaking should be exponential with exponent with base lower than one in the complexity of the thing. Because if you have even one extra component that needs to be added for a basic replicator to be possible, then the number of things that need to come together at the same time by pure chance increases by one. And so it takes maybe twice or some number of times as long.
And so plausibly abiogenesis just with very high probability created the simplest possible thing that is capable of being a replicator. Right. But then the question is if there are other types of things that evolution can't reach - I agree, there's lots of very valuable things that evolution can't reach. There's a reason we don't have zebras that shoot lions with machine guns, or wheels for that matter.
But then intuitively, it's more likely to be more efficient if it's simpler in some way. And if it's simpler than why wouldn't abiogenesis have produced it first? And then if it's more complex than are we even sure that it would actually be more efficient than it'd be able to outcompete life.
Liron: But you do agree that in general, when you have a human engineer at a work bench, they have quite a lot of options that the process of evolution won't have.
Vitalik: Yes. Absolutely.
Liron: The reason I bring up the sci-fi stuff is just so from my perspective, and by the way, I'm also hunting for a disagreement. Because I do think you and I will have a disagreement. So I want viewers to enjoy the fireworks here.
I personally think that there's a lot of headroom above human intelligence. And I think that the universe we live in, it's pretty low entropy. At the end of the day, the laws of physics probably are pretty simple. They don't take that many bits to specify, is my impression.
And I think that the AI pretty quickly will just figure it out. It'll just kind of be pressing up against the walls of what there is to know about how to engineer things in our universe. And I do think it'll subjectively feel like a blueprint where you can just decide where the atoms go. It has a lot of mastery over how to configure this universe in a way that we humans, I would argue we're getting there if you extrapolate our trajectory. That actually just seems like the natural extrapolation. But we are a century or two away, but the AI is going to get there very soon.
And so I just brought up nanotech as an example to test the waters with you of do you also think that that is what seems likely to happen?
Vitalik: Nanotech is this interesting example where if it's possible then it's definitely super scary because it's at that level that would outcompete anything that isn't nanotech. But then on the other hand, for nano bots to really be nano, their source code has to be in the kilobytes and tens of kilobytes. And that does feel like a range within which it's possible for evolution to really try a huge number of different things.
And of course you could argue that there are ideal solutions where something that's even one step away from the ideal solution doesn't get you anywhere. And so evolution just never discovers it. Like if you think of algorithms from computer science, fast Fourier transform - a fast Fourier transform with one bit flipped the wrong way is not something that's almost as good as a fast Fourier transform. It's just random junk.
So I think if you start going through different examples of specific types of super technology, I think you do have to just get very empirical about each one.
Liron: So we could talk about space travel and Dyson swarms. Just harvest all the energy of the star. That seems like the kind of thing where physics totally says it's possible. So it's just a matter of getting in the weeds and engineering and it seems like we as humans, given enough decades, we are gonna do it. I just think that the AI can just kind of jump to the solution. That's my guess about how the AI is going to work. What the level of intelligence will just make engineering seem so easy to it.
Vitalik: I agree. So with Dyson swarms in particular, I think AI will definitely create them eventually. I think the question is how much time it will take. There's definitely fundamental constraints on the speed of rearranging atoms, and its ability to expand out to a full industry that rearranges the atoms of that planet to something that could be a Dyson sphere.
From an AI safety perspective, the relevant variable is basically is it something where it's able to just suddenly snap its fingers and take over? Or is it something that would take time? Because if it's the former, then we're screwed. And if it's the latter, then that's a process where there's a lot of potential places for basically people aided by every other AI to participate and intervene in.
Liron: A common pushback when I bring up this line of reasoning is people are like, well, chaos man, you can't really do that much with the universe beyond what humans can do. Because you stop being able to predict what's going to happen. Because there's chaos. Are you one of those people?
Vitalik: I mean, I think there's definitely things that ASI will not be able to do for chaos related reasons. My probability is that 90% chance that ASI will not be able to find pre-image for SHA-512.
And also, similarly, I expect that there will be bounds on how much long-term predicting ASI will be able to do. I think 50% confidence for the three body problem. I believe in the idea that there is such a thing as fundamentally chaotic systems, and it's actually easy to get to them, even accidentally. And they're unpredictable, but I definitely don't think that that stuff is an argument for why you can't have Dyson spheres or why you can't make big tech advancements compared to today.
Liron: It seems like the pattern is pretty well established with human engineering of yes, there's chaos. Yes. You can't predict anything, but in practice, you make the airplane and only one in a million people die on it. The airplane works. Even though Navier-Stokes or whatever is really complicated, but we still do it.
Vitalik: Yeah, there's definitely a lot of phenomena that are chaotic at the micro level, but then actually follow reasonably understandable laws where sometimes you have to model one or two things by randomness at the macro level, and that just ends up being totally navigable.
Techno-Optimism and Historical Patterns
Liron: So it sounds like a lot of agreement here so far. Let's move on to talk about this concept of techno optimism. This is all the way back in 2023. Marc Andreessen published something about techno optimism, and then you responded to it with d/acc, which we'll get to in a second. But before that, let me just ask you, are you a techno optimist?
Vitalik: I think, yes. I think historically speaking technology has been probably the most powerful source of good in the world. And I think by default there are a lot of just amazing things that we should be able to expect out of technology.
I expect, to the extent that politics doesn't interfere, technology will make food and housing ultra cheap. It will cure all diseases. It will do a lot of things to potentially even help us become better people. Technology has done all of these things. Lots of diseases have been cured. Because of the internet we're all able to be much more informed than we were before.
Even technology does sometimes create problems. But often a trend that I believe in is that when wave n of technology creates a problem, wave n plus one of a technology solves it. The best example of this for me is air pollution, where air pollution was an incredibly big deal in the 1950s. By our standards, cities like London and Los Angeles were unlivable. But then basically decided like, hey, we want this problem solved and we solved it. And now those cities are relatively speaking great.
Liron: And even before air pollution, you had horses crapping all over the road.
Vitalik: Yep. Exactly. I believe in that. But I think one of the big motivating charts for this is if you look at the chart of life expectancy over the past century, it just looks like an up and up. And probably the biggest argument against most sort of unbridled techno optimism is oh, well what about unintended consequences? This stuff can be used for bad, but then what is the worst of the worst that can happen?
And the answer in the 20th century was World War II. And World War II is incredibly bad. And it made life expectancies drop by a lot that was visible on the charts, but at the same time, life expectancy was higher in Germany in 1955 than in 1935.
And so it's this insanely powerful force where if it can even wash away things as bad as World War II, then you gotta give it some respect. But but then of course, there is this caveat that I think both of us agree in which is basically that super intelligent AI is not like other technologies.
And the other thing is that I think the pattern of technology making things better is not automatic. It depends on human beings saying like, wait, this wave of technology caused this problem, we have to go and fix it. And I think we are human beings that are part of this system and we have to be of that process too.
So I think those are the two big caveats on my techno optimism. Basically. One, ASI is different and two, the good things are not automatic and we have to do actually do things and often even do things that involve complicated stuff like government policy to actually make the good things happen.
Liron: I am also a techno optimist. Some people may not realize this, but I definitely am. I am also impressed by this crazy graph that is exponential showing median human wealth. My favorite chapter of a book about techno optimism is the Rational Optimist by Matt Ridley.
Vitalik: I've not read it. I've heard of it.
Liron: Chapter one was my favorite. The other chapters are more forgettable for me, but chapter one is so good because it's basically just talking about how the past was so bad and the present is so much better, and I think it's spot on. And that is very profound. Not enough people appreciate this basic fact. Clearly you do.
If you were to ask me, hey, why do we have nice things? I'd be like, well, obviously technology and then also capitalism. I don't know which one's a bigger force. The technology part or the capitalism part, I feel like they're both needed.
Vitalik: Technology is the biggest force. And I think improved quality of institutions. Capitalism is definitely a big one. And then I think more inclusive and large coalition political orders are definitely - things like democracy, I think contributed quite a bit as well. And the reduction in war contributed quite a bit as well. Improvements in our ability to cooperate now compared to something like a thousand years ago.
Liron: So you're a techno optimist. Are you a transhumanist?
Vitalik: Let's see, how would I even define transhumanism? I think probably you just have to define it through a cluster definition. Do I want to have longevity technology to live 10,000 years, or potentially longer. Yes. Would I want to eventually be uploaded? I mean sure. Once it's safe.
And on the other hand, do I want to force the entire world to go into those things? No. So, but at the same time I think Transhumanists themselves, people who self identify as transhumanism, generally say like, if someone doesn't want to self upload. If someone just wants a regular human life, then they should absolutely have that.
So yeah, I think I'm a transhumanist.
Liron: All right, same here. Okay, so you pointed out when we were talking about techno optimism, that we have to put an asterisk because it seems like AI is not gonna be that much like other tech. It seems to pose more danger. For instance, with other tech, like you pointed out like, oh, we have cars and they're polluting. Oftentimes the answer is just plow forward, just iterate. It'll work itself out. More tech fixes everything.
But in the specific case of AI, it seems like there may be some dynamics that break the historical pattern for us techno optimists. Maybe elaborate on that.
Vitalik: The biggest thing is independent agency. Historically, technologies that we've invented. There are technologies that do a lot of things, but ultimately there has to be a human somewhere that presses something like a button to make the thing happen. And so power ultimately still stays in human's hands.
And even that it can be worrying in a lot of ways. Outside of AI, I think one of my bigger worries of technology is concentration of power, that some of modern digital technology leads to like the fact that it's even possible to create billions of devices that you sell to people, but then those devices ultimately still phone home and you can see everything that people are doing. You have access to an off switch. You're essentially not selling a product, you're selling sort of membership in your digital empire, and there's basically no limits to how that can scale. These things worry me.
But I think the really big thing is independent agency. And the fact that basically ASI is not a tool. It is a species and it's a species that can outcompete humans.
Liron: Now there's another angle to look at this, which is the danger of the development process and the inability to redo. Because if you look at why is tech so great, why is tech progress so worthwhile to invest in? Well, people work on tech and they iterate and they keep making it better. And the first version is usually janky and then the second version is better and so on.
But in the case of AI, you kind of have this positive feedback loop and there's a big danger that there won't be a second version. Correct.
Vitalik: Absolutely. I think the agency is a big part of making that the case. Because if you look at extremely destructive technologies that don't have agency, which could be super pandemics under the most pessimistic assumptions, it could be nuclear war under the most pessimistic assumptions, like these kinds of things. They're gonna kill a lot of people, but they're not gonna kill 100% of the population. At the very least you have Antarctic bases, you have submarines, you have individual places in far corners of the world. And they're going to survive it.
And the thing that killed most people, it's not a smart thing. It's not going to go and take active steps that where it says, hmm, my objective function is to kill people. Let me go and find people and make sure I haven't missed them. It's just a very dumb thing that executes one strategy. It succeeds where it succeeds, it fails where it fails and then it's done.
But ASI - no, it is goal directed and it is going to hunt those stray submarines and Antarctic bases down and kill them. So that is the danger.
d/acc: Decentralized/Democratic/Defensive Acceleration
Liron: Well if you go back a little, do a little history of tech discourse in 2023, 2024, there was this weird dichotomy where the effective altruism movement got mixed up with this idea of tech deceleration. As if it's synonymous when it's two very different ideas.
And then as a reaction to that, some people went off and invented this idea that tech acceleration is the opposite of effective altruism. And they called it effective acceleration. And they said that we should kind of throw caution to the wind and double down on tech acceleration. And they didn't seem sensitive to the idea that maybe AI is fundamentally different.
Vitalik: And I think one of the kind of meta things to keep in mind with today's ideologies is that honestly, the level of quality has gone down. And this is true at every level. If you even look at the manifestos of crazy people. The Unabomber - he killed people. But his manifesto on the future of industrial civilization has cogent arguments and they're very wrong arguments, but they're intellectual arguments that you can go grapple with. But then if you look at people of his category today, it's just vibes and it's like Bronze Age mindset. If you read that, it's all just vibes.
Liron: The political party platforms are also like nothing now.
Vitalik: Yeah. There's no kind of deep vision of this is a set of core fundamental principles that I believe in. These are the goals that I have. These are things that need to be different in order to try to achieve this goal. It's just extremely vibes based. It's extremely just rapidly make a thing that achieves a particular aesthetic. It's definitely less reasoned than before.
And honestly e/acc is that. E/acc does not have intellectual, deeply thought through arguments for why things like AI safety are wrong. It's just, hey, deceleration is darkness and this is light, and that's basically as far as it goes.
But at the same time, vibes do speak to people. And there are a lot of people who really feel that tech deceleration has gone too far. And I mean, this also gets tied into the whole discourse about wokeness having gone too far.
With vibes is like, they're so fundamentally about wanting to entangle with everything and the exact opposite of decoupling. And there's a lot of points of view from which effective accelerationism sounds totally correct. No, Europe should have more air conditioning. And degrowth is very wrong and ruinous and counterproductive.
And I think we should be much braver on many forms of biotech. And I think the people that want to ban synthetic meat are bad shit insane. There's lots and lots of tech questions where the acceleration side is totally correct, and two, where there are a lot of people that sincerely are pushing for the deceleration side. And often it just is because of this kind of very abstract and vibes based fear of change.
And so people are attracted to the vibe that takes the opposite side. But the thing with vibes as these sort of collective super agents is that they are less agent - they have a much lower IQ. And then they're not able to handle the nuance that well actually AI needs to be handled carefully.
Liron: So you stepped into the void late 2023, you noticed that the conversation was getting bogged down. It wasn't productive, so you basically pulled a Scott Adams high ground maneuver. You basically came in and said something that nobody can really disagree with. The idea that ideology aside. We all have a lot in common. Hoping AI goes well, we don't wanna miss out on Pareto improvements. There's a lot of low hanging fruit of stuff we can do.
And so thus enters d/acc, which means defensive, decentralized, democratic differential acceleration, correct?
Vitalik: Correct. And I think if you had to narrow it down to three, I would say it's defensive decentralized acceleration.
So Jason Crawford from Progress Studies movement was at the d/acc event in SF and he had this interesting analogy where he basically talked about d/acc as being something that integrates the strengths of three different eras.
Where the late 19th century was the era of being decentralized and being accelerationist, but it did not have defense. And so we had lots of market failures. We had lots of things like air pollution. We had a lot of these negative outcomes that people disliked.
Then the early 20th century is the era of peak statism in often the worst possible ways. It was the era of dominance of national security states. And basically that was the era of being defensive and accelerationist, but without being decentralized. And of course that led all kinds of extremely horrible things of its own culminating in World War II.
And then the second half of the 20th century is basically the hipster era. And hipsters are decentralized and they are democratic, but they don't like acceleration.
I grew up in Canada and I think in Canada, when I was in high school, there was this kind of dominant vibe where basically the ideal world is exactly like the world as it exists today, but with less greed and more public healthcare.
That's really the kind of sentiment that's common in I think definitely Europe. I think definitely many places in the US though some parts of the world don't feel it at all. And this is the thing we have to move past. And the question is can we get the benefits of all three of those things?
Liron: Remind me if you have, you said you had to pick two words that start with D, so what do you pick?
Vitalik: It was defensive and decentralized.
Liron: Alright, so for short it's just defensive, decentralized acceleration. All right. And it's a middle ground between pausing and accelerating. It's a high ground maneuver.
Now, you've pointed out that defense and decentralization are closely related, your top two, because your ideal vision. If I understand correctly, your ideal vision for d/acc is that you're enabling an equilibrium of many self-sovereign parties.
Vitalik: Correct. The motivating analogy I had there is if you compare the political environment of Switzerland to the political environment of the Eurasian steppes. Switzerland is amazingly able to protect its independence. It has a super decentralized structure internally. It's famously a country where nobody knows who the hell the president is. I'm actually not even gonna be able to tell you if it's supposed to be a president or a Prime Minister. I'm sure many people in Switzerland probably won't either.
But at the same time, it works and it's quite effective. And one of the things that enables Switzerland surviving is the fact that the country is surrounded by a whole bunch of freaking high mountains. In World War II, even the Nazis ended up not even trying to invade it.
But then on the other side, the Eurasian steppes are the most offense favoring terrain possible. It's just flat land and you could just have soldiers or tanks or whatever, just plow through it. And if you're the attacker, arguably you have the advantage because you can pick the place in time. And those are the places where tens of millions of people die in great wars over the course of many centuries in history.
And so the question is sort of how do we make the world metaphorically structurally more like Switzerland and less like the Eurasian steppes? And I would argue that's good for safety and it's good for democracy and good for freedom.
Liron: So you're basically saying, how do we somehow decentralize the world into defended regions like Switzerland?
Vitalik: Right, exactly. And I think both physically and also metaphorically. If you talk about things like cyber defense and even biodefense, those are things where you can't quite think of it as a region by region thing. But it is something where following principles of locality can be a very good thing. But locality can be physical locality and digital locality. We should be mindful of both.
d/acc Examples and Biodefense Work
Liron: So my current analysis of d/acc is that, so first of all, I respect the high ground maneuver. I don't think anybody can disagree with that. This is a nice ideal. I like the ideal. I just think that it's probably going to be impractical. I don't actually expect us to successfully do d/acc.
Uh, but before I hit you with my objections, let's look at the upside. So in the last 18 months since you published the first d/acc post, what is your favorite example of recent research in the d/acc direction or recent d/acc engineering success stories? What's going well with d/acc?
Vitalik: So, I mean, first of all, I think all basically everything that Balvi is funding is squarely within the bio side of the d/acc space. So for example, one of the companies that Balvi is funding is doing COVID and other kinds of disease testing in a way that is airborne.
So basically imagine version one of this, as you imagine a device where you just breathe into it and it just tells you after a minute, with PCR quality, do you have any one of these diseases or not. And then version two is like a box that just sits passively on a table and then after a few minutes, if anyone has a disease, it beeps.
The reason why this is valuable is that if you look at why China was able to fight against COVID so effectively. A big part of it was basically PCR testing everyone every 48 to 72 hours was de facto mandatory. And so they were able to catch lots of cases very early. And then even if outbreaks started, they were able to stop them in a lot of cases.
And you know, there was definitely a lot of darker sides to that whole story. The Shanghai lockdowns that lasted for months. Basically lot of people in their apartments, stuck for months were extremely traumatizing to a lot of people. But a lot of things, and especially this sort of rapid testing, was very effective and China was able to hold out all the way until the Omicron era.
And so if you imagine that kind of technology, but without requiring human beings to do anything - PCR quality of testing just like a basic part of anyone's ambient environment. That's the kind of technology that Balvi's funding. And a lot of that particular company, it's actually making some pretty significant amounts of progress. So I'm personally looking forward to them coming out with something good in the next one to two years.
So that's the sort of thing that you would think about on the biodefense kind of d/acc side. But there's also other planks and I think there's also a lot of people doing d/acc related things without calling themselves part of d/acc. And I think there's a lot of bright stuff in computer security. But we can go down those rabbit holes.
Liron: Gotcha. So, I mean, I acknowledge that some technologies are really nice and they're pretty clearly not bad. What you're talking about. That sounds really good. That sounds asymmetrically as you say, toward defense. And similarly, I think a classic example is public key encryption. That seems like a defense promoting. More decentralized islands that have this power now, it seems helpful.
So I agree that we can find some examples that seem like d/acc wins. So my objection is just that when you look at the most powerful tools, they're just not going to have this asymmetrical defense favoring flavor. They're just going to be really powerful general tools. Like vision, can you really say a vision is more defensive or offensive for example?
Vitalik: Yeah, no, it can easily go both ways.
Liron: Right? And the tool in question is superintelligence.
Vitalik: I mean, vision's an interesting example. Because the rise of modern surveillance is one of these really big topics and it's generally believed to be something that's very worrying in a lot of ways. So yeah, there's a huge amount of technologies that have very ambiguous consequences. And I fully agree with that.
Whether offense wins or whether defense wins to me is something that's not binary to me. It's a slider and different technologies push that slider in all kinds of directions. Sometimes they increase capability without really affecting the slider. And but then to me, every marginal unit of movement toward the defensive side is a really valuable and good thing.
Liron: Well when we're talking about super intelligence though, maybe, do you have any specific thoughts on what it looks like to build super intelligence in a way that's d/acc.
Vitalik: The biggest question there is what and what type of super intelligence and what is the context in which it gets released into the world. So a very not d/acc version of super intelligence is where if you have this one very powerful, dominating thing that's very far ahead of everyone, and so basically no one else is even able to put up any kind of resistance against it and essentially just is able to take over the whole world.
And if we do that, then effectively there is this - we have one shot and either it's aligned with us, in which case it creates utopia and it's a benevolent dictator, or it ends up being unaligned, in which case we're screwed.
I think realistically the d/acc version of super intelligence is something more pluralistic, where basically intelligence improves over time in a way that many different people around the world have access to it. And essentially we're able to continue maintaining a world with roughly at least a similar type of structure to what we have today. Where there's multiple actors. There isn't something with any one queer level of dominance and effectively any gains that one makes end up being transparent enough that they get adopted by others.
And that would be the decentralized kind of version.
Liron: What if we say that policies that try to target a world where AI super intelligence is decentralized and grows slowly and manageably in capabilities - that would be like a d/acc approach.
Vitalik: I'd say so. Policies can have all kinds of different consequences. Because I mean, I've actually, even in favor of some AI decelerationist policies. I think that having some of these international compute treaties that do things like creating globally controllable off switches - that seems like a sensible tool to have in the toolbox.
If it can be done in a way where if the button is pressed, it kind of evenly slows down everyone as opposed to basically it being a tool where it's basically a tool for control by one party, because if that's what it is, that just creates maximum incentives for everyone else to resist it.
So I'm in favor of some decel things as well. I think d/acc things, in the sense of — I don't think I'm in favor of any policy that tries to accelerate AI. I think I'm in favor of policies that ensure that AI developments that are distributed more, I think I would agree.
And then of course there's the caveat that we haven't really yet talked about the defensive side yet. And I think there's interesting things that you can do there. The concept of just having hardware that's able to make credible claims about what kind of code is running on it. This is one of those famous things that Eliezer talks a lot about how you can have more cooperation if agents are able to prove things about what their source code is.
And so I think more things like that is valuable and I think it's also very valuable in sub intelligent cases. So where if for biosafety reasons, we end up having lots of sensors to do early detection. It would be very nice to have stronger assurances that these sensors are not simultaneously spying on everyone and are not able to just push data about everyone's private lives into some kind of global panopticon.
Challenges and Criticisms of the d/acc Approach
Liron: I wanna just pause on the fact that you are saying that you acknowledge the value of these kind of deceleration-ist global coordination policies. Because I think we're both on the same page that it's like, we don't like this. This doesn't make us happy that to propose this kind of stuff. It's like we both like building tech as fast as possible.
Vitalik: Absolutely. There are a lot of approaches to the decelerationist thing that strike me as being much worse than others. And I think the version of this stuff that is the worst is the version that slows things down by de facto putting a very concentrated authority in charge of what can happen and what can't.
Because if you have an organization that controls that lever, then effectively that's an organization that can control the world. And to me that's both a bad outcome just from the perspective of my preferences and values of what kind of political structure I want to have. And I think it's also something that will contribute to derailing the entire project from succeeding.
And so if we can find deceleration things that actually credibly decelerate everyone, including militaries, then that's the kind of pause button I'm much more on board with than a pause button where people can exempt themselves by proving to political power brokers that they're important enough.
Liron: What do you think of MIRI's proposal where it's not like there's one actor controlling all the AIs, but it's more like a shared international space? Huge international data center space where every country can come on and work together, but it's just all being monitored. Everybody can monitor everybody and that way if things go crazy, they can all contribute to vote on pressing the off switch, something like that.
Vitalik: I mean, I think things like this are interesting. I mean obviously, I have a strong prior that things of that type that are created by the kinds of people that have built present day technologies will basically end up looking like a very big trust-me system that just ends up super concentrating authority.
But basically my willingness to be warm to that kind of thing is in proportion to the extent to which it's not like that. So if you have a data center where literally I, as a private individual would be able to mail in my camera, and the camera would actually be pointed at the hardware and I would be able to see it running. Things like that would improve my confidence in the thing.
Liron: To recap here, you're showing a lot of nuance. For you it’s no problem, but I think for most people in the discourse, it is difficult. So just to recap here, the nuance - d/acc isn't as simple as just saying, rah rah, decentralization, rah rah, libertarianism. There's kind of a paradox here.
Accelerationists and libertarians, they think of themselves as pro tech for everybody, pro-freedom, pro decentralization. But if you do that, you've pointed out before, the natural equilibrium of AI isn't just that everybody can have their own AI and everybody plays together nicely. The natural equilibrium very likely is that somebody's AI just starts getting ahead and seizing more resources. And then you've broken the decentralized model, even though you let everybody be decentralized and free. But then you break the model if and when one AI takes the lead.
Vitalik: Right. Exactly. I think one of the big both non-existential and leading to existential risks that I'm worried about is basically this historical trend that we've had where the agents that make up humanity - if they make progress, that progress diffuses by default, and there isn't really a way to opt out of diffusion.
That trend breaking is one of the things that worries me. And so I looked at the AI 2027 story, for example, and one of the conclusions that I came away with is basically that the extent to which I am worried and the extent to which that feels risky is proportional to the extent to which Agent 5 actually is very far ahead of everyone else.
Basically this is actually one of those reasons where this anti open source direction that's popular among some AI safety people, actually really worries me. Basically the conclusion there is, well actually, if your ability to be defended from one AI going crazy is based on actually having other AIs that are powerful enough that they can do something about it and that they're able to absorb any new gains in capabilities growth, then you want policy that pushes the level of diffusion up.
And so basically silos bad. Chinese spies good, that's an important one. Open source good. That's an important one. Basically capabilities being accessible to more groups of people good.
So that's one of the ways in which I would depart from some of the standard ideas. And so I think actually this is one of the reasons why the hardware side is the place where I really believe in where I believe it's most practical and to concentrate the regulation, because that's fully compatible with diffusion of ideas if the thing that you're doing is you're ensuring that execution of them, especially at the largest scales, is something that we have some control over.
Liron: If I'm understanding correctly, and also from reading your other stuff, your mainline non-doom scenario of how AI progress can go well is if there's no really fast foom that overwhelms everybody. As you said, there's diffusion, there's open source, so you're basically pro open source, and we're just really hoping for things to just progress at a reasonable pace that we can handle and not get overwhelmed.
I feel like that's your mainline non doom scenario.
Vitalik: Right. Basically where the world becomes reasonably pluralistic, in terms of centers of power and agency and that keeps going as a stable pattern, basically. All the way up until we reach the technological ceiling.
Liron: What about foom though? Because from my perspective, I actually think that foom at some point is very likely.
Vitalik: Yeah, and I think scenarios that are more foomy are definitely scenarios where this kind of thing becomes less likely to happen. I think at some point, if we want human beings to be part of the story, then at some point this is one of the reasons why I believe in things like brain computer interfaces and then eventually things like uploading, is that's a way in which we can continue to be a leading part of the story and we can actually continue to have agency in the world, even as the timescales even shrink. But it's definitely a far from guaranteed scenario.
Liron: Yeah I did see that in your posts. So basically you think that it would be ideal to merge with AI. Correct.
Vitalik: Correct.
Liron: But people have pointed out that that may not be plausible. I think Rob Wiblin in your podcast was saying that, isn't that like saying that maybe birds can merge with airplanes, but what does that mean for a bird to be merged with an airplane? Isn't it really just the airplane?
Vitalik: Right. It's an interesting analogy. It's like, what if birds had much more agency back in the year 1500. And then they realized that humans are gonna want to have these things at some point. And so they somehow had agency over the process of evolution and they basically decided like, hey, let's actually turn ourselves into air horses and then let's actually figure out how to give ourselves jet engines and then we can actually be competitive.
And then, I mean, if that succeeded, then we would have a world where birds have quite a bit of agency today.
Liron: Yeah, but in that hypothetical, it feels to me like we just figured out how to build an adequate plane and the part where we stuck the bird on. I don't know. It's kind of hard to make the analogy. I mean, if we, okay, if, forget the analogy. If we're just talking about human brains and AIs, it's like, I just don't know what the human is doing in real time. It's because it seems like, because the AI is so much more powerful, so it's like, is the human just evaluating, like, what does that mean to merge?
I'm already, my neurons are already so much weaker than the AI that why not just say, I'm not merged. I'm just sitting here with my terminal and the AI is asking me questions. Isn't that as good as merging?
Vitalik: I mean, in a lot of ways, especially at early to mid stages, that is the type of interaction that we're going to see. I think the long term technological ceiling future of this does involve uploading and it does involve actual human minds moving to digital substrate, at which point we'll actually be able to think at the levels that will let us grapple with the kinds of ideas that are at the technological ceiling.
And I think some people will make that choice and some people won't. But if we're talking about the far future, then to me, basically there's two options where one is spooky and two is total disempowerment. That basically just because there is going to be someone who's thinking at many times human speed and I'm sure we'll get there, but also it is something that I'll have to go through quite a bit of iterations in order to get to.
Liron: You recently spearheaded a big and successful merge of the proof of stake chain into Ethereum. Is it possible that you're getting too cocky about merging?
Vitalik: I mean, possible, right? But also there are good scenarios that don't flow through the kind of path that I described. There is this scenario where we get an aligned ASI and then eventually the ASI decides like, hey, human beings value agency. And so let me figure out how to give human beings some more agency in this world at the technological ceiling that I've figured out how to go through the more dangerous stages and get everyone to.
So maybe we can unpack here to what extent some of the sort of spookier things about this, about these kinds of scenarios are specifically properties of the d/acc world and which of them are properties of any sufficiently technologically advanced world in general.
Because to me there's basically two options. One of them is that something very spooky happens to humans, which ourselves getting uploaded as one of those examples. And the other option is total disempowerment. And then there's of course a third option, which is perma pause. But to me, perma pause just becomes more and more unstable over time. And I think that ends up being politically spooky in a lot of ways that most people in the world what ends up really disliking.
And so there is a fundamental question of you get into the really far technological future, which of those three bullets do you bite? And there's definitely a lot of people whom I respect who basically bite the bullet of human disempowerment and basically say, well, let's, okay, let's will have a world where essentially we're retired and the AIs are thinking a millions of times faster than us, and they're doing all of the important stuff. And life basically just becomes a game for us.
And I respect the kind of thinking that leads to that kind of answer, but I feel like there's a fundamental human drive for not just enjoying the world, but actually being a productive part of contributing to it. And so if you take that to the logical conclusion, then I think the second one of those three becomes the most palatable option.
Liron: Which was the second again.
Vitalik: So option one is humans retire and become cows grazing on the fields and AIs that are a million times faster do the important stuff. Option two is basically humans upload and we turn into something which feels to our minds like a continuous psychological experience of being one mind all the way from where we are now to something that can think at levels that are at the technological ceiling.
But still, the end stage of that feels as radically different compared to where we are now as where you are now felt to where you were when you were two years old. And then the third option is perma pause.
Liron: I mean the second option seems like the ideal, like Eliezer's fun theory sequence.
Vitalik: I mean, I think there's a version of the fun theory sequence where basically we are cows grazing on the fields and we're basically put inside of games and those games give us the feeling of things like progression and meaning and doing valuable things. But at the same time, the AIs can totally run circles around us. But I think ultimately the second one is probably still more compatible if it can happen.
The Good Outcome and AI Company Incentives
Liron: Now that we're talking about the good outcome, I think you've made this observation that the AI companies aren't really being clear about what they're imagining as the good outcome. Like Sam Altman is talking about things changing surprisingly little. That's his recent line. Things are gonna change so much, but you're gonna be surprised how little they're changing. What do you think about that?
Vitalik: I think the AI companies, everything that they do and say at this point is very heavily guided by all kinds of political incentives. And there was a period where I think they were guided by these very idealistic causes. And maybe to some extent they still pretend to be, but that basically died sometime between 2023 and now.
Regardless of what you actually believe, it's definitely better to say for marketing that you are bringing into the world a future that is fundamentally very relatable and recognizable where the only thing that's different is that we have tools that are better, that do unpleasant stuff for us.
And then also that's the story that's optimized for the general public and the media. And then of course, this is the thing that really worries me, the story that's optimized for the government is basically we're gonna make your military great. Military use of this stuff. That really worries me.
That's something where there's been this very rapid cultural shift where a few years ago, everyone was adamant that what they were building tools for peace and not for war. And Anthropic had explicit covenants to that regard. I forget, I think maybe OpenAI did too, but then they've all just explicitly dropped them and they've basically embraced being parts of the national security state of which country that they're in. And the use of these things as weapons is something that is increasingly part of the world already, and it'll go further.
Liron: When you mentioned the military, I was gonna ask, do you think that your mainline doom scenario has to do with the military building AI?
Vitalik: It's definitely a large part of my probability mass. Basically because I mean, if we get into a serious war then the thing that continues to be the trend in serious wars is basically that often people come in with kind of high minded norms about yes, we're in a war, but even still there are fundamental principles that we're not going to violate.
And so in World War I the UK, one of those was not doing conscription because they value freedom. Then in World War II at the beginning there wasn't that much of just full on bombing of cities at the beginning. But then as war progresses and as people get more and more desperate, the ladder just keeps on ratcheting up.
And anything that gets done at the beginning to say, well, okay, there is a war, but we're not gonna use this to rush to ASI - eventually, once it gets into year four, year five, more and more people just say like, screw it. We're gonna do things that get closer and closer to rushing to ASI with less and less safety measures. And you have to do it because the other guys are doing it, and then you get doom.
The Intractability Question for AI Alignment
Liron: Okay, I think I'm putting my finger on the crux of disagreement between me and you because there's a reason why my P(Doom) is 50% and yours is 12%. I think yours is too low, and I think this is why.
This goes back to a comment you made in your response to AI 2027. You were saying making the world less vulnerable is possible. So you seem optimistic about laying down protections basically so that the AI won't foom too fast and run over them.
So for example, you said, you're optimistic that the end game of cybersecurity will actually be that we have well-defended systems. Is that the kind of thing that gives you confidence that P(Doom) is significantly under 50%?
Vitalik: It's one of the things. I think I also just have very big probability mass for just total unknown unknowns - important variables come out that none of us have been thinking about.
But maybe just to lay out that side of the scenario - basically the thing that I argue is that in the AI 2027 scenario, they argue that AI is gonna get to this superintelligent, crazy level. And by 2029, we're going to have things like cure for cancer, cure for aging, nanotech, early Dyson swarms, automated robot economy.
And the thing that I notice is basically that they're giving all of these technologies to the attacker, but they're giving none of them to the rest of humanity to try to use in defense. And then in that story, we get wiped out by a pandemic. But we actually know pandemics are pretty dumb agents. And we are currently orders of magnitude away from the optimum in terms of defending against pandemics.
Even sort of normie technologies, some of the type that Balvi has funded. My estimate is that if those get adopted, then we're able to get a 10 to 20x reduction in airborne spread. And that if you get the upper end of that range, then basically even measles becomes unviable as a disease.
Liron: Well, I mean with AI 2027, and you know, I'm not as concerned with defending their scenario. It's more like, I think the larger crux between me and you is that yeah, sure, okay, we can defend against this particular pandemic vector or whatever. But I still just think that we're going to be in a plant situation. We're just not going to be reacting on the timescale with the intelligence level that the AI is going to be presenting us with.
Vitalik: So, okay, so I do think that there will come a time when these new waves of technology are happening really fast. And right now, these things are happening on the order of years to decades. And then at some point that scale is gonna be months, and then it's gonna be weeks, and then it's going to be days.
I don't think that rules out any of my scenarios. Because effectively as AI creates new scenarios that we have to worry about then humans plus AI responding to scenarios, the rate at which that's going to happen is also going to continue to increase.
And so I do think that we as humans aided by powerful technologies can be quite effective if we really have to be. And this is a world that is going to be chaotic, and I definitely do not expect that we will come out clean, but I also do not expect that it'll be clean for the attacker either.
Liron: Do you imagine that we're even going to stay in the loop of these kind of decision processes? You don't think AI will just be operating independently pretty soon? Assuming sufficient intelligence.
Vitalik: Right, well, this gets back to the topic that we had about an hour and a half ago where we're talking about AGI and ASI. It's the same question as what are our timelines? And if you were saying that AI operating independently on timelines that are too fast for human beings to make an impact, that's basically saying the why ASI.
Liron: And I think it's very likely ASI is going to come in the next couple decades.
Vitalik: Right. And I think my story definitely becomes less likely in the worlds where ASI comes sooner and more likely in the worlds where we basically have more iterations of all of this happening in the pre ASI regime, and potentially we can get close to the technological ceiling in a pre ASI regime.
But I think in the world where timelines are shorter, then I would argue that the thing that we need is some kind of pause. And but the thing that we want to do is figure out how to get that kind of pause in a way that is as not power concentrating and as viable as possible.
And but then if we succeed at that, then what we've done is we've basically put into the world where ASI timelines are longer again. And then the question is, well, okay, we're delaying, but then what is the thing that we're delaying for? And then I argue that the thing that's valuable to do is basically do d/acc style things anyway.
Liron: Well, I definitely see the appeal of your mainline non-doom scenario. It's like, so we get the pleasure of watching AI get smarter and cooler and solve all these problems for us, and keep it on the edge where it hasn't overrun us or gone independently and taken away all our power to intervene in these kind of loops. I definitely feel you that I would love to live through the scenario where we somehow hang on and don't die and stay in the loop. It just feels like wishful thinking to allow myself to think that that's going to be the case.
Vitalik: And I can always say there's some probability that you're totally right.
Liron: There's this question that I call the intractability question, which is we maybe need to come to grips with looking at the difficulty of super intelligent AI alignment on a 20 year timeframe and be like, you know what, this may be an intractable problem. I feel like this rarely gets discussed.
Vitalik: It's very possible that super intelligent AI alignment is intractable. I think if that ends up being true, then I would still feel more confident in any ecosystem where there isn't one agent dominating everything. But there is some collection of different agents that are kind of aligned based on hopefully various different methodologies centered around various different groups of people - that still feels safer to me, even if it ends up being the case that all of these super intelligences are acting on timescales where we as humans are just watching things whizzing past us and we can't contribute.
Liron: So my threat model here, which is Eliezer's and MIRIs, is the “if anyone builds it, everyone dies.” We're like playing shuffleboard getting closer and closer to the edge where we get zero points. And all I can think to do is just not get closer to the edge. I don't think we have a plan for the edge at all. I feel like what you just described is you're trying to describe a plan for the edge, and I'm like, I think we should just back away from the edge.
Vitalik: And I think I would argue that the rest i think about this is that if you had to criticize any of the plan for the edge ideas, whether it's mine or whether it's the e/acc or whether it's anyone's, is that they're kind of technologically naive. But I would argue that the back away from the edge approach is kind of politically naive in the sense that we're basically talking about an n party prisoner's dilemma and deep levels of political and cooperation in the context of a reality where the dominant theme of 2025 seems to be that more and more crazy dictators have discovered that you can just bomb people and then they're starting to just bomb people.
Liron: It sounds like we both agree that the nature of the problem is that we're stuck between a rock and a hard place. And the only difference is that you're saying like, well, obviously we can't go to the rock. And I'm like, well, obviously we can't go to the hard place. We both see something as impossible, but it's like, well, we just have to talk about which one's less impossible.
Vitalik: Right. I agree. And I think there is value in trying to do the right kind of both. I think the thing that I think we should try to avoid is doing the wrong kind of both.
So on the technology side, the things that I'm against are basically making capabilities progress happen even faster. I would be against that, even if the even faster capabilities progress is fully open source.
Liron: But if you're against making it happen faster, doesn't that imply that you likely just want it to happen slower?
Vitalik: I do, yeah.
Liron: All right. Same here. Same here.
Vitalik: And then on the political side, I think the thing that would want to really avoid is these visions of safety that are predicated on some kind of centralized approach that basically involves the good guys winning the game first.
One of the things that really worried me is the whole Leopold Aschenbrenner situational awareness approach where basically says like, look, if we want to be safe, then we have to win the game. And then basically accelerate faster than everyone else. And then basically present China with an argument that look, we already won and now you have to sign onto this where all computers are governed by us, and we get to choose what all the limits are.
And it's like, well, okay, so you're basically saying that you want China to sign onto a protocol that limits how many boxes they can run. What do we know about how willing they are to do that?
And I think these kinds of safety by one small group winning approaches, safety by saying, we're going to self-select as the elite that is going to shepherd humanity through the transition safely. And we're going to win and then you should trust us. That sort of thing to me is possibly as dangerous as the naive, unbridled acceleration. In large part because the first step in the plan is acceleration, and then step two and step three are safety.
So if we figure out ways to make AI progress happen at least less likely to happen crazy quickly and at the same time make sure that diffusion continues to happen and at the same time do what we can to keep the world more cooperative, and at the same time do more to keep the world more well defended against near term and midterm threats that are going to come up, then that's the best path to improving things.
AI Companies: OpenAI, Anthropic, and Corporate Incentives
Liron: So getting back to the intractability thing though. So from my perspective, if we were a sane civilization, we would have a way to step back and evaluate how tractable or intractable the problem seems.
This first occurred to me in 2023 when I was just on Twitter, and I saw Jan Leike, who was leading safety at OpenAI at the time, before he resigned and left about a year later. But he was leading safety at OpenAI, and I remember he tweeted something that was just optimistic and positive. Like, oh, we're making some good progress on the safety team. I'm optimistic, it's a hard problem, but we'll eventually solve it.
And my reaction to that was like, well, wait a minute. Whose job is it to point out that the problem is intractable? If it was intractable? I'm not even saying it is, but if it was, whose job would it be to point out, Hey, this is an intractable problem?
Because it seems like the guy who's in that seat is just assuming that he needs to plow forward and do his best. He's not stepping back and meta evaluating whether to sound the alarm.
Vitalik: Yeah, and they don't have incentives to. These are corporations where basically the entire investment pitch involves the probability that these companies will dominate a big part of the world economy as a result of the AGI and ASI boom.
And in eras past, you would have strong publicly funded academia that has also unaligned incentives, but at least a different vector of unaligned incentives. And ideally it would be independent from both the tech world and the military world. And it would be able to say and make these arguments, but right now, that's not super strong.
And then also OpenAI, of course. At the beginning had this idealistic vision that it would have this nonprofit governance and then it would have this very strong alignment team internally. But then as we saw, basically the company ended up giving up openness for safety and then giving up safety for winning the race.
Liron: Speaking of OpenAI, I wanted to ask you about this situation, on the topic of policy and on the topic of whose job is it to say it's intractable. So I remember during the OpenAI board coup, Helen Toner, there was that famous drama where she observed, ensuring AGI benefits humanity, OpenAI's charter, that might be consistent with OpenAI collapsing, rather than keeping Sam as CEO. So she was kind of in that seat of being like, “Hey, this isn't going to work out. We shouldn't just go do our best. We should potentially shut it down.”
Vitalik: I agree. And it's a common pattern. I forget where this was from, but the standard joke is like, I want to save the world and I want to be the one doing it. And organizations and entities of all kinds, whether for-profit or non-profit or government or charity or whatever, they often are very averse to the idea that the best way to accomplish their mission or goal might just be to stop working on what they're working on and dissolve.
Liron: I want to ask you about Anthropic and Dario Amodei. Anthropic, there's obviously a lot to like, I mean, the team is obviously rock stars, lots of great people, and I know that they're conflicted. I know that when I personally with a handful of people from Pause AI, when we protested outside their office, I know that they felt pangs of guilt.
Clip: Anthropic AI - reckless! Dario Amodei - reckless!
Liron: They took the protest seriously. And they're good people. So I can empathize with where they're coming from, even though my position is that they should just quit.
Anthropic in one sense is the best actor among the AI companies because they truly feel that safety is important. And P(Doom) is in the same zone, more than 10%.
Now, having said all those good things about Anthropic, there's also an argument that they're actually doing the most damage because from my perspective, they are tractability-washing the problem.
Remember, I'm saying it's so important to say this might be intractable, but when Anthropic is saying, yeah, there's a big chance of catastrophe, but we are just going to do our best. It's like, no, wait, wait, stand back. Don't do your best yet - first evaluate if it's tractable. And I feel like Anthropic is the one that's most guilty of tractability washing.
Vitalik: I think very plausible. From an institution design perspective, I think if you gave me a company whose goal is to both advanced progress and ensure safety, then one of the first things that would pop into my head is basically that hey, maybe we want to really ensure the independence of the safety side so it doesn't feel the need to have opinions that are compliant with what the progress side wants.
And then if you take that logic to its conclusion, then basically what you get is that the company should just take a third or a half of its treasury and just plump it into a totally independent non-profit whose job is to critique both the remaining progress division of that company and all the other companies from the outside.
And this is clearly not the sort of thing that they're doing. And this is the sort of thing that they're moving away from. I mean, as Robin Hanson keeps saying, human beings are great at rationalization.
And then on the other hand, being brave in the context of an organization and just saying like, let's we want to do this radical thing that a bunch of people will think it's crazy. That's hard. But I hope that people in these companies do get the courage and try to actually do something like that.
Liron: They tried to do that at OpenAI with the 20% of the resources.
Vitalik: Right. But that's a team inside of a company. And then I mean, theoretically there's a nonprofit, it has a mission, but then we just saw how unfortunately, as much as I really respect the governance experiment of making hybrid structures that are both profit making and that have some kind of social mission that actually has teeth, I think that particular experiment definitely got worn down over time. And I think it lasted a decade and at this point I just model it as a profit making entity. And I think the main social value of basically all of the nonprofit parts of the governance seems to be that it puts uncertainty into the corporate structure that scares away investors, that reduces the amount of capital that they have access to. And that's a very important contribution to AI safety in itself, but even so, it's limited.
Race to the Top vs Race to the Bottom
Liron: So from my perspective, we're locked into this dynamic that's kind of like a race to the bottom, as you say. They're just capitalist actors. But I want to ask you about Dario Amodei's race to the top. Because he's basically saying, “Hey, we're gonna do a race to the top. We are going to make a company that people want to join” - this is Dario's words - “a company that people want to join that engages in practices that people think are reasonable while managing to maintain its position in the ecosystem. If you can do that, people will copy it.” What are your thoughts?
Vitalik: This feels like the sort of thing that can work well in some eras, but that breaks down in more chaotic eras. And I think we are gonna be in a more chaotic era for the next couple of decades.
So I think it's very admirable that he has that sentiment. I think it's great that Anthropic is doing that as opposed to doing what OpenAI is doing or what most of the other companies are doing. But is that alone going to be sufficient? Again, I don't think so, just because I think the human capability for rationalization is crazy powerful.
Human beings are very strongly influenced by social factors and incentives and motivations that correlate with the thing you're in becoming bigger in both monetary and non-monetary ways. And so it's true for nonprofit things too.
And if you want to be sustainable, you have to come up with a way to actually give yourself incentives that are aligned with yourself doing the right thing. And I haven't seen OpenAI doing that yet. I think if they do do that, then that would be amazing and that would make me much happier.
Liron: So when I heard Dario's race to the top, I was just asking myself what is the game theory of race to the top? Because I thought it was race to the bottom. Can you really just flip it?
And my conclusion is just that what he's saying does have a kernel of truth. When Anthropic comes out with like, Hey, let's have responsible scaling policies or just ideas they have or let's do some research and poke at our AIs and do mechanistic interpretability. All of that stuff can be part of the game theory equilibrium if it's free or cheap. So essentially Pareto improvements. That's basically what he's pitching.
Vitalik: Well, the way that I would steelman the case is basically that top talent wants to be part of making the world better and not making the world worse. And actual top talent is often very willing to take large salary cuts in order to be part of that kind of thing.
And so if they do that, then that can actually attract top talent. And I actually think Ethereum itself, its success has been to a large part because of that. Ethereum is the thing in the crypto space that really tries hard to stick to the ideals to be decentralized. That values open source, to value security. To value things like censorship resistance, when there is a lot of blockchains that try to cut corners for the sake of speed and enterprise deployments and consumer use cases.
And it does somehow happen that despite Ethereum being one of the more ideological of the bunch, it just keeps succeeding, and I think that effect is probably a big part of the reason why. But the question is are you operating in a regime where that kind of pressure is decisive or are you not?
And I think blockchains are very coordination dependent, and so that can work. I think AI is much more, is less like a community and more like a tool. And so that kind of pressure is probably somewhat lower.
So less optimistic there, but also definitely very glad that people are thinking in that direction.
Liron: I mean, that's a good point. What you're saying about like, well, look, if you want to attract the best talent and the talent has these opinions, then you're getting a win. So you can get paid essentially and better talent when you're racing to the top. So, yeah, I mean, I gotta give Dario credit for having a kernel of truth to what he's saying. And the question is just how much does it fight the race to the bottom? I would argue not much. But anyway.
Discourse Quality and Defending Against Ad Hominem Attacks
Liron: Okay, so last topic for you. We've gone object level. I think it was very interesting to contrast our views. I don't see a huge gap between our views. As I said, the crux is just how plausible we think it is that things will go at a manageable, decentralized, defensive pace. You seem significantly more optimistic, but I think we can - it feels like it's not crazy to think that one of us over time will update toward the other.
Vitalik: I think so.
Liron: Okay, so bringing it home here, I want to just zoom out here, go meta level and talk about the discourse.
You are very much an accelerationist in terms of inventing top tier crypto stuff. Being on the forefront of that, you're clearly a pioneer and yet you really didn't hesitate to point out like, oh yeah, well, if AI is that threatening, we have to consider pause solutions, regulations solutions. So you're really showing a lot of mental flexibility and non-ideological-ness. Just openness in a way that when I go on social media and I look at the discourse there, I just don't really see that.
Vitalik: Yeah, and I think those are ideals that I definitely consciously strive for. I think they're virtues that I've recognized as virtues. Definitely since I was reading the Sequences and Scott and also other things as well. And it's everything that I was looking at when I was a teenager.
So I think those things are important to me. I think if we want to get into the question of how can we make better discourse norms the norm rather than something that depends on individual people. And I definitely don't think that over time I'm immune to any pressures.
Then I think we do have to look at both the incentives and the intellectual environment, but then also ultimately the social structure. Because in general, people are very averse expressing views and then ultimately to holding views that are offensive to their social circles.
And this is just an incredibly powerful force and a lot of people in AI are specifically located in the Bay Area, are in social circles where people who are either part of AI companies and people who are investing in AI companies and people who are in the tech industry in general and who really believe in these ideological meme-plexes are really at the center.
And when you're in something like that, then having that escape that is definitely hard. And then of course on the other hand, there's definitely an AI safety bubble as well. And that's also something that is very geographically concentrated in one place. It definitely - there definitely is a pattern that I worry about where 10 years ago it was about people who are attracted to ideas, but then now it's becoming more and more of a social cluster that sees itself as a social cluster. And that probably subconsciously biases toward protecting that social cluster's relevance.
And so I actually wonder this interesting question of to what extent is me being sane in the ways that I hopefully am sane as opposed to me being insane, or both of us being insane, which we have to have non-zero probability for, actually just a result of me being a nomad and not being in the Bay Area.
And if more people did that, then would we have more independent thought? And then another big part of it is just being in the crypto space, which is definitely a tech industry. And so you get, it's tech and you have to learn math and you have to really be up to date with the math, but at the same time, it's a different kind of math. And if AI just ends up completely stalling for a century, then that's actually totally fine for us from the perspective of our own interests.
So basically the value of being disconnected in terms of personal interest is something that I wonder if that's something that's positive and that we should find ways to try to benefit from more.
Liron: This is how I see AI X-risk discourse right now. I think that in the larger population, the average Americans, they're reporting in surveys, you get results like, I don't know the exact number, but it's something like, oh yeah, 70% of people think this is a moderate to high concern that AI will be a serious threat and it should be regulated and slowed down. That's kind of the default average American opinion, which to be fair, a lot of people aren't even techno optimist in a way that you and I think that they should be. So they're not even necessarily having a problem for the right reasons from our perspective.
But I still get the sense that the average person is rightly concerned about AI X-risk. But then I go in the tech sphere and I see a lot of ad hominem attacks of AI doomers like myself, people with a high P(Doom).
So let me run some ad hominem attacks by you because I feel like you can diffuse the attacks because you have enough credibility as a tech leader that you can basically shoot down ad hominem attacks from other people in tech.
So for instance, is high P(Doom) a fringe position?
Vitalik: I would say no. I would say high P(Doom) is a position that is held by a lot of people within the AI space. I would think among the public in general, probably the one nuance is that there's a lot of people who are pessimistic in ways of things are going to be medium bad. There's a lot of people whose just general casual vibe is like, oh, the world is going to hell. Things are getting worse and worse. And this could involve things like World War III. It could involve climate change, it could involve misinformation.
And but then those kinds of people - among them, the probability that humanity will literally go extinct because of AI by 2040 is more fringe, but at the same time, they are people who I think still align on a lot of important goals.
And if we talked about some of the issues that I talked about in terms of AI driven power concentration, a lot of them would probably use different words, but sign on to something very similar. And then even any of these individual issues, dealing with misinformation is part of d/acc. And at the same time, dealing with misinformation is just upstream of having a sane discourse that's actually even able to deal with any of these issues as they arise.
So I think there definitely are differences of opinion between the mainstream and people like ourselves. And then there's differences of opinion between people like ourselves and accelerationists. And sometimes all of the different sides have valid points.
Liron: Okay, because today I was on Twitter and I was going back and forth with Grady Booch, who's a well-known software engineer, the inventor of UML, Unified Modeling Language. So definitely a legit guy. But he told me, he said, I posit that yours is the fringe position, but it may seem mainstream to you because you appear to be locked in an echo chamber.
Okay, so he is clearly wrong. We're not in an echo chamber. Maybe he's in an echo chamber. Boo-yah. All right.
Vitalik: I mean, I think there's a kind of fractal effects where there are echo chambers that we're in, and then those echo chambers exist. But then there's also, at the same time, particular beliefs that are actually much more widespread than that.
And so the average person who believes in AI doom probably has never heard of instrumental convergence before. And if you asked them, they would not repeat that theory. And they might even have a different theory, which is super interesting that we might actually need to listen to more. But they would not have that theory.
And so I think bubbles exist, but at the same time, just saying it's a bubble is not some kind of knockdown argument that AI is not worth worrying about.
Liron: So I'll tell you some more ad hominems that I see pretty often, and you can just say yes or no if they're true or not. Okay. Doomers are non builders. They're not in the arena.
Vitalik: I think very false. I think so first of all, a lot of the people that have signed on to worrying about AI risk, including the statements, including people who have become very big at just talking about this, there are some of the forefathers of the modern machine learning movement.
I think also another thing is that the idea that doomers are not builders and that doomers are a well-funded influential group - those are somewhat incompatible because to be well-funded, the funders have to have built something. And I am concerned about AI and of course I've built things in crypto. Jaan Tallinn is concerned about AI and he founded Skype. Then Dustin Moskovitz is very basically a lot of the early people in people who believe in AI risk, whether for the last 20 years or whether for the last two years, are people who have been effective builders in all kinds of places.
Liron: Exactly, yes, I'm glad you brought that up. Okay, how about this? No one who actually understands how AI works is a doomer.
Vitalik: I think very false. I think clearly people like Eliezer have very deep knowledge. I think more than the average accelerationist in the nuances of how AI works and how things like gradient descent work and how things like transformers and chain of thought work.
I think there's people who have spent years and decades researching AI alignment methods, like the alignment teams at all of these AI companies, the kind of people who go out and say like, Hey, P(Doom) is 50%. Even if their final conclusion is, well, I'm working on it anyway. These are people who are inside of these labs and who have done a lot of very deep and nuanced AI interpretability stuff.
So I think the people who are worried in many cases know a lot about how AI works.
Liron: You know, your lifestyle, you seem like you're very well connected in tech and you're always meeting new groups of people. So when I'm asking you these questions about what's fringe, what do people think? It sounds like you're in a good position to answer.
Vitalik: In a lot of cases, definitely far from all.
Liron: Exactly. Okay. So this segment is great. I'm glad we're doing it. I mean that's basically what we do here on Doom Debates. The mission is to just move the Overton window where people are like, oh wow, okay. This is in fact something that respectable people are talking about.
Okay, so next ad hominem attack here. Doom is a narrative that big companies use in a bid for regulatory capture.
Vitalik: I think true to some extent. I think if you look at especially the anti open source kind of meme, I think it's very plausible that a big part of the reason why that meme was so popular and it got adopted so quickly is because there has historically been a strong pro open source sentiment in tech.
And in the absence of safety arguments, there's a strong line that you're working on AI that is running in data centers, that is concentrated within a few particular companies, then you're making something like Facebook except something that has an even more intimate, even more invasive impact into people's lives. And that the only ethical thing to do is to work on open source AI that you can run locally and potentially run on whatever hardware that you can trust.
And it is very convenient that anti open source safety sort of emerged just in time to give people moral license to go and just make a closed thing that effectively is Facebook in terms of data sorting capability but worse and feel righteous that they're doing the right thing about it. So I do worry about that.
But at the same time, that's clearly far from what motivates a huge portion of AI safety people. Jan Tallinn is not someone who is interested in regulatory capture. If you want regulatory capture, then Eliezer and Scott Alexander are not the ones succeeding at it. Sam Altman is.
And then you ask, well, why hasn't Sam Altman gotten on a doom train more? Especially among the intellectuals, it doesn't feel at all like caring about AI risk is correlated with any kind of power mongering among AI companies.
I think some of the sort of pro closed and control and also pro rapid winning the race, which Dario Amodei has said - that stuff is definitely, I think, connected to political incentives and we should be worried about that.
Liron: So I mean, regulatory capture is basically a way for companies to build a moat, but when I think about these AI giants, the ones who have the most benefit from disingenuous fear-mongering, it sounds like they already have a moat in terms of GPUs and capital and talent. Why would they need to go build this other moat?
Vitalik: I mean, in an emerging industry, it's never clear how big your moat is. It's always possible that AI could get commoditized or LLMs could get commoditized. It's very possible that the value accrual layer is not gonna end up being the foundation model layer for various reasons.
And then I mean, there's regulatory capture, and then there's subtler forms of capture, which are basically just your own employees and the public feel good about business models that keep things more closed and concentrated. And to me, that is a form of capture, even though that's more about the meme layer than it is about the government.
So I mean, I think the incentives for them to do these things are non-zero, though at the same time a lot of the people that have these worries are definitely very sincere about them.
Liron: Last couple ad hominems here. Doom is fake. People don't seriously believe doom.
Vitalik: Very, very false. I think there's lots of people in a lot of these rationalist circles who really internalize doom, who have psychological mental breakdowns because of doom, who change professions because of doom, who even change personal lifestyle habits because of doom. And I think do things that are irresponsible, not stopping saving for retirement. I think people should keep saving for retirement. And I even go so far as to say you should not put your entire retirement savings into crypto.
But there are people whose entire pattern of behavior and demeanor and emotional state just shows like for them this is not an act.
Liron: I was gonna ask you about, you know, Tyler Cowen is famously questioning. He's saying if you look at any AI doomer, their investing or their betting behavior belies that they don't really believe it, but you already answered it. You're saying, well, there's a lot of people who are being kind of irresponsible with their life savings.
Vitalik: Absolutely. Yeah.
Liron: Now I personally, I'm not one of those people right now because I have a 50% P(Doom) and I'm still living out the 50% good scenario. So I am saving for retirement. I'm frontloading more of my spending than I otherwise would. I'm indulging a little more in the short term and slightly screwing myself in the long term. I'm changing the weight balance of that a little bit, but it's modest. I'm not doing anything extreme and I've never understood what Tyler Cowen wants me personally to do differently.
Vitalik: I mean, I think when I read some of the posts in which he makes these kinds of arguments a lot of the time, it is about financial and investing behavior. It is true that buying GPU companies, computer hardware companies stocks in 2020, 2021 was an extremely good idea. And from what I understand, it's definitely something that a lot of AI X-risk people did actually effectively capitalize on, though it's also something that a lot of other people have not.
Basically I think he makes arguments like, if you have a non-standard belief that something big and crazy will happen, then there always are market consequences of that big and crazy thing. And you should look for them and you should bet on them. And I think sometimes it's actually not so simple. You can't actually predict necessarily what the value accrual layer is gonna be.
A lot of the time it's in private held companies, a lot of the time the technology just goes in a completely crazy direction a lot of the time. Total value is high, but marginal value is low. And so it doesn't end up being super investible a lot of the time. The incumbents end up successfully adapting and capturing a lot of the market value.
So I think it's definitely hard to do that kind of thing. But at the same time, I guess it is a super valuable thought experiment. And there are situations where people took the market consequences of their beliefs seriously and did big. I think a lot of rationalists did that very well in early COVID. In 2020 actually put a bunch of shorts on things. And when this was back when people were saying like, oh, it was only 10 cases last week and it's 100 this week. What's the big deal? And then they were thinking what's the function that was 10 last week and 100 this week look like? And then they realized they basically had to short the global economy, and they did. And it worked extremely well.
So I think there's merit to his case, his arguments and you should take it seriously, but it's also often not quite that simple.
Liron: Yep. All right, last ad hominem attack. Doom. AI doom. It's just the rapture for nerds. It's filling the religion shaped hole. It's a doomsday cult.
Vitalik: I think that just doesn't feel realistic from the perspective of any kind of inside view of the psychology of these people. The way that I view the psychological journey is basically that they first saw this argument for why AI risk is worth taking seriously. And then they thought about it as a fun, intellectual puzzle and like, Ooh, this is a fun, intellectual puzzle. Then a few years pass and then there's no counterarguments to it. Then a few more years pass, and then there's no counterarguments to it, and then a few more years pass and then they realize like, wait, this belief is not something about the magical lands of Narnia that I just have as a hobby.
This is a belief that makes claims about the world and the preconditions for the claims that it makes about the world are coming. And then wait, I actually have to take this seriously and then wait. Oh my God, this is actually inconvenient for a whole bunch of my other ideological preconceptions.
A lot of the time it's more like being uncomfortably dragged kicking and screaming than it is about enthusiastically embracing a religion. To the extent that there is something psychologically appealing, I would say it's the idea that you can learn something unique and surprising about the world just by doing math and thinking really hard without being pre-embedded in existing social structures. If you had to find something that is appealing about AI risk at a non-rational level, I would say it's that. I would not say it's doom as a consequence specifically.
I think it would've appealed to basically the same people, even if it turned out that the outcome of the reasoning was, say that Christian theocracy is the correct political system. If it turned out that somehow that was the outcome of all of these AI risk arguments, then probably a similar group of people would've bought into them.
So I think it's more about the process than about the conclusion.
Closing Thoughts on Flexibility and High-Quality Discourse
Liron: Great. I'm thrilled that you've said all that stuff because people really need to hear it. These ad hominem attacks have gone too far. We're just two calm, reasonable people evaluating the arguments for doom.
Okay, I think that's a good place to end. So viewers, I think two really good takeaways from this conversation, from everything Vitalik has been saying is first, how to be a flexible, non-ideological thinker weighing the most likely possible futures. I think you can see some insight into how he's processing his P(Doom). He is updated it to 12%. Hopefully he and I can converge sooner rather than later.
But the other thing to observe is how to have high quality discourse. You don't have to do ad hominem attacks. You can respect both sides. You can respect all the positions. You can say what your crux is, say what it would take to change your mind. And I think this was a tour de force. So thank you very much for coming on.
Vitalik: Thank you very much, Liron.
Big thanks to Vitalik for coming on the show. He is the kind of thought leader whose opinion everybody should be weighing into their P(Doom) and their analysis of how to solve the problem. And so it's meaningful that his P(Doom) is at least in the same zone of greater than 10%.
My goal for this show is for more people like Vitalik, more of these really respectable, good thinkers whose opinion matters - more of these people to come on, do debates, let's hear their perspective, debate it against other people's perspective, and try to get closer to the truth before it's too late.
I want to create a phenomenon for this show where it's like that 2023 Center for AI Safety letter where so many people signed it. The analogy would be so many people are coming on Doom Debates that you start to notice, wait, why aren't these last few people signing it? So in the case of that letter, it's like, oh, Yann LeCun. Kind of the odd one out for not signing it when you've got people from Microsoft, OpenAI, Anthropic, Google DeepMind, all signing it, and it's like, oh, okay. Yann LeCun and Mark Zuckerberg didn't sign it, but everybody else signed it. Everybody else who matters.
That's what I'm looking to create with Doom Debates. I think this is an important enough conversation that it's not enough to be tweeting from your own echo chamber. It's not enough to be scheduling a friendly reporter to come interview you on your terms. We need to do one better when it comes to discourse and Vitalik really stepped right up. I mean, you saw, he and I disagree 50% versus 12% P(Doom), but he was totally game. He stepped on up.
I think, and I hope that this is a beginning of a trend where more discourse is happening on Doom Debates or equally productive debate and discourse forms.
—
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates
