DEBATE: Should we ban the development of artificial superintelligence until there’s a scientific consensus that it’ll be done safely and controllably and strong public buy-in?
Arguing FOR banning superintelligence until there’s a scientific consensus that it’ll be done safely and controllably and with strong public buy-in:
👤 Max Tegmark is an MIT professor, bestselling author, and co-founder of the Future of Life Institute whose research has focused on artificial intelligence for the past 8 years.
Arguing AGAINST banning superintelligent AI development:
👤 Dean Ball is a Senior Fellow at the Foundation for American Innovation who served as a Senior Policy Advisor at the White House Office of Science and Technology Policy under President Trump, where he helped craft America’s AI Action Plan.
Two of the leading voices on AI policy engaged in high-quality, high-stakes debate for the benefit of the public!
This is why I got into the podcast game — because I believe debate is an essential tool for humanity to reckon with the creation of superhuman thinking machines.
Timestamps
0:00 - Episode Preview
1:41 - Introducing The Debate
3:38 - Max Tegmark’s Opening Statement
5:20 - Dean Ball’s Opening Statement
9:01 - Designing an “FDA for AI” and Safety Standards
21:10 - Liability, Tail Risk, and Biosecurity
29:11 - Incremental Regulation, Timelines, and AI Capabilities
54:01 - Max’s Nightmare Scenario
57:36 - The Risks of Recursive Self‑Improvement
1:08:24 - What’s Your P(Doom)?™
1:13:42 - National Security, China, and the AI Race
1:32:35 - Closing Statements
1:44:00 - Post‑Debate Recap and Call to Action
Show Notes
Statement on Superintelligence released by Max’s organization, the Future of Life Institute — https://superintelligence-statement.org/
Dean’s reaction to the Statement on Superintelligence — https://x.com/deanwball/status/1980975802570174831
America’s AI Action Plan — https://www.whitehouse.gov/articles/2025/07/white-house-unveils-americas-ai-action-plan/
“A Definition of AGI” by Dan Hendrycks, Max Tegmark, et. al. —https://www.agidefinition.ai/
Max Tegmark’s Twitter — https://x.com/tegmark
Dean Ball’s Twitter — https://x.com/deanwball
Transcript
Episode Preview
Max Tegmark 00:00:00
I would argue that artificial superintelligence is vastly more powerful in terms of the downside than hydrogen bombs would ever be.
If you think of it as actually a new species which is in every way more capable than us, there’s absolutely no guarantee that it’s going to work out great for us. If we treat AI like we treat any other industry, we would then have safety standards. Here are the things you have to demonstrate to the “FDA for AI” or whatever.
Dean Ball 00:00:27
I think the fundamental thing to think about here is really assumptions.
There are many worlds in which humans can thrive amid things that are better than them at various kinds of intellectual tasks. I just have very serious issues with the idea that we’re just going to be able to pass a new regulatory regime, everything’s going to go fine, and there will be no side effects.
And these analogies of FDA to AI are not really very good. It’s not to say that I don’t think we need something like an FDA.
Max 00:00:50
But then I’m confused by why you don’t think we should have the same for AI. So what’s the—
Dean 00:00:54
Let me make an uninterrupted point for a few minutes if you don’t mind.
Ok.
I think that there will be tons of side effects, and I think that we will stave off a lot of wonderful possibilities for the future.
Liron Shapira 00:01:08
Maybe the real crux of disagreement is your mainline scenario. And so let me ask both of you this question. What is your P(Doom)?
Max 00:01:18
If we go ahead and continue having nothing like the FDA for AI? Yeah, I would think it’s definitely —
Dean 00:01:24
I just kind of have this sneaking suspicion that if the models seemed like they were going to pose the risk of overthrowing the US government or anything in that vicinity, I don’t think OpenAI would release that model. Or Anthropic, or Meta, or xAI, or Google. I just don’t think they would.
Introducing The Debate
Liron 00:01:41
Welcome to Doom Debates. Today I’m excited to bring you a debate between two of the world’s leading voices on AI policy. The question at hand: should we ban the development of artificial superintelligence?
The stakes are high. Advances in AI have become the key driver of our economic engine. Artificial intelligence is increasingly facilitating breakthroughs in manufacturing, healthcare, education, even basic science.
The prediction market Metaculus estimates that the first true AGI, the first fully general human-level AI systems, will be achieved by 2033, less than 10 years from now. Many experts believe that milestone will soon be followed by the creation of artificial superintelligence, a system that surpasses the capabilities of the entire human species.
That brings us to our debaters, two of the clearest voices who disagree about how society should approach these developments. On one side of the debate, we have Max Tegmark, an MIT professor who believes we should ban superintelligence development until there’s a consensus that it’ll be done safely and controllably and strong public buy-in.
His research has focused on artificial intelligence for the past eight years. He’s also the co-founder of The Future of Life Institute, a leading organization dedicated to addressing existential risks from AI and other transformative technologies. Max, welcome to Doom Debates.
On the other side of the debate, we have Dean Ball who completely disagrees with banning superintelligence. Dean is a senior fellow at the Foundation for American Innovation, has served as a senior policy advisor at the White House Science and Technology Policy Office under President Trump, where he helped craft America’s AI Action Plan, the central document for US federal AI strategy. Dean, welcome to Doom Debates.
Dean 00:03:36
Thank you so much for having me.
Max Tegmark’s Opening Statement
Liron 00:03:38
Okay, let’s do opening statements. Max, the starting point of our debate today was a dispute over Future of Life Institute’s statement on superintelligence published on October 23rd.
The statement says: “We call for a prohibition on the development of superintelligence not lifted before there is: one, broad scientific consensus that it will be done safely and controllably; and two, strong public buy-in.” So why should we ban superintelligence?
Max 00:04:09
Well, if you negate that statement, then you’re saying that we should be allowed to go ahead and build artificial superintelligence even if there’s no real consensus at all that it can be kept under control or that people even want it.
If we were to say that, we would basically be doing the most spectacular corporate welfare because we don’t do that in any other industries.
Right now there are more regulations on sandwiches than superintelligence in the US. If you want to sell drugs, medicines, cars, airplanes, you always have to demonstrate to the satisfaction of some independent scientists who don’t have a conflict of interest that this is safe enough—that the benefits outweigh the harms.
I’m just saying we should treat superintelligence the same way. Right now, 95% of all Americans in a new poll don’t actually want to race to superintelligence. And most scientists who work on this agree that we have no clue at the moment how to keep something which is so vastly smarter than us under control.
Dean Ball’s Opening Statement
Liron 00:05:19
Okay. And Dean, you oppose the public statement and you don’t share Max’s views on prohibiting superintelligence. Give us your opening statement. Why do you think we shouldn’t ban superintelligence?
Dean 00:05:29
I think the concept of a ban and of superintelligence in general is just quite nebulous. That is the fundamental issue that I have. AI systems that could pose substantial danger to humans are not disallowed by the laws of physics, at the very least.
I think there are really serious questions about how close those things are and how likely we are to build those things in the near future. My guess is, five years ago, if you were to try to describe general superintelligence in a law that a lot of people could agree to—which would be the way that you would affect something like a ban—you would run into problems.
All the things Max referenced are things where we impose those requirements through laws, right? On airplanes and drugs and whatnot. So if you’re going to have a law, you are going to have to define superintelligence in a statute. The problem you will run into there is that you will define it in such a way that you actually end up banning many things that we would want.
There are many ways that you could plausibly define superintelligence that would negate technologies that I think would be quite beneficial to humanity. Imagine an AI system that has largely solved mathematics. It’s solved all the outstanding problems that we have in mathematics. It has advanced certain domains of science—maybe many domains of science—by the famous “century compressed into a decade” or compressed into five years.
It’s accelerating the development of AI research itself. It’s doing that in meaningful ways because one of the areas of science that it knows how to do experiments in is computer science and AI research. It’s a better legal reasoner than you or me or anybody else. It’s better at coding than you or me or anybody else.
I can imagine such a system existing. In fact, my guess is that such a system will exist by roughly 2030 without posing the kinds of risks that Max is worried about. Again, I don’t think those risks are impossible; I just place a lower probability on them.
So I worry that in practice, if you tried to affect such a ban, you end up with: “We’re going to ban N plus GPT N plus two.” So you have GPT-5, and that’s N. There’s GPT-6, which would be allowed. And then GPT-7 would be the thing where we say, “No, we’ve decided that’s too scary, so we’re going to basically ban that.”
Then what happens after that? In order to figure out anything about whether superintelligence is safe or not, you can’t just do that research speculatively. You have to actually build the thing to some extent and put it in a constrained setting to figure out if it’s safe. You have to build at least big parts of it.
Once you’ve done that, it’s like, “Well, okay, but there’s a ban.” So only the specially sanctioned group is allowed to conduct this research. At that point you have a monopoly, perhaps a global governmental cartel of some kind that is developing this. This I also think could be dangerous. And that is, of course, assuming that you were able to get the international cooperation you would need to affect such a ban, which I also doubt.
Designing an “FDA for AI” and Safety Standards
Liron 00:09:01
Okay. Max, Dean raised a few points about the practical difficulties of doing this kind of superintelligence regulation, even going so far back as to define what superintelligence is for the purpose of this ban. How would you respond to that?
Max 00:09:17
I’m afraid that we might disappoint you here, Liron, by agreeing more than you want. You want the fierce debate to clobber each other.
I think it’s actually quite easy to write this law, and I don’t think it requires defining superintelligence at all. Let me explain what I mean by that.
If we treat AI like we treat any other industry that makes powerful tech, we would then have safety standards. There are safety standards for restaurants; before they can open, they have to have someone check the kitchen.
So if you had safety standards for AI, they wouldn’t need to define superintelligence. They would just say that if there’s a system that plausible experts think could cause harm, here are the things you have to demonstrate to the “FDA for AI” or whatever—that this is not going to do.
You might want to demonstrate that it’s not going to teach terrorists how to make bio-weapons. If it’s a very powerful system, you probably put one of the safety standards being: you have to demonstrate that you can keep this under control. If the company selling it can’t convincingly make the case that this thing is not going to cause the overthrow of the US government, then reject. Come back when you can.
I didn’t mention superintelligence here at all. It’s the company’s obligation to demonstrate that they meet the standards.
To take an analogy that might help clarify what I’m talking about here, let’s talk about thalidomide for a little bit. This was the medicine that was given to women in the US to reduce morning sickness and nausea during pregnancy, and it caused over a hundred thousand American babies to be born without arms or legs.
The dumb way to prevent such harm would have been if the FDA had a special rule that “we have a ban on medicines that cause babies to be born without arms and legs.” What if someone comes out with a new medicine now and the arms and legs are fine, but the baby has no kidneys or no brain? That’s not the way to go about it.
The way you instead go about it is you ask the companies to do a clinical trial and provide quantitative evidence of what are all the different side effects that people might not want. Quantify them. How many percent get each? Then quantify the benefits.
You give this to some independent experts who don’t have money on the line—so they can’t work for the companies—who look at the benefits and the harms and they decide: is this a net positive for the American people? Then they approve it. This is how we do regulations in all other areas.
This is how I think it’s quite easy to do also for AI. In summary, you don’t define superintelligence. You just define the harms that society is not okay with. Very broadly, it boils down to demonstrating that the harms are small enough to be acceptable, and then it’s the company’s job to make all the definitions they want, quantify things, and persuade these independent experts. Does that make sense?
Liron 00:12:22
I’m happy to let you guys cross-examine each other pretty freely, and I’ll just step in once in a while.
Dean 00:12:26
Okay, cool. So basically, instead of saying we should ban superintelligence, what you’re saying instead is we should have a kind of licensing regime, a regulatory regime of some kind with respect to Frontier AI systems.
Max 00:12:43
Very much inspired by how we do it for other tech.
Dean 00:12:46
Yeah. So I’d say a couple of things about that. First of all, most preemptive regulatory regimes that I’m aware of don’t generally require you to prove a negative. So you couldn’t prove that the plane won’t crash. The FAA (Federal Aviation Administration) doesn’t require you to prove that.
It requires you to make affirmative statements about really not the plane itself, but many subsystems of the plane. So, the turbines of this jet engine have X, Y, Z chemistry, which conforms to X, Y, Z technical standard, which conforms to... et cetera.
In fact, the way that a lot of times that ends up working is there are layers and layers of regulation. The plane maker has to buy jet engines that are only from people that conform to certain standards. And those standards have to do oftentimes not just with the object-level properties of the component in question, but also things like: how does information flow through the business?
If you’re a turbine manufacturer—if you make turbine blades for jet engines—you are probably subject to implicit and explicit regulations that have to do with risk management inside of your company and who is the designated risk officer and all these sorts of things.
Max 00:14:23
Can I just jump in? I agree with everything you said here. What the companies need to demonstrate in the safety case is the high-level thing the government wants to know. How many flight hours on average do you have until a failure and so on?
The companies can solve that whatever way they want. It’s in the interest of the companies to not use flaky manufacturers, to have good procedures, and have people study crack formation, the physics of it, and so on—and then switch to another alloy if that works better.
It’s the same for medicines. The government doesn’t come in and micromanage, “Oh, this chemical is allowed, this ingredient is not allowed.” Rather, if they have a medicine that seems pretty good—suppose there’s a new antibiotic that seems really good against bronchitis, but it contains lead and aluminum and cyanide—some small doses people will look in the company and be like, “You know, we’re having a hard time demonstrating the safety of lead. Maybe this works even without the lead. Maybe we can swap out this thing.”
So all the innovation is driven by capitalism, by market forces, to come up with the risk bounds that they want to meet.
Nuclear reactors are a great example. What the law actually says there is the company has to make a real quantitative calculation and demonstrate that the risk of a meltdown is less than one in 10,000 years to even get permission to start building it. So the company has free rein to come up with whatever reactor design they want and then they will innovate. Whoever meets those first gets the big bucks.
Dean 00:16:00
I think it’s considerably more complicated. In principle that’s true, but in practice it is considerably more complicated than that. There’s what’s called soft law—guidelines and all these other things—that push people in certain technological directions and away from others.
But that’s actually not even my point. My point is that at the end of the day, you have to be able to make affirmative statements about safety.
The problem I think would be: what are the affirmative statements about safety when you consider that the systems we are talking about are by their very nature extremely general?
Obviously, AI systems today are being used in areas that already have regulatory structures of the kind you’re describing that affect them. This regulator would either have to be so general and have such a broad projection of authority, or it would have to be really, really narrow.
I kind of doubt that it would end up being really narrow in the context of democratic politics. Because the issue that you’ll have is there’s going to be more than just X-risk type issues. Even if you could formulate some statement about existential risk and say, “Okay, you have to prove that the model will not do X, Y, Z that demonstrates catastrophic misalignment,” okay, fine.
But I would say in practice, you’re likely to end up with a situation where, for example, “the model cannot result in job loss.” And then you have—this gets back to an article that I wrote more than a year ago called The Political Economy of AI Regulation—because this is so general and because the technology is going to, in its positive adoption (not like existential risk, just positive adoption), challenge many entrenched economic actors and aspects of the status quo.
When those people, if a regulatory regime of the kind you are describing exists, then those people are going to be able to use it as a cudgel to prevent technological change. That I think we would all agree—well, probably the three people on this discussion would probably agree—is good for the world.
Max 00:18:43
I will push back in a bit on this idea that it’s so hard to get started on this, but I’d love to just give you a chance first to answer a very simple question.
Do you think it’s reasonable to have zero safety standards on AI right now? Do you think it feels reasonable that there should be less regulations on superintelligence than sandwiches now in 2025?
Dean 00:19:07
Well, I certainly think we overregulate sandwiches. For the listener who doesn’t have context, I think what Max is probably referring to is sandwiches served in restaurants, public health regulations, and local public... all sorts of things like this.
It’s true, there are probably ways in which we do overregulate those things and probably many other ways in which we don’t. I would say that generally speaking in America, we succeed when we regulate at the level of the restaurant that serves the sandwich.
The restaurant probably has many computers in it. It probably uses computers in many different ways, including to get the ham and the bread brought to the restaurant. We don’t regulate those computers with respect to their conveyance of ham to the restaurant. We just treat them as general-purpose technologies that can do lots of different things.
Max 00:20:01
But if you go to that sandwich shop and you notice that across the street from it is OpenAI or Anthropic or Google DeepMind or xAI... If they had developed superintelligence this year—which I think is highly unlikely, but suppose they did—then they would be legally allowed to just release it into the world without breaking any law because there are no safety standards they have to meet. Do you feel that that’s at all reasonable?
Liability, Tail Risk, and Biosecurity
Dean 00:20:29
Well, fundamentally, I think you should be able to develop new technology and release it so long as you’re not behaving with reckless disregard or gross negligence.
This is the thing that actually already exists. To say it’s “illegal” would imply it’s a violation of criminal law, which may or may not be true, but certainly it’s a violation of civil law. So if release of that system were to result in physical harm, loss of property, death anywhere...
Max 00:21:10
Yeah. Human extinction.
Dean 00:21:10
Well, human extinction, sure. But again, I’m sort of skeptical that that’s what we’re going to have on Day One. That company is subject to common law liability.
Max 00:21:26
Who do they pay if we’re all extinct?
Dean 00:21:30
So yes, in the tail risk case that we all die, then common law liability does not help you. And in general, it’s true that common law liability is not a great solution for most tail risks to the extent that the damages incurred dwarf the balance sheet of even the largest companies.
If you created a pandemic—someone made a pandemic with your model—and we’ve decided that that was reckless misconduct for which OpenAI bears some form of liability... Well, that’s a lawsuit you can bring against them. But if the damages are a hundred trillion dollars or something, then it’s very unlikely that you’re going to be able to recoup that amount of money from OpenAI, even with all the money that they have. You’ll bankrupt OpenAI and still be not fully compensated for the harms that you suffered.
So it is true that as a general matter, tail risks are one of the classic examples of where public policy outside of reactive liability makes sense. I don’t dispute that.
Now, I think when it comes to the sort of foreseeable tail risks that AI models might pose, the current ones that people talk about are things like catastrophic cyber and bio. I think there’s a lot of things that you can do downstream of that which avoids creating this large-scale regulatory regime.
Max 00:23:04
You talked about extinction too, wouldn’t you say?
Dean 00:23:06
I mean a lot of people do, but there still is not the kind of persuasive evidence for extinction in terms of mechanically how would that work. I just don’t think we’ve seen that to nearly the same extent.
Max 00:23:22
We haven’t seen any extinction yet, of course; by definition, we wouldn’t be talking here today. But I mean, to put it on the risk list...
It’s interesting what you mentioned there about the pandemic example because I think it’s quite relevant. As you know, it’s very controversial right now whether COVID-19 was actually the result of gain-of-function research by the US government and Peter Daszak or not.
But if you just consider that there’s some probability P that Peter Daszak in his research group did create it with help from others... If someone were to sue them for the millions of deficits it cost, it would be pretty meaningless because Peter Daszak doesn’t have that kind of money. The university where he worked doesn’t have that kind of money.
For that reason, the US government has now kind of clamped down on gain-of-function research again and said “No more of this gain-of-function research until we better understand what you’re doing.”
We have biosafety labs level one, level two, level three, level four. So if you’re doing something even that seems less scary than what they did, you have to do it in a special facility. You have to get some pre-approvals.
Contrast that now with digital gain-of-function research. We had Sam Altman on a press conference the other week being so excited about building automated AI researchers. And ultimately, a lot of people are excited about recursive self-improvement, which is very analogous to biological gain-of-function research.
Why should we have regulations on biological gain-of-function research and still be content with having no binding regulations at all on digital gain-of-function research? That makes no sense.
Dean 00:25:08
I’m trying to answer the first question you asked me, which has to do with safety standards. First of all, let me just say, what you said is completely consonant with my assertion that tail risks are not typically contemplated very well by the common law liability system.
With that being said, you can do essentially automated gain-of-function research with the nucleic acid language model today. You can basically simulate the evolutionary process that allows for more virulent viruses or whatever else. We’ve seen the early stages of this from people like the Arc Institute in California. Those are not like ChatGPT-style models, but it’s the same architecture trained on nucleic acid sequences. So we know that’s a thing.
I think as a practical matter though, the issue that you have is that those things are bits, and it’s very, very hard to just purely regulate bits. So what do we do?
Instead of imposing regulations at the layer of the model—which is a really difficult layer of abstraction on which to do it, in the same way that we don’t tend to place regulations at the layer of computers or software, because these things are really important general-purpose technologies (undoubtedly all three of those things have killed lots of people at this point)—we don’t regulate at that layer of abstraction because it’s not a good conceptual unit of account for regulation.
So what do we do? Well, there are all these chokepoints in the physical world. Some of them are labs of certain biosafety level categories three and four, as you said. Some of it is at the layer of nucleic acid synthesis screening. Basically, you have to say, “If you’re going to order the creation of a certain kind of nucleic acid, we are going to, as a matter of policy, require that you screen that against some sort of methodology that allows us to test for whether or not you’re trying to make a pathogen.”
I worked on some of those policies when I was in the Trump administration. So these are all things that we do. Those are safety standards that exist, that are emerging, that are downstream in many ways of advancements in AI. I think the urgency of policies like nucleic acid synthesis screening goes up because of AI.
I think when it comes to safety standards for something at the very general level of a generalist artificial intelligence model, in the long run we will build those. I don’t think anybody in the spectrum is saying that there won’t be standards for safety, security, etc., of large language models.
Max 00:28:02
“Long run”? Because Sam Altman talked about a thousand days to superintelligence, and he might be wrong, but I’m curious if you’re thinking less than three years or more than three years.
Dean 00:28:10
I’m thinking that it will happen gradually over the course of the next decade or maybe even more than that.
Max 00:28:17
Maybe after superintelligence?
Incremental Regulation, Timelines, and AI Capabilities
Dean 00:28:18
Maybe. Yes. But I think the broader point here is that this is traditionally the way that we kind of do things in the United States.
You build a technology. You gain experience in practice with its utility and you sort of diffuse it throughout the economy in this very complicated way. Sometimes there are demonstrated harms. And when there are demonstrated harms, the first thing we do is we deal with that through the liability system.
Again, I would point out that OpenAI, Google, and other companies have common law liability cases—not copyright cases, but “you caused physical harm to me” type of liability cases—against them for chatbots. I think that at least some of those, the companies are likely to lose. They’ll be determined by courts.
Gradually over time, we codify around a set of standards that are shaped by experience that are broadly agreed to by many different actors. And then eventually we codify those in the form of government standards. And eventually that becomes part of an international standards body.
Nobody is disagreeing that that is not a process in which we need to invest substantial time and money and energy.
In fact, I would say the Trump administration should get points in your book because part of the reason that the administration renamed what was called the AI Safety Institute by the Biden administration—they renamed it the Center for AI Standards and Innovation—was to reflect this reality. That the ultimate goal of an organization like that is to produce technical standards.
It takes time to do, but when you actually have these things, they are coherent because they’re formulated through experience. I think the problem is when you try to change the sequencing of that and try to come up with standards without any experience, sitting in the ivory tower or the regulator’s conference room. You have a tendency to create standards that are unrealistic and burdensome.
Max 00:30:38
I completely agree with you, Dean. It’s great that the current administration is taking biosecurity seriously, and I get a sense that they’re also taking AI-assisted hacking more seriously.
I completely agree with you also that this is how things have been done in the past. People invent the car, a bunch of people die, and then gradually you mandate the seatbelt and traffic lights and speed limits to make the product more safe.
But I think it’s important to remember that science has been getting progressively more powerful from ancient times until now. As a result, technology also keeps growing exponentially in its power. At some point, the technology gets powerful enough that this old traditional strategy of just learning from mistakes goes from being a good strategy to being a bad one.
I think it served us well for cars. It served us well for things like fire. We invented fire first. We didn’t regulate it to death. It was later we decided to put fire codes in and have fire extinguishers and fire trucks.
I would argue that nuclear weapons are already above this threshold. We don’t want to just let everybody who wants to buy hydrogen bombs in supermarkets do so, and then “oopsie, that didn’t go so well, we had the nuclear winter now, and 99% of Americans starved to death, let’s regulate.”
For those things, it was very obvious to people that one mistake was one too many. And we already have a bunch of proactive laws about how to deal with hydrogen bombs.
In fact, even despite all your work in the government, you are not allowed to buy your own hydrogen bomb. Even though I would trust you with it—I know you’re a nice guy. I’m not allowed to start doing new plutonium research in my lab at MIT even if I pinky promise that I’m going to be careful, just because one mistake there is viewed by society as one too many and they know I don’t have enough cash to pay the liability if I get sued afterwards.
I would argue that artificial superintelligence is vastly more powerful in terms of the downside than hydrogen bombs would ever be.
There have been some pretty careful calculations recently showing that in the worst-case scenario, only about 99% of all Americans would die and starve to death if there’s a global nuclear war with Russia. So there’s still 3 million who have survived. Whereas if we lose control of artificial superintelligence because somebody sloppily built a new robot species that just kind of took over, it really is game over in a way that nuclear war wouldn’t be.
So the way I see this is not that there’s anything wrong with the traditional wisdom for how to regulate things. I think that’s very appropriate for all tech below a certain risk threshold. We’re very lucky with AI that so many of the great benefits we have are not particularly risky.
AlphaFold, an absolutely superhuman tool for folding proteins, great for drug discovery. Autonomous vehicles, they can save soon, I believe, over a million lives every year from pointless road deaths. There’s so much productivity that can be gained from building controllable AI tools. For those, it’s very feasible to continue having the sort of liability system you’re describing—the traditional way, learn from mistakes and then fix.
It’s only fringe stuff, like in particular artificial superintelligence, which is on the wrong side of that threshold. Right now there’s not much upside, frankly, to sprint to building superintelligence in three years. If we could do it safely in 20 years instead, we would be much better off just doing controllable tools until then.
That’s why it irks me so much that I think people conflate these two things a lot. I’m not saying you do, but a lot of folks I have spoken to on the Hill think that the only choice we have is more AI or less AI, or go forward or stop.
Whereas I see the development as branching into two paths. Either we continue going very aggressively forward to build all these great tools, just insisting that companies demonstrate to us that they are controllable tools. Versus going all in on building superintelligence.
I have to give you a compliment, Dean. I was so pleasantly surprised when I read the Action Plan that it didn’t mention the word “superintelligence” a single time and not AGI. I think that was really wise because it highlights that there’s so much great stuff we can do with AI tools without having to even get into the whole question of superintelligence.
Emergent Capabilities and the Action Plan
Dean 00:35:38
Well, a lot of people contributed to the Action Plan, but thank you very much. I appreciate the sentiment.
Actually, I’d say the reason we didn’t use terms like AGI and superintelligence in the Action Plan, at least from my perspective, is because it’s really hard to know that we’re talking about the same thing.
The Action Plan is, in the public world, very heavily associated with me. But of course, the Action Plan was written by many people within the government. I played a big role in it for sure, but by no means the only one. It was not my unilateral product.
Part of the reason the Action Plan doesn’t talk about superintelligence is because it’s very hard to build consensus among a document that has so many authors as to what we really mean. And this maybe gets into where my concerns are with laws and drafting and exactly what you mean and what you don’t mean.
My Substack does from time to time talk about superintelligence. But I think about a model like GPT-7, this ostensible GPT-7. And I think to myself: if this is a model that advances the frontiers of science in many different domains, solves a lot of different math problems that have flummoxed humans in some cases for centuries, is better at legal reasoning and coding than any human... That doesn’t seem inherently dangerous to me. And also seems like—how is that not superintelligent? It’s not like Bostromian superintelligence, that specific definition.
But I guess my view is that concept of superintelligence was created quite a long time ago in the grand scheme of things, with respect to how fast AI advances. It’s not obvious to me. I think that concept of superintelligence was a really useful way of thinking about advanced AI systems. Bostrom wrote that book, Superintelligence, in 2014, I want to say. Like 11 years ago.
Dario Amodei, CEO of Anthropic, talks about this sometimes with respect to AGI, where he’s like: AGI 10 years ago was “we’re driving to Chicago.” But then once you actually get closer to Chicago, it’s like, “Okay, well what neighborhood are we going to? What street? What’s the house number?”
I think that as we get closer, we need to develop new and more specific abstractions for what we are talking about. In the fullness of time, we will probably have really specific kinds of technical standards and also maybe even statutory requirements for what you can and can’t build with AI.
One thing I want to be very clear about is I’m not saying this needs to be unregulated for all time. In fact, I would say you made the point earlier about how we regulate different medicines with different levels of rigor based on their potential risks. I think we already do that with the frontier language models.
Max 00:48:59
There are no binding regulations right now on anything. There’s no binding regulation preventing people from launching things. As opposed to drugs, where you can’t release any drug in the US until you’ve talked to the FDA about it.
Dean 00:49:22
Well, not quite. This gets into technical definitions where these things matter. You can release, for example, CRISPR-engineered bacteria without consulting the FDA because those are probiotics according to the statute.
A company called Lumina released a CRISPR-engineered bacteria that you’re supposed to brush your teeth with that will ostensibly eliminate cavities. You’re infecting yourself with a bacteria that you’ll be infected with for the rest of your life, and every person you kiss will also be infected with it.
Max 00:50:01
There was a lab in Wisconsin that’s been taking this bird flu strain that kills 95% of humans—but it’s pretty harmless because it’s not airborne—and they’ve been working on trying to make it airborne. So there’s room for improvement there.
But I think we agree on the basic premise here: you can’t open your restaurant or really release a new type of opioid before you’ve been FDA approved.
There are some obvious differences in opinion about things, but there are some things which I think are just more in the confusion category, which is really helpful to clear up. And one of them is around definitions.
Max 00:50:42
Whenever you have any term historically that starts to catch on, every hipster is going to try to latch onto it and have it mean something else.
Alan Turing, when he said in 1951 that if we build machines that are way smarter than us, the default outcome is that they take control... and Irving J. Good talked about superintelligent machines in the sixties and recursive self-improvement.
The definition of superintelligence that was implicit in that was obviously that they could do everything way better than us. Which meant that they could also do better AI research than we could. They could build their own robot factories, make more robots that didn’t need us anymore, and therein lied the risk.
After that, I agree with you right now. There’s just so much hype and BS about this. Mark Zuckerberg talked about superintelligence in a way that almost made it sound like it has something to do with Meta glasses. We have so many different ways people have redefined AGI from the original definition.
I don’t know if you saw the paper that I was involved in that we did with Dan Hendrycks and many others called “Defining Superintelligence.” We welcome people to come up with other actual empirically useful definitions.
But we found with this definition that we’re absolutely not there yet. GPT-4 was 27% of the way to AGI. GPT-5 was 57% of the way there. So there are still a lot of areas where today’s best AI systems really lack long-term memory, for example.
But we’re getting closer. And I think that if we’re thinking about only putting the first FDA-style safety standards on AI in three or four or five years, there is some reasonable chance that that’ll happen only after AGI and maybe even superintelligence have been created. And that would be, I think, a pretty big “oopsie” for humanity.
So I think there are very useful, clear definitions of what we mean. And as I said in the beginning, the way to write a law is not to define superintelligence and ban it, but I think instead ban the outcomes that you don’t want. Something overthrowing the US government, something making bio-weapons for terrorists—which are very easy to define.
As soon as that law is in place, it’s going to spur just massive innovation in the companies. I love comparing pharma companies’ budgets with AI companies’ budgets. The leading AI companies now spend maybe 1%—give or take—on safety. Whereas if you go to Novartis or Pfizer or Moderna, they spend way more than that on their clinical trials and safety because that’s the financial incentive.
They’re in a race to the top; whoever can be the first to come out with a new drug that meets the safety standards makes a ton of money.
People really respect the safety researchers in those companies. They don’t think of them as whiners who slow down the progress. They think of them as people who help them win the race against the other companies to make the big bucks. So I think as soon as we start treating AI companies like we treat companies in other industries, we will incentivize amazing innovation.
Max’s Nightmare Scenario
Liron 00:54:01
Can I also throw out a question? I want to clarify Max’s nightmare scenario here, because I think that’s important to frame.
I think you’re not even just concerned about something like thalidomide where a bunch of people die—hundreds of thousands or whatever it was. You’re concerned about this runaway process where it just becomes too late to regulate forever. Is that fair to say?
Max 00:54:21
Yeah. Can I take a minute and just clarify a little bit for listeners who haven’t thought so much about this?
Many times humanity has been thinking a little too small. Nuclear weapons were science fiction until they suddenly existed. People started going to the moon; it was science fiction until we did it.
From my perspective as a scientist, as an AI researcher: if you think of the brain as a biological computer, then there’s no law of physics saying you can’t make computers dramatically better at all tasks than we are.
A lot of people used to say, “Yeah, but that sounds so hard. It’s probably decades away.” In fact, most professors I know thought that even six years ago, we were probably decades away from making AI that could even master language and basic knowledge at human level, and they were all wrong. It turned out because we already have it now in systems like ChatGPT and 4.5.
So if we consider what would happen if we actually built huge numbers of humanoid robots that were better than us at all jobs—including research, including mining, including building robot factories and so on—we would have built something which is not just a new technology like the printing press. We really built a new species. It can replicate; these robots can build new robots in robot factories and they don’t need us anymore.
It could be great. It could mean that we don’t have to do the dishes anymore and we’re going to live in abundance with them taking care of us. But it’s not guaranteed.
Alan Turing, as I mentioned—the godfather really of our field—said in 1951 that the default outcome he thought was them taking control. We have the two most cited AI professors on the planet, Jeff Hinton and Yoshua Bengio, saying similar things today.
So if you let go of this idea that AI is like the new internet or whatever and you just think of it as actually a new species which is in every way more capable than us, there’s absolutely no guarantee that it’s going to work out great for us.
I’m not just talking about how we obviously couldn’t get a job because they could do it all cheaper. I’m talking even about the fact that we don’t really have any say after that necessarily on what happens on the planet.
A lot of people are working on this—it’s called the control problem or the alignment problem. There’s broad scientific consensus that they’re not solved yet.
So this is the scenario which I think will end up... If we just race as fast as possible to build these superintelligent machines rather than instead focusing on the controllable tools that can cure cancer and do all the other great stuff, and then taking it nice and slow with the things that we don’t yet know how to control.
The Risks of Recursive Self‑Improvement
Dean 00:57:12
So, okay. There’s a lot there that I think I can respond to. First of all, I would start just by pointing out that you correctly observed that lots of people will take terms like superintelligence and redeploy them to mean completely different things. I would submit to you that maybe Sam Altman, when he talks about this existing in three years, is maybe doing a bit of the same thing.
So you can’t talk out of both sides of your mouth. You can’t say, “Well, this happens, but also these people say they’re going to build it tomorrow.” You have to pick one. But the other thing I would say that’s more serious is...
Max 00:57:49
Although I wasn’t talking about... I’m concerned about this regardless of whether it happens in three years or ten. The key thing is just whether—I mean, I think right now we’re closer to figuring out how to build superintelligence than we are figuring out how to control it.
The way we fix that is simply sure no one is allowed to build it before it can be controlled.
Dean 00:58:12
Okay, so let me just respond now. So you have a... basically, the fundamental difference here is that I am saying the technology will be regulated in a wide variety of different ways, which are fundamentally and mostly reactive.
Doesn’t mean that we won’t pass laws. There’s already laws that I’ve supported, which have to do with AI regulation and I don’t think impose substantial burdens on the development.
I would also say the development of this technology is a national security priority, and it seems really hard to... it seems like a really big cost to impose something where we would self-consciously slow ourselves down when others are not doing that. But I’m not even going to... I think that’s a valid point, but that’s not even where I want to go.
The Problem with an “FDA for AI”
Instead, I feel like our crux is: you kind of want this precautionary principle-based, preemptive regulatory regime that would require some group of people to say affirmatively “Yes” before you’re allowed to do something like build superintelligence.
Max 00:59:11
Just like drugs.
Dean 00:59:15
Which is what those are. I think that it’s really hard. I think that there are huge costs associated with a regulatory regime of that kind. I don’t think the government could do it very well.
I think it’s very possible that by doing that in practice—as someone who has observed lots and lots of these regimes play out—I think it’s very possible that by doing that in practice, you would actually end up not just being worse for innovation, but being worse for safety.
Max 00:59:50
Should we close the FDA, or did I misunderstand?
Dean 00:59:53
I mean, I would say the FDA is an organization that is in need of deep, deep and profound reform. Because one of the things that happens when you impose a top-down regulatory regime like this is you lock in all sorts of assumptions that you have about the world.
Dean Ball 01:00:09
So let’s take the FDA as an example.
Max Tegmark 01:00:13
You give this answer, which we were super interested in. It’s very different saying the FDA needs to be reformed, less regulatory capture, et cetera, from saying it should be shut down.
Max 01:00:28
So are you saying that we’re worse off having it than we would if we didn’t have it at all? Or are you arguing simply for a better FDA?
Dean 01:00:34
Let me explain where I’m coming from here. I think these analogies of the FDA to AI are not really very good. It’s not to say that I don’t think we need something like an FDA. It’s that I don’t think we need to test drugs before they go into people.
Max 01:00:51
But then I’m confused by why you don’t think we should have the same for AI. What’s the—
Dean 01:00:56
Let me make an uninterrupted point for a few minutes if you don’t mind. First of all, when it comes to the FDA, we have this huge problem right now with a lot of drugs. What we have realized after several decades of modern science, as opposed to when the FDA was made a hundred years ago, is that diseases are way more complicated.
Dean 01:01:23
They’re not really discrete things. There’s kind of no such thing as cancer and there’s kind of no such thing as Alzheimer’s. They are much more complicated, broader failures of very complex biological systems and circuits.
Dean 01:01:50
The issue that you run into with that is that you need highly personalized treatments for things in order to solve them. Your cancer is different from my cancer. The FDA’s regulatory regime turns out to be entirely unsuited to deal with that because it was based on this industrial-era assumption that diseases manifest themselves the same way over large populations.
Dean 01:02:12
So what you want to do is test it over a big population and get average statistical results as opposed to safety results for one person. What that means is that we have locked into place an entire economic structure for the way that we treat disease that is wrong for modern science.
Dean 01:02:35
It’s really non-obvious how we change it because there are a lot of entrenched interests associated with the current system, including the people who run the clinical trials that we operate at great expense. That would be one of a huge number of examples of problems that can manifest themselves with top-down regulatory regimes of this kind.
Dean 01:02:59
The idea that we need such a thing for AI—I am saying that doing that carries an enormous cost. I don’t think we really have the evidence that the cost is worth paying with respect to AI compared to the many benefits that we get from regulating it much more like we have regulated things like the computer, the internet, and software.
Dean 01:04:14
Those general-purpose technologies have actually worked and grown and made our lives so much better. I think there’s a high burden of proof. Not to say it hasn’t been met, but to say every single top-down regulatory system we have carries with it a similarly high burden of proof.
Dean 01:04:33
If you made a statement that said, “We need to investigate,” or “We need to come up with guarantees that we want AI Labs to be able to make regarding empirical evaluations about their models,” I would be open to it depending on the specifics. But certainly, I would not have had the visceral negative reaction that I did to the “Ban Superintelligence” statement.
Max 01:04:58
Cool. There is a lot of good stuff in there. Let me pick out three things I’d like to respond to: one about regulation, one about perceived vagueness, and then one about national security.
Max 01:05:24
On the first one regarding regulation, it sounds like we’re actually in agreement. Even though you would like to see reforms on the FDA, you would not like restrictions on biotech to be completely eliminated. You would not want people to be able to do BioSafety Level 4 research to make that 95% lethal bird flu airborne, for example, just because it’s cool and people can sue them later.
Max 01:06:05
Whereas for AI, you feel still there should be nothing to prevent companies from deploying things. Maybe you’re open to it in three or four years, but for now, my position is that if someone releases actual true superintelligence that takes over the world, it’s going to be too late to regulate it then.
Max 01:06:33
On the second point regarding vagueness, this is really important. Many people have said to me, “This statement that you put out on superintelligence, why isn’t it written much more concretely so you can make a law out of it?” That was very deliberate. If you look historically in the US when we’ve had new laws passed, like for example a law against child pornography.
Max 01:06:52
You could have pushed back and said, “If someone says they’re against child pornography, that’s too vague. How do you define a child? Is it under 16 or under 18? How do you define pornography?” The law can’t just say, “You know it when you see it.”
Dean 01:06:54
You totally can do that.
Max 01:06:56
But there started to be a broad consensus that we need some kind of ban on child pornography that created the political will for experts to sit down and hash out all these details. This is something you are very good at—looking at how you would actually draft the laws.
Max 01:07:10
The idea with our statement was very analogous. We see 95% of Americans don’t want a race to superintelligence. A lot of people are super excited about AI tools, but view the idea of losing control of Earth to a new robot species as kind of dystopian—including David Sacks no less.
Max 01:07:35
If we can start getting a public knowledge that most people actually don’t want an uncontrolled race to superintelligence, just like most people want some kind of ban on child pornography, then that can create political will where brilliant policymakers like you sit down with stakeholders and carefully craft language for how this would actually work.
Max 01:08:04
So in summary, the vagueness was not a bug, but a feature. We were going for moral leadership basically stating we would like there to be some kind of restrictions on a race to superintelligence.
Dean Ball, What’s Your P(Doom)?™
Liron Shapira 01:08:24
Let me jump in for a sec because I think you guys may actually dovetail more on policy itself than it’s sounding like. Maybe the real crux of disagreement is your mainline scenario of what things would look like if we just kind of went on cruise control and didn’t do much more than we’ve already done in the way of policy.
Liron 01:08:43
So, Dean, let me ask you this question. What is your P(doom) defined as just letting AI play out, not layering on additional regulation, waiting years? What is the probability that it goes wrong and we get this runaway superintelligence that’s now too late to control?
Dean 01:09:00
Doom being defined as human extinction?
Liron 01:09:05
Yeah, like a catastrophe of extinction scale. Maybe half the human population dies and then we go back to being cavemen, or just something extremely catastrophic or extinction.
Max 01:09:15
Would a permanent 1984 also count?
Dean 01:09:19
If what we’re talking about is AI systems taking control over the world and killing large numbers of people... My P(doom) is very low. It’s sub 1%, it’s 0.01% or something like that. It’s very low.
Dean 01:09:46
Not to say that I don’t think there are all sorts of other outcomes from AI that seem very bad, that seem way more plausible to me, and that I work on a lot. But the specific doom scenario just doesn’t really seem all that likely.
Dean 01:10:21
If you passed a law that said a group of people—let’s say the Supreme Court—has to look at every frontier language model release and take a straight up or down public vote on “Do we think this model is likely to take over the world?”, I wouldn’t be particularly concerned about that.
Dean 01:11:02
I wouldn’t really support that law for a lot of reasons, but I wouldn’t be concerned about the outcome. The problem is that it’s not the law. A lot of people that signed the “Ban Superintelligence” statement, I would predict, have a much more nebulous set of concerns about AI than the very specific ones that you have, Max.
Dean 01:11:17
I’m not saying you aren’t worried about misinformation or deepfakes or job loss. But when it comes to the job loss thing, it is really complicated. Matt Walsh had a tweet recently saying AI is going to cause 5 million lost jobs over the next 10 years.
Dean 01:11:53
I thought that was an extremely optimistic scenario. Eliminating 5 million jobs over 10 years is very slow compared to the normal churning of the economy where millions of jobs are created and destroyed every year.
Dean 01:12:24
The issue is, if we had a regulatory regime staffed with union representatives and various stakeholders, and their task was, “Do you think this will be good for the economy? Do you think it could create job loss? Do you think it could be dangerous?”—that group might vote against the release.
Dean 01:13:01
There is a plausible version of GPT-7 that is really good and pro-social that might also displace jobs. If the vote was “Will this take over the world?”, it would be 0-9. But with a stakeholder group worried about nebulous harm, they might block it. Do you believe that failure mode is a real one?
National Security, China, and the AI Race
Max 01:13:40
I totally agree. It was very interesting for me because I spent a lot of time talking to many of the initial signatories. There were indeed many different reasons.
Max 01:13:57
The NatSec people, like former head of Joint Chiefs of Staff Mike Mullen, for him control loss was very central because he viewed that as a national security threat. Regardless of whether the US government gets overthrown by a foreign power or by superintelligence, it’s a security threat.
Max 01:14:15
On the other hand, there were people from Steve Bannon to Bernie Sanders who felt that if we have superintelligence that makes all humans economically obsolete, then American workers would be dependent on handouts.
Max 01:14:38
Conservatives view handouts in the form of UBI as socialism. People like Bernie Sanders view company handouts, like Sam’s Worldcoin, as incredibly dystopian—the most massive power concentration in human history to a tiny clique in San Francisco who don’t necessarily share their moral values.
Max 01:15:10
Then we had faith leaders who signed this for different reasons, feeling this is harmful for human dignity. A lot of people in San Francisco joke about superintelligence being the “San God.” These faith leaders are saying, “Wait, I already believe in a God. Why should I support some atheists building a new one to run the show?”
Max 01:15:51
But in short, there are two separate questions. One: Should there be any kind of safety standards? Two: What exactly should be on the list? I’ll be very happy if we could start by just having one very light requirement: companies have to make a good quantitative case that it is not going to overthrow the US government before we launch it.
Liron 01:16:09
Before Dean responds, Max, speaking of reasons to sign the statement and these nightmare scenarios, what is your P(doom)?
Max 01:16:18
We actually wrote a paper with three grad students from MIT where we took the most popular approach for how humans can control superintelligence, known as recursive scalable oversight. We got very nerdy and tried to calculate the probability that the control fails.
Max 01:16:39
We found in our most optimistic scenario that it fails 92% of the time. I would love if people who think they have a better idea for controlling superintelligence were to publish it openly so it can be subject to scrutiny.
Max 01:17:05
But until that time, if we go ahead and continue having nothing like the FDA for AI, so people can legally just launch superintelligence and worry about getting sued later... yeah, I would think it’s definitely over 90% that we lose control.
Liron 01:17:21
Wow. 0.1% versus 90%.
Dean 01:17:25
I just have this sneaking suspicion that if the models seemed like they were going to pose the risk of overthrowing the US government, I don’t think OpenAI, Anthropic, Meta, xAI, or Google would release that model.
Dean 01:17:46
I think they probably wouldn’t do that and would call the US government. It just doesn’t seem like a realistic scenario to me.
Max 01:17:59
I think the makers of Thalidomide would not have released that either if they had known it was going to cause a hundred thousand babies to be born without arms or legs. But it was complicated and they just didn’t realize it.
Max 01:18:15
Similarly, it is very complicated for these companies to know. Dario Amodei has talked about 15 to 25% risk. Sam Altman has also talked about how it could be lights out for everybody. So they’re clearly comfortable with 5% or higher.
Dean 01:18:32
You can’t deny the possibility. I can’t prove a negative. That’s why you can’t say it’s zero. If you’re being intellectually honest, you can’t say zero.
Dean 01:18:48
Here’s the thing. First, some of the negative effects you are describing, including the labor market stuff, are going to be emergent outcomes of a general-purpose technology interacting with society. That is very hard to model in advance.
Dean 01:19:34
If a group of people sit around thinking about potential risks, they tend to overstate them. People model AI as an exogenous shock, like a meteor coming to society, assuming we will just remain in place and do nothing.
Dean 01:20:09
If I showed you today’s generative AI tools five years ago, you would have guessed elections would be over, the media environment destroyed, and no software engineers left. In reality, society is an adaptive complex system. Humans are ingenious.
Dean 01:21:23
The other point is about this recursive self-improvement thing. Every general-purpose technology in human history exhibits what you would call recursive self-improvement.
Max 01:21:24
With humans in the loop.
Dean 01:21:27
Kind of. We use iron to make better iron. We use computers to make better computers. We use energy to get more energy. Every general-purpose technology exhibits these recursive loops.
Dean 01:22:28
So you can’t just cite the fact that AI is likely to have recursive loops of self-improvement.
Max 01:22:30
We don’t disagree on anything here.
Dean 01:22:32
But in the case of every other technology, the feedback loop tends to be autocatalytic and produces nonlinear improvement, but never results in a runaway process where we blew up the entire universe.
Liron 01:23:11
Okay. Max, maybe you can explain why you still think there’s a doom scenario despite Dean’s point.
Max 01:23:40
I completely agree that technological progress has always involved self-improvement loops. But there have always been humans in the loop. When there are not humans in the loop, things can go quite fast.
Max 01:23:56
If you look at a slow motion of a nuclear bomb exploding, there is no human in the loop. You get one uranium atom decaying, then two, then four, then eight. The reason we’ve never seen anything blow up fast with our technology is because we’ve always had humans as the moderator.
Max 01:24:25
Unless you think there is some secret sauce in human brains that you can’t build into machines, it is possible to build machines that don’t need us. If those machines think a hundred times faster than us and can instantly copy knowledge, we could see more progress in a month than in a thousand years.
Max 01:24:44
This is not my idea; I.J. Good articulated this in the sixties. We can’t say “it never happened before so it won’t happen again” because we’ve never built superintelligence before. The industrial revolution replaced muscles; we’ve never replaced cognitive abilities.
Max 01:25:31
I’d love to comment on the NatSec angle because it is the main reason given in Washington for why we should not regulate. AI lobbyists say, “But China.” They say if we don’t race to superintelligence, China is going to do it first.
Max 01:25:52
I think that is baloney. It’s not one race. There are two separate races. One is a race for dominance—economic, technological, military—which Dean articulated in the AI Action Plan. The way to win that is by building controllable tools.
Max 01:26:15
Then there’s a second race: who can be the first to release superintelligence that they don’t yet know how to control. I argue this is a suicide race. The Chinese Communist Party and Xi Jinping clearly like control.
Max 01:27:00
They would never permit a Chinese company to build technology if there were a significant chance it could overthrow them. Elon Musk told me in spring 2023 he met with high-up CCP officials and said, “If someone in China builds superintelligence, China is not going to be run by the CCP. It’s going to be run by the superintelligence.”
Max 01:27:23
Elon said there were a lot of long faces. Within a month, China rolled out their first AI regulations. I am confident the Chinese have much more surveillance on DeepSeek and their companies than the US government has on ours.
Max 01:27:58
When I said P(doom) of over 90%, that was if we do no regulation. I am actually quite optimistic because I don’t think China will allow a race to superintelligence since we don’t know how to control it.
Max 01:28:44
I think there are a growing number of people in US National Security who are beginning to view this as a threat. Maybe they listened to Dario Amodei talk about a “country of geniuses in the data center” and thought, “Wait, I have a list of countries I track as threats. Did Dario say country? Maybe I should add that to my watch list.”
Max 01:29:04
We could end up in a great situation where the US prevents anyone from building stuff they don’t know how to control, and we have a race to build the best, most powerful, helpful tools.
Liron 01:29:25
Okay, Dean, let me ask you the last question. Max has laid out his nightmare scenario of uncontrollable recursive self-improvement. Dean, you see that as very low probability, but you have your own nightmare scenarios regarding regulating AI too much.
Liron 01:30:09
If I understand correctly, your two nightmare scenarios are losing the AI race or an overregulation that leads to a tyranny situation. Explain your nightmare scenarios.
Dean 01:30:20
Certainly, I think all manner of tyranny is possible with AI. AI is going to challenge the structure of the nation-state no matter what. It requires institutional evolution, conceivably revolution in certain places.
Dean 01:31:10
There is a version of that evolution where we get a rentier state. Think of the Middle Ages. We get a state run by a small number of people that control something—a tool of violence—and they are not quite legitimate in the way we think of democratic legitimacy.
Dean 01:31:32
There’s a middle class of rent-seeking humans who have legal protections, and then a large underclass of people with very low practical agency. That seems very likely to me. I think there are many regulatory regimes, including licensing, that make that outcome substantially likelier.
Liron 01:32:01
Do you think losing the AI race or the tyranny scenario is the main nightmare?
Dean 01:32:08
Basically tyranny. Losing the AI race to China is hard to know... certainly there is a world where China becomes the dominant technological power, and that’s a bad world too. But it’s not my nightmare scenario; it’s not the worst possible thing.
Closing Statements
Liron 01:32:35
Okay. All right. We’ve covered a lot, so let’s go to closing statements, starting with Max.
Max 01:32:40
We’ve talked a lot about doom here, but I’d like to end on an optimistic note. The real reason I’m so engaged with this topic is because I’m fundamentally a quite optimistic person.
Max 01:33:00
I’m very excited about the potential for an amazing future where we don’t have to worry about dying of cancer and we can prosper like never before, potentially for billions of years, spreading out into the cosmos. We’ve completely underestimated the opportunity we have.
Max 01:33:22
That’s why I think it’s important we don’t squander this by making hasty chess moves. We are choosing between two paths right now. One is the pro-human future. America was founded to be run by the people, for the people—not for the machines of America.
Max 01:34:26
That path is very pro-tech: full steam ahead with ever-better AI tools. The other scenario is we race to build superintelligence. By definition, none of you can earn any money after that’s been built. You’re dependent on handouts from the government or a tech CEO.
Max 01:35:07
Why should we, after hundreds of thousands of years of working to become captains of our own ship, throw away all this empowerment by building something that takes over? That is incredibly unambitious.
Max 01:35:52
A journalist asked me what Steve Bannon has in common with faith leaders and Susan Rice. I said, “They’re all human.” Of course they want the pro-human future. If there was an alien invasion, we would work together.
Max 01:36:20
Now you have a small fringe group from Silicon Valley saying, “Yeah, we should build these aliens.” To me, the inspiring future is where we remain in charge and keep AI as a tool to create a future cooler than sci-fi authors could imagine.
Liron 01:37:01
Right, great. Let’s go to Dean.
Dean 01:37:03
I think the fundamental thing to think about here is assumptions. AI doom debates usually revolve around one of the interlocutors assuming their conclusions.
Dean 01:37:33
We’ve talked about how superintelligence has many different definitions. There’s one version that implies bad things. But it takes a big leap to assume that is what we are actually going to build.
Dean 01:37:57
As Max said, the future is often profoundly stranger than we can imagine. The future today would be alien to someone 50 or 100 years ago. The things we assume today about the technology of the future are probably wrong, and we don’t want to embed too many of those assumptions into the law.
Dean 01:38:42
We want to maintain adaptability. I wouldn’t assume that superintelligence means the bad thing. I would consider that there are many worlds in which humans can thrive amid things that are better than them at various intellectual tasks.
Dean 01:39:45
Max talked about how America is by the people and for the people. True, but we also have a system that makes it quite hard to pass new laws. Our founding fathers were deeply distrustful of raw democratic impulse.
Dean 01:40:13
They believed you had to balance raw democratic will with deliberative bodies because laws are passed by people that have the monopoly on legitimate violence. We don’t want to give them new powers willy-nilly.
Dean 01:41:19
I have serious issues with the idea that we can pass a new regulatory regime and everything goes fine with no side effects. I think there will be tons of side effects and we will ban tons of technological progress.
Dean 01:42:03
There are many ways to investigate and interrogate superintelligence without banning it. I note that Max did not spend that much time defending the actual “Ban Superintelligence” statement.
Dean 01:42:28
You can build a society capable of grappling with this technology and institutions that evolve with it. That involves taking risks seriously, but not being a radical in either direction. Details matter. It’s not going to be a matter of taking regulatory concepts off the shelf.
Liron 01:43:15
Thank you. I’m thankful to both of you for stepping up to debate the difficult policy questions around superintelligent AI. It’s not black and white. It won’t work in an echo chamber.
Liron 01:43:33
Respectful debates between smart people with different views is what we need right now. I’d go so far as to say debate is a social infrastructure. So thank you again, Max and Dean.
Max 01:43:52
Thank you, Dean, for a really great conversation.
Dean 01:43:55
Thanks to you, Max. Thanks to you, Liron. This was great.
Post‑Debate Recap and Call to Action
Liron 01:44:00
Wow, what an illuminating debate from two people who are actually in the room for these kinds of policy discussions. So regarding America’s AI Action Plan, the document that Dean Ball helped draft.
Liron 01:44:12
Both Max and Dean were happy that it doesn’t mention superintelligence, but for different reasons. Max was saying we need a whole other statement about superintelligence, proposing a conditional ban until there is consensus. Dean is saying it’s good we didn’t mention it because it’s too vague right now.
Liron 01:44:32
The crux of disagreement really comes down to their P(doom). Max was saying his P(doom) is greater than 90% without tough regulations. Dean’s P(doom) is only about 0.1%.
Liron 01:45:15
Dean is basically not worried about plowing forward and dealing with issues as we get to it, whereas Max says we better be preemptive because it might be too late.
Liron 01:45:28
I think their policy recommendations are totally downstream of what they see as the probability of doom. If they were to meet halfway—say 25% or 40%—they’d start coming up with very similar policy ideas.
Liron 01:46:12
They went on to talk about the FDA analogy. Dean didn’t go full libertarian saying the FDA is evil, but he pointed out its baggage and industrial-era focus. His analogy is that when we get to superintelligence, this kind of straightjacket regulation could be an extreme case of why he doesn’t like the FDA.
Liron 01:47:28
In conclusion, what a stark divide. A scientist saying high risk of catastrophe in potentially less than 10 years, and a policymaker saying you haven’t made a strong enough case so we shouldn’t ban valuable research.
Liron 01:48:01
My prediction is that we’re going to keep seeing policy that’s downstream of the policymakers’ P(doom). One of the benefits of having Dean Ball here is that we heard his perspective on P(doom) explicitly because it wasn’t in the AI Action Plan.
Liron 01:48:25
It’s not just about this one disagreement. It’s about building the social infrastructure for high-quality debate. It has to be informed, respectful, nuanced, and productive for policymaking.
Liron 01:49:03
If you think this debate was productive, you can support me and my team at Doom Debates by donating to the show. Go to doomdebates.com/donate to learn more.
Liron 01:50:00
If you’re new to the show, check out the Doom Debates YouTube channel. I’ve been having debates with top thinkers like Gary Marcus (P(doom) 1-2%) and Vitalik Buterin (8-12%).
Liron 01:50:23
While you’re on that YouTube channel, smack the subscribe button. I look forward to bringing you the next episode of Doom Debates.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
