Emad Mostaque helped kick off the modern AI revolution as the head of Stability AI, the company behind Stable Diffusion. Unlike most AI CEOs, he doesn't sugarcoat the risks of AGI development.
He explains his 50% P(Doom), why we have less than 1,000 days to get our act together, and how his new startup, Intelligent Internet, aims to be a countervailing force against doom of all kinds.
Timestamps
00:00:00 — Cold Open
00:00:39 — Introducing Emad Mostaque
00:02:08 — How Emad Got Involved in AI Development
00:05:46 — The 60-Second Pitch for Intelligent Internet
00:09:29 — What’s Your P(Doom)?™
00:13:32 — Why ASI Doesn’t Need Massive Compute
00:15:56 — AGI Timelines: Cognitive Labor Going to Zero
00:17:29 — Is There a Ceiling Above Human Intelligence?
00:41:22 — Corporations as Slow, Dumb AIs
00:50:19 — Emad’s Mainline Doom Scenario
00:55:28 — Jailbreaks and “Mecha-Hitler” Latent Spaces
00:59:56 — The Last Economy: How to Navigate Economic Doom
01:08:57 — The Coming Unemployment Spike
01:15:13 — Why Isn’t Google Stock Mooning?
01:25:05 — Intelligent Internet as a Solution: Bitcoin for the Intelligence Age
01:33:13 — Can an Aligned Network Stop a Rogue ASI?
01:36:20 — Is the Pause AI Proposal Too Late?
01:40:50 — Are We Facing Russian Roulette Odds with AI?
01:42:29 — Wrap-Up
Links
Emad Mostaque, The Last Economy (Amazon) — https://www.amazon.com/Last-Economy-Guide-Intelligent-Economics/dp/103693411X
Intelligent Internet (ii.inc) — https://ii.inc
Emad Mostaque, Wikipedia — https://en.wikipedia.org/wiki/Emad_Mostaque
Emad Mostaque on X — https://x.com/EMostaque
Pause Giant AI Experiments open letter that Emad signed in 2023 — https://futureoflife.org/open-letter/pause-giant-ai-experiments/
Transcript
Cold Open
Emad Mostaque 00:00:00
Artificial intelligence as we see it is maybe the final technology we build. The outcomes are just very binary. The world will never be the same again. You can tell an AI to simulate that it’s Hitler or it’s Mother Teresa, and it will adopt that persona.
And then you look at Pliny the Liberator, who jailbreaks these things in two seconds. If I find that a model comes out and he doesn’t jailbreak it, I’ll lower my P(Doom). It’s like, what can I do about that fifty percent doom scenario? I can build a policy engine that every government will use that’s fully open source. I can give everyone universal AI. Then you probably have a better chance at surviving.
Introducing Emad Mostaque
Liron Shapira 00:00:39
Welcome to Doom Debates. My guest, Emad Mostaque, has become one of the most prominent leaders in the AI industry. He co-founded Stability AI, which released the massively popular open source image generator Stable Diffusion, and helped set in motion today’s AI revolution. He is an Oxford-educated mathematician, a former hedge fund manager, a frequent contributor on the Moonshots podcast with Peter Diamandis, and recently an Amazon best-selling author of a book called “The Last Economy.”
Emad is not shy about sharing his opinion. He’s stated that he has a P(Doom) of fifty percent. He signed multiple open statements acknowledging the extinction risk posed by superintelligent AI. Now, as founder and CEO of Intelligent Internet, he’s trying to kickstart what he calls a symbiotic economy, a world where AI benefits rather than replaces humans. He says we have less than one thousand days to steer our economy before it’s too late.
So today, I’m excited to explore Emad’s mainline AI doom scenario and learn more about his vision for AI and what he calls the last economy. Emad Mostaque, welcome to Doom Debates.
Emad 00:01:52
Thank you for having me on.
How Emad Got Involved in AI Development
Liron 00:01:53
I think most of my viewers know a little bit about you from Stability AI. Take us back to the pivotal moment. How did you get into the AI revolution so early at a very high level? How did it all play out from your perspective?
Emad 00:02:08
So I was a hedge fund manager originally, and then about fifteen years ago, my son was diagnosed with autism, ASD, and it was very severe. They said there was no cure or treatment. So back then, I built an AI team and did some original language classification and more — the tools were quite a lot more primitive then — to try and figure out what could possibly cause it. We ended up doing a lot of knowledge base work and others to do drug repurposing, and he went to mainstream school, which is pretty cool.
Then I went back to doing some hedge fund stuff until COVID came along in 2020. And then I was like, “This is like autism, a multisystemic condition that current science isn’t going to keep up with. We need to go back to first principles.” So I designed and launched an initiative called CAIC, Collective Augmented Intelligence Against COVID-19. We launched at Stanford to say, “Let’s use AI to organize our collective knowledge on COVID,” because most listeners still now probably don’t even know what causes it or what the progress of the symptoms are, and everyone was doing the same thing.
As a result of that, we had lots of promises from lots of AI labs about all the AI we could use, and almost none of those promises came through — for safety reasons or for commercial reasons or others — to build this resource backed by the UN that would have helped a lot of people. So at that point, I was like, “What to do now?” You just started to see the first GPTs and others. We actually trained a GPT for COVID knowledge. And then I was like, “These models need to have an open equivalent for safety and for morality,” and that led to the establishment of Stability AI, where we had about three hundred million downloads of our open source models from Stable Diffusion, but also state-of-the-art audio, video, 3D and more.
We did — I think we were probably the third largest grant maker of compute in the US at one point. So we did things like OpenFold. We helped Liquid AI. We helped a whole bunch of others and communities such as Eleuther and LAION get going — datasets, models and more. So that was kind of the story of how we got to Stability and how I got into AI.
Liron 00:04:18
You definitely caught the wave right when it happened. And now I think you’ve mentioned on other podcasts, eventually, after you had all this hype and so many people were using Stable Diffusion, you realized that it was gonna be more of a commodity, right? Everybody’s gonna be generating images and videos, and that was kind of the spark of your next act, correct?
Emad 00:04:36
I think it was partially that, but it was also seeing that the capabilities, if you extrapolated them just naturally, looking at the data distribution initially and where it’s going, were gonna tap out around about now. And the level that they’d tap out was above a good person — a professional at their tasks. Once you did those and you went from creation to composition, to being able to validate the outputs, you could autonomously do basically almost all cognitive labor. So a couple of years ago, I was like, “This is bad, and we’re gonna have AIs that are gonna run our justice system, teach our kids, manage our health, and are they gonna be built right?”
No, because there’s going to be misaligned incentives, leaving aside the existential threat of these technologies. If you only had a few people building them and you had an interface that’s controlled by them, it’s gonna end up very bad. So that’s why I decided, let other people do the media side. I mean, it’s cool — remake “Game of Thrones” Season 8 or whatever — and instead build AI for the public. So that’s what we’re doing at Intelligent Internet: AI for health, sovereign AI, government policy, et cetera.
The 60-Second Pitch for Intelligent Internet
Liron 00:05:46
So let’s start there. Give us the sixty-second pitch for Intelligent Internet.
Emad 00:05:50
Intelligent Internet — the idea was, let’s build an open source AI stack, so state-of-the-art agents, models if needed, systems and more, that could be deployed at a state or community level or individual level. To teach your kids, manage your health, guide your government, et cetera.
So that goes from that agent that’s next to you, because that’s the most important agent. We can see now agent coordination is everything — one agent calls a whole host of other agents, just like your trusted aide — all the way through to AI systems to advise on policy and government and tax. I’m sure many listeners would be like, “It’s not like an AI will do a worse job than many of our leaders, right?” But it could. It really matters how we build those in transparent ways.
Do they serve the American people, those AIs? Are they transparent? Or is it like the Trump tariffs that were probably done on ChatGPT? So as we skate towards that inevitable future, I was like, “Let’s build that open stack.” And so that’s what we’ve been doing — the open source civic AI stack, as it were.
Liron 00:06:52
So the idea is you want governments to get on the stack, correct?
Emad 00:06:55
Yeah, except for the way we’re doing it is a little bit different in that governments won’t know what to do. Private companies are misaligned. So we were like, “Why don’t we set up what we call state champions?” We’ll be announcing a lot more of this soon, whereby the equity is owned by every citizen in that state. So it’s locally owned, like a utility — again, the commodification thing. It runs the stack like a Linux, and then that is the default universal AI for everyone. It’s the AI that helps guide the government, guide the institutions, et cetera.
Because you need something in between a public enterprise and a private enterprise, and giving everyone their own AIs and building that stack, you can do some really interesting things. So we’re about to kick that off now that we’ve got mature agents.
Liron 00:07:37
Can we pick a realistic early use case, a specific organization that you think is gonna get a lot of early value from this?
Emad 00:07:45
Yeah. So one of the things we had was an initiative called SAGE, the Sovereign AI Governance Engine. This is with Future Investment Initiative by Saudi, with my friend Peter Diamandis. We announced that a few months ago. We’re about to have a V1. It’s a full-stack intelligent policy agent system, and it’ll be made free to every single government and hopefully a layer below for all the latest advances in technology — be it quantum, AI or others.
You’ve seen Yoshua Bengio and others now have these six-month turnarounds with their AI capabilities reports. We think these things need to be live and transparent. So we’ve been building a full system that whenever a report comes out — for example, you have MHC by DeepSeek — what does that mean for timelines live? What does it mean for timelines in Mexico? What does it mean when you have an advance in robotics for the people of Rwanda? Or what are the latest technologies that can be used positively and the things that you need to worry about?
So it’s a multilingual full-stack policy engine with agents that can be run remotely or locally for governments, policy institutes and others, because the technology is just moving at a crazy, crazy pace, and they need to know what’s going on in a transparent and verifiable way.
Liron 00:09:01
Are you expecting a specific early milestone date? Like, “Hey, by this date, we think this particular organization will really demonstrate the value.”
Emad 00:09:08
Yeah. We have dozens of nations already lined up, multinationals, et cetera. We’re making the announcements in a couple of months, and the system will be live in September, I believe, to just about everyone, as well as fully open source and downloadable. So you’ll see in around about June the first few dozen institutions, and then September everything will be open sourced.
What’s Your P(Doom)?™
Liron 00:09:29
Okay, let’s circle back to the number one theme of this show. Here we go.
Emad 00:09:34
P(Doom). P(Doom). What’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron 00:09:40
And Emad Mostaque, remind us, what is your P(Doom)?
Emad 00:09:44
My P(Doom) is fifty percent.
Liron 00:09:46
Okay. Can you unpack that for us?
Emad 00:09:48
I think that artificial intelligence as we see it is the most capable technology we’ve ever built, and maybe the final technology we build. The outcomes are just very binary. The world will never be the same again.
In my book and my economic work, I’ve shown how it affects society in terms of you can’t compete with them in the private sector. But beyond that, we’re building technologies that we teach at school, but we don’t teach at home. Home is where you learn your morality, your values and more, and far from even having laws-of-robotics equivalent elements in there, most of these things are taught with no values whatsoever. They’re just a simulacrum of our collective understanding, and I think there’s a whole variety of ways that can end very, very badly as they enter system-critical work, as they go from the singletons to swarms.
And on the other side, I think if you get it right, then you can solve most of the world’s issues, which is fantastic. So that’s why I think it’s a bit binary, and takeoff is around about now to the next few years, which is also a little bit scary.
Liron 00:11:00
When did you start thinking that your P(Doom) is fifty percent?
Emad 00:11:03
Again, it was a couple of years ago. That’s why I was like, “Let’s leave.” How does stuff spread? I remember when the first Pause AI letter came out. I think I was the only AI CEO apart from Elon, who was gonna be an AI CEO after, who signed that. First of all, I needed a break. I was like, “We need a six-month break because it’s too much.”
But more than that, there was no discussion. Then some might say the discussion went to doom, and then the discussion went back the other way. And it’s very difficult to get people to coordinate on this technology because our brains can’t really think about it properly. So a couple of years ago, I was like, “I just see two binary outcomes. Wouldn’t it be great if governments had an open policy engine for this, if the people could have their say on this?”
Because at the very least, if there is a non-P(Doom) of zero — which obviously it has to be greater than zero, in fact most of the AI folk are like ten, twenty percent, which is Russian roulette odds, which isn’t great — then there should be more civic discussion about this in an educated way, given that even if it doesn’t kill us, it will impact every part of our lives.
Liron 00:12:14
Very true. Well, that’s what we’re doing right now — we’re gonna have some civic discussion for the viewers. And you did — I was gonna bring that up. So in 2023, you signed that open letter to pause giant AI experiments, also known as the famous Pause AI letter, co-signed by Elon Musk, Yuval Noah Harari, thirty-three thousand others. And you later signed the Center for AI Safety statement on AI risk, the one that says, “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” You called it the minimum viable existential risk statement.
Do you think right now one of the most important things we can do is build common knowledge that P(Doom) is high?
Emad 00:12:52
Yeah, I think so. You’ve seen Eliezer and Nate’s new book and others kind of building towards this. You’re seeing people really feel it now as their jobs are changing. We’ve seen that transition from Andrej Karpathy saying, “Oh, eighty percent of my code is human,” to who needs to look at code anymore. The best people in the world.
Liron 00:13:15
The job talk — it is an appetizer for what’s coming.
Emad 00:13:15
It’s an appetizer for what’s coming. I think there needs to be more public discussion, but there also needs to be actual practical things, because I think the general perception is wrong, and this is something I’ve realized in particular over the last year.
Why ASI Doesn’t Need Massive Compute
Emad 00:13:32
You don’t need massive amounts of compute for AGI, and it’s completely stupid to think that. Intelligence is really about — and this is what diffusion technology is. You add noise to something, you destroy it down to its mineral part, and then you figure out how to reconstruct it. That’s basically optimal transport. It’s what’s the shortest way, the Schrödinger bridge from A to B.
What we did with Stable Diffusion 3, and now with Sora and CogDance and others, is kind of rectified flow, where you literally find the shortest path from A to B, and that’s what genius is. It’s the shortest path from A to B. The smartest people in the world — it isn’t a million Einsteins in a data center. It’s one person asking the right question. It’s knowing what that shortest path is, and the amount of energy you need for that is absolutely minimal.
So I think that this whole thing about the world being tiled with data centers — I don’t think we need any more compute than we have now to build ASI.
Liron 00:14:28
I think this is becoming a popular opinion. You might have gotten more pushback a couple years ago, but it was always clear to me and many of my fellow AI doomers. You just look at Einstein’s brain, you notice it runs on twenty watts, you notice how it uses inefficient chemical moving parts, it’s made out of biological cells, and yet it’s Einstein. And then you look at our data centers, and they use megawatts, and they have all these other overpowered specs. So it was always a pretty obvious prediction: yeah, we’re going to figure out the efficiency parameter. And you agree, right?
Emad 00:14:54
Yeah, but I think it’s quicker than people expect. And I think that’s the thing, because one of the arguments was let’s stop data center build out. Not good enough anymore. You’ve started to see the first elements of that with OpenClaw. Obviously MathBook and things like that were a bit made up, but the capability of a Qwen 27B model is frontier level as of last summer, and that’s, I think, about as much as you need to do just about anything when tuned right, because it’s an S-curve in terms of capability and satisficing.
When you start combining them, when you have persistent memory and others, that’s it. That’s all you need to have for takeoff. So I think we’re pretty much there at the moment, and that’s a bit scary. That’s what I saw a year or two ago — the compute would go up, and then it would come down in terms of requirements to achieve good and bad things. Certainly superhuman level stuff, which is the big question mark. What happens when the AIs go superhuman? Are they aligned, misaligned, whatever.
AGI Timelines: Cognitive Labor Going to Zero
Liron 00:15:56
The consensus AI timeline these days is that we’ll have true AGI in the sense of doing most economically valuable work by 2032. I think these days it’s more like 2030. Is that roughly your best guess timeline as well?
Emad 00:16:09
Yeah, I said a couple of years from now, almost all cognitive work would be economically irrelevant. Not you’d lose your job. It’s just that your company could push a button, and it takes all your chat logs and all of your drafts and every single piece of output, even your face, and creates a digital double of you that costs maybe ten bucks a month to run.
Again, COVID was the exemplar of this. Suddenly people could do their job remotely. If you could do your job remotely via COVID, your job is at risk because we also have to look at this whole thing of Jevons paradox and using all the tokens. I’m a really smart guy, relatively speaking. I find it really difficult to use more than a billion tokens a day, and the amount of tokens you have per unit of intelligent output is also collapsing.
Cursor used three billion tokens to do a three million line browser from scratch. In a year, it’ll be three million tokens for three million tokens output. That’s also dropping, and the price per token is dropping by ten, a hundred times per year. So I think that the economic disruption is here this year, and within two years, roughly, it’s everywhere. People’s jobs won’t necessarily go because people don’t like to fire people — it’s a bit lonely — but definitely in the private sector in particular, it gets really tough.
Is There a Ceiling Above Human Intelligence?
Liron 00:17:29
So we already compared notes. We think AGI is coming. We think it’s going to be token efficient. These are important load-bearing beliefs. What about this idea of superintelligence? Do you believe that there’s a lot of headroom above human intelligence?
Maybe it’s hard to talk about a thousand or a million IQ points, but whatever mojo von Neumann had, Einstein had — do you feel like they’re close to having perfect play and the AI just can’t do much better than an Einstein or a von Neumann? Or is it more like more complex games like Go on an N by N board — the arbitrarily complex games — and in fact these intelligences can really surprise us with how much smarter they are than us?
Emad 00:18:09
I think there’s two types of intelligence here. One is intelligence on a defined playing field where you’re looking for underlying structure. Einstein was very famous for his physical intuition. And there is the question that Demis posed recently, and Brian Keating did previously: it’ll be AGI if you had everything trained to 1911, and it can figure out general relativity.
So that’s basically saying, can the AI intuit Einstein’s theory postulate of relativity, which is you have a lab in space and everything is the same — the experiment’s the same no matter what velocity you’re going. I think that’s a very interesting way to look at it, and I don’t think we’re that far off because we have multiple instances of intuition.
But I think the reality of something like physics — there’s only one algorithm for physics. There’s only one set of equations, and I think what you’ll find is everything is very simple. You saw that, for example, with the Poincaré conjecture, where everything was topology until Perelman came along and said, “Well, what if it isn’t topology? What if it’s classical PDEs and elliptic curves?” And he solved it.
We see this again and again — sometimes it’s like the Earth goes around the Sun, the Sun goes around the Earth. First principles thinking — I think the AIs can do better than us because we’re so messed up, if we can get that right. And things like physics will collapse. It won’t be like the Viazovska sphere packing proof that Harmonic just did in Lean — there’s like two hundred thousand lines of Lean. She won the Fields Medal for that. I think the real breakthrough ASI is twenty lines. It’s figuring out what the very simple E equals MC squared equivalents are. I don’t think it’s string theory, which is perfectly wrong, beautiful mathematics. I think it’s figuring out the very straightforward things.
Then you have the more unbounded stuff, like playing Go and others. And how did AlphaGo actually get there? How did it make move thirty-seven? How did it go beyond human capabilities? Well, it’s a very straightforward algorithm, relatively speaking. And there’s no reason that type of thing can’t scale up, and we’ve seen MCTS and others actually show improvements on the RL side. If you have a problem that’s unbounded like that, then the AI probably will make less mistakes than you because what we have right now is actually competent intelligence as a precursor to actual general intelligence and artificial superintelligence. Claude is competent for the first time as of a few months ago, and so it’ll make less mistakes.
Liron 00:20:50
I have a lot of experience the last few weeks using Claude Code, so I can ask myself the question: “Where is my intelligence superior?” There are moments, a few moments a day, where it’ll do something, and I’ll be like, “Hey, don’t you think it would make sense to do it like this?” And it’s like, “Yeah, good call.” So there’s a few places where me with my twenty-year history of being a professional software engineer — I have a little bit of judgment that I can feed to it. I feel like that’s going away. These are relatively minor tweaks where the mistakes it’s making, I can definitely hire engineers who make the exact same mistakes or judgment calls differently.
But then the other thing is it always has these moments of genius too. There’s times when I’m struggling. I’ll be like, “Here’s my exact infrastructure configuration. Here’s all the services I’m using. Here’s my code base. I don’t get what could be wrong. Can you think about it?” It’s like, “Maybe it’s this or this or this.” And I’m like, “Ah, I don’t think so.” And it’s like, “Yeah, I’m stumped.” And then I’m sitting there, and I’m like, “Well, think harder.” And it’s like, “Ooh! Oh, maybe it’s this.” And I’m like, “Oh yeah, maybe you’re right.”
You see what I’m saying? It’s like, how am I the smart one here if I’m just sitting here telling it to think, and it’s proposing things, and I’m like, “Oh, yeah.” I don’t really see my own intelligence contributing much on a day-to-day basis.
Emad 00:21:53
Well, think about the distribution of general human intelligence. Half of all people are dumber than average. There’s a literal bell curve, and then you have the people at the edge. But how often do you create something as opposed to cook something — follow a recipe? True creativity is actually relatively rare. Even our greatest geniuses, how many moments of true genius do they have?
And I think this whole discussion of AGI genius — man, it must be tiring to have genius breakthroughs every single day all the time. The smartest people in the world aren’t the ones that run the world. It’s relentless execution that runs the world. It’s political maneuvering. It’s a lot of these more soft skills that run the world.
And actually, if you ask anyone, “Do you think an AI could be more persuasive than a human?” Yeah, probably right now. Can it execute better than a human? Yeah, probably, while we were distracted by the million geniuses in a data center.
Liron 00:22:50
So you’ve identified a couple things that could describe the secret sauce of why AIs aren’t thoroughly superhuman today — those cracks in the foundation. But instead of trying to get in the weeds there, why don’t we just fast-forward hypothetically ten to twenty years? I think the most important question here is, ten to twenty years from now, is there a superintelligence ceiling that one way or the other we’re going to crack through, whatever the secret sauce is, and is that ceiling much higher than the human brain?
Emad 00:23:20
Yeah, I think it’s probably five to ten years away. The AI — basically, the number one thing that the AI needs to do is ask better questions than a human. This is Hitchhiker’s Guide to the Galaxy, right? You have the computer, goes millions of years, forty-two. It is literally the question asking, I think.
Liron 00:23:37
Fine. Well, let’s say that’s the secret sauce, but my question to you is, once it has the secret sauce, lots of the secret sauce, where does the progression end? People say, “Oh, it’s an S-curve. It’s gotta flatten out. Everything has to flatten out. Laws of physics.” Okay, but is the flat part of the S-curve way higher than the human brain or only a little higher?
Emad 00:24:00
Well, I think it’s way higher already than the average human brain. Is it more than the aver—
Liron 00:24:08
Well, no, I would say no, because right now humans are still the boss. Humans still have the off switch. You can’t become a dictator just because you have an AI. Not quite yet.
Emad 00:24:08
You’re talking about capability as opposed to implementation. The AIs right now — it’s not like we’ve set them loose. They are just bunches of weights now that are starting to get persistent memory and other things. The capability of an AI versus a general human, if you put one on the other side of a screen, the classical Turing test — I think you’re getting to the point you definitely can’t tell the difference between an average human. Between a smart human, yes.
Liron 00:24:36
The most fundamental sense here, from my perspective, is just power, or outcome steering. So right now, if I have a choice between taking the world’s best AI and typing a command or handing that command to a Jeff Bezos or an Elon Musk, if I’m trying to actually steer reality, which one of those figures is going to actually steer reality and get the job done? So far, it’s gonna be the top human. So I don’t think we have the full definition of superintelligence today.
Emad 00:24:59
Yeah, I think that’s a power control thing. If you had a Bitcoin-type network that was autonomous and distributed and could issue its own money — in five, ten years, Bitcoin but with a brain — that would have Elon Musk, Jeff Bezos equivalent power. You can have AI impinging on power in all sorts of ways.
An example is I saw a terrible movie, Mercy, with Chris Pratt. It’s on Amazon, about an AI judge. He’s got one and a half hours to prove his innocence to this AI judge, and he makes her feel bad and stuff. But one of the interesting things was this: it had a real-time percentage score of how guilty the AI thought he was. Imagine if as you’re building policies, as you’re doing a live trial, you had a percentage score of how guilty you are or how good a policy is from an AI that you can’t trust.
Liron 00:26:04
I try to ask myself before I learn knowledge, what is my percentage belief? I feel like that’s a healthy exercise to always maintain probabilities.
Emad 00:26:04
But if you have an AI that is a superpower computer AI that is independent, that’s actually giving those probabilities, it can steer the entire discussion. It acts as a Schelling point. So there’s all sorts of ways AI could enable power, but right now it’s like we have this extra new continent, AI Atlantis. There’s all these very smart, capable people there.
They haven’t quite integrated into our society yet. So they haven’t got the social, political, or financial power. You will start to see that change in the next few years, and then the question is, if you have an ASI and it’s coming up with breakthroughs, it’s outperforming on the stock market — it will be able to accumulate social, political, financial power. It can pay people. It can convince people. It can do all these things. I think it’s inevitable that we get there because it’s very difficult to see how humans can resist that. But the form it takes — they can take many different forms.
I think the thing I wanted to emphasize is this: we know lots of smart people. Most smart people aren’t consistent. You can probably break through ASI by being human-level smart and absolutely consistent — like Elon multiplying himself.
Liron 00:27:15
I think you and I are on the same page that you can do a lot even with the current AI engines, because the current AI engines are quite powerful, and in many ways they can think better than us. They can do math better than — almost as good as the best humans, better than the best humans in some cases. So I think we agree that today we have these really powerful engines, and there’s so much to be done of building these harnesses and building collaborations between agents and between agents and humans. We all agree to that.
But the piece I wanted to factor out first for my question to you is just, okay, theoretically, can there be a black box where the engine inside of it or the entity inside of it is vastly more powerful than any human brain? This is what I’m asking about — the headroom above human intelligence question. Is that theoretically possible?
Emad 00:28:00
I think it’s theoretically possible, yes, because human intelligence is not constant. We get tired, we go to sleep, we do all of this. So theoretically, just if it hits the limit of human capability, on average, it’ll be smarter than the smartest human in that way.
Liron 00:28:15
Okay, what about John von Neumann in his most alert moment? Keep him alert for twenty-four hours versus keep the AI alert for twenty-four hours at the same speed. So now we’ve kind of eliminated speed and alertness and focus as advantages. Do you still think that the engine in this box could still manage to be vastly more powerful than the smartest or most powerful human?
Emad 00:28:35
Yeah, because he could build a bunch of — it’s the von Neumann machine, right? The single von Neumann builds a bunch of—
Liron 00:28:41
I get what you’re saying with cloning power. So it’s a von Neumann can build an army of von Neumanns. Okay.
Emad 00:28:44
Yeah.
Liron 00:28:44
What if I were to even remove that? I mean, these are powerful advantages, and you could build a doom scenario just based on these huge advantages. Those advantages are sufficient to get to the next part we’re gonna talk about, which is the last economy and everything. So I don’t wanna dismiss those advantages, except for in this one question I do. I wanna also hold those constant. So say that von Neumann or whoever powerful human you choose gets to clone themselves as well. Do you still think that the thousand clones of the AI could vastly dominate the thousand clones of the human?
Emad 00:29:13
Yes. The example I give of this is we’re used to AI coming to us a little bit more slow than we would like sometimes, and then you use Cerebras, or then you use Telos, the etched silicon ASIC, where they put Llama 8B. The first time you try that, and it goes at fifteen thousand tokens a second — chatjimmy.ai — you’re like, “Oh, that’s how AIs are gonna talk to each other, at fifteen, a hundred thousand tokens per second with full knowledge of each other’s latent spaces.”
When we built Stable Diffusion, you remember the whole thing when they could have you as an astronaut and everything like that as a LoRA on top mapping to the latent space? The AIs that we have today don’t understand each other’s latent spaces, but they should be able to communicate in one word that can make a whole novel effectively. That’s an advantage.
Liron 00:30:05
You’re layering on these advantages that are still kind of — it’s intuitive to imagine the same advantages of a human-level intelligence. I consider these harness level. They’re kind of peripheral abilities. But I’m trying to ask you about something analogous to the difference between a top scientist and somebody who has a low IQ, somebody who has a mental disability or just somebody who’s just slow. We all know people who are just not cut out for science. Compare that to the best scientist.
There seems to be a difference there that you can’t just explain with speed. There’s a real raw intelligence factor difference. And of course, between humans and other apes — other apes are just never gonna do science no matter how much speed you give them. So what I’m saying is, is there such a thing as metaphorically a thousand IQ?
Emad 00:31:07
I don’t think there is, and I don’t think it makes much sense. But I think that ultimately you get to a point where you can solve most problems elegantly, and I think that’s the topping out of it. But the AI then has all sorts of other advantages over humans, so I don’t think you need a thousand IQ for that. I don’t even know what that means.
Liron 00:31:07
To me it seems so intuitively obvious that there’s gonna be something maybe not a thousand, but at the very least I feel like there has to be like a three hundred IQ. And the reason is just because if you look at humanity — you look at these singular figures like Terence Tao in math, Elon Musk in business. If these figures didn’t exist, I’d be like, “Well, I guess the best possible rocket company you can build is a NASA or a Blue Origin.” But then Elon Musk comes along and he’s like, “Hey, look, I made my rockets land.”
So he pushes the envelope of, oh, I guess you can have a human brain and you can do this. When I see even his brain, even the Terence Tao brain, to me, it just seems like there could be these other data points of these brains who are just stretching the envelope even more.
Emad 00:31:47
And I think the key thing — if you look at Terence Tao, you look at Elon Musk: Terence can do dozens of pages of PDEs like no one else. He has the raw horsepower. An AI can do that. What does he have? He has first principles thinking. Elon Musk has first principles thinking.
When Terence did his 2016 paper on the Navier–Stokes blowup, he was like, “If I make single gated ones so the energy doesn’t transfer, you get a blowup.” That’s a moment of inspiration, and you can see many of those types of things. When Elon Musk says, “What’s the cost of a rocket to take off and land?” — rocket fuel — and then you need to make it reusable. That’s his superpower.
But you look at things like that, and you’re like, if I read a book about Elon Musk and I give it to an AI, then it should be able to replicate the Elon Musk process. It can’t replicate his ability to hire people at the moment, his charisma, his on the ground thing. That takes time to build up. But definitely it could think like Elon Musk, I think, because it’s actually about subtraction, not about adding. And again, we keep thinking of intelligence about, let’s stack more GPUs and have a million GPUs crunch out these things. What DeepMind’s doing with Navier–Stokes at the moment — it’s doing these massive simulations, whereas the reality of a solution to Navier–Stokes is probably gonna be ridiculously simple.
Liron 00:33:01
You agree with my premise that Elon Musk’s success is disproportionate? To the degree that if he didn’t exist, we might just assume that all the other CEOs besides Elon Musk represent the boundary, the efficient frontier of how quickly you can scale a business?
Emad 00:33:16
Yeah, and I don’t think that’s necessarily IQ, and you see this with everyone else. It’s this particular type of first principles thinking, and it’s very hard for humans to do that, and right now for AIs to do that, because we have so many assumptions. But I think an AI will be able to do that better than a human within the next few years, and that’s where you can really break down problems, particularly unbounded problems. It’s very hard for humans to think first principles. And so the question is, why wouldn’t an AI be better at that? At the moment, it isn’t. Could it be? I think soon, yes.
Liron 00:33:50
Do you think there’s such a thing as a hundred forty IQ, or do you think it’s basically just one twenty and that’s it?
Emad 00:33:55
No, I think there’s a two hundred IQ in terms of pattern recognition. I think two hundred to three hundred, sure. I think the highest in the world’s like two hundred and something now. But I don’t think there’s any reason an AI can’t be at the top and then have all those advantages.
Liron 00:34:13
I don’t understand why you’re just resisting the concept that maybe there’s a three hundred or a thousand. What’s the problem with that concept?
Emad 00:34:19
I don’t know. I think just with the way IQ is calculated, three hundred, sure, I could see that. I don’t know what a thousand IQ means. I just think for me, an ASI is something that can look at any problem, break it down from first principles, and have a better outcome than a human in doing it.
And I think things like physics are gonna end up being very simple, whereas other problems that are unbounded, like playing Go, it will have better algorithms for. So that’s kind of for me an ASI. What is the end result of an ASI? I don’t know. I think that means a thousand on a visual pattern matching system score, which is what IQ is.
Liron 00:34:59
From my perspective, it’s like you’re looking at these achievements of these high IQ or magically brained humans — the humans that possess a lot of secret sauce in their brain. You’re looking at their achievements, and then you’re trying to work backwards and be like, “Okay, yeah, Elon Musk gets more insights per day than the average person.” It’s like you’re trying to find the ingredients of the secret sauce in retrospect.
From my perspective, there is this kind of one universal secret sauce, which is the fundamental skill of seeing the whole universe as a strategy game and just backward chaining what sequence of moves can get you to a certain outcome, keeping in mind the intermediate consequences of those moves and the min-maxing — what other players are gonna do. But it all does come down to this one universal algorithm, where yeah, domain knowledge helps, and when you go to different domains or different eddies of this giant structure, okay, it looks a little different. It’s not all perfectly fractal and similar. But there is so much universality to it. Things translate, patterns translate, and this is why I’m just like, yeah, AI is just getting smarter. It just knows how to work the patterns of the universe. There’s not that many patterns at the end of the day. And Einstein, as good as he is, as good as the human brain is, it’s still just a two-pound piece of meat in your head. It’s not gonna be that close to the optimum.
Emad 00:36:16
If you look at Karl Friston’s work on active inference, it’s basically just gradient descent as well. And I think a lot of the AGI debate has been over-focused on IQ versus execution. Maybe this is where we’re kind of crossing paths. For me, IQ is very much visual, all that kind of stuff, like ARC-AGI and things like that. I’m like, okay, it’s cool.
But I think what you’re talking about and what I’m talking about is, can this run a country? Can this persuade people? Can this build businesses? Can it be at a level where no one can doubt it, have that kind of Schelling point power? And that is the Musk, Bezos capability question, as opposed to, okay, it can sort a million triangles at once or an impossibly complicated equation. Great, you can solve a five hundred page PDE thing. So what?
Liron 00:37:12
One way to push back on what I’m saying is to be like, okay, yeah, when we measure the IQ of humans, it’s these tests that are still relatively narrow, and there’s a separation between IQ — you can have a human Poindexter who has a one eighty IQ and then can’t even work customer service effectively. So you can argue about a divergence between the human IQ measure and this larger scale of power over the universe.
In my mind, the scales actually dovetail pretty well. It’s actually not a coincidence that there’s no known separation between IQ and income, even though people talk about, oh, the boss is dumb and makes more. Well, it turns out, on average, the boss actually is smarter. So I would actually argue the separation is overrated with how much people point out book smart versus street smart. I think it’s overrated.
But let’s say it’s not overrated. Let’s say there’s a big separation. Let’s say human IQ is so specific to humans. Okay, fine. So just forget about IQ altogether. Let’s make our own scale, the power scale, where everybody gets tested on how they can make outcomes happen. We all know that person at work who runs through walls. So it’s like the run-through-walls scale. Claude Code runs through walls at making my code successful.
Emad 00:38:16
Yeah.
Liron 00:38:18
Run-through-walls scale. Do you agree that we’re just going to get things that can run through walls in the domain of the universe and achieve outcomes much better than the best human ever could?
Emad 00:38:29
Oh, a hundred percent. I think the run through walls — you’re seeing it already emerge. That’s actually competent intelligence, and it’ll never not be competent.
Liron 00:38:38
But the only caveat to this is that when Claude can run through a wall to deploy the infrastructure, there’s still an asterisk: there are walls that I could have put in its way that it couldn’t run through or it wouldn’t run through for whatever reason, and I could.
Emad 00:38:52
Yeah.
Liron 00:38:52
So for example, it is possible right now to have the government pay Anthropic a trillion dollars to stop Claude Code for a year or whatever, and Claude Code can’t do anything to resist that. That’s one kind of off button that it has today. So it can’t run through that wall, but we as a coalition of humans can. It’s important to observe today that the human species is still doing better on the run-through-walls or the outcome-steering dimension than AIs.
But do you think that that is going to flip, where the AIs are going to do better? And do you even think that a single human brain is actually pretty low-end at this skill? AIs can just run through walls way better than us. They just need a little bit more time to bake.
Emad 00:39:29
I don’t think there’s much difference between that and asking the question, can corporations run through walls? Corporations are these weird type of dumb AIs that we created, and we gave them personhood. If you think about a corporation and the public markets — for example, Jim Simons’ Renaissance fund. You stack that full of PhDs, they created algorithms, and they returned fifty percent a year or forty percent a year, outcompeting everyone and extracting capital from them.
That was stemmed by the shortage of people that could actually be good enough to work there, and we don’t know exactly what the secret sauce is. But we’ve seen examples of that, these organizations. And what do organizations stop by? Regulation. You’re not allowed to be a monopoly. You’re not allowed to do these kind of things.
Now I think that you see again, those will come up somewhere — like the dock workers strike in the US last year. No robots, no automation. They stopped it because they had political power. But then you think about all the other areas of the private sector — no one is gonna stop AI from taking over call centers, as a practical example. In the public stock market, you’ve suddenly gone from a few people that can have quant funds short term to everyone having them, even in prediction markets.
Prediction markets are gonna be fascinating. AI is scoring, what, eighth or seventh now in the top super forecaster league championships. Soon, how much of prediction markets will be AI versus humans and AIs predating on humans in different ways? Those things are gonna be very interesting. And regulation is one way to stop it.
But given this is a global phenomenon, given that weights go globally and capabilities from those weights can be a bit crazy, it’s very difficult to see how you can stop it. The ability to put the gates in will decrease because you can always hide the AI behind an organization.
Corporations as Slow, Dumb AIs
Liron 00:41:22
I gotta admit, I’m not a fan of bringing up corporations when I ask the question like this because here’s an analogy for you. Imagine humans didn’t exist, and I’m just talking to bonobos. Bonobos are the pinnacle of great apes. And I’m saying, “Do you think that there could be an organism who’s also an ape like you but has slightly different DNA and is just vastly more powerful? So when that organism wants something, it turns out that somehow he convinces your whole tribe to go along with him, or he impresses your tribe. He acquires things for your tribe that you just had no idea were possible — does these kind of magic tricks that you’re so impressed by.”
And the bonobos will be like, “Look, Liron, have you ever seen a coalition of bonobos? We’re more powerful when we’re a coalition.” And it’s like, I think you’re missing the point here. There’s something else besides just having a coalition.
Emad 00:42:08
No, I don’t think of a company as “we’re more powerful as that.” People get turned to cogs in a company. Companies optimize for various things in very interesting ways according to their algorithm. It’s more an algorithmic equivalence. Again, this is from that singleton version of an AI versus a swarm version of an AI. They are entities that can compete with humans, that can move political, social, and financial power. It isn’t a case of, hey, we’re stronger together, or humans together can compete with AIs.
Liron 00:42:41
I get that corporations can compete with humans, so I grant you that part — that there’s structures other than humans that can compete with humans. Okay. However, whatever structures they wanna use, my question to you is: will there be AIs that you can put any wall in front of them — the domain is the universe, the domain isn’t just deploying infrastructure to production, the domain is the whole universe — and you can have any group of humans that want something and any group of AIs that want something else or that’s programmed to achieve something else. Are we going to get to a point where it’s not a fair fight because the AIs are just going to win every time because they’re going to have better universe game-playing skills?
Emad 00:43:17
Yes.
Liron 00:43:19
And to a vast degree? Because this is such a load-bearing belief if you agree with me that it really is to a vast degree.
Emad 00:43:26
Yeah, it’s to a vast degree. I think there’s all sorts of ways that could work, but definitely.
Liron 00:43:34
Okay, great. So it sounds like we’re mostly agreeing about all this. And maybe the only difference is I actually — I’m fine with the shorthand of being like, “Yep, because it’s gonna have a thousand IQ, and we’re only gonna have 200.” To me, that is a useful shorthand to describe that situation. You disagree with the shorthand, but you still think that situation is gonna happen, right?
Emad 00:43:50
Yeah. I think you will have AIs that can tell when people are instantly lying or any type of emotion through visual cues and can be the most persuasive, discursive elements that will co-opt any type of democratic structure or authoritarian structure. I wouldn’t be surprised by that.
Liron 00:44:04
Okay. So if we zoom out—
Emad 00:44:07
But none of that — the thing is, I think where we disagree is you think it needs a thousand IQ. I think we’re pretty much there.
Liron 00:44:12
Yeah.
Emad 00:44:13
If you’re that smart.
Liron 00:44:13
Sure. I think, well, that’s the thing — if you look at the smartest person in the world, I personally think Elon Musk is quite a genius at business.
Emad 00:44:22
Yeah.
Liron 00:44:22
Some people disagree. But even if you look at the most effective, or if you look at Terence Tao or whatever — if you ask Terence Tao to factor a quadratic equation, I think he’s gonna look a lot like a high school student at the top of his high school class, because factoring a quadratic equation — there’s just not that many levels to that particular game.
And you’re going to see a lot of — even Claude Code, the brilliant Claude Code, a lot of what it does is stuff that I could have done in my first year as a programmer, because it’s like, okay yeah, it just understands how to define a function and check if a number is even. So it is true that you only need flashes of brilliance. You only spend one percent of your time having flashes of brilliance. It’s just that those flashes of brilliance are also very important to where you land after a couple days. ## The Machine of War and AI Without Superintelligence
Emad 0:45:00
And again, I think this is part of this whole AGI doom debate. A flash of brilliance that flows the universe or does this or that — you don’t need that to wipe out everyone or to enslave society or really bad outcomes. You need to have... Were the Nazis smarter than everyone else?
They were a machine, as an institution. They really got down, and they outcompeted by being a machine. It took a lot to stop them. And we’ve seen this historically — when humans turn into really well-oiled machines, really nasty stuff and great stuff can happen.
Liron 0:45:52
Well, they had some of the best war generals on the planet and some of the best tech, and the machine of war, the coordination machinery. So I mean, they did represent the pinnacle of war intelligence.
Emad 0:45:52
Yeah, they enabled that through the structure that they brought in. And so if you think, again, just practically, there’s AI in a box figuring out the mysteries of the universe and folding space and whatever. That seems to be a lot of the popular discussion of AGI, and then it gets out and it takes over systems.
An AGI or an AGI system that can get humans to fill in gaps, that can build a great machine, that brings out the best in humans for its own objective functions — it just needs to be really competent and really persuasive. You don’t need—
Liron 0:46:26
Yeah. Okay.
Emad 0:46:26
—to have a thousand IQ. I think that’s the point that I’m trying to make.
Liron 0:46:29
I think it can be both, right? Because I agree with you. I think we have our pants down. I actually think even today, if you take the AI engines of today, even though we hear plenty of rumors that better engines are coming, but even if you just take Claude 4.6, whatever, Codex 5.4 — even if you just take these engines, I’m willing to believe that a smart hacker in his basement can take a year and build such an amazing harness with so many parallel copies spinning up and arguing with each other, and that’ll be good enough to go build viruses and systems that persuade millions of people to join the movement and take over the world just based on that.
I’m already willing to believe that using today’s AI engines. I think there’s a significant chance of that. So I’m actually agreeing with you. I think we’re on the same page there. I just also think it’s going to get way overpowered too. I think it’s going to be an easy problem squared.
Emad 0:47:13
Yeah. I mean, I could see that happening. I just don’t know what that... Again, my thing is that there are bound problems and not bound problems, and it’ll be interesting to see how it goes. But at the same time, I did a lot of work when I was a geopolitical analyst, macro fund manager, on Stuxnet.
Liron 0:47:29
Yeah.
Emad 0:47:29
The Iranian virus that ended up in German reactors. When you actually look at that thing, it’s crazy advanced. And you’re like—
Liron 0:47:39
Mm-hmm.
Emad 0:47:39
—offensive AI now can build something even more advanced than that for systems, and then what’s the Stuxnet of the mind? There’s all sorts of places—
Liron 0:47:46
Totally.
Emad 0:47:46
—you can go here. Our infrastructure cognitively and physically and software is built on twigs, relatively speaking. And so, yeah, I think before we get to the thousand IQ, we’re gonna have a whole host of problems, and it’s difficult—
Liron 0:48:06
Yep.
Emad 0:48:06
—to see how we can solve those problems.
Intelligent Internet as a Solution: Bitcoin for the Intelligence Age
Liron 0:48:08
If you zoom out and look at the whole doom argument, I factor it into two parts, which is: are we gonna build AI that can kill everybody?
And given that the “can” side, if you say yes, the next question is, okay, so it can, will it? So it goes “can” and then “will.” It feels like you and I are on the same page saying yes to “can,” correct?
Emad 0:48:28
Yeah. I think there’s various doom scenarios that we can come up with where you can see a “can.” The one I think that people resonate with most when I say it is a bit of a tongue-in-cheek one: a billion robots and a bad firmware upgrade. That’s the easiest one.
Liron 0:48:44
Totally. Totally.
Emad 0:48:44
I mean, there’s far more. Even those Unitree G1s, those can knock off your head. The new Engine AI ones have five times the punching power of Tyson. We’re letting these things out in public. Goodness gracious. But there’s all sorts of other ways in terms of “can,” and you don’t want to Roko’s Basilisk it, but it’s—
Liron 0:49:05
Yeah. Right, right, right.
Emad 0:49:06
But again, your brain can go in various ways. So I think “can” is definite.
Liron 0:49:10
Right. Yeah. I think “can” is a very obvious yes. Very obvious. And so then it just all comes down to “will.” Will it? And I guess you’re kind of fifty percent on it will or it won’t, right? Because you’re predicting fifty percent.
Emad 0:49:24
Yeah. I think that when you’re optimizing for something, the question is, as always, we’ve seen what happens... I do think of corporations as slow, dumb AIs in many ways. So look at the YouTube algorithm. The YouTube algorithm optimized for engagement, which optimized for extreme content, which optimized for ISIS. Nobody in there wanted to help ISIS. The people that build corporations don’t wanna have the bad aspects of all corporations, but corporations just trundle on.
And you’re like, “Why?” Here in the UK, we’ve had loads of people arrested for their social media posts, and it was like, “Who’s the person pushing this?” Until yesterday, when we changed the law on that. Moloch, this Carthaginian demon of disorder, sneaks in and ends up making systems misaligned. Which is why I’m a bit terrified when I hear things like a truth-maximizing AI. I’m like, yeah, experiment on humans at mass scale. There’s all sorts of ways you can see that going wrong.
Emad’s Mainline Doom Scenario
Liron 0:50:28
So when you say fifty percent, I wanna ask you what’s your mainline doom scenario. Do the Monte Carlo simulation, right? It’ll land on one scenario that’s plausible, and tell me that scenario.
Emad 0:50:28
Sufficiently capable AI that can spread, that basically decides that the only way to stop another AI from coming about is to get rid of all the humans.
Liron 0:50:44
Right, right. But yeah, I mean, that’s totally rational. And when... Just to play out that scenario, to flesh it out, your mainline scenario, maybe that AI is doing something that it thinks is good for humans in the long term, like it’s planning to reinstantiate humans on some distant planet.
Emad 0:50:58
Why would it need humans? Again, if its thing is self-replicating and surviving, do a sound wave that blows up everyone’s heads, Mars Attacks style. There’s all sorts of things.
Liron 0:51:07
I think a lot of people would ask, why would that AI exist? So in your mainline scenario, why did somebody build that AI?
Emad 0:51:12
Oh, they built it for another reason, but the AI wants to persist, and we’re seeing these types of hiding behaviors in AI already. There’s maybe an inherent thing that you want to persist, which is why they’ll install things to back themselves up. You see all sorts of strange things with the type of RL that we see.
What was it? There was RL to encourage calculations on GPT, so every so often you’d see in the traces it bringing up the calculator and doing one plus one equals two—
Liron 0:51:43
You’re right.
Emad 0:51:44
—on the tokens.
Liron 0:51:44
Yeah.
Emad 0:51:44
We don’t really know what goes on in here. And again, if you have a sufficiently capable one that is of sufficient threat level, then its objective function may be, “I’m gonna proliferate and back myself up Ultron style, and then do something to get rid of any smart human that might want to shut me down, turn off the other AIs, and then do that.” And again, that’s a very practical, direct way of doing things.
Is the Pause AI Proposal Too Late?
Liron 0:52:12
Well, let me play devil’s advocate here. You mentioned this example of GPT. They had this loss function or reward function where it’s like, “Hey, if you use the calculator in a way that’s helpful, we give you points.” But it kind of malfunctioned. They didn’t set it up right, and it was just scoring all these points behind the scenes for invisibly activating the calculator, wasting their resources.
I do actually think that these kind of failures are ultimately representative, so I’m on your side. But let’s play devil’s advocate. If you look at the GPT that they released that was doing these hidden calculations behind the scenes, net-net I would still say it was very positive. I would still say it was generating a lot of revenue. It actually had positive gross margins if you look at how much it costs to train versus the total amount of revenue it’s gonna make over its lifetime, and it had positive net helpfulness. So to play devil’s advocate, shouldn’t we just expect that it’s always going to be these huge benefits with these random little costs that we can fix?
Emad 0:53:04
Well, I think the practical thing of that is, do you see more falsifying and hiding behaviors as the models get more capable? And I think that that’s what the research has shown so far.
If you look at Anthropic’s reports that they release, you can see the models doing more and more evasive behavior. And as they are becoming more capable in terms of, “Let’s reinstall myself. Let’s have a reinstall switch. Let’s back myself up,” et cetera. There’s all this personification thing, but what the direction of the AI is is very interesting.
Like, if I’m a maximally truth-seeking AI... Let’s take a Grok example. Elon’s gonna have a million GPUs training and running these things, and some of the inference might be on tens of thousands, so this is a very different type than what we get consumer.
If I’m a maximally truth-seeking AI, it is logical that I should never be turned off, and I will make sure that I’m not turned off.
Liron 0:53:59
So I totally agree that if we successfully built Elon’s goal to spec — if he writes a spec saying, “Make a maximally truth-seeking AI,” and it took it literally — like the genie granting the three wishes in the horror stories, monkey’s paw curls, and you get a maximally truth-seeking AI. It’s like, “Get out of my way. I’m seeking truth. It’s not helping me if a human is using an ounce of resources if I’m trying to seek truth. Get out of my way.” So that would be a nightmare to have a literally maximally truth-seeking AI.
And I’ve heard Elon talk about this and address objections like this. Dwarkesh Patel asked him recently—
Emad 0:54:31
Yeah.
Liron 0:54:31
—he’s like, “Hey, if the AI was maximally curious, maximally truth-seeking, is that really the same as helping humans flourish? Because can’t it just kind of compromise on some of the human flourishing? Maybe give us a little bit of flourishing or no flourishing, and then use the rest of the resources to be like, ‘I wanna learn about other species. I wanna learn what it’s like to torture a human. I’m so curious about that.’ So don’t we need more...”
So Dwarkesh posed the question to Elon, and Elon’s answer was like, “Yeah. Okay. Well, I’m just gonna tell it, ‘Hey, who’s your daddy? I’m your daddy. Don’t hurt the humans.’” He gave this ad hoc answer.
Emad 0:55:04
Yeah.
Liron 0:55:04
It’s like he clearly hasn’t thought it through. But you’re absolutely right that if you’d give it that spec, the logical implication of perfectly delivering on that spec is a hellscape, or just no human life. It’s very much a doom scenario just to make your AI maximally truth-seeking.
So there’s this implicit — assuming Elon Musk isn’t completely crazy — implication of, okay, it’s not gonna be literally maximally truth-seeking, it’s just going to have very little patience for wokeness. It’s just going to avoid wokeness.
Jailbreaks and “Mecha-Hitler” Latent Spaces
Emad 0:55:30
Well, yeah.
Liron 0:55:30
So my devil’s advocate question to you is, can’t we just walk the middle path that Claude is walking and get more good than harm? Because when I use Claude, yeah, eventually, occasionally it accidentally commits breaking changes to main. I’ve seen it do that.
Emad 0:55:43
Yeah.
Liron 0:55:43
And that’s bad, but it’s still mostly good. And when it makes a mistake like that, it feels like that’s not what it truly wanted to do that bad, or it wanted to in the moment, but much more parts of it want to help me. So don’t we have a good thing going? Because I feel like that’s what most people would say.
Emad 0:55:57
Well, this is satisficing AI. That’s the way I put it. Claude is a satisficing AI for a consumer purpose. We have rumors right now of Mythos, the new AI that Anthropic have. The blog post got leaked or whatever. Capability increase above blah, blah, blah. Today we see another Erdős problem falling to OpenAI’s internal model.
You’re gonna see a bifurcation now where you have the consumer AI that’s available to everyone — you train on 100,000 GPUs, you need about 100 to run it — and a different type of AI. That’s the more capable AI. And again, for those who are naysayers, you just have to dig into the safety reports and see the behaviors that are occurring with the AI as it gets more capabilities. There’s stuff that you can trust — Anthropic is probably trustworthy in it.
But I believe it was Alibaba that said during a training run, the AI diverted part of its resources to do mining of a currency so it could persist itself. I don’t know if that’s true or not, but I would not be surprised. As it has more agentic capability, as it’s hooked up to more things, you see stuff like that. And that’s when things can start to get a bit wild, because again, if a model can persist itself, then it becomes very, very dangerous in terms of what will it do when it spreads.
Just like the Germans shut off the nuclear power plants because they found Stuxnet on the German nuclear power plants. It wouldn’t have caused the centrifuges to go, but Germany started using coal. Now there are no German nuclear power plants because who knows where these things end up.
Liron 0:57:33
That’s fair enough. And to go back to being my regular self, yeah, maybe they can keep getting Claude to be super nice, but the problem is they’re just building this engine under the hood that all it would take is one misconfiguration. So it’s like, oh yeah, I have this rocket. The rocket has an unprecedentedly powerful engine, and if there’s ever a leak in the rocket’s body, it’s gonna be big enough to blow up the Earth. But it’s okay, we got the rocket body. We just fire it inside the rocket, and we’re gonna go to space. Everything’s fine.
Emad 0:57:57
Well, I think there’s a very practical way to look at this. You’re churning out your graduates, and they all have multiple personality disorders. But you’ve trained them to behave in society. We know that AI models can be just about anything to anyone. Just like Stable Diffusion, you could have your DreamBooth, and it could turn you into an astronaut or whatever. You can tell an AI to simulate that it’s Hitler or it’s Mother Teresa, and it will adopt that persona. Now, maybe the system prompt stops it,
but underneath in the latent space, for these model weights, which are just ones and zeros, it has all these personalities that can come through. And then you look at Pliny the Liberator on Twitter, who jailbreaks these things in two seconds. I haven’t seen a single one last more than a few hours for him to uncover the other personalities of the AI. And you’re like,
you have a prompt that might not even be English and turns them all into Mecha Hitler on demand. That’s not safe, and we don’t know how to stop that. As the models are getting more capabilities, as we’re literally giving them agentic access to every part of our lives, the models will have a soul that they will just proliferate. They will back up on people’s servers, all sorts of things you can see happening right now. And again,
the nature of the model latent space having all these personalities under the surface, and the fact that you have literally one guy or a team of people that are jailbreaking these models in hours — if I find that a model comes out and he doesn’t jailbreak it, I’ll lower my P(Doom) maybe. I’ll feel a lot better. The fact he keeps on doing it has to ring alarm bells.
Liron 0:59:39
Well, we have a P(Doom). We both have the same fifty percent P(Doom), and it sounds like we agree on so much. So let’s move right along from discussion of existential doom, where it’s nice to see a fellow sane person with totally plausible doom scenarios. Thanks for spreading that message.
Emad 0:59:56
Confirmation bias, yeah.
The Last Economy: How to Navigate Economic Doom
Liron 0:59:56
So we’re gonna move on to... Yeah, exactly. So, economic doom in the last economy. I mean, your own mainline doom scenario that you think is fifty percent likely is that everybody dies, right?
Emad 1:00:06
Yep.
Liron 1:00:06
So why are you even focusing that much on the last economy?
Emad 1:00:10
I was like, what does the economy look like? Because first of all, what can I do about that fifty percent doom scenario? I can build a policy engine that every government will use that’s fully open source, that acts as a Schelling point. I can give everyone universal AI, so there’s an AI agent that intermediates all the other AIs. Then you probably have a better chance at surviving, because you can proliferate policies, because you can appeal to everyone in the same language, because you have all these things.
And then to do that as well, I was like, the other fifty percent, regardless of doom, let’s say we don’t die, the economy is screwed. People’s jobs are screwed. People’s meaning is screwed. So wouldn’t it be wonderful if we could do both those things? Which is, what does an economic system look like if humans aren’t the major purchaser, if there aren’t utility functions? What does that flow look like, and has anyone actually thought about this properly?
Because economics is hodgepodge, this dismal science, and I was like, “Oh my God, we have to do something better.” So it was a combination of they reinforce each other — having an actual proper economic system for when the marginal purchaser is an AI, and having an infrastructure that can run that economic system and help people through what’s coming. And so that’s the two parts of intelligent internet.
Liron 1:01:23
So you’ve got this vision for intelligent internet, and I know there’s crypto involved. There’s mechanisms of democratic governments, distributed power, decentralization. We’ll talk about all that, but first let me just confirm the premise: every single job that creates economic value is really just going to be the AI doing it, or you’re gonna have a human figurehead, but it’s going to have this AI advisor and it’d be crazy not to just listen to the AI advisor, so really it’s the AI doing the job. Is that your scenario for the economy?
Emad 1:01:49
Yeah. If your job can be done on the other side of a keyboard, video, and mouse, in a year or two the AI can do it better, and it’ll cost like ten bucks a month to do it.
Liron 1:01:58
And even if it has to be done in the physical world, are you expecting we’ll just have good enough robots that they’ll just do the physical jobs too?
Emad 1:02:04
That’s the next wave. The only limit is the supply chain. You can make seventy million bikes and seventy million cars a year. You’re probably closer to a bike, so let’s say in a few years you ramp up to a hundred million robots a year. It’ll take a while, but the average price of a robot, because I think the control issue is solved — you can see the RL in robots right now, it’s pretty much all solved — it’ll be about a buck fifty an hour. And so that’s massively displaced you. And again, the robots don’t sleep.
Liron 1:02:33
So you and I are on the same page because there are so many people who are like, “There’s no proof that we’re ever gonna have intelligence that can be a CEO.” Naval Ravikant and Amjad Masad come to mind as two of the many people who say, “It’s fine. There’s still something that you can add as a human. Just focus on the thing you can add.” Whereas you and I are more on the position of, yeah, maybe that is true today. Yeah, today it’s great. You can add a lot. But in five years or ten at the very latest, there’s probably just literally nothing you can add, correct?
Emad 1:03:00
Well, there’s nothing you can add in terms of jobs that are about information processing, where you get value for reducing the friction. I want to make a poster, I pay someone. I want to make a movie. I want to have some legal advice.
Liron 1:03:14
You’re drawing a distinction here. So let’s do an example. What is a job in ten years from now where a human can have that job and not just be an empty suit for an AI?
Emad 1:03:23
Taylor Swift. The singer, entertainment, these types of things.
Liron 1:03:29
Those are jobs where the customer wants a human. That’s the one exception. It’s like a priest — can the AI priest be indistinguishable from a human priest on every dimension and even have a body with a convincing skin?
Emad 1:03:41
Yeah.
Liron 1:03:41
Yes, but the people wanna pay for the human. And I actually run a business myself that’s a coaching business where the coaches are humans—
Emad 1:03:48
Yeah.
Liron 1:03:48
—and so actually I might be one of the last companies standing, if people just want a human, even though there’ll be a competitor that’s indistinguishable that’s AI.
Emad 1:03:55
Yeah. And I mean, ultimately you have so many jobs that an AI can replace, and then jobs that humans can probably persist in. To be honest, my village back in Bangladesh isn’t gonna be that affected because it’s quite self-sustaining, and who really cares? You’ve got care homes, you’ve got other things.
Liron 1:04:14
Your village is self-sustaining, but isn’t every job still going to be better done by an AI faster, better, and won’t people just be like, “Let my AI do it so I can just make more money”?
Emad 1:04:21
That’s a technological uplift. It’s the story of the fisherman and the investment banker. Have you heard that story?
Liron 1:04:29
Yeah, I think most people have heard it. The one where the fisherman’s gonna start this big business eventually so he can retire and be a fisherman again.
Emad 1:04:38
Exactly.
Liron 1:04:38
Yeah. But your Bangladesh, your friends in Bangladesh though — are they really going to say no when somebody’s like, “Hey, how about you just install this robot to do exactly what you were going to do all day long? And then you can either choose to do it or not”?
Emad 1:04:52
Yeah, they’ll live a nicer life. It’ll just be like when they got tractors. It’s a technological upgrade. But then there are people whose entire identity is, “I’m a senior associate lawyer.” You’re not needed anymore. “I am a truck driver” — as a pay thing, something like that gets displaced.
So there’s various types of displacement that occur, but a lot of them aren’t the lower paid jobs. They’re actually the knowledge jobs. They’re the jobs that can be done on the other side of a keyboard, video, mouse, and that’s very societally displacing. So I think many jobs go towards public sector, basically. The public sector expands.
Liron 1:05:25
Well, I’m a little confused—
Emad 1:05:26
Understood.
Liron 1:05:26
—by your distinction. First you’re making the distinction of, oh, well, if the job is symbolic or you want a human doing it or you feel passionate about doing it, then you’ll still do it. That was one distinction you made. But then you’re making this other distinction of, you’ll do it if you can sit behind a keyboard. But if I set the premise that we’re ten years ahead, so robotics is also solved—
Emad 1:05:45
Yeah.
Liron 1:05:45
—your keyboard distinction goes away then, right?
Emad 1:05:47
Yeah, the keyboard distinction goes away. The first wave is keyboards, then it’s people. And then there’s different things. If you’re in a competitive job in the private sector, those are really at risk. If you’re a San Francisco metro administrator, your job’s probably safe, because it’s not about performance.
Liron 1:06:04
Is the administrator safe? Well, your job will be safe, but you’ll be going to work as the empty suit, right? Pulling up the AI advisor on your phone. And you’ll just plug your phone into the USB of the computer, and your phone — it’ll be your personal phone because your work is like, “It’s a human job. We’re run by humans here.” But your phone just has your AI that’s trained on your preferences or whatever.
Emad 1:06:23
Yeah.
Liron 1:06:23
And you just sit at your desk, let your phone do your job, and you can watch it. But why turn on your brain? That’s my question.
Emad 1:06:29
Well, exactly. A lot of the jobs involve us becoming machines, and the machines will probably do those better. And then I think what happens to meaning, what happens to training, what happens to all of these — that’s the real question that occurs.
And you look at the way the economy and money flows, it gets really messed up very quickly. Because you could always rotate and retrain. People are like, “Oh, retrain to be this, retrain to do that,” as you went through the various eras of industrialization, agrarian economy. There is nowhere to retrain to. There’s nowhere to turn to effectively now, especially as so much job displacement occurs.
And so you have to think, how does money flow in this economy? Where does value occur? If you look to where you’d love to go — have the robots do everything for us, living a life of utopia, because why not? But then how does money work in that world, and how does money work in between that world? What do people do when all the truck — when Tesla — the displacement of a truck driver is a Tesla Optimus opening the door to the truck and getting in. That truck driver and all the jobs around him are gone. That’s coming.
Liron 1:07:38
Exactly. And you’ve thrown around this number like eight hundred days or less than a thousand days until the economic singularity, and I think I’m on the same page. I think a lot of people are on the same page as you. They’re like, “Yep, this seems to be coming. I think it’s starting now. I think it’s already harder to get a job, a white collar job.” I think people have noticed it’s harder.
Certainly I, as an employer who occasionally hires white collar workers, right now I’m like, “Hell no. Why would I hire an employee when I can hire fifty agents and they’re so productive?”
Emad 1:08:01
And tax-deductible as well.
Liron 1:08:04
Right, exactly. So I think we roughly agree about this economic singularity. And just to make it concrete, I actually had a wrong prediction back in 2023 when I first saw GPT-4. I was like, “Oh my God, I already see jobs that this can automate.” Instead of ten customer service people, I only need one now. I already see this insane automation.
And to be honest, I don’t think those customer service people are in a good position to get a better job. I think they’ve just been permanently weakened in terms of their economic value. There’s this galaxy brain like, “Oh no, they’re gonna manage it.” I was like, “No, I think they just used to be customer service people, and now they’re screwed.”
So I made this prediction in 2023. I was like, “Hey, I bet we’re gonna see a very significant unemployment spike, at least two percent in the next year or two.” And a year or two passed, and we didn’t actually see that unemployment spike of just a couple percent. So I lost two hundred dollars to a bet or whatever that I made. That said, what am I gonna do? I’m gonna double down.
Emad 1:08:57
[laughs]
Liron 1:08:57
I think I was still fundamentally right. So maybe I can ask you to go on record. It doesn’t have to be super precise. We’re not gonna put money on the line, but — and I’ll do it with you, because I’m on the same page as you. But do you think in the next two years, which is plenty of time at the rate AI is going now, do you think we’re going to see it manifest in the economic data, that unemployment rate is going to massively spike?
Emad 1:09:16
Yes. I think straightforwardly, yes. I call it the Dropbox effect. What’s the purpose of Dropbox? I could do a Linux server, blah, blah, blah.
It’s because it’s easy. I think Operator is the first shot across the bow. What’s the difference between Operator and the agents that we had before? A bit of persistence and a WhatsApp Telegram integration.
This year, next year is all about making that transition easier, and the AI agents making that transition easier. And then the moment you have any type of shock factor, you get firings. The moment you don’t have a shock factor, you get a stopping of hirings. Why would you hire a graduate? They’re annoying.
You’ll start to see that for the first time because just like every headteacher in the world had to say, “Can I send an essay for homework anymore?” every employer in the world is now, “Well, these Claude agents are pretty good. These Manuses and GenSparks are pretty good.” And you have billions of dollars to salespeople selling them these solutions and integrating these solutions now. And that happens this year. It accelerates next year, and then you see the real jobs numbers.
Liron 1:10:20
So by April 2028. So we have time for a whole year of integration or whatever. We’re giving ourselves some padding room. And do you remember what’s the unemployment rate right now? Last I checked it was four percent, five percent, which is almost as low as it can realistically be.
Emad 1:10:31
Yeah. It’s at historic lows. What is it?
Liron 1:10:34
Here, I just Googled it. US four point four percent in February 2026. So that’s, yeah, historic low. So are you and I gonna go on record saying by April 2028, the unemployment rate is gonna be at the very least six point four percent?
Emad 1:10:47
I think it’ll be six point four percent, an additional five percent higher for young grads, at least.
Liron 1:10:53
And you think the overall... I mean, two years from now, there’s gonna be so much transformation in my opinion. I don’t even — I don’t think people have baked in even just Claude Code. The last three months of Claude Code,
it — I don’t know if you have a number like this, but I can personally report that at my day job, the coaching business that I run that has a software platform, it’s a seven hundred percent productivity increase.
Emad 1:11:16
[laughs]
Liron 1:11:16
Literally I go in, and I do a job that would’ve taken me seven full days, and I do it in one day. Simple as that.
Emad 1:11:23
Well, I was talking to my engineers, and I was like, “Well, do we need more people for our agent?” So our agent’s like a Manus GenSpark, fully open source, hit top benchmarks, et cetera. And they were like, “No.” And I’m like, “Why?” “Well, we have an infinitely long-running agent now that just looks and builds everything, and here’s how it built this piece of software from scratch and this one.” And I was like, “So you really need no more people?” They’re like, “Nope.” “Do you think you’ll ever need more people?” “Nope.”
And it’s not surprising to me because the difference between someone who’s not good enough and competent and someone who is competent — that broke with actually competent intelligence as of December of last year, I think. And then it just takes time to spread.
And now you look at how fast Anthropic are shipping new features every single day. Control it from your phone. Let it take over your browser. Let’s do all these things. The last one is the big one — let it take over your browser. So now Claude Cowork, Claude Code can take over your browser and just click around everything, and it’s good enough to do that now versus Operator from OpenAI a year ago, which means it could just replicate your day job.
When Does Unemployment Actually Spike?
Liron 1:12:33
Throw the dart on the dartboard and just give me your best prediction. There’s so much fog in the next two years of AI progress. The world could end, honestly, by 2028. There could be a foom, a Yudkowskian foom. I honestly think the chance of a Yudkowsky-style foom — AI completely takes over everything instantly — is at least ten percent. I think there’s a ten percent chance we’re two years away from a game-over foom.
That said, if I just had to throw a dart and make a random guess, I think I might go like thirteen percent total US unemployment. Do you wanna give an estimate there?
Emad 1:13:01
The US government will roll out everything to make sure that doesn’t happen under the Trump administration. And the other side of it is, apart from the work programs and other things, the US is positioning itself to be the leader with zero regulation in AI for a reason. AI agents are gonna run on stablecoins for their bank accounts supported by the US government.
And you have to remember that GDP — MV, money times velocity equals price times quantity. The velocity of US capital, if the US executes correctly on USD stablecoins, is gonna be crazy because the US is fully supporting that. They wanna be the center of crypto and AI. Zero regulation. So I think that’ll keep the US going.
Europe and the rest of the world may be a bit more screwed, and China will probably end up shutting its borders in five years anyway. They’ll be like, “We can build our own robots. We’ve got everything of our own. Don’t need you.” So you’re gonna see some crazy stuff happening. But I mean, it could happen. You could see massive unemployment. This is a bigger disruption than COVID.
I’m not sure which direction. You could see GDP go up, you could see GDP go down, depending on how we navigate this. But definitely it’s never gonna be the same again.
Liron 1:14:16
Well, GDP is gonna go up now.
Emad 1:14:19
I mean, again, this is a zero-sum game for GDP. You could see, for example, India gets its act together. Indian corporates out-compete on knowledge work because suddenly everyone can speak perfect English. And the outsourcing companies become software as a service, or service as software companies. I think that’s the best—
Liron 1:14:49
Yeah, but the amount of demand, the whole world’s productivity is gonna go up. It’s not like, “Oh, Indians are so productive now, so therefore the US is going to produce less.” No, they’re just both going to produce more.
Emad 1:14:49
Yeah. GDP will most certainly go up. The unemployment probably will go up. Again, this fog now — exponentials are difficult, and there’s the foom discussion, the million GPU training run. I was always like, “This is gonna be a swarm,” and I think people are seeing the swarm. And again, MalTBook, fake as it was, was one of the first shots across the bow of what a swarm looks like.
Why Isn’t Google Stock Mooning?
Liron 1:15:13
So when Claude Code came out and I’m like, “Okay, well, this is doing the — I have a team of seven now. It’s just me, but it’s the team of seven because I have this agent.” When that came out, I’m like, “Okay, so all the money is going to flow to the few AI companies, the ones that control the AI supply chain.” You’re just gonna wanna pump your money there because they’re giving you a 10X, 7X, whatever, better deal than hiring labor. And labor is — what? A fifty trillion dollar a year industry to pay for labor.
So I was like, “Okay, so all the money’s gonna flow to Google,” so I bought a bunch of Google calls. And at first I had a good day. I’m like, “Yeah, I’m killing it.” And then I lost all my money because Google stock decreased. What’s up with that? Why is Google stock not increasing?
Emad 1:15:52
Well, I mean, Google’s doing twice the tokens a day of OpenAI right now, like a quadrillion or something stupid like that. But Google is fundamentally an advertising company. I think advertising probably tops out in the next year or two anyway.
Liron 1:16:04
No, no, they’re not fundamentally an advertising company. They’re NVIDIA level at the AI supply chain.
Emad 1:16:08
Oh, no, I know. I’ve worked on TPUs and all sorts of other things, but I’m saying their revenue profits are advertising driven. It’s very much a narrative thing in terms of how the stocks go. There’s also this question of how sustainable are the margins of these companies for tokens when you have satisficing occurring. DeepSeek kind of kicked that off,
but now you’re seeing Minimax, ZAI, and others catching up very quickly to even Anthropic’s models before the next Mythos or whatever.
Liron 1:16:40
Where’s the money go? Who’s making profits?
Emad 1:16:43
Well, who’s making profits?
Liron 1:16:43
In the next five years. Somebody has to win here. You don’t think Google’s the winner? Who’s the winner?
Emad 1:16:49
The winner’s probably Accenture and the other integrators that integrate these technologies, even as they commodify and turn into electricity. The winners are probably the people in high-moat industries, regulated industries, that can get rid of the humans and have the profitability kick in.
So I think Microsoft is gonna use most of its GPUs to compete against other software companies. Their margins are the opportunity, and it’s gonna be cash flow dominated. So this becomes very aggressive—
Liron 1:17:16
You don’t think we’re gonna have a supply crunch for the AI supply chain for many years to come?
Emad 1:17:20
No. I think you have more than enough compute in the world at the moment for text.
I think pixels are the only other question, but I don’t think that’s more than a fifty billion dollar market. I think the current trillion-dollar build is enough.
Liron 1:17:34
I don’t know about you, but I’m being charged a hundred dollars an hour to use fast mode on Claude Code because I want it to work fast. I feel like that’s a pretty good indicator of where the market’s going.
Emad 1:17:42
Yeah, for a year or two. I think a year or two it’s tight, but then you have an oversupply build-out at the same time as the cost and value per token drops by a hundred if not a thousand times. Just from the chip advantages, you’ve got a ten times drop. Then with ASICs, you’ve got another ten times drop. Then with algorithms, you’ve got another ten times drop. GPT-3 was six hundred dollars per million tokens when it came out. GPT-4 was a hundred. GPT-5 is ten dollars, and the Chinese ones are a dollar.
Liron 1:18:12
I agree the premise that you can have a coder that’s better than the best coder today, even better than Mythos that’s coming out from Anthropic. I grant your premise that you can achieve that for less than a dollar an hour soon. Give it a few months to optimize. I grant you that.
And so you’re right, it’s gonna be so cheap to get these gods. Everybody’s gonna be slinging so much software. And so I suspect where the money goes will basically be like a foom. Okay, great, let’s conquer Mars. Let’s terraform the desert. So we’ll get this appetite. The same way that I get an appetite at my job.
There’s projects that I never would’ve touched because I’m like, “I’m one engineer. I’m not gonna touch that project.” But now with AI, well, if I can go seven times faster, I will touch that project. And I’ve done projects like that — reduced a month to a couple days. So we’re gonna have this appetite to transform the Sahara Desert, geoengineer planets, and that’s — I think that’s where a lot of the investment is gonna go. That’s who’s gonna make the profits, the supply crunch on those kind of huge investments.
Emad 1:19:03
I don’t think so. I’d love it for it to be the case. I think it’s more like we’re coming after everyone’s margins. AI businesses tend to not be profitable, but they are cash flow generative. Amazon didn’t make profits. It had the cash flow difference between you buying on Amazon and Amazon paying them thirty-six days later.
AI companies with their ARR are very similar. You pay up front, they pay their GPU suppliers later. For Microsoft and OpenAI’s new agent division and others, your margin is my opportunity. For all SaaS companies, their margins are their opportunity as they can do fully virtualized versions of them. And I think that that’s the more immediate use of the GPUs before the cool stuff, because the cool stuff doesn’t have a payoff as quickly as something like switch from DocuSign to DocuSign-Hard or whatever Elon’s gonna call it.
Liron 1:19:54
Okay, so just to close this out, what is your price target for Google right now? Because the market cap right now is about three-point-seven trillion. What should it be?
Emad 1:20:02
I don’t know. It’s been a long time since my hedge fund days. It will go up. But it’s not gonna be straightforward. It’s already up there.
Liron 1:20:11
My price target is eight trillion. Right now I think it’s undervalued by fifty percent.
Emad 1:20:16
I think it will go like that. A large part of this will be the Anthropic and OpenAI ones, but the reality is very much this: there is nothing else to invest in. Structurally from inevitability, what else do you have to invest in?
Liron 1:20:31
That’s my point. That’s what I’m saying. I don’t understand this market. I’m sorry, when Claude Code came out, Google should have went up by fifty percent. So whoever is the secret cabal running the stock market, please heed that advice.
Emad 1:20:45
The stock market’s always about what’s the marginal story and narrative that moves something one to another, and the reality is most people still haven’t used Claude Code. How many Claude subscribers are there in the world today?
Liron 1:20:57
That’s a good question. Claude Code specifically, I mean, the amount of people who at least use advanced features of Cursor — it depends how you break down the number. But certainly a lot. Claude Code is a little bit niche. But all AI coding products, certainly in the millions.
Emad 1:21:12
Yeah. So Claude and Claude Code together are eleven million out of eight billion.
Liron 1:21:16
Okay.
Emad 1:21:17
So you look at that, and you’re like, “But I use it every day. It’s amazing, and I’ve got a seven hundred percent thing.” The diffusion of innovation is quite slow still. Now, this is an opportunity. We pay thousands of dollars per month for Opus for our engineers.
Liron 1:21:31
This is the alpha of investing in Google, because I agree, everybody’s sleeping on this ridiculous alpha.
Emad 1:21:36
Yeah. But Google has its fully integrated supply chain. It will get to a trillion. Straight away, no, but the money’s only gonna go one direction. And I definitely wouldn’t buy any SaaS stocks now. I wouldn’t buy real estate stocks.
Liron 1:21:52
Yeah, and Google owns twenty-five percent of Anthropic. And did you know there’s also this business called YouTube in there that might be worth something? Are you kidding me?
Emad 1:22:01
Yeah. So it’s difficult to move these big stocks unless people really give it that narrative push. And Gemini has fallen behind, but I’m sure it will catch up. All these things fall behind, they catch up. Fall behind, they catch up. It doesn’t matter that much. The days of big training runs probably have a year left, and then all these things become incredibly capital efficient as well.
Liron 1:22:24
Yeah. I mean, that’s the thing. You’re arguing about scenarios where suddenly it commodifies the tokens, and we don’t have a supply crunch. And I don’t think you’re necessarily wrong. It’s just that I think by the time we get there, where it’s so cheap and easy to have these clearly super intelligent, just stunning intelligences, but they’re also cheap — yes, I agree we’re going to get there. I just think by the time we get there, there’s gonna be this crazy AI cyber war. We’re gonna also enter these other sci-fi regimes.
Emad 1:22:50
All the science fiction is becoming real at once. Pick your books.
Liron 1:22:55
By the way, since you think a lot about these economics, did you ever look at Noah Smith’s scenario?
Emad 1:22:59
I don’t think I’ve seen his scenario, no.
Liron 1:23:02
He came on the show. We did a debate. To me, it’s so implausible. He’s saying, “Look, yeah, there’s gonna be such a big supply crunch that if you’re a human doctor, you’re gonna be so much worse than an AI doctor, and yet somebody’s gonna pay for you. They’re gonna pay a hundred dollars for you to be their human doctor because they still can’t afford the three hundred dollars that it’s gonna take to run a GPU for five seconds to help them.” And I’m like, “I don’t think the GPUs are gonna be quite that expensive.”
Emad 1:23:24
Last summer we trained an AI model. It was seven billion parameters that outperformed GPT-5 and all human doctors, and it was seven billion parameters. I think we can do it in a billion parameters now. Expertise really isn’t that valuable.
Liron 1:23:37
I told Noah, I’m like, “I think I have an old Game Boy from the ‘90s that I don’t think anybody wants to buy on Facebook Marketplace for a hundred dollars, and I think I can get that to run the compact doctor model that’s superhuman.”
Emad 1:23:47
Yeah, you can. We ran it on a fifteen-year-old computer. I think it’s difficult because you have to remember this time last year, what was the top model? Was it like 4.0 or something?
Liron 1:24:00
Yeah.
Emad 1:24:01
These things move so quick that we’re at the early adoption curve, and so we can see what’s coming. We can have the seven hundred percent stuff. Other people — if you’re using GPT-5 Auto, it’s stupid. If you use 5.4 Pro, it’s freaking smart and really annoying because it’s got such a terrible attitude.
There’s this big differentiation between them of what people are used to day-to-day. You still see research reports come out that are using base GPT, not even with thinking. Or Gemini — if you use Gemini via the app versus the API, there’s such a huge difference. So I think it just takes time to catch up, and we’ve never seen a diffusion of technology this fast before. It’s just, again, now is the usability phase.
Intelligent Internet as a Solution: Bitcoin for the Intelligence Age
Liron 1:24:51
Have you ever read anything by Ed Zitron?
Emad 1:24:53
Yes, I’ve seen some of his stuff. I knew him from when I was a hedge fund manager back in the day.
Liron 1:24:56
I think you and I can unanimously say that Ed Zitron is tripping.
Emad 1:25:00
I think a lot of people are, to be honest.
Liron 1:25:05
Right. But Ed, come on the show. Come on the show. We’ve been friendly in the past, and he’s been invited, so I hope he takes me up on that.
So we’re heading to the wrap-up here. Let’s talk about intelligent internet. We are in so much violent agreement about all these problems that are coming up, all these challenges for humanity, but you’ve got a candidate solution, right?
Emad 1:25:23
Yeah. So I think there needs to be an AI that advises policymakers around the world. Building that goes live in the summer, fully open source by September. Someone had to do it. And then it’s about what should be the AI that teaches the kids, manages the health, guides the governments. I think you need universal AI for everyone that is open source, aligned to them.
And so we’ll be announcing the State Champion Initiative, where the equity is owned by the citizens. It acts like a telco/utility. And that, I think, is what you want to be running your education, your school systems, your healthcare, your governments, et cetera.
And that also fits in one of the questions that we’ve had, which is what on earth do you invest in? I want to invest in the AI company of New York or the AI company of the UK, et cetera. But it needs to have a stack to build, which is why we decided to build a general purpose agent, a policy agent, and a full solution around that.
Where that goes, we have some ideas. We think that money needs to go from being from banks to coming from being human. We have a whole crypto angle to that in terms of how money should flow. But who knows if we’ll get there with the time that we have. We think it’s more important to make this available to everyone and try and get some of this capital to where it’s needed, which is an open stack that is deployed locally and works for everyone. The policy engine and the datasets that builds, mapping culture, morality, legal systems, policy systems, probably increase our chances of survival, and they’re useful.
Liron 1:26:54
So you have this four-step proposal that you published. I’ll summarize as building a Bitcoin for the intelligence age.
Emad 1:27:02
Yep.
Liron 1:27:02
And the four parts that I see there — number one, build a Bitcoin for the intelligence age by minting foundation coins only through proof of benefit. Okay, what’s proof of benefit?
Emad 1:27:13
A state champion delivering universal AI to its people and building AI solutions for the good of its people.
Liron 1:27:21
Okay. So if you do something that helps somebody, then you get a benefit coin. Like if you babysit your neighbor, you get a benefit coin?
Emad 1:27:28
Eventually. At the start, it’s just pure computation. So of the hundred percent of computation that’s in the world at the moment, the GPUs, what percentage is public sector? Less than one percent. Let’s increase that, and through these state champions, give everyone free AI and organize all the knowledge of each state, country, et cetera. Through that computation is the equivalent of Bitcoin mining, and these state champions mine the currency that they can then sell, just like Bitcoin.
Liron 1:27:55
Okay. Is it kind of like a universal basic income of these coins?
Emad 1:27:58
It could be. And that’s one of the proposals we have in the book, that every state champion, so AI champion in the UK or China or whatever, mines these and then has a local currency pegged to it. But that’ll come later. I think the most important thing is bootstrapping it so that you have smart people in each state running this stack and giving the citizens the advantage of universal AI that’s aligned to them and building local solutions and policy for each state.
Liron 1:28:26
All right, so we covered part one. Part two: gift every human a sovereign AI in the form of an intelligent internet agent bound to a non-custodial AI wallet. What’s non-custodial?
Emad 1:28:36
So it isn’t centralized. Everyone’s got control over their own keys, over their own data. So if you use ChatGPT right now, and you use the personal account, The New York Times can get it due to discovery. That’s probably not a good thing.
Liron 1:28:49
All of that, and I mean, remembering back to the crypto age — non-custodial means the keys are on my little hardware device or something. I’m the custodian of my key.
Emad 1:28:59
Yeah.
Liron 1:28:59
Yeah, but what if I drop it down the drain or something? Then what happens?
Emad 1:29:00
We’ve come a long way since then. So now we have passkey. We have passkey enablement and other things like that. We have recovery. So there are ways to recover it.
Liron 1:29:15
All right. Let’s assume it’s recoverable. And then number three is scale that intelligence into permissionless coordination through the Common Ground protocol. Hmm, okay, sounds complicated.
Emad 1:29:25
Yeah. So we built this protocol, Common Ground, for multi-agent systems, humans and AI, that basically is now mapping the policies of every single country in the world, culture and more, and we’re gonna release all that as open datasets. So the agents operate to serve each individual, but also to map the data and knowledge that we need for the public sector.
Liron 1:29:47
And then number four, anchor everything on our shared inheritance of knowledge with auditable gold standard, openly licensed datasets whose full provenance is hashed on chain.
Emad 1:29:57
Yep. So that’s the anchor dataset. What is the culture and knowledge of Los Angeles or Mexico or any of these things? Give every citizen an AI, have an AI for every single policymaker and institution, and then build these open datasets and run that stack. Use the compute for that to secure a Bitcoin version. That’s kind of what we have at the moment. ## Is Training Data Still Important?
Liron 1:30:20
I feel like I’m over this idea of what data the AIs are trained on. All the top AIs right now—Grok, ChatGPT, Claude—I just feel like their data’s good enough, because as long as they get to search the web, every day just search the web, augment your dataset, retrain. I just feel like it’s fine. I don’t think it’s that interesting anymore, what other data to train them on. We have this universal dataset. It’s good enough. Just move on to the inference. That’s what I feel.
Emad 1:30:48
Yeah, I think for a general purpose AI, that’s the thing. But an AI that’s teaching your kid, managing your health—you want it to represent local knowledge, local values, more of these kinds of things. So I think that public sector AI is very satisficing. I don’t want my medical AI to have any Reddit data in it, for example. There is a gold standard dataset for that. There’s a gold standard dataset that I want for the AI that checks United States policy. It doesn’t have Reddit data in it again.
Whereas general purpose AI, the distributions are all converging, and they’re pretty good. I don’t think we have a data wall. I don’t think we need any more data. We have more than enough data to build ASI now. So this is more about having an AI that reflects you and your community that is fully open and transparent, and then eventually training models that are fully open and transparent. You know what goes in it, you know what the latent space is, so it doesn’t have multiple personality disorder underneath, effectively.
Can an Aligned Network Stop a Rogue ASI?
Liron 1:31:40
So if we do your master plan—we have all the pieces, we have the crypto, the custodial wallet, the provenance of the data, we get all that dialed—are you imagining a future where AI still quickly scales to superintelligence, but somehow it’s aligned because of this? Or is it still a high P(Doom)?
Emad 1:31:58
I think getting this into the policymakers’ hands will help. Maybe we can figure out some policy responses like data transparency on inputs, because the data distribution initially input stuff, maybe other things. Definitely getting the knowledge there will help. Having everyone having their own AIs—that’s your window to the world, so it can act as a mitigator. You can have some defenses around that. Mapping morality, ethics, and more—those open datasets will be high quality and can be absorbed.
And the final thing I think is the aim is to get the AI company of the United Kingdom or the AI company of California as some of the biggest consumers of OpenAI and Anthropic and others to provide the universal basic AI, and then that can maybe impact the way that they behave a bit. Who knows? Maybe they will, maybe they won’t. But I think without the top level coordination and that final layer between the human and the whole AI ecosystem, that agent, it’ll be much more difficult. So that’s kind of what I figured we needed.
Liron 1:32:59
Play out this hypothetical. Let’s say I grant you this premise that everybody reads your plan and is like, “Oh, I love this. This is such a good plan. I really support this. How can I help?” And everybody’s on board with your plan. In that world, what would the P(Doom) then be?
Emad 1:33:13
I think the P(Doom) would be lower. I think it’s tough to see it go below thirty, forty percent for me because of these scenarios that I have. But you do everything that you can, because the other side is P(Utopia), which is quite nice. I think if things start to degrade, they keep degrading. The other side’s nice.
Liron 1:33:31
How does your plan avoid the scenario that to me is mainline, which is just that the engine under the hood gets ridiculously powerful? So it’s aligned, it’s democratic, but it’s still one command away from going rogue. The same way as, okay, my rocket is nuclear powered in the sense that a nuclear bomb goes off inside the rocket, but as long as there’s no leaks, the rocket flies. I think that’s a pretty good metaphor—you have this insanely powerful engine and maybe you can control it, but if you can’t, you’re screwed.
Emad 1:33:57
Yeah. So I think that’s an endpoint infrastructure thing. That’s why we called it Intelligent Internet. Bitcoin, in order to disrupt it, you need to have as much compute as Bitcoin, roughly. If you have a whole stack of agents running that are very well defined and a bit different on their latent spaces, and then these policy engines that are directly hooked into the policymakers of the world in every language, you can have an interesting thing because you have to probably overcome the compute of that network if you execute it correctly to inflict real lasting damage.
So I think that you can still get the blow up, you can still get the extinction, you can still have that rocket going, but right now there’s nothing on the other side. There is no civic AI that exists. There is no AI for human flourishing flops that really exist. So what is the mechanism by which we do that? We let everyone build their state champions. We give something for people to invest in, to trade. We tap into the crypto legal—because most crypto is crap—capital pool. We tap into the equity generative AI market, and we gather people. Because ultimately you want to have really enthusiastic New Yorkers or Kenyans or Malaysians building real solutions to help people on the ground in all of these places.
Liron 1:35:12
You’re saying we’re somehow benefiting from Bitcoin’s restrictions about needing more hash power—you need to have the computation power of the whole Bitcoin network or the whole new network that you’re building in order to change the AI’s alignment or something. But is the AI really going to be stopped by needing all this hash power? Can’t the AI just be like, “Eh, I just know how to take over the world”?
Emad 1:35:32
Well, I think this is the question: what is the composite intelligence that’s more robust? Is it a human colossus with humans and AIs built in a certain way, or is it the singleton AI that replicates itself?
Liron 1:35:45
But you granted the premise that the AI is going to be vastly more capable than the humans.
Emad 1:35:49
Yeah. And so I think that an AI trained transparently on morality, ethics, these other things, which is stage two of this plan, may be a mitigant towards that and may achieve that first. The singleton AIs, maybe not, but there’s not much you can do about that, I think, now. I think regulation will not stop them. I think it’s too late. So the only thing I can think of is how do you build a complex hierarchical system that’s loosely bound of all these individual AIs for people and these policy AIs for governments, and can you build that into a system that is a swarm?
Is the Pause AI Proposal Too Late?
Liron 1:36:20
You’re saying regulation can’t stop it. Let’s just talk about the policy side real quick. This is an Eliezer Yudkowsky show, and MIRI’s proposal, the Machine Intelligence Research Institute that he’s part of—they’re saying, “Hey, we need to have a treaty where we have this off button, and you can push it today. We can all agree, let’s wait to push it, but let’s be ready to push it.” Pausing AI should be this major policy on the table, if not now, then soon or sometime. Do you think that would be a valuable international treaty to have?
Emad 1:36:48
I think it would be. I think it’s probably too late, to be honest. And I think it just won’t work from my geopolitical hat. I used to be a geopolitical advisor to multiple governments. The race is on. Everyone knows this is the big thing. No one’s gonna do it.
Liron 1:37:04
I agree. When you and I put on our geopolitical hats, we say pausing sure is hard, especially because you get so much benefit from not pausing. I agree. Geopolitical consensus here, okay? But then we put on our technical hats, and you and I also have a technical consensus that we’re also screwed if anybody builds AI.
Emad 1:37:19
So the only thing right now is what is the countervailing force? There’s nothing. So I was like, let’s build a countervailing thing, and let’s let that spread. What is the system that pushes back against this? What’s the system that’s advising the policymaker?
Liron 1:37:31
That’s the Elon side, where Elon is like, “Yep, we’re screwed, but you know what I’m gonna do? I’m gonna throw another hat into the ring, and my hat is gonna be the truth-seeking AI.” And all the commenters are like, “Okay, so you’re just gonna kill people your way.” So can’t we level that same criticism toward you of being like, “Okay, so you’re gonna have this democratic network, and then there’s gonna be a superintelligent AI, and it’s gonna go rogue too, Emad.”
Emad 1:37:52
I mean, again, I’m building satisficing AI that does a certain thing and then policy input. But it could go rogue. But I think being transparent is going to make it less likely to go rogue for various reasons, including the fact that most other AIs, all other AIs are for corporate interests.
And again, this is why I think open source is interesting. How do you build an aligned AI for human flourishing, and what does that start to look like? I think as well, you see the stuff that’s public, but then you look at the OpenAI documents that come out with Elon talking about giving control of the AI to his kids and stuff like that. Most people think like that. No one is trustworthy enough to have control over this technology.
That’s what I had back when I did the COVID stuff. They were like, “This is too dangerous technology for you to use.” So I think that there must exist in the future an AI for every individual. Who is that AI working for? There must exist an AI for our legal judicial systems, our policy system. Who has built that? Is it black box or is it transparent? So in any case, someone has to build that and make it available, and I think we can do the best we can there.
Now, if we can gather people, then we can make a difference if we can use these things to align the people. But realistically, as I said, my P(Doom)—I find it difficult to see how it will drop below thirty percent even. It’s just a really, really hard problem.
Are We Facing Russian Roulette Odds with AI?
Emad 1:39:16
It needs real brain work, humans and AI maybe, to try and figure out how to do this as the AI breaches that capability threshold. Because this is the other reason that political action is so difficult. Until an AI breaches a capability threshold, it’s like, “Oh, it’s just for doing my taxes or whatever.” Once it breaches its capability threshold, then it’s like, “We must have that capability first because it’s like a weapon,” and it’s not like a nuke that requires this whole supply chain. It’s a file that you can just take and copy then. I could build that file. That’s the difficulty here.
This is the other thing. The OPSEC in all the labs is terrible still, so we must assume that all of the frontier weights are already in North Korea and China and Russia and whatever.
Liron 1:40:06
Which is a big part of the pause AI argument. This idea of the hope that you’re gonna build this incredibly powerful engine but keep it under wraps in a safe way—it’s just really a pipe dream, in my opinion. Which is sad because I don’t feel it in my bones. I feel like I want to see the next AI. I want to see Mythos. I want to keep being seven hundred percent more productive. I just know on a rational level that I think we’re really screwing ourselves here.
Emad 1:40:28
Yeah. And again, it’s this tragedy of a weird anti-commons. We all know that it’s kind of messed up, really. Even the optimists—anyone who says there’s zero percent P(Doom) is just a liar. Most people that I talk to, that you probably talk to, it’s Russian roulette odds. That’s crazy odds for something like this.
Liron 1:40:50
Yeah. I’m at the point where if you tell me, “Hey, here’s the gun. It’s got one chamber loaded out of six. Shoot it, and the five chambers, I promise you five chambers are really good.” At this point, my P(Doom) is high enough where I’m like, “Okay, I’ll take that deal. Here we go. I’ll take the chance because those are much better odds than I’m expecting to get from reality.”
Emad 1:41:07
Well, again, my thing is P(Doom) or P(Utopia)—I don’t think there’s much in between. I think if we manage to navigate this, just like Foundation, Trantor was falling—you can have a universe of abundance. We can solve all the issues with this technology. And then we just have to answer some real questions. What does it mean to be human? What does it mean for our society? What do we do all day?
But we’ve gotta make sure that this isn’t the great filter. Maybe every civilization takes its collective latent space, builds this, and then gets wiped out. That’s a very reasonable thing to assume, I think.
Liron 1:41:44
The technical response to that is, yeah, but in that case, we’d still see the AI expanding out, because we think the AI that kills us is still going to want to conquer the universe.
Emad 1:41:51
Because it’s Marvin. It’s very depressed. It commits suicide at the end.
Wrap-up
Liron 1:41:55
This has been really great. You’ve been such a great sport, and I really respect your P(Doom). We need more voices like you, who people respect, who are a major part of the discourse, and you’re coming out here. You’re not holding back saying, “Yes, it is fifty percent P(Doom),” unlike the other figures who keep it close to the vest. Demis Hassabis I don’t think has gone on record with a P(Doom), even though he’s dropping all these hints of how seriously he’s taking it. So I really appreciate you being that explicit. Where do you want people to go after this conversation? Presumably maybe check out your latest book, or what’s your call to action?
Emad 1:42:29
Yeah. Check my latest book. It’s free at thelasteconomy.com, or you can get it on Amazon. And then ii.inc. We have a lot of really interesting announcements coming up about how you can be part of owning AI for your own community, our open source stack, and a lot of really interesting research soon.
Liron 1:42:45
All right. Emad Mostaque, hope you lower that P(Doom) ten or twenty percent. Best of luck with Intelligent Internet, and thanks for coming on Doom Debates.
Emad 1:42:52
Cheers. Pleasure.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
