I Called Out a16z Partner (Martin Casado) for Downplaying AI Capabilities in 2024

Now that Doom Debates is almost 2 years old, enough time has passed that we can revisit my claims from 2024. I claim they’ve been aging well!

In this early episode, I react to Martin Casado, a General Partner at Andreessen Horowitz (a16z) who claims that AI is basically just a buzzword for statistical models and simulations. As a result of this worldview, he only predicts incremental AI progress that doesn’t pose an existential threat to humanity, and he sees AI regulation as a net negative.

From my perspective, Martin’s problem is that he needs to go beyond analyzing AI as just statistical models and simulations, and analyze it using the more predictive concept of “intelligence” in the sense of hitting tiny high-value targets in exponentially-large search spaces.

If Martin appreciated that intelligence is a quantifiable property that algorithms have, and that our existing AIs are getting close to surpassing human-level general intelligence, then hopefully he’d come around to raising his P(doom) and appreciating the urgent extinction risk we face.

Links

Watch the original episode of the Cognitive Revolution podcast with Martin and host Nathan Labenz:

Follow Martin — https://x.com/martin_casado

Follow Nate — https://x.com/labenz

Follow Liron — https://x.com/liron

Timestamps

00:00 Introducing Martin Casado

01:42 Martin’s AGI Timeline

05:39 Martin’s Analysis of Self-Driving Cars

15:30 Heavy-Tail Distributions

38:03 Understanding General Intelligence

38:29 AI's Progress in Specific Domains

43:20 AI’s Understanding of Meaning

47:16 Compression and Intelligence

48:09 Symbol Grounding

53:24 Human Abstractions and AI

01:18:18 The Frontier of AI Applications

01:23:04 Human vs. AI: Concept Creation and Reasoning

01:25:51 The Complexity of the Universe and AI's Limitations

01:28:16 AI's Potential in Biology and Simulation

01:32:40 The Essence of Intelligence and Creativity in AI

01:41:13 AI's Future Capabilities

02:00:29 Intelligence vs. Simulation

02:14:59 AI Regulation

02:23:05 Concluding Thoughts

Transcript

Introducing Martin Casado

Martin Casado 00:00:00
I literally think this whole problem comes down to simulation, and maybe it’s just because of my simulation background. The only way to simulate the universe is to be the universe.

Liron Shapira 00:00:17
Welcome to “Doom Debates.” Today, we’re gonna be unpacking the worldview of a16z general partner, Martin Casado. Martin was a successful entrepreneur who had a billion-dollar exit before joining a16z, so obviously a smart, capable guy.

Liron 00:00:33
I’ve seen Martin tweet a lot about how AI regulation is bad, and AGI isn’t a threat, and Nick Bostrom and Eliezer Yudkowsky are misleading people about a threat that’s not real. So I kind of have a sense of his position. But the first time I’ve really seen it fleshed out was in a recent podcast that he did with my friend Nathan Labenz on the “Cognitive Revolution,” highly recommend it. I’ll link to it in the show notes.

Liron 00:00:58
So we’re gonna go through that podcast, or most of it at least, and I’m gonna give you an analysis of what he’s saying and why I disagree. I think this is a pretty unique episode because Martin gets off the doom train at a place that you don’t really see that many people get off.

Liron 00:01:13
His stop on the doom train seems to be that he just doesn’t see superintelligence being possible in our universe, as much as I can gather. Because he thinks that computationally, it’s just so hard to simulate pieces of our universe, and that’s going to prevent a kind of superhuman engineer in the form of an AI.

Liron 00:01:30
I hope I’m summarizing him correctly. You can listen for yourself. I think that’s a good summary. And it’s a pretty unique stop on the doom train. People mention it, but they usually don’t double down as that’s where they’re dying on that hill. That’s usually not what you see. So this will be an interesting analysis. Hopefully, you’ll enjoy it. We’ll kick it off with a very traditional question that Nate asked Martin at the beginning of his podcast, which is basically, what are your AGI timelines? How much time do you think we have?

Martin’s AGI Timeline

Nathan Labenz 00:01:51
How powerful do you think AI is gonna be over the next couple of years? We’ve obviously heard AGI 2027. Is that a story you’re buying? What do you think we’re gonna see over the next two to three years?

Martin 00:02:04
I think, like all things, the past is probably the best predictor of the future. What is interesting is if you actually look at the past 80 years of, quote-unquote, “AI,” it’s been steady progress independent of there being winters and summers. It’s been very, very steady progress. There’s been progress on economics, progress on problem solves.

Martin 00:02:22
And every time we tackle a problem, everybody’s like, “Oh, goodness, this is it. We’re almost at AGI.” And then it just turns out that’s one modality, and then we go on to the next, and we say, “Oh, that wasn’t really actually AGI or AI.” So this whole AGI thing has been a moving goalpost for 70 years.

Martin 00:02:38
And so I would say, listen, there’s been this kind of very steady progress. We’ve gotten good at many things. It’s been a very useful tool. I hope it continues to be so. I think it’s a very important thing for us to continue to develop and use. But I don’t think that there’s any step change or it changes the nature of computers or software in ways that we haven’t seen before. I don’t think that.

Liron 00:02:57
Okay, that’s basically the Robin Hanson take, the idea that, yeah, AI progress looks really exciting now, but it’s actually such a long journey and everything’s incremental, and this is just the next incremental step. And for all we know, it’s a hundred-year journey or a thousand-year journey. It’s a fair point to ask, why isn’t that the case? What is special about this moment? But I have an answer, which is just—

Liron 00:03:21
We no longer can easily name what the AI can’t do. So if you’d asked me 10 years ago, “Hey, what’s AI progress looking like?” I would have told you, “We’re making good incremental steps, but AI definitely can’t draw a picture based on a prompt. AI definitely can’t recognize images as well as humans.” That wasn’t the case 10, maybe 12 years ago. I would have said, “Hey, AI definitely can’t chat and use English.”

Liron 00:03:48
So those were all very clear, cut-and-dried, objective metrics I could have told you, “Yeah, AI sucks at this.” Today, I have to tell you, “Look, AI is making incremental progress, but it definitely can’t be a human employee that does the full job, but it can do moments of the job. But if you try to chain a few actions together, it kind of gets unreliable.” Even me just describing what AI can’t do, it’s getting tough.

Liron 00:04:15
I could be like, “Okay, yeah, it can’t wash your dishes,” but they are working on really, really good dexterous robots, and what kind of dexterity does the robot not have? It’s hard for me to even draw a box around what robots can’t do these days. It seems like they’re very much in production, kind of crashing through the last obstacles.

Liron 00:04:33
And so it would be kind of surprising if 20 years from now or 50 years from now, a robot definitely couldn’t open that dishwasher door and take out some dishes and hold them with the right amount of force. Yes, technically, there’s not a product on the market that will walk into your home and do that today, but what’s the bottleneck? This seems like a very much less than a decade away type of expectation.

Liron 00:04:55
So built into what Martin Casado is saying, where he’s saying, “Yeah, it’s all incremental,” he’s building in these claims implicitly. He’s saying, “Yeah, all these things that seem like they’re really close might actually be a lot farther than they seem because I know some kind of limitations.”

Liron 00:05:10
So I don’t share his perspective this is all incremental. I think that we are so close. We’re knocking on the door of human level to the point where it’s even hard to say what obstacle is left. That’s how close we are to just banging down that door. If Martin can draw a boundary around what he thinks AI can do versus what he thinks it’s a long way from doing, maybe he can convince me that there’s this whole separate realm that we’re far away from. I just don’t see it. I think that the line has just become so fuzzy, and we’re recklessly walking through it. But let’s see what kind of distinctions he makes.

Martin’s Analysis of Self-Driving Cars

Martin 00:05:39
So again, I think I like to draw from a historical analogy because I think they’re very useful in this context because I just feel like we see these very often, and they all look kind of the same, and it’s played out.

Martin 00:05:52
And so when I first joined Stanford to do my PhD, so I did my PhD in computer science and systems. It was in 2003. And around then, I don’t know, it was 2003 or 2004, Sebastian Thrun had won the DARPA Grand Challenge. So the DARPA Grand Challenge is he drove a van autonomously, fully autonomously for 1,200 miles. And everybody was like, “Hooray! AGI is solved. Robotaxi is solved. This is amazing.”

Martin 00:06:18
And 100%, we’ve hit one of these threshold moments, 100%. You could actually do things you couldn’t do before. It was the start of a new era of vision and perception in self-driving. And now, 20 years later, and about $100 billion invested, a little bit less than $100 billion invested, the unit economics of self-driving are still three times worse than Uber.

Liron 00:06:42
Wait, an AI driving a car has 3X more expensive unit economics than Uber? That doesn’t really make sense to analyze it that way. Maybe what he’s saying is today, if you look at Waymo, they have all kinds of overhead. They have a human support team. They have a lot of quality assurance. And so today it’s unprofitable while Uber is slightly profitable. Maybe that’s the point he’s making, but that’s such a weird choice of measure.

Liron 00:07:05
Such a temporary contingent data point that the unit economics are 3X worse, whatever he means by that, however he’s analyzing that. Obviously, there’s a dichotomy of possible outcomes. Either we get to the point where self-driving works and it’s better than a human driver and it’s robust, so that it can drive many hundreds of miles before you need any sort of human intervention. And in that case, of course, it’s going to be much cheaper to run a self-driving service than it is to run Uber, because the driver cost is currently something like 60% of the cost of the ride, and now you don’t have that 60%, and instead maybe you have a small fraction of that. It’s not even going to be close.

Liron 00:07:48
So it seems like the point Martin is making about self-driving is almost the opposite of how he’s using the evidence. The point I would make is, yeah, we’re probably a couple years away from self-driving that actually works. And then at that point, all the billions that have been invested are going to easily pay off. I think that’s almost the consensus opinion that the investment is going to pay off massively.

Liron 00:08:08
I mean, a single decade of being able to have self-driving cars for everybody, for eight billion humans or whatever high fraction of those humans rides in a car, that’s a massive return on however many billions have been invested into this. So I’m just confused about what the lesson is that Martin is taking away from this.

Liron 00:08:25
Maybe to be charitable, he’s saying, “Hey, we get excited that stuff is going to work for a long time, but then it takes a decade longer than we think.” Okay, and I agree. I’m not saying AGI has to happen next year. I agree it might take a decade, but that’s just not the primary lesson here when we’re talking about the end of humanity from superintelligent AI.

Martin 00:08:39
And so I am a systems person, meaning all I’ve been doing for the last 30 years of my life is building computer systems, large distributed systems. This is what I do. And I just know that scalar capabilities—they’re not necessarily parametric, so they don’t necessarily just go up and to the right, and there’s so many examples of this. I know that the universe is this very heavy-tailed system, and there’s no single solution that tends to rein in its complexity.

Liron 00:09:04
I think he’s gonna come back to this point a lot, that the universe is, quote-unquote, “heavy-tailed,” that it has a lot of edge cases or it can’t be simulated or something like that, I think is the gist of where he’s going. We’ll see.

Martin 00:09:16
I know that the economics for these things are very, very hard. What gets lost in these conversations is AI has been better than humans for a very long time at many things. Handwriting detection, diagnosis for a very long time, game playing for a very long time, mathematics, since the creation of the computer, and four orders of magnitude better at mathematics than us. And yet none of these things have resulted in kind of general economically viable solutions for everything. Just for subsets.

Liron 00:09:47
Okay, because they didn’t know how to speak English. They didn’t know how to have a natural conversation. They didn’t know how to see, how to process visual input. They didn’t know how to generate visual output. These are all barriers that have now come down to the point where it’s hard to say what they can’t do. So again, I’m looking at you, Martin, to tell me what exactly is it that they can’t do. What’s the last barrier?

Martin 00:10:12
And so for—we can talk about this language. So another way to rephrase what you said about the language stuff is these models are very good at predicting what a mean person would do next, a mean person would do next. So sure, they basically do kernel smoothing over positional embeddings. That’s great.

Liron 00:10:29
Whoa, whoa, whoa. Who are you calling kernel smoothing?

Martin 00:10:31
They basically do kernel smoothing over positional embeddings.

Liron 00:10:35
That’s not accurate. They’re doing a lot more than that. But you know what? When he uses that term kernel smoothing, I recognize that dog whistle, Martin. What you really wanna say is the SP word. Stochastic parrot. I gotcha. Think about a racist guy who wants to say the N word, but he doesn’t wanna be seen as racist, so he says, “Hey, look at that urban thug.” Say what you wanna say, Martin, that that AI is a good-for-nothing stochastic parrot.

Martin 00:11:00
They basically do kernel smoothing over positional embeddings.

Liron 00:11:03
He’s trying to cloak the slur here. But I’m sorry. That’s not what it’s doing. It’s not kernel smoothing over positional embeddings because when you have the transformer architecture, you now get something that is qualitatively different. It’s nonlinear, which is my same beef with people who try to say, “Look, it’s just linear algebra. Why are you afraid of linear algebra?”

Liron 00:11:22
Well, hold on a second. Once you apply all these layers of the neural network, it’s very much not linear. It’s a totally different phenomenon, which as you would expect when you have something that’s never been done before, where it’s talking English, doing better drawings, better essays than humans could do. Obviously, that’s something more than being a stochastic parrot, something more than kernel smoothing over positional embeddings. So I just wanted to point that out because that is actually a low blow that he’s leading off with.

Martin 00:11:45
Which means if you have a corpus of data, it’ll give you the average response, which is great, and that’s very useful for a number of things. Does that provide economic utility for a broad range of stuff? I mean, I’ll tell you, I look at these companies basically full-time, and I have for three years. We probably have the largest portfolio of them, and I don’t see that yet.

Liron 00:12:03
I get the temptation to say it’s just the average response, but that’s just so not accurate about what it’s doing. When you tell it to draw you a picture with a five-sentence description and it draws something plausible, complete with plausible details, and okay, there’s six fingers on one hand, but it’s an entirely new scene. That’s just giving you the average response? What does that even mean?

Liron 00:12:27
It’s frustrating to me because these people who are so smug about writing off what the AI is already doing as if they know what the limitation is, these same people were not saying five years ago, “Yeah, you’re gonna get amazing artwork. You’re gonna put artists out of business.” These people had no predictive power in their mental model. They’re just retroactively looking at what the AI can do and finding a way to dismiss it.

Liron 00:12:48
They don’t have a sound methodology when they’re making the claims. They’re very much just making it up as they go along. When they make claims like, “Yeah, it’ll just give you the average response,” they’re actually not self-aware about the limits of their own knowledge and their own insight, which is frustrating.

Martin 00:13:02
Now, there are areas where they do work, but it’s not that. It’s not this kind of general knowledge worker. That’s actually not working. And the areas where it is working is stuff like the marginal cost of content creation goes to zero, so now we have amazing new content, or computers are creating an emotional connection with humans. That’s new, and that’s exciting, and that’s amazing.

Martin 00:13:22
But this idea of this general knowledge worker working, I haven’t seen it yet. And maybe it’ll happen, but just like self-driving 20 years ago, that’s just not how the universe tends to play out.

Liron 00:13:32
Oh, okay. So the marginal cost of content creation goes to zero, and creating an emotional connection with humans is gonna be a solved problem. But the general knowledge worker is the last barrier standing? How are you making this categorization? This is such a retroactive move. You would not have told me this five years ago. You would not have predicted this, and you don’t understand why it’s happening now.

Liron 00:13:55
Now, what about my own epistemic state? I’ve never claimed to know the order in which the last pieces of AI are coming. I’ve always just been saying that the end state is superhuman AI. I can’t tell you if the last pieces are gonna come in two months or two years or 20 years, or maybe even longer. I don’t claim to know that. I don’t have special predictive power over how fast it’s coming.

Liron 00:14:15
All I can do is observe that all these different obstacles that we used to say would be extremely impressive if they were solved, they’re pretty much all in the rearview mirror except these last AGI milestones, to use Martin’s terminology, a general knowledge worker. Okay, yeah, we don’t have that. That’s one of the last things standing.

Liron 00:14:33
I mean, we’re already getting medical exams with case studies getting passed better than doctors. I’m not saying I literally wanna trust my life to an AI doctor, but we’re getting there. We’re getting there. I think we’re all confused on why it has the set of abilities that it has.

Liron 00:14:48
And in fact, researchers are actually discovering that you take the same AI that exists today and you try different methods of prompting it, and you try chaining together different prompts, and we’re still getting surprised at what today’s AIs can do. This is very much a situation in flux. We do not deeply understand what’s going on in these AIs. It’s still very much a black box.

Liron 00:15:07
And here’s Martin with this fake distinction where he’s like, “Yeah, creativity’s automated, of course. Automating human emotion, generating human emotion, pfft, of course that’s easy. Knowledge work is hard.” What? Where are you getting this distinction besides just retroactively describing what happens to be the case today?

Heavy-Tail Distributions

Liron 00:15:30
Okay, now Martin is gonna explain something that to him is apparently a core belief that drives a lot of his claims about AI. To me, it doesn’t really make any sense, but let’s listen to how he tells it and see what the crux of my disagreement is.

Martin 00:15:39
I don’t think people appreciate how heavy-tailed the universe is. So maybe I just wanna describe this, ‘cause it just comes up right now and it’s gonna be so relevant to what we’re gonna be talking about going forward. There are many things that systems deal with that are heavy-tailed. But what heavy-tailed means is if you draw an occurrence at random, the chances are that it’s a very rare occurrence.

Martin 00:15:58
So a very classic case of this is search. And so if you draw a unique search query, say Google, at random, the chances are it’s pretty unique actually, even after all of this time. Now, this is unique searches. So if you take all of the searches that go into Google and you don’t dedupe them, you can have a lot of repetitive ones. The vast majority of searches are the same. Let’s say 90% of them are common.

Martin 00:16:25
But if you reduce it to just singular intents, so you don’t have any duplications, the majority are exceptions. And that’s how the universe tends to be. I come from networking. It’s a very famously heavy-tailed discipline. A lot of things in the tail.

Martin 00:16:40
So if you’re building a general system, the majority of new things it has to do, not things it has to do, the majority of new things it has to do are exceptions. Now, it’s very easy to get tripped up in these conversations because the majority of the stuff that you’re doing is not new. The majority of stuff you’re doing is very common. But anytime you get into new areas, then you have to come up with new stuff.

Liron 00:16:55
Okay, so the concept of a heavy-tailed distribution, that’s obviously a coherent concept from statistics. The idea is that if you compare it to a bell curve, like the bell curve of human height, if you imagine skewing it to the right and having more weight on values on the right side of that curve than you would normally get from a bell curve, then you’re looking at a heavy-tailed distribution. For example, if you look at the distribution of income, which is also the example that Nassim Taleb uses in his Black Swan analyses.

Liron 00:17:26
If you look at the distribution of income or wealth, you have billionaires. So you don’t just have everybody clustered around the hundred thousand or million dollar range. You have these billionaires that are off to the right a thousand times higher, and you’ve got a fat tail connecting the normal people to the billionaires. So that’s an example of a heavy-tailed distribution.

Liron 00:17:45
And I guess the way that Martin would think would be, “Hey, if we train AIs to do a lot of analysis on people with average income, it’s just not going to learn insights when it encounters a billionaire for the first time.” And that’s all analyzed on a single dimension, because there’s a single quantity of income or wealth. Imagine that there are many dimensions, and then you’re dealing with heavy tails across this multidimensional space. I think this is what Martin is getting at.

Liron 00:18:09
And yeah, that’s a coherent concept. You can argue this is the problem with AIs, is they’re just connecting dots that are near the center of the bell curve, and they get screwed up when you get a heavy tail. I think that I’ve paraphrased him accurately. He can correct me if I’m wrong.

Liron 00:18:24
Now, the problem I see is, okay, what’s the metric by which you’re saying that things have never been seen before or things are part of the heavy tail? Because in the case of Google Search or in the case of prompting ChatGPT, you can give it a query or a prompt that it’s never seen before, and in fact, with ChatGPT, presumably if you’re writing a two-sentence prompt, there’s gonna be so many words in that prompt that it really is the first time that that exact prompt has ever been seen, especially if you consider the preceding conversation. So it’s a unique input if you consider the context as a whole.

Liron 00:18:55
So does Martin just automatically grant that ChatGPT is successfully handling heavy tails? No, I don’t think he would. I think he would introduce a metric and he would say, “Sure, that’s a novel sequence of tokens, but it’s still located close to other token sequences that have been seen before. So on my similarity metric, it’s not part of the heavy tail.” But that’s really where he sneaks in his own arbitrary judgment, just sneaking in this arbitrary label that he likes, heavy tail versus light tail, as if it applies to the kind of complex jobs that modern AI is doing.

Liron 00:19:25
Labeling ChatGPT as if it’s light tail, it’s like, come on. I’ve asked ChatGPT to come up with stuff that seems creative to me. When I asked for a new name for my podcast, it said, “The Singularity Scenario,” “Doomsayers Code,” “Future Fallouts,” “Machine Mayhem Meditations,” “Endgame AI Enigmas,” “Intelligence Insurrection.”

Liron 00:19:48
So Martin can just dismiss all that stuff like, “Well, it’s just taking words and connecting the words and encoding what the words mean,” but from my perspective, you’re clearly just making up rationalizations because you wanna put the current level of AI capabilities into some bucket that you wanna call light-tailed, but I just don’t see it. I don’t see how you could have a priori told me what ChatGPT can’t do.

Liron 00:20:10
Could you really have a priori told me, “Sure, ChatGPT can suggest new names for your podcast that nobody’s ever invented before. That’s totally the kind of task that doesn’t have a heavy tail”? How could you tell me that? I don’t get it. What is the metric by which these suggestions are not heavy-tailed?

Liron 00:20:28
And then Martin might say, “Well, if you look at the internet’s corpus of data, there’s so many examples where somebody was trying to name a podcast and there’s a bunch of principles for how to come up with a good name, and there’s this idea of which words sound catchy and alliteration and which words sound novel and new.”

Liron 00:20:45
So I get why he might think, “Hey, it’s a type of problem that’s already been done.” And that’s really the crux of the issue, is can you really take something the AI did that seems impressive and always just find some other thing that to you seems similar enough that you can then write off the AI as being, “Ah, it really just connected these two dots. It really just interpolated this other thing that it found together with what you’re saying”?

Liron 00:21:07
And it’s a tough question because the thing is, it’s a matter of degree. So yeah, I agree that it probably did use some corpus of naming podcasts or marketing or all these different related fields, but we didn’t know how to make this algorithm to interpolate five or ten years ago. So obviously it’s easy to say that it’s trivial now, but people have tried. People have had similarity metrics before. They’ve had embeddings before. They didn’t have the transformer architecture and they weren’t able to produce the output black box, these inscrutable matrices. That’s the new thing here. That’s the new insight that’s able to make these connections that you’re rationalizing away as being, “Oh, it’s just interpolating. It’s just kernel smoothing.”

Liron 00:21:48
But it’s not. It’s something that we don’t understand. But I agree that you can point to any example and you can say, “For me as a human, this feels similar, and AI can only do things that to me feel similar.” You can make that argument, but of course the problem is every time a new AI release comes out, every time a larger model comes out, it somehow seems to stray farther and farther away from what seems like short hops away from its training data.

Liron 00:22:10
I don’t think GPT-2 could have made this hop to name my podcast. GPT-3 probably could have done it. GPT-4 definitely can do it. GPT-5 can probably do much harder problems. And I don’t think the Martin worldview saying that AI just interpolates things lets you make any predictions whatsoever in terms of how far away from its training data GPT-5 can hop.

Liron 00:22:32
That hop distance can surprise us. It can be surprisingly short. It can be surprisingly long. Nobody is prepared to make a prediction. In fact, I don’t see Martin on record with a prediction. I would happily make a prediction except I don’t know. I don’t claim to know. My prediction is a high uncertainty distribution.

Martin 00:22:54
And it’s like self-driving is a very great case of this. Nobody expected basically a 2D vision problem. Let’s be honest, self-driving is 2D. It’s not even 3D. You have streets and you have signs. You don’t really worry about the Z dimension, really.

Martin 00:23:10
And yet $100 billion in, 100 billion of investment, we’re almost there but not in an economically positive way. The reason I wanted to make this point now is I think so much of these discussions is people, I believe, underestimating how heavy-tailed the universe is and how hard it is to make progress. And so we should be working as hard to make progress so that we can do stuff like self-driving and not slow it down because the task is enormously complex.

Liron 00:23:42
Okay, I think I get how Martin’s argument works. What he’s doing is he’s observing domains like the domain of chatbots, the domain of self-driving, and he’s observing, hey, there’s a few percent of the distribution of inputs where you’re not getting a good output. And then he’s going from there to his preferred explanation that those outputs that it’s failing at, they must be part of the heavy tail. They must be outside of the center of the distribution.

Liron 00:24:05
They might be inputs that you can’t just interpolate your way to solving, and those inputs are so far beyond the reach of what AI can do because AI is just an interpolator, and there’s some fundamentally different type of intelligence that you need when you solve those last percent of inputs.

Liron 00:24:22
Now, of course, he may be right. It may be that when we finally figure out how to make AI that can do the last .1% of self-driving or the last 5% of ChatGPT queries or however you wanna analyze it, it may be that we do it by discovering such a new type of AI, and Martin will have been right: “Aha, yes, this is just a matter of you can’t interpolate your way to success.”

Liron 00:24:45
I personally don’t see it that way. I don’t follow him when he makes the leap from there’s a percentage of inputs that AI can’t do to therefore some inputs can’t be interpolated. I don’t see it that way. I just think that the inputs that it can’t do tend to be more like, okay, it can’t reflectively look back and kind of edit its own output.

Liron 00:25:05
It doesn’t do what humans do, where they kind of take a first stab at something, think back on how they should change it, take another stab, kind of a loop where you get iteratively closer and closer to the answer. And you can apply a certain robust logic.

Liron 00:25:20
Because applying logic is also another type of interpolation. When a human looks at something and is like, “Aha, I made a mistake. There’s kind of a logical inconsistency here,” that process of a human doing that, you can still describe it as interpolation. It’s not fundamentally different. I mean, AIs can do logic. They can reason to a degree. They just don’t reason in long, robust chains.

Liron 00:25:42
It’s not black and white like AI can reason versus AI can’t reason. Most experts agree that there’s some amount of reasoning or proto-reasoning going on. So I’m just not seeing the connection between what AIs currently can’t do and some fundamental distinction of interpolation versus not. I think that’s where Martin goes wrong here.

Nathan 00:26:01
I always say that the best systems are closing in on expert performance on routine tasks, and the word routine there definitely is critical, ‘cause when you get outside of routine tasks, they are not comparable to expert performance.

Liron 00:26:19
Yeah, that’s a good summary by Nate. That terminology of routine tasks I think is being used synonymously with Martin’s terminology of in distribution, and then heavy tail is just the idea of if you wanna be economically valuable, there’s always gonna be so much that lies outside your distribution, and you better get working on it.

Liron 00:26:38
So the crux of disagreement is going to come down to basically this concept of the threshold of general intelligence. This thing that humans have. I mean, the human brain is lots of copy and pasting. It’s a relatively small amount of genetic information, a small amount of genetic divergence from the apes that is letting us reflect on new domains, go to space within one or two generations.

Liron 00:27:02
So this extra module that the human brain has, I don’t think that AI is gonna be that far away from it. Maybe it needs a new architectural insight, but I think if you take the raw material of this magical high-dimensional interpolation that we’ve already cracked, and you add a little bit of extra secret sauce of being like, “Oh, here’s some extra logic. Here’s how you can really quickly train yourself on a new domain and start applying the new domain so you don’t need a human trainer,” whatever extra secret sauce we need to put into that AI, I don’t think it’s gonna take that much secret sauce.

Liron 00:27:35
And then suddenly I think you’ll be able to break out of a lot of different kinds of routines at once. So if people keep working on the self-driving problem and they keep doing 99.9%, 99.99% of situations, if they keep working on it, eventually there’s no way to get 99.99999% without just having core general intelligence.

Liron 00:27:58
Because the amount of crazy scenarios you can encounter is actually as crazy as the amount of scenarios you can encounter period. Because you can just take any situation and embed it as a self-driving car scenario.

Liron 00:28:10
You can take any kind of crazy celebration that humans are having in the middle of town square where the car’s trying to drive, and you can be like, “Okay, in this celebration, here’s how you have to conduct yourself if you’re trying to drive through. Here’s a bunch of rules. You have to be respectful. You can’t make a left turn during this particular celebration.”

Liron 00:28:28
Or you could be like, “Oh, here’s a truck that has a bunch of traffic lights on it, but the traffic lights shouldn’t count as real traffic lights.” The amount of crazy scenarios that you can concoct is unbounded.

Liron 00:28:40
Or, okay, this other driver gets out of their car, and they’re trying to tell you something. You should really listen to what they’re saying. Oh, okay, so now you just have to understand what they’re saying about the world, and maybe they’re arguing with you, but they’re wrong, so you have to argue back to them.

Liron 00:28:55
So you’re never going to fully solve self-driving before your full general intelligence just comes online, which is true about a lot of different problems. Whenever a problem gets complex enough, you’re never gonna fully solve it without just having general intelligence.

Liron 00:29:12
So when you point at the problem and you say, “Hey, this problem has a heavy tail,” I don’t think that that’s as good of an analysis as saying, “This problem is a Turing complete problem. It’s an AI complete problem. It can only be fully solved with general intelligence.” I think that’s the useful framework here because general intelligence will come. We have it as humans. Even Martin would agree, we don’t have a problem with heavy tails.

Liron 00:29:38
So something has snapped. Something has reached some threshold in the human brain. So why analyze it as heavy tails? I guess that might be an analysis that works when you pre-assume that AI has to work exactly like today’s AIs.

Liron 00:29:52
I guess if you wanna draw a circle around all the things that today’s GPT can do, you can probably come up with a metric where you can accurately say, “Okay, yeah, it’s part of the heavy tail. It’s out of distribution.” I think you’d largely be accurate. I think you’d be surprised how many times it can surprise you and go into that tail that you kind of made out to be untouchable. I don’t think it’s untouchable. I think you’re gonna get surprised. But I think that you’re making sense when you talk about, okay, today’s AIs do better in distribution.

Liron 00:30:20
So we can split the difference. I can say, “All right. Fine, Martin. You’re speaking some sense.” It’s accepted that AIs do better when things look more like their training data. There’s some sense in which you’re correct. But I also think that you’re trapping yourself in this mindset where you’re about to be blindsided by reaching the threshold of general intelligence, which is something that happened in human evolution.

Liron 00:30:42
You would’ve been blindsided when the ape brain had a few genetic modifications, and then you got the human brain. You would not have predicted the human brain. So I’m here trying to tell you, wake up because something analogous to the human brain is coming with minor modifications probably to the AIs that we have today.

Liron 00:31:00
You need to have a mental model that will not be blindsided when that happens. And of course, what I’m saying is similar to what a lot of experts are telling you. I’m just telling you the Geoffrey Hinton position, the Ilya Sutskever position. I even think Sam Altman probably agrees with what I’m telling you right now.

Liron 00:31:12
So it’s interesting which sides form in different arguments. A lot of these people that I mentioned might not be quite as big doomers as I am, especially Sam. He’s more of an optimist. But he would take my side of this argument versus Martin in terms of how close we probably are to AI that’s going to solve those heavy tail problems.

Martin 00:31:13
As far as I can tell, what LLMs are doing, and there’s actually papers that have shown this pretty concretely, is they take a corpus of data, they create positional embeddings, and they basically average over it. And based on that, they can predict what a human being would do or say given a certain situation. And they do that basically via averaging.

Liron 00:31:36
Again, I think you’re giving the wrong intuition here. It’s true that token embeddings are a linear structure, so your intuition around, yeah, we’re going to encode the meaning. We’re gonna solve the symbol grounding problem using a high-dimensional space. We’re going to use similarity metrics in high-dimensional space to relate different meanings. Your linear intuitions and the concept of an average, that all makes sense when you’re just talking about embeddings.

Liron 00:32:02
But when you’re talking about an LLM that’s using the transformer architecture to chew through thousands of tokens and a large prompt and then come up with a response, taking into account all of that different context and combining it in a combinatorially novel way—okay, I’m not just asking it to come up with a title. I’m asking it to come up with a title for my podcast, which is about people debating doom, and maybe I want it to be silly.

Liron 00:32:28
Combinatorially adding all these things in, it’s more than just finding some point inside of the embedding space. It’s more than that. It’s an operation that has a bunch of parameters and context. I mean, it’s a black box, frankly. I can’t really tell you how it works because nobody knows.

Liron 00:32:45
I can just tell you that there’s a bunch of parameters describing this multi-step nonlinear operation, and you’re not describing that when you’re just hand-waving and saying that it’s averaging and interpolating. You’re missing a key part of what is actually happening here, and as a result, you’re lacking the predictive power.

Liron 00:33:08
You couldn’t have told me in advance if I had told you five years ago, “Hey, you’re gonna have AI that all it does is average things. All it does is interpolate. Now tell me what it can and can’t do. Can it suggest names for my podcast? Can it write a poem that rhymes and is about maritime law but also makes you think of this other historical figure? Can it do that if all it can do is interpolate and average?”

Liron 00:33:32
Tell me, Martin, what does your analysis imply about specific input outputs that AI can do or not? I don’t think that he could have reliably made the connection between his own mental model, “Ah, yes, it only averages things,” and the correct answer of, “Why, yes, this is something that AI can do. It can make those kind of crazy novel poems.”

Liron 00:33:52
I don’t think the connection is there. I think he’s smoothing over a picture retroactively that he can kind of sell. He can sell it to himself. He can sell it to others. But it doesn’t logically hold together. He’s not accurately describing what AI can currently do and where it might be going soon.

Martin 00:34:01
We’re nowhere close to understanding what these distributions look like, how heavy-tailed they are, and how they can handle that tail.

Liron 00:34:10
I agree that there actually is some uncertainty on what’s gonna happen when we train the next LLMs. How far into the heavy tail are they going to be able to reach? Is scale really all you need? Does a 10X or 100X or 1,000X bigger GPT-4 get the job done in terms of cracking into general intelligence, in terms of full self-driving in every situation, in terms of taking over the world? And I can’t tell you.

Liron 00:34:35
So I agree with Martin that we are nowhere near understanding what’s going on with these distributions. So I guess we can see eye to eye in that sense.

Liron 00:34:45
The only difference is I see us as being architecturally close or just close in design space to general intelligence and general superintelligence because I think if you crack human-level intelligence, you’re very likely going to foom into smarter intelligence. I think it’s not going to quickly stop at human level. I think we’re screwed at that point.

Liron 00:35:05
So I’m not suggesting that scale is all you need necessarily. I think it may or may not be. I think we may try to 1,000X GPT-4, and we still get an AI that’s having trouble with edge cases. That is very plausible to me. But I think just a few little tweaks, a few other modifications to the architecture that are as insightful as the concept of a transformer, just throw in a couple more concepts, and that’s it, and then it’s game over.

Liron 00:35:30
I don’t think that we’re that many large concepts away from game over, from a superintelligent AI. And part of the reason I feel that way, I’m estimating that way, is because when you look at the human brain, there weren’t that many major architectural modifications between the ape brain and the human brain.

Liron 00:35:48
And compared to the ape brain, the human brain has foomed. The human brain has taken over. It has reached the point of no return, where no other species can hope to control their niche, no other species can hope to defend their niche in the face of humans. The idea of competition between niches is now completely gone on the current trajectory of humans.

Liron 00:36:08
So that kind of escape velocity, I think it didn’t take that many genetic modifications to go from the ape brain to the human brain, and I don’t think it’s going to take that many architectural modifications now that we’re at the point of LLMs. Probably not. Maybe it’ll take 50 years. I just don’t see it taking 500 years. I don’t see us having that much time, and of course, we may only have one year.

Liron 00:36:24
So I guess that’s where we disagree in terms of extrapolating what’s coming next.

Martin 00:36:29
I actually think this is how it’s gonna play out. Right now, anytime we’re always at the beginning of what looks like an exponential but is actually a sigmoid, and what we think is a fat head system but ends a heavy tail system, it always feels like this.

Martin 00:36:43
It feels right now, and everybody’s saying all the stuff that they’re saying right now. And we’re all like, “It’s amazing, everything’s gonna work out.” And then we’re gonna realize, oh my goodness, the universe is heavy-tailed. This is really hard. These things are good at 80% of stuff, but the 80% of stuff that it’s good at is not that hard. And the stuff that I need it to be good at is really hard.

Liron 00:36:59
I think we’ve figured out what’s going on between Martin and me. Martin looks at problems like self-driving, where there’s a few percent of situations that the AI hasn’t figured out, and customer service tickets where, sure, we got to 90%, we got to 95%, but that last 5% is just still thorny, still requires a human. Martin’s looking at all these domains and he’s calling them fat tails, and he’s referring to the idea that we haven’t solved general intelligence, which lets us solve these fat tails.

Liron 00:37:28
I’m looking at those same problems and I’m agreeing that, yes, this is definitely a pattern that you’re observing. The idea that AI hasn’t solved everything perfectly, there is definitely a key reason why, and it’s the reason that I’ve been calling lacking general intelligence. It’s not all the way to general intelligence.

Liron 00:37:48
And so whenever you have an AI complete domain, whenever you have customer service tickets that are able to pull in an arbitrary amount of context of understanding different people or a bunch of different systems without a clear bound, anytime you have a situation like that, I would frame it as, okay, yeah, then you need to just use your general intelligence.

Understanding General Intelligence

Liron 00:38:08
You need to use intelligence whose domain is over a system as complicated as the universe. Not in terms of understanding every atom in the universe, but in terms of understanding some basic laws. A little bit about physics and a little bit about human psychology and a little bit about reasoning and making arbitrarily long connections and doing it robustly. I’m just describing what it’s like to have general intelligence, and I agree that today’s AIs don’t fully have it. They’re not human level.

AI’s Progress in Specific Domains

Liron 00:38:38
I do think it’s worth noting, and I think Martin would agree, that when you look at domains like handwriting recognition or recognizing objects better than humans, it is notable that there are domains where they’re pretty much done solving it, including domains that we used to think were really hard and thorny.

Liron 00:38:58
So I do think the only thing that’s missing is this very large domain general intelligence. That last few percent of things where the only way to solve it is to solve everything in one fell swoop. There’s no such thing as being able to master 99.99999% of self-driving situations without also mastering reasoning about the universe as a whole.

Liron 00:39:18
There’s no such thing as being able to solve 99.999% of customer service tickets without also being insightful about the universe as a whole. But here’s the thing, the universe as a whole is still a finite problem. So it’s not like this new, mysterious, different type of problem. It’s just a larger domain. It’s still a domain, it’s just larger than other domains.

Liron 00:39:38
But we’re getting there. The same way that we totally solved board games, at least the popular board games—chess, checkers, Go. The same way we solved those and we’re solving video games, we’re going to solve the physical universe. The physical universe is not that hard.

Liron 00:39:58
So talking about fat tails isn’t a convincing reason to tell me that AI is not going to expand into solving problems in the domain of the universe. We are getting there. That’s where the trend is going. And once you start being able to reason about the universe as a whole robustly, better than humans, you’re not gonna be able to be like, “Hey, look, there’s a fat tail here. This is out of distribution.”

Liron 00:40:18
That argument is just not going to reflect what’s actually happening. What’s actually happening is that the same way that humans think about strategic moves and answers to problems within the universe, the AI will be doing that too. It will now be going toe-to-toe with humans in that particular domain, the same way that it keeps going toe-to-toe with humans in other bigger and bigger domains.

Liron 00:40:38
And so that’s why the current emphasis on the fat tails, that’s totally fine for now. If you’re just investing in companies that are trying to make money in the next three years, which is his job—although he likes to think that he’s investing for 10 plus years, so maybe not. But if you’re just looking very short term, sure. But zoom out and look what’s going on here. You have to talk about general intelligence. You have to look ahead a few years and see where these things are going.

Liron 00:40:47
Okay, in this next section, I’m not gonna play everything that Nathan said, but basically he was coming around to the question of how do you make sense of high-level concepts that exist inside the AI’s internal representation?

Nathan 00:40:59
If you were to keep scaling and keep training these models on distributed systems or software or whatever, how do we know that they don’t start to learn things that people don’t know and truly generalize beyond the training set? ‘Cause it does seem like that’s happening in some profound way.

Martin 00:41:15
Yeah, no, I think that’s a great question. And I think maybe this is why I come to this from a different angle. Prior to my PhD, my life was computational physics. I worked at Lawrence Livermore National Lab. I did large physics simulation. I just lived this life.

Martin 00:41:32
And I think the Occam’s razor in all of this is distribution in, distribution out. These things are very good at learning distributions of the training set. And so this is just a weird, quirky coincidence of this moment. I actually worked on protein folding at IBM T.J. Watson Research Center in 1999 on the Blue Gene project.

Martin 00:41:55
And at that time, there was a belief that we knew enough of the fundamental forces in order to actually, from first principles, calculate protein folding. That was the thesis, and that we just didn’t have enough compute. And so they actually built a whole computer to do this called Blue Gene, and through corporate machinations, that ended up becoming an entirely new computer ‘cause they couldn’t fund it, and it never really happened.

Martin 00:42:20
But there was the belief that if you had enough compute, that you could do protein folding and a bunch of other stuff. And so I think it’s very reasonable. This is just strict Occam’s razor that if you have enough compute and you have enough data and you’re dealing with an axiomatic system that you can reduce to first principles, then you will learn that distribution and you can use it for predictive stuff.

Martin 00:42:45
And if you wanna call predictive emergent, that’s fine, but it’s just predictive. Anything is predictive. I’ve worked on simulation codes that were predictive. Could I have predicted the yield of a nuclear weapon? No, I could not have predicted that as a human being. Only a computer can. But it’s really just understanding first principles to create these things.

Martin 00:43:05
I very strongly believe these models are very useful in science where you’ve got fundamental laws of nature that are being learned that can be predictive. I think the models that are good at that are not gonna turn around and solve a different problem most likely, because I think you’ve probably learned one distribution from one domain, and that’s a very useful tool that we should use.

Liron 00:43:19
Okay. So what does Albert Einstein’s brain do? What does Elon Musk’s brain do? They understand some principles, some axioms of how their universe works. They don’t know how quarks work, but they know enough building blocks that describe their universe and the people in it and the objects in it. That’s all they need to know to treat that as a well-defined challenge and make powerful predictions about it and take powerful actions in it. That’s it.

AI’s Understanding of Meaning

Liron 00:43:48
So when you’re describing, “Hey, these systems are possible,” why does that not describe the challenge that you give to a human who’s super effective in the world? And why does that not describe throwing computation at that problem and beating humans at their own game? Where’s the difference? Where’s the distinction? Where’s the firewall that AIs aren’t going to cross? And you’d probably answer, “Okay, it has to do with the fat tails.” But—

Liron 00:44:05
Where exactly are the fat tails? Because Elon Musk’s brain understands enough building blocks that those building blocks can combinatorially combine and handle any situation in Elon Musk’s life. So where does the AI stop being able to do that?

Liron 00:44:22
You’d probably be like, “Ah, well, the AI under the current architecture needs to be fed so much data.” But I guess I would point out that the AI is still crawling away from its distribution. You can still ask it to combine things in a novel way. You just can’t ask it to do it for 50 steps. But it can take a few steps. There’s actually a lot of evidence that it can take a few steps.

Liron 00:44:45
Sometimes you might have to hold its hand. You might have to be like, “Hey, what if you take this example and make a few modifications to it and then try to make an extra modification? Can you do that?” It can to some degree. It’s showing a willingness and an ability to crawl away from whatever you think is the center of its distribution of training data. It just starts to get bad at it after a certain amount of steps.

Liron 00:45:10
And that’s the whole problem right now is how many more steps of this kind of iteration before it’s like, “Oh, wait. Hey, I’m just generally intelligent. I kind of know how to go back and edit myself when I made a mistake.” That’s what everybody’s looking at. That’s what everybody’s wondering about. That’s what humanity as a species is being kept alive by, the fact that we can’t do this yet.

Liron 00:45:30
But anyway, my question to Martin would just be, okay, you agree that computers can kind of master all these different problems that are based on principled systems, lawful systems predicting what they’re gonna do. You agree that AIs can do that. So what’s so hard about an AI reasoning about the universe? Why is that a fundamentally hard problem compared to other problems that AIs can solve? ## Structure in Text and Compression

Martin 00:45:44
Now let’s go ahead and move that over to language. So if my Occam’s razor is these things learn structure in a data corpus, it’s distribution in, distribution out, that means that they’re learning structure of the text corpus that they’ve been fed. A great example of this — there’s a recent paper I saw, I don’t know if it got accepted or not, I think it was just on archive — but that showed that if you could gzip a text corpus and the compression was good, then the accuracy was good. Which suggests almost everything you need to know about this, that all it’s doing is learning structure in the text.

Liron 00:46:20
So I haven’t seen the paper he’s talking about, but it seems to me he’s really missing the point when he says, “Hey, look, AI can operate better at things that have small gzips.” He’s missing the point that we’ve made a breakthrough in compression. The fact that we have more powerful LLMs means that we’re finding deeper structure than ever before in the data we’re getting.

So we’re processing a stream of text, and we’re mapping it down to: this describes a world. The world has actors. The world has big chunks. The world has deep connections. And gzip is blind to that structure. So when Martin is saying, “Hey, look, I read a research paper that says gzip actually describes how the LLM’s gonna do,” he’s missing the breakthrough that happened, that we’re actually getting deeper in our ability to compress things. We actually stored a deeper generator of the next token.

And that’s why, if you were to just look apples to apples, how small can you compress text — it’s called the Hutter Prize when it’s done with a relatively small corpus, I think about 100 gigabytes of Wikipedia. I think it’s too small to really reflect the progress with AI. Unfortunately, it’s kind of a bad contest for our needs. But if you had a better version of the Hutter Prize, you would just see it getting done using smaller and smaller data sizes. Because as we make advancements in intelligence, we’re making advancements in compression because we’re seeing deeper. Intelligence is predictive power, which is compression. There’s a deep connection here that it seems Martin is not acknowledging in this segment of his interview.

Compression and Intelligence

Martin 00:48:01
There’s nothing to do with underlying meaning. It’s just structure that’s in the text.

Liron 00:48:05
It has nothing to do with the underlying meaning? It’s just structure within the text? Wait, but sufficiently deep structure is meaning. LLMs have already solved what was traditionally called the “symbol grounding problem.” That’s been solved. Symbols are able to be grounded. When you embed a token in a high-dimensional space, especially with context, that’s it. That’s the meaning of the token — its relationship to other symbols.

LLMs have pretty much mastered that. Sure, they can’t fully reason from those embeddings to output any problem, to take in all the context perfectly. I mean, we know they’re not generally intelligent yet. But in terms of knowing what a token really truly means for the purpose of writing essays about it and answering arbitrary prompts, they’ve got it. They’re grounding those symbols. So I’m actually kind of surprised to hear Martin say—

Symbol Grounding

Martin 00:48:55
There’s nothing to do with underlying meaning. It’s just structure that’s in the text.

Liron 00:48:59
Because you can grant me that the structure that’s been learned by today’s LLMs actually does map to the meaning of the tokens that they’ve processed. You can grant me that and still make your argument that they can’t reason about long-tail situations. Those are two totally separate points.

So it’s interesting to me that Martin is making the separate point of, “No, the structure that it’s learned is just structure. It’s not meaningful structure.” I think he’s on very shaky ground there. To me, it’s obvious that it is meaningful structure.

Martin 00:49:28
So you can understand the distribution of text. You can actually spit out text. But this doesn’t say anything about learning fundamental principles of the world from which the text is based.

Liron 00:49:39
Wow, really doubling down here. You really don’t think that the AI is seeing through to some of the depth behind the tokens that are being generated all over the web? When it processes this text, you don’t think that it’s mapping to something deeper that is actually real, is actually meaningful about how that text was generated?

For instance, there is a famous example of Yann LeCun going on record saying, “I bet an AI can’t tell you the answer of what happens when I put a book on a table, and then I shake the table or I move the table. I bet the AI won’t realize that the book moves with the table.” But sure enough, today’s LLMs do realize when you ask them. They do realize that the book moves with the table, and you can try to make a bunch of variations of the problem, and they’ll still reason and give you an English explanation saying, “Hey, this object, whatever you call the object, is on the table, and friction will cause it to hold with the table. So unless something unusual is going on, it will end up in the same position.”

And the amount that you can vary the prompt — you can vary it pretty deeply. The only thing that has to stay constant is a mapping to an actual physical model of something on top of something, or at least something isomorphic to that in a deep way. So I’m not saying you can’t trick it no matter how hard you try, but I’m saying you can’t trick it in a shallow way. It’s non-trivial to trick it. It’s going so far beyond statistics, so far beyond being a stochastic parrot. You have to give it credit for something where it actually understands some deep level of meaning.

Liron 00:51:34
By the way, here’s that infamous clip of Yann LeCun that originally went viral when I posted it on Twitter last year. If you’re just listening on the podcast audio feed, what happens after Yann LeCun says that GPT-5000 could never solve a simple physics problem is that I type it into GPT-3.5, and it spits out the exactly correct answer, and then I play the music from “Requiem for a Dream” to symbolize that he has kind of screwed humanity by not making these obvious predictions correctly.

Yann LeCun 00:51:35
I don’t think we can train a machine to be intelligent purely from text. So for example, I take an object, I put it on the table, and I push the table. It’s completely obvious to you that the object will be pushed with the table, right? Because it’s sitting on it.

There’s no text in the world, I believe, that explains this. And so if you train a machine as powerful as it could be, your GPT-5000 or whatever it is, it’s never gonna learn about this.

That information is just not present in any text.

Liron 00:52:26
Another example, you can ask it to draw the situation. You can describe any arbitrary situation in words and then ask it to draw it. How is it mapping from words to an image? The only way to do that is with a deep representation, unless you’ve seen something very, very similar on the web.

And that’s a go-to move of AI skeptics or people who just don’t think AI is that impressive. They’re all, “Ah, it’s just repeating back a pattern it’s already seen.” But you can vary the input quite a lot. And so I guess the crux is always gonna come down to: is it really that similar? Because your concept of similarity is a concept that five years ago everybody would have told you, “It’s not that similar because we can’t build an AI to do it.”

So some really deep breakthrough has been made where this level of similarity that you as a human can perceive and seems obvious to you is only obvious to AIs now. Now that they have these giant matrices of meaning and context, suddenly they’ve figured out what your obvious similarity metric is. That’s where we stand today.

Human Abstractions and AI

Martin 00:53:24
And so it almost seems to be that the magic in this sequence of text is you’ve got these humans that have spent, say, 3,000 years looking at the universe. And the universe is, again, heavy-tailed, and it’s nonlinear, and it’s very complex, and it’s fractal, and it’s self-similar, and these are notoriously complicated systems to simulate.

So then the human brain is, “I’m gonna abstract this out as a tree, and I’m gonna abstract this out as this concept.” And so we’re these machines that take the universe and abstract it into things that are words, but it’s a very lossy representation. You can’t describe something and get back the universe. But it turns out because we have those abstractions, they’re very predictable. We’ve made the universe more linear, and we’ve made the universe less heavy-tailed, and we’ve made the universe less self-similar. And once we’ve done that, we’ve added structure that’s predictable.

Martin 00:54:12
So the fact that you can take a corpus of text which a human being has pulled from the universe and then made the universe much simpler and find structure in that is not surprising at all. But to think that somehow then you can go from that to the universe is a step that just simply has not been demonstrated. And it actually doesn’t even stand to reason.

Liron 00:54:32
There’s no doubt that when you’re ingesting words of text that humans have outputted, you’re getting something that’s high information density compared to a random pixel hitting a camera. If you look at a token of text, the amount of generality you can get, the amount of reasoning power you can get using that token as your starting representation — you’ve definitely got a head start. There’s no doubt about that.

But that said, we also do have camera systems that go from really messy pixels all the way to a tagging of everything you see in the scene, and that’s now become human level and even superhuman level. So okay, maybe it got bootstrapped by human labeling, but we now have a system that can open its eyes, look out at the world, identify what it’s seeing, and now reason about what it’s seeing.

Liron 00:55:22
So where do we go from here? This isn’t really a new point that he’s making right now. It’s just entirely the question of: okay, great, you’ve now got abstractions over what you’re seeing, and now can you combine those abstractions robustly, correctly? Can you reason? Can you have general intelligence? So we’re just back to the original question. I don’t get what the new point is besides just pointing out that a single token conveys a lot of meaning. It’s high information compared to a single pixel. And I agree with him on that, but AIs can now generate their own tokens in many contexts too.

You can ask an AI to output its internal state when it’s playing a video game that it’s really good at, and I’m sure it’ll have some information-dense tokens there too. So what? Maybe Martin would extend the argument and be, “Whenever you think that an AI is reasoning or making inferences or trying to crawl its way outside of its distribution, it’s just an illusion where it’s finding some tokens that a human has written, and it’s using those as a crutch, and it’s just regurgitating those back out to you.”

I guess you could try to make an argument like that, but the reason that argument tends to be weak is just because these people never actually predict or describe what the boundaries are. They just use these hand-wavy terms that are mostly applied retroactively and then kind of get broken in the next version of AI. There’s no predictive power telling you what the next AI can’t do or even in some cases what the current AI can’t do.

There’s been a number of challenges posed where people with Martin’s attitude of, “Hey, I know what AIs can’t do,” they pose a challenge and then the challenge gets broken by current AIs. So they’re not even experts in practice on what the current AI can’t do. Most recently, we have the ARC challenge, the Abstraction and Reasoning Challenge by François Chollet and Mike Knoop — a really interesting challenge where there’s visual puzzles that you can encode as tokens, and they’re pretty easy for humans, and AIs struggle with them. But even that challenge, we’ve been slowly climbing up ever since it was posed, 1% at a time, something like 1% every few days, climbing toward 100% just using current AI models.

So nobody, in my view, is wrapping their head around what current AIs can’t do. They’re just hand-waving. And in this particular case, what Martin is saying — “Yeah, humans have done all the work to generate these tokens” — I consider that a hand wave. What exactly is your point about the limitation of AI?

Martin 00:57:39
The fact that you can take a corpus of text which a human being has pulled from the universe and then made the universe much simpler and find structure in that is not surprising at all. But to think then somehow then you can go from that to the universe is a step that just simply has not been demonstrated, and it actually doesn’t even stand to reason.

Liron 00:57:57
So there’s two phases to the process. Phase one is you take the messy universe and you output some nice high-level structure like tokens or three-dimensional vectors — some high-level representation. That’s step one. And then step two is you’ve got this high-level representation in your head, and then you understand strategic action plans, or you can reason, or you can reach conclusions.

So there’s two different steps. It seems like Martin is now focusing on how amazing it is that humans, in step one, took the messy universe and got these tokens. But we’re seeing step one happening. We’re seeing AIs take pixels and output a bunch of objects. We’re seeing them look at a room and output exactly all the nooks and crannies of every object in the room.

So I don’t see what’s the fundamental challenge with step one. Step one actually seems like you can do it using system one — you can do it using simpler systems, and you can rapidly surpass human abilities during step one. So it seems like all the magic, all the secret sauce that we don’t know the answer to yet is in the step two part. We don’t know how fully general reasoning is going to work. That’s still an unsolved problem. That’s the part that I can see maybe it’ll take 50 years if we’re unlucky — or lucky, as I see it. I just don’t see why Martin is dwelling on part one of this right now.

Liron 00:59:33
Okay, now Nate asks the question that I had about why aren’t you granting that the AI is mapping these symbols to structure that’s worthy of being called meaning? It’s correctly associating internal mental models in a way that’s corresponding to structures in reality. That’s basically what meaning is. That’s symbol grounding.

Nathan 00:59:33
If I understood you correctly, you’re saying that doesn’t mean that they have any understanding of meaning. How would you square that with the sparse autoencoder line of research, or sort of Golden Gate Claude, if you will? They’re able to now say through these sparse autoencoder techniques that they can isolate, I think it’s 30-some million different features, each of which is a direction in activation space.

And then inject those at runtime. They’re starting to create these sort of control mechanisms where if they jack up the Golden Gate Bridge feature, then all it wants to talk about is Golden Gate Bridge.

Or more practically, insert kindness or insert deviousness or whatever.

There seems to be some meaning there, right?

Martin 01:00:17
There’s structure. That’s very different than meaning. So — we’re talking about three things. This is a great conversation because I think it gets to the heart of it. So the universe is self-similar, it’s fractal, meaning no matter what zoom level you look at it, it has the same stochastic properties. So you can spend an entire life studying a cell or a planet. That’s how much complexity is in the universe.

Liron 01:00:51
I’m not sure I understand what Martin is saying here. You can spend your life studying a cell or a planet, yeah, because those have a lot of moving parts and a complex interplay.

But if you study the laws of physics, the laws of physics actually describe things at a much simpler factored level. You can understand all the different rules that govern an electron, and then you’re done. And yeah, you can start combinatorially putting different electrons in the universe’s memory, and now you’ve got a complex system, sure. But writing down all the laws of physics and talking about fundamental interactions is going to be simpler. I don’t know why he’s saying self-similar. I don’t know why studying a cell or a planet is self-similar to studying a small number of fundamental particles when you’re just doing base-level physics.

There’s also parts of the universe that just don’t necessarily have that much detail. If you’re just studying how to play chess, chess is part of the universe, and the level of detail is constrained.

So both the outer walls of the universe — the fundamental laws of physics, the beginning of everything in that sense — that has a very finite amount of complexity in terms of bits of information. And then you also have regions of the universe that are limited to a finite amount of complexity, like the game of chess played within the physical universe. You have these different regions with different amounts of complexity, so I just don’t know why he’s choosing to use the word fractal or the word self-similar. There’s a distinction where some parts are simple, and he seems to be neglecting that when he says that the universe is a fractal. But I may be nitpicking.

Martin 01:02:14
The universe is heavy-tailed, which means the exception is the norm if you dedupe. And it’s nonlinear, which means that you can’t computationally predict out too far just because we don’t have closed-form solutions for nonlinear stuff, and it’s just a very hard computation problem. That’s the universe.

Liron 01:02:31
Okay, but at least the entropy is low as fuck. I mean, don’t forget to name that advantage, because most universes that are possible to define have much higher entropy than our universe. They just have much more chaos. They don’t factor into these super simple laws.

And it’s these super simple laws, it’s this low entropy that lets an ape in the savanna or an ape in a suburb look out into the stars and make a ton of correct predictions. I mean, we know the motions of the planets, for God’s sakes. That’s not something you could know in an arbitrary universe. So let’s not neglect the advantage of how easy mode this universe is given its low-ass entropy.

Martin 01:03:10
Now, human beings have had to navigate this crazy universe. And so we’ve created this amazing engine, which is the brain, and it has reduced this universe to concepts and words and stuff that we use and we talk about that kind of makes it a little bit predictable, and so at least you and I can communicate about it.

But if I tell you, “This is a tree,” there’s a concept tree in my brain, but it’s almost an arbitrary distinction that it’s a tree. I could talk about branches versus leaves. I could talk about networks of trees. That’s actually one tree, like the big aspen grove. I could talk about cells of the tree. It’s this kind of arbitrarily useful distinction. So it has some semblance to the real world. But if I say a tree, it’s probably not accurate relative to how the world is. Is it one tree? Is it a family tree? It’s just a useful abstraction.

So these models will 100% recover the abstractions that we’ve put in text because the structure is all there. That’s not surprising. Compression would do that. All it’s doing is taking advantage of structure, and that structure is real. But let’s say that it’s finding a tree. Does a tree actually map to the universe in a meaningful way?

Liron 01:04:18
Wow, okay. He doesn’t seem to be aware that the low entropy of the universe is what gives it joints that you can carve it along. So if you’re an AI or an alien looking at planet Earth, looking at the landscape of a forest, you’re gonna circle the trees. The trees are very clearly the correct factors of that landscape, as opposed to taking 2.5 trees and drawing a circle around that and splitting the tree down the middle. That is clearly the wrong circle to draw.

You’re gonna want to put a whole number of trees in the circle that you’re drawing when you’re just trying to reason about a forest. An alien can spot a tree. It’s not a human-specific concept. It’s not a culture-specific concept. You can see humans from different cultures don’t have a problem agreeing on what a tree is.

Now, of course, there’s edge cases. Is a sufficiently big bush a tree? Blah, blah, blah. Okay, fine. But realistically, in a forest, most trees, we all agree where the tree is. The alien is going to agree. Why? Because all these different properties — hey, it has a life cycle, it reproduced, it started from a single cell — all of these different properties apply to this unit of one tree.

And it’s the same with a cell. Aliens are going to analyze life forms in terms of their cells because cells are a layer of abstraction, just like a module in software. A cell has an input/output. The cell wall keeps its contents together. Why would you not draw a circle around a region in spacetime where the contents stay together?

So my point here is that the universe is low entropy, and as a result, it factors into units that any intelligence is going to recognize. I may have never looked in a microscope and seen cells before. I may have never opened a textbook and seen cells before. But the first time you show me, you can be damn sure I’m gonna be, “Oh, okay, this looks like modular units,” because I’ve done software engineering. I know what a module is. I know why abstraction is useful. I know why a black box that has input/output guarantees is useful. This is how the universe works in a way that’s understandable to intelligence.

So to me, it’s kind of a red flag that Martin is talking past this whole low-entropy module structure, and he’s just saying, “Hey, there’s probabilistic long tails.” Yeah, okay, but intelligence is sensitive to the low-entropy factorable structure of the universe around it. That’s the game being played here. That’s the nature of cognitive work — to carve reality at its joints, to model reality, to use high-level representations that map to how the universe is simple in those same ways.

Martin 01:06:44
It’s a human-created concept that has some vague semblance to something we all agree on, unless you’re a scientist, then you probably disagree with the common understanding. And oh, by the way, my concept of a tree also includes a toy and a cartoon picture of a tree, which is entirely different. And so text is a way that we as humans represent the world that’s very different than the actual universe because we find it useful. There’s structure there, and these LLMs are exploiting that structure just like compression would or anything else. And it’s very useful for us, but it doesn’t necessarily mean that these things can enact on the world. These are very different domains.

Liron 01:07:25
If I understand Martin correctly, I think what he’s saying is we humans are the ones who originally made the mental connection between different types of tree. In my mind, I’ve connected the idea of a biological tree with a cartoon tree with a toy tree, and the LLMs wouldn’t be making those kinds of connections by themselves, but they’re ingesting tokens that humans like me have been putting out, and that’s why they can reason about this stuff because now it’s in their distribution. I think that is a good recap of what Martin is saying about the different kinds of trees, for instance.

The problem with that is that I’m here talking with GPT-4, and it seems to have abstracted the tree concept beyond its distribution. For instance, I just asked it to invent a new type of tree within the domain of music just to show that it’s understood at some deep level what it means to be a tree.

And sure enough, it came up with a couple different ideas like the Rhythm Evolution Tree or the Harmonic Growth Tree. It’s basically saying, “Hey, when you have a musical composition, the way you have your tonic and your main harmony, and then it evolves into other harmonies that branch out into subharmonies, but then they collapse back into their parent harmony, but it might branch off to another child harmony.”

I don’t know how accurate this is as a matter of musical analysis, but as a matter of just understanding what a tree is and what it would mean to have a tree in a different domain — and of course, this is just one abstract conception of a tree: root, leaves, branches — but it sure is a popular one. The point is, it’s working with the concept. It’s applying the concept in a domain where I searched Google and nobody’s ever talked about the Harmonic Growth Tree before. So it is using a few leaps of reasoning here, as far as I can tell.

And Martin’s point about how, “Hey, we humans, we’re the token generators. We’re the only ones who could ever tell it what’s a tree and what’s not because that connection was made in our head” — if I understand that that’s Martin’s point, I don’t see that as a robust claim. Maybe that’s true about the dumber LLMs, but I think we’re just about past that point where you can say that only humans can invent the structure behind these tokens.

I think the AI has got a good grasp of the structure of what it means to be a tree at a very high level, such that it can work with that concept as flexibly as many humans can, especially humans with two-digit IQs. Do you really think that a human with a two-digit IQ is going to be working with the abstract concept of a tree better than the AI just did that proposed the idea of a Harmonic Growth Tree? That seems pretty competent by human standards.

So bringing it back to the high-level disagreement between me and Martin — I’m just not seeing robust distinctions. It seems like he’s throwing out this terminology and acting like it’s a useful way to think about AI, but it just seems rough, and it seems like it’s not going to hold when the next AI comes, even assuming that it’s still holding today. I just don’t see the accuracy and robustness of the distinctions he’s making.

Nathan 01:10:46
I guess I’m not quite getting the gap. There’s so many interesting results to point to recently. Did you see the one about GPT-4 finding and exploiting new zero-day exploits? This was just in the last week or two, so... And it’s increasingly impossible to keep up with everything, so I don’t expect you’ve seen everything.

Martin 01:10:48
Yeah. No, that one I have. I think when you’re dealing with this much compute and this much data, the human intuition just totally fails. And we as humans are really bad about thinking in distributions anyway. That’s not how we think. We kind of assume the world is parametric. And by the way, which is why the text that we create is so well-structured and can be exploited by LLMs. You have to navigate this universe. You have to make the simplifying assumptions, which we do.

Liron 01:11:42
Again, the key word that’s explaining what’s going on here is low entropy. The reason humans are able to specialize in having low-entropy mental representations is because they correspond to low-entropy parts of our low-entropy universe. That’s the reason why you can have a cognitive engine. Cognition works in a low-entropy universe. That’s why the neurons in our head are able to successfully perform cognitive work, and that’s why AIs are able to perform cognitive work with increasing success on a broader and broader domain, up to and including the physical universe.

Martin 01:12:05
And so it’s good to actually come up with mental frameworks about how these things work. So I’ll tell you a few of mine. The first one is, as far as I can tell, these things are exploiting structure in whatever data that they’re reading, as we’ve mentioned before, and it’s not clear whether that distribution extends beyond that. And if it does, then you’re basically back to simulating the universe, which, yeah, I’ve spent a lot of time with. I think that’s very tough.

Liron 01:12:25
I think this is a key point that Martin is gonna be making repeatedly. He hasn’t really fleshed it out in this podcast, but I’ve seen him tweet similar things. So just keep an eye out for the dichotomy he likes to draw between interpolation within a distribution and simulation. I think for him, those are kind of two sides of some spectrum. We’ll see.

Martin 01:12:54
The second one is the actual mechanisms. We’re talking about transformers. The actual mechanism is basically kernel smoothing. It’s averaging, which means to me the further you get out to where the data is rare, the greater the inaccuracies come. And that doesn’t mean that for systems that you can actually extrapolate from — that you don’t get great results. That would be, quote-unquote, out of distribution. It turns out some systems are linear or you have enough data, you can map the distribution. So that one is totally fine.

Liron 01:14:06
Yeah, he’s really committed to using concepts from statistics and linear analysis to make predictions about AI — or rather, refuse to make predictions but make retroactive analysis. He’s pretty committed to doing this, and I think he’s in for a rude awakening.

My daily experience using ChatGPT just doesn’t match this idea that it’s only picking things within a distribution. I get that it’s better when it can find examples that are similar to what I’m asking it, but I’m asking it to do novel things. I’m asking it to process inputs and map them to outputs in a novel way, and I think his mental model, which is very attached to probability and linearity — he’s missing something important, and he’s just going to get increasingly blindsided.

And once again, he’s not making any predictions about what GPT-5 won’t be able to do. He’s only retrodicting, which is much easier. So I invite him to make predictions since he’s so insightful, apparently, about how limited this LLM architecture is. Please be my guest. Make a prediction. I’ll happily take your bet if you put any kind of attractive odds — if he thinks something is three-to-one likely or unlikely, and I have the more one-to-one uncertainty position, maybe we can set up a bet.

Martin 01:15:21
Now, the third one is this in-context learning one, and I think Vishal Mishra, who’s a professor at Columbia, did the best work on this, where he actually shows that for in-context learning, where you actually put the context in the prompt and you can move the posterior distribution to get interesting results, he mapped it specifically to Bayesian learning. And it’s a beautiful paper. I don’t know why more people don’t read it.

So listen, we know that these things can do some basic Bayesian reasoning, and this is where the prompt is basically the new evidence which changes the posterior function. So you’ll get new stuff there. We know that if you average enough stuff, you’ll get new stuff there. It just has to be linearly interpolatable. If it’s not linearly interpolatable, you’re not gonna get new stuff.

So none of these things suggest that you’re not gonna get new stuff. It just puts constraints. We know how Bayesian systems work. We’ve got 20 years of understanding convergence properties to them. We have work that’s specifically mapped ICL to Bayesian learning, so let’s just go ahead and use that corpus of work to understand the properties. It doesn’t say it’s out of distribution or in distribution. It doesn’t say that at all. It just bounds what that means. And then we also know the mechanics of the way transformers work, which is this kernel smoothing — I mean, it’s more complex than that. And so that can create new things, but it means there’s a linear interpolation.

Liron 01:16:27
It’s just linear interpolation, but it’s Bayesian, and also there’s kernel splitting. Yeah, there’s a lot of stuff you threw into the mix there, and I don’t think that it adds up to somebody who just listens to what you’re saying and tries to use that to understand an AI that they haven’t seen yet, tries to make predictions into the future.

Again, send a message like that with that insight that you have. Package it up, send it to Martin from five years ago. Have Martin from five years ago try to predict what the AIs of 2024 can and can’t do. I think that the Martin from five years ago would be stumped because I don’t think that you’re making sense when you say descriptions like that.

You’re just saying descriptions that you can say, “Look, my description maps to the AI we see today.” And people who are already familiar with the AI that we see today can nod along and be, “Sure, sounds like you’re making sense.” But I think that these words are meaningless the way that you’re using them, and they have no predictive power. And given the stakes — that being stuck in a description like that can blindside you to the emergence of superintelligence that destroys humanity — I would encourage you to try saying a description of AI that actually compiles into something that lets somebody predict something.

Martin 01:16:59
I feel like when we have these discussions, they should be a bit more principled as opposed to, “I’ve got this anecdote that seems like something’s new,” because nobody says that you’re not gonna see new stuff. I mean, it’s very obvious if you’re doing interpolation, it’s new. Very obvious if you’re doing Bayesian reasoning, something’s gonna be new. And talk more about the distributions and the theory of why we’re doing that. But as far as I can tell, that’s totally missing. Nobody’s come and said, “Here’s my theory of out-of-distribution stuff. Here is my thesis for what is going on functionally to create this new data.”

Liron 01:17:02
Yeah, my thesis is that you’ve got an architecture with a shitload of parameters, and that architecture is capable of learning any function in principle. And then you get a bunch of data from a low-entropy universe, and then you learn a bunch of deep patterns in the data, and then you just truly understand a bunch of concepts from the universe around you.

And the last question is, how much can you extend your reasoning robustly so that you have general intelligence? And that part isn’t answered yet. But the fact that you can pass the freaking Turing test when you couldn’t before, I would call that potentially out of distribution. It certainly wasn’t in the distribution that any AI was able to extrapolate before we had this totally new transformer architecture.

So Martin is now trying to shift the burden of proof to the camp that’s saying we’re getting close to general intelligence. He’s saying, “Why would you think that this can reason outside of distribution? I’m the one with the default hypothesis, which is it’s just stuff that we know before.” It’s like, really? Passing the Turing test is just stuff that we know before?

You use the term Bayesian reasoning. That’s a very powerful term. I’m guessing that you meant some simplified version of it, like just a single neuron doing one layer of Bayesian reasoning or something. I don’t know what you meant. You’re being kind of hand-wavy.

The thing that is less in doubt is the frontier of applications. We are now at this new frontier of applications. What AI is doing with language, with images, with being useful as a question-answering engine or writing essays to a degree that we’ve never seen before — this is new stuff. You can’t deny that this is new stuff. This is stuff that we used to be confused about building, and it’s all coming fast. It’s all coming from the same architecture. We’re running that architecture on a low-entropy universe. It’s going to keep absorbing stuff, and we just don’t know where that leads. Hand-waving and saying, “Ah, yes, I understand everything. It’s just part of a distribution of stuff” — that’s not reassuring. You may be right, but it’s not like you should be confident in what you’re saying here.

The Frontier of AI Applications

Martin 01:18:55
And on the other hand, you’ve got mounting and tons of evidence that map these to existing systems that we know that people just seem to not want to follow. And I just feel like, listen, humans love to see things in clouds and complex systems. There’s so many facets, and they’re so complicated, and they’re so huge, and they’re so ethereal, and then we see things. And we just do this historically.

And we’ve got these kind of amazing compute elements that are huge, and they surprise us, and they’re amazing. But we can map them to formal systems, and we know how they work. And that doesn’t mean that they’re dangerous or not dangerous. I’m not saying that. I’m just saying that we can actually map them to reasonable systems to have a discussion, and that just seems to be missing. This conversation’s a great example. I’m very happy to map these things to formal systems we understand and have that discussion. But it’s always this kind of anecdotal whack-a-mole instead, which I just don’t know how to answer to every instance of what seems like emergent behavior when, of course, emergent behavior is expected anyways.

Liron 01:19:46
What I would like to know from you is just be clear about what your boundary is when you talk about being out of distribution, because my subjective opinion is I get plenty of stuff that’s out of distribution every time I use ChatGPT. I’m not just looking for the stuff that’s in distribution. I’m looking to connect things together in a novel way.

So you need to be clear. You need to have a criterion for how to distinguish things that are outside versus inside distribution. What’s your similarity metric? Because the naive metric of just a new token string — obviously things are outside distribution. You would agree that things are outside distribution. So you need to clarify what your similarity metric is to define what’s outside of distribution, something I asked earlier in the podcast.

And then if you can do that, then go ahead and put down a prediction of what GPT-5 definitely can’t do because of your insight about it not being able to go outside of distribution. Go ahead and predict what that would mean for GPT-5. I would be very impressed if you did that. Hell, I would update my beliefs. That’s what I’m here for. I’m here to learn from your insight. Teach me your insight in the form of a prediction. Let me update on your wisdom. Thanks.

Nathan 01:20:49
On the sigmoid question, I tend to also agree that it does not seem like there’s reason to believe that this is gonna be an exponential forever. It does seem like it probably levels off. But then I’m also reminded of the old joke of two guys in the woods, and the bear is coming, and the one’s putting on his shoes, and he says, “I don’t have to outrun the bear. I just have to outrun you.”

Martin 01:21:09
Yeah.

Nathan 01:21:09
And so I do wonder if we imagine continuing to scale up as we have been scaling up, and there’s all these trend lines and X times more compute and however much advantage from algorithmic efficiency or whatever. Let’s imagine we continue to scale up and it’s a few more orders of magnitude, and let’s say we don’t just put in the text, but we also put in this sort of low-level solution data and the protein — you have the DNA data and the protein and the gene expression, and we work our way up all these levels of orders of magnitude.

And then it’s computer systems — all the cloud logs from AWS and Google Cloud, and all this stuff gets in there, and you’ve got all these different self-similar but overlapping orders of magnitude of ways of understanding the world.

Even if it asymptotes or levels off at some point, I have a hard time imagining that doesn’t level off at a higher point than a human is able to achieve today. But I feel like you probably see that differently still.

Martin 01:22:09
So a lot of this reduces to how you view the universe. If you don’t view the universe as fractal self-similar, and if you don’t view it as heavy-tailed, and if you don’t view it as nonlinear, then you could imagine that. But it is all of those things. And we know it is all of those things. And so there’s no distribution of data that we know of that’s not the universe that will produce something that’s predictive of the universe.

Liron 01:22:32
Really? What about a description of Elon Musk’s mental model of the universe? If you can operate that, can’t you be quite powerful and have quite high predictive power about our universe? Seems like it. And again, the thing that makes this possible is that the universe is low entropy. So looking at Elon Musk’s model of the universe really does get you, in practice, most of what you need to get from looking at the real universe.

Now, can you surpass Elon Musk just by having that model? Not necessarily, but this already climbs you up to a human level just for a start. So before we even talk about the secret sauce of general intelligence, there’s already evidence that you can already suck in a lot of the distribution that you need to operate in the universe. Even before we get to the essence of general intelligence and what’s missing there, the idea that, oh my God, the universe is so big, how can you ever ingest enough data to be in distribution?

I think a good intuition is look at how little data you need to ingest everything that Elon Musk has ever seen and everything that Elon Musk’s genes have ever stored from their evolution. It’s just a few megabytes of data, for God’s sakes. So appealing to the vastness of possible distributions doesn’t move me very much when I see how humans barely know anything, and it still makes them very powerful.

Human vs. AI: Concept Creation and Reasoning

Martin 01:24:06
It’s really that simple. Now, that doesn’t mean that we can’t focus on an area and reproduce that distribution. I could become very good at predicting whatever, protein folding. I could get really good at playing chess. But it’s distribution in, distribution out.

Liron 01:24:06
I don’t agree with distribution in, distribution out. I think there are reasoning steps being made, and there’s a few right now. There’s gonna be more later as the models get more sophisticated.

I guess at this point, I’d be interested to ask Martin, “Okay, so what do you think is gonna happen when we tweak the architecture?” Let’s say I even grant you, okay, yeah, the transformer architecture will plateau because it’ll always stick relatively close to its distribution in some sense. Let’s say I even grant that. I think it would be a pretty wild sense — I don’t think you can get as much amazing stuff as we’ve seen and have that be an accurate description. But okay, I’m granting it to you. It’s staying within its distribution.

So what do you think is gonna happen next? Do you think maybe somebody will have a way to take a few reasoning steps and get out of distribution that way? You do seem kind of frozen in the status quo.

Well, in the beginning of the podcast, Martin referenced the idea that it’s always incremental progress, and it always takes longer than you think. So I’m just curious, what timeline are we working with in his mind? Does he think it’s gonna take 20 years? Or is he thinking 100 years like Robin Hanson? I’m just curious where he’s going with all this, because he’s dwelling a lot on the limitations that you can argue exist today.

Martin 01:25:17
Have you noticed the ones that stuff like recursive self-similarity works? And control loops work and simulated data works, synthetic data works. There are these axiomatic areas where the axioms constrain the search space and you’re basically converging on search.

And you can get very good at those. I can get much better at unit arithmetic. I can get much better than you at game playing. I can get much better than humans at all of these things. But none of that talks to the fact that can you find the right level of abstraction in a fractal system, and can you tackle a heavy-tailed universe where not by occurrence, but by uniqueness, the complexity is in the tail?

Even in this discussion, the use cases that you point to tend to be these kind of axiomatic — of course we can do full search. We thought in the late ‘90s we could do protein folding by fully searching the search space. It’s just not surprising to me that we can learn distributions and spit them out.

The Complexity of the Universe and AI’s Limitations

Liron 01:26:20
Okay, this seems like an important issue to take up with Martin. When he talks about protein folding being a problem that’s obviously just one of distribution — and obviously we’re gonna make some AI that’s going to interpolate the correct prediction of a protein’s folding just by looking at other data about protein folding — of course we’re gonna get there, it’s that type of problem.

That’s quite a distribution with quite a lot of degrees of freedom that we’re now treating as not a big deal. I mean, this was an unsolved problem for a very long time for a good reason. It’s even an NP-complete problem. But of course, we’re not solving it in the general worst case. We’re using heuristics, and it’s not always accurate.

But point is, protein folding was not a statistical distribution problem. And I know that modern AI approaches use an architecture that you think is heavily statistical distribution-based, but I just don’t think that’s right. I think you’re seeing a new power that’s more than just statistics. It’s parameterized learning of a nonlinear system, of a system that’s incredibly flexible in what it can represent. And it’s defying your simple description.

I think you’re missing something important that’s happening, and you’re brushing it off just because we did it. You’re acting like you predicted it. Because you were familiar with this problem, I guess it didn’t come as a surprise to you when it was solved, but I don’t understand why. NP search problems aren’t tractable and don’t necessarily allow heuristics based on statistical distributions. I would like to unpack what he’s saying here because I think he may just be confusing himself.

Martin 01:27:44
I feel that’s a very different statement than saying, “Now we have a model that can navigate the universe in a way that is predictive of all of the complexities of it.” Maybe another way to think of this is we’ve spent 3,000 years doing our best and writing it down, and we can create a model that can learn from all of that and do what the mean human being would have done in the last 3,000 years.

But the problem is the stuff that we’re doing tomorrow — by uniqueness, a lot of it’s gonna be new, and that’s just how the universe works. And we’re gonna have to either build a machine that can do that, which we don’t know how to do, or we’re gonna have to do it ourselves and let these machines do the mean task.

AI’s Potential in Biology and Simulation

Liron 01:28:23
But I didn’t do anything today that was new, did I? I was just home all day, took care of my kids, used my computer, recorded this podcast. What am I doing that’s new? And why can’t I have an AI come and do this? What’s the issue? Why are we talking about the full complexity of the universe? Isn’t there enough data in the last 3,000 years of what humans have done to let the AI come and take care of my kids and record this podcast? I think Martin is wrong to dwell so much on the idea that the universe is big and has fat tails.

Liron 01:28:53
I think Martin may not have a healthy respect for search problems if he can see them as being well-defined. So he sees what ChatGPT can do today, and he’s, “Eh, I have enough data points in that distribution. It doesn’t matter that you’re searching this exponentially large combinatorial space of possible essays that you could write me because you’ve seen other essays.”

It’s like, hello — other essays? You can see a trillion essays, and that’s a microscopic fraction of the entire space of essays. Microscopic is an understatement. I think Martin doesn’t have a healthy respect for how crazy it is that you can locate an acceptably good essay in that kind of vastly combinatorial space.

I think he thinks that’s a lot less impressive than predicting how a simulation is going to go down in a heavy-tailed universe. He has a higher level of respect for that for some reason. But I would encourage you to have more respect for searching in well-defined yet exponentially vast spaces.

When your search space is exponentially vast — 10 to the power of a million, much, much larger than the size of the universe — that’s your search space, and then you have a mere trillion examples. That’s not interpolation. That is true creativity when you can do a search like that on just a trillion examples, because finding the answer is highly improbable relative to any kind of naive algorithm.

Relative to the best algorithm that humanity could field five years ago, the search space was intractable. Even if you’d given humanity those same trillion examples five years ago, the search space was intractable. So you’re missing an actual achievement of creativity, an artificial intelligence advance that we still don’t fully understand because it’s achieved not by just multiplying a bunch of matrices, but also by applying nonlinearity and also by preserving context using the trick that is transformers and by using billions of parameters that are capable of learning any function.

There’s a lot going on here. You gotta stop dismissing this amazing feat that’s happening when ChatGPT spits out an essay even though it’s seen other essays. You gotta stop focusing on the universe having fat tails and having the three-body problem and chaotic systems. That’s not where the action is. That doesn’t explain why AI can’t live Liron Shapira’s life, which takes place mostly at home in a suburb. You’re just distracted by the wrong thing. ## Experiencing the Universe and Creating Concepts

Nathan 01:31:14
So do you have an account of or theory of what it is that you think humans are doing that the models—

Liron 01:31:23
Can’t do today?

Martin 01:31:24
Yeah. Two things. One of them, and the most important one, is we experience the universe and we abstract it into concepts.

Liron 01:31:31
Abstracting things into concepts does seem like something that current chatbots can do, but even if they can’t, we’ve given them tons and tons of concepts. So just being able to operate the concepts that we gave them, you would think would make them good at different jobs.

How many jobs require new concept creation? Is that really the bottleneck displacing people, concept creation? I thought it was more like reasoning. So it’s interesting that you’re going with that as the differentiating quality of humans.

Martin 01:31:58
What I would argue is models are very good at taking the output that humans create and being able to reproduce that distribution. They’re very good at that. That’s not the game. The game is looking at the universe and creating the supervised data. That’s the game.

Liron 01:32:14
Yeah, that’s the game if you’re a physicist maybe, but is that the game if you’re a software engineer? You really have to look at the universe. You can’t just look at tokens created by humans?

It’s getting kind of funny how he’s so obsessed with the physical universe having long-tailed chaotic phenomena while ignoring that other professions may not need to construct their own concepts and tokens, but they certainly need to be creative to find an optimal solution in an exponentially large search space. That’s where a lot of cognitive work happens.

So I definitely recommend reframing intelligence to searching vast combinatorial spaces, even if they don’t have fat tails, as if it matters in an exponentially sized space.

The Essence of Intelligence and Creativity in AI

Martin 01:32:53
I think one model to look at this is, again, human beings have been around, let’s say, in a capacity for writing things down for 5,000 years. So you’ve got humans for 5,000 years that have been looking at the universe and doing this thing that models cannot do, which is making a decision: “That is a rock, and that is dust, and this is a concept, and this is a relationship, and I’m gonna write it down. And then as a group, we’re gonna synthesize these ideas and work at these.”

We’ve got this almost platonic representation in our heads of the universe, and that is very structured. So we did all the hard work. That is hard work to take this untamed universe and reduce it into words and concepts. That’s hard. No LLM that I know can do that. Not even close.

But then once we’ve done all of that hard work, is it surprising to you that there’s structure? We did all this work. Of course there’s structure. So you take that structure, you put it into an LLM, it learns the structure, and it can spit it back out. So the very specific thing that these LLMs can’t do is look at the universe and recreate this kind of structure.

Liron 01:33:58
How did the human brain figure out that the pixels coming into our eye are often pixels that have bounced off three-dimensional objects? Did we figure that out using our intelligence? No, it’s hardwired. So the brain already comes factory-installed with some firmware — the idea that three-dimensional solid objects exist in our universe, and light is going to be a surface-level representation of those 3D objects. That’s firmware. Animals have the same firmware. We can bootstrap the AI with that firmware.

In fact, if you use Oculus or Apple VR, you’re getting a construction of your surroundings using a really good AI algorithm in addition to your camera. So that level of mapping low-level input to high-level concepts is clearly a solved problem.

And you’ve got another solved problem that LLMs can do, which is when you describe a scene, you can get out a drawing of that scene. You can talk to ChatGPT, and you can say, “Hey, the furniture is here. The room looks like this. Okay, now draw it.” Even if you describe it in a very roundabout way, in a very novel way, it’s going to draw what you described. It has a mental model. It has a world model.

So I don’t know what you think is unique to humans on this front. I’m willing to acknowledge that maybe humans are better at it. But don’t you think that the AI is nipping at our heels? How much of a lead do you think we have here?

Martin 01:35:14
To be super clear, that structure is not the universe. That’s very different. A rock is a rock. It is an idea in our head. It’s not representative of any single thing. Is a grain of sand a rock? I don’t know. Is a boulder a rock? I don’t know. These are concepts we’ve created in order to navigate the universe. They’re not the universe.

Liron 01:35:33
Yeah, but it’s the natural way that the universe factors. So aliens would also have that idea of the grain of sand. And it really does only take one universal algorithm, one general intelligence algorithm, to map the visual stimulus of looking at the beach to the 3D model of grains of sand, where you treat an individual grain of sand as a concept in your model because it’s a useful high-level concept to have. And it’s not just humans. It’s a convergent concept.

Yeah, how big does the grain of sand have to be before it becomes a rock? Who knows? But concepts — it’s in the nature of concepts to have those kind of fuzzy boundaries. That doesn’t change the fact that many grains of sand are just obviously grains of sand and not rocks, and that’s what makes it a useful concept. And aliens would know that too.

Martin 01:36:14
So human beings take the universe and create the concepts. And that’s structured because we need structure in order to do anything. The LLMs learn that structure. It feels like the only thing that works is basically you can do exhaustive, quote-unquote, “AI,” which converges on search to learn distributions, or you do supervised learning, which human beings are doing the hard work, in my opinion, by labeling things and everything else, and then you just learn what the human beings have done.

But to take the universe and actually to rein structure and all that complexity, that’s what we do. And it would be great if machines can do it. I’ve never seen any evidence that they can. Maybe some glimmers of it. That’s just not where we are.

Liron 01:36:53
A neural network is all about creating higher and higher levels of representation in the different layers of the network. Creating concepts is what it’s doing. When you put a weight inside of a neural network, that weight is saying when to activate a concept. That’s what neural networks do.

If you’re really impressed by the fact that humans invented rocks, guess what? A visual object recognition AI is training itself to have all these different concepts, and that’s how it recognizes things. So it’s very much doing the same kind of concept instantiation that the human brain is doing. I don’t know why you’re picking on this, the example of a human chunking the universe into a rock. I don’t know why you’re picking on this as the ability that AIs don’t have yet. That doesn’t seem to be what’s missing right now.

Martin 01:37:32
Yeah, I think the—

Liron 01:37:32
I just wanna be super clear because you asked me a very specific question. I’ll give you a very specific answer. Look at the universe and then come up with these concepts that are useful—

Martin 01:37:41
That are not the universe. They’re concepts. They’re totally separate. A rock is not a thing. It’s a human concept. But to take the universe and decide something is a rock, that’s actually all of the complexity and all of the energy is that step, which LLMs just don’t do.

Liron 01:37:55
The training step of an AI where it processes tokens and tries to predict the next tokens and repeats and trains all the different weights in this huge neural network, that whole training process is designed to encode high-level concepts in those neuron weights. And nobody knows if all of those concepts are similar to what we as humans would have as our concepts. We know for sure that some of them are gonna be human-like because after all, the universe does naturally factor in many ways.

But if you look at the more complex abstractions that humans use, nobody knows for sure whether GPT-4 is using the same high-level abstractions, how it does its complex reasoning. When you ask it to write an essay and it just spits out this brilliant essay really quickly, nobody knows if the high-level types of reasoning that are happening within GPT-4 are the same as the ones humans are doing. But the point is, it’s doing it.

When we talk about training an AI, we are talking about creating inside the neural network a high-level concept given only low-level tokens as input. Now, of course, you can say the tokens were high level in the first place, humans thought of the tokens. Fine, but the concepts are even higher level than the tokens.

So when he’s saying LLMs can’t take the universe and come up with a concept that describes it, that’s literally what it means to train a deep learning algorithm. It’s to take a low-level stimulus and come up with a high-level representation that makes sense of it and reflects the underlying low entropy of it. That’s the name of the game. That’s cognitive work. Humans do it. Evolution did it when it built us. Neural networks do it when they’re training.

Do they do it at inference time? Maybe not as much. I would argue they still do it. But if you need to think about it this way, just imagine an AI that can do a training run in the course of running itself. If that makes it easier for you to think about the idea that AIs can instantiate new novel concepts, just imagine it does another training run while it’s running.

Martin 01:39:47
We know enough about the natural sciences to build predictive models, and we have for a very long time. I can simulate a supernova on a computer pretty accurately.

I think it’s phenomenal. It’s phenomenal that we have an approach to throw compute at a problem of learning a fundamental law of physics. But how is that any different than the fact that we’ve been modeling physical systems for a very long time, other than the fact that in these cases, it allows us to apply more compute at the problem, and we can solve problems that we don’t have closed-form analytical solutions to?

It almost feels to me like an extension of simulation, which would be very different than the claims around general reasoning. It’s literally learning laws of physics, which we’ve been doing since the beginning of compute.

The creation of computers was for ballistics. The reason that we created computers is because ballistics are nonlinear. The trajectory of ballistics are nonlinear, so we had people in rooms that would create logarithm tables by hand. They were called computers. That’s where the name came from. ENIAC shows up. ENIAC does it four orders of magnitude faster. It was about 5,000 times faster than a human being, and we have a computer. And this is a perfect example of a nonlinear system, which is a trajectory with gravity being done by a computer. Why is this not just a straightforward extension of exactly that, like we’ve been doing for the last 80 years?

AI’s Future Capabilities

Liron 01:41:13
So yeah, I’ve diagnosed Martin’s problem, or at least the crux of where he and I disagree. He keeps underestimating what a breakthrough it is when you have a problem in a giant exponential search space, and suddenly you can solve that problem. He’s acting like an AI playing Go is just an extension of the ENIAC computer running a brute force search. He’s like, “Oh, look, computers are faster now.” No, when you can play Go, at that point, you’re actually getting into creativity.

The essence of creativity is to take an exponentially vast search space that a brute force search can’t even begin to search efficiently, can’t even begin to find acceptable solutions, and then to somehow come up with acceptable solutions anyway. To somehow find an ordering in this vastly exponential search space, an ordering where you can pluck something out at the top of your own search ordering, which seems like it should be way far down a naive search ordering.

Somehow you’ve flipped the search ordering, you’ve narrowed down, you’ve overcome the a priori improbability of finding a certain satisfactory point in the search space. You’ve somehow done it. You’ve beaten the exponentially tiny odds, and there’s no way to explain it other than to say there’s a bunch of complex algorithms — true artificial intelligence. There’s the same kind of tricks that the human brain is doing, where you’re noticing the low entropy of the universe. You realize that the exponential search space has a bunch of deep types of structure. You model the structure. You have interplay between the elements of the structure. You have a lot of different context shaping how you navigate the structure. We’re just talking about the essence of intelligence here.

He’s really just trying to reduce the problem. He’s trying to write off all the different problems that AI can solve and being like, “Eh, that’s just search.” No, I’m sorry. The way that GPT is able to converse with me and build models based on the words I say and create novel constructions, do a little bit of inference using the concepts that it knows, operate high-level concepts, know what I’m talking about when I refer to something obliquely, understand what I’m even talking about, make sense of it — the way that it’s doing all of these things is not an extension of other types of computing algorithms that we’ve had. It’s really a new type of algorithm. We have made actual breakthroughs with LLMs, and yes, we’re not 100% of the way there. They can’t fully reason robustly. I get that something is missing, but he’s not putting his finger on it.

Nathan 01:43:27
Here’s my expectation. Tell me if you — maybe we could make a little friendly wager on this.

Martin 01:43:31
Yeah, yeah.

Nathan 01:43:32
I think that over the next year, we are going to start to see scaled up foundation models for biology that are going to start to understand the super complicated interactions between genes, between proteins in cells in ways that are inferred from inputs and outputs, learning these higher order concepts in the middle—

Liron 01:44:00
Yeah.

Nathan 01:44:00
—which we could not simulate because it’s computationally—

Liron 01:44:04
Right.

Nathan 01:44:04
—just intractable, and which we certainly don’t have a closed form solution for either.

Liron 01:44:09
I agree.

Nathan 01:44:09
And if that does happen, that would seem to constitute to me an instance of looking at the world, looking at—

Liron 01:44:16
Mm-hmm.

Nathan 01:44:16
—basically raw data of sequences and just lysed cells and what proteins were found in them and whatever, and learning meaningful abstractions. And I would expect that we’ll start to discover stuff by doing counterfactual experiments on those models. In other words, tweak a thing, see what happens. Find medicines. Find disease patterns by make a tweak, see what happens. What if we change this counterfactually and then go validate those things in the wet lab?

Liron 01:44:44
Yeah.

Nathan 01:44:44
If we start to see that happening, would that to you represent—

Liron 01:44:48
I agree.

Nathan 01:44:49
—the phenomenon?

Liron 01:44:52
I fully expect that. Listen, I think it—

Nathan 01:44:52
But how does that not constitute looking at the universe and figuring out what’s what?

Liron 01:44:55
I’m glad Nate is asking Martin to make a prediction because as I’ve said before, I do think that the way he thinks he’s explaining things has no predictive power, and I think he’s deliberately refusing to make predictions. So let’s see if he tries to take his mental model and say, “Aha, I know what AI can’t do. AI can’t form a deep understanding of what’s going on inside these cells because it doesn’t have enough concepts.”

I wonder if he’ll make a prediction of some kind of limitation that AI has where it can never tell us what’s gonna happen inside these cells because it doesn’t have enough deep concepts the way a human scientist would have. So it’s blocked on doing human level science. I wonder if he’ll apply his model to make a prediction like that or if he’ll just keep hand waving and refusing to make a prediction. Let’s see.

Martin 01:45:33
Oh, for sure. For specialized subdomains, you absolutely can learn distributions. 100%. Listen, I’ve implemented Navier-Stokes — fluid dynamics where this is a turbulent, chaotic system. So we’ve known how to implement very complex systems with computers, for sure, and especially in very specific domains where we can reduce these things to a few fundamental forces or we can reduce these things to mostly linear systems.

The previous version of this is just all the computational methods where we would take a problem that we know and we’d actually experimentally determine — just experimentally we’d say, “Okay, this material behaves this way under this pressure and this heat, and it has these properties.” And we take what we learned experimentally and we’d create these models, and they would do pretty good at simulation.

And I would say this is a very straightforward extension of that, which is you look at how the world works in a constrained situation, and then you can predict what would happen in the constrained situation. But just like simulations, remember we could do this with simulation. We’ve been able to do this for a very long time. You could experimentally determine how different materials work, and then you can actually simulate a new system based on those. This is how we do most industrial design today anyways. So my question to you is, how is this fundamentally different than that—

or not a natural extension where you’re giving it a system, you’re learning some fundamental properties, you can do something new? But that doesn’t mean that you can disobey the laws of physics in predicting stuff that computers can’t predict. It’s not obvious to me that it can simulate complex nonlinear systems that are chaotic for long periods of time. I think this is just yet another step on this kind of — we have computers simulate physical stuff and we’re simulating the next thing with the next tool. Does the parallel make sense?

Liron 01:47:29
Let me try to recap what I’m hearing. He’s saying, sure, you can build this new type of AI that can analyze a cell and go beyond human scientists in coming up with conclusions: “Hey, I think this is what’s happening in the cell. I think this drug might work for this reason.” Basically surpass human scientists in the domain of coming up with practical solutions to get stuff done inside the cell.

But he can write that all off because he’s like, “Look, we have enough equations that describe these low level phenomena, so it’s actually just kind of like running a simulation, and we’ve had simulations before. This isn’t a novel breakthrough in that sense. After all, I one time worked on a Navier-Stokes system, a system that would simulate currents in the atmosphere, how the air flows, and I was able to parallelize that, have a bunch of different chips, and I was able to get a decent approximation of what’s happening in the air. So therefore, I’m not gonna be impressed when the AI comes up with good answers to what’s gonna happen in the cell.”

But wait a minute. He’s once again missing a very important concept — the concept that when you wanna narrow down an exponential search space that has some kind of deep structure, if you can do that successfully, that is true intelligence. There’s nothing else to it in terms of observed behavior.

Intelligence is that which can look at a non-trivial exponential search space and then somehow get to that tiny, tiny point in it which is surprisingly good, that there is no naive way to get there. That’s the essence of creativity, the essence of intelligence, and in Martin’s mind, that’s not really a concept that he’s invoking to explain what’s going on.

His mental model is this dichotomy where some algorithms create concepts or do whatever secret sauce humans are doing, and then other algorithms merely simulate using concepts we have or merely do statistical interpolation. So he’s not really seeing this other thing that I’m seeing, which is intelligence is somehow modeling an exponential search space and then getting to the good parts.

But to me, it’s very revealing that he’s describing the “do cell science” problem as something that’s just like doing a computer simulation of Navier-Stokes. No, that’s not what it’s doing because that would require too much computation. He’s not minding the gap. He’s not realizing how much computation it would take for his description to be accurate as to what the computer is doing. It’s taking a shortcut. It’s taking an exponentially powerful shortcut that no human knows how to take. I’ve just described actual intelligence.

So I just don’t get why you won’t grant that that’s intelligence. Now, on the subject of prediction, I guess he actually is predicting that this could happen. This isn’t something that’s going to surprise him. I thought he was gonna be like, “Oh yeah, sure, AI could never do that.” But no, he’s going the other way. He’s saying, “AI can do that, but I’m not gonna be impressed.”

So it is still interesting to see where he draws the boundary of what AI can do and what it can’t do. If he’s just going to answer that, “Oh yeah, it can do that, it can do that, it can do that,” at some point it’s — okay, can it just take over the Earth? What’s going to stop it? So now I’m interested to ask him to make a prediction of where he would draw the line. What’s the least impressive thing that he thinks that AI can’t do in the next two years? I would love to know what he thinks that answer is.

Martin 01:51:28
If you go to the ‘80s and ‘90s, we would literally empirically test physical matter. We didn’t know how the physical matter worked. We just empirically tested. We’d have, it has this opacity under this heat. It has this tensile strength. We’d use that to build databases of materials, equations of state, we call them. This is how they interoperate. We’d use those when we’re doing simulations, and we’d simulate what happens when a car explodes, what happens if an airplane runs into a building. All of these things. None of those instances worked. They were simulations, and they were very accurate. And none of them came from first principles. They were all empirical.

But that has its limits because it didn’t solve climate prediction past 15 days. It didn’t allow us to simulate life. And so to me, this is just computers being attached to another domain, which thank goodness we’ve come up with a great tool that’s gonna give us a little bit more insight. But it’s a little bit more insight, is what it is.

Liron 01:51:36
Whoa, hold on a second. You think Nate’s scenario about an AI doing cell science is just a little bit more insight? That’s all you think it is? Here’s the scenario again.

Nathan 01:51:57
Scaled up foundation models for biology that are going to start to understand the super complicated interactions between genes, between proteins in cells in ways that are inferred from inputs and outputs.

Liron 01:52:21
One of Martin’s main points that he keeps repeating is that the universe has heavy tails. You can’t just use a computer simulation to predict what the universe is going to do, according to Martin, because it quickly diverges into chaos, into non-linearity. That’s a major point that he likes to make. And he says the universe is self-similar. It has this fractal structure where there’s a surprising amount of detail on every level. He even specifically used the example of a cell.

Martin 01:52:26
You can spend an entire life studying a cell in a planet. That’s how much complexity is in the universe.

Liron 01:52:26
And now Nate is asking him the hypothetical: “Hey, what if AI gets really good at predicting what’s happening in the cell?” Kind of the essence of intelligence, the problem that’s supposed to be the hardest problem that Martin is saying AI doesn’t have a handle on, and that’s Nate’s scenario. So it seems like Martin is flip-flopping and saying, “Yeah, if AI could pull that off, that would be a little bit more insight compared to what we have today.”

Martin 01:52:45
It’s a little bit more insight, is what it is.

Liron 01:52:47
No, it’s more than a little bit more insight. That is the exact prediction that’s supposed to reveal whether your mental model is useful or not. You gotta help us make your prediction falsifiable here.

So there’s a general pattern where Martin is just not drawing the boundary of what he thinks AIs can’t do. He’s just saying, “Yeah, maybe it’ll do that, and that’ll justify my model. Maybe it won’t do that, and that’ll justify my model.” I think, unfortunately, that his MO is to just retroactively justify everything that happens as if it fits his model, but his model is incoherent.

Martin 01:53:15
This is why, by the way, we stopped PlayStations from going to the Middle East. That was exactly this. So again, I worked, I actually worked in the nuclear weapons program at Livermore, so I was very close to the previous version of these discussions. We’re like, “Oh my goodness. If Saddam Hussein gets PlayStations, he’s gonna be able to simulate nuclear weapons.” Just totally misunderstanding that the ability to simulate something is not some runaway process that’s gonna allow you to recreate a world or anything like that.

Liron 01:53:43
But simulation isn’t an accurate description of what’s happening in Nate’s proposed hypothetical scenario. Nate’s scenario is we’re going to tell you what happens counterfactually when you do different things to a cell, and essentially how to engineer a cell more powerfully than we could ever engineer before.

So you’re telling me that upgrading human engineering on a fundamental level better than any human could ever do with their own brain, that’s just simulation to you? Simulation normally describes when you have a set of low-level operations, and then you just run them with a ton of computing power, and then you just see what happens as an output. But that doesn’t work for the cell. You can’t use a simulation algorithm to get that many high-level insights about what’s happening in a cell because as you yourself say, it’s a chaotic phenomenon.

That’s why we’ve never had giant clusters that we use to simulate cells and get that many useful results out of. It’s just never been a tractable approach because the cell has too many moving parts that have too many complex relationships. So we don’t have computers big enough to do actual simulation.

It’s crazy to me that he is dismissing Nate’s scenario as just simulation. It’s obviously using shortcuts that no human being understands. It has created its own shortcuts. The shortcuts work by using different levels of understanding that the human brain doesn’t have.

It might have concepts in it like protein, organelle, Golgi apparatus — those are the kind of concepts that it probably has something close to when it analyzes a cell because they’re so useful to humans, and the universe really does kind of carve like that. But inside of these giant neural networks with so many billions of parameters, it’s going to have other concepts that you’re not going to find in any human textbook that are useful concepts. It’s doing a version of science that’s a sped up, much more subtle version of what the human brain will ever be able to do.

And Martin is looking at this hypothetical scenario that Nate is throwing at him, and he’s just saying, “Oh yeah, it’s just simulation. I’ve done simulation before.” How can I give you a more intelligent scenario than doing cell engineering? How is he not seeing this?

Martin 01:55:39
It’s a very specific tool for a very specific situation. But we were here before whenever it was 25 years ago.

Liron 01:55:45
So in addition to using the word simulation as a way to dismiss this upcoming breakthrough, he also wants to dismiss it as being too specific. “Oh yeah, cell engineering, that’s just too specific.”

I often talk about domain expansion, how we’re seeing AI be able to surpass humans in optimizing larger and larger domains. When you’re getting into cell engineering, a domain that you yourself said is incredibly complex, has incredibly fat tails in the distribution — when you yourself said it’s that kind of nightmarish domain and the AI is now crossing it, how many more levels of domain expansion do we have before it’s at the whole universe?

Imagine you’re sharing the world with an AI that can engineer the crap out of cells to the degree that it can basically make its own bacteria. You’re not getting any ideas about what this bacteria might do when let loose on the world? That’s not troubling to you? It’s just a specific domain. It’s just cell engineering. Extrapolate a little bit, please.

Nathan 01:56:39
I’m still a little confused around what would count. What would be the evidence of the fundamental thing that humans can do and have done through our history that the AIs can’t do?

Liron 01:56:49
Exactly, because I think Martin is starting to contradict himself or kind of show the incoherence of his model and kind of flip-flop on what he says is hard and what he says is easy. Specifically when he said, “Oh yeah, cell engineering, that’s no big deal.” So I like that Nate is asking him to be like, “Okay, tell me what’s actually hard.” Show me the boundaries of what your mental model actually says AIs can’t do so that you won’t just retroactively say everything counts as what you predicted. This is a good line of questioning by Nate.

Nathan 01:57:15
If the fundamental thing that humans can do and have done through our history that the AIs can’t do is look at the universe and figure out the right abstractions and come up with the right concepts that compress it in order to make sense of it — and I agree, it’s very hard to say what they’re doing in the language domain because we already did that work and they’re learning it from us.

I try to describe something in the biology domain where it seems like they’re starting to show signs of doing that, and I could believe that they would, and then you sort of agreed, but then now I’m confused as to — wouldn’t that count as doing that?

And then it seemed like the response was, “Well, that’s just in one domain.” But it doesn’t seem like there’s anything that would prevent it. Certainly, there’s a huge leap in generality with this latest generation of systems. So I do imagine just shoveling all the modalities into one model. We’ve already got text, vision, and audio in GPT-4o. Why not the true GPT-5o would be biology data and weather data and pictures of deep space and solution simulations and battery simulations, material science, whatever. Throw that all into one thing, and if it can do that, then it’s definitely not gonna be constrained by one domain anymore.

So I’m still a little lost as to exactly what the limit that you see is in terms of why it doesn’t become—

Martin 01:58:25
I’m so glad—

Nathan 01:58:25
—a system that’s more powerful than people.

Martin 01:58:27
I’m so glad you reduced it to this. I think this is great. And there’s two things. There’s this notion that language reasoning is general, which a lot of people believe, but you don’t seem to be on that kick, so let’s put that one aside. And you’re more on the learning properties and simulating properties of the universe side, which is totally fine.

So what I would do is I would just bring you back to everything we’ve learned about simulation, which is even when you know all of the properties of the system and you can simulate, you just don’t have the computational capacity to simulate nonlinear systems. We don’t have the materials or the energy or anything. It’s literally a compute problem. We have code bases that have been around for 20, 30 years that simulate all sorts of crazy stuff, and yet they’ve got limited utility for exactly this reason. The universe is just so complex that they’re useful for a little bit.

Liron 01:59:19
Okay, but the scenario Nate gave you was asking about a very high utility scenario. Remember, this is the scenario Nate gave you.

Nathan 01:59:25
Scaled up foundation models for biology that are going to start to understand the super complicated interactions between genes, between proteins in cells in ways that are inferred from inputs and outputs.

Liron 01:59:46
I described it as cell engineering. We can just ask the AI what we want to output out of a cell, and it’ll tell us how to engineer it and make it happen. That kind of scenario. It sounds like you’re dismissing it as, “Oh, I don’t have to answer about it because it’s not gonna happen.” So let’s be specific. What’s going to surprise you when it happens? Because that’s what Nate was trying to ask you — would you be surprised when this happens in as little as one year?

Martin 02:00:05
If it turns out that these models somehow change that compute trade-off where it can simulate nonlinear systems in ways that traditional stuff can’t, I’m 100% with you.

Liron 02:00:20
Okay. I’m glad that you’re admitting that the scenario Nate proposes is outside of what you predict is allowed to happen, so you’re allowing yourself to lose base points when that happens, even though you’re being kind of vague.

I object to you describing that scenario as requiring simulation. You’re making an assumption that a system that can relate inputs to outputs the way Nate describes must be doing simulation. And of course, it’s impossible to do low-level simulation, but you can’t just have a few simple building blocks combined to get the simulation you need.

So the only way to describe it is deep intelligence. It’s a multi-layered simulation that requires making inferences. There’s not even a compact way for me to explain what’s going on except to say the AI is intelligent and it’s using its intelligence to figure out what’s going to happen. It’s really hard to reduce it beyond an explanation like that.

It’s a very complex, smart system that Nate is describing if it happens. So don’t just call it a simulation. If you see the kind of outputs that Nate is describing, then you’re not witnessing a simulation. You’re witnessing something fundamentally different.

Intelligence vs. Simulation

Martin 02:01:19
But that’s not what they’re doing. What they’re doing is they’re inferring stuff that we haven’t been able to infer by looking at data. That doesn’t mean that they can be predictive in a way that kind of disobeys our understanding of compute requirements.

Liron 02:01:34
Just to repeat back what Martin is saying, I think he’s saying, “Okay, maybe you can figure out the cell’s outputs without simulating it, but it’s never been done before by humans using the dataset we have, so maybe it will never be done.”

But it will be done. Nate’s prediction is probably correct that we’re going to see AI pulling off these kind of feats because at the end of the day, despite how complex cells are, they’re still very low entropy machines compared to the maximum entropy possible.

Cells are low entropy, which is how evolution is able to successfully mutate them and select the next generation of genes to build better cells. There’s enough structure that you can understand well enough to just select better genes and have them predictably perform well in the organism’s niche in the next generation. The fact that that process works is already plenty of evidence that the entropy of a cell is still relatively low.

And in AI, the algorithms we have to parameterize these giant AI models, those algorithms are very good candidates to start making really useful predictions about the cell, even though a human looking at that data is not going to have the right complicated models to give you the same predictions. And even though it’s, quote-unquote, “out of distribution” on whatever kind of distribution of human text on the internet you think it’s looking at, or whatever kind of distribution of other cells doing other stuff.

I don’t think you can usefully describe it as, “Oh yeah, it’s just finding something in a distribution.” I think the only way to describe it is it built up a model, all these different concepts, and the relationship between these concepts and the structure of those concepts is somehow similar, isomorphic to the low entropy structure patterns that are happening at very different levels, at high complexity, high interconnectedness, and patterns nonetheless. Low entropy nonetheless.

It’s modeling the universe’s low entropy inside of the AI model’s low entropy. And again, that’s the essence of intelligence. That mirroring process, that multi-level low entropy mirroring process — that’s the good stuff. It’s not everything. It’s not reasoning. It’s not self-reflection. It’s not everything, but it’s a lot. And Martin is just not giving it credit for what it is. He keeps using his hammer to try to hit the nail. The hammer that he likes to use is it’s just a simulation and it’s just interpolating. He’s missing something very deep and important that’s happening.

Martin 02:03:46
Let me just give you a specific. Rayleigh-Taylor instability is basically if you have two liquids that are on top of each other with different densities. And that is one Rayleigh-Taylor unstable system. And if you perturb it, you get these just amazing, kind of chaotic, turbulent things that happen. We have been looking at this problem for 30 years, and we have no idea how to actually predict what will happen. We just know roughly what they will look like, but we don’t know the specifics.

And AI systems are way less efficient than an actual code written for simulation. So to think that it can tackle those types of problems, I just don’t see any indication.

Liron 02:04:24
This is the common argument people make where they say, “Look, the universe is chaotic. All you have to do is take a pendulum, connect another pendulum under it. It’s called a double pendulum. Swing the double pendulum, and then wait 30 seconds, and suddenly the motion of that double pendulum is going to depend on the exact positions of the air particles all around the room. So even just a little pendulum is a chaotic system, or even just three gravitational bodies are a chaotic system. The universe is just so hard to predict. How will AI ever take over the universe when it’s so hard to predict?”

So this is a common type of argument people are making. Martin is making the same argument using the case of Rayleigh-Taylor instability. Yeah, I agree. If you wanna model that exact system, you better have a lot of really precise measurements. You better have a lot of context if you wanna model the evolution of the system, and your model is probably going to become inaccurate.

But here’s the trick. You just wall off the parts of the universe that you can’t predict, and what you find left over is a ton of parts that you can predict. So I live in my house. The air is super chaotic. I have no idea where the air particles are gonna go. And guess what? I can walk down to the kitchen and get a tasty snack. That’s not a problem. And guess what? I can run an online business. I can make money. I can do quite a lot, even though I have no idea what a lot of the universe’s chaos is doing.

I have no idea what the gravitational interactions between all the objects in my house and all the objects in the solar system are doing. I have no idea how to predict that on a fine-grained level. And it doesn’t matter because the universe is a combination of parts that you can predict with very high accuracy and then parts that you can’t.

Now, you might wonder, in practice, which part wins? But you just have to look at the human world and the biological world to conclude there’s plenty that you can predict. Engineering is possible. It’s a settled question that you can engineer things. There’s no doubt that humans can terraform other planets, that humans can conquer the galaxy the same way that life has conquered Earth, the same way that humans are conquering Earth. There’s no doubt that our particular physical universe is a universe in which engineering can succeed over chaos.

Now, we’re never going to perfectly micromanage the position of every atom. There’s always going to be some heat, some waste. In fact, the second law of thermodynamics says there’s gonna be an increasing amount of heat and waste. That’s fine. There’s still plenty of stuff that we can engineer and build.

Now, in the case of cell engineering, Martin is basically saying, “Look, if we can’t solve Rayleigh-Taylor instability, how are we ever going to tell you exactly how to engineer the cell to do whatever you want the cell to do?” The answer is just there’s plenty that we know about cell engineering. Humans already do some small amount of cell engineering successfully. We give each other medicine. Gene therapies are coming out. CRISPR is coming out. It’s just small. It’s hard to deal with, but we’re getting there.

But the AI — we’re predicting that the AI is going to take a big leap because the AI’s specialty is modeling a system that has a bunch of low entropy that is understandable, but it’s just hard for the human brain to understand because there’s degrees. A cell just has so many moving parts. There’s some chaos in a cell, but there’s also a lot of order. The genetic code, the parts of the genetic code that natural selection has decided are worth selecting on and passing on — all of that structure is low entropy. All of that structure is modelable because it works for a reason. Otherwise, natural selection wouldn’t bother copying it across the generations.

That reason is a reason that’s really hard for the brain to model because there’s so many dependencies. The answer to why it works could be a 50-page explanation, a 50-page proof with a lot of detail. The human brain is not optimized for that. Natural selection can handle it fine because it uses the physical universe to play it out. But you don’t need the whole physical universe to understand what key pieces of the structure are, why it works. You don’t need the whole universe to understand why engineered systems work, as long as those engineered systems are low entropy and they work across contexts.

So the same piece of DNA — it can work for your parent, and it can work for you, and it can work for your child. In that case, it must not be that hard for an AI to understand why that piece of DNA is useful because there’s enough structure to it. There’s enough regularity to it.

So I think Martin is making a common type of mistake when he brings up chaotic systems like Rayleigh-Taylor instability, and he’s not looking at how hackable the universe is and how limited the human brain is as an engineer by the fact that our brain is kinda small. We don’t have the billion parameters. We don’t have the capacity to just grok an entire cell and be like, “Okay, I get how this cell works. I get how to engineer this cell.” We have very limited working memories. We have limited visual imaginations. We’re better than the animals, but we’re not that great, and we’re about to get surpassed.

Martin 02:08:57
Now we’re in AI where you don’t even know what the end state is. You’re just like, “I have a whole bunch of data, and I want you to find patterns in that data.” That requires even more compute, so it’s even less efficient. So it’s another modality of compute. It’s one we’ve been fighting for a long time, but it doesn’t change the nature of computers. They’re still systems, and they’re still computers, and they still have the same limitations independent of what distribution that they learn.

And so if it turns out that these things can simulate systems for a period of time longer than a normal simulation, then I’m with you. I’m like, “This is breaking the laws of physics.” But until then, it feels to me like simulation where you just don’t know all of the rules, but it’s learning some of the rules.

Liron 02:09:39
I disagree with that analysis. I disagree that what AIs are doing is fundamentally inefficient. For instance, when they write an essay, sure, they’re using a lot of H100 GPUs for now, but that essay sure does come out fast, and these models sure are getting optimized to be smaller and smaller.

At the end of the day, I think the best way to understand the optimization or the compute resources of an AI is to see that it’s trending toward the architecture of the human brain. Human brain runs on 12 watts, seems to have a lot in common with the architecture of the new AIs we’re seeing. So I don’t really get what he’s saying now about this AI being inefficient. I don’t get what he’s saying now about, “Oh my God, can this AI simulate stuff?” I just don’t think he’s using useful abstractions right now.

Nathan 02:10:16
Yeah, I think about it less in terms of simulation and more in terms of how effective the choice of actions can be at any given time step. I’m not simulating the universe. Humanity as a whole is not simulating the universe. But we’re all just taking our local conditions and our sort of general sense of our own selves and goals, and trying to do the next step at any given time.

Liron 02:10:44
Yes. Thank you. The idea of calling AI just simulation or just interpolation is just ignoring this capability that the human brain is doing. You can claim that the AI is not doing the secret sauce that the human brain is doing, but you at least have to acknowledge there’s this big secret sauce that the AI could be doing. If it’s not doing it today, it might be doing it soon. You have to at least explain the secret sauce and not just call everything simulation and interpolation.

Nathan 02:11:07
And it seems like our overall efficacy through our lives is the integral, if you will, over how good our choices are at each given time step. And that doesn’t depend on any huge simulation of anything irreducibly complex.

So then if I imagine an AI, it seems within reach to imagine an AI that can do something very similar to what I’m doing, which is have a goal, look at its immediate surroundings, look at what it just did, look at whatever other context it may be given, and pick a next action and potentially be better than me at it and potentially quite a bit better.

And then that to me seems like enough. If it can do that, then I feel like we’re in an unprecedented environment where we now have fundamentally pretty alien and not super well understood things that can take more effective actions in many given contexts than I could. And then that to me is where I start to turn the conversation toward what sort of safeguards should we have in place.

Martin 02:12:07
Just again, because these conversations tend to be so muddled — if that action requires interacting with the physical world, it has to simulate the physical world. It just does. It has to understand dynamics and ballistics. It has to understand what happens if someone throws a rock at it, or if it’s in water or if the weather’s — I mean, that’s how you navigate the physical world. That’s really why we created computers. It was because these are very hard things to do.

Liron 02:12:34
But there’s a difference between simulating something and understanding something. When you understand, you just have something isomorphic in your own mental model. You have high-level pieces, and you can operate interactions between those high-level pieces. And what you get is isomorphic, sufficiently isomorphic to the states you’re going to observe in the real world.

So you don’t have to simulate atoms. You don’t have to simulate molecules or even cells to effectively navigate the world. Even using that term simulation is highly misleading. Understanding how to navigate the world does not mean you have to simulate the world.

Now, if you wanna abuse the terminology and say, “Ah, yes, the fact that your brain contains these representations of 3D objects that you’re trying to navigate around, that’s simulation” — okay, but you’re abusing the notation because the dynamics of these high-level objects and your choice of high-level objects, Martin himself said that it’s so amazing that humans are able to chunk these objects like rocks and sand and trees. If that’s amazing to you, then it’s not simulation. So why are you saying now that the human’s ability to navigate the world requires simulation?

Martin 02:13:33
If that action requires interacting with the physical world, it has to simulate the physical world. And then if it’s not that, if it’s not interacting with the physical world, then it is interacting in this kind of language domain that we’ve created. And I agree it’d be very good at some subset of those things. There’s zero indication it’d be good at new things. And that’s what we’re actually very good at.

And again, without actually having a model for all of these things that we understand — the distributions, we understand the mechanisms — I feel like we just use words, and the words all make sense. But complex systems, we never know convergence and divergence without actually specifying the system.

And I feel like for these conversations, we just don’t have a system we talk about, and so it’s always we live in the world of rectangles and arrows, and somebody takes a rectangle, and they have an arrow that goes back to the rectangle. They’re like, “Ah, we’ve got a virtuous cycle,” without actually specifying that if you hit diminishing marginal returns, you don’t go anywhere or you’re doing the same stuff or whatever.

And so I think this is incumbent on all of us to actually understand the systems we’re working with and then come up with these basic views and properties to make sure at least we understand what the convergence properties are. I’m sorry, that was a very muddled thing to say, but I feel that until we talk about specifics, it’s very hard to make concrete statements in this.

Liron 02:14:48
Yeah, I also find that too muddled to respond to, so let’s move on.

Nathan 02:14:51
Let’s change gears, because I think this probably certainly gives everybody enough to get at least a good intuition for our relative philosophies on this.

AI Regulation

Martin 02:14:59
Yep. ## AI Regulation

Nathan 02:15:00
So what do you think we should do right now in practical terms to regulate AI, if anything?

Martin 02:15:10
The regulation one is sticky for me for two reasons. The first one is we don’t even have a definition of AI.

Liron 02:15:16
Okay, feel free to use my definition. Systems that can map inputs to outputs better than humans in broad domains.

Martin 02:15:23
And so I think it reduces to regulating software. And then for that, I would say we’ve been regulating software for a very long time, and there’s a broad, robust discourse around that, and I think we should make whatever conversations we have part of that broader discussion.

Liron 02:15:38
No, most pieces of software like Microsoft Word, Google Chrome, these are pieces of software that you can look at their domain and you’re like, “Okay, it compiles this code.” It parses HTML, it parses JavaScript, it does word processing of arbitrary DOCX files.

Those domains are much narrower than the universe, and yeah, it’s going to get a little bit fuzzy when you’re talking about, okay, this system helps you do a medical diagnosis. Can it help you engineer a medicine? That seems pretty broad. It’s going to get fuzzy, but at least we know where it’s black and white and where it’s fuzzy, and then we can spend more effort on the parts that are fuzzy.

But that’s the name of the game. That’s just how regulation goes. To just use AI synonymously with software, you’re just ignoring the important thing happening around you. It’s almost like a head-in-the-sand approach.

Martin 02:16:20
I don’t know what the distinction between AI and software is. I really don’t. I have seen the definition used in these regulations. It’s so broad that it really could include all non-trivial software. And I don’t say this to be a polemic, and I don’t say this to be difficult. I’m saying this very clearly. They literally say a system that can navigate and change a virtual or physical system. These are so broad. So we’re really talking about software. That’s what we’re really talking about.

70 years of history regulating software in many domains. And I think that regulation’s very important. I’m not a libertarian. I’m a lifelong liberal, a very moderate person. I’m just saying this discourse has been around for a very long time, and we should continue. And if there’s an area that software’s being pushed, an area that we need to have some sort of protections, we should add them to it.

But that’s a very different statement than saying AI is somehow paradigmatically different. There’s just literally zero indication that it is. And then trying to somehow regulate a computer science primitive—that’s like regulating a database.

Liron 02:17:25
Maybe there’s some similarities to software, but it’s ridiculous to me to say that there’s zero indication that AI is different. If you have a virtual girlfriend who starts manipulating people, isn’t that different? If you have a super intelligent virus that takes the entire world a week of lost productivity to clean out, isn’t that different?

And yeah, that hasn’t quite happened yet. The girlfriends are getting scary, but the virus hasn’t quite happened yet, not that I know of. But isn’t that something that we should get ready to regulate? What do you mean that there’s no difference? It does seem like there’s something new brewing here.

Not to mention what I’m worried about, which is a permanently uncontrollable super intelligent AI, where in that case, regulating a punishment is useless. So you just have to regulate the prevention of it ever being created in the first place.

Nathan 02:18:07
I guess to venture a distinction or what makes the technology a paradigm shift, I would probably zero in on the fact that they are trained, not engineered, and that maybe a better thing even than that would be that the creators of the models generally don’t know what they’re going to be able to do and, even at deployment time, don’t have a very robust account of what the capabilities of the systems are.

You could point to things in the past and be like, “Oh, you didn’t expect this out of whatever,” but this does seem to be qualitatively different that they just train, train, train, train, train a long time. Especially if you look at base models. Base models are totally unpredictable and nobody really knows. I think one of the reasons that people are putting so much resource into post-training is to try to get control and it’s only sort of working.

Liron 02:19:02
Yeah, that’s another great distinction you could use if you really don’t know what’s AI and what’s other software. You can definitely throw in that criterion of, does it get trained and then not give you an account of all its different capabilities? If so, then it’s potentially dangerous AI. That’s a nice distinction.

Martin 02:19:16
Yeah, so this is the thing—when you’re talking to an internet guy and a distributed systems guy, it’s just none of the systems that we worked on we understood the implications of. Think about the internet. Every sociopath becomes your next-door neighbor. What does that even mean?

What does it mean to put kids on the internet? What does it mean to have your business on the internet? What does it mean to put critical infrastructure on the internet? There is no model for how any of this behaves. There is no way to make computer systems provably correct.

Liron 02:19:46
So it’s often a good approach when you have a new technology to be like, “Ah, crap, all these different things can go wrong, but let’s just take it step by step. Let’s move forward. Let’s deal with the consequences. Let’s iterate.” So I’m all for that approach. Normally I’m a techno-optimist. When it comes to VR, go hog wild. Make a bunch of VR worlds. If there’s problems with some of them, recall them, punish the people who made them. That’s fine. That’s a typical technology.

The internet was, I guess, a little bit more dangerous than VR in the ways that Martin is describing. Fine. Of course, the disanalogy here is that experts are warning—not all, but many of them—experts are warning that you might have an uncontrollable extinction scenario in the near term.

So that is qualitatively different. You’re not going to find a large fraction of serious experts warning that the internet is going to cause human extinction within a couple decades. So I don’t know what to say. That breaks your analogy. When that’s the downside that we’re dealing with, potential imminent human extinction, that some people like me think has a very high probability, like 50%.

Jeff Hinton, his personal probability, he said, was more than 50%. To be precise, he updated it down to 10% to 20% because he said a lot of his friends are saying 10% to 20% or lower, and he wanted to update down to be with his friends. But he said that he independently assessed the risk as being more than 50%. So when you have a situation like that, bringing out your analogies of how some people had some worries about the internet and we managed to overcome those worries, it’s just not analogous. It’s almost like, why are you even bringing that up?

Martin 02:21:13
We weren’t putting compute limits on databases, and we weren’t regulating computer science primitives, and we weren’t inhibiting innovation of startups, and that’s what we’re doing now, and that is a paradigm shift, and that is a doctrine shift, and it’s really scary.

Liron 02:21:28
I agree. It’s a paradigm shift. It’s a scary scenario. I agree with that. If the government’s coming down on regulation—the government sucks in many ways. It’s not necessarily staffed with all competent people. Absolutely. I hear you. It sucks. It’s scary. It’s a paradigm shift.

Of course, we have to deal with the issue that experts are warning about near-term human extinction. So I get that Martin’s position is, well, there’s a very low risk of near-term human extinction. I agree with him that if you accept the premise that there’s a very, very low risk of imminent human extinction, then these regulations are so crazy. What are you so happy to regulate everything? Everything’s fine. I agree.

But if he grants my premise that there’s a near-term human extinction risk, a high one, if you grant my premise, then you gotta do something. You gotta prevent some hacker in a basement from taking the latest Llama model that’s on the verge of super intelligence, pushing it all the way over and losing control, and then goodbye for humanity forever. You have to do something.

So at this point, the argument is getting less interesting to me because when you live in a mental model like Martin’s, where there’s just a low risk of extinction, then I agree regulation is bad. We don’t disagree about that stuff. The crux of our disagreement, I don’t think, is on the topic of regulation. I think the crux of our disagreement is just, are we doomed? Because if we are, I’d like to think that Martin would then be on the same page as me being like, “Okay, let’s do something about the possibility of being imminently doomed.”

And that’s why my whole podcast is called Doom Debates, because I think that if we can get on the same page about doom, then a lot of other things are going to fall into place. To me, it’s a little ridiculous to have a policy discussion with somebody who doesn’t realize that we’re very likely doomed.

Concluding Thoughts

Liron 02:24:39
So I’m not going to play you the last third of Martin’s podcast with Nate because that whole section is operating under the premise that we’re not imminently doomed, so let’s just talk about good ideas for regulation. And when you pre-assume that we’re not imminently doomed, that there’s no large human extinction threat, at that point, I actually think Martin makes reasonable arguments. I don’t really have a major crux between his worldview and my worldview after conditioning on the assumption that we’re not doomed.

So if you go and listen to that discussion, I’m actually sympathetic to a lot of Martin’s points, and I think it’s a high-quality discussion. It’s just not interesting to me because I don’t think that we live in a world where we’re not doomed. So it’s just like talking in fantasy land. But they’re perfectly fine points. The points make sense if we’re just talking about regulating VR or regulating the next social network or, hell, even regulating crypto. Go wild. I don’t care that much.

The only other content I want to show you from Martin is the back and forth that we’ve had on Twitter. There’s a Twitter thread from July 27th where Martin tweeted, “LLMs don’t reason. They’re a reason cache with fuzzy matching. The extent they generalize is a function of how prior reasoning applies to future knowledge and configurations of the universe. I suspect the answer to that is not very much.”

And then a follow-up tweet by Martin, he says, “Game playing in an axiomatic system like Go can be exhaustively searched with more compute. Clearly, some problems fit in that domain, e.g. protein folding, where there is a relatively constrained set of solutions or maybe even code, but it’s not at all clear this generalizes broadly.”

So I didn’t really get why he thinks there’s a dichotomy between playing in an axiomatic system and playing in a non-axiomatic system. I think he’s barking up the wrong tree. I think he’s looking for a distinction that’s going to explain what AIs can’t do, and he’s just going to be wrong about that distinction.

So I replied to him. This is what I wrote: “When you say can be exhaustively searched with more compute, do you mean orders of magnitude more compute than could ever be physically realizable? Because that’s trivially true about every problem we know how to recognize solutions to.”

My point was basically: if you can win at chess, if you can win at Go, you’re already not exhaustively searching. You’re already doing something smarter. You’re doing the essence of intelligence to some degree. The essence of intelligence is to search an exponential space in a way that a brute search could never begin to search, but your search can somehow prioritize some really good candidate options within that space somehow.

The algorithm may vary. You look at what it’s doing, you look at the input-output relationship, and that’s sufficient for you to conclude there must be intelligence here. Even a black box is something that you can look at and conclude that it’s intelligent. And the axiomness of the system, the fact that chess has rules or Go has rules or StarCraft has rules or driving on the road has rules—it doesn’t really matter. It’s just that mapping from inputs to candidate outputs located within a giant exponential space. That is the essence of intelligence.

So again, this is why I’m just confused why Martin thinks he’s really onto something with axiomatic rules. When I asked him about why he’s so fixated on the game Go having axiomatic rules, he replied, “Basically, it converges to simulation, which we understand the bounds of very well.”

And I replied, “I don’t understand the distinction, e.g. is reasoning about Conway’s Game of Life on the exhaustively searchable side of your distinction? Why or why not?”

The reason I asked that is because I see Conway’s Game of Life as a perfect intermediate between the dichotomy that he’s trying to set up. He’s trying to set up a dichotomy where on one side you have things like the game of Go, and on the other side you have things like the physical universe, which he thinks you can’t axiomatize the rules of—which I’m not even sure is true. I think it’s probably false.

But in his mind, we don’t understand string theory yet, so you can’t axiomatize the rules of the universe, and that’s what makes life in the universe hard, I guess, according to Martin. But anyway, the reason I threw out the Game of Life is because the Game of Life does have axioms. It does have very simple rules. We can describe exactly how it evolves, and yet it’s chaotic, it’s Turing complete, so you can’t really make predictions about it. You can’t really solve problems about it in the general case, or there’s arbitrarily hard problems within the Game of Life universe.

So that’s why I asked him to analyze Conway’s Game of Life, and his response is this: “Conway’s Game of Life is an entirely different class. There’s no end state.” So there’s a new distinction—no end state.

“Put it this way: initial conditions described axiomatically, e.g. positions on a chessboard, plus fixed rules for evolving state, e.g. chess moves, plus well-defined end state—game won or lost—equals search.”

Contrasted to: “Initial conditions described via physical phenomena, e.g. a car, plus physical laws for evolving state, e.g. materials, fluid dynamics, heat transfer, chemical reaction, plus end state described over physical phenomena—does the car catch fire—equals simulation. We have 50 plus years of computer science that has tackled both of these domains, and we very well understand the bounds, compute requirements, et cetera.”

Liron 02:28:35
Very interesting. He’s just doubling down on this distinction. Now he’s calling it search versus simulation. That doesn’t seem to map really nicely to what he was saying in our podcast. I felt like the distinction he made in the podcast with Nate was more like there’s simulation, but on the other side it’s not search, it’s interpolating patterns. I felt like that was the distinction he was emphasizing during the podcast.

But now it seems like he’s making a distinction between search and simulation, which I frankly don’t understand. I feel like it’s a very fuzzy distinction, so I followed up on Twitter. I wrote, “How about the problem of design a region of size N by N in Life neighboring a randomly initialized region and at time step 10 trillion have over G gliders gliding? I’m asking this test case because I’m still not clear on why your distinction is meaningful.”

And he says, “Which distinction? Between rules-based and physical?” And I said, “You just defined a particular distinction. I think you would term it search versus simulation. It just doesn’t seem like it has any fundamental distinctions to me, which is why I’m trying to understand how this seemingly hybrid example, Game of Life, gets analyzed by you.”

And he replied, “Well, for one, we don’t know all the laws of physics, which is why nearly all simulation relies on empirical equations of state. Further, for all practical purposes, physics is non-discrete. With cellular automata, we know all the rules”—Game of Life is a cellular automaton—”because we define them and we have full understanding of the state.”

And then I answered, “If I understand correctly, your claim is that we can classify problems into two buckets, and AI is stuck only being able to do problems in the first bucket”—the one that he called search in this thread—”and your answer to my latest question is to add rules to your distinction so that my problem, the N by N gliders problem, goes into your easier bucket.”

So that was where I left it and he didn’t reply, so I don’t know exactly where he’s going with this. But honestly, I think it’s the same situation that we’re seeing with Martin in Nate’s podcast, where he has a distinction, he has a background working on simulations. He thinks he’s constraining today’s AIs to fit into buckets that he’s familiar with, but he’s missing this important abstraction of the essence of intelligence being somehow taking shortcuts in an exponential output space. He doesn’t seem to focus much on that mental model.

He doesn’t seem to focus much on the idea of an agent in a low entropy universe making a multi-level low entropy model of that universe and then engineering it. He never talks about that, which to me, that is what’s going on. You’re missing what’s going on with AI. And at least if you don’t think that’s what’s going on in AI, at least that’s how you need to talk about what’s going on in the human brain. And then you need to answer the question of, well, why isn’t AI catching up to the human brain if that’s what’s going on in the human brain? I never see Martin talk about that. He left the Twitter thread at that point, so I don’t have anything else to report to you there.

Liron 02:31:05
Just to close it out, I’ll play one more quote from the last half of his podcast with Nate. I didn’t play you the last part of the podcast because it’s so much about policy in a world where we’re not doomed, but let me just play you this one little bit where Martin gives a closing statement about simulation.

Martin 02:31:05
I literally think this whole problem comes down to simulation, and maybe it’s just because of my simulation background. The only way to simulate the universe is to be the universe. It literally comes down to the universe is a big computer that’s simulating itself.

Liron 02:31:18
And I’ll reiterate my response, which is that he’s talking past what AI is, what intelligence is. He’s just pointing to one particular type of limit, one particular type of ceiling. Yes, you can’t use finite resources to model arbitrary chaos. That’s true. Complexity theory also tells us that you can’t use finite resources to crack any type of encryption or to solve arbitrary problems that can be stated in a relatively compact form.

I can give you a traveling salesman problem or an NP complete problem, various types of problem where I can write down a problem that looks easy enough, and there’s just no way to solve it in the length of time that you have in the universe. So there’s all these ceilings that tell us what we can’t do. That’s great.

But now let’s look at reality. You have these agents called humans. They engineer stuff in the world. They transform the whole world to their liking. We’re about to have AIs. They’re also going to have this amazing superpower called engineering, called intelligence, and the fact that this universe has these limits, the fact that complexity theory has these limits, the fact that the speed of light is a limit—great. We’re entering a universe that has all these high ceilings. That’s great. It’s not going to affect what we actually come and do, how we’re actually going to take over the universe.

So it’s mind-blowing to me that he thinks that he’s explaining what’s going to happen with AI when he’s just obsessed with fat tails in simulations of low-level physical systems. That’s just not how intelligent agents go and take over the universe, go and engineer, build what they want to build.

So I encourage him to just look at the relevant thing you need to look at if you want to have a hope of staying alive, having utility in the universe. Look at what’s actually happening. Look at hierarchies of concepts that are self-trained. LLMs are self-trained on low-level inputs to create high-level concepts to a degree that, as Geoffrey Hinton says, it’s already beyond humanity’s ability to do that. So they’re already doing it in a more subtle way.

They might prove themselves, as Nathan says. We might have biological engineering systems that surpass the best human scientists and the best human engineers. And from there, we might finally get to general intelligence, where they connect everything together, they reason robustly, they plan toward goals.

Which is the funny thing, by the way—normally in all my podcasts I’m always talking about goal optimization. I’m always talking about utility functions and optimizing the universe. But Martin got off the doom train at such an early stop, the stop that you basically can’t do engineering at a superhuman level because of this idea of fat tails and this idea of everything has to be in distribution.

He got off at such an early stop that I never even got around to talking about instrumental convergence, taking over the universe. I just had to spend the whole podcast explaining what it looks like when you have an intelligence, explaining what it looks like when the human brain is doing what appears to be the impossible from the perspective of other animals or from the perspective of a naive analyst.

The human brain appears to be doing the impossible. So I’m just explaining. I’m giving him the tools to analyze the phenomenon that you’re seeing inside of your head, and it also helps you analyze the phenomenon that you’re seeing inside of AIs. These are the tools you need if you want to extrapolate successfully into what’s going to happen.

And I’ll also reiterate my point that his own model is rather vague and refuses to make predictions, and the one time that he had an opportunity to make a prediction about what AI potentially can’t do—maybe AI can’t engineer biology as well as humans—he actually said, “Oh yeah, that would just be a specific problem.” So he kind of dismissed the one impressive thing that Nate brought up as not even being that impressive.

So I encourage him to reflect on the quality of his own epistemics, of whether he’s even saying something meaningful or whether he just likes talking about a certain thing. He just likes talking about the universe. He likes talking about fat tails and simulation, and he just keeps going back to those because those are topics that he’s familiar with. But he’s getting blindsided. He’s just missing the actual phenomena that are happening around him, and of course, those phenomena have huge implications.

Now that said, to say something nice, I think that he’s giving a perfectly capable analysis of the economics of AI in the next two years. And if you want to go listen to that part of the podcast, that’s the last half that I didn’t really excerpt many clips from. So overall, I think he’s a smart guy. He’s making many capable points.

I just have to highlight the part that I disagree with, which is: hey, you have an intelligence. You don’t know what the boundaries actually are. You’re about to get surprised, and you should be humble about that. You shouldn’t think that you understand because of these two hammers that you have, the hammer of distributions and the hammer of simulation. You really gotta update your understanding of what’s going on, because you’re not being accurate.

Okay, that’s all I got with Martin. Stay tuned for more episodes where I go and review other people saying stuff on other podcasts. You might call it a takedown episode. Of course, Martin, if you want to come on the podcast and debate me, I’m game anytime. I think it would be a productive discussion. I wouldn’t throw any gotchas at you. It would be a high-quality discussion. We can get a moderator if you want.

And then one more thing—Martin is pretty closely associated with Marc Andreessen, who I’ve written about on Twitter in the past. So at some point, I’ll be doing a Marc Andreessen review episode where I go through all the stuff that he’s said on podcasts and what I think about that. Spoiler alert, I’m going to strongly disagree in many ways.

Okay, that’s it for today. I think this is the longest episode so far, so let me know in the comments—is this a good length? What do you think? I always love the feedback. I find it very motivating, so keep it coming. And of course, if you’re watching this on YouTube, smack that like button. If you’re listening to this in your podcast player, leave a review. Go to youtube.com/@doomdebates. Subscribe to my channel. Go to doomdebates.com. Subscribe to my Substack. Go to x.com/liron. Subscribe to me on Twitter. You got a lot of homework. And hey, how about this—tell a friend. It’s a good idea. All right, that’ll be all for today, and I’ll see you back here for the next episode of Doom Debates.

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏