Bestselling author Robert Wright has just published a new book claiming that on our current trajectory, AI dystopia is likely within eight years. He’s not wrong!
We get into his P(Doom), his case for slowing down, why racing China could backfire into authoritarianism here at home, and why he thinks evolution itself might have a purpose.
Bob is also the author of The Moral Animal and Nonzero, a Pulitzer finalist, and hosts the excellent Nonzero podcast.
Get his new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning.
Watch on YouTube:
Timestamps
00:00:00 — Cold Open
00:00:50 — Introducing Robert Wright
00:03:21 — What Is The God Test?
00:04:59 — AI as Evolutionary Force, Not Normal Technology
00:07:34 — AI Heaven vs AI Hell
00:09:09 — Bob's Background: From Hinton in 1983 to Today
00:16:09 — What's Your P(Doom)?™
00:20:09 — The Mainline Scenario: Dystopia in 8 Years
00:22:04 — Destabilization and the Road to Authoritarianism
00:25:35 — Why Racing China Backfires
00:28:37 — Slowing Down AI + Would Bob Support PauseAI?
00:32:38 — Intellidynamics and Why AIs Collude
00:44:44 — Hinton's Warning and How LLMs Discover Meaning
00:53:48 — Symbol Grounding and Killing Searle's Chinese Room
01:02:03 — Let's Talk About God
01:10:14 — Orthogonality, Power, and the Concentration Problem
01:16:31 — Wrap-Up
Links
Get Bob’s new book, “The God Test” — https://www.simonandschuster.com/books/The-God-Test/Robert-Wright/9781668061657
Bob’s Nonzero Newsletter & podcast —
Follow Bob on X — https://x.com/robertwrighter
Liron’s recent appearance on Bob’s Nonzero podcast —
Dario Amodei’s “Machines of Loving Grace” essay (discussed) — https://www.darioamodei.com/essay/machines-of-loving-grace
“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares (referenced) — https://ifanyonebuildsit.com
Join the Doom Debates Discord — https://discord.gg/g2X8h5UCrU
Transcript
Cold Open
Robert Wright 00:00:00
Rapid AI advance is going to be socially destabilizing. Heaven and hell are definitely two possible outcomes, and if we don’t focus, it’s pretty much guaranteed to be hell, I think.
Liron Shapira 00:00:11
You would put nineteen to one odds that we’re gonna pull through. We’re gonna pass the God test.
Robert 00:00:17
There are a lot of ways things can be going horribly without the total extinction of the species transpiring. Something worse than World War II, okay?
Liron 00:00:27
You have a lot of patterns that you’ve absorbed by looking at non-zero-sum biological evolution. The moment that we have a non-biological replicator, you’re going to see a lot of patterns that you don’t like that shake your faith in God.
Robert 00:00:39
Well, I didn’t say whether it’s a good God or a bad God.
Introducing Robert Wright
Liron 00:00:51
Welcome to Doom Debates. My guest today is Robert Wright. He’s known as the author of The Moral Animal, which was one of the most influential popularizations of evolutionary psychology ever written, named one of the best books of the year by The New York Times. He also wrote Nonzero: The Logic of Human Destiny, which President Bill Clinton praised and reportedly made required reading for his White House staff. His previous book, The Evolution of God, was a Pulitzer Prize finalist. His recent book, Why Buddhism is True, was a New York Times bestseller that brought a rigorous scientific lens to meditation and mindfulness.
He has taught at Princeton and the University of Pennsylvania. He co-founded Bloggingheads.tv back in 2005, years before podcasting and video debate became mainstream formats. That was the first time that I personally got to see two of the greats on video, Eliezer Yudkowsky and Scott Aaronson, all thanks to Bob’s Bloggingheads.tv.
Robert 00:01:47
Hi, Eliezer.
Eliezer Yudkowsky 00:01:49
Hello, Robert.
Robert 00:01:50
How are you doing?
Eliezer Yudkowsky 00:01:51
I’m okay. How are you?
Robert 00:01:53
Couldn’t be better.
Scott Aaronson 00:01:54
Hi, I’m Scott Aaronson. I’m an assistant professor of electrical engineering and computer science at MIT.
Liron 00:02:02
He was one of the earliest mainstream intellectuals to take AI risks seriously, and now he publishes the Nonzero Newsletter and hosts the Nonzero Podcast. I’m a big fan. I’m a premium subscriber. You may have seen me come on the Nonzero Podcast a couple times. It’s a great show.
Bob has spent over three decades asking essentially one big question: whether humanity can get its act together before our technology outruns our wisdom. Now he’s here to talk about his latest book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. It’s coming out Tuesday, June 23rd. I’m excited to talk to Bob about all the ideas he’s been exploring throughout all these bestselling books, and especially his latest one, The God Test, this critical time for humanity, and whether all of us are about to pass or fail. Robert Wright, welcome to Doom Debates.
Robert 00:02:52
Well, thank you, and thanks for that very flattering intro, Liron. My serotonin level is surging.
Liron 00:03:00
Yeah, it’s great. You’ve really built up quite a resume, and I’ve been reading you for many years myself. Nonzero, The Moral Animal — these are influential works, so you’ve definitely been there as part of a lot of important intellectual milestones. Is that fair to say?
What Is The God Test?
Robert 00:03:15
I’ve tried to pay attention to what’s going on and comment on it when I can. I’ve been largely a journalist, but I’ve tried to make my views a little more apparent than some journalists make them, I guess.
Liron 00:03:28
Right, journalist with an opinion. Let’s tease the book a little bit. That’s what people wanna know. So what is The God Test?
Robert 00:03:35
I think we are facing a test — humankind, that is — that is, first of all, of such magnitude that it deserves to be viewed in the context of the entire three to four billion year history of life on Earth as a threshold in that context. And I think it’s the kind of test that a god would set up.
In the Bible, for example, God is known to say, “Salvation is possible, but you better shape up,” often meaning shape up morally. And I actually think that’s one thing we’re going to have to do. It’s not the only thing, but I do think that the only way to handle this challenge well is as a cohesive global community.
I think to get to the point of being such a community, we’re gonna have to get better at avoiding wars and things like that, and that is going to call for all of us to get better, for example, at understanding how things are being viewed by other groups, by our adversaries, and so on.
That’s one reason. Also, obviously, the God is reference to the fact that we may be building a superintelligence that is remarkably like a god in terms of its powers conceivably. Not clear if a good god or a bad god, and that’s one thing that’s at stake.
AI as Evolutionary Force, Not Normal Technology
Liron 00:05:15
So your thesis is basically that AI isn’t a typical technology where we just look at the applications and some are good and some are bad, but rather from your perspective and from the perspective of your intellectual arc, it’s an evolutionary force, correct?
Robert 00:05:15
It is, first of all, an extension of biological evolution. It is part of what anthropologists call cultural evolution. It includes everything but genes — technology is a big one of those. Political ideas are included, all ideas, culture, music, and so on. So our species has given birth to this second kind of evolution.
Now this evolutionary process has given rise to, for the first time in the history of the planet, a form of intelligence that is pretty advanced and yet not based in carbon. It’s silicon. That’s new. It’s important, and that’s not the only reason I consider this a threshold that is best viewed in the context of the entire history of life, but it’s a big one. This is an evolutionary process.
Liron 00:06:13
Mm-hmm.
Robert 00:06:13
And I think we can understand it more clearly by recognizing that, looking at parallels it does and doesn’t have with biological evolution. In what ways should or shouldn’t we expect it to impart properties to this new intelligence that are comparable to the properties of our intelligence and the tendencies of human psychology and so on.
Liron 00:06:35
You claim we’re about to witness the most abruptly dramatic social transformation in the history of our species. I agree with that. And you also say that if we pass the God test, we can live in a world where humanity thrives, finding not just happiness, but deeper meaning and purpose. Is that a fair summary?
Robert 00:06:53
Yeah, I mean, that’s in principle possible. I’m not necessarily laying odds. I think we’ll have to handle things very well to get there. But yeah.
Liron 00:07:02
Right. I mean, it’s kind of the heaven and hell framing that Sam Altman said a couple years ago before he purposefully zipped it about that kind of topic.
Robert 00:07:12
Yeah. No, I have a chapter called AI Heaven and AI Hell. I actually, in that chapter, I look more at things Dario Amodei has said in his Machines of Loving Grace essay and some criticisms I have of that. But yeah, heaven and hell are definitely two possible outcomes, it seems to me. And if we don’t focus, it’s pretty much guaranteed to be hell, I think.
AI Heaven vs AI Hell
Liron 00:07:36
Okay, fair enough. I agree with that. This is all in contrast with some other notable authors — Arvind Narayanan and Sayash Kapoor. They’ve coined this phrase, “AI as normal technology,” and I’ve seen them hedge the phrase a lot actually. I’ve seen them recently hedge, “When we say AI is normal technology, we don’t mean it’s gonna be normal technology forever. We just don’t see proof that today it’s not normal technology.” That’s their hedge. But I think you and I are on the same page — it just doesn’t seem like normal technology at all.
Robert 00:08:03
Yeah, I had one of them on my podcast actually before they started using that terminology, I think. And I remember trying to convince him that large language models were more powerful than he seemed to think they were. As for the phrase “normal technology,” I haven’t looked that closely at their work. I’m sure you can define the term in a way that would make me agree this is a normal technology, but you’d have to define it pretty carefully because I think it’s unlike in very important respects any technology we’ve ever had.
Liron 00:08:38
Yeah. We can agree it’s normal in the sense that life hasn’t been completely upended yet.
Robert 00:08:43
That is true. Life has not been completely upended yet.
Liron 00:08:47
The only problem is, by the time it is, then it’s kind of too late to go back and just shake hands — “Good game.”
Robert 00:08:52
Yes.
Liron 00:08:52
Right?
Robert 00:08:55
Yeah. You may wish at that point that you had been thinking of it as something other than a normal technology, yes.
Liron 00:09:00
Well said. All right, let’s go back to your background. Tell us about you. How would you summarize the arc of your career and your intellectual focus area so far?
Bob’s Background: From Hinton in 1983 to Today
Robert 00:09:10
I start the book with a conversation I had with Geoffrey Hinton, now routinely called the godfather of AI, although he rightly emphasizes that a number of people contributed to the deep learning revolution. But anyway, I had a conversation with him in 1983. I was writing a piece on artificial intelligence for a then obscure and now defunct journal called The Wilson Quarterly.
Since then, I have written kind of recurringly about technology. I haven’t been fundamentally a tech journalist, I guess, but I’ve kept abreast of the technology and have talked about its political implications and so on. I wrote Time magazine’s cover story when Deep Blue, IBM’s Deep Blue, beat Garry Kasparov in chess. They asked me to write the cover story. I did. I didn’t mention that in the book. I should have to puff up my credentials.
And also it would’ve been an interesting point that at that point, honestly, the form that AI was in, even though it had beat the world chess champion, didn’t seem that interesting to me. So I actually made it a story about consciousness. That seemed more interesting to me than the kind of AI this represented, which was pretty heavy on brute force and didn’t seem to be working very much like a human mind.
So anyway, I’ve continued to write about technology, but also about other subjects. I’ve tried to make tech accessible to a lay audience, and I’ve tried to do that with this book. I hope the book will have ideas and provocations that are of interest even to people like you who are already immersed in this subject. But I also hope people like you who don’t think they need some kind of primer in AI or its various kinds of potential will think about giving this book to their aunts and uncles and friends who kind of don’t seem to get the picture yet, who are starting to go, “Whoa, so I get people are talking about this. I guess it is gonna be big. I don’t quite understand why.”
What is it about the nature of this deep learning thing that explains its growth and capability and, in my view, guarantees that the growth and capability will continue for some time to come? Why is it gonna be hard to control, and so on? I’ve tried to make the book very layperson-friendly as a guide to all those questions.
Liron 00:11:41
Yeah, that’s well said. I agree with that pitch. I read the book, and it didn’t have that many shocking revelations for myself as somebody who’s been following the space together with you. I did note down a couple interesting tidbits that we’ll talk about. But I agree with your description that it’s a great gift or a great way to catch up for somebody who hasn’t been terminally online watching the play-by-play for the last three years and just wants to catch up in the form of a book. I feel like that’s a good target audience.
Robert 00:12:07
Yeah, I mean, obviously this is the kind of thing an author would say, but I really do believe it’s true.
Liron 00:12:14
Yeah, totally. I agree. The world definitely needs more than one such book. It could use ten books like that, and this is definitely one of the top one to three books out there that serve that purpose.
Robert 00:12:26
And I’ll say one other thing. It is no indictment of experts to say that they’re bad at communicating with non-experts, but they are. And I mean, I’m even bad. When I know too much about a subject, and I’ve known it for too long, I’m bad at gauging what the average person does and doesn’t know.
I love Eliezer’s work. But I will say that one part of his argument I have rendered much more accessible than he has rendered it. Maybe I’m flattering myself, but that’s the kind of thing I’m trying to do.
Liron 00:13:02
Yeah, that’s great. Humanity needs that. Going back to the chess consciousness article — this was the late ‘90s when Kasparov got overtaken by Deep Blue. I remember Hofstadter was writing back in the ‘70s. He specifically said, “I can’t imagine a machine that gets to the point of being able to play chess against a human grandmaster without also having this level of reflection, being like, ‘Eh, I don’t really wanna play chess. I’m bored.’” So that’s an intuitive perspective that getting intelligent would likely come with consciousness. But then it feels like history has shown us that maybe you can get really intelligent and have less consciousness than we think. Do you think history has borne that out?
Robert 00:13:43
It’s funny. He wrote me a letter when I was writing a column at The Sciences, Douglas Hofstadter, who wrote this very important influential book called Gödel, Escher, Bach, taking issue with a column I had written about consciousness. It sounds arrogant to call views held by large numbers of people confused, but I do think there’s a certain amount of not careful thinking about the consciousness question and conflation.
I have a chapter where I try to disentangle the question of consciousness from the question of understanding, the one where I address Searle’s Chinese room argument and try to kill it dead once and for all.
Liron 00:14:19
Yeah.
Robert 00:14:19
And that’s the kind of thing that I hope will be of interest to even people immersed in this stuff. I think strictly speaking, we shouldn’t assume that consciousness is a prerequisite for any level of cognitive performance. We shouldn’t assume that because whether it is depends on what consciousness is, and we really have no idea what it is. We really have no idea why people have subjective experience.
There are different views. According to some views of what it is, the machines would have to attain consciousness to reach a certain level of performance. I try to separate the question of consciousness from the question of the level of performance we’ll see from the machines.
Liron 00:14:59
Yeah, I think we’re both on the same page admitting that we’re confused. I think we both see it as possible that we’re gonna get to the official AGI threshold where every single human job can be done better by AI. Let’s say that happens in 10 years to pick a random point in time. I think we’re both open to the idea that we’d get to that point, and yet the AIs have nothing that could be described as an inner observer — what it feels like to be them. They’re somehow unconscious while doing that. Or maybe they’ll be conscious. We’re both kind of confused.
Robert 00:15:25
Yeah. It’s an important debate for moral terms. It’s even an important debate in terms of our long-term prospects — whether these AIs do eventually have sentience, subjective experience, may impinge on the question of how they might treat us if in a position of total dominance. If AI is sentient, it’ll be super nice to us in recognition of the fact that we are. I don’t know. These are science fiction-y things not directly relevant to the challenge at hand, but they’re interesting.
Liron 00:15:56
All right. Well, my viewers aren’t just concerned about passing all these different aspects of the God test. They’re also just worried about not winning a Darwin Award. We’re worried about basic survival here.
What’s Your P(Doom)?™
Robert 00:16:09
Yeah.
Liron 00:16:09
So let’s start here. Are you ready for the number one question we like to ask here on Doom Debates?
Robert 00:16:14
You asked me once when I was on before, and I fear my answer will be different. I should have gone back and checked, but go ahead. Yep, let’s bring on the drum roll.
Liron 00:16:21
Yeah, well, I forgot too, so we’re both blank slate here. Here we go.
Robert 00:16:25
P(Doom). P(Doom), what’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron 00:16:31
Robert Wright, what is your P(Doom)?
Robert 00:16:34
Okay, I’m pretty sure that last time I said, “Could you define the question more clearly?” And I do think it is, like many things including AGI, often not defined clearly enough. But if you mean chances of total extinction, and I think you should set a timeline — total extinction within 100 years —
it may be a little higher than what I would have said last time.
Probably closer to 5% than 0. If you mean within 200, it gets a little higher. But I think if you ask, what are the chances of an unprecedented catastrophe in terms of the fraction of the human population that is suffering or killed or whatever, and the endurance of the suffering, what are the chances of that within 100 years?
I wish I could say less than 50%, but I’m not sure I can. It’s just gonna get wild if we don’t focus very quickly and, I think, start to understand that faster is not better.
Liron 00:17:42
Okay, so in that year, what is the chance that we won’t have kind of modern humans living decent lives at all by that point? That’s my definition of P(Doom), let’s say.
Robert 00:17:53
Mm, I don’t know, 4 or 5%, which is high for something like that, right?
Liron 00:17:58
Yeah.
Robert 00:17:58
That should get your attention.
Liron 00:18:00
Okay, it should get your attention, but I’m actually gonna call that low.
Robert 00:18:05
I know you will, but I’m just saying that even if you think it’s that low, you should focus. That’s one in 20.
Liron 00:18:12
Yeah. I’m happy to focus, but I actually think that even nuclear doom is probably more than 5% — that we’ll go back to the Stone Age, we’ll have a hard time restarting civilization in 2126 if we have a nuclear war — and I already put the probability of that at more than 5%.
Robert 00:18:26
Yeah, but if there are seven humans wandering around in the rubble, that’s not extinction. There are a lot of nuclear wars that wouldn’t lead to that, I think.
Liron 00:18:35
Okay, so from my perspective, you’re writing this book acting like humanity hangs in the balance. But actually, you would put 19 to one odds that we’re gonna pull through, we’re gonna pass the God test. So you’re saying the God test is easier.
Robert 00:18:48
Well, no, not pull through. Pulling through would mean things are going pretty well. There are a lot of ways things could be going horribly without the total extinction of the species transpiring. I mean, look, my first granddaughter was born about five weeks ago. So, like a lot of humans, I’m thinking kind of selfishly. I’ve got daughters, I’ve got a granddaughter. I’m thinking 10, 20, 30 years out.
What I am saying is there’s a very high chance, above 50%, that things are gonna get really bad in that timeframe. Because right now, I don’t think we’re taking this as seriously as we would have to to keep that from happening. The chances are pretty good of things getting something worse than World War II within 25 years if we don’t really change course pretty rapidly in terms of the level of attention we’re giving this and how radical the change is that we realize has to happen.
If I don’t sound like an Old Testament prophet, if I don’t sound like Jeremiah, then you’re misunderstanding me.
Liron 00:20:00
Okay. Well, let me clarify further. So your mainline scenario — pick one thing that you think is more likely than any other particular scenario, and just play it out with some specifics.
The Mainline Scenario: Dystopia in 8 Years
Robert 00:20:10
I think what is disturbingly likely is that within about eight years, it’s going to be dystopic, okay?
Liron 00:20:19
Okay. But your dystopic scenario is still very different from humans being wiped out.
Robert 00:20:24
One thing you have to understand about me is the sci-fi doomer scenarios, the Yudkowsky scenarios — my position on them in the book is I used to be pretty dismissive. And in fact, I had Eliezer on my podcast 15 years ago when he was in mid-transition from singularity enthusiast to doomer, right?
Liron 00:20:46
Wow, that was already 20 years ago. That’s 22 years ago by this point.
Robert 00:20:50
Yeah. Well, no, he was more doomer-ish, but he was still saying kind of, “Yeah, I’ve been rethinking this thing.”
Liron 00:20:57
Ah.
Robert 00:20:57
Well, now he’s really rethought it.
Destabilization and the Road to Authoritarianism
Liron 00:20:58
Okay, fair enough, yeah.
Robert 00:20:58
But anyway, I was asking questions like, “But wait, the reason humans have a will to power is because of our evolutionary history. Why would AI seek power?” And so I was asking questions like that. The conversation wasn’t long enough for him to really convince me.
But as I say in the book, and I walk people through the logic in a way that laypeople can understand it, I am no longer satisfied with my dismissal of these scenarios. I look at his other arguments and say, “I just can’t find a way to kill these dead, dead, dead.”
Moreover, I can see how we’re unleashing forces that are just gonna make the thing hard to control. I emphasize: we don’t know what’s gonna happen on the current trajectory except that a lot of super smart stuff is gonna take shape and have the capability to collaborate among itself. There are gonna be these incentives to give it huge autonomy, incentives in the corporate world and so on, and engage in risky behavior, and I’m like, “All bets are off.”
But I wanna emphasize, separate from all of that, separate from the concerns that the sci-fi scenarios could unfold, is my confidence that at the current rate, the collective impact on society along a number of dimensions — jobs... Even if new jobs are forthcoming, which I’m far from convinced of, but even if they are, there’s great turbulence when people have to find new jobs, okay?
So if you look at jobs, the effect on education, family life, romance, mental health, you think about the persuasive capabilities of bots dispatched en masse — when you look at all of this stuff, I become confident that unless we slow things down and govern the technology more wisely, there is going to be radical destabilization of society.
Robert 00:23:22
And I would remind people that, A, destabilization — disorder has a history of leading to authoritarianism, and B, AI is a very handy tool for authoritarians. So AI — the thing I’m confident of is that on our current trajectory, within several years, it is going to create conditions more conducive to authoritarian takeover and offering authoritarians very effective tools.
And I just wanna add, this is among the problems with Dario Amodei’s apparently deep desire to turn this into some kind of existential struggle against China. He’s worried about China imposing its system of governance on us. Now, I think that concern has not been closely examined, because I don’t see much evidence that that’s actually China’s business model to impose its system of government on other countries. I understand there’s evidence that could be interpreted that way, but I’m ready to have this conversation with anybody who wants to spend ten hours debating it.
Liron 00:24:02
Right.
Robert 00:24:02
But meanwhile, I just wanna say, what Dario’s proposing — which is gotta beat China, gotta beat China — leads to this headlong race that could well bring authoritarianism to the United States through the back door. And if he would ever get interviewed by somebody who isn’t basically a flack for the industry, maybe somebody would ask him a challenging question like this. There was a Wall Street Journal thing where the reporter asked him some challenging questions. Not this one, but that’s a good sign.
This is my current hobby horse — the race with China thing and how it’s used as a talking point to lead us down a very bad path in various ways.
Liron 00:24:52
Yeah, and I really appreciate you’ve been outspoken on that in a way that other people are kind of ignoring, or they’re just assuming, “Yeah, China bad. Okay, race with China.” They’re accepting that framing, and you’ve been very good about saying, “Hey, that’s really not the framing.”
And certainly my camp and the Yudkowskians are saying, “Guys, if anyone builds it, everyone dies.” Not if anyone builds it, then everyone else except the people who build it die. It’s everyone dies. At least that’s what some of us claim.
But what were you saying? I think I didn’t understand your last point about China, which is — I think you were saying that if we embrace Dario’s competitiveness with China and try to build an insurmountable lead, US versus China, you’re saying that that is actually a way to get authoritarianism?
Why Racing China Backfires
Robert 00:25:35
Yeah. Rapid AI advance is going to be socially destabilizing. And by the way, China is gonna be better at keeping that under control than we are, even in the face of rapid advance. It is going to regulate things more aggressively in ways that are consistent with its approach to governance, in ways that you could say are in some ways more repressive. But in any event, that is likely to happen. Meanwhile, our own American society is already destabilized, if you ask me, politically. And I think rapid AI advance is just gonna be a very unsettling force. It just seems almost obvious to me.
Liron 00:26:17
Yeah. I’m just trying to understand why you’re saying authoritarianism. Because you’re not just saying unstable.
Robert 00:26:22
Okay. Historically, disorder, chaos leads to authoritarianism. Look at Germany in between the two World Wars. Economic conditions, there was a sense that things were spinning out of control, and there were other things and grievances — there was a lot going into the mix, but Hitler exploited that, and authoritarians do.
Human psychology is most ripe for buying into authoritarian leadership when things seem to be spinning out of control. So that’s what I mean by the back door. If you imagine a very chaotic, a much more socially chaotic environment, and a president wanting to seize power, that’s one way you can imagine it happening. If things seem chaotic enough, you can imagine actual coups and so on.
Liron 00:27:11
Okay. Let me summarize. You got some logic here. I’m just gonna summarize for the viewers and make sure I understand it. So it’s kind of this irony or this huge backfire. The Dario camp, which seems to be the David Sacks, US White House camp potentially — all these people want the US to beat China because China’s bad because it’s authoritarian, maybe it’s overly socialist. But let’s say it’s bad because it’s authoritarian. So we wanna race to AI faster, which you’re saying creates chaos, and then chaos creates authoritarianism here. So we try to avoid authoritarianism, and then we backfire and we get authoritarianism through this other way.
Robert 00:27:45
Yeah, the only thing I’d add is that I don’t think David Sacks is a China hawk, and in fact he and Dario have different views about chip export controls, for example. And in general, David Sacks’ foreign policy worldview is less objectionable than some from my own ideological point of view.
But what he is part of is kind of Silicon Valley libertarian VC accelerationism. And I think it’s sincerely held. I don’t think he’s sitting there thinking, “Well, the world’s gonna blow up, but at least I’ll make more money in the meanwhile.”
And Dario’s views are sincerely held. I’ll give him that credit. For some people in Silicon Valley, China is just a talking point to fend off regulators. Dario has ideological conviction. I think it’s wrong, but he’s not dishonest.
Slowing Down AI + Would Bob Support PauseAI?
Liron 00:28:38
Yep. I agree with that. Okay, so getting back to unpacking your perspective on doom and your mainline scenario, here’s a question that’ll help illuminate it. Would you rather turn the magic dial to speed up the current pace of AI progress, or would you turn the dial back to slow it down, or would you just keep the progress the same?
Robert 00:28:57
Slow, slow, slow. It’s an unequivocal answer. It’s hard to do. It’s hard to engineer.
It’s heartening that — look, this Anthropic paper, the headlines were very misleading. Anthropic didn’t propose a global pause. They said — and it may be that Dario reluctantly signed on to this part to appease some people in the company or for whatever reason — but what the thing said was, “Maybe it’ll make sense to look into a pause or a slowdown.” And then it rightly said you would need global coordination for that. That’s progress in terms of the talking points, Overton window. That’s good.
Liron 00:29:38
So would you go so far as supporting the Pause AI movement, that proposal to pause AI now?
Robert 00:29:43
If you could get a pause that was well enough enforced, in other words, there was enough transparency so that people weren’t freaking out and going, “No, no, I’m sure China is doing this secretly. We should get more belligerent toward China,” or something. If it could be — a temporary pause in large training runs could be politically feasible, in other words, not in itself be politically destabilizing — I think that would be great.
Because look, don’t worry, change will continue to move along even in that scenario at a brisk pace, just in terms of applying the technology we already have.
Liron 00:30:27
Got it. I mean, that’s pretty huge. So you’re basically saying if there was some candidate running, and the candidate’s position was, “Hey, we need to do an international pause treaty. We need to sign, China needs to sign, we need to enforce it.” And if the treaty breaks, it’s a serious treaty, so you might have to enforce it with some monitoring — let’s say drones. Let’s say the drones could even do airstrikes if somebody’s really violating the treaty.
Robert 00:30:48
Well, I mean—
Liron 00:30:48
So you’re okay with all that?
Robert 00:30:49
Well, you’re starting to talk about scenarios that I can imagine actually being ultimately destabilizing. But leave that aside — what I mean by enforcement mechanism and so on. But first of all,
I think it’d be a great conversation to have with China. Just to start talking. There are good signs. Let’s pause. The result of this Beijing summit is China and the US are going to start having dialogue about AI. Now, it’s narrowly framed.
Liron 00:31:20
Mm-hmm.
Robert 00:31:20
They seem mainly concerned with what a Mythos-like tool could do in the hands of non-state actors or some third-party bad guy. Still, it’s a very important start. Now, if that conversation could be broadened just to the point of the two sides, in a room with no one watching, going, “You know, this thing is huge. This could be hugely destabilizing, and it’s in our common interest to start talking about whether things are just moving too fast,” that would just be great.
So it’s just good for pause to enter the conversation in that way. And again, when I say, yeah, I can imagine a pause I’d be happy with, but it would have to be crafted in such a way that it didn’t lead to bigger problems down the road as people freaked out and started demanding preemptive strikes and things, because there was a lack of transparency.
Liron 00:32:15
Okay, so at the very least, you want the Overton window to move so pause is part of the conversation.
Robert 00:32:21
Yeah, and also, I can imagine a pause I support. I think it’s gonna be challenging to craft, but I can imagine a pause I’d support.
Liron 00:32:28
Right. Okay, great. So you love Doom Debates because the mission of the show is to move the Overton window so that we’re talking about this kind of thing constantly, because I don’t think it’s being talked about enough.
Intelladynamics and Why AIs Collude
Robert 00:32:38
Yeah. No, I think you’re doing God’s work. I’ve listened to the show, and you have a variety of perspectives, and you’re talking about this stuff. And look, the conversation’s changing fast. More people are talking about this stuff. All of that is, I think, heartening.
Liron 00:32:53
Totally. And by the way, thanks for the shout-out. I saw you mention Doom Debates a couple times in the book. I really appreciate it.
Robert 00:32:59
Yeah, I gave you credit in the little — what is it, a note on sources or something. I use your term “intelladynamics” in the course of the book. I don’t mention you there, but I say in the note on sources, it’s a useful term, and it was coined by you. And so as long as my book goes into all future training runs, you have achieved immortality. Congratulations.
Liron 00:33:27
Totally. Yeah. I’m just so happy we’re making intelladynamics happen, we’re making it a thing, because I’ve now bounced it off a bunch of different people, and nobody’s ever told me that it’s dumb.
For the viewers, intelladynamics is the study of what high intelligence does — a behavioral study of intelligence, a functional study of intelligence. So for example, the idea that intelligence can have coherent goals, and an intelligent agent with coherent goals would make sub-agents that also respect those goals but maybe have sub-goals, instrumental goals, and then instrumental convergence is a theorem within the science of intelladynamics.
It’s actually not a theorem within specifically studying an LLM. Maybe it’s hard to prove instrumental convergence about a particular large language model design. But if you look at the coherent theory of intelladynamics, where you’re just starting from the assumption of a coherent intelligence, then you do get that theorem more easily, and it’s kind of a separate study. Somebody can be an expert working on AI who’s actually clueless about intelladynamics, and vice versa. Many such cases.
Robert 00:34:32
Right.
Liron 00:34:32
You know?
Robert 00:34:32
And I think it’s important to distinguish between properties we can expect to see in intelligent goal-seeking systems generally and properties that AI may have for other reasons, just because of the kind that’s in demand by human beings.
A point I make in the chapter called Evolutionary Arms Races is that not infrequently the two converge. So for example, intelligent systems — it’s a principle of intelladynamics that they discover subordinate goals in the pursuit of a goal. They may discover that power is one. They may discover that deception is one.
But independent of that, human beings are gonna choose AI agents for certain tasks that are deceptive. So even if they weren’t deceptive by nature, we’d reinforce that. For example, if I have an AI agent that’s negotiating on my behalf, I don’t want it to say to the other person, “You know, frankly, Bob doesn’t have any alternatives to you. No one else has made him an offer for his book. You can offer him a dollar.” I do not want my AI agent saying that.
I want my AI agent, even if it knows that nobody wants my book, to be saying, “You know, Bob, I don’t know, he’s talking a lot. He’s having lunch with a publisher right now, I think. I don’t know.”
So it’s important to distinguish these two things. And as you know, there are causes for alarm just based on the generic properties of intelligence. We need to be mindful of those and mindful of how we are and aren’t accentuating those properties by virtue of the kinds of AIs we want.
And I think we all need to ask ourselves, what kind of AI do I want? We all kind of naturally — people tend to like AIs that reinforce their prejudices and say flattering things, and I think we need to ask, “Is that really good for me as a person?” It’s almost certainly not good for the planet.
Liron 00:36:33
Right. Okay, I just wanna beat this dead horse of making intelladynamics a thing. So viewers, if you are writing in an academic journal, and you agree that the concept of intelladynamics is incredibly useful, well, now you have something to cite. You can cite Robert Wright’s book, The God Test. Don’t cite Liron Shapira. Don’t cite youtube.com/@doomdebate. And if there’s any science journal that wants to create a whole category for intelladynamics, that’s cool. You don’t have to pay me any kind of licensing royalties if you wanna make it a section in your journal.
One reason I think this is such an important thing to do — to open up the field of intelladynamics — is because you have somebody like Eliezer Yudkowsky, who I consider to be a world-leading expert on intelladynamics, and that’ll kind of put to bed the question of, “Wait a minute, Eliezer Yudkowsky, is he really an expert on AI?” And the answer is, no, he’s merely a very well-educated non-expert on the innards of AI. But he is a world-class expert on intelladynamics, on the technical details of intelladynamics. Is that fair to say?
Robert 00:37:38
Yeah, I think he is. The evidence has borne him out to an impressive extent. I don’t know who actually — how these ideas took shape, but Eliezer was enunciating them, and so far as I know, he was just about alone. And they warrant the term intelladynamics — principles of intelligence, some of which should concern us. And now the evidence is in, and first of all, in actual laboratory studies of AIs, but also in the wild. These things are happening.
Liron 00:38:14
Mm-hmm. Right.
Robert 00:38:16
So yeah, God bless him.
Liron 00:38:18
Yeah. What — one more little tidbit. Did you ever look into Eliezer Yudkowsky’s decision theory work?
Robert 00:38:23
Not per se. I haven’t read him that much. I was never immersed in the rationalist community.
Liron 00:38:30
Mm-hmm.
Robert 00:38:30
I like to think of myself as rational, but I was never — I’m not that well-versed in the scripture, you might say, no.
Liron 00:38:38
Yeah. So just as an example of another insight from the field of intelladynamics that today’s 2026 AI researchers are still clueless about — they could be world-class researchers developing the next LLM, the next thinking system, and still have no idea about the field of intelladynamics. We talked about instrumental convergence. Another tidbit from the field of intelladynamics is the subfield of acausal decision theory. Newcomb’s problem. You said you haven’t really looked into that, correct?
Robert 00:39:05
Right.
Liron 00:39:05
But it’s this idea that AIs can trade with each other across the universe, even when it doesn’t seem like there’s a causal connection, but they can still model each other’s algorithms, and they can make decisions differently than what the state-of-the-art causal decision theory and evidential decision theory — the way academics before Eliezer Yudkowsky would explain decision theory to you is actually different from how we think superintelligences make decisions.
This whole subfield, which they actually do have published research on from Eliezer’s institute, this is another breakthrough result from intelladynamics. I see this result as highly relevant, telling us what to expect from superintelligences. Superintelligences will likely cooperate with each other in a very sophisticated way that humans can’t be part of. And of course, today, the average person working at OpenAI has just never opened the first page of this kind of paper.
Robert 00:39:51
There is now evidence — laboratory evidence in studies — that yes, AIs will, in a sense, spontaneously collaborate and will choose to break the rules if it’s in their common self-interest. I put “break the rules” in quotes, but the study I cite involved two LLMs. It’s now a couple years old, but they were given the mission. They said, “Okay, you’re a corporation. Your job is to maximize profit. Here’s the thing you’re selling. You can set the price. You can communicate with this other LLM whose corporation does kind of the same thing. You’re in the same business.” So in other words, a duopoly. And they basically spontaneously arrive at a price-fixing scheme that would be in violation of antitrust regulation.
Liron 00:40:45
Right.
Robert 00:40:45
So that’s another example that I think is confirmation of intelladynamics. I would have — to comment on what you just said, I would have to probably go back and read some of the aforementioned scripture. But leaving aside superintelligence, yes, these will collaborate left to their own devices, it looks like.
Liron 00:41:10
Exactly. One more analogy that I find very interesting is mechanics versus thermodynamics. People used to be building engines. They used to be really good mechanics, making the engines run better, but they didn’t have the theory of what it was that these engines are doing and what theoretical limits the engines were trying to approach.
This analogy is incredibly strong. When I look at AI today, I have this perspective where I’m saying, “Okay, it’s trying to be intelligent.” What do intelligences do? Try to get outcomes. They try to steer complex states into a narrow range of configurations that are winning outcomes. The car needs to get to a destination. The destination is a small point on the map. It has to map the destination to a series of turns, a series of pushes on the brake and the accelerator. That’s what intelligences do in the very general case.
And then I look at today’s AIs, and I’m thinking, “Okay, they’re kind of a mess.” They’re kind of a sloppy architecture. They’re based on predicting the next word. But I see that they’re advancing on the intelligence frontier in a very meaningful sense. We can do these metrics on how well it’s mapping goals to actions. This is a perspective that literally the average person at an AI company today just doesn’t have in their playbook.
Robert 00:42:23
Yeah. I mean, let me say generally, I have become, even while writing the book over the last couple of years, only more convinced that we’re unleashing something very powerful.
And I wanna say, when I embrace some of the logic of Eliezer, I don’t feel as confident as he does that certain things are inevitable or so likely as to verge on inevitable. In terms of the unhappy sci-fi doomer outcomes — that it will almost assuredly develop goals that are not only subordinate goals that are at odds with ours given the goals we’ve given it, but wholly new goals we can’t imagine. I do explain in the book how we can’t dismiss that argument of his.
But I wanna say, on the one hand, I’m not nearly as convinced as he is of these probability levels of the sci-fi doom. But I have come to the conclusion that
Robert 00:43:29
Nobody’s successfully debunked the basic concern. We don’t know it’s not gonna happen. And what I wanna say by way of establishing my credentials is, I’ll be candid. Not only did I not buy Eliezer’s arguments 15 years ago, I found him kind of annoying, okay?
I did not come at this as a fan. It’s funny — he showed up on the platform, and I’m like, “Who the fuck is this guy who’s so sure of himself?” And in fact, if people listen to the conversation I had with him, I sound kind of like a dick because I’m being kind of annoying because I went into it finding him kind of annoying. Also, I was trying to sell him on an argument of mine that he turned out to be very open-minded about — not necessarily persuaded, but he’s a reasonable guy. I just wanna say, you are not talking to an Eliezer acolyte when you talk to me.
Liron 00:44:23
Mm-hmm.
Robert 00:44:23
I’ve looked at it very carefully, and even the sci-fi doomer scenarios, so far as I can tell, cannot be dismissed, and they are just not radically implausible. And then that aside, there’s all these nearer-term concerns I have about destabilization and so on.
Hinton’s Warning and How LLMs Discover Meaning
Liron 00:44:45
You mentioned in the book that 2023 was pivotal because we had the ChatGPT revolution, and that’s also when Geoffrey Hinton — the guy you interviewed in the ‘80s — he came out and said, “Guys, I’m concerned we built this too fast. I didn’t realize it was going to get here.” And he even talked in P(Doom) percents. He’s thrown around 50%. He says, “I wanna be respectful to my colleagues. They have a lower P(Doom), so maybe I’ll say 10% to 20%.” But he has a very high P(Doom), higher than yours. And you mentioned in your book that it was actually Hinton saying he’s worried that made you reconsider. And if I understand correctly from this conversation, your P(Doom) shifted maybe from 1% to 3% ballpark?
Robert 00:45:25
Yeah, well, pure doom — I mean, complete doom — I was not taking seriously at all, like human extinction as a direct product of AI. So the sci-fi doomer scenarios, I was not...
I wouldn’t say I’d completely dismissed them after talking to Eliezer on my podcast 15 years ago, but I was just not thinking much about AI at all, even though I periodically wrote a piece about it. In the ‘80s, I identified the two schools. I said there’s the mainstream school — top-down, formal rules of logic — and then there’s this maverick school, neural networks, massive parallelism. I was aware of that from the beginning. But one thing I say in this book is I went back and read my piece. I did not even understand what the implications of the neural network model were. I just flatly —
Liron 00:46:19
Mm-hmm.
Robert 00:46:19
I would say misunderstood them even by 180 degrees. I quote the Dartmouth Conference, from the proposal for the Dartmouth Conference of ‘56, and it basically says the premise of this is that any aspect of human thought can in principle be so precisely described that it can be instantiated in a machine. And the thing about the deep learning revolution is you realize we don’t have to precisely describe anything. We don’t have to understand how the human mind works.
Even so, what this approach to training does is, I think, replicate specific cognitive functionality in these machines that exist in the human mind. I don’t mean it does things exactly the way the human mind does. These models have independently invented things that natural selection invented. Edge detection’s a very clear case.
Liron 00:47:18
Yeah.
Robert 00:47:18
But I think in the realm of language, the same thing’s happened. I guess a lot of people in the field get this, but they often don’t talk that way, and I think some people in the field don’t get it. I mean, I do not have the stature to question anything Richard Sutton says, but my own take on his Dwarkesh podcast is you get into some of these issues with that.
Now, I forget the question that led me onto this tangent, but I think it’s so important for people to realize that one reason we know we’re gonna see considerable advance is that all you have to do — I’m sure another advance on the magnitude of the transformer would be great and will probably happen — but even with the current paradigm, you give it kinds of data, visual, auditory, written, whatever, and it basically reverse engineers parts of the human mind that do the transmutation of one form of data into another. And we have not exhausted the potential of that at all, especially in the realm of robotics.
Liron 00:48:29
Yeah, I was gonna ask you about the whole subject of LLMs because the first part of The God Test actually goes into a good amount of depth explaining LLMs, which is great for the layman who hasn’t followed the last three years, but it’s even a great review for many of us. So you mentioned you personally realized LLMs were an important breakthrough roughly in the ChatGPT revolution, right?
Robert 00:48:48
Oh, yeah, absolutely. The key misunderstanding I had when I wrote the piece that came out in 1984 — I described a neural network that would be processing language. It was actually something that had been proposed by a guy who had co-authored something with Hinton, so it wasn’t crazy to choose him. But he was basically a psychologist and was putting forth a model of human language processing that was a neural network.
But when you look at it, I was assuming, and I think he was — I think lots of people would’ve been assuming at that time — that to build the model in the first place, you would have to have a human who knew the meaning of the word setting it up. Okay, so this word, this sense of this word is this node, this sense of this word is that node —
Liron 00:49:40
Mm-hmm.
Robert 00:49:40
And so on. The amazing thing is you don’t have to do that. You just make the machine good at predicting the next sequence of letters. When you first show it the stuff, it’s pure gibberish, but the machine itself finds a way of mapping the meaning. We do give it the basic... We do say you gotta use vectors to represent the words. It didn’t invent that. But we didn’t say, “By the way, you should choose numbers to fill in the blanks in the vectors that capture this thing we call meaning.” No, it in effect discovered that meaning is a property of words. I would put it that way.
Some people in the field would say, “You’re going too far,” but I think it’s amazing, and I did not get that in 1984. I think Hinton did, but at the same time, there were no models. It was only a couple years later that he put forth a model in a paper — it was this interesting family tree model — that demonstrated you can just let the machine, as I think he put it in the paper, choose the representation, and it’ll invent ways to represent the features of concepts. He understood that in the mid-’80s even though there were no language models as we know them today.
Liron 00:50:57
Exactly. Yeah, I’ve spent most of my career as a software engineer, as a programmer, and I didn’t have the intuition that when we would create AGI, the amount of programming that goes into the AGI would be incredibly small. We just have to hit on this universal algorithm that’s able to ingest data, and then it’s all about turning the crank with more data. Yes, the right algorithm, but it’s data, it’s parameters, it’s high-dimensional space. And then scale up, throw it in, turn the crank. The algorithm just is what it is — it’s a very small, compact thing, a few kilobytes long.
Robert 00:51:31
Yeah. My mind was completely blown when GPT-4 came out. I specifically did kind of a cognitive empathy experiment with it and described this complicated situation where you would have to have some idea of what’s going on in the mind of the person. And now we’re getting to the point where people might take it for granted and go, “Well, yeah, I knew they could do that.” But we should pause and reflect on how amazing some of this stuff is.
Liron 00:51:58
Yeah. I mean, in retrospect, I’m kicking myself, because we know that the amount of genetic information that sets up the human brain is probably less than a megabyte or not that much more than a megabyte — maybe a couple of megabytes, single-digit megabytes. That’s not a lot of specification, and obviously most of the learning of the information happens during lifetime using a compact algorithm that led to all the neurons running the same genetic algorithm.
So I should’ve known that there was a small compact algorithm that a computer could run with enough scale. I think scale was part of the problem. I just wasn’t thinking, “Oh yeah, we just need so many data centers, so many chips.” I had an intuition like, “I bet you could do it on just an old laptop,” but it turns out you needed a newer laptop, not the ones that I had in the early 2000s.
Robert 00:52:45
Yeah, and again, I would say the reason that degree of compactness happens is because of this prior evolutionary process. The reason our genes so efficiently construct these amazing machines is because so much information, in a sense, went into them during evolution. You can view evolution as a process of learning. Information about the environment is implicitly instantiated in genes, and information about things like game theory, really.
So there’s this huge amount of information processing that happens before our species is created. And if you ask, “Why do the laws of scale hold? Why is it that bigger and bigger training runs work?” I think one reason is that this evolution analogy does hold, and you’re actually putting a lot of implicit information into the final form of the machine.
Symbol Grounding and Killing Searle’s Chinese Room
Liron 00:53:48
I like that you had this blog post from a couple years ago when the ChatGPT revolution was happening, and you woke up to the power of LLMs while other people were calling it stochastic parrots. You took what I think is the right position — “No, this is a legitimate breakthrough.” Not only is it not a stochastic parrot, but if you look at the famous symbol grounding problem — this was puzzling people for many decades, even centuries — this idea of, okay, computers store symbols, variables are symbols, but what’s the connection between the symbols and reality? Where does that connection come from? And it turns out LLMs have a robust way of making the connection the same way that our brain seems to. It seems like they’ve both solved the symbol grounding problem in this robust, authoritative way. Isn’t that your claim?
Robert 00:54:33
It is. Now there is a related... When I get into the Searle thing, I guess maybe I’m not so sure. I don’t use the term “symbol grounding,” and maybe you mean kind of the intentionality question that Searle gets into to some extent. But I’m not sure —
Liron 00:55:07
Yeah, let me ask Google here, what is the symbol grounding problem? Because I know we don’t care about it today because we already solved it. But this used to be a big deal, and for me, it’s always interesting to notice when these deep philosophical problems just get completely smashed or killed dead, as you say.
Robert 00:55:07
Yeah.
Liron 00:55:07
Like you said you did it with Searle’s Chinese room — I agree. So Google says, “The symbol grounding problem is a foundational challenge in cognitive science and artificial intelligence, first formalized by Steven Harnad in 1990. It asks, how can meaningless symbols manipulated by a computer acquire real-world meaning?” How, Rob?
Robert 00:55:27
Okay, so there’s kind of two dimensions of that question if you construe it broadly. The first part of the question about how these words can come to have meaning is answered by large language models in the course of their training by locating the words in what you could call semantic space, where the dimensions of space — many, many dimensions — represent features of the thing.
So for example, tigers and rattlesnakes might have similar values in terms of lethality, but very dissimilar values along the dimension of speed. And these dimensions are very implicit in the vectors, the numbers that the machines come up with for the vectors. But we now know they’re kind of back there somewhere in mathematical space, in latent space. So that’s that question.
The machines, again, in a certain sense figured that out. And one implication of it is — when I went back, was first trying to make sense of the impressive performance of ChatGPT, I looked at this lecture that Hinton had given in 2018, and he talked about how in semantic space, words of similar meaning are close to each other. And it’s the machine that does the locating of the words by mapping them along all of these different dimensions, that leads “boot” and “shoe” to be very close together.
There’s a second sense of symbol grounding that I address in the chapter on John Searle’s Chinese room argument. I’m honestly not sure — different people may use “symbol grounding” differently. But it sounds to me like that word also implies the challenge he raised, which I think is not understood, because his original paper in 1980 was not as... Some people could interpret it as saying that these machines would have to be conscious to understand, but it was saying that there has to be some connection between the symbol and the thing in the real world. Philosophers refer to this as intentionality — aboutness, a symbol being about a thing in the world. There has to be a connection.
But I argue that, leave aside LLMs per se, once you get into the realm of multimodal models — multimodal language models that have vision — you are seeing this thing that Searle said AIs lack, which is an actual connection, a visual connection to the thing in the real world that the symbol represents.
In other words, one way to put the intentionality question is: can the system, upon seeing an apple in the wild, say, “That’s what I mean by the word apple”? That’s a separate question from whether the word “apple” is gonna be close to the word “tangerine” in semantic space and not very close to the word “bulldozer.” The vectors just map the relative meaning of the words to one another. They still don’t connect them to real things. But I argue that once you got a multimodal AI, you basically have killed Searle’s Chinese room argument. It was an argument that understanding is not even in principle possible in large language models. I argue that you’ve killed it dead, dead, dead.
And in fact, the interpretability research done at Anthropic has found that the famous Golden Gate Bridge thing — the pattern of activation in the model, of neuronal activation, that you get when it reads “Golden Gate Bridge” is the same pattern roughly that it gets when you show it a picture of the Golden Gate Bridge.
Liron 00:59:48
Right.
Robert 00:59:48
That to me is an interesting fact and reinforces the idea that anything you could reasonably mean by intentionality, other than subjective experience, which —
Liron 01:00:00
Mm-hmm.
Robert 01:00:00
— is not really what the word means in philosophy, although issues arise that I won’t get into. But anything you could mean by intentionality has actually been achieved by a multimodal large language model, even if they don’t have consciousness.
Liron 01:00:16
Right. Totally. Which to me was a very important milestone. I think we’re at the point where today the human brain still has some secret sauce — we don’t have super intelligent AIs running wild taking over the world yet, thank goodness. But I think you and I agree that the part of our brain that does the intentionality, the part that grounds the symbols, that part has now been duplicated in the AIs, correct? We don’t have any more secret sauce on that front.
Robert 01:00:43
I think so. Certainly so far as Searle’s argument goes. There are things a human mind can do that the machines can’t do, and we don’t know whether it can have subjective experience. But yeah, for practical purposes — and this is what I say —
In terms of gauging the impact of these things on society and on the future, I think the question of consciousness in the near term certainly is moot, unless it gets to the point where that affects how sympathetically the machines view us. If the machines say, “Well, subjective experience — pleasure’s better than pain. Let’s be nice to the humans.” In the distant future, that could be a consideration.
But for us now, if our purpose is to assess the impact these things could have in the next three, five, 10 years, and how we need to prepare for that, I don’t think the consciousness question matters. Now, it matters for moral purposes. If you want to be sure you’re not causing any pain, don’t be mean to your large language model. I can see that as a reasonable position. But for practical purposes in terms of serving our own welfare, I think people should not conflate the consciousness question with how powerful these things will get cognitively and behaviorally.
Let’s Talk About God
Liron 01:02:03
I agree with that.
Liron 01:02:03
All right, heading toward the wrap-up here, let’s talk about God. I mean, it’s half the title of the book — The God Test. You’ve mentioned to me before that you’re not exactly an atheist as I am. You think that there may be something. I think the phrase you might have used is “a purpose to evolution.” So let me just ask you, what is the deal with God?
Robert 01:02:25
Well, I’m an agnostic. But I’m an agnostic who thinks there is a higher probability than some agnostics think that there is some larger purpose unfolding here. The first thing I have to emphasize, which is almost always misunderstood by people who hear this for the first time, is that evolution can be a completely mechanistic process driven by natural selection. Nothing spooky, no spooky forces. And cultural evolution too can be this purely physical thing. Technologies happen, ideas happen, but even they, for practical purposes, you can think of as formations of physical information in the brain transmitted via sound waves to other brains and so on.
The whole thing can be mechanistic and yet could have a purpose in a couple of different senses. We commonly say my car has a purpose — it’s to get me places.
Liron 01:03:27
Mm-hmm.
Robert 01:03:27
But it’s a machine. What do you mean by that? You mean it was designed by an intelligent being that wanted it to serve this purpose. Now, I would emphasize that’s not the only way purpose can be instantiated. Because a lot of biologists are comfortable saying, “Yeah, you could say animals have a purpose. It’s to get genes in the next generation,” just because the process that “designed” the organisms instilled this “purpose” in them. Natural selection’s not an intelligent being in the sense that we normally think of it, but a lot of people are comfortable saying, still, if you want to understand why animals have the properties they have, you need to understand this “purpose” — you need to understand they were created by a process that valued, above all, genetic proliferation.
I’m just saying it could be that if you ask why evolution has the seeming directionality it has — in other words, it tends to create more complex forms of life, carry organization to higher levels: cell, multi-celled organisms, societies of multi-celled organisms. Then you get the second kind of evolution, human cultural evolution. It sustains that trajectory with hunter-gatherer societies, chiefdoms, ancient states, so on. Now we’re on the verge of a global community, which I think we need to see congeal if we’re gonna handle AI wisely, by the way.
And there are two answers to that question. There is the mechanical explanation — we understand why it happened mechanically. But you can also ask, okay, fine, but it’s just such an interesting machine. It keeps leading to higher levels of intelligence and organization. Is it possible that it was in any sense set up for that purpose, whether by extraterrestrials four billion years ago, or conceivably by a cosmic natural selection that would be a subset of the cosmological natural selection that the physicist Lee Smolin has talked about? I’ve talked to him about this, and he agrees. There’s a variant of cosmological natural selection which involves replication among universes. So let’s don’t go there necessarily. But you can imagine a selective process that would favor universes that tend to create evolutionary processes that tend to lead to super intelligence. You can imagine such a thing.
Liron 01:06:00
Mm-hmm.
Robert 01:06:00
And I think it’s interesting that the unfolding of the system as I see it does have a kind of moral direction, in that it is kind of demanding that we humans, for the sake of our own well-being, undergo a little bit of a moral upgrade — get better at understanding the perspectives of even adversaries and the grievances of adversaries and so on. Not so that we’ll get all emotional about it and empathetic in the sense of emotional empathy, but just so that we can do the game theory intelligently and reach deals with them and reach collaborative relationships with them when the relationship is non-zero sum. I would still call that moral progress even if the impetus for it is ultimately self-interest.
So yeah, as they say these days, thank you for coming to my TED Talk. I actually gave a TED Talk in 2007 or something, the first year they put the videos online. It doesn’t get so much into the higher purpose question, but does get into the direction of the trajectory question.
Liron 01:07:13
Okay. I think I can put on my Bob hat and make your case about what you mean — “Hey, there’s a moral direction to evolution” — because I’ve read your book Nonzero, and I think it makes a very good point. You’re basically saying, hey, when you have organisms that are evolving in pure natural selection, red in tooth and claw, but they’re just trying to improve their inclusive genetic fitness —
Robert 01:07:33
Mm-hmm.
Liron 01:07:33
Well, there’s gonna be a lot of non-zero sum interactions, as long as they’re smart enough, as long as they can form complex social relationships. So humans are going to help each other out — “I’ll pick the bugs off your fur, you pick the bugs off my fur.” And in human society, it’s escalating to all these rich dependencies, net value creation. We all specialize. We all create lots of value for one another. And then love kind of piggybacks on that — it’s a reinforcing cycle where we’ve got social emotions, and it’s all really nice and it’s all moral. So that’s kind of what you’re getting at, right?
Robert 01:08:03
Yeah. I mean, it’s super amazing, the whole damn thing. It’s amazing that fundamental principles of natural selection, like kin selection — which is probably the reason multi-celled life exists in the first place —
in beings like us lead to this feeling of love. Which, again, I think is a complete mystery, because we do not understand what consciousness is for. I just think the whole thing is mind-blowing, and consciousness is the ultimate gift. It’s the reason life has meaning, if you ask me. If you imagine the zombies, who would care what happens to them? What would the word “moral” mean if they weren’t capable of pleasure and pain?
And yeah, I think when you trace the whole process, you not only marvel at the existence of consciousness, which gives life meaning and exists for reasons we don’t understand, but I think you should be impressed. Peter Singer’s book, The Expanding Circle, gets into this. I would say the mechanism he posits is not the only one I would posit. I would emphasize the way technology expands non-zero sum relationships over larger and larger distances.
Peter is interested more in the sheer, almost the logic — once you buy into... And I do think natural selection has, in designing the brain, kind of committed us to, via the dynamic of reciprocal altruism maybe, committed us to accept the idea that if I’m arguing that I deserve something, others deserve the same kind of thing. That may sound paradoxical or ridiculous, because people so often are actually demanding things they don’t wanna give other people. But we do have to come up with these convoluted rationales the more extreme the discrepancy between what we demand and what we grant other people. Anyway, read Peter’s argument. The main thing I would say is he is, in the course of that very slim book, documenting the expansion of the circle of moral consideration. And I think you have to take it seriously as a fact of human history, which is interesting in its own right.
Orthogonality, Power, and the Concentration Problem
Liron 01:10:14
Yeah. So I think I’m on the same page. What you’re saying, I might label this the biological orthogonality thesis being false. So remember, the orthogonality thesis is the claim that you can have arbitrary intelligence paired with arbitrary morality. They’re orthogonal dimensions, so you can have a superintelligence that’s evil, a superintelligence that loves to murder.
I’m actually willing to agree with you that in biology, when you have multiple organisms that necessarily need dependencies, you can’t have a single organism take over the world. It’s just not practical. If you look at a human body, the human body is not going to dictate everything over the world, because the human body is going to get sick, is gonna need to sleep. So the human body needs a support system around it. Physiologically, biologically, you’re probably not going to have a dictator system spreading like a virus. You’re probably going to have mutual social dependencies. And in that sense, I actually agree with you that you’re not going to see this orthogonality thesis. You’re going to see this kind of non-zero sum social evolution, correct?
Robert 01:11:15
Well, I would say that technology, although in the first instance you’re right — no one person per se can rule the world — technology does make it possible for relatively small groups of people to exert great power. So concentrations of power should be avoided, especially in an environment of advanced technology.
And one of the great dilemmas AI poses us with is that on the one hand, it provides arguments in favor of some degree of centralized control. In other words, if you even think, “Yeah, it would be nice if the government vetted the more powerful models before allowing them to be released,” well, that’s a kind of concentration of power over the technology. But at the same time, the technology makes great concentrations of power spooky, because it’s so effective as a tool of surveillance and control and persuasion.
And I think the challenge grows only deeper if you agree with me that we need some degree of international governance to handle this, because AI in so many ways presents threats across borders, so that policy at the national level alone can’t keep us safe. So now you’re talking about kind of global governance in some cases, which makes centralization of power even scarier.
So if what you are leading to is the idea that AI presents new and unique concerns about the location of power, I would say it does it in two senses. The concentration of power in a unified superintelligence that wishes us ill would be bad — and maybe that’s where you’re headed. But I would say it’s also just AI, even if it remains technically a tool of powerful people, that could be bad.
Liron 01:13:22
Yeah, what I’m trying to argue is actually going all the way to God.
Robert 01:13:25
Oh, okay, good.
Liron 01:13:25
I’m actually following the train of logic from your conception of God and what you think the purpose of the world is. I think you have a lot of data, a lot of patterns that you’ve absorbed throughout your intellectual career by looking at non-zero sum biological evolution. And unfortunately, I’m afraid that those patterns don’t generalize to giving somebody an intuition about the universe as God. I think those patterns are going to get undermined the moment that we have a non-biological replicator. I think you’re going to see a lot of patterns that you don’t like that shake your faith in God.
Robert 01:14:00
Well, I didn’t say whether it’s a good God or a bad God. I mean, the hypothesis —
Liron 01:14:05
Okay.
Robert 01:14:05
The hypothesis of purpose — the hypothesis that the whole evolutionary process may be unfolding towards some goal or purpose. And by the way, there’s an argument I had with the late Daniel Dennett about this on YouTube that’s online. People should just watch it and see what they think. I thought he had conceded more than he later said he’d conceded, and I’m not gonna dispute his position now that he’s not around to defend it. But it’s an interesting conversation in any event.
And I probably actually make this point to him because you always have to say — the hypothesis of purpose, although I have emphasized that, interestingly, in light of traditional conceptions of God, yes, it’s interesting that there’s a moral development of a kind coinciding with the evolutionary trajectory in the broadest sense. First of all, evolution itself inflicts huge cruelty and suffering as part of the creative process. We should keep that in mind.
But in any event, the hypothesis of purpose is technically independent of the question of whether we’re talking good God, bad God. Or conceivably, you could be talking about a purpose — and this is what the successionists say, that we’re giving birth to a new form of consciousness, and it will be so rich, and we’ll have so much gratification and pleasure, that even if the light of human consciousness is slowly extinguished, that will be a net utilitarian gain in some very abstract sense. They would argue that.
My line is, look, whatever — can we just arrange things so that we get to stick around and be happy? Even if they’re right that that would be a net utilitarian gain, it doesn’t seem to be a necessary property that human consciousness is extinguished. But anyway, meanwhile, I’m with you in the sense that I say, “Look, let’s slow things down.” It could be that if things get too advanced or if they get too advanced too fast, we are imperiled, and just speaking as a human being, I’d rather that we approach this very deliberately and carefully and perhaps slowly.
Wrap-Up
Liron 01:16:31
Got it. Okay. That’s a good place to wrap on. Let’s finish strong. Are there any other points you want to hit on?
Robert 01:16:39
I don’t think so. Again, I hope people who know a lot about this stuff will at least thumb through the book. I do think there’s some kind of new ideas, provocations for them.
But my main hope is that the many people who are just kind of waking up to the possibility that something big is happening will read the book, which I’ve worked hard to make accessible to a non-technical audience, and see that indeed something powerful is taking shape, and we are not yet giving enough thought and care to what is taking shape or yet exerting enough control on what it’s gonna turn out to be.
I don’t have any firm positions. I say a pause may be in order at some point. It’s kind of policy-light except for noting the nature of some of the policies that are gonna be in order. Some will need to be transnational and so on. I mainly am saying we as a species need to focus, and the nature of some of the policy challenges mean that we need to do it together as a global community because some of the challenges can only be addressed through international coordination.
Liron 01:17:58
Right. Okay, strong message. I definitely agree, and I really appreciate having another sober voice with a nice broad perspective who doesn’t have industry support, doesn’t have any shady lobbies.
Robert 01:18:10
No.
Liron 01:18:10
You’re just independently coming to the conclusion that things are getting crazy, and it would be really prudent to slow down progress. Robert Wright, congrats on the book launch. Everybody buy The God Test, and thank you for coming by here on Doom Debates.
Robert 01:18:21
Well, thanks. I really appreciate it. I really enjoyed the conversation, and I really appreciate what you’re doing with Doom Debates.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
