Dr. Quintin Pope, PhD, is one of the few critics of AI doomerism who is truly fluent in the concepts and arguments. In Oct, 2023 he joined me for a debate in Twitter Spaces where he argued that AI alignment was basically already solved.
His inside view on machine-learning forced me to update my position, but could he knock me off the doom train?
Timestamps
00:00:00 — Cold Open
00:00:43 — Introductions
00:01:22 — Quintin's Opening Statement
00:02:32 — Liron's Opening Statement
00:05:10 — Has RLHF Solved the Alignment Problem?
00:07:52 — AI Capabilities Are Constrained by Training Data
00:10:52 — Defining ASI and Could RLHF Align a Superintelligence?
00:13:13 — Quintin Is More Optimistic Than OpenAI
00:14:16 — What Is ASI in Your Mind?
00:15:57 — AI in 5 Years (2028) & AI Coding Agents
00:19:05 — Continuous or Discontinuous Capability Gains?
00:19:39 — DEBATE: General Intelligence Algorithm in Humans
00:30:02 — The Only Coherent Explanation of Humans Going to the Moon
00:34:01 — Are We "Fully Cooked" as a General Optimizer?
00:35:53 — Common Mistake in Forecasting Superintelligence
00:42:22 — 'Neat' vs 'Scruffy': Will Interpretable Structure Emerge Inside Neural Nets?
00:48:57 — Does This Disagreement Actually Matter for P(Doom)?
00:54:33 — Thought Experiment: Could You Have Predicted a Species Would Go to the Moon?
00:57:26 — The Basin of Attraction for Superintelligence
00:59:35 — Does a Superintelligence Even Exist in Algorithm Space?
01:09:59 — Closing Statements
01:12:40 — Audience Q&A
01:19:35 — Wrap Up
Links
Original Twitter Spaces debate (Quintin Pope vs. Liron Shapira) — https://x.com/i/spaces/1YpJkwOzOqEJj/peek
Quintin Pope on Twitter/X — https://twitter.com/QuintinPope5
Quintin Pope, Alignment Forum profile — https://www.alignmentforum.org/users/quintin-pope
InstructGPT, Wikipedia — https://en.wikipedia.org/wiki/InstructGPT
AIXI, Wikipedia — https://en.wikipedia.org/wiki/AIXI
AlphaZero, Wikipedia — https://en.wikipedia.org/wiki/AlphaZero
MuZero, Wikipedia — https://en.wikipedia.org/wiki/MuZero
DeepMind AlphaZero and MuZero page — https://deepmind.google/research/alphazero-and-muzero/
Midjourney — https://www.midjourney.com/
DALL-E, Wikipedia — https://en.wikipedia.org/wiki/DALL-E
OpenAI Superalignment announcement — https://openai.com/index/introducing-superalignment/
Shard Theory sequence on LessWrong — https://www.lesswrong.com/s/nyEFg3AuJpdAozmoX
“Evolution Provides No Evidence for the Sharp Left Turn” — https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn
“My Objections to ‘We’re All Gonna Die with Eliezer Yudkowsky’” — https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky
“AI is Centralizing by Default; Let’s Not Make It Worse” — https://forum.effectivealtruism.org/posts/zd5inbT4kYKivincm/ai-is-centralizing-by-default-let-s-not-make-it-worse
Singular Learning Theory, Alignment Forum sequence — https://www.alignmentforum.org/s/mqwA5FcL6SrHEQzox
Transcript
Cold Open
Quintin Pope 00:00:00
I honestly think alignment has been basically solved.
Liron Shapira 00:00:04
So you are currently defending something that is more optimistic than what OpenAI themselves would officially endorse.
Quintin 00:00:09
Yeah, I think they’re kind of not updating sufficiently strongly.
Liron 00:00:13
The problem with your approach, it makes me feel like if we were both surveying the planet Earth a million years ago, and you’d be saying, “Look at all this empirical evidence. The fish are trained on swimming. The beavers are trained on building dams. Why are you telling me that suddenly one day a species will go to the moon? Do you know that there’s no air in space?” You know what I’m saying?
Quintin 00:00:32
I would not have made that mistake.
Liron 00:00:32
Okay. How would you not have made that mistake?
Introductions
Liron 00:00:43
Hey, everybody. Welcome to the space. We’re going to chat. We’re going to compare notes, compare our beliefs about what’s going on with AI doom. Seems like an important subject. You’ve been writing about a lot on LessWrong. I’ve been writing a lot on Twitter, and I think the reason people are excited about this debate, and I’m excited, is just because there’s not that much debate going on where both people kind of know a lot of the concepts. So it’s always starting over from ground level.
So how about start by giving an intro of who you are, and then maybe state your main claim and how you think it differs from mine, and then we’ll try to drill down to what the crux is. So Quintin, take it away.
Quintin 00:01:22
Sure thing. So my name is Quintin Pope. I am an alignment researcher and a fifth-year PhD student at Oregon State University. I study natural language processing and AI systems as my PhD subject and also have done a bunch of alignment-related thinking and writing, mostly on LessWrong.
In terms of my position on doom and alignment, I honestly think that alignment has been basically solved for the past couple of years, or rather, I think it probably has been solved for the past couple of years with basically the discovery of RLHF and imitative modeling of human demonstrations. There are, of course, a bunch of unknowns that factor into this, or latent variables in my understanding of how machine learning works that factor into this, and I do acknowledge that there could be settings of those variables that imply we’re in an alignment-is-very-hard world, but to the limit of my ability as a rational person, having thought about this quite a lot, I think we’re probably in an alignment-is-pretty-easy world.
Liron 00:02:32
All right, great. I’ll do my side, and then we can try to find where the crux is.
So my name’s Liron. My background is in startups and tech, computer science. I’ve just been dabbling. I’ve been following LessWrong and AI safety research since back in 2007, so it’s been interesting to watch AI progress toward the Turing test. Feels like burning the last timeline that we have left, from my perspective, over the last 15 years.
And my position is a lot like Eliezer Yudkowsky. He’s my single biggest influence, reading LessWrong. And so sometimes I call myself a stochastic parrot for his writings just because I think that they’re still kind of underappreciated if you go deep, read all the little stuff that he’s put out. So I hope to communicate some of the insights there.
My P(Doom) by 2040, I put it at about 50%. For no reason in particular, should it be 10%? Should it be 90%? I feel like that’s the right ballpark. I think it’s crazy to say it’s only 1%. I think it’s clearly more than that. But I admit that there’s unknown unknowns.
But I don’t really disagree with any of Eliezer Yudkowsky’s claims. I do share the perspective that he’s put out, that all the signs are kind of pointing toward doom. Besides the obvious sign of, well, we don’t want that, but where we actually seem headed — I know you mentioned, hey, we look like we’re in the world where alignment’s easy. So I guess I’m the opposite. It seems like a lot of things are conspiring to make us screwed six ways from Sunday, and we’re so far off from having a hope of survival.
We’ll nail down the crux soon, but the single biggest reason that affects me is probably just the shape of algorithm space. This intuition that AIs generally want to optimize, they want to map goals to actions. And we’re about to wander into the space where they start doing that. We figure out how to make them do that, and then we’re screwed. We do not know how to navigate that space. We are basically just entering — we are summoning the demon, to use Elon Musk’s term. I think he’s spot on. Don’t agree with everything he says, but we absolutely are summoning the demon.
So anyway, that’s my high-level perspective. I’d love to pick at each other’s perspective and try to see — the crux refers to the idea of what could I do to change your mind? What’s the one key belief that, if I change my mind, then I’d get closer to your belief. If you change your mind, you’d get closer to my belief.
And I will say, I actually think you have stuff to teach me. Your posts do have a lot of good thinking. So I actually think it’s very plausible that I’ll make at least a minor update in this conversation. I think it’s unlikely that I’ll make a major update. But if I do make a major update, I think that’s great, because I would love to not think that all our lives are going to get cut short. So I am open to changing my mind. Would you say you’re open to changing your mind?
Quintin 00:05:14
Yeah, I would say that. But to be totally honest, I don’t expect to in this conversation, because I’ve read the sequences, I’ve read Yudkowsky’s back and forth with Richard Ngo, Scott Alexander, a few other people. I’ve read some of the Yudkowsky-Christiano debates. So I do think I’m — I’m not maximally up-to-date with everything Yudkowsky has written, but I’m fairly familiar with his writing, and I don’t find it convincing.
So, if you’re representing yourself as this specter of Eliezer Yudkowsky’s doom model, I don’t think I’ll update much on that. No offense to you.
Liron 00:05:53
Yeah, no problem.
Quintin’s Opening Statement
Liron 00:05:55
All right, great. Let me kick things off then. Let’s try to find the crux. So for me, a good thing to pick at is your claim that we’ve kind of already solved a lot of alignment because we just proved out the RLHF technique that OpenAI has used for GPT-4. That’s one of your big claims, right?
Quintin 00:06:11
My actual position is that if you just sort of look at the trajectory of alignment technique progress versus capabilities technique progress — and they’re arguably the same thing anyway — but if you just look at this trajectory and think about how it’s been going so far, things have been looking pretty good for the alignment techniques. I think GPT-4 is clearly more aligned than, say, InstructGPT-3, the first version of that.
And if you just walk these two trajectories forwards to the point of strong superintelligence, I don’t see any reason to think that capabilities will jump so far forwards that all of alignment will break. So I guess I’d say that alignment has been solved in the way that you might say that the problem of car suspension has been solved if you’re looking at a car from the 1960s or whatever.
If you took the suspension from that car and moved it to a really high-performance car from now, probably things don’t go that well. But if you look at the engineering discipline of building car suspensions for automotive systems, I think it’s entirely reasonable to look at the state of those two engineering disciplines and think, “Yeah, we basically know how to do car suspension, and we’re going to be able to build suspensions for future cars.”
Liron 00:07:31
Okay. So the reason I disagree with that position is because I think that there’s a pretty clear separation between doing alignment when you’re able to give feedback and you’re competent to give feedback, and then you enter a regime that we will in the future where we’re not competent to give feedback — where we can be deceived.
So in the future — today, we look at a statement, and it’s like, “Hey, does this feel like it supports Hitler? Okay, that’s a no-no. We don’t want GPT-4 supporting Hitler.” That’s kind of easy to say, “Bad GPT-4.”
In the future, GPT-5 or GPT-10 is going to come to us and be like, “I got a really great DNA string. Upvote or downvote? Should I put this into an organism? What do you think? Yes or no?” And I’ll be like, “Um, yes?” And it puts it into an organism. The organism is a gene drive, it infects a bunch of people, and it’s like, “Look, harmless.” And I’m like, “Yeah, I guess that’s harmless.” Two days later, everybody drops dead. So we’re going to enter a regime where giving feedback through this process, treating it as a black box where you just give RLHF feedback, is going to stop working at the superintelligent scale.
Liron’s Opening Statement
Quintin 00:08:36
Yeah. So the reason I object to this is because I think you’re relying on a fundamentally leaky abstraction for the word intelligence. My perspective on learning systems and learning processes is that their capabilities very closely hew to the structure and the geometry of the data that they’re fed. And it’s very, very difficult to get a system that’s competent in an area where you don’t actually have the data necessary to isolate competency as a feature of all the possible policies that a system could have in the domain.
So you’re basically, in my view, providing a situation that I don’t really expect to happen. You’re speculating that the competencies of these systems will greatly expand beyond the realms for which we have the data to differentiate between competent and incompetent policies.
Liron 00:09:34
Right.
Quintin 00:09:35
So I’d ask, what process do you think gave rise to GPT being competent at the bioengineering or whatever it is? And then, ultimately, I think that source of competency is going to bottom out in some data distribution that it was trained on. And that data distribution is going to have to have some interpretable to us, or at least partially interpretable to us, labels of what a good action or bad action in this context represents.
So I’ll give a concrete, specific example here. Suppose we’re training an AI to write really compelling, emotionally compelling poetry. There needs to be some sort of data distribution that distinguishes things that are compelling versus non-compelling poetry. And that data needs to have some sort of labels to highlight this capability in capability space.
There are, of course, an enormous variety of little hacks you can do here in order to try and get your limited labeling ability to go as far as possible. But ultimately, this grounds out in a system that was exposed to some data manifold of good and bad poetry, and labels about good and bad poetry, which are interpretable to humans, where the human sees this was labeled as good poetry, this was labeled as bad poetry. Or there was some experimental process where the AI was trying out different poetry and seeing what poetry humans burst into tears upon seeing or something like that.
And so when you talk about this AI’s capabilities vastly exceeding the domains over which we’re able to give feedback, I think you’re sort of implicitly allowing capabilities to generalize beyond what I think is actually possible with learning processes relative to the data that’s available to them. Does that make sense?
Liron 00:11:36
Yeah. All right. So I think this factors into two subclaims that you have. You’re basically saying you’re kind of pushing back on my whole premise of superintelligence anytime soon because you’re saying it’s going to hew close to its data. So we have this fine-grained control over how it becomes superintelligent.
Has RLHF Solved the Alignment Problem?
Liron 00:11:53
Okay, what if we put a pin in that, and what if I just ask you the second part of the question, which is if the AI was superintelligent, just as an ASI — just has 300 IQ, very general — if that did exist, suspend your disbelief for a second, could RLHF solve alignment for that?
Quintin 00:12:12
Let’s see. It depends on what you mean by ASI here.
Liron 00:12:18
Sure. A general optimizer where I can input any goal and it’ll map its way back to an action in the domain of the physical universe.
Quintin 00:12:25
That’s not a thing that I think exists in the way I think you think it exists. And in particular, I think that plausible ML systems that we’ll build in the future and that we will call superintelligence are not like that thing you’re imagining.
Liron 00:12:43
Sure. But this is the part of the argument where you accept that premise, and then we can go back and see if the premise is true.
Quintin 00:12:48
Yeah. The issue is that this is an underspecified premise, in my opinion. You’ve described this system using this collection of words, but I don’t understand exactly what type of system this is supposed to be pointing to.
Liron 00:13:01
Sure. So, if you want to be a little bit more formal, it’s kind of like the outcome pump, or AIXI, except some computable version of it.
Quintin 00:13:09
Okay. So I think this is a really bad model of intelligence, and if you actually tried to build a system that worked this way, it would be either actually not possible in our universe, at least, or massively more difficult than basically just scaling up deep learning.
So one of the very deep objections I have, and differences of opinion I have with a lot of more pessimistic people, is that I think AIXI or even computable versions of AIXI are complete red herrings in terms of trying to understand the fundamental nature of intelligence.
Liron 00:13:50
I got you. And we don’t have to get too deep in the details because we’re still in the early stages here, just trying to find the high-level disagreement.
So, going back a bit. I was asking you — you started by saying you’re optimistic about RLHF, and then I asked you hypothetically, what if we had a superintelligence, would RLHF work on that? And I want to go back to that stage and just point out that OpenAI’s official position is that the RLHF that they have for GPT-4 will not work in the future for superintelligence. So what you’re saying seems to be more optimistic than OpenAI’s stated position. Will you agree to that much?
Quintin 00:14:22
Yeah. I think they’re kind of not updating sufficiently strongly on the evidence that deep learning—
Liron 00:14:30
I just wanted to point that out. So you are currently defending something that is more optimistic than what OpenAI themselves would officially endorse, because they have a superalignment project with the admission that you do need something better than RLHF to align superintelligences.
But it doesn’t matter what they think. Let’s go back to what we think. So I guess we don’t have to worry about the if/then question — if you thought superintelligence existed, then would you think RLHF would work on it — because we’re getting very stuck on the original premise, which I think is earlier on the doom train. You just don’t really imagine that true superintelligence, the way I imagine it, is coming anytime soon, right?
Quintin 00:15:11
I believe that things that are correctly called superintelligences will arise fairly soon. I’m confident that we have a lot of disagreements about what exactly is entailed by the word superintelligence and what those sorts of systems will tend to look like.
And I’d like to move back a bit to answer your question about whether I think current alignment techniques will work on a superintelligence. I think there are superintelligences that they will work on. I think probably our median developmental trajectory is to build such superintelligences that current alignment techniques will not perfectly work on, but let’s say work on well enough to avoid complete extermination of the human race, which I think is your key question of interest.
Liron 00:15:58
Right. Yeah. So I understand that. And so, okay, let’s unpack, because you have said over email you think ASI is possible in five years. So, what do you imagine this ASI being able to do, if not kind of essentially everything? What is ASI in your mind?
Quintin 00:16:15
So basically, it’s a system that knows a lot of really useful things and is able to combine them and apply them in really useful ways. So it’s much more of a functionalist, or I guess you could say pragmaticist view on what an ASI means. The issue is that the term ASI is massively underspecified.
Liron 00:16:42
Sure. Well, just what do you think is going to be the interesting thing that’s going to happen five years from now?
Quintin 00:16:46
Yeah. I think you’ll have conversational agents that when you interact with them, they don’t easily appear to have large deficits in their knowledge of stuff and reasoning over things. I think this range of conversational capability will extend to domains on which there’s a fair amount of data, and those domains will tend to be pretty — a lot of those domains will be quite economically important.
So, for example, coding. Let’s look at coding specifically. I think we’ll have AI systems that know all the useful libraries in the world and are really quite experienced at writing effective code with all of those libraries, and they do it very, very quickly. And so there’s this AI system that’s able to — you could imagine it as sort of the best programmer for every language, but for all of them simultaneously, and also sped up very quickly and very cheap to run. Let’s say something like that.
AI Capabilities Are Constrained by Training Data
Liron 00:17:47
Okay. So let’s say that happens. If I give it a prompt like, “Clone this strawberry,” meaning take a strawberry and make another strawberry that’s identical on the cellular level — every cell that one strawberry has, the other strawberry needs to have a similar cell. It can be different molecularly, but it’s essentially a cellular strawberry clone, which is something that would be incredibly difficult for humans to do using 2023 technology, presumably difficult for humans using 2028 technology. But do you think that the ASI that you think will exist can do that?
Quintin 00:18:22
Probably not the ASI from five years from now. I think that this is quite a difficult problem for either humans or AIs because there’s not a lot of relevant data or theory for how you’d want to do this. It seems pretty unlikely, in my opinion, that you’re going to have this sudden jump in frontier AI capabilities where now it’s able to do the strawberry cloning thing, despite the biotech industry being well behind on strawberry cloning technology or technology that can be repurposed to strawberry cloning.
Liron 00:18:54
Okay. I think I’m getting a sense of where your head is. So I think you are describing AI progress the way a lot of people are going to find intuitive. So I think you’ve got that intuitive appeal. It sounds to me like you’re basically saying, look, the same way GPT-4 today is kind of a chill dude, it’s kind of a buddy, it talks back to you. Doesn’t seem like it’s close to taking over the world. Doesn’t seem like it’s close to making a nuclear weapon or anything like that. It’s very nice. We’re in a good place.
It seems like you intuitively just think that it’s always going to be the case, that it’s just going to be this entity that you can kind of get the answers to test questions from, and it can be as smart as some of the smarter humans when you’re talking about subjects that humans already have familiarity with and trained it on. But it’s not just going to run way outside of the human species and start being kind of in its own league the way that we are to the other apes. Is that kind of what you have in mind?
Quintin 00:19:48
So you alluded to two different things there. One was the alignment property of future AI systems, and the other is about the spikiness of capabilities gains. And I was talking about, in that particular scenario, the spikiness and distribution over capabilities gains for our AI systems versus the rest of the world. And in that context, I do believe that things are going to be a lot more continuous, distributed, and slow-takeoff-y, but also kind of short-timeline-y, is my position.
Defining ASI and Could RLHF Align a Superintelligence?
Liron 00:20:23
Okay. All right. Maybe let’s go over to my position and maybe you can try to poke a hole in mine.
So mine is there’s going to be a discontinuity, which is you can’t just always look to the data that we’re giving these things to train because you’re going to see a discontinuity that’s similar to what happened with humans. Unlike beavers — beavers build dams because they were trained on dams. They got adapted to building dams, and every generation built the dam a little better.
Unlike with beavers, the human space program was not a product of adaptation over the generations. It is an artifact## Reasoning vs. Pattern Matching
Liron 0:45:00
People are going and doing mechanistic interpretability. I think I’m pushing back against your original premise of, look, these LLMs just vacuum up textbooks and then they interpolate their answer to what they saw in the textbooks. I’m saying they’ve got neat structure where they’re actually reasoning. They’re doing things that we can understand that’s not just interpolating from a textbook.
Quintin 0:45:21
I’d say they also vacuum up reasoning and interpolate across reasoning patterns.
Liron 0:45:27
I think that’s where you’re, from my perspective, doing the fudge move — the move that I think is not the best way to think about things. It seems like you’re able to sneak in a lot.
It sounds like any time I call something reasoning, you’re saying, “That’s not reasoning. That’s just pattern matching on how you step from one thing to the next thing.”
Quintin 0:45:44
I think that things that learn reasoning do it by gradient descent, or at least deep learning systems that learn reasoning do it by gradient descent. And then there’s going to be reasoning in the data that drives—
Liron 0:45:55
But the thing is, reasoning is reasoning. There’s a platonic structure. There’s a correct way to reason, a correct way to do science. Maybe there’s a few variations of a correct way, but there’s not that many correct ways. And so to the extent that all of these training methods work, they’re working because they’re building something isomorphic to actual neat reasoning.
Quintin 0:46:12
Okay, but regardless of the number of correct ways to do reasoning, the thing you actually use reasoning for and the things you’re good at reasoning about and the context in which you use different types of reasoning or different reasoning approaches — these are all going to be determined by the data distribution that the model was trained on. And this is the underlying reality I’m referencing whenever—
Liron 0:46:34
Okay, so how about this? Don’t you think that once you got trained on any data distribution, if you’re a sufficiently good reasoner, you can quickly just reason your way into uncharted territory, just like humans did with the space program? We reasoned our way into uncharted territory.
And I think you’re going to say, “Ah, but then we quickly wrote textbooks for one another, and we read each other’s textbooks, so we are still doing the same feedback loop.” Whereas from my perspective, no, we reasoned into outer space, and then we just purely relied on our wits.
Quintin Is More Optimistic Than OpenAI
Quintin 0:47:01
I think that you’re greatly overestimating the degree to which reasoning can allow you to move off the manifold of data from which you were trained.
Liron 0:47:12
It sounds like no matter what insight a human comes up with, you can probably find some data that it’s connected to, but it’s not really. The human just made it up. We can try some examples.
Quintin 0:47:21
Sure. Einstein’s theory of general relativity — he had a bunch of prior context of relevant data about the... Sorry, let’s do special, not general. I know general is more complicated.
Liron 0:47:36
Sure.
Quintin 0:47:36
He had a bunch of prior context about what physical laws tend to look like, what the general pattern of physical laws is, and also about the relativity principle and needing to have the same physical laws in different observer frames.
Liron 0:47:52
Right.
Quintin 0:47:53
And then I will agree, he did a certain amount of reasoning from those, and probably other—
Liron 0:48:00
Yeah. No matter what Einstein did, I feel like he was very stubborn to insist on certain constraints. He’s saying, “These seem like constraints that we should satisfy.” And then he buckled down and did a hard constraint satisfaction problem — finding what mathematical model satisfies these constraints.
Quintin 0:48:15
Yeah. And my perspective is that you actually do need quite a bit of data about what physical laws tend to look like and what the appropriate constraints are in order for you to be set up to perform this sort of reasoning.
Liron 0:48:30
Okay, but you’re saying data, what physical laws look like. I can give you a pamphlet on a bunch of physical intuitions that have been built up over the years that are enough to power a giant chain of different theoretical breakthroughs. There’s not that many fundamental intuitions, like a sense of beauty of what good physics is.
Quintin 0:48:48
I don’t think that process gets you nearly as far as you seem to be thinking it does.
Liron 0:48:53
Yeah. Well, we’re getting closer to the crux. I feel like we’re making progress.
What Is ASI in Your Mind?
Quintin 0:48:57
Are we though? Okay, let’s imagine that we agree you’re right, versus let’s imagine we agree I’m right. Does this actually have much implication for P(Doom)?
Liron 0:49:07
I think it does because I have this mental model of, hey, there’s a search problem where I give you a goal, and you have to give me actions that are likely to yield that goal when you play causality forward. That is the doomy search problem. You discover that, it’s game over. Which I think every day we’re kind of getting the AIs closer to discovering that.
I see a lot of convergence toward that for a number of reasons. And then you’re kind of being like, “No, what are you talking about? That’s not a thing.” It’s all just training and lookup tables and interpolating data. So we’re just trying to reconcile these mental models.
Quintin 0:49:41
Okay, but after this, let’s assume we go maximally to your position. And after this, you’re going to be saying there’s consequentialism in the structure of the world and what it means to be competent in the world. And this means we are doomed because the AIs will use their consequentialism for bad things, and they will do stuff outside of the training.
And then I’m going to say, “No, even in this maximally Liron-esque world, what the AIs use their consequentialism for is going to be determined by the data, just as it is in the context of humans.” And then—
Liron 0:50:17
Well, that’s why I set up the premise. Remember I asked you if I’m right that these kind of optimizers exist, do you really think RLHF is an adequate tool for that? And I pointed out that OpenAI says no. So do you want to accept the premise and talk about whether RLHF still works?
AI in 5 Years (2028) & AI Coding Agents
Quintin 0:50:32
Even in the world where I fully agree with your perspective on the extent to which general reasoning can push capabilities beyond the data manifold, I still don’t think that superintelligences look like the fully general — what did you say — goal-to-action mapping thing.
Liron 0:50:51
Okay.
Quintin 0:50:52
And I think you just end up in a world where AIs are better at generalizing from the data, and then this has, on the one hand, a good alignment implication where they’re better at generalizing from alignment feedback, and a bad alignment implication where they’re better at generalizing to unexpected capabilities.
Liron 0:51:13
Okay, so it sounds like you’d rather just not ever accept my premise and talk about what happens because you think it’s just too crazy of a premise?
Quintin 0:51:20
Probably that’s true, but I’m also still unsure of what exactly you mean by this fully general goal-to-action mapper.
Liron 0:51:31
Right. Yeah, because I’m just wondering, are we ever going to talk about how the world goes if my premise is true? How are we going to align AI if my premise is true? And remember, my premise is goal-to-action mapping, powerful superhuman goal-to-action mapping.
So we can talk about, hey, if I’m right, how does alignment work? Or we can just spend the rest of the conversation — do you think 20 minutes left before we lose steam and let the crowd weigh in? Does that sound good?
Quintin 0:51:54
I want us to spend as much time as possible on how does alignment work.
Liron 0:51:59
Okay. But it’s going to be hard to talk about that when, if you don’t accept my premise, then I agree, alignment seems a whole lot easier. It’s a very load-bearing premise for why I think alignment is hard.
Quintin 0:52:10
All right. Let’s discuss the premise then.
Liron 0:52:13
Okay.
Quintin 0:52:13
Have you been thinking that we’ve been doing this the whole time with the—
Liron 0:52:17
Yeah, I think that we have spent the whole time just talking about digging into my mental model of does it even make sense to talk about a generalized goal-to-action mapper as ever being a thing that gets produced by some process, the way evolution produced what I think is a generalized goal-to-action mapper. The way evolution produces a space program. Are we about to produce something that’s going to go way beyond its data, do its own reasoning? FOOM? I think we were talking about that.
Quintin 0:52:42
Yeah. So, as I previously said, I don’t think evolution’s outcome is at all evidence for basically anything about AI, because we can explain it using principles of learning theory or processes which do not then also predict a similar explosion in capabilities for AI.
From my perspective, our whole discussion was about a relatively unimportant point of how far can you generalize from data manifolds based on the inductive biases or plausible chain-of-thought-esque plausible future AI systems. And my thinking was that it was relatively disconnected from the odds of success for alignment.
And in fact, this is one general objection I have to a lot of doom arguments. They seem to discount what I consider to be key determining factors of the system’s behavior. So, arguments that the AI will behave maliciously which don’t mention the training data of the AI.
Liron 0:53:44
Right. Okay, so there’s a fundamental difference in our approach, which I knew going in, but it’s coming into play here. I’m looking at this huge constraint being the system has gone into the framework of a goal optimizer, and a lot of what it’s doing, it’s doing because it’s picking the right action for a goal.
And then your frame tends to be, well, I have some insights into the internals, and so I know kind of what it tends to lean toward as it works. Is that fair to describe? You’re very much inside the box. I’m very much more properties of the black box — two different viewpoints.
Quintin 0:54:15
Yeah, I’d say I put a lot more weight on empirical evidence regarding how current AI systems and plausible future AI systems work, and then have an inside view of how I expect their dynamics to be. And the alignment that arises or doesn’t from those.
Continuous or Discontinuous Capability Gains?
Liron 0:54:30
The problem with your type of approach — when I hear people like you talk like that, it makes me feel like if we were both surveying the planet Earth a million years ago, we’re looking around, and you’d be saying, “Look at all this empirical evidence. The fish are trained on swimming. The beavers are trained on building dams. Why are you telling me that suddenly one day a species will go to the moon? Do you know that there’s no air in space?” You know what I’m saying?
Quintin 0:54:58
That mistake.
Liron 0:55:00
Okay, how would you not have made that mistake?
Quintin 0:55:00
So there’s this naive empiricism that I think you’re pointing to where you look at points on the graph and sort of draw a line through them and assume the future will continue as the past. And that’s not what I’m doing. What I’m doing is I’m constructing a causal graph of how current AI systems work, constructing a causal graph of how the human brain works, constructing—
Liron 0:55:24
And you’d be constructing a causal graph of how the human brain works too.
Quintin 0:55:29
Yeah. And then I’m back-propagating from the observations about AIs, humans, beavers, whatever, to the latent variables associated with those causal graphs. And then from that, updating those variables with those observations and then constructing a causal graph of the thing I intend to predict.
Liron 0:55:43
So have the conversation with me. It’s a million years ago, and I’m saying, when these brains generalize — the brains that are being built by genes — one day they will generalize, and they won’t just build the thing that their fathers built. They will build a new thing, and it will leave the planet. How would you say, “Yes, you’re right, Liron”? What would you say?
Quintin 0:56:02
I’d say, well, there seems to be this huge inefficiency here in how capabilities-relevant information accumulates across generations and over time, where every generation the beavers die and their additional within-lifetime learning just goes away and is largely not transmitted to their offspring.
And so you have very limited ability to gather capabilities over time because most of the optimization power that their brains exerted across a lifetime went to waste. And if you have an organism which is capable of accumulating capabilities-related information across the generations, then you should expect their rate of capabilities progress to increase massively relative to the capabilities progress that evolution provides.
So this is back to what I was originally explaining in terms of why I don’t particularly update from evolution. The key insight here is that I’m in an epistemic position — or I think I’m in an epistemic position — where I can predict the sharp left turn of human capabilities progress with respect to evolution without having to have a general tendency for such sharp left turns to happen from other types of optimization processes.
And so all the variants of this argument of updating from evolution to AI, I don’t think they work in my epistemic universe.
Liron 0:57:28
Yeah. And by the way, I know that you have a good case, or at least an interesting case, why we’ll take it step-by-step on LLMs, and it’ll never make a sharp left turn, and it’ll never go to the nightmare that I fear. Rather than talking about that line of argument, I’m not that concerned about the exact chain of descent coming down from today’s LLMs.
I’m more interested in the observation that there’s a basin of attraction. Maybe it’ll take one malevolent actor in a basement who wants to kill humanity to seed one modification of open source and go in a different direction. I don’t really care about the next few steps. I care about where we’re going to land inside of algorithm space.
And I want to point out that there’s a big basin of attraction where if we land anywhere near the goal optimizer region — whether it’s GPT-6 landing there just by training or whether it’s one terrorist doing one GitHub check-in — however we get there, it’s a big, dangerous black hole. That’s the claim I want to make.
DEBATE: General Intelligence Algorithm in Humans
Quintin 0:58:20
I don’t believe that claim. Let me rather say I’m not entirely sure of the claim you’re making in terms of how do I mechanistically think about this assertion. But I’m pretty sure if I did know what you were talking about, I wouldn’t believe it.
Liron 0:58:36
Do you know the sense in which I mean it’s an attractor state?
Quintin 0:58:39
Probably not. Let’s see. I’m guessing you’re saying things that are generally capable of self-modifying and changing themselves are going to self-modify to be better at doing stuff and more consistent, more coherent, and more of what you intuitively associate with the words “goal-to-action outcome mapper” — something like—
Liron 0:59:02
Exactly. And then you also see instrumental convergence, where they sweep humanity out of the way. Once you start getting near that region, the reason I call it an attractor state is when you start getting near it, when it’s doing it a little bit, it’ll be very tempting to do it more and more and more, and then FOOM.
Quintin 0:59:19
Yeah. So I don’t believe that entering this thing you’re thinking about actually provides you much in the way of capabilities. And in fact, probably it substantially worsens your capabilities. But I’m not confident on that last one.
Liron 0:59:37
But regardless, let me ask you this. In algorithm space, do you think there is some point that is a superintelligence in my sense, which is you can input a goal to it — very much like AIXI — and it will give you superhuman actions? Actions that can outmaneuver the human race starting from just being inside of a data center? Do you think that point exists?
Quintin 0:59:54
Obviously it does. Probably for whatever you’re imagining, there exists some algorithm in all possible algorithm space that does this. I don’t think we’ll find it without—
Liron 1:00:02
So it’s good to know that you’re not getting off the doom train too early because I wasn’t 100% clear. There’s some people who will say human IQ is pretty close to how smart something can be. There’s no evidence that something can be that much smarter than an individual human. I think this is actually Robin Hanson’s perspective.
Quintin 1:00:17
I don’t know. But I’m maybe not that incredibly far from that position in the sense that, while I think there are clearly things that are much smarter than humans — there can be such things — from a learning process perspective, it’s not clear that there are things that learn that much vastly better than humans.
Liron 1:00:39
Okay, wait. It’s not clear, but you also said it’s obvious that there is a point in algorithm space that’s just a much better optimizer than humans. And in order to be a much better optimizer, presumably you learn as you go what you need to learn.
Quintin 1:00:49
Yeah. So take the example of randomly initialized GPT-4 versus final trained GPT-4. Final trained GPT-4 is much better at learning things than randomly initialized GPT-4.
And so when I talk about learning processes, I’m sort of taking a quasi-blank-slate view on things and being saying, this is a thing that has not yet learned anything that needs to be able to learn anything. But my mental model is that the things you’re referencing don’t actually get dropped into the universe because they’re not—
Liron 1:01:27
Sure. They get dropped on Earth in an Earth data center, let’s say. That’s the starting point.
Quintin 1:01:31
Okay, but I don’t think there’s a way for us to actually create these things using currently available techniques.
Liron 1:01:38
So, okay. This is interesting. If I give you a data center, the right configuration of bits on the SSD drives of that data center would code for a massively superintelligent goal-to-action mapper. But we don’t know how to get to that configuration. We may never, but it’s there.
Quintin 1:01:57
Probably. Or I suppose it depends on the size of the data center, but for current state-of-the-art data centers, probably.
Liron 1:02:03
And then I would say, okay, so the point exists in this bit space, and I would argue it’s also a basin of attraction. So if we can get nearby it — do you at least agree there’s some inward sliding toward it if we can get within the vicinity?
Quintin 1:02:16
Oh, I don’t think you can get into a basin of attraction that leads to that thing. Or let me clarify. I think that it would be very difficult to do so, and you are definitely not going to accidentally find yourself there.
And even if you did find yourself there and slid down the slope, so to speak, I don’t think it would be particularly more capable than alternative approaches — the alternative approaches we’re probably going to use.
Liron 1:02:46
Do you think the LLM training — you said alternative approaches. So in your mind, we’re doing the LLM approach, it’s going to go so far, but I’m guessing you think we need an additional approach, so we’re going to find some other approach, and it’s going to get farther, and eventually we’ll get to superintelligence. Is this just your mainline probability of what’s going to happen?
Quintin 1:03:01
There’s probably approaches that are better than LLMs. I think LLMs do scale to superintelligence. In terms of whether we’ll find those approaches before we hit superintelligence with LLMs, I think it’s somewhat more likely than not, but I’m uncertain.
Liron 1:03:19
And superintelligence, even by my definition, where it’s not really a matter of reading more textbooks and interpolating more — it’s really kind of generalized where it’s, “Okay, I understand the universe. Give me a goal, I will reason what I need to reason.” That’s what you think we may get to with LLMs?
Quintin 1:03:33
No. As we discussed previously, I think it’s really hard to generalize from data. And to be clear, when I said that there could be an AI that would be very bad — a take-over-the-world goal-to-action mapper thing — I’m still not imagining it as having the degree of generalization off the data manifold that I think you imagine.
Liron 1:03:57
Would it have a superhuman level? Could it generalize better than Einstein can generalize?
Quintin 1:04:02
Clearly, but as I’ve said previously, I don’t think Einstein generalized as far as you seem to think. And also keep in mind, Einstein is basically the most impressive example of generalizing off data manifolds that at least either of us easily knows. And I think most intellectually relevant, economically relevant intellectual work involves way less generalizing than Einstein demonstrated.
The Only Coherent Explanation of Humans Going to the Moon
Liron 1:04:29
Okay. I do want to get to a closing statement. Let me just throw one thing at you before I get to my closing statement, which is going back to Neat versus Scruffy and chess AIs.
I think it’s very interesting because, again, my mental model is you have these optimizers. Yes, chess was an optimizer, but it was so limited by its domain that you can just kick it over. You can shoot it with a gun. And so it just didn’t do the job of being a physical world optimizer because it was too narrow domain.
Okay, but the trend I’m seeing is that you have these optimizers whose domain is growing, and they’re happening right in front of our eyes. After AlphaGo and Stockfish, you got AlphaZero, which trained on chess without even looking at the human dataset, which created its own dataset. And then you had MuZero, which didn’t even look at the rules of the game when it started training.
And similarly, in terms of broad domain generalization, we’re seeing now that Midjourney seems to be getting outcompeted by DALL-E, which is taking advantage of the multimodal, where it’s understanding simultaneously the language and the images. So it seems like we’re seeing a domain broadening while keeping the optimization power high, and that is a scary trend to me. That trend seems to be going somewhere — it seems to be generalizing soon, the domain will expand to be the whole universe.
So what do you think about that trend? And what does it say about maybe this is how the future of our current AI trajectory is going, where you’re seeing optimization over a larger and larger domain?
Quintin 1:05:58
As I said previously, with deep learning systems, you have their data available, so you can just look at their data, and they are generative models over that data. And they’re more controllable than the sorts of argmaxers that you seem to be anchoring your notion of superintelligence to, because they’re—
Liron 1:06:17
Well, I think you’re being very quick to look at the internals. I encourage you to just look at what’s happening in terms of the input-output relationship they’re able to achieve.
Quintin 1:06:26
Yes, AI progress is happening.
Liron 1:06:31
One thing is—
Quintin 1:06:32
You keep making references to optimization as this broad, scary umbrella term. But whenever I think about systems on a mechanistic level of detail, they don’t seem that scary to me. I think this is the general pattern of our discussion.
Liron 1:06:48
Yeah, that is the general pattern. Absolutely. Because it’s easy to — I see what you’re doing. You look at the chess optimizer and you’re saying, it’s not just that the domain is narrow, it’s that the algorithm inside is not that scary because it’s just argmax.
Quintin 1:07:03
No, it’s scary because it is argmax. If you wanted a general agent that did things, that planned things in the real world that was structured in the same way as a chess agent, you would need a criterion, some classification criterion that so precisely differentiated between good and bad trajectories that it could consider ten to the twenty or whatever enormous number of causal paths through physical reality of accomplishing things.
And this classification criterion would have to perfectly differentiate between those paths that helped humans versus those paths that exploited the—
Liron 1:07:41
Yeah, okay. Sure, I get that in theory. But I want you to look at the actual trend we’re seeing, which is now we have MuZero that’s able to play multiple board games. What does that mean to you?
Quintin 1:07:48
It means we’re transitioning away from this classifier, argmax-driven generation of policies to a more amortized imitation of known good trajectories, which I think is much safer because the base objective for that—
Liron 1:08:06
You’re calling it an imitation of known good trajectories, but this one doesn’t have training data.
Quintin 1:08:11
It definitely does. MuZero has self-play. Look at how self-play—
Liron 1:08:16
Okay. Self-play is training data?
Quintin 1:08:18
Of course it is. You know how the RL update equations work, right? You have some trajectories, you score each trajectory with a reward value, and then you update the policy to more closely imitate the higher-scoring trajectories and be more or less likely to generate the lower-scoring trajectories. It is imitating its past successes.
Liron 1:08:43
I see why you can imagine self-play is going to be a type of interpolation. I don’t really see it just because the amount of self-play is still minuscule compared to the scenarios it finds itself in at playtime.
Quintin 1:08:55
I mean, it plays what, 30 million games?
Liron 1:08:59
Okay. I don’t know the figure, but it’s certainly some food for thought just looking at MuZero. But let me continue. Then you look at LLMs, and now suddenly you can ask it an English question about chess or ask it a bunch of other English questions. The domain of the language is suddenly very broad, and the optimization still manages to be superhuman in some cases, subhuman in some cases.
Do you perhaps see the same trend I’m seeing where it’s like, uh-oh, domain seems to be getting bigger?
Quintin 1:09:24
I don’t see how this wouldn’t occur in any situation where we’re making non-zero AI progress. And in my mind, the trend that matters for alignment is this trend of using more and more amortized imitation of known good trajectories as the base driver of capabilities progress. And I think that’s a good trend.
And it’s the thing that I think undercuts the sorts of goal-to-action mapper intuitions you bundle in with your conception of a scary agent.
Are We “Fully Cooked” as a General Optimizer?
Liron 1:09:59
Okay, let’s do closing statements. My closing statement is I just want to identify the crux, which is what would it take to really shift me over to more of the Quintin view?
I feel like I still have a hang-up. I still just think that what the human brain is doing is going to get surpassed. Whatever it is that we’re doing that let us really murder the other animals, have our way with the other animals, go to the moon, this unprecedented stuff — I feel like there’s a lot more of that stuff coming.
The reason we don’t have nanotech seems like just an artifact of how slow we’ve been going because we’re running a piece of meat that’s 12 watts. So it just seems like there’s all this headroom that’s about to be entered into that it’s hard for us to get a handle on.
But I guess maybe I understand your position is, well, maybe there’s a bunch of reasons to think that getting a handle on it might work out to be kind of easy and natural. Sure, I guess it helps me think a little bit more clearly.
Imagine a scenario where we don’t die because this path opens up of one step at a time where somehow we get into this much more difficult space. I’m trying to update here. I think I kind of see it. I guess the thing that would convince me the most is to understand how optimization capabilities aren’t just generalizing and going off to the races.
Because I wasn’t really convinced when we talked about going to the Moon and you were saying, “Oh, well, the human is just reading textbooks.” I guess that would be the weakest part for me. All right, so that’s my closing statement — how to make me update. Give me your closing statement.
Quintin 1:11:34
Let’s see. I’m not entirely sure what to say as a closing statement here. I guess I do agree with your assessment of the relatively weakest portions of my argument, though I still believe mostly the same position as I did.
I do think there’s some mysterious stuff going on in terms of figuring out exactly how far one is able to generalize from training data via using logical reasoning.
Maybe I might ask you whether or not you consider self-play in the context of LLMs to be self-modification, or entering into an attractor that doesn’t end up at your notion of a scary goal optimizer thing.
Liron 1:12:16
Interesting. Yeah. I don’t know that I have a ready-made answer. Sweet. Okay, great. Well, thanks very much. I feel like that was a quality discussion. Definitely got me thinking, and I’ll keep sleeping on some of these points.
I think it’s fair to say that I’ve slightly updated because you’ve made some scenarios concrete. You’ve made it a little less impossible to imagine a hypothetical future where we don’t die. So thanks for that. All right, so let’s open the floor. If you guys want to come talk, ask your question or comment. First person, Tyler. Go for it, Tyler.
Common Mistake in Forecasting Superintelligence
Tyler 1:12:47
Yeah, Liron and Quintin, great discussion. I wanted to hear from you, Liron. Imagine that we do get this aligned AGI in the future. So step one of the problem is solved. How do we prevent this aligned AGI from creating an unaligned AGI?
So you get your perfect future, Liron, but what’s the next steps after we actually solve this problem? Because it seems like there’s a bigger societal issue of, okay, well, we’ve got to make sure no one creates an unaligned AGI ever. And this seems like a more political, harder problem to solve. If you had your perfect world, what would that end up as?
Liron 1:13:27
Yeah, sure. I’ll take a stab at that. One model of the perfect world, which I don’t think we’re ever going to get to because I think we’re going to crash and burn long before we ever have a hope of this, is the AI that’s able to just reason through what everybody wants, find a good equilibrium, and is kind of — I guess it would be the AI central government.
Now, before you say, “Oh, Liron wants a central government,” I haven’t even thought this through. I think it is a logically possible solution. I don’t think it’s feasible. I think we’re so far from even thinking about that. I usually just spend my time thinking of how overdetermined our doom is. So I don’t think I have a super intelligent answer to give you there. What do you think, Quintin?
Quintin 1:14:02
I don’t think that misalignment gives you superpowers. I think that compute gives you power insofar as AIs go. This is surprisingly more true for AIs than it is for humans because of adversarial examples.
If you have a million times more compute than your opponent, and you’re also a thousand times better at doing AI research than your opponent, then your opponent is basically fucked, because you can generate adversaries that just make your opponent go completely insane.
And so these scenarios of there’s this individual bad actor who has a relatively small amount of compute and who decides to make a misaligned AI — I think they just get completely steamrolled by the majority of people who don’t want to die of misaligned AIs and have devoted millions of times more compute and power to the aligned AIs to prevent that.
Liron 1:14:57
Wow. So Quintin is not only imagining a nice path forward to AI that doesn’t kill us, but also a resulting equilibrium where we remain not killed.
Quintin 1:15:05
Well, my P(Doom) is 4%, so yes, both of those things need to be imagined.
I think that the default outcome of AI technology is to promote centralization of power. And it isn’t even necessarily so much that a handful of evil capitalists or dictators or whatever get control of the AI, so much as AIs let bureaucracy expand a whole lot.
I think we’re going to soon discover that one of the things keeping bureaucracy from being very intrusive into our lives has just been how inconvenient it is to monitor and control a bunch of people and fill out forms and run forms and do compliance. But LLMs are really great at giving feedback over things and writing forms and writing new laws and assessing whether you’re behaving in accordance with the law, and so on and so forth.
And one of my concerns is that this is going to enable a much greater degree of intrusiveness into our lives than previous equilibriums of what has been convenient for bureaucrats to do has allowed.
I wrote a post as part of the AI pause debate, basically making the case that AIs are likely to centralize things by default or promote centralization by default. And we should tune our regulatory intervention ideas towards things that split the hair towards less centralization or move on the margin in that direction and be aware of this as a threat model. The name of that post is “AI Development is Centralizing by Default. Let’s Not Make It Worse.”
Liron 1:16:46
Nice. All right, Sichu.
Sichu 1:16:48
Hello. Yeah, so I have a question for Quintin. My question being that, in this argument, basically you’ve denied there is a possibility of some sort of phase transition in the behavior of these models. So why do you think so? Because in mechanistic interpretability work, we’ve already seen the existence of some level of phase transition.
Quintin 1:17:13
I think phase transitions are most significant in the very small models, such as grokking, where you see them have massive differences in model behavior, and they seem much less significant in larger models.
I actually do think that RLHF is surprisingly robust to internal phase transitions in your models. So my answer would be the accounting of models as undergoing phase transitions as provided by, say, mechanistic interpretability or the singular learning theory people doesn’t seem to magnify into massive catastrophic variations in the applicability of alignment techniques to those models.
And the current trend is bigger models, less relative change due to phase transitions. So I don’t see a reason to think this should change when we get to even bigger models.
Finally, this is a point I didn’t get to in our previous discussion, but you don’t actually need to be as smart as the model in order to provide oversight to the model. In fact, in the human brain, there’s a thing called the learning system and a thing called the steering system. And what the steering system does is it provides rewards associated with different sorts of actions.
And this process, despite the fact that the steering system is much stupider than the learning system, is actually the process by which human values arise. So it’s not perfectly the case that the learning system does exactly what the steering system, quote-unquote, “wants” or would reward highly. But the degree of alignment between things that you find rewarding versus things you actually do seems pretty good to me.
Liron 1:19:04
It’s unclear whether that would scale to the superintelligence level, because I think the reason why it’s able to even chime in and give feedback is because when the action-picker part, the smart part, goes off and does its thing, it usually doesn’t do this irreversible FOOM. Usually there’s a chance for the reinforcement part to weigh in again. So that might not necessarily be the case.
Quintin 1:19:26
Yeah. We didn’t get into this at all, and we probably shouldn’t now, but I don’t think FOOM is going to be a thing.
Liron 1:19:34
Okay.
‘Neat’ vs ‘Scruffy’: Will Interpretable Structure Emerge Inside Neural Nets?
Liron 1:19:35
All right, guys. Cool. We’ll call it here. Just want to thank Quintin. It was an awesome debate. Gave everybody a lot of food for thought, including myself. I think it’s such an important topic, and I hope to hear more thoughtful people debating AI doom.
Quintin 1:19:50
Thanks so much for having me. It was a very interesting discussion.
Liron 1:19:53
All right. Thanks, everybody. Have a good one.
Quintin 1:19:53
Yeah. All right. Have a good one as well.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
