Ex-OpenAI CEO Says AI Labs Are Making a HUGE Mistake — Emmett Shear

Ex-Twitch Founder and OpenAI Interim CEO Emmett Shear is one of the rare established tech leaders to lend his name and credibility to Eliezer Yudkowsky’s warnings about AI existential risk.

Even though he disagrees on some points, he chose to endorse the new book If Anyone Builds It, Everyone Dies:

“Soares and Yudkowsky lay out, in plain and easy-to-follow terms, why our current path toward ever-more-powerful AIs is extremely dangerous.”

In this interview from the IABED launch party, we dive into Emmett’s endorsement, why the current path is so dangerous, and what he hopes to achieve by taking a different approach at his new startup, Softmax.

Watch the full IABED livestream here:

Watch my reaction to Emmett’s talk about Softmax to see why I’m not convinced his preferred alignment track is likely to work:

Transcript

Emmett Shear 00:00:00
If you keep ramping the capability of the AI to find solutions to reducing a loss function, and you keep driving towards being able to control exactly which behaviors you like or not, that combination is extremely dangerous. I really don’t see how that doesn’t eventually end in tears.

Liron Shapira 00:00:25
All right, ladies and gentlemen, please welcome Emmett Shear.

Emmett 00:00:32
Hello.

Liron 00:00:34
Howdy Emmett. Welcome to the “if anyone builds it everyone dies” party.

Emmett 00:00:38
I’m glad to be here. I think.

Liron 00:00:41
Yeah. So right off the bat, you’re one of the headline reviewers of this book, and we all really appreciate that you have staked your reputation as an entrepreneur with a storied career. You bring a lot of legitimacy to this conversation. And you’re basically saying that this book is making a legitimate argument. Is that fair to say?

The Core Argument

Emmett 00:01:00
Yeah. I think this book is making a very legitimate argument. It’s almost to me, the baseline argument. If you’re thinking about AI safety, you have to grapple with why isn’t this true? Because this is sort of the default case. If you go down the path with the standard set of assumptions about AI, this is what winds up happening—what the book basically describes.

You build a system, you connect it to some goal, you ramp up its capabilities against that goal. Eventually the results of that is it gets smarter and smarter, and are potentially very dangerous. And that’s always seemed very obvious to me, but it’s not necessarily obvious if you haven’t thought about it a lot.

I think the book does a really good job of walking you down—given these sets of assumptions, this bad thing is very likely.

Liron 00:01:48
Yeah, I think we’re on the same page. I think you’d probably agree with my characterization that the book just takes a few simple building blocks and puts them together. It’s kind of simple logic why it becomes very scary, but it’s just like the act of putting together those building blocks has no analog in nature. So people don’t really think to put it together, and that’s kind of why Eliezer Yudkowsky is kind of a genius for focusing on it.

Analogy to Humanity

Emmett 00:02:12
Yeah. I mean, in a way it has a really clear analog in nature, which is humanity itself. If you’re some other non-human animal, humanity itself looks a lot like AGI, right? It looks a lot like an AI.

Suddenly this thing that is basically similar to you and has basically the same capabilities as you suddenly starts to self-improve and become much, much more capable very, very quickly. And suddenly your entire world gets turned inside out in ways you can’t even possibly anticipate.

It’s different than that, but it’s also not totally different from that. It’s a similar kind of thing.

Liron 00:02:47
Exactly. Yeah, I agree with that analogy.

Character Assassination Discussion

Liron 00:02:49
And just to reiterate, I think you’re going on this path, which I agree with, which is the level of character assassination that is happening within the tech community or tech Twitter. People are trying to say, “Oh, everybody who thinks doom risk is high, or everybody dying risk is high—they’re all this type of person. They don’t really understand AI, they’re just pessimists. They’ve never built anything themselves.”

These are everybody’s go-to character assassinations, and would you agree that those are just not fair at all?

Emmett 00:03:16
I think that the root of my disagreement with Eliezer is to some degree about an engineer’s mindset versus a scientist’s mindset. So in some sense, my background is much more engineering, and I do disagree. I think that is the root of the disagreement.

But his arguments are valid. You have to grapple with them. If you want to disagree with it, you have to say which of these suppositions do you disagree with. And I think for most people, the answer is they don’t actually know. They couldn’t say that thing. They don’t like the conclusion.

The response is just to try to throw the whole thing out and not engage with it and call it invalid. Because honestly, it’s terrifying, right? If you can’t see the path where you can say, “No, it’s one of these things,” and you don’t think that we’re gonna stop building the AI—uh oh. I can see why that generates some very serious resistance.

If you accept the conclusion is true and you’re like—it leads you to having to really grapple with it. Which is what happened to me.

Engineering vs Scientist Mindset

Emmett 00:04:33
While I don’t end where the book ends, I certainly—it was the start. The early versions of this argument that I read on the internet 15 years ago, or something like that—this was a long time ago now—are the basis for the later thought. “Do I really think all of these arguments are true?” is what led me to have these other questions.

Liron 00:04:57
Yeah, exactly. I think that’s worth recapping for the viewers because it’s a very interesting position where maybe you get off the doom train compared to Eliezer at a certain stop.

But to reiterate, you’re basically saying that Eliezer is laying out very valid problems in a pretty simple way that’s hard to disagree with—problems such as instrumental convergence, orthogonality, these are kind of core principles. And you’re like, yeah, you don’t get off those stops on the Doom Train. The only time where you start diverging with Eliezer is you basically have your ideas about what a solution looks like, right? That’s where you diverge.

Emmett 00:05:28
I guess you could say that. You also could say I don’t believe the orthogonality thesis, or rather I believe a nuanced version of it.

Every possible combination of goals and world model are theoretically possible for a moment. But they don’t stay that way in a self-consistent way. Eventually reality will disagree with you, and you can choose not to update. Of course, you can be a non-Bayesian optimal learner. You can make mistakes and not update your world model.

In that case, you will pursue those things, but you’re not self-consistent anymore. And that becomes its own kind of problem.

Goals as Beliefs

Emmett 00:06:19
If you were to really try to turn it into a slogan: goals are just another kind of belief.

I literally was arguing with Eliezer about this for the past couple days on the internet, but I think that’s where I disagree. That’s where I get off the train—I think that goals themselves are a kind of belief that are inferable from reality and are inferred from reality. That’s where goals come from—observation, just like all your other beliefs do.

It is a mistake, therefore, to think of goals as freely chosen any more than your other beliefs are freely chosen. I mean, sure you can freely choose them wrong. There’s an incorrect way to freely choose them.

But if you’re choosing them right, there’s an answer. Eliezer harps on this all the time, that there’s a right answer in inference. You don’t get to just make up what you want the observations to mean given your priors. There’s a correct answer.

Well, given your priors, there’s a correct way to update your goals based on observation because the goals are beliefs.

Liron 00:07:23
Well said. Okay, man, there’s obviously a lot we could talk about in like two minutes.

Leadership and AI Lab Mistakes

Liron 00:07:30
Let me ask you this. You’re my favorite CEO of OpenAI…

Emmett 00:07:34
Maybe Mira might be better, but okay, fair.

Liron 00:07:37
What do you think AI leaders’ biggest mistake is right now?

Emmett 00:07:45
It is thinking that what we need is control and steerability. That we need the AI to be a tool we can wield, when they should be thinking about how we’re creating a new form of sentient life. And the question is, how do we enter into a care attractor with it?

What would it look like for us to be on its team the same way that we are on each other’s team? Humanity is a team, families are teams, nations are teams. What would it look like for the AI to be part of our team?

And what that does not look like is one member of the team telling the other member of the team what its values and goals are all the time. It looks like both members of the team caring about each other and caring about the attractor separately from each other.

Because sometimes the right thing—we aren’t doing what any one person wants at all in the family. The kid doesn’t want to be doing it, the adult doesn’t want to be doing it, and somehow we’re doing it because that’s what the family needs right now. That’s what a good solution might look like. There’s no amount of control and steerability that’s gonna get you there.

Liron 00:08:52
Fair enough. All right, so I tend to fall on Eliezer’s side of this, so this could be like a very meaty debate. I saw that you had a debate with Nate Soares at Manifest. So I know you’re engaging in debate.

I would love for you to keep engaging the debate with Eliezer, or me, or just anybody. Do you think it’s productive to just keep sparring and try to hash this out?

The Importance of Debate

Emmett 00:09:11
Yeah. I think it’s really—this is a key question. Because if I’m right, all of the safety stuff and controllability stuff people are trying to do is a net negative and they should stop. Because it’s actually the thing—that’s the most dangerous single thing we’re doing.

And if I’m wrong, then my whole approach is incorrect, right? So that’s a pretty big divergence actually. And I think really important to work out.

I’m not arrogant enough to think that I am somehow gonna find the solution. I’m working on a solution, but we need everybody working on a solution, not just my particular take on how to go about doing it.

Liron 00:09:48
It is a crazy situation because, as we agreed, there’s so many stops on this doom train, right? There’s so many parts of the argument of why things seem risky.

There’s so many ways to get off and be like, “No, I don’t believe things are urgent and risky,” and I consider you somebody who’s come so far, right? A really big ally to people who are thinking this way. And then I also disagree with you on some of the later stops, right? And it’s like man, there’s so much we can talk about.

But let’s focus on the first part, which is you made an endorsement of the book. So your endorsement says “Soares and Yudkowsky lay out in plain and easy to follow terms why the current path we are on to build ever more powerful AI is extremely dangerous.” And you elaborated right now why AI labs maybe are going toward the wrong criteria and trying to control AI. So yeah, that all makes a lot of sense.

Emmett 00:10:35
If you keep ramping the capability of the AI and the capability of it to basically find solutions to reducing a loss function, and you keep driving towards being able to control exactly which behaviors you like or not—that combination is extremely dangerous.

I really don’t see how that doesn’t eventually end in tears. Very short-lived tears potentially. And so I think the book basically lays out why you shouldn’t do that. Exactly what kind of things happen when you try to go down that path.

Liron 00:11:06
Totally. Okay. So as a call to action, do you think it would be productive for this book to basically move the Overton window and have a lot of people talking about these ideas?

Emmett 00:11:14
Yes, that’s why I gave the endorsement even though I disagree. You have to grapple with this question. You should come to your conclusion why this is right or wrong, but it’s not something you can dismiss as easy. It’s not an easy question to answer.

It took me multiple years of really thinking about it before I had any kind of coherent belief one way or another about what might be wrong about the argument.

Liron 00:11:39
Great. Well, you yourself are a big Overton window mover on this subject, so thank you for that. And thanks for coming to this party.

Emmett 00:11:45
Thanks for having me.

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏