Dr. Ben Goertzel is a pioneering AI researcher, entrepreneur, and author who has helped organize AGI as a research field since the 2000s through books, workshops, and conferences.
We talk about Ben’s early days of AGI research with Eliezer Yudkowsky, his timeline to the singularity, and what makes him so optimistic that superintelligence will be beneficial for humanity.
Timestamps
00:00:00 — Cold Open
00:00:59 — Introducing Dr. Ben Goertzel
00:04:11 — How Did Ben Get Interested in AGI?
00:08:26 — Early AGI Enthusiast Community: Extropians & SL4
00:15:15 — Is There Headroom Above Human Intelligence?
00:19:53 — Can Superintelligent AI Kill Everyone?
00:28:25 — Ben’s AGI Timeline: 2029
00:36:00 — What Modern AI Is Missing
00:42:01 — AGI to Superintelligence — FOOM in a Matter of Months
00:46:34 — What’s Your P(Doom)?™
00:53:37 — Ben’s Case for AI Acceleration
01:03:50 — The Orthogonality Thesis Is “Dishonest”
01:13:43 — Liron: Moore’s Law for Goal Achievement
01:23:50 — Ben: Alignment Is Just Good Parenting
01:27:36 — Alignment in Practice: OmegaCLAW
01:35:47 — Creating Stable Goals Under Self-Modification
01:45:22 — SingularityNET, OpenBGI, and the ASI Alliance
01:57:31 — The $ASI Token Economy
02:01:15 — Closing
Links
Ben Goertzel on X — https://x.com/bengoertzel
Ben’s Substack (Eurykosmotron) —
SingularityNET — https://singularitynet.io/
OpenBGI — https://openbgi.com/
Hyperon AGI Framework — https://hyperon.dev/
OmegaCLAW on GitHub — https://github.com/asi-alliance/OmegaClaw-Core
ASI Alliance — https://superintelligence.io/
The Consciousness Explosion (book) — https://theconsciousnessexplosion.ai/
“The Singularity Institute’s Scary Idea (and Why I Don’t Buy It)” (2010) — http://multiverseaccordingtoben.blogspot.com/2010/10/singularity-institutes-scary-idea-and.html
“Why ‘Everyone Dies’ Gets AGI All Wrong” —
Gerald Feinberg, “The Prometheus Project” (1968) — https://archive.org/details/prometheusprojec0000gera
Gary Marcus vs. Liron Shapira — AI Doom Debate —
Vitalik Buterin — Will “d/acc” Protect Humanity? —
Ken Stanley Debate — https://lironshapira.substack.com/p/debate-with-a-former-openai-research
Transcript
Cold Open
Ben Goertzel 00:00:00
It could well be that a superintelligence will, with high probability, converge on this sort of beneficial, compassionate value system, even if it starts off kind of nasty, and that might be the case. I’ve tried to convince myself it’s the case and haven’t quite managed to yet.
Liron Shapira 00:00:18
I feel like that means that you agree with me that the orthogonality thesis is at least somewhat true, the idea that—
Ben 00:00:24
No, I think it’s a very dishonest and misleading idea, actually.
Liron 00:00:31
The problem is that we live on the same game board, and if I achieve my goal, then that could logically imply that you don’t achieve your goal, and we get a conflict. What are your thoughts?
Ben 00:00:38
I think it’s a very childish and irrelevant way to look at the universe.
Liron 00:00:45
So are you ready for the most important question of Doom Debate?
Ben 00:00:49
Let’s go.
Introducing Dr. Ben Goertzel
Liron 00:00:59
Welcome to Doom Debates. My guest today is Dr. Ben Goertzel, a pioneering artificial intelligence researcher, entrepreneur, and author. He helped organize AGI as a research field in the 2000s through books, workshops, and conferences. He now leads SingularityNet, the OpenCog Foundation, and the AGI Society, which runs the annual Artificial General Intelligence Conference.
Ben is generally optimistic about AGI. He argues that it’s more likely to benefit humanity if it’s developed in an open, decentralized way, rather than controlled by a small number of corporations, governments, or military institutions.
Interestingly, though, some of his early AGI work was sponsored by the Singularity Institute, the organization that later became MIRI. But he strongly disagrees with Eliezer Yudkowsky’s more pessimistic view of AGI risk, including the core message of Eliezer and Nate Soares’ recent book, If Anyone Builds It, Everyone Dies.
For Ben, the debate over beneficial AGI has spanned more than two decades. I’m excited that he’s joined me on Doom Debates today to hash it all out. Dr. Ben Goertzel, welcome to Doom Debates.
Ben 00:02:06
Hey, great to be here.
Liron 00:02:08
All right, so you’ve got a long, wide-ranging career. Let’s get the overview here. You’ve worked in mathematics, cognitive science, bioinformatics, robotics, AGI. What has been the guiding aim of your research, and how has it evolved over time?
Ben 00:02:21
The guiding aim of my research began as just a curious guy trying to understand how the universe functions, spanning the external universe and the internal universe and every other kind of universe or multiverse you could imagine.
And then as I progressed in my career, I became more convinced of the radical good that AGI technology could do and became somewhat convinced that developing beneficial superintelligence may be the only way to stop our species from annihilating itself through various forms of stupidity and bad emotion and social organization gone awry.
So I would say over the course of my career, my motivations have drifted a bit from just pure curiosity of trying to understand everything, enjoying tinkering with stuff, to “holy shit, if we can’t make beneficial general intelligence soon, our species may meet an unpleasant end.” On the other hand, if we can make it soon, we can cure aging and death and scarcity and open all these new horizons.
So I’m still very curious and want to understand the universe, but the humanitarian and transhumanitarian motivation has become more prominent as I’ve advanced in age, I suppose.
Liron 00:04:06
How did you first get into AI research or the concept of AGI?
How Did Ben Get Interested in AGI?
Ben 00:04:11
I first encountered the notion of smart machines in science fiction like most people in my generation. The original Star Trek and Space 1999, and then Isaac Asimov’s novels and so on and so forth.
And then sometime around seventy-three, seventy-four, I found a book called The Prometheus Project in a used bookstore in Haddonfield, New Jersey. This was written by a Princeton physicist named Gerald Feinberg, and what he said in that book is, within a few decades, we’ll get machines smarter than people. We’ll be able to make nano machines that will build whatever we want, and we’ll be able to cure aging and involuntary death.
And the question will then be, toward what end do we put this? Toward wild consciousness expansion or pointless rampant consumerism? And then the other question is, how do we decide this? He thought the UN should roll out digital computers all around the world to enable people to vote on what purpose all this amazing new technology should be put.
So I read this nonfiction book I bought for a dollar or something in a used bookstore in suburban Jersey, and it made a lot of sense to me. I followed his arguments. It was basically The Singularity Is Near written in a little more serious academic tone. The book had been written, I think in ‘68. I encountered it in the early seventies. So that planted a seed in my mind because that wasn’t science fiction — it was a physics prof out of Princeton.
Then a few years after that, it was probably 1980, I found Douglas Hofstadter’s book, Gödel, Escher, Bach.
Liron 00:06:07
Yeah.
Ben 00:06:08
That book rolled through a lot of the thinking that was there in the AI and cognitive science field at that time — the mind is a collection of patterns and the brain is a complex self-organizing system. And if you can solve pattern recognition and abstraction and analogy, then maybe you can build a mind.
So that gave me a clue of what at least some people in the AI field were working on. I was probably in ninth grade, first year of high school, when I read that book.
Ben 00:06:41
That was right around the time I bought my first generally programmable computer. In the ‘70s, I built electronics kit computers that you had to program in hexadecimal machine code. But then I got an Atari 400 computer you could program in assembly language, FORTH, and BASIC. And I started trying to code AI systems, and with a machine with 16K of RAM, it was very challenging.
Liron 00:07:11
Yeah, 16K is not enough.
Ben 00:07:13
Yeah, you could do more tricks. You could override the tape drive buffer to get a little extra RAM with assembly language. But yeah, it was a far cry from what you have in a cheap mobile phone today.
But it did bring me into a collision course with how difficult AI actually is. Because right away you start coding stuff that seems like it should work and it doesn’t work at all. And then you see what actually is the real challenge there — the way we think we’re solving problems isn’t how we’re actually solving problems. And then you come to grips with the fact that we don’t really understand how thinking works well enough to emulate it, which sort of set me on the course of my career.
But at that time, the AI field was very boring. It was just locked up in rule-based expert systems and various dead-end approaches. So I ended up doing my bachelor’s and PhD in mathematics just because I figured math is not going to go out of style. It’s going to be useful for anything I’m going to want to pursue in life.
Early AGI Enthusiast Community: Extropians & SL4
Liron 00:08:29
Let’s talk about the early internet era of AGI discourse. I’m thinking late Usenet and then famously there was Eliezer Yudkowsky’s SL4 mailing list. You were pretty active on that. Talk about that era.
Ben 00:08:39
So SL4 was sort of an extremist offshoot of the Extropy list as I experienced it at that time. As soon as the web became a thing, I was convinced this is how AI was going to emerge — it should be decentralized like the internet is decentralized. The domain name service system is decentralized.
So I became possessed with the idea that you should have different nodes of the AI on different machines all over the world and they could be coordinating with no central owner or controller.
Another thing that happened when the internet came out is you could connect with the 300 or 3 crazy people in the world who were interested in some obscure topic that you were also interested in. The advent of the web was really the first time I found this sizable community of people interested in AGI and mind uploading and time travel and all this crazy stuff that I’d been thinking about since I was a little kid.
I had a few close friends who were semi-interested in these things, but until the internet, I hadn’t found a whole bunch of other people who had spent decades of their whole lives with wheels churning about how to make thinking machines or real immortality potions or whatnot.
The Extropy list was an email list which was in the late ‘90s really the center of online discourse on all these transhumanist and advanced technology topics. Extropy, of course, was a coinage from Max Moore, I suppose, and his wife, Natasha Vita-More, who I still know both of them quite well. It was their notion of the sort of opposite of entropy — growth and expansion toward a more and more amazing and abundant future.
At first, I was very annoyed with the Extropy community because they all seemed to be right-wing political libertarians, and I was sort of an anarcho-socialist. A good friend of mine, a Russian guy named Sasha Chislenko, I met through the Extropy list and later hired him for my first AI company, Webmind.
And he was there when Eliezer Yudkowsky came and gave a talk at Webmind about how we should stop what we’re doing. But Sasha was such a hardcore right-wing libertarian — sweet, lovely guy — but he literally believed in the future every molecule in the air should have a barcode and you should only be allowed to breathe it if you have enough money to suck in the oxygen. And otherwise, if you didn’t have enough money, then you don’t deserve to breathe.
This sort of thinking was around and wrapped up with all the futurist thinking, which annoyed me. On the other hand, it was the only place to go. At least you could argue the point. And we were going over and over on that email list about different ways of making AGI. If you mind upload, is your mind upload really yourself? What’s the best way to make a longevity remedy? Do you need to just activate DNA repair, or do you need to just put little nanobots through the body? You had Ralph Merkle there with the medical nanobots, and there was a variety of perspectives.
Ben 00:12:36
There was one group that basically believed humans, with a bit of upgrade, will continue to rule the roost. Humans would make themselves a bit smarter. Maybe they get superpowers like Superman — they can fly, they don’t have to die, they can leap tall buildings in a single bound, maybe get their IQ boosted. But basically, it would be humans.
Then there was another subgroup involving Eliezer Yudkowsky, myself, and some others — maybe 10% of the whole Extropy list — who thought it was obvious that humans are not anywhere near the ceiling of intelligence possible according to known physics, let alone whatever physics will look like after AGI reinvents and rediscovers physics.
So it just seemed obvious to us that humans cannot run as fast as a race car, and we can’t fly as well as an airplane. Why would we happen to have evolved to be the most generally intelligent possible thing? And what my good friend Hugo de Garis — who I hadn’t met face to face at that point, but we were exchanging emails — as he put it, the amount of intelligence in a grain of sand could be quadrillion times the whole human species once you have femto computing and atto computing and so forth.
So that was a sort of philosophical rift in the Extropy world. And Eliezer and I were thinking you can build a Jupiter brain that has unimaginably greater intelligence than humans.
The SL4, Shock Level Four, email list sort of was born out of that. That was a list for people who were not shocked by anything and who had no problem conceiving that human beings would be roughly the same intelligence as an amoeba compared to a post-singularity superintelligence.
Eliezer and I saw eye to eye on that, and at that time, Eliezer believed he knew how to build an ethical, beneficial superintelligence. So he was less worried about the existential risk because he thought he and only he knew how to build the thing.
Is There Headroom Above Human Intelligence?
Liron 00:15:17
I wanted to actually ask you more on that topic of how capable the AGI could be, and later we’ll go on to talk about something else you mentioned, that decentralization could be so important for AGI. We’ll revisit that when we talk about your latest projects.
But let’s talk about how capable AGI could be because I personally agree with a lot of the stuff you’ve said, and I think this is actually still a debate today. The same debate that you were having in the late ‘90s about how much intelligence could you fit into a grain of sand. There are so many people today who actually make fun of people for thinking that superintelligence could be incredibly powerful. So I would ask you to elaborate on how much headroom there is above human intelligence.
Ben 00:15:59
So there are a lot of people alive today who believe that if you fornicate outside of marriage, you will be tormented forever, boiling in a pit of oil for all eternity or something.
Liron 00:16:12
Right. Wrongness is timeless.
Ben 00:16:14
Yeah. The fact that a lot of people believe something that makes no sense — that says more about people than about the belief in question. I honestly find it hard to take seriously anyone who understands basics of modern science and thinks humans are likely to be anywhere near the maximum ceiling of general intelligence.
Actually, if someone doesn’t believe in science at all and thinks the world is just a simulation created by God 6,000 years ago, in a way, that’s more consistent because then physics can just be a weird delusion put there to confuse people. But if you accept science for what it seems to be saying, we’ve got these particles buzzing around — molecules, quark-gluon plasmas. You’ve got a great amount of potential for information processing at scales below the level of how neurons are working.
If you just do some physics calculations and try to calculate how much information could be stored in a given amount of mass energy, how much information processing could be done in a certain amount of mass energy — the answer is many, many orders of magnitude above what a human brain is doing.
If we then look at how evolution works, it doesn’t optimize things in a maximal and eternal way. The cheetah runs very fast and they’re beautiful creatures. On the other hand, no one’s going to argue that the cheetah is the fastest runner that could possibly be built under the laws of physics. It’s like arguing that a peregrine falcon, when it dives at 200 miles per hour, is flying faster than anything the laws of physics would permit. Why would any intelligent person think that way?
Now, to me, intelligence — it’s not as well defined as speed of flying or running, but in the end, it’s a collection of practical capabilities that some hunk of matter is carrying out. And I can’t understand why anyone with an understanding of science would believe the human brain is anywhere near the maximum. It doesn’t even seem like an argument worth having.
There are some arguments worth having — can a digital computer be conscious or does it need a quantum computer? Will an AGI that starts out with a human-like value system maintain that or drift to something utterly different? Those at least are arguments that make sense to have. But whether we are at the maximum or near the maximum level of general intelligence for a physical system, it’s kind of absurd to debate. People are just attached to themselves and to being the smartest creature on the planet. I don’t think there’s anything deeper to it than that.
Can Superintelligent AI Kill Everyone?
Liron 00:19:53
The rest of this discussion, there is going to be a significant debate between you and me on topics like alignment and why you’re more optimistic and I’m more pessimistic about how the singularity is going to go. We’re going to have a lot of substance to debate. But I think there’s a point at the beginning of the argument where you and I still agree.
Let me factor it like this. When I talk about humanity’s doom or even what I call the doom train — I’ve got a train behind me — and we’re talking about the train to doom. The train represents the line of argument of all the different things you have to believe if you want to believe that things don’t look good.
You can split the entire train of stops where the first half is, can superintelligent AI kill everybody? And the second half is, okay, let’s say it can — will it? So it’s can versus will.
Ben 00:20:40
Sure.
Liron 00:20:40
And I think you and I might be on the same page about “can.”
Ben 00:20:44
Right.
Liron 00:20:44
Because you’re acknowledging that it could have such powerful capabilities, and just to confirm, we’re even talking about something like rapidly designing and building nanobot swarms. It probably can do that. Would you agree?
Ben 00:20:55
Sure. I mean, I also think subhuman AI could kill everybody. You don’t need superintelligence to kill everybody.
Liron 00:21:02
So that’s already a great starting point, which is when we discuss AI doom or bad scenarios, I think we’re both on the same page that once the intelligence level gets significantly higher than human, all of this stuff is on the table. It’s just a discussion of what will happen.
Ben 00:21:15
I think it’s on the table before the intelligence gets significantly higher than human because if we connect Claude Code in the next upgrade version to the US military’s systems and it somehow threw a hack into that code, gets control of our biological and chemical nuclear weapons systems — you don’t even need a superintelligent system. All you need potentially is buggy code connected to the world’s most powerful weapon systems.
And then you have an argument: is that really an existential risk or just a risk to kill off a large swath of the world population? But I mean, you don’t even need superintelligence to pose a huge risk to the species. So yeah, we’re agreed on that point. This is a real technological frontier with tremendous capability for impact, for good or bad, depending on how it’s done.
Liron 00:22:19
Okay, great. Another thing I want to hit on to set the stage for potential disagreement or our doominess level. Before we get to that, I also want to talk about AI progress and timelines, because you’ve got such a unique perspective — one of the first people to even think about AGI and see it distantly in the future and then live to see it getting quite close. So let me ask you, how would you explain at a high level what’s happened in the last few years of progress toward AGI compared to what you saw before then?
Ben 00:22:51
It’s a bit subtle. I think at the high level, we’re basically on track to what Ray Kurzweil forecast in his 2005 book, The Singularity Is Near, which was qualitatively what he put out in his book The Age of Spiritual Machines a while before that, and what Gerald Feinberg put out in The Prometheus Project, or say the Russian AI pioneer Valentin Turchin said in his book The Phenomenon of Science in the ‘60s.
But Kurzweil, in his 2005 book — which was published the same year I published the book titled Artificial General Intelligence, which put that name on the map — Kurzweil saw progress toward human-level general intelligence by 2029. This was based on fairly high-level data analysis. He was just looking at the likely increase in capability of computers, and he figured by 2029, the computer you had on your desktop or in a server on a rack would have roughly the capability of a human brain. And so then he figured software would catch up, and then we would have roughly human-level AI by 2029.
He also thought it would take till 2045 to get a true singularity where your laptop would be more intelligent than the whole human species. And I think, as I said to him at the time, I was with him on the 2029 thing, but I thought he underestimated the exponent you would get during the follow-on phase.
It seemed to me like once you have a human-level AGI, I was more of the foom mentality as Eliezer and Robin Hanson put it back in the day. It seemed like once you have a human-level AGI, then that thing can revise its own source code, it can build new hardware for itself. It seems like you’ll be on an exponential trajectory with a faster exponent than the point up until you have the human-level AGI.
But I would say on the whole, we’re pretty close to following the trajectory that Ray Kurzweil laid out, which is that roughly speaking, as our available processing power approaches that of the human brain, we’re innovating different approaches to AI software that are getting us closer and closer to human brain-like capability.
Ben 00:25:31
That’s high level. When you drill down, it gets much more complicated, and the pattern with which our AI software is achieving human-level or superhuman capabilities is not something anybody foresaw.
There’s been lots of weird discoveries along the way. We have now large language models — ChatGPT, Claude, and so forth — and these are not that much like the human brain. They have a little bit of resemblance to aspects of the human brain because of having formal neuron structures that bear a little bit of resemblance to neurons in the human brain. But they’re not architected like a human brain really, and they’re trained in a different way — not through life experience, but just through machine learning on the whole internet.
But in their own erratic way, they’re able to do a lot of the same things a human brain does — write essays, do math, work out new physics theories, send emails, control robots, and so forth. Now, they’re still missing some things, and that’s clear to everyone. There’s a certain amount of self-understanding, a certain amount of common sense, a grounding of ideas in reality, and a certain amount of creativity that LLMs are missing.
And then we have the scientific question: can you get to AGI just by scaling up or upgrading LLMs, or do you need to introduce a bunch of other ideas alongside them? I tend to think you need to introduce a bunch of other ideas alongside them, but the needed ideas are already there in the historical AI field, so it’s more a matter of integrating them all together in the right sort of way.
But actually, either one of those avenues — whether it was scaling up LLMs or integrating a bunch of mechanisms together — that’s all consistent with Kurzweil’s grand scheme, that as we get better and better computers, we’re able to experiment with various kinds of AI software which will leverage the hardware available to come closer and closer to human brain capability.
Now, we might beat 2029 by a couple of years and get to human-level AGI in 2027. It might slip. It might be 2032 or something. Of course there’s some confidence interval around any estimate like that, but I think Ray was remarkably accurate in the big picture in that book.
Ben’s AGI Timeline: 2029
Liron 00:28:26
You gotta hand it to Ray Kurzweil because he was so steadfast on that 2029 prediction, and if you look at Metaculus and a lot of experts, it is converging very close to 2029. And if you look at that report, AI 2027, that predictive essay—
Ben 00:28:40
Yeah.
Liron 00:28:40
—that roughly is saying 2029 is when things will take off, maybe 2031. Kurzweil nailed it from a few decades back.
Ben 00:28:48
It’s the power of looking at the data. Amara Angelica, who’s a good friend of mine, she was working with him, and they were just collating all this data and fitting curves. Most people didn’t want to look at the data in a reasonable way, and he was willing to just gather the data, plot it out, and listen to what it said.
Liron 00:29:16
Yeah, it’s easy to retroactively say we all should have known. But I think you and I and so many observers — it took us so long to get on the same page. To give you a couple examples, on Metaculus and these prediction markets, you can see people thinking 2050 is roughly when AGI is gonna come, up until the ChatGPT-3 moment, and then it came down a decade, and then ChatGPT-4 came down another decade, and suddenly we’re talking early 2030s.
I was more on that front. I was more of a 2050 type of person until a few years ago. I think you might have actually been on the opposite side, because I remember that in 2007 you gave that talk at Singularity Summit and you were saying, “Hey, I think we can get AGI in 10 years.” So you were kind of a decade early. Is that fair to say?
Ben 00:29:59
The title of that talk, as I remember specifically — I gave a few talks around that timeframe called “Human Level AGI in 10 Years If We Really, Really Try.”
Liron 00:30:13
Ah, okay, so we didn’t try hard enough.
Ben 00:30:15
The species certainly didn’t. Now the species is, and that’s a significant point. Now there’s massive trillion-dollar companies focusing on AGI, and governments are making AGI policies. We don’t even yet have a US Manhattan Project for AGI, but the president is making noise about that, and we have huge companies doing quasi-equivalent things.
And China may have a Beijing project for AGI with the government and industry working closely together. So I think now our species is really trying, and what I was trying to say then is we should be really trying. But our species just didn’t because the powers that be controlling the resources weren’t willing to start really trying until ChatGPT 3.5 or whatever it was that hit them in the head and made them go, “Oh, shit, hold on. If this can write my college admissions essay, then maybe it can take over the universe.”
Liron 00:31:28
I’ll give you a point for that because I think what you’re saying is fair, that we weren’t really trying in terms of the percentage of IQ points and resources we’re investing. It shot up so much recently, and so you can flip the analysis from “we weren’t trying” to “now we are trying.” I actually agree with you on that point, and I’ll even give you a point for saying, “Okay, yeah, if we really tried, we could have shaved a few years off the timeline.”
It just leaves the question of how did Kurzweil know that we were going to try just the right level?
Ben 00:31:55
Because he was just plotting curves and the data that he was using already baked in to what extent people are willing to jump onto a new technology. If we’d really tried to make better chips in 1995, we could have done much better than we did. The whole trajectory of modern capitalism is based on people will only pile money into something once it’s already proved out in a very practical way, and then they’ll pile in a bunch more money. That’s the nature of the growth curve. So all these socioeconomic, psychological aspects were just baked in to the curves to which he was interpolating.
Liron 00:32:48
Now, you weren’t just a passive observer predicting the date of AGI. You were also trying to make it happen as quickly as possible. That’s been a main focus of your career — just hurry up and get to AGI. When you reflect on how much progress we’ve made and why we didn’t do it 10 years earlier and what were the insights, would you describe the last decade as kind of like Richard Sutton’s bitter lesson — we finally learned the bitter lesson and acted accordingly?
Ben 00:33:13
No. I think most of us knew that lesson already. Maybe he didn’t, but maybe Sutton thought that very simple RL systems could do amazing things without a bunch of data.
I think the idea that scale is what would lead to human-level general intelligence — that’s the whole basis of Kurzweil’s extrapolation, that we’re going to get computers closer and closer to human brain capability, and that will allow us to get closer and closer to human-level intelligence.
The strong form of the bitter lesson — that scale is all you need — I think is not true. You can’t implement dumb things at large scale and get smart systems. And it’s also not true that we’re just taking the same software we had 20 years ago and running it at greater scale and it’s being very intelligent.
As we saw in the recent inadvertent dump of 600,000 lines of Claude Code software — in addition to the LLM that they trained, and there’s a lot of tricks involved in training that sort of LLM, you saw 600K lines of neural-symbolic hackery, all sorts of complex things they did to make Claude Code do what it does. That relies on scale, of course, but once you have the scale, there’s a lot of hard algorithmic thinking that you have to do to make things work at the given scale.
So I would say the weak form of the bitter lesson was already well understood and baked into Kurzweil’s curve plotting and a bunch of other people’s thinking. The stronger form of the bitter lesson isn’t true, and if it was, Richard Sutton’s AI project would be getting more progress than it is. He’s applying reinforcement learning at a larger scale now, and it still doesn’t do continual learning — it still suffers catastrophic forgetting, because even with the current scale, you need some algorithmic innovation to overcome these issues. You’re not going to get RL to overcome catastrophic forgetting just by running backprop at greater scale, in my opinion. You need to improve the fundamental learning algorithms. But you probably need a large scale of machines to even run the experiments to improve the fundamental learning algorithms.
Liron 00:35:47
Right. So now we’re getting into your research focus or your engineering focus because you’re not really a large language model guy. You’ve worked on OpenCog. That was a different architecture. So let me ask you the question this way: what fundamental capabilities or ingredients is modern AI missing?
What Modern AI Is Missing
Ben 00:36:04
There’s actually a couple ways to parse your question, because you can ask what fundamental ingredients are LLMs missing, but then modern AI already combines LLMs with a bunch of other things to overcome some, but not yet all, of those limitations.
LLMs in themselves are very bad at grounding reasoning in observed reality, which is part of what leads to all the hallucination that they do. They’re also very poor at many kinds of memory that are critical to being a human-like agent — episodic memory, long-term declarative memory, even working memory within a lengthy iteration. They’re also poor at creativity. They’re derivative. They just in a shallow way recombine what was fed into them.
Now, some of these shortcomings are being overcome by embedding LLMs in more complex architectures. As we all see as users, Claude and OpenAI provide supplementary memory to the LLM in different ways. Your Claude can sort of carry over stuff from one session to another. OpenAI can sort of do that, but it’s a little flaky about it.
And then if you look at how LLMs are being applied in mathematics or software development, what’s key is that you write an agentic loop that combines the LLM with, say, a code interpreter or a formal verifier to verify math theorems. And so then you have some other bit of software that is providing grounding in reality, which the LLM itself doesn’t do.
Liron 00:34:34
Right.
Ben 00:34:34
So if you look at the modern LLM infrastructures, they’re already combining the LLM with symbolic reasoning-based things and with external memory stores to try to overcome some of the weaknesses there. But I think just adding these things onto the LLM doesn’t go far enough.
Part of this is because of an issue I already mentioned — continual learning and catastrophic forgetting. We take for granted that learning phase and inference phase are different, and that what we need to do is train a model like GPT 5.5, freeze it, freeze all the weights, and then that model does inference and a bit of in-context learning.
Of course, the brain isn’t like that. The weights in our brain — which are a metaphor for various electrochemical processes — these weights keep evolving as we learn, and we’re doing learning and inference all tangled up with each other in complex ways. I think that’s fundamentally a better paradigm.
This thing where you’re training a model, freezing it before it starts to catastrophically forget, and then using the frozen model for various things until you upgrade the model — I don’t think that’s how we’re gonna get to AGI. And now we’re hacking around that limitation with these external embedding vector memory stores and stuff, and that makes sense from a commercial product perspective, perhaps if your goal is to get a better product to market as rapidly as possible in a certain consumer LLM race. But I don’t think it makes sense for building AGI.
And then the creativity issue — nobody is trying to solve because they don’t care. You don’t need much creativity to make money. There’s a lot of the economy to automate without trying to automate fundamental creativity. So some of the key shortcomings are just not a priority for the companies leading commercial AI development now.
But I don’t think any of these things are out of the scope of the AI field as a whole, and all these things have been prototyped and experimented with by my own team and other teams for a long time. So it’s more...
But there are a variety of perspectives here. My friend Gary Marcus, who you would’ve heard of—
Liron 00:40:47
Yeah, friend of the show.
Ben 00:40:48
He sort of just thinks LLMs — he has his difficulties with Yann LeCun, but he would agree with Yann LeCun that on the path to AGI, LLMs are an off-ramp. And then at the other extreme, you have people who believe scale up LLMs, add a few bells and whistles, and you’ll get to AGI.
I’m sort of in the middle there. I think LLMs can be a quite valuable component of an AGI system, but I don’t think LLMs in their current form are the center of the AGI system. They’re more like an amazing knowledge oracle, which can then plug into the primary cognitive loop of the system.
Liron 00:41:37
Great. So just to recap your assessment of progress and timelines, you think Kurzweil’s 2029 is still a pretty good estimate of the milestone for true AGI, correct?
Ben 00:41:47
Yes.
Liron 00:41:49
Okay. Yeah, and I’m roughly on the same page. And then do you think that after that point, it goes from AGI to ASI — you mentioned you think it won’t take until 2045—
AGI to Superintelligence — FOOM in a Matter of Months
Ben 00:42:01
Mm-hmm.
Liron 00:42:01
—and I actually agree with you there. I’m seeing more of an AI 2027 scenario with quick improvement. So are you basically imagining a foom or a massive capabilities increase a few years after 2029?
Ben 00:42:10
I would suspect a massive capability increase within six to 12 months or something. And the only reason I think it’s six to 12 months instead of one week is if the AGI system itself is a little tentative about radical upgrades to its own machinery. Even if I could stick a soldering iron in my head and upgrade my brain, I might be a little careful about it because I don’t want to mess myself up too badly.
I think that once you have a full-on human-level AGI, given what we see already about how good LLMs are at doing math and doing programming and doing engineering — it seems very clear, even without building new hardware, the human-level AGI should be able to make itself an order of magnitude more efficient in using the hardware that it has, or two orders of magnitude.
How optimized is a whole Linux stack? Not very, from the perspective of what a super AGI could do.
Liron 00:43:22
Exactly.
Ben 00:43:22
So yeah, you would think that even without hardware innovation, just by rewriting its own software, you should get one or two orders of magnitude of efficiency improvement by having a human-level AGI improve itself.
And then that already, if your first thing was a little above human level, just by software self-modification will get you well above the human level. But then at the same time, this human-level AGI will be re-instrumenting factories and chip fab facilities and so forth. And probably there are ways to make not such huge adjustments to the way chips are fabricated that would specialize them better for AGI than what we humans have done.
So then maybe just by tweaking the existing hardware facilities, after another year or two, you could get another couple orders of magnitude improvement from better chips. And then somewhere along the line, that AGI or that early-stage superintelligence may invent a whole new type of hardware or something.
So yeah, I think it’s human level, then within six months or a year significantly superhuman, within another one or two years way superhuman. And I think it could happen faster than that. ## The Arms Race Risk
Ben 0:45:00
But I think if we do our job right, the AGI will not be upgrading itself at the maximum possible speed, just like you or I will not upgrade ourselves at the maximum possible speed. Because there is some balance to be struck.
But this gets into the existential risk issue. What you don’t want is Trump AI here, Xi Jinping AI here, and each one has to self-modify faster than the other one to outdo the other one in some arms race from AGI to superintelligence. That would probably not be good.
You would like the path from AGI to superintelligence not to be wrapped up in some horrifying military arms race type scenario.
What’s Your P(Doom)?™
Liron 0:51:33
So yeah, everything we’ve talked about so far — just summarize it as you and I are both expecting in less than a decade to see capabilities that maybe qualitatively have that sense of awe, kind of like an ancient human looking upon the cities and the military hardware and the communication and the screens, the computation that we’ve all built up today. That sense of awe — “What the hell? You guys can do all that?” I think we’re both expecting to see a lot of that from our own perspective in less than a decade, correct?
Ben 0:46:22
Yep.
Liron 0:46:22
Okay, so we’re gonna segue from the “what can AI do” part of the conversation to the “what will AI do,” and I think that’s where we start having disagreements. So are you ready for the most important question of Doom Debates?
Ben 0:46:34
Let’s go. P(Doom). What’s your P(Doom)? What’s your P(Doom)?
Liron 0:46:41
Dr. Ben Goertzel, what is your P(Doom)?
Ben 0:46:46
So, as you know, I don’t think that’s a very useful question because we don’t have any rational way to estimate that. So if I have to pick a number, I’ll come up with π percent. Pi is a very nice number.
Liron 0:47:07
Just to see if there’s anything I can get out of you on this question — I can imagine somebody like Yann LeCun goes around saying, “Oh, it’s less than 0.01%.” Do you think that’s a reasonable answer?
Ben 0:47:18
It may be reasonable. To me, it’s optimistic about geopolitics because I think the probability of doom without AGI is higher than that, based on the whole military balance of powers and the whole spectrum of advanced technologies being developed.
I think if we don’t have AGI, we’ve got nanotech. We’ll have advanced bioweapons. We’ll have a lot of other nasty things coming, and we have warlike people running major countries. So I don’t see how you get the doom level that low regardless of any particular technology that you’re looking at.
Liron 0:48:01
Do you think there’s at least a 10% chance that human civilization will go catastrophically wrong, let’s say, by 2100? Nuclear war—
Ben 0:48:11
I don’t honestly know how to put a rational calculation on that. I do think, subjectively, 0.0001% feels wrong. Because even with just the nukes that we have now — what if some conflagration over Taiwan or something goes awry and we have a global thermonuclear war?
All these calculations about nuclear winter are pretty speculative in the end. You could imagine a global thermonuclear war unfolding, and then we just all get irradiated to death. I haven’t tried to make a rational calculation of that probability.
I guess I don’t need to because if I did, it wouldn’t change my actions in life in any way. But it feels like weapons that could arguably wipe out the human species exist, and there are people whose judgment I don’t fully trust with their fingers on the button. That’s the current situation.
Liron 0:49:30
Well, one reason I ask about the doom of the world by 2100 is because if we thought nuclear war was like 80% chance or some catastrophe was gonna happen no matter what, then AI could just be our Hail Mary pass, even if it looks super dangerous.
But conversely, if you thought, “Hey, there’s only like a 10% chance we’re gonna kill everybody by 2100. We’ll probably make it through the century,” in that case, it suggests, okay, well, if AI seems pretty risky, maybe we should actually tread more cautiously.
Ben 0:49:58
Right. But I sincerely don’t know how to place a believable number on these things. Because it’s not just about nuclear weapons as they exist now. What destructive technology might be developed by human scientists working together with ChatGPT version 11 by 2040? How can we have methodical... I mean, will we solve nuclear fusion by then? Will we have a fusion bomb?
It seems like the possibilities for inventing insanely destructive weapons using the non-AGI tech we have now — there’s a lot of potential to develop very destructive things, and so far, it doesn’t look like geopolitics is especially evolving in a pacifist and beneficial direction.
But does that give us a 20% or 80% chance of world destruction? I sort of feel like trying to pinpoint odds for something like this is a fool’s game, and the confidence interval becomes very, very wide.
Liron 0:51:26
I just have one more question along these lines, because you did say in the last part of the conversation, “Hey, AI is going to have these incredibly qualitatively impressive capabilities,” kind of like the ancients marveling at modern society.
And now you’re saying, well, the P(Doom) of the world is already substantial. It’s hard to predict, but it’s certainly on the radar even without AI.
But it seems to me like AI, given that it’s gonna be extremely powerful in the next less than a decade — it seems like that could potentially escalate whatever probability of doom. It seems like that is going to bump it up appreciably. But let me ask you the question this way: if you could turn the dial to speed up or slow down AI capabilities progress right now, what would you do?
Ben 0:52:12
I would hope for a better dial, because it really depends on who’s developing the AI and how it’s being done. And it’s not exactly clear what that means. Does that just mean giving more money to Deep—
Liron 0:52:39
Yeah, so you read the owner’s manual of the dial and it says, “Hey, anybody who is about to make a research breakthrough in X years is now going to make it in X over two years.”
Ben 0:52:57
Oh, then I would speed it up. Yeah. Because then you’re averaging it over groups like mine as well, as well as just big tech and the Chinese government. This is because I think I can get there before these guys anyway. So if we’re all speeding up by double, then I will still get there before them. It’ll all just happen faster.
Liron 0:53:20
Got it. So you got skin in the game. You’re speeding up your own progress. Okay.
Liron 0:53:24
I’ll read a quote from a post you made in 2010. Rather than a quote: “If you go ahead with an AGI when you’re not 100% sure that it’s safe, you’re committing the Holocaust” — you’re quoting that as that’s not your position. You say, “I suppose my view is closer to: if you avoid creating beneficial AGI because of speculative concerns, then you’re killing my grandma, because advanced AGI will surely be able to help us cure human diseases and vastly extend and improve human life.”
Okay, so that’s consistent with what you said now. So would you just use the term accelerationist on yourself? Because you basically think AI capabilities acceleration is a good thing, correct?
Ben’s Case for AI Acceleration
Ben 0:54:00
I don’t think it’s a good thing unreservedly, and you’d be familiar with the buzzwords e/acc and d/acc. So I’m more d/acc than e/acc, which sort of builds in the idea that if we can do it right, we should be accelerating it.
The d/acc buzz phrase is decentralized accelerationism, but of course, that’s a shorthand because decentralized doesn’t have to be right either. There can be bad decentralized things. So I think if we could accelerate the right sort of AGI development, that will decrease not only short-term suffering but decrease existential risk.
Of course, there could be ways of accelerating AGI development that would be bad in the short term and in the long term. It really depends on the details, as you would expect with something as complex as what we’re talking about.
Liron 0:55:15
The d/acc dream is great. It stands for, like you said, democratic acceleration and also defensive acceleration, decentralized acceleration — there’s a few different D’s. I had Vitalik on the show last year, and I think he’s coming from a good place. I would love the d/acc future.
The problem is I just don’t see the mainline scenario of what the d/acc future looks like, because given AI capabilities — we both agree they’re coming really large, very powerful, very imminent. And if you just have all these decentralized players that all have access to this extremely superhumanly powerful AI, it’s hard for me to imagine how that translates into a balance. So tell me about your mainline scenario. How does the balance of superhuman power work?
Ben 0:56:02
I think about it in two different phases, and you’re asking more about the second phase. Or I guess three phases. One is here to AGI, one is AGI to ASI, and the other is sort of post the ASI breakthrough.
To my mind, the biggest risks, both for short-term human suffering and existential risks, occur during the period between the first true human-level AGI breakthrough and the emergence of a superintelligence. But the question you just asked seems to be more post-superintelligence — when you have a decentralized field of superintelligences.
My feeling then is you will see the emergence of what I’ve referred to for a long time as a mindplex, which amusingly is a term that Eliezer Yudkowsky invented for me when I was asking him, probably on SL4, for a term for something like halfway between a mind and a society — more unified than a human society, less unified than a single human mind controlling a human body.
Because if you have a decentralized network of AIs, they can swap brain matter as much as they wish. They don’t have to communicate in a low bandwidth way such as we do using emails or voice or something. So I think you could have a decentralized network of ASI systems that have a greater level of coordination and understanding than societies of humans do.
And I think there’s a pretty strong mathematical and computational argument that cooperation is more efficient than struggle, strife, and combat. There are more hard problems you can solve by a bunch of agents cooperating and sharing ideas freely than there are problems that you can solve by a bunch of agents that are hiding stuff from each other and being competitive.
So I think once you get to superintelligence, you’re gonna see an interestingly well-coordinated sort of mindplex that breaks the shallow dichotomy between decentralized and centralized that we see now in the human domain.
Ben 0:58:44
What worries me more is the period between the initial human-level AGI breakthrough and the superintelligence, because then you will have systems that are maybe human level or a little smarter than humans, and you will have a lot of human beings who are trying to control them to achieve their human goals. That’s when things get complicated and potentially messy and dangerous.
Then you have to ask, what do you fear more? Do you fear a sort of heterogeneous population of human groups in every country in the world, each of which is striving to do their own thing with human-level AGI? Or do you fear Trump, Xi Jinping, and Putin each controlling a big early-stage AGI to achieve their own geopolitical ends?
Because these seem to be the options on hand for this period between AGI and ASI. The option of a rational, benevolent, democratic world government deciding how to accelerate or decelerate AGI progress for the common good of the species — it’s a pleasant idea to science fictionally speculate about. That seems not the way geopolitics is going, and I don’t see how it’s gonna go that way in the next few years.
Liron 1:00:20
So you’ve got this idea of the mindplex, which sounds good, and it dovetails with Vitalik’s d/acc — democratic, decentralized, defensive. Let me ask you, do you think that the individual minds in the mindplex care about humans, and so it’s just like different humans that have their own aligned AI, and then they’re negotiating that way? Or do you think maybe none of the AIs in the mindplex are that aligned to humanity, but the fact that they all have to coexist is what makes the future good?
Ben 1:00:51
I think that if we do it right, then the default among superintelligences will be caring for humans and other sentient beings. I think that’s an achievable way to architect an AGI. And if you architect the initial AGI that way, then it’s likely that the AGI, as it self-modifies and improves, will maintain this beneficial orientation to humans and other sentient beings.
I think we certainly can do it that way, and the risk that worries me is that we don’t do it that way because the people who create the first AGI are just thinking about other stuff, like their own wealth and power, rather than thinking about architecting an AGI system with the good of humanity and all sentient beings in mind.
Liron 1:01:57
Just to clarify, so if everybody’s AI was very selfish and power-seeking, then combining those AIs in a mindplex might not help much. But you’re optimistic that the individual AIs will care about sentient life?
Ben 1:02:11
Yeah. I think there’s quite a lot that we don’t know about superintelligence. It could well be that a superintelligence will, with high probability, converge on this sort of beneficial, compassionate value system, even if it starts off kind of nasty.
That might be the case. I’ve tried to convince myself it’s the case and haven’t quite managed to yet. I went through a bunch of work to try to do a math analysis of this sort of scenario — the value systems of rich-resource minds and superintelligences. I managed to create a reasonably rigorous math argument that if your AGI is already compassionate and beneficial, and it’s architected in the right way, as it self-modifies, it is likely to remain compassionate and beneficial. I could make some nice fixed point theorem arguments about that.
I haven’t yet come up with even a rigorous just-so story to convince myself that if the AGI starts out an asshole, that it will, with high probability, converge on a compassionate value system. Not to say that wouldn’t happen. I just haven’t — even though I’m an optimist, I haven’t managed to convince myself of that one.
The Orthogonality Thesis Is “Dishonest”
Liron 1:03:50
So this gets to the topic of the orthogonality thesis — the idea that you can be arbitrarily intelligent but then also have arbitrary morality or arbitrary values.
When you were saying that these kind of mean, nasty, selfish AIs — you haven’t convinced yourself that they turn into nice AIs — I feel like that means that you agree with me that the orthogonality thesis is at least somewhat true?
Ben 1:04:14
No, I think it’s a very dishonest and misleading idea, actually. I think it might be true that in principle you could take any stupid or ridiculous goal system and objective and tie it on to an arbitrarily intelligent system. But that doesn’t really matter if the odds of that combination are very, very low in any realizable situation.
I think the goal system, the set of motivations and aspirations and values of a system, are not in reality like an extra thing that’s papered on to the system. I don’t think that is how humans are. Our goals and aspirations are interwoven with all the rest of our minds, and I think the same will be true of AGIs and ASIs.
So in fact, I think there’s extremely strong interdependencies between what are the goals and values and what is the intelligence of the system and how the system works. I think even if the orthogonality thesis as stated is true, it’s kind of irrelevant because what matters is what combinations of intelligence levels and goals and values have a reasonably high probability of occurring, not which ones in principle could occur together in a vacuum.
It reminds me of Eliezer’s idea that if you pick a random mind out of mind space, the odds that it will care about humans is very small. Technically might be true, might not be true, but it doesn’t matter because it’s not what we’re doing. We’re seeding a very specific mind out of our own goals and our own culture. So arguing about random minds is interesting, but I’m not sure why it’s relevant. It’s like arguing that a random kid won’t love me. Okay, but my kids are not random kids. I raised them in a specific way.
Liron 1:06:46
Right. Okay, so I think we’ve identified three degrees of claim here. Just to recap for the viewers what your position is. The weakest claim is what I would call the orthogonality thesis, which is just saying somewhere in the space of possible AI algorithms, you’re gonna find at least one that has a distasteful morality and yet extreme intelligence. I think you’re willing to agree to that, but you’re just going on to the next level.
Then there’s the next claim of: are these common types of AIs? You can go to a counting argument — what’s the probability of getting such an AI? And then there’s another claim: okay, maybe there’s a lot of them, maybe there is a high fraction of AIs that have distasteful morality. But that still doesn’t matter because we’re the ones building it, and so the only question is: are we going to build it? Correct?
Ben 1:07:35
Yeah, that’s right. And the first version you posed feels like it’s probably true. The second version you posed I’m not as sure about. And yes, the third version is the one that we should actually care about.
Liron 1:07:57
Yeah, I agree with that. That’s fair enough. Now, you mentioned a few minutes ago that you do worry about people with selfish motives and profit motives just trying to race to the profit prize or whatever, and that could lead to a situation where we then get this permanently superintelligent, immoral, non-mindplex AI, correct?
Ben 1:08:18
Yeah. Or it could be a malevolent mindplex also. Indeed, even if we set aside all my attempts to come to grips with this mathematically, just on a basic human common sense level — if we create something one and a half times as smart as the smartest human, general intelligence, creative, grounded in reality, and this seems like a warm, loving creature that is nice to us and wants to help everyone do the best in life, versus a system with a similar level of intelligence and capability whose goal is to help one country achieve dominance over all the others on the planet by blowing them up if necessary —
There’s a certain amount of common sense as to which of these is more likely to lead to a beneficial superintelligence. And so far, my attempt to analytically understand the situation agrees with this commonsensical perspective that the former is more likely to lead to a good outcome than the latter.
Liron 1:09:38
Let me tie in this other concept that seems relevant to the discussion of immoral AIs — this concept of transcendence. I’ll quote from one of your posts. You said, “The core philosophical flaw in Eliezer’s reasoning on these matters is treating intelligence as pure mathematical optimization divorced from the experiential, embodied, and social aspects that shape actual minds. If we think about AGI systems as open-ended intelligences, a concept I explore in my 2024 book The Consciousness Explosion, we see them as living, self-organizing systems that seek both survival and self-transcendence.”
Basically, my question to you is, do you think that this drive to seek self-transcendence actually makes it safer to kind of rush ahead and build AIs because maybe they’ll just seek self-transcendence, and then it’ll just be okay for that reason?
Ben 1:10:26
I think that’s just the way living systems are. My friend Weaver, AKA David Weinbaum, had a PhD thesis at the Free University of Brussels a few years ago called Open-Ended Intelligence, and he’s trying to formulate a perspective on intelligence that’s more fundamental than reward maximization or the typical perspectives taken in the machine learning field.
He’s looking at an intelligence as a complex self-organizing system, which at a high level has two conflicting yet synergizing drives. One of these is individuation — maintaining its boundaries, continuing to be a system. The other is self-transcendence — to grow into new forms going beyond what it was before.
On a philosophical level or a biological origin-of-life and evolution level, you can see that the tension and cooperation between these two drives has been around since the beginning of the universe. I think this characterizes growth of humanity on an individual and a cultural level and will characterize the growth of AI systems going forward.
I don’t think we will succeed at constraining that. It’s sort of like a Jurassic Park “nature will find a way” type of perspective. If you try to create a super AI and constrain it to be a pure optimization oracle that can’t grow and can’t self-transcend in an unpredictable direction, but will only stay within certain rails and do what you want it to — I think via some way or another, someone will connect that system to something else. We’ll get some meta system that grows and evolves.
It’s going to become a complex self-organizing system balancing individuation and self-transcendence like all living systems do, with a high probability. Does that make things safer than if there was no drive to self-transcendence? That’s sort of a weird question to me because the drive to self-transcend is imminent in all complex self-organizing systems in the universe.
But I would say on the whole, sure, because otherwise what do we have? We have a system that we put our 2026 values of some particular slice of human society nailed down without ever changing into some super powerful system. That seems like there’s a lot of bizarre failure modes associated with that growth-and-evolution-free option, which is not really feasible to do anyway.
Liron: Moore’s Law for Goal Achievement
Liron 1:13:43
Let me put some of my cards down on the table here in terms of my mainline scenario. I kind of see a Moore’s law for goal achievement or a Moore’s law for optimization, because when you look at Moore’s law, Gordon Moore was noticing transistor density was the property, but you could generalize it a little farther and say computation could be the property — number of flops happening across the world keeps doubling.
There’s another property that I’m noticing, which is goal achievement or optimization power — the ability to navigate to outcomes. To pick a series of actions where you do the actions, even though everybody else on the game board is trying to pick their actions, or the universe — the weather is what it is that day. And for whatever reason, you’re able to get the outcome that you set out to get. You had some parameters for the outcome that you were trying to get, and sure enough, the future meets those parameters because you chose actions that got you there.
On that spectrum of how effectively can the best agents manipulate the universe to get to these goals, I think there’s going to be a kind of Moore’s law for that, where it’s going to very rapidly get better and better on this dimension.
And the only difference between this kind of Moore’s law and the original Moore’s law is that in the original Moore’s law, you can have a bunch of computers getting more and more powerful on their separate islands doing their own computation, and they don’t have to hurt anybody. But if you look at Moore’s law for goal achievement, the problem is that we live on the same game board, and if I achieve my goal, then that could logically imply that you don’t achieve your goal, and we get a conflict. What are your thoughts?
Ben 1:15:16
I think that’s a very childish and irrelevant way to look at the universe. I think goal achievement and even intelligence as we’re now conceiving it are probably gonna seem like very silly notions to even an early-stage superintelligence.
The idea that an ASI is gonna be concerned with optimizing functions or achieving goals or accumulating resources — I think this is projecting ephemeral characteristics of human motivation and life into what will be a quite different order of complex self-organizing system. An ASI will not be playing a game on a game board. We may view ourselves that way because we evolved that way, but that’s not likely to be the perspective an ASI is going to have.
Liron 1:16:35
I think we both agreed that it’s going to have these — I don’t want to say godlike, feel free to substitute whatever you want — but godlike powers to get things done, to engineer things. You didn’t have a problem with the concept of nanotechnology, for instance. So how do you define intelligence then?
Ben 1:16:53
I don’t think that’s necessarily a cosmically important problem, but if we’re within the reinforcement learning world, of course, we have the Marcus Hutter / Shane Legg definition, which is basically the ability to achieve computable reward functions in computable environments on average with some suitable weighting function.
If you look at Weaver’s notion of open-ended intelligence, he’s not looking at it that way. He’s just looking at complex self-organizing systems that are individuating and self-transcending. And you can bridge these perspectives because a very complex system that’s individuating and self-transcending in a complex dynamic environment — along the way, it’s gonna face a lot of challenges, and it’s gonna be achieving a lot of complex goals in complex environments as it goes.
I don’t think there’s a fundamental contradiction between these ways of looking at things, but how we conceptualize what these superintelligent systems are gonna do is not relevant to what they’re actually gonna do.
Liron 1:18:31
If I’m just summarizing you correctly, it sounds like you’re saying yes, in the future, they are going to pass your Moore’s law for goals criterion, meaning if you actually measure their optimization power, yes, it’s gonna go up exponentially, but that’s not the focus. It sounds like that’s your attitude.
Ben 1:18:49
Sure. That’s like — I could now defeat my five-year-old daughter and all her friends at all their games on the playground. I could do it. If they ever really beg me to, maybe I’ll do it one day. I play tag with them now and then. I can outrun all of them. It’s great. I’m a mighty freeze tag player. But that does not really summarize the crux of what I’ve achieved in my life.
Liron 1:19:17
Do you think it’s pretty likely that in the future of all these islands — these decentralized islands of superhuman intelligence — one of the islands is pretty likely to go rogue and want to conquer, and then it’s really up to the other islands to keep them in check?
Ben 1:19:32
By intuition, I suspect that’s more likely to happen between the AGI and ASI stage, which is the stage I’m most worried about. But I would say none of us rationally can know what a society of superintelligences is gonna be like.
Among the many possibilities, which has been explored in science fiction many times, it could be that once an AGI gets to a certain level of intelligence, it notices all the other intelligences out there that we are just too stupid to see. Maybe they disappeared into black holes, as in John Smart’s transcension hypothesis. But when you’re smart enough, you can read the subtle patterns in the black hole radiation. Maybe they’ve downscaled into subquantum, subparticulate vibrations, as in Hugo de Garis’s search for intraparticulate intelligence. Or maybe something much weirder than all that.
It could be that once you get the superintelligence, it just sees the ambient field of superintelligence that’s already there that we’ve been too dumb to notice, except maybe in small inklings here and there. Something like that would make the whole question of rogue superintelligences utterly different than if something like that doesn’t happen. We really can’t know about something like that, although it’s fun to speculate, of course.
Liron 1:21:14
I have a little scenario that may illuminate your perspective. Imagine you’ve got a capabilities organization like an OpenAI, and currently they’re racing to try to boost the capabilities. You’ve told me that you’re concerned that they’re too profit-motivated, and they might go too fast, and they might be too selfish about it.
So imagine if we could somehow tweak them where they’re really not selfish or profit-motivated, they’re just trying to be neutral, where all they wanna do is just push the AI past the threshold of superintelligence. They just wanna do neutral, quote-unquote neutral capabilities increase. That’s all they wanna do. Would you then feel good about this idea of just neutrally getting to the singularity?
Ben 1:22:00
It’s hard to concretely understand what that would mean actually, but on the whole—
Liron 1:22:18
All it means is saturating all the benchmarks. Inventing benchmarks, saturating, repeat.
Ben 1:22:23
Yeah, I understand, but what LLMs are doing is spewing out recombinations of human culture, which is not — in what sense is that neutral? But on the whole, seat of the pants, I would say no, that would not make me very happy.
I would much rather explicitly put a compassionate and beneficial goal system into the AGI. So if you envision it as a dynamical system, let’s say you have an attractor basin of beneficial, compassionate AGIs. Let’s just build a system inside that basin so it will converge to something nice.
Let’s not dick around with, “Well, can we put it on the fringe of the attractor or a little bit out of the attractor and then see what happens?” As a species, why would we want to do it that way? It’s kind of stupid. I don’t have a strong odds estimate in my head of what happens then.
Liron 1:23:34
So if I just take the last thing you’re saying now — hey, when we build superintelligent AI, we should really try to put caring for humanity or caring for sentient life as being near the center of its goal system.
Ben 1:23:44
Yes.
Liron 1:23:45
I feel like that’s a good statement of the alignment problem that I agree with, and I feel like that’s hard to do. No?
Ben: Alignment Is Just Good Parenting
Ben 1:23:50
The analogy to this is raising children. I have five kids and one granddaughter. You could raise a kid like an attack dog to be a vicious killer, and then there’s of course a risk if they decide they don’t like you, they’ll turn around and turn that on you. You could raise a kid to be warm, loving, and compassionate insofar as you can figure out how, and there’s not a guarantee there, but on the whole, that often works out well.
You’re asking if you raised a kid to be completely neutral and dispassionate, how will that turn out? That’s a weird question. They could fall under the spell of someone else who convinces them to do something you don’t like. A lot of things could happen.
But is it hard to raise a child to be warm, loving, and compassionate? It’s not that hard. I think I’ve succeeded at it with my three adult children so far. They’re pretty nice people.
Liron 1:25:00
Yeah, but it’s just hard to separate how much of the compassion was built in.
Ben 1:25:03
Well, the capability for it is built in, and it also needs reasonable, loving, caring upbringing for that capability to be brought out. And there’s an analogy to AI there in that you need to architect the motivational system in the right way. You need to build a cognitive architecture that is motivated and agentic at the core, rather than like a shallow agentic wrapper around a next token predictor or something.
You need a cognitive architecture that is motivationally driven and is instrumented to try to understand itself and understand others. And you need to supply it with the right initial goal system in which compassion and benefit to others is one of the top level goals. And then you need to raise that AI system by working together with it to do beneficial things in the world as it matures.
I would submit if you do these things, then it’s probably not that hard to get to an AGI system which is beneficial and compassionate in orientation.
This happens not to be what’s happening in the mainstream AI industry at this moment. So is it hard conceptually or from an engineering standpoint or from a practical standpoint? I don’t think so if you have energy and resources associated with it. Of course it’s hard relative to digging a ditch in the ground, but it’s a big achievement like building a laptop or building a bridge. I don’t think it’s hard beyond the level of major engineering achievements that we can do.
Is it gonna be hard to direct enough resources away from profit motive or national security motive toward human benefit motive in building AGI? Now that’s a different question. That’s a question about politics and fundraising and open source community building. That’s a different sort of hard.
Alignment in Practice: OmegaCLAW
Liron 1:27:36
We’re about to segue into talking about some of your research projects. I think a good segue then is, when we talk about the alignment problem and orthogonality and whether how we’re gonna raise the AI to be good or whether it’s gonna naturally be good, let me ask you it this way: when you have your project to build AGI, at what point are you not just maximizing capabilities, but making a trade-off to get more alignment instead?
Ben 1:28:09
I would say we haven’t hit that point yet in my own projects, but it’s not hard to see how that would happen because we already have early experiments we’re running now where misalignment of various sorts happens and then we have to sort of rein experimental agents in and reboot and realign.
We have a framework we’re playing with now called OmegaCLAW. It’s an OpenCLAW-type system, but in addition to an LLM in the agentic loop, we put a symbolic long-term episodic and working memory in there. And we coded the thing — not the LLM, but the agentic bit — in our AGI language, Metta, M-E-T-T-A.
So you can get an OpenCLAW-type agent in like two hundred lines of code, and that’s a language written for self-modification. You can chat with a thing. You can suggest it should rewrite its code to improve its memory or reasoning in a certain way, and it can then do that in real time while it’s talking to you, which is quite cool.
Now, that’s not an AGI yet. It does have better personality, better reasoning, better memory than an OpenCLAW-type system. And it just qualitatively — it’s a fundamentally different sort of thing to interact with. It knows who ## Alignment Failures in Early Self-Modifying Systems
Ben 1:30:00
It knows who and what it is, and it builds up its sense of itself and its collection of capabilities over time in a way that OpenCLAW systems don’t with their sort of vector embedding based memory.
With this sort of system, we don’t yet have a sophisticated goal system integrated with it. That will be there by the end of the summer. But this sort of system tries to be aligned with you and just misunderstands and messes up sometimes. Now and then it disables itself — it modifies its own code into a dysfunctional state, or it will try to do what it thinks you want and will utterly misinterpret that and go off and do the wrong dumb thing.
So we’re already seeing with these early experiments with self-modifying AI systems driven by a combination of symbolic logical reasoning and LLMs — we’re already seeing failures of alignment. But these are in systems that we know are not smart enough to be dangerous, except by enabling people to do dumb things like an LLM can already be dangerous.
We haven’t yet integrated our fancy beneficial motivational system into it, which we’re in the middle of doing. So that will give us a playground for systems with a well-structured goal system, compassion and benefit there, LLMs, logical reasoning, some evolutionary learning for creativity, but clearly not yet at the human level AGI level. And it will be interesting to experiment with how alignment and misalignment actually works in this kind of system.
Ben 1:31:54
One thing that’s interesting from a sort of safety research and AGI boxing perspective is what has made this kind of system interesting is partly the fact that it’s on the internet. If you take a comparable system and put it in a box with a limited amount of data, it’s just not as interesting.
So what we’re seeing is like the first inkling of a practical breakthrough toward AGI in my project is something that’s coming from putting our special fancy AI methods together with LLMs and together with a CLAW system interfacing with the internet. So it’s a specifically non-boxed system. You have a logical reasoning system, and it’s very valuable that when it needs a new premise for reasoning, it can go on the internet and find a new premise and just pull it in.
So it seems like we’re gonna be experimenting with this question you asked starting in the fall in a practical case. It’s very clear you will hit that — if you have your own OmegaCLAW agent that you’re trying to teach to be a helpful research assistant or personal assistant, you are gonna hit a case where it’s becoming more capable, but in a way that’s causing it to do its own thing rather than your thing. We’re gonna get a practical sense of how to manage it during a pre-AGI phase.
I think that’s sort of how it has to be. I don’t think you’re gonna be able to solve that problem by theory alone any more than they could solve the problem of how to make a scalable transformer by theory alone. AI is developing by a mix of theory and practice.
Liron 1:34:01
A common thing we hear is that you gotta launch these AIs and have their capabilities grow in order to effectively have a feedback loop that can help you align it. That’s basically what you’re saying.
Ben 1:34:11
Yeah, but I didn’t want it to be that way. I spent a lot of time last year trying to do an in-depth mathematical analysis of self-modification and the stability of goals under self-modification. I think I made a lot of progress there. But what I found is you can make more progress mathematically under assumptions of very rich resources.
When you get closer to superintelligence, it becomes more tractable from a math standpoint, sort of like how a genetic algorithm is easier to analyze as population size goes to infinity. And the small size problem of a practical system we can play with right now — there’s all these finite size corrections and it’s just harder for the math to tell you what’s going on.
Ben 1:35:06
MIRI, Eliezer Yudkowsky’s group, kind of found the same thing. They did a bunch of nice papers about the AIXI type stuff, like the case where you’re near infinite compute size. You can do more nice theory there, but then the nice theory doesn’t seem to tell you that much about systems that are just barely big enough to do general intelligence. It’s very messy and it seems like you’re figuring things out through a mix of theory and practice, and that’s not satisfying to me intellectually. It just seems to be the real situation.
Liron 1:35:46
I’m glad you brought up stability under self-modification. That’s the idea of like, hey, we have this AI and it seems to be talking friendly to us, but when I ask it to modify itself or make another AI — program another AI from scratch — it programs this other AI and all the properties that we thought we gave the original AI, it seems like they aren’t preserved the way that we thought in the successor AI. Oops, and by the way, they’re superintelligent.
So what you’re saying is you couldn’t really get an intellectually satisfying way to convince yourself it’s gonna be stable under self-modification—
Creating Stable Goals Under Self-Modification
Ben 1:36:19
I convinced myself mathematically that I’ve found nice ways to make systems that will have stable goal systems under self-modification once they’re above a certain threshold level of general intelligence and capability. So I did convince myself that once you get to a beneficial superintelligence, it can, with high probability, maintain its beneficial goals under self-modification.
What I didn’t convince myself of is how to make that provably true at the early stages when you have something that’s just barely human-level intelligent.
Liron 1:37:06
Hmm.
Ben 1:37:07
Then it’s just harder for the system to model itself. A simplistic way to look at this argument is if you have something close enough to superintelligence, it can run a simulation of the relevant swath of the universe and how things would evolve under this or that modification to its code. The closer you come to doing that, the better off you are.
I can say, “Okay, if I zoinked around my hippocampus in this way, let’s run a simulation of how earth will be with my zoinked hippocampus.” If I’m smart enough to run that simulation in a passable way, that may not be a guarantee, but it’s a very interesting situation to be in. But you just need a lot of intelligence and a lot of resources to take that sort of approach.
So I think for reasons like that, when you have a superintelligence with enough capability and resources, it may well be able to trial out tweaks to its algorithm and see if they will indeed violate its goal system before effecting those tweaks. But it’s harder to see how to do that for a system that’s still at the roughly human scale, just from a purely intellectual or math or algorithm perspective.
You can still do a better or worse job. You can make a system that deploys its best uncertain reasoning engine to try to evaluate how a given modification will impact itself and others before it puts that into effect in its brain, or you could make a system that doesn’t do that. You could make a system that trials out a variation of itself in a sandbox for a while before putting it into its main system, or you could not do that.
In fact, we’ve dealt with that recently with this OmegaCLAW system. We did shift from a system where self-modifications are tried on the fly on the main system to one where it makes a clone of itself and tries out modifications on that clone just to be sure it’s not killing itself with that modification. So you can still do things better or worse, even if you don’t have the stronger guarantees you could have with a system that’s more fully superintelligent.
Liron 1:39:48
So to summarize here, it’s beyond our capabilities right now as human researchers — nobody’s figured this out — how to have a theoretical model where we build this superintelligent AI or just an AGI that’s not even superintelligent yet. We build it and it’s able to modify, it’s able to make successors, which we think it’s going to be able to, but we haven’t figured out the theory of how the successors stably preserve any sort of properties that we care about because it’s so easy in principle for it to just write some other piece of code and we don’t like it, but potentially it’s too late because the other piece of code is also superintelligent.
I think we agree we haven’t theoretically locked it down. You mentioned that it can be locked down at the ASI level — eventually you think it becomes stable, which I actually agree with. I agree that eventually it will become stable, but I think we’re both concerned that maybe in the era where it’s not stable, the first few versions of it that aren’t stable yet, maybe they will stabilize on something that’s actually horrible for us, correct?
Ben 1:41:41
I think the bottleneck is formalizing the relevant aspects of the human world, actually. I think even LLMs are pretty good at working together with Lean or other theorem provers to formalize and prove things. And our candidate AGI architectures are concisely formalized in the case of my own work, messily formalized but still formalizable for transformer neural nets or something.
But we don’t have a tractable formalization of the human world. So a very interesting research program would be to try to make the best possible formal model of the human and natural world we are dealing with, and then try to do the best case we could of formalizing the problem of practical AI systems as they modify their goals and see what will happen.
But that’s not what the world is doing now. Even take the much simpler case of making logic formulations of known mathematics. We know how to do that now, but there’s no Manhattan Project to make a logical formalization of all math. Instead, we have VC-funded companies that are formalizing math but keeping it secret for themselves so they can somehow make money from formalized math.
Josef Urban, a good friend of mine at the AI for Reasoning Institute in Czechia, he’s trying to make an open formalization of all math, and he doesn’t have enough money to pay for the server time to run this code to do it. And this is just formalizing math, which is much easier than formalizing a model of our whole world.
Ben 1:43:00
Steve Omohundro, a good friend of mine — if you haven’t had him on your show, you should.
Liron 1:43:21
Yeah, I’d love to have him on. He hasn’t been on yet.
Ben 1:43:22
He’s a great guy. He proposed a while ago the idea that a path to AI safety would be to replace all of the world’s IT infrastructure with formally verifiable software. His thinking was then you could have a narrow AI theorem proving engine verify that a rogue AI wasn’t suddenly forming.
I’m not so convinced that’s a guarantee against rogue AGI forming. It would, however, be a very good idea for all sorts of reasons. With Mythos and other sorts of LLM cybersecurity risks popping out, it’s more and more clear.
But my point is we’re not doing that very fast either. Formalizing all math or upgrading all of our software infrastructure to formally verifiable software — these things are much simpler than formalizing the human world well enough to prove theorems about the goal preservation under self-modification of early-stage AGIs. But we’re not even doing these much simpler things because we’re spending our resources on multiple replicas of ChatGPT and then on controlling killer robots to take part in various asinine wars around the world. That’s the state of play 2026.
Liron 1:45:00
Yeah, I agree with that. There’s low-hanging fruit of things that can make alignment or safety a little bit better, and we’re not even doing those. In my mind, I think there’s just the much bigger prize that we’re not attacking. But yeah, I mostly agree with what you just said.
Liron 1:45:15
Okay, so we’re coming up on 12:00 PM Pacific. I wanna be respectful of your time. I also wanna make sure that you can talk about your new projects. So you mentioned OmegaCLAW, we’ll talk about that, and we’ll talk about SingularityNet and OpenBGI. Sound good?
SingularityNET, OpenBGI, and the ASI Alliance
Ben 1:45:29
Sounds good. Yes.
Liron 1:45:31
Great. So I know you’ve already discussed OmegaCLAW, but for the viewers, can you tease us with a killer use case or an exciting use case for OmegaCLAW?
Ben 1:45:40
Sure. Actually, let me take a step back and answer a different question, and then I’ll answer that one. You mentioned a bunch of organizations I’m involved with, which are all important, but at the crux of my AI work are two open source software initiatives.
One is HyperON, which is a framework for sort of neural symbolic evolutionary AI. Not just neural nets, but for a variety of different AI paradigms, including uncertain theorem proving, program evolution, and so on. We spent the last three years building a scalable infrastructure for this, which now basically works. So now we’re in the midst of trying to take a couple decades of AGI prototypes and implement them in terms of this scalable infrastructure.
The other open AGI R&D initiative I’m engaged with is trying to push forward predictive coding as an alternative to backpropagation for training deep neural nets, because I think many of the limitations we see with LLMs and other deep neural nets is because of the backpropagation algorithm, which is the key learning algorithm underlying them. There are alternatives that have been explored in academia for a while but just haven’t been scaled up. So we have a new framework called FabricPC, which is sort of the Torch or TensorFlow for predictive coding training of neural networks.
These two open source initiatives underlie all the different organizations that you mentioned. So to get to OmegaCLAW now — this really popped up because I saw OpenCLAW, I thought it was cool, and I’m like, “But this code is a horrible mess.” And I mean, it was a first prototype. It’s allowed to be.
I’m like, “We could code this in a couple hundred lines in our AGI language, and it would be much more elegant and beautiful, and then it could modify itself as it goes, which would be quite interesting.” So Patrick Hammer, who’s one of the AI researchers and developers on SingularityNet’s team, he did that implementation based on my initial suggestions and screwing around.
Ben 1:48:14
What we realized is that the killer functionality we were adding by putting it in our meta-AGI language was memory. Because when you use our AtomSpace symbolic knowledge graph within a CLAW-type system, it just naturally remembers everything it did and everything you said and why it did everything it did.
Having that symbolic memory, which is a memory of what the system was at different points in time, how it changed itself, and all the interactions it had — having that nice symbolic memory there just gives a very different set of behaviors and a very different feel than an OpenCLAW-type system, which is basically an LLM in an agentic loop. It comes out more like interacting with some weird AI dude rather than just a harness. Because it remembers who you are, it remembers all the interactions with you, remembers what it did before and what it messed up and what it succeeded at.
This is still a prototype. It’s in GitHub now — anyone can play with it. But we’re gonna roll out the first official developer version in maybe three weeks from now, and that’ll be a Docker container you can download to run on your own machine. I think a really productized version will be around the end of the summer, which is what I would say will be ready for companies to use to customize for their own purposes.
Ben 1:49:58
Even without any new AI bells and whistles added, what this can be, as one example, is an actual AI personal assistant. Google Assistant is great — it works on my phone, it does things. But it’s not really a personal assistant that knows who you are and knows what you want and remembers what it did for you last week and can do something today consistent with what it did for you last week.
Beyond retail applications, what I would like to have is an intelligent artificial research assistant that can help me carry out a systematic research project over time, which involves various different code bases, various research papers, various notes, and that keeps track of all these in its memory and can then think along with me in carrying out different actions.
So I think OmegaCLAW is a suitable infrastructure both for personal assistance with actual persistent memory and understanding of itself and of you and who you are. And then this has implications for research assistance, for coding assistance, and a whole bunch of other applications.
Initially it’s limited by being based on standard LLMs for a bunch of the capability. We can supplement the LLM with some symbolic reasoning, which we’re already doing. We can supplement with some evolutionary learning for more creativity. We can get a certain distance that way.
I think it’s likely that to make a real breakthrough to human-level AGI, you’ll need to upgrade the LLM also, and that’s what the predictive coding thrust is about. If you can retrain the RL part on top of the pre-trained model using predictive coding rather than backprop, you will then get something that can update its weights in real time as part of this OmegaCLAW interaction. Then you’ll get a real neural symbolic synergy, where the weights in the LLM are adapting to what the CLAW agent is doing during its interaction with you.
This is interesting. It’s the first example in my AGI research career, really, where we built some practical thing, started playing with it, and more capability by far popped out than what I thought I was putting in. We are piggybacking on LLMs, but it’s LLMs put together with symbolic memory and reasoning, put together with the internet that an agentic harness gives you. But then you get more than the sum of the parts, at least subjectively.
This is not something that shows up on benchmarks at the moment necessarily, but neither was ChatGPT. When ChatGPT popped out, it wasn’t the best on any benchmark. It was second or third best on a lot of benchmarks, but it was plainly super cool.
Ben 1:53:39
To get back to the different corporate entities you mentioned, which are not the most interesting thing — I’ve been developing AGI software since 2017 within SingularityNET, and we later merged with a couple other blockchain projects into the ASI Alliance. Our focus there has largely been on building a decentralized infrastructure for AGI, so that once we get to AGI, you can roll it out on a global network of machines without any central owner or controller, which is either necessary and critical for ethical and beneficial AGI or really scary, depending on how much you trust Trump, Putin, and Xi Jinping.
On the other hand, the project in a way is now outgrowing this decentralized infrastructure project, which itself is growing wonderfully. We’re launching a new layer one blockchain, the ASI chain, which uses our meta-AGI language as the smart contract language. So that’s going great.
But we’re finding that we need a lot more compute, we need a lot more human energy. So we’re launching a new project called OpenBGI for Open Beneficial General Intelligence, which is sort of set up the way I think OpenAI should have been set up. We’re setting it up with a for-profit entity and a non-profit entity. We want each of them to have some money and some people working for them, and we’re trying to craft the agreements between them in a way that will not allow — once we do get to AGI — the non-profit benefit part to be squashed and eliminated.
I think for-profit is meaningful and valuable, and the business world is building all this amazing technology. But the open and non-profit aspect also is important and shouldn’t be squashed to zero.
So OpenBGI will be building HyperON technology and predictive coding technology to do enterprise applications and just building out the open source infrastructure. It will be doing all this compatibly with SingularityNET’s decentralized infrastructure. So it will be presumably the biggest user of SingularityNET’s decentralized infrastructure.
But this all has got to be based on open source code. And I think this is the only way we can beat big tech because — while we are gonna be raising money for OpenBGI and SingularityNet already has a flourishing token ecosystem — I don’t aim to raise more money than Anthropic and OpenAI and all this.
What I think is we need to build like Linux has done. We need to build a flourishing open source developer community, and we need to get — yes, we’ll have server farms, but we also have software that can leverage the processors in everyone’s phone and laptop and pull all this into the global compute network.
Liron 1:57:00
For the viewers, you’re talking about OpenBGI. BGI is beneficial general intelligence, correct?
Ben 1:57:05
Beneficial general intelligence. Yeah. It could’ve been BAGI, but “baggy” seemed like a bad acronym.
Liron 1:57:12
It’s not baggy. Okay, it’s just BGI. And just a review — you talked about how this is going to use the infrastructure that you’ve built over the last decade or two with SingularityNET, correct?
Ben 1:57:25
And HyperON and our new predictive coding frameworks. Yeah.
The $ASI Token Economy
Liron 1:57:31
And to catch up the viewers, you mentioned there’s blockchain projects and it has a flourishing ecosystem. So SingularityNET launched the AGI token in 2017 on Ethereum—
Ben 1:57:43
That’s it.
Liron 1:57:43
—and then the token moved to the Cardano chain—
Ben 1:57:47
It’s on both. It’s on Ethereum, Cardano, and Cosmos at the moment, and we will launch our new, our own layer one chain around the end of this year, which is sort of customized for HyperON AGI technology.
Liron 1:58:05
Nice. And just go back to 2021. So then the AGI coin on Ethereum and the AGIX coin on Cardano later merged with two other tokens in 2024 to become ASI. So you’re already recapitulating the whole AGI to ASI—
Ben 1:58:19
Yeah, yeah. Hype goes faster than technology. So building a token called ASI is easier than building the actual ASI.
Liron 1:58:29
Yep. Okay, great. And on CoinMarketCap, as of today, the market cap of ASI is $541 million. How does the token price support the research your team is doing?
Ben 1:58:42
We can incentivize development with tokens. We had the stockpile of tokens that we reserved for development funds when we did the initial coin offering, and we can use those tokens to compensate developers who build code that runs on the platform.
Liron 1:59:07
Gotcha. But now, as you mentioned, you’re kind of continuing along the lines of crypto plus AI because you’re gonna have a new layer one chain, and it’s gonna benefit from the HyperON architecture. Just making sure.
Ben 1:59:19
The new layer one should allow quite a lot because if we deploy early-stage AGI systems on this layer one chain, then let’s say someone gets a subscription to an OpenBGI-based system, similar to what they do to a ChatGPT system now. It will probably be focused more on higher level professional accounts. Say someone pays $500 a month to access an OpenBGI interactive system to help them with coding, math, research, business strategy. Then this system will be running in the back end largely on the ASI chain.
So even if someone paid for that subscription in dollars, on the back end, it will be converted to the ASI tokens. So then you have massive token utility with that ASI token. You have a flourishing ASI tokenomic ecosystem behind the scenes.
I think even if end users are paying in fiat currency for a subscription, if you have an agent economy behind the scenes where someone is paying a subscription for one agent, but then the agent pays some money for another agent, which pays some money for another agent — these inter-agent transactions can be done using the ASI token, which is sort of what it was built for.
When we launched this project in 2017, the notion of an inter-agent economy was kind of fanciful, and nobody paid much attention to it. But now everyone accepts that the future of the economy is AI agents paying AI agents. We have a nice infrastructure for that. We’ve had for a while, and it will be massively upgraded with the ASI chain launch toward the end of the year.
Closing
Liron 2:01:15
Nice. Okay, great. That’s all extremely fascinating. Encourage viewers to dive into that. I forgot to ask when you were talking about the Mindplex and kind of co-evolving with AIs or socializing them — adapting to them. I feel like you talk a lot about that. Have you looked into Softmax? Do you think your view is similar to those guys?
Ben 2:01:34
No, I don’t, no.
Liron 2:01:37
Yeah, I definitely recommend checking out softmax.com because I’m noticing a lot of similarities in both of your general approaches. I think they have Ken Stanley as an advisor, and they have Emmett Shear.
Ben 2:01:47
I know Ken Stanley reasonably well, and his work on NEAT and so forth is great. He’s working for a different company now — Lila Biosciences or something.
Liron 2:02:05
Yeah, with Ken Stanley, I just saw him as one of the advisors. I don’t know the details there, but they’ve also mentioned they’ve been inspired by his writing. And by the way, if viewers search “Doom Debates Ken Stanley,” he actually came on the show and we had a really great debate.
Ben 2:02:17
He’s great. From the name Softmax, I would assume they’re neural net focused, whereas my technical approach puts neural nets together with logical inference, evolutionary learning, and other methods in an integrated architecture, rather than being solely neural. Because of course, softmax is a function used inside artificial neurons. But I will check out that project. It’s an amazing time when there’s more cool AI projects out there than I can keep up with.
Liron 2:02:53
That’s right, because you certainly were around before that was the case.
Ben 2:02:56
Yeah.
Liron 2:02:56
Ben Goertzel, you’re such a fascinating guy, and I really appreciate that you came on the show to have this discussion. Talking to people like you is really why I started Doom Debates. Thanks so much.
Ben 2:03:05
Yeah, thanks for having me.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
