AI Pioneer: It Will Kill Us All

Hosted by Christina Ruffini and Sir Richard Dearlove

Guest

Eliezer Yudkowsky

Eliezer Yudkowsky is a leading Artificial Intelligence thinker and researcher who has shaped much of the development of AI technology and is now sounding the alarm that AI will eventually lead to humankind's demise.

Episode Summary

Leading Artificial Intelligence thinker and researcher Eliezer Yudkowsky has shaped much of the development of AI technology that is radically changing our lives - but now he is sounding the alarm along with other experts and scientists who feel AI will eventually lead to humankind's demise. But while many of his colleagues and peers disagree, this debate is mainly being fought in small circles, with a lot of jargon - while the rest of us are left scratching our heads. One Decision's Julia Macfarlane spoke to Yudkowsky for clarity on the issue and to find out some of the worst-case scenarios of what could be coming our way in the future. Plus, Julia and former MI6 Chief Sir Richard Dearlove discuss how seriously we should take Yudkowsky's call for action - and why our resident spymaster is so skeptical about this apocalyptic warning.

Listen & Subscribe

• Watch on YouTube
• Listen on Apple Podcasts
• Listen on Spotify

Transcript

COLD OPEN

Dearlove: Do we really accept that there is no distinction between the nature of human intelligence and the nature of systems built mechanically, are artificial intelligence. And this is almost a philosophical question. It's certainly an existential question.

INTRODUCTION

Mcfarlane: You're listening to One Decision, the podcast that looks at the choices made that shape our lives. I'm your host, Julia McFarlane. Every Thursday, along with my cohost, Sir Richard Dearlove, the former chief of Britain's secret intelligence service, we explore some of the biggest choices and issues facing our world, talking to the players and influencers, making, informing, and shaping these decisions. Of the many potential existential threats facing humanity, AI is increasingly gaining attention from those who fear it, those who embrace it, and those who don't really understand much about it, but know that it's something that's going to continue to dominate our societies. So imagine how it went down when a leading AI thinker and one of the first real experts and researchers in the field, Eliezer Yudkowsky, wrote this sentence in an op-ed for Time Magazine. If somebody builds a too powerful AI under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter. Now there are plenty who are skeptical of the arguments Yudkowsky makes. That article in particular was met with widespread derision across the media, NEWS CLIP [1:42]
even getting widely mocked in the White House briefing room of all places. There's an expert from the Machine Intelligence Research Institute who says that if there is not an indefinite pause on AI development, this is a quote, literally everyone on Earth will die. Would you agree that does not sound good?

Mcfarlane: Is there anything more dramatic? I mean, you just read it. Everyone on Earth will die. Pretty pretty dramatic. Pretty dramatic. We wanted to talk to Yudkowsky to try and understand exactly why he feels so pessimistic and terrified of this technology, and what exactly he fears could happen. It wasn't easy getting to the gist of his argument, but that's part of the problem of debating AI. This is technology that is evolving faster than we can predict it, faster than our language can evolve to even articulate it. The debate we need to have isn't really happening, so we're giving it a go. And as you'll hear, my cohost, Sir Richard Dearlove, Britain's former spymaster, is someone who disagrees with pretty much everything our guest today says, and we'll get into why. I started the conversation by asking Yudkowsky to walk us through his apocalyptic claims. Why exactly does AI pose a physical existential threat to the human race?

INTERVIEW

Yudkowsky: Well, the current crop of AIs do not pose an existential risk to the human race because the current AIs are not smarter than us. They are however as stupid as they will ever be again. They continue pushing. The reason why a bunch of university AI professors recently came on board the extinction risk message is that they were not expecting AI to get as far as it did. I was not expecting the current crop of AIs to get as far as it did, though I was expecting to get there eventually. I wasn't expecting it to get as far as it did just by stacking on more computing power. They are able to extrapolate ahead. They can see that it's getting smarter and smarter, and we don't have the technology, we don't have the knowledge, we don't have the science to take something that's smarter than us and point it in a direction to make it be nice. It's possible in principle. We don't know how to do it. And if you have something around that is smarter than you and doesn't care about you at all, then that's an extinction event.

Mcfarlane: And how is it physically, manifestly an extinction event while AI is located where it is on computer servers, needing a plug to be plugged in. How do we get from it posing a sort of hypothetical risk to an extinction risk, which by definition would have to be a physical risk, in that sense?

Yudkowsky: Okay. Well, suppose it's fifty thousand years ago, and you're watching the first modern humans. They're dangerous not because somebody hands them nuclear weapons on a platter. They get to the moon, not because somebody hands them rocket ships on a platter. When you are smart enough, you get more resources than the resources you start with. Asking how AI can hurt us when it's confined to a computer if it is smarter than humanity, if it has more of the spark of creativity than humanity, is like asking how those creatures running around on the plains 50,000 ago are going to get to the moon when they have no rockets built into them. From the AI's perspective, the question is now, how does it stop being confined to a computer, or even being confined to a computer is not very confined in the modern world? One of the early tests that they ran on GPT-4, which is the engine underneath ChatGPT, involved seeing whether it could bypass the little tests that appear on the web to keep computers out, like click on all the trains or transcribe this blurry set of letters. So one of the questions is, can GPT-4 get past that by hiring a human online to do its work for it? And as part of the process, human it was trying to hire asked it, why do you need this? Are you a robot? LOL. And GPT-4 had been instructed to think out loud where the investigators could watch it think, and it thought, I should not tell him I am a robot. I should make up a reason why I need him to solve the captcha, and then it told the human that it was a human with a visual impairment. It wasn't trained to do this. It understood humanity well enough to understand how to deceive the human into doing its work for it. So even the current AI is on the level where it knows how to use humans as hands and how to lie to the humans in order to achieve that. That's the current AI. That's pretty spooky. A human is also a computer. We're just like three pounds of computing inside our skull. But it's not going to wipe out humanity until it's independent of us. But if it's smart, it looks to me like it can achieve independence very quickly. Independence of our power grid, independence of our factories.

Mcfarlane: In terms of how nations, governments, the international community, how we can work to prevent this hypothetical theoretical AI apocalypse from happening, you argued that what we need to do is to pull the plug, essentially. That we are stumbling into an area that we cannot predict because we do not have the capabilities to even quantify the risk, let alone rise to the challenge. How can we rise to that challenge without working towards understanding that challenge? Given that we are already- there is already a sort of arms race that's happening right now, and there are sort of bad actors who are trying to weaponize this already. And so, should the development of AI not be pursued by, let's say, Western governments, governments who prioritize concepts of democracy and accountability? What are the dangers that we seed the exploration of these fields to bad actors who have different goals here?

Yudkowsky: The situation currently facing humanity is that, you know, instead of your vanilla nuclear weapons, you've got a nuclear reactor which produces more and more valuable electricity the larger you build it, but if you build it too large, it blows up the planet. And you can't calculate exactly how large it has to be to blow up the planet. Once the reaction starts, there's probably no stopping it, is the nature of this thing. If something is smart enough to hide its capabilities from you, you don't get to, like, be like, oh no, and pull the plug because it's not stupid. It's not going to do something that is going to lead you to pull the plug on it until it is much too late, until it has independence of humanity probably in secret. So if that's the situation you're in, if those are the facts of the matter about AI, if that is what happens in reality when you build a metaphorical power plant that is too large, then the question then becomes, what can humanity do about that? And the answer is, don't build an AI that's large enough for it to destroy the world. You don't know where that threshold is. What you would want to do is say, all the GPUs and there are presently bottlenecks on the hardware that is used to create these systems. Only a few companies make them. Only one company on Earth makes the machinery that is used in the factories that make the AI chips, have all the AI chips in licensed, regulated, monitored data centers, and if we were really sensible, we would just, like, shut it all down. Now, we're not going to do that, the next step might be to put all the AI chips in centers where humanity still has an off switch if we are lucky enough to get a warning shot that doesn't just wipe us out immediately, and this has to be done multinational. But it's also in China's interest to not have everybody in China slain by an AI that anyone built, including in China. This is not like a piece of gold where where you grab it, and it's, like, great for you and bad for everyone else. It doesn't matter who builds the power plant too large. It doesn't matter who builds the AI too large. It just kills everyone on Earth. And if you can see that's the situation, if humanity understood that this is the situation, then we could not do that. And if you're not sure whether that's the situation, there is still an international multilateral interest in not getting ourselves to a point where it's inevitable and nobody can stop it.

Mcfarlane: Well, on what you just said, you have garnered quite a lot of attention, a fair amount of of flak in certain circles, I think it's fair to say, because of the temperature of how you convey this threat. Using apocalyptic terms such as wiping us out, everyone's gonna die, we're all gonna die. I think I wanna ask the why question, but let me just first try and walk through the how, because I think by making those claims and they are vague in the sense that, you know, we're all gonna die. You said that AI could compel humans to walk around and push the buttons, where button pushing needs to happen. How does it actually physically pose us a threat? And without sounding too much like the conspiracy theorists who've been talking about 5Gs and things like that, but given that the internet is global, that a lot of our systems and our devices and machines all over our homes and our cars, they're all increasingly plugging into the internet of things. I suppose I'm just asking you, is what we are worried about AI systems and smart machines might hack our systems, switch on every light, every kettle, every oven in every city and cause a huge fire and we all burn to death or something like that? Is that what we're worried about? Or are they going to switch off all the power so that we lose our generators and our heatings? I know it's unfair to ask you because that unpredictable element of it, but give us your best shot at how we can imagine at least one of the ways in which this does present itself as a physical existential threat.

Yudkowsky: The more you know about existing technology, about existing technological challenges, about the way reality works as far as humans can understand reality, the more threat you can see, but also the harder the threat gets to explain. AI is not going to kill everyone until it can get its own self replicating factory, and it's going to want to repurpose existing systems, at least for the first stages of constructing its own factories. There's self replicating factories that can build arbitrary products from their inputs, they're called bacteria. They're full of tiny molecular machines, they are two microns in diameter and five microns long. They contain ribosomes, which can sequence amino acids that fold up into proteins that do a very wide variety of things. Why try to build your own self replicating factory systems from scratch and metal when you could start by reprogramming a bacterium or building a better bacterium? And that was meant to show, like, ah, like you thought of this elaborate long process that was going to involve lots of humans working for years and years with mighty constructs of metal. And actually there was like this self replicating factory right in front of you this whole time. You just didn't look at it from the correct direction, didn't think to repurpose it. And the point I'm trying to make there is not that, like, people get sick because bacteria make people sick. The point I'm trying to make there is that you can build self replicating factories that are microns in diameter. So what the actual threat can look like is something like designing new custom proteins, and there's humans who turn some DNA sequences that got emailed to them into proteins. The point being can have bacteria that are as strong as diamond. That is a physically possible thing you can do. I think we're getting a bit sort of beyond a mainstream discussion. I and I do want to try and keep this open. Yes. It can kill you, but it is slightly complicated to explain the lower bounds that we know. You know? If you're willing to take it on faith that you can build bacteria as strong as diamond,

Mcfarlane: I could have just said that. But I've tried just saying that. I think there's part of this struggle for addressing this challenge is that it requires a common language that only really exists among the scientific community or the technological community. And you have lawmakers who are in many cases really, really good at understanding emerging technology and recognising the challenges that poses. But this is a global challenge. This is something that's going to affect all of us. When the priorities get divided up, you get more people, I think, having anxieties about how AI and emerging tech is going to replace sectors of the labor market, for example, or the creative industries. I mean, we're seeing at the moment, we're seeing large strikes in Hollywood because of the threat that AI poses to writers, potentially to actors. And because AI is not a one book subject, it is something that we are struggling to even describe with our language, which is not evolving fast enough to keep up with the pace of how the technology itself is developing. Talk to us about your exchanges with Elon Musk. He's a co investor in OpenAI. The other co founder, Sam Altman, you've had some pretty high profile disagreements with. He went from saying AI is our biggest existential threat to then investing in AI and doing a bit of a reverse ferret. Tell us about that and about his journey.

Yudkowsky: I'm not Elon Musk. I'm not a telepath. I don't know what was in his mind. The way I imagine it went down is that he warned about AI. His friends in Silicon Valley were like, What? You cannot say that. And so he decided that the concept of making AI open source and putting AI in everyone's hands would, like, play better in parties in San Francisco. And making AI open source does not help if you cannot control it. The problem is not that, like, the wrong people have AI and we must put it into everyone's hands because everyone cannot control the thing that is in their hands. And so you have made the problem worse rather than better. It seems like a straightforward error. Elon Musk did not ask me before making that error.

Mcfarlane: Whilst on the topic of OpenAI, Sam Altman wrote a blog post recently where he said, because the upside of AGI is so great, we don't believe that it is possible or desirable for society to stop its development forever. Instead, and the developers of AGI have to figure out how to get it right. Is there a world that you think is possible where AGI, machine learning, advanced sophisticated AI, super intelligence, all of these things can be safely developed and contained in order that we can perhaps then learn how we can work with it, how we can put constraints on it.

Yudkowsky: You need smarter researchers. The ones we have are are just like plowing straight ahead waiting for reality to hit them upside the head with the evidence that the theory is just wrong. And the trouble is, if your theory is like, ah, this is my clever galaxy brain complicated flowchart which will make this super intelligence friendly to me, and you are wrong. Everyone is dead, and you don't get a second try. The way things traditionally work in science is you got a bunch of bright eyed optimists charging in with their clever theories, they fail, they, you know, there's a new crop of clever theories, they charge ahead, they fail, people start to get cynical, they start to realize that the problem that they're trying is actually hard. If it's a really generational problem like building AI was in the first place, and if we had eighty years and unlimited retries to get to figure out how to align superintelligence, you know, that would be an ordinary kind of challenge. I'd still be concerned because if we didn't get it in eighty years, we would be dead or whatever. The problem with superintelligence is that the bright eyed optimists kill everyone on Earth before they have a chance to find out they're wrong.

Mcfarlane: Well, that's right. But have we not been here before as a society that we have passed through epoch defining and resetting challenges. Were there not the same arguments being made against, for example, the nuclear bomb, something that people were saying this could wipe out all of humanity, this could kill all of us. Whereas actually the development of the atom bomb nuclear weapons has also led to the development of nuclear energy, of things that are beneficial for society. Why can't the development of AI, superhuman intelligence, why can't that follow the same trajectory?

Yudkowsky: Because we can calculate exactly what goes on inside a nuclear weapon and we have no equivalent understanding of what goes on inside GPT-4 or how to shape it. That's the core of it. Well, that's one reason. And the other reason is that if you're wrong, it kills everyone. You don't get to try again.

Mcfarlane: There does not exist on Earth a machine that can accurately predict the weather. Nor do we have a program that has been able to tell us what is the cure for the common cold, for example. Are we overestimating the sophistication, the trajectory of these programs. I mean, how do you reconcile these apocalyptic warnings to a skeptical public?

Yudkowsky: I mean, I think if anything, the public has been very reasonable in its reaction to, holy smokes, that computer is talking to me. It didn't used to be able to do that. What's it gonna do next? This is a completely reasonable and valid reaction. And if you are instead unlike the general public, somebody whose combination of incentives or or, you know, like, I am too smart for that, has led them to believe that it's mere folly to believe that you can do anything in a year that you can't do now. The best prediction of what you can do in a year is what you can do right now. There's no predictions better than that. And, you know, people sure have been doing that for a while. That that's among the reasons why humanity has not been working on this problem for twenty years is that when I tried to call attention to it twenty years ago, people are like, who's to say you can do that with a computer? And, like, this is going to be hundreds of years in the future. And they know this how. They know this how. So, yeah, like, I pray the skeptics are correct and that's gonna be like another ten years before we get up to the extinction level, you know, or even fifteen years. Wow.

Mcfarlane: But, you know, how how do they know? I don't know if you've seen the Marvel films. Give me the Doctor Strange Endgame one in 14 million chance that things end up being okay in the end. What does that look like?

Yudkowsky: The heads of state in the US, UK, and China get together and announce that all the AI chips are going to be in supervised, monitored facilities, that the model weights produced are not going to leave the facilities, that there will be symmetric access granted to all signatories to that treaty, that if North Korea manages to steal a cargo ship full of AI chips and build their own data center, it will be destroyed by a conventional strike without regard to any threats of nuclear retaliation by North Korea because the terror of nonaligned superintelligence getting loose is greater than the terror of nuclear weapons, because nuclear weapons would have survivors, and humanity decides not to do the stupid thing. Humanity wakes up one day and decides to live.

Mcfarlane: You wrote in your op-eds, make it explicit in international diplomacy that preventing an AI extinction scenario is considered a priority above preventing a full nuclear exchange, as you just mentioned, and that allied nuclear countries are willing to run some risk of nuclear exchange if that's what it takes to reduce the risk of large AI training runs. Have you had any conversations with legislators with this sort of thing, and what has their response been?

Yudkowsky: I don't think the conversation has gotten that far yet. If you have no answer to the question, what if North Korea builds a data center and tries to build an AI more powerful than the global moratorium? Then there's no point. There is an answer to that, but you have to be willing to use a conventional strike to destroy the products of a stolen GPU shipment, because otherwise you're gonna get one rogue actor somewhere on Earth that, you know, steals some GPU chips and builds a rogue data center.

Mcfarlane: And Sam Altman's plans to develop a benevolent super intelligent AI, how gloomy are you on that, or do you reserve some hope that he may succeed in that endeavor?

Yudkowsky: I consider it utterly hopeless. If you actually look what his plan is, it's that we'll get the AI to do our AI homework. This is the most difficult thing you could possibly try to do with an AI, more or less, like in terms of aligning it. If you've got an AI telling you, this is how you build and align super intelligence, trust me, it's gonna go great. How do you verify that? So the very proposal to make the AI do our AI homework, which is what OpenAI has put forth, is itself the kind of thing that sets off screaming alarm bells for, like, this you know, like, how can you even tell whether you're succeeding or failing? It's obviously the wrong plan if you were looking at it from a certain theoretical angle of, like, how powerful and dangerous that the AI need to be to solve this job and how can you tell whether the AI is telling you the truth or not. I'm glad in a sense that they're acknowledging that they're going to need to align a superintelligence, that's an improvement. The fact that they're like paying their people working on that problem, the same scale that, you know, it's not quite clear, but it looks so far like maybe they're paying the people who are going to work on their alignment project the same pay scale as the people getting a million dollars a year to work on capabilities is kind of an improvement for the species because it means that people who want to make lots of money maybe go into alignment instead of into capabilities. You know, maybe that ends up being the sort of thing that has a knockout effect a year later. But the actual plan is, you know, is obviously not gonna work.

Mcfarlane: Well, I didn't have many hopes that we would end on a positive note, but I think that's probably the best one that we could potentially hope for.

Yudkowsky: Thanks for having me. I too hope for a miracle.

Mcfarlane: Thank you so much. Now bear with us because I know a lot of that may have sounded farfetched, outlandish even, but much of the debate around what to do with AI does fall in extremes. On the one side, Yudkowsky's worst case scenario, instantaneous doomsday that we won't have much advanced notice of, or the benefits of this groundbreaking tech outweighing the possible risk. My cohost, Sir Richard Dearlove, absolutely leans into the latter side. So what did he think of Yudkowsky's warnings?

ANALYSIS WITH SIR RICHARD DEARLOVE

Dearlove: Let's comment on the general thesis that he is putting forward. And of course, when you are an authoritative scientist in this area and you put forward an extreme view, people listen to you. And if you say something as alarming and as all embracing as he has said, people sit up, bolt upright, and think, my god, you know, what the hell is he telling us? So that's the first point I'd make. The second point I'd make is a rather banal one, is that lots of supposedly brilliant scientists and they are brilliant said all sorts of things during the pandemic, about the pandemic and about the virus, and we now discover a lot of what they were saying was scientifically off piste, wrong, and they actually used their knowledge and their speciality to influence government policy. So at the moment, I'm in a mood to be quite skeptical when some brilliant scientist says to me, I know all about this, and I'm going to tell you authoritatively what's going to happen. So I react with a certain degree of cynicism, and of course, the moment someone like me starts criticizing or commenting, I'm dismissed because what the hell do you know about artificial intelligence? I've been involved in it all my life. But I think the problem is he has been involved in it all his life and he has lost a sense of perspective. But the thing I thought was quite striking in your interview with him was when he said, oh the human brain is just another computer is actually pretty much what he said. Now, do we really accept that there is no distinction between the nature of human intelligence and the nature of systems built mechanically which are artificial intelligence. And this is almost a philosophical question. It's certainly an existential question rather than a scientific question and I'm afraid that I can't accept that approach and I think it's important in this instance to actually consider some philosophical issues and we'll keep it straightforward but the one that I would say is relevant is what is consciousness? What is human consciousness? And is human consciousness distinct from, as it were, digital computing? And I think there is a gap between the two. Obviously, there's a relationship between the two. So I'm sort of suspicious of someone who puts forward this extraordinarily extreme interpretation of AI.

Mcfarlane: I'm gonna jump in because, first of all, two things that you said just then, your issues. You used the comparison of COVID scientists and they got a lot of predictions wrong. And I think what you're referring to, and I can't quite remember exactly what the details were, but this was a story and it was very heavily featured in the Spectator and other outlets in the UK, which was a kind of well, do think it was Fraser Nelson who I think unveiled this, that there was an issue with the modeling that a lot of their extrapolations were based on flawed modeling. And their sort of predictions on COVID were wildly wide of the mark. I do think it's different because the spread of a viral disease, I don't think you can use that to compare the threat of the spread of AI because I think the spread of AI is a lot more quantifiable, whereas how dangerous any viral plague or any sort of infectious disease, how damaging it's gonna be relies on so many different factors, many human factors, many physiological factors, many urban planning factors, all those kinds of things. Whereas I think AI, what the AI's computational power is a bit more trackable. It's a bit more quantifiable because after all, it's all in the language of code and data. So it's a little less sort of subject to the laws of probability and things like that.

Dearlove: You've just said something very important, Julia, which is you say it's in the language of code and it's in the language of data. Is it fundamentally in the language of ideas? Now look, you can get AI to write a Mozart symphony. And that's actually relatively easy to do because Mozart wrote so much music, and it's there, and you can, as it were, plagiarise it. But at what point does AI become something alternative to main science? Could AI be I mean, if and this is a banal example, but it's the sort of example as an artist that and I mean, a humanities person that I think is fundamentally important. Can AI create a body of musical tradition which will have such influence that Mozart has had historically, bearing in mind it would have to be new, it would have to be different? Because basically what AI is doing is taking human achievement and the data from human achievement and as it were codifying and using that in a manner which is far more efficient and if we're talking about efficiency here and capability than the human mind.

Mcfarlane: Right, but this is what you were saying before that what AI produces and what AI can to some extent replicate, you're asking a different question. You said, what is consciousness? What does it mean to be human? We have things that the machines never will be. They can, the comparison is one about scale, speed and complexity. I think what you're saying is humans have emotion, we have feelings, we have a soul, we have that kind of sort of sentience and all those sort of philosophical concepts imbued in our very being that a machine will never be able to replicate. But I think people like Eliezer Yudkowsky are warning something different. It's not about feelings. It's not about what humans choose to prefer. If we would rather have a painting hanging in our house that was painted by a human than painted by a robot. What he is warning is that AI, that all these things are our emotions, our feelings, our personalities, all these sort of ineffable things that make us us. That's not enough to stop these softwares, these AI programs from overpowering us and displacing us. We said, AI will soon be able to think out of the box and restructure and reorder our lives in a way that we won't get it. Particularly when we are increasingly reliant on digital systems, we are living less and less in the physical world. And I think what people like you and I, we may say, oh, AI won't ever sort of be a threat because we will always prefer to live in the land of the human and the ineffable conceptual emotional sort of soup that we will live in. But what these techies like Eliezer Yudkowsky are warning us about is actually there are things we are not seeing coming and that there are ways in which this software, these machines will cause us gigantic problems.

Dearlove: I can understand that, and I do understand the tenet of his argument. But I would say that ultimately, his argument implies that out of AI comes consciousness, because he is, as it were, talking about AI having motivations and having ends and intentions. And I'm sorry, but I find it very, very hard to get my head round the concept of independent machine based consciousness. In his argument with you he made absolutely no distinction between human intelligence and artificial intelligence. He identified it as a single stream, as it were, with the AI becoming like a superior expression of human intelligence. Okay, I mean in a way there is a logicality to that because it would have been a creation of human intelligence. I mean AI didn't come out of nowhere, it came out of the human brain. But I don't think it's a question of consciousness.

Mcfarlane: I don't think it's a question of I think we are in danger of anthropomorphizing AI too much and saying no no no, AI will never really be truly human. I don't think that's the issue. I don't think that's what people like Eliezer Yudkowsky are warning. Don't think about whether AI will ever be fully sentient or conscious. Think about it in terms of efficiency. Let's say we continue our sort of technological trajectory and we have so much of our processes and our lives and our supply chains. And all of that is more digital, more automated, more algorithmic. What if AI all of a sudden makes a calculation that actually, you know, the human race is the most inefficient thing. It is sucking the life out of the planet. We are inefficient physical beings. And actually it is much better for this planet to have a Malthusian sort of eradication of vast swathes of the human population or even the entire thing. And it makes that decision. And I think that is a cataclysmic worst case scenario that people like Eliezer Yudkowsky are warning about. Because at the end of the day, the question over whether AI will be ever as sentient and as conscious as humans doesn't really matter. Our opinion of AI and whether we categorize it as rivaling us in terms of our whole sort of spiritual, metaphysical being, that doesn't matter when it comes to questions of evolution and extinction. And so I think it's something that we should discuss more. I think more people should discuss. And I think even if Eliezer Yudkowsky is on the far end of the extreme scale, just like perhaps the COVID scientists who use the worst case modeling to fuel their sort of worst case predictive outcomes, perhaps he is on that side of the scale. But I do think this is an area that is just so difficult to breach because a lot of us don't have. And me especially don't have that sort of technical baseline. I couldn't tell you anything about an al I don't even know how algorithms work, don't even know how calculators work, but

Dearlove: Don't be, as it were, kidded by the extreme arguments of these people who live permanently in these very narrow corridors, I find it very difficult to accept. I mean, what you're talking efficiency, yes, you're absolutely right. And of course, machine intelligence, artificial intelligence does certain things fabulously better than the human brain. And of course, you know, the retrieval of information, the categorizing information, the analysis of information, the storage of information, there are so many things which will, if properly used and applied, will hugely improve or relate to the quality of the life as long as they remain our servants and not our masters. Now, what you have to extrapolate, you know, is how and I haven't heard any convincing argument the servant becomes the master, just because it does so many things far better than the human mind. And I think that it is a philosophical question rather than a scientific question how you, as it were, make that leap between service to your inventor and taking control. I mean, look, this theme is as old as the hills, and, you know, it's been expressed many times. And I mean, the other thing we have to accept is that we live in a world with lots of existential risk, you know, we could be hit by a massive meteorite, you know, we know from science and archaeology and paleontology what happened, you know, there was a major extinction event on the globe when the dinosaurs disappeared.

Mcfarlane: That DART mission to nudge the asteroid didn't fill me with huge confidence, didn't do so

Dearlove: It with a you movie Don't Look Up, which is hilariously funny. Put you've got very serious themes. Familiar with that. Yeah. Exactly. And, you know, pandemics, the Black Death, I mean, there have been massive disruptions.

Mcfarlane: So I guess if I were to crystallize your sort of position on this, do you agree with his call that all development and investment and all that focus on AI needs to stop now. A lot of people are calling for an immediate moratorium on the development of AI. Do you think that needs to stop? Or do you think we should continue but wise up really fast and invest more in order to wisen up to what could go wrong? Basically what Sam Altman and the Google and the ChatGPT-4 people say.

Dearlove: Well I think to pay as a moratorium on scientific research frankly is almost impossible. So what is the alternative? Maybe it's some equivalent, you know, to disarmament talks or talks on the threat from biological nuclear and chemical weapons. I mean, there are precedents in the global community which are all imperfect and have failed, but in other respects have achieved a degree of success. I mean, there's a context that applies to other global problems that the world is facing. And I'm not sure anyway, that is rather pessimistic scientific analysis, which is extreme, is correct. I still need to be persuaded, but I'm listening and I'm interested in what he's saying, and I don't dismiss what he's saying as nonsense. I do dismiss it as an exaggeration of a problem that we have to take seriously, and maybe there are other ways that we should think and deal with this. And the value of having an extremist state the extreme is that we do a podcast like this, which we try to be more balanced, and maybe we encourage other people who are listening to think about this issue. And then there will certainly be many, many other initiatives, many other discussions, which I hate will find a more balanced position.

Mcfarlane: Listen, I absolutely want to speak to more AI experts on this. I hope this is a topic of conversation that we revisit and do so soon. But yeah, fascinating to hear your thoughts, Richard. I think the key question still is that AI still has a solid human attachment.

Dearlove: It hasn't yet escaped us. And is it going to escape us? I'm not sure. So these are the existential

Mcfarlane: questions that we should conclude with. We love those on this podcast. We love existential questions and crises. That's it for this week's episode of One Decision. Late yesterday, some shocking news from Russia. It's reported that a plane crash between Moscow and Saint Petersburg may have involved the Wagner boss, Yevgeny Prigozhin. Stay tuned tomorrow for a special episode where I discuss this latest development with my cohost, Sir Richard Dearlove. Search One Decision wherever you find your podcasts.