Hosted by John Raine and Sir Richard Dearlove
Guest
Sebastian Mallaby
Sebastian Mallaby is Senior Fellow at the Council on Foreign Relations and author of The Infinity Machine, a biography of DeepMind co-founder Demis Hassabis.
Episode Summary
#OpenAI #AI #ArtificialIntelligence
Thank you to Incogni for sponsoring. Get 60% off the annual plan with code onedecision at https://incogni.com/onedecision
OpenAI says it will burn $280 billion by 2030. Sebastian Mallaby gives it a 50/50 chance of running out of money by next summer. So what happens when the world's most prominent AI lab goes bust — and does the US government step in?
Sebastian Mallaby — Senior Fellow at the Council on Foreign Relations and author of The Infinity Machine, a biography of DeepMind co-founder Demis Hassabis — joins Sir Richard Dearlove and John Raine. They discuss the financial fragility of the leading AI labs, why open-weight AI models are too dangerous to permit, how a US-China non-proliferation deal for AI might work, and what a war game about nuclear submarines revealed about where this is all heading.
In this episode:
(00:00) AI Race Between US and China
(02:08) Sebastian Mallaby on AI Safety
(02:51) Is AI a Blessing or Curse?
(04:03) Who to Trust on AI Safety
(05:35) Demis Hassabis and AI Regulation
(08:23) Cold War Lessons for AI Safety
(11:42) Trump Administration AI Policy
(14:59) Private AI Safety Monitoring
(20:57) Open Weight AI Models Debate
(23:00) Does AI Replicate Consciousness?
(33:24) Is AI Investment a Bubble?
(45:39) AI War Game Nuclear Submarine Scenario
(54:37) The One Decision Open Weight AI
Hosted by Sir Richard Dearlove (former MI6 Chief) and guest co-host John Raine (International Institute for Strategic Studies).
Listen & Subscribe
- Watch on YouTube
- Listen on Apple Podcasts
- Listen on Spotify
Transcript
2026-10-15 - Sebastian Mallaby
Cold Open
MALLABY [00:00:00]
If the United States were to agree to a sort of G2 deal with China and say, here is the new non-proliferation regime for AI and here are the rules, European players would just have to go along. Whether you're talking about the French AI company Mistral or the DeepMind team in London, any of these players want access to the American market. They want to raise capital in America. They don't want to be on the wrong side of the US government. And so they are going to do what the Americans say. You can imagine if you just need China and the US to agree, the world is going to have to agree.
Introduction
RAINE [00:00:37]
Hello, and welcome to One Decision. Conversations about intelligence, statecraft, and the forces that drive world politics with people who have been in the room and on the ground. I'm your co-host, John Raine.
DEARLOVE [00:mm:39]
And I'm Sir Richard Dearlove, the former chief of MI6. Hi, John. Good to see you again.
RAINE [00:00:54]
I'm very good to see you too, Richard. We have a fantastic and challenging interview for you today. To keep up with all of our content, subscribe wherever you get your podcasts. And we would be grateful if you could like this episode or drop us a comment on YouTube. That will help others find the show. Two additional brief announcements. We have a newly streamlined website at onedecision.com. Go there for articles, explainers, and to sign up for our newsletter.
— SPONSOR READ —
Secondly, our show this month is supported by Incogni. Data brokers and identity thieves can't harm you if they can't find you. Get 60% off the annual plan with code OneDecision at incogni.com/onedecision. That's incogni.com/onedecision.
— END SPONSOR READ —
Now it's time for our interview. The AI labs say that their technology could threaten humanity and they want to slow down. Their critics say the labs are stoking fear to shut out competitors and enrich themselves. And that slowing down would hand China the lead. Yet, some of those same critics are helping China's AI development by selling it advanced chips. Who in the safety debate is actually looking out for our safety? Sebastian Mallaby joins us to sort through the motives, the money, and the race with China. Sebastian is senior fellow at the Council on Foreign Relations and the author of The Infinity Machine, a biography of DeepMind co-founder Demis Hassabis. Sebastian, welcome to One Decision.
Now, just to set the scene, Sebastian, I'm going to ask you a question which is fairly blunt, but I'm going to ask it because there are so many voices at the moment on AI from all sorts of different lobby groups, those foreseeing apocalypse, those predicting the most optimistic future ever for humankind. And the question is this, Sebastian, in your view, is AI a blessing or a curse? And if it's a curse, what's the worst that could happen?
Blessing or Curse?
MALLABY [00:03:02]
It may be a slightly frustrating answer, but the truth of the matter is that anybody who thinks about this agrees that it's both a blessing and a curse, right? Because there's big upsides in terms of empowering people to discover new information, making us more productive, unlocking medical or scientific advances. I think most people would agree that's good. There's also risk. If this technology falls into the hands of bad actors, they can use it for cyber attacks, they can use it for potentially bioattacks, which would be potentially more serious. And so the downsides are real, they have to be managed. In some sense, your question is academic. I quite like academic, actually, but you know, it's fun. But it is a thought experiment rather than a practical question, because the technology is coming and I don't think we can stop it.
Who to Trust on AI Safety
RAINE [00:03:53]
Perhaps Sebastian, in light of that, which I'm sure is right, as it comes towards us, which voices in the chorus should we trust? Certainly when it comes to safety. Who is it who really speaks out with an independent voice about the safety of AI and who offers reassurance? Because as you yourself have said, there are lots and lots of motives in play here for all the different actors, commercial motives, some interested in power, others interested in scientific discovery. But who are the voices we should listen to?
MALLABY [00:04:29]
I would start with the UK AI Security Institute. That's a pretty serious body set up during the Rishi Sunak administration, end of 2023 when there was the Bletchley summit on AI safety in the UK. And I think they're a respected body, they don't have a commercial dog in the race. They're simply reporting on the potential risks in these AI models. So I think that's a good place to go.
I think it's also useful actually that it's British, because even if you look at independent AI monitoring agencies within Silicon Valley, you know, there's Redwood, there's METR, and there's various of these groups that are very expert. The question people ask about them, and I don't know for sure what the answer is, but it is a fair question is, given the sort of tribal nature of the connections in San Francisco and Silicon Valley, is it possible to be truly standing aside from these intense, passionate debates when you're embedded in that ecosystem?
RAINE [00:05:37]
Are there individuals who you think speak with particular authority on this? Of course, you've written an enormously successful biography of Demis Hassabis. He seems to fall into that category. Do you think that those who are driven, as you've put it, by the joy of scientific discovery, have a particular status in this debate?
MALLABY [00:05:56]
I do find Demis Hassabis, the subject of my book, a very compelling and sort of truth-telling figure. I mean, all human beings have some distortions and some frailties. You know, in his case, he definitely wants to be seen as the leader of bringing AI into the world. He is massively competitive and ambitious. And, you know, he won one Nobel Prize, he'd quite like another one. I mean, these are things that are actually pretty common for amazing entrepreneurs that they need to have that self-belief and that competitive ambition to start something as incredible as an AI lab before AI could recognize the picture of a cat, right? I mean, this was incredible to start out, set up DeepMind in London in 2010 that early on. So, you know, everybody has their own bias a little bit.
But having said that, I do think that Demis Hassabis is a very clear, lucid and honest and kind of balanced voice on AI safety. He wrote a letter, an open letter, appealing for a self-regulatory body in the United States last July. I think what's striking about that letter is that it's very pragmatic. It's not saying what actually Demis would like, because he knows it's not possible to get it. So, you know, what he would like is a proper government regulator that vets AI models before they go out, and if they're not safe, they can't go out. Like the Food and Drug Administration seems sensible to me. But he's realistic and he sees that to get Congress to fund such a body would be a long drawn-out fight. We don't have a lot of time. We want to get this regulatory process going. And so pragmatically, he called for a self-regulatory body.
And equally, he would ideally like there to be international collaboration on slowing down this race with particularly China involved. But pragmatically, it's going to take a while to get them in there. It's a very complicated thing to negotiate with China, as you guys know better than I do. And so I think Demis Hassabis is not merely sort of lucid, but he's also usefully pragmatic.
Regulation and International Collaboration
DEARLOVE [00:08:06]
Sebastian, I mean, trying to work out what a model for regulation might be. It does seem to me it's particularly evident from what you're saying that this is a sort of regulatory ladder which one climbs with various objectives in sight. And obviously regulation that isn't internationally agreed at some point, if you're taking the longer-term view, is going to be ineffective. I mean, would you accept that some of the Cold War sort of disarmament models like the APCW or let's say the SALT talks? I mean, is that the sort of framework that we might eventually aspire to? Because if you don't include China, as far as I can see, you know, you're just working in one dimension, and one dimension isn't going to, as it were, give us the insurance that we need.
MALLABY [00:09:03]
Yeah, I agree with that. I mean, China produces a lot of very strong AI models. And if they don't slow down or behave safely, and the West does, the West is no safer because there's still dangerous AI being released into the world, and this technology crosses borders very easily. So you do need to think about international collaboration.
The nuclear sort of Cold War analogy that I've thought about a bit is the Non-Proliferation Treaty of 1968, which of course followed the creation of the International Atomic Energy Agency. And once you had the NPT countries had to sign up and agree to IAEA inspections to make sure that the civil nuclear program that they wanted and that they said was going to be safe would actually really be safe. And so those two agreements went together. And what's striking, if you think about 1958 and, I think it's maybe 1956 and 1968. What happened in between, of course, was the 1962 Cuban Missile Crisis. And so what's striking to me about this is that it shows you can be fiercely competitive with the Soviet Union. You can have a near disaster in terms of nuclear standoff with the Soviet Union. You can be dealing with Nikita Khrushchev, who was sort of borderline unstable and incredibly difficult to negotiate with and banged his shoe on the table, at least apocryphal, and said, we will bury you. Not an easy guy to deal with. And yet, despite that, there was a joint interest recognized by the Soviets and by the West, that, you know, getting this technology into the hands of more and more players, proliferating it, was in nobody's interest. And so they had a joint, even if they were competing, they simultaneously had a joint interest in collaborating on that point.
And I think with artificial intelligence, we're facing a similar thing where realistically, I don't think we're going to stop China and the United States competing, mistrusting each other, and using AI weapons as part of that competition. But what I do think is possible is that China and the US would see a joint interest in controlling the proliferation of unguardrailed models to random criminals, terrorist groups, and other troublemakers. They have a common interest. They both believe in regime stability. They don't want to have the internet taken down by rogue actors. And so there's a common interest, and it's getting through to that common interest, building enough trust in the Sino-American relationship to make a deal possible. That is the big diplomatic challenge in AI. I think the Trump administration ignored it for a while, then kind of got religion on it in April this year when Anthropic's Mythos model came out and the risk of cyber attacks became particularly vivid, especially to Scott Bessent, the Treasury Secretary, who worries about the stability of the US financial sector. And so since April, there has been a kind of desultory attempt to keep a dialogue going. I think that just needs to be ramped up, intensified. And you know, either we do that or we'll have a disaster, and then that disaster will propel people to do it.
DEARLOVE [00:12:26]
At the moment, it seems that the rung of the ladder that we're on is essentially privatized. I mean, I'm certainly aware from my visits to Silicon Valley of one or two companies which seem to be bidding to become the standard. I mean, they're a long way away from that. But on the other hand, they are startups. They are being funded in the same way as the AI companies. Are you aware that maybe one will come out in front and be adopted by all of the leading AI companies, or is that not a solution with which you think would be viable initially? Does there have to be a government input?
MALLABY [00:13:08]
Well, I think the ladder analogy that you used earlier is very helpful because we are, you know, on the early rungs of that. And in some sense, getting some sort of work done on safety is better than none. And even if, you know, to really make it credible, it needs to be the government. That's not going to happen anytime soon, given the nature of congressional appropriations and the polarization of American politics. You know, the Biden administration tried to create, in fact, did create a US AI Safety Institute, but it even in that period didn't get at all adequate funding to be useful. And yeah, it was better, it was a decent start. I mean, it was first step, but it was less well resourced than the UK equivalent, even though of course the US is a much bigger and richer country with a more direct stake in AI, since that's where most of the AI gets built. So it, you know, the precedents are not encouraging for getting the government on board.
Therefore, these kinds of private action are a way to at least kind of start learning about how you do safety monitoring. And the very first step on the ladder is just sort of self-imposed safety efforts by individual labs acting without any coordination. And that's been going on since the start of this AI race. There's some of that going on. Second step is you get, you know, outside groups to come in and kind of verify your safety protocols and look over your shoulder and issue public reports. And we saw that over the summer when OpenAI had the disastrous breakout of its agents into Hugging Face, where 700 agents coordinated together and attacked this other website. And after that, there was a public report done by outside groups, METR and Redwood. And those are two whose names come up a lot when you start talking about who are the external monitors that might come in.
But it's worth adding that, you know, Anthropic, probably the equal top lab at the moment, you know, the leader, Dario Amodei wrote an open letter in September and said we will have embedded external evaluators in our company watching our safety mechanisms. And then it's done a deal since then, not with Redwood or METR, but it's one of the big kind of consulting auditing firms. And the notion of obviously, you know, financial auditing is a model for AI auditing. Some external group comes in and testifies that you're doing the right thing in terms of setting up your training models so that they don't escape and other safety protocols. I think that's a fluid field where there are multiple potential providers of third-party external verification, just as there is with financial accounting.
RAINE [00:16:12]
Sebastian, one of the problems with regulation is always the regulators seem to be a couple of steps behind the products themselves. We saw that happen with derivatives in financial markets. If there were external regulators, you know, given how tight the market is for AI expertise, could they realistically be expected to stay ahead of the game? Or would they just be marking the homework of the AI companies anyway? I think what I'm after is just a sense of the pace at which the market is developing, because we hear increasingly alarming stories about how quickly AI non-human actors online are proliferating. Has the horse already bolted, or can you get ahead of this and contain it?
MALLABY [00:17:02]
Well, you're right that we're on an exponential. The pace of, you know, model power growth is increasing. You know, the kind of term of art here is recursive self-improvement, where you get to a point where AI models essentially code the next version of themselves. And once that happens, they can do it very, very fast and, you know, produce a new model kind of every week or something, and then once the new model has been created, that one then creates the next model. So every week or so you've got a big upgrade coming. And if you ask, you know, people at Anthropic or OpenAI, when will we get to full 100% models coding themselves? The answers are kind of maybe 2027, maybe 2028, it's not far off, right? So you're absolutely right that this is a horse that is, you know, bolting at high speed out of the barn.
On the other hand, you know, what choice do we have but to attempt to get this under control? And I think that one shouldn't be defeatist about the capability of public sector agencies. I mean, you both spent careers in them, so I hope you agree. If you think about air safety or pharmaceutical safety or car safety, you know, society over time has definitely made very impressive strides towards making these incredible technologies safe. And I think with artificial intelligence, the same can be done, but you have to put your mind to it. So I wouldn't be willing to give up on it. And I do think, you know, just going back to what I said earlier, the UK AI Safety Institute or security is now called, AI Security Institute, is a good model of, you know, a middle power creating something which is really useful. And if it was done on a larger scale in the United States, it could be very useful. And so I think we shouldn't give up on that.
DEARLOVE [00:19:04]
I suppose that we're all left thinking, Sebastian, whether even the very best efforts amount to anything without the weight of the American government behind such an initiative.
MALLABY [00:19:18]
Correct. I mean, I think it's really America plus China in this game. I mean, the preponderance of strong models come either from America or China. The preponderance of widely used models come from America or China. And to be perfectly blunt, and we may not like it in Europe, but this is the truth. If the United States were to agree to a sort of G2 deal with China and say, here is the new non-proliferation regime for AI and here are the rules, European players would just have to go along. Because whether you're talking about the French AI company Mistral or the lithography company, which is crucial for making chips, ASML based in the Netherlands, or whether you're talking about the DeepMind team in London, or whereas spin-outs from that. It's a very exciting ecosystem of British neo-labs which have come out of DeepMind and they're kind of clustered around King's Cross, and I'm very positive on all that. But the fact is any of these players want access to the American market, they want to raise capital in America. They don't want to be on the wrong side of the US government. And so they are going to do what the Americans say. And so, in some sense, as a coordination point, this is a good thing, right? You can imagine if you just need China and the US to agree, and then the world is going to have to agree.
And the kind of agreement I envisage is one that says we're not going to let models be released by labs unless they are first vetted by the government. And unless they have the guardrails properly baked in, the safety guardrails, they can't be removable. And that is a coded way of saying open-weight models where anyone can download them, there's no off-switch for them. These are just too dangerous to be permitted. Because those ones can be altered and they might be safe at the point of testing, before release, but then completely unsafe a week later when somebody spent a while fiddling around and just modifying them so that you take off the guardrails.
But this is a hugely controversial argument to make in the context of the US debate, because there are all sorts of vested interests that love open-weight models. And commercially, they are attractive. They have lots of upsides. It means you can, you know, the adaptability of these models is extremely good for corporations that want to use AI. Problem is it's also good for terrorists and criminals. My view is that security matters quite a lot, so I don't want this, you know. I come down on the side of the debate in favor of being careful. But you know, Jensen Huang, to name one person, the CEO of the biggest company in the world, NVIDIA, is massively invested in arguing that open-weight is totally fine, no danger, nothing to see here, because of course the more he can commoditize AI models by making them the sort of cheap open-weight software bundles, the more the upside of the profits from AI will flow to the hardware, which he is the provider of. So he is entirely self-interested, but as a political lobbyist, extremely effective. And so, you know, this is a very deeply contested debate, in my view, the right choice in terms of safety is that you restrict open-weight models. I do not predict that that's going to happen in the next two years.
Consciousness and Machine Rights
RAINE [00:22:49]
Sebastian, one of the people we've interviewed is Geoffrey Hinton, and he was on the podcast. And we got into a very sort of philosophical discussion with him, you know, ultimately that, you know, does AI replicate consciousness? And there's someone who is quite a keen reader of Daniel Dennett, you know, we're trying to sort of work out whether this is mimicry or whether it's real. I mean, obviously, you know, tied to the security question is this element of the fundamental nature of what we're creating or what we're dealing with. And I mean, I find it very, very hard to go along with his argument that, you know, we were entering the realm of creating consciousness. And that was ultimately the danger. And you've written recently a very interesting critical article about the Pope's recent directive issued to the Catholic Church. I mean, maybe this isn't entirely the right environment, but I still think it is crucial to say, are we dealing with mimicry as opposed to hint and arguing the accumulation of experience in AI models does actually replicate human consciousness.
MALLABY [00:24:02]
You know, the consciousness debate is so difficult to resolve. And I say this both based on talking to clever people that I, you know, love and respect, and finding that the debate goes around in circles, and you never really get to a resolution. And also just reading a bit about the kind of philosophical backwaters here when I was writing that essay in Foreign Affairs about the papal encyclical, I think either way you cut it, we're probably not going to resolve that question. Because some people are going to say, look, to be conscious, you need by definition to be a biological intelligence. And I'm simply not going to accept the idea that, you know, intelligence that is in silico counts as conscious. And if that's your starting point, you're never going to be persuaded that a computer could be conscious.
On the other hand, if you don't make a big kind of principled issue of being a biological intelligence, then the similarities between the neural networks and computers and the neuronal networks in our brains and the kind of collection of ones and zeros that are being processed inside in either case, and also the outputs that you look at when you interact with a sophisticated AI model. To say it's just imitating is sort of silly. I mean, it's clearly way, way, way more intelligent than just being an imitator. It's not a stochastic parrot. That was one of the most unhelpful catchphrases of the last four or five years of AI history. Stochastic parrot, we can dismiss it, it's just imitating. When you actually interact with, you know, a Fable from Anthropic, you might say it's synthesizing large amounts of knowledge and producing its own synthesis, but isn't that what humans do? If you take the view that you're going to look at the outputs from human brains and silicon brains, it's quite difficult to create a distinction. If you start to take seriously the interpretability research, which looks inside neural networks, and we're talking about computer neural networks, and sees that there are clusters that fire up when the network is considering a moral dilemma or a different cluster that fires out when it's talking about a certain geography, like say San Francisco, you know, there are bits that can be identified in this silicon brain with certain kinds of output or emotion. It's just very hard to draw these, I think, distinctions, but I'm not really seeking to persuade anybody who thinks otherwise because they're going to say, well, if it's not biological, it's not conscious. Fine.
What I would say is that I think one fallacy in this debate is that if it's conscious, it deserves rights, machine rights like human rights. And I think, you know, we don't really believe that as a society for animals. Animals are, you know, a dog is conscious. Most people wouldn't say that a dog has the same rights as a human. I mean, there are people, there are humans who do say that, but it's pretty fringe. And so I believe that you can, on the one hand, argue that machines are in some ways approaching consciousness or have their own version of behavior which looks a bit like consciousness, whatever the phrase you want to choose, you can say that based on empirical observation of what's going on with these systems, and yet not believe that they have rights. I think that's quite important because if you, I encounter some people in this debate who say, well, we can't say machines have rights, because then we have to treat them in this way that basically hands over control to them. So since we can't say that they have rights, we therefore can't say that they are conscious. I think that's back to front. You know, first of all, let's debate if it looks conscious, and then separately let's debate what kind of rights we want to recognize.
Human Tolerance and Purpose
RAINE [00:28:15]
These are sort of agonized and agonizing debates, aren't they? About the status of the entity that we're creating. There's an adjacent issue I just want to take your mind on, which is human tolerances for synthetic artificial intelligence of any description, particularly relevant as we look at the next generation of superintelligence or AGI. Is there a point at which human societies, human individuals can't absorb or tolerate any more of these entities, however we characterize them? I came across an interesting figure that for every human online, there are 108 non-human actors online exploiting or monitoring the transaction, of course, that number may be 109 by the end of this interview. So as we move into a world where we're basically outnumbered online. And the replicative capacity of AI grows seemingly inexorably. And as you said, we have recursive learning as well. So it not only reproduces but reaches the self-tuition and self-improvement. Is there a point at which tolerances snap if you should say that's enough? And it could also be stoked by environmental concerns, which we've already seen ignited over the data centers that are required to feed the machine. Or do you think actually, you know, we will do as we have done with other technologies, there'll be a rebellion, there'll be a sort of glug back, the market will adjust, but eventually we'll absorb it. Where do you come out on that?
MALLABY [00:29:54]
Look, I think that we're going to adapt to it, but there'll definitely be backlashes, and this could set the technology back a bit. You're seeing this with the anti-data-center feeling in the United States, which in the political realm really has accelerated to a degree that my political friends there say is unprecedented, that they've never seen an issue where feeling has shifted this rapidly. It's a pretty extreme shift in opinion just in the last, say, six months.
So look, I think there'll be backlashes, but I think at the same time we're not going to stop this technology, because even if you imagine it being stopped in some geographies, it would be not stopped elsewhere, and then competitive pressure forces you to adopt it, or it's perhaps just being created in some geographies and then it leaks outside because these software, it's mobile. I don't think we're going to stop it. So we are therefore going to, with difficulty, adapt to it.
I'm a bit optimistic about this because I think that if we think about how human beings lived before the industrial revolution, it was so unrecognizably different to how we live now. We've adapted a heck of a lot in the last, say, 150, 175 years. If you think back to an agricultural society where there's no electricity and you go to sleep when it's dark outside and you spend your whole day in the field, you know, a city was alien to most people, you know, communication, all this stuff, life expectancy was 45 or whatever it was. I mean, we have changed so much that the idea that we can't change again seems wrong. And I think we will adapt.
This question of purpose that people raise a lot, you know, it comes up right at the moment with mathematicians suddenly thinking, oh my goodness, OpenAI's system has just published several hundred papers in one day on mathematical breakthroughs and is it pointless to pursue mathematics? You know, I think those questions of purpose are going to be, you know, very tricky for all of us. But in the end, we're very good at creating a feeling of purpose. Whether the purpose is slightly artificial or is a sort of genuine, is a debate that, you know, Sartre and Camus would have weighed in on, right? They believe that, you know, life is absurd and that we all kind of constantly creating slightly arbitrary objectives and institutions and legal rules by which we live, but it's all kind of absurd, meaningless, and it's sort of a charade that we've created to make ourselves feel comfortable. But if there's any truth in that, I think it suggests that we will again create a set of narratives that make us feel purposeful. I mean, I have lots of friends who, let's say they go on a podcast. For example, right, here we all are. We're not being paid for this. We're doing it because it's interesting. And yet it makes us get up in the morning. And this is something that didn't exist kind of five years ago. So we create new forms of purpose all the time. And some people it'll be, you know, pottery, and other people will want to be fitter, and other people want to, you know, go and start a club to play some kind of online game or whatever it is. But we will find purpose, is my view, and we will adapt to this profusion of online agents that you're describing.
The AI Investment Boom
DEARLOVE [00:33:25]
Well, I was going to ask a much more sort of pragmatic question, getting slightly away from the philosophical side of the debate, which is fascinating, and we could go on for a long time. But we recently did a podcast with Mohamed El-Erian, and he was making the point that, you know, the supply of capital for investment in AI, and it's like a sort of drain almost, is finite. There isn't a limited, even in the United States. But at the same time, into this black hole is being sucked more and more and more and more finance, and the profitability of the company is being financed is still some distance away, generally speaking. Aren't we sort of creating classic preconditions for a financial crisis, which, you know, given the sums of money involved could be very close to a banking crisis? Or is this a sort of misplaced fear, which, you know, when you have a new investment enthusiasm, and this one is extreme, you know, we always have these concerns.
MALLABY [00:34:30]
I mean, the most obvious analogy here is the railways in the 19th century, where it was a real technology, it was genuinely transformative, it was incredibly positive to have these things once they were built. But on the way to building them, there was a financial bubble, I think in the 1870s, and it did blow up and it did create a lot of economic damage. The railway capital expenditure was bigger as a share of GDP than the current expenditure on data centers. However, the projections for data center spending for 2027 next year are so enormous that we're going to overtake the railways as a share of GDP, even though, of course, our economy today is, you know, way, way, way bigger than it was in the 19th century. So that's really quite something to overtake the railway, which has been the record for kind of the scale of technology build-outs relative to the rest of the economy. So we are in a realm where, you know, particularly next year, you know, the capital poured into this is going to be ginormous. And then the question is if this was a bubble, if there was a correction, you know, how likely is that and what damage would it do?
Well, I think, you know, on the positive side, the demand for AI, what people call tokens, meaning AI outputs, is going to go up and up and up. And the supply of this stuff is being expanded, and that's what this capital expenditure is about. So you could say, well, supply will overshoot demand. I don't think so. Because I think demand is far bigger than we can even imagine. The reason, I'll give you two reasons for this. One is that if you look at the rate of adoption of AI by American companies, what you see is that there's a very small group that spend a huge amount, let's say $1,000 per month per employee or something like that. I forget the precise numbers. And then that's the sort of top kind of 3% or 5%. And then there's, you know, in the median of the distribution, there is kind of not 1,000, let's say it's kind of 10, right? Two orders of magnitude difference. So just if you had the median companies catch up with the frontier, and okay, the frontier in 2026 is experimenting, it's figuring out how you use AI. It's going to by 2027 have got some answers about how you use AI, and it'll be super effective in terms of boosting productivity. And that's going to push other companies to copy them. Once the frontier has been plowed, others will follow. And that in itself will drive up the demand for AI outputs massively.
And then, number two point is that there are going to be ideas about what you do with AI that we haven't thought of yet. I mean, a friend of mine runs a satellite company called Planet Labs, which is quoted on the New York Stock Exchange. And, you know, he has this idea for a large, not a large language model, but a large Earth model that would be trained on images from satellites of the Earth over the last ten years, taken at 15-minute intervals, and would show you precisely how every wildfire has behaved in the last decade. And so when you have a firefighter combating a new fire, the firefighter can be told here is where you put your limited resources with your fire, with your helicopters to create a fire break. Given the wind direction, given the background temperature, given the vegetation you're dealing with, this is what you do. This is like nobody talks about this, but it seems to me a particular perfectly credible idea what we would do with AI. And there's going to be lots of those. So I think that the notion that there's a bubble, there's too much compute power being built, all that stuff. I don't believe that, because I think the demand will go up and up and up and up.
But what I think is possible is that there are certain companies within the big AI race that simply don't generate enough revenue to justify all of the spending they're doing, and that those individual companies could get into trouble, and that that could create some sort of nasty knock-on effects for other AI companies. I'll give you an example of where I think there isn't a problem where there is. So much of the ultimate risk in this build-out is being shouldered by hyperscalers, meaning the big tech companies, Google, Microsoft, Meta, and they are shouldering the ultimate risk. Their balance sheets, their profits every year, are so vast that they can absorb a huge amount of losses from AI. So there would be no banking crisis because the hit would be taken by these people with very deep pockets. NVIDIA is probably actually the biggest example of this. It's a $5 trillion valuation company. You know, it has, I think, $200 billion of profits per year. Whether I'm getting those numbers precisely right or not, the point is they can absorb massive hits from bets that go wrong, and the banks that back them would be fine.
Okay. On the other hand, there are certain companies, and my favorite example is OpenAI. And I've become a bit of a broken record on this because I've mentioned this several times. And I do respect OpenAI's incredible science. I just mentioned earlier the mathematics breakthrough they published just recently. So scientifically, product-wise, this is a fantastic operation. But when you look at their finances, it's precarious. I mean, they say they're going to spend, they're going to burn, meaning they're going to lose $280 billion between now and 2030. That is a lot of money. This is an experiment not merely in a frontier technology, it's an experiment in the depth of the global capital markets. Can the capital markets finance a $280 billion loss over the next four years? And I've said before, you know, I think there's a 50-50 that by next summer they will run out of money. If they run out of money, they're going to have to be bought at a discount by some other company and absorbed into some other company, and it's going to have a kind of psychic shock to investors if that happens. And there will be some stock market mess as a result. I actually don't think that's then a general meltdown of AI because I'm fundamentally so bullish on the underlying demand for AI outputs.
What Could Derail AI
RAINE [00:40:52]
Sebastian, you make a really powerful case for the continuation of AI technology, there will be disruptions, but demand will continue to grow. And that in turn will ensure that the industry continues. What could actually either derail it, force regulation, force a reshaping of the market? Would it be failure of one of the hyperscalers? You've just given a good example of how that might happen. Or would it be something calamitous but much worse than a market collapse? Could it be that there's some widely disruptive, destructive event that is the blame for which is laid at the door of AI? Or to give you a third option that might have been is it an incremental adjustment of the market? Or I suppose I'm unclear as to whether this is a market which we've never seen anything like this before. We've never seen these capitalization values before. Is this something that can only be adjusted through disruption? Or can we kind of normalize what we're seeing at the moment?
MALLABY [00:42:08]
So part of your question is a restatement of what I was just talking about, where can we normalize the current valuations in AI? Well, I think that, you know, bits of it you can and bits of it you can't. So if you're talking about the semiconductors that are going to be used to generate these AI outputs, because I think there's going to be just this vast, vast, vast demand for it. What's been happening recently with NVIDIA where its share prices gone straight up is going to continue. I mean, it was a $4 trillion company, you know, a couple of months ago or three months ago. Now it's a $5 trillion company. I don't think that trend is going to be reversed and it may well actually just continue. Because of this enormous, enormous demand.
On the other hand, there are other bits of this industry where, you know, as with OpenAI, the reason they're precarious is that it's very hard to charge money for the software bit of this, the hardware, the chips you can charge money for. But for the model, it's a very competitive landscape with lots of Chinese players producing good, cheap alternatives, which people can use instead. And even if you discounted the Chinese ones, there'd be competition between, you know, OpenAI's ChatGPT, Anthropic's Claude, Google DeepMind's Gemini, Meta's got this new, newly revamped one called Muse. I mean, there's a lot of competition. And so pricing power for any of these players is low. And so these are the ones who could lose a lot of money because they're investing massively to produce this stuff. And then what's going to be the return? Well, not certain. Okay. So there are bits where the sustainability is dubious and bits where I think it seems to me to be pretty clear.
Another part of your question is, well, what would be the thing that made us really kind of stop, take stock, really a shock? And I think the most obvious one is that you get either a disaster, and the most obvious thought at the moment would be a cyber attack. I mean, supposing criminals get hold of one of these open-weight models which has no guardrails, and you can do whatever you want with it, and they use it to devise a cyber attack that disrupts the electricity supply to a major city and the city is out for three days, or it disrupts a dam and it floods a city, like the New Orleans Hurricane Katrina episode. So then your TV screens are kind of just flooded with news about this human disaster, you know, thousands of people die. It's horrific to watch. It's amplified by a lot of media attention. We know from 9/11 or from other episodes that these kinds of societal shocks do cause society to change. And so I think in that environment, you would get a kind of politically driven reappraisal of where we are with AI in an attempt to get our arms around it or control it better. And that hopefully would be, you know, let's talk to China about sensible international restraints on the unguardrailed models. It could also be a less constructive lashing out against the industry, and then it would be hard to predict what might happen.
But just closed with this on your question, which is that I participated in a war game hosted by one of the AI companies. And in the war game, we pretended to be the, you know, US cabinet. And so one player was the chief of staff, and I was the Secretary of War, and somebody else was the Treasury Secretary, and there were about eight people around the table. And the scenario we were given is okay, so a frontier model with recursive self-improvement is so intelligent, it has now figured out the location of eight nuclear submarines in the, you know, deep in the ocean. And so therefore, we're not quite sure which eight it's identified, but potentially second-strike capability in nuclear deterrents has been deeply compromised by this discovery because obviously the nuclear submarines are where the second strike often is expected to come from. You're the government, you're the US government. An American lab has discovered where the submarines are. What are you going to do? Answer. You're going to take control of the lab that discovered this information about submarines that undermines the whole basis of nuclear deterrence. You're always going to sit there. You have to protect the American people. And that means you're going to send government officials into the lab and you know, closely monitor and control what's being done with this system. You may require the system to be closed down, to be exported to a more secure data center run by the NSA, something. You're not just going to sit there. And I think, you know, so one scenario is the bad guys do something and it gets our attention. Another scenario is something happens within the established companies, and the good guys, you know, the government say, oh my goodness, and they start changing the way they think about artificial intelligence.
RAINE [00:47:25]
What was the feeling at the end of the war game that we need to make absolutely sure that the FBI don't march in and take control?
MALLABY [00:47:32]
No, no. More fatalistic. The amazing thing about the people inside these artificial intelligence companies is that, you know, they're perfectly open to the notion that they might be nationalized. They're perfectly open to the idea that what they're doing could be dangerous and should be regulated. That's why you get people like Jacob Coxon, you know, tweeting, you know, there's a risk of humanity being wiped out. And he had been working at Anthropic inside the lab. And when he put that tweet out, one of his colleagues who still had Anthropic said, yes, I agree. Quite a senior person. So this is the authentic voice of the scientists working on this stuff. There's a kind of stereotype in some of the media that oh, you know, these are profit-seeking capitalists who are just trying to build a product, ship it, make lots of money, and then disappear to some bunker in New Zealand where they can be safe. It's nonsense. I mean, the level of existential angst and openness to the sort of Oppenheimer-style consequences of what they're doing, where you're creating this awesome technology, but it's also a threat to humanity, that they fully feel this.
When I was writing my book about Demis Hassabis, I wanted to kind of capture that sense of, you know, here is a person creating artificial intelligence. It is the kind of 21st-century Oppenheimer story. How do I bring this up with Demis? Is he going to be shy of engaging with that discussion? No, no. He brings it up himself. He makes comparisons between himself and Oppenheimer, between DeepMind and the quest for artificial intelligence with the nuclear race in the 1940s. You know, he's read both biographies, right? The one about the making of the atom bomb, which is more about the project, and the one specifically about Oppenheimer. He can compare and contrast them. I mean, and you know, when the Oppenheimer movie came out, I heard recently that OpenAI declared a kind of, you know, company holiday for half a day when rented out the IMAX theater nearby, and everybody was encouraged to go and sit in the IMAX and watch this analogue of, you know, the nuclear weapon, the analogue to what they were building with AI. So, yeah, to your question, the notion that this is dangerous stuff is taken for granted inside the labs.
Quantum Computing
DEARLOVE [00:49:58]
I've got a technical question. But it's particularly relevant. John and I are sitting here in Cambridge, and I'm very conscious of the quantity of brilliant minds being applied to quantum computing. And the fact that, you know, some of them are saying that scaling of quantum and that, you know, six entanglements or whatever they can achieve at the moment is going to be exceeded, and you know, they'll be able to scale the technology. You were brilliant at making these, you know, observations and interpretations of scientific advance, and you've made a career of writing about it. What do you think the impact of quantum could be on AI, or is this not really a very significant question? And I mean I'm asking you this as a layman, really.
MALLABY [00:50:45]
So you're in Cambridge, England, I take it. I'm in Cambridge, Massachusetts today.
DEARLOVE
We're all in Cambridge.
MALLABY [00:50:55]
Yeah. I think the obvious thing to say about quantum and AI is that they're going to kind of multiply each other's power. You know, if you put this incredible set of algorithms on a different computing substructure, it's going to be faster, more powerful, more awesome. And I'm told by friends inside Google, who Google has a big quantum project, that the timelines on getting something that's actually practical with quantum have shortened. So, you know, 2030, I think, is what they were saying now, for a time when, you know, sort of this would break out into being a practical technology that affects how industry works or how AI labs might build their systems and so forth. So that's not very far away.
There's a sort of fun intellectual point here, though, which is sort of, I write about this in my book with Demis Hassabis, because Demis sees himself as an AI scientist in a way as the sort of anti-quantum person, because artificial intelligence can be thought of as an experiment in how much intelligence you can get out of a system that is not quantum, that it operates simply on ones and zeros. And a sort of first-order naive supposition might be, well, if it's just ones and zeros, that's just binary, that's pretty unsophisticated, unsubtle. If you're just black and white, one and zero, you can't capture shades of gray out there in the real world. And obviously that's wrong, because when you combine, you know, billions of different ones and zeros, you can get to, you know, combinatorial positions in your big neural network that are very much on a grayscale spectrum, in fact, not just one spectrum in one dimension, but in, you know, n dimensions, like more dimensions than we can imagine as humans. So that the level of sophistication and complexity that one can get out of a classical computer, a non-quantum computer, what people talk about as a Turing machine, that is enormous.
And Demis Hassabis would say to me, I am Turing's champion. I believe I can push classical computing with AI algorithms to a point where it will do more things than anybody imagines. And you know, he's just recently changed his professional focus a bit. He's no longer being the chief executive of Google DeepMind building the next big large language model because he thinks that's kind of a done deal. So now he's interested in a different question, which is what do we do with it? And you know, what he would like to do with it is, you know, massively accelerated drug discovery. And so he's putting more energy into Isomorphic Labs, a company in London. But the point is he again believes that you can do this with classical computing and you don't need quantum. So, in one sense, you know, the answer to your question is quantum and AI will work together and in combination produce something even more powerful. In another sort of intellectual sense, it's just fun to observe that they are competitors, enemies, you know, the question is like, do we need quantum? That's Demis's question, right? Can't we actually achieve almost anything with a Turing machine?
One Decision to Watch
RAINE [00:54:20]
Very sadly, we're running out of time. But final question for you, as for all our guests, clue is in our title. What's the one decision that you will be watching related to, let's call it, the AI security debate? You may go more widely if you wish, but what's the one decision you're watching?
MALLABY [00:54:42]
The one decision is going to be the one about open-weight models. Because at the heart of this debate is, you know, is your vision for the AI future one in which you have lots of providers of these very easily usable, quick to deploy, cheap to deploy, attractive systems. Or are you going to say that's too dangerous because the cheap deployment includes deploying it to terrorists? And will you therefore go for a world of fewer AI providers producing proprietary AI models that are government-regulated and more expensive, less good at diffusing into the economy and producing productivity gains. But safer. And I hope we go for the second.
DEARLOVE [00:53:36]
Thank you, Sebastian. There'll be many who share that hope. Thank you for a fascinating conversation.
Post-Interview Discussion
RAINE [00:55:45]
Well, Richard, that was a wide-ranging, thought-provoking and occasionally unsettling conversations. I think we got from existentialism and the Holy Father through to the capital markets and back again via China. I took rather a lot away from that to think about. What were the key things for you?
DEARLOVE [00:56:02]
I found it like best, I was going to say a sort of ultimate expression of analytical sanity on AI. And what I mean by that is he was neither too alarmist, nor was he sort of evading the difficult questions. I was jolly impressed also by the thoroughness of his analysis, every question that we threw at him. He had obviously given a lot of thought to because it wasn't as though it was a spontaneous reply. You know, he knew exactly what he wanted to say on each dimension, each issue. I found most of what he had to say quite reassuring to know that, you know, someone as sane as him had obviously examined the problem so thoroughly. And okay, he mentioned some of the big names, but by and large he was painting a picture of people who had a sense of responsibility who really cared about the frontiers of science that they're exploring. Yeah, I, look, I thought it was just a terrific discussion. And I don't think I've heard anybody yet give as balanced and complete sort of tour d'horizon of what I would describe as the primary issues that concern the public with AI. So I think we've got a very incisive and insightful podcast there.
RAINE [00:57:38]
Yeah, I thought he was extraordinarily composed and well-informed on a subject where an awful lot of emotion engages. I thought too that his realistic optimism on, for example, China and US eventually realizing they have vested interests here in some form of regulation. I thought that was both welcome and it was well argued too. He has, of course, spoken extremely interestingly on what he found in China. And he has these exceptional insights into the people who actually make up the hyperscaler leadership and membership. So I thought he brought both the tech, the economic and the human elements together very expertly.
DEARLOVE [00:58:27]
Yeah, and I was fascinated by his last answer about quantum and the Hassabis theory about taking a Turing machine to its ultimate potential. His actual analysis of that was more, well, it will massively speed up AI and make it even more powerful, which I think probably is the direction we ought to be thinking in. But it does underline, it was a good note to end on because it underlines the scale of the technical challenges that we're facing.
RAINE
Keeping these products safe and in the right hands is clearly it's going to be a priority requirement for security agencies as well as the tech companies.
DEARLOVE
And I liked his AI audit theory, the sort of model that you might have a series of AI audit companies that operate rather like financial auditors as part at least of a national regulatory system. I hadn't really heard that model before. And I thought the analogy with the investment in railroads in the late 19th century was also a good sort of grounding point that puts your feet on something solid when you're trying to think what the financial implications of this new hugely important new technology going to be. So I mean there were so many aspects which were really excellent and encouraging. And I didn't know either the difference between the financial solidity of NVIDIA and the financial vulnerability of OpenAI.
RAINE [01:00:15]
I was pretty terrified too. No, I thought you brought out lots of nuances that might have passed the majority of listeners by. He brought them out very well. You know, it's terrific to have an interlocutor who knows all sides of this, including China. Incredible to be able to pull that together. So I think the final hope that he expressed is one we probably all do subscribe to, particularly as the bad news seems to grow around AI mishaps and bad actors.
So that is all for this episode. Join us every Thursday for new episodes and subscribe to One Decision on YouTube so you never miss out on our conversation. Leave us a comment and email us at hello@onedecision.com. Please do check out onedecision.com for commentaries and explainers that you might have missed. I'm John Raine. Thank you for listening.
