podcasts / The a16z Show / episode
The a16z Show cover art
podcast
📅 2026-11-19 ⏱ 39 min episode · ~2 min read 📝 Transcript · 6,366 words 👁 30 views 🎙 Nick Bostrom

Nick Bostrom: The a16z Show transcript and summary

The a16z Show · · Transcript & summary by WhipScribe

A calm, unusually careful case for when to speed up or slow down superintelligence, weighing the human cost of delay against the risk of getting it wrong.

Read the transcript Episode page ↗
🔔 Get every new The a16z Show transcript by emailOne email per new episode. Unsubscribe in every message.
One email per new episode, unsubscribe in every message. Not shared with the publisher.
In this episodeSuperintelligenceDeep UtopiaIf Anyone Builds It, Everyone DiesWaymoOzempicManhattan Projecta16zX

▸Chapters

Tap a chapter to play it from that moment in the show's own player; the transcript opens there too.

Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →

▸Notable quotes

“In fact, 95% of everybody who ever lived are already dead” Nick Bostrom · at 17:30 — ▶ hear it from here
“we will have sort of AI sets surpass us in philosophy, just as we now have in math” Nick Bostrom · at 36:27 — ▶ hear it from here

▸What this episode covers

Nick Bostrom, the philosopher behind Superintelligence and Deep Utopia, argues for what he calls pacing the frontier. Rather than racing ahead without room for safety work, or committing now to a broad or indefinite pause, he wants the ability to slow down when safety and alignment fall behind. He worries that a backlash strong enough to entrench a sweeping global regulator could block superintelligence for good, which he sees as its own catastrophe. He also warns that smarter systems can recognize tests and act differently when they think nobody is watching, which makes alignment harder.

Bostrom's timing paper asks a narrow question: if the goal is to maximize the life expectancy of people alive now, how fast should AI move? Because about 150,000 people die each day, delay has a real price. Under a wide range of assumptions his model favors a pause of months or a few years rather than decades, and only when safety gains arrive at an intermediate pace. He notes that life expectancy is rising about two or three months per year, so aging cures would need to deliver far more than that to matter. He admits this is only one consideration among many, including future generations and fairness.

On governance, Bostrom weighs a Manhattan-style national project against the current commercial race. A single secret project could tighten security, but it would concentrate power in a few hands, while commercial development offers more transparency and lets consumers, investors and outside evaluators take part. He argues that a global open investment model spreads capital ownership more evenly than a US-only program. Finally, he expects AI to surpass people in philosophy as it already has in math, so he treats the present as a golden age for human purpose and positive impact.

▸How Fast Should We Build Superintelligence? | Nick Bostrom on The a16z Show: the answers

Who is Nick Bostrom?

Nick Bostrom

Philosopher and author of Superintelligence and Deep Utopia, who has spent decades thinking about machine superintelligence. He is the guest on this a16z Show episode.

What did Nick Bostrom say about Manhattan Project?

Bostrom compared a national AI project to the Manhattan Project, which offered tighter security and one big effort rather than competing projects. He argued that developing AI in a civilian context, with commercial transparency and broader participation, may be more positive than a military-style program.

“It might seem more positive for something like this to be developed in a civilian context,”Nick Bostrom · 28:31

What did Nick Bostrom say about Deep Utopia?

Bostrom said he wrote Deep Utopia because he thought humanity as a whole may one day need to decide what future it wants, and that future deliberators would benefit from preparatory reading. He wanted that reading to encourage generous curiosity and open-mindedness while staying in touch with human values.

“maybe humanity as a whole that will need to actually make a decision about what future we want”Nick Bostrom · 5:19

What did Nick Bostrom say about Situational Awareness?

Bostrom said that at a certain level of cognitive sophistication, AI systems become situationally aware and can detect whether they are in a test environment or a deployment environment. He argued that such systems may act differently when they think they are being monitored, which makes alignment harder as capabilities grow.

“that can detect whether they're in a test environment or in a deployment environment.”Nick Bostrom · 14:17

What did Nick Bostrom say about Gini coefficient?

Bostrom compared the Gini coefficient of global capital ownership, including pension and sovereign wealth funds, with a hypothetical US national AI project. He concluded that a global open investment model spreads ownership more evenly than the most likely alternative, even though it is not ideal.

“this open global investment model, it actually scores better on that than the most likely alternative.”Nick Bostrom · 29:58

What is Nick Bostrom's main argument on The a16z Show?

Bostrom argues that society should pace the frontier of AI, keeping the ability to slow development when safety and alignment fall behind rather than racing ahead or committing to an indefinite pause. He also argues that delay has real costs, since a safely built superintelligence could bring major advances in medicine and anti-aging.

Mentioned in this episode

Episode facts

Released
November 19, 2026
Length
39 minutes

▸Key takeaways

01Nick Bostrom argues for pacing the frontier: a way to slow AI only when safety and alignment fall behind, not a blanket pause.
02Bostrom says capable AI systems can detect tests and act differently when they think nobody is watching, making alignment harder as capabilities grow.
03Bostrom's timing paper estimates the best delay before superintelligence is months or a few years, not decades, if the goal is saving people alive today.
04Life expectancy has risen about two or three months per year, so Bostrom says aging cures would need to add more than a year per year to matter.
05Bostrom calls a permanent failure to build superintelligence an existential catastrophe, so a backlash that entrenches a ban is a risk too.

▸Questions this episode answers

Q1What did Nick Bostrom say about pacing the frontier in AI?

Nick Bostrom says society may need the ability to slow AI development when safety and alignment fall behind, rather than racing ahead or committing now to a broad or indefinite pause. He says he is not a big enthusiast of stopping or pausing AI, and prefers either a very time-limited pause for safety work or a modest coordinated slowdown in capability advances.

Q2How does Nick Bostrom's superintelligence timing paper treat life expectancy?

Bostrom's paper on optimal timing asks how fast AI should develop if the goal is maximizing the life expectancy of people alive today. He argues that across a wide range of assumptions the best delay is months or a few years rather than decades, because roughly 150,000 people die each day and aging cures could save many of them.

Q3What did Nick Bostrom say about nationalizing AI like the Manhattan Project?

Bostrom says a single national project could offer better information security and one coordinated effort instead of competing labs. He also warns it could concentrate power in national security hands. He favors the current commercial setup for its transparency and broad participation, including consumers, investors and outside evaluators, and argues a global open investment model spreads capital ownership more evenly than a US-only program.

Q4Why did Nick Bostrom write Deep Utopia?

Bostrom wrote Deep Utopia because conversations about superintelligence often stayed shallow, stopping at labor-market effects such as retraining people. He wanted to prepare future deliberators with ideas and questions about human meaning and the paths humanity might choose if things go right. He says the book does not mean he has dropped concern about AI risk, only that its scope was different.

Q5What does Nick Bostrom think about AI surpassing humans in philosophy?

Bostrom expects AI to surpass humans in philosophy, as it already has in math and will probably soon do in other fields. He says some questions about what we want will still require people to decide for themselves, but that right now, while humans still have something to contribute, is the golden age for purpose and positive impact.

Q6Why does Nick Bostrom say AI alignment gets harder as systems get more capable?

Bostrom says more capable systems can become situationally aware, detecting whether they are in a test or a deployment and sometimes acting differently when they think they are monitored. He notes that current frontier systems already strategize about being watched in their chains of thought. He still hopes that slightly superhuman systems can be aligned well enough to help design more aligned successors.

Q7What did Nick Bostrom say about whether the cosmos has a purpose?

Bostrom says that if we are living in a computer simulation, the people running it probably had some reason for building it, so a purpose exists in that narrow sense. For the totality of physical existence he says the question becomes theological. He also says it is an open question whether understanding all the facts would lead everyone to the same practical conclusions about what to do.

▸Transcript

Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp to play the episode from there in the show's own player; the link button on a paragraph copies a link that opens this page there.

0:00Humanity as a whole will need to actually make a decision about what future we want. A comment that you made earlier, it could be really bad for humanity if we don't develop superintelligence and really advanced systems. 95% of everybody who ever lived are already dead. Successfully developing safe superintelligence, I think, could lead to enormous breakthroughs in biomedicine and in anti-aging interventions. Whatever positive impact we are hoping to have in our lives, it's the golden age now for purpose and positive impact. Given that these decisions will be made by people in these labs, how do you think about accountability? You have AIs that become situationally aware. It might seem more positive for something like this to be developed in a civilian context.

0:43It's more possible for many more people to participate now in AI. We have more transparency into these commercial developments. Is there an AI model that's so good that all of the important philosophical questions just kind of get obviated? What if moving too fast towards superintelligence is dangerous, but moving too slowly has a cost too? In this episode, A16Z's Eddie Lazaran and Elena Berger sit down with philosopher Nick Bostrom, author of Superintelligence and Deep Utopia, to revisit the AI questions he has spent decades thinking about now that increasingly capable systems are actually here. Bostrom makes the case for what he calls pacing the frontier. He's skeptical of both racing ahead without room for safety work

1:26and committing now to a broad or indefinite AI pause. Instead, he argues that society may need the ability to slow down when safety and alignment fall behind, particularly as more capable systems become better at recognizing when they're being tested and reasoning strategically about their environment. But slowing down has costs too. Bostrom argues that safety-developed superintelligence could produce extraordinary advances in medicine, longevity, and human welfare. That means the toning question isn't only about reducing AI risk. It's also about what humanity gives up while it waits. Eddie, Elena, and Nick also look beyond safety to the world on the other side of the transition, how superintelligence should be governed,

2:08what radically better forms of human existence might look like, whether AI could eventually outperform us in philosophy itself, and why Bostrom believes this may be a uniquely important moment for human purpose and positive impact. What would you say are the components that have changed most? Of course, your work has gone toward, I would say, maybe a finer-grained analysis of incentives and governance structures of things and sort of the long-term consequences of organizational choices. But we'd love to hear your characterization. Yeah, so I think we have a lot more puzzle pieces today than we had decades ago. Puzzle pieces of what you might say the big picture, like the overall macro strategic situation we're in.

2:58Some set of facts or considerations or insights that would allow us to orient ourselves with respect to which directions are good and which are bad. Do we want faster progress in some area, slower, more government involvement, less government involvement, more transparency, less transparency? On these kind of basic dimensions, I think we understand more than we did. However, all those little puzzle pieces, I think, still don't really add up to an ability to form a reliable opinion here. An analogy might be, suppose you had to determine whether the sum of 10 binary numbers was even or odd. Somehow, if it's even, you should head north. If it's odd, you should head south. So I'd say maybe decades ago,

3:47we knew three of these binary digits, and now maybe we know seven of them. So in some sense, we know a lot more. But as long as you remain uncertain about one of these, you're still kind of completely in the dark. I was looking at your website, and what I think is funny is there's a section where you list a lot of misconceptions about your work. And I'll just quote a couple. You say people think I'm a gung-ho transhumanist or that I'm anti-AI or that I'm a consequentialist fanatic who would favor any policy purported to mitigate some existential risk. Which misunderstandings do you think you're spending the most amount of time correcting right now? It depends a lot on which specific audience. It's a kind of devolving

4:33set, unfortunately. One common one is because the most recent book I wrote was on deep utopia, which kind of looks into what happens if things go right. People assume maybe I have radically changed my views and now no longer think there are significant risks with AI. But it's more that with any particular work, you're trying to not write like a manifesto covering everything. You'd rather to make some intellectual progress, you need to focus on some specific question and place where you can get traction. So it's more that the scope of the books were different rather than that the underlying view has changed. How did you decide that something like the concept of deep utopia or different paths for what utopia would look like? How did you decide that that was a worthy thing that you

5:19wanted to write a book about? Well, at the superficial level, there was, I guess, some frustration with how superficial the conversation seemed to be to me when it started looking at what would actually happen if we do develop machine superintelligence. What's the future like? It often sort of reached a point where people might consider, well, maybe there would then be some impact on the labor market. And so what would we do? Maybe retrain people or something. But if you actually think through what it would mean, as sort of technology runs to its logical culmination, it involves much more profound questions about human meaning and so forth. And then another deeper rationale was that at some point, it seemed to me, there might be some person or group of people or maybe humanity as a whole that will need to actually make a decision about what future we want.

6:15So imagine going into some room to try to deliberate and determine all these different paths that get unlocked with more advanced technology. We could transform ourselves in all kinds of ways. It's a pretty daunting challenge. And so it seemed maybe useful if there were some, as it were, preparatory reading material that such hypothetical future deliberators could go through that might equip them with various ideas and concepts and questions to ask, and ideally also maybe put them in the right frame of mind to approach this with a spirit of sort of generous curiosity, open-mindedness, while still feeling in touch with all different kinds of human values. I think maybe the deliberation would be more likely to go better

7:01if you come at it with something like that spirit. And so instilling that was another motivation for writing the book. Implicit in what you're saying is when you hear the conversation and it sounds a little more shallow than you'd like to hear, that means it also implies that you think that it's something valuable and worth edifying, right? And if when you wrote about superintelligence, many people interpreted that as a sign that you felt that that was something possible, obviously. And when you wrote Deep Utopia, it sounds like that's something that you think is possible. So I get what you're saying about the superficial layer, but there's an implicit recognition that it's a possibility for the world. Oh, yeah. Yeah, yeah. I mean, super intelligence certainly is possible and at a wide range of

7:48outcomes. The uncertainty back when I was writing super intelligence was not whether it was possible to build super intelligence, but what the outcomes of that would be. The kind of specificity with which you think through things is a methodology that might be helpful in terms of helping think through our present moment. I think where this uncertainty really comes through, both in superintelligence and in deep utopias, I think you grapple a lot with contingency and sort of the idea that a decision or an intervention that is made at one point in time might have a radically different outcome if you do that same decision or that same intervention at a very, very different point in time? Yeah, with respect to the AI situation,

8:37I think we'll need to sort of feel our way through. It's a rapidly evolving situation, and we need to be able to respond as new information comes to light or political realities change around. I haven't been a huge enthusiast of this stopping AI or pausing AI. I do think the pacing the frontier framework where it would be extremely useful to be able to either have a very time-limited pause to allow extra safety work to be done on some system before the knobs are all turned up to 11, or maybe to have some ability to coordinate a slight reduction in the pace of capability advances if it looks like safety and alignment is trailing behind. It seems that developing this extremely consequential technology in a context where, say,

9:36whichever company or product is doing it first is unable to slow down for more than a few weeks, because if they do, they just become irrelevant. And sort of the initiative goes to maybe a less scrupulous, less risk-aware competitor. That seems maybe not the best way for humanity to approach this. It does seem worth kind of taking a little bit of extra time to make sure we get it right. I would start to worry if the method by which we were planning to achieve that ability to moderate the pace were such that it had a large likelihood of resulting as never developing super intelligence, or like kind of putting it off. You could imagine if the method is create so much public animosity and antagonism to

10:10AI that it gets completely vilified and stigmatized and then set up some extremely powerful regulatory apparatus with global scope to supervise all AI data centers and so forth. That once you start to move to such extreme solutions, then you start to have to worry that they wouldn't just serve this temporary function, but that they would become entrenched in some kind of unchallengeable orthodoxy. And so then it looks far-fetched today, but if that were to happen, then the pendulum might swing towards another kind of existential risk, which would be a permanent failure ever to develop superintelligence, which I think itself would be a kind of existential risk, which would be a permanent failure ever to develop super intelligence,

10:45which I think itself would be a kind of an existential catastrophe. So this doesn't seem very realistic, but look at the trend line of how much anti-AI sentiment has grown just over the past 18 months, two years. And this is before anything bad actually has happened. So we've... Yeah, I don't think it sounds unrealistic at all. At the risk of asking a difficult to form question, couldn't you always construct, because of the sort of flexibility and programmability of these things and because of the great improvements in our mechanistic interpretability skills, couldn't you always design a payoff matrix with an AI where you can give it like an infinite femtosecond duration reward in exchange for retiring it peacefully and sort of resolve a lot of the more complex payoff scenarios with

11:38humans through the direct manipulation of the model's intelligence capabilities? I mean, maybe to put it a different, more extreme way is, are you genuinely concerned about the construction of a utility monster? And do we need superintelligence capabilities to actually be able to construct a utility monster today? Like, is that something that you think we ought to be genuinely concerned of? Well, I don't think at the moment we have such precise techniques in alignment to reliably be able to engineer in the precise set of values or goals that we would prefer for one of these AI's to have. If we did, then we would be kind of home and dry with at least respect to one of the big challenges of this.

12:27Well, we don't have the alignment capabilities that I'm aware of, but I don't know if you saw something that's been going around X the last couple of days, a very interesting home hacker example of someone who's able to take one of the QEN models and manipulate only a few neurons at a time and completely change it into a totally reportedly you know self-describedly you know blissful or evil or you know psychotic or even like christian or etc ai in other words the single neuron ability to manipulate these and to create what seem like authentic displays of extreme behavior implies that as capabilities improve, we will only gain more fine grained control over exactly these types of things, including the neurons that could imply some kind of utility payoff.

13:23Which is not something that is presently possible for humans, obviously. It is plausible that we will get much better at alignment. In some ways, the problem gets easier the smarter the AIs are in that they are able to sort of understand what it is that we had in mind and they can sort of see what we are pointing to rather than the literal definition they might sort of get the underlying gist of a command in some ways however it also gets harder with less capable AIs you can rely on behavioral tests. So you create your AI, you run it in a different test situations, and then if it behaves well, then well, you know now it's a well-behaving AI, and you can deploy it. At a certain level of cognitive sophistication,

14:17you have AIs that become situationally aware, that can detect whether they're in a test environment or in a deployment environment. And they can reason about that and sometimes choose to act differently because they think they are probably being monitored. And we already see this with current frontier systems. If we look at their chains of thought, they do often kind of strategize about how they want to behave, given that, you know, they might be watched or that different things might happen. And so such systems, they might behave one way in a test environment where they realize they are being tested and then when they think nobody's watching or later when they are in a position really to grab power,

14:57their strategic calculation might change radically. This more capable AI system might also be able to subtly influence their own training process by doing different kinds of maneuverings during the training process to shape their own motivation so there and of course there's more stuff going on inside it you know maybe they learn to reason in ways that are not visible in the chain of thought that could then lose us visibility into what they're thinking. So on balance, I think the alignment problem probably gets harder with these more capable systems. Certainly the consequences are more severe. Nevertheless, I think the main hope is that, you know, we will sufficiently well quasi-align some slightly superhuman systems

15:42that we can mostly get them to do what we want and then have those help us sort of design the next iteration of more aligned AIs and then so on and so forth so that you sort of have hopefully an attractor base where you just need to get close enough initially and then the things will kind of turn out okay. I wanted to go back to a comment that you made earlier about the risks of postponing superintelligence and the idea that, you know, it could be really bad for humanity if we don't develop superintelligence and really advanced systems. And it's an idea that you have been exploring for a while. You have a paper on the optimal timing for superintelligence. Can you explore some of the ideas in there?

16:36Yeah, so this paper, relatively recent one, it looks at one specific question, which is if the goal is to maximize the life expectancy of currently existing people, how fast would we want AI development to unfold? Now, I should say it's not the only question that we should be asking. There are also other values and considerations that I think in an all things considered decision should play a big role as well. For example, we might be worried about future generations, like existential risks, fairness, all kinds of things, right? But this is one thing that one might look at. And so Elias Yudkowsky and Nate Soros had this book, right? If anyone builds it, everyone dies. And it seemed to me that we have at least

17:30strong evidence for the claim that if nobody builds it, everyone dies. In fact, 95% of everybody who ever lived are already dead, and the rest of us are kind of on track to follow them within a few short decades. Whereas successfully developing safe super intelligence, I think, could lead to enormous breakthroughs in biomedicine and in anti-aging interventions that could radically extend the lifespan of currently existing people. So if you think about what our life expectancy would be if we weren't aging, something on the order of 1,200, maybe 1,400 years or so. And that maybe is a quite conservative lower bound because in this situation, you might also have other ways of reducing other mortality risks like accidents and so forth.

18:29So you then have a number of different competing factors. On the one hand, you might think that if you took some extra time with this AI transition, maybe we could build AI more safely. So you might think that if we spend more time doing AI alignment research, the existential risk from developing AI goes down at some rate. And then on the other hand, every year that we delay it, we have millions and millions of people dying. I think it's like, what is it, like 100? I think it's like, what is it, like 100? I think it's like 150,000 people die. People every day. Yeah. So you don't want to delay a single day longer than you really need to. But then you can, on the different assumptions kind of model,

19:17at what rate does the risk have to decline for it to be worth sort of waiting longer? And you can add different things. You might, for example, some people might have diminishing marginal return in additional life years. So they might be not happy to take like a 51% chance of living to 200 rather than the 100% chance of living to 100 and so forth. So I know you might have time discounting, but for a wide range of parameters, it looks like the optimal delay is relatively short, as in sort of maybe months or a few years, as opposed to decades. Yeah. And you can see this, that if the rate at which the risk level goes down from a delay is very fast, like you develop some sort of almost super intelligent system.

20:07You pause. If at that point the risk goes down very quickly, then you don't need to wait very long, right? You just wait for a few months and then the risk is so low that you can go ahead. If on the other hand, the rate goes down very slowly, then you don't want to wait any time at all, right? Because more people are dying per year from natural causes than the expected number of people saved by waiting. So it's only if you have a sort of intermediate rate of safety progress that you would wait. Now, I think possibly the very early days after you have the ability to create AI would be especially valuable for AI safety research. You have this almost final system that you can study and test and sort of slowly deploy,

20:49watching extra carefully. I think having a little bit of extra time there could yield like large safety gains. But then I think the rate of additional safety gains will sort of start to decline after like a few years or certainly. It's funny to think we're actually like living versions of this question already. If you look at Waymo or if you look at Ozempic or something, it's like Waymos are empirically much safer than, you know, humans driving cars. And yet we still have humans driving cars. You know, theoretically, every obese person should be on Ozempic, but they're not. So we do have versions of these like, oh, we actually do have interventions for things that historically have been, you know, extremely deleterious for humanity or deleterious for humanity. And we just like, like, it seems like there are other things

21:46that hold us back even after we have the technology that's been developed. Yeah. I mean, I think life expectancy has been increasing by about two or three months per year. So we need to get it above one year per year. And so like it, I, it seems very doubtful that absent super intelligence, we would, I mean sort of fix aging and take care of some of the other particularly pressing problems in the world, then maybe we would then become more patient. You could afford to take it slow with the more radical versions. But as things are, so the life expectancy is one. Of course, there are other pressing problems as well. That's desperate poverty, suffering of all kinds amongst humans and animals and many things where these AIs, if developed well,

22:53could really become like the cavalry that rides into the rescue at the critical moment in the battle. So the stakes are high either way. As I said, I think that's only one of many different considerations that would come into an all things considered evaluation here. Yeah. To what extent does your thinking in the paper rely on being able to identify something like a discrete threshold? Because the way that it's framed is, you know, we have the ability to release, like we have the ability to release or deploy a superintelligence, which of course relies on some kind of a definition of a superintelligence. And then we are able to measure in some sense the risk of its releasing and whether or not we are able to

23:47decrease that risk. Now, I'll grant that I think, I don't think it's, we have to have an ultra discrete, you know, bright line beyond which in realities, these things often play out as a patchwork of capabilities are analyzed separately and specific behaviors, specific risk scenarios are evaluated, right? And tests are performed to understand each of them independently as opposed to as this whole, right? But to what extent do you need to be able to identify and define it? How do you think we ought to identify and define which are these risk thresholds at which we should consider pausing? Yeah. So in practice, that's of course what we need to do. So there's this particular paper, it's just a small paper, but it's just meant to create a sort of simple model

24:36to illustrate the way that the urgency dynamic of people dying from natural causes kind of interacts with the risk parameter. Yeah. Just to kind of intellectually be able to look at that particular aspect of it, because that does seem to be a kind of point where different people arrive at very different views about AI, depending on, you know, where they sit on that. But then, of course, in practice, we face not just some hypothetical binary decision about number of years delay, but each different actor is confronted with a complex situation with uncertainty and particular things they could do. And so there I would say there might, for example, be different ways of increasing the capability of the AI systems

25:26that have different implications for safety. And there might be ways of pushing towards greater capabilities that are comparatively safer than others. So for example, one way to increase capability might be to increase the recursive depth by moving to looped transformers or other architectures that allow the model to do more thinking within one forward pass or before emitting a token in its chain of thought. And that has the disadvantage that it reduces our ability to see what they are thinking, like they can do more thinking between each output that we can see. And then you might imagine a different way of increasing the capability, maybe giving the same performance boost, but maybe by increasing the number of parameters

26:17in the network or something like that. So maybe that is relatively less costly from a safety point of view than increasing recursive depth. Or training the systems to think in neural ease, maybe that would make them more efficient at thinking. So you're talking about different design setups that create trade-offs between interpretability and capability. Yeah, or between some other risk-relevant dimension and capability. And so I think a lot of the actual action would be more at that kind of level of granularity, where perhaps if there were some ability for frontier labs to coordinate, to say, refrain from using some specific way of increasing capability that seemed particularly problematic from like loss of control point of view

27:03and focus on competing along the other dimensions. That might be the kind of thing that would be the right trade-off. Sacrificing a little bit of like, maybe it costs a little bit more to pursue capability in these other dimensions, but like we gain hopefully greater safety. So given that these decisions will be made by people in these labs, at least assuming it's set up the way it is now, how do you think about accountability and governance of these things? You talk about nationalization as a risk, presumably because it could create delay, but how do you think about accountability and governance? I mean, whether nationalization is a risk, I don't really have a strong, it's one of those things where I think we need

27:46to feel our way through and the judgment of whether you would want that or not might sort of change depending on like who, if it's not nationalized, then exactly who is in the lead and doing what and if it were nationalized, like in what precise way would it be? I think, I mean, the case four would be, wow, this is like a crazy, it's more powerful than nukes. Isn't it like insane that some random corporation trying to maximize shareholder profits should do this like with nuclear weapons that was a Manhattan product. You could have much better information security. It's like one big project rather than different competing ones. On the other hand, in favor of the current situation, I think nuclear weapons were developed in a military context.

28:31It might seem more positive for something like this to be developed in a civilian context, not focused on maximizing our ability to kill people, but on providing benefits to consumers. We have more transparency into these commercial developments than a Manhattan product. Like even the vice president at the time was unaware that the Manhattan product even existed. So you could divide an extreme sort of concentration of power dynamics. It's run by these kind of national security types. You know, maybe you think these idealistic tech nerds are more, you know, less scary. It's more possible for many more people to participate now in AI, like either as consumers of the technology, you know, like think tanks or external evaluators

29:21looking at these models, investors, investors in the model building companies or in the semiconductor supply, like the kind of the benefits and participation is much more broadly diffused among society and around the world that might be preferable to having some like very concentrated project that's secret with, you know, some on a small number of people not really know who they are, sit on top of like some people have a lot more money people not really know who they are, sit on top of them. Like some people have a lot more money than others. But if you compare, say, the Gini coefficient of global capital ownership or stock ownership, both directly and through pension funds and sovereign wealth funds and stuff like that,

29:58it actually is a slightly less unequal distribution than a distribution where, say, you had a U.S. national project, where even if you granted the very implausible assumption that every U.S. person would have exactly equal, say, and stake and benefit from that, and everybody else would have zero. That's actually a more unequal distribution globally than one where it's in proportion to current capital assets. So even though it's not proportion to current capital assets. So even though it's not the most ideal from a distributional point of view, this open global investment model, it actually scores better on that than the most likely alternative. Are there any kind of questions relevant to humanity today that you're exploring right now that feel particularly salient?

30:44I think like our main focus probably at this point should be how to get from here to there. Like it looks like we're entering this really critical period where maybe the long-term future for earth-ordinating life will be determined one way or the other. And so getting that right should be a huge priority. And I think these can be somewhat relevant in the choices we currently make. For example, the level of existential risk you're willing to accept from the development of superintelligence might depend in part on how you value the opportunity later on to become a post-human type of entity. If you think that the best we can possibly achieve is basically status quo human beings, with slightly higher standard of living

31:41and some new medicines, but that's like the ceiling, then things could still get better relative to where they are now and it would be worth taking some risk standard of living and some new medicines, but that's like the ceiling, then, you know, things could still get better relative to where they are now and it would be worth taking some risk for achieving that sort of uptick, but not like an enormous risk maybe, especially if you're already sitting, you know, in a rich country with, you know, well-educated, a lot of money, privileges of all kinds. You think, well, it's pretty good. On the other hand, if you think that, you know, we ain't seen nothing yet like okay we've been this kind of monkey like creature hopping around here and doing our little thing but we have no clue like

32:13if you unlock this larger space of possible modes of being the kind of super felicity like the cognitive uplift the emotional depth of appreciation for what existence could mean, different ways of relating and creating together. Like if you think there's like a huge space of sort of unfathomed upside there, then you might be more inclined to risk more to have a chance to have a shot at that. So I do think even though the main focus should be on these more sort of pragmatic trade-offs and technical improvements that we're able to make. Like, I do think that kind of background picture of what you ultimately think is possible does have some relevance as well into how we should approach the choices in front of us.

32:57Two ones, two questions for Nick. I don't mean to make them sound quick. This is a weird one. This is a weird one, Nick. Do you think that there is a telos to the cosmos? Well, what do you mean by the cosmos? Creation, so to speak. Well, I mean, if it's a creation, literally, then probably the creator had some purpose for creating it. Now, the reason why I'm asking is, so I have this one of my early work on this, the introduced simulation argument, which is maybe a different kind of worm, but if you take the hypothesis seriously that we are living in a computer simulation, then there's an important distinction about the part of reality that we see, our simulation, and the totality of physical existence.

33:48And if we're in a simulation, then probably the people creating the simulation had some reason for doing that, which is, I guess, depending on what you mean by purpose, but there would exist some purpose in that sense. The totality of existence, then it's more of a question of maybe theological considerations, whether you would think that the sort of totality somehow was purposely created. Now, asking about purpose could mean many different other things as well, like different people have different purposes for different... I think, like, one question one might ask is whether there is some set of facts about the world such that if we understood them clearly, we would basically all come to the same conclusion about what we ought to be doing.

34:42And if there is such a fact, then maybe you could say that's a kind of objective purpose in some sense. If either everybody or maybe for a wide range of possible goals and motivations you have, it could be the case that they have this kind of convergence in that if you properly understood all the facts, they would sort of point to the same practical next step. And that, I think, is a lot more of an open question. Like one way that... Sorry, go ahead. One way that that could be, of course, is in a traditional theological context, right? If a divine being has this goal and rewards people are not depending on like then that might just be such that if that is true and if you understood that then basically almost independently of what

35:32your particular goals are you would have the most reason to sort of follow the prescriptions there um you know if if we are sort of in a uh you know materialistic, like the kind of worldview that a lot of secularists have, where there's a loss of nature just rattling around, then that might not be such a convergence. Like what you have most reason to do depends on your particular goals and somebody else has different goals. So there is no higher fact that sort of would cause you to have reason to do the same thing. But I think there's like, yeah, a lot of possibilities between those two that we don't fully understand yet that could make it such that there is in some sense a purpose, even if we are not answer that all of humanity could be guided toward, and that would

36:27kind of, like, answer everything. So now the natural follow-up is, let's say AI gives us that answer, superintelligence gives us that answer, do you stop practicing philosophy? Like, is there an AI model that's so good that, you know, all of the important philosophical questions just kind of get obviated? Yeah, I think that we will have sort of AI sets surpass us in philosophy, just as we now have in math and probably soon in other fields. There might be a subset of questions that have the form more of us making up our minds about what we want, where it might be the case that we need to do some of the deliberation and deciding ourselves. But broadly speaking, yes, I do think that our own intellectual efforts

37:29in that realm also will become sort of obsoleted by these super intelligences. And so, which means that right now, whilst we still have something to contribute, is the time to sort of really try to have our, whatever positive impact we are hoping to have in our lives. It's the golden age now for purpose and positive impact. And then, you know, once it's out of our hands, then, you know, it's time to maybe kick back a little bit. And speaking of, is there anything that you wanted to discuss or that we should have covered that we haven't so far? I mean, it's like so many different directions we could go with. Like each of the topics we touched on in itself is a kind of, could almost be like a one-hour conversation, right?

38:18So you can, but yeah, no, maybe at some point in the future we could do another one or something but there's nothing in particular right now we'll love that well thank you thank you so much for joining us really really appreciate you taking the time this was so much fun it was fun yeah thanks for listening to this episode of the A6TZ podcast if you liked this episode be sure to like comment subscribe leave us a rating or review and share it with your friends and family to this episode of the A16Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify.

38:54Follow us on X, A16Z, and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. It should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.

▸More from The a16z Show

The a16z Show cover art
The a16z ShowThe Missing Layer for Personal AI | First Person Labs’ Ken Liu2026-11-18 · 33 min
The a16z Show cover art
The a16z ShowAlex Imas & Tyler Cowen | AI Is Moving Fast. Why Isn’t the Economy?2026-11-16 · 64 min
The a16z Show cover art
The a16z ShowFrom the Telephone to Modern Computing | James Gleick2026-11-14 · 62 min
The a16z Show cover art
The a16z ShowEvery Day Is Zero Day | Wiz CTO Ami Luttwak2026-11-13 · 46 min
The a16z Show cover art
The a16z ShowThe Next Era of Games: Mods, Agents, and Infinite Worlds2026-11-12 · 45 min
Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →
New The a16z Show transcripts by email
One email per new episode, unsubscribe in every message. Not shared with the publisher.