What Senior Engineers Are Actually Doing About AI

0:00Everything's chaotic right now. Which do you see as the more likely outcome? That we will still need some understanding what the LLMs and the AIs are doing within our systems, or we don't need to have any sort of understanding about what's going on underneath the hood? I'm holding on, and I don't know what's going to happen. I think that would be a fool's errand, is thinking that we have a stable workflow pattern for AI. So let's start with juniors. What's the advice that you're giving them? Whatever it is you do with AI, make sure that... Today we're speaking with Dimitri Spanos. Dimitri, you have a long history of AI and ML over the course of decades. I'd like to zoom in on what your sense is,

0:47like what you're seeing on the effect of AI to careers. Yes. So this is something I think about a lot and I also hear about a lot because separate from my own professional work, I have an informal mentoring network of engineers who are generally in the early 20s. And so I hear from them a lot about concerns that they have about AI in their careers that certainly you and I never had. I can speak to a few different angles and you can tell me what you think is most productive for the audience. The one angle is long-term adoption trends in AI of organizations over the course of decades. And one of the patterns that I've seen there is AI and ML trickling down from something that maybe a few specialists would do at a place like Google or Facebook,

1:38that was 20 years ago, to something where you would have a team, like an ML team of four or five people at many organizations in like 2015. And then now we're reaching a point where AI and ML are touching almost everybody's job, at least in the tech business, one way or another. A second angle is seeing how the junior people in the very different concerns that they face from the concerns that my peers and colleagues face who are mid to late career. And both are being maybe not threatened exactly, but at least disrupted by progress in AI and especially coding, but AI in general. Those two groups face very different incentives and very different problems that they have to solve. I'd love to do everybody. So let's start with juniors.

2:28What are you seeing there? The biggest thing I see with juniors is that if you're fresh out of university, I just see a lot of anxiety about whether any of the skills they got will matter in five years. You know, there are AI coding boosters and AI coding skeptics. And even if you just take the full range of that, even on the side of the AI coding skeptics, I think most of them would agree that it's getting better enough, fast enough, that probably in three or four years, it will be a substantial part of at least low stakes coding engineering. And I know that sounds very minimal. I'm specifically giving what I think the view of the world is from the skeptics case. From the boosters case, we're basically already there for anything, but the hardest problems

3:14you can just use current frontier models. And you don't really have to think about it very much. You don't even have to look at the code. You can just have a testing system and have AIIs generating testing systems and you can ship. Right. So I recognize that there's a range of opinion there. But I think even at the most skeptical part of the range, I see people who are who were six months ago were saying, probably I would never use this, even though it seems interesting, to now saying, it seems inevitable that we will be using this heavily within a couple of years. So that's one part. What's the advice that you're giving them? Everything's chaotic right now. So it is chaotic. And I've been doing this mentorship program for about 10 years now.

3:59And so 10 years ago, I knew with substantial confidence what to tell people for, you know, if you are just out of school and maybe you're not from the kind of school that easily places people into, you know, big tech positions, but you want to get there somehow. I had, I think, pretty confident advice about, you know, here are the advantage, for example, of investing in Go in, let's say, 2016 or investing in Rust in 2016. Those were things that I could say pretty confidently, these are going to be growth industries over here and these are going to be growth industries over here. It's harder now to separate that from AI tools in general. And so the advice I have been giving with seemingly some success, right now there are probably, I think, seven people that one way or another I'm kind of informally watching them as they develop their career in the early 20s stage.

4:51The ones that I end up connecting with are also ones that are very interested in maintaining the ability to write high quality code by hand. So that's just a personal preference, right? I'm not saying everybody has to be like that. But I personally want to maintain that skill, even if I understand that its role in the industry is changing. And so the people I end up mentoring also tend to have that preference. But increasingly, I see that those people are doing most of the work for their jobs as maybe like 70 or 80% AI, and then a 20 or 30% part that is human review and editing on top, that seems like it will only increase. It's hard to see how that cohort of people who are just now, for example, getting into

5:31Google or Amazon or any of those companies, it's hard to imagine that when your career starts that way, that the AI use will substantially decrease. That's my sense. But you pointed out this important contrast, which is 10 years ago, I was very confident Google would be a growth skill and would help you for sure getting into one of the big techs eventually. Now, I don't know what to say about AI, except that it seems clear that you can't ignore it. And so somehow we have to figure out, all of us, juniors and seniors alike, have to figure out how to live with it. What do you think about the concept of understanding what the code is that the AI creates? If you're a junior, should you still be going into data structures and algorithms?

6:17Should you still understand what's going on under the hood, how the hardware works, what the design of the system is? I'll give you an answer in two parts. One is just my personal bias, which is that's partly why I got into this business in the first place. So for me, just speaking personally, that's something that I like and the people I work with usually like that as well for its own sake. So that's my bias. Let me not tell you what I think is practical for someone who's thinking about career growth. Quick word from today's sponsor, Linear. Linear has been a sponsor for almost a year now. And in that time, a lot has changed in how teams build software. Coding agents like Clog Code blew up, and they're being used to write more and more code.

7:01Writing code got fast, but reviewing it didn't. A lot of the speed that you get out of agents just gets swallowed back up in review. I spent 18 years as an engineer at Amazon, a lot of it reviewing other people's code, so I know the feeling. You get a huge pull request with changes in some random order, the context is scattered across other tools, and at some point you're either grinding through it or rubber stamping it. What people forget is that code review is how accountability spreads across the team. When you approve a PR, you're putting your name next to it. You own that change now too. That's the part that only a person can do, deciding whether the change fits the system and solves the problem the customer actually had.

7:43So a rubber stamp just puts your name on code that you didn't read. That's what Diff's is for. It's a code review inside of Linear, right next to the issue that started the work. It opens up instantly and breaks a giant PR into readable chapters. The Y sits next to the code, so you can decide whether it solves the problem instead of hunting through tabs to remember what the problem was. Linear used to track the work. Now it reviews it too. I think you could, at least in the very short term, like the next two or three years, you could just ignore all that stuff. And it would probably work because the models are getting so good and people are so motivated to hire people who are AI natives and who are adopting AI

8:26agentic workflows, I think in the short term, you could get away with it. And so anything past the short term, I'm speculating. I worry that if you don't do that, then eventually you are not going to know how to evaluate the answers that an AI gives you in, say, three years when you have to do something that is above the scope of a typical junior engineering task, right? So a typical junior engineering task is maybe a single feature or a single cluster of bugs on one part of the system, and everything's tightly focused, and it's clear what success and failure mean. It's clear why we're doing it. If you resolve the bug or you ship the feature, then it's done, and you don't have to think about

9:04it very much anymore. That's typical, I think, of junior engineering work. As you become more senior, the decisions you make and the systems you build, and especially the systems that you choose to integrate or adopt, those systems have long-term consequences that are harder to measure. Just picking something very concrete, it's difficult to, for example, to evaluate, we changed everything to go or we changed everything to rust. Just picking something very concrete, it's difficult to, for example, to evaluate, we changed everything to Go or we changed everything to Rust. And now everything is 20% faster, whatever, right? How much of that was the fact that you chose Go? How much of that was the fact that you rewrote it? How much of that is changing demand patterns in the usage? All of

9:40those things are very hard, very hard to evaluate in a strict quantitative way. And that's where intuition and fundamental understanding are, for me anyway, deeply tied, right? Whatever intuition you have about the behavior of your system, that's coming from you having an understanding that's at least one level below wherever you're working. Let's just say, hypothetically, you're working in Python. One level below that is at least understanding what's the Python interpreter doing? What are the native data structures of the Python interpreter? What's the memory consumption of a dict versus a list versus whatever, right? Those are all things that in the short term, I think you could just ignore right now.

10:19The thing that is less clear to me is if for most people that will bite you in three years and how that will bite you. Because the LLMs also, they're getting better at this kind of broad informal reasoning. So most of the talk lately these days is just about how much better it's getting at coding and closing linear tickets, right? Like very, very quantitative, specific objective things. We don't have good metrics for the broader reasoning powers yet, but just my own subjective opinion is that they're getting substantially better there as well. I know that's not very concrete in being able to tell you, hey, it's 7% better at informal engineering reasoning this year, and it'll be 12% better next year. But that's something that is changing. Maybe I can summarize that by saying,

11:09I think you could get away, I'm not endorsing this, but I think you could get away with not understanding data structures and algorithms for a couple of years at a minimum. And then it's a question of how quickly LLM progress catches up to the informal engineering wisdom that is not easily captured in metrics. Are you team read the code or are you team like Vibe, you just care about the outputs? For my own personal work, I am on team read the code for sure. For client work, my job working with clients is I'm not usually building the application. I'm helping them figure out how to adopt AI technology and AI patterns. I think that with clients, there's, I think the split, the read split versus the just watch the output split,

11:52you can sort of predict it just by knowing the team. It really seems very personality driven. I know people who are very heavy users of AI generated coding, but still they read everything. Maybe it generates 10,000 lines, but they still read everything. And it's a little bit slower, still much faster than writing it yourself. My experience has been, at least with clients, that it's much more of a personality choice than it is like an engineering management choice. For me personally, I'm on team read the code, but also I do pretty specialist stuff like mathematical computing and AI internals. And there it really, really matters. If you get like a minus sign wrong, or if you're multiplying by one as an integer versus like

12:31one point something as a float, those are all things that are very subtle. And you have to read the code to verify the correctness because you can't easily verify the correctness just from the outputs. You might get, for example, let's say you built an LLM from scratch, right? Most people wouldn't do this, but if you built an LLM from scratch and got outputs, you can't easily verify from the outputs that you implemented everything correctly. That's the kind of thing that I do for my own personal work. And so naturally I'm driven more toward having to read it. But I have friends I respect who are very smart. They just generate artifacts and test them and see whether or not it passes the test suite. I think really the big difference

13:08there is partly personality and partly what kind of work you're doing. So, Michiel Hashimoto states that he has a whiteboard test, basically. So, the idea is that you should understand how your system works such that you could actually whiteboard the system out and talk about what each one of the components do and what their interactions are with each other. What do you think about that? I think it's very reasonable. I have a parallel thing. His might be better actually for communicating to junior engineers. But the way that I think about it is not the whiteboard, but what kind of control surface do you as the engineer have into the project, right? So if you tell me, for example, here's the bad end, the bad end is I asked it to make me an application. It generated whatever, 100,000 lines

13:49of code. I have no idea what the subsystems are. I have no idea what the database schema is. I don't know if there are many different microservices involved. All I know is that when I go to this URL, I get the result that I want. Okay, that's the bad version. The good version is someone can say, okay, you don't like, that's the bad version. The good version is someone can say, okay, you don't like, for example, the latency. The latency is coming from these three components. And here are some of the ways that we can control that. You don't like the fact that the styling is the same in both, you know, like the cat section of the dog section. We need to come up with a way to have different styling

14:21for cats and dogs. And maybe we need to look at some kind of factoring. And that factoring belongs over in this module over here, right? This is a very similar kind of thing where I think about it as the person responsible for the project. What knobs do you have? What's your mental model of the project? And what are the control surfaces, you know, sliders and knobs and inputs that you can put into your mental model of the project so that you can improve it, control surfaces, you know, sliders and knobs and inputs that you can put into your mental model of the project so that you can improve it, control it, adapt it to whatever the new needs are. And that sounds very similar. Maybe it's not exactly the same thing, but that sounds very similar to what you described from Ashwin.

14:56I haven't seen that myself, but it makes a lot of sense that you would want to address the same concerns, that if I put it on the whiteboard, why am I putting it on the whiteboard? I'm putting it so that I can explain to somebody else why do they want to understand because they might want to change something, right? So in the end, I think the most important thing for me is having some kind of sufficiently detailed understanding of what's happening so that you can aim your changes in a reasonable way. If the only thing you can do is say, please, LLM, make it so that I can have separate styling for dogs and cats. Again, maybe you can get away with that for a few years, actually. But I think in the long run, having an understanding of where internal engineering decisions have been made, I think that will remain valuable even if the way we do it is shifting.

15:39Well, I think it's interesting. It feels like there's a lot of compression going on in terms of like frontline and middle managers and the senior engineers or staff level engineers that are doing their work. In this new AI age, I see less and less of a distinction. Kun Chen, he's a L8 engineer at Meta and Atlassian and Microsoft. He says, directors don't know what the code looks like. Directors can't really go in and whiteboard how the systems are going to go. They order the people underneath them to go and do their work, and they just care about outcomes. If I hear you correctly, you're basically saying like, hey, you can probably get away with it for a couple of years, but there is a universe where it sort of catches up to you.

16:25What I'm getting at is like, which do you see as the more likely outcome? That we will still need some understanding of what the LLMs and the AIs are doing within our systems, or far future, we don't need to have any sort of understanding about what's going on underneath the hood. I would separate that from the pure technology question from what I would advise younger people to be thinking about. So one of the reasons I have this framework of what is your understanding that you can use to exert control over the project? That is the value that you are bringing as someone separate from an LLM, right? Because hypothetically, the director kind of person that you mentioned, they could just go type into the

17:04LLM, hey, make the P95 latency better. I don't care how it happens, right? I don't even want to see a whiteboard diagram, just go do it, right? So they could do that. I'm encouraging people to maintain the engineering expertise that lets them do that better than the director so that you still have a role even when the director says, hey, I want P95 latency to be better. Obviously, I'm not going to get involved hey, I want P95 latency to be better. Obviously, I'm not going to get involved. But then that person is going to be wondering, can I just hand that off to an ancient swarm or should I hand that off to Steve or whatever? Someone more junior than you. But, you know, Steve, make sure that someone at the director level still prefers to route that through you rather than routing it directly to the AI.

17:50And maybe I'll make an aside, which is saying you could do this badly, right? Like we all know the one engineer who has the sneaky system for job security that no one else understands. You can do it badly, okay? So I'm recognizing you can do it badly and in a pathological way. Just please don't do that. But you can do it badly. Okay. So I'm recognizing you can do it badly and in a pathological way, just please don't do that, but you can do it in a virtuous way as well. And I think at least over the next five years, which is the timeframe that I'm keeping in mind for, for advising the junior people I'm mentoring, at least over that timeframe, I think we will still, we will still have directors who want to route tasks through junior people, even if that ends up

18:27in the nuts and bolts going to an AI. So I encourage people to develop their skills and their knowledge of the system, going back to the reading the code versus not reading the code, so that when the director talks to you, you can say, yes, actually, I have read the code, and I understand the internal architecture, and I can direct the LLM to work here and here and here, and the director wouldn't know to do that. And at least for now, you can keep an edge over the LLMs in this sense. But that brings back the earlier point I was making about AI is getting better and better at informal holistic engineering judgment. That's very hard to quantify. And I can't tell you that that's going to be 5% better or 20% better next year. But to the extent that you're thinking, how do I keep justifying

19:10myself in this human organization, right? The reason people find junior engineers valuable is that as a more senior person, you can say, look, this P95 thing, it's killing us in the market. Our competitor has a much better P95. We just really need to improve P95, right? That's the kind of decision that happens at a more senior engineering management level. And they want to delegate, right? They want to have humans they trust and say, hey, you person who is five years more junior than I am, I really want to improve P95. Give me the five things that you think we can do. And then I can evaluate the business trade-offs. They have to figure out what's the business impact right because whatever maybe one of

19:47one of the things you do has some impact on cloud services bill or something right and that's not that's not a decision you can make purely technically that's a decision that incorporates business decisions as well so just wrapping up i think whatever it is you do with ai try to make sure that you remain useful to the people who would like to delegate to you. And don't leave them asking, why wouldn't I just delegate to the AI? What's the point of delegating to the junior person what I could delegate to the AI? Yeah, that totally makes sense. Like just getting back to like the control service, what are the knobs? What are the sliders? How can I actually change the system to have the types of outcomes that I want? You should always have a really good sense of that, right? And I

20:31think it's difficult to do if you don't really understand what's going on underneath the hood. I do, though, see the big conflict here, which is that as we learn to use these LLM systems, there's been a marked shift between being prescriptive about how the LLM does the thing versus like being declarative, just being like, this is what I want. I don't care how you get there. I would say that right now that is still iffy. And I think it's reasonable to think, is it going to be that in three years it's no longer iffy? Right. And I can just be completely declarative and i don't have to be at all prescriptive about the internals that's a very reasonable concern and that's that's partly why i personally struggle with the people that i mentor it's hard for me to say here's my best

21:16advice for the next couple of years and i'm sure that at a minimum that's not going to hurt you over 10 years right or that's not going to be missing an important opportunity over 10 years. Now, my ability to model people's careers, maybe it extends out two or three years. Whereas 10 years ago, I could easily model someone's career out for 10 years. So coming back to the question of the iffiness, for sufficiently simple projects with sufficiently simple business goals, I think it's no longer iffy, right? Like a moderate amount of code, tens of thousands of lines of code. If you don't mind what your AI bill is going to be and you don't mind that maybe it crunches for a couple of days before you have the thing that you want to ship.

21:59Moderate projects with moderate goals, with moderate quality expectations. I think it's no longer iffy. I think you can more or less just insert your credit card and press go and get something reasonable. It is still iffy anywhere that you're trying to do something close to the state of the art. So examples that my friends happen to have touched, not necessarily representative of the broader industry, state of the art rate tracing or new strategies in attention for LLMs or different formulations of risk pricing for complicated options. These are all things where maybe the distinction I would make there is that even if you forget the code, even knowing what is right to do in the first place is hard. Like I can't just come and tell you, hey, Steve, here's the best options pricing strategy.

22:46Just go do it and you'll be a billionaire in no time, right? But that's a place still where a lot of ambitious software engineering happens in all of those domains and many similar domains, right? Those are just ones that I personally have friends who are working there. There, I would say it's definitely iffy. If you just tell the AIs, even with an unlimited budget or whatever, like the highest budgets I've seen among client projects are in the range of, even with an unlimited budget or whatever, like the highest budgets I've seen among client projects are in the range of like $5,200,000. Okay. So it's not anywhere like what the frontier labs are doing, where they're spending millions or tens of

23:14millions of dollars, right? So maybe if you have tens of millions of dollars to burn, it's different. But if you only have tens of thousands of dollars to burn, things that are ambitious in the way that those three examples are ambitious, the results are still iffy. And expert-level engineers, people with 15 years of experience, those people still produce substantially better overall architecture than the LLMs would, but still they benefit from the acceleration provided by the LLMs. So let's talk about that. Maybe some people with 10 years, 15 years of experience, people our age, we're in our mid-40s. What sort of trend do you see for people that are happening at this layer? Things are definitely changing.

23:56How are you thinking about the shift in engineering? Is there a way to sort of anticipate what the future is going to be or maybe to buy insurance for the future? I can tell you what my cohort of people, which are people similar to you, roughly in experience and accomplishment. I can tell you what that cohort is doing. And I'm thinking about maybe a group of like 20-ish people right now. People I know really well and I know the internals of their careers. Roughly a third are just worried, right? They're like, I'm at Google. I'm at Amazon, I'm at Meta, I'm holding on and I don't know what's going to happen. That's roughly a third. The pay is good. It's a high quality job. It's a high status job. And it's not totally obvious what that's going to be in 10 years. And if you're

24:40like 22, I understand that these ages maybe seem distant or hard to relate to. But if you're like 22, I understand that these, these ages may be seem distant or hard to relate to, but if you're 45 and you think what's going to happen in 10 years, you're 55, you're not dead yet at 55. And you still need to have a story about what's going to happen in your life when you're 55 or 65. Right. I would say roughly one third of this cohort is like, I don't know what's happening and I'm just trying to hold on. Right. That's one category. Another category is, I don't know who coined this term, solo unicorn. Have you heard this? Yeah. So roughly another third of them are taking this solo unicorn idea pretty seriously.

25:16And they're thinking, okay, maybe you could make an amazing business or at least an amazing product with one great engineer and like 50 pretty good engineers. But those 50 pretty good engineers are just AI. I know people who are trying that. They only really started trying that like Q2 of this year. So results are not in yet. That seems to be one other chunk of how people are responding to it. And it makes sense, right? That you're thinking, okay, I'm whatever, mid-40s in my career. I have lots of domain expertise and I know something about engineering management and how engineering projects get decomposed and what has business value and what has market value, right? If you're thinking you're that kind of a person,

26:00maybe I can just provide all that at the top and try to build a product that I sell myself or maybe gets acquired by a bigger company or whatever, right? Like there are many, many paths to making that financially viable. I know several people who are trying to do some version of that one way or another. A third thing that I'm seeing is people who are, it's kind of that, but like rotated 90 degrees. So they're again thinking, okay, probably, probably my position in this big tech is not stable for 10 years at some point they're going to say hey we just don't need that many java programmers with 25 years experience right that does not dig on job whatever go pick your language right but it's a reasonable concern that within 10 years the executive suite might say we might need some of

26:42these people we definitely don't need all of these people right if you're worried about that and also if you're not someone who has a an idea for a solo unicorn kind of product strategy another thing i've seen people try and again the results are not in is to kind of build um solo app development shops so like up until now we've had like web dev shops and mobile app shops. And very briefly, I met someone who was working, believe it or not, at a metaverse development shop, right? Or like, so, but this is a recurring pattern of, okay, I have this expertise. I don't have a business thesis, right? I don't have a product that I want to build, but I know there are other people out there who want to build products. So maybe what I can do is I can use my expertise and my ability to drive a network of AIs,

27:30and I can go and work with small businesses who want to build custom apps or build marketing presences or whatever, right? The range is wide here for what this kind of economic unit does, right? I know web dev shops and really the only thing they do is websites for restaurants and they do quite good business, right? That's not necessarily economically a bad business, but I'm just saying I know web dev shops who do something that specialized and then I know other, I know mobile app shops that will do almost anything, right? That you come to them and say, I need to ship this on iOS and can you please do that? So this last one third category is people who are trying to do some version of that, where they're planning on eventually exiting the big tech and then

28:11essentially being an AI powered development shop. Those are the three things that I see among my cohort and N equals 20 is not a huge number. So take all of that with a grain of salt. Having said that, I don't see obvious better strategies that people could be adopting other than either hold on or try to do a solo unicorn or try to do a solo dev shop so that the thing you're bringing is the engineering maturity and engineering process management and AI does the rest. Those are the three big categories that I see. Given all the developments in AI, where's all the software? We can create code as quickly as 10 times, 100 times faster than we used to be able to produce code. And if these LLMs are so sophisticated, where is this millions of flowers that are blooming in terms of applications?

29:06Like the application stack, like I'm still using the same apps that I was using five years ago. They haven't had like an explosion of quality and feature set. I don't think, at least from my side, if anything, I think availability has been quite challenged on a lot of the things like GitHub. GitHub has like zero nines of availability. Last time I checked, it feels like there's no there there. You know, software is very expensive. And then we got some set of applications and then software got really cheap to write. And it seems like we're still there. We're still at this like we still have the same feature set and applications that we did when software was expensive. I agree. Let me tell you maybe two parts there.

29:49The first part is just to state what I think is a reasonable skeptics or naysayers case, which is just riffing on what you already said. And then the second part is to tell you something that isn't my idea, but I heard from a friend of mine who is an MBA business school professor. So the first part. The way I look at the negative case is this. One is, is there even one example of an application that was built that is amazing? And we look at it and say, it's amazing just standing on its own. We don't care how we got it. And also, we couldn't have gotten it without AI. I'm not aware of anything like that. So that's the application as end result. we got it. And also we couldn't have gotten it without AI. I'm not aware of anything like that.

30:30So that's the application as end result. Then there's the human business process. Is there an application that's really very good and it got built at 10% of the cost or in 10% of the time or at 10% of the risk, right? Like you can pick your business metric, but that's saying, okay, fine. Maybe AI is not going to build, maybe it's not going to raise the ceiling on what applications you can have, but it can at least radically change the business process that leads to that. I still have not seen that either. My sort of finger in the wind estimate of bulk productivity. So not, not like I used AI to generate this particular PR, but the bulk productivity, which means I look at a team of engineers, I give them AI, I watch them for six

31:11months. What's the economic value that they produce in that six months versus what's the economic value that they were going to produce without AI? My best estimate for sophisticated teams is that that economic value estimate improvement is in the ballpark of 20%. But there are caveats there. And, you know, on the one hand, for one person, I see many businesses, I see, you know, three, four, five a year. On the other hand, that's still not 100 businesses or 1000 businesses that I've seen do this. So that's the second part, which is, is there an obvious home run in, at least in the business process, even if the quality of the software hasn't changed? Those are the two prongs of the skeptics case. And I don't know of

31:52any answers to that. I would love to hear from people, like if you have the app that is amazing and was only possible because of AI, or you have the app that is amazing and is like whatever, as good as anything else on the market. And it was done in 10% of the time or cost. I would want to see that, but I haven't seen that. I've got a friend, he created an app that helps you practice, I think it's called elk calling. So I don't know anything about this, but like if you go hunting for elk, you have to lure them somehow. And so you might do this type of like bull call, I think is what it's called. He's not a software developer. He's very tech savvy. So he was able to vibe code this app. Basically, you practice your call and then it uploads the call to the cloud.

32:38And then it does some analysis. And then anyways, that's the app that's there. This app would not exist without AI, but from the outside, like you just don't know that it was created like completely vibe coded. He's trying to make a, an iOS version right now. It's just sort of like mobile web. So I think that's, that's my first like counter argument to the skeptical case that you just brought up, which is like, if it was completely AI generated, how would you know? But secondly, I think the bigger point is like, I just haven't seen the explosion of apps. So I know of this, you know, again, N equals one case. And when I was talking with him, I'm like, why can't you just play a recording of like a elk call that you know is going to work? He's like, oh no,

33:21that's totally illegal. You're not allowed to use these devices. It has to be like a fair hunt. Oh, wow. Okay. And I was like, okay, I didn't realize there was this whole universe that was going on underneath it. You know, I think the big thing is just like, you wouldn't be able to know if it was completely AI created from the outside. I would just expect like a ton of apps and a ton of functionality and a ton of SaaS to be created. Whereas the order of magnitude change in the outside, I would just expect like a ton of apps and a ton of functionality and a ton of SaaS to be created. Whereas the order of magnitude change in the marketplace, not this 10% stuff, right, that you're calling out. Let me give you an analogy there that I think is,

33:56it's shaping up and your outcall fits into this. So consider this maybe a trend that will materialize over the next year or two. Just a hypothesis. So you're probably familiar with website builders like Squarespace or Wix or Weebly. And like a generation ago, we had GeoCities and whatever, right? So there's all been this recurring pattern of here's a way to make a software-shaped thing without knowledge of software. And it's okay because maybe it's just an amateur effort and you don't care, right? Like if you're just some kid making a personal website, you don't care, right? Or if you're a hole-in-the-wall coffee shop, you don't care that Wix is maybe not whatever, like the most impressive CDN in the

34:47world, right? It lets you bring your personal small context and put it into a technological artifact. That's good enough, right? So if I tell you, for example, that there are, I have no idea how many webpages are on Wix or Weebly or any of those, right? But let's just say it's millions, probably it's millions. And you and I have never heard of any of them. I can't name one, but they're out there, right? I wonder if something similar is happening with AI apps, where it's, you can see the parallel in the pattern, right? That's the, I need to make this elk calling app. and it's like it's not anything anybody's thinking about it only really matters to me and maybe a few of my friends were i guess elk hunters or elk aficionados i don't know why they want the elk right to me

35:34that feels like the app version of a squarespace or wix or like in the elder geocities website right that it's like a geoCities-ish app. That's how I feel about that. I don't know if that will play out, but that's a thing we've seen happen before across two generations. That's why I brought up GeoCities, right? We saw it happen in the 2000s. We saw it happen again in the 2010s. A technology adoption pattern we've seen across two generations. And maybe we'll see the third time now with AI. And I think your elk example is an instance of that. It explains the question of why haven't I heard about any of this? It's because you haven't heard about anybody's GeoCities website or anybody's Squarespace website either, right?

36:21These are just things that are on a small scale that nobody hears about. I think I understand. So yeah, there's more web pages than there's billions of them, right? Google indexes them and then they're serving up the top, who goes to the second page of Google search results. And so it's basically like, okay, we have these recommender systems or these search systems, they're surfacing the best of the best. And I think right now, like you can't just really vibe code a thing that's going to reach that level, that cream of the crop level. So I would maybe buy that argument. I will push back a little bit, which is like, there are plenty of WordPress sites that are out there that people do interact with. It's not because the WordPress site is particularly impressive,

37:06but it's like the fastest way to get a business website up. You know, one of those services and the business can be definitely successful by using some of these things. Yeah, so I agree. That's a good addition, which is there are things that you've heard of that are WordPress sites, but you haven't, probably haven't heard of them as WordPress sites, but you haven't, probably haven't heard of them as WordPress sites. Most people haven't anyway, right? They've developed their own presence, their own brand, their own quality, and it happens to be on WordPress. I probably know someone who's done exactly what you're saying with WordPress, and I'm not even aware because it's, it never occurs to me to check, hey, like I can read the source

37:44and see whether this is generated by WordPress or not, right? Let's shift back to careers. Meta just recently laid off a large percentage of their workforce in their latest round. There are reports that certain groups were going to be eliminated, I think like 75% or 80%. Like they're doing a radical restructuring despite the fact that they have record profits. There are also reports that they're desperately trying to hire back a bunch of managers that knew where all the dead bodies were buried. And then they're also desperately trying to convince people that were managers that went to IC to come back to being managers. And so do you have any commentary or thoughts around the cycle of layoffs?

38:29Is it a situation where seeing companies trying to get more efficient and maybe cutting a little bit too close to the bone with their efficiencies? Or is this a situation where there's like a radical paradigm change in the way that we're thinking about tech workforces. Well, I think there are at least two things happening here. One is that some of the C-suite and the VPs right under them do believe in this radical change. was bolstered by, like him or hate him, what Musk ended up doing with Twitter and X, which is opinions vary on the ongoing quality of the user experience on X. But the thing you cannot deny is that they had radical layoffs. Many people were predicting that it would just fall over.

39:23It did not fall over. It's serving much more traffic now. It's now also serving live video streams. And so at a minimum, it didn't die, right? There are reasonable arguments about how much it's prospering, but at a minimum, it didn't die despite, I remember something like 80% layoffs. It might have been more than that. It might have been more than that, right? I think some people are looking at that and saying, yeah, this supports the idea that maybe it will be ugly in the short term, but actually it's possible. Right. And imagine now, hypothetically, I have no such information, but just imagine you are the C-suite at Meta or Google and you're thinking, could we get away with firing 80% of people?

39:59Right. Because I mean, that's a big bill. That's a really, if I look at the financial statements, that's a lot of money. Right. I mean, that's a big bill. That's a really, if I look at the financial statements, that's a lot of money, right? And maybe we just tank a year or two of really bad PR and unpleasantness. Again, I have no such information. I'm only saying that I can imagine someone saying, look, it seems not to have killed Twitter or Musk. So maybe it's possible, right? I think that's one thing that people are thinking about. Another thing is, this is outside my economic or financial expertise, but I lived through the 2007, 2008 era as well, just as you did. And I saw what was happening in big tech at the time, but also partly in finance at the time, because I'm not in finance, but I have friends who are. It feels a little bit like that. Certainly not 100% like that, but it feels like maybe 50% in that direction. The things are blowing up in the line go up sense,

40:51not in the explosion sense. The lines are going up so quickly and people are wondering, hey, maybe this is a time when we can afford to do some risky things because the lines are going up so quickly. I don't know. That's pure speculation, but it does, it feels a little bit the same. I emphasize not 100%, but like 50% the same as what I saw happening in tech and finance in 07, 08, that was also followed by substantial layoffs. I don't have the economic or financial expertise to judge how similar the situations are, but just from the inside, it sort of feels the same. I think the Twitter case is particularly interesting because I think that was much more like Elon Musk stating that most people here are useless and not actually contributing.

41:38I don't think it was an AI layoff per se. It was basically like he had a thesis that there's just a small amount of people that are actually keeping this place running and I'm going to fire most of the people and then we will try to patch up the holes afterwards. I was one of those groups of people that thought Twitter was just going to keel over and die. It was pretty clear, I think, that Elon Musk didn't understand GraphQL. And so he thought that, you know, there were like hundreds of calls that were being made from the client to fetch all of the little pieces of data. Not that the backend was, you know, had some sophisticated technology to actually aggregate all of the information that was there. Super weird. Sometimes I'm like, wow, this Musk guy must be, must be pretty smart. And

42:18then he starts to talk about, you know, software development. And I'm like, wow, is this, is this what he's talking about? Like, is this his level of sophistication with electric vehicles and with space? Because then maybe things might be in trouble. But I was I was totally one of those people that thought that Twitter was going to keel over and die. It did not. And you might make the argument that there's you know, there's the video stuff that's there. A bunch of people move to other platforms, but it might be stronger than ever. I think Twitter and X are more relevant than ever. Possibly. And I think if I put on my Musk skepticism hat for a moment and say, OK, they fired 80% of people, maybe it wasn't that motivated by deep analysis of the technology stock,

42:59right? I think that's probably fair that it was not motivated by deep analysis of the technology stock. I think the fact that despite all that, despite the crudeness, despite the fact that it was not motivated by deep analysis of the technology stock. I think the fact that despite all that, despite the crudeness, despite the fact that it was such a large layoff, that it's still working. I think if you're a C-suite person, you have to be thinking, okay, well, could we do like 20%, right? I don't know anything about Google C-suite, right? But probably Google C-suite does not think that most of Google is useless. But they might think, hey, I mean, if he got away with 80%, could we get away with 20%? Again, pure speculation on my part. And just connecting

43:28this back to the AI, sorry, but the parallel that I see with the 07-08 was that at the time, people were very, very skeptical that the layoffs really needed to happen. And they were wondering whether they were using external financial conditions as cover for justifying the layoffs so that they would look good in the stock markets. And maybe now AI is being used in a parallel way. Again, I emphasize I'm not any kind of financial or economic analyst, but these are just as experiences of being in the tech business, things I was hearing back then, I'm starting to hear now as well. I'm highly suspicious that a lot of these layoffs were, you know, a big financial move. Stock price goes up when you lay off a bunch of people. That's just the reality of

44:09it. But also using it as cover to get rid of low performers that are difficult to fire, to get rid of, right? And so we're sort of like using that as cover to like get more efficient and more lean, but also just like move people out that were kind of hard to manage out through other mechanisms. When it comes to the meta case, I think it's really interesting because it's like, it's such a self-inflicted wound. When you have record profits, like why rattle the cage and try to like remove so many people from the workforce? You know, and this is kind of the thesis that I'm thinking through. High performers, people that are valuable to the thesis that I'm thinking through. High performers, people that are valuable to the company, people that are making really big contributions,

44:49people that really understand AI technology and how to control agents, those people are being rewarded more than ever in the workplace. Six months ago, we thought that all engineering teams were going to get replaced with product managers and a Cloud Code subscription. We don't need any of these people. Turns out, at least from what I'm seeing, people that are really leaning into AI and applying it to their current workforce, those are the people that are being retained. Where it's really apparent that they're not making a really big contribution, they're not really leaning into AI. Those are the people that are more likely to be let go. But what's your counterpoint? I'm super curious. So I agree with everything you said.

45:35And then I want to add, and then. The and then is all of that is contingent on being a high performer viewed as a high performer through AI. But what if you're a high performer not through AI? And I know several such people who are having unpleasant discussions at their firms and people who are very specialist knowledge. Probably if I told you the domains, it would be enough to localize where these problems are. But anyway, it is not computer security engineering. So let's take computer security engineering as an example. You can imagine someone who has decades of experience in computer security engineering and cryptography and protocols and business policies and whatever, such a person saying, I can't do my job at the level of diligence that the job demands of me using AI in the way you want me to use AI.

46:29And so I know I emphasize this is a hypothetical example, but I know people who are in a parallel situations who have just said, look, I'm putting in my notice. I can't work this way. And this is a person who was prior to AI objectively very high performer, but they feel that they can't do what they should be doing in this AI first style. I agree with everything you said about the people who are most rewarded right now are the people who are most able at least to appear to be increasing productivity through AI. I think some of that is performance. I don't know if it's 20% or 40%. I mean, I certainly know people who tell me, they tell me that I have to be using AI to close at least 20% of my tickets.

47:11So I just pick 20% and call it AI and that's it, right? So some amount of that flow of reward is I think being maybe misdirected is not the right word, but there is some amount of, like if you tell me that I have to say that I used AI to get a reward, then I'm going to say that I used AI, right? That's just ordinary corporate dynamics. Well, I think it's very similar to like the lines of code metric that we might've used in prior years. Yes. Which was like, we're going to measure your productivity based off of the amount of code that you create. Well, that just led to bloated software, right? Like you can unroll a for loop and make as many lines of code as you'd like to, right? Remember when token maxing was a thing

47:58and everybody's like, look how many tokens I'm burning for all of this stuff. Well, if you just simply counted tokens, like you could just, you could basically tell the AI, like, hey, go do a bunch of stuff and burn these tokens. And then like, you know, spend even more tokens to go and clean it up. Voila, right? The thing you want is good code. You observe that lots of ambitious software has a high line count. Therefore, having a high line count is good. Therefore, we reward engineers for producing many lines. And now you have engineers who are incentivized to produce many lines, not engineers who are incentivized to produce useful software. And you just substitute AI in there for line count,

48:35as you say. I think the big reason why lines of code or token maxing is a bad measure is because they're input metrics. Ultimately, what we care about is outputs. When we talk about productivity, it's about the thing that was produced after that. Increasingly, I think a good way to have an insurance policy is to use AI to ship more things at higher speed. Yes, I agree. And this connects back to what I was saying earlier about if you really try to understand what your organization has gained from whatever you did with AI, whatever adoption pattern you chose, I think the relevant metric that is very difficult to fake is take your team of five or ten engineers, see what they produce in six six months and try to figure out what they would

49:27have produced in those six months without ai and what's the gap i've done informal versions of this study with with clients before my estimate is that you can get maybe 20 percent 20 boost in the economic value of what you produce by using AI reasonably. And I have not, again, in my personal experience, I have not seen anything that's like, we produce 10 times the economic value in those six months. Let's talk a little bit about this labor thing that we had talked about, which is at the end of the day, we understand now that PMs with a cloud code subscription aren't going to be taking over engineering teams. And so there's going to be some sort of mix between a human that's a software engineer and then how much we spend on the tokens. Can you elaborate a little bit more on this? I

50:17think we had talked about this in our pre-podcast call. Yes. So this is a discussion that I've had a few times with clients. And I think some version of this is useful framing for anyone who's thinking about team-wide AI adoption or company-wide AI adoption. have the budget for four junior big tech quality engineers, right? So people who are like 23, 24, just past entry level, big tech engineers, and I give you the budget for those four people. And then I ask, would you rather have those four people without AI? Or would you rather have those three of them, and then the salary of the fourth in frontier model AI credits. And then how does that change as the size of the team changes, right? Because you might think differently about three versus four and nine versus ten, right?

51:14Nine engineers. Is the tenth engineer adding as much value as just giving AI to the previous nine? I don't have hard answers there. And I mean, ultimately, I don't make the decisions anyway. I don't have hard answers there. And I mean, ultimately, I don't make the decisions anyway. I just help clients frame the decisions. I see ballpark somewhere around like the three, four range of engineers that people say, okay, yeah, maybe it actually just makes more sense. Instead of having a fifth engineer, we have four engineers plus AI and we accept that the bill is going to be similar to the bill for an engineer's salary all in. That aligns separately with my 20%-ish bulk productivity improvement. These are coming at it from completely different angles, right? I didn't, these are not arranged from the same calculation, but they're in client conversations.

52:06I see both one, which is, what did we accomplish with our team? What extra did we accomplish in six months in economic value? And the other side saying, yeah, if I think about how we work, probably I would take the four engineers with AI rather than five engineers without AI. And at least that's kind of an order of magnitude of the boosting effect seems similar. And there is like five or six, not 500 or 5,000. The difference here is like, you can have the AI tokens or, you know, however you want to meter it. They can be working at all hours of the day. Human software developers, you know, they're going to be putting in on the order of eight hours every day, five days a week. And so I think the economics and the calculus

52:52is really interesting because Dario talks about things like, oh, we'll have a data center of geniuses in a couple of years. I think the big thing is like, is that data center of geniuses going to cost like a hundred geniuses, right? You didn't have access to a hundred of geniuses before, but now you do, but they're going to cost you. The critical piece here is like, I really want to see the balance sheets to these frontier models and to these frontier labs. I know they release top line revenue, but they do not release what net revenues are. And so like, are they just like burning a ton of money on data structure construction, burning a ton of money on compute, burning a ton of money on pre-training and on inference?

53:37And actually margins are really thin. And the costs for replacing a software developer with an AI is roughly the costs of a software developer. It's important to think about this relative to the whole world and not just the West Coast big tech economy because it's one thing if you're talking about replacing the salary of a 24-year-old at Google in San Francisco, and it's another thing if you're in Japan or Shanghai or Mumbai or Riyadh, right? The economics of AI may end up looking very different in different parts of the world just because of different local labor market effects. Yeah, absolutely. I think, you know, I really want to know what happens in the next three to five years with things like outsourcing. Is it that the labor over there is inexpensive and so that we change it over to AI or maybe, so you mentioned that one of the advantages of the AI is they can be running 24 hours a day. Okay. So that's true

54:49mechanistically. But if I ask about the business process of building something useful, my question is how frequently are there what I call judgment interrupts where the process has to stop. And someone who is an arbiter of the value says, yes, this was good. Not just it's good on paper, but it's actually good. I've been calling it judgment interrupts. You just can't proceed with the work unless someone comes in and says, yes, we thought that whatever, changing the style to be based on red instead of based on blue, we thought it would be good. And indeed it is good. Or we thought it would be good, but actually we were wrong, right? So AI, you did everything we asked. You changed all the styles to be red themed rather than blue themed, but actually what we wanted was wrong.

55:33Now that you've built it, we see that we were wrong. So a thing that can work 24 hours a day, it eventually gets blocked on interrupts like this, right? Which is I built the thing you asked me to do. Did this actually make anything useful or not? I do wonder, actually, if lower cost labor power in other emerging markets might have an advantage there in that they could have more people available for that kind of work than we might have in higher cost of labor markets. I get total speculation, but I think that there is something there about the things can grind technologically for 24 hours a day. But if you're trying to build something for a business, something for a purpose, then there are checkpoints where someone has to say, yes,

56:15technically you did solve the problem, but it turns out that the solution doesn't give us what we want. So we need to try something else. How frequently those checkpoints need to be and to what extent maybe an AI swarm could partially simulate that and have judgment AI. I mean, I don't know if you're familiar with this jargon from the literature about LLM as how much we can free that process from having to be blocked on someone saying – a designer or a business manager saying, yes, okay, you finished the work and it's good. Let's continue. Or you finished the work. You did what we asked, but actually we asked for the wrong thing, so now we have to start over. I'm a little skeptical of having human arbiters be outsourced.

57:12The reason is because this is the cycle that I used to see with outsourcing. They take a look at their balance sheet, labor costs are astronomical for stateside engineers. And so they're like, we could get that done for a fraction of the costs in South Asia. So then we go and spend the time to spin up a team over there, train them up. We're able to find a bunch of highly skilled people that are equivalent to the hiring bar over stateside. And then what happens is there's some sort of drift that occurs in terms of like what needed to be built. Communication breaks down, deadlines slip. There's not a good explanation as to why. And then so some people start to go and fly over to the outsource site

57:58and they try to set things straight and things look good. And then they fly back home and then it goes back to where we were before. Then the pendulum swings the other way and it's just like, yeah, maybe the outsource team wasn't a good idea. We're going to, you know, really bolster the team in the state side. I've seen that cycle happen probably three or four times over the course of my career, at least within my career at Amazon. And so when there are these decision interrupts that occur with AI where judgment needs to happen, I think in a perfect world, you would have around the world judgment for every time zone that was there so that you could unblock the development. But in reality, it's going to be in the eight hours,

58:41wherever the company is located in the US or not. This is not a condemnation of outsourcing. I think the best way to have an outsourced team is not by like a division of labor sort of thing where it's like, hey, we come up with the ideas, you go and implement. The best way to do outsourcing is to actually have the team own something from top to bottom. And so basically you've created an organization over there with ownership and they're not just sort of seen as labor. I agree. And I will echo the, I have seen that pathology cycle many times myself where like swinging back and forth between we're drifting and then we sent the executives over there to talk to the local management. And then it looks like it gets better for a week or two.

59:27And then we're just back to the old problems. So I have seen that myself. I completely agree with that. I do wonder how much it matters really to be crunching 24 hours a day, right? So the idea of the – hypothetically, right? If you had executives all over the world or vice presidents all over the world, so that that decision interrupt could always be routed to someone who was in the middle of their workday. I do wonder how much that matters because the speed up on the code generation is creating the actual artifact. The speed up is so high. I do wonder if it makes much of a difference getting that decision four hours earlier than you would otherwise. Certainly, there are people who believe that it matters and that having this continuous like agents, you know, almost driving the humans saying, OK, we finished with our work.

1:00:16Tell us, is it good? And then humans being constantly on call to to provide those judgments. And, you know, some people I see this less now, but six months ago, I'm sure you knew people who were in this cycle of, my agents are working. I need to be up at 3 a.m. in case my agents need something, right? It hasn't gone away. I think that there's still some vampirism that's going on where it's just like, hey, one more. I think prompts are too low level now, but one more prompt, one more task that I want to set them on. And I do think that there's a lot more tooling around trying to automatically keep your agents working. If I took a survey of all of my tech friends, they would all say that they feel some amount of anxiety

1:00:58when they let their subscriptions and their AI, like if there's any sort of slack that's going on. They constantly feel like there has to be something working in the background. Otherwise, you know, they're not utilizing everything to their fullest extent. They have so many things that they want to build. It's on them. The responsibility is on them to keep them active and at capacity at all times. So I know some people like that. I am definitely not like that. And just as a matter of my personal opinion, maybe the market will prove me wrong. I don't see those people obviously shipping more stuff that matters than people who don't do that. I do wonder if it's a kind of, I don't know if you, if you play video games, but I do wonder

1:01:36if it's a kind of factorial mindset that's being applied to, to the AI production process, because that I can at least relate to that you're playing a game and you want to maximize how much how quickly you're producing the parts that go into your rocket ship or whatever happens at the end of factorio right i get that playing a game and wanting to be continually involved in unblocking bottlenecks that's how i see it that's just my opinion and purely subjectively i don't see the people who have this AI factorial mindset, obviously shipping things that seem any better than what anybody else is doing. Yeah, it just seems I completely agree with you. It seems to me like they have sort of like a ton of projects.

1:02:16And then when the AI agents go, you know, they have a little bit more slack. It's like, OK, well, to keep them working working i'm just going to create another project and then i think on the other hand is somebody like steve yegi you know i was talking with him on the podcast and you know he created gas town and it's like it's a token max here if you will and he was like he has this game that he's been working on for 20 years you know before ai about two years ago he was basically like i had so many feature requests that he was basically like, I had so many feature requests that he was basically triaging them based off of like whether he would get to them before he died or not, right? And so now with AI agents,

1:02:53he has this gigantic backlog of work that's there. And that's why he created Gastown and all these other things. He's like, I can actually see line of sight for getting rid of all of that backlog. And so we can have everything that we ever wanted for this game. And so I think to myself, like, is there a world, and I'd love to get your take on this, where with AI, we can have a concept of a defect-free piece of code, a defect- free system, or even like a system of arbitrary complexity, of pretty large complexity, where there is zero tech debt. It's a beautiful aspiration. I am skeptical. Let me just talk a little bit about the specific example of the Wyvern backlog. And I don't know Yagi personally. I don't know the,

1:03:37certainly the Wyvern code, but I know similar kinds of situations where people tell me, we have a thousand Jira tickets and we just have AI crunching on clearing out all of it. Right. So I think that's a similar kind of situation. So this is, I'm now no longer specifically talking about Yagyut. I'm talking about just the general pattern. My question to those people is, why were they in the backlog? And they were in the backlog, I think, because they didn't matter very much. Right. You always had something more important to do. I think we underestimate, like, especially people who are very ambitious and want to accomplish things. I think we underestimate the informative, the information carried by the fact that you haven't

1:04:13done it yet. If you haven't done it yet, either you didn't think you could do it, or you didn't think that it was worth it. And now it's totally fair to say, hey, now with AI, that changes, right? Now I can do it. Now it is worth it because all it costs me is some cloud subscription money, right? That's totally fair. And it inverts the question. Then the question is, okay, you have these 500 euro tickets that you're clearing out from the backlog. How much does that matter to the business? What's the latent business value in clearing out all the stuff that you for years, you specifically have said doesn't really matter, right? If you've been saying for years, these things just don't matter enough for us to prioritize them. How much business value do you expect to extract by

1:04:55clearing them all out? I'm sure it's some, right? I'm sure it's some. There's like a units problem maybe where, you know, if you're looking at your team and thinking yeah we usually close whatever 10 tickets per sprint right i'm just making up numbers right those tickets are not really in the same units uh so to speak as the tickets in the five-year-old jira backlog they're in units of tickets that mattered enough that they became top of mind right that's the unit of a ticket that you actually work on. And the unit of a ticket that's been in the tree of backlog for five years is it's something that someone thought about doing at some point and was never important enough to be top of mind, right?

1:05:35And that's kind of a different category of thing. And again, I'm sure that there's some value from just clearing all of that out. If nothing else, you get peace of mind, right? And peace of mind is actually worth a substantial amount in engineering management. I do wonder maybe if most of the value is in the peace of mind that you say, hey, look, even all this low priority stuff, even that we've cleared it out. So now we really know that we're working on a clean slate. There are no gremlins hiding deep in the code that we kind of know about, but we only documented three years ago. And the person who documented it is gone. And can we actually solve it, right? So I think the peace of mind is maybe more important than the 500 tickets themselves.

1:06:13Again, just my theory. Amazon, they have an annual planning process. It's called, well, there's two parts to it. It's called OP1 and OP2, but it doesn't matter what it stands for. But basically the idea is we do yearly planning. We propose all of these different projects that we could do for the upcoming year. OP1 for 2027 is coming up. So this is everything that we could do for 2027. We scope out all of those projects and then we say, okay, this is how many software developers that we have. Almost inevitably, the amount of software developers that we have only allows us to do the top 10, say, of those projects when we might have identified 35. And so what we do is we draw the line. So, hey, which projects are above the line? Which projects

1:06:58are below the line? We spend a bunch of time at that boundary and people come in and they, you know, they make passion pleas that we should, we should do this project versus the other project. I think it would be really awesome if we were like, Hey, actually we don't have to choose. We can just do all 35 of these things, you know, hiring a software developer that meets the bar. And you know, it's, that's really expensive and time consuming and we don't have an unlimited budget there. I think maybe the point you're saying is like all 35 of those projects probably shouldn't be done. There's probably a whole bunch of garbage that's there, right? And the things that are below the line, really they belong there.

1:07:40Certainly there are human resource constraints. And so if you said that you pick 10 out of the 35, I'm sure that there's a high chance that if you could bump that to 11, that that 11th thing adds something, right? So I'm not saying that there's no value at all past the line. The thing that's in slot 35, it's in slot 35 for a reason, right? It's not a near competitor to the thing in slot 10. And if you had 500 slots, even worse, right? And if you had 500 slots and even worse, right? And if you had 500 slots and those 500 have accumulated over years, so it's in slot 500 and it's been there for five years, that really is a lot of information that just doesn't matter. I agree with the positive vision of that,

1:08:17which is whenever you do enterprise planning, at some point, as you say, you draw the line, you say, we're doing these 10, nothing past that. Maybe you can do 11 or 12 or 13. I don't know how many you do now. And the other thing I would add is that with AI and really any kind of rapid development tool, I mean, AI is an extreme version of that, but there have been rapid development tools of different kinds before. That kind of shifts what it means to do something because something that used to just be a sketch on a whiteboard and a coffee conversation can now go to like whatever, a 2000 line, single file HTML sketch. And at least you can click around at some stuff on some stuff and show it to each other

1:08:56and say, Hey, if it kind of worked like this, would that be valuable? Or what else could we do? Or how does it connect to something? Right. Something that was, you were already going to build a prototype, right? Let's say that, let's say that you agreed, I don't know how Amazon works, but let's say that you agreed we're committing to the 10 and then we're going to build prototypes of 11, 12, and 13 just to see whether that changes our minds, right? So that thing where you committed to a prototype, that prototype can be richer, right? Because you can say, look, we, we put intention and focus into the core thing, but things like styling or like multi-user interaction or whatever, right? There's all this stuff that's around any serious project. There's like the

1:09:38core thing and then there's lots of stuff that's around it, right? If AI or any rapid prototyping tool lets you turn the prototypes prototypes of 11 12 and 13 into something fuller that's also kind of shipping shifting shifting the balance right so the thing that used to just be a coffee conversation becomes a sketch implementation the thing that used to just be a sketch implementation now becomes a crude prototype the thing that used to just be a crude prototype is like a focus a focused pillar with some AI backfill of the supplemental features. There's definitely value in that. And that's playing at the 10 versus 11 boundary, right? Of kind of pushing out how far into our priority list can we meaningfully capture value given the human resources we have.

1:10:20I think this 10 projects that we're going to do for the year for the organization, that itself was idealized and points to a flaw of it, which is like you scope it at a period of time where there might be high ambiguity. And then what ends up happening is projects one, three, and five blow up in scope. Now we can't get, you know, seven, eight, or nine done at all. So it didn't even matter, like all of that arguing and horse trading that we were going to do to get those things to be above the line. But yeah, I think, you know, I really, the situation that, you know, the takeaway for me is like, if you're doing planning past six months, like, what are you doing anyways? Like we were in a completely different world six months ago and we have no idea where we're going to go now. So it's

1:11:04time to buckle up. I have a question. So what's the biggest trend that you see right now with all of the AI stuff that's going on that you think is just, it's a fool's errand. It's just going to fizzle. You have no idea why people are so excited about it. I do think we're past the point now of people trying to replace entire engineering teams with AI. So that's an easy answer. But the reason I am uncertain is because this changes so much about how we work and how, and it's just moving so fast that anything we say now, who knows what GPT-7 is going to do. Okay. Let's bring it back to the discussion of Gastown and similar things. Six months ago era orchestration frameworks. Okay. So it's not just about Gaston,

1:11:45this is about six months ago orchestration frameworks. And now we have things with orchestration frameworks that are aware of loops and now swarms. And the thing that I think would be a mistake would be to assume that we're going to be doing whatever that is in six months. So I think anything like that, I would not bet my team's future on that, even though I would invest my team's resources in the short term to explore that. So maybe a thing that would be a fool's errand is thinking that we have a stable workflow pattern, that we have finally figured it out, and we know the stable workflow pattern for AI. I think all the evidence points to there will be some other disruption in six months, right?

1:12:28And we don't know what it is. I absolutely agree with you. I think things like computer use, things like OpenClaw, a lot of that stuff just gets completely brought in to the frontier models. It's really, really difficult to say like, hey, this tool chain, this approach that we're using right now, it's going to be around in six months. And so I think this is a big thing for, say, startups. You know, if you're in YC right now, how can you build a business on such shifting sands? I have people in YC asking me such questions. So six months ago, they were like, oh, okay, cool. Well, we're going to make a pivot and everything's going to be an open claw. We're going to set up your open claw for you.

1:13:11And like, that's the basis of your business, right? And that stuff is sort of fizzled out. Yeah. I mean, if you just focus on small startup teams just for a moment, which is only one slice of the world, it's a very difficult question. What can you do that relies on frontier AI that is not just going to be swallowed by frontier AI? I have no good answer for that. And I know many people who are trying to figure it out and they don't have good answers either. Well, I think there's this, they're getting absolutely pinched right now because I don't think you can get into YC or there's not a lot of investor dollars for anything that doesn't have to do with AI right now. But at the same time, if you build anything that relies on the frontier models, they're just going to use you as an idea.

1:13:53If you build something useful, they're just going to pull the next app or they're going to go and be a solopreneur. That doesn't bode well for them. What could you create right now that isn't just going to be obsolete in three to six months? In AI tools, possibly nothing. But the people that I'm talking about are building ways to book exotic vacations that cost $50,000, right? And are extremely difficult to coordinate. This is the kind of thing that something that would be too niche to build an application for in the past, but now is maybe viable. And is given that you have the app, the app is not super challenging, right? And then it's just brand and access and marketing and distribution, right? And so far, so far, the frontier AIs don't have that built in yet.

1:14:51No, absolutely. That's true. I think what was going through my head when you had gone through the different buckets that people could be in is, man, like marketing and go-to-market and sales are such like critical pieces of building your own business. It doesn't really seem like AI is necessarily helping out with that. It's certainly helping with artifacts. I don't know how helpful it is in the actual strategy part. But also I'm not any kind of expert, so I couldn't judge. I do know that it's helping in the artifacts. What type of opportunities are you seeing as you work with the various companies that you work with or just around ambiently? What sort of opportunities do you see people have in front of them

1:15:37that they aren't necessarily leveraging? This is a small thing, but it connects to a larger thing with this. I don't know if you've been following what's happening with Jeff the last couple of days. Oh, absolutely. Yeah. So this is a small thing that points in a Jeff ish direction. Jeff is much more much more ambitious than this. Right. But one of the things I've I found useful in helping really more more conservative businesses, not conservative politically, just conservative, like they're old, they're like mining or, you know, lumber or textiles. So many of my clients are more traditional businesses that I help them adopt some of the more modern stuff from AI in the last 10 to 15 years. One of the things that I have helped that has been very helpful for them.

1:16:21And so if you're in one of those conservative industries, maybe useful to you, is thinking about LLMs as a kind of text calculator. And so instead of going to it like, here's all my data and I want you to do this. Instead say, here's an email. I'm using this as a parallel to the Jeff example, right? The common Jeff example about processing an email. Here's an email. Please extract the name of this person or tell me if it's undetermined because they didn't include it, right? And then give that to me as a JSON and a confidence score. So like AI engineers, I understand this is basic stuff, but I'm frequently presenting things that are basic for us to much more conservative industries. So the text calculator idea is, okay, I have this record.

1:17:10You already have the record, right? You already have the email. So have the LLM just ask it one very narrow question. What's the name? Are they asking for a refund? Yes or no. Do they have evidence that they're within the warranty period? Yes or no. Do they have a callback number? Yes or no. So this is what I mean by text calculator. This has the advantage that you can cross-reference it against like a policy document, for example, right? You can give the AI your policy document on returns. Again, like fellow AI engineers, I understand this is a very basic thing, but 80% of the world doesn't understand this idea yet. That's still diffusing out into the universe. So Jeff kind of takes this idea and does it much more seriously in both mathematically

1:17:52seriously and like in a structured data series kind of way. So I hope that in a year I can just say, we'll use Jeff to do this because it does this. It does what we want and better, right? But right now you can do a basic version of that, chaining together very small, isolated LLM computations and then processing it in the normal way you would process code. And Jeff makes the same argument, right? That you have isolated injection of AI into your pipeline and then deterministic, ordinary code processing, saying the rest of the way so you localize where there might be weirdness and where you have to have quality control and all the rest is just ordinary software engineering which i think is a great idea and they i'm glad that

1:18:34somebody has taken that idea seriously and taken it farther than uh than uh than i certainly have i absolutely love jeff you know um the idea that you could do a lot of these operations in parallel instead of sequentially like an LLM. Sometimes we don't, at least myself, don't appreciate like how much electricity and compute you're burning every time you put a prompt in. Like when it goes to the cloud, like, yeah, it does take a little bit of time for it to stream back, but it's a not insignificant amount of power that you're burning. And if that power is coming from like a dirty source or whatever it is, like it's, why would you do that? Like if you wanted to, if you can ask an LLM to add two plus

1:19:13two and you'll get the right answer, the better way to do it is in terms of, you know, bits and bytes. The terrible way to do it is to load it up into these transformers, right? And have the information come back. So I do love the idea of a classifier that doesn't blow up scope. It's basically like, hey, we've actually predetermined the types of decisions for your query, and then we'll give you a distribution over it. Like, that's amazing. I have another buddy who was like, man, these new models, when you go and ask it to do the equivalent of make you a sandwich, it like blows up scope so much that it starts to remodel your kitchen. You can't have that. I also hate the, you know, you've ever seen these like chatbots where people will be like, oh, there's an AI chatbot for my return or for customer service.

1:20:02And then they get the chatbot to like start writing code for you. They're like, wow, look at all these free tokens. I can just use this customer service agent to go and do all of that stuff. I think with Jev and these types of primitives, like, you know, they're really going to clamp down on that type of stuff, right? It's not going to be in the distribution that the customer wants to implement a red black tree. Yes. Let me edit my previous response of what is a fool's errand. A fool's errand is using like unstructured LLMs to handle unstructured human input in a business chat like customer service. That I think is just begging for trouble, obviously. But it's like so easy to hook it up that way.

1:20:47Yeah. You know, you're a software developer and you're basically like, hey, you know, look at what I've written. You know, it calls to LLM to do this general purpose thing. And it looks like from the outside that it's going to work, but there's so many different guardrails that you're going to have to put in place before it, you know, either blows up on you in the bad way of blowing up or there's, you know, you're just introducing a ton of security vulnerabilities, prompt injection, et cetera. Yeah. And also just bad user experience, right? Absolutely. I'm really bullish on Jeff. I do wonder a little bit, like I'd seen the founder with his launch video and he's basically like, oh, our service is like too cheap to meter.

1:21:26You know, these computations, you know, can come in at very high speed and at very low latency. And I had somebody else had made the joke like, hey, this $10 credit that they were giving out for Jev for the early access, they were going to be able to like have that be part of their inheritance for when they pass away and give it to their kids. And so part of me is like, is it too cheap for that particular company? I think very quickly that the Frontier Labs are either going to acquire Jeff or be able to develop their own. I wonder what sort of moat that they have around this type of work. How do you stay up to date on everything that's going on? I think one of the hallmarks as I listened to your podcast

1:22:11with Casey is that you really like to do deeper research and, you know, go to the sources on top of your extensive experience within the industry. Sometimes I fall into the trap of doing research on X and Twitter, right? And like, that's how I get my news. Like, how do you stay up to date on what's going on and make sure that there's like, like real research and science that's backing it up? Well, so definitely partly that. I think if you want to know, if you want to know what's happening like a month ago, there are many sources. If you want to know what's happening the last two or three days, I think it has to be X and partly Blue Sky. There is some stuff on Blue Sky that does not also appear on X.

1:22:55The other thing is that I don't know how broadly useful this advice is, but for me it's been useful just directly following researchers. I mean, Karpathy, obviously, an underrated person I would recommend following. I don't know him personally, but accessible and informative and successful in the business is Jeremy Howard. And he posts somewhat frequently on X. And if you just start from there and follow the following recommendations, you will find a cluster of people. Really, that's how I mainly do it there is just following a cluster of people and seeing what they're talking about on a day-to-day basis. And then similarly, I mean, you are on that list as well. In my following AI news, you're in there as well.

1:23:45I mean, not as an AI researcher, obviously, but I'm also interested in what's happening in the business, even though business and finance is not my main thing. Well, Dimitri, thanks so much for having this conversation today. I learned a ton. If they wanted more from you, where can my viewers go and get it? You can find me at dimitrispanis.com. Everything is there. Well, thanks so much for the conversation today. My pleasure. Thank you, Steve.