Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub

0:00All told, it was about 136 billion tokens. It came out to approximately $120,000. This is Stephen Taube, distinguished engineer at Microsoft who single-handedly rewrote a 400,000-line codebase in Rust. And I asked him all about what he learned from the process. These projects that were once unaffordable are now quite doable. How did you discover and fix regressions along the way? Me reading every line would not prevent bugs. In fact, agents reading every line is a lot more effective. Something I would do differently if I had to do it again was... Here's the full episode. I wanted to focus this conversation around the port that you did for the GitHub Copilot runtime and basically rewriting it into Rust because I think this is a new type of project that

0:53is now feasible with the recent models. So I imagine a lot of people in industry are going to attempt something similar, maybe with their legacy code bases into some more performance language so i wanted to start off with asking you what was the project and why did you rewrite everything in rust what was the problem that you were solving yeah that's a great question especially because people sometimes i think assume that rewriting things into rust is kind of all the rage these days and so people assume uh i've had questions that assume like oh you're doing this just because or like you can and so you will and uh that is not at all what this was there were you know there are certain shapes the current the project could

1:39have existed in where rewriting it wouldn't have been a conversation at all. But the impetus is largely that AI has been this incredibly fast-moving target, as we all know, and over the last few years, the capabilities that it has enabled has skyrocketed. And so across GitHub, across Microsoft, there are all of these projects that have kind of blossomed from the ground up to incorporate AI capabilities into them. And all of them basically started from, oh, we can make these LLM calls. Oh, we have to wrap a little bit of goo around it. We have to add more around it and more around it and building up effectively what is now called agentic harnesses or agentic loops or agentic runtimes, whatever you want to

2:31call them. And we ended up with a lot of them because everyone kind of had their own homegrown implementation. And that's not a good place to be, especially when you're trying to have all of them kind of be best of breed and be on the forefront. And in this world of AI, like there are new models coming out weekly, if not daily. Like, you know, a couple of weeks ago, there were all these releases from both OpenAI and Anthropic Cup, you know, one upping each other. And there are new techniques that a month ago were research and today every harness has to have or they're immediately out of date. And you don't want this myriad of different implementations all needing to stay up to date. And so you really want a small number, ideally one, but a small number of extensible harnesses or runtimes that everyone can use and customize and extend to their needs.

3:27But the Copilot runtime that was being used for the Copilot CLI and the GitHub app and Copilot Cloud Agent, like when you're on GitHub and you ask Copilot to do stuff or code review like um that was written in typescript um using node using v8 uh and that has implications for where you can use that as an embeddable library so as a like one of the main blockers for that runtime being able to be used everywhere i slash we wanted to be usable was fundamentally what it was implemented in the frameworks it was using the runtimes it was using and so on and that bet basically not moving to rust but rather moving away from typescript and node to something that was more embeddable, had better efficiency, lower working set, faster throughput, better able to be consumed into

4:34any myriad of environments, regardless of what language they're implemented in, regardless of what frameworks they're using. That was basically the catalyst for this. And to your point earlier, there are any number of reasons folks might be looking to do these kinds of ports or migrations. They are incredibly common across the industry, whether moving from one deprecated framework to a newer version or from one language to another or from a legacy code base that has become an unwieldy you know mess of spaghetti to something cleaner and modern and fresh these migrations and ports are everywhere and a lot of times people just don't take them on because they are too expensive they They take too long. And we would have been in that

5:25situation a year or two ago as well. But with modern models and modern harnesses, these projects that were once unaffordable are now quite doable in a very small period of time. In the post that you wrote, I saw that there was a line that said, I didn't set out to move to Rust. I set out to move away from Node.js and V8. And that makes me wonder, if you didn't need to go to Rust, did you consider other alternatives and why did you end up going to Rust specifically? So the very first thing I did was basically create a prototype, agentically, because you can now create prototypes super fast in um still with typescript but with different runtimes so you know node versus bond versus whatever uh as well

6:13as i think i did python go c-sharp java and rust um and created a very small end-to-end sort of steel thread through one of these systems kind of backed by a local chat completion, you know, dummy server. Probably five to ten thousand lines for each of them. And looked at them in terms of end-to-end performance, in terms of working set, throughput, what kind of scale that we could get if we threw a lot of load at it, and then also what kind of, whether it could actually meet requirements and what it would take for someone, for us, we currently have six language SDKs that are thin bindings on top of the Copilot runtime. What it would look like for this to be usable from all of those. And in particular right now, because we don't want to force, or previously because we didn't want to force Node into the consuming applications process,

7:22the SDK basically spawned another process to host the actual execution of the sessions. But we'd really like as an option for it to be in proc as well, like pretty much every other library in the world. And so basically evaluating all of these languages and runtimes against these performance goals and constraints and just looking at the data to see what worked and then how well it worked. And for the most part, TypeScript and Python were very quickly out from kind of all of the performance perspective and the embeddability goals. Go had pretty good perf and, you know, had a pretty clean code base. But it's not really designed for embeddability. It's not real, like interop is not its raison d'etre, right? It's, you can do it,

8:26but you're kind of cutting against the grain. And then, and Java is also not great for that. Java's perf was okay. The two main contenders ended up being C Sharp, in particular C Sharp with native AOT, so that it could be compiled ahead of time down to assembly and, uh, and rust. And from a perf perspective, uh, they were both very much on par with each other, uh, across pretty much all the metrics in some places, rust was a little better in some places, C sharp was a little better. Um, and so to your, your, you know, question earlier, if the project was already in C Sharp, we wouldn't be having this conversation. We would have stayed with it. There would have been no reason to rewrite it. But it wasn't, and so then it came down to these criteria.

9:18So from a perf perspective, they both look good. From an embeddability perspective, they both look good. C Sharp has very strong interrupt characteristics as well. And then it came down to primarily softer factors. So there are other projects around GitHub in particular that are built in Rust and very few that are implemented in C Sharp. There are more people around GitHub that know Rust and fewer that know C Sharp. That's not the case within Microsoft, obviously, but within GitHub. And so factoring in those kinds of criteria, we ended up going with Rust. I know Anders recently did, I think the project started a bit before your project started, but he did a port with the team of the TypeScript compiler from being written in TypeScript to

10:08being written in Go. I was curious if you compared notes with him. Definitely. Yeah. And this is a great example where the different projects can reach different conclusions and be right. There's no requirement to embed the typescript compiler into other applications um and conversely the shape of the um the shape of the work that is compiling typescript looks very different from the shape of the work that is um sending requests to an llm and manipulating prompts and dealing with sandboxes and all that kind of stuff um the nature of the projects was fundamentally different although interestingly in the last few weeks there have been several uh agentic ports of uh which you can you know if you search X or Twitter or whatever it's called now,

11:08you know, you can find people that are porting the TypeScript Go compiler into Rust. And doing so very fast with modern models, given the size of the code base and the cleanliness of the code base. And so it's interesting to see. So once you decided that there was motivation for the project and it was a clear problem to solve, I wanted to ask you a little bit about the migration strategy because I've seen different strategies in different projects. So for instance, you mentioned Bun earlier. Bun, it looked like they kind of built up a giant, basically a new branch that's just completely entirely in Rust. And then they shipped all in all at once, stable Rust release, and they did it all in one one go. Whereas when I was reading what you had done, it looked

12:05like a lot more incremental and little by little. How did you think through the different strategies and when might you select one versus the other? The interesting thing about Bunn, first, I loved Jared who wrote the Bunn post. I loved the post that he wrote. It was really exciting because that post, I wasn't aware that they were doing that port. And then that post came out about halfway through our migration. And it was really cool to see that this other project of approximately, you know, order of magnitude, same amount of code in it was in different language. They were in Zig rather than TypeScript, but that they had, you know, thought that this was a sensible thing to do and had landed it.

12:47And in the post, they talk about it taking, I think, 11 days for them to get to like 98% of their tests passing or something like that, which is really cool. And to your point, like they did it sort of in a, an en masse, like rewrite the world and go. It is interesting. I am not involved in the project, so I don't know all the details what's going on behind the scenes but it is interesting to me that um if you look at their npm page they were shipping near daily builds uh i think you know a lot of canary builds basically through mid-april like for a long time daily builds and then mid-april it stops uh and then they talk about doing the you know the 11 days of porting which is an impressive feat

13:32to get from nothing to this you know a 500 000 line code base ported to rust in 11 days and have a passing 98 or 99 of the tests but then they didn't have another npm release of any kind at least as far as i could see for another two and a half months um so there was this window where yes the the core port took a small period of time but it seems then like there was some aspect of stabilization and things like that happening such that there was this three month gap between releases um which interestingly is also approximately the same amount of time that it took copilot the other fortune and a half weeks that I spent porting copilot. So, you know, there's lots of reasons why they differ, but fundamentally it seemed to be about the same amount of time.

14:22And we just chose sort of different ways of going about it. In the post, Jared makes a point about not wanting to do the incremental approach that Copilot did because it would incur lots of temporary interrupt junk. And he's right. It did. We just chose that approach, knowing full well what we were getting ourselves into because of the sort of the constraints that we were under so uh you know for a javascript runtime um obviously fixing bugs and new features like that's all valuable um but you can pause for three months and the world isn't going to fall over. JavaScript three months ago isn't all that different from JavaScript today. And you can do sort of a rewrite the world approach, pause everything, and then continue

15:20shipping when you're done. Agenda Karnas is in the summer of 2026. If you pause for three months, the world you ship into will be very different from the worlds that you started in. And so trying to pause for a long period, for a period of time, even if it's a few days, was not something that we were willing to entertain um and you know even if we've been able to do it in two weeks that would have been two weeks worth of either blocking the whole team or uh dealing with a constant influx of of changes across a very large code base and having to rationalize and you know all of those changes in and merge all those changes in as ports instead we opted for the approach which i think made sense for copilot and made sense for bun where we just incrementally moved the code base

16:20more and more from typescript to rust in pieces in main in the shipping code base shipping releases daily and it's just that at the beginning it was 100% TypeScript and 0% Rust and at the end it was 100% Rust and 0% TypeScript and it was just you know more and more over time and for the most part it worked out quite well it did mean that there was to the point of the post of the the bun post there was a lot of temporary code that came in and left right i think at we um we were using uh nappy nappy rs for interop between typescript and rust and at the peak point we had several thousand interop signatures for, I think, two or three thousand for calling between them. And then by the end, it had dropped

17:12to zero because there was no more interop required. So for this project, it made a lot of sense. There are other approaches. You could do a rewrite the world. You could have everyone stop what they're doing and just rewrite in main you could have a feature branch and allow main to keep going and then constantly be taking ports in and for the incremental approach you can do what we did but rather than do the approach of what i chose which was sort of atomically swap uh typescript with rust you could have sort of an ab thing going thing going where you have both the TypeScript and the Rust component and you ship with both and you have a feature flag that allows you to ship between them and do sort of flights and experiments of those components side by side.

17:57There are various reasons you might choose which, and I think what we chose ended up working out well for us, but all the other options would be valid or probably better for other projects. So it sounds like that A-B approach, you would also still have some temporary testing harness that would be comparing the two legs anyways. Well, in the case where you have a separate branch, and then new commits that come in on the old project, or I guess main, need to be continuously ported. I imagine that would be relatively doable with modern models as well, because each of those single commit ports is just probably trivial for a model. Did you consider doing that one or how did the math compare between doing that

18:52versus incurring the interop costs? So there's a couple of questions there. It's not just about having a separate feature branch and bringing everything over. If we had a separate feature branch and bringing everything over if we had a separate feature branch that we were maintaining and at the end we did this swap where we deploy like the whole new stack we would only be testing in the wild everything all at once and that's a pretty pretty risky gambit um whereas if you uh one of the big benefits of the approach that we chose was we were effectively able to use production for the last mile of testing and have each deployment be a small diff on top of what the previous one was so So over the course of the port, I think we shipped about 130

19:45releases, something like that, each one of which contains just incrementally more. And so if there were issues, it was very pretty straightforward and fast to root cause those issues, even without using computer at all. Like someone reports an issue like, oh yeah, I know exactly what that is, right? And then you can very quickly narrow it down whereas the the whole rewrite the world it's harder to do that kind of validation um and there were absolutely regressions that came about from this a good chunk of those were because of issues with merging in changes from me because you know i still had feature branches they were just much more time boxed instead of being weeks they were a day for you

20:35know porting a component but within that day there might still be 10 or 20 prs that came in and touched that component and a non-trivial percentage of the regressions were places where the the agents just missed something as part of doing the the merge um and and the port and the merge at the same time um one of the the nice things about sort of the atomic uh replacing from typescript to rust versus having an ab thing going is the agents were instructed to delete the TypeScript that had been ported, it instantly created a conflict, which sort of raised red flags and pointed the agent's attention at that code. Whereas if we weren't deleting it, it would have been very easy for more changes in main to that

21:38TypeScript to find their way into the existing untouched TypeScript in the branch. And then the agent would have to reason about all of the code that it had ported, looking for changes to that to bring that in, versus it was sort of instantly pointed to those places because we were doing this swap at the time. Yeah, I can imagine a lot of people, a lot of teams or engineers that want to rewrite their entire code base, probably the biggest concern is the correctness concerns, which is there's so much code that you don't have the luxury of a human even reading the code to confirm that it's correct. So how did you discover and fix regressions along the way so the the most important thing was our

22:27end-to-end tests our integration tests or whatever you want to call them the we basically had two flavors of tests for copilot we had tests, which were largely sort of, you know, they're testing micro functionality throughout the code base, and a lot of them have internals access. So they're able to test this little function here that's not exposed or that piece of code there that's not exposed. And then there are the end-to-end tests, either at the runtime level, the SDK level, or the CLI is in the same code base. And so its end-to-end tests also effectively exercise the runtime. And this isn't 100% true, but the CLI can largely be thought of as sort of just exposing the runtime's capabilities and sort of the lowest layer possible in a human interface.

23:31And so we had all of those end-to-end tests to validate behaviors. And by and large, if not 100%, but the vast majority of regressions that we had were because we didn't have end-to-end tests covering that functionality. We generally had unit tests covering it, but the problem with unit tests in this situation is they were also implemented in TypeScript, testing TypeScript with internals access. And so those, in many situations, those unit tests had to be rewritten at the same time that the code was being poured. Now, because of how we did it we got the benefit of um if i replace this little piece down here and then the typescript so this is now rust and the typescript sitting above it was interacting with this through interop this guy had

24:17its own unit tests and so indirectly those unit tests are still validating this ported rust component so we got benefits from that but But the safeguard was the integration test or the end-to-end test across everything. And I had written a document for the team sort of motivating this port and laying out exactly how we would want to go through it and approximate timeframes and approach and strategies and technology we'd use and so on. One of the things that called out was we needed to beef up our end-to-end tests as the very first thing we did. And I did. I think one of my failings and something I would do differently if I had to do it again was I would have done it more. Because, again, the vast majority, it kind of goes without saying,

25:02but if you have tests for something and those tests don't regress, then like the behavior is good as long as you have tests covering all the behaviors you care about. And so the fact that we had any regressions sort of highlights that there were gaps. Now, that's not 100% true. The funniest or saddest cases were where we neglected to prevent the agents from rewriting the end-to-end tests at the same time that they were changing the behavior that was being tested. You know, the agents want to get to green. Like, that's their goal. And so if you let them, they will change the oracle. Like, they will change the tests at the same time to make it pass. Or we had one case where this one didn't slip through I noticed this one but like the the agent was

25:52making changes and it ended up changing some public APIs and uh CI caught this and said no no no you can't do that but we have an escape hatch in ci where if someone really knows what they're doing they can put a label on the the pr to basically say no this particular break is okay and the agent found that and applied the label to say like oh this this break is okay even though it you know it wasn't um and then there are other things that are just really hard for you you could have end to end tests but they're sort of in the weeds. Things like, you know, you're you're sending data out on the wire and making sure that every byte that you send in all possible situations is what's expected at the end of the end of the wire

26:38that requires an extensive suite. And if you have any gaps gaps there's the potential for things to slip through and then there's also you know there's a typescript and rust have very different threading models um concurrency models and uh as much as people like to say that you know if your rust compiles that's correct and it prevents all these kinds of bugs that's true but it doesn't prevent like uh you know all kinds of bugs and there are plenty of concurrency bugs that slip through um and so a bunch of the issues were lifetime issues concurrency issues things that the rust compiler does not does not catch um and when you're coming from a largely serialized, single-threaded execution model and going to one where on purpose we wanted much better scalability,

27:34we wanted things to run concurrently, there are plenty of issues that creep in there. So anyhow, to your question, the number one thing is to make sure that um all of the behaviors that you care about are encoded in tests um and then let the agents port the code while maintaining you know keeping everything passing um and making sure that there are guardrails in place to make sure that they don't change the thing that you're using for the validation at the same time that you need that validation to be correct. It still feels like a regression is inevitable because the test coverage is it's like it's opt-in kind of where it relies on a human saying these are the things that are important but you could always forget

28:26something um so i i kind of wonder if this type of project it's just inevitable that there will be some breakage i think so i mean it's it's hard to imagine i mean who knows what things will look like in a year things Things are moving really fast. But right now, yes, 100%, there will be regressions. If there aren't, I would be astounded. Unless we're talking about like, you know, a tiny, tiny little code base. But you're talking about rewriting hundreds of thousands or millions of lines of code. It's just law of numbers. If the agents get it right, 99.99% of the time, which is probably a nine or two higher than most humans, that still leaves 0.01% of the time where they're going to get it wrong in some way, shape, or form.

29:32And if you're dealing with million lines, that 0.1% is a very real, small, but real number. You can throw more agents at it. And we did. We just, before Merge, doing lots of agentic reviews, and then after Merge, running agents to continually continually look for bugs look for concurrency issues look for whatever and they find things but they never find it's never at least in my experience it's not a hundred percent perfect for such a huge volume of of code i imagine most of it wasn't human reviewed how did you handle handle code review at scale? Or did you do any at all? Yeah, so definitely did some. But I did not read every line. That would be impossible. And if I did, it would also be a waste. There's no like me reading every line would not

30:21prevent bugs. In fact, agents reading every line is a lot more effective not just faster but more effective in quality than me doing so um i i absolutely looked at the code for every pr that i put up but i was focusing more on big picture stuff, right? Is the direction of this correct? Is the architecture of this correct? Or is the componentization of this correct? Is the approach or the algorithms that are being used appropriate? Is there anything fishy about like, you know, if I, a lot of it is like squinting at a 30,000 foot level and looking at the two code bases and does anything look off, right? And then relying on the agents to focus on the line by line details. And in many situations, I would sort of

31:20do a quick scroll through all the changes to just see if anything popped out. And I might notice a few things here and there, but for the most part, a really good agentic reviewer is going to be way better at sort of the line level details than certainly me. And I would argue most humans. Um, but at the macro level, there are lots of times when I basically stopped the agents and said, Whoa, they're like, this is, this is not the direction to be going. Um, or the inverse where the agents would stop and be like, we're done. And I'd be like, no, you're not, uh, the, this, this is, this is not what we're going for. Um, and so it was a bigger picture than sort of the micro level. I saw in the blog post that you have this cool bar chart where you basically categorized

32:16every single message from you to the agents and what your intent was looking over that and reflecting on how you guided the agents, where was most of your time spent and what were the most impactful times where you would step in and change something? There were two or three even higher level categories of things that I was doing. There was one or two categories where I think this just speaks to the state of sort of long horizon tasks with models and harnesses today, where it was me being frustrated with their progress in one way, shape or form, and either telling them to keep going in some fashion when they had stopped or telling them to stop when they were going off down, you know, bad paths. But the majority were more, were in a category that was more of,

33:13there were trade-offs to be made. And I wanted to make sure, and reasonable people or reasonable agents could choose multiple paths. And I wanted to make sure that we were going the decisions that needed to be made, and look and kind of talking through rubber ducking through sort of the pros and cons of those choices, given the current situation we were in. And some of that to the to your earlier questions about sort of the, the incremental versus sort of pause the world or big bang rewrite approaches. Some of those were less about the end to end and more about where we like what I wanted to do for this one slice that we were putting up. So, you know, the agents would pause at a particular place saying, you know, I'm done

34:23with this component and i'd look at it and say this is not a good place to leave things in terms of the the the tendrils that this has into other systems you know temporarily or either maybe we need to do just a little bit more to kind of get beyond this architectural boundary and then that'll clean up a bunch of stuff so that when we merge this the code base is in a reasonable place a lot of times it would sort of declare success for the slice and if you squint yeah okay but if it were do just a little bit more or in some case revert and do a little bit less it would have left the code base in a better place or situations where it actually tried to do more than i wanted it to do like one of the one of the rules that i

35:15put in place with instructions that wasn't always abided by was i wanted these ports to be as strict ports as possible not removing functionality not adding functionality not making any huge changes and algorithmic approaches not trying to fix arbitrary bugs that it comes across along the way i would have it note those for me so i could revisit those separately but i wanted these to be very strictly as one-to-one functionality ports as possible. And there were times when I would see it violating those rules. And I would basically have it back off from that and go back to what it previously had, or at least talk through what it did to make sure that I was comfortable with that particular change being included.

36:03A non-trivial percentage of the regressions that we have had were from places where it violated those rules and i didn't notice and it included fixes uh for things that um shouldn't have actually been done in fact there was just one yeah i for the most part we're not seeing any more uh regressions being reported from the porting phase we see regressions all the time that's the nature of merging hundreds of prs a day into a repo but um you know it's just to be expected um and we fix them generally before they ship. And it's just where we see them in Maine. But we had one yesterday or the day before where there was this really corner case thing. And I looked at it. It's like, that's really weird.

37:03Like, I don't it's saying that I added this and I don't remember adding this and looked back what it was. So I had created the pr as one of the porting efforts and it wasn't in the initial pr uh and then some feedback from a peer or a peer's agent came in recommending doing something and the uh agentic response to that was to basically address this security feedback by locking something down. But it locked it down a little bit tighter than it needed to be locked down. And that resulted in some things that had been working before the port, you know, no longer working after the board. And those are really interesting things to go and figure out how they happened. That's interesting. So your agent will auto-apply and auto-listen on the PR

37:59and if it seems reasonable, may take action without your input. Is that right? So for the most part, I use the GitHub Copilot app. And it's got this agent merge functionality where you can basically have it automatically do various things watching a PR. It can automatically rebase if there are conflicts or merge if there are conflicts. It can automatically drive CI to green. So if tests are failing, fix the tests. It can automatically address review feedback that comes in. And it can automatically merge once everything is green. And you get for each of those four things, you can choose which ones you want. For the most part, I leave all three of those checked except for the last one so that I have a time where after everything is green and before I click the

38:51merge button, I go back and I review to see everything that it did to drive this PR to green. What tests did it fix? What feedback came in and how did it address it? To make sure that I know, again, at a high level, I'm not reviewing every single line, what it did. And this particular one, I remember actually reading, but I didn't fully comprehend how much it was locking things down and how much it was blocking. Because if I had, I would have said, actually, those cases are okay. Let those through. And had it back off from that. If you had all of those checked off, I mean, no one would do this because they're on your team. But imagine I was on your team. I could probably craft a PR comment that is malicious, I guess.

39:43You know, it's like, hey, it sounds reasonable, but I did request you to delete like everything or. Yeah, I did ask you to format your C drive. Did you not format your C drive? Yeah, exactly. Just take everything down. Yeah, and you know, certainly different projects, I would have different boxes of those checked. Like if this was an open source project, the only one I would have different boxes of those checked. Like if this was an open source project, the only one I would consider leaving checked would be the CI one. You know, driving, there's no way, and maybe rebasing, but there's no way that I would allow it to automatically address arbitrary review feedback coming in on a PR and certainly not automatically merge.

40:25On a close, on a private closed source thing where I trust, you know, the people that I know to be working on the project, it's less risky. But even then, I still, you know, that less checkbox, you have to be very, very trusting. Yeah. One of the things I like about the app as well is it does, there are built-in prompts as part of agent merge for what those different things are, but the app does permit me to add my own prompting for those different capabilities. And so I have additional constraints and guidance for the agent as part of doing each of those things about how it should do it. Like, you know, always rebase instead of merge for one of them or for review feedback. Here are the kinds of things I want

41:16you to think critically about before you just blatantly accept it. And these are the ones where it's okay. Just go ahead and address it. You know, those sorts of things. OpenAI, Anthropic, and Cursor all use this product to make their lives easier. WorkOS is an API platform that adds enterprise features like SSO, SCIM, and RBAC to your app. These are things that you need to sell to other companies, but aren't the main focus of what you're building. And now they've added more support for AI agents. For instance, Auth for Agents gives each agent its own identity with short-lived scoped tokens instead of a long-lived API key. So if something goes wrong, you can revoke one session and everything under it dies. Another feature they have is Airlock, which is an

41:55authorization layer. Every action an agent takes is checked against rules that you can write in plain English. So it's not rubber stamping everything and it's not running unsupervised either. And lastly, they have Pipes, which connects your agents to over 300 third-party tools without you ever needing to manage a token. So if you're building SaaS or AI agents and selling to enterprises, WorkOS fills all these gaps in a few lines of code. Check them out at workos.com to learn more and get started. And I appreciate them for sponsoring my show. We talked a bit about regressions. And you mentioned that a lot of them were caught before they went out to prod, it sounds like. And I think a lot of people might hear regressions

42:36and think, oh, you're you're taking down the app or, you know, things are exploding all the time or the incidents. Could you explain maybe the further protections after end-to-end testing that made it so these aren't bad regressions? I would say probably two-thirds of these things that I'm calling regressions were caught very soon after merging to main and before we shipped any pre-release. Even though we're shipping, at this point, we're shipping, I think, three pre-releases Even though we're shipping, at this point we're shipping, I think three pre-releases a day. So they were caught like very quickly. Then there was, there were some that made it into a pre-release and our form of ring deployments with the CLI

43:18is basically pre-release and stable where pre-release on any given day is probably hitting five to 10% of the, um, the audience, um, of the customer base. And then, you know, 90 or 95% is only unstable. And so a reasonable number of these issues, uh, I don't know exactly what percentage 10, 20%, um, found their way into a pre-release. And then pretty quickly we'd get an issue opened or someone saying, Hey, like I'm seeing this difference in behavior was this by design or not and so a lot of these were things that were different not necessarily bad but different enough that someone noticed and then asked us like was this on purpose um and some of those cases were ones where I'd failed to prevent the agent from fixing bugs as part of the port.

44:08And so some of them are actually desirable changes, but we hadn't intended to, you know, they weren't purposeful in that regard. And then there were a number that made it into stable releases and the ones that made it into stable releases because we hadn't noticed them in either main or pre-release they often ended up staying in and being unnoticed in stable releases for quite a while um and this is mostly the trickle that we have been seeing over the last you know well you don't really see much anymore but a month ago would have been the trickle that we were seeing, you know, over a few weeks where some super corner case thing just behaved slightly differently. And then we get an issue open like, hey, this this isn't working the way I expected.

44:55You know, is this on purpose or not? And oftentimes they weren't. And we'd fix them as soon as we heard about them. There were also ones that we would see from various telemetry systems, you know, monitoring for exceptions, crashes, things like that. And obviously those are unintentional when we fix those. In the post, you shared this graphic of which models you used throughout the course of the project. So you have this nice, I guess, bar chart that's showing what portion of each week's, I guess, tokens are consumed by either Claude or GPT, and in rare cases, Gemini. At the beginning of the project, it was almost entirely Claude. And then by the end of the project, it was almost entirely GPT.

45:43So what made you shift your model of choice in the middle of the project to GPT? For the most part, I was constantly trying different ones. And because, you know, over that period, like both labs were releasing new models at a very fast pace. And so I was experimenting. And when, when I would move to one and I found it to work well, I would stick with it until, uh, something bothered me and then I would try a different one. And then, you know, so, um, it doesn't in any way reflect sort of a long-term preference. It was more of at that moment, what was working best for me then. Um, and so at the beginning of it, I think I was it 5.5 5.6 one of those and that started working even better for the these particular workloads so then i jumped to that and then opus 5 came out

46:55and it was all this is cool and jump to that one right and then um recently i've been having a lot of luck with uh 6.1 sol um and i'll use uh i'll use opus 5.5 for code review uh for the work that i've done with 6.1 sol um and so i'll like i'll sort of split the balance there um sometimes if i get really frustrated with something like both over the course of the port both models both the gpt and claude would exhibit some very frustrating traits at certain points like um a particular point or if i was working on it for some period of time one of the models might get stuck and kind of like no matter what i did it kept kind of refocusing on a particular thing and not stepping out and looking at the forest for the trees. And there are various things I would do in those situations. Sometimes I

47:52would start up like keep the code changes, but start over with a clear session and try and use that to, you know, clear context window and try and use that to get it to reset itself. But often it would find itself back into a rut. And so in those situations, I would switch models, um, and then keep going with the new model until it found itself exhibiting some frustrating behavior. And then I would, you know, switch to something else. Um, I did experiment using Gemini and Grok as my main drivers. Um, but I would often leave those to be as part of like a multi-model code review. So a lot of the reviewing that I had that I was doing, I would be using either GPT or Claude as for the main work and then use three or four different models from different families to do a line-by-line

48:46review of the porting that was done. And I would do that very frequently, at least as frequently as every time that I was rebasing to bring in additional functionality from changes that had been coming into main, but often even more frequently than that. Yeah, I did see that you had a prompt that was, I guess, auto applied on rebase or something like that, where you requested it to be reviewed, I guess, adversarially, or, you know, all, all of them at the same time. What made you come to that conclusion rather than, I guess, code review with the same model, but I don't know, different focuses for each thread? Yeah, it's basically to get effectively different perspectives. Every little change that you make

49:40to a non-deterministic system ends up sending it down, you know, different paths and looking at different things. And one of the biggest changes you can make is completely changing the model. And so the best way to get different opinions and focusing on different things that I found is by having different models, especially than the one that I used initially. So after you finished the rewrite, what was the concrete impact and the cost? So if you look at it from a business perspective, what was the benefit and the cost of this project? So the benefits are stark. I mean, the benefits are the ability for all these products and services and teams across GitHub and Microsoft to be able to rely on a shared harness that is constantly sort of,

50:38you know, best of breed and getting better and better every day and not have to spend their own resources trying to do the same thing they can focus on their business value and and reuse this component it's the same as with any library that you want to rely on and this library is a particularly large library right you want to be able to focus on your business value and reuse the constantly evolving work for someone that is spending all their time focusing on on that thing and so from a business value perspective it's it's like incalculable across github and microsoft being able to leverage this shared harness um whereas before the, a lot of that would have been a harder sell and harder to justify.

51:27And part of that is because of very real costs, like performance costs. So you know, something that was taking 10 times as much working set, waving my hands means 10 times less density, which means 10 times as many servers you know those sorts of things um and uh if you can you know make the um if you can scale the number of users that are able to be on the same machine running concurrently, that has impacts on your bottom line. If you can scale the throughput and reduce the latency by five or six X or whatever it is, that has real benefits. One of the biggest impacts from a Perf perspective, a lot of these systems, especially services, will do, you know, a request will come in and the agent

52:29loop needs to be started to make one more turn, which could be a very short thing, and then the request is done. And so for these things, the startup cost ends up being one of the dominant factors and the startup of a node application versus the startup of a a native rust or c application is there's a big difference there we're talking hundreds and hundreds of milliseconds uh difference or in some cases seconds um on a per request basis that latency can be extremely measurable and extremely impactful for bottom line if you're even able to adopt it because you might not be able to and then you're back in the previous world that i was talking about so from a business perspective it's worth a lot to do this port. In terms of what it actually cost us,

53:27it was about, so I stored all of my session logs or 99% of my session logs from this effort. And from those session logs, I was then able to sort of deduce the token counts for fresh tokens, cash writes writes cash reads uh output tokens and all told it was about 136 billion tokens broken up in such a way that if you were to use the public pricing of those models at those times it came out to approximately $120,000. This is not factoring any discounts or things that companies might have for what those tokens cost. And this is just sort of what's on the website of, you know, OpenAI and Anthropic, plus my time, which, you know, it took, it was about 14, 15 weeks. But I was doing other things at the same time.

54:31You know, a lot of these agentic projects, if I look at my co-pilot app that I currently have on my screen, I've got about 35 sessions right now for things that I'm working on. Some of them in the same area, some of them are across different projects. And so I'm not, developers are no longer working on one thing at a time. So if you look at the percentage, just as a rough estimate for my time, if you look at the percentage of PRs that I was submitting over that same period, about a fifth of them, give or take, was to do with this project and about 80% was for other things. And so if you figure, you know, about 20% of about 15 weeks, you end up with about three weeks of my time spread out over three months plus the the token costs um and it's only getting cheaper the the the models are becoming

55:33like at cheaper at an astronomical rate i think like when open ai released i'm not gonna remember the exact things but i think like six one sol is half the price of six sol and six was half the price of five six salt like the it's just getting cheaper and cheaper and cheaper um and the models are also getting better and better and better which means that the amount of time that i need to put in to steer the models to do the right thing is also decreasing um and so the costs of these ports, the cost of these migrations is only getting cheaper over time. Wow. So three weeks of, I guess, of developer's time plus $120,000, you're able to rewrite the entire code base. If you were to scope this out pre-AI, let's say you wanted to undertake this, what would that look like?

56:29First, it wouldn't have happened. And I'm actually serious. Like for this particular thing, it wouldn't have happened because by the time we were done, we would have ported the wrong thing. Like this space is moving so fast that we would have basically never finished. I think if you can kind of see that in the charts, in the blog post, in that for a good portion of the porting effort, the amount of rust is increasing and the amount of TypeScript is largely remaining the same, but increasing a little bit. And that's because as the port was happening, there was just a huge amount of new functionality coming into the repo. And at the beginning of the porting effort, because the majority of the code base was TypeScript, the majority of the new incoming code was also TypeScript.

57:29By the end of the project where the majority of the code base was Rust, the majority of the new incoming code was Rust. But if you look at like the first half of the graph where we're generally 50% or less Rust, the amount of TypeScript was doing this, even as the amount of Rust was, you know, doing that. You know, even if that wasn't the case, this we're talking it was about 450 000 lines of typescript just production source never mind tests that were ported i think there was a few hundred thousand lines of unit tests there in typescript as well that also needed to be ported to rust um just think about the productivity of an individual developer and how much they can reasonably do in a given day,

58:06if they're top notch. And we're talking a whole team of developers for a year or two to do something like this, right? It is, it is a, if you're able to do it at all, it is, it would have been a monumental investment. And I struggle to think about even writing that proposal because I know that it would be sort of, I would look bad. Like I would be dismissed immediately as of course we're not doing that. Whereas this was a proposal like, I'm gonna take 20% of my time over three months and by the end, this whole thing is going to be moved over and for the most part no one else on the team is going to be impacted um and you know that's not 100 true but like waving my hands and we'll have all these business benefits in a few months um and it you know this will be the bill that's a much easier proposal to write

59:08and even just the even if you were paying that actual you know uh non-discounted public facing uh token rates of 120 000 you compare that to the cost of a senior developer you know for a year uh with all their benefits and their bonus like even that is a good deal it sounds like at least for this type of work where it's super what we want is very obvious and encoded somewhere in the existing functionality. So like this type of work is at least one order of magnitude cheaper, maybe even more. I mean, if you had a team of engineers and they're, you know, engineers, maybe two to 400k or something like that per, this is a drop in the bucket, the 120k. When you were proposing the project, did you have a sense of how much this would cost in tokens before you went and did it?

1:00:13Yes. And the bill ended up being larger than I expected. Mostly because the line count ended up being larger than I expected. I hadn't fully factored in the growth rate of how much incoming code there would be. So this was, you know, there were a few things. I think I did a lot of things right. I did a few things wrong. And, you know, 20-20 hindsight and all that. One of them I already mentioned, I didn't spend enough time really fleshing out the end-to-end tests to make sure that they were as comprehensive as they could have been. I could have done that better. The other thing I could have done better was really factor in the growth rate of code that would need to be ported. So when I looked at the code base

1:01:03as part of writing this proposal, it was somewhere on the order of 150,000 lines of production typescript 130 something like that I don't remember the exact number um and based on that and some back of the napkin calculations and looking at how long it took to create the prototypes that I'd alluded to earlier for, you know, in the different languages to sort of measure perf and things like that. I think I'd estimated about six to eight weeks to get it done. But when you look at what we actually needed to port, I didn't factor in a few things one that there were you know tens of uh developers using agents uh very quickly augmenting the code base with lots of a different additional functionality um

1:01:55and uh you know this code base is one that's it's not public open source but it's effectively internal open source so there's a bunch of folks that come in and add the things that they need. And then the core team is helping to review all that. And so there's this very significant influx of functionality that I hadn't accounted for. I also hadn't accounted for the fact that because of how the code base came about, the CLI, the Copilot CLI and the runtime were kind of very intertwined, more so than they ideally should have been. And so in order to port just the runtime, and the CLI can be ported separately if we wanted to, but to actually tease them apart to the point where you don't need the CLI,

1:02:41you don't need to deploy the CLI as part of your application that's using the runtime. There was this effort of teasing them apart, but then there's also an effort of moving functionality that really doesn't need to be in the TUI and that could be shared in the runtime down. And I failed to account for how much of the code that was currently marked as CLI would actually be beneficial to move down. And so over this effort, there was also code that was coming down. So by the end, we ported somewhere between four and I think 450,000 lines of TypeScript. So about three times what I'd estimated, which impacted both the end-to-end time that it took to do it and then also sort of token counts. So after having gone through this

1:03:27process, if there was a software engineer out there on some team considering doing some sort of rewrite, what criteria should they look for to decide if it's worth undertaking it or not? It really comes down to what is the business value i mean there's there's the ones you can do for fun where it's your dime and you get to decide you know what fun is worth um just like you know i i really don't like going to casinos because i really don't like what i consider to be losing money gambling but friends of mine love it and they consider it paying for a nice worth of entertainment. They're not losing money. They're paying for fun, right? And so if you've got your own personal development project and you want to spend tokens on porting that to something else,

1:04:16great. Have fun. That sounds awesome. But you need to understand what the benefits are. The benefits are your enjoyment. When it comes to a business, it's what is the business value that this is accruing to? And then like anything else, what is it going to cost? And what is the return on that investment? And that needs to be very clear. And there are lots of these migration projects that people all over the world and businesses all over the world are doing, even before agents, where they have done that cost assessment and they have decided that the return on investment is very much worthwhile. people moving from .NET Framework to .NET Core. And we invested a lot in making .NET Core incredibly fast. And so there would be very real performance benefits that would come from doing

1:05:15that migration. And they could figure out the cost associated with, well, we're spending X millions of dollars running our services today, and we would save $3 million a year by moving, you know, by doing this port and this port is going to take a year and it's going to cost us a developer for that year. So they do the math and, you know, this is, this is absolutely worthwhile. Even at a more micro level, not even for ports, like what features do you invest in? When I was spending a lot of my time working, for example, on the .NET regex engine, I could look to see how much are we spending, we as a company, you know, or how much is this particular service spending on running on regular expressions every year?

1:06:06If I invest two weeks in making these changes to the .NET RegEx engine and it makes the patterns that they're using four times faster, what is the return on investment for my two weeks and how much they save as a service over the course of the year, right? These kinds of trade-offs are everywhere. And you just need to be very clear about why you're doing it, what it will cost and what you get for it. And if you can write all that down and you can justify it, then it's a good project to do. If you can't justify it, it's very suspect. When I was doing research for this interview, I looked into your career. Well, on your LinkedIn, there's no information, but I found an interview you did. Yeah, I'm very, I'm, I basically, I've been at Microsoft for 25 years.

1:06:53I love it. I've jumped around a lot internally, but I've had no interest in going to different companies. And so I haven't really invested in keeping my LinkedIn profile up to date. Yeah. And so I haven't really invested in keeping my LinkedIn profile up to date. But so when I, when I did the digging though, or like, uh, listen to an interview that you had done about the different roles, your path to distinguished engineer is really interesting because you did a bunch of different roles and it wasn't just SWE on team A and then SWE on team B, you held a variety of ones starting as almost a like a forward deployed engineer, consultant engineer. And then you did technical writing at some point. And then you were an advisor to a technical advisor to a VP. Oh, and you also

1:07:41jumped to a PM at some point. From all those different roles that you tried, which of them would you say was the most, like you would recommend someone dabble in that direction if they wanted to try something that wasn't the pure SWE role? Yeah, every role that I did taught me something different. And so from that perspective, like I have great respect for people that you know are come out of college and want to be developers and become an engineer and that's and that's what they do and they just stay on that path like that's a very valid path for me i think i picked up a whole bunch of skills by having more of a circuitous route so like you you mentioned as a forward deployed engineer it was more at the time was Microsoft's consulting arm.

1:08:31And so I would spend anywhere from three weeks to three months at a particular customer site helping them solve their problems. And I would learn a whole lot about their industry. I would learn a whole lot about their particular breed of problem. I'd learn about Microsoft's offerings from the outside and places that it was great in places that it wasn't so great. And learn how our customers, like their real struggles and what they were dealing with and being able to bring that back to then later building solutions for them was incredibly valuable um when uh i then worked on like msdn magazine and was doing a bunch of technical writing and technical editing and figuring out what i wanted to share with

1:09:13people and how i wanted to share it and some amount of development building things to then talk about like i i gathered a set of skills around how to think through what was really interesting what was really relevant uh how to communicate it in a clear fashion uh what didn't need it to be said what did need to be said how to tell a story that would really resonate um from a pm perspective it became you know how do i how do we figure out what actually needs to be built? And what are all the pieces that need to come together to really deliver this thing? Obviously, from a dev perspective, all the coding and the I am incredibly thankful and glad that I did and will never do again, was being sort of this technical advisor for a VP at Microsoft.

1:10:18It was an amazing experience. It was a couple of years where I was there right hand, sort of like invisible partner, um, shadowing everything they were doing and ghostwriting. And after every meeting, having a little, you know, review kind of going through what we heard and what we think we need to do about it and bouncing ideas back and forth. Um, it was incredibly insightful into how things actually work, how decisions actually get made. Um, being a part of things that I wouldn't otherwise have been a part of, at least not for many, many years later. Um, it was awesome. It was also incredibly draining, incredibly selfless. Uh it took a lot out of me. And so I cherish those experiences and what I learned from them. And I would highly encourage anyone who has the opportunity to do so, to do that kind of thing for a period of time. That role is really interesting because it sounds like you're basically a chief of staff

1:11:30to someone super high up, but technical. So you're not like an executive assistant. You're pulled into all the really, I guess, privileged conversations and you're providing that information that they need to make the calls. You mentioned that you learned how things are really done. Do you have some examples of stuff that someone might learn if they were kind of in that role or something that was really helpful for you? There are lots of times where if you're sort of rank and file engineer at a company and you see these decisions that get made an email gets sent out about some new path we're following or some partnership we're forming or some investment that we're making some acquisition that is being done

1:12:19winding down a certain product investing in a new product, spinning up a new team, like all of those things look like they just sort of come out of thin air. And they're, you know, they're just sort of handed down as this is the finished decision, and it's fully formed. And you don't really get insights into everything that led up to that point, and all of the alternatives that were considered and dismissed and why and what the trade-offs were that were factored in and how experienced people in the industry are thinking through those alternatives and those problems and which which things they're choosing to focus on, which things they consider to be irrelevant. All of that is stuff that I didn't have exposure to.

1:13:10And being able to be a fly on the wall, just soaking in, not to mix metaphors, but soaking in all of that, and then being able to have conversations about it. We get out of a meeting and he basically say, so what did you hear? What do you think? Like help, you know, help reason through it. And sometimes I would have valuable insights and sometimes he would say no. That's not how we think about these things. And every one of those was a learning experience for me. Yeah, I've heard engineers say similarly about the value of trying management, even if they ultimately decide they hate it and they don't want to do it again, is you, yeah, you get to see how the sausage is made a little bit more as a manager. You see all those decisions in managing the org that can be helpful in operating uh just in your your career in general yeah and

1:14:08you know i also i i'm i didn't mention it before but i also did that like i i thought that what i really wanted to be was a manager and um so i basically set things up where at a certain point i trade you know i jumped around a lot internally in microsoft and i changed to a role where i sort of handshake agreement that six months later i would be able to sort of shift into a lead position and have a team of people and i arranged all that and it all worked out and i became a lead and six months into being a lead, I was like, I'm not sure if this is actually what I thought it was. And six months after that, I was like, Nope, it's definitely not. This is not for me. And then it took me about a year after that, further to sort of

1:14:59safely work my way back to an IC role where I've been since. And again, to your point, like super grateful for the experience, incredibly grateful to Microsoft for having an individual contributor career track that allows one to sort of progress their whole career and continue to be promoted and whatnot, but still not have to have direct reports. For me, what I realized was the thing that I loved about or that I thought the reason I wanted to be a manager was because I thought it was how I could help others and how I could kind of provide career guidance or help other people solve problems or whatever it was. And that was misguided on my part. And it took becoming a manager to realize that I could do all of that

1:15:51as an individual contributor. And the things that I loved about being a manager were things that I didn't have to be a manager to do. Do you have any top book recommendation that maybe impacted you? Yeah. I mean, it's old at this point, but I've read it several times. And I absolutely love it. There's a book by Charles Petzold called Code. It's probably from 2003, maybe. I don't know, something in that time frame. But I love it. And it is one of the things that I really value is being able to explain complicated concepts in a way that doesn't require the person to know anything about what you're talking about. I think that is a valuable skill, even if you're talking to people that do know what you're talking about,

1:16:41because being able to break down complicated concepts into very clear concise pieces and communicate that clearly and then at a 30,000 foot level is a is a great way of getting everyone onto the same page and sort of arguing your case and pushing things in the direction that you want them to go. And Code is a book where Charles basically does that, where he talks about how computers work from the ground up, teaching people that know nothing about computers that aren't technically experienced about binary, about logic gates, about circuits, about programming, about computers, about the internet, and kind of building things up from basic principles from nothing to a reasonable understanding of how everything works.

1:17:37And I was blown away by the book. And I've since reread it on multiple occasions, even though I know everything that's in there. I just love the way that it communicates things. So if I had one recommendation, that's it. Looking back on all your experience, you've kind of been working in the industry for a while. What professional advice would you give a young engineer just entering the industry today that you believe could be helpful? That's a great question. And it's one I've been struggling with. A couple years ago, I would have had a much more clear answer. But with AI, I still don't know what the right answer is. A lot of the times that I spend working with AI agents, I'm bringing to bear 25 years of experience,

1:18:28for better or worse, in being able to very quickly sniff out things when they're doing the right thing, or they're doing the wrong thing, or how I want to guide them. Or an agent says, I'm 100% confident that the right answer is A. And I'll say, yeah, but have you also considered B and C and D? Because these trade-offs might not be there, right? And then, oh, of course. Just those little things, being able to guide them in the right way, requires you to have gone through it. And I'm concerned that because of how AI agents are progressing, that they're taking away the opportunities to have gone through it. And I don't yet know what the answer then is. I'm a very firm believer in really learning kind of the basics and building everything up from core principles. And so like when, you know, learning

1:19:25a language, I still think it's valuable to learn and to start out with a language like C because you're understanding how computers actually work and you understand what memory management means. And you learn about all the various trade-offs between this algorithm and that algorithm, rather than working at a layer where a lot of that is obscured from you, where the language and environment are providing a ton of services to you, but you don't necessarily understand everything that's happening. And so from that perspective, I still think that there's value in focusing on those core concepts and really internalizing algorithms and how machines actually work and being able to reason and extrapolate from that. And I suspect that that will,

1:20:18in whatever domain you're in, that kind of core primitive knowledge about distributed systems or about databases like that foundational stuff will help to be successful in a world where the vast vast vast majority of the code is actually being written by agents and you're helping to guide them but i don't know that for sure and at the rate that these agents are learning and evolving i don't know what the world's going to look like in two or three or four years. I suspect it'll look vastly different than it does today. I think we're in for a wild ride. Awesome. Well, yeah, thank you so much for your time today, Stephen. I really appreciate it. Pleasure. This has been a fun conversation. Thank you.

1:21:00Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskov, Mike Stonebreaker, Mark Brooker, these were all people that I brought on because someone left a comment. On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype. It's a split keyboard, so there's two sides. This is in the case. But yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of

1:21:39the early units. We're now working on the long journey of building the tooling now. And so if you still want to pick one up, I've left the late pledges open on Kickstarter. So you can grab one there. I'll put a link in the description. Thank you again for watching the podcast and I'll see you in the next episode.