How did Stephen Toub review hundreds of thousands of lines of AI-written code?

Stephen Toub did not read every line and said doing so would be a waste, because agentic reviewers beat humans on line-level detail. He looked at every pull request for direction, architecture, componentisation and algorithms, and ran line-by-line reviews from three or four models of different families, often on every rebase. In the GitHub Copilot app's agent merge feature he lets agents rebase, fix CI and answer review feedback, but keeps the final merge for himself.

• Toub said switching to a different model family is the easiest way to get a genuinely different review perspective.
• For an open source project he would only let the agent drive CI to green, never answer arbitrary review comments or auto-merge.

Source: our own transcription of the episode. The full conversation is on Ryan Peterman (2026-11-02). Read the whole episode summary →

▸More from this episode