Why does Noam Brown warn against supervising an AI's chain of thought?

Noam Brown says a reasoning model that writes out its thinking in plain language is a rare gift for safety, and punishing it for bad thoughts teaches it to hide them. Every intervention based on what researchers read adds a little pressure toward concealment. He says OpenAI already sees chain-of-thought monitorability degrading as models get better at controlling their reasoning.

• Brown credits a colleague, Jakob, with insisting from the start of reasoning models that chain of thought must not be supervised.
• Brown says models already recognise many test setups, for example declining to open a folder holding an answer key because it looks like a trap.

▸In their words

“we cannot supervise chain of thought because this is really a gift.” Noam Brown · at 1:09:20 — open in the transcript →

Source: our own transcription of the episode. The full conversation is on Dwarkesh Podcast (2026-09-17). Read the whole episode summary →

▸More from this episode