Interpretability

Interpretability is the study of what happens inside a trained neural network, to explain why it produces a given output. Mechanistic interpretability tries to map the network's internal units and connections to human-understandable concepts and step-by-step procedures, much as a biologist traces a circuit in the brain. The work matters because a network's behavior is learned, not written by programmers, so no one knows in advance how it works.