Meditation on LLM Interpretability

I have been spending more time on interpretability lately, doing several more projects to see if I can find a new angle for understanding what LLMs are doing. I even brought out a few old friends, including HTM, to see if tricks and obscure old ML methods could make a dent in stopping humanity from going out... That's mostly a failure. Welp, expected: if interpretability is anywhere remotely close to hard, we'd already have done it. But that got me thinking about something that's been in the back of my mind: what exactly is the limit of interpretability?

Seriously. What do we mean when we say we are doing interpretability? There's already a very concise and accurate explanation of how the model works: the model itself! It's not like LLMs are magic and we need to burn a part of our soul as an energy source. They are just numbers and math operations applied in a very specific, predefined order. RoPE here, MatMul there, attention somewhere else, etc. The real question interpretability is asking, from my understanding, is how to establish high-level understanding and control over the model. Note that I didn't say human understanding -- the model's weights may implement complex operations via a short chain of nonlinear vector operations, and it could take many more words in English than there are bits in the weights to explain them with all their important nuances. For LLMs or humans alike, exploding circuits and algorithms implemented in weights into a long English explanation doesn't help anyone, and there's no guarantee that the English explanation is even remotely understandable.

There is this tension of abstraction. The more precisely we attempt to describe things, the closer we get to just describing model weights. And the higher-level and more concise we go, the less accurate we are. Yet at the same time, interpretability's original goal was to completely understand the operations of models. Even if we can do that, the complete functionality of the model might be so long to describe that we are just better off not doing so.

The interpretability world has largely given up on completely understanding models -- we are yet to fully understand how GPT-2 works. The focus now seems to be either trying to extract internal state or inventing new ways to understand and, more importantly, induce an expected change in the model. The former includes methods like using Sparse Autoencoders to decompose the residual stream into its components and see if any of the basis vectors correlate with English concepts (though not necessarily being able to write anything back into the residual stream). The latter includes attempts like Anthropic's J-lens: finding a new way to look into the model, write back, and prove there's a predictable effect.

Either way, I find the theme is always some unparameterized or unsupervised approach. And it's fair that we can't exactly train against ground truth; the job is mostly information decomposition.

And yet I wonder: There must be a way to do better. While it is tempting to explain the model's behavior using only linear algebra or by constructing circuits, and while doing so would contribute greatly to our understanding of models, at some point control using low-level schemes becomes impossible, as the sheer amount of code needed surpasses practical limits.

Maybe there's an impossible triangle here. The primitives have to be high-level, we ought to expose detailed mechanisms, and the explanations of those mechanisms have to be short enough to understand.

I just don't think it is possible to satisfy all three of them.

The current approach of SAEs and J-lens kind of gives us an "I don't know what is happening, but we can prove effects" approach. I am wondering if we can take it further. Maybe we can distill enough of the internal mechanisms of an LLM into other, more explainable architectures, then study those as a proxy. Or better, maybe we can find an explainable architecture that can replicate the internals of LLMs closely enough. Maybe we can gradually replace the internals of models and deploy that instead, or even use it as a source of diffs in case the model is doing something we don't recognize.

I don't know what I am trying to say. These are just things on my mind.