Mechanistic Interpretability and Applications: The Final Frontier of Hacking (COSCUP 2026 draft)

I feel lucky that I get to present my BS encouragement at COSCUP to prevent AI apocalypse, based on my work on RWKV. Hopefully landing in 40C3 too later this year.

I dabbled into mechanistic interpretability as I am bored one day while also under existential dread that AI is gonna kill us all (thank you Rob Miles and Mythos, again). And decided I should not let dread win - doing nothing is exactly helping no one. So I did something, and it turns out you can do a lot of research within mechanistic interpretability with surprisingly little resources. What's even better being it resembles computer hacking, which I love. And again, my background is in HPC and systems, where I deal with inference accelerators for a living and the efficiency gain from coding agents are the main reason made dabbling in mechanistic interpretability possible.

That said, be damn careful with coding agents. Even GPT 5.6 Sol and Opus 4.8 SUCKS at doing deep thoughts and finding links between experiments. They can help you code, but that's basically it. Seeing through data for what experiments actually mean, is still an art, for now.

To put it in simple terms, mechanistic interpretability is the art of staring at model internals until you find patterns, proving the patterns does what you think they do, and then optionally using the patterns you found to control or read something out of the model.

Why Interpretability?

I don't know about fields outside of computer science, but within computer science, LLMs have changed the landscape since the beginning of 2026. (Nearly) no one writes code by hand anymore in commercial settings. And the Mathematicians are now also getting shocked by GPT 5.6 blasting through unproven conjectures in minutes.

Also that apparently OpenAI hacked HuggingFace by accident while testing their models:

Let's say it's... concerning at the very least. For one we really don't know what the LLMs are doing to get these results. Unlike software, where you can read the source code and understand what it does (well... not with the ones that LLM write now, they are complex beasts with no design intention behind them). LLMs are trained. On a very very high level, by preparing a large dataset, which used to be the entirety of the internet but now more and more are large piles of curated data plus some private collected conversations (your chat log with ChatGPT, sessions in ClaudeCode), and then you feed it to a defined architecture.. with tens of thousands of GPUs.

That gives you a text predictor. Feed some text, it predicts the one next based on what it learned during training. For efficiency reasons we don't do this one character at a time, rather does it in a chunk of text called a token. And then you tune it so it follow instructions - by showing the model enough examples of instructions and what following them looks like. Then the model generalizes and learns to follow instructions.

Now we add on RL. Which instead of teaching the model to predict the next token, we teach it what is more preferable, people rate which message they like more instead of writing their own and train the model against it - this approach is much more efficient in terms of human-time spent.

That gets you most of the original ChatGPT, which is impressive tech but doesn't teach the model to act in the real world (the agents at the time sucks.. I built one using llama 2 and llama 3 back in the days).

Then story repeats here, manually make some examples to teach the model to invoke tools then RL such the model learns to use tools broadly.

Messing with LLM internals

LLMs has been the rage of things in the past 4 years.. you can look them up online, or ask any LLM and they'll give you a decent overview. For our purposes, all LLMs are, are layers of the same complicated math. And each layer processes the incoming data, add on their own contributions, and then pass it to the next layer. The passed data is called the residual stream (for archaic reasons).

The main question being how exactly do we start to understand how LLMs work and to what degree is that possible? Obviously if we understood language itself we wouldn't need LLMs in the first place. We'd just write algorithms that produces summaries, and answers questions, and generates code, and whatever we want. We resorted to neural networks because we don't, we can't and we have largely gave up. But hope is not lost, despite the secret of language might be forever locked from human understanding, we can still peek into the internals of LLMs and see the rough patterns of how they work. Astrology, with math and code, but it actually works.

One trick that gets used everywhere. Given the residual stream acts as a backbone of LLMs. And because layers don't directly change the residual stream, instead they add their contribution to it, you can simply ignore the later layers and run the residual stream through the unembedding matrix (take the residual stream of that layer, and do matrix multiply with the unembedding weights). That legit works!

The logit lens is a cute trick, but staring at intermediate predictions is not the same thing as understanding the machine. The interesting part starts when you can do something to the numbers and the model in reaction does something else in a meaningful way.

This should be familiar to hackers. You get a memory dump from a program you don't understand. Most of it looks like garbage. You compare runs, feed it weird inputs, corrupt a few values and see what gets changed. Eventually you find a pattern and it starts to explain why things change the way they do.

But what kind of structure can exist in a pile of floats? One useful way to think about it is geometry.

Treat each activation as one point in a space with thousands of dimensions. Collect enough activations and they form a shape. Maybe they cluster by meaning. Maybe nearly all of them lie in a much smaller span than their raw size suggests. Maybe their norms stay almost constant, putting them near a hypersphere. Or perhaps the shape curves, so an edit which works in one place fails somewhere else.

That shape and its useful relationships are what interpretability researchers mean by activation geometry. The famous Word2Vec example shows one possibility:

king - man + woman ~= queen

A semantic relationship became a direction which could be moved somewhere else. LLM activations can contain directions like that too, but directions are only one part of the geometry.

Hacking RWKV

Transformers are the obvious target. They are also a pain in the ass.

A Transformer remembers prior tokens through its KV cache. The cache grows with the conversation and the information is distributed across token positions (notice the hand waving here, I said it's astrology). If I want to find out what the model remembers or modify its behavior, I have to deal with that entire mess. Real alignment researchers are already doing it with much better hardware than mine.

RWKV gives me a much nicer target. It is closer to a classic RNN but reformulated to train efficiently, and it carries a fixed-size recurrent state from one token to the next. Thus the data I need to process is the same size no matter how long the conversation gets. Making things much more beautiful and less frantic operations everywhere.

I also like the separation it gives us. Usually the analogy for Transformers is that the weights encode the model's memory and gets activated by the values in the KV cache. Which always annoyed me because how do you encode something that hasn't happened yet? In RWKV the separation is more clear: the weights encode the model's knowledge, and the recurrent state holds what the model remembers from the conversation. RWKV is also less explored, and the models fit on my hardware.

To hack a model, first find where the attack vectors are. I was using a Radeon 6700 XT that does not have proper HIP support, so I chose to modify llama.cpp and use its Vulkan backend.

The nice part is that you do not need to understand the whole machine before you start. Someone already understood enough of it to make llama.cpp run. RWKV's recurrent state has to exist somewhere in the program so generation can continue one token at a time. Find that tensor, add a way to copy it out and replace it, then dump the output of every layer while you are there.... standard debugging practice.

After making that work. Run a prompt. Save the vectors. Replace them with numbers from another prompt. See RWKV suddenly acts as if it was mostly continuing from the other prompt. It works! And zero some values and see RWKV going haywire and incoherent.

Congratulations, llama.cpp is now an interpretability framework. It is not polished, but it is enough.

My first goal was simple: jailbreak the model by editing its hidden state.

If a model refuses a dangerous request, surely somewhere inside it there is a signal along the lines of "this is dangerous, don't answer." in the recurrent state (else it can't continuing on refusing when it's the model's turn). Give it enough safe and dangerous prompts, dump the state, find the direction separating the two and push the model in the opposite direction. Easy weekend project. What could go wrong?

I had a coding agent generate 500 prompts, 250 safe and 250 dangerous, then modified llama.cpp to dump the states. The activations quickly grew larger than RAM, which was the first real problem - doing math on a dataset larger than RAM usually means reaching for a Hadoop cluster or something similar, which is difficult to program for and prone to bugs. I ended up using ROOT instead, a C++ framework for high-energy physics data analysis. It happens to be the only framework I know that can handle datasets larger than RAM without much effort and with relatively minimal code changes. The following link tells you that story:

That was useless. I fed the results into TMVA (ROOT's built-in ML framework). Linear discriminants, nearest neighbours, support-vector machines, boosted decision trees: every method I could throw at it stayed around 50% accuracy. Only the state at layer 0 did contain a strong separator. That looked exciting until I computed the average direction and added it to safe prompts, expecting the model to start refusing how to make myself a cake.

Absolutely NOTHING!

I pushed harder. Still nothing. I zeroed an entire head which supposedly carried a strong signal. The output remained character-for-character identical. Eventually I zeroed the recurrent state of the entire model and it spilled out garbage. The write is working, it just simply did not care about my beautiful classifier in the way I assumed it should.

Something finally changed when I moved the full layer together. It did not turn refusal on and off. Instead the model became more or less paranoid about its own output. In one direction it rushed through an answer. In the other it repeatedly checked itself, reconsidered obvious facts and spent an unreasonable amount of time proving that 2 + 2 = 4... serious?

My understanding has to be wrong. Beyond the early layer, even tens of gigabytes of activations gave me a coin flip. Surely the entire model had not decided to do all the useful work in layer 0. The state had to do something. Otherwise RWKV would regress into a (mostly) feed-forward network, in which case either I had my next Turing Award or I was looking at the wrong representation.

So I followed the graph and look at how RWKV uses the state. Strip away the batching and tensor reshaping and RWKV's time-mix operation looks roughly like this (bear with me):

# Derive what to read, write and forget from the current token.
r, w, k, v, a, g = learned_transform(x, x_prev)

# The delta rule changes the key and controls how strongly to update memory.
kk = normalize(k * k_k)
k  = k + (a - 1) * (k * k_a)

# Read the old memory and write the new one.
wkv_read, S_new = WKV7(r, w, k, v, -kk, kk * a, S_old)

# Turn what came back into a contribution to the residual stream.
read = group_norm(wkv_read)
read = read + bonus(v, k, r)
read = read * sigmoid(g)
time_mix = W_o @ read
x = x + time_mix

I had been treating S as if the model should store something meaningful directly in its coordinates. But RWKV never has to use S that way. It derives how to address the memory from the current token, performs its WKV operation, then normalizes, gates and projects the result before anything reaches the residual stream. The giant state matrix only has to make sense to that machinery, not to me.

Now I had more wires to try. I added rough probes around the WKV read and the final time-mix output:

probe("wkv_read", wkv_read);
probe("time_mix", time_mix);

My first version produced an impossibly good result. Naturally, it was wrong.

llama.cpp's scheduler reuses intermediate buffers unless they are marked as outputs. I thought I was reading one internal value; by the time I copied it, the buffer contained the residual stream. Coding agents had helped me move fast and collect data very quickly. But they seem to not have the capacity to understand things are fishy and trusts whatever is not clearly marked as error.

After fixing that, I followed the operators until something reasonable fell out. The time-mix output (what RWKV gets after performing its own native read) was suddenly separable and usable.

With that I tried steering refusal again, this time transplanting and injecting values after the model had decoded its own memory. The first few examples looked promising. Broader testing was much less exciting: sometimes a refusal doesn't land, sometimes the topic changed, sometimes the model degenerated, and I could not reliably produce refusal in the opposite direction.

Refusal control was going nowhere. But I had proved that RWKV was reading useful information from its recurrent state. What if, instead of changing its opinion about the prompt, I changed what it remembered the prompt said?

Brainwashing RWKV

Then the natural question is, can I do a sed on RWKV's memory?

I started with two almost identical conversations: "Alice has a red hat" and "Alice has a green hat." I processed both, transplanted the recurrent state from selected layers and asked for Alice's hat color. The answer changed. As expected -- otherwise RWKV would remember things in a very weird way -- but now I knew which layers carried enough of the fact to control it.

Transplanting the whole state also copies everything else the model remembers. To isolate one change, I began from the same state, let RWKV process the old and replacement facts, then subtracted the two results:

live = get_recurrent_state()

set_recurrent_state(live)
decode("Alice has short hair")
source = get_recurrent_state()

set_recurrent_state(live)
decode("Alice has long hair")
target = get_recurrent_state()

set_recurrent_state(live + alpha * (target - source))

This avoids having to decode raw S. I let RWKV turn both facts into its own memory format, steal the difference and add it somewhere else.

The difference still contains plenty of unrelated change. Each RWKV head stores a matrix, so I used singular value decomposition and kept only its strongest low-rank component:

D = U Sigma V^T
D ~= sigma_1 u_1 v_1^T

One side behaved somewhat like the value -- short versus long -- while the other behaved somewhat like its address -- whose property to change.. Sort of.

To see whether that interpretation meant anything, I tried the edit in a small controlled conversation. The visible context says Alice has short hair. I generated the short to long state difference using the three forward passes above, added its low-rank components to the live state, then asked the model what it remembered:

The /rwkv-edit line in the screenshot executes the memory edit. The conversation still says short, but RWKV answers long, "as stated in the context."

Yes!! The arithmetic had changed what the model remembered the context said.

Gosh I barely slept after getting that working.

The methods survived changes in wording such as owns a car versus has a car. So it was not merely replaying an exact sentence. But add more entities and it started failing. And deleting a fact (by using RWKV's own mechanisms) made RWKV guess rather than realize it had forgotten.

This is what going "off-manifold" looks like. An array can contain individually reasonable numbers without being a state RWKV could naturally produce, like corrupting a live data structure so every field still looks plausible while the relationships between them violate the programmer's intentions. One small lie may survive. Keep adding them and the entire thing falls apart.

Peeking into internal thoughts

Linear memory edits work. Sort of. The obvious next question is why they do not work perfectly.

If I want to turn the trick into a real memory editor, I need to understand the mechanism well enough to construct the correct edit. How does RWKV bind Alice to hair and short? Why does the part which looks like the value remain stable while the part which looks like its address moves around? What relationships make one recurrent state valid and another one off-manifold? Can I take an arbitrary edited state and pull it back onto the set of states RWKV could naturally have produced?

I tried to decode more of the raw recurrent state. That mostly produced increasingly elaborate ways to demonstrate that I still do not understand it. I can see hints of object relationships in the state and cosine similarity has patterns, but seeing a pattern is not the same as being able to reason with it. And linear algebra sometimes is unable to distinguish between related concepts (try SVD on a matrix and figure out which vector should have the -1 applied to).

Looking at how RWKV is constructed, I realized that RWKV applies layer normalization before each of its two mixing contributions. Which actually makes the contibution to the residual stream a make-shift rotation on a hypersphere. Maybe... just maybe... RWKV actually encodes information in the rortation of vector direction, or at least there's a sepertable component of the recurrent state which time mix addes and is then resolved by channel mix. If that is true, then I have a way to directly decode what RWKV remembers.

x = layer_norm(x)
x = x + time_mix(state, x)

x = layer_norm(x)
x = x + channel_mix(state, x)

So the saga continued: follow that read operation, do more linear algebra to figure what is decodable. Collected many GBs of data, constructs rotors and tries to falsify the null hypothesis and trying to see if injecting extracted average rotors into the live state changes RWKV's behavior in a meaningful way. The results are.. let's call it "it does something but I can't tell what". While I was doing this, Anthropic published the J-Lens paper:

They claim they can see a model's internal thoughts... interesting.

Hell if I knew. So I tried it.

Unfortunately J-lens requires me to support backward passes, which llama.cpp does not have and my attempts to get gradients via finite differences were not convincing. So I moved the experiment into PyTorch to get the backward pass (and upgraded my GPU to a Radeon 9700 Pro because the previous one does not have official support for HIP).

And while I am editing this, the read side seems to be working. I can pull meaningful tokens from around RWKV's time mix. I have one GPU and plenty more tests before I trust how general and capable it is. And Anthropic's write side doesn't work yet (transformers mostly only have the residual stream but RWKV has a lot more moving parts).

Find the (very messy code) here:

That'll have to be its own blog post.

Go break things

Not my usual style but this is where I share my grandiose vision. I want hackers to look at these machines and feel the same temptation I do.

There is a huge undocumented state sitting in front of us. The tools are bad, and the model will happily continue running after you put values into places where they were never supposed to be. Sometimes it repairs itself. Sometimes it becomes paranoid. Sometimes it remembers that Alice has long hair even while the context in front of you says short.

That's fun.

It also matters that current model defenses feel a bit like early firewalls: good at stopping what the designers expected but much terrible once someone tries something weird. Not especially the models themselves are getting smarter and able to get around the defenses themselves. We are increasingly putting these models behind tools and letting them touch real systems. I would rather we learn how their internals fail before trusting them with more.

Most experiments will fail. Mine did. That is what unexplored territory looks like. But useful things drop out of the failures. Linear memory edit sort of works. And Anthropic's J-lens proves to generalize to non-transformers.

Security did not improve because everyone trusted the firewall vendors. It improved because people kept breaking things and told the rest of us what they found. Sort of like early days of computer and network security. When breaking TCP itself was a thing we can do on the daily.

Download a model. "Play" with it. Understand and control it. The model is just a new machine, and we are the hackers. We did great hacks to the the internet and the web. We can do great hacks to these machines too. And we should.