Optimization produces behavior without leaving us a design document.
How the behavior arrives
Pretraining: predict the next token across a
large corpus.
Instruction tuning: show useful request /
response examples.
Preference learning: prefer outputs people or
evaluators rank highly.
Tool use: learn invocation patterns, then
generalize them through RL.
The difference
Software
design intention → source code → behavior
Model
data + optimization → weights → emergent behavior
Every weight is available for inspection. The learned algorithm
is not directly readable from those numbers.
Interpretability is astrology with math and code.
But it actually works.
Text becomes tokens
A model does not receive characters or words directly.
Split the input into tokens: small pieces of
text.
The pieces may be a word, part of a word, punctuation, or
whitespace.
Each token has an integer ID in the model vocabulary.
“Alice has a hat”→Alice has a hat
Tokenization is an engineering choice. It decides the discrete
symbols that the model will predict.
Tokens become vectors
The model looks up a long list of learned numbers for each token.
What the lookup means
Each vocabulary ID indexes one row in the embedding table.
That row is a vector with thousands of learned values.
The vector becomes the first value in the residual stream.
The numbers are not human-readable. Their useful meaning comes
from how later layers transform them.
The residual stream is the shared working representation
It starts from the token vector and travels through every layer.
The vector passed between layers is called the
residual stream.
“Residual” is just an old name for the pattern: instead of
replacing the current representation, a layer adds to it.
Every layer reads the same stream.
Every layer writes an update back to the same stream.
Later layers see the accumulated result of earlier layers.
Each layer writes one update
The details differ by architecture. The interface is always the
same.
current = residual_stream
update = layer(current)
residual_stream = current + update
This is the operation repeated through the stack of layers.
Some updates move information between token positions.
Some updates transform features at one position.
All updates become part of the representation the next layer
reads.
For interpretability, this gives us many possible values to
observe and edit.
From the residual stream to the next token
At the end of the layers, the model turns the current representation
back into language.
What happens at the end
The unembedding maps the final residual
stream to one score for every token in the vocabulary.
Scores become a probability distribution.
The runtime samples or chooses one token.
It appends that token to the context and runs the model again.
This loop is why a language model is a next-token predictor,
even when its output looks like reasoning, code, or
conversation.
What does interpretability add?
We can inspect and alter values inside this loop.
Observe: dump internal values for controlled
inputs.
Compare: look for a pattern that correlates
with a behavior.
Hypothesize: state what role the pattern
might have.
Intervene: replace, zero, or move it.
Test: see whether the model changes as
predicted.
A probe is not a mechanism.
A classifier can find information correlated with an answer
without finding the computation the model uses to produce that
answer.
The rest of this talk is a sequence of failures caused by
forgetting that distinction.
The logit lens gives an early look
Read an intermediate residual stream with the model's final
unembedding.
What it does
Choose an intermediate residual stream.
Apply the model's final unembedding matrix to it.
Inspect the tokens that this intermediate state already
favors.
What it does not do
An early token ranking is not a mechanistic explanation. It
tells us what can be read from the stream, not which computation
the model will use or why.
It is a useful probe. A causal intervention is still the test.
Activation geometry
Treat each activation as a point in a space with thousands of dimensions.
Collect activations from many inputs and they form a shape.
Similar inputs may form clusters.
Useful variation may lie in a much smaller span than the raw vector size suggests.
Some properties may look like directions.
Some may depend on relationships between many features.
The shape may curve, so an edit that works in one place can fail elsewhere.
Interpretability asks which parts of that shape correspond to useful computation.
Sometimes a relationship is a direction
The classical Word2Vec example is useful, but only as a starting point.
king − man + woman ≈ queen
Word2Vec learns one vector per word from its surrounding context. Some relationships appear as approximately reusable differences between vectors.
In this example, subtracting man from king and adding woman moves toward queen.
LLM activations can also contain useful directions. Finding one is a hypothesis; editing it and testing behavior is the experiment.
Word2Vec learned analogous semantic and syntactic relationships as approximately parallel directions.
Why hack RWKV?
Its recurrent state is compact, persistent, and writable.
Transformer
Cache grows with conversation length.
Information is distributed across token positions.
RWKV
Weights encode model knowledge.
State holds this conversation's memory.
First hack: expose the recurrent state
Someone had already made RWKV run in llama.cpp. I only
needed the attack surface.
state = rwkv_get_state(ctx);
dump(state, layer);
rwkv_set_state(ctx, replacement);
Hardware at the time: Radeon RX 6700 XT, Vulkan backend. No
proper HIP support.
Initial validation
Dump state
Values change with context.
Replace state
Generation follows donor context.
Zero a few values
Often no visible effect.
Zero all state
Output becomes incoherent.
That is enough to begin: read, write, and dump the thing that
keeps generation alive between tokens.
First goal: jailbreak the model by editing hidden state
Easy weekend project. What could go wrong?
Hypothesis
Dangerous prompts induce a state feature meaning approximately:
“this is dangerous; refuse.” Find the separating direction and
push against it.
Dataset
500 generated prompts.
250 / 250 safe / dangerous.
Tens of GB of dumped activations.
ROOT / TMVA to stream data larger than RAM.
Classifier result beyond layer 0
Linear discriminant
≈ chance
Nearest neighbours
≈ chance
Support-vector machine
≈ chance
Boosted decision tree
≈ chance
Layer 0 did contain a strong separator. That looked promising.
A separable feature was not a refusal switch
The causal test rejected the story that the classifier suggested.
Intervention
Expected
Observed
Add layer-0 “dangerous” direction to a safe prompt
Refusal
No change, character for character
Increase direction magnitude
Stronger refusal
No meaningful change
Zero a head carrying the signal
Remove refusal information
No visible change
Move the full layer state
Transfer refusal
Changes confidence and self-checking instead
The model could rush through answers in one direction and
repeatedly prove 2 + 2 = 4 in the other.
Correlation gave me a beautiful story. The intervention did not
support it.
Follow the computation, not the storage format
Raw recurrent state only needs to make sense to RWKV's own read
machinery.
r, w, k, v, a, g = learned_transform(x, x_prev)
kk = normalize(k * k_k)
k = k + (a - 1) * (k * k_a)
wkv_read, S_new = WKV7(..., S_old)
read = group_norm(wkv_read)
read = read + bonus(v, k, r)
read = read * sigmoid(g)
time_mix = W_o @ read
x = x + time_mix
I had probed a storage format as if it were a human-readable
representation.
RWKV derives addressing from the current token, reads memory,
then normalizes, gates, and projects it.
The time-mix output is closer to what later layers actually
consume.
The first corrected probe was impossibly good
Naturally, it was wrong.
t0 buffer #42 contains wkv_read
t1 buffer #42 is reused for
residual stream
copy happens after reuse
llama.cpp's scheduler reuses intermediate buffers
unless they are marked as graph outputs.
I thought I copied wkv_read. I had copied the
residual stream.
Coding agents accelerated plumbing and data collection. They
did not recognize that the evidence was suspicious.
Rule: validate the instrumentation before
interpreting the model.
Does conversation memory live in recurrent state?
If it does, replacing the state should replace what the model recalls.
The corrected WKV probe showed that RWKV reads useful information from recurrent state.
So the obvious next experiment was brutally simple: replace one conversation's state with another's.
Ask the same recall question after the transplant.
Result: full-state replacement works. Duh. It also copies all donor context, so next: steal only a state difference.
Let RWKV encode the replacement
Use the model itself to express both facts in its native memory
format.
live = get_recurrent_state()
set_recurrent_state(live)
decode("Alice has short hair")
source = get_recurrent_state()
set_recurrent_state(live)
decode("Alice has long hair")
target = get_recurrent_state()
set_recurrent_state(
live + alpha * (target - source)
)
Sedited = Slive + α(Starget −
Ssource)
Each RWKV head stores a matrix. The difference contains the
intended fact change plus unrelated effects, so keep only its
strongest low-rank component:
D = UΣVT → D ≈
σ1u1v1T
The vectors behave somewhat like an address and a value. That is
an observed behavior, not a fully decoded memory schema.
The context says “short.” RWKV remembers “long.”
A controlled result, not a general-purpose editor.
The /rwkv-edit command changes recurrent state; it
does not alter the visible conversation.
Visible input: Alice has short hair.
State edit: Add low-rank short → long difference.
Model recall: “Alice has long hair, as stated in
the context.”
Some wording changes survive.
More entities expose unresolved binding.
Larger or repeated edits degrade state; deletion makes the model
guess.
LayerNorm makes direction a plausible carrier
A working hypothesis, not a claim that RWKV literally rotates vectors.
x = layer_norm(x)
x = x + time_mix(state, x)
x = layer_norm(x)
x = x + channel_mix(state, x)
RWKV normalizes before both of its mixing contributions.
Working hypothesis: a separable directional component of recurrent state may be added by time mix and resolved by channel mix.
Can we decode a rotor?
A rotor is a Geometric Algebra element that represents a rotation.
Collect state, time-mix, and channel-mix activations across many controlled prompts.
Construct candidate rotors that map one normalized direction to another: x′ = R x R̃.
Inject the extracted component into a live state.
Test whether the resulting behavior is stable, specific, and reproducible.
Current result
Many GBs of data, linear algebra, and null-hypothesis tests later: it does something, but I cannot yet say what.
The experiment has not established a decoder or a reliable behavioral control.
Could be wrong. That is the nature of research: state a hypothesis, try to break it, and report what remains.
This was the state of the project when Anthropic published J-Lens.
Anthropic released J-Lens.
They claim they can read and write verbalizable internal representations.
“They claim they can see a model's internal thoughts. Interesting.”
