A Model That Writes Its Own Adapter — and the Bug That Almost Hid It

The premise of Memory-LoRA is simple enough to state in one line: point it at a GitHub repository, and it emits a LoRA adapter for that repository in a single forward pass. No fine-tuning job. No RAG. Zero repository tokens at inference time.

Ask the same model the same question with the adapter off and on — the adapter is the only variable:

click was never seen during training, and no part of the repository is in the prompt.

That is the finished version. The interesting part is how close I came to shipping a broken one.

Six views, not one

Each repository is summarised along six axes, embedded by a frozen Qwen3-Embedding-0.6B:

One view would capture what the code looks like. Six capture how the project is organised, which is the thing a coding assistant actually needs to know.

The head maps that vector to an (A, B) pair per shape-qualified module type, shared across the transformer layers of that shape:

Δ = (α / r) · (x·Aᵀ)·Bᵀ        A: [r, in]   B: [out, r]

That is identically PEFT's LoRA convention, which is deliberate — the output drops straight into a standard adapter with no re-derivation and no approximation. Gemma stays frozen throughout; the LM loss backpropagates through the non-detached A and B tensors into the hypernetwork. Only the head trains.

The number that lied

An earlier checkpoint reached an eval loss of 2.606. Respectable. Trending down. The kind of number you screenshot.

It was worse than applying no adapter at all. It was also worse than random noise of the same magnitude.

The cause took a while to find, and it is embarrassing in hindsight: 64% of every repository embedding was a constant vector — the frozen encoder's mean response to the general concept of "source code", present identically in every repo. It dominated the input, and the trunk duly collapsed onto it.

Read the last bar. The repositories were perfectly distinguishable in the raw input. The network threw that away, and then produced essentially the same adapter for every repository on Earth — an adapter that was, on average, mildly harmful.

The fix is three lines of statistics: fit_input_stats() standardises the conditioning input using training-set statistics, stored in the checkpoint so training and inference apply the identical transform. Emitted-adapter cosine across repos fell 0.96 → 0.21 in 40 steps.

The lesson is about the metric, not the bug

The bug was recoverable in an afternoon. What should have been alarming is that it survived weeks of training runs, and the reason is that training logged only the adapted loss.

A loss of 2.606 says nothing on its own. It only means something next to the frozen-model baseline on identical data — and that number was never on the dashboard.

Every evaluation now reports delta_vs_baseline and diag/adapter_cosine, and those are the two metrics I watch. There is also a zero-B control in diagnose_head.py that must reproduce the baseline exactly: it proves the adapter is being applied correctly before any other number is worth reading.

If you take one thing from this post: an absolute loss is not evidence. A delta against the thing you claim to improve on is.

Results, after the fix

Cross-repository held-out loss against the frozen base, lower being better:

The absolute losses behind those deltas are 2.606, 2.7168 and 2.6811 — note that the raw number went up between the first two while the model got dramatically better. The eval sets differ; only the delta on identical data is comparable. The final run doubles the corpus to 2,146 repositories and 55,700 QA pairs.

Benchmarked across three unseen repositories, three task families each:

Nine of nine combinations won.

What it does not do

The third row of that table is the honest one. AudioBench keyword accuracy stayed at 0% — unconventional projects, whose identity is not inferable from their structure, transfer poorly.

More generally, the adapter learns a repository's stack and conventions, not what the project does. On requests it will confidently tell you it is an XML library. That is exact factual recall, which a rank-16 LoRA structurally cannot hold, and retrieval is the right tool for it.

And the base losses of 12–16 on short gold targets inflate those deltas. Keyword accuracy and the raw generations are the trustworthy evidence; the nats are the headline.