For the first AI hackathon held at the French National Assembly, the theme was trustworthy AI in the legislative process. I built Légitus, a French legal assistant, in 25 hours of training.
The base model choice was the first design decision, and it was about compliance rather than benchmarks. I started from Apertus, the open LLM from Swiss AI: fully open training data and weights, developed transparently. For a legal-domain assistant, where auditability is the entire point, a model whose training corpus is a trade secret defeats the exercise before it starts.
Four stages, each one answering a problem created by the previous one.
Stage 1 · CPT
Continued pretraining on LEGI
LoRA r=64 / α=128 on every linear layer, plus embeddings and output head trained on a higher learning rate. Packing at 2,048 tokens.
gains Knows the shape of French law
costs Cannot hold a conversation
Stage 2 · Data
27k synthetic instructions
Generated across Gemini, DeepSeek, Nemotron and Qwen, then validated example by example — language, length, and the expected abstention markers.
gains Citation and abstention are teachable
costs Synthetic data inherits its teachers' habits
Stage 3 · SFT
Instruction tuning
LoRA r=32 / α=64 with embeddings and head frozen. Loss on assistant turns only, masked by hand against the Apertus delimiters.
gains Grounds and cites reliably
costs Does not always stop cleanly
Stage 4 · Merge
Interpolate with Apertus-Instruct
0.55 × Légitus + 0.45 × Apertus-Instruct, chosen to recover turn-taking without washing out the specialisation.
gains Concise, terminates cleanly
costs Self-identification regresses
1. Continued pretraining — put the law in the weights
Before any instruction-tuning, the base model gets continued pretraining on LEGI — the consolidated French codes, laws and regulations, via Légifrance.
The configuration is a little unusual for a LoRA run: rank 64, alpha 128 across every linear layer, plus full training of the embeddings and the output head on a separate, higher learning rate, with packing at 2048 tokens. The embeddings matter here because legal French is not general French. Article references, codified numbering, and the specific register of statutory text are poorly served by token representations tuned on web text, and a frozen embedding table caps how much of the corpus the adapter can actually absorb.
The result is a model that knows the shape of French law, and cannot yet hold a conversation.
2. The dataset is where abstention is taught
~27,000 synthetic instruction examples, generated across multiple model families — Gemini, DeepSeek, Nemotron, Qwen, through OpenRouter — rather than from one teacher, so the failure modes do not all point the same direction.
They cover nine categories, and the choice of categories is the design:
Grounded RAG with citation
Abstention — insufficient sources
teaches refusal
Abstention — incorrect sources
teaches refusal
Legal search tool calls
Extraction
Closed-book reasoning
Structured citation
Situational questions, plain language
Identity & conversational turns
Two of those nine are explicitly about not answering. That is deliberate. The failure mode of a legal assistant is not ignorance; it is a confident, well-formatted, fabricated article number. A model that has never been rewarded for abstaining will never abstain.
Every generated example is validated before it is kept — language, length, and the presence of the expected abstention or problem-detection markers — rather than trusted because a large model produced it. The set is public as moncefem/legi-instruct-fr.
Two smaller sets go in alongside it: a 200-example bilingual identity set (who the assistant is, greetings, corrections of mistaken identity — written deliberately short and cleanly terminated), and a filtered selection of general instructions from four external French and multilingual datasets, for conversational fluency without diluting the specialisation.
3. SFT — and a chat template with no generation block
Instruction-tuning runs LoRA at rank 32 / alpha 64 on top of the CPT model, with the embeddings and output head frozen this time, to preserve what stage 1 put there.
The loss is computed only on assistant turns. That sounds routine, and it was the fiddliest part of the whole pipeline: Apertus's chat template has no native generation block, which is what most training stacks rely on to locate the assistant's tokens. The masking is done by hand against the <|assistant_start|> and <|assistant_end|> delimiters instead.
One operational detail that earned its keep: the adapter is pushed to the Hub periodically during training, with automatic resumption. On rented GPUs, over 25 hours, a cold restart is not a hypothetical.
4. The merge — and the trade I made with open eyes
At this point Légitus grounded and cited well, and did not reliably know when to stop talking. Apertus-Instruct stops cleanly every time. So the final model is a linear interpolation between the two:
The goal was to recover general conversational quality without washing out the legal specialisation.
It worked, and it cost something. Concision and clean turn termination improved markedly. Self-identification regressed at this ratio — the merged model is less reliable about saying who it is than either parent, because the identity set was 200 examples against a half-weighted general instruct model. That regression is documented in the model card rather than quietly tuned away, because the ratio is a dial someone else may want to set differently.
The evaluation at this stage is qualitative — side-by-side comparisons against the original Apertus-Instruct, which is what 25 hours buys you.
What it is, honestly
The model still hallucinates. I tested it against adversarial prompts and it is only fair to say there is real work left.
But on the legal topics it was actually trained on, answers became noticeably more accurate and more consistent, with genuine citation discipline instead of confident guessing — and it will tell you when its sources are not enough.
For a legal assistant, that last behaviour is worth more than a few points of accuracy. The dataset and the merged model are both open on Hugging Face, and the pipeline code is organised stage by stage.
Read more
The Reward Is a Service: What RLAIF Taught Me About Grading Reasoning
Training Apertus Nash on a single DGX Spark, the reward model lived in its own container and re-read its rubric on every request — because RL exploits whatever the reward forgets to punish, and that only shows up mid-run.
A Model That Writes Its Own Adapter — and the Bug That Almost Hid It
Memory-LoRA generates a repository-specific LoRA for Gemma-4-E2B in one forward pass. For weeks it looked like it worked. It was worse than no adapter at all, and the training loss never said so.
From Legal Text to Quantitative Insights: Agentic AI for Competition Authority Decisions
A walkthrough of my PhD project — scraping the French Competition Authority archive, ranking 18 LLMs on a manually annotated benchmark, and building a deep-research agent that turns the extracted data into econometric findings.