MiniCPM5 German Address Repair: Teaching a 1B Model When to Shut Up

🔧 MiniCPM5 German Address Repair


What it is	fine-tune of MiniCPM5-1B that repairs messy German address records into validated JSON
Stack	MiniCPM5-1B with LoRA SFT on a Colab T4, GGUF through llama.cpp, rules floor plus field scorer
Status	v2 in progress, clean retrain and frozen eval still open
Code	local-slm-de-address-repair

I wanted a small local model that fixes dirty address data and says so when it cannot. The SFT version beats my rules floor on recall by about 3x and holds review precision near 1.0. Those numbers come with a warning though. The test split leaked, so I am reporting them as contaminated and retraining next.

What I learned

UUID-disjoint splits can still leak badly. My train, val, and test IDs never overlapped, yet 1,035 of 2,000 test rows duplicated training pairs under different IDs. Overlap rows scored F1 0.47 while truly new rows scored 0.27. I now group every split by the full dirty-plus-gold fingerprint.
A good refusal lowers your recall score. When the postcode field arrives empty, the right move is to leave it empty and flag it for review. My scorer still counts that as a miss. SFT logged 3,150 such correct abstentions, so read its recall of 0.244 as share of fields fixed without guessing.
The rules floor is annoyingly strong on precision. Deterministic rules hit precision 0.96 with recall 0.07. SFT moved that to precision 0.82 with recall 0.24. I keep coming back to this trade. You pay real precision to buy recall, and you should name the price out loud.
Almost all damage lives in the name field. Of 87 SFT damage events, 55 hit name. Clean Pfarrhaus became Pfarrhaus Kalkhorst because the locality leaked in. Same story for Stadtkirche turning into Stadtkirche Delmenhorst. Names carry the least mechanical evidence, so the model over-edits exactly there.
Zero empty-field fills is not zero inventions. Base MiniCPM5 filled 898 empty fields unasked. SFT filled zero, which felt like victory until I counted 248 unsupported additions to non-empty fields. That Pfarrhaus case counts here. I changed the wording in the paper and the metric.
Review coverage needs two numbers or it lies. SFT flags 0.21 of all field slots, which sounds calm. But 1,697 of 2,000 records carry at least one flag, so 0.85 of records still reach a human. The second number runs your staffing plan. Report both.
Provenance turned out to be half the project. The T4 needs fp16 since it lacks bf16 support. Winner checkpoint-939 scored 0.769 on honesty metrics. And my so-called v2 GGUF hashed byte-identical to v1, which stung. Without pinned configs, hashes, and exact merge commands, none of that is checkable. I committed all of it.

How it works

A dirty record goes in with six fields, prompt v2 asks for normalized JSON plus a change list plus needs_review flags, and the tuned 1B model answers through a local llama.cpp server. A scorer then grades repair precision and recall, damage to clean fields, review precision, and schema validity on the same frozen 2,000 records for rules, base, and SFT. All work uses public AddressTable records, not private customer data.

# src/addr_repair/io.py, simplified: one fingerprint, one split, no leakage
groups = {}
for record in rows:
    groups.setdefault(pair_fingerprint(record), []).append(record["id"])
# pair_group_split then assigns each group wholly to train, val, or test

Steal this if: you fine-tune small models on paired data and want an eval that punishes guessing instead of rewarding it.

MiniCPM5 German Address Repair: Teaching a 1B Model When to Shut Up