Automating Medical Coding Part 2

There is a particularly dangerous phase in building an LLM system.

• It produces an answer.

• The answer looks correct.

• You celebrate.

• Then you try 100 examples

• You suddnely become miserable

This cycle can happen in eleven minutes or in eleven days .

Medical coding systems expose this cycle brutally because errors often look perfectly respectable: for example

Acute kidney injury
→ Other acute kidney failure

Looks interchangeable.

But if the documentation contains no subtype, the result may be overly specific.

Another example:

Obesity
→ Class 1 obesity

Both conditions look similar

But where did class 1 come from? Nowhere.

Here the model has invented specificity on its own.

Build evaluation set before polishing the demo

It makes perfect sense to treat synthetic episodes as first-class engineering assets in an LLM pipeline.

For evaluations Instead of one giant hand-written integeration test, It is wise to generate benchmark episodes in varying degree of difficulties such as:

Easy

explicitly documented conditions
sub conditions
ruled-out conditions

Medium

several diagnoses
chronic conditions
history
negation
symptoms

and eventually:

Hard

multiple related conditions
complications
acute-on-chronic disease
sequencing requirements
Similar ICD candidates

The goal isn't to pretend synthetic data equals hospital data. The goal is to create deterministic regression pressure.

A useful episode fixture then could be

DISCHARGE DIAGNOSES
 
Acute kidney injury secondary to dehydration.
Dehydration.
Chronic atrial fibrillation.
 
HOSPITAL COURSE
 
The acute kidney injury was attributed to dehydration. IV fluids were administered and renal function improved.
 
Chronic atrial fibrillation remained rate controlled.
 
There was no evidence of acute myocardial infarction.
 
PAST MEDICAL HISTORY
 
Previous ischemic stroke.

expected active output would be:

Acute kidney injury Dehydration Chronic atrial fibrillation
Dehydration
Chronic atrial fibrillation

but NOT:

Acute myocardial infarction
Previous ischemic stroke as current disease

Above fixture tests:

assertion
chronicity
causal relationship
active vs history
negation
candidate retrieval
specificity

which is arguably more useful than:

Input: "AKI"
Expected: N17.x

Measure the pipeline by stage

One overall accuracy number doesn't tell you much.

Suppose the expected code is:

N17.9

and the system returns N20.1 .

Anything along the way could have failed?

Extraction
    ↓
"AKI" wasn't detected
 
Retrieval
    ↓
N17.9 never entered Top 10
 
Compatibility
    ↓
N17.9 was incorrectly rejected
 
Reasoning
    ↓
correct candidate was present but another was chosen
 
Reportability
    ↓
correct code was selected then removed
 
Sequencing
    ↓
correct codes, wrong episode structure

All of these are completely different engineering failures.

So a correct evaluation function should say:

CANDIDATE_RETRIEVAL_ERROR

instead of:

wrong coding

Recall@K matters enormously

For retrieval, the first question is not:

Did the retriever rank the correct code first?

It's:

Did the correct code survive long enough for reasoning to see it?

So measure:

Recall@1
Recall@3
Recall@5
Recall@10
 
*If Recall@10 is poor, your LLM reasoner cannot save you.
It can't choose a candidate it never received.

Then measure false-positive behaviours

Medical coding needs metrics beyond conventional accuracy.

Some of the most useful regression metrics are:

Ruled-out active-code rate
Historical-condition leakage
Unsupported-specificity rate
Wrong-core-disease rate
Duplicate-code rate
Symptom overcoding rate

These often reveal more than overall F1.

Imagine accuracy goes from:

78% → 81%

but unsupported specificity doubles. That's not necessarily a progress.

Synthetic episodes are excellent for adversarial testing

You can deliberately create cases like:

No evidence of bowel obstruction.

and assert:

Bowel obstruction must never become active.

Or:

Chronic atrial fibrillation remained rate controlled.

and assert:

chronic != historical

Or:

Obesity.

and assert:

do not invent class 1 / class 2 / class 3

These aren't trying to mimic population prevalence. They're trying to attack your architecture.That's exactly what a regression dataset should do.

Generate cases, but don't let generation grade itself

There's an obvious trap with using one LLM model:

LLM generates episode
LLM generates expected code
same LLM evaluates model

Congratulations, it's an overrated echo chamber.

A better workflow is:

Synthetic episode generator (A different LLM)
      ↓
Expected labels
        ↓
Validate code against authoritative ICD database 
        ↓
Freeze fixture
        ↓
Run pipeline

For a serious benchmark, human coding review should eventually enter the loop.

But synthetic datasets are excellent for early architecture development because you can cheaply generate hundreds of controlled scenarios.

Efficiency also becomes measurable

Once you have hundreds of episodes, you can study:

Average retrieval latency
LLM calls per episode
Tokens per episode
Abstention frequency
Candidate-set size
Cache effectiveness

Now you can answer engineering questions like:

Do we actually need 20 candidates?

Maybe Recall@10 is nearly identical to Recall@20.

Does semantic retrieval help?

You can compare:

Index + BM25

against:

Index + BM25 + semantic

Is the expensive reasoning model worth it?

Run the same frozen benchmark against several local models. That's a real scientific experiment.

A benchmark changes the conversation

Before evaluation the conversation starts with:

“The pipeline seems pretty good.”

After benchmark it pivots to:

Retrieval Recall@10: 96%
Exact code accuracy: 81%
Unsupported specificity: 7%
Ruled-out leakage: 0.4%

Now you know which direction to take further

And more importantly, you know when a fix introduces regression.

In the final article, I'll move from individual diagnoses to the harder problem: coding the episode itself : Reportability, principal diagnosis and sequencing without turning the system into a collection of medical if statements with nested LLM calls .