ML systems · BANKING77 · 2026
A small model.
A selective second opinion.
Keep confident predictions local. Ask the LLM when the specialist is unsure.
Three policies. One test set.
3,080 held-out casesA shared 0 to 100% scale. The full LLM is more accurate; the hybrid uses fewer calls.
Routed hybrid: 2,713 correct. 268 LLM requests. +2.50 accuracy points over the local specialist.
See exact values
| Policy | Accuracy | Correct | LLM calls |
|---|---|---|---|
| Local only | 85.58% | 2,636 | 0 |
| Routed hybrid | 88.08% | 2,713 | 268 |
| LLM only | 92.18% | 2,839 | 3,080 |
The route a request takes
The API charge tradeoff
Offline routing replay91.1% lower known LLM charges.
With 4.09 percentage points less accuracy than the full LLM. This is an offline policy comparison, not measured production savings.
Method, accounting & limitations
One seed, one training pool, one public benchmark. The specialist used 1,540 training labels (20 per intent), plus 770 reused validation/development labels. Thresholds were frozen before test evaluation.
The study queried the LLM on all 3,080 cases. Routing counts describe an offline replay, not the fraction of calls purchased. Three attempts have unknown usage; invoice spend is unreconciled. Specialist hosting, retrieval, and production serving costs are excluded.
These results do not establish production reliability or generalization to other intent taxonomies.