← ELI5 · Nestor G Pestelos Jr

Systems Design · AI

Model Router

Sending simple chores to small bikes and heavy cargo to big trucks.

1. Not every question needs a supercomputer

Asking for a synonym does not need a giant model that costs five cents per query.

Small Model Fast · Fractions of a cent Frontier Model Deep Logic · Heavy Cost
Why have different sizes?

Big models can solve complex math, but they are slow and expensive. Small models can format text or answer simple facts in a fraction of a second for almost nothing.

2. The traffic sorter

The router inspects the question and picks the cheapest model that will do the job well.

Question Router Check Difficulty Easy → Small Hard → Giant
How does the router know?

A quick mathematical check or tiny classifier reads the structure of the prompt. If it spots multi-step code, it sends it to the expert.

3. The escalation ladder

Try the fast model first; if the answer fails automated checks, call the expert.

1. Fast Model Answers task 2. Auto Check Did it pass? 3. Escalate Call Giant
Why not always use the giant model?

Because most requests succeed on step 1. You get 90% of your answers ten times faster for a tiny fraction of the bill.

Back to top

Longer version: Reference Entry on Model Router

Sources: Ong et al., RouteLLM (2024); Chen et al., FrugalGPT (2023).