← ELI5 · Nestor G Pestelos Jr

Systems Design · AI

LLM Gateway

The switchboard between your application and AI models.

1. One plug for every brain

Your app talks only to the gateway, and the gateway handles every different AI provider.

Your App Gateway (Switchboard) Provider A Provider B Local Model
Why not connect directly?

If you connect directly, changing an AI vendor means rewriting your code. The gateway acts as a translator so your app never cares who answers the question.

2. Automatic backup when models crash

When your main model is down or overloaded, the gateway instantly reroutes to a backup.

Gateway Primary: Overloaded ❌ Backup: Active ✅
How fast is the switch?

The gateway measures response times and error codes. If a request hits a rate limit or times out, it switches models in milliseconds without user intervention.

3. Protecting your budget

The gateway counts tokens and enforces spend limits before sending expensive requests.

User Request Budget Guard Check Limits Pass
What happens if budget runs out?

The gateway can reject excess requests or gracefully switch to a cheaper small language model instead of incurring expensive frontier bills.

Back to top

Longer version: Reference Entry on LLM Gateway

Sources: Mitra, System Design for the LLM Era (2026); CNCF AI Gateway Working Group.