The Waterfall Router Pattern: Fast Models First, Smart Models Last
Don't send every prompt to your most expensive brain. Route like you've been routing compute for decades.
Every team discovers the same way: one glorious week where everything runs on the frontier model, then the invoice arrives and someone asks why hashtag suggestions cost four dollars each. The fix isn't a cheaper model — it's a router.
Here's the thing: you've run waterfall architectures forever — CDN edge, app servers, batch queues. LLM routing is the same pattern with new nouns. Cheap pass first, measure confidence, escalate on ambiguity or policy triggers. Users get speed on easy tasks; hard tasks still get brains.
Building the router
- Classify intent — tiny model or rules; don't overthink
- Score confidence — logprobs, self-check pass, or cheap verifier
- Escalate with budget — cap reasoning spend per user/session
- Log everything — you'll tune thresholds weekly at first
Routing is product behavior, not infra trivia. Users feel latency before they feel model names.
Confidence scoring without PhD infrastructure
You don't need a bespoke ML classifier on day one. Start simple: regex for intent buckets, token length heuristics, keyword triggers for escalation ("legal," "contract," "delete all"). Add a cheap self-check pass — ask the fast model "are you confident?" — before burning reasoning tokens. Log disagreements; that's your training data for v2.
Here's the thing: routing mistakes are product-visible. Escalate too much and you're slow and expensive. Escalate too little and you're wrong. Tune in production with feature flags, not in a lab with twenty prompts.
- Per-user budgets — cap premium tier spend per session
- Sticky routing — multi-turn threads stay on the tier that started them
- Timeout fallbacks — if tier three hangs, degrade gracefully with apology
- Shadow mode — run expensive tier in parallel, compare offline before switching
Waterfall routing is boring infrastructure — boring infrastructure is what keeps AI products alive past the demo.
Organizational buy-in for "good enough"
Stakeholders romanticize the biggest model. Your job is to show quality per dollar and p95 latency on real traffic. Dashboard tier distribution weekly — when leadership sees 80% served fast with acceptable scores, the religion shifts from "always GPT-max" to "right tool, right job."
What this means for you: instrument every request with tier, latency, cost, and user feedback thumbs. The router becomes a living product with OKRs, not a one-time if-statement.
| Metric | Why |
|---|---|
| Escalation rate | Detect classifier drift |
| Cost per successful task | CFO-friendly |
| p95 latency by tier | SLA enforcement |
| Downgrade after escalation fail | Graceful degradation |
Edge cases in routing
Multi-turn threads need sticky tier decisions — escalating mid-conversation confuses users who saw fast answers then hit a wall. Language detection, PII presence, and regulatory keywords should be first-class escalation triggers, not afterthoughts.
What this means for you: document override rules — when must tier three run regardless of confidence? Medical, legal, and safety categories often need hard escalation. Encode in config, not tribal prompt lore.
Cost attribution per tier
Finance will ask which product area burned tier-three budget. Tag requests with team, feature, and customer ID before routing. Without tags, optimization debates become religion.
What this means for you: monthly report — spend and volume by tier. Kill features that escalate 40% of traffic for 2% quality gain.
The router is product infrastructure — tune it weekly or pay monthly in tokens and latency.
What this means for you: celebrate when tier-one traffic grows — that means routing is working.
Cold start routing
Day one you have no logs. Start with rules from domain experts — legal always tier two, hashtags always tier one — then replace rules with data as traffic arrives. Perfect classifier day one is fiction; honest heuristics ship.
Publish router thresholds in the team wiki with last-changed date and owner. Mystery routing configs become mystery bills. Transparency lets product argue for tier-three budget with data instead of fear.
Start with cold-start rules, not a perfect classifier
You don't need logs to build tier one — you need domain experts willing to say "hashtags are always fast, legal is always escalated" and a wiki page with a last-changed date. What this means for you: the router that ships this week with honest heuristics beats the router that waits a quarter for a trained classifier. Replace rules with data as traffic accumulates, not before.
Here's the thing: the 80/15/5 split only holds if you keep tuning it. Watch escalation rate weekly, tag spend by team and feature, and kill the features that escalate 40% of traffic for 2% quality gain. A router nobody owns quietly drifts back into "always send it to the expensive model."
Celebrate when tier-one traffic grows — that means routing is working.
Your move: This week, profile your inference traffic by model cost. If the most expensive model handles more than 20% of requests, you need a router — not a better model. Implement a simple two-tier waterfall: run every prompt through a fast 8B model first, then escalate to 70B+ only when confidence is below 0.7. Measure cost per correct answer before and after.
Stay with us · dig next
How do you plan to implement the Waterfall Router in your organization?
Share your thoughts or experiences with implementing tiered routing strategies for LLMs.
No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.
