Why Model Routing Is Broken in Production

Everybody loves the idea of a smart model router. You take a pile of incoming prompts, assess their complexity, and dispatch them to the cheapest possible LLM that can get the job done. It sounds clean. It looks great on architecture diagrams. But once you actually deploy one into a real agentic workflow, that neat little classification problem violently morphs into a chaotic systems optimization nightmare.

Take pricing, for instance. You look at the sticker rates, do some quick math, and assume the model with the lowest per-token cost wins. I ran into a scenario where a supposedly more expensive model ended up cutting total execution costs in half. Why? Caching. Agentic loops drag massive context windows back and forth across every single step. If your routing logic relies purely on static pricing sheets instead of factoring in live cache hit rates, you are essentially flying blind.

There, then is the delusion of task difficulty. People love to build routers that scan a prompt, guess how hard it is, and route accordingly. Good luck with that. A seemingly trivial request can suddenly spiral into a multi-step labyrinth of tool calls, API retrievals, and endless error-handling loops. You rarely know the true complexity of a workload until the execution is already halfway done.

And don't even get me started on latency. We treat tokens-per-second benchmarks like gospel, as if raw model generation speed is the only thing that dictates user experience. It ignores routing overhead, queue times, cold starts, and hardware constraints. Building a reliable router isn't just about picking the right brain for the job – it's about orchestrating an entire moving ecosystem, when you stack all of this up.

If you are building your own routing layer today, throw away the textbook assumptions. Real-world constraints will break them by Tuesday afternoon.