Building LLM Router

Most LLM applications begin with an llm call

response = client.chat.completions.create(
    model="premium-model",
    messages=messages,
)

This is a reasonable approach for prototyping, but does not hold up in production.

Without intelligent routing, the same expensive model ends up handling everything : from diagnosing a production incident to glorified string manipulation such as

Convert this sentence to lowercase

Unsurprisingly, this kind of architectural misadventure irritates already-overstressed CFO, and is how the team lunch budget mysteriously disappears

This is the problem RouteWise is determined to solve

RouteWise is an OpenAI-compatible LLM gateway that sits between applications and multiple LLM providers.

It examines each request, identifies what it needs, filters unsuitable models, applies operational policies, and routes to an appropriate provider.

It also handles budgets, retries, fallbacks, circuit breakers, usage tracking, and routing visibility through an administrative dashboard.

Repository:

https://github.com/uncle-voh-max/routewise

Why model routing matters

Every LLM request does not require the default capability.

Consider these prompts:

Extract the customer ID from this JSON.
 
Summarise this support request.
 
Analyse a production incident and propose likely causes.
 
Inspect this image and return structured findings.

The first request needs reliable structured output.

The second request could be handled by a small, inexpensive model.

The third needs stronger reasoning and possibly a larger context window

The fourth requires vision support.

Sending every request to the strongest available model may produce good answers, but wastes money and capacity.

Similarly Sending everything to the cheapest model reduces cost but undermines quality

A good tradeoff is:

Selecting the least expensive model that serves the query reliably.

This is the essence of model routing.

cheap incorrect answers by app are not cost savings. They are incidents wearing discount codes.

RouteWise in the request path

With route wise applications call LLMs through an OpenAI-compatible gateway rather than directly integrating with Ollama, Azure, OpenAI, and other providers.

The query path becomes:

The important detail worth highlighting here is that filtering of llm models precedes actual scoring.

RouteWise first filter models that can perform the task. After that it compare models by cost, quality, latency, and faithfulness.

This avoids a common routing mistake:- allowing an unsuitable model to remain a contender merely because it is cheap.

Runtime conditions narrow the filter further

A model may be eligible for a request but could be currently unavailable. Such models are automatically excluded from the filter

RouteWise can exclude a candidate when it's

1. circuit breaker is open

2. cost budget has exhausted

3. predicted latency exceeds the request limit

4. quality is below the required threshold

5. Is explicitly excluded by the caller

Applications can express these constraints during instantiation of gateway

{
  "gateway": {
    "maximum_estimated_cost": 0.02,
    "maximum_latency_ms": 2000,
    "minimum_quality_score": 0.8,
    "minimum_faithfulness_score": 0.75
  }
}

The application is effectively saying

Give me this level of quality,
within this latency,
and below this cost.

RouteWise owns the decision about which models satisfies the given contract.

This is crispier than spreading specific model concerns throughout the application.

LLM landscape is highly uncertain. Model names could change, Pricing may evolve, Providers might deprecate older models and whatnot.

Applications should not request a release every time the model catalogue has a mood swing.

Routing policies precisely express intent

Once unsuitable models are filtered out RouteWise applies the selected policies and scores the remaining candidates .

The built-in policies represent common trade-offs

Balanced

{
  "gateway": {
    "routing_policy": "balanced"
  }
}

This is the default mode. It considers quality, cost, latency, and faithfulness in same manner, forbidding any one dimension to dominate.

It works well when the caller does not have any preference.

Cost optimized

{
  "gateway": {
    "routing_policy": "cost_optimized"
  }
}

Useful for:

Formatting
Routine classification
Internal automation
Development traffic
High-volume, low-risk requests

Cost optimisation does not mean blindly selecting the cheapest model. Candidates must still satisfy capability and quality constraints.

Quality preferred

{
  "gateway": {
    "routing_policy": "quality_preferred"
  }
}

Useful for:

Complex reasoning
Code review
Important customer-facing reports
Incident investigation
High-value decisions

Latency optimized

{
  "gateway": {
    "routing_policy": "latency_optimized"
  }
}

Useful for:

Interactive assistants
Autocomplete
Real-time support
Interfaces where responsiveness matters more than marginal quality gains.

Users rarely describe a slow answer as “high quality.” They normally describe the application as broken.

Configurable routing registry for all models

RouteWise stores model capabilities and operational characteristics in a registry.

A model registry looks like:

models:
  - id: premium-reasoning-model
    provider: openai_compat
    provider_model: provider-model-name
    tier: 1
    enabled: true
 
    capabilities:
      tool_calling: true
      structured_output: true
      vision: true
      streaming: true
 
    context_window_tokens: 128000
    max_output_tokens: 16384
 
    input_cost_per_million_tokens: 2.50
    output_cost_per_million_tokens: 10.00
 
    quality_score: 0.97
    faithfulness_score: 0.97
    latency_score: 0.75

This allows application to separate business code from model specific APIs

Complexity should influence route

A simple LLm task and a difficult task should not receive identical routing.

Formatting and rewriting
    → small local model
 
Structured extraction
    → inexpensive model with good schema support
 
Routine summarisation
    → mid-tier model
 
Complex reasoning
    → high-quality model
 
Long, expensive conversation
    → multi-model cascade

This lets local models handle predictable workloads while higher tier models remain available for difficult tasks.

The idea is not new. Distributed systems have always placed work on resources appropriate for the workload.

LLMs simply add probabilistic behaviour and more entertaining invoices.

Cascades can split expensive work

Some requests are better handled via several stages.

For example:

Compact a long conversation
        ↓
Extract important facts
        ↓
Draft a response
        ↓
Finalise with a stronger model

RouteWise supports cascading where each stage can use a different tier.

complex_reasoning:
  steps:
    - name: compact
      task: conversation_compaction
      model_selector:
        maximum_tier: 4
 
    - name: extract
      task: fact_extraction
      model_selector:
        maximum_tier: 3
 
    - name: finalise
      task: final_response
      model_selector:
        maximum_tier: 1
 
  min_complexity: 3
  max_direct_cost_multiplier: 1.5

A smaller model can reduce or structure the input before a stronger model performs final reasoning.

This can lower cost, especially when the original context is large.

But cascading needs measurements or evals

Four inexpensive calls are not automatically cheaper or faster than one capable call.

Retries and fallbacks are part of routing

Providers fail. They time out, return rate limits, reject payloads, or become temporarily unhealthy.

RouteWise can retry retryable failures and automatically moves to the next eligible model.

Circuit breakers prevent repeated traffic being sent to a provider that is already failing.

This is an important reason to centralise routing.

Without a gateway, every application team eventually invents its own retry policy:

Team A retries everything
 
Team B retries nothing
 
Team C retries authentication failures ten times
 
Team D falls back directly to the most expensive model

Centralisation turns improvisation into an explicit operational policy

The dashboard makes routing visible

Dynamic model selection creates an obvious question

Why did this request use that model?

RouteWise includes an administrative dashboard covering usage, cost, latency, models, health, circuit breakers, decisions, and audit events.

Overview

The overview helps answer:

• How many requests are succeeding?

• Which providers receive traffic?

• How much are we spending?

• Are fallbacks increasing?

• Has latency changed?

• Are circuit breakers open?

A gateway that reduces costs while hiding its behaviour eventually becomes an expensive mystery.

Model management

Models can be enabled or disabled at runtime.

This is useful during:

• provider incidents

• model deprecations

• controlled rollouts

• quality regressions

• unexpected pricing changes

Every automated platform eventually needs a professionally labelled “absolutely not” switch.

Routing decisions

The decisions view records information such as

• selected model

• provider

• active policy

• latency

• estimated cost

• success or failure

• fallback usage

When an answer is unexpectedly poor, engineers can identify which model handled the request and which policy influenced the decision.

A router without decision history is just a secretive oad balancer.

Evaluating routing quality

A router should not be evaluated merely by checking that requests return HTTP 200.

Compare metrics along several strategies:

Always use the strongest model
 
Always use the cheapest model
 
Balanced routing
 
Cost-optimised routing
 
Quality-preferred routing

Use a representative request dataset (golden dataset) and record:

• answer quality

• selected model

• token usage

• cost

• latency

• failure rate

• fallback count.

The most useful measurements are:

Quality retention

How close is routed quality to the strongest model?

Cost reduction

How much spend is saved without unacceptable quality loss?

Selection accuracy

Did the router forwarded tasks to suitable models?

Recovery effectiveness

Did retries and fallbacks recover successfully?

Routing stability

Does a small score adjustment unexpectedly move most traffic?

The North Star is not just cost reduction

A gateway that cuts model spend by 70% while doubling support escalations has merely moved the expense to another department.

Running RouteWise locally

RouteWise can be started with Docker Compose on local machine

The local stack uses PostgreSQL and Redis, Ollama for local inference.

Clone and configure

git clone https://github.com/uncle-voh-max/routewise.git
cd routewise
 
cp .env.example .env

Review the environment file and add credentials for any hosted providers you intend to use.

Start with Docker Compose

docker compose up -d

Apply database migrations if required:

uv run alembic upgrade head

Check the service:

curl http://localhost:8000/health

Run the API directly

Install dependencies:

uv sync

Apply migrations:

uv run alembic upgrade head

Start the server:

ROUTEWISE_ENV=local \
uv run uvicorn routewise.main:app --reload

A local Ollama installation commonly runs at:

http://localhost:11434

Redis and PostgreSQL can be configured through REDIS_URL and DATABASE_URL.

Send the first routed request

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Explain circuit breakers in two paragraphs."
      }
    ],
    "gateway": {
      "routing_policy": "cost_optimized",
      "explain_routing": true
    }
  }'

With routing explanation enabled, the response includes metadata describing the model that was selected and rational behind selection.

Open the dashboard at:

http://localhost:8000/admin/dashboard

* * Use the configured admin API key, and change any development default before exposing the dashboard outside the local machine.

The senior-engineering perspective

RouteWise establishes a designated demarcation.

Applications express:

Required capabilities
Quality expectations
Latency tolerance
Cost limits
Routing policy
Tenant context

The gateway owns:

Provider integrations
Model inventory
Pricing
Health
Budgets
Retries
Fallbacks
Circuit breakers
Usage tracking
Routing decisions

This allows application teams to change models and providers without rewriting every integration.

It also creates one place to enforce organisational cost and reliability controls.

But centralisation carries the weight of bigger responsibilities

Model scores must be calibrated. Policies need owners. Dashboard access must be secured. Routing decisions must be evaluated continuously.

Intelligent routing should not make model choice mysterious. It should make model choice deliberate.

The best model is not always the largest or cheapest.

It is the model that satisfies the request, remains reliable under production load, and does not make finance reaching out to your manager.