Model routing

AI model routing you can predict

With LLMFlux your application decides which model answers. The gateway decides which provider serves that model, and moves to another provider when one fails — so you get control over model choice and resilience against provider outages.

Two layers of routing

  • Model selection — your code

    The model field selects the model. Use any id from GET /v1/models, or an alias an administrator has published. Unknown ids return 404 invalid_model instead of silently falling back.

  • Provider selection — the gateway

    Providers serving a model are grouped by priority tier. Within a tier, load is shared by weighted random choice. Failures move the request to the next candidate.

Keeping model choice in your code makes results reproducible: the same request with the same model id is answered by the same model, and costs follow the published price for that model.

How provider failover works

  1. The request arrives with a model id the gateway recognises.
  2. A provider is chosen from the highest-priority tier that serves the model.
  3. On a retryable failure — network error, HTTP 429 or 5xx — the next candidate provider is tried.
  4. The failing provider cools down for 60 seconds so later requests skip it.
  5. Non-retryable errors (for example an invalid request) are returned immediately, without retrying elsewhere.

Streaming requests fail over before the first byte is sent to you; once a stream has started it continues from the same provider.

Routing patterns for applications

  • Route by task

    Use a fast model for classification, extraction and short replies; a larger model for planning, coding and long-form reasoning.

  • Route by budget

    Pick models by their per-1M-token prices from the catalogue, and check actual spend per request in the logs.

  • Fallback in your code

    If a model returns an error after the gateway's own failover, retry with a second model id you have tested.

  • A/B comparisons

    Send a sample of traffic to a candidate model and compare quality and cost before switching.

Python — same request across modelspython
# Same request, different model: only the "model" field changes.
for model in ["claude-sonnet-5-5", "deepseek-v4-flash", "grok-4.7"]:
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Reply with your model name."}],
    )
    print(model, "→", resp.choices[0].message.content)
Find valid model idsbash
curl https://api.llmflux.dev/v1/models \
  -H "Authorization: Bearer $LLMFLUX_API_KEY"

Prices and context windows for every id are in the model catalogue. For the gateway overview, see AI API gateway.

Documentation

Frequently asked questions

Does LLMFlux choose a model for me?
No. You name the model in each request and that is the model you get. The special id "auto" maps to one model configured in the catalogue; it is a fixed mapping, not a per-request prediction.
What is provider failover?
Some models can be served by more than one upstream provider. If the provider handling your request fails with a network error, a 429 rate limit or a 5xx error, the gateway sends the request to the next provider for the same model.
Will failover change the model I asked for?
No. Failover only changes which provider serves the request, never the model id you requested.
How long is a failing provider skipped?
A provider that fails is taken out of rotation for a 60-second cooldown, then tried again.
What do I receive if every provider fails?
A documented gateway error such as 502 upstream_error or 504 gateway_timeout. Provider error text is not passed through. Retry with backoff, or try another model.
Can I route between models in my own code?
Yes. Because every model shares the same request format, routing by task, cost or user plan is a matter of choosing a model id before you send the request.

Start with one key

Create an account, generate an API key, and send your first request in a few minutes.