inferoute

reference

Documentation

This page explains what inferoute does, the ideas behind it, and every knob it exposes. No prior context needed. The source and issue tracker live on GitHub.

What inferoute is

inferoute is a gateway that sits in front of one or more large language model servers and makes them look like a single, reliable endpoint. Your application talks to inferoute using the ordinary OpenAI chat-completions API. inferoute decides which server actually handles each request, retries elsewhere when one fails, enforces per-caller rate limits, and can answer repeat questions from a cache without touching a model at all.

It is one small Go binary (inferouted) with one JSON config file. It stores nothing on disk, holds no database of its own, and adds a few milliseconds to a request. You run it next to your inference servers and point your clients at it instead of at them.

The problem it solves

Running a language model is now the easy part: vLLM, SGLang, llama.cpp, and Ollama all do it well. The awkward part is everything around the model once more than one team, or more than one GPU, is involved:

inferoute is that layer, and only that layer. It does not run models, fine-tune them, or store your conversations.

What happens to a request

Every call to POST /v1/chat/completions goes through the same sequence. This is the actual order of operations in the code, not a simplification.

  1. 01
    Parse and identify

    inferoute reads the JSON body and pulls out the "model" field. A body that is not valid JSON, or has no "model", is rejected with HTTP 400 before anything else runs. If the name matches a model alias, it is swapped for the real name here.

  2. 02
    API-key check

    If you configured an api_keys allowlist, the "Authorization: Bearer <key>" header must carry one of those keys. Otherwise the request gets HTTP 401. If the list is empty, this step does nothing.

  3. 03
    Rate limit

    If rate limiting is on, the caller's token bucket (keyed on their API key, or their IP address if there is no key) must have a token available. If not, the caller gets HTTP 429 and the request stops here.

  4. 04
    Cache lookup

    If caching is on, the prompt is turned into a vector and looked up in NuclaDB. If a stored prompt is within max_distance, its saved response is returned immediately, a streamed answer is replayed byte for byte, and no backend is contacted. The response carries X-Inferoute-Cache: hit.

  5. 05
    Pick a backend

    On a cache miss, inferoute takes the healthy backends that serve this model and picks one using the configured "load_balancing" strategy (round_robin by default). If none are healthy or none serve the model, the caller gets HTTP 503.

  6. 06
    Forward and, if needed, fail over

    The request is forwarded to the chosen backend. On a connection error or a 5xx, that backend is marked unhealthy and the request is retried on the next one, up to three attempts total. If all three fail, the caller gets HTTP 502.

  7. 07
    Stream back and record

    The backend's response is written to the caller as it arrives, never buffered. The response gets X-Inferoute-Backend naming the server that handled it. A successful, non-streamed answer to a cacheable request is stored in the cache in the background. Counters and the latency histogram are updated.

Concepts and vocabulary

Terms this documentation uses, defined once.

Backend
One inference server inferoute forwards requests to: an Ollama process, a vLLM server, a hosted provider like OpenAI or Groq, anything that speaks the OpenAI chat-completions API. You list your backends in the config file.
Model
The name a caller puts in the "model" field of the request body, for example "llama3". Each backend declares which models it can serve. inferoute uses that field to decide where a request can go.
Model alias
A rename. If your callers expect "gpt-4" but your servers actually run "llama3:70b", an alias ({"gpt-4": "llama3:70b"}) lets the request come in as "gpt-4" and get routed as "llama3:70b". Nothing about the caller changes.
Routing
Choosing which backend handles a given request. inferoute looks at the model, filters to the backends that serve it and are currently healthy, and picks one.
Load balancing
The rule for picking among a model's healthy backends, set by "load_balancing" in the config. round_robin (default) spreads requests evenly, one after another. least_pending sends each request to the backend with the fewest in-flight right now. weighted picks at random, biased by each backend's "weight".
Failover
If the chosen backend refuses the connection or returns a server error, inferoute marks it unhealthy and immediately retries the request on the next healthy backend, up to three attempts, before giving up.
Health check
A background loop that sends a plain GET to each backend on a path and interval you configure. A backend that stops responding is taken out of rotation until it recovers, so failover usually happens before a caller ever sees an error.
Rate limit (token bucket)
A cap on how fast one caller can send requests. Each API key gets a bucket that refills at a steady rate (requests_per_second) and holds a small reserve (burst). A request takes one token; when the bucket is empty the caller gets HTTP 429 until it refills.
Semantic cache
An optional response cache keyed on the meaning of the prompt rather than an exact string match. Two prompts that are close enough in an embedding vector space are treated as the same question, so the second one is answered instantly from storage instead of hitting a backend.
Embedding distance
A number measuring how far apart two prompts are in meaning. 0 is identical, larger is less similar. The cache counts a stored prompt as a match when its distance to the new prompt is below max_distance. Smaller max_distance means stricter matching.

Getting started

The goal of this walk-through is a running gateway in front of two model servers, with one request going through it.

1. Have something to route to

You need at least one backend. Two makes load balancing and failover visible. Two local Ollama instances serving the same model:

OLLAMA_HOST=127.0.0.1:11434 ollama serve &
OLLAMA_HOST=127.0.0.1:11435 ollama serve &
ollama pull llama3

2. Install and run inferoute

One command if you have Go, plus the sample config, which already points at those two ports:

curl -fsSL https://raw.githubusercontent.com/Rakshit-gen/inferoute/main/install.sh | sh
curl -O https://raw.githubusercontent.com/Rakshit-gen/inferoute/main/config.example.json
inferouted -config config.example.json

Or build from source with go build -o bin/inferouted ./cmd/inferouted. The binary takes exactly one flag, -config, and defaults to ./config.json.

No GPUs to hand? ./scripts/local-stack.sh in the repo builds and runs the gateway in front of two mock backends that echo prompts back (and can stream), so you can try routing, failover, and this dashboard with only Go installed.

3. Send a request

Same shape as a call to OpenAI or Ollama. The only change your application ever makes is the base URL.

curl localhost:8081/v1/chat/completions \
  -d '{"model":"llama3","messages":[{"role":"user","content":"hi"}]}'

You get back exactly what the backend returned. inferoute only chose which of the two servers handled it, which you can confirm from the X-Inferoute-Backend response header. Kill one ollama serve and send the request again: it fails over to the survivor instead of erroring.

4. Watch it here

Sign in to this dashboard, open Connections, and add your gateway's URL. The dashboard and the playground then read from it live. See Using this dashboard.

Configuration reference

Everything is one JSON file passed with -config. Every field has a working default except backends, which is required. A complete example:

{
  "listen_addr": ":8081",
  "health_check_path": "/",
  "health_check_interval": "10s",
  "load_balancing": "round_robin",
  "backends": [
    { "name": "ollama-1", "url": "http://localhost:11434", "models": ["llama3"], "weight": 1 },
    { "name": "ollama-2", "url": "http://localhost:11435", "models": ["llama3"], "weight": 1 }
  ],
  "rate_limit": { "enabled": false, "requests_per_second": 5, "burst": 10, "redis_addr": "" },
  "cache": {
    "enabled": false,
    "nucladb_addr": "http://localhost:8080",
    "embedding_backend_addr": "http://localhost:11434",
    "embedding_model": "nomic-embed-text",
    "max_distance": 0.05,
    "tenant_id": "inferoute-cache"
  },
  "model_aliases": {},
  "cors_origins": ["*"],
  "api_keys": []
}
listen_addr
The address inferoute itself listens on. Default ":8081".
health_check_path, health_check_interval
The path to GET on every backend to confirm it is alive, and how often to do it. Defaults: "/" every 10 seconds.
backends (required)
The list of servers to route to. Each entry is { name, url, models, path_prefix?, api_key?, weight? }. "models" is the list of model names that server can handle; two backends listing the same model get load-balanced between. "path_prefix" is prepended to the forwarded path for servers that mount their API under a prefix (Groq serves its OpenAI-compatible API under "/openai", so path_prefix ":/openai" turns "/v1/chat/completions" into "/openai/v1/chat/completions"). "api_key" is sent as that backend's Authorization header, overriding whatever the caller sent, so you can put a paid provider behind inferoute without callers knowing the key. "weight" (default 1) biases the "weighted" load-balancing strategy toward bigger boxes.
load_balancing
How to pick among a model's healthy backends: "round_robin" (default, even rotation), "least_pending" (fewest in-flight requests, best when request durations vary), or "weighted" (random, biased by each backend's "weight").
rate_limit.enabled
Turn per-key rate limiting on or off. Off by default.
rate_limit.requests_per_second, .burst
The steady refill rate and the reserve size. A caller can spend up to "burst" requests quickly, then is held to "requests_per_second" after that.
rate_limit.redis_addr
Leave empty and each inferoute process keeps its own count (fine for a single instance). Set it to a Redis "host:port" and several inferoute instances behind a load balancer share one limit.
cache.enabled
Turn semantic response caching on or off. Off by default. Requires a running NuclaDB instance.
cache.nucladb_addr
Where that NuclaDB instance is reachable.
cache.embedding_backend_addr, .embedding_model
Which Ollama-compatible server and model turn a prompt into a vector for cache lookups.
cache.max_distance
How close a stored prompt must be to count as a hit. This is a distance, not a similarity score: 0 means identical, and a smaller number is a stricter match. See the caching section below, this one is easy to set backwards.
cache.tenant_id
The NuclaDB tenant the cache vectors are stored under.
model_aliases
Maps a requested model name onto one your backends actually serve, for example {"gpt-4": "llama3"}. Empty by default (no aliasing).
cors_origins
Browser origins allowed to call the HTTP API directly. ["*"] allows any origin and is the default. Non-browser clients send no Origin header and are never affected.
api_keys
An allowlist of bearer tokens for POST /v1/chat/completions. When the list is non-empty, a request without a listed "Authorization: Bearer <key>" gets HTTP 401. Empty (the default) leaves the proxy open. Reloaded on SIGHUP. The read-only endpoints are not gated by this.

HTTP API reference

Six endpoints. One does the proxying, the rest are read-only introspection used by tools and by this dashboard.

POST /v1/chat/completions
The proxy itself. Reads "model" from the JSON body, resolves aliases, checks the cache when enabled, picks a healthy backend, and streams the response straight back. Returns 400 if the body is not JSON with a "model" field, 429 if rate limited, 401 if an API key is required and missing, 503 if no backend serves that model, 502 if every attempt failed.
GET /v1/models
An OpenAI-compatible model list: every model a backend serves plus every alias clients can ask for. SDKs call this to fill a model picker. Not gated by api_keys.
GET /v1/backends
A JSON array of every backend with its name, url, models, and current health.
GET /v1/config
The active model aliases, the health-check interval, and whether rate limiting, caching, and API-key auth are on. This is what the dashboard reads to describe your gateway.
GET /metrics
Prometheus text format: request counts and latency by model and backend, and cache hit / miss / error counts.
GET /healthz
A liveness probe for load balancers and orchestrators. Always 200 while the process is running.
GET /docs
inferoute's own documentation page, served straight from the binary.

Response headers

Every proxied response tells you how it was handled. The playground sends a real request and shows both headers along with the path it took.

X-Inferoute-Backend
The name of the backend that served the response, or "cache" when the answer came from the semantic cache.
X-Inferoute-Cache
Set to "hit" when the response was served from the cache. Absent otherwise.

Changing config without downtime

Sending the process a SIGHUP reloads the backends, load_balancing, model_aliases, and api_keys sections from the same file, with no restart and no dropped in-flight requests. This is how you add or drain a backend, or switch strategy, in production.

kill -HUP $(pgrep inferouted)

Changes to rate_limit and cache are not hot-reloaded. Those take a full restart.

Rate limiting in depth

Rate limiting protects your capacity from a single runaway caller. It is off until you set rate_limit.enabled to true.

Each caller gets a token bucket. The bucket refills at requests_per_second and can hold up to burst tokens in reserve. Every request spends one token. A caller who has been quiet can spend their whole reserve in a short spike, then is held to the steady rate. A caller with an empty bucket gets HTTP 429 and should retry after a moment.

The bucket is keyed on the caller's API key, taken from the Authorization: Bearer header. Requests with no key share a bucket keyed on the client IP address instead.

By default the counting happens in memory, inside the one inferoute process. If you run several inferoute instances behind a load balancer and want them to enforce one shared limit, set rate_limit.redis_addr to a Redis address and they will all count against the same buckets.

Semantic caching in depth

The cache lets inferoute answer a question it has effectively seen before without running inference again. It is off until you set cache.enabled to true, and it needs a running NuclaDB instance to store vectors in.

How a lookup works:

  1. Embed. The prompt text is sent to the embedding model at cache.embedding_backend_addr and comes back as a vector.
  2. Search. NuclaDB returns the closest stored prompt and its distance from this one.
  3. Decide. If that distance is below cache.max_distance, it is a hit: the stored response is returned as-is, and a streamed response is replayed as the exact bytes first captured.
  4. Store. On a miss, once the backend has answered, the new prompt and response are written to the cache in the background.
two things that fail silently if set wrong

max_distance is a distance, not a similarity. NuclaDB's score is 1 - cosine_similarity, so 0 means identical and a bigger number means further apart. A smaller max_distance is a stricter cache. This is the opposite of what a "similarity threshold" would do, and it was the first version's bug.

NuclaDB's vector dimension must match your embedding model. NuclaDB is started with a fixed vector size for the whole database. Point cache.embedding_model at a model whose output is exactly that length, or every insert is rejected.

Measured effect, against the real binary with a backend held at 700ms to stand in for inference time: a miss took about 705ms, a hit took about 0.8ms. That is roughly 880 times faster on a hit, confirmed from the X-Inferoute-Cache: hit header, not from timing alone.

Metrics and observability

GET /metrics serves Prometheus text. Point Prometheus or Grafana at it. The collectors:

inferoute_requests_total{model, backend, status}
A counter of every proxied request. "backend" is the server name, or "cache" on a cache hit, or "none" when no backend could be reached. "status" is the HTTP status returned to the caller.
inferoute_request_duration_seconds{model}
A latency histogram per model, measured from receiving the request to finishing the response. This dashboard reads percentiles off it.
inferoute_cache_lookups_total{outcome}
A counter of cache lookups by outcome: "hit", "miss", or "error". Hit rate is hits over hits plus misses.

The dashboard renders all of this live: throughput, latency, per-backend share, and cache effectiveness.

Deploying to production

inferoute is a stateless HTTP proxy, so it deploys like any other small web service, with no database or persistent disk of its own. A container image is provided (Dockerfile at the repo root).

Using this dashboard

This dashboard is a separate application from the gateway. The gateway serves one tenant; the dashboard lets several people each watch their own gateways without seeing each other's.

  1. Sign in. Use the Sign up button in the nav. Authentication is handled by Clerk.
  2. Add a connection. On the Connections page, enter your gateway's base URL and, if it has an api_keys allowlist, a key. The key is stored server-side and never sent to the browser.
  3. Watch it. The Dashboard and Playground read from your active connection. You can register several gateways and switch between them.

Every request the dashboard makes to a gateway goes through this app's own server, which attaches your connection and its key. One account's requests can never be routed to another account's gateway. Because the browser talks only to this app and never to a gateway directly, the gateway's cors_origins does not need to include the dashboard.

Current limitations

Things inferoute does not do yet, so you can plan around them: