OpenAI-compatible inference gateway
Route, cache, and rate-limit
your inference backends
inferoute sits in front of your Ollama, vLLM, or hosted LLM endpoints and load-balances across them, health-checks them, fails over on error, and semantically caches responses. One binary, one JSON config.
Drive the load, toggle cache and the rate limit, knock backends offline. The pipeline responds the way the real gateway would.
what happens to a request
- 01
Ingress
A caller sends an OpenAI-shaped POST /v1/chat/completions. inferoute reads the model field from the body. Nothing else about the request needs to change.
- 02
Rate limit
A token bucket keyed on the API key (or client IP) decides whether this request gets through. In-process by default; back it with Redis to share one budget across a fleet.
- 03
Semantic cache
If caching is on, the prompt is embedded and looked up in NuclaDB. A close-enough match is served straight from cache (a streamed response is replayed chunk for chunk).
- 04
Route + failover
On a miss, inferoute load-balances across the healthy backends registered for that model (round-robin, least-pending, or weighted). A connection error or 5xx moves to the next one; health checks keep the pool current.
- 05
Stream back
The upstream response is flushed to the caller as it arrives, never buffered, with X-Inferoute-Backend and X-Inferoute-Cache headers describing how it was handled.
what it does
Routing + failover
Round-robin across healthy backends per model, automatic retry on the next one on error.
Streaming passthrough
SSE responses are forwarded chunk by chunk as the backend produces them.
Per-key rate limiting
Token bucket on the API key or IP. In-process, or Redis-backed for a shared limit.
Semantic cache
Prompts are embedded and matched in NuclaDB; close matches skip the backend entirely.
Config hot-reload
SIGHUP reloads backends, model aliases, keys, and limits without dropping in-flight requests.
Prometheus metrics
Volume and latency by model and backend, cache hit / miss / error, on /metrics.
API-key allowlist
Optional: gate /v1/chat/completions behind a set of keys, constant-time compared.
Model aliases
Map a friendly name (gpt-4o) onto whatever a backend actually serves (llama3:70b).
put it in front of anything that speaks the API
config
{
"listen_addr": ":8081",
"backends": [
{ "name": "ollama-a", "url": "http://10.0.0.11:11434",
"models": ["llama3", "mistral"] },
{ "name": "vllm-c", "url": "http://10.0.0.20:8000",
"models": ["llama3:70b"] }
],
"model_aliases": { "gpt-4o": "llama3:70b" },
"rate_limit": { "enabled": true, "requests_per_second": 20, "burst": 40 },
"cache": { "enabled": true, "max_distance": 0.15 },
"api_keys": ["sk-team-1"]
}quickstart
# install (needs Go), grab the sample config, run
curl -fsSL https://raw.githubusercontent.com/Rakshit-gen/inferoute/main/install.sh | sh
curl -O https://raw.githubusercontent.com/Rakshit-gen/inferoute/main/config.example.json
inferouted -config config.example.json
# send it a request
curl localhost:8081/v1/chat/completions \
-d '{"model":"llama3","messages":[{"role":"user","content":"hi"}]}'Then add it as a connection on the Connections page to watch it live here.
questions
- Does my client code change?
- Only the base URL. inferoute speaks the OpenAI chat-completions API, so any OpenAI SDK works by pointing it at the gateway.
- What is the cache matching on?
- Embedding distance between prompts, checked against NuclaDB. You set the max distance; anything closer is treated as a hit. Caching is off until you enable it.
- How does it hold state?
- It does not. inferouted is a single stateless binary reading one JSON config. Rate-limit and cache state can live in Redis / NuclaDB if you want them shared.
- Is it multi-tenant?
- The gateway itself is single-tenant. This dashboard adds per-account isolation: you register your own gateway connections and only ever see your own traffic.