Every model endpoint gets a live score from four signals: accepted-token error rate, p95 time-to-first-token, price drift, and a soft penalty for upstreams near their own rate ceiling. The router never compares models across families on quality — your fallback chain decides quality; the score only decides which of the allowed models answers now. Since we shipped it, fallback churn dropped from 2.1% to 0.4% of requests.
We keep the scoring loop deliberately boring: one tick every 300 s, EWMA with a 6-tick half-life, and a circuit breaker that quarantines an upstream after 8 consecutive accepted-stream failures. Quarantined upstreams still get 1% of traffic as probes, so recovery is fast and visible on the status page.
Response caching: 72% cache hits without hurting quality
· Cache squad · 8 min read
We hash (model-allowed-set, normalized messages, sampling params) rather than raw bodies: whitespace, key ordering and system-prompt timestamps are normalised away. The win comes from support workloads, where 40% of prompts are near-duplicates within a single shift. Cache entries expire after 24 h by default; we never cache tool-call traces, because replaying them is unsafe.
Cache is per-key with optional team namespaces. A customer on the Growth tier can share hits across their own keys while staying isolated from everyone else — that is where most of the 72% came from.
EU-resident inference: what “region pinning” actually guarantees
· Compliance · 5 min read
With "region": "eu" the gateway accepts the request only on EU edges, forwards only to upstreams with EU processing addenda, and routes open-weight models exclusively to our Frankfurt fleet. Vendor APIs may still carry request metadata through non-EU control planes — that is vendor policy, not ours — so we document it per model on the catalogue page and surface the applied route in the x-gateway-region header. For strict residency, the open-weight fleet is the safe default.
Terms of service (summary)
Plain-language summary; the full agreement is served to authenticated accounts.
Pay-as-you-go token billing; acceptable use follows the policies of each upstream vendor; we do not store prompt content beyond 24-hour request metadata unless archiving is enabled per key; uptime SLA and support response times per tier; accounts may be suspended for credential abuse or prohibited content under the applicable upstream policy.
Privacy notice (summary)
GDPR-aligned; DPA available for Growth and Scale tiers.
Data controller: CI-AI Gateway B.V., Rotterdam, NL. We process API payloads solely to deliver inference under zero-retention agreements with upstreams; account data is kept for billing compliance (10 years); sub-processors: upstream model vendors, eu-Central object storage, transactional email. Requests: privacy@ci-ai.duckdns.org.