Skip to content
SecondStack SecondStack SecondStack Docs

API Routing

API Routing centralizes every LLM request in your organization through one gateway. Instead of scattered provider keys with no visibility or controls, chat users, internal apps, agents, and IDE plugins all call a single endpoint — built on LiteLLM — that handles authentication, routing, guardrails, budgets, and observability.

Claude CLI Codex CLI Cursor IDE VS Code IDE OpenClaw Obsidian Chatbot Agents User Virtual API Keys LiteLLM Routing · Guardrails Budgets · Observability Enterprise API Keys LLM PROVIDERS Anthropic OpenAI Google OpenRouter Databricks Local Inference Claude CLI Codex CLI Cursor IDE VS Code IDE OpenClaw Obsidian Chatbot Agents User Virtual API Keys LiteLLM Routing · Guardrails Budgets · Observability Enterprise API Keys LLM PROVIDERS Anthropic OpenAI Google OpenRouter Databricks Local Inference

Clients authenticate with virtual keys; the gateway routes to upstream providers with policy applied.

Virtual API keys

Section titled “Virtual API keys”

Users and applications authenticate with virtual API keys rather than raw provider keys. Each key carries its own budget and policy, and users can self-serve keys for their apps, agents, and IDE tools in the User Dashboard. Provider credentials stay held by the platform.

OpenAI-compatible endpoints

Section titled “OpenAI-compatible endpoints”

Applications call standard, OpenAI- and Anthropic-compatible endpoints — for example /v1/chat/completions, /v1/messages, and /responses — with a virtual key. The gateway handles routing, failover, and load balancing across the models it fronts, so client code stays simple.

Access control

Section titled “Access control”

Model access is governed centrally. Access is default-deny and additive, granted by team, group, user, or team API key, and applied separately to two channels — the Chat App and direct API traffic — so you can allow a model in chat while restricting it for programmatic use, or vice versa. See ControlTower for the management UI.

Guardrails, budgets, and observability

Section titled “Guardrails, budgets, and observability”

Every request can pass through the platform’s guardrails and controls:

  • Guardrails — the SecondGuard service inspects inbound prompts and outbound responses for PII, secrets, and other policy-sensitive content, on the request path.
  • Budgets — per-user, per-team, and per-key limits are enforced here at request time. See Cost Management.
  • Observability — detailed usage is captured for analytics, and client request tags help attribute background traffic.

Minimal exposed surface

Section titled “Minimal exposed surface”

Only the inference endpoints are exposed externally through the OpenResty gateway; the gateway’s full management API is not reachable from outside. This keeps the public surface small.

Supported providers

Section titled “Supported providers”

Through LiteLLM, the gateway can route to Anthropic, OpenAI, Azure, Google (AI Studio and Vertex), Amazon Bedrock, Databricks, xAI, OpenRouter, and self-hosted open-weight models via vLLM.

Section titled “Related”