What makes it different is what it refuses to throw away. Vertex AI's context caching, Cloud Storage media and strict response schemas survive the trip, and a routing model can decide how capable a model, and how much reasoning, each request deserves. This post explains why we built it, how it is put together and where it is going.

We tried not to write one​

We started where most teams start: put a generic OpenAI-compatible proxy in front of every provider and move on. That approach reaches Gemini through an OpenAI-shaped adapter, and the adapter is where the features we depend on disappear. There is no OpenAI field for a Vertex cachedContents resource or a gs:// video, and an OpenAI response_format schema only becomes a strict Vertex responseSchema if the adapter translates it, so a translation layer either drops these features or never learns about them.

We wanted multi-provider routing and fallback, and we wanted Vertex's native features, not one or the other. So Synapse keeps a dedicated native lane for Vertex, alongside the OpenAI-compatible lane that serves everything else, and keeps the codebase small enough that the routing, fallback, ledger and metrics code is yours to read, run and embed.

Three lanes, one endpoint​

Every chat request goes to the same endpoint. The request body alone decides which of three lanes serves it, so a client opts into native features by adding an extension block, with no second API to learn. The architecture overview walks through the whole request flow.

• The standard lane calls providers through the genai crate. OpenAI, Qwen (DashScope) and self-hosted vLLM, Ollama or TGI through the oai_compat provider can all appear in one fallback chain, and so can Vertex, without its native-only features.
• The native Vertex lane takes any request whose vertex block carries cached_content, response_schema, thinking_config or a gs:// media URI. Synapse translates the OpenAI messages to Vertex's format and calls Vertex AI's :streamGenerateContent endpoint directly, buffering the stream for clients that did not ask for one. Only the route's vertex legs take part; if a route has none, the request fails with native_feature_unsupported instead of silently losing the features. See the native Vertex guide.
• The Jev lane sends typed questions to TypeSafe System One (Jev), which returns structured decisions instead of free text. It can also judge and then extract in a single call. See the Jev lane guide.

A native request looks like any other chat completion, plus a vertex block:

{
  "model": "gemini-flash",
  "messages": [{ "role": "user", "content": "Summarise the video." }],
  "vertex": {
    "media_uris": ["gs://cloud-samples-data/video/animals.mp4"],
    "response_schema": { "type": "object", "properties": { "summary": { "type": "string" } } }
  }
}

Here gemini-flash is a route alias for gemini-3.5-flash-lite on Vertex, as in the quickstart.

Routing by difficulty​

Lanes decide how a request reaches a provider. Routes decide which legs it tries. A static route is an ordered list of legs, tried until one succeeds. A strategy = "jev" route instead declares difficulty tiers, described by the kind of work they suit. For each request, the Jev router asks Jev how demanding the conversation is and whether it needs step-by-step reasoning, then serves it from the matching tier with a matching reasoning effort. The response says what happened in x-synapse-routing, x-synapse-tier and related headers.

A companion post, Jev routing explained, covers the decision in detail, and the Jev router guide is the reference.

The baseline, not an add-on​

A few things we consider table stakes come with every deployment:

• Real streaming. On the standard and native Vertex lanes, Synapse always streams from upstream, so stream: true clients get token-by-token server-sent events. Non-streaming clients get the buffered result and, on the standard lane, keep the full fallback chain. See Streaming and tool calling.
• Tool calling on the standard and native Vertex lanes. The native lane also honours tool_choice through Vertex toolConfig; the standard lane forwards tools but drops tool_choice.
• A cost ledger you own. Every request is attributed to a tenant from the x-synapse-tenant header and priced from your pricing.toml into SQLite or Postgres, with optional fan-out to Google Cloud Pub/Sub and AWS SNS. See the cost ledger guide.
• Metrics. OpenTelemetry synapse_* metrics, served in Prometheus format on port 9090 and optionally pushed over OTLP. The OpenTelemetry metrics post explains the pipeline.
• Input guardrails. Named scanner policies that block or observe requests before they reach a provider. See Guardrails policy.

Synapse is licensed under MPL-2.0, so you can build it into commercial products, and none of this sits behind a paid tier.

Run it, or embed it​

You can run Synapse as a single binary or as the sustentabilitas/synapse-gateway Docker image, with the API on port 8080 and metrics on port 9090. The quickstart gets a gateway answering in a few minutes.

If your service is written in Rust, you can skip the extra process entirely. Depend on the synapse-gateway crate with default-features = false, build a Gateway in code, and call Gateway::chat() in-process, with the same routing, fallback and ledger behaviour as the binary and no HTTP hop. See Embedding Synapse as a library.

The family​

Synapse is a Cargo workspace, and the gateway has siblings that grew out of the same needs:

• synapse-proxy is a config-driven reverse-proxy sidecar. It routes by path prefix and stamps a bound identity, such as a tenant, on every forwarded request, so a sandboxed workload cannot choose its own.
• synapse-a2a is an agent-to-agent (A2A) registry with admin registration and public discovery, served by the gateway binary.
• synapse-mcp is an on-demand MCP gateway library that routes Streamable HTTP tool calls per server and injects the current tenant's identity.
• synapse-context is the shared context store behind the proxy and the MCP gateway.

Workspace crates shows how they depend on each other.

What's next​

We would rather you learn the gaps here than in production. Inbound authentication, rate limiting and dynamic route reloading are planned; today, run Synapse behind your own API gateway, ingress or service mesh. The gateway records metrics but emits no trace spans, and chat legs get one attempt each, with no retries or circuit breakers. The limitations and roadmap page lists every one of these, with links to the details.

The code is on GitHub, and contributions are welcome. Try the quickstart, and tell us what breaks.

This post covers why we moved, what the exposition looks like, what the metrics cover and how to get the example Grafana dashboard running.

Why opentelemetry-rust​

The gateway was the odd one out in its own workspace. synapse-proxy and synapse-mcp already recorded through OpenTelemetry instruments, while the gateway used the metrics crate and its own exporter. Moving the gateway onto opentelemetry-rust 0.32 puts all three crates on one pipeline: instruments created from an OpenTelemetry Meter, exported by the OpenTelemetry Prometheus exporter, with OTLP available beside it. (The proxy keeps the exporter's default naming; see the metrics catalogue.)

That matters most when you embed the gateway. A Rust service that builds a Gateway in code now passes it a GatewayMetrics made from a meter of its own MeterProvider, and the gateway's instruments land wherever that provider exports. The default is a no-op, so an embedded gateway records nothing until you opt in; a global metrics recorder in the host process no longer receives gateway series. With the server feature, synapse::telemetry::install builds exactly the exporters the binary uses. See Embedding Synapse as a library.

Prometheus by default, OTLP when you ask​

The binary always serves Prometheus text on its metrics listener, SYNAPSE_METRICS_ADDR, which defaults to 0.0.0.0:9090, at GET /metrics. The API port does not serve metrics.

curl -s http://localhost:9090/metrics | grep '^synapse_'

Set OTEL_EXPORTER_OTLP_ENDPOINT to a collector's base URL, such as http://otel-collector:4318, and the gateway also pushes the same metrics over OTLP/HTTP to <endpoint>/v1/metrics, every 60 seconds by default, tagged with service.name from OTEL_SERVICE_NAME (default synapse-gateway). Both paths carry the same names and labels. The startup line synapse-gateway metrics listening tells you whether OTLP is on. The Telemetry variables are listed with the rest of the configuration.

An exposition that looks hand-written​

The OpenTelemetry Prometheus exporter adds things by default that would have broken existing dashboards: a second _total on counters, unit suffixes, otel_scope_* labels and a target_info series. The gateway turns all of those off, so names come out exactly as the catalogue lists them, such as synapse_requests_total and synapse_request_duration_seconds.

Two things did change, both deliberately:

• Durations are histograms. The *_duration_seconds metrics now expose _bucket, _sum and _count series, with buckets from 5 ms to 120 seconds, instead of summaries. Queries on {quantile=...} move to histogram_quantile(...) over _bucket, and you can now aggregate latency across instances, which summaries never allowed.
• Cardinality is capped. Each metric keeps at most 2,000 label combinations, the OpenTelemetry SDK default. Beyond that, new combinations fold into one series labelled otel_metric_overflow="true".

What the metrics cover​

The metrics catalogue lists every instrument with its type and labels. In short, the gateway counts:

• Chat requests: requests, latency, and input and output tokens, labelled by route, serving model, lane and system, the provider family in OpenLLMetry's gen_ai.system vocabulary (vertexai, openai, dashscope, oai_compat).
• Embeddings and passthrough: embedding requests and latency, and passthrough calls by action and status for the Gemini and Jev passthrough endpoints, plus Gemini passthrough model fallbacks.
• Jev: routing decisions by tier and outcome, decision latency, and hybrid extraction responses.
• Guardrails: scans by outcome, matches by scanner and severity, and scan latency.
• The cost ledger: rows dropped because the queue was full, and rows a sink failed to write.

Two things are deliberately missing. Tenants are not labels, because their values come from clients and are unbounded; per-tenant usage and cost live in the cost ledger, which you can query directly. And there is no metric for failed chat completions: the request metrics count requests that produced a response, so measure error rates at the load balancer or mesh in front of the gateway. What the request metrics count spells out the edge cases, such as abandoned streams.

The gateway records metrics only. It creates no OpenTelemetry trace spans; logs go through tracing, filtered with RUST_LOG. See Logging.

A dashboard to start from​

The repository ships an example Grafana dashboard, synapse-gateway-dashboard.json. It charts traffic by route, lane and model; latency percentiles; token rates; cost-ledger health; and embeddings. The latency panels are the ones the move to histograms fixed: they query histogram_quantile over _bucket series, which the old summaries never produced, so they used to stay empty.

Two things to check before you import it: every panel names its Prometheus data source by UID prometheus, and every query filters on a $job variable that defaults to synapse-gateway. The Grafana dashboard page covers importing it by hand, provisioning it with Docker Compose, and loading it on Kubernetes through the kube-prometheus-stack sidecar. Its Resilience row stays empty, because chat legs have no retries or circuit breakers to report on.

Where to start​

Point Prometheus at port 9090, import the dashboard, and add the starter alerts from Metrics: lost ledger rows, a failing ledger sink and degraded Jev routing are problems no client error will tell you about.

Synapse's Jev router lets the request decide. A strategy = "jev" route declares difficulty tiers, and for each request the gateway asks TypeSafe Jev how demanding it is, then serves it from the matching tier with a matching reasoning effort. This post walks through how that decision is made, what it costs, and what happens when it can't be made.

Tiers describe work, not models​

A Jev route replaces a route's single list of legs with two to ten tiers, ordered from easiest to hardest. Each tier has a name, a description of the work it suits, a reasoning effort and its own legs:

[routes."auto"]
strategy = "jev"

[routes."auto".jev_router]
default_tier = "moderate"

[[routes."auto".tiers]]
name = "trivial"
description = "Greetings, chit-chat, one-line lookups or rewrites"
effort = "none"
legs = [{ provider = "vertex", model = "gemini-3.5-flash-lite", region = "us" }]

[[routes."auto".tiers]]
name = "moderate"
description = "Everyday Q&A, summarising, simple extraction or code edits"
effort = "low"
legs = [{ provider = "vertex", model = "gemini-3.6-flash", region = "global" }]

[[routes."auto".tiers]]
name = "hard"
description = "Multi-step analysis, maths or proofs, non-trivial code or debugging"
effort = "medium"
legs = [{ provider = "vertex", model = "gemini-3.1-pro-preview", region = "global" }]

A jev route needs TYPESAFE_API_KEY; without it, the default strict provider validation refuses to start the gateway.

The descriptions matter more than anything else in the file, because they are what Jev scores the request against. "Multi-step analysis, maths or proofs" gives Jev something to judge; "Gemini Pro" does not. The order matters too: it is what the difficulty score indexes, and it drives fallback between tiers. The Jev tiers tutorial builds this route step by step.

Two questions per request​

Before serving a request, Synapse sends Jev a bounded summary of the conversation: the latest user message, the system prompt and as much recent history as fits, about 24,000 characters in all, plus whether the request carries tools or media. Jev answers two typed questions about it.

Difficulty is a score against your tier descriptions, in order. Synapse rounds it to the nearest tier. Jev also reports its confidence in that score; below min_confidence (default 0.5), the gateway ignores the score and serves the request from default_tier.

Needs reasoning is the probability that a good reply needs careful step-by-step reasoning, such as maths, logic, planning or debugging. At or above reasoning_threshold (default 0.7), the tier's effort goes up one step, even when the difficulty score was too uncertain to use.

The decision call is bounded by timeout_ms, 400 ms by default, so it adds up to that much latency to every request on the route. That is the price of the decision, and the reason a static route is still the better choice when you already know which model each use case needs.

Effort, translated per provider​

Each tier's effort is one of none, minimal, low, medium, high, xhigh or max, and the reasoning bump moves one step along that list, stopping at max. Synapse translates the effort for each leg:

• On the standard lane, OpenAI-style providers get reasoning_effort, with max sent as xhigh, and Vertex Gemini 3 models get a thinkingLevel.
• On the native Vertex lane, Synapse sets thinkingBudget itself: 512 tokens for minimal, 1,024 for low, 4,096 for medium, 8,192 for high, 16,384 for xhigh and 24,576 for max.
• none sends nothing, so the model's own default applies. On some Gemini models that default is dynamic thinking, so use minimal when you want thinking kept small.

The Effort table in the routes reference has every mapping. Thinking tokens are billed as output, which is why easy tiers usually run at none or minimal and only hard tiers get medium or more.

A client's own effort always wins on its lane. A reasoning_effort in the request body replaces the tier's effort on the standard lane, and a vertex.thinking_config replaces the tier's thinkingBudget on the native lane. Jev still picks the tier, and the response reports x-synapse-reasoning-effort: client.

Every response says what happened​

Every chat completion carries x-synapse-routing: jev on a Jev route, static-override when the client sent "routing_strategy": "static" to skip the decision, and static on ordinary routes. When they apply, x-synapse-tier names the tier that served, x-synapse-tier-decided the tier the decision picked (Jev's choice, or default_tier when routing is degraded) if a different one served, and x-synapse-reasoning-effort the effort the serving leg ran with. For streams, the headers describe the leg that produced the first chunk. The response headers section lists every value.

A Jev problem never fails the request​

The router is designed to degrade, not to break. If Jev times out, errors, answers with low confidence, or the gateway has no Jev client at all, the request goes to default_tier and the response carries x-synapse-routing-degraded with the reason: timeout, error, low_confidence or jev_unavailable. Pick a default_tier that handles most requests acceptably, usually a middle one.

If the serving tier's legs fail, Synapse tries each harder tier, nearest first, then each easier tier. Requests with native Vertex features only use vertex legs, so they go to the nearest tier that has one. Only when every eligible leg fails does the request fail, exactly as it would on a static route. Failure behaviour has the details.

Decisions you can audit​

Each decision Jev answers writes its own row to the cost ledger, with provider typesafe, lane jev and the same request_id as the chat request it routed, priced as typesafe:<model> from your pricing.toml. You see the cost of deciding next to the cost of answering.

Two metrics track the router: synapse_routing_decisions_total, labelled by route, tier and outcome, and synapse_routing_decision_duration_seconds. A rising share of timeout or error outcomes means requests are landing on default_tier without a real decision, so raise timeout_ms or check Jev's availability. Metrics has a ready-made query for that ratio.

Try it​

Start with the Jev tiers tutorial, then keep the Jev router guide at hand while you tune tier descriptions and thresholds against your own traffic.