Gateway Caching

Gateway caching serves a previously-seen request entirely from PassingRight without forwarding it to the upstream provider. Requests are matched on the parsed cache-key fields — JSON formatting outside string values, and parameters outside the cache key, do not create a separate entry; the text of your messages and the key order of message and tool objects do. Repeated identical calls cost $0 — there is no inference and no provider charge. It is most useful for API workloads with deterministic inputs (classification, batch jobs, FAQ lookups, retries) rather than free-form chat.

How It Works

When you make an API request:

1. PassingRight generates a cache key based on the request parameters
2. If a matching cached response exists, it's returned immediately
3. If no cache exists, the request is forwarded to the provider
4. The response is cached for future identical requests

This means repeated identical requests are served instantly from cache without incurring additional provider costs.

Cost Savings

Caching can dramatically reduce costs for applications with repetitive requests:

Requirements

To use caching:

1. Enable Caching in your project settings under Preferences
2. Configure the cache duration (TTL) as needed
3. Make requests as normal—caching is automatic

Cache Key Generation

The cache key is scoped to your project and includes the resolved provider and model — so two projects never share cache entries, and the same request routed to a different provider is a separate entry. The key is a hash of these request parameters:

• Resolved provider and model
• Messages array (roles and content, including the system prompt)
• Temperature, max tokens, top P
• Frequency and presence penalty
• Response format
• Tools/functions, tool choice, and the web search tool
• Reasoning effort and reasoning max tokens
• prompt_cache_key, prompt_cache_retention, prompt_cache_options
• n and service_tier
• The response mode — streaming and non-streaming entries are kept separate, so an otherwise identical stream: true request never shares an entry with a non-streaming one

Cache Behavior

Cache Hits

When a cache hit occurs:

• Non-streaming responses are returned immediately
• No provider API call is made
• No inference costs are incurred

Cache Misses

When a cache miss occurs:

• Request is forwarded to the LLM provider
• A successful response is stored in cache — errored, client-cancelled, or empty responses are never stored (a response truncated by max_tokens is cached like any other)
• Normal inference costs apply
• Future identical requests will hit the cache

Streaming and Caching

Caching works with both streaming and non-streaming requests:

• Non-streaming: Full response is cached and returned immediately on a hit
• Streaming: The complete stream is cached chunk by chunk (only once it finished successfully) and replayed on a hit, reproducing the original chunk timing with each gap capped at one second — so a streamed replay takes roughly as long as the original stream

Cache TTL (Time-to-Live)

Cache duration is configurable per project in your project settings. You can set the cache TTL from 10 seconds up to 1 year (31,536,000 seconds).

The default cache duration is 60 seconds. Adjust this based on your use case—longer durations work well for static content, while shorter durations are better for frequently changing data.

Identifying Cached Responses

A cache hit replays the stored completion — same id, same content, same token counts. The metadata envelope is rebuilt for the current request (fresh log_id, and a fresh request_id on non-streaming responses) and all cost fields are zeroed, so the body is not byte-for-byte identical to the original response. Two markers tell you it was a replay:

• the x-llmgateway-cache: HIT response header — set on /v1/chat/completions, /v1/messages (which has no metadata envelope), and streaming AI SDK requests; it is not surfaced on /v1/responses or on non-streaming AI SDK responses
• metadata.cached: true on the response body (and on the final metadata chunk of a streamed replay)

All cost fields are zeroed on a replay, because no upstream call was made:

{
	"usage": {
		"prompt_tokens": 12,
		"completion_tokens": 48,
		"total_tokens": 60,
		"cost": 0,
		"cost_details": {
			"total_cost": 0,
			"input_cost": 0,
			"output_cost": 0
		}
	},
	"metadata": {
		"cached": true
	}
}

Token counts are not zeroed — they still describe the completion you are receiving, and are what the dashboard records for analytics. Note that any prompt-caching fields (prompt_tokens_details.cached_tokens, cache_write_tokens) describe the original upstream call, not the replay.

Bypassing the Cache for a Single Request

Send x-no-cache: true to skip the cache for one request — the call goes upstream and its response is not stored. Useful when a client retries an identical request and expects a fresh sample (for example an agent loop) without disabling caching for the whole project. The header works on /v1/chat/completions, /v1/messages, and the AI SDK endpoints; it is not forwarded on /v1/responses.

curl https://api.passingright.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-no-cache: true" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello"}]}'

Use Cases

Development and Testing

During development, you often send the same prompts repeatedly:

// This prompt will only incur costs once
const response = await client.chat.completions.create({
	model: "gpt-4o",
	messages: [{ role: "user", content: "Explain quantum computing" }],
});

Chatbots with Common Questions

FAQ-style interactions often have repeated questions:

// Common questions are served from cache
const faqs = [
	"What are your business hours?",
	"How do I reset my password?",
	"What is your return policy?",
];

Batch Processing

Processing large datasets with potentially duplicate items:

// Duplicate items in batch are served from cache
for (const item of items) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages: [{ role: "user", content: `Classify: ${item}` }],
	});
}

Best Practices

Maximize Cache Hits

• Use consistent prompt formatting
• Normalize input data before sending
• Use deterministic parameters (temperature: 0)
• Avoid including timestamps or random values in prompts

Appropriate Use Cases

Caching is most effective for:

• Static knowledge queries
• Classification tasks
• FAQ responses
• Development/testing
• Retry scenarios

When to Avoid Caching

Caching may not be suitable for:

• Real-time data requirements
• Highly personalized responses
• Time-sensitive information
• Creative tasks requiring variety
• Chat or coding tools where prompts overlap but are not identical — use Provider Cache Control instead

Pricing

Caching is completely free. Cached responses are held in a short-lived Redis-backed cache (bounded by your configured TTL) and do not incur storage charges. Storage costs only apply if you separately enable Data Retention for full request/response payloads.