What is Prompt Caching?

Prompt caching is an API-level optimisation that stores the processed state of your prompt prefix, allowing subsequent requests with the same prefix to skip redundant computation. When you send a prompt to an LLM, the model processes each token through multiple transformer layers—a computationally expensive operation. Prompt caching stores this intermediate state (called the KV cache) so it doesn't need to be recomputed.

Think of it like preparing ingredients for a recipe. Without caching, you chop vegetables from scratch every time you cook. With caching, you prepare them once and store them ready to use—you only need to add the final fresh ingredients each time.

How Prompt Caching Works

Here's the technical flow of prompt caching:

1. First Request (Cold Cache): The full prompt is processed through all transformer layers. The computed key-value pairs for the prompt prefix are stored in cache.
2. Cache Storage: The provider stores the KV cache associated with your prompt prefix, typically for a limited time (TTL).
3. Subsequent Requests (Warm Cache): When you send a request with the same prefix, the cached KV pairs are loaded directly. Only the new tokens (your dynamic content) need full processing.
4. Cost Reduction: You pay a reduced rate for cached tokens since they require less computation.

Provider Implementations

Major LLM providers offer prompt caching with different specifications:

Anthropic Claude

• Minimum prefix: 1,024 tokens (approximately 4,000 characters)
• Cache TTL: 5 minutes (extended with each cache hit)
• Cost savings: Cached input tokens cost 10% of standard input pricing (90% reduction)
• Write cost: 25% premium on first write to cache
• Activation: Automatic for eligible prompts, or explicit with cache_control parameter

OpenAI

• Minimum prefix: 128 tokens for GPT-4 Turbo and later models
• Cache TTL: Variable (typically 5-10 minutes)
• Cost savings: 50% reduction on cached input tokens
• Activation: Automatic—no code changes required

When to Use Prompt Caching

Prompt caching delivers the most value in these scenarios:

• Long System Prompts: Applications with detailed instructions, personas, or policies that remain constant across requests.
• RAG Applications: When retrieved context documents are reused across multiple user queries within the cache window.
• Multi-turn Conversations: Chat applications where conversation history grows but earlier messages remain unchanged.
• Code Analysis: Tools that include large codebases or documentation as context.
• Batch Processing: When processing many items with the same instructions or context.

When NOT to Use Prompt Caching

Prompt caching isn't always the right solution:

• Short prompts: If your system prompt is under 1,024 tokens (Claude) or 128 tokens (OpenAI), caching won't activate.
• Highly dynamic prefixes: If every request has unique context at the start (e.g., personalised user profiles), cache hit rates will be near zero.
• Low request frequency: With 5-minute TTLs, infrequent requests mean your cache expires before the next hit.
• One-off analysis: For single queries against unique documents, there's no reuse opportunity.

Structuring Prompts for Caching

To maximise cache hits, structure your prompts with static content first:

[#ff7b72]">class=[#ff7b72]">class="text-[#a5d6ff]">"text-[#8b949e]">// Optimal structure [#ff7b72]">for prompt caching
[#ff7b72]">const prompt = {
  [#79c0ff]">system: `
    [STATIC - CACHEABLE]
    You are a customer support agent [#ff7b72]">for TechCorp.
 
    COMPANY [#79c0ff]">POLICIES:
    - Returns accepted within [#79c0ff]">30 days
    - Premium members get priority support
    - ... (thousands of tokens of policies)
 
    PRODUCT [#79c0ff]">CATALOGUE:
    - ... (product details)
 
    RESPONSE [#79c0ff]">GUIDELINES:
    - Be professional and empathetic
    - Always verify customer identity first
  `,
 
  [#ff7b72]">class=[#ff7b72]">class="text-[#a5d6ff]">"text-[#8b949e]">// Dynamic content comes last
  [#79c0ff]">user: `
    [DYNAMIC - NOT CACHED]
    [#79c0ff]">Customer: [#ff7b72]">class="text-[#a5d6ff]">"I want to [#ff7b72]">return my laptop"
  `
};

The key principle: static content first, dynamic content last. The cache matches from the beginning of your prompt, so any variation in the prefix will invalidate the cache.

Cost Analysis Example

Let's calculate the savings for a typical enterprise scenario:

Prompt Caching vs Response Caching

It's important to distinguish prompt caching from response caching—they're complementary techniques:

For optimal cost efficiency, consider using both: prompt caching for the API layer and response caching (perhaps with semantic similarity matching) at the application layer for frequently repeated queries.

Conclusion

Prompt caching is one of the easiest wins in LLM cost optimisation. For applications with consistent system prompts and high request volumes, it can reduce costs by 80%+ with minimal code changes.

The key is intentional prompt architecture: place static content at the beginning, keep dynamic elements at the end, and monitor your cache hit rates to continuously optimise.