OpenAI recently introduced prompt caching features as a part of its annual DevDay announcements. Prompt caching—which OpenAI claims can benefit users with a 50% discount on inputs—will now applied to various models, including GPT-4o and its mini versions. Unsurprisingly, this has generated excitement among developers, with many already drawing comparisons between OpenAI’s and Claude’s prompt caching features.
But what is prompt caching? What are its practical use cases? We’ll answer these questions in this blog. We’ll discuss OpenAI’s Prompt Caching, compare it with Claude’s approach, and discuss their pricing, speed, methodology, and availability.
What is OpenAI Prompt Caching?
OpenAI’s Prompt Caching is a new feature that allows developers to reuse previously seen input tokens across multiple API calls. This is particularly useful for applications that frequently use the same context, such as chatbots or coding assistants. With prompt caching, developers can save on costs and processing time.
Understanding OpenAI Prompt Caching’s Features
1. Cost Reduction: Cached prompts can reduce up to 50% of the input cost for the data that the model has recently processed. This optimization will benefit applications involving extended dialogues or repetitive tasks where context remains consistent (eg code generation, RAG applications).
2. Latency Improvement: The caching system enhances response times by reusing previously computed tokens, effectively reducing the computational load on the model for repeated or similar prompts. In technical terms:
- With token-level caching, the system caches at the token level, not just for entire prompts.
- With partial cache hits, even partial matches in the prompt can lead to performance improvements.
3. Automatic Application: The caching feature can be seamlessly integrated into supported models, requiring no modifications to existing API integrations. Technical implementation:
- Client-side: No changes are needed in API calls.
- Server-side: OpenAI’s infrastructure handles caching logic transparently.
- Supported models: As of the last update, this feature is available for GPT-4o, GPT-4o mini, OpenAI o1-preview, and o1 mini.
4. Cache Management: OpenAI implemented a time-based expiration policy for cached prompts to balance performance benefits with up-to-date model responses. Cache lifecycle:
- Short-term retention: Cached prompts are typically maintained for 5-10 minutes after the last use.
- Hard expiration: All cached entries are forcibly evicted within 1 hour of their last access, regardless of usage patterns.
Pricing Structure
OpenAI has provided a detailed pricing structure for its models with prompt caching:
| Model | Uncached Input Tokens | Cached Input Tokens | Output Tokens |
| GPT-4o | $2.50 | $1.25 | $10.00 |
| GPT-4o (Fine-Tuning) | $3.75 | $1.88 | $15.00 |
| GPT-4o Mini | $0.15 | $0.08 | $0.60 |
| GPT-4o Mini (Fine-Tuning) | $0.30 | $0.15 | $1.20 |
| o1 | $15.00 | $7.50 | $60.00 |
| o1 Mini | $3.00 | $1.50 | $12.00 |
Claude’s Approach to Prompt Caching
Anthropic’s Claude also offers a prompt caching feature that allows developers to cache frequently used context between API calls. This feature is available for Claude 3.5 Sonnet, Claude 3 Opus, and Claude 3 Haiku.
How is Claude Prompt Caching Useful?
Prompt caching shines in situations where the same prompt is used repeatedly. It’s especially useful for AI assistants like Perplexity or Bind AI, where many users might ask similar questions. In code generation, it helps when developers reuse prompts or templates, such as creating web applications or writing scripts for data conversion. Web search tools also benefit, particularly when they need to fetch the same information multiple times with consistent context.
Key Features of Claude Prompt Caching
- Cost Efficiency: Claude claims that prompt caching can reduce costs by up to 90% and latency by up to 85% for long prompts.
- Use Cases: The caching feature benefits various applications, including conversational agents, coding assistants, large document processing, and iterative tool use.
- Performance Metrics: Early adopters have reported substantial improvements in speed and cost across different use cases, such as reducing latency from 11.5 seconds to 2.4 seconds when chatting with a book using a cached prompt of 100,000 tokens.
Pricing Structure/Benefits
Claude’s pricing model for cached prompts includes the following:
| Model | Input Tokens | Cache Writes | Cache Hits |
| Claude 3.5 (Sonnet) | $3 / MTok | $3.75 / MTok | $0.30 / MTok |
| Claude 3 (Haiku) | $0.25 / MTok | $0.30 / MTok | $0.03 / MTok |
| Claude 3 (Opus) | $15 / MTok | $18.75 / MTok | $1.50 / MTok |
Comparison of Methodology for OpenAI and Claude Prompt Caching
When comparing the methodologies behind OpenAI’s and Claude’s prompt caching:
- Caching Mechanism: Both systems cache previously seen input tokens but differ in how they apply discounts and manage cache retention.
- Discount Rates: OpenAI offers a straightforward 50% discount on cached prompts, while Claude provides a more complex structure with up to 90% savings on usage but incurs additional costs when writing to the cache.
- Latency Reduction: OpenAI focuses on improving processing times through efficient token reuse, while Claude emphasizes substantial reductions in both cost and latency across various use cases.
OpenAI vs Claude Prompt Caching Comparison Table
| Feature | GPT-4o | Claude 3.5 Sonnet |
| Speed/Latency | Average latency of 0.32 seconds | 2x faster than Claude 3 Opus, but slower than GPT-4o |
| Cost Savings | 50% cheaper than previous models | 5x cheaper than Claude 3 Opus |
| Price per million cached writes | $1.25/mtok | $3.75/mtok |
| Cache Mechanism | Partial match caching | Exact match caching |
| Time Limit for Cache | Resets every 5-10 minutes | Clears every hour |
| Caching access | Automatically applied | Requires explicit API parameter and additional cost |
Speed and Latency
Speed is a critical factor for developers when choosing between OpenAI and Claude:
- OpenAI: The exact speed improvements are not detailed but imply faster processing due to token reuse.
- Claude: Reports indicate dramatic reductions in latency—up to 79% for specific use cases like chatting with large documents compared to non-cached scenarios.
Summary of Speed Improvements
| Use Case | Latency w/o Caching | Latency w/ Caching | Cost Reduction |
| Chat with a book (100k tokens) | 11.5s | 2.4s (-79%) | -90% |
| Many-shot prompting (10k tokens) | 1.6s | 1.1s (-31%) | -86% |
| Multi-turn conversation | ~10s | ~2.5s (-75%) | -53% |
These metrics highlight the significant advantages of using prompt caching for OpenAI and Claude.
Availability and How to Use
Both OpenAI and Anthropic have made their prompt caching features accessible through their respective APIs, but alternatively, you can add your API on Bind AI and use prompt caching. Join the waitlist today by signing up on Bind AI!
Using OpenAI Prompt Caching
To use OpenAI’s prompt caching:
- Ensure you are using one of the supported models (e.g., GPT-4o).
- Make API calls with prompts longer than 1,024 tokens.
- Monitor your API responses for the cached_tokens value to see your savings.
- No additional setup is required; caching is automatic once you meet the criteria.
Using Claude Prompt Caching
For Anthropic’s Claude:
- Access the Claude chat by signing up or logging into your Anthropic account.
- Implement prompt caching in your API calls as per their documentation.
- Keep track of your token usage to optimize costs effectively.
The Bottom Line
OpenAI’s and Anthropic’s prompt caching features represent significant advancements in AI model efficiency, offering developers powerful tools to enhance performance while managing costs effectively. While both approaches share similarities in concept, they differ in execution regarding pricing structures, latency improvements, and overall methodology.
Choosing between OpenAI and Claude will depend on specific use cases, budget constraints, and desired performance metrics. As both companies continue to innovate in this space, developers can expect even more enhancements in AI capabilities moving forward.
You can try prompt caching with advanced models like GPT-4o and Claude 3.5 Sonnet with Bind AI Copilot. Besides prompt caching, you will also get benefits like GitHub and Google Drive integration, built-in IDE, and no daily limitations for advanced queries. (900/month) Start your free 7-day premium trial today.