GPT-realtime API features and comparison with ElevenLabs

OpenAI has officially released updates to its gpt-realtime and Realtime API for production-grade voice agents. What’s new? Advanced speech-to-speech model with new API features: MCP server support, image input, and SIP phone calling. All this at a cheaper price. OpenAI promises benefits like enhanced nuance retention, dramatically reduced latency, and more fluid, human-like voice interactions.

Let’s take a detailed look at this release, its features and pricing, and compare it with industry benchmarks like ElevenLabs.

GPT-realtime Features

GPT-realtime API, unlike its previous versions, was trained with real customers in mind, so it’s built to excel in practical use cases such as customer support, personal assistants, and even educational tools.

What makes it stand out is how well it handles natural conversation — the audio sounds more fluid, it follows instructions more reliably, and it’s smarter when it comes to reasoning and calling functions. Here’s a detailed look at its features:

GPT-realtime Model Capabilities

• Audio Quality & Expressiveness: The model produces speech with greater emotion, intonation, and pacing—capable of following nuanced instructions like “speak empathetically in a French accent.”
• Intelligence & Comprehension: It captures non-verbal cues (like laughter), seamlessly switches languages mid-sentence, and accurately detects alphanumeric information (e.g., phone numbers, VINs) across languages like Spanish, Chinese, Japanese, and French. In benchmarks, gpt-realtime scored 82.8 % on the Big Bench Audio reasoning evaluation—up from 65.6 % achieved by the previous model in late 2024.
• Instruction Following: Adherence to developer-supplied instructions improved significantly, with a 30.5 % score on the MultiChallenge audio benchmark, versus 20.6 % for the older model.

• Function Calling & Integration: Enhanced accuracy now allows gpt-realtime to trigger functions at the right times with appropriate arguments. On the ComplexFuncBench evaluation, performance jumped to 66.5 %, compared to 49.7 % previously. Asynchronous function calling also runs smoothly—long-running calls no longer break the conversational flow.

New Developer-Friendly Capabilities

• Remote MCP Server Support – Developers can now link remote tool servers (via MCP) directly in sessions, enabling built-in integration with backend tools like Stripe without manual wiring.
• Image Input Support – Voice agents can now receive image inputs (photos, screenshots) alongside audio or text and respond contextually—e.g., “what do you see in this screenshot?”
• Session Initiation Protocol (SIP) Support – This integrates voice agents with PBX, desk phones, and the public phone network directly from the Realtime API.
• Reusable Prompts – Prompt templates (developer messages, tools, conversation examples) can now be saved and reused across sessions for easier development and consistency.

Voices & Expressiveness

Two new voices—Marin and Cedar—were introduced exclusively for the Realtime API. Additionally, all existing preset voices received upgrades for greater realism and expressiveness.

Safety, Privacy & Compliance

OpenAI incorporated multiple safeguards such as content classifiers to halt misuse, enforced user transparency (making clear that the agent is AI), default preset voices to reduce impersonation risks, and compliance with EU data residency and enterprise privacy standards.

GPT-realtime and Realtime API Pricing & Availability

The Realtime API and gpt-realtime are now available to all developers. Pricing reflects a 20 % reduction compared to the earlier gpt-4o-realtime-preview model:

• Audio input: $32 per 1 M tokens (just $0.40 when hitting cache).
• Audio output: $64 per 1 M tokens.

These controls, coupled with fine-grained context control, help keep long-session costs in check. The introduction of prompt caching, image input, and other enhancements support wider adoption and cost efficiency.

Historically, the original Realtime API (beta) pricing was:

• Text input: $5 / 1 M tokens
• Text output: $20 / 1 M tokens
• Audio input: $100 / 1 M tokens
• Audio output: $200 / 1 M tokens (approx. $0.06/min input, $0.24/min output).

Real-world testing revealed voice interactions cost around $1 per minute—higher than nominal estimates. Prompt caching and pricing updates help address these concerns.

GPT-realtime Use Cases and Applications

GPT-realtime and Realtime API are ideal for:

• Customer support voice agents – e.g., Zillow uses gpt-realtime for interactive home search conversations.
• Voice assistants and personal aid – including language tutors, coaching apps, and accessibility tools.
• Phone-based AI agents – thanks to SIP support.
• Interactive media – combining voice and image inputs for richer user interfaces.
• Proactive tools – voice agents that can call internal APIs or trigger functions while maintaining conversational fluidity.

GPT-realtime vs. ElevenLabs – Comparison

Now, let’s contrast OpenAI’s GPT-Realtime model and Realtime API with ElevenLabs, a leading player in speech synthesis and voice AI.

GPT-realtime vs ElevenLabs: Voice Customization & Emotional Expression

ElevenLabs stands out for voice emotion, customization, and cloning—offering tools for creating custom voices with minimal data and accurately capturing emotional nuance.

Their TTS latency is impressive (<400 ms) and they offer a high price-to-quality ratio, especially for creative applications like audiobooks or character voiceovers.

If you’re curious as to how to use ElevenLabs voice AI in your applications, follow this guide.

GPT-realtime vs ElevenLabs: Pricing & Affordability

• ElevenLabs offers a free tier (e.g., 15 minutes of conversational AI) and a Business plan priced at $0.08 per minute for voice minutes, presumably scaling with volume.
• This is significantly cheaper than the original OpenAI Realtime API cost (~$1 per minute), though OpenAI’s recent cuts have narrowed the gap.

GPT-realtime vs ElevenLabs: Integration & Developer Experience

• OpenAI Realtime API offers a single-model pipeline, low-latency, seamless function calling, image input, and SIP support—all via one API. It’s engineered for production-grade, multimodal voice agents.
• ElevenLabs, in contrast, provides developer-friendly TTS and voice tools with straightforward APIs and strong creative control—but may require chaining with other models for speech-to-text, logic, or understanding tasks.

GPT-realtime vs ElevenLabs: Applications & Use Cases

• OpenAI is tailored for voice agents that need reasoning, interactivity, tooling, and multimodal input, such as conversational support systems or intelligent voice interfaces.
• ElevenLabs shines in creative, content-driven applications—audiobook narration, character voices, emotionally driven TTS, and voice cloning with emotional nuance.

GPT-realtime vs ElevenLabs: Latency & Real-Time Flow

• OpenAI’s solution is designed for low-latency, streaming, real-time interactions with continuous flow and interruption handling.
• ElevenLabs maintains low latency (<400 ms) but relies on a chained architecture, potentially requiring separate speech recognition or logic layers for full interactivity.

GPT-realtime vs. ElevenLabs Summary Table

The Bottom Line

OpenAI’s GPT-realtime and Realtime API offer a robust solution for building interactive voice agents, focusing on reasoning, tool integration, and context awareness, along with SIP integration and improved performance at lower costs.

In contrast, ElevenLabs specializes in voice quality and customization, making it ideal for creative tasks where expressiveness and voice cloning are key, thanks to its user-friendly API and affordable pricing for content creators and designers.

Choosing between them comes down to needs:

• For conversational agents with dynamic logic, tool access, and multimodal inputs, OpenAI Realtime API is the more robust option.
• For emotionally rich, creative, voice-first content, ElevenLabs offers exceptional realism, style, and affordability.

Easy.