OpenAI

Supported

• LLM: gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.4-mini (streaming + vision)
• STT: whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe
• TTS: tts-1, tts-1-hd, gpt-4o-mini-tts (instructable), streaming
• Realtime speech-to-speech: gpt-realtime-2.1, gpt-realtime-2.1-mini (experimental relay mode)

Setup

import { OpenAILLMProvider, OpenAIWhisperProvider, OpenAITTSProvider } from '@llmrtc/llmrtc-provider-openai';

const llm = new OpenAILLMProvider({ apiKey: process.env.OPENAI_API_KEY, model: 'gpt-5.6-terra' });
const stt = new OpenAIWhisperProvider({ apiKey: process.env.OPENAI_API_KEY, model: 'gpt-4o-mini-transcribe' });
const tts = new OpenAITTSProvider({ apiKey: process.env.OPENAI_API_KEY, model: 'tts-1', voice: 'nova' });

Speech-to-text models

OpenAIWhisperProvider runs on OpenAI's transcription endpoint and accepts any transcription model:

Model	Notes
whisper-1	Default. Battle-tested, widest language coverage
gpt-4o-transcribe	Higher accuracy, better with noisy audio and accents
gpt-4o-mini-transcribe	Near-gpt-4o-transcribe accuracy at lower cost - a good default for voice agents
gpt-realtime-whisper	Native streaming over the Realtime API - use OpenAIRealtimeSTTProvider (below)

const stt = new OpenAIWhisperProvider({
  apiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-4o-mini-transcribe',
  language: 'en' // optional hint
});

Streaming transcription (Realtime API)

OpenAIRealtimeSTTProvider streams audio to a transcription-type Realtime session and yields interim transcripts while the user is still speaking. Billing follows the transcription model's audio-duration pricing, not realtime LLM tokens.

import { OpenAIRealtimeSTTProvider } from '@llmrtc/llmrtc-provider-openai';

const stt = new OpenAIRealtimeSTTProvider({
  apiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-realtime-whisper',  // default
  delay: 'low'                    // optional latency/accuracy trade-off
});

Enable streamingSTT: true on the server (or STREAMING_STT=true + STT_PROVIDER=openai-realtime in CLI mode) for live interim transcripts - see Streaming Speech-to-Text.

Realtime speech-to-speech (experimental)

OpenAIRealtimeSpeechProvider connects sessions directly to the gpt-realtime-2.1 family for ~300–500ms voice-to-voice responses with native interruption handling — used with the server's realtime relay mode:

import { OpenAIRealtimeSpeechProvider } from '@llmrtc/llmrtc-provider-openai';

const provider = new OpenAIRealtimeSpeechProvider({
  apiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-realtime-2.1' // or 'gpt-realtime-2.1-mini' (~1/3 cost)
});

Realtime audio is priced per audio token (~10x an equivalent pipeline); see the relay-mode docs for budgets and cost math.

Text-to-speech

Available voices: alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer on all models; ballad and verse additionally on gpt-4o-mini-tts.

Steerable delivery with gpt-4o-mini-tts

The gpt-4o-mini-tts model accepts natural-language instructions that control tone, pacing, emotion, and accent - useful for giving your voice agent a consistent persona:

const tts = new OpenAITTSProvider({
  apiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-4o-mini-tts',
  voice: 'coral',
  instructions: 'Speak like a friendly, upbeat concierge. Keep a brisk pace.'
});

Instructions can also be set per call:

await tts.speak('I found three options for you.', {
  instructions: 'Sound pleased, as if delivering good news.'
});

Instructions are only sent when the model name starts with gpt-. On any other name - tts-1, tts-1-hd, or a proxy/deployment alias - they are ignored with a one-time warning, because the API rejects them there. If you run an instructable model behind an alias, name it with a gpt- prefix or pass the real model name per call.

Env vars

• OPENAI_API_KEY
• Optional: OPENAI_MODEL, OPENAI_STT_MODEL, OPENAI_TTS_MODEL, OPENAI_TTS_VOICE, OPENAI_TTS_INSTRUCTIONS, OPENAI_BASE_URL

Notes

• Vision is supported via message attachments.
• Use gpt-5.6-luna for latency-sensitive or cost-sensitive flows; gpt-5.6-sol is the flagship tier.
• For the lowest TTS latency, use format: 'pcm' with speakStream (24kHz, 16-bit signed LE, mono).