Skip to main content

LLMRTC Docs

LLMRTC is an Open Source TypeScript SDK for building real-time voice and vision AI agents. It combines WebRTC for low-latency audio/video streaming with LLMs, speech-to-text, and text-to-speech—all through a unified, provider-agnostic API.

With LLMRTC you can add real-time capabilities to any LLM wether it is self-hosted or cloud-hosted.

You can connect multiple LLMs together, and launch AI agents playbooks, delegating different tasks to different LLMs, and combine them to create a powerful system.

What is LLMRTC? - Add Real-Time capabilities to any LLM​

LLMRTC handles the complex infrastructure needed for conversational AI:

You focus on your application logic. LLMRTC handles:

  • Real-time audio/video streaming via WebRTC
  • Voice activity detection and barge-in
  • Provider orchestration and streaming pipelines
  • Session management and reconnection

Key Features​

Real-Time Voice​

Stream audio bidirectionally with sub-second latency. Server-side VAD detects speech boundaries, and barge-in lets users interrupt the assistant naturally.

Vision Support​

Send camera frames or screen captures alongside speech. Vision-capable models can see what users see.

Provider Agnostic​

Switch between OpenAI, Anthropic, Google Gemini, AWS Bedrock, or local models without changing your code. Mix providers freely (e.g., Claude for LLM, Whisper for STT, ElevenLabs for TTS).

Tool Calling​

Define tools with JSON Schema. The model calls them, you execute them, and the conversation continues seamlessly.

Playbooks​

Build multi-stage conversations with per-stage prompts, tools, and automatic transitions. Two-phase execution separates tool work from responses. Six transition types (tool calls, intents, keywords, LLM decision, timeouts, custom) give precise control over conversation flow.

Streaming Pipeline​

Responses start playing before generation completes. Sentence-boundary detection ensures TTS starts at natural pause points, reducing perceived latency. STT → LLM → TTS streams end-to-end.

Hooks & Observability​

20+ hook points for logging, debugging, and custom behavior. Built-in metrics track TTFT, token counts, and durations. Plug into your existing monitoring stack.

Session Resilience​

Automatic reconnection with exponential backoff. Conversation history survives network interruptions. Graceful degradation when providers fail.

Architecture​

LLMRTC consists of three packages:

PackagePurpose
@llmrtc/llmrtc-coreTypes, orchestrators, tools, hooks—shared foundation
@llmrtc/llmrtc-backendNode.js server with WebRTC, VAD, and all providers
@llmrtc/llmrtc-web-clientBrowser SDK for audio/video capture and playback

Supported Providers​

Cloud Providers​

ProviderLLMSTTTTSSpeech-to-SpeechVision
OpenAIGPT-5.6 family, GPT-5.5Whisper, GPT-4o Transcribe, Realtime (streaming)TTS-1/HD, GPT-4o Mini TTS (steerable)GPT Realtime 2.1GPT-5.6
AnthropicClaude Sonnet 5, Opus 4.8, Haiku 4.5---Claude vision
Z.aiGLM 5.2 (1M context, open weights)----
Google GeminiGemini 3.5 Flash, 2.5 Flash/Pro--Gemini Live (preview)Gemini Vision
AWS BedrockClaude, Llama, etc.---varies
OpenRouter100+ models---varies
ElevenLabs-Scribe v2 (+ realtime streaming)Multilingual v2, v3, Flash--

Speech-to-speech models run through the experimental realtime relay mode — voice-to-voice in ~300–500ms with tools, playbooks, and budgets intact.

Local Providers​

ProviderLLMSTTTTSVision
OllamaLlama 4, Gemma 4, Qwen, etc.--Qwen3-VL, LLaVA
LM StudioAny GGUF model---
Faster-Whisper-Whisper (fast)--
Piper--Many voices-

Use Cases​

Voice Assistants​

Build Siri/Alexa-style assistants with custom capabilities. Add tools for your domain—check orders, book appointments, control devices.

Customer Support​

Multi-stage playbooks guide conversations through authentication, triage, and resolution. Tools integrate with your CRM and ticketing systems.

Multimodal Agents​

Combine voice with vision for screen-aware assistants. Users can share their screen or camera and ask questions about what they see.

On-Device AI​

Run entirely locally with Ollama, Faster-Whisper, and Piper. No cloud dependencies, no API costs, full privacy.

Developer Experience​

  • TypeScript-First: Full type safety with IntelliSense support across all APIs
  • Tool Validation: JSON Schema validation catches malformed LLM arguments before execution
  • Smart Error Handling: Automatic retry with error classification (retryable vs non-retryable)
  • Comprehensive Types: Every provider, hook, and event is fully typed

Production Deployment​

For production use, WebRTC requires a TURN server to ensure reliable connections for users behind NAT/firewalls.

Recommended: The OpenRelay Project by Metered provides a free global TURN server network with 20GB of monthly TURN usage at no cost — sufficient for most applications.

const server = new LLMRTCServer({
  providers: { llm, stt, tts },
  metered: {
    appName: 'your-app-name',
    apiKey: 'your-api-key'
  }
});

See Networking & TURN for detailed configuration options.

Quick Example​

Backend (Node.js):

import { LLMRTCServer, OpenAILLMProvider, OpenAIWhisperProvider, OpenAITTSProvider } from '@llmrtc/llmrtc-backend';

const server = new LLMRTCServer({
  providers: {
    llm: new OpenAILLMProvider({ apiKey: process.env.OPENAI_API_KEY! }),
    stt: new OpenAIWhisperProvider({ apiKey: process.env.OPENAI_API_KEY! }),
    tts: new OpenAITTSProvider({ apiKey: process.env.OPENAI_API_KEY! })
  },
  systemPrompt: 'You are a helpful voice assistant.'
});

await server.start();

Frontend (Browser):

import { LLMRTCWebClient } from '@llmrtc/llmrtc-web-client';

const client = new LLMRTCWebClient({
  signallingUrl: 'ws://localhost:8787'
});

client.on('transcript', (text) => console.log('User:', text));
client.on('llmChunk', (chunk) => console.log('Assistant:', chunk));

await client.start();
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
await client.shareAudio(stream);

Getting Started​

Ready to build? Follow our quickstart guides:

  1. Installation - Set up packages and dependencies
  2. Backend Quickstart - Run your first server
  3. Web Client Quickstart - Connect from the browser
  4. Tool Calling - Add custom capabilities
  5. Local-Only Stack - Run without cloud APIs

Documentation Structure​

SectionContents
Getting StartedInstallation, quickstarts, first application
ConceptsArchitecture, streaming, VAD, playbooks, tools
BackendServer configuration, deployment, security
Web ClientBrowser SDK, audio/video, UI patterns
PlaybooksMulti-stage conversations, text and voice agents
ProvidersProvider-specific configuration and features
RecipesComplete examples for common use cases
OperationsMonitoring, troubleshooting, scaling
ProtocolWire protocol for custom clients

Community​

Next Steps​

Getting Started

Build your first voice assistant in minutes

Architecture

Understand how LLMRTC works