Skip to main content

Message Types

All messages are JSON objects with a type field. Messages are sent over WebSocket and/or WebRTC data channel.

Client → Server Messages​

ping​

Heartbeat to keep connection alive. Server responds with pong.

{
  type: 'ping',
  timestamp: number  // Date.now() for RTT calculation
}

offer​

WebRTC SDP offer to initiate peer connection.

{
  type: 'offer',
  signal: RTCSessionDescriptionInit  // From RTCPeerConnection.createOffer()
}

reconnect​

Attempt to recover a previous session after disconnect.

{
  type: 'reconnect',
  sessionId: string  // Previous session ID to recover
}

audio​

Audio data for transcription (fallback when WebRTC audio track unavailable).

{
  type: 'audio',
  data: string,                    // Base64-encoded audio (WAV/WebM)
  attachments?: VisionAttachment[] // Optional images to include
}

attachments​

Vision attachments sent via data channel, queued for next speech segment.

{
  type: 'attachments',
  attachments: VisionAttachment[]  // Array of { data, mimeType?, alt? }
}

Server → Client Messages​

ready​

Sent immediately after WebSocket connection. Contains session info and ICE servers for WebRTC.

{
  type: 'ready',
  id: string,                      // Unique session ID assigned by server
  protocolVersion: number,         // Currently 1
  iceServers?: RTCIceServer[]      // STUN/TURN servers for WebRTC connection
}

ICE Servers: The iceServers array contains STUN and TURN server configurations that the client should use when creating its RTCPeerConnection. This enables the server to centrally manage ICE configuration, including Metered TURN credentials.

Example with Metered TURN:

{
  "type": "ready",
  "id": "abc123",
  "protocolVersion": 1,
  "iceServers": [
    { "urls": "stun:stun.metered.ca:80" },
    { "urls": "turn:global.relay.metered.ca:80", "username": "abc", "credential": "xyz" },
    { "urls": "turn:global.relay.metered.ca:443?transport=tcp", "username": "abc", "credential": "xyz" }
  ]
}

See Networking & TURN for configuration details.

pong​

Response to ping. Echoes timestamp for RTT calculation.

{
  type: 'pong',
  timestamp: number  // Echoed from ping message
}

signal​

WebRTC SDP answer in response to client's offer.

{
  type: 'signal',
  signal: RTCSessionDescriptionInit  // SDP answer
}

reconnect-ack​

Response to reconnect request.

{
  type: 'reconnect-ack',
  success: boolean,        // Whether reconnection succeeded
  sessionId: string,       // Session ID (may be new if original expired)
  historyRecovered: boolean // Whether conversation history was restored
}

Conversation Flow Messages​

transcript​

Speech-to-text transcription result.

{
  type: 'transcript',
  text: string,       // Transcribed text
  isFinal: boolean    // true when transcription is complete
}

llm-chunk​

Streaming LLM response chunk.

{
  type: 'llm-chunk',
  content: string,    // Partial response text
  done: boolean       // true for final chunk
}

llm​

Complete LLM response (non-streaming mode).

{
  type: 'llm',
  text: string        // Full response text
}

TTS Messages​

tts-start​

TTS synthesis is starting. Sent before first audio chunk.

{
  type: 'tts-start'
}

tts-chunk​

Streaming TTS audio chunk.

{
  type: 'tts-chunk',
  format: string,     // 'pcm' | 'mp3' | 'ogg' | 'wav'
  sampleRate: number, // e.g., 24000
  data: string        // Base64-encoded audio data
}

Note: When WebRTC audio track is available, TTS audio is sent directly over the track instead of via tts-chunk messages.

tts​

Complete TTS audio (non-streaming mode).

{
  type: 'tts',
  format: string,     // 'mp3' | 'wav' | 'ogg'
  data: string        // Base64-encoded audio data
}

tts-complete​

TTS playback finished successfully.

{
  type: 'tts-complete'
}

tts-cancelled​

TTS playback was interrupted (user barge-in).

{
  type: 'tts-cancelled'
}

Speech Detection Messages​

speech-start​

VAD detected user started speaking.

{
  type: 'speech-start'
}

speech-end​

VAD detected user stopped speaking. Processing begins.

{
  type: 'speech-end'
}

Playbook Messages​

These messages are only sent when using playbook mode with tool calling.

tool-call-start​

Tool execution is starting.

{
  type: 'tool-call-start',
  name: string,                        // Tool function name
  callId: string,                      // Unique ID for correlation
  arguments: Record<string, unknown>   // Arguments passed to tool
}

tool-call-end​

Tool execution completed.

{
  type: 'tool-call-end',
  callId: string,          // Matches tool-call-start
  result?: unknown,        // Tool result (on success)
  error?: string,          // Error message (on failure)
  durationMs: number       // Execution time in milliseconds
}

stage-change​

Playbook transitioned to a new stage.

{
  type: 'stage-change',
  from: string,            // Previous stage ID
  to: string,              // New stage ID
  reason: string           // Why transition occurred
}

Error Messages​

error​

Server-side error occurred.

{
  type: 'error',
  code: ErrorCode,         // Structured error code
  message: string          // Human-readable description
}

Error Codes:

CodeDescription
WEBRTC_UNAVAILABLEWebRTC not supported on server
CONNECTION_FAILEDWebRTC connection failed
SESSION_NOT_FOUNDSession ID not found for reconnect
SESSION_EXPIREDSession expired (TTL exceeded)
STT_ERRORSpeech-to-text failed
STT_TIMEOUTSTT request timed out
LLM_ERRORLLM inference failed
LLM_TIMEOUTLLM request timed out
TTS_ERRORText-to-speech failed
TTS_TIMEOUTTTS request timed out
AUDIO_PROCESSING_ERRORAudio processing failed
VAD_ERRORVoice activity detection failed
INVALID_MESSAGEMalformed message received
INVALID_AUDIO_FORMATUnsupported audio format
TOOL_ERRORTool execution failed
PLAYBOOK_ERRORPlaybook execution error
INTERNAL_ERRORUnexpected server error
RATE_LIMITEDToo many requests

Type Definitions​

interface VisionAttachment {
  data: string;        // Base64 data URI or URL
  mimeType?: string;   // e.g., 'image/jpeg'
  alt?: string;        // Description for accessibility
}

interface RTCIceServer {
  urls: string | string[];
  username?: string;
  credential?: string;
}