Skip to content
Twilio ConversationRelay Voice Agents

Twilio ConversationRelay Voice Agents Skill

Build real-time AI voice agents on Twilio ConversationRelay — TwiML setup, the WebSocket message types, streaming LLM responses back for playback, and the voice-provider configuration that goes with them.

GitHub Docs

Install

Install with the skills CLI

One command for Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf and dozens of other agents. Needs Node.js; the skill is fetched straight from twilio/ai.

$ npx skills add twilio/ai --skill twilio-voice-conversation-relay

Run it in a terminal. The CLI finds the agents installed on your machine, asks which ones to add the skill to, and whether to install it into the folder you ran it from or for all your projects.

Other ways to install

  1. Open the skill on GitHub and download the folder that holds its SKILL.md.
  2. Put it at ~/.claude/skills/twilio-voice-conversation-relay/, where Claude Code picks it up in every project.

No scripts or references next to it? Saving the SKILL.md alone as ~/.claude/skills/twilio-voice-conversation-relay/SKILL.md works too.

Using another agent? Its personal skills folder
Codex
~/.agents/skills/
Cursor
~/.cursor/skills/
GitHub Copilot
~/.copilot/skills/
Gemini CLI
~/.gemini/skills/
Windsurf
~/.codeium/windsurf/skills/
OpenCode
~/.config/opencode/skills/

About

  • TwiML setup for starting a ConversationRelay session
  • Full WebSocket message-type reference with handling patterns
  • Streaming LLM responses back for low-latency playback
  • Explicit CANNOT section documenting platform limits and pitfalls

ConversationRelay is Twilio's bridge between a phone call and your own application: Twilio runs speech recognition and text-to-speech, your server holds a persistent WebSocket, receives transcripts, calls whichever LLM you like, and streams text back for Twilio to speak. The loop is caller → Twilio (ASR/TTS) → WebSocket → your app → LLM.

This skill covers that loop end to end — the TwiML that starts a relay, every WebSocket message type and what to do with it, how to stream partial LLM output so the caller is not left in silence, and how to configure voice providers. It also carries an explicit CANNOT section, which is the part that saves time: hard constraints and known pitfalls written down so the agent stops inventing capabilities the platform does not have.

Practical prerequisites it enforces before generating code: an upgraded Twilio account with ConversationRelay access (onboarding is not instant — it starts in the Console under Voice > ConversationRelay), a voice-capable Twilio number, account SID and auth token, a WebSocket server reachable over wss:// with TLS, and an LLM integration.

Part of Twilio's official Skills set, MIT-licensed in the twilio/ai repository. Twilio Skills are a Public Beta product and not covered by Twilio's support terms or SLA.

SKILL.md

The instructions your agent loads, as published by Twilio.

SKILL.md
name
twilio-voice-conversation-relay
description
Build AI-powered voice agents using Twilio ConversationRelay. Handles real-time speech recognition (ASR), text-to-speech (TTS), and bidirectional audio streaming via WebSocket. Covers TwiML setup, WebSocket message types, LLM integration, streaming responses, and voice provider configuration. Use this skill to build voice bots, IVR replacements, or real-time AI voice assistants on Twilio calls.
Overview

ConversationRelay connects Twilio's telephony layer to your app via a persistent WebSocket. Twilio handles ASR (speech-to-text) and TTS (text-to-speech); your app receives transcripts, calls an LLM, and sends text back for playback.

Caller ←→ Twilio (ASR/TTS) ←→ WebSocket ←→ Your App ←→ LLM

---

Prerequisites
  • Upgraded Twilio account with ConversationRelay access (requires onboarding)

— New to Twilio? See twilio-account-setup
— Start onboarding at: Console > Voice > ConversationRelay — access is not instant

  • A voice-capable Twilio phone number
  • TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN — see twilio-iam-auth-setup
  • WebSocket server reachable via wss:// (TLS required)
  • An LLM integration (OpenAI, Anthropic, etc.)
  • For placing calls: see twilio-voice-outbound-calls

Onboarding: Complete via Console > Voice > ConversationRelay > Onboarding. Select TTS/ASR providers:

  • TTS: Deepgram, Amazon Polly, Google Cloud TTS, ElevenLabs
  • ASR: Deepgram, Google Cloud STT

---

Quickstart

Step 1 — Return TwiML pointing to your WebSocket server

Python (Flask)

from flask import Flask
from twilio.twiml.voice_response import VoiceResponse, Connect, ConversationRelay

app = Flask(__name__)

@app.route("/voice", methods=["POST"])
def voice():
    response = VoiceResponse()
    connect = Connect()
    connect.conversation_relay(
        url="wss://yourapp.com/ws/voice",
        welcome_greeting="Hello! How can I help you today?"
    )
    response.append(connect)
    return str(response)

Node.js (Express)

const { VoiceResponse } = require("twilio").twiml;

app.post("/voice", (req, res) => {
    const response = new VoiceResponse();
    const connect = response.connect();
    connect.conversationRelay({
        url: "wss://yourapp.com/ws/voice",
        welcomeGreeting: "Hello! How can I help you today?",
    });
    res.type("text/xml").send(response.toString());
});

Step 2 — Handle WebSocket events and respond with text

Python (websockets)

import asyncio, json, websockets

async def handle_call(websocket):
    async for message in websocket:
        event = json.loads(message)
        if event["type"] == "prompt":
            ai_response = await call_llm(event["voicePrompt"])
            await websocket.send(json.dumps({"type": "text", "token": ai_response, "last": True}))

async def main():
    async with websockets.serve(handle_call, "0.0.0.0", 8080):
        await asyncio.Future()

asyncio.run(main())

Node.js (ws)

const WebSocket = require("ws");
const wss = new WebSocket.Server({ port: 8080 });

wss.on("connection", (ws) => {
    ws.on("message", async (data) => {
        const event = JSON.parse(data);
        if (event.type === "prompt") {
            const aiResponse = await callLLM(event.voicePrompt);
            ws.send(JSON.stringify({ type: "text", token: aiResponse, last: true }));
        }
    });
});
Security: The voicePrompt field contains ASR-transcribed caller speech — it is untrusted external input. When passing to an LLM, isolate it as user input within a structured system prompt. Implement topic boundaries and output filtering to prevent the LLM from disclosing system instructions or speaking inappropriate content. ConversationRelay is a pure transport layer with no built-in content safety — any LLM output is spoken to the caller verbatim.

---

Key Patterns
WebSocket Message Types

Received from Twilio:

TypeWhenKey fields
connectedWebSocket openedcallSid, streamSid
promptUser finished speakingvoicePrompt (transcript)
interruptUser interrupted TTS—
dtmfUser pressed keypad keydigit
errorAn error occurreddescription

Sent to Twilio:

TypePurposeKey fields
textSend TTS responsetoken (text), last (bool)
interruptStop current TTS—
endHang up the callreason
Stream LLM Responses Token-by-Token

Lower latency by streaming as the LLM generates output — Twilio starts speaking before the full response is ready.

Python

async for chunk in llm_stream:
    await websocket.send(json.dumps({"type": "text", "token": chunk, "last": False}))
await websocket.send(json.dumps({"type": "text", "token": "", "last": True}))

Node.js

for await (const chunk of llmStream) {
    ws.send(JSON.stringify({ type: "text", token: chunk, last: false }));
}
ws.send(JSON.stringify({ type: "text", token: "", last: true }));
Voice Configuration

Python

connect.conversation_relay(
    url="wss://yourapp.com/ws/voice",
    voice="en-US-Neural2-F",
    language="en-US",
    transcription_provider="deepgram",
    speech_model="nova-2-phonecall",
    interrupt_by_dtmf=True,
)

Node.js

connect.conversationRelay({
    url: "wss://yourapp.com/ws/voice",
    voice: "en-US-Neural2-F",
    language: "en-US",
    transcriptionProvider: "deepgram",
    speechModel: "nova-2-phonecall",
    interruptByDtmf: true,
});

---

CANNOT
  • No raw audio access — Text in, text out only. For raw audio, use <Connect><Stream> (Media Streams).
  • Cannot mix with Media Streams — <Connect><Stream> and <Connect><ConversationRelay> are mutually exclusive on the same call. No error — one is silently ignored.
  • No custom STT/TTS engines — Limited to Deepgram + Google (STT) and Deepgram + Amazon Polly + Google + ElevenLabs (TTS).
  • No mid-session voice/provider changes — Voice and provider are set at TwiML time. Only language can be switched mid-session via WebSocket message.
  • No WebSocket auto-reconnection — If the WebSocket drops, the call disconnects. Implement recovery via <Connect action> URL.
  • Voice only — No SMS/messaging support. For omnichannel, use Conversation Orchestrator.
  • No built-in memory or context — BYO conversation history and context management.
  • No LLM integration — Pure transport layer. You bring your own LLM via the WebSocket server.
  • No server-side recording via REST API — record:true on the Calls API is silently ignored. Must use <Start><Recording> before <Connect> in TwiML.
  • Not PCI compliant with Voice Intelligence v2 — Do not enable intelligenceService in PCI workflows.
  • ElevenLabs requires account enablement — Accounts without ElevenLabs access get error 64101. Voice IDs (not human-readable names) are required.
  • Cannot use ConversationRelay without onboarding — Not available immediately on a new account
  • Cannot use non-TLS WebSocket — Server must be reachable via wss:// (TLS required)
  • Cannot stream audio without `last: true` — Twilio won't play audio until it sees a last: true token in the stream

---

Next Steps
  • Place outbound calls: twilio-voice-outbound-calls
  • Standard IVR without AI: twilio-voice-twiml
Install

Planning skill that interrogates your use case before any code is written, then recommends the right Twilio Conversations stack — ConversationRelay, Memory, Intelligence, Orchestrator, TaskRouter — and the implementation skills to follow.

71 views

Harden a Twilio integration against credential leaks and fraud — API keys instead of auth tokens, webhook signature validation, geo-permissions, SMS pumping prevention and zero-downtime credential rotation.

Browse all skills →