Skip to main content

Building a voice AI agent on DialStack with ElevenLabs

Answer calls to a DialStack number with an agent you built in ElevenLabs. A small relay server sits between the two: DialStack streams the call's audio to it over the attach WebSocket, and the relay forwards it to ElevenLabs' agent WebSocket. No ElevenLabs phone number or SIP trunk is involved.

This is the BYO Receptionist pattern with ElevenLabs as the AI. The worked example in the BYO guide shows the whole relay in one short file. This page sets it up end to end with the runnable voice-ai-agent example in the DialStack SDK, started with --provider elevenlabs.

1. Set up the agent in ElevenLabs​

  1. In the ElevenLabs dashboard, create an agent under Agents. Give it its prompt, first message, voice and any tools there; the relay sends none of its own.
  2. In the agent's settings, set both the user input audio format and the TTS output audio format to μ-law 8000 Hz. Telephone audio is μ-law 8 kHz, so the relay then forwards it in both directions without transcoding.
  3. Copy the agent ID (agent_…).
  4. Create an API key under Developers → API Keys, with write access to Agents (Conversational AI). Read-only access isn't enough: fetching a signed URL needs the convai_write permission. The relay sends it as the xi-api-key header to fetch a signed URL for each call. The key stays on your server, so the agent can keep authentication enabled.

The relay sends no overrides, so none need to be enabled in the agent's Security tab.

2. Set up DialStack​

Create a Voice App whose webhook URL points at your relay, and route a number or dial-plan node to it. See Voice Apps and the Routing section of the BYO guide.

3. Run the relay​

Bash
git clone https://github.com/dialstack/dialstack-sdk.git
cd dialstack-sdk
npm install && npm run build   # the example links the local server package
cd examples/voice-ai-agent
npm install
cp .env.example .env

Set these in .env:

VariableValue
PUBLIC_URLThe public HTTPS URL of the relay (for example an ngrok URL).
DIALSTACK_API_KEYYour DialStack secret key.
VOICE_APP_WEBHOOK_SECRETThe signing secret of the Voice App.
ELEVENLABS_API_KEYThe ElevenLabs API key.
ELEVENLABS_AGENT_IDThe ElevenLabs agent ID.
TRANSFER_TARGETOptional. Where to transfer the caller; see below.

Then start it and call the number:

Bash
npm run dev -- --provider elevenlabs

Audio formats​

DialStack's media WebSocket carries μ-law at 8 kHz. The audio formats are agent settings, not per-call options, and ElevenLabs reports them in the conversation_initiation_metadata event when the conversation starts. The relay reads them there:

  • μ-law 8000 Hz (recommended): passed through unchanged.
  • PCM 16000 Hz (the ElevenLabs default): the relay converts between μ-law 8 kHz and PCM 16 kHz. It works, at some cost in audio quality.
  • Any other format: the relay logs unsupported agent audio format and ends the attachment.

Barge-in and hang-up​

  • When ElevenLabs interrupts the agent because the caller spoke (the interruption event), the relay drops the agent audio it has queued but not yet played, so the caller isn't talked over. If echo on the caller's line interrupts the agent too often, tune the agent's turn-taking settings in ElevenLabs.
  • The relay answers ElevenLabs' ping events with pong, which keeps the conversation alive.
  • When the caller hangs up, the relay closes the ElevenLabs socket. When ElevenLabs ends the conversation, the relay closes the DialStack media socket. That ends the attach, and DialStack runs the next action you chained after it, if any.

The caller's number​

DialStack's call.received webhook carries the caller's number (from_number) and the number they dialled (to_number). The relay keeps them from the webhook and sends them to ElevenLabs as the dynamic variables caller_number and called_number when the conversation starts. Reference them in the agent's prompt or first message, for example {{caller_number}}:

JavaScript
agent.send(
  JSON.stringify({
    type: 'conversation_initiation_client_data',
    dynamic_variables: { caller_number: event.from_number, called_number: event.to_number },
  })
);

Transfer to a human​

Hand the caller off with DialStack's transfer action, not with ElevenLabs' phone transfer tools: those act on calls ElevenLabs carries itself, and this call is carried by DialStack.

  1. On the agent in ElevenLabs, add a client tool named transfer_to_human, with a description telling the agent when to use it (for example, "Transfer the caller to a person when they ask for one").
  2. Set TRANSFER_TARGET in the relay's .env to an extension, an E.164 number or a sip: URI.

When the agent calls the tool, the relay sends the call a transfer, which replaces the running attach:

JavaScript
await dialstack.calls.update(
  callId,
  { actions: [{ type: 'transfer', target: process.env.TRANSFER_TARGET }] },
  { dialstackAccount: accountId }
);

DialStack closes the media socket, the relay closes the ElevenLabs conversation, and the caller is connected to the target. The relay answers the tool call only once DialStack has accepted the request. A rejected target is reported to the agent as an error, so it doesn't promise the caller a hand-off. See Replacing Actions.

Troubleshooting​

  • get-signed-url failed: 401: ELEVENLABS_API_KEY is wrong, or the key lacks Agents write access (missing the permission convai_write).
  • get-signed-url failed: 404: ELEVENLABS_AGENT_ID is wrong.
  • The agent says or shows tags like [friendly]: those are expressive tags from the agent's prompt or voice settings. Change them in ElevenLabs; the relay passes audio through untouched.
  • The socket closes before agent ready: the relay logs the close code and reason from ElevenLabs. Run with LOG_LEVEL=debug for every message type.
On this page