Skip to main content

Building a voice AI agent on DialStack with Telnyx

Answer calls to a DialStack number with an assistant you built in Telnyx. A small relay server sits between the two: DialStack streams the call's audio to it over the attach WebSocket, and the relay forwards it to Telnyx's assistant conversation WebSocket. No Telnyx phone number, SIP trunk or Call Control application is involved.

This is the BYO Receptionist pattern with Telnyx as the AI.

1. Set up the assistant in Telnyx​

  1. In the Telnyx Mission Control portal, create an AI Assistant. Give it its instructions, greeting, voice and any tools there; the relay sends none of its own.
  2. Copy the assistant ID (assistant-…).
  3. Create an API key (API v2) under API Keys. The relay sends it as Authorization: Bearer <key>.

2. Set up DialStack​

Create a Voice App whose webhook URL points at your relay, and route a number or dial-plan node to it. See Voice Apps and the Routing section of the BYO guide.

3. Write the relay​

The relay answers the webhook with an attach action, accepts DialStack's media WebSocket, and pipes audio between that socket and the assistant. Telnyx speaks PCM16, so the relay decodes caller μ-law, and downsamples and re-encodes the assistant's audio:

JavaScript
import http from 'node:http';
import express from 'express';
import WebSocket, { WebSocketServer } from 'ws';
import { mulaw } from 'alawmulaw';
import { DialStack, MediaStream } from '@dialstack/sdk-server';

const dialstack = new DialStack(process.env.DIALSTACK_API_KEY);
const app = express();

// 1. A call reached the Voice App: attach it to our media endpoint.
app.post('/webhook', express.raw({ type: 'application/json' }), async (req, res) => {
  const event = DialStack.webhooks.constructEvent(
    req.body,
    req.header('x-dialstack-signature'),
    process.env.VOICE_APP_WEBHOOK_SECRET
  );
  if (event.event === 'call.received') {
    await dialstack.calls.update(
      event.call_id,
      { actions: [{ type: 'attach', url: 'wss://relay.example.com/media' }] },
      { dialstackAccount: event.account_id }
    );
  }
  res.status(200).end();
});

// 2. DialStack opens the media WebSocket: bridge it to the Telnyx assistant.
const server = http.createServer(app);
new WebSocketServer({ server, path: '/media' }).on('connection', (ws) => {
  const call = new MediaStream(ws);
  const assistant = new WebSocket(
    `wss://api.telnyx.com/v2/ai/assistants/${process.env.TELNYX_ASSISTANT_ID}/conversation` +
      '?input_sample_rate=8000&input_format=pcm16&output_format=pcm16',
    { headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` } }
  );
  assistant.on('open', () => assistant.send(JSON.stringify({ type: 'session.update' })));

  let ratio; // assistant output rate / 8000, from session.created
  let queued = Buffer.alloc(0);

  // Agent audio goes out in 20 ms frames (160 bytes of μ-law).
  const pacer = setInterval(() => {
    if (queued.length < 160) return;
    call.sendAudio(queued.subarray(0, 160).toString('base64'));
    queued = queued.subarray(160);
  }, 20);

  // Caller audio: μ-law 8 kHz → PCM16 8 kHz, no resampling.
  call.on('audio', (frame) => {
    if (!ratio || assistant.readyState !== WebSocket.OPEN) return;
    const pcm = mulaw.decode(Buffer.from(frame.payload, 'base64'));
    const audio = Buffer.from(pcm.buffer, pcm.byteOffset, pcm.byteLength).toString('base64');
    assistant.send(JSON.stringify({ type: 'input_audio_buffer.append', audio }));
  });

  assistant.on('message', (data) => {
    const msg = JSON.parse(data.toString());
    if (msg.type === 'session.created') {
      ratio = msg.session.audio.output.format.rate / 8000; // set by the assistant's voice
    } else if (msg.type === 'response.output_audio.delta') {
      const bytes = Buffer.from(msg.delta, 'base64');
      const pcm = new Int16Array(new Uint8Array(bytes).buffer, 0, bytes.length >> 1); // copy: aligned
      // Average each window down to 8 kHz. Integer ratios only (16, 24, 48 kHz).
      const out = new Int16Array(Math.floor(pcm.length / ratio));
      for (let i = 0; i < out.length; i++) {
        let sum = 0;
        for (let j = 0; j < ratio; j++) sum += pcm[i * ratio + j];
        out[i] = sum / ratio;
      }
      queued = Buffer.concat([queued, Buffer.from(mulaw.encode(out))]);
    } else if (msg.type === 'input_audio_buffer.speech_started') {
      queued = Buffer.alloc(0); // the caller barged in: drop what the agent hadn't said yet
    }
  });

  const hangUp = () => {
    clearInterval(pacer);
    assistant.close();
    call.close();
  };
  call.on('close', hangUp);
  assistant.on('close', hangUp);
});

server.listen(8080);

The voice-ai-agent example in the DialStack SDK is the same relay, with error handling, drift-free pacing and logging. Its downsampler also handles non-integer rates such as 22.05 kHz, and chunks that end mid-window. To run it:

Bash
git clone https://github.com/dialstack/dialstack-sdk.git
cd dialstack-sdk
npm install && npm run build   # the example links the local server package
cd examples/voice-ai-agent
npm install
cp .env.example .env

Set these in .env:

VariableValue
PUBLIC_URLThe public HTTPS URL of the relay (for example an ngrok URL).
DIALSTACK_API_KEYYour DialStack secret key.
VOICE_APP_WEBHOOK_SECRETThe signing secret of the Voice App.
TELNYX_API_KEYThe Telnyx API v2 key.
TELNYX_ASSISTANT_IDThe Telnyx assistant ID.
TRANSFER_TARGETOptional. Where to transfer the caller; see below.

Then start it and call the number:

Bash
npm run dev -- --provider telnyx

Audio formats​

DialStack's media WebSocket carries μ-law at 8 kHz. Telnyx takes and returns PCM16 only, so the relay converts:

  • Caller → Telnyx. The relay connects with input_sample_rate=8000 and decodes μ-law to PCM16. No resampling.
  • Telnyx → caller. The output rate is chosen by the assistant's voice, not by the client, and Telnyx reports it in the session.created event (session.audio.output.format.rate). The relay reads it there and downsamples to 8 kHz before encoding μ-law. Changing the assistant's voice needs no relay change.

Barge-in and hang-up​

  • When Telnyx detects the caller speaking (input_audio_buffer.speech_started), the relay drops the assistant audio it has queued but not yet played, so the caller isn't talked over.
  • When the caller hangs up, the relay closes the Telnyx socket. When Telnyx ends the conversation, the relay closes the DialStack media socket. That ends the attach, and DialStack runs the next action you chained after it, if any.

The caller's number​

DialStack's call.received webhook carries the caller's number (from_number) and the number they dialled (to_number). The relay keeps them from the webhook and sends them to Telnyx as the dynamic variables caller_number and called_number, in the session.update that opens the conversation. Reference them in the assistant's instructions or greeting, for example {{caller_number}}:

JavaScript
assistant.send(
  JSON.stringify({
    type: 'session.update',
    session: {
      assistant: {
        dynamic_variables: { caller_number: event.from_number, called_number: event.to_number },
      },
    },
  })
);

session.update must be the first frame, before any audio: Telnyx rejects one sent after the session has started.

Transfer to a human​

Hand the caller off with DialStack's transfer action.

  1. On the assistant in Telnyx, add a client-side tool named transfer_to_human, with a description telling the assistant when to use it (for example, "Transfer the caller to a person when they ask for one"). The relay ignores its arguments. Telnyx requires a required array in the schema, so either give it an empty one through the API ({"type": "object", "properties": {}, "required": []}) or, in the portal, add one optional string parameter such as reason. If the assistant also has a built-in Transfer tool, remove it or tell the assistant to use transfer_to_human instead: Telnyx doesn't carry this call, so its own transfer can't move it.
  2. Set TRANSFER_TARGET in the relay's .env to an extension, an E.164 number or a sip: URI.

When the assistant calls the tool, Telnyx sends the relay a conversation.item.created event whose item is a function_call. The relay sends the call a transfer, which replaces the running attach:

JavaScript
await dialstack.calls.update(
  callId,
  { actions: [{ type: 'transfer', target: process.env.TRANSFER_TARGET }] },
  { dialstackAccount: accountId }
);

DialStack closes the media socket, the relay closes the Telnyx conversation, and the caller is connected to the target. The relay answers the tool call (a function_call_output item) only once DialStack has accepted the request. A rejected target is reported to the assistant as an error, so it doesn't promise the caller a hand-off. See Replacing Actions.

On this page