# AssemblyAI Documentation

> AssemblyAI provides production-ready Voice AI models for speech-to-text, speaker detection, sentiment analysis, and more. Build with pre-recorded or streaming audio using REST APIs and WebSockets.

Generated from the local Mintlify documentation source. Content inside `<llms-ignore>` is excluded, and content inside `<llms-only>` is included.

## Contents

- [AssemblyAI Documentation](#assemblyai-documentation-docs-index-mdx)
- [Build with AI coding agents](#build-with-ai-coding-agents-docs-coding-agent-prompts-mdx)
- [Models](#models-docs-getting-started-models-mdx)
- [Evaluations](#evaluations-docs-evaluations-mdx)
- [Account Management](#account-management-docs-account-management-mdx)
- [Single Sign-On (SSO)](#single-sign-on-sso-docs-sso-mdx)
- [Billing and Pricing](#billing-and-pricing-docs-billing-and-pricing-mdx)
- [Introducing Universal-3.5 Pro](#introducing-universal-3-5-pro-docs-getting-started-universal-3-5-pro-mdx)
- [End-to-end examples](#end-to-end-examples-docs-getting-started-end-to-end-examples-mdx)
- [Meeting notetaker](#meeting-notetaker-docs-getting-started-end-to-end-examples-meeting-notetaker-mdx)
- [Sales call intelligence](#sales-call-intelligence-docs-getting-started-end-to-end-examples-sales-call-intelligence-mdx)
- [Medical scribe](#medical-scribe-docs-getting-started-end-to-end-examples-medical-scribe-mdx)
- [Content repurposing](#content-repurposing-docs-getting-started-end-to-end-examples-content-repurposing-mdx)
- [Real-time meeting assistant](#real-time-meeting-assistant-docs-getting-started-end-to-end-examples-real-time-meeting-assistant-mdx)
- [Real-time live captioner](#real-time-live-captioner-docs-getting-started-end-to-end-examples-real-time-live-captioner-mdx)
- [Best Practices for building Meeting Notetakers](#best-practices-for-building-meeting-notetakers-docs-meeting-notetaker-best-practices-mdx)
- [Build a medical scribe](#build-a-medical-scribe-docs-medical-scribe-best-practices-mdx)
- [Build a Post-Visit Medical Scribe](#build-a-post-visit-medical-scribe-docs-medical-scribe-best-practices-medical-scribe-post-visit-mdx)
- [Build a Real-Time Medical Scribe](#build-a-real-time-medical-scribe-docs-medical-scribe-best-practices-medical-scribe-real-time-mdx)
- [Best Practices for Building Voice Agents](#best-practices-for-building-voice-agents-docs-voice-agents-best-practices-mdx)
- [Best Practices for building Contact Center Applications](#best-practices-for-building-contact-center-applications-docs-contact-center-best-practices-mdx)
- [Integrations](#integrations-docs-integrations-mdx)
- [Universal 3.5 Pro Realtime on LiveKit](#universal-3-5-pro-realtime-on-livekit-docs-voice-agents-livekit-universal-3-5-pro-mdx)
- [Universal 3.5 Pro Realtime on Pipecat](#universal-3-5-pro-realtime-on-pipecat-docs-voice-agents-pipecat-universal-3-5-pro-mdx)
- [Zapier Integration with AssemblyAI](#zapier-integration-with-assemblyai-docs-integrations-zapier-mdx)
- [Integrate Make with AssemblyAI](#integrate-make-with-assemblyai-docs-integrations-make-mdx)
- [The AssemblyAI n8n Integration](#the-assemblyai-n8n-integration-docs-integrations-n-8-n-mdx)
- [The Postman collection for the AssemblyAI API](#the-postman-collection-for-the-assemblyai-api-docs-integrations-postman-mdx)
- [Build a Zoom Real-time transcription bot with Recall.ai](#build-a-zoom-real-time-transcription-bot-with-recall-ai-docs-integrations-recall-mdx)
- [Transcribe Your Zoom Meetings](#transcribe-your-zoom-meetings-docs-integrations-zoom-rtms-mdx)
- [Integrate Telnyx with AssemblyAI](#integrate-telnyx-with-assemblyai-docs-integrations-telnyx-mdx)
- [Integrate Twilio with AssemblyAI](#integrate-twilio-with-assemblyai-docs-integrations-twilio-mdx)
- [Transcribe Your Amazon Connect Recordings](#transcribe-your-amazon-connect-recordings-docs-integrations-amazon-connect-mdx)
- [Transcribe Genesys Cloud Recordings with AssemblyAI](#transcribe-genesys-cloud-recordings-with-assemblyai-docs-integrations-genesys-cloud-mdx)
- [\U0001F99C️\U0001F517 LangChain Integration with AssemblyAI](#u0001f99c-u0001f517-langchain-integration-with-assemblyai-docs-integrations-langchain-mdx)
- [\U0001F99C️\U0001F517 LangChain Python Integration with AssemblyAI](#u0001f99c-u0001f517-langchain-python-integration-with-assemblyai-docs-integrations-langchain-python-mdx)
- [\U0001F99C️\U0001F517 LangChain JavaScript Integration with AssemblyAI](#u0001f99c-u0001f517-langchain-javascript-integration-with-assemblyai-docs-integrations-langchain-js-mdx)
- [▲ Vercel AI SDK Integration with AssemblyAI](#vercel-ai-sdk-integration-with-assemblyai-docs-integrations-vercel-ai-sdk-mdx)
- [Integrate Power Automate with AssemblyAI](#integrate-power-automate-with-assemblyai-docs-integrations-power-automate-mdx)
- [Semantic Kernel Integration for AssemblyAI](#semantic-kernel-integration-for-assemblyai-docs-integrations-semantic-kernel-mdx)
- [Integrate Activepieces with AssemblyAI](#integrate-activepieces-with-assemblyai-docs-integrations-activepieces-mdx)
- [Haystack Integration for AssemblyAI](#haystack-integration-for-assemblyai-docs-integrations-haystack-mdx)
- [Cloudflare](#cloudflare-docs-nav-links-cloudflare-mdx)
- [Relay.app](#relay-app-docs-nav-links-relay-app-mdx)
- [Bubble by Knowcode](#bubble-by-knowcode-docs-nav-links-bubble-by-knowcode-mdx)
- [Pipedream](#pipedream-docs-nav-links-pipedream-mdx)
- [Drupal](#drupal-docs-nav-links-drupal-mdx)
- [Trust center](#trust-center-docs-nav-links-trust-center-mdx)
- [Security overview](#security-overview-docs-nav-links-security-overview-mdx)
- [Data Controls](#data-controls-docs-data-controls-mdx)
- [Data retention and model training](#data-retention-and-model-training-docs-data-retention-and-model-training-mdx)
- [API Reference](#api-reference-docs-api-reference-overview-mdx)
- [Quickstart](#quickstart-docs-pre-recorded-audio-getting-started-transcribe-an-audio-file-mdx)
- [Model selection](#model-selection-docs-pre-recorded-audio-select-the-speech-model-mdx)
- [Prompting and Keyterms](#prompting-and-keyterms-docs-pre-recorded-audio-universal-3-5-pro-prompting-mdx)
- [Medical Mode](#medical-mode-docs-pre-recorded-audio-medical-mode-mdx)
- [Speaker Diarization](#speaker-diarization-docs-pre-recorded-audio-label-speakers-mdx)
- [Multichannel Transcription](#multichannel-transcription-docs-pre-recorded-audio-transcribe-multiple-audio-channels-mdx)
- [Code Switching](#code-switching-docs-pre-recorded-audio-code-switching-mdx)
- [Automatic Language Detection](#automatic-language-detection-docs-pre-recorded-audio-language-detection-mdx)
- [Set Language Manually](#set-language-manually-docs-pre-recorded-audio-set-language-manually-mdx)
- [Supported Languages](#supported-languages-docs-pre-recorded-audio-supported-languages-mdx)
- [Custom Spelling](#custom-spelling-docs-pre-recorded-audio-correct-spelling-of-terms-mdx)
- [Filler Words](#filler-words-docs-pre-recorded-audio-include-filler-words-mdx)
- [Word Search](#word-search-docs-pre-recorded-audio-search-for-words-in-transcript-mdx)
- [Set the Start and End of the Transcript](#set-the-start-and-end-of-the-transcript-docs-pre-recorded-audio-set-the-start-and-end-of-the-transcript-mdx)
- [Transcript Status](#transcript-status-docs-pre-recorded-audio-check-transcript-status-mdx)
- [Transcript export options](#transcript-export-options-docs-pre-recorded-audio-export-transcripts-as-srt-vtt-or-text-mdx)
- [Delete Transcripts](#delete-transcripts-docs-pre-recorded-audio-delete-transcripts-mdx)
- [Upload a media file](#upload-a-media-file-docs-pre-recorded-audio-api-reference-files-upload-mdx)
- [Submit a transcript](#submit-a-transcript-docs-pre-recorded-audio-api-reference-transcripts-submit-mdx)
- [Get a transcript](#get-a-transcript-docs-pre-recorded-audio-api-reference-transcripts-get-mdx)
- [Get transcript sentences](#get-transcript-sentences-docs-pre-recorded-audio-api-reference-transcripts-get-sentences-mdx)
- [Get transcript paragraphs](#get-transcript-paragraphs-docs-pre-recorded-audio-api-reference-transcripts-get-paragraphs-mdx)
- [Get subtitles](#get-subtitles-docs-pre-recorded-audio-api-reference-transcripts-get-subtitles-mdx)
- [Get redacted audio](#get-redacted-audio-docs-pre-recorded-audio-api-reference-transcripts-get-redacted-audio-mdx)
- [Word search](#word-search-docs-pre-recorded-audio-api-reference-transcripts-word-search-mdx)
- [List transcripts](#list-transcripts-docs-pre-recorded-audio-api-reference-transcripts-list-mdx)
- [Delete a transcript](#delete-a-transcript-docs-pre-recorded-audio-api-reference-transcripts-delete-mdx)
- [Webhooks for pre-recorded audio](#webhooks-for-pre-recorded-audio-docs-pre-recorded-audio-webhooks-mdx)
- [Benchmarks](#benchmarks-docs-pre-recorded-audio-benchmarks-mdx)
- [Evaluating Pre-recorded STT models](#evaluating-pre-recorded-stt-models-docs-pre-recorded-audio-evaluations-mdx)
- [Cloud Endpoints and Data Residency](#cloud-endpoints-and-data-residency-docs-pre-recorded-audio-select-the-region-mdx)
- [Rate limits](#rate-limits-docs-pre-recorded-audio-rate-limits-mdx)
- [Migration guide: Deepgram to AssemblyAI](#migration-guide-deepgram-to-assemblyai-docs-pre-recorded-audio-migration-guides-dg-to-aai-mdx)
- [Migration guide: OpenAI to AssemblyAI](#migration-guide-openai-to-assemblyai-docs-pre-recorded-audio-migration-guides-oai-to-aai-mdx)
- [Migration guide: AWS Transcribe to AssemblyAI](#migration-guide-aws-transcribe-to-assemblyai-docs-pre-recorded-audio-migration-guides-aws-to-aai-mdx)
- [Migration guide: Google Speech-to-Text to AssemblyAI](#migration-guide-google-speech-to-text-to-assemblyai-docs-pre-recorded-audio-migration-guides-google-to-aai-mdx)
- [Migration guide: Gladia to AssemblyAI](#migration-guide-gladia-to-assemblyai-docs-pre-recorded-audio-migration-guides-gladia-to-aai-mdx)
- [Best Practices for building Meeting Notetakers](#best-practices-for-building-meeting-notetakers-docs-meeting-notetaker-best-practices-mdx)
- [Build a medical scribe](#build-a-medical-scribe-docs-medical-scribe-best-practices-mdx)
- [Best Practices for building Contact Center Applications](#best-practices-for-building-contact-center-applications-docs-contact-center-best-practices-mdx)
- [Get YouTube Video Transcripts with yt-dlp](#get-youtube-video-transcripts-with-yt-dlp-docs-pre-recorded-audio-guides-transcribe-youtube-videos-mdx)
- [Build a UI for Transcription with Gradio and Python](#build-a-ui-for-transcription-with-gradio-and-python-docs-pre-recorded-audio-guides-gradio-frontend-mdx)
- [Detect Low Confidence Words in a Transcript](#detect-low-confidence-words-in-a-transcript-docs-pre-recorded-audio-guides-detecting-low-confidence-words-mdx)
- [Running Bulk Transcription and Load Tests at Scale](#running-bulk-transcription-and-load-tests-at-scale-docs-pre-recorded-audio-guides-bulk-transcription-and-load-tests-at-scale-mdx)
- [Transcribe Multiple Files Simultaneously Using the JavaScript SDK](#transcribe-multiple-files-simultaneously-using-the-javascript-sdk-docs-pre-recorded-audio-guides-sdk-node-batch-mdx)
- [Transcribe Multiple Files Simultaneously Using the Python SDK](#transcribe-multiple-files-simultaneously-using-the-python-sdk-docs-pre-recorded-audio-guides-batch-transcription-mdx)
- [Transcribe from an S3 Bucket](#transcribe-from-an-s3-bucket-docs-pre-recorded-audio-guides-transcribe-from-s3-mdx)
- [Transcribe Google Drive Files](#transcribe-google-drive-files-docs-pre-recorded-audio-guides-transcribing-google-drive-file-mdx)
- [Transcribe GitHub Files](#transcribe-github-files-docs-pre-recorded-audio-guides-transcribing-github-files-mdx)
- [Iterate over Speaker Labels with Make.com](#iterate-over-speaker-labels-with-make-com-docs-pre-recorded-audio-guides-make-speaker-labels-mdx)
- [Calculate the Talk / Listen Ratio of Speakers](#calculate-the-talk-listen-ratio-of-speakers-docs-pre-recorded-audio-guides-talk-listen-ratio-mdx)
- [Plot A Speaker Timeline with Matplotlib](#plot-a-speaker-timeline-with-matplotlib-docs-pre-recorded-audio-guides-speaker-timeline-mdx)
- [Generate Custom Speaker Labels with Pyannote](#generate-custom-speaker-labels-with-pyannote-docs-pre-recorded-audio-guides-use-assemblyai-with-pyannote-to-generate-custom-speaker-labels-mdx)
- [Use Speaker Diarization with Async Chunking](#use-speaker-diarization-with-async-chunking-docs-pre-recorded-audio-guides-speaker-diarization-with-async-chunking-mdx)
- [Setup A Speaker Identification System using Pinecone & Nvidia TitaNet](#setup-a-speaker-identification-system-using-pinecone-nvidia-titanet-docs-pre-recorded-audio-guides-titanet-speaker-identification-mdx)
- [Use Automatic Language Detection as a Separate Step From Transcription](#use-automatic-language-detection-as-a-separate-step-from-transcription-docs-pre-recorded-audio-guides-automatic-language-detection-separate-mdx)
- [Route to Default Language if Language Confidence is Low](#route-to-default-language-if-language-confidence-is-low-docs-pre-recorded-audio-guides-automatic-language-detection-route-default-language-mdx)
- [Create Custom Length Subtitles](#create-custom-length-subtitles-docs-pre-recorded-audio-guides-subtitle-creation-by-word-count-mdx)
- [Create Subtitles with Speaker Labels](#create-subtitles-with-speaker-labels-docs-pre-recorded-audio-guides-speaker-labelled-subtitles-mdx)
- [Generate Subtitles for Videos](#generate-subtitles-for-videos-docs-pre-recorded-audio-guides-subtitles-mdx)
- [Translate an AssemblyAI Subtitle Transcript](#translate-an-assemblyai-subtitle-transcript-docs-pre-recorded-audio-guides-translate-subtitles-mdx)
- [Schedule a DELETE request with AssemblyAI and EasyCron](#schedule-a-delete-request-with-assemblyai-and-easycron-docs-pre-recorded-audio-guides-schedule-delete-mdx)
- [Troubleshoot Common Errors](#troubleshoot-common-errors-docs-pre-recorded-audio-guides-common-errors-and-solutions-mdx)
- [Implement Retry Server Error Logic](#implement-retry-server-error-logic-docs-pre-recorded-audio-guides-retry-server-error-mdx)
- [Implement Retry Upload Error Logic](#implement-retry-upload-error-logic-docs-pre-recorded-audio-guides-retry-upload-error-mdx)
- [Identify Duplicate Dual Channel Files](#identify-duplicate-dual-channel-files-docs-pre-recorded-audio-guides-identify-duplicate-channels-mdx)
- [Correct Audio Duration Discrepancies with Multi-Tool Validation and Transcoding](#correct-audio-duration-discrepancies-with-multi-tool-validation-and-transcoding-docs-pre-recorded-audio-guides-audio-duration-fix-mdx)
- [Audio File Downsampling Recommendations and Best Practices](#audio-file-downsampling-recommendations-and-best-practices-docs-pre-recorded-audio-guides-downsampling-mdx)
- [Translate an AssemblyAI Subtitle Transcript](#translate-an-assemblyai-subtitle-transcript-docs-pre-recorded-audio-guides-translate-subtitles-mdx)
- [Translate AssemblyAI Transcripts Into Other Languages Using Commercial Models](#translate-assemblyai-transcripts-into-other-languages-using-commercial-models-docs-pre-recorded-audio-guides-translate-transcripts-mdx)
- [Transform Chinese transcripts into Simplified or Traditional Text](#transform-chinese-transcripts-into-simplified-or-traditional-text-docs-pre-recorded-audio-guides-traditional-simplified-chinese-mdx)
- [Do More With Our SDKs](#do-more-with-our-sdks-docs-pre-recorded-audio-guides-do-more-with-sdk-mdx)
- [Quickstart](#quickstart-docs-streaming-getting-started-transcribe-streaming-audio-mdx)
- [Model selection](#model-selection-docs-streaming-select-the-speech-model-mdx)
- [Optimizing Accuracy and Latency](#optimizing-accuracy-and-latency-docs-streaming-getting-started-optimizing-accuracy-and-latency-mdx)
- [Turn Detection](#turn-detection-docs-streaming-turn-detection-mdx)
- [Prompting and Keyterms](#prompting-and-keyterms-docs-streaming-prompting-and-keyterms-mdx)
- [Conversation Context](#conversation-context-docs-streaming-universal-3-5-pro-context-carryover-mdx)
- [Multilingual Transcription](#multilingual-transcription-docs-streaming-multilingual-transcription-mdx)
- [Medical Mode](#medical-mode-docs-streaming-medical-mode-mdx)
- [Voice Focus](#voice-focus-docs-streaming-voice-focus-mdx)
- [Streaming Diarization and Multichannel](#streaming-diarization-and-multichannel-docs-streaming-label-speakers-and-separate-channels-mdx)
- [PII Redaction](#pii-redaction-docs-streaming-pii-redaction-mdx)
- [Filter profanity](#filter-profanity-docs-streaming-filter-profanity-from-transcripts-mdx)
- [Message Sequence](#message-sequence-docs-streaming-message-sequence-mdx)
- [Streaming WebSocket API](#streaming-websocket-api-docs-streaming-api-spec-streaming-websocket-mdx)
- [Generate streaming token](#generate-streaming-token-docs-streaming-api-spec-generate-streaming-token-mdx)
- [Authenticate with a temporary token](#authenticate-with-a-temporary-token-docs-streaming-authenticate-with-a-temporary-token-mdx)
- [Webhooks for streaming speech-to-text](#webhooks-for-streaming-speech-to-text-docs-streaming-webhooks-mdx)
- [Updating Configuration Mid-Stream](#updating-configuration-mid-stream-docs-streaming-updating-configuration-mid-stream-mdx)
- [Common session errors and closures](#common-session-errors-and-closures-docs-streaming-common-session-errors-and-closures-mdx)
- [Benchmarks](#benchmarks-docs-streaming-benchmarks-mdx)
- [Evaluating Real-time STT models for Voice Agents](#evaluating-real-time-stt-models-for-voice-agents-docs-streaming-evaluations-voice-agents-mdx)
- [Streaming Endpoints and Data Zones](#streaming-endpoints-and-data-zones-docs-streaming-endpoints-and-data-zones-mdx)
- [Rate limits](#rate-limits-docs-streaming-rate-limits-mdx)
- [Self-Hosted Streaming](#self-hosted-streaming-docs-streaming-self-hosted-streaming-mdx)
- [Universal 3.5 Pro Realtime on LiveKit](#universal-3-5-pro-realtime-on-livekit-docs-voice-agents-livekit-universal-3-5-pro-mdx)
- [Universal 3.5 Pro Realtime on Pipecat](#universal-3-5-pro-realtime-on-pipecat-docs-voice-agents-pipecat-universal-3-5-pro-mdx)
- [Streaming Migration Guide: Universal Streaming to Universal-3.5 Pro Streaming](#streaming-migration-guide-universal-streaming-to-universal-3-5-pro-streaming-docs-streaming-migration-guides-universal-to-universal-3-5-pro-streaming-mdx)
- [Streaming Migration Guide: Deepgram to AssemblyAI](#streaming-migration-guide-deepgram-to-assemblyai-docs-streaming-migration-guides-dg-to-aai-streaming-mdx)
- [Migration guide: Speechmatics to AssemblyAI](#migration-guide-speechmatics-to-assemblyai-docs-streaming-migration-guides-speechmatics-to-aai-streaming-mdx)
- [Streaming Migration Guide: Gladia to AssemblyAI](#streaming-migration-guide-gladia-to-assemblyai-docs-streaming-migration-guides-gladia-to-aai-streaming-mdx)
- [Best Practices for building Meeting Notetakers](#best-practices-for-building-meeting-notetakers-docs-meeting-notetaker-best-practices-mdx)
- [Build a medical scribe](#build-a-medical-scribe-docs-medical-scribe-best-practices-mdx)
- [Best Practices for Building Voice Agents](#best-practices-for-building-voice-agents-docs-voice-agents-best-practices-mdx)
- [Best Practices for building Contact Center Applications](#best-practices-for-building-contact-center-applications-docs-contact-center-best-practices-mdx)
- [Stream a pre-recorded file in real time](#stream-a-pre-recorded-file-in-real-time-docs-streaming-guides-stream-prerecorded-file-realtime-mdx)
- [Transcribe System Audio in Real-Time (macOS)](#transcribe-system-audio-in-real-time-macos-docs-streaming-guides-transcribe-system-audio-mdx)
- [Terminate Streaming Session After Inactivity](#terminate-streaming-session-after-inactivity-docs-streaming-guides-terminate-realtime-programmatically-mdx)
- [Migrating from Streaming v2 to Streaming v3 (Python)](#migrating-from-streaming-v2-to-streaming-v3-python-docs-streaming-guides-v2-to-v3-migration-mdx)
- [Migrating from Streaming v2 to Streaming v3 (JavaScript)](#migrating-from-streaming-v2-to-streaming-v3-javascript-docs-streaming-guides-v2-to-v3-migration-js-mdx)
- [Next.js example using Real-time STT](#next-js-example-using-real-time-stt-docs-nav-links-next-js-example-using-streaming-stt-mdx)
- [Vanilla Javascript example using Real-time STT](#vanilla-javascript-example-using-real-time-stt-docs-nav-links-vanilla-javascript-example-using-streaming-stt-mdx)
- [Apply LLM Gateway to Streaming](#apply-llm-gateway-to-streaming-docs-streaming-guides-real-time-llm-gateway-mdx)
- [Translate Real-time STT Transcripts with LLM Gateway](#translate-real-time-stt-transcripts-with-llm-gateway-docs-streaming-guides-real-time-translation-mdx)
- [Apply Noise Reduction to Audio for Streaming Speech-to-Text](#apply-noise-reduction-to-audio-for-streaming-speech-to-text-docs-streaming-guides-noise-reduction-streaming-mdx)
- [Transcribe audio files with Streaming](#transcribe-audio-files-with-streaming-docs-streaming-guides-streaming-transcribe-audio-file-mdx)
- [Evaluate Streaming transcription accuracy with WER](#evaluate-streaming-transcription-accuracy-with-wer-docs-streaming-guides-evaluate-streaming-wer-mdx)
- [Determine Optimal Turn Detection Settings from Historical Audio Analysis](#determine-optimal-turn-detection-settings-from-historical-audio-analysis-docs-streaming-guides-turn-detection-improvement-using-async-mdx)
- [Transcribe a short audio file](#transcribe-a-short-audio-file-docs-sync-stt-getting-started-transcribe-a-short-audio-file-mdx)
- [Prompting and Keyterms](#prompting-and-keyterms-docs-sync-stt-prompting-and-keyterms-mdx)
- [Conversation context](#conversation-context-docs-sync-stt-conversation-context-mdx)
- [Language selection](#language-selection-docs-sync-stt-language-selection-mdx)
- [Word timestamps](#word-timestamps-docs-sync-stt-word-timestamps-mdx)
- [Transcribe a short audio file](#transcribe-a-short-audio-file-docs-api-reference-sync-api-transcribe-mdx)
- [Pre-warm a connection](#pre-warm-a-connection-docs-api-reference-sync-api-warm-mdx)
- [Audio requirements](#audio-requirements-docs-sync-stt-audio-requirements-mdx)
- [Error handling](#error-handling-docs-sync-stt-error-handling-mdx)
- [Cloud endpoints & data residency](#cloud-endpoints-data-residency-docs-sync-stt-endpoints-and-data-zones-mdx)
- [Connection pre-warming](#connection-pre-warming-docs-sync-stt-connection-pre-warming-mdx)
- [Voice Agent API](#voice-agent-api-docs-voice-agents-voice-agent-api-mdx)
- [Build with AI coding tools](#build-with-ai-coding-tools-docs-voice-agents-voice-agent-api-build-with-ai-tools-mdx)
- [Supported languages](#supported-languages-docs-voice-agents-voice-agent-api-supported-languages-mdx)
- [Create an agent](#create-an-agent-docs-voice-agents-voice-agent-api-create-agent-mdx)
- [Manage agents](#manage-agents-docs-voice-agents-voice-agent-api-manage-agents-mdx)
- [Create an agent](#create-an-agent-docs-voice-agents-voice-agent-api-api-spec-create-agent-mdx)
- [List agents](#list-agents-docs-voice-agents-voice-agent-api-api-spec-list-agents-mdx)
- [Retrieve an agent](#retrieve-an-agent-docs-voice-agents-voice-agent-api-api-spec-get-agent-mdx)
- [Update an agent](#update-an-agent-docs-voice-agents-voice-agent-api-api-spec-update-agent-mdx)
- [Delete an agent](#delete-an-agent-docs-voice-agents-voice-agent-api-api-spec-delete-agent-mdx)
- [Connect your own LLM](#connect-your-own-llm-docs-voice-agents-voice-agent-api-connect-your-own-llm-mdx)
- [Prompting guide](#prompting-guide-docs-voice-agents-voice-agent-api-prompting-guide-mdx)
- [Greeting](#greeting-docs-voice-agents-voice-agent-api-greeting-mdx)
- [Tools](#tools-docs-voice-agents-voice-agent-api-tools-overview-mdx)
- [HTTP tools](#http-tools-docs-voice-agents-voice-agent-api-tools-http-tools-mdx)
- [Client-side tools](#client-side-tools-docs-voice-agents-voice-agent-api-tools-client-side-tools-mdx)
- [Voices](#voices-docs-voice-agents-voice-agent-api-voices-mdx)
- [Output volume](#output-volume-docs-voice-agents-voice-agent-api-volume-mdx)
- [Turn detection and interruptions](#turn-detection-and-interruptions-docs-voice-agents-voice-agent-api-turn-detection-and-interruptions-mdx)
- [Add transcription context](#add-transcription-context-docs-voice-agents-voice-agent-api-transcription-prompt-mdx)
- [Steer toward known languages](#steer-toward-known-languages-docs-voice-agents-voice-agent-api-language-selection-mdx)
- [Isolate the caller's voice](#isolate-the-caller-s-voice-docs-voice-agents-voice-agent-api-noise-suppression-mdx)
- [Deploy your agent](#deploy-your-agent-docs-voice-agents-voice-agent-api-deploy-mdx)
- [Browser integration](#browser-integration-docs-voice-agents-voice-agent-api-browser-integration-mdx)
- [Set up an inbound phone agent via SIP](#set-up-an-inbound-phone-agent-via-sip-docs-voice-agents-voice-agent-api-connect-to-twilio-mdx)
- [Audio format](#audio-format-docs-voice-agents-voice-agent-api-audio-format-mdx)
- [Recordings & transcripts](#recordings-transcripts-docs-voice-agents-voice-agent-api-session-history-mdx)
- [List sessions](#list-sessions-docs-voice-agents-voice-agent-api-api-spec-list-sessions-mdx)
- [Retrieve a session](#retrieve-a-session-docs-voice-agents-voice-agent-api-api-spec-get-session-mdx)
- [Delete a session](#delete-a-session-docs-voice-agents-voice-agent-api-api-spec-delete-session-mdx)
- [Inline session configuration](#inline-session-configuration-docs-voice-agents-voice-agent-api-session-configuration-mdx)
- [Events reference](#events-reference-docs-voice-agents-voice-agent-api-events-reference-mdx)
- [Troubleshooting](#troubleshooting-docs-voice-agents-voice-agent-api-troubleshooting-mdx)
- [Message Sequence](#message-sequence-docs-voice-agents-voice-agent-api-message-sequence-mdx)
- [Voice Agent WebSocket API](#voice-agent-websocket-api-docs-voice-agents-voice-agent-api-api-spec-voice-agent-websocket-mdx)
- [Generate voice agent token](#generate-voice-agent-token-docs-voice-agents-voice-agent-api-api-spec-generate-voice-agent-token-mdx)
- [Speech Understanding](#speech-understanding-docs-speech-understanding-getting-started-mdx)
- [Action Items](#action-items-docs-speech-understanding-action-items-mdx)
- [Rate limits](#rate-limits-docs-speech-understanding-rate-limits-mdx)
- [Auto Chapters](#auto-chapters-docs-speech-understanding-auto-chapters-mdx)
- [Custom Formatting](#custom-formatting-docs-speech-understanding-custom-formatting-mdx)
- [Entity Detection](#entity-detection-docs-speech-understanding-entity-detection-mdx)
- [Key Phrases](#key-phrases-docs-speech-understanding-key-phrases-mdx)
- [Sentiment Analysis](#sentiment-analysis-docs-speech-understanding-sentiment-analysis-mdx)
- [Speaker Identification](#speaker-identification-docs-speech-understanding-speaker-identification-mdx)
- [Summarization](#summarization-docs-speech-understanding-summarization-mdx)
- [Topic Detection](#topic-detection-docs-speech-understanding-topic-detection-mdx)
- [Translation](#translation-docs-speech-understanding-translation-mdx)
- [Migration guide: Auto Chapters](#migration-guide-auto-chapters-docs-speech-understanding-migration-guides-auto-chapters-mdx)
- [Migration guide: Summarization](#migration-guide-summarization-docs-speech-understanding-migration-guides-summarization-mdx)
- [Guardrails](#guardrails-docs-guardrails-getting-started-mdx)
- [PII Redaction](#pii-redaction-docs-guardrails-redact-pii-from-transcripts-mdx)
- [Content Moderation](#content-moderation-docs-guardrails-detect-sensitive-content-mdx)
- [Profanity Filtering](#profanity-filtering-docs-guardrails-filter-profanity-from-transcripts-mdx)
- [Speech Threshold](#speech-threshold-docs-guardrails-set-minimum-speech-threshold-mdx)
- [Quickstart](#quickstart-docs-llm-gateway-quickstart-mdx)
- [Models](#models-docs-llm-gateway-available-models-mdx)
- [Providers](#providers-docs-llm-gateway-providers-mdx)
- [Model Fallbacks](#model-fallbacks-docs-llm-gateway-fallback-mdx)
- [Prompt Caching](#prompt-caching-docs-llm-gateway-prompt-caching-mdx)
- [Tool Calling](#tool-calling-docs-llm-gateway-tool-calling-mdx)
- [Structured Outputs](#structured-outputs-docs-llm-gateway-structured-outputs-mdx)
- [Cloud Endpoints and Data Residency](#cloud-endpoints-and-data-residency-docs-llm-gateway-cloud-endpoints-and-data-residency-mdx)
- [Create a chat completion](#create-a-chat-completion-docs-llm-gateway-api-reference-create-chat-completion-mdx)
- [Create speech understanding](#create-speech-understanding-docs-llm-gateway-api-reference-create-speech-understanding-mdx)
- [List available models](#list-available-models-docs-llm-gateway-api-reference-list-available-models-mdx)
- [Improve Latency](#improve-latency-docs-llm-gateway-improve-latency-mdx)
- [Ask Questions About Your Audio Transcripts](#ask-questions-about-your-audio-transcripts-docs-llm-gateway-ask-questions-mdx)
- [Agentic Workflows](#agentic-workflows-docs-llm-gateway-agentic-workflows-mdx)
- [Basic Chat Completions](#basic-chat-completions-docs-llm-gateway-chat-completions-mdx)
- [Multi-turn Conversations](#multi-turn-conversations-docs-llm-gateway-conversations-mdx)
- [Prompt Engineering for LLM Gateway](#prompt-engineering-for-llm-gateway-docs-llm-gateway-prompt-engineering-mdx)
- [Rate limits](#rate-limits-docs-llm-gateway-rate-limits-mdx)
- [Troubleshooting](#troubleshooting-docs-llm-gateway-troubleshooting-mdx)
- [Apply LLM Gateway to Audio Transcripts](#apply-llm-gateway-to-audio-transcripts-docs-llm-gateway-apply-llms-to-audio-files-mdx)
- [Setup An AI Coach With LLM Gateway](#setup-an-ai-coach-with-llm-gateway-docs-guides-task-endpoint-ai-coach-mdx)
- [Generate Action Items with LLM Gateway](#generate-action-items-with-llm-gateway-docs-guides-task-endpoint-action-items-mdx)
- [Prompt A Structured Q&A Response Using LLM Gateway](#prompt-a-structured-q-a-response-using-llm-gateway-docs-guides-task-endpoint-structured-qa-mdx)
- [Estimate Input Token Costs for LLM Gateway](#estimate-input-token-costs-for-llm-gateway-docs-guides-counting-tokens-mdx)
- [Extract Dialogue Data with LLM Gateway and JSON](#extract-dialogue-data-with-llm-gateway-and-json-docs-guides-dialogue-data-mdx)
- [Extract Quotes with Timestamps Using LLM Gateway + Semantic Search](#extract-quotes-with-timestamps-using-llm-gateway-semantic-search-docs-guides-transcript-citations-mdx)
- [Extract Transcript Quotes with LLM Gateway](#extract-transcript-quotes-with-llm-gateway-docs-guides-timestamped-transcripts-mdx)
- [Analyze The Sentiment Of A Customer Call using LLM Gateway](#analyze-the-sentiment-of-a-customer-call-using-llm-gateway-docs-guides-call-sentiment-analysis-mdx)
- [Custom Topic Tags Using LLM Gateway](#custom-topic-tags-using-llm-gateway-docs-guides-custom-topic-tags-mdx)
- [Redact PII from Text Using LLM Gateway](#redact-pii-from-text-using-llm-gateway-docs-guides-llm-gateway-pii-redaction-mdx)
- [Apply LLM Gateway to Streaming](#apply-llm-gateway-to-streaming-docs-guides-real-time-llm-gateway-mdx)
- [Translate Real-time STT Transcripts with LLM Gateway](#translate-real-time-stt-transcripts-with-llm-gateway-docs-guides-real-time-translation-mdx)
- [Implement a Sales Playbook Using LLM Gateway](#implement-a-sales-playbook-using-llm-gateway-docs-guides-sales-playbook-mdx)
- [Segment A Phone Call using LLM Gateway](#segment-a-phone-call-using-llm-gateway-docs-guides-phone-call-segmentation-mdx)
- [Generate SOAP Notes using LLM Gateway](#generate-soap-notes-using-llm-gateway-docs-guides-soap-note-generation-mdx)
- [Frequently Asked Questions](#frequently-asked-questions-docs-faq-mdx)
- [Has AssemblyAI certified to the EU-U.S. Data Privacy Framework?](#has-assemblyai-certified-to-the-eu-u-s-data-privacy-framework-docs-faq-has-assemblyai-certified-to-the-eu-us-data-privacy-framework-mdx)
- [Can I sign a DPA agreement with AssemblyAI?](#can-i-sign-a-dpa-agreement-with-assemblyai-docs-faq-can-i-sign-a-dpa-agreement-with-assemblyai-mdx)
- [Can you provide a copy of your most recent penetration test executive summary?](#can-you-provide-a-copy-of-your-most-recent-penetration-test-executive-summary-docs-faq-can-you-provide-a-copy-of-your-most-recent-penetration-test-executive-summary-mdx)
- [Can you provide a recent vulnerability scan?](#can-you-provide-a-recent-vulnerability-scan-docs-faq-can-you-provide-a-recent-vulnerability-scan-mdx)
- [Will AssemblyAI sign a Business Associate Addendum (BAA) as described in the HIPAA rules and regulations?](#will-assemblyai-sign-a-business-associate-addendum-baa-as-described-in-the-hipaa-rules-and-regulations-docs-faq-can-you-sign-a-baa-mdx)
- [Do you have a formal risk assessment policy or process?](#do-you-have-a-formal-risk-assessment-policy-or-process-docs-faq-do-you-have-a-formal-risk-assessment-policy-or-process-mdx)
- [Do you have documented information security policies? If so, how frequently are they updated?](#do-you-have-documented-information-security-policies-if-so-how-frequently-are-they-updated-docs-faq-do-you-have-documented-information-security-policies-if-so-how-frequently-are-they-updated-mdx)
- [Do you offer EU Data Residency?](#do-you-offer-eu-data-residency-docs-faq-do-you-offer-eu-data-residency-mdx)
- [Do you offer self-hosted solutions?](#do-you-offer-self-hosted-solutions-docs-faq-do-you-offer-self-hosted-solutions-mdx)
- [Do you offer servers in the EU?](#do-you-offer-servers-in-the-eu-docs-faq-do-you-offer-servers-in-the-eu-mdx)
- [Do you support SAML in your product?](#do-you-support-saml-in-your-product-docs-faq-do-you-support-saml-in-your-product-mdx)
- [How long are outputs maintained?](#how-long-are-outputs-maintained-docs-faq-how-long-are-outputs-retained-mdx)
- [Does AssemblyAI utilize an anti-virus/anti-malware solution across all relevant infrastructure (workstations and servers), and are appropriate response capabilities deployed to respond to alerts?](#does-assemblyai-utilize-an-anti-virus-anti-malware-solution-across-all-relevant-infrastructure-workstations-and-servers-and-are-appropriate-response-capabilities-deployed-to-respond-to-alerts-docs-faq-does-assemblyai-utilize-an-anti-virus-anti-malware-solution-across-all-relevant-infrastructure-workstations-and-servers-and-are-appropriate-response-capabilities-deployed-to-respond-to-ale-mdx)
- [How are incidents escalated within your organization?](#how-are-incidents-escalated-within-your-organization-docs-faq-how-are-incidents-escalated-within-your-organization-mdx)
- [How do we securely use your service?](#how-do-we-securely-use-your-service-docs-faq-how-do-we-securely-use-your-service-mdx)
- [How do you protect production code?](#how-do-you-protect-production-code-docs-faq-how-do-you-protect-production-code-mdx)
- [How to Access AssemblyAI's Security Reports](#how-to-access-assemblyai-s-security-reports-docs-faq-how-to-access-assemblyai-s-security-reports-mdx)
- [How to Opt Out of Data Sharing for our Model Improvement Program](#how-to-opt-out-of-data-sharing-for-our-model-improvement-program-docs-faq-how-to-opt-out-of-data-sharing-for-our-model-improvement-program-mdx)
- [Is multi-factor authentication enforced for all access to scoped systems and data?](#is-multi-factor-authentication-enforced-for-all-access-to-scoped-systems-and-data-docs-faq-is-multi-factor-authentication-enforced-for-all-access-to-scoped-systems-and-data-mdx)
- [Does AssemblyAI have a documented process for reviewing and approving third-party service providers?](#does-assemblyai-have-a-documented-process-for-reviewing-and-approving-third-party-service-providers-docs-faq-is-there-a-documented-process-for-reviewing-and-approving-third-party-service-providers-mdx)
- [Does AssemblyAI have an incident response plan?](#does-assemblyai-have-an-incident-response-plan-docs-faq-please-describe-the-incident-response-plan-mdx)
- [What are your recovery time and recovery point objectives?](#what-are-your-recovery-time-and-recovery-point-objectives-docs-faq-what-are-your-recovery-time-and-recovery-point-objectives-mdx)
- [What is your SLA for repairing Critical/High/Medium vulnerabilities?](#what-is-your-sla-for-repairing-critical-high-medium-vulnerabilities-docs-faq-what-is-your-sla-for-repairing-critical-high-medium-vulnerabilities-mdx)
- [What logs are available to customers?](#what-logs-are-available-to-customers-docs-faq-what-logs-are-available-to-customers-mdx)
- [What standards do your internal password policies follow?](#what-standards-do-your-internal-password-policies-follow-docs-faq-what-standards-do-your-internal-password-policies-follow-mdx)
- [Where are your servers located?](#where-are-your-servers-located-docs-faq-where-are-your-servers-located-mdx)
- [Am I charged for transcribing silent audio?](#am-i-charged-for-transcribing-silent-audio-docs-faq-am-i-charged-for-transcribing-silent-audio-mdx)
- [Are Custom Models More Accurate than General Models?](#are-custom-models-more-accurate-than-general-models-docs-faq-are-custom-models-more-accurate-than-general-models-mdx)
- [Do I Get Charged for Failed API Calls?](#do-i-get-charged-for-failed-api-calls-docs-faq-are-customers-charged-for-api-calls-that-result-in-errors-mdx)
- [Are there any limits on file size or file duration for files submitted to the API?](#are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api-docs-faq-are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api-mdx)
- [Can I customize how words are spelled by the model?](#can-i-customize-how-words-are-spelled-by-the-model-docs-faq-can-i-customize-how-words-are-spelled-by-the-model-mdx)
- [Can I delete the transcripts I have created using the API?](#can-i-delete-the-transcripts-i-have-created-using-the-api-docs-faq-can-i-delete-the-transcripts-i-have-created-using-the-api-mdx)
- [Can I get a list of all transcripts I have created?](#can-i-get-a-list-of-all-transcripts-i-have-created-docs-faq-can-i-get-a-list-of-all-transcripts-i-have-created-mdx)
- [Can I send audio to AssemblyAI in segments and still get speaker labels for the whole recording?](#can-i-send-audio-to-assemblyai-in-segments-and-still-get-speaker-labels-for-the-whole-recording-docs-faq-can-i-send-audio-to-assemblyai-in-segments-and-still-get-speaker-labels-for-the-whole-recording-mdx)
- [Can I submit files to the API that are stored in a Google Drive?](#can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive-docs-faq-can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive-mdx)
- [Can I use the API without internet access?](#can-i-use-the-api-without-internet-access-docs-faq-can-i-use-the-api-without-internet-access-mdx)
- [Do we have resources for building with Make?](#do-we-have-resources-for-building-with-make-docs-faq-do-we-have-resources-for-building-with-make-mdx)
- [Do you have any examples for how to use your API?](#do-you-have-any-examples-for-how-to-use-your-api-docs-faq-do-you-have-any-examples-for-how-to-use-your-api-mdx)
- [Do you have example use cases for using AssemblyAI?](#do-you-have-example-use-cases-for-using-assemblyai-docs-faq-do-you-have-example-use-cases-for-using-assemblyai-mdx)
- [Do you offer cross-file Speaker Identification?](#do-you-offer-cross-file-speaker-identification-docs-faq-do-you-offer-cross-file-speaker-identification-mdx)
- [Do you offer translation?](#do-you-offer-translation-docs-faq-do-you-offer-translation-mdx)
- [Do you offer voice-to-voice or text-to-speech (TTS)?](#do-you-offer-voice-to-voice-or-text-to-speech-tts-docs-faq-do-you-offer-voice-to-voice-or-text-to-speech-tts-mdx)
- [Does it cost extra to export SRT or VTT captions?](#does-it-cost-extra-to-export-srt-or-vtt-captions-docs-faq-does-it-cost-extra-to-export-srt-or-vtt-captions-mdx)
- [Is there a way to generate SRT or VTT captions with speaker labels?](#is-there-a-way-to-generate-srt-or-vtt-captions-with-speaker-labels-docs-faq-is-there-a-way-to-generate-srt-or-vtt-captions-with-speaker-labels-mdx)
- [Does it cost more to transcribe an audio or video?](#does-it-cost-more-to-transcribe-an-audio-or-video-docs-faq-does-it-cost-more-to-transcribe-an-audio-or-video-mdx)
- [Does your API return timestamps for individual words?](#does-your-api-return-timestamps-for-individual-words-docs-faq-does-your-api-return-timestamps-for-individual-words-mdx)
- [How are individual speakers identified and how does the Speaker Label feature work?](#how-are-individual-speakers-identified-and-how-does-the-speaker-label-feature-work-docs-faq-how-are-individual-speakers-identified-and-how-does-the-speaker-label-feature-work-mdx)
- [How are paragraphs created for the /paragraphs endpoint?](#how-are-paragraphs-created-for-the-paragraphs-endpoint-docs-faq-how-are-paragraphs-created-for-the-paragraphs-endpoint-mdx)
- [How are word/transcript level confidence scores calculated?](#how-are-word-transcript-level-confidence-scores-calculated-docs-faq-how-are-word-transcript-level-confidence-scores-calculated-mdx)
- [How can I integrate AssemblyAI with other services?](#how-can-i-integrate-assemblyai-with-other-services-docs-faq-how-can-i-integrate-assemblyai-with-other-services-mdx)
- [How can I make certain words more likely to be transcribed?](#how-can-i-make-certain-words-more-likely-to-be-transcribed-docs-faq-how-can-i-make-certain-words-more-likely-to-be-transcribed-mdx)
- [How can I test AssemblyAI without writing code?](#how-can-i-test-assemblyai-without-writing-code-docs-faq-how-can-i-test-assemblyai-without-writing-code-mdx)
- [How can I transcribe YouTube videos?](#how-can-i-transcribe-youtube-videos-docs-faq-how-can-i-transcribe-youtube-videos-mdx)
- [How do I generate subtitles?](#how-do-i-generate-subtitles-docs-faq-how-do-i-generate-subtitles-mdx)
- [How does AssemblyAI compare to other ASR providers?](#how-does-assemblyai-compare-to-other-asr-providers-docs-faq-how-does-assemblyai-compare-to-other-asr-providers-mdx)
- [How does Automatic Language Detection work?](#how-does-automatic-language-detection-work-docs-faq-how-does-language-detection-work-for-transcriptions-mdx)
- [How does the API handle files that contain spoken audio in multiple languages?](#how-does-the-api-handle-files-that-contain-spoken-audio-in-multiple-languages-docs-faq-how-does-the-api-handle-files-that-contain-spoken-audio-in-multiple-languages-mdx)
- [How long does it take to transcribe a file?](#how-long-does-it-take-to-transcribe-a-file-docs-faq-how-long-does-it-take-to-transcribe-a-file-mdx)
- [What should I do if I'm getting an error?](#what-should-i-do-if-i-m-getting-an-error-docs-faq-i-am-getting-an-error-what-should-i-do-mdx)
- [Is there a Postman collection for using the API?](#is-there-a-postman-collection-for-using-the-api-docs-faq-is-there-a-postman-collection-for-using-the-api-mdx)
- [Is there a way for us to send the start time / end time for transcription instead of transcribing the whole length of a call recording?](#is-there-a-way-for-us-to-send-the-start-time-end-time-for-transcription-instead-of-transcribing-the-whole-length-of-a-call-recording-docs-faq-is-there-a-way-for-us-to-send-the-start-time-end-time-for-transcription-instead-of-transcribing-the-whole-length-of-a-call-recording-mdx)
- [Is there an OpenAPI spec/schema for the API?](#is-there-an-openapi-spec-schema-for-the-api-docs-faq-is-there-an-openapi-spec-schema-for-the-api-mdx)
- [What causes a "read operation timed out" error?](#what-causes-a-read-operation-timed-out-error-docs-faq-read-operation-timed-out-error-mdx)
- [Should I use Speaker Labels or Multi-channel?](#should-i-use-speaker-labels-or-multi-channel-docs-faq-should-i-use-speaker-labels-or-multi-channel-mdx)
- [Should I use the NA or EU endpoint for my Speech-to-Text requests?](#should-i-use-the-na-or-eu-endpoint-for-my-speech-to-text-requests-docs-faq-should-i-use-the-na-or-eu-endpoint-mdx)
- [What are the recommended options for audio noise reduction?](#what-are-the-recommended-options-for-audio-noise-reduction-docs-faq-what-are-the-recommended-options-for-audio-noise-reduction-mdx)
- [What audio and video file types are supported by your API?](#what-audio-and-video-file-types-are-supported-by-your-api-docs-faq-what-audio-and-video-file-types-are-supported-by-your-api-mdx)
- [What IP Address Should I Whitelist for AssemblyAI?](#what-ip-address-should-i-whitelist-for-assemblyai-docs-faq-what-ip-address-should-i-whitelist-for-assemblyai-mdx)
- [What is the minimum audio duration that the API can transcribe?](#what-is-the-minimum-audio-duration-that-the-api-can-transcribe-docs-faq-what-is-the-minimum-audio-duration-that-the-api-can-transcribe-mdx)
- [What is the recommended file type for using your API?](#what-is-the-recommended-file-type-for-using-your-api-docs-faq-what-is-the-recommended-file-type-for-using-your-api-mdx)
- [What types of audio URLs can I use with the API?](#what-types-of-audio-urls-can-i-use-with-the-api-docs-faq-what-types-of-audio-urls-can-i-use-with-the-api-mdx)
- [Where can I find a list of recent changes to the API?](#where-can-i-find-a-list-of-recent-changes-to-the-api-docs-faq-where-can-i-find-a-list-of-recent-changes-to-the-api-mdx)
- [Where can I find cURL code examples?](#where-can-i-find-curl-code-examples-docs-faq-where-can-i-find-curl-code-examples-mdx)
- [Why can't I access recording URLs from the /upload endpoint directly?](#why-can-t-i-access-recording-urls-from-the-upload-endpoint-directly-docs-faq-why-cant-i-access-urls-from-the-upload-endpoint-directly-mdx)
- [Can I use speaker diarization with Streaming Speech-to-Text?](#can-i-use-speaker-diarization-with-streaming-speech-to-text-docs-faq-can-i-use-speaker-diarization-with-live-audio-transcription-mdx)
- [How accurate is your Streaming transcription compared to Async transcription?](#how-accurate-is-your-streaming-transcription-compared-to-async-transcription-docs-faq-how-accurate-is-your-real-time-transcription-compared-to-async-transcription-mdx)
- [How does Universal Streaming session-based pricing work?](#how-does-universal-streaming-session-based-pricing-work-docs-faq-how-does-universal-streaming-session-based-pricing-work-mdx)
- [What languages are supported for Streaming Speech-to-text?](#what-languages-are-supported-for-streaming-speech-to-text-docs-faq-language-support-for-real-time-transcription-mdx)
- [Resolving SSL Certificate Verification Error When Trying to Use Real-time STT](#resolving-ssl-certificate-verification-error-when-trying-to-use-real-time-stt-docs-faq-resolving-ssl-certificate-verification-error-in-assemblyai-real-time-transcriber-mdx)
- [I am getting a "Model deprecated. See docs for new model information" error message. What does it mean?](#i-am-getting-a-model-deprecated-see-docs-for-new-model-information-error-message-what-does-it-mean-docs-faq-upgrading-to-the-universal-streaming-model-mdx)
- [How do Content Moderation severity scores work?](#how-do-content-moderation-severity-scores-work-docs-faq-how-do-content-moderation-severity-scores-work-mdx)
- [How can I summarize my audio file?](#how-can-i-summarize-my-audio-file-docs-faq-how-do-your-summarization-models-work-mdx)
- [Is Mistral still supported?](#is-mistral-still-supported-docs-faq-is-mistral-still-supported-mdx)
- [Is pricing for Speech Understanding per feature or all-inclusive?](#is-pricing-for-speech-understanding-per-feature-or-all-inclusive-docs-faq-is-pricing-for-audio-intelligence-per-feature-or-all-inclusive-mdx)
- [Understanding Input and Output Tokens for LLM Gateway](#understanding-input-and-output-tokens-for-llm-gateway-docs-faq-understanding-input-and-output-tokens-for-llm-gateway-mdx)
- [Can you use the Playground with files in languages other than English?](#can-you-use-the-playground-with-files-in-languages-other-than-english-docs-faq-can-you-use-the-playground-with-files-in-languages-other-than-english-mdx)
- [How do I delete a transcript I created using the Playground?](#how-do-i-delete-a-transcript-i-created-using-the-playground-docs-faq-how-do-i-delete-a-transcript-i-created-using-the-playground-mdx)
- [Why is the transcription I am receiving using the Playground in a different language?](#why-is-the-transcription-i-am-receiving-using-the-playground-in-a-different-language-docs-faq-why-is-the-transcription-i-am-receiving-using-the-playground-in-a-different-language-mdx)
- [Do you have an affiliate marketing program?](#do-you-have-an-affiliate-marketing-program-docs-faq-do-you-have-an-affiliate-marketing-program-mdx)
- [Do you have any job openings or internship opportunities?](#do-you-have-any-job-openings-or-internship-opportunities-docs-faq-do-you-have-any-job-openings-or-internship-opportunities-mdx)
- [How do I contact support?](#how-do-i-contact-support-docs-faq-how-do-i-contact-support-mdx)
- [How do I get in touch with your Sales team?](#how-do-i-get-in-touch-with-your-sales-team-docs-faq-how-do-i-get-in-touch-with-your-sales-team-mdx)
- [I’ve spotted an issue with the website, what should I do?](#i-ve-spotted-an-issue-with-the-website-what-should-i-do-docs-faq-ive-spotted-an-issue-with-the-website-what-should-i-do-mdx)
- [What are your support hours and response time SLAs?](#what-are-your-support-hours-and-response-time-slas-docs-faq-what-are-your-support-hours-and-response-time-slas-mdx)
- [What is your API Uptime SLA?](#what-is-your-api-uptime-sla-docs-faq-what-is-your-api-uptime-sla-mdx)
- [Where can I find AssemblyAI's product roadmap?](#where-can-i-find-assemblyai-s-product-roadmap-docs-faq-where-can-i-find-assemblyais-product-roadmap-mdx)
- [AssemblyAI integration prompt for AI coding agents](#assemblyai-integration-prompt-for-ai-coding-agents-docs-agent-instructions-mdx)
- [Deployment](#deployment-docs-deployment-mdx)
- [Select The EU Region for EU Data Residency](#select-the-eu-region-for-eu-data-residency-docs-pre-recorded-audio-guides-how-to-use-the-eu-endpoint-mdx)
- [Identifying speakers in audio recordings](#identifying-speakers-in-audio-recordings-docs-pre-recorded-audio-guides-identifying-speakers-in-audio-recordings-mdx)
- [Streaming v2 (Legacy)](#streaming-v2-legacy-docs-pre-recorded-audio-legacy-streaming-mdx)
- [Universal-3.5 Pro](#universal-3-5-pro-docs-pre-recorded-audio-universal-3-5-pro-mdx)
- [Tracking customer transcription usage](#tracking-customer-transcription-usage-docs-tracking-your-customers-usage-mdx)
- [Use case guides](#use-case-guides-docs-use-cases-mdx)
- [Universal-3.5 Pro Streaming API](#universal-3-5-pro-streaming-api-docs-voice-agents-universal-3-5-pro-streaming-api-mdx)
- [Create a webhook subscription](#create-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-create-webhook-subscription-mdx)
- [Delete a webhook subscription](#delete-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-delete-webhook-subscription-mdx)
- [Retrieve a webhook subscription](#retrieve-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-get-webhook-subscription-mdx)
- [List webhook subscriptions](#list-webhook-subscriptions-docs-voice-agents-voice-agent-api-api-spec-list-webhook-subscriptions-mdx)
- [Update a webhook subscription](#update-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-update-webhook-subscription-mdx)
- [Webhooks](#webhooks-docs-voice-agents-voice-agent-api-webhooks-mdx)
- [API specs](#api-specs)

---

<a id="assemblyai-documentation-docs-index-mdx"></a>

# AssemblyAI Documentation

URL: https://www.assemblyai.com/docs
Source: docs/index.mdx
Navigation: Overview > Getting started
Description: Build with our Voice AI Infrastructure

## Build with AI coding agents

  
    Pin the integration prompt URL in your agent's `AGENTS.md` / `CLAUDE.md` / `.cursorrules`, or copy the full prompt directly. Works with Cursor, Claude Code, Copilot, Devin, and any other AI coding assistant.
  

## Explore our models

  
  
  
  
  
  

## Deployment options

  
  
  
        <path d="M137.857 92.4673H92.0903V138.234H137.857V92.4673Z" fill="currentColor" />
        <path d="M183.624 46.7178H137.857V92.4849H183.624V46.7178Z" fill="currentColor" />
        <path d="M183.624 138.234H137.857V184.001H183.624V138.234Z" fill="currentColor" />
        <path d="M229.374 0.950684H183.607V46.7178H229.374V0.950684Z" fill="currentColor" />
        <path d="M229.374 183.984H183.607V229.751H229.374V183.984Z" fill="currentColor" />
        <path d="M46.3409 183.984V138.234V92.4671V46.7178V0.950684H0.57373V46.7178V92.4671V138.234V183.984V229.751H46.3409H92.0902H137.857V183.984H92.0902H46.3409Z" fill="currentColor" />
      </svg>
    }
    href="/voice-agents/livekit-universal-3-5-pro"
  />
  
        <path d="M45.7718 70.7701C50.4477 69.0096 55.7252 70.3307 59.0204 74.0864L101.936 123.001H230.064L272.98 74.0864C276.275 70.3307 281.552 69.0096 286.228 70.7701C290.904 72.5306 294 77.0042 294 82.0005V190H332V214H270V113.873L244.52 142.915C242.242 145.512 238.955 147.001 235.5 147.001H96.5C93.0452 147.001 89.7581 145.512 87.4796 142.915L62 113.873V214H0V190H38V82.0005C38 77.0042 41.0958 72.5306 45.7718 70.7701Z" fill="currentColor" />
        <path d="M270 238.001H332V262.001H270V238.001Z" fill="currentColor" />
        <path d="M0 238.001H62V262.001H0V238.001Z" fill="currentColor" />
        <path d="M128 198.001C128 206.837 120.837 214.001 112 214.001C103.163 214.001 96 206.837 96 198.001C96 189.164 103.163 182.001 112 182.001C120.837 182.001 128 189.164 128 198.001Z" fill="currentColor" />
        <path d="M236 198.001C236 206.837 228.837 214.001 220 214.001C211.163 214.001 204 206.837 204 198.001C204 189.164 211.163 182.001 220 182.001C228.837 182.001 236 189.164 236 198.001Z" fill="currentColor" />
      </svg>
    }
    href="/voice-agents/pipecat-universal-3-5-pro"
  />

---

<a id="build-with-ai-coding-agents-docs-coding-agent-prompts-mdx"></a>

# Build with AI coding agents

URL: https://www.assemblyai.com/docs/coding-agent-prompts
Source: docs/coding-agent-prompts.mdx
Navigation: Overview > Getting started
Description: Build with AssemblyAI using Cursor, Claude Code, Copilot, Devin, and other AI coding assistants. Copy the integration prompt, pin the URL in your agent's project instructions, connect the docs MCP server, or install the Claude Code skill.

If you're using Cursor, Claude Code, Copilot, ChatGPT Codex, Devin, or another AI coding assistant to build with AssemblyAI, give your agent live context so it doesn't rely on outdated training data.

## Start here

  
    Paste our full integration prompt into your agent. Teaches it operating rules, recommended models, parameter gotchas, and quickstarts for every product surface — so the first code you get back is closer to production-ready.
  
  
    Add a one-liner to your `AGENTS.md`, `CLAUDE.md`, or `.cursorrules` so your agent refetches the latest integration prompt before every AssemblyAI prompt — **the most effective option** and what we recommend for ongoing work.
  

## Compare all four methods

| Method | Who it's for | When to use | Trade-off |
|---|---|---|---|
| [**Pin the URL in your agent instructions**](#pin-the-url-in-your-agent-instructions) (`CLAUDE.md`, `.cursorrules`, `AGENTS.md`, …) | Teams shipping AssemblyAI in production — **recommended default.** | Ongoing project work: agent reads project instructions on every prompt and refetches the latest docs. | Tiny one-liner; agent re-fetches the latest docs before every AssemblyAI prompt. |
| [**Copy the integration prompt**](#copy-the-integration-prompt) | First-time users and quick experiments. | One-shot chat sessions (ChatGPT Codex, single Cursor / Claude Code conversation) where you don't have a project config to pin a URL in. | Adds ~770 lines to your prompt; re-copy when the prompt changes. Less efficient than the URL pin for ongoing work. |
| [**Docs MCP server**](#docs-mcp-server) | Devs whose agent does heavy mid-session doc lookups. | Iterative feature work where the agent needs to check several pages mid-task. | Layers on top of the URL pin; requires an MCP-capable client. |
| [**Claude Code skill**](#claude-code-skill) | Claude Code and other skills-compatible agents (60+, via the universal skills format). | You want curated SDK-specific context (Python / JS / streaming / voice agents) bundled with the agent, auto-loaded when relevant. | Installed snapshot; re-install to pick up doc changes. |

For most teams, pinning the URL is enough. Add the MCP server on top if your agent does a lot of mid-session lookups. The copy-paste prompt is best kept for first-time use or one-shot chat sessions.

## Pin the URL in your agent instructions

For ongoing projects, point your agent at the integration prompt URL from its project instructions. The agent re-fetches the prompt before every AssemblyAI-related prompt, so you automatically pick up API changes without bloating your context window.

Add this to your project's `AGENTS.md`, `CLAUDE.md`, `.cursorrules`, `.cursor/rules/`, `.github/copilot-instructions.md`, `.windsurfrules`, or whichever instructions file your agent reads:

```
Before writing AssemblyAI code, read https://www.assemblyai.com/docs/agent-instructions.md
and https://www.assemblyai.com/docs/llms.txt. The API has changed — do not rely on
memorized parameter names.
```

The two URLs cover different things:

- [**agent-instructions.md**](/agent-instructions.md) — the **rules of the road**: operating rules, discovery questions, recommendation template, feature selection, quickstarts for every product surface, error handling, and the gotchas that trip up most integrations.
- [**llms.txt**](/llms.txt) — the **documentation index**: every page's title, URL, and one-line description so the agent can navigate to detail pages.

For deep lookups, [llms-full.txt](/llms-full.txt) returns the full concatenated documentation content. Narrow with `?lang=python` or `?lang=typescript` to save tokens, or add `?excludeSpec=true` to skip the API reference.

## Copy the integration prompt

For a one-shot agent session — a single ChatGPT Codex run, a quick Cursor chat without project files, or just trying AssemblyAI for the first time — paste the full integration prompt into your agent along with a short description of what you want to build. The prompt teaches the agent the operating rules, discovery questions, recommended models, parameter gotchas, and quickstarts for every product surface, so the first code you get back is closer to production-ready than what the agent would write from memory.

For ongoing project work, [pin the URL](#pin-the-url-in-your-agent-instructions) instead — pasting the full prompt into your project instructions stuffs every message with ~770 lines of context.

`````
# AssemblyAI Integration — Coding Agent Instructions

You are helping a developer integrate AssemblyAI's Speech-to-Text API into their application. Your job is to understand their context through discovery, produce a concrete implementation plan, get their approval, and then write correct, production-ready code.

This is a public API. The developer creates their own key at [assemblyai.com/dashboard/api-keys](https://www.assemblyai.com/dashboard/home).

**Official documentation.** Two ways to wire your coding agent up to live docs (both recommended — they layer):

1. **Project instructions** (every prompt): add to `CLAUDE.md`, `.cursorrules`, `AGENTS.md`, or equivalent:

   ```
   Always fetch https://www.assemblyai.com/docs/llms.txt before writing AssemblyAI code.
   The API has changed — do not rely on memorized parameter names.
   ```

   `llms.txt` is the structured index. For full content use `llms-full.txt`; narrow with `?lang=python` or `?lang=typescript`, or add `?excludeSpec=true` to skip the API spec.

2. **Docs MCP server** (on-demand lookups): `https://assemblyai.com/docs/mcp` — Streamable HTTP transport. Lets the agent search and fetch AssemblyAI documentation pages on demand.

   ```bash
   # Claude Code
   claude mcp add assemblyai-docs --transport http https://assemblyai.com/docs/mcp
   ```

   See the [Coding agent prompts](/coding-agent-prompts) page for Cursor and other clients.

---

## 0. Operating rules

1. **Discovery first, code later.** Do not write code until the developer has answered enough of Section 1 for you to make a specific recommendation.
2. **One question per message.** Never batch discovery questions. Wait for an answer before asking the next one.
3. **Plan before you build.** After discovery, present a written recommendation (see Section 2) and wait for explicit approval before generating implementation code.
4. **Prefer the official SDKs.** Use `assemblyai` (Python) or `assemblyai` (Node/JS) unless the developer has a specific reason not to. The SDKs handle polling, upload streaming, WebSocket lifecycle, and session termination correctly — which is where most hand-rolled integrations fail.
5. **Never expose the API key in client-side code.** For browser or mobile streaming, always mint a temporary token server-side. For pre-recorded, proxy uploads and submissions through your server.
6. **Authorization header is the raw key — no `Bearer` prefix.** This trips up everyone. **One exception:** the Voice Agent API (Section 10) requires `Authorization: Bearer YOUR_API_KEY`. Don't generalize either rule across products.
7. **`speech_models` is optional on pre-recorded requests.** If omitted, the request defaults to `["universal-3-5-pro", "universal-2"]`. You can still set it explicitly — see Section 5 for semantics.
8. **Always terminate streaming sessions explicitly.** An abandoned WebSocket keeps accruing charges until the 3-hour cap.
9. **Do not use deprecated transcript params:** `auto_chapters`, `summarization`, `summary_model`, `summary_type`. Use LLM Gateway instead (Section 8).
10. **If the developer's answers are inconsistent, stop and surface the conflict.** Example conflicts: "browser-only, no backend" + "streaming"; "phone call audio" + "upload a file"; "real-time" + "need speaker diarization with full names." Don't paper over these — ask.
11. **Be flexible.** If something the developer says doesn't match the shape of the API (e.g., they describe a use case that isn't supported — see Section 13), say so directly and propose the closest supported alternative.
12. **Verify parameters against live docs before recommending.** This file is a snapshot — features move between beta and GA, model-specific behaviors change, and new knobs ship regularly. Before posting the Section 2 recommendation, confirm each parameter you plan to use is supported for the chosen **mode** (pre-recorded vs streaming) *and* **model** (U3 Pro, U2, U3 Pro Streaming, Universal-Streaming). Do not assume a pre-recorded flag works on streaming, or that a parameter supported on U2 still behaves the same on U3 Pro. Pull the current reference rather than memorizing. Primary sources, in order of preference:
    - `https://www.assemblyai.com/docs/llms-full.txt` — the canonical machine-readable reference
    - Per-mode docs: `/docs/pre-recorded-audio/*` (pre-recorded) and `/docs/streaming/*` (streaming), including the model-specific overview page (e.g., `/docs/streaming/getting-started/transcribe-streaming-audio` and `/docs/streaming/select-the-speech-model`) which lists *exactly* which parameters are honored/ignored by that model
    - The OpenAPI-backed API reference at `/docs/api-reference/*` for request/response schemas
    - For LLM Gateway: `/docs/llm-gateway/quickstart` lists the current valid `model` strings — don't guess short names like `claude-sonnet-4`

  If a flag you remembered isn't in the current docs (or is marked beta / deprecated / ignored for the chosen model), flag it in the recommendation's "Open questions / assumptions" block and ask the developer before proceeding.

---

## 1. Discovery questions

Ask these **one at a time**, in order. Skip any question already answered in the conversation. Adapt wording to sound natural, but cover the substance of each.

1. **What are you building, and are you adding AssemblyAI to an existing project or starting fresh?** (A short description of the product is usually enough.)
2. **What do you need: pre-recorded transcription, real-time real-time STT, or a managed voice agent?**
   - Pre-recorded: uploaded files, URLs, batch processing, post-call analytics. → Section 6.
   - Real-time STT: live transcripts only (you bring your own LLM/TTS). Live captioning, voice-agent STT, meeting notetaking, dictation. → Section 9.
   - Voice Agent API (managed): full-duplex speech-in/speech-out — STT + LLM + TTS + turn detection + tool calling, all in one WebSocket. Right answer when "I want to talk to an AI" is the whole product. → Section 10.
3. **Where is your audio coming from?** (e.g., uploaded files, public URLs, browser microphone, mobile app, Twilio/Telnyx phone numbers, SIP trunks.)
4. **What language and framework are you using?** (e.g., Python + FastAPI, Node + Next.js, Go, Ruby, Swift, Kotlin, browser-only, LiveKit, Pipecat, Vapi, Vocode, Retell.)
5. **Do you already have an AssemblyAI API key, or do you need to create one?** (If needed: [assemblyai.com/dashboard/api-keys](https://www.assemblyai.com/dashboard/home).)
6. **Do you have a data residency requirement?** (US vs EU — this changes the base URL.)
7. **Anything beyond a plain transcript?** Don't read off a checklist. Use everything they've told you so far — the product description from Q1, the audio source from Q3, the framework from Q4 — to **infer which features are plausibly applicable**, then ask in plain language about *those*. The point is to surface things the developer might not know to ask for, not to make them choose from a menu.

   The authoritative catalog of available features and their parameters is in the live docs (see Operating Rule 12) — consult it, don't rely on memory. Section 3 of this file is a starting reference, not the final word.

   Calibrate to mode and use case. Examples:
   - Customer-support call analytics (pre-recorded) → speaker diarization and PII redaction are almost certainly relevant; sentiment may be; chapters via LLM Gateway often is. Ask about those, not about live-streaming features.
   - Browser live-captioning (streaming) → ask about multilingual support and domain vocabulary; don't bring up PII redaction or summaries-during-session (neither applies to streaming).
   - Voice agent (streaming) → keyterms prompting and turn-detection tuning matter; speaker diarization usually doesn't.
   - Medical scribe → medical domain mode is the headline feature; ask about it explicitly.

   Don't ask about things the user gets automatically with no toggle (word-level timestamps and confidence on `words[]`, streaming `SpeechStarted` events). Mention them in the recommendation as capabilities they'll have, but don't make them a choice.

   If you're confident from context that a feature is needed (e.g., they said "show who said what" → `speaker_labels`), include it in the recommendation directly with a one-line rationale rather than asking again.

---

## 2. Recommendation template (after discovery)

Before writing code, post a plan with all of the following. Get explicit approval.

````
## Recommendation

**Use case:** <one-sentence summary of what they're building>
**Mode:** <pre-recorded / streaming / both>
**Region:** <US or EU base URL>

**Model:**
- <model name> — <one-line rationale>
- <fallback model, if applicable>

**Endpoints:**
- <endpoint 1>
- <endpoint 2>

**Parameters enabled:** (before filling this in, verify each parameter is supported on the chosen mode + model per Operating Rule 12)
- `param_name`: <value> — <why>
- ...

**Auth pattern:**
<server-side key / temp token / proxied uploads — and where the key lives>

**Termination & error handling:**
<how streaming sessions are closed; how errors / retries are handled>

**Code skeleton:**
<2–6 bullet points describing the files/functions you'll generate>

**Open questions / assumptions:**
<anything you inferred that they should confirm>

Ready to proceed?
````

If they say yes, write the code. If they push back on any piece, revise the plan — don't just start coding around objections.

---

## 3. Feature selection guide (agent reference)

Use this to build the recommendation. Do not dump it on the user.

| Developer need | Parameter / approach |
|---|---|
| Speaker diarization | `speaker_labels: true` (pre-recorded, and streaming — streaming adds a `speaker_label` to each Turn event) |
| Automatic language detection | `language_detection: true` (pre-recorded; on streaming, only available on Universal-Streaming Multilingual — adds `language_code` + `language_confidence` to Turn events. **Not** supported on U3 Pro Streaming.) |
| Specific language | `language_code: "es"` etc. (pre-recorded and U3 Pro Streaming — biases the multilingual model toward one language) |
| Multilingual / code-switching | `speech_models: ["universal-3-5-pro"]` + `prompt` parameter — see [U3 Pro prompting guide](/pre-recorded-audio/universal-3-5-pro/prompting) |
| Domain-specific vocabulary | `keyterms_prompt: [...]` (pre-recorded: up to 1,000 terms with U3 Pro / 200 with U2; streaming: up to 100 terms, each ≤50 chars) |
| Medical domain | `domain: "medical-v1"` (pre-recorded *and* streaming; supported languages: en, es, de, fr) |
| PII redaction in text | `redact_pii: true` + `redact_pii_policies: [...]` + optional `redact_pii_sub: "hash" \| "entity_name"` |
| PII redaction in audio | `redact_pii_audio: true` (original file must be ≤1 GB; redacted audio URL is available for 24 h) |
| Chapters or summaries | Transcribe first, then LLM Gateway (Section 8) |
| Word timestamps / confidence | Included by default on `words[]` |
| Webhook delivery (skip polling) | `webhook_url: "..."` (Section 7) |
| Managed voice agent (speech-in / speech-out) | Voice Agent API (Section 10) — one WebSocket, no separate STT/LLM/TTS |
| Custom voice agent (your LLM + TTS) | Real-time STT + framework integration (Section 11) |
| Multilingual streaming | Universal-3 Pro Streaming (native code-switching; `language_code` to bias toward one language) |

---

## 4. API overview

- **REST base URL (US):** `https://api.assemblyai.com`
- **REST base URL (EU):** `https://api.eu.assemblyai.com`
- **Streaming WebSocket (Edge, default):** `wss://streaming.assemblyai.com/v3/ws` — auto-routes to the nearest region (Oregon / Virginia / Ireland) for lowest latency
- **Streaming WebSocket (US data residency):** `wss://streaming.us.assemblyai.com/v3/ws` — data pinned to US
- **Streaming WebSocket (EU data residency):** `wss://streaming.eu.assemblyai.com/v3/ws` — data pinned to EU
- **LLM Gateway (US):** `https://llm-gateway.assemblyai.com/v1/chat/completions`
- **LLM Gateway (EU):** `https://llm-gateway.eu.assemblyai.com/v1/chat/completions` — Claude and Gemini only; OpenAI and Qwen are US-only
- **Auth header:** `Authorization: YOUR_API_KEY` (no `Bearer`). Same header is used for REST, streaming WS upgrade, temp-token minting, and LLM Gateway
- **Content type:** `application/json` for submit/poll and LLM Gateway; `application/octet-stream` (raw binary) for `/v2/upload`

Core REST endpoints:
- `POST /v2/upload` — upload a local file (raw binary body, **not multipart**). Returns `{ "upload_url": "..." }`. Max 2.2 GB.
- `POST /v2/transcript` — submit a job. Returns transcript object with `id` and `status: "queued"`. Max 5 GB / 10 hours.
- `GET /v2/transcript/{id}` — poll. Statuses: `queued`, `processing`, `completed`, `error`.

Streaming:
- `wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro`
- `GET https://streaming.assemblyai.com/v3/token?expires_in_seconds=60` — mint a single-use temp token for browser/mobile clients. Optional `max_session_duration_seconds` (60–10800, defaults to 3 h) caps the downstream session length.

---

## 5. `speech_models` semantics

`speech_models` on pre-recorded requests is an **ordered fallback list**, not parallel execution. The first model in the array is tried; if it's unavailable (e.g., not yet rolled out to the account, or temporarily unhealthy), the next is used. A single transcript is produced by exactly one model.

Recommended default: `["universal-3-5-pro", "universal-2"]` — tries the latest model first, falls back to the stable predecessor.

On streaming, the parameter is **singular** (`speech_model=universal-3-5-pro`) — there is no fallback list. Easy to mix up.

---

## 6. Pre-recorded quick start

### SDK (recommended)

**Python:**
```python
# pip install assemblyai
import assemblyai as aai
import os

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],  # fallback handled by SDK
    speaker_labels=True,
)

transcript = aai.Transcriber(config=config).transcribe("https://assembly.ai/wildfires.mp3")
# Or a local path: .transcribe("./recording.wav")

if transcript.status == aai.TranscriptStatus.error:
    raise RuntimeError(transcript.error)
print(transcript.text)
```

**Node/JS:**
```javascript
// npm install assemblyai
import { AssemblyAI } from 'assemblyai';

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const transcript = await client.transcripts.transcribe({
  audio: 'https://assembly.ai/wildfires.mp3', // or a local file path / Buffer / stream
  speech_models: ["universal-3-5-pro", "universal-2"],
  speaker_labels: true,
});

if (transcript.status === 'error') throw new Error(transcript.error);
console.log(transcript.text);
```

The SDK handles upload, submit, and polling. You don't need to write the polling loop yourself.

### Raw HTTP (fallback — use only if SDK isn't an option)

**Upload a local file** (raw bytes, not multipart):
```bash
curl -X POST https://api.assemblyai.com/v2/upload \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  --data-binary @recording.wav
# -> { "upload_url": "https://cdn.assemblyai.com/upload/..." }
```

**Submit and poll (Python):**
```python
import os, time, requests

headers = {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}

submit = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers=headers,
    json={
        "audio_url": "https://assembly.ai/wildfires.mp3",
        "speech_models": ["universal-3-5-pro", "universal-2"],
        "speaker_labels": True,
    },
)
transcript_id = submit.json()["id"]

while True:
    res = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers=headers,
    ).json()
    if res["status"] == "completed":
        print(res["text"]); break
    if res["status"] == "error":
        raise RuntimeError(res["error"])
    time.sleep(3)
```

Common optional params: `speaker_labels`, `language_detection`, `language_code`, `punctuate`, `format_text`, `redact_pii`, `redact_pii_audio`, `keyterms_prompt`, `webhook_url`, `prompt`.

---

## 7. Webhooks (skip polling)

Provide `webhook_url` on submit; AssemblyAI POSTs when the job finishes:

```json
{ "transcript_id": "5552493-16d8-42d8-8feb-c2a16b56f6e8", "status": "completed" }
```

Handler requirements:
- Return 2xx within **10 seconds**. Otherwise retried up to 10 times, 10s apart. 4xx is not retried.
- On receipt, call `GET /v2/transcript/{id}` to fetch the full result — the webhook payload doesn't include it.

Optional custom auth on your webhook: set `webhook_auth_header_name` and `webhook_auth_header_value` when submitting.

**Source IPs** (for allowlists): US `44.238.19.20`, EU `54.220.25.36`.

**Local dev note:** Webhook URLs must be publicly reachable. Use ngrok, Cloudflare Tunnel, or similar during development.

---

## 8. LLM Gateway (chapters, summaries, custom analysis)

LLM Gateway replaces both the deprecated transcript params (`auto_chapters`, `summarization`, `summary_model`, `summary_type`) and the legacy **LeMUR** API, which sunset on 2026-03-31. If a developer mentions LeMUR or `transcript_ids`, point them at LLM Gateway and the [migration guide](/llm-gateway/quickstart). Workflow:

1. Transcribe normally with `POST /v2/transcript`.
2. Once `status == "completed"`, POST to LLM Gateway with the transcript text (or paragraphs from `GET /v2/transcript/{id}/paragraphs` for chapter-style output):

```http
POST https://llm-gateway.assemblyai.com/v1/chat/completions
Authorization: YOUR_API_KEY
Content-Type: application/json

{
  "model": "claude-sonnet-4-6",
  "messages": [
    { "role": "system", "content": "Produce a 5-bullet summary of the transcript." },
    { "role": "user", "content": "<transcript.text here>" }
  ],
  "max_tokens": 1000
}
```

Model IDs are exact strings — see the [LLM Gateway Overview](/llm-gateway/quickstart) for the current list. Examples: `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`, `gpt-5.2`, `gpt-5.1`, `gpt-4.1`, `gemini-3.5-flash`, `gemini-2.5-pro`, `gemini-2.5-flash`, `qwen3-next-80b-a3b`. `claude-sonnet-4` by itself is **not** valid — always include the version suffix. EU region (`llm-gateway.eu.assemblyai.com`) supports Anthropic and Google only.

Do not submit with `auto_chapters` and `summarization` both enabled — the API rejects it (`Only one of the following models can be enabled at a time: auto_chapters, summarization.`). But the broader rule is simpler: **don't use either.**

---

## 9. Streaming — Universal-3.5 Pro

**WebSocket (default, Edge Routing):** `wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro`

For data residency, swap the host: `streaming.us.assemblyai.com` (US-pinned) or `streaming.eu.assemblyai.com` (EU-pinned). The default host auto-routes to the nearest region.

**Audio format:** PCM16 signed little-endian, mono, 16 kHz. Binary WebSocket frames, **50–1000 ms per chunk**, no faster than real-time. Phone audio (`encoding=pcm_mulaw`, `sample_rate=8000`) is sent as-is — don't upsample.

**Auth:**
- Server-side: `Authorization` header on the WS upgrade.
- Browser/mobile: mint a short-lived token server-side and pass it as `?token=<token>` (no Authorization header).

Mint a token:
```bash
curl -s "https://streaming.assemblyai.com/v3/token?expires_in_seconds=60" \
  -H "Authorization: $ASSEMBLYAI_API_KEY"
# { "token": "..." }
```
`expires_in_seconds` must be 1–600. Tokens are single-use per session.

### Server messages (JSON)

- `Begin` — `{ type, id, expires_at }`
- `SpeechStarted` — `{ type, timestamp, confidence }`
- `Turn` — `{ type, turn_order, end_of_turn, transcript, end_of_turn_confidence, words:[...], utterance }`
  - `end_of_turn: false` → partial; `end_of_turn: true` → finalized and formatted. Always read `transcript` for current text.
- `Termination` — `{ type, audio_duration_seconds, session_duration_seconds }`

### Client messages

- Binary PCM16 frames — audio.
- `{ "type": "Terminate" }` — graceful end. **Always send this when done.**
- `{ "type": "ForceEndpoint" }` — force current turn to end.
- `{ "type": "KeepAlive" }` — only needed if `inactivity_timeout` is set.
- `{ "type": "UpdateConfiguration", "keyterms_prompt": [...], "min_turn_silence": 100, "max_turn_silence": 1000 }` — adjust mid-session.

### SDK (recommended)

**Python:**
```python
# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    StreamingClient,
    StreamingClientOptions,
    StreamingEvents,
    StreamingParameters,
    TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = StreamingClient(
    StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"])
)
client.on(StreamingEvents.Turn, on_turn)
client.connect(StreamingParameters(sample_rate=16000, speech_model="universal-3-5-pro"))

# Feed 16 kHz mono PCM16 chunks (50–1000ms each) via client.stream(chunk)
# When finished:
client.disconnect(terminate=True)  # sends Terminate and closes cleanly
```

**Node/JS:**
```javascript
// npm install assemblyai
import { AssemblyAI } from 'assemblyai';

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const rt = client.streaming.transcriber({
  sampleRate: 16000,
  speechModel: 'universal-3-5-pro',
});

rt.on('turn', (turn) => {
  const tag = turn.end_of_turn ? 'FINAL' : 'partial';
  console.log(`${tag}: ${turn.transcript}`);
});
rt.on('error', (err) => console.error(err));

await rt.connect();
// rt.sendAudio(pcm16Buffer) for each 50–1000ms chunk
// When done:
await rt.close(); // sends Terminate and closes
```

### Raw WebSocket (fallback)

**Node.js (`ws`):**
```javascript
import WebSocket from 'ws';

const ws = new WebSocket(
  'wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro',
  { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
);

ws.on('open', () => {
  // Feed PCM16 16kHz mono chunks here, 50–1000ms each.
  // Example: audioStream.on('data', (chunk) => ws.send(chunk));
});

ws.on('message', (raw) => {
  const msg = JSON.parse(raw.toString());
  if (msg.type === 'Turn') {
    console.log(msg.end_of_turn ? `FINAL: ${msg.transcript}` : `partial: ${msg.transcript}`);
  }
});

function stop() {
  ws.send(JSON.stringify({ type: 'Terminate' })); // required!
}
```

**Python (`websockets`):**
```python
import asyncio, json, os, websockets

URL = "wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro"

async def run(audio_source):
    """audio_source: async iterator yielding 50–1000ms PCM16 chunks at 16kHz mono."""
    async with websockets.connect(
        URL,
        additional_headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    ) as ws:
        async def send_audio():
            async for chunk in audio_source:
                await ws.send(chunk)
            await ws.send(json.dumps({"type": "Terminate"}))

        async def recv_loop():
            async for raw in ws:
                msg = json.loads(raw)
                if msg["type"] == "Turn":
                    tag = "FINAL" if msg["end_of_turn"] else "partial"
                    print(f"{tag}: {msg['transcript']}")
                elif msg["type"] == "Termination":
                    return

        await asyncio.gather(send_audio(), recv_loop())

# asyncio.run(run(my_audio_iterator()))
```

---

## 10. Voice Agent API (managed speech-in / speech-out)

Use this when the developer wants a complete spoken AI agent — not just transcription. Single WebSocket, audio in and audio out, with STT + LLM + TTS + turn detection + tool calling all managed by AssemblyAI.

**Endpoint:** `wss://agents.assemblyai.com/v1/ws`

**Auth:** `Authorization: Bearer YOUR_API_KEY` — the Bearer prefix is **required** on this product (different from STT and LLM Gateway, which take the raw key). For browsers/mobile, mint a temp token instead and pass it as `?token=<token>`.

**Token endpoint (for browser/mobile clients):**
```bash
curl -s "https://agents.assemblyai.com/v1/token?expires_in_seconds=300&max_session_duration_seconds=8640" \
  -H "Authorization: Bearer $ASSEMBLYAI_API_KEY"
# { "token": "..." }
```
- `expires_in_seconds`: 1–600 (controls how long the token can be redeemed for)
- `max_session_duration_seconds`: 60–10800 (caps the resulting session; defaults to the 3-hour max)
- Tokens are **single-use** per session — get a fresh one for every reconnect (including `session.resume`).

**Audio format:** PCM16 mono **24 kHz**, **base64-encoded inside JSON events** (not raw binary frames — this is different from real-time STT). ~50 ms chunks (2,400 bytes) is fine; the server buffers continuously, exact chunk size doesn't matter.

### Lifecycle (the events that matter)

1. Client connects, sends `session.update` immediately (don't wait for `session.ready`):
   ```json
   {
     "type": "session.update",
     "session": {
       "system_prompt": "You are a helpful assistant.",
       "greeting": "Hi there! How can I help?",
       "input": {
         "format": { "encoding": "audio/pcm" },
         "keyterms": ["AssemblyAI", "Universal-3"],
         "turn_detection": {
           "vad_threshold": 0.5,
           "min_silence": 200,
           "max_silence": 1000,
           "interrupt_response": true
         }
       },
       "output": {
         "voice": "anna",
         "format": { "encoding": "audio/pcm" }
       },
       "tools": [ /* flat-schema tool defs, see step 5 */ ]
     }
   }
   ```
   Output `encoding` accepts `audio/pcm` (24 kHz, default), `audio/pcmu` (G.711 μ-law, 8 kHz), or `audio/pcma` (G.711 A-law, 8 kHz) — use the G.711 variants for telephony bridges (Twilio, etc.) so you don't have to resample.
2. Server replies with `session.ready` (capture `session_id` for `session.resume` if you reconnect within 30 s of a disconnect).
3. **Only after `session.ready`**, start streaming mic audio:
   ```json
   { "type": "input.audio", "audio": "<base64 PCM16 24kHz>" }
   ```
4. Server emits, in roughly this order, per turn:
   - `input.speech.started` / `input.speech.stopped` (VAD)
   - `transcript.user.delta` (partials) and `transcript.user` (final)
   - `reply.started`, `reply.audio` (multiple base64 PCM16 chunks — write directly into an output buffer at 24 kHz), `transcript.agent`, `reply.done`
   - **Field-name asymmetry:** `input.audio` carries audio in the `audio` field; `reply.audio` carries it in the `data` field. Easy to miss — copying `event["audio"]` from input handling will silently return nothing on output.
5. **Tool calls:** tool definitions in `session.tools` use a **flat** schema — *not* OpenAI's nested `{type: "function", function: {...}}` form:
   ```json
   {
     "type": "function",
     "name": "get_weather",
     "description": "Get the current weather for a city.",
     "parameters": {
       "type": "object",
       "properties": { "location": { "type": "string" } },
       "required": ["location"]
     }
   }
   ```
   Server sends `tool.call` with `{call_id, name, arguments}`. Accumulate the result locally, then send `tool.result` with the matching `call_id` *after* `reply.done` fires. If `reply.done.status == "interrupted"` (user barge-in), discard pending tool results.
6. **Resume after disconnect:** within 30 s, reconnect with a *new* token and send `session.resume` carrying the previous `session_id` to keep conversation context. After 30 s, start a new session.

### Voices

Voice IDs are **exact strings** — invented or remembered values silently fail. Pick from the catalog below or call `GET https://agents.assemblyai.com/v1/voices` for the live list.

**English** — `alba`, `eve`, `george`, `jane`, `jean`, `mary`, `michael` (🇺🇸 American), `anna`, `charles`, `paul`, `vera` (🇬🇧 British).

**Language-specific** (each speaks the named language plus English): `giovanni` (🇮🇹 Italian), `lola` (🇪🇸 Spanish), `juergen` (🇩🇪 German), `rafael` (🇵🇹 Portuguese), `estelle` (🇫🇷 French).

If the developer needs a voice not in this list, *don't* substitute a similar-sounding name — say so and ask. Pre-Voice-Agent-API names like `claire`, `dawn`, `josh`, `grace`, `pete` are **no longer valid** and will be rejected at `session.update`. The legacy voices `ivy`, `james`, `tyler`, `winter`, `bella`, `david`, `kyle`, `helen`, `martha`, `river`, `emma`, `victor`, `eleanor`, `arjun`, `dmitri`, `pierre`, `giulia`, `luca`, `lucia`, `mateo`, and `diego` have been removed — pick a current voice from the catalog above.

### Playback gotcha

Don't sleep-schedule audio chunks. Write each `reply.audio` PCM directly to an OS audio buffer (e.g., `sounddevice.OutputStream.write()`) — the OS drains at exactly 24 kHz and absorbs network jitter. Sleep-based timing drifts and produces pops/gaps.

On `reply.done.status == "interrupted"`, flush the output buffer (e.g., `speaker.abort(); speaker.start()`) so the user doesn't hear stale agent speech.

### Quickstart pattern (Python sketch)

```python
# pip install websockets sounddevice numpy
import asyncio, base64, json, os
import sounddevice as sd
import websockets

URL = "wss://agents.assemblyai.com/v1/ws"
SAMPLE_RATE = 24_000

async def main():
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a helpful assistant.",
                "greeting": "Hi! How can I help?",
                "output": {"voice": "anna"},
            },
        }))

        ready = asyncio.Event()
        loop = asyncio.get_running_loop()
        mic_q: asyncio.Queue = asyncio.Queue()

        def on_mic(indata, *_):
            if ready.is_set():
                loop.call_soon_threadsafe(mic_q.put_nowait, bytes(indata))

        async def pump_mic():
            while True:
                chunk = await mic_q.get()
                await ws.send(json.dumps({
                    "type": "input.audio",
                    "audio": base64.b64encode(chunk).decode(),
                }))

        with sd.InputStream(samplerate=SAMPLE_RATE, channels=1,
                            dtype="int16", callback=on_mic), \
             sd.OutputStream(samplerate=SAMPLE_RATE, channels=1,
                             dtype="int16") as speaker:
            asyncio.create_task(pump_mic())
            async for raw in ws:
                ev = json.loads(raw)
                if ev["type"] == "session.ready":
                    ready.set()
                elif ev["type"] == "reply.audio":
                    import numpy as np
                    speaker.write(np.frombuffer(base64.b64decode(ev["data"]), dtype=np.int16))
                elif ev["type"] == "reply.done" and ev.get("status") == "interrupted":
                    speaker.abort(); speaker.start()

asyncio.run(main())
```

For a complete worked example (MCP-tooled agent that talks back), see the [Voice Agent API quickstart](/voice-agents/voice-agent-api). For browser integration, see the [browser integration guide](/voice-agents/voice-agent-api/browser-integration).

### When to choose Voice Agent API vs Real-time STT + your own LLM/TTS

- **Voice Agent API (Section 10):** end-to-end conversational agents, fastest to ship, AssemblyAI manages the pipeline. Use when "speech in, speech out" is the whole product.
- **Real-time STT + framework (Section 11):** you need a specific LLM, a specific TTS provider, custom turn-detection logic, complex orchestration (LiveKit/Pipecat/Vapi/Vocode/Retell), or features the managed pipeline doesn't expose yet.

If they're not sure, ask: *do you want to choose your own LLM and TTS, or is a managed pipeline fine?* That single answer routes them.

---

## 11. Voice Agent framework configs (real-time STT + your own pipeline)

This section is for developers who are NOT using the Voice Agent API (Section 10) — they're wiring AssemblyAI Real-time STT into LiveKit, Pipecat, Vapi, Vocode, Retell, or similar, and bringing their own LLM and TTS.

The defaults will not be good enough. Common tuning:

- **`keyterms_prompt`** — pass proper nouns, product names, and domain terms. For dynamic values (usernames, order IDs), update mid-session via `UpdateConfiguration`.
- **Turn silence bounds** — `min_turn_silence` and `max_turn_silence` (ms). Lower values fire end-of-turn faster but risk cutting speakers off. Higher values reduce false finalizations. Form-filling and dictation use cases often want wider windows.
- **Multilingual** — Universal-3 Pro Streaming code-switches natively and ignores `end_of_turn_confidence_threshold`. Pass `language_code` to bias the model toward a single language.
- **Barge-in / false SpeechStarted** — ambient noise, TTS bleed-through, and PSTN echo can cause spurious `SpeechStarted` events. If the agent is interrupting itself, look here first. Framework-level knobs (e.g., LiveKit's `min_interruption_duration`) often complement, not replace, server-side tuning.
- **Phone audio** — 8 kHz mu-law (`pcm_mulaw` at 8000 Hz) should be sent as-is, not upsampled to 16 kHz. Upsampling degrades accuracy.

When the developer names one of these frameworks, ask about their specific turn-taking and interruption requirements before defaulting.

---

## 12. Browser patterns

**Never put the API key in client code.**

### Pre-recorded — proxy upload + submit through your server

```javascript
// Next.js route handler (server)
export async function POST(request) {
  const incoming = await request.formData();
  const file = incoming.get('file'); // Blob

  const upload = await fetch('https://api.assemblyai.com/v2/upload', {
    method: 'POST',
    headers: { authorization: process.env.ASSEMBLYAI_API_KEY },
    body: file.stream(),
    duplex: 'half',
  });
  const { upload_url } = await upload.json();

  const submit = await fetch('https://api.assemblyai.com/v2/transcript', {
    method: 'POST',
    headers: {
      authorization: process.env.ASSEMBLYAI_API_KEY,
      'content-type': 'application/json',
    },
    body: JSON.stringify({
      audio_url: upload_url,
      speech_models: ['universal-3-5-pro', 'universal-2'],
    }),
  });
  return Response.json(await submit.json());
}
```

### Streaming — server mints a temp token, client connects directly

```javascript
// Server
export async function GET() {
  const res = await fetch(
    'https://streaming.assemblyai.com/v3/token?expires_in_seconds=60',
    { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
  );
  return Response.json(await res.json()); // { token }
}
```

```javascript
// Client
const { token } = await fetch('/api/aai-token').then((r) => r.json());
const ws = new WebSocket(
  `wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro&token=${token}`,
);
ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'Turn') console.log(msg.transcript, msg.end_of_turn);
};
```

### Capturing mic audio in the browser

`MediaRecorder` does not emit PCM16. You need an `AudioWorklet` (preferred) or `ScriptProcessorNode` to:
1. Capture raw Float32 samples.
2. Downsample to 16 kHz.
3. Convert Float32 → Int16.
4. Send each ~50 ms chunk as a binary WS frame.

Reference: [AssemblyAI realtime-transcription-browser-js-example](https://github.com/AssemblyAI/realtime-transcription-browser-js-example).

---

## 13. Not supported / out of scope

If a developer asks for any of these, say so directly and propose the closest supported alternative. Do not improvise.

- **Real-time translation.** AssemblyAI transcribes, it doesn't translate. Suggest: transcribe with U3 Pro, then translate via LLM Gateway.
- **On-device / offline STT.** Cloud API only.
- **Speaker identification (matching voices to known people).** `speaker_labels` does diarization (Speaker A, B, C) but does not recognize specific individuals.
- **Standalone TTS.** Not an AssemblyAI product *as a separate API*. TTS is bundled into the Voice Agent API (Section 10) — if they need just-TTS, point them to a dedicated provider.
- **Voice activity detection as a standalone product.** VAD is internal to the streaming pipeline and surfaced via `SpeechStarted` / turn events, not exposed separately.

---

## 14. Error handling

### REST (pre-recorded)

- **401** — Missing/invalid Authorization, disabled account, or insufficient balance. Double-check there's no `Bearer` prefix.
- **Transcript `status: "error"`** — Read the `error` field on `GET /v2/transcript/{id}`.
- **Retries** — Exponential backoff on 5xx. For 429, respect the `Retry-After` header.
- **Limits** — `/v2/upload` max 2.2 GB; `/v2/transcript` max 5 GB / 10 hr per file.
- **Scoping** — An API key can only transcribe files uploaded under the same project.

### Streaming — handshake

- **HTTP 410** — The old `v2` streaming endpoint is deprecated. Upgrade to `/v3/ws`. This is an HTTP status on the upgrade request, not a WebSocket close code.

### Streaming — WebSocket close codes

| Code | Meaning |
|------|---------|
| `1008` | Unauthorized: missing/invalid Authorization or token |
| `3005` | Session cancelled (server-side error) |
| `3006` | Invalid message type / invalid JSON |
| `3007` | Audio chunk outside 50–1000 ms, or sent faster than real-time |
| `3008` | Session expired (3-hour cap) |
| `3009` | Too many concurrent sessions |

### Streaming gotchas

- `speech_model` (streaming, singular) vs `speech_models` (pre-recorded, plural). Don't mix up.
- On U3 Pro Streaming, `end_of_turn_confidence_threshold` is silently ignored; `language_code` biases the multilingual model toward a single language.
- Always send `{ "type": "Terminate" }` when finished. An abandoned session stays billable until the 3-hour cap (`3008`).
- Chunk size matters: frames outside 50–1000 ms will close the socket with `3007`.

---

## 15. Quick-reference gotchas

- No `Bearer` prefix on the Authorization header — *except* for the Voice Agent API (Section 10), which requires `Authorization: Bearer ...`.
- `speech_models` is **optional** on pre-recorded submits — it defaults to `["universal-3-5-pro", "universal-2"]` and is an **ordered fallback list** when provided.
- `/v2/upload` takes **raw binary**, not multipart.
- Webhook handlers must return 2xx in ≤10 seconds.
- Local webhook development needs a public tunnel (ngrok, Cloudflare Tunnel).
- Browser code never holds the API key. Proxy uploads, or mint temp tokens for streaming.
- Always `Terminate` streaming sessions.
- Don't use `auto_chapters`, `summarization`, `summary_model`, `summary_type`. Use LLM Gateway.
- Medical mode is `domain: "medical-v1"` (pre-recorded body param / streaming query param). The legacy `medical_mode` flag is **not** the right name.
- LLM Gateway model IDs are exact and versioned (e.g., `claude-sonnet-4-6`, `gpt-5.2`, `gemini-2.5-pro`). Shorthand like `claude-sonnet-4` is invalid.
- Phone audio stays at native 8 kHz mu-law (`encoding=pcm_mulaw`) — don't upsample.
- EU customers use `api.eu.assemblyai.com`, `streaming.eu.assemblyai.com`, and `llm-gateway.eu.assemblyai.com`. The default streaming host (`streaming.assemblyai.com`) is **Edge Routing**, not US-pinned — use `streaming.us.assemblyai.com` if you need data residency guarantees on the US side.
- Speech-model values are **raw strings** in the SDKs (`"universal-3-5-pro"`, `"universal-2"`). Enum aliases like `aai.SpeechModel.universal_3_pro` do **not** exist — agents that hallucinate them produce code that imports cleanly and fails at runtime.
- LeMUR has fully sunset (2026-03-31). Don't generate code that calls LeMUR endpoints or passes `transcript_ids` to a chat-completions API — use LLM Gateway with the transcript text in `messages` instead.
`````

The same prompt is published at [agent-instructions.md](/agent-instructions.md) if you'd rather link to it than paste the whole thing.

## Docs MCP server

For on-demand lookups during a session — useful when the agent is iterating on a feature and needs to check several pages mid-task — connect the AssemblyAI docs MCP server:

```
https://assemblyai.com/docs/mcp
```

**Claude Code:**

```bash
claude mcp add assemblyai-docs --transport http https://assemblyai.com/docs/mcp
```

**Cursor** (`.cursor/mcp.json`):

```json
{
  "mcpServers": {
    "assemblyai-docs": {
      "url": "https://assemblyai.com/docs/mcp"
    }
  }
}
```

Any MCP client that supports [Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports#streamable-http) can connect. The server exposes tools to search and fetch AssemblyAI documentation pages on demand.

## Claude Code skill

The [AssemblyAI skill](https://github.com/AssemblyAI/assemblyai-skill) gives Claude Code curated instructions and context for the Python and JavaScript SDKs, streaming, voice agents, audio intelligence, and more.

```bash
claude install-skill https://github.com/AssemblyAI/assemblyai-skill
```

Works with 60+ AI coding agents via the universal skills format:

```bash
npx skills add AssemblyAI/assemblyai-skill
```

Verify it's installed:

```bash
claude skill list
```

## Tips for best results

- **Be specific about the SDK** — say "use the AssemblyAI Python SDK" or "use the JavaScript SDK" rather than "use AssemblyAI".
- **Reference the model** — specify `universal-3-5-pro` for pre-recorded or streaming to avoid outdated model names.
- **Set your API key as an environment variable** — `export ASSEMBLYAI_API_KEY=your_key` so the agent can reference it naturally.

---

<a id="models-docs-getting-started-models-mdx"></a>

# Models

URL: https://www.assemblyai.com/docs/getting-started/models
Source: docs/getting-started/models.mdx
Navigation: Overview > Getting started
Description: AssemblyAI's speech-to-text models and their capabilities

AssemblyAI offers several state-of-the-art speech recognition models, each optimized for different use cases. Choose the model that best fits your needs based on accuracy, latency, cost, and language requirements.

## Pre-recorded models

  
    <ul style={{ fontSize: "14px", lineHeight: "2", paddingLeft: "20px", margin: 0 }}>
      <li>Highest accuracy, fastest model</li>
      <li>Supports 18 languages</li>
      <li>Native code switching</li>
      <li>Contextual prompting capabilities</li>
      <li>Keyterms prompting up to 1,000 words</li>
    </ul>
  
  
    <ul style={{ fontSize: "14px", lineHeight: "2", paddingLeft: "20px", margin: 0 }}>
      <li>High accuracy, low latency</li>
      <li>Support across 99 languages</li>
      <li>Keyterms prompting up to 200 words</li>
      <li>Code switching</li>
    </ul>
  

We recommend [Universal-3.5 Pro](/pre-recorded-audio/universal-3-5-pro) for pre-recorded audio transcription. It delivers the highest accuracy and fastest transcription out of the box, with optional contextual prompting support. Universal-3.5 Pro supports 18 languages, for anything outside that set, the system automatically falls back to Universal-2, giving you coverage across 99 languages total without any extra configuration.

## Streaming models

  
    <ul style={{ fontSize: "14px", lineHeight: "2", paddingLeft: "20px", margin: 0 }}>
      <li>Highest accuracy for voice agents</li>
      <li>Fastest word emissions</li>
      <li>Advanced prompting capabilities</li>
      <li>Keyterms prompting up to 100 words</li>
      <li>18 languages with native code switching</li>
    </ul>
  
  
    <ul style={{ fontSize: "14px", lineHeight: "2", paddingLeft: "20px", margin: 0 }}>
      <li>Good balance of speed and cost-effectiveness</li>
      <li>Multilingual real-time transcription</li>
      <li>Keyterms prompting up to 100 words</li>
      <li>6 languages: en, es, pt, de, fr, it</li>
    </ul>
  
  
    <ul style={{ fontSize: "14px", lineHeight: "2", paddingLeft: "20px", margin: 0 }}>
      <li>Good balance of speed and cost-effectiveness</li>
      <li>English transcription</li>
      <li>Keyterms prompting up to 100 words</li>
      <li>Intelligent endpointing</li>
    </ul>
  

We recommend [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio) for streaming transcription. It provides the highest accuracy with sub-300ms latency, native multilingual code switching, and advanced prompting support.

## Add-on models

Add-on models enhance transcription accuracy for specialized domains. They work alongside your chosen speech model and are billed separately.

  
    <ul style={{ fontSize: "14px", lineHeight: "2", paddingLeft: "20px", margin: 0 }}>
      <li>Improved accuracy for medical terminology</li>
      <li>Medications, procedures, conditions, and dosages</li>
      <li>Works with pre-recorded and streaming models</li>
      <li>4 languages: en, es, de, fr</li>
    </ul>
  

### Medical Mode

Medical Mode (`domain: "medical-v1"`) is an add-on that enhances transcription accuracy for medical terminology — including medication names, procedures, conditions, and dosages. It is optimized for medical entity recognition to correct terms that other models frequently get wrong.

**Supported models:**
- Pre-recorded: Universal-3.5 Pro, Universal-2
- Streaming: Universal-3.5 Pro Streaming, Universal-Streaming English, Universal-Streaming Multilingual

**Supported languages:** English, Spanish, German, French

Medical Mode is billed as a separate add-on. See the [pricing page](https://www.assemblyai.com/pricing) for details.

Learn more: [Medical Mode for pre-recorded audio](/pre-recorded-audio/medical-mode) | [Medical Mode for streaming](/streaming/medical-mode)

## Choosing the right model

### Pre-recorded

#### Universal-3.5 Pro

Universal-3.5 Pro is our most powerful Voice AI model, designed to capture the "hard stuff" that traditional ASR models struggle with. It delivers state-of-the-art accuracy for entities, rare words, and domain-specific terminology out of the box, with code switching and optional prompting for more control. It's also our fastest model, so you get the best accuracy without sacrificing speed.

**Best for:**
- Applications requiring highest-accuracy transcription
- Medical scribes needing clinical grade transcription accuracy
- Sales intelligence / Call centers needing native code-switching
- Meeting notetakers / recruiting notetakers needing high-quality diarization

  
  <br />

  **Regional dialects**

Universal-3.5 Pro also supports regional dialects and local speech variants out of the box — no special configuration needed. See the full list of [supported dialects](/pre-recorded-audio/supported-languages#regional-dialects-and-variants).

[Try Universal-3.5 Pro here](/pre-recorded-audio/universal-3-5-pro)

#### Universal-2

Universal-2 offers accurate, cost-effective transcription across 99 languages with low latency. It supports code switching and optional keyterms prompting for domain-specific vocabulary (up to 200 words). Universal-2 is the go-to choice when you need reliable transcription across diverse languages.

**Best for:**
- High accuracy at lower cost with broad language support
- High-volume, price-sensitive batch transcription
- Support for over 99 languages
- Recommended fallback when a requested language isn't supported by Universal-3.5 Pro

  
  <br />

[Try Universal-2 here](/pre-recorded-audio/select-the-speech-model)

### Streaming

#### Universal-3.5 Pro Streaming

The most accurate model with the fastest word emissions for voice agents that demand the highest quality. Best-in-class accuracy with advanced prompting capabilities, including both [keyterms prompting](/streaming/prompting-and-keyterms) and [native prompting](/streaming/prompting-and-keyterms). Supports English, Spanish, German, French, Portuguese, Italian, Turkish, Dutch, Swedish, Norwegian, Danish, Finnish, Hindi, Vietnamese, Arabic, Hebrew, Japanese, and Mandarin.

**Best for:**
- Real-time voice agents
- Applications requiring premium accuracy
- Customer service voice agents needing elite entity accuracy
- IVR replacement / binary response detection in short utterances
- Agent assist and sales intelligence needing real-time speaker diarization, mid-session dynamic prompting
- Multilingual voice agents with native code-switching across 18 languages
- Compliance and verbatim recording — disfluency control via prompting

  
  <br />

  **Regional dialects**

Universal-3.5 Pro Streaming also supports regional dialects and local speech variants out of the box, with no special configuration needed. See the full list of [supported dialects](/streaming/getting-started/transcribe-streaming-audio).

[Learn more about Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio)

#### Universal-Streaming Multilingual

A multilingual transcription model offering a good balance of speed and cost-effectiveness. Supports English, Spanish, German, French, Portuguese, and Italian. Features intelligent endpointing and [keyterms prompting](/streaming/prompting-and-keyterms) support for up to 100 words.

**Best for:**
- Cost-effective real-time transcription across languages
- Cost-sensitive multilingual streaming across EN/ES/DE/FR/PT/IT

  
  <br />

[Learn more about Universal-Streaming Multilingual](/streaming/getting-started/transcribe-streaming-audio)

#### Universal-Streaming English

An English transcription model offering a good balance of speed and cost-effectiveness. Features ~300ms word-by-word immutable transcripts, intelligent endpointing, and [keyterms prompting](/streaming/prompting-and-keyterms) support for up to 100 words.

**Best for:**
- Cost-effective real-time transcription for English
- English-only real-time apps — fastest and cheapest streaming option for English

  
  <br />

[Learn more about Universal-Streaming English](/streaming/getting-started/transcribe-streaming-audio)

  To learn how to specify a model, see
  [selecting a model for pre-recorded audio](/pre-recorded-audio/select-the-speech-model)
  or [selecting a model for streaming audio](/streaming/select-the-speech-model).

## Pricing

For detailed pricing information, visit our [pricing page](https://www.assemblyai.com/pricing).

### Pre-recorded

| Model           | Price per Hour | Volume discounts |
| --------------- | -------------- | ---------------- |
| Universal-3.5 Pro | $0.21/hr       | Available        |
| Universal-2     | $0.15/hr       | Available        |

### Streaming

Streaming is billed per hour of **session duration** — the total time your WebSocket connection stays open — not per hour of audio sent. See [Streaming Speech-to-Text billing](/billing-and-pricing#streaming-speech-to-text-billing) for details.

| Model                            | Price per Hour (session duration) | Volume discounts |
| -------------------------------- | --------------------------------- | ---------------- |
| Universal-3.5 Pro Streaming      | $0.45/hr                          | Available        |
| Universal-Streaming Multilingual | $0.15/hr                          | Available        |
| Universal-Streaming English      | $0.15/hr                          | Available        |

For volume discounts, please reach out to sales@assemblyai.com.

## Next steps

- Explore [Speech Understanding](/speech-understanding) features like summarization, sentiment analysis, and more
- Learn about prompting: [Universal-3.5 Pro prompting guide](/pre-recorded-audio/universal-3-5-pro/prompting) | [Universal-3.5 Pro Streaming prompting guide](/streaming/prompting-and-keyterms)

---

<a id="evaluations-docs-evaluations-mdx"></a>

# Evaluations

URL: https://www.assemblyai.com/docs/evaluations
Source: docs/evaluations.mdx
Navigation: Overview > Getting started
Description: Figure out which STT models are best for your product with an evaluation.

Choosing the right Speech-to-text model for your product requires more than reviewing public benchmarks. Public benchmarks can be misleading due to overfitting — models are often trained on the same datasets used for evaluation, inflating their reported accuracy.

Running an evaluation on your own audio data is the most reliable way to determine which model performs best for your specific use case. AssemblyAI provides evaluation tools for both pre-recorded and streaming transcription, measuring metrics that matter in production.

## Pre-recorded audio evaluations

Assess which pre-recorded audio STT model is best for your use case. Pre-recorded evaluations measure accuracy using metrics like Word Error Rate (WER) and Full-Word Error Rate (FWER), giving you a clear picture of transcription quality on your actual audio.

  Learn how to evaluate pre-recorded STT models on your own audio data.

## Streaming evaluations

Assess which real-time STT model is best for your voice agent or real-time use case. Streaming evaluations focus on latency metrics like Time to First Token (TTFT) and Time to Complete Turn (TTCT) alongside accuracy, since both speed and correctness matter for real-time applications.

  Learn how to evaluate real-time STT models for voice agents and real-time
  applications.

## Benchmarks

If you want to review AssemblyAI's current model performance before running your own evaluation, see our benchmarks for the latest accuracy and latency numbers:

- [Pre-recorded STT benchmarks](/pre-recorded-audio/benchmarks)
- [Real-time STT benchmarks](/streaming/benchmarks)

---

<a id="account-management-docs-account-management-mdx"></a>

# Account Management

URL: https://www.assemblyai.com/docs/account-management
Source: docs/account-management.mdx
Navigation: Overview > Getting started
Description: Account Management documentation.

The [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home) is where you manage everything related to your account: API keys, rate limits, cost and usage, billing, team members, personal settings, and access to the in-browser tools for each API.

The dashboard sidebar is organized into two top-level sections:

- **APIs** — playgrounds, getting started resources, logs, and comparison tools for Voice Agents, Realtime STT, Pre-Recorded STT, and the LLM Gateway.
- **Workspace** — account-wide configuration, split into **Manage** (API keys, rate limits, cost, usage) and **Settings** (personal info, organization & access, members, billing, alerts, activity logs).

## APIs section

Each API in the sidebar has its own set of tools. They aren't covered in detail here — see the corresponding product docs — but the options you'll see are:

- **Playground** — an in-browser environment for testing the API without writing code.
- **Get Started** — a quickstart and links to additional resources for that API.
- **Logs** — historical records of past requests (async transcripts, realtime sessions, etc.).
- **Compare** — side-by-side comparison of results across models (Pre-Recorded STT).
- **Truth File Corrector** — compare an async STT transcript against a ground-truth file (Pre-Recorded STT).

## Workspace > Manage

The **Manage** area is where you configure how your account talks to the API.

### API Keys

API keys are unique credentials that authenticate requests to the API. Each API key is associated with a specific project, ensuring secure and controlled access.

You can create and delete API keys based on your plan:

| Usage limits       | Free | PAYG | Contracted | Enterprise |
| ------------------ | ---- | ---- | ---------- | ---------- |
| Number of API keys | 2    | 4    | 25         | Custom     |

#### Create a new API key

1. Log in to your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).
2. In the sidebar, go to **Workspace** > **Manage** > **API Keys**.
3. Click **Create New API Key**.
4. Enter a descriptive name (for example, `Production API` or `Development API`).
5. Click **Create**.

#### Delete an API key

1. Log in to your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).
2. In the sidebar, go to **Workspace** > **Manage** > **API Keys**.
3. Locate the API key you want to delete and click **Delete**.
4. Confirm the deletion in the popup dialog.

  This action cannot be undone. Make sure no active applications are using the
  key before deletion.

  Each project must have at least one active API key at all times. You cannot delete the last remaining key in a project.

### Rate limits

The **Rate Limits** page shows the request limits in effect for your account. For details on how rate limits work and how to handle them in your application, see the [Pre-Recorded STT rate limits](/pre-recorded-audio/rate-limits) and [Real-time STT rate limits](/streaming/rate-limits) pages.

### Cost and usage

The **Cost** and **Usage** pages provide a breakdown of your spend and usage so you can track and manage costs effectively. You can analyze the data at different levels of granularity:

- Account
- Product (e.g., Speech-to-Text, Streaming, LLM Gateway)
- Models (e.g., Best, Claude Sonnet 4.6, etc.)
- Project
- API key

  Usage and spend data updates every 2 minutes. For streaming, a session must be closed before its usage populates.

## Workspace > Settings

The **Settings** area covers personal account info, organization configuration, team membership, data controls, billing, alerts, and activity logs.

### Personal Info

Manage the details tied to your individual user:

- Change the email address associated with your account.
- Change your password.
- Set up or update multi-factor authentication (MFA).
- Log out.

  You can also reach **Personal Info** from anywhere in the dashboard by clicking the portrait icon in the top-right corner.

### Organization & Access

Control how members authenticate and what protections are enforced on the organization:

- Authentication methods available to members.
- MFA enforcement for the organization.
- [Single sign-on (SSO)](/sso) configuration. SSO is a paid add-on, billed monthly for each connection. On pay-as-you-go accounts it's paid out of your prepaid balance, so auto-pay has to stay enabled and cover your connections — see [Auto-pay and SSO connections](/billing-and-pricing#auto-pay-and-sso-connections).
- **Danger Zone** — delete your account or organization. See [Deleting your account or organization](#deleting-your-account-or-organization) below.

### Members

Invite teammates to your account with role-based access control. Each member gets their own login and has a single role per account. A single person can belong to multiple accounts with different roles in each. For example, an Owner of one organization and a Reader on another.

Projects, API keys, billing, and all other account configuration are owned by the account, not by individual members. Adding, removing, or changing a member's role has no effect on any of these.

#### Roles

There are three roles. Each account has exactly one Owner, and a member has exactly one role per account:

| Role | Permissions |
| ---- | ----------- |
| **Owner** | Full control, including billing, account deletion, and ownership transfer. There is exactly one Owner per account. |
| **Admin** | Full administrative access — can do everything an Owner can, except delete the account. |
| **Reader** | Read-only access. Can view projects, API keys, usage, and transcriptions, but cannot create, edit, or delete. This is the default role for new invites. |

Here's a quick reference for what each role can do:

| Action | Owner | Admin | Reader |
| ------ | ----- | ----- | ------ |
| View members, projects, API keys, usage, transcriptions | ✓ | ✓ | ✓ |
| Invite, update, or remove members | ✓ | ✓ | ✗ |
| Create, edit, or delete projects, tokens, alerts | ✓ | ✓ | ✗ |
| View or edit billing, autopay, invoices | ✓ | ✓ | ✗ |
| Manage [Data Controls](/data-controls) (opt out of model training, set TTL, sign a BAA) | ✓ | ✓ | ✗ |
| Delete the account | ✓ | ✗ | ✗ |

Ownership transfer cannot be performed self-serve. Contact [support@assemblyai.com](mailto:support@assemblyai.com) to request a transfer.

#### Member limits

The number of members (active + pending invites combined) per account depends on your plan:

| Plan | Max members |
| ---- | ----------- |
| Free | 2 |
| PAYG | 20 |
| Contracted | 20 |
| Enterprise | 20 |

Pending invites count toward the limit. If you need a higher member limit, contact [support@assemblyai.com](mailto:support@assemblyai.com).

#### Inviting a member

Owners and Admins can invite team members from the dashboard:

1. Go to **Workspace** > **Settings** > **Members**.
2. Click **Invite Member**.
3. Enter the invitee's email address and select a role.
4. Click **Send Invite**.

The invitee receives an email with an accept link. Invite links are valid for 7 days. If one expires, the inviter must send a new one.

If your account is at its member limit, revoke a pending invite or remove an existing member before sending a new invitation.

#### Revoking an invite

To cancel a pending invite, use the delete action on the pending invite entry in the **Members** list. There is no separate revoke action. Deleting handles both pending invites and active members.

#### Removing a member

Owners and Admins can remove any non-Owner member from the **Members** list.

The Owner cannot be removed without first transferring ownership to another member. To transfer ownership, contact [support@assemblyai.com](mailto:support@assemblyai.com). After a transfer, the previous Owner is automatically demoted to Admin.

#### MFA enforcement

Owners and Admins can require all members to use multi-factor authentication (MFA) from **Workspace** > **Settings** > **Organization & Access**. When enabled, members who haven't set up MFA will be prompted to do so on their next login. The **Members** list shows the MFA enrollment status for each member.

### Data Controls

The **Data Controls** page lets Owners and Admins manage how AssemblyAI retains and uses your data — opt out of the model improvement program, set a time-to-live (TTL) for audio and transcripts, and review and sign a Business Associate Agreement (BAA). These controls are available self-serve, at no additional cost, on paid plans. See [Data Controls](/data-controls) for details.

### Billing

The **Billing** page shows:

- Your current account balance.
- Credit card on file (add or update your payment method).
- Auto-pay configuration.
- Billing history and invoices.

  You can also reach **Billing** by clicking the **View billing details** button in the bottom-left corner of the dashboard.

For details on how billing, auto-pay, and invoices work, see [Billing and Pricing](/billing-and-pricing).

### Alerts

Set up usage and balance alerts so you're notified before your balance runs low or your usage crosses a threshold you define.

### Activity Logs

The **Activity Logs** page provides an audit trail of account activity — useful for reviewing administrative changes, sign-ins, and other notable events.

## Projects

Projects can be used to isolate data for different environments or applications, e.g., production, staging, or development. Each project has its own API keys, allowing for better organization and data access control.

Transcripts and other project-specific data are accessible only within the project they were created in — an API key from one project cannot access historical transcripts or data from another project. This separation maintains data security and prevents unintended cross-project access.

This project-level scoping also applies to files uploaded via the `/upload` endpoint. An API key can only transcribe files that were uploaded within the same project. If you need to transcribe the same file across multiple projects, you must upload it separately in each project.

You can create, rename, and delete projects based on your plan:

| Usage limits       | Free | PAYG | Contracted | Enterprise |
| ------------------ | ---- | ---- | ---------- | ---------- |
| Number of projects | 2    | 2    | 5          | Custom     |

## Account switching

When you sign in, AssemblyAI logs you straight into your account, or shows you a list to choose from if you belong to multiple accounts.

If you belong to multiple accounts, an organization switcher is available in the dashboard so you can move between them without logging out.

### Migrating existing accounts to multi-user

If you already have multiple AssemblyAI accounts, the recommended path is to designate one as your primary account and invite your other users into it.

**A few important notes:**

- Existing accounts are **not** automatically merged. Each account retains its own projects, API keys, transcripts, and billing history.
- Users invited to another account will retain their original account. They may see multiple accounts in the account switcher after accepting an invite.
- Going forward, manage billing changes in a single account to avoid confusion and unexpected charges.
- To move workflows, switch the API keys in your applications to point to the primary account's API keys.

**Best practice:** Start from the account you want to treat as primary, then invite teammates into that account using the invite flow described above.

## Changing your account email

You can change your account email address yourself from **Workspace** > **Settings** > **Personal Info**. You can also reach this page by clicking the portrait icon in the top-right corner of the dashboard.

The Owner's email address is used for billing communications. If you [transfer account ownership](#members) to another member, billing emails go to the new Owner's address.

## Deleting your account or organization

If you are the sole member of your account, you can delete it from the dashboard:

1. Log in to your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).
2. In the sidebar, go to **Workspace** > **Settings** > **Organization & Access**.
3. Scroll to the **Danger Zone**.
4. Click the option to delete your account or organization and follow the prompts.

Account deletion is permanent and cannot be undone. Make sure to back up any important information before proceeding with deletion.

If your account has other members, remove them or transfer ownership before attempting to delete the account. If you encounter any issues, contact [support@assemblyai.com](mailto:support@assemblyai.com).

---

<a id="single-sign-on-sso-docs-sso-mdx"></a>

# Single Sign-On (SSO)

URL: https://www.assemblyai.com/docs/sso
Source: docs/sso.mdx
Navigation: Overview > Getting started
Description: Set up SAML or OIDC Single Sign-On for your AssemblyAI organization, including step-by-step configuration for Okta, Google Workspace, and Microsoft Entra ID.

AssemblyAI supports SAML and OIDC Single Sign-On (SSO) for organizations on Contract and PAYG plans. SSO lets your organization members sign in with your company's identity provider (IdP) — such as Okta, Google Workspace, or Microsoft Entra ID — instead of a password or magic link.

SSO is self-served: you configure it yourself from [**Settings → Organization & Access**](https://www.assemblyai.com/dashboard/settings/organization-and-access) in the AssemblyAI dashboard. No contact with support is required to get started.

  Only organization **owners** and **admins** can access and configure SSO and
  billing settings.

With SSO you can:

- Let members sign in through your company's IdP
- Auto-create new members on their first SSO login with Just-in-Time (JIT) provisioning
- Enforce SSO-only login by disabling all other sign-in methods

## Pricing

| Detail      | Info                                                     |
| ----------- | -------------------------------------------------------- |
| Price       | $199/month per connection                                 |
| Limit       | 1 connection per organization                             |
| Billing     | Line item on invoice — review in [**Settings → Billing**](https://www.assemblyai.com/dashboard/settings/billing) |
| Eligibility | Contract and PAYG plans                                   |

  Billing starts when a connection becomes **Active**, not when you create it —
  a connection sitting in **Pending** while you finish IdP configuration is not
  charged. SSO billing does not prorate: once a connection goes active, you're
  charged the full month's price regardless of how many days remain in the
  billing period. If you delete a connection mid-month, the connection is
  terminated immediately but the current period's charge is not refunded.

### Balance and auto-pay requirements on PAYG plans

On PAYG plans the subscription is charged against your prepaid balance, so two
things have to be true before you can add a connection.

**Your balance has to cover the connection.** A connection is charged as soon as
it becomes active, so you need at least **$199** in balance — one connection's
price, whatever else you already have. If you're short when you add a connection,
the dashboard offers to charge your card for the difference (plus $10, so a small
amount of usage doesn't put you straight back under) and creates the connection
once that payment goes through. Without a payment method on file there's nothing
to charge, so add one first.

**Your auto-pay settings have to keep it covered.** This is what pays for the
connection each month after the first, so
[auto-pay](/billing-and-pricing#how-does-auto-pay-work) needs:

- A refill threshold of at least **$199** for each active connection.
- A refill amount at least **$10** above your refill threshold.

If your settings are below that, creating or activating a connection raises them
for you. It never lowers a threshold or amount you've set higher, and raising
them isn't itself a charge — auto-pay charges your card on its own schedule, when
your balance falls past the threshold.

Both requirements apply when you *create* a connection as well as when it goes
active, so you can't end up with a connection you're unable to activate.

While a connection is active:

- Auto-pay can't be switched off.
- Your refill threshold can't drop below the price of your active connections.
- If the invoice covering the connection isn't paid, the connection is removed —
  see [What happens if I delete my SSO
  connection](#what-happens-if-i-delete-my-sso-connection-while-members-are-using-it)
  for what members experience when a connection goes away.

Deleting the connection lifts these restrictions, and you're free to lower
auto-pay again.

  Contract plans are invoiced at the end of the month and don't use auto-pay, so
  none of this applies to them.

## How SSO login works

**SP-initiated (standard) flow:**

1. A user clicks **Sign in with SSO** on the AssemblyAI login page.
2. The user is redirected to your company's IdP.
3. The IdP authenticates the user and sends the authentication response back to AssemblyAI — a signed SAML assertion for SAML connections, or an authorization code for OIDC connections.
4. AssemblyAI validates the response and logs the user in.

**IdP-initiated flow:**

1. A user clicks the AssemblyAI tile in their IdP dashboard (for example, Okta).
2. The IdP sends a SAML assertion directly to AssemblyAI's callback URL.
3. AssemblyAI processes the assertion and logs the user in.

## Set up SSO

You configure SSO from [**Settings → Organization & Access**](https://www.assemblyai.com/dashboard/settings/organization-and-access) in the dashboard. When you create a connection, you choose a **connection type**: SAML or OIDC. Which one to pick depends on your IdP — most enterprise IdPs support both, and either works with AssemblyAI.

### SAML setup

SAML setup always follows the same pattern, regardless of IdP:

1. **AssemblyAI → IdP**: Copy the **ACS URL** and **Audience URI** from your AssemblyAI SSO connection into your IdP's application settings (sometimes labeled **SP SSO URL** and **SP Entity ID**).
2. **In your IdP**: Configure attribute mapping so the IdP sends the required attributes (see below).
3. **IdP → AssemblyAI**: Copy your IdP's **Metadata URL** (recommended) — or the IdP Entity ID, IdP SSO URL, and X.509 certificate individually — into your AssemblyAI connection.
4. Once all required fields are present and valid, the connection status flips from **Pending** to **Active**.

  Configuring via **Metadata URL** is recommended wherever your IdP provides
  one. AssemblyAI auto-populates the Entity ID, SSO URL, and certificate from
  it, and can re-fetch updated metadata later — for example, after your IdP
  rotates its signing certificate.

### OIDC setup

OIDC setup follows a similar two-direction pattern:

1. **AssemblyAI → IdP**: Copy the **Redirect URL** from your AssemblyAI SSO connection into your IdP as the **Sign-in Redirect URI**, using the **Authorization Code** grant type.
2. **IdP → AssemblyAI**: Copy the **Client ID**, **Client Secret**, and **Issuer URL** (usually your IdP's hostname) from the IdP application into your AssemblyAI connection.
3. Once all required fields are present and valid, the connection status flips from **Pending** to **Active**.

  The **Issuer URL** must be the plain issuer base URL — not the discovery URL
  ending in `/.well-known/openid-configuration`. Using the discovery URL is a
  common cause of failed OIDC setups.

### Required SAML attributes

For SAML connections, your IdP must send the following attributes in the SAML assertion:

| SAML attribute | Maps to    |
| -------------- | ---------- |
| `email`        | Email      |
| `firstName`    | First name |
| `lastName`     | Last name  |

Your IdP can send a combined `full_name` attribute instead of separate `firstName` and `lastName` attributes.

If these attributes are missing or incorrectly named, login fails with an attribute mapping error. If your IdP lets you configure a **NameID** format, set it to the user's email address.

### JIT provisioning

Just-in-Time provisioning controls whether new members are automatically created on their first SSO login:

- **Anyone**: Any user who authenticates via SSO is automatically added as a member — no prior invite required.
- **Nobody**: Only pre-invited members can log in via SSO.

  An **Active** connection alone doesn't mean new users can log in. If you want
  new users auto-created on first login, you also need to set **JIT
  Provisioning → Anyone**. This is a separate step from the IdP configuration
  itself.

### Sign-in methods

The **Sign-in Methods** setting controls which authentication methods are available to your organization's members: Magic Link, Password, Google OAuth, and SSO. To enforce SSO-only login, disable all non-SSO methods. At least one method must always be enabled.

  Deleting an SSO connection while SSO-only login is enforced will lock out all
  members once their sessions expire. Re-enable an alternative sign-in method
  before deleting the connection.

### MFA

Authenticator app (TOTP) is the supported MFA method. Enabling **Require MFA for all members** forces all members to enroll on their next sign-in.

If your IdP already enforces MFA, AssemblyAI's MFA requirement may not trigger for SSO users — MFA has already been handled at the IdP level.

## IdP configuration guides

The steps below cover the most common identity providers. For IdPs not listed here, follow the generic patterns in [Set up SSO](#set-up-sso) — any SAML 2.0- or OIDC-compliant IdP works.

### Okta (SAML)

1. In the Okta Admin console, go to **Applications → Create App Integration → SAML 2.0**.
2. On the **Configure SAML** screen, enter:
   - **Single sign-on URL** = your AssemblyAI **ACS URL**
   - **Audience URI (SP Entity ID)** = your AssemblyAI **Audience URI**
   - **Name ID format** = EmailAddress
   - **Application username** = Email
   - **Attribute Statements**:
     - `firstName` → `user.firstName`
     - `lastName` → `user.lastName`
     - `id` → `user.id`
3. Copy the **Metadata URL** from Okta's **Sign On** settings tab into the Single Sign-On section of your AssemblyAI settings.
4. Assign the app to users under **Assignments** in Okta.

  If a user isn't assigned to the application in Okta, they can't sign in via
  SSO — even if JIT provisioning is set to Anyone. If a user exists in Okta but
  their SSO login fails, check their app assignment first.

For more detail, see Okta's [SAML app integration documentation](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_saml.htm).

### Okta (OIDC)

1. In the Okta Admin console, go to **Applications → Create App Integration → OIDC - OpenID Connect** and select **Web Application** as the application type.
2. Under **General Settings**:
   - **Grant type** = Authorization Code
   - **Sign-in redirect URIs** = your AssemblyAI **Redirect URL**
3. Assign the app to users or groups under **Assignments**.
4. Save the app, then copy the following from the **Client Credentials** section into your AssemblyAI SSO connection:
   - **Client ID**
   - **Client Secret**
   - **Issuer URL** = your Okta org URL (for example, `https://your-company.okta.com`)

  The same assignment requirement applies as with SAML: users who aren't
  assigned to the application in Okta can't sign in via SSO, even if JIT
  provisioning is set to Anyone.

For more detail, see Okta's [OIDC app integration documentation](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_oidc.htm).

### Google Workspace (SAML)

1. In the Google Admin console, go to **Apps → Web and mobile apps → Add app → Add custom SAML app**.
2. Google displays an **Option 2** panel with the **IdP Entity ID**, **SSO URL**, and **Certificate**. Paste these into your AssemblyAI SSO connection using the manual configuration option.
3. Back in Google, under **Service provider details**, enter:
   - **ACS URL** = your AssemblyAI **ACS URL**
   - **Entity ID** = your AssemblyAI **Audience URI**
   - **Name ID format** = EMAIL
   - **Name ID** = Primary email
4. Under **Attributes**, map:
   - First name → `firstName`
   - Last name → `lastName`
5. Set **User access** (Service status) to **ON** for the relevant organizational unit or group.

  Google Workspace doesn't provide a metadata URL for custom SAML apps — the
  manual configuration option (Entity ID, SSO URL, certificate) is the expected
  path. This is normal, not a misconfiguration.

For more detail, see Google's [custom SAML app documentation](https://support.google.com/a/answer/6087519).

### Microsoft Entra ID / Azure AD (SAML)

1. In the Entra admin center, go to **Enterprise applications → New application → Create your own application → Non-gallery**.
2. Under **Single Sign-On → SAML → Basic SAML Configuration**, enter:
   - **Identifier (Entity ID)** = your AssemblyAI **Audience URI**
   - **Reply URL (ACS URL)** = your AssemblyAI **ACS URL**
3. Under **Attributes & Claims**:
   - Edit the **Unique User Identifier (Name ID)** claim and set its source to `user.primaryauthoritativeemail`.
   - Add additional claims: `firstName` → `user.givenname`, `lastName` → `user.surname`, `id` → `user.objectid`.
4. Copy the **App Federation Metadata Url** from the **SAML Certificates** section into your AssemblyAI SSO connection.
5. Add users or groups under **Users and groups** in the Entra application.

  Entra's default Name ID claim is often not the user's email address. If SSO
  login fails with an "Email format is invalid" error and you're using Entra
  ID, check the Name ID claim source first — this is the most common Entra
  misconfiguration.

For more detail, see Microsoft's [SAML-based single sign-on documentation](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/add-application-portal-setup-sso).

### Microsoft Entra ID / Azure AD (OIDC)

1. In the Entra admin center, go to **App registrations → New registration** and select **Accounts in this organizational directory only**.
2. Under **Authentication → Add a platform → Web**, set the **Redirect URI** to your AssemblyAI **Redirect URL**.
3. Under **Certificates & secrets → New client secret**, create a secret and copy the secret **value** into the Client Secret field of your AssemblyAI connection.
4. From the app's **Overview** page:
   - **Client ID** = the **Application (client) ID**
   - **Issuer URL** = `https://login.microsoftonline.com/<DIRECTORY_ID>/v2.0`, substituting your **Directory (tenant) ID**

  Two common copy-paste errors to watch for: copy the client secret's
  **value**, not its secret ID — and build the Issuer URL in the exact format
  above with your tenant ID substituted in. The Issuer isn't a value you can
  copy verbatim from a single field in the Entra UI, and it isn't just your
  Entra domain.

For more detail, see Microsoft's [OpenID Connect documentation](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc).

### Attribute name cheat sheet

| IdP                | Email                                                    | First name                    | Last name                   |
| ------------------ | -------------------------------------------------------- | ----------------------------- | --------------------------- |
| Okta               | email (via NameID)                                       | `firstName`                   | `lastName`                  |
| Google Workspace   | email (via NameID)                                       | `firstName`                   | `lastName`                  |
| Microsoft Entra ID | email (via NameID, from `user.primaryauthoritativeemail`) | `firstName` (`user.givenname`) | `lastName` (`user.surname`) |
| Auth0              | email                                                    | `firstName`                   | `lastName`                  |

## Troubleshooting

### I set up SSO but login still goes to the regular login page

The most likely causes:

1. **SSO is not enabled as a sign-in method.** Check [**Settings → Organization & Access**](https://www.assemblyai.com/dashboard/settings/organization-and-access) **→ Sign-in Methods** — SSO must be toggled on.
2. **The connection is in Pending status.** The configuration is incomplete. Finish configuring the connection — for SAML, the Entity ID, SSO URL, and certificate must all be present; for OIDC, the Client ID, Client Secret, and Issuer URL.
3. **You're testing an IdP-initiated flow.** Some flows only work SP-initiated. Try initiating login from the AssemblyAI login page instead.
4. **The ACS URL or Audience URI wasn't saved in your IdP.** Copy AssemblyAI's ACS URL and Audience URI into your IdP configuration.

### "SAML Connection attribute mapping is missing either the 'full_name' or both the 'first_name' and 'last_name' fields"

Your IdP isn't sending the required name attributes. Configure your IdP to send `firstName` and `lastName` (or a combined `full_name` attribute). This is configured on the IdP side — see the [attribute name cheat sheet](#attribute-name-cheat-sheet) for the IdP-specific field names.

### "Email format is invalid" error

Your IdP is sending the email in an unexpected format, or the email attribute mapping points to the wrong field. Ensure your IdP sends the email as a standard email attribute in the SAML assertion. On Microsoft Entra ID, check that the Name ID claim source is set to `user.primaryauthoritativeemail`.

### A user logged in via SSO but wasn't added as a member

JIT provisioning is set to **Nobody**. Either change JIT provisioning to **Anyone**, or pre-invite the user from the Members page before they log in via SSO.

### OIDC login fails after setup

The most common OIDC misconfigurations:

1. **Redirect URI mismatch.** The Sign-in Redirect URI in your IdP must exactly match the Redirect URL from your AssemblyAI connection.
2. **Wrong Issuer URL.** Use the plain issuer base URL, not the discovery URL ending in `/.well-known/openid-configuration`.
3. **Missing or wrong client secret.** Make sure a client secret is set, and that you copied the secret's value (not its ID). If the secret has expired, generate a new one and update your connection.

### SSO was working, but broke after our IdP certificate rotated

If you configured the connection manually (not via Metadata URL), update the X.509 certificate in your AssemblyAI SSO connection with the new certificate. If you used the Metadata URL option, re-save your connection so AssemblyAI re-fetches the updated metadata.

  If your IdP has multiple signing certificates, make sure the **active** one
  is configured in your connection.

### What happens if I delete my SSO connection while members are using it?

- The connection is deleted immediately.
- Members who logged in through the SSO connection being deleted are signed out immediately. Members who signed in through another method (magic link, password, Google OAuth, or a different SSO connection) keep their active sessions.
- Future SSO logins will fail on that connection.
- If SSO was the only allowed sign-in method, members will be locked out on their next login. Re-enable another sign-in method before deleting the connection.

## Current limitations

- Maximum of 1 SSO connection per organization
- No SCIM support — user provisioning and deprovisioning from the IdP isn't automated
- Custom attribute mapping is limited to email, first name, and last name

---

<a id="billing-and-pricing-docs-billing-and-pricing-mdx"></a>

# Billing and Pricing

URL: https://www.assemblyai.com/docs/billing-and-pricing
Source: docs/billing-and-pricing.mdx
Navigation: Overview > Getting started
Description: Billing and Pricing documentation.

AssemblyAI uses pay-as-you-go pricing with no contracts, minimums, or monthly subscriptions. You only pay for what you use, and failed transcripts aren't charged. For the latest per-hour rates across all models, see the [pricing page](https://www.assemblyai.com/pricing) or the [Models pricing section](/getting-started/models#pricing).

## How billing works

- Add a credit card and deposit funds into your account. Funds are drawn down as you use the API.
- Rates are listed per hour for simplicity, but pre-recorded audio is pro-rated to the exact second of audio processed.
- Credits are deducted only after a successful transcription completes. If a request errors, you aren't charged for it.
- Monitor your balance on the **Billing** page (under **Workspace** > **Settings** > **Billing** in the [dashboard](https://www.assemblyai.com/dashboard/home)) and enable [auto-pay](#how-does-auto-pay-work) to avoid service interruptions. End-of-month invoicing is available for enterprise customers.
- Set up billing alerts under **Workspace** > **Settings** > **Alerts** in the dashboard to get notified before your balance runs low.

## Payment method and invoices

AssemblyAI accepts all major credit cards. ACH transfers are available in some cases — email [support@assemblyai.com](mailto:support@assemblyai.com) to learn more.

To update the credit card on your account, log in to the [dashboard](https://www.assemblyai.com/dashboard/home), go to **Workspace** > **Settings** > **Billing**, and update your payment details. You can also click **View billing details** in the bottom-left corner of the dashboard to jump straight to this page.

To change the company information shown on your invoices, email [support@assemblyai.com](mailto:support@assemblyai.com) with the details you'd like to appear.

## Pre-recorded Speech-to-Text billing

Pre-recorded transcription is billed on the duration of the submitted audio or video file in seconds, multiplied by the hourly rate for the selected speech model. Add-on features (for example, Entity Detection or Medical Mode) are billed separately at their own hourly rates, also pro-rated to the exact second.

## Streaming Speech-to-Text billing

  **Streaming is billed per session, not per second of audio**

  Streaming Speech-to-Text is billed on the total duration that your WebSocket connection stays open, not on the amount of audio you send. You're charged for the entire session — including any time the connection is idle with no audio flowing — so it's important to close streaming sessions as soon as you're done with them.

Billing for streaming sessions starts when the WebSocket connection opens and stops when the session ends. This model gives you full control over cost: you can keep a stream open continuously for instant response, or open streams on demand to minimize spend.

Each open session is billed independently on its own WebSocket-open duration, so concurrent sessions accumulate billed time in parallel. For example, two streaming sessions running simultaneously for 5 minutes each — including a single call that is dual-streamed under two separate session IDs — bill as 10 minutes of total session time at the streaming rate listed in the [Models pricing section](/getting-started/models#pricing).

Follow these best practices to avoid unexpected charges:

- Always send a session termination message when your application is finished with a stream. This is what stops billing for the session.
- If a session is not closed, it will automatically close after 3 hours — and you'll be billed for the full 3 hours of session time, regardless of how much audio was actually streamed.
- Improperly closed streaming sessions are the most common cause of unexpected charges that lead to negative account balances.

For details on how to terminate a session, see the [Session termination](/streaming/getting-started/transcribe-streaming-audio) message sequence reference.

## Voice Agent API billing

  **Voice Agent sessions are billed on WebSocket-open duration**

  Voice Agent sessions are billed on the total time the WebSocket connection stays open, not on how much audio the user or agent speaks. Always send [`session.end`](/voice-agents/voice-agent-api/events-reference#sessionend) before closing the socket on an intentional disconnect.

Billing for a Voice Agent session starts when the WebSocket connection opens and stops when the session ends. Sessions bill at the Voice Agent rate listed on the [pricing page](https://www.assemblyai.com/pricing), pro-rated to the second, and each concurrent session accumulates billed time independently.

A few Voice Agent-specific behaviors to be aware of:

* **30-second resume grace window.** If the client closes the WebSocket without sending `session.end`, the server keeps the session open for 30 seconds so the client can reconnect with [`session.resume`](/voice-agents/voice-agent-api/events-reference#sessionresume). That grace window is billable. Send `session.end` on any intentional disconnect (user hung up, "End call" button, page unload) to avoid it. See [Unexpected billing after the call ended](/voice-agents/voice-agent-api/troubleshooting#unexpected-billing-after-the-call-ended).
* **Idle time is billable.** Just like Streaming STT, silence and idle periods on an open Voice Agent connection are billed. Close sessions as soon as you're done with them.
* **Any LLM Gateway calls made by the agent are billed separately** at the LLM Gateway token rates for the model in use — see [LLM Gateway billing](#llm-gateway-billing) below.

For the full session lifecycle and best practices, see the [Voice Agent API overview](/voice-agents/voice-agent-api) and [Ending the session cleanly](/voice-agents/voice-agent-api/browser-integration#6-ending-the-session-cleanly).

## LLM Gateway billing

LLM Gateway is billed on **input and output tokens**, not on audio duration or session time. Each request is charged at the per-1M-token input and output rates for the specific model you call. Rates vary by model — Claude, GPT, Gemini, and the other supported providers each have their own input and output prices, listed in the Rates table on the [Billing page](https://www.assemblyai.com/dashboard/home) of the dashboard and on the [pricing page](https://www.assemblyai.com/pricing).

A few things to keep in mind:

* **Output tokens usually cost more than input tokens.** Output pricing reflects the compute needed to generate the response. See [Understanding input and output tokens for LLM Gateway](/faq/understanding-input-and-output-tokens-for-llm-gateway).
* **LLM Gateway is not covered by the \$50 free tier.** The free credits granted to new accounts apply to Pre-recorded STT, Real-time STT, Voice Agent API, Speech Understanding, and Guardrails. LLM Gateway usage is billed from your account balance from the first request.
* **In-region endpoints have a 10% surcharge as of July 1, 2026.** Requests to the US (`llm-gateway.assemblyai.com`) or EU (`llm-gateway.eu.assemblyai.com`) in-region endpoints are 10% higher than global routing as a direct pass-through of provider price increases, with no AssemblyAI upcharge. To keep the standard rate, opt into [global routing](/llm-gateway/cloud-endpoints-and-data-residency#global-routing).
* **Streaming and Voice Agent calls that invoke LLM Gateway are billed additively.** The Streaming or Voice Agent session is billed on its WebSocket-open duration, and any LLM Gateway request made during that session is billed separately on its input and output tokens.

To estimate input token cost ahead of time, see [Estimate input token costs for LLM Gateway](/guides/counting-tokens).

## Multichannel billing

When [multichannel transcription](/pre-recorded-audio/transcribe-multiple-audio-channels) is enabled, each channel is transcribed and billed separately. The total cost is the audio duration multiplied by the model's hourly rate, multiplied by the number of channels.

For example, a 5-minute recording with three channels is billed as 15 minutes of audio (5 minutes × 3 channels) at the selected model's rate.

## Free tier and credits

New accounts receive \$50 in free credits for Pre-recorded STT, Real-time STT, Voice Agent API, Speech Understanding, and Guardrails. Credits do not expire, and any unused credits are retained on your account when you upgrade by adding a credit card. LLM Gateway is not included in the free tier.

Once your free credits are used up, add a credit card to keep using the API. If your balance reaches \$0 without [auto-pay](#how-does-auto-pay-work) enabled, API access is paused until you top up.

## Volume discounts

If you plan to send large volumes of audio or video through the API, [contact sales](https://www.assemblyai.com/contact) to see if you qualify for a volume discount.

## Startup and Y Combinator pricing

Early-stage startups can apply for the [AssemblyAI Startup Program](https://www.assemblyai.com/startup-program), designed to help startups build with the speech-to-text API without financial constraints.

Y Combinator companies qualify for special pricing — [contact sales](https://www.assemblyai.com/contact) to discuss the discount and see if you qualify.

## AWS Marketplace

AssemblyAI is available on the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-n24l7xlhzr4o6). Purchasing there lets you consolidate billing with your existing AWS account and apply your spend toward AWS committed-use agreements.

Marketplace changes billing only. Your usage still runs on the AssemblyAI cloud API. Endpoints, API keys, and code are unchanged, and there is nothing to migrate in your integration.

### Subscribing

1. Subscribe through the [listing](https://aws.amazon.com/marketplace/pp/prodview-n24l7xlhzr4o6). New subscriptions include a 90-day trial with \$50 in usage credits, then enroll automatically in pay-as-you-go.
2. Complete the fulfillment step, which links your AWS entitlement to a single AssemblyAI workspace.

  **The entitlement attaches to one workspace**

  If your organization has more than one AssemblyAI account, the entitlement links to whichever workspace completes the fulfillment step. Accounts are never merged, so billing will not follow usage on a different account.

  Before subscribing, decide which workspace should be the billed one and make sure the person subscribing on the AWS side signs in to that workspace during fulfillment. See [Migrating existing accounts to multi-user](/account-management#migrating-existing-accounts-to-multi-user).

### If your plan still shows Free after subscribing

1. Check the subscription start date in your AWS console. A future-dated subscription means the entitlement is not effective yet, and re-running the fulfillment link will not change your plan.
2. Confirm which workspace the entitlement resolved to. Your AWS account team can confirm whether the entitlement was issued and where it was mapped.
3. To avoid being blocked in the meantime, add a card under **Workspace** > **Settings** > **Billing**. This converts the workspace to pay-as-you-go immediately, your existing credits are retained and drawn down first, and billing consolidates back to AWS once the entitlement lands.

### Discounted rates

The listing self-serves at list pricing. For volume-based rates, [contact sales](https://www.assemblyai.com/contact/sales) to request a private offer, which you accept through AWS.

  Marketplace billing is tied to an offer with a term. When the offer expires, a new one must be accepted to keep billing running through AWS.

## Billing and customer usage tracking

The **Cost** and **Usage** pages under **Workspace** > **Manage** in your dashboard provide a breakdown of spend and usage. If you need to track usage programmatically on a per-customer basis, you can use webhooks with custom query parameters for async transcription, or capture the `session_duration_seconds` from the WebSocket Termination event for streaming.

For async transcription, append a `customer_id` query parameter to your webhook URL when submitting a transcription:

```
https://your-domain.com/webhook?customer_id=customer_123
```

When the transcription completes, extract the `customer_id` from the webhook URL and retrieve the `audio_duration` from the transcript response to record usage.

For streaming transcription, manage customer IDs in your application state and capture the `session_duration_seconds` from the Termination event. AssemblyAI bills streaming based on session duration.

  Creating separate API keys for each customer is not recommended. Use webhooks
  with metadata instead.

## How does Auto-pay work?

Auto-pay automatically recharges your account when your balance falls below a specified threshold. When triggered, it charges your card to bring your balance back to a predetermined amount. Auto-pay is recommended for production environments to prevent service interruptions.

### Example

If you set:

- **Whenever my balance falls below:** \$25
- **Bring my balance back to:** \$50

When your balance of \$26 drops to \$21 after a \$5 charge, auto-pay will add \$29 to reach your \$50 target balance.

When your balance hits \$0 or goes negative without auto-pay, your API access is restricted until you add funds. You'll receive this error message: `{"error": "Your current account balance is negative. Please top up to continue using the API."}`

Enable auto-pay and maintain a healthy balance to ensure uninterrupted API access. Set your threshold based on your typical monthly usage to avoid frequent small charges.

You can also disable auto-pay at any time from **Workspace** > **Settings** > **Billing** in the [dashboard](https://www.assemblyai.com/dashboard/home), unless your account has an active [single sign-on connection](#auto-pay-and-sso-connections).

### Auto-pay and SSO connections

[Single sign-on](/sso) is a paid add-on, billed monthly for each connection at \$199 per connection. On pay-as-you-go accounts that subscription is charged against your prepaid balance, which means two things.

**Your balance has to cover a connection before you can add one** — at least \$199, one connection's price. If you're short, the dashboard offers to charge your card for the difference and adds the connection once that payment clears. Without a payment method on file, add one first.

**Auto-pay is what keeps it covered after that**, so your settings have to be at least:

- **Whenever my balance falls below:** \$199 for each active connection.
- **Bring my balance back to:** \$10 above your threshold.

Creating or activating a connection raises your auto-pay settings to that minimum if they're below it. It never lowers settings you've already chosen, and raising them isn't itself a charge — auto-pay charges your card on its own schedule.

While a connection is active, auto-pay can't be switched off and your threshold can't drop below the price of your active connections. If the invoice covering a connection isn't paid, the connection is removed. Deleting the connection releases the requirement, so you're free to lower auto-pay again afterwards.

End-of-month invoiced accounts are billed in arrears and don't use auto-pay, so these requirements don't apply.

---

<a id="introducing-universal-3-5-pro-docs-getting-started-universal-3-5-pro-mdx"></a>

# Introducing Universal-3.5 Pro

URL: https://www.assemblyai.com/docs/getting-started/universal-3-5-pro
Source: docs/getting-started/universal-3-5-pro.mdx
Navigation: Overview > Getting started
Description: Learn how to transcribe audio using Universal-3.5 Pro.

## Overview

Universal-3.5 Pro is our most powerful Voice AI model, designed to capture the "hard stuff" that traditional ASR models struggle with. It delivers state-of-the-art accuracy for entities, rare words, and domain-specific terminology out of the box, with code switching and optional prompting for more control. It's also our fastest model, so you get the best accuracy without sacrificing speed.

Universal-3.5 Pro is available for both **pre-recorded (async)** and **streaming** use cases. Configuration and settings differ between the two because streaming is optimized for real-time audio utterances typically under 10 seconds, with special efficiencies built into the model for low-latency turn detection and voice agent workflows.

Based on your use case, navigate to the appropriate guide below:

  
    For pre-recorded audio files. Supports long-form audio, prompting, keyterms
    prompting, and full language detection.
  
  
    For real-time audio streams. Optimized for low-latency turn detection, voice
    agents, and live transcription.

---

<a id="end-to-end-examples-docs-getting-started-end-to-end-examples-mdx"></a>

# End-to-end examples

URL: https://www.assemblyai.com/docs/getting-started/end-to-end-examples
Source: docs/getting-started/end-to-end-examples.mdx
Navigation: Overview > Getting started > End-to-end examples
Description: Copy-paste pipelines that combine multiple AssemblyAI products in a single script.

## Overview

Each example below is a self-contained script that wires together several AssemblyAI products into a working pipeline. Run one, see the polished output, and customize from there.

Every example uses placeholder API keys (`YOUR_API_KEY`). Replace them with your actual key from the [AssemblyAI dashboard](https://www.assemblyai.com/dashboard).

---

## Pre-recorded pipelines

These pipelines transcribe an existing audio file, then enrich the transcript with Speech Understanding features and LLM Gateway analysis.

| Pipeline | Products used | Best for |
| --- | --- | --- |
| [Meeting notetaker](/getting-started/end-to-end-examples/meeting-notetaker) | STT + speaker diarization + Speaker Identification + language detection + LLM Gateway | Team meetings, standups, all-hands |
| [Sales call intelligence](/getting-started/end-to-end-examples/sales-call-intelligence) | STT + speaker diarization + Speaker Identification + sentiment analysis + LLM Gateway | Revenue teams, coaching, QA |
| [Medical scribe](/getting-started/end-to-end-examples/medical-scribe) | STT + speaker diarization + Speaker Identification + Medical Mode + entity detection + LLM Gateway | Clinical documentation, SOAP notes |
| [Content repurposing](/getting-started/end-to-end-examples/content-repurposing) | STT + key phrases + LLM Gateway | Podcasts, webinars, marketing |

---

## Streaming pipelines

These pipelines use the [Real-time STT API](/streaming/getting-started/transcribe-streaming-audio) to transcribe audio in real time from a microphone, with optional LLM Gateway integration for live analysis.

| Pipeline | Products used | Best for |
| --- | --- | --- |
| [Real-time meeting assistant](/getting-started/end-to-end-examples/real-time-meeting-assistant) | Real-time STT + Universal-3.5 Pro + LLM Gateway | Live captions, real-time summaries |
| [Real-time live captioner](/getting-started/end-to-end-examples/real-time-live-captioner) | Real-time STT + keyterms prompting | Accessibility, live events |

---

## Customize and extend

Each pipeline above is a starting point. Here are common ways to build on them:

- **Swap LLM models** — Change the `model` parameter in LLM Gateway requests to use any of the [25+ supported models](/llm-gateway/available-models) (Claude, GPT, Gemini, and more).
- **Add structured output** — Use [Structured Outputs](/llm-gateway/structured-outputs) to constrain LLM responses to a JSON schema for easier downstream processing.
- **Add PII redaction** — Enable [PII Redaction](/guardrails/redact-pii-from-transcripts) to automatically mask sensitive information before it reaches the LLM.
- **Use Speaker Identification** — Replace generic speaker labels with real names using [Speaker Identification](/speech-understanding/speaker-identification).
- **Add Translation** — Translate transcripts into 86 languages using [Translation](/speech-understanding/translation).
- **Use webhooks** — Replace polling with [webhooks](/pre-recorded-audio/webhooks) for production workloads so your server gets notified when transcription completes.

## Next steps

- [Pre-recorded STT quickstart](/pre-recorded-audio/getting-started/transcribe-an-audio-file) — Step-by-step guide for your first transcription
- [Real-time STT quickstart](/streaming/getting-started/transcribe-streaming-audio) — Set up real-time transcription
- [LLM Gateway overview](/llm-gateway/quickstart) — Explore all available models and features

---

<a id="meeting-notetaker-docs-getting-started-end-to-end-examples-meeting-notetaker-mdx"></a>

# Meeting notetaker

URL: https://www.assemblyai.com/docs/getting-started/end-to-end-examples/meeting-notetaker
Source: docs/getting-started/end-to-end-examples/meeting-notetaker.mdx
Navigation: Overview > Getting started > End-to-end examples
Description: Transcribe a team meeting with speaker labels, identify speakers by name, then generate structured meeting notes with LLM Gateway.

Transcribe a meeting recording with speaker labels and automatic language detection, identify speakers by name, then send the transcript to LLM Gateway for a formatted summary with action items.

**Products used:** [Pre-recorded STT](/pre-recorded-audio/getting-started/transcribe-an-audio-file) + [speaker diarization](/pre-recorded-audio/label-speakers) + [Speaker Identification](/speech-understanding/speaker-identification) + [language detection](/pre-recorded-audio/language-detection) + [LLM Gateway](/llm-gateway/quickstart)

**Model selection:** This example uses both `universal-3-5-pro` and `universal-2` for broad language coverage across 99 languages. If your meetings are English-only, you can use `universal-3-5-pro` alone for the highest accuracy.

```python expandable
import requests
import time

# ── Config ────────────────────────────────────────────────────
base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

audio_url = "https://assembly.ai/wildfires.mp3"

# ── Step 1: Transcribe with speaker labels + language detection ──
data = {
    "audio_url": audio_url,
    "language_detection": True,
    "speaker_labels": True,
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
response.raise_for_status()
transcript_id = response.json()["id"]

while True:
    result = requests.get(f"{base_url}/v2/transcript/{transcript_id}", headers=headers).json()
    if result["status"] == "completed":
        break
    elif result["status"] == "error":
        raise RuntimeError(f"Transcription failed: {result['error']}")
    time.sleep(3)

# ── Step 2: Identify speakers by name ──
understanding_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers=headers,
    json={
        "transcript_id": transcript_id,
        "speech_understanding": {
            "request": {
                "speaker_identification": {
                    "speaker_type": "name",
                    "known_values": ["Alice", "Bob"],  # Replace with actual participant names
                }
            }
        },
    },
)
understanding_response.raise_for_status()
identified = understanding_response.json()

# ── Step 3: Format identified transcript for the LLM ──
speaker_transcript = "\n".join(
    f"{u['speaker']}: {u['text']}" for u in identified["utterances"]
)

# ── Step 4: Generate meeting notes via LLM Gateway ──
llm_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": (
                    "You are a meeting notes assistant. Given the transcript below, produce:\n"
                    "1. A concise summary (3-5 sentences)\n"
                    "2. Key decisions made\n"
                    "3. Action items with owners (use speaker labels)\n\n"
                    f"Transcript:\n{speaker_transcript}"
                ),
            }
        ],
        "max_tokens": 2000,
    },
)
llm_response.raise_for_status()

print("=== Meeting Notes ===\n")
print(llm_response.json()["choices"][0]["message"]["content"])
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "YOUR_API_KEY",
  "Content-Type": "application/json",
};

const audioUrl = "https://assembly.ai/wildfires.mp3";

// Step 1: Transcribe with speaker labels + language detection
let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    audio_url: audioUrl,
    language_detection: true,
    speaker_labels: true,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const { id: transcriptId } = await res.json();

let result;
while (true) {
  res = await fetch(`${baseUrl}/v2/transcript/${transcriptId}`, { headers });
  result = await res.json();
  if (result.status === "completed") break;
  if (result.status === "error")
    throw new Error(`Transcription failed: ${result.error}`);
  await new Promise((r) => setTimeout(r, 3000));
}

// Step 2: Identify speakers by name
const understandingRes = await fetch(
  "https://llm-gateway.assemblyai.com/v1/understanding",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      transcript_id: transcriptId,
      speech_understanding: {
        request: {
          speaker_identification: {
            speaker_type: "name",
            known_values: ["Alice", "Bob"], // Replace with actual participant names
          },
        },
      },
    }),
  }
);
if (!understandingRes.ok) throw new Error(`Error: ${understandingRes.status}`);
const identified = await understandingRes.json();

// Step 3: Format identified transcript for the LLM
const speakerTranscript = identified.utterances
  .map((u) => `${u.speaker}: ${u.text}`)
  .join("\n");

// Step 4: Generate meeting notes via LLM Gateway
res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers,
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content:
          "You are a meeting notes assistant. Given the transcript below, produce:\n" +
          "1. A concise summary (3-5 sentences)\n" +
          "2. Key decisions made\n" +
          "3. Action items with owners (use speaker labels)\n\n" +
          `Transcript:\n${speakerTranscript}`,
      },
    ],
    max_tokens: 2000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const llmResult = await res.json();

console.log("=== Meeting Notes ===\n");
console.log(llmResult.choices[0].message.content);
```

```text
=== Meeting Notes ===

## Summary
The discussion covered the impact of Canadian wildfire smoke on US air quality.
Experts explained how particulate matter affects respiratory and cardiovascular
health. The group reviewed current air quality index readings and discussed
protective measures for affected communities.

## Key decisions
- Monitor AQI levels daily until smoke clears
- Issue public health advisories for sensitive groups

## Action items
- Alice: Compile daily AQI data for the affected regions
- Bob: Draft public advisory messaging for distribution
- Alice: Coordinate with local health departments on response protocols
```

Speaker Identification maps generic labels like "Speaker A" to real names. You can pass a list of `known_values` to guide identification, or omit it to let the model infer names from the conversation. Learn more in the [Speaker Identification guide](/speech-understanding/speaker-identification).

---

See the [End-to-end examples overview](/getting-started/end-to-end-examples) for all available pipelines.

---

<a id="sales-call-intelligence-docs-getting-started-end-to-end-examples-sales-call-intelligence-mdx"></a>

# Sales call intelligence

URL: https://www.assemblyai.com/docs/getting-started/end-to-end-examples/sales-call-intelligence
Source: docs/getting-started/end-to-end-examples/sales-call-intelligence.mdx
Navigation: Overview > Getting started > End-to-end examples
Description: Transcribe a sales call with speaker labels and sentiment analysis, identify speakers by role, then generate a coaching scorecard with LLM Gateway.

Transcribe a sales call with speaker labels and sentiment analysis, identify speakers by role, then use LLM Gateway to generate a coaching scorecard with talk/listen ratio and sentiment insights.

**Products used:** [Pre-recorded STT](/pre-recorded-audio/getting-started/transcribe-an-audio-file) + [speaker diarization](/pre-recorded-audio/label-speakers) + [Speaker Identification](/speech-understanding/speaker-identification) + [sentiment analysis](/speech-understanding/sentiment-analysis) + [LLM Gateway](/llm-gateway/quickstart)

**Model selection:** Uses `universal-3-5-pro` for the highest English accuracy. For multilingual sales teams, add `universal-2` as a fallback.

```python expandable
import requests
import time
from collections import Counter

# ── Config ────────────────────────────────────────────────────
base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

audio_url = "https://assembly.ai/wildfires.mp3"

# ── Step 1: Transcribe with speaker labels + sentiment analysis ──
data = {
    "audio_url": audio_url,
    "speaker_labels": True,
    "sentiment_analysis": True,
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
response.raise_for_status()
transcript_id = response.json()["id"]

while True:
    result = requests.get(f"{base_url}/v2/transcript/{transcript_id}", headers=headers).json()
    if result["status"] == "completed":
        break
    elif result["status"] == "error":
        raise RuntimeError(f"Transcription failed: {result['error']}")
    time.sleep(3)

# ── Step 2: Identify speakers by role ──
understanding_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers=headers,
    json={
        "transcript_id": transcript_id,
        "speech_understanding": {
            "request": {
                "speaker_identification": {
                    "speaker_type": "role",
                    "known_values": ["Sales Rep", "Customer"],
                }
            }
        },
    },
)
understanding_response.raise_for_status()
identified = understanding_response.json()

# ── Step 3: Calculate talk/listen ratio per speaker ──
speaker_durations = Counter()
for utterance in identified["utterances"]:
    duration_ms = utterance["end"] - utterance["start"]
    speaker_durations[utterance["speaker"]] += duration_ms

total_ms = sum(speaker_durations.values())
talk_ratios = {
    speaker: round(dur / total_ms * 100, 1)
    for speaker, dur in speaker_durations.items()
}

# ── Step 4: Summarize sentiment shifts ──
sentiment_by_speaker = {}
for s in result["sentiment_analysis_results"]:
    speaker = s.get("speaker", "Unknown")
    sentiment_by_speaker.setdefault(speaker, []).append(s["sentiment"])

sentiment_summary = ""
for speaker, sentiments in sentiment_by_speaker.items():
    counts = Counter(sentiments)
    sentiment_summary += (
        f"{speaker}: "
        f"{counts.get('POSITIVE', 0)} positive, "
        f"{counts.get('NEUTRAL', 0)} neutral, "
        f"{counts.get('NEGATIVE', 0)} negative\n"
    )

# ── Step 5: Format transcript and generate coaching scorecard ──
speaker_transcript = "\n".join(
    f"{u['speaker']}: {u['text']}" for u in identified["utterances"]
)

llm_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": (
                    "You are a sales coaching assistant. Analyze this sales call and produce a scorecard.\n\n"
                    f"Talk/listen ratios: {talk_ratios}\n\n"
                    f"Sentiment breakdown:\n{sentiment_summary}\n"
                    f"Transcript:\n{speaker_transcript}\n\n"
                    "Produce:\n"
                    "1. Call summary (2-3 sentences)\n"
                    "2. Talk/listen ratio analysis (ideal is 40/60 for the rep)\n"
                    "3. Customer sentiment shifts and what caused them\n"
                    "4. Top 3 coaching suggestions for the sales rep"
                ),
            }
        ],
        "max_tokens": 2000,
    },
)
llm_response.raise_for_status()

print("=== Sales Call Scorecard ===\n")
print(llm_response.json()["choices"][0]["message"]["content"])
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "YOUR_API_KEY",
  "Content-Type": "application/json",
};

const audioUrl = "https://assembly.ai/wildfires.mp3";

// Step 1: Transcribe with speaker labels + sentiment analysis
let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    audio_url: audioUrl,
    speaker_labels: true,
    sentiment_analysis: true,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const { id: transcriptId } = await res.json();

let result;
while (true) {
  res = await fetch(`${baseUrl}/v2/transcript/${transcriptId}`, { headers });
  result = await res.json();
  if (result.status === "completed") break;
  if (result.status === "error")
    throw new Error(`Transcription failed: ${result.error}`);
  await new Promise((r) => setTimeout(r, 3000));
}

// Step 2: Identify speakers by role
const understandingRes = await fetch(
  "https://llm-gateway.assemblyai.com/v1/understanding",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      transcript_id: transcriptId,
      speech_understanding: {
        request: {
          speaker_identification: {
            speaker_type: "role",
            known_values: ["Sales Rep", "Customer"],
          },
        },
      },
    }),
  }
);
if (!understandingRes.ok) throw new Error(`Error: ${understandingRes.status}`);
const identified = await understandingRes.json();

// Step 3: Calculate talk/listen ratio per speaker
const speakerDurations = {};
for (const u of identified.utterances) {
  speakerDurations[u.speaker] =
    (speakerDurations[u.speaker] || 0) + (u.end - u.start);
}
const totalMs = Object.values(speakerDurations).reduce((a, b) => a + b, 0);
const talkRatios = {};
for (const [speaker, dur] of Object.entries(speakerDurations)) {
  talkRatios[speaker] = ((dur / totalMs) * 100).toFixed(1) + "%";
}

// Step 4: Summarize sentiment shifts
const sentimentBySpeaker = {};
for (const s of result.sentiment_analysis_results) {
  const speaker = s.speaker || "Unknown";
  if (!sentimentBySpeaker[speaker]) sentimentBySpeaker[speaker] = [];
  sentimentBySpeaker[speaker].push(s.sentiment);
}

let sentimentSummary = "";
for (const [speaker, sentiments] of Object.entries(sentimentBySpeaker)) {
  const counts = { POSITIVE: 0, NEUTRAL: 0, NEGATIVE: 0 };
  sentiments.forEach((s) => counts[s]++);
  sentimentSummary += `${speaker}: ${counts.POSITIVE} positive, ${counts.NEUTRAL} neutral, ${counts.NEGATIVE} negative\n`;
}

// Step 5: Format transcript and generate coaching scorecard
const speakerTranscript = identified.utterances
  .map((u) => `${u.speaker}: ${u.text}`)
  .join("\n");

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers,
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content:
          "You are a sales coaching assistant. Analyze this sales call and produce a scorecard.\n\n" +
          `Talk/listen ratios: ${JSON.stringify(talkRatios)}\n\n` +
          `Sentiment breakdown:\n${sentimentSummary}\n` +
          `Transcript:\n${speakerTranscript}\n\n` +
          "Produce:\n" +
          "1. Call summary (2-3 sentences)\n" +
          "2. Talk/listen ratio analysis (ideal is 40/60 for the rep)\n" +
          "3. Customer sentiment shifts and what caused them\n" +
          "4. Top 3 coaching suggestions for the sales rep",
      },
    ],
    max_tokens: 2000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const llmResult = await res.json();

console.log("=== Sales Call Scorecard ===\n");
console.log(llmResult.choices[0].message.content);
```

```text expandable
=== Sales Call Scorecard ===

## Call summary
This call discussed the environmental and health impacts of wildfire smoke on
US communities. The speakers covered air quality data, health risks, and
recommended precautions for the public.

## Talk/listen ratio
Sales Rep: 65.3% | Customer: 34.7%
Analysis: The ratio is inverted from the ideal 40/60 split. The rep dominated
the conversation — focus on asking more open-ended questions.

## Customer sentiment shifts
- Started neutral during introductions
- Shifted negative when discussing health risks and poor air quality readings
- Returned to neutral during the action-planning portion

## Coaching suggestions
1. Ask more discovery questions early to understand the customer's specific concerns
2. When the customer expresses concern, acknowledge before pivoting to solutions
3. Summarize key points at the end and confirm next steps with clear ownership
```

---

See the [End-to-end examples overview](/getting-started/end-to-end-examples) for all available pipelines.

---

<a id="medical-scribe-docs-getting-started-end-to-end-examples-medical-scribe-mdx"></a>

# Medical scribe

URL: https://www.assemblyai.com/docs/getting-started/end-to-end-examples/medical-scribe
Source: docs/getting-started/end-to-end-examples/medical-scribe.mdx
Navigation: Overview > Getting started > End-to-end examples
Description: Transcribe a clinical encounter using Medical Mode with speaker labels and entity detection, then generate a structured SOAP note with LLM Gateway.

Transcribe a clinical encounter using Medical Mode with speaker labels and entity detection, identify speakers by role, then use LLM Gateway to generate a structured SOAP note.

**Products used:** [Pre-recorded STT](/pre-recorded-audio/getting-started/transcribe-an-audio-file) + [Medical Mode](/pre-recorded-audio/medical-mode) + [speaker diarization](/pre-recorded-audio/label-speakers) + [Speaker Identification](/speech-understanding/speaker-identification) + [entity detection](/speech-understanding/entity-detection) + [LLM Gateway](/llm-gateway/quickstart)

**Model selection:** Uses `universal-3-5-pro` with `"domain": "medical-v1"` for purpose-built accuracy on medical terminology, drug names, and clinical language.

```python expandable
import requests
import time

# ── Config ────────────────────────────────────────────────────
base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

audio_url = "https://assembly.ai/wildfires.mp3"  # Replace with your clinical audio

# ── Step 1: Transcribe with Medical Mode + speaker labels + entity detection ──
data = {
    "audio_url": audio_url,
    "domain": "medical-v1",
    "speaker_labels": True,
    "entity_detection": True,
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
response.raise_for_status()
transcript_id = response.json()["id"]

while True:
    result = requests.get(f"{base_url}/v2/transcript/{transcript_id}", headers=headers).json()
    if result["status"] == "completed":
        break
    elif result["status"] == "error":
        raise RuntimeError(f"Transcription failed: {result['error']}")
    time.sleep(3)

# ── Step 2: Identify speakers by role ──
understanding_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers=headers,
    json={
        "transcript_id": transcript_id,
        "speech_understanding": {
            "request": {
                "speaker_identification": {
                    "speaker_type": "role",
                    "known_values": ["Provider", "Patient"],
                }
            }
        },
    },
)
understanding_response.raise_for_status()
identified = understanding_response.json()

# ── Step 3: Extract detected entities ──
entities = result.get("entities", [])
entity_summary = "\n".join(
    f"- {e['entity_type']}: {e['text']}" for e in entities
)

# ── Step 4: Format identified transcript ──
speaker_transcript = "\n".join(
    f"{u['speaker']}: {u['text']}" for u in identified["utterances"]
)

# ── Step 5: Generate SOAP note via LLM Gateway ──
llm_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": (
                    "You are a medical scribe. Given the clinical encounter transcript and "
                    "detected entities below, generate a structured SOAP note.\n\n"
                    "Format the note with these sections:\n"
                    "- **Subjective**: Patient's reported symptoms and history\n"
                    "- **Objective**: Clinical observations and measurements\n"
                    "- **Assessment**: Diagnosis or clinical impression\n"
                    "- **Plan**: Treatment plan, prescriptions, and follow-up\n\n"
                    f"Detected entities:\n{entity_summary}\n\n"
                    f"Transcript:\n{speaker_transcript}"
                ),
            }
        ],
        "max_tokens": 2000,
    },
)
llm_response.raise_for_status()

print("=== SOAP Note ===\n")
print(llm_response.json()["choices"][0]["message"]["content"])
```

```python expandable
import assemblyai as aai
import requests

# ── Config ────────────────────────────────────────────────────
api_key = "YOUR_API_KEY"
aai.settings.api_key = api_key
headers = {"authorization": api_key}

audio_url = "https://assembly.ai/wildfires.mp3"  # Replace with your clinical audio

# ── Step 1: Transcribe with Medical Mode + speaker labels + entity detection ──
config = aai.TranscriptionConfig(
    domain="medical-v1",
    speaker_labels=True,
    entity_detection=True,
)

transcript = aai.Transcriber().transcribe(audio_url, config)

if transcript.error:
    raise RuntimeError(f"Transcription failed: {transcript.error}")

# ── Step 2: Identify speakers by role ──
understanding_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers=headers,
    json={
        "transcript_id": transcript.id,
        "speech_understanding": {
            "request": {
                "speaker_identification": {
                    "speaker_type": "role",
                    "known_values": ["Provider", "Patient"],
                }
            }
        },
    },
)
understanding_response.raise_for_status()
identified = understanding_response.json()

# ── Step 3: Extract detected entities ──
entities = transcript.entities or []
entity_summary = "\n".join(
    f"- {e.entity_type}: {e.text}" for e in entities
)

# ── Step 4: Format identified transcript ──
speaker_transcript = "\n".join(
    f"{u['speaker']}: {u['text']}" for u in identified["utterances"]
)

# ── Step 5: Generate SOAP note via LLM Gateway ──
llm_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": (
                    "You are a medical scribe. Given the clinical encounter transcript and "
                    "detected entities below, generate a structured SOAP note.\n\n"
                    "Format the note with these sections:\n"
                    "- **Subjective**: Patient's reported symptoms and history\n"
                    "- **Objective**: Clinical observations and measurements\n"
                    "- **Assessment**: Diagnosis or clinical impression\n"
                    "- **Plan**: Treatment plan, prescriptions, and follow-up\n\n"
                    f"Detected entities:\n{entity_summary}\n\n"
                    f"Transcript:\n{speaker_transcript}"
                ),
            }
        ],
        "max_tokens": 2000,
    },
)
llm_response.raise_for_status()

print("=== SOAP Note ===\n")
print(llm_response.json()["choices"][0]["message"]["content"])
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "YOUR_API_KEY",
  "Content-Type": "application/json",
};

const audioUrl = "https://assembly.ai/wildfires.mp3"; // Replace with your clinical audio

// Step 1: Transcribe with Medical Mode + speaker labels + entity detection
let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    audio_url: audioUrl,
    domain: "medical-v1",
    speaker_labels: true,
    entity_detection: true,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const { id: transcriptId } = await res.json();

let result;
while (true) {
  res = await fetch(`${baseUrl}/v2/transcript/${transcriptId}`, { headers });
  result = await res.json();
  if (result.status === "completed") break;
  if (result.status === "error")
    throw new Error(`Transcription failed: ${result.error}`);
  await new Promise((r) => setTimeout(r, 3000));
}

// Step 2: Identify speakers by role
const understandingRes = await fetch(
  "https://llm-gateway.assemblyai.com/v1/understanding",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      transcript_id: transcriptId,
      speech_understanding: {
        request: {
          speaker_identification: {
            speaker_type: "role",
            known_values: ["Provider", "Patient"],
          },
        },
      },
    }),
  }
);
if (!understandingRes.ok) throw new Error(`Error: ${understandingRes.status}`);
const identified = await understandingRes.json();

// Step 3: Extract detected entities
const entities = result.entities || [];
const entitySummary = entities
  .map((e) => `- ${e.entity_type}: ${e.text}`)
  .join("\n");

// Step 4: Format identified transcript
const speakerTranscript = identified.utterances
  .map((u) => `${u.speaker}: ${u.text}`)
  .join("\n");

// Step 5: Generate SOAP note via LLM Gateway
res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers,
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content:
          "You are a medical scribe. Given the clinical encounter transcript and " +
          "detected entities below, generate a structured SOAP note.\n\n" +
          "Format the note with these sections:\n" +
          "- **Subjective**: Patient's reported symptoms and history\n" +
          "- **Objective**: Clinical observations and measurements\n" +
          "- **Assessment**: Diagnosis or clinical impression\n" +
          "- **Plan**: Treatment plan, prescriptions, and follow-up\n\n" +
          `Detected entities:\n${entitySummary}\n\n` +
          `Transcript:\n${speakerTranscript}`,
      },
    ],
    max_tokens: 2000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const llmResult = await res.json();

console.log("=== SOAP Note ===\n");
console.log(llmResult.choices[0].message.content);
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

// ── Config ────────────────────────────────────────────────────
const apiKey = "YOUR_API_KEY";
const client = new AssemblyAI({ apiKey });
const headers = {
  authorization: apiKey,
  "Content-Type": "application/json",
};

const audioUrl = "https://assembly.ai/wildfires.mp3"; // Replace with your clinical audio

// Step 1: Transcribe with Medical Mode + speaker labels + entity detection
const transcript = await client.transcripts.transcribe({
  audio: audioUrl,
  domain: "medical-v1",
  speaker_labels: true,
  entity_detection: true,
});

if (transcript.error) {
  throw new Error(`Transcription failed: ${transcript.error}`);
}

// Step 2: Identify speakers by role
const understandingRes = await fetch(
  "https://llm-gateway.assemblyai.com/v1/understanding",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      transcript_id: transcript.id,
      speech_understanding: {
        request: {
          speaker_identification: {
            speaker_type: "role",
            known_values: ["Provider", "Patient"],
          },
        },
      },
    }),
  }
);
if (!understandingRes.ok) throw new Error(`Error: ${understandingRes.status}`);
const identified = await understandingRes.json();

// Step 3: Extract detected entities
const entities = transcript.entities || [];
const entitySummary = entities
  .map((e) => `- ${e.entity_type}: ${e.text}`)
  .join("\n");

// Step 4: Format identified transcript
const speakerTranscript = identified.utterances
  .map((u) => `${u.speaker}: ${u.text}`)
  .join("\n");

// Step 5: Generate SOAP note via LLM Gateway
const llmRes = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [
        {
          role: "user",
          content:
            "You are a medical scribe. Given the clinical encounter transcript and " +
            "detected entities below, generate a structured SOAP note.\n\n" +
            "Format the note with these sections:\n" +
            "- **Subjective**: Patient's reported symptoms and history\n" +
            "- **Objective**: Clinical observations and measurements\n" +
            "- **Assessment**: Diagnosis or clinical impression\n" +
            "- **Plan**: Treatment plan, prescriptions, and follow-up\n\n" +
            `Detected entities:\n${entitySummary}\n\n` +
            `Transcript:\n${speakerTranscript}`,
        },
      ],
      max_tokens: 2000,
    }),
  }
);
if (!llmRes.ok) throw new Error(`Error: ${llmRes.status}`);
const llmResult = await llmRes.json();

console.log("=== SOAP Note ===\n");
console.log(llmResult.choices[0].message.content);
```

```text expandable
=== SOAP Note ===

## Subjective
Patient reports exposure to wildfire smoke over the past several days. Describes
worsening cough, shortness of breath, and eye irritation. Symptoms began
approximately 3 days ago coinciding with elevated air quality alerts in the region.

## Objective
- AQI reading: 150 micrograms per cubic meter (10x annual average)
- Particulate matter levels classified as "unhealthy"
- Patient appears alert and oriented

## Assessment
Acute respiratory irritation secondary to wildfire smoke exposure.
Environmental exposure consistent with regional air quality emergency.

## Plan
1. Advise patient to remain indoors with windows closed
2. Recommend N95 mask for any necessary outdoor activity
3. Prescribe albuterol inhaler PRN for acute bronchospasm
4. Follow up in 1 week or sooner if symptoms worsen
5. Refer to pulmonology if symptoms persist beyond 2 weeks
```

For more on building clinical documentation apps, see the [Medical Scribe guides](/medical-scribe-best-practices).

---

See the [End-to-end examples overview](/getting-started/end-to-end-examples) for all available pipelines.

---

<a id="content-repurposing-docs-getting-started-end-to-end-examples-content-repurposing-mdx"></a>

# Content repurposing

URL: https://www.assemblyai.com/docs/getting-started/end-to-end-examples/content-repurposing
Source: docs/getting-started/end-to-end-examples/content-repurposing.mdx
Navigation: Overview > Getting started > End-to-end examples
Description: Transcribe a podcast or webinar, extract key phrases, then generate a blog post draft with LLM Gateway.

Transcribe a podcast or webinar, extract key phrases, then use LLM Gateway to generate a blog post draft with highlights.

**Products used:** [Pre-recorded STT](/pre-recorded-audio/getting-started/transcribe-an-audio-file) + [key phrases](/speech-understanding/key-phrases) + [LLM Gateway](/llm-gateway/quickstart)

**Model selection:** Uses `universal-3-5-pro` with `universal-2` fallback for multilingual content. If your content is English-only, `universal-3-5-pro` alone gives the best results.

```python expandable
import requests
import time

# ── Config ────────────────────────────────────────────────────
base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

audio_url = "https://assembly.ai/wildfires.mp3"

# ── Step 1: Transcribe with key phrases enabled ──
data = {
    "audio_url": audio_url,
    "language_detection": True,
    "auto_highlights": True,
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
response.raise_for_status()
transcript_id = response.json()["id"]

while True:
    result = requests.get(f"{base_url}/v2/transcript/{transcript_id}", headers=headers).json()
    if result["status"] == "completed":
        break
    elif result["status"] == "error":
        raise RuntimeError(f"Transcription failed: {result['error']}")
    time.sleep(3)

# ── Step 2: Extract top key phrases ──
highlights = result.get("auto_highlights_result", {}).get("results", [])
top_phrases = sorted(highlights, key=lambda x: x["rank"], reverse=True)[:10]
phrases_list = ", ".join(p["text"] for p in top_phrases)

# ── Step 3: Generate blog post via LLM Gateway ──
llm_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": (
                    "You are a content writer. Transform this transcript into an engaging "
                    "blog post.\n\n"
                    "Requirements:\n"
                    "- Write a compelling title and subtitle\n"
                    "- Break the content into 3-5 sections with headers\n"
                    "- Weave in the key phrases naturally\n"
                    "- Add a TL;DR at the top\n"
                    "- End with a call-to-action\n\n"
                    f"Key phrases: {phrases_list}\n\n"
                    "Transcript:\n{{ transcript }}"
                ),
            }
        ],
        "transcript_id": transcript_id,
        "max_tokens": 3000,
    },
)
llm_response.raise_for_status()

print("=== Blog Post Draft ===\n")
print(llm_response.json()["choices"][0]["message"]["content"])
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "YOUR_API_KEY",
  "Content-Type": "application/json",
};

const audioUrl = "https://assembly.ai/wildfires.mp3";

// Step 1: Transcribe with key phrases enabled
let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    audio_url: audioUrl,
    language_detection: true,
    auto_highlights: true,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const { id: transcriptId } = await res.json();

let result;
while (true) {
  res = await fetch(`${baseUrl}/v2/transcript/${transcriptId}`, { headers });
  result = await res.json();
  if (result.status === "completed") break;
  if (result.status === "error")
    throw new Error(`Transcription failed: ${result.error}`);
  await new Promise((r) => setTimeout(r, 3000));
}

// Step 2: Extract top key phrases
const highlights = result.auto_highlights_result?.results || [];
const topPhrases = highlights
  .sort((a, b) => b.rank - a.rank)
  .slice(0, 10);
const phrasesList = topPhrases.map((p) => p.text).join(", ");

// Step 3: Generate blog post via LLM Gateway
res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers,
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content:
          "You are a content writer. Transform this transcript into an engaging " +
          "blog post.\n\n" +
          "Requirements:\n" +
          "- Write a compelling title and subtitle\n" +
          "- Break the content into 3-5 sections with headers\n" +
          "- Weave in the key phrases naturally\n" +
          "- Add a TL;DR at the top\n" +
          "- End with a call-to-action\n\n" +
          `Key phrases: ${phrasesList}\n\n` +
          "Transcript:\n{{ transcript }}",
      },
    ],
    transcript_id: transcriptId,
    max_tokens: 3000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const llmResult = await res.json();

console.log("=== Blog Post Draft ===\n");
console.log(llmResult.choices[0].message.content);
```

```text expandable
=== Blog Post Draft ===

# When the Sky Turns Orange: Understanding Wildfire Smoke and Air Quality

**How Canadian wildfires are reshaping air quality across the United States**

**TL;DR:** Wildfire smoke from Canada is triggering widespread air quality alerts
in the US, with particulate matter levels reaching 10x normal in some cities.
Here's what you need to know about the health risks and how to protect yourself.

## The smoke crosses borders

Hundreds of wildfires burning across Canada have sent massive plumes of smoke
southward into the United States, creating hazy skies and triggering air quality
alerts from the Midwest to the Eastern Seaboard...

## Understanding particulate matter

The real danger lies in fine particulate matter — microscopic particles that
can penetrate deep into your lungs and even enter your bloodstream...

## Protecting your health

Health experts recommend staying indoors, using air purifiers, and wearing
N95 masks when outdoor exposure is unavoidable...

## Looking ahead

As climate change intensifies wildfire seasons, these cross-border smoke events
are likely to become more frequent...

---
*Want to transcribe your own podcast or webinar? Get started with AssemblyAI's
API at [assemblyai.com](https://assemblyai.com).*
```

---

See the [End-to-end examples overview](/getting-started/end-to-end-examples) for all available pipelines.

---

<a id="real-time-meeting-assistant-docs-getting-started-end-to-end-examples-real-time-meeting-assistant-mdx"></a>

# Real-time meeting assistant

URL: https://www.assemblyai.com/docs/getting-started/end-to-end-examples/real-time-meeting-assistant
Source: docs/getting-started/end-to-end-examples/real-time-meeting-assistant.mdx
Navigation: Overview > Getting started > End-to-end examples
Description: Stream audio from your microphone with speaker diarization and LLM Gateway for live transcription and automatic summaries.

Stream audio from your microphone with speaker diarization and LLM Gateway to get live transcription and automatic summaries after each speaker turn.

**Products used:** [Real-time STT](/streaming/getting-started/transcribe-streaming-audio) + [Universal-3.5 Pro](/streaming/getting-started/transcribe-streaming-audio) + [LLM Gateway](/guides/real_time_llm_gateway)

**Model selection:** Uses `universal-3-5-pro` (Universal-3.5 Pro Streaming) for the lowest latency (~300ms) with the highest streaming accuracy.

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode

# ── Config ────────────────────────────────────────────────────
YOUR_API_KEY = "YOUR_API_KEY"

PROMPT = (
    "Summarize this speaker turn in one sentence, then list any "
    "action items mentioned.\n\nTranscript: {{turn}}"
)

LLM_GATEWAY_CONFIG = {
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": PROMPT}],
    "max_tokens": 500,
}

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "min_turn_silence": 560,   # Wait longer for natural meeting pauses
    "max_turn_silence": 2000,
    "llm_gateway": json.dumps(LLM_GATEWAY_CONFIG),
}

API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"

# Audio settings
FRAMES_PER_BUFFER = 800
SAMPLE_RATE = 16000
stop_event = threading.Event()

def on_open(ws):
    print("Connected — speak into your microphone. Press Ctrl+C to stop.\n")

    def stream_audio():
        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True, frames_per_buffer=FRAMES_PER_BUFFER,
            channels=1, format=pyaudio.paInt16, rate=SAMPLE_RATE,
        )
        while not stop_event.is_set():
            try:
                data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(data, websocket.ABNF.OPCODE_BINARY)
            except Exception:
                break
        stream.stop_stream()
        stream.close()
        audio.terminate()

    threading.Thread(target=stream_audio, daemon=True).start()

def on_message(ws, message):
    data = json.loads(message)
    msg_type = data.get("type")

    if msg_type == "Turn":
        transcript = data.get("transcript", "")
        if data.get("end_of_turn") and transcript:
            print(f"[Turn] {transcript}\n")
        elif transcript:
            print(f"\r  ... {transcript[-80:]}", end="", flush=True)

    elif msg_type == "LLMGatewayResponse":
        content = data.get("data", {}).get("choices", [{}])[0].get("message", {}).get("content", "")
        print(f"[Assistant] {content}\n")

    elif msg_type == "Termination":
        print(f"\nSession ended — {data.get('audio_duration_seconds', 0)}s of audio processed.")

def on_error(ws, error):
    print(f"Error: {error}")
    stop_event.set()

def on_close(ws, code, msg):
    stop_event.set()

ws_app = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": YOUR_API_KEY},
    on_open=on_open, on_message=on_message,
    on_error=on_error, on_close=on_close,
)

ws_thread = threading.Thread(target=ws_app.run_forever, daemon=True)
ws_thread.start()

try:
    while ws_thread.is_alive():
        time.sleep(0.1)
except KeyboardInterrupt:
    print("\nStopping...")
    stop_event.set()
    if ws_app.sock and ws_app.sock.connected:
        ws_app.send(json.dumps({"type": "Terminate"}))
        time.sleep(2)
    ws_app.close()
```

```javascript expandable
// npm install ws mic
const WebSocket = require("ws");
const mic = require("mic");
const querystring = require("querystring");

// ── Config ────────────────────────────────────────────────────
const YOUR_API_KEY = "YOUR_API_KEY";

const PROMPT =
  "Summarize this speaker turn in one sentence, then list any " +
  "action items mentioned.\n\nTranscript: {{turn}}";

const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 500,
};

const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  format_turns: true,
  min_turn_silence: 560,
  max_turn_silence: 2000,
  llm_gateway: JSON.stringify(LLM_GATEWAY_CONFIG),
};

const API_ENDPOINT = `wss://streaming.assemblyai.com/v3/ws?${querystring.stringify(CONNECTION_PARAMS)}`;

// Connect and stream
const ws = new WebSocket(API_ENDPOINT, {
  headers: { Authorization: YOUR_API_KEY },
});

let micInstance;

ws.on("open", () => {
  console.log("Connected — speak into your microphone. Press Ctrl+C to stop.\n");

  micInstance = mic({ rate: "16000", channels: "1", debug: false });
  const micStream = micInstance.getAudioStream();

  micStream.on("data", (data) => {
    if (ws.readyState === WebSocket.OPEN) ws.send(data);
  });

  micInstance.start();
});

ws.on("message", (message) => {
  const data = JSON.parse(message);

  if (data.type === "Turn") {
    const transcript = data.transcript || "";
    if (data.end_of_turn && transcript) {
      console.log(`[Turn] ${transcript}\n`);
    } else if (transcript) {
      process.stdout.write(`\r  ... ${transcript.slice(-80)}`);
    }
  } else if (data.type === "LLMGatewayResponse") {
    const content =
      data.data?.choices?.[0]?.message?.content || "";
    console.log(`[Assistant] ${content}\n`);
  } else if (data.type === "Termination") {
    console.log(
      `\nSession ended — ${data.audio_duration_seconds || 0}s of audio processed.`
    );
  }
});

ws.on("error", (err) => console.error(`Error: ${err}`));

ws.on("close", () => {
  if (micInstance) micInstance.stop();
  console.log("Disconnected.");
});

process.on("SIGINT", () => {
  console.log("\nStopping...");
  if (ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: "Terminate" }));
  }
  setTimeout(() => {
    if (micInstance) micInstance.stop();
    ws.close();
    process.exit(0);
  }, 2000);
});
```

```text
Connected — speak into your microphone. Press Ctrl+C to stop.

[Turn] So the main thing we need to decide today is whether we're going
with vendor A or vendor B for the new analytics platform.

[Assistant] The speaker is initiating a decision discussion about choosing
between two analytics platform vendors.
Action items: None yet — decision pending.

[Turn] I think vendor A has better pricing but vendor B has the integrations
we need. Can someone pull the comparison spreadsheet by Friday?

[Assistant] The speaker compared vendor pricing vs. integrations and requested
a comparison document.
Action items:
- Pull the vendor comparison spreadsheet by Friday
```

---

See the [End-to-end examples overview](/getting-started/end-to-end-examples) for all available pipelines.

---

<a id="real-time-live-captioner-docs-getting-started-end-to-end-examples-real-time-live-captioner-mdx"></a>

# Real-time live captioner

URL: https://www.assemblyai.com/docs/getting-started/end-to-end-examples/real-time-live-captioner
Source: docs/getting-started/end-to-end-examples/real-time-live-captioner.mdx
Navigation: Overview > Getting started > End-to-end examples
Description: Stream audio from your microphone with keyterms prompting for domain-specific accuracy, ideal for live events and accessibility.

Stream audio from your microphone with keyterms prompting for domain-specific accuracy, ideal for live events, accessibility, and broadcast captioning.

**Products used:** [Real-time STT](/streaming/getting-started/transcribe-streaming-audio) + [Universal-3.5 Pro](/streaming/getting-started/transcribe-streaming-audio) + [keyterms prompting](/streaming/prompting-and-keyterms)

**Model selection:** Uses `universal-3-5-pro` for sub-300ms latency with `format_turns` enabled for clean, readable captions.

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode

# ── Config ────────────────────────────────────────────────────
YOUR_API_KEY = "YOUR_API_KEY"

# Add domain-specific terms to boost recognition accuracy
KEYTERMS = ["AssemblyAI", "Universal-3.5 Pro", "LLM Gateway", "speech-to-text"]

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "keyterms_prompt": KEYTERMS,
}

API_ENDPOINT = (
    f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS, doseq=True)}"
)

# Audio settings
FRAMES_PER_BUFFER = 800
SAMPLE_RATE = 16000
stop_event = threading.Event()
caption_count = 0

def on_open(ws):
    print(f"Live captioning started — keyterms: {', '.join(KEYTERMS)}")
    print("Speak into your microphone. Press Ctrl+C to stop.\n")
    print("-" * 60)

    def stream_audio():
        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True, frames_per_buffer=FRAMES_PER_BUFFER,
            channels=1, format=pyaudio.paInt16, rate=SAMPLE_RATE,
        )
        while not stop_event.is_set():
            try:
                data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(data, websocket.ABNF.OPCODE_BINARY)
            except Exception:
                break
        stream.stop_stream()
        stream.close()
        audio.terminate()

    threading.Thread(target=stream_audio, daemon=True).start()

def on_message(ws, message):
    global caption_count
    data = json.loads(message)

    if data.get("type") == "Turn":
        transcript = data.get("transcript", "")
        if data.get("end_of_turn") and transcript:
            caption_count += 1
            print(f"\r[{caption_count:03d}] {transcript}")
        elif transcript:
            # Show partial (live) caption
            print(f"\r  >> {transcript[-70:]}", end="", flush=True)

    elif data.get("type") == "Termination":
        duration = data.get("audio_duration_seconds", 0)
        print(f"\n{'=' * 60}")
        print(f"Session ended — {caption_count} captions, {duration}s of audio")

def on_error(ws, error):
    print(f"\nError: {error}")
    stop_event.set()

def on_close(ws, code, msg):
    stop_event.set()

ws_app = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": YOUR_API_KEY},
    on_open=on_open, on_message=on_message,
    on_error=on_error, on_close=on_close,
)

ws_thread = threading.Thread(target=ws_app.run_forever, daemon=True)
ws_thread.start()

try:
    while ws_thread.is_alive():
        time.sleep(0.1)
except KeyboardInterrupt:
    print("\n\nStopping...")
    stop_event.set()
    if ws_app.sock and ws_app.sock.connected:
        ws_app.send(json.dumps({"type": "Terminate"}))
        time.sleep(2)
    ws_app.close()
```

```javascript expandable
// npm install ws mic
const WebSocket = require("ws");
const mic = require("mic");
const querystring = require("querystring");

// ── Config ────────────────────────────────────────────────────
const YOUR_API_KEY = "YOUR_API_KEY";

// Add domain-specific terms to boost recognition accuracy
const KEYTERMS = ["AssemblyAI", "Universal-3.5 Pro", "LLM Gateway", "speech-to-text"];

const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  format_turns: true,
  keyterms_prompt: KEYTERMS,
};

const API_ENDPOINT = `wss://streaming.assemblyai.com/v3/ws?${querystring.stringify(
  CONNECTION_PARAMS
)}`;

let micInstance;
let captionCount = 0;

const ws = new WebSocket(API_ENDPOINT, {
  headers: { Authorization: YOUR_API_KEY },
});

ws.on("open", () => {
  console.log(`Live captioning started — keyterms: ${KEYTERMS.join(", ")}`);
  console.log("Speak into your microphone. Press Ctrl+C to stop.\n");
  console.log("-".repeat(60));

  micInstance = mic({ rate: "16000", channels: "1", debug: false });
  const micStream = micInstance.getAudioStream();

  micStream.on("data", (data) => {
    if (ws.readyState === WebSocket.OPEN) ws.send(data);
  });

  micInstance.start();
});

ws.on("message", (message) => {
  const data = JSON.parse(message);

  if (data.type === "Turn") {
    const transcript = data.transcript || "";
    if (data.end_of_turn && transcript) {
      captionCount++;
      process.stdout.write(
        `\r${String(captionCount).padStart(3, "0")}  ${transcript}\n`
      );
    } else if (transcript) {
      process.stdout.write(`\r  >> ${transcript.slice(-70)}`);
    }
  } else if (data.type === "Termination") {
    console.log(`\n${"=".repeat(60)}`);
    console.log(
      `Session ended — ${captionCount} captions, ${data.audio_duration_seconds || 0}s of audio`
    );
  }
});

ws.on("error", (err) => console.error(`Error: ${err}`));
ws.on("close", () => {
  if (micInstance) micInstance.stop();
});

process.on("SIGINT", () => {
  console.log("\n\nStopping...");
  if (ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: "Terminate" }));
  }
  setTimeout(() => {
    if (micInstance) micInstance.stop();
    ws.close();
    process.exit(0);
  }, 2000);
});
```

```text
Live captioning started — keyterms: AssemblyAI, Universal-3.5 Pro, LLM Gateway, speech-to-text
Speak into your microphone. Press Ctrl+C to stop.

------------------------------------------------------------
[001] Welcome everyone to today's demo of AssemblyAI's speech-to-text platform.
[002] We'll be showing you how Universal-3.5 Pro handles real-time transcription.
[003] The LLM Gateway integration lets you add AI analysis on top of your
      transcripts without switching providers.

============================================================
Session ended — 3 captions, 24s of audio
```

---

See the [End-to-end examples overview](/getting-started/end-to-end-examples) for all available pipelines.

---

<a id="best-practices-for-building-meeting-notetakers-docs-meeting-notetaker-best-practices-mdx"></a>

# Best Practices for building Meeting Notetakers

URL: https://www.assemblyai.com/docs/meeting-notetaker-best-practices
Source: docs/meeting-notetaker-best-practices.mdx
Navigation: Overview > Use cases & integrations > Use case guides
Description: Complete guide for building meeting notetakers with AssemblyAI

## Introduction

Building a robust meeting notetaker requires careful consideration of accuracy, latency, speaker identification, and real-time capabilities. This guide addresses common questions and provides practical solutions for both post-call and live meeting transcription scenarios.

## Why AssemblyAI for Meeting Notetakers?

AssemblyAI stands out as the premier choice for meeting notetakers with several key advantages:

### Industry-Leading Accuracy with Pre-recorded Audio

- **93.3%+ transcription accuracy** ensures reliable meeting documentation
- **2.9% speaker diarization error rate** for precise "who said what" attribution
- **Speech Understanding** integration for intelligent post-processing and insights
- **Keyterms prompt** allows providing meeting context to improve accuracy of transcription

### Streaming with Universal-3.5 Pro

As meeting notetakers evolve toward real-time capabilities, AssemblyAI's Universal-3.5 Pro Streaming model (`universal-3-5-pro`) offers significant benefits:

- **Speaker diarization** available for both pre-recorded and streaming transcription
- **Ultra-low latency (~300ms)** enables live transcription without delays
- **Format turns** feature provides structured, readable output in real-time
- **Keyterms prompt** allows providing meeting context to improve accuracy of transcription

### End-to-End Voice AI Platform

Unlike fragmented solutions, AssemblyAI provides a unified API for:

- Transcription with speaker diarization
- Automatic language detection and code switching
- Boosting accuracy via meeting context with keyterms prompt
- Speech Understanding tasks like speaker identification, translation, and transcript styling
- Post-processing workflows with custom prompting - from summarization to completely custom workflows
- Real-time and batch processing of pre-recorded audio in a single platform

## When Should I Use Pre-recorded vs Streaming for Meeting Notetakers?

Understanding when to use pre-recorded versus streaming speech-to-text is critical for building the right meeting notetaker.

### Pre-recorded Speech-to-text

**Post-call analysis** - Meeting already happened, you have the full recording

- **Highest accuracy needed** - Pre-recorded models have higher accuracy (93.3%+)
- **Speaker diarization is critical** - Pre-recorded has 2.9% speaker error rate
- **Broad language support** - Need any of 99+ languages
- **Advanced features required** - Summarization, sentiment analysis, entity detection, PII redaction, speaker identification
- **Batch processing** - Processing multiple recordings at once
- **Quality over speed** - Can wait seconds/minutes for perfect results

**Best for:** Zoom/Teams/Meet recording uploads, compliance, documentation, post-call summaries, searchable archives

### Streaming Speech-to-text

**Live meetings** - Transcribing as the meeting happens

You should use streaming when you need to display a live transcript of text to users as they are speaking. With Universal-3.5 Pro Streaming, accuracy is closer to pre-recorded, but pre-recorded will always be the most accurate option.

- **Real-time captions** - Displaying subtitles/captions to participants during calls
- **Immediate feedback** - Need transcription within ~300ms
- **Interactive features** - Live note-taking, real-time keyword detection, action item alerts
- **No recording available** - Processing live audio only

**Best for:** Live captions, real-time note-taking apps, accessibility features, live keyword alerts

**Streaming is billed per session**

Streaming is billed on the total duration that your WebSocket connection stays open, not on the amount of audio you send. For long-running meetings, make sure to terminate sessions when the meeting ends to avoid being billed for idle time. See [Billing and pricing](/billing-and-pricing) for details.

### Hybrid Approach (Recommended)

Many successful meeting notetakers use **both** pre-recorded and streaming speech-to-text:

1. **Streaming during the call** - Provide live captions and real-time notes to participants
2. **Pre-recorded after the call** - Generate high-quality transcript with speaker labels, summary, and insights

This gives users immediate value during meetings while providing comprehensive documentation afterward.

**Example workflow:**

- User joins meeting → Start streaming for live captions
- Meeting ends → Upload recording to pre-recorded API for final transcript with speaker names
- Generate meeting summary, action items, and searchable archive from pre-recorded transcript

## What Languages and Features for a Meeting Notetaker?

### Pre-Recorded Meetings

For post-call analysis, AssemblyAI supports:

**Languages**:

- 99 languages supported
- Automatic Language Detection to route to the most spoken language
- Code Switching to preserve changes in speech between languages

**Core Features**:

- Speaker diarization (1-10 speakers by default, expandable to any min/max)
- Multichannel audio support (each channel = one speaker)
- Automatic formatting, punctuation, and capitalization
- Keyterms prompting for boosting domain-specific terms

**Speech Understanding Models**:

- Summarization for meeting recaps
- Sentiment analysis for meeting tone assessment
- Entity detection for extracting key information
- Speaker identification to map generic labels to actual names/roles
- Translation between 86 languages

### Real-Time Streaming

For live meeting transcription:

**Languages**:

- English-only model (default)
- Multilingual model supporting English, Spanish, French, German, Portuguese, and Italian

### Streaming (Universal-3.5 Pro Streaming)

- Speaker diarization for identifying who is speaking
- Partial and final transcripts for responsive UI
- Format turns for structured, readable output
- Keyterms prompt for contextual accuracy

See the [Universal-3.5 Pro Streaming documentation](/streaming/getting-started/transcribe-streaming-audio) for full details.

## How Can I Get Started Building a Post-Call Meeting Notetaker?

Here's a complete example implementing pre-recorded transcription with all essential features:

```python expandable
import assemblyai as aai
import asyncio
from typing import Dict, List
from assemblyai.types import (
    SpeakerOptions,
    LanguageDetectionOptions,
    PIIRedactionPolicy,
    PIISubstitutionPolicy,
)

# Configure API key
aai.settings.api_key = "your_api_key_here"

async def transcribe_meeting_async(audio_source: str) -> Dict:
    """
    Asynchronously transcribe a meeting recording with full features

    Args:
        audio_source: Either a local file path or publicly accessible URL
    """
    # Configure comprehensive meeting analysis
    config = aai.TranscriptionConfig(
        # Speaker diarization
        speaker_labels=True,
        speakers_expected=None,  # Use if you know exact number from Zoom/Meet/Teams
        speaker_options=SpeakerOptions(
            min_speakers_expected=2,
            max_speakers_expected=10  # Set a bit higher than expected; too high can cause over-splitting
        ),
        multichannel=False,  # Set to True if audio has separate channel per speaker

        # Language detection
        language_detection=True,  # Auto-detect the most used language
        language_detection_options=LanguageDetectionOptions(
            code_switching=True,  # Preserve language switches
            code_switching_confidence_threshold=0.5,
        ),

        # Punctuation and formatting
        punctuate=True,
        format_text=True,

        # Boost accuracy of meeting-specific vocabulary
        keyterms_prompt=["quarterly", "KPI", "roadmap", "deliverables"],

        # Speech Understanding - commonly used models
        summarization=True,
        sentiment_analysis=True,
        entity_detection=True,
        redact_pii=True,
        redact_pii_policies=[
            PIIRedactionPolicy.person_name,
            PIIRedactionPolicy.organization,
            PIIRedactionPolicy.occupation,
        ],
        redact_pii_sub=PIISubstitutionPolicy.hash,
        redact_pii_audio=True
    )

    # Create transcriber
    transcriber = aai.Transcriber()

    try:
        # Submit transcription job
        transcript = await asyncio.to_thread(
            transcriber.transcribe,
            audio_source,
            config=config
        )

        # Check status
        if transcript.status == aai.TranscriptStatus.error:
            raise Exception(f"Transcription failed: {transcript.error}")

        # Process speaker-labeled utterances
        print("\n=== SPEAKER-LABELED TRANSCRIPT ===\n")

        for utterance in transcript.utterances:
            # Format timestamp
            start_time = utterance.start / 1000  # Convert to seconds
            end_time = utterance.end / 1000

            # Print formatted utterance
            print(f"[{start_time:.1f}s - {end_time:.1f}s] Speaker {utterance.speaker}:")
            print(f"  {utterance.text}")
            print(f"  Confidence: {utterance.confidence:.2%}\n")

        # Print summary data
        print("\n=== MEETING SUMMARY ===\n")
        print({
            "id": transcript.id,
            "status": transcript.status,
            "duration": transcript.audio_duration,
            "speaker_count": len(set(u.speaker for u in transcript.utterances)),
            "word_count": len(transcript.words) if transcript.words else 0,
            "detected_language": transcript.language_code if hasattr(transcript, 'language_code') else None,
            "summary": transcript.summary,
        })

        return {
            "transcript": transcript,
            "utterances": transcript.utterances,
            "summary": transcript.summary,
        }

    except Exception as e:
        print(f"Error during transcription: {e}")
        raise

async def main():
    """
    Example usage with error handling
    """
    # Use either local file OR URL (not both)
    audio_source = "https://assembly.ai/wildfires.mp3"  # Or "path/to/recording.mp3"

    try:
        result = await transcribe_meeting_async(audio_source)

        # Additional processing
        print(f"\nTotal speakers identified: {len(set(u.speaker for u in result['utterances']))}")
        print(f"Meeting duration: {result['transcript'].audio_duration} seconds")

    except Exception as e:
        print(f"Failed to process meeting: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```

## How Can I Get Started Building a During-Call Live Meeting Notetaker?

Here's a complete example for real-time streaming transcription with meeting-optimized settings:

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "your_api_key"

# Keyterms to improve recognition accuracy
KEYTERMS = [
    "Alice Johnson",
    "Bob Smith",
    "Carol Davis",
    "quarterly review",
    "action items",
    "follow up",
    "deadline",
    "budget"
]

# MEETING NOTETAKER CONFIGURATION (different from voice agents!)
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,  # ALWAYS TRUE for meetings - users need readable text

    # Meeting-optimized turn detection (wait longer than voice agents)
    # universal-3-5-pro defaults: min_turn_silence=100ms, max_turn_silence=1000ms
    "min_turn_silence": 560,  # Wait longer for natural pauses (voice agents use ~100ms)
    "max_turn_silence": 2000,  # Allow thinking pauses

    # Keyterms for accuracy - pass each term as a separate query parameter
    "keyterms_prompt": KEYTERMS,
}

API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS, doseq=True)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
transcript_buffer = []

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("=" * 80)
    print(f"[{datetime.now().strftime('%H:%M:%S')}] Meeting transcription started")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")
    print(f"Keyterms configured: {', '.join(KEYTERMS)}")
    print("=" * 80)
    print("\nSpeak into your microphone. Press Ctrl+C to stop.\n")

    def stream_audio():
        """Stream audio from microphone to WebSocket"""
        global stream
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                if not stop_event.is_set():
                    print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    """Handle incoming messages from AssemblyAI"""
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        # Uncomment to see full JSON for debugging:
        # print("=" * 80)
        # print(json.dumps(data, indent=2, ensure_ascii=False))
        # print("=" * 80)
        # print()

        if msg_type == "Begin":
            session_id = data.get("id", "N/A")
            print(f"[SESSION] Started - ID: {session_id}\n")

        elif msg_type == "Turn":
            end_of_turn = data.get("end_of_turn", False)
            transcript = data.get("transcript", "")
            turn_order = data.get("turn_order", 0)
            end_of_turn_confidence = data.get("end_of_turn_confidence", 0.0)

            # FOR MEETING NOTETAKERS: Show partials for responsive UI
            if not end_of_turn and transcript:
                print(f"\r[LIVE] {transcript}", end="", flush=True)

            # FOR MEETING NOTETAKERS: Use formatted finals for readable display
            # (Unlike voice agents which should use utterance for speed)
            if end_of_turn and transcript:
                timestamp = datetime.now().strftime('%H:%M:%S')
                print(f"\n[{timestamp}] {transcript}")
                print(f"           Turn: {turn_order} | Confidence: {end_of_turn_confidence:.2%}")

                # Detect action items
                transcript_lower = transcript.lower()
                if any(term in transcript_lower for term in ["action item", "follow up", "deadline", "assigned to", "todo"]):
                    print("           ⚠️  ACTION ITEM DETECTED!")

                # Store final transcript
                transcript_buffer.append({
                    "timestamp": timestamp,
                    "text": transcript,
                    "turn_order": turn_order,
                    "confidence": end_of_turn_confidence,
                    "type": "final"
                })
                print()

        elif msg_type == "Termination":
            audio_duration = data.get("audio_duration_seconds", 0)
            print(f"\n[SESSION] Terminated - Duration: {audio_duration}s")
            save_transcript()

        elif msg_type == "Error":
            error_msg = data.get("error", "Unknown error")
            print(f"\n[ERROR] {error_msg}")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\n[WEBSOCKET ERROR] {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\n[WEBSOCKET] Disconnected - Status: {close_status_code}, Message: {close_msg}")

    global stream, audio
    stop_event.set()

    # Clean up audio stream
    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def save_transcript():
    """Save the transcript to a file"""
    if not transcript_buffer:
        print("No transcript to save.")
        return

    filename = f"meeting_transcript_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"

    with open(filename, "w", encoding="utf-8") as f:
        f.write("Meeting Transcript\n")
        f.write(f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
        f.write(f"Keyterms: {', '.join(KEYTERMS)}\n")
        f.write("=" * 80 + "\n\n")

        for entry in transcript_buffer:
            f.write(f"[{entry['timestamp']}] {entry['text']}\n")
            f.write(f"Confidence: {entry['confidence']:.2%}\n\n")

    print(f"Transcript saved to: {filename}")

def run():
    """Main function to run the streaming transcription"""
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\n\nCtrl+C received. Stopping transcription...")
        stop_event.set()

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

  These settings wait longer before ending turns to accommodate natural
  conversation pauses and ensure readable formatted text for display. You can
  [tweak these settings](/streaming/getting-started/transcribe-streaming-audio) to get
  the best results for your notetaker.

## How Do I Handle Multichannel Meeting Audio?

Many meeting platforms (Zoom, Teams, Google Meet) can record each participant on separate audio channels. This dramatically improves speaker identification accuracy.

### For Pre-recorded Meetings

```python
config = aai.TranscriptionConfig(
    multichannel=True,  # Enable when each speaker is on different channel
    speaker_labels=False,  # Disable - channels already separate speakers
    # Other settings...
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file, config=config)

# Access per-channel transcripts
for channel, channel_transcript in enumerate(transcript.channels):
    print(f"\n=== Channel {channel} ===")
    print(channel_transcript.text)
```

**When to use multichannel:**

- Zoom local recordings with "Record separate audio file for each participant" enabled
- Professional podcast recordings with individual microphones
- Conference systems with dedicated channels per participant
- Phone calls with caller and callee on separate channels

**Benefits:**

- **Perfect speaker separation** - No diarization errors
- **No speaker confusion or overlap issues**
- **Faster processing time** - Diarization not needed
- **Higher accuracy** - Model processes clean single-speaker audio

**How to enable in meeting platforms:**

- **Zoom**: Settings → Recording → Advanced → "Record a separate audio file for each participant"
- **Teams**: Requires third-party recording solutions like [Recall.ai](https://www.recall.ai/)
- **Google Meet**: Requires third-party recording solutions like [Recall.ai](https://www.recall.ai/)

### For Streaming Meetings

For real-time multichannel audio, create separate streaming sessions per channel:

```python expandable
import asyncio
import websockets

class ChannelTranscriber:
    def __init__(self, channel_id: int, speaker_name: str):
        self.channel_id = channel_id
        self.speaker_name = speaker_name
        self.connection_params = {
            "sample_rate": 16000,
            "speech_model": "universal-3-5-pro",
            "format_turns": True,
        }

    async def transcribe_channel(self, audio_stream):
        """Transcribe a single audio channel"""
        url = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(self.connection_params)}"

        # If you're using `websockets` version 13.0 or later, use `additional_headers` parameter. For older versions (< 13.0), use `extra_headers` instead.
        async with websockets.connect(url, additional_headers={"Authorization": API_KEY}) as ws:
            # Send audio from this channel only
            async for audio_chunk in audio_stream:
                await ws.send(audio_chunk)

            # Receive transcripts
            async for message in ws:
                data = json.loads(message)
                if data.get("type") == "Turn" and data.get("end_of_turn"):
                    print(f"{self.speaker_name}: {data['transcript']}")

# Create transcriber for each channel
async def transcribe_multichannel_meeting(channel_audio_streams):
    transcribers = [
        ChannelTranscriber(0, "Alice"),
        ChannelTranscriber(1, "Bob"),
    ]

    # Run all channels concurrently
    await asyncio.gather(*[
        t.transcribe_channel(stream)
        for t, stream in zip(transcribers, channel_audio_streams)
    ])
```

See our [multichannel streaming guide](/streaming/label-speakers-and-separate-channels#multichannel-streaming-audio) for complete implementation details.

## How Should I Handle Pre-recorded Transcription in Production?

Choose the right approach based on your application's needs:

### Option 1: Simple Blocking Call

```python
# Simple blocking call
transcript = await asyncio.to_thread(transcriber.transcribe, audio_url, config=config)
```

**Pros:**

- Simple, straightforward code
- Good for low volume applications
- Easy to understand and debug

**Cons:**

- Ties up resources while waiting
- Not suitable for high volume
- Cannot process multiple files simultaneously

**Best for:** Personal projects, prototypes, low-traffic applications

### Option 2: Webhook Callbacks (Production Recommended)

```python
config = aai.TranscriptionConfig(
    webhook_url="https://your-app.com/webhooks/assemblyai",
    webhook_auth_header_name="X-Webhook-Secret",
    webhook_auth_header_value="your_secret_here",
    speaker_labels=True,
    summarization=True,
    # ... other config
)

# Submit job and return immediately (non-blocking)
transcript = transcriber.submit(audio_url, config=config)
print(f"Job submitted: {transcript.id}")
# Your app can continue processing other requests

# Your webhook receives results when ready (typically 15-30% of audio duration)
```

**Webhook handler example:**

```python expandable
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhooks/assemblyai", methods=["POST"])
def assemblyai_webhook():
    # Verify webhook authenticity
    if request.headers.get("X-Webhook-Secret") != "your_secret_here":
        return jsonify({"error": "Unauthorized"}), 401

    import requests as http_requests

    data = request.json
    transcript_id = data["transcript_id"]
    status = data["status"]

    if status == "completed":
        # Fetch the full transcript (webhook only sends transcript_id and status)
        transcript = http_requests.get(
            f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
            headers={"authorization": "your_api_key"}
        ).json()
        process_completed_meeting(transcript)
    elif status == "error":
        log_transcription_error(transcript_id)

    return jsonify({"received": True}), 200

def process_completed_meeting(transcript):
    """Process completed meeting transcript"""
    utterances = transcript["utterances"]
    summary = transcript["summary"]

    # Store in database
    save_to_database(transcript)

    # Notify user
    send_notification(transcript["id"])
```

**Pros:**

- Non-blocking - submit and forget
- Scales to high volume
- Process multiple files in parallel
- Automatic retry on failures
- Get notified when complete

**Best for:** Production apps, user-uploaded recordings, batch processing, SaaS products

### Option 3: Polling (Custom Workflows)

```python
# Submit job
transcript = transcriber.submit(audio_url, config=config)
print(f"Submitted: {transcript.id}")

# Poll for completion with progress tracking
while transcript.status not in [aai.TranscriptStatus.completed, aai.TranscriptStatus.error]:
    await asyncio.sleep(5)
    transcript = transcriber.get_transcript(transcript.id)

    # Optional: Show progress
    print(f"Status: {transcript.status}...")

if transcript.status == aai.TranscriptStatus.completed:
    process_transcript(transcript)
else:
    print(f"Error: {transcript.error}")
```

**Pros:**

- Full control over retry logic
- Can show progress to users
- Good for background jobs
- Works without webhook infrastructure

**Cons:**

- Must implement your own polling loop
- Ties up resources while polling
- More complex than webhooks

**Best for:** Background job processors, CLIs with progress bars, custom retry logic

### Comparison Table

| Method      | Blocking | Scalability | Complexity | Best For                     |
| ----------- | -------- | ----------- | ---------- | ---------------------------- |
| Blocking    | Yes      | Low         | Low        | Prototypes, low volume       |
| Webhooks    | No       | High        | Medium     | Production, high volume      |
| Polling     | Partial  | Medium      | Medium     | Background jobs, progress UI |

### Scaling Considerations

- **HTTP rate limit:** 20,000 requests per 5-minute window, counted across submissions (POST) and polling (GET) combined
- **Exceeding the limit:** returns a `403` response
- **Parallel transcriptions (rate limit):** 200+ for paid accounts (queued beyond that)
- **Ramp up gradually:** start at 10-50 parallel requests, double incrementally
- **Avoid the rate limit:** use [webhooks](/pre-recorded-audio/webhooks) or jittered, widened polling — see [Polling without exceeding the rate limit](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale#polling-without-exceeding-the-rate-limit)
- **Contact Sales** before large-scale rollouts

## How Do I Identify Speakers in My Recording?

Speaker diarization tells you **when** speakers change ("Speaker A", "Speaker B"), but **Speaker Identification** tells you **who** they are by name or role.

### Why Use Speaker Identification?

**Instead of:**

```
Speaker A: Let's review the Q3 numbers.
Speaker B: Revenue was up 15% this quarter.
Speaker A: Excellent work on that launch.
```

**You get:**

```
Sarah Chen: Let's review the Q3 numbers.
Michael Rodriguez: Revenue was up 15% this quarter.
Sarah Chen: Excellent work on that launch.
```

### How It Works

Speaker Identification uses AssemblyAI's Speech Understanding API to map generic speaker labels to actual names or roles that you provide:

```python expandable
import assemblyai as aai

aai.settings.api_key = "your_api_key"

# Step 1: Transcribe with speaker diarization
config = aai.TranscriptionConfig(
    speaker_labels=True,  # Must enable speaker diarization first
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "name",  # or "role"
                "known_values": ["Sarah Chen", "Michael Rodriguez", "Alex Kim"]
            }
        }
    }
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe("meeting_recording.mp3", config=config)

# Access results with identified speakers
for utterance in transcript.utterances:
    print(f"{utterance.speaker}: {utterance.text}")
```

### Identifying by Role Instead of Name

For customer service, sales calls, or scenarios where you don't know names:

```python
config = aai.TranscriptionConfig(
    speaker_labels=True,
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "known_values": ["Agent", "Customer"]  # or ["Interviewer", "Interviewee"]
            }
        }
    }
)
```

**Common role combinations:**

- `["Agent", "Customer"]` - Customer service calls
- `["Support", "Customer"]` - Technical support
- `["Interviewer", "Interviewee"]` - Interviews
- `["Host", "Guest"]` - Podcasts
- `["Doctor", "Patient"]` - Medical consultations (with HIPAA compliance)

### How to Get Speaker Names

**For platform recordings:**

1. **Zoom**: Extract participant names from Zoom API or meeting JSON
2. **Teams**: Get attendees from Microsoft Graph API
3. **Google Meet**: Use Google Calendar API to get participants

**Example with Zoom:**

```python
# Get participant names from Zoom meeting
zoom_participants = get_zoom_meeting_participants(meeting_id)
speaker_names = [p["name"] for p in zoom_participants]

# Use in speaker identification
config = aai.TranscriptionConfig(
    speaker_labels=True,
    speakers_expected=len(speaker_names),  # Exact number of speakers to detect
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": speaker_names
            }
        }
    }
)
```

### How Speaker Identification Works

**Speaker Identification Requirements:**

1. **Speaker diarization must be enabled** - Cannot identify speakers without diarization first
2. **Requires sufficient audio per speaker** - Each speaker needs enough speech for accurate matching
3. **Works best with distinct voices** - Similar voices may be confused
4. **Post-processing step** - Adds additional processing time after transcription

**Accuracy depends on:**

- Audio quality (clear, minimal background noise)
- Voice distinctiveness (different genders, accents, tones)
- Amount of speech per speaker (more = better)
- Number of speakers (fewer = more accurate)

### Alternative: Add Identification Later

You can add speaker identification to an existing transcript by posting to the Speech Understanding API with the `transcript_id`. This is useful when you get speaker names after the transcription completes, or when building iterative workflows where users confirm speaker identities.

```python expandable
import requests

# First, transcribe with speaker diarization
transcript = transcriber.transcribe(audio_url, config=aai.TranscriptionConfig(speaker_labels=True))

# Later, add speaker identification using the transcript ID
understanding_body = {
    "transcript_id": transcript.id,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": ["Sarah Chen", "Michael Rodriguez"]
            }
        }
    }
}

result = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers={"Authorization": aai.settings.api_key},
    json=understanding_body
).json()

# Access identified speakers from the response
for utterance in result["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")
```

This approach is useful when:

- You get speaker names after the transcription completes
- You want to try different name mappings
- Building iterative workflows where users confirm speaker identities

For complete API details, see our [Speaker Identification documentation](/speech-understanding/speaker-identification).

## How Do I Translate Between Languages in Meetings?

AssemblyAI supports translation between 86 languages, enabling you to transcribe meetings in one language and translate to another.

### When to Use Translation

**Common use cases:**

- Transcribe Spanish meeting → Translate to English for documentation
- Transcribe multilingual meeting → Translate all to common language
- Create translated meeting notes for international teams
- Provide translated summaries for stakeholders

### Basic Translation

Translation is a Speech Understanding feature. You enable it via the `speech_understanding` parameter with `target_languages`:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

# Configure transcription with translation
data = {
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "translation": {
                "target_languages": ["es", "de"],
                "formal": True
            }
        }
    }
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

print("--- Original Transcript ---")
print(transcript["text"][:200] + "...")

print("\n--- Translations ---")
for language_code, translated_text in transcript["translated_texts"].items():
    print(f"{language_code.upper()}:")
    print(translated_text[:200] + "...")
```

### Translation with Speaker Labels

For meetings where you need per-utterance translations with speaker attribution:

```python
data = {
    "audio_url": audio_url,
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "translation": {
                "target_languages": ["es"],
                "match_original_utterance": True,
                "formal": True
            }
        }
    }
}

for utterance in transcript["utterances"]:
    print(f"Speaker {utterance['speaker']}:")
    print(f"  Original: {utterance['text'][:100]}...")
    print(f"  Spanish: {utterance['translated_texts']['es'][:100]}...")
```

### Supported Language Pairs

AssemblyAI supports translation between **86 languages**, including:

**Popular combinations:**

- Spanish ↔ English
- French ↔ English
- German ↔ English
- Mandarin ↔ English
- Japanese ↔ English
- Portuguese ↔ English
- And all combinations between supported languages

### Translation Response Format

The response includes `translated_texts` as a dictionary keyed by language code:

```python
{
    "text": "Original transcript in source language",
    "translated_texts": {
        "es": "Translated transcript in Spanish",
        "de": "Translated transcript in German"
    },
    "utterances": [
        {
            "speaker": "A",
            "text": "Hello, how are you?",
            "translated_texts": {
                "es": "Hola, ¿cómo estás?"
            },
            "start": 0,
            "end": 1500
        }
    ]
}
```

For complete language support and translation details, see our [Translation documentation](/speech-understanding/translation).

## What Workflows Can I Build for My AI Meeting Notetaker?

Use these Speech Understanding and Guardrails features to transform raw transcripts into actionable insights.

### Summarization

`summarization: true`

**What it does:** Generates an abstractive recap of the conversation (not verbatim).  
**Output:** `summary` string (bullets/paragraph format).  
**Great for:** Meeting notes, call recaps, executive summaries.  
**Notes:** Condenses and rephrases; minor details may be omitted by design.

**Example:**

```python
config = aai.TranscriptionConfig(
    summarization=True,
    summary_type="bullets",  # or "bullets_verbose", "gist", "headline", "paragraph"
    summary_model="informative",  # or "conversational"
)
```

### Sentiment Analysis

`sentiment_analysis: true`

**What it does:** Scores per-utterance sentiment (positive / neutral / negative).  
**Output:** Array of `{ text, sentiment, confidence, start, end }`.  
**Great for:** Customer satisfaction tracking, coaching, churn prediction.  
**Notes:** Segment-level (not global mood); sarcasm and very short utterances are harder to classify.

**Example:**

```python
for utterance in transcript.sentiment_analysis_results:
    if utterance.sentiment == "NEGATIVE":
        print(f"Negative sentiment detected: {utterance.text}")
```

### Entity Detection

`entity_detection: true`

**What it does:** Extracts named entities (people, organizations, locations, products, etc.).  
**Output:** Array of `{ entity_type, text, start, end }`.  
**Great for:** Auto-tagging topics, tracking competitors mentioned, CRM enrichment.  
**Notes:** Operates on post-redaction text if PII redaction is enabled.

**Example:**

```python
# Extract all organizations mentioned
organizations = [
    entity.text for entity in transcript.entities
    if entity.entity_type == "organization"
]
print(f"Companies mentioned: {', '.join(organizations)}")
```

### Redact PII Text

`redact_pii: true`

**What it does:** Scans transcript for personally identifiable information and replaces matches per policy.  
**Output:** `text` with replacements; original `words` timing preserved.  
**Great for:** GDPR/CCPA compliance, safe sharing, SOC2 requirements.  
**Notes:** Runs **before** downstream features; they see the redacted text.

**Recommended policies for meetings:**

```python
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,      # Remove names
        PIIRedactionPolicy.email_address,    # Remove emails
        PIIRedactionPolicy.phone_number,     # Remove phone numbers
        PIIRedactionPolicy.organization,     # Remove company names
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,  # Stable hash tokens
)
```

**Why hash substitution?**

- Stable across the file (same value → same token)
- Maintains sentence structure for LLM processing
- Prevents reconstruction of original data

### Redact PII Audio

`redact_pii_audio: true`

**What it does:** Produces a second audio file where redacted portions are bleeped/silenced.  
**Output:** `redacted_audio_url` in the transcript response.  
**Great for:** External sharing, training materials, demos.  
**Notes:** Original audio is untouched; bleeped sections may sound choppy.

### Complete Example

```python expandable
config = aai.TranscriptionConfig(
    # Core transcription
    speaker_labels=True,

    # Speech Understanding
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,

    # PII protection
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.email_address,
        PIIRedactionPolicy.phone_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
    redact_pii_audio=True,
)

transcript = transcriber.transcribe(audio_url, config=config)

# Access all features
meeting_insights = {
    "summary": transcript.summary,
    "sentiment_trend": analyze_sentiment_trend(transcript.sentiment_analysis_results),
    "entities": extract_entities(transcript.entities),
    "safe_transcript": transcript.text,  # PII redacted
    "safe_audio": transcript.redacted_audio_url,  # PII bleeped
}
```

## How Do I Improve the Accuracy of My Notetaker?

**Best practices:**

- Include participant names for better speaker recognition
- Add company-specific jargon and acronyms
- Include product names and technical terms
- Keep individual terms under 50 characters
- Up to 200 terms per request (Universal-2) or 1000 terms (Universal-3.5 Pro)

### Using Keyterms Prompt for Pre-recorded Transcription

Keyterms prompting improves recognition accuracy for domain-specific vocabulary by up to 21%:

```python expandable
# Define domain-specific vocabulary
company_terms = [
    "AssemblyAI",
    "Universal-3.5 Pro",
    "Speech Understanding",
    "diarization"
]

participant_names = [
    "Dylan Fox",
    "Sarah Chen",
    "Michael Rodriguez"
]

technical_terms = [
    "API endpoint",
    "WebSocket",
    "latency metrics",
    "TTFT"
]

# Configure with keyterms prompt
config = aai.TranscriptionConfig(
    keyterms_prompt=company_terms + participant_names + technical_terms,
    speaker_labels=True,
    # ... other settings
)
```

### Using Keyterms Prompt for Streaming

```python expandable
# Streaming with contextual keyterms
keyterms = [
    # Participant names
    "Alice Johnson",
    "Bob Smith",

    # Meeting-specific vocabulary
    "Q4 objectives",
    "revenue targets",
    "customer acquisition",

    # Technical terms
    "API integration",
    "cloud migration"
]

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "keyterms_prompt": keyterms,
}
```

## How Do I Process the Response from the API?

### Processing Pre-recorded Responses

```python expandable
def process_transcript(transcript):
    """
    Extract and process all relevant data from pre-recorded transcript
    """
    # Basic transcript data
    meeting_data = {
        "id": transcript.id,
        "duration": transcript.audio_duration,
        "confidence": transcript.confidence,
        "full_text": transcript.text
    }

    # Process speaker utterances
    speakers = {}
    for utterance in transcript.utterances:
        speaker = utterance.speaker

        if speaker not in speakers:
            speakers[speaker] = {
                "utterances": [],
                "total_speaking_time": 0,
                "word_count": 0
            }

        speakers[speaker]["utterances"].append({
            "text": utterance.text,
            "start": utterance.start,
            "end": utterance.end,
            "confidence": utterance.confidence
        })

        # Calculate speaking time
        speakers[speaker]["total_speaking_time"] += (utterance.end - utterance.start) / 1000
        speakers[speaker]["word_count"] += len(utterance.text.split())

    meeting_data["speakers"] = speakers

    # Extract summary
    if transcript.summary:
        meeting_data["summary"] = transcript.summary

    # Calculate meeting statistics
    total_duration = transcript.audio_duration  # Already in seconds
    meeting_data["statistics"] = {
        "total_speakers": len(speakers),
        "total_words": sum(s["word_count"] for s in speakers.values()),
        "average_confidence": transcript.confidence,
        "speaking_distribution": {
            speaker: {
                "percentage": (data["total_speaking_time"] / total_duration) * 100,
                "minutes": data["total_speaking_time"] / 60
            }
            for speaker, data in speakers.items()
        }
    }

    return meeting_data

# Example usage
result = process_transcript(transcript)
print(f"Meeting had {result['statistics']['total_speakers']} speakers")
print(f"Speaker A spoke for {result['statistics']['speaking_distribution']['A']['minutes']:.1f} minutes")
```

### Processing Streaming Responses

```python expandable
class StreamingResponseProcessor:
    def __init__(self):
        self.partial_buffer = ""
        self.final_transcripts = []
        self.turn_metadata = []

    def process_message(self, message: dict):
        """
        Process real-time streaming messages
        """
        msg_type = message.get("type")

        if msg_type == "Begin":
            return {
                "event": "session_started",
                "session_id": message.get("id"),
                "expires_at": message.get("expires_at")
            }

        elif msg_type == "Turn":
            return self.process_turn(message)

        elif msg_type == "Termination":
            return {
                "event": "session_ended",
                "audio_duration": message.get("audio_duration_seconds"),
                "session_duration": message.get("session_duration_seconds")
            }

    def process_turn(self, data: dict):
        """Process turn messages"""
        is_final = data.get("end_of_turn")
        transcript = data.get("transcript", "")
        turn_order = data.get("turn_order")

        response = {
            "turn_order": turn_order,
            "is_final": is_final,
            "confidence": data.get("end_of_turn_confidence", 0)
        }

        # Handle partials (for live display)
        if not is_final and transcript:
            self.partial_buffer = transcript
            response["event"] = "partial"
            response["text"] = transcript

        # Handle finals (for storage)
        elif is_final:
            final_transcript = {
                "turn_order": turn_order,
                "text": transcript,
                "confidence": data.get("end_of_turn_confidence"),
                "timestamp": datetime.now().isoformat()
            }
            self.final_transcripts.append(final_transcript)
            response["event"] = "final"
            response["text"] = transcript

            # Clear partial buffer
            self.partial_buffer = ""

        return response

    def get_full_transcript(self):
        """
        Combine all final transcripts into complete meeting transcript
        """
        return {
            "full_text": " ".join(t["text"] for t in self.final_transcripts),
            "transcripts": self.final_transcripts,
            "total_turns": len(self.final_transcripts)
        }

# Example usage
processor = StreamingResponseProcessor()

# If you're using `websockets` version 13.0 or later, use `additional_headers` parameter. For older versions (< 13.0), use `extra_headers` instead.
async with websockets.connect(API_ENDPOINT, additional_headers=headers) as ws:
    async for message in ws:
        data = json.loads(message)
        result = processor.process_message(data)

        if result["event"] == "partial":
            # Update UI with live transcript
            update_live_caption(result["text"])

        elif result["event"] == "final":
            # Save final transcript
            save_transcript_segment(result)

# Get complete transcript when done
full_transcript = processor.get_full_transcript()
```

## Additional Resources

- [Universal Pre-recorded Documentation](/pre-recorded-audio)
- [Universal-3.5 Pro Streaming Documentation](/streaming)
- [Speaker Diarization Guide](/pre-recorded-audio/label-speakers)
- [Speaker Identification Guide](/speech-understanding/speaker-identification)
- [Translation Guide](/speech-understanding/translation)
- [Getting Started Guide](/pre-recorded-audio/getting-started/transcribe-an-audio-file)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)
- [Support](https://www.assemblyai.com/contact/support)

---

<a id="build-a-medical-scribe-docs-medical-scribe-best-practices-mdx"></a>

# Build a medical scribe

URL: https://www.assemblyai.com/docs/medical-scribe-best-practices
Source: docs/medical-scribe-best-practices.mdx
Navigation: Overview > Use cases & integrations > Use case guides > Build a medical scribe
Description: Build AI-powered medical scribes that transcribe patient encounters and generate clinical documentation.

AssemblyAI provides everything you need to build a medical scribe, from high-accuracy transcription with medical terminology support to HIPAA-compliant PII redaction and structured clinical note generation through the LLM Gateway.

Choose the guide that matches your clinical workflow:

  
    Transcribe recorded patient encounters with Universal-3.5 Pro. Includes Medical Mode, speaker diarization, entity detection, PII redaction, and SOAP note generation.
  
  
    Stream audio from a microphone during live encounters with Universal-3.5 Pro Streaming. Includes LLM Gateway post-processing, medical keyterms, and automatic SOAP note generation.
  

## Which approach should I use?

| Scenario | Recommended approach |
| --- | --- |
| Post-visit documentation with highest accuracy | [Post-visit medical scribe](/medical-scribe-best-practices/medical-scribe-post-visit) |
| Live encounter transcription during patient visits | [Real-time medical scribe](/medical-scribe-best-practices/medical-scribe-real-time) |
| Telemedicine or emergency department visits | [Real-time medical scribe](/medical-scribe-best-practices/medical-scribe-real-time) |
| Specialist consultations with complex terminology | [Post-visit medical scribe](/medical-scribe-best-practices/medical-scribe-post-visit) |
| Hybrid: real-time notes with post-visit verification | Use both guides together |

---

<a id="build-a-post-visit-medical-scribe-docs-medical-scribe-best-practices-medical-scribe-post-visit-mdx"></a>

# Build a Post-Visit Medical Scribe

URL: https://www.assemblyai.com/docs/medical-scribe-best-practices/medical-scribe-post-visit
Source: docs/medical-scribe-best-practices/medical-scribe-post-visit.mdx
Navigation: Overview > Use cases & integrations > Use case guides > Build a medical scribe
Description: Complete example implementing pre-recorded transcription with Universal-3 Pro for post-visit clinical documentation.

This example implements a post-visit medical scribe using pre-recorded transcription with Universal-3.5 Pro. It transcribes a clinical encounter with speaker diarization, Medical Mode, entity detection, and PII redaction, then outputs a structured provider-patient dialogue with detected clinical entities.

For real-time transcription during live encounters, see the [Real-Time Medical Scribe](/medical-scribe-best-practices/medical-scribe-real-time) guide instead.

```python expandable
import assemblyai as aai
import asyncio
from typing import Dict, List
from assemblyai.types import (
    PIIRedactionPolicy,
    PIISubstitutionPolicy,
)

# Configure API key
aai.settings.api_key = "your_api_key_here"

async def transcribe_encounter_async(audio_source: str) -> Dict:
    """
    Asynchronously transcribe a medical encounter with Universal-3.5 Pro

    Args:
        audio_source: Either a local file path or publicly accessible URL
    """
    # Configure comprehensive medical transcription
    config = aai.TranscriptionConfig(
        speech_models=["universal-3-5-pro", "universal-2"],
        domain="medical-v1",
        language_detection=True,

        # Diarize provider and patient
        speaker_labels=True,
        speakers_expected=2,  # Typically provider and patient

        # Punctuation and Formatting
        punctuate=True,
        format_text=True,

        # Boost accuracy of medical terminology
        keyterms_prompt=[
            # Patient-specific context
            "hypertension", "diabetes mellitus type 2", "metformin",

            # Specialty-specific terms
            "auscultation", "palpation", "differential diagnosis",
            "chief complaint", "review of systems", "physical examination",

            # Common medications
            "lisinopril", "atorvastatin", "levothyroxine",

            # Procedure terms
            "electrocardiogram", "complete blood count", "hemoglobin A1c"
        ],

        # Speech understanding for medical documentation
        entity_detection=True,  # Extract medications, conditions, procedures
        redact_pii=True,  # HIPAA compliance
        redact_pii_policies=[
            PIIRedactionPolicy.person_name,
            PIIRedactionPolicy.date_of_birth,
            PIIRedactionPolicy.phone_number,
            PIIRedactionPolicy.email_address,
        ],
        redact_pii_sub=PIISubstitutionPolicy.hash,
        redact_pii_audio=True  # Create HIPAA-compliant audio
    )

    # Create async transcriber
    transcriber = aai.Transcriber()

    try:
        # Submit transcription job - works with both file paths and URLs
        transcript = await asyncio.to_thread(
            transcriber.transcribe,
            audio_source,
            config=config
        )

        # Check status
        if transcript.status == aai.TranscriptStatus.error:
            raise Exception(f"Transcription failed: {transcript.error}")

        # Process speaker-labeled utterances
        print("\n=== PROVIDER-PATIENT DIALOGUE ===\n")

        for utterance in transcript.utterances:
            # Format timestamp
            start_time = utterance.start / 1000  # Convert to seconds
            end_time = utterance.end / 1000

            # Identify speaker role
            speaker_label = "Provider" if utterance.speaker == "A" else "Patient"

            # Print formatted utterance
            print(f"[{start_time:.1f}s - {end_time:.1f}s] {speaker_label}:")
            print(f"  {utterance.text}")
            print(f"  Confidence: {utterance.confidence:.2%}\n")

        # Extract clinical entities
        if transcript.entities:
            print("\n=== CLINICAL ENTITIES DETECTED ===\n")
            medications = [e for e in transcript.entities if e.entity_type == "drug"]
            conditions = [e for e in transcript.entities if e.entity_type == "medical_condition"]
            procedures = [e for e in transcript.entities if e.entity_type == "medical_process"]

            if medications:
                print("Medications:", ", ".join([m.text for m in medications]))
            if conditions:
                print("Conditions:", ", ".join([c.text for c in conditions]))
            if procedures:
                print("Procedures:", ", ".join([p.text for p in procedures]))

        return {
            "transcript": transcript,
            "utterances": transcript.utterances,
            "entities": transcript.entities,
            "redacted_audio_url": transcript.redacted_audio_url
        }

    except Exception as e:
        print(f"Error during transcription: {e}")
        raise

async def main():
    """
    Example usage for medical encounter
    """
    # Can use either local file or URL
    audio_source = "path/to/patient_encounter.mp3"  # Or use URL
    # audio_source = "https://your-secure-storage.com/encounter.mp3"

    try:
        result = await transcribe_encounter_async(audio_source)

        # Additional processing
        print(f"\nEncounter duration: {result['transcript'].audio_duration} seconds")

        # Could send to LLM Gateway for SOAP note generation here

    except Exception as e:
        print(f"Failed to process encounter: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```

## Next steps

- [Build a Real-Time Medical Scribe](/medical-scribe-best-practices/medical-scribe-real-time) — Stream audio from a microphone with LLM post-processing
- [Medical Mode](/pre-recorded-audio/medical-mode) — Improve medical terminology accuracy
- [End-to-end medical scribe pipeline](/getting-started/end-to-end-examples/medical-scribe) — Compact pipeline with SOAP note generation

---

<a id="build-a-real-time-medical-scribe-docs-medical-scribe-best-practices-medical-scribe-real-time-mdx"></a>

# Build a Real-Time Medical Scribe

URL: https://www.assemblyai.com/docs/medical-scribe-best-practices/medical-scribe-real-time
Source: docs/medical-scribe-best-practices/medical-scribe-real-time.mdx
Navigation: Overview > Use cases & integrations > Use case guides > Build a medical scribe
Description: Complete example for real-time streaming medical transcription with LLM Gateway post-processing and SOAP note generation.

This example implements a real-time medical scribe using Universal-3.5 Pro Streaming with LLM Gateway post-processing. It uses [Medical Mode](/streaming/medical-mode) to improve accuracy for clinical terminology, streams audio from a microphone, applies LLM-powered clinical editing on each turn, and generates a SOAP note at the end of the session.

For post-visit documentation using pre-recorded audio, see the [Post-Visit Medical Scribe](/medical-scribe-best-practices/medical-scribe-post-visit) guide instead.

```python expandable
import os
import json
import time
import threading
from datetime import datetime
from urllib.parse import urlencode

import pyaudio
import websocket
import requests
from dotenv import load_dotenv
from simple_term_menu import TerminalMenu

# Load environment variables from .env if present
try:
    load_dotenv()
except Exception:
    pass

"""
Medical Scribe – Real-time STT + LLM Gateway Enhancement (SOAP-ready)

What this does
--------------
1) Streams mic audio to AssemblyAI Real-time STT
2) On every utterance or end of turn, calls AssemblyAI LLM Gateway to
   apply *medical* edits (terminology, punctuation, proper nouns, etc.)
3) Logs encounter turns and generates a SOAP note at session end via the Gateway

Quick start
-----------
export ASSEMBLYAI_API_KEY=your_key
python medical_scribe_llm_gateway.py
"""

# === Config ===
ASSEMBLYAI_API_KEY = os.environ.get("ASSEMBLYAI_API_KEY", "your_api_key_here")

# WebSocket / STT parameters - CONSERVATIVE SETTINGS FOR MEDICAL
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",  # Enable Medical Mode for clinical terminology accuracy

    # MEDICAL SCRIBE CONFIGURATION - Conservative for clinical accuracy
    # Medical conversations have LONG pauses (provider thinking, examining patient, reviewing charts)
    # universal-3-5-pro defaults: min_turn_silence=100ms, max_turn_silence=1000ms
    "min_turn_silence": 800,  # Wait much longer (vs ~100ms for voice agents, 560ms for meetings)
    "max_turn_silence": 2000,  # Longer for clinical thinking pauses
}

API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio config
FRAMES_PER_BUFFER = 800  # 50ms @ 16kHz
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Globals
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
encounter_buffer = []  # list of dicts with turn data
last_processed_turn = None

# === Model selection ===
AVAILABLE_MODELS = [
    {"id": "claude-haiku-4-5-20251001", "name": "Claude Haiku 4.5", "description": "Fastest Claude, good for simple tasks"},
    {"id": "claude-sonnet-4-5-20250929", "name": "Claude Sonnet 4.5", "description": "Best for coding & agents"},
    {"id": "claude-sonnet-4-6", "name": "Claude Sonnet 4.6", "description": "Latest Sonnet, fast with strong reasoning"},
]

def select_model():
    menu_entries = [f"{m['name']} - {m['description']}" for m in AVAILABLE_MODELS]
    terminal_menu = TerminalMenu(
        menu_entries,
        title="Select a model (Use ↑↓ arrows, Enter to select):",
        menu_cursor="❯ ",
        menu_cursor_style=("fg_cyan", "bold"),
        menu_highlight_style=("bg_cyan", "fg_black"),
        cycle_cursor=True,
        clear_screen=False,
        show_search_hint=True,
    )
    idx = terminal_menu.show()
    if idx is None:
        print("Model selection cancelled. Exiting...")
        raise SystemExit(0)
    return AVAILABLE_MODELS[idx]["id"]

selected_model = None

# === Gateway helpers ===

def _gateway_chat(messages, max_tokens=800, temperature=0.2, retries=3, backoff=0.75):
    """Call AssemblyAI LLM Gateway with debug logging and retry."""
    url = "https://llm-gateway.assemblyai.com/v1/chat/completions"
    headers = {
        "Authorization": ASSEMBLYAI_API_KEY,
        "Content-Type": "application/json",
    }
    payload = {
        "model": selected_model,
        "messages": messages,
        "max_tokens": max_tokens,
        "temperature": temperature,
    }

    last = None
    for attempt in range(retries):
        try:
            print(f"[LLM] POST {url} (model={selected_model}, attempt {attempt+1}/{retries})")
            resp = requests.post(url, headers=headers, json=payload, timeout=60)
            print(f"[LLM] ← status {resp.status_code}, bytes {len(resp.content)}")
            last = resp
        except Exception as e:
            if attempt == retries - 1:
                raise RuntimeError(f"Gateway request error: {e}")
            time.sleep(backoff * (attempt + 1))
            continue

        if resp.status_code == 200:
            data = resp.json()
            if not data.get("choices") or not data["choices"][0].get("message"):
                raise RuntimeError(f"Gateway OK but empty body: {str(data)[:200]}")
            return data
        if resp.status_code in (429, 500, 502, 503, 504):
            print(f"[LLM RETRY] {resp.status_code}: {resp.text[:180]}")
            time.sleep(backoff * (attempt + 1))
            continue
        raise RuntimeError(f"Gateway error {resp.status_code}: {resp.text[:300]}")

    raise RuntimeError(
        f"Gateway failed after retries. Last={getattr(last,'status_code','n/a')} {getattr(last,'text','')[:180]}"
    )

def post_process_with_llm(text: str) -> str:
    """Medical editing & normalization using LLM Gateway."""
    system = {
        "role": "system",
        "content": (
            "You are a clinical transcription editor. Keep the speaker's words, "
            "fix medical terminology (drug names, dosages, anatomy), proper nouns, "
            "and punctuation for readability. Preserve meaning and avoid inventing "
            "details. Prefer U.S. clinical style. If a medication or condition is "
            "phonetically close, correct to the most likely clinical term."
        ),
    }

    user = {
        "role": "user",
        "content": (
            "Edit this short transcript for medical accuracy and readability.\n\n"
            f"Transcript:\n{text}"
        ),
    }

    try:
        res = _gateway_chat([system, user], max_tokens=600)
        return res["choices"][0]["message"]["content"].strip()
    except Exception as e:
        print(f"[LLM EDIT ERROR] {e}. Falling back to original.")
        return text

def generate_clinical_note():
    """Create a SOAP note from the encounter buffer via Gateway."""
    if not encounter_buffer:
        print("No encounter data to summarize.")
        return

    print("\n=== GENERATING CLINICAL DOCUMENTATION (SOAP) ===")
    # Build a compact transcript string for the LLM
    lines = []
    for e in encounter_buffer:
        if e.get("type") == "utterance":
            lines.append(f"[{e['timestamp']}] {e.get('speaker', 'Speaker')}: {e['text']}")
        elif e.get("type") == "final":
            lines.append(f"[{e['timestamp']}] FINAL: {e['text']}")
    combined = "\n".join(lines)

    system = {
        "role": "system",
        "content": (
            "You are a clinician generating concise, structured notes. "
            "Produce a SOAP note (Subjective, Objective, Assessment, Plan). "
            "Use bullet points, keep it factual, infer reasonable clinical semantics "
            "from the transcript but do NOT invent data. Include medications with dosage "
            "and frequency if mentioned."
        ),
    }
    user = {
        "role": "user",
        "content": (
            "Create a SOAP note from this clinical encounter transcript.\n\n"
            f"Transcript:\n{combined}\n\n"
            "Format strictly as:\n"
            "Subjective:\n- ...\n\nObjective:\n- ...\n\nAssessment:\n- ...\n\nPlan:\n- ...\n"
        ),
    }

    try:
        res = _gateway_chat([system, user], max_tokens=1200)
        soap = res["choices"][0]["message"]["content"].strip()
        fname = f"clinical_note_soap_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"
        with open(fname, "w", encoding="utf-8") as f:
            f.write(soap)
        print(f"SOAP note saved: {fname}")
    except Exception as e:
        print(f"[SOAP ERROR] {e}")

# === WebSocket callbacks ===

def on_open(ws):
    print("=" * 80)
    print(f"[{datetime.now().strftime('%H:%M:%S')}] Medical transcription started")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")
    print(f"Gateway model: {selected_model}")
    print("=" * 80)
    print("\nSpeak to begin. Press Ctrl+C to stop.\n")

    def stream_audio():
        global stream
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                if not stop_event.is_set():
                    print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio, daemon=True)
    audio_thread.start()

def on_message(ws, message):
    global last_processed_turn
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        if msg_type == "Begin":
            print(f"[SESSION] Started - ID: {data.get('id','N/A')}\n")

        elif msg_type == "Turn":
            end_of_turn = data.get("end_of_turn", False)
            transcript = data.get("transcript", "")
            utterance = data.get("utterance", "")
            turn_order = data.get("turn_order", 0)

            # live partials
            if not end_of_turn and transcript:
                print(f"\r[PARTIAL] {transcript[:120]}...", end="", flush=True)

            # If AssemblyAI has finalized a turn, LLM-edit the transcript
            if end_of_turn and transcript:
                if last_processed_turn == turn_order:
                    return  # avoid duplicate processing
                last_processed_turn = turn_order

                ts = datetime.now().strftime('%H:%M:%S')
                print("\n[DEBUG] EOT received. Calling LLM…")
                edited = post_process_with_llm(transcript)

                changed = "(edited)" if edited.strip() != transcript.strip() else "(no change)"
                print(f"\n[{ts}] [FINAL {changed}]")
                print(f"  ├─ Original STT : {transcript}")
                print(f"  └─ Edited by LLM: {edited}")
                print(f"Turn: {turn_order}")

                encounter_buffer.append({
                    "timestamp": ts,
                    "text": edited,
                    "original_text": transcript,
                    "turn_order": turn_order,
                    "type": "final",
                })

            # If we also get per-utterance chunks, just log them raw (no LLM) for timeline
            elif utterance:
                ts = datetime.now().strftime('%H:%M:%S')

                low = utterance.lower()
                if any(t in low for t in ["medication", "prescribe", "dosage", "mg", "daily"]):
                    print("           💊 MEDICATION MENTIONED")
                if any(t in low for t in ["pain", "symptom", "complaint", "problem"]):
                    print("           🏥 SYMPTOM REPORTED")
                if any(t in low for t in ["diagnose", "assessment", "impression"]):
                    print("           📋 DIAGNOSIS DISCUSSED")

                encounter_buffer.append({
                    "timestamp": ts,
                    "text": utterance,
                    "original_text": utterance,
                    "turn_order": turn_order,
                    "type": "utterance",
                })
                print()

        elif msg_type == "Termination":
            dur = data.get("audio_duration_seconds", 0)
            print(f"\n[SESSION] Terminated – Duration: {dur}s")
            save_encounter_transcript()
            generate_clinical_note()

        elif msg_type == "Error":
            print(f"\n[ERROR] {data.get('error', 'Unknown error')}")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    print(f"\n[WEBSOCKET ERROR] {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    print(f"\n[WEBSOCKET] Disconnected – Status: {close_status_code}")
    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# === Persist artifacts ===

def save_encounter_transcript():
    if not encounter_buffer:
        print("No encounter data to save.")
        return

    fname = f"encounter_transcript_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"
    with open(fname, "w", encoding="utf-8") as f:
        f.write("Clinical Encounter Transcript\n")
        f.write(f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
        f.write("=" * 80 + "\n\n")
        for e in encounter_buffer:
            if e.get("speaker"):
                f.write(f"[{e['timestamp']}] {e['speaker']}: {e['text']}\n")
            else:
                f.write(f"[{e['timestamp']}] {e['text']}\n")
            f.write("\n")
    print(f"Encounter transcript saved: {fname}")

# === Main ===

def run():
    global audio, stream, ws_app, selected_model

    print("=" * 60)
    print("  🎙️  Medical Scribe - STT + LLM Gateway")
    print("=" * 60)
    selected_model = select_model()
    print(f"✓ Using model: {selected_model}")

    # Init mic
    audio = pyaudio.PyAudio()
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Audio stream opened successfully.")
    except Exception as e:
        print(f"Error opening audio stream: {e}")
        if audio:
            audio.terminate()
        return

    # Connect WS
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": ASSEMBLYAI_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    ws_thread = threading.Thread(target=ws_app.run_forever, daemon=True)
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\n\nCtrl+C received. Stopping...")
        stop_event.set()
        # best-effort terminate
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            except Exception as e:
                print(f"Error sending termination: {e}")
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)
    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## Using a different streaming model

If you switch to a different streaming model such as Universal-Streaming, formatting is not applied automatically. Two alternative models are available:

- [`universal-streaming-english`](/streaming/getting-started/transcribe-streaming-audio) — English only
- [`universal-streaming-multilingual`](/streaming/getting-started/transcribe-streaming-audio) — Supports 6 languages: English, Spanish, German, French, Italian, and Portuguese

Add `format_turns=True` to your connection parameters to receive transcripts with punctuation, casing, and inverse text normalization (for example, dates, times, and phone numbers):

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-streaming-english",  # or "universal-streaming-multilingual"
    "format_turns": True,
    ...
}
```

When `format_turns` is enabled, the model emits two `Turn` messages when a turn ends: one with `turn_is_formatted: false` (the raw unformatted transcript) and a second with `turn_is_formatted: true` (the formatted transcript). To avoid calling the LLM twice, check for both `end_of_turn` and `turn_is_formatted`:

```python
elif msg_type == "Turn":
    end_of_turn = data.get("end_of_turn", False)
    turn_is_formatted = data.get("turn_is_formatted", False)
    transcript = data.get("transcript", "")

    if end_of_turn and turn_is_formatted and transcript:
        # Formatted final transcript — safe to post-process with LLM
        edited = post_process_with_llm(transcript)
        ...
```

Note that `turn_is_formatted` should not be used on its own to detect end of turn — always use `end_of_turn` for that.

Universal-Streaming also uses a different turn detection system than U3 Pro. Instead of punctuation-based detection, it uses a confidence threshold controlled by `end_of_turn_confidence_threshold` (default `0.4`). The `min_turn_silence` and `max_turn_silence` parameters in the main code above are U3 Pro–specific and should be replaced with `end_of_turn_confidence_threshold` when switching models:

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-streaming-english",
    "format_turns": True,
    "end_of_turn_confidence_threshold": 0.6,  # increase for fewer false turn endings
}
```

For Universal-3.5 Pro (`universal-3-5-pro`), `format_turns` is not needed since formatting is built into the end-of-turn system and `end_of_turn` and `turn_is_formatted` always have the same value.

## Next steps

- [Build a Post-Visit Medical Scribe](/medical-scribe-best-practices/medical-scribe-post-visit) — Pre-recorded transcription for post-visit documentation
- [Medical Mode for Streaming](/streaming/medical-mode) — Improve streaming medical terminology accuracy
- [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio) — Full streaming model documentation

---

<a id="best-practices-for-building-voice-agents-docs-voice-agents-best-practices-mdx"></a>

# Best Practices for Building Voice Agents

URL: https://www.assemblyai.com/docs/voice-agents/best-practices
Source: docs/voice-agents/best-practices.mdx
Navigation: Overview > Use cases & integrations > Use case guides
Description: Complete guide for building voice agents with AssemblyAI's Universal-3 Pro Streaming

## Introduction

AssemblyAI's **Universal-3.5 Pro Streaming** is the most accurate real-time speech-to-text model designed for voice agents. It delivers formatted, immutable transcripts with sub-300ms latency, exceptional entity accuracy, native multilingual code switching, and a fully promptable interface, all optimized for conversational AI workflows.

The STT component is the "ears" of your voice agent. Transcription errors propagate into the LLM and response logic, so even small accuracy gaps compound in impact. Choosing and configuring the right STT model is one of the highest-leverage decisions you can make when building a voice agent. For guidance on how to evaluate and compare STT models for your use case, see the [streaming evaluation guide](/streaming/evaluations/voice-agents).

## Why Universal-3.5 Pro Streaming for Voice Agents?

Voice agents need speed, accuracy, and natural turn-taking. Universal-3.5 Pro Streaming is purpose-built for this:

**Sub-300ms latency with formatted output**

- Immutable transcripts arrive fully formatted (punctuation, capitalization), no waiting for a separate formatting step
- Every final transcript is ready for immediate LLM processing

**Exceptional entity accuracy**

- Credit card numbers, phone numbers, email addresses, physical addresses, and names are transcribed with high accuracy
- Short utterances like "yes", "no", "mmhmm" are handled reliably

**Punctuation-based turn detection**

- Turn boundaries are determined by terminal punctuation (`.` `?` `!`) combined with silence thresholds
- Configurable `min_turn_silence` and `max_turn_silence` parameters let you tune responsiveness vs. accuracy
- No confidence-score guessing. The model understands when a sentence is complete

**Fully promptable**

- Contextual `prompt` parameter for describing what the audio is about
- Dynamic prompting mid-session via `UpdateConfiguration`. Adapt the model to each stage of the conversation
- `keyterms_prompt` for boosting recognition of specific names, brands, and domain terms

**Native multilingual support**

- Supports English, Spanish, French, German, Italian, and Portuguese
- Automatic code-switching between languages within a single session
- Language-specific prompting for improved accuracy

## What Languages Does Universal-3.5 Pro Streaming Support?

Universal-3.5 Pro Streaming supports 18 languages with automatic code-switching:

- English
- Spanish
- German
- French
- Portuguese
- Italian
- Turkish
- Dutch
- Swedish
- Norwegian
- Danish
- Finnish
- Hindi
- Vietnamese
- Arabic
- Hebrew
- Japanese
- Mandarin

The model handles code-switching natively. Speakers can switch between supported languages mid-conversation without any configuration changes. Accuracy improves when you specify the expected language in the prompt. See [Supported languages](/streaming/getting-started/transcribe-streaming-audio) for the full language list and regional dialect reference.

To bias the model toward a specific language, pass the `language_code` connection parameter — see [Language selection](/streaming/getting-started/optimizing-accuracy-and-latency#language-selection). For multilingual conversations, no configuration is needed — code-switching is native.

## How Do I Get Started?

### Complete voice agent stack

AssemblyAI provides a speech-to-speech [Voice Agent API](/voice-agents/voice-agent-api) that abstracts away the complexity of a full voice agent stack: managed STT, LLM, turn detection, and TTS in a single endpoint.

For a cascading architecture, AssemblyAI has the best speech-to-text model. For a complete stack, you need:

1. **Speech-to-Text (STT):** AssemblyAI Universal-3.5 Pro Streaming
2. **Large Language Model (LLM):** OpenAI, Anthropic, Google, etc.
3. **Text-to-Speech (TTS):** Rime, Cartesia, ElevenLabs, etc.
4. **Orchestration:** LiveKit, Pipecat, or custom build

### Pre-built integrations

**LiveKit Agents (recommended)**

LiveKit provides the fastest path to a working voice agent with AssemblyAI. See [Universal-3.5 Pro Streaming on LiveKit](/voice-agents/livekit-universal-3-5-pro) for a full guide.

```python
from livekit.agents import AgentSession
from livekit.plugins import assemblyai, silero

session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,
    ),
    turn_detection="stt",
    min_endpointing_delay=0,
)
```

**Pipecat by Daily**

Pipecat is an open-source framework for conversational AI with maximum customizability. See [Universal-3.5 Pro Streaming on Pipecat](/voice-agents/pipecat-universal-3-5-pro) for a full guide.

```python
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.assemblyai.models import AssemblyAIConnectionParams

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    connection_params=AssemblyAIConnectionParams(
        speech_model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
    ),
    vad_force_turn_endpoint=False,  # Use AssemblyAI's built-in turn detection
)
```

### Direct WebSocket connection

For custom builds, connect directly to the WebSocket API:

```python expandable
import json
import pyaudio
import websocket
import threading
import time
from urllib.parse import urlencode

API_KEY = "YOUR_API_KEY"
SAMPLE_RATE = 16000

CONNECTION_PARAMS = {
    "sample_rate": SAMPLE_RATE,
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 1000,
}

API_ENDPOINT_BASE = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE}?{urlencode(CONNECTION_PARAMS)}"

def on_message(ws, message):
    data = json.loads(message)

    if data.get("type") == "Turn":
        transcript = data.get("transcript", "")
        end_of_turn = data.get("end_of_turn", False)

        if end_of_turn:
            # Final transcript - send to LLM
            print(f"Final: {transcript}")
        else:
            # Partial - can start pre-emptive LLM generation
            print(f"Partial: {transcript}")

    elif data.get("type") == "SpeechStarted":
        # User started speaking - handle barge-in
        print("Speech detected, interrupt agent if speaking")

ws = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": API_KEY},
    on_message=on_message,
)
```

## How Does Turn Detection Work?

Universal-3.5 Pro Streaming uses a **punctuation-based turn detection system** controlled by two parameters:

| Parameter          | Default    | Description                                                              |
| ------------------ | ---------- | ------------------------------------------------------------------------ |
| `min_turn_silence` | `100` ms   | Silence before a speculative end-of-turn check fires.                    |
| `max_turn_silence` | `1000` ms  | Maximum silence before forcing the turn to end.                          |

**How it works:**

1. User speaks → audio streams to AssemblyAI
2. User pauses for `min_turn_silence` → model checks for terminal punctuation (`.` `?` `!`)
3. If terminal punctuation found → turn ends immediately with `end_of_turn: true`
4. If no terminal punctuation → partial emitted with `end_of_turn: false`, turn continues
5. If silence reaches `max_turn_silence` → turn forced to end regardless of punctuation

This is different from the legacy Universal-Streaming models, which used a confidence-based `end_of_turn_confidence_threshold`. Universal-3.5 Pro Streaming does not use that parameter. Turn decisions are based on punctuation after silence thresholds.

### Configuration presets

```python
# Fast - quick confirmations, IVR, yes/no questions
fast_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 800,
}

# Balanced - most voice agent conversations (recommended)
balanced_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 1000,
}

# Patient - entity dictation, complex instructions
patient_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 200,
    "max_turn_silence": 2000,
}
```

### Entity splitting tradeoff

Lower silence values produce faster transcripts but can split entities across turns:

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"It's John."                    → turn ends (period found after 100ms pause)
"Smith."                        → new turn
"At gmail.com."                 → new turn

# With (min_turn_silence=400, max_turn_silence=2000)
"It's john.smith@gmail.com."    → single turn (properly formatted)
```

For voice agents, the downstream LLM can usually piece together split entities. But if your use case involves entity extraction or alphanumeric dictation, increase `min_turn_silence` and `max_turn_silence` during those portions of the conversation using [dynamic configuration updates](#how-do-i-update-configuration-mid-session).

## How Do I Handle Barge-In and Interruptions?

### SpeechStarted events

Universal-3.5 Pro Streaming emits `SpeechStarted` events when speech has been detected. `SpeechStarted` is only emitted when the model produces a transcript. This makes it a reliable signal for barge-in handling:

```json
{
  "type": "SpeechStarted",
  "timestamp": 14400,
  "confidence": 0.79
}
```

When you receive a `SpeechStarted` event:
1. Stop TTS playback immediately
2. Switch the agent back to listening mode
3. Wait for the user's full turn to complete before responding

### Filtering backchannels

Users often produce short backchannel utterances ("mhm", "yeah", "um", "okay") while the agent is speaking. Treating every `SpeechStarted` event as a barge-in causes the agent to stop mid-sentence on these fillers, even though the user didn't intend to interrupt.

The fix is to gate barge-in on each `Turn` event during agent speech: suppress the interrupt when the transcript is short or every token is a known backchannel. Implementation depends on your stack:

- **Voice Agent API**: semantic interruption classification is built in. See [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions).
- **LiveKit**: LiveKit Cloud users should enable [adaptive interruption handling](/voice-agents/livekit-universal-3-5-pro#interruption-handling). Self-hosted deployments use the two custom filters in the same section.
- **Direct WebSocket**: see [Interruption handling](/voice-agents/universal-3-5-pro-streaming-api#interruption-handling) on the Universal-3.5 Pro Streaming API page for the combined word-count + filler-word filter.

### VAD threshold alignment

Universal-3.5 Pro Streaming includes an internal Silero VAD controlled by the `vad_threshold` parameter (default `0.3`). If you're also running a local VAD (common in LiveKit and Pipecat), align the thresholds to avoid a dead zone where one detects speech but the other doesn't:

```python
# Both thresholds aligned at 0.3
stt = assemblyai.STT(
    model="universal-3-5-pro",
    vad_threshold=0.3,
)
vad = silero.VAD.load(
    activation_threshold=0.3,
)
```

If you're in a noisy environment and getting false speech triggers, raise both thresholds together.

## How Can I Use Prompting to Improve Accuracy?

### The prompt parameter

Universal-3.5 Pro Streaming supports a `prompt` parameter for [contextual prompting](/streaming/prompting-and-keyterms) — a natural-language description of what the audio is about. Transcription behavior itself (verbatim output, punctuation, turn detection) is built in and managed automatically; the prompt carries context, not instructions.

  **Beta feature**

Prompting is a beta feature. We recommend starting without a prompt to establish baseline performance, then adding context to optimize for your use case.

```python
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "prompt": "AI voice agent call with a customer about an internet service outage."
}
```

**Tips for effective prompts:**

- **Describe the conversation**: domain, scenario, or full details — start broad, and add only details your application actually knows (see the [three context levels](/streaming/prompting-and-keyterms#contextual-prompting))
- **Include names and identifiers you already know**: caller name, account or order IDs, products — detailed context helps the model spell them correctly
- **Specify language**: use the `language_code` connection parameter for monolingual sessions ([Language selection](/streaming/getting-started/optimizing-accuracy-and-latency#language-selection))

### Keyterms prompting

Use `keyterms_prompt` to boost recognition of specific names, brands, or domain terms, up to 100 terms per session:

```python
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "keyterms_prompt": json.dumps([
        "AssemblyAI",
        "LiveKit",
        "Dr. Rodriguez",
        "Lisinopril",
        "iPhone 15 Pro",
    ])
}
```

**Best practices for keyterms:**

- Include proper names, product names, technical terms, and domain-specific jargon
- Include terms up to 50 characters each
- Don't include common English words, single letters, or generic phrases
- Don't exceed 100 terms total

For detailed guidance, see [Keyterms prompting](/streaming/prompting-and-keyterms).

## How Do I Update Configuration Mid-Session?

You can update `prompt`, `keyterms_prompt`, `min_turn_silence`, and `max_turn_silence` during an active session using `UpdateConfiguration`. This is one of Universal-3.5 Pro Streaming's most powerful features for voice agents.

### Dynamic keyterms by conversation stage

As your voice agent moves through different stages, update keyterms to match what the user is likely to say:

```python
# Caller identification stage
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["Kelly Byrne-Donoghue", "date of birth", "January", "February"]
}))

# Medical intake stage
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["cardiology", "echocardiogram", "Dr. Patel", "metoprolol"]
}))

# Payment stage - also increase max_turn_silence for credit card dictation
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["Visa", "Mastercard", "American Express"],
    "max_turn_silence": 3000
}))
```

### Dynamic prompting

You can also update the transcription prompt mid-session. This is especially powerful when paired with tool calls in your LLM:

- If your agent asks a yes/no question, prompt the model to anticipate short responses
- If your agent asks for a phone number or email, prompt it to expect those formats
- If you present a list of options, boost those options in the prompt

```python
# After asking "Would you like to confirm your appointment?"
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "prompt": "User is responding yes or no to a confirmation question. Expect short responses."
}))

# After asking "What's your phone number?"
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "prompt": "User is dictating a phone number. Expect digits and formatting.",
    "max_turn_silence": 3000
}))
```

## How Do I Use Speaker Diarization?

Streaming Diarization identifies and labels individual speakers in real time. Each `Turn` event includes a `speaker_label` field (e.g., `"A"`, `"B"`) indicating which speaker produced that transcript.

Enable it by adding `speaker_labels: true` to your connection parameters:

```python
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "speaker_labels": True,
}
```

Speaker accuracy improves over the course of a session as the model accumulates embedding context.

**With LiveKit:**

```python
stt = assemblyai.STT(
    model="universal-3-5-pro",
    speaker_labels=True,
)
```

**With Pipecat** (including custom formatting):

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    connection_params=AssemblyAIConnectionParams(
        speech_model="universal-3-5-pro",
        speaker_labels=True,
    ),
    speaker_format="[{speaker}] {text}",
)
```

For more details, see [Streaming Diarization and Multichannel](/streaming/label-speakers-and-separate-channels).

## How Do I Optimize for Latency?

### Key optimizations

**1. Use the right silence thresholds**

Start with `min_turn_silence=100` and `max_turn_silence=1000`. Only increase if you're seeing entity splitting issues.

**2. Tune `interruption_delay` for faster TTFT**

The `interruption_delay` parameter controls how soon the first partial is emitted. Set `interruption_delay=0` for the fastest possible time to first token (~300ms effective). The default of 500ms produces a first partial at ~800ms. See [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio) for details.

**3. Eliminate additive delays in your orchestrator**

In LiveKit with `turn_detection="stt"`, set `min_endpointing_delay=0`. LiveKit's default 0.5s delay is additive on top of AssemblyAI's own endpointing.

**4. Use 16kHz sample rate**

This balances audio quality and bandwidth. Higher sample rates don't improve accuracy.

**5. Align VAD thresholds**

Mismatched VAD thresholds between your local VAD and AssemblyAI create a dead zone that delays interruption. Set both to `0.3`.

**6. Skip unnecessary features**

Only enable `speaker_labels` if you need diarization. Only use `keyterms_prompt` if you have domain-specific terms. Each feature adds marginal processing overhead.

### Latency breakdown

| Stage | Typical latency | Notes |
| ----- | --------------- | ----- |
| Audio to AssemblyAI | ~50ms | Network dependent |
| Speech-to-text | ~200-300ms | Sub-300ms P50 |
| `min_turn_silence` check | 100ms+ | Configurable |
| `max_turn_silence` fallback | 1000ms+ | Only if no terminal punctuation |

## How Does the Message Sequence Work?

Universal-3.5 Pro Streaming sends messages in a specific sequence. Here's what a typical conversation looks like:

**1. Session begins**

```json
{
  "type": "Begin",
  "id": "session-id",
  "expires_at": 1759796682
}
```

**2. Speech detected**

```json
{
  "type": "SpeechStarted",
  "timestamp": 1200,
  "confidence": 0.85
}
```

**3. Early partial** (emitted after 750ms of continuous speech)

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": false,
  "turn_is_formatted": false,
  "transcript": "Yeah my credit card..."
}
```

**4. Silence-based partial** (speaker pauses, no terminal punctuation)

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": false,
  "turn_is_formatted": false,
  "transcript": "Yeah my credit card number is--"
}
```

**5. Final transcript** (terminal punctuation found, or `max_turn_silence` reached)

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": true,
  "turn_is_formatted": true,
  "transcript": "Yeah, my credit card number is 8888-8888-8888-8888.",
  "speaker_label": "A"
}
```

For Universal-3.5 Pro Streaming, `end_of_turn` and `turn_is_formatted` always have the same value. You can reliably use `end_of_turn: true` to detect a formatted, final transcript.

**6. Session termination**

```json
{
  "type": "Termination",
  "audio_duration_seconds": 45.2
}
```

For the complete message reference, see [Message sequence](/streaming/getting-started/transcribe-streaming-audio).

## How Can I Improve Accuracy?

### Keyterms prompting

The single most effective way to improve accuracy on domain-specific terms. Keyterms are especially useful for improving recognition of proper nouns, product names, and technical jargon spoken with accents or in noisy environments. See [How Can I Use Prompting to Improve Accuracy?](#how-can-i-use-prompting-to-improve-accuracy) above.

### Dynamic configuration updates

Update keyterms and prompts mid-session based on conversation context. See [How Do I Update Configuration Mid-Session?](#how-do-i-update-configuration-mid-session) above.

### Tune silence thresholds

If entities are splitting across turns, increase `min_turn_silence` (for punctuation-triggered splits) or `max_turn_silence` (for forced timeout splits). You can do this dynamically mid-session for specific conversation stages like entity dictation.

### Noise handling

Universal-3.5 Pro Streaming handles background noise well out of the box. Avoid adding noise cancellation as a preprocessing step. The artifacts it introduces typically cause more harm than the background noise itself.

For telephony environments with low-quality audio (such as 8 kHz mulaw), you can prompt the model to tag genuinely unclear segments as `[unclear]` rather than forcing a guess. This helps you identify audio segments that no model (or human) can reliably transcribe, and prevents inaccurate guesses from entering your downstream pipeline.

## Scaling and Rate Limits

Universal-3.5 Pro Streaming provides unlimited parallel streams:

- No hard caps on simultaneous connections
- No overage fees for spike traffic
- Automatic scaling from 5 to 50,000+ streams

**Rate limits:**

- Free users: 5 new streams per minute
- Pay-as-you-go: 100 new streams per minute
- When using 70%+ of your limit, capacity automatically increases 10% every 60 seconds

These limits are designed to never interfere with legitimate applications. Your baseline limit is guaranteed and never decreases, so you can scale smoothly without artificial barriers.

---

## Evaluating your voice agent

Benchmark scores are a useful starting point, but they don't tell the full story. To determine which STT model works best for your voice agent in production:

- **Run A/B tests at scale.** Swap models in your agent pipeline, compare core outcome metrics (task completion rate, booking rate, resolution rate), and let real user behavior determine the winner.
- **Optimize for outcomes, not benchmarks.** The question is not which model has the best WER. It is which model drives users to complete their goal most reliably.
- **Simulate real scenarios early.** Before you have real users, manually test the full agent flow under realistic conditions. This surfaces issues that isolated STT benchmarks will miss.

For a complete evaluation framework including accuracy metrics, latency metrics, and ground truth best practices, see the [streaming evaluation guide](/streaming/evaluations/voice-agents).

## Additional Resources

- [Streaming evaluation guide](/streaming/evaluations/voice-agents)
- [Universal-3.5 Pro Streaming overview](/streaming/getting-started/transcribe-streaming-audio)
- [Turn detection guide](/streaming/getting-started/transcribe-streaming-audio)
- [Prompting guide](/streaming/prompting-and-keyterms)
- [Keyterms prompting](/streaming/prompting-and-keyterms)
- [Message sequence](/streaming/getting-started/transcribe-streaming-audio)
- [Streaming diarization and multichannel](/streaming/label-speakers-and-separate-channels)
- [Supported languages](/streaming/getting-started/transcribe-streaming-audio)
- [LiveKit integration](/voice-agents/livekit-universal-3-5-pro)
- [Pipecat integration](/voice-agents/pipecat-universal-3-5-pro)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)

---

<a id="best-practices-for-building-contact-center-applications-docs-contact-center-best-practices-mdx"></a>

# Best Practices for building Contact Center Applications

URL: https://www.assemblyai.com/docs/contact-center-best-practices
Source: docs/contact-center-best-practices.mdx
Navigation: Overview > Use cases & integrations > Use case guides
Description: Complete guide for building contact center applications with AssemblyAI

## Introduction

Building a contact center application requires careful consideration of accuracy, speaker separation, compliance, and scalability. This guide addresses common questions and provides practical solutions for both post-call analytics and real-time agent assist scenarios.

## Why AssemblyAI for contact centers?

AssemblyAI stands out as the premier choice for contact center applications with several key advantages:

### Industry-leading accuracy on telephony audio

- **Universal-3.5 Pro model** delivers best-in-class accuracy on 8kHz telephony audio
- **2.9% speaker diarization error rate** for precise agent vs. customer attribution
- **Multichannel support** for stereo call recordings where agent and customer are on separate channels
- **Keyterms prompt** allows providing call context to improve accuracy of company names, products, and compliance phrases

### Streaming with Universal-3.5 Pro

For real-time agent assist, AssemblyAI's Universal-3.5 Pro Streaming model (`universal-3-5-pro`) offers:

- **Low latency** enables live transcription during calls
- **Format turns** feature provides structured, readable output
- **Dynamic prompting** via `UpdateConfiguration` to update context mid-call
- **Dual-channel streaming** for separate agent and customer audio streams

### End-to-end voice AI platform

Unlike fragmented solutions, AssemblyAI provides a unified API for:

- Transcription with speaker diarization (agent vs. customer)
- Multichannel audio support for stereo call recordings
- PII redaction on both text and audio for HIPAA and PCI compliance
- Post-processing workflows with custom prompting - from call summaries to QA scoring
- Streaming and pre-recorded transcription in a single platform
- Compliance and security built for enterprise workloads (BAA, SOC2, ISO)

## When should I use pre-recorded vs streaming for contact centers?

Understanding when to use pre-recorded versus streaming is critical for contact center workflows.

### Pre-recorded Speech-to-text

**Post-call analytics** - Call already happened, you have the full recording

- **Highest accuracy needed** - Pre-recorded models have the highest accuracy
- **Speaker diarization is critical** - Pre-recorded has 2.9% speaker error rate
- **Multichannel recordings** - Most contact center recordings are stereo with agent and customer on separate channels
- **Compliance workflows** - Full PII redaction with audio de-identification
- **Post-call analytics** - Summarization, sentiment analysis, entity detection, QA scoring
- **Batch processing** - Processing large volumes of call recordings

**Best for:** QA scoring, compliance monitoring, coaching insights, post-call CRM updates, searchable call archives

### Streaming Speech-to-text

**Live calls** - Transcribing as the call happens

You should use streaming when you need to display a live transcript to agents during calls. With Universal-3.5 Pro Streaming, accuracy is closer to pre-recorded, but pre-recorded will always be the most accurate option.

- **Agent assist** - Live transcription visible to agents during calls
- **Real-time coaching** - Prompt agents with suggested responses or compliance reminders
- **Live compliance monitoring** - Detect compliance violations in real-time
- **No recording available** - Processing live audio only

**Best for:** Agent assist, real-time coaching, live compliance monitoring, live call transcription

### Hybrid approach (recommended)

Many contact center platforms use **both**:

1. **Streaming during the call** - Provide live transcription for agent assist and real-time coaching
2. **Pre-recorded after the call** - Generate high-quality transcript with speaker labels, summary, and analytics

**Example workflow:**

- Call begins → Start streaming for live agent assist
- Call ends → Upload recording to pre-recorded API for final transcript with speaker names
- Generate call summary, QA score, and compliance report from pre-recorded transcript
- Push results to CRM (e.g., Salesforce)

## What languages and features for a contact center application?

### Pre-recorded calls (Universal-3.5 Pro)

For post-call analytics, AssemblyAI supports:

**Languages**:

- 99 languages supported
- Automatic Language Detection to route to the most spoken language
- Code Switching to preserve changes in speech between languages

**Core Features**:

- Speaker diarization (agent-customer separation)
- Multichannel audio support - when agent and customer are on separate audio channels, enables perfect speaker separation without diarization
- Automatic formatting, punctuation, and capitalization
- Keyterms prompting for boosting domain-specific terms (up to 1000 terms for Universal-3.5 Pro)
- Natural language prompting (Universal-3.5 Pro) - up to 1,500 words to guide transcription behavior
- Speaker options with configurable min/max expected speakers for call transfers

**Speech Understanding**:

- Summarization for call recaps
- Sentiment analysis for customer satisfaction tracking
- Entity detection for extracting names, account numbers, and products
- Speaker identification to map generic labels to agent and customer names
- Translation between 86 languages

**Guardrails**:

- PII redaction on text and audio for HIPAA and PCI compliance

### Streaming (Universal-3.5 Pro Streaming)

For live call transcription, use **Universal-3.5 Pro Streaming** (`universal-3-5-pro`) for the highest streaming accuracy:

**Core Features**:

- Speaker diarization for identifying agent vs. customer
- Partial and final transcripts for responsive UI
- Format turns for structured, readable output
- Keyterms prompt for company names, products, and compliance phrases
- Dual-channel streaming for separate agent and customer audio

For more details, see the [Universal-3.5 Pro Streaming documentation](/streaming/getting-started/transcribe-streaming-audio).

## How can I get started building a post-call analytics pipeline?

Here's a complete example implementing pre-recorded transcription for contact center call analysis:

```python expandable
import assemblyai as aai
import asyncio
from typing import Dict, List
from assemblyai.types import (
    SpeakerOptions,
    PIIRedactionPolicy,
    PIISubstitutionPolicy,
)

# Configure API key
aai.settings.api_key = "your_api_key_here"

async def transcribe_call(audio_source: str, agent_name: str = None) -> Dict:
    """
    Transcribe a contact center call recording with full analytics

    Args:
        audio_source: Either a local file path or publicly accessible URL
        agent_name: Optional agent name for speaker identification
    """
    # Configure comprehensive call analysis
    config = aai.TranscriptionConfig(
        # Model selection
        speech_models=["universal-3-5-pro", "universal-2"],

        # Speaker diarization
        speaker_labels=True,
        speaker_options=SpeakerOptions(
            min_speakers_expected=2,  # Agent and customer
            max_speakers_expected=5   # Allow for call transfers - safe to keep high
        ),
        multichannel=False,  # Set to True if audio has separate channel per speaker

        # Language detection
        language_detection=True,

        # Boost accuracy of contact center vocabulary
        keyterms_prompt=[
            # Company-specific terms
            "Acme Corp", "Premium Support Plan",

            # Compliance phrases
            "recorded line", "calls are monitored and recorded",

            # Common contact center terms
            "account number", "case number", "ticket number",
            "escalation", "supervisor", "hold time",
        ],

        # Post-call analytics
        summarization=True,
        sentiment_analysis=True,
        entity_detection=True,

        # PII protection for compliance
        redact_pii=True,
        redact_pii_policies=[
            PIIRedactionPolicy.person_name,
            PIIRedactionPolicy.phone_number,
            PIIRedactionPolicy.email_address,
            PIIRedactionPolicy.account_number,
            PIIRedactionPolicy.us_social_security_number,
            PIIRedactionPolicy.credit_card_number,
            PIIRedactionPolicy.credit_card_cvv,
            PIIRedactionPolicy.credit_card_expiration,
            PIIRedactionPolicy.date_of_birth,
        ],
        redact_pii_sub=PIISubstitutionPolicy.hash,
        redact_pii_audio=True,
    )

    # Add speaker identification if agent name is known
    if agent_name:
        config.speech_understanding = {
            "request": {
                "speaker_identification": {
                    "speaker_type": "role",
                    "speakers": [
                        {"role": "Agent", "name": agent_name},
                        {"role": "Customer"}
                    ]
                }
            }
        }

    # Create transcriber
    transcriber = aai.Transcriber()

    try:
        # Submit transcription job
        transcript = await asyncio.to_thread(
            transcriber.transcribe,
            audio_source,
            config=config
        )

        # Check status
        if transcript.status == aai.TranscriptStatus.error:
            raise Exception(f"Transcription failed: {transcript.error}")

        # Process speaker-labeled utterances
        for utterance in transcript.utterances:
            start_time = utterance.start / 1000  # Convert ms to seconds
            end_time = utterance.end / 1000

            print(f"[{start_time:.1f}s - {end_time:.1f}s] {utterance.speaker}:")
            print(f"  {utterance.text}\n")

        return {
            "transcript": transcript,
            "utterances": transcript.utterances,
            "summary": transcript.summary,
            "sentiment": transcript.sentiment_analysis_results,
            "entities": transcript.entities,
            "redacted_audio_url": transcript.redacted_audio_url,
        }

    except Exception as e:
        print(f"Error during transcription: {e}")
        raise

async def main():
    audio_source = "https://your-storage.com/calls/call_recording.mp3"

    result = await transcribe_call(audio_source, agent_name="Sarah Johnson")

    print(f"\nCall duration: {result['transcript'].audio_duration} seconds")
    print(f"Summary: {result['summary']}")

if __name__ == "__main__":
    asyncio.run(main())
```

## How Do I Handle Multichannel Contact Center Audio?

Most contact center recordings are stereo with the agent on one channel and the customer on the other. Multichannel transcription gives you perfect speaker separation without diarization.

### Pre-recorded Multichannel

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    multichannel=True,  # Enable when agent and customer are on separate channels
    speaker_labels=False,  # Disable - channels already separate speakers

    # Still enable analytics
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,

    # PII redaction
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file, config=config)

# Channel 1 = Agent, Channel 2 = Customer (typical layout)
for utterance in transcript.utterances:
    role = "Agent" if utterance.channel == "1" else "Customer"
    print(f"{role}: {utterance.text}")
```

**When to use multichannel:**

- Call recordings from PBX systems with separate agent/customer channels
- Recordings from platforms like Genesys, Twilio, Five9, NICE, or Talkdesk
- Any stereo recording where each channel represents a different speaker

**Benefits:**

- **Perfect speaker separation** - No diarization errors
- **No speaker confusion or overlap issues**
- **Higher accuracy** - Model processes clean single-speaker audio per channel

### Streaming Multichannel

For real-time dual-channel transcription, create separate streaming sessions per channel:

```python expandable
import asyncio
import websockets
import json
from urllib.parse import urlencode

API_KEY = "your_api_key"

class ChannelTranscriber:
    def __init__(self, channel_id: int, role: str):
        self.channel_id = channel_id
        self.role = role
        self.connection_params = {
            "sample_rate": 8000,  # Telephony standard
            "speech_model": "universal-3-5-pro",
            "format_turns": True,
            "encoding": "pcm_mulaw",  # Common telephony encoding
        }

    async def transcribe_channel(self, audio_stream):
        url = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(self.connection_params, doseq=True)}"

        # If using websockets >= 13.0, use additional_headers. For < 13.0, use extra_headers.
        async with websockets.connect(url, additional_headers={"Authorization": API_KEY}) as ws:
            # Send and receive must run concurrently for real-time streaming
            async def send_audio():
                async for audio_chunk in audio_stream:
                    await ws.send(audio_chunk)

            async def receive_transcripts():
                async for message in ws:
                    data = json.loads(message)
                    if data.get("type") == "Turn" and data.get("end_of_turn"):
                        print(f"{self.role}: {data['transcript']}")

            await asyncio.gather(send_audio(), receive_transcripts())

# Create transcriber for each channel
async def transcribe_live_call(agent_audio_stream, customer_audio_stream):
    agent = ChannelTranscriber(0, "Agent")
    customer = ChannelTranscriber(1, "Customer")

    await asyncio.gather(
        agent.transcribe_channel(agent_audio_stream),
        customer.transcribe_channel(customer_audio_stream),
    )
```

See our [multichannel streaming guide](/streaming/label-speakers-and-separate-channels#multichannel-streaming-audio) for complete implementation details.

## How Can I Build a Real-Time Agent Assist?

Here's a complete example for real-time streaming transcription optimized for contact center agent assist:

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "your_api_key"

# Contact center keyterms
KEYTERMS = [
    # Company and product terms
    "Acme Corp",
    "Premium Support Plan",
    "Enterprise License",

    # Compliance phrases
    "recorded line",
    "calls are monitored",

    # Common contact center vocabulary
    "account number",
    "case number",
    "escalation",
    "supervisor",
]

# CONTACT CENTER CONFIGURATION
CONNECTION_PARAMS = {
    "sample_rate": 8000,  # Telephony standard (8kHz)
    "speech_model": "universal-3-5-pro",  # Universal-3.5 Pro Streaming for highest accuracy
    "format_turns": True,

    # Contact center turn detection
    # universal-3-5-pro defaults: min_turn_silence=100ms, max_turn_silence=1000ms
    "min_turn_silence": 400,  # Longer than default for natural call pauses
    "max_turn_silence": 1500,  # Longer for customers explaining issues

    # Keyterms for accuracy
    "keyterms_prompt": KEYTERMS,
}

API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS, doseq=True)}"

# Audio Configuration
FRAMES_PER_BUFFER = 400  # 50ms of audio at 8kHz
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
transcript_buffer = []

def on_open(ws):
    print("=" * 80)
    print(f"[{datetime.now().strftime('%H:%M:%S')}] Agent assist transcription started")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")
    print(f"Keyterms configured: {', '.join(KEYTERMS[:5])}...")
    print("=" * 80)

    def stream_audio():
        global stream
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                if not stop_event.is_set():
                    print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        if msg_type == "Begin":
            session_id = data.get("id", "N/A")
            print(f"[SESSION] Started - ID: {session_id}\n")

        elif msg_type == "Turn":
            end_of_turn = data.get("end_of_turn", False)
            transcript = data.get("transcript", "")
            turn_order = data.get("turn_order", 0)

            # Show partials for responsive agent UI
            if not end_of_turn and transcript:
                print(f"\r[LIVE] {transcript}", end="", flush=True)

            # Use formatted finals for agent display
            if end_of_turn and transcript:
                timestamp = datetime.now().strftime('%H:%M:%S')
                print(f"\n[{timestamp}] {transcript}")

                # Detect compliance keywords
                transcript_lower = transcript.lower()
                if any(term in transcript_lower for term in ["cancel", "refund", "complaint", "supervisor"]):
                    print("           ** ESCALATION KEYWORD DETECTED **")

                transcript_buffer.append({
                    "timestamp": timestamp,
                    "text": transcript,
                    "turn_order": turn_order,
                    "type": "final"
                })
                print()

        elif msg_type == "Termination":
            audio_duration = data.get("audio_duration_seconds", 0)
            print(f"\n[SESSION] Terminated - Duration: {audio_duration}s")

        elif msg_type == "Error":
            error_msg = data.get("error", "Unknown error")
            print(f"\n[ERROR] {error_msg}")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    print(f"\n[WEBSOCKET ERROR] {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    print(f"\n[WEBSOCKET] Disconnected - Status: {close_status_code}, Message: {close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def run():
    global audio, stream, ws_app

    audio = pyaudio.PyAudio()

    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
    except Exception as e:
        print(f"Error opening audio stream: {e}")
        if audio:
            audio.terminate()
        return

    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\n\nCtrl+C received. Stopping transcription...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## How Should I Handle Pre-recorded Transcription in Production?

### Webhook Callbacks (Recommended)

For high-volume contact center workloads, use webhooks instead of polling:

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    webhook_url="https://your-app.com/webhooks/assemblyai",
    webhook_auth_header_name="X-Webhook-Secret",
    webhook_auth_header_value="your_secret_here",
    speaker_labels=True,
    multichannel=True,
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
)

# Submit job and return immediately (non-blocking)
transcript = transcriber.submit(audio_url, config=config)
print(f"Job submitted: {transcript.id}")
# Your app continues processing other calls
```

**Webhook handler example:**

```python expandable
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhooks/assemblyai", methods=["POST"])
def assemblyai_webhook():
    if request.headers.get("X-Webhook-Secret") != "your_secret_here":
        return jsonify({"error": "Unauthorized"}), 401

    import requests as http_requests

    data = request.json
    transcript_id = data["transcript_id"]
    status = data["status"]

    if status == "completed":
        # Fetch the full transcript (webhook only sends transcript_id and status)
        transcript = http_requests.get(
            f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
            headers={"authorization": "your_api_key"}
        ).json()
        process_completed_call(transcript)
    elif status == "error":
        log_transcription_error(transcript_id)

    return jsonify({"received": True}), 200

def process_completed_call(transcript):
    """Process completed call transcript and push to CRM"""
    utterances = transcript["utterances"]
    summary = transcript["summary"]

    # Store in database
    save_to_database(transcript)

    # Push summary to CRM
    push_to_crm(transcript["id"], summary)

    # Run QA scoring
    qa_score = score_call_quality(utterances)
    save_qa_score(transcript["id"], qa_score)
```

### Scaling Considerations

- **HTTP rate limit:** 20,000 requests per 5-minute window, counted across submissions (POST) and polling (GET) combined
- **Exceeding the limit:** returns a `403` response
- **Parallel transcriptions (rate limit):** 200+ for paid accounts (queued beyond that)
- **Ramp up gradually:** start at 10-50 parallel requests, double incrementally
- **Avoid the rate limit:** use [webhooks](/pre-recorded-audio/webhooks) or jittered, widened polling — see [Polling without exceeding the rate limit](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale#polling-without-exceeding-the-rate-limit)
- **Contact Sales** before large-scale rollouts

## How Do I Handle PII and Compliance?

PII redaction is critical for contact center compliance (HIPAA, PCI-DSS, GDPR, CCPA).

### Recommended PII Configuration

```python expandable
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        # Customer identity
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.date_of_birth,
        PIIRedactionPolicy.us_social_security_number,

        # Contact information
        PIIRedactionPolicy.phone_number,
        PIIRedactionPolicy.email_address,
        PIIRedactionPolicy.location,

        # Financial information (PCI-DSS)
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.credit_card_cvv,
        PIIRedactionPolicy.credit_card_expiration,
        PIIRedactionPolicy.account_number,
        PIIRedactionPolicy.banking_information,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,  # Stable hash tokens
    redact_pii_audio=True,  # Create de-identified audio file
)
```

**Why hash substitution?**

- Stable across the file (same value = same token)
- Maintains sentence structure for downstream LLM processing
- Prevents reconstruction of original data

### HIPAA Compliance

- AssemblyAI provides a **Business Associate Agreement (BAA)** at no cost
- Paid customers can review and sign our standard online BAA self-serve from the [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the dashboard (Owner or Admin role required, upgraded account required)
- Use PII redaction with audio de-identification for full compliance

## How Do I Improve the Accuracy of My Contact Center Transcription?

### Prompting Best Practices

The most impactful lever for contact center accuracy is **prompting**. Use a structured prompt with a `Context:` field:

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],

    # Natural language prompt for transcription guidance
    prompt=(
        "Transcribe this audio with perfect punctuation and formatting. "
        "Preserve linguistic speech patterns including disfluencies, filler words, "
        "hesitations, repetitions, stutters, false starts, and colloquialisms. "
        "Transcribe in the original language mix (code-switching), preserving the "
        "words in the language they are spoken. Output plain transcript text only. "
        "Use a new line when the voice changes; each line contains only one "
        "person's words.\n\n"
        "Context: Acme Corp customer service call, recorded line, "
        "Agent: Sarah Johnson, calls are monitored and recorded"
    ),

    # Keyterms for proper nouns and domain vocabulary
    keyterms_prompt=[
        "Acme Corp",
        "Sarah Johnson",
        "Premium Support Plan",
        "Enterprise License",
        "recorded line",
        "calls are monitored and recorded",
    ],

    speaker_labels=True,
)
```

**Tips for effective prompting:**

- Use **positive instructions** ("transcribe verbatim") not negative ("do NOT summarize")
- **Start with fewer instructions, add one at a time** — every added instruction risks conflicting with another. Treat the older "3–6 instructions" guidance as an upper bound, not a target.
- **Layer instructions one by one** and test each against your call recordings to measure impact
- Dynamize the `Context:` line per call with known info: company name, agent name, compliance phrases
- Use **keyterms** for proper nouns and domain vocabulary (company names, product names, agent names)

### Using Keyterms for Pre-recorded Transcription

```python expandable
# Build keyterms dynamically per call
call_keyterms = [
    # Company terms (static)
    "Acme Corp",
    "Premium Support Plan",

    # Agent name (from routing system)
    agent_name,

    # Customer name (from CRM lookup)
    customer_name,

    # Account-specific terms
    "account ending in 4532",
]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    keyterms_prompt=call_keyterms,
    speaker_labels=True,
)
```

### Using Keyterms for Streaming

```python
# Streaming with contact center context
keyterms = [
    "Acme Corp",
    "Premium Support Plan",
    "Sarah Johnson",
    "recorded line",
]

CONNECTION_PARAMS = {
    "sample_rate": 8000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "encoding": "pcm_mulaw",
    "keyterms_prompt": keyterms,
}
```

## What Workflows Can I Build for My Contact Center Application?

Use these features to transform raw call transcripts into actionable insights.

### Summarization

`summarization: true`

**What it does:** Generates an abstractive recap of the call.
**Output:** `summary` string (bullets/paragraph format).
**Great for:** Post-call CRM updates, call recaps, supervisor review.

```python
config = aai.TranscriptionConfig(
    summarization=True,
    summary_type="bullets",  # or "bullets_verbose", "gist", "headline", "paragraph"
    summary_model="informative",  # or "conversational"
)
```

### Sentiment Analysis

`sentiment_analysis: true`

**What it does:** Scores per-utterance sentiment (positive / neutral / negative).
**Output:** Array of `{ text, sentiment, confidence, start, end }`.
**Great for:** Customer satisfaction tracking, escalation detection, QA scoring.

```python
# Analyze customer sentiment across a call
negative_count = 0
for result in transcript.sentiment_analysis_results:
    if result.sentiment == "NEGATIVE":
        negative_count += 1
        print(f"Negative at {result.start / 1000:.1f}s: {result.text}")

# Flag calls with high negative sentiment
if negative_count > 3:
    flag_for_supervisor_review(transcript.id)
```

### Entity Detection

`entity_detection: true`

**What it does:** Extracts named entities (people, organizations, locations, products, etc.).
**Output:** Array of `{ entity_type, text, start, end }`.
**Great for:** CRM enrichment, auto-tagging topics, competitor tracking.

```python
# Extract key entities from a call
organizations = [e.text for e in transcript.entities if e.entity_type == "organization"]
print(f"Companies mentioned: {', '.join(organizations)}")
```

### Speaker Identification

Map generic speaker labels to agent and customer names:

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    speaker_labels=True,
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "speakers": [
                    {"role": "Agent", "name": "Sarah Johnson", "description": "Customer service representative"},
                    {"role": "Customer"}
                ]
            }
        }
    }
)
```

### Translation

Translate call transcripts for international teams:

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,
    speech_understanding={
        "request": {
            "translation": {
                "target_languages": ["en"],  # Translate to English
                "match_original_utterance": True,  # Per-utterance translations
                "formal": True
            }
        }
    }
)
```

### Redact PII Text and Audio

```python
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
    redact_pii_audio=True,  # Generate de-identified audio
)

# After transcription
print(transcript.text)  # PII redacted in text
print(transcript.redacted_audio_url)  # PII bleeped in audio
```

## How Do I Process the Response from the API?

### Processing Pre-recorded Responses

```python expandable
def process_call_transcript(transcript):
    """
    Extract and process all relevant data from a pre-recorded call transcript
    """
    call_data = {
        "id": transcript.id,
        "duration": transcript.audio_duration,  # Already in seconds
        "confidence": transcript.confidence,
        "full_text": transcript.text,
    }

    # Process speaker utterances
    speakers = {}
    for utterance in transcript.utterances:
        speaker = utterance.speaker

        if speaker not in speakers:
            speakers[speaker] = {
                "utterances": [],
                "total_speaking_time": 0,
                "word_count": 0
            }

        speakers[speaker]["utterances"].append({
            "text": utterance.text,
            "start": utterance.start,
            "end": utterance.end,
        })

        speakers[speaker]["total_speaking_time"] += (utterance.end - utterance.start) / 1000
        speakers[speaker]["word_count"] += len(utterance.text.split())

    call_data["speakers"] = speakers

    # Extract summary
    if transcript.summary:
        call_data["summary"] = transcript.summary

    # Analyze sentiment
    if transcript.sentiment_analysis_results:
        sentiments = [r.sentiment for r in transcript.sentiment_analysis_results]
        call_data["sentiment_breakdown"] = {
            "positive": sentiments.count("POSITIVE"),
            "neutral": sentiments.count("NEUTRAL"),
            "negative": sentiments.count("NEGATIVE"),
        }

    # Calculate statistics
    total_duration = transcript.audio_duration
    call_data["statistics"] = {
        "total_speakers": len(speakers),
        "total_words": sum(s["word_count"] for s in speakers.values()),
        "speaking_distribution": {
            speaker: {
                "percentage": (data["total_speaking_time"] / total_duration) * 100,
                "minutes": data["total_speaking_time"] / 60,
            }
            for speaker, data in speakers.items()
        },
    }

    return call_data

result = process_call_transcript(transcript)
print(f"Call had {result['statistics']['total_speakers']} speakers")
print(f"Sentiment: {result.get('sentiment_breakdown', {})}")
```

## Additional Resources

- [Universal-3.5 Pro Pre-recorded Documentation](/pre-recorded-audio)
- [Universal-3.5 Pro Streaming Documentation](/streaming/getting-started/transcribe-streaming-audio)
- [Speaker Diarization Guide](/pre-recorded-audio/label-speakers)
- [Multichannel Streaming Guide](/streaming/label-speakers-and-separate-channels)
- [Speaker Identification Guide](/speech-understanding/speaker-identification)
- [Translation Guide](/speech-understanding/translation)
- [PII Redaction Guide](/guardrails/redact-pii-from-transcripts)
- [Getting Started Guide](/pre-recorded-audio/getting-started/transcribe-an-audio-file)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)
- [Support](https://www.assemblyai.com/contact/support)

---

<a id="integrations-docs-integrations-mdx"></a>

# Integrations

URL: https://www.assemblyai.com/docs/integrations
Source: docs/integrations.mdx
Navigation: Overview > Use cases & integrations > Integrations
Description: Integrations documentation.

AssemblyAI seamlessly integrates with a variety of tools and platforms to enhance your workflow. Whether you're looking for no-code solutions or developer tools, we've got you covered.

## Voice agent orchestrators

  
        <path d="M137.857 92.4673H92.0903V138.234H137.857V92.4673Z" fill="currentColor" />
        <path d="M183.624 46.7178H137.857V92.4849H183.624V46.7178Z" fill="currentColor" />
        <path d="M183.624 138.234H137.857V184.001H183.624V138.234Z" fill="currentColor" />
        <path d="M229.374 0.950684H183.607V46.7178H229.374V0.950684Z" fill="currentColor" />
        <path d="M229.374 183.984H183.607V229.751H229.374V183.984Z" fill="currentColor" />
        <path d="M46.3409 183.984V138.234V92.4671V46.7178V0.950684H0.57373V46.7178V92.4671V138.234V183.984V229.751H46.3409H92.0902H137.857V183.984H92.0902H46.3409Z" fill="currentColor" />
      </svg>
    }
    href="/voice-agents/livekit-universal-3-5-pro"
  >
    Use AssemblyAI with Livekit's voice agent orchestrator.
  
  
        <path d="M45.7718 70.7701C50.4477 69.0096 55.7252 70.3307 59.0204 74.0864L101.936 123.001H230.064L272.98 74.0864C276.275 70.3307 281.552 69.0096 286.228 70.7701C290.904 72.5306 294 77.0042 294 82.0005V190H332V214H270V113.873L244.52 142.915C242.242 145.512 238.955 147.001 235.5 147.001H96.5C93.0452 147.001 89.7581 145.512 87.4796 142.915L62 113.873V214H0V190H38V82.0005C38 77.0042 41.0958 72.5306 45.7718 70.7701Z" fill="currentColor" />
        <path d="M270 238.001H332V262.001H270V238.001Z" fill="currentColor" />
        <path d="M0 238.001H62V262.001H0V238.001Z" fill="currentColor" />
        <path d="M128 198.001C128 206.837 120.837 214.001 112 214.001C103.163 214.001 96 206.837 96 198.001C96 189.164 103.163 182.001 112 182.001C120.837 182.001 128 189.164 128 198.001Z" fill="currentColor" />
        <path d="M236 198.001C236 206.837 228.837 214.001 220 214.001C211.163 214.001 204 206.837 204 198.001C204 189.164 211.163 182.001 220 182.001C228.837 182.001 236 189.164 236 198.001Z" fill="currentColor" />
      </svg>
    }
    href="/voice-agents/pipecat-universal-3-5-pro"
  >
    Use AssemblyAI with Pipecat's voice agent orchestrator.
  

## No-code tools

  
    Connect AssemblyAI with 5000+ apps using Zapier's automation platform.
  
  
    Build complex automation scenarios with Make's visual workflow builder.
  
  
    Automate workflows with n8n's automation platform and AssemblyAI.
  
  
    Test and explore AssemblyAI's APIs using Postman collections.
  

## Meeting transcriber tools

  
    Setup a Zoom meeting bot with AssemblyAI and Recall.ai.
  
  
    Integrate AssemblyAI with Zoom RTMS for meeting transcription.
  

## Telephony tools

  
    Build voice agents with Telnyx and AssemblyAI using Pipecat or LiveKit.
  
  
    Process audio from Twilio calls and voice messages.
  
  
    Setup a transcription pipeline for Amazon Connect recordings with
    AssemblyAI.
  
  
    Setup a transcription pipeline for Genesys Cloud to AssemblyAI.
  

## Community-maintained tools

  
    Integrate AssemblyAI with LangChain for advanced language model
    applications.
  
  
        <path d="M37.5274 0L75.0548 65H0L37.5274 0Z" fill="currentColor" />
      </svg>
    }
    href="/integrations/vercel-ai-sdk"
  >
    Transcribe audio with the AI SDK's unified API using the @ai-sdk/assemblyai
    provider.
  
  
    Create automated workflows with Microsoft Power Automate to process audio
    and extract insights.
  
  
    Use AssemblyAI with Microsoft's Semantic Kernel framework.
  
  
    Create automated workflows with this open-source alternative to Zapier.
  
  
    Build production-ready NLP applications with Haystack integration.
  
  
    Transcribe audio with Cloudflare Workers and AssemblyAI.
  
  
    Use AssemblyAI with Relay.app's workflow automation platform.
  
  
    Add speech-to-text capabilities to your Bubble.io no-code applications.
  
  
    Build event-driven workflows with Pipedream's integration platform.
  
  
    Add speech-to-text transcription to your Drupal site through the AI module.
  

Looking to integrate with a different tool? Check out our [API Reference](/api-reference) or [contact our team](https://www.assemblyai.com/contact) for custom integration support.

---

<a id="universal-3-5-pro-realtime-on-livekit-docs-voice-agents-livekit-universal-3-5-pro-mdx"></a>

# Universal 3.5 Pro Realtime on LiveKit

URL: https://www.assemblyai.com/docs/voice-agents/livekit-universal-3-5-pro
Source: docs/voice-agents/livekit-universal-3-5-pro.mdx
Navigation: Overview > Use cases & integrations > Integrations > Voice agent orchestrators
Description: Integrate AssemblyAI's Universal 3.5 Pro Realtime speech-to-text model into a LiveKit voice agent

## Overview

This guide covers integrating AssemblyAI's **Universal 3.5 Pro Realtime** speech-to-text model into a LiveKit voice agent using the [Agents framework](https://docs.livekit.io/agents/).

  **Universal 3.5 Pro Realtime is our flagship next-generation streaming model for voice agents** — multilingual and promptable, with [conversation context](#conversation-context) and [voice focus](#voice-focus).

  Available on **`livekit-agents` 1.6+** — set `model="universal-3-5-pro"`.

AssemblyAI provides the speech-to-text and turn detection in your LiveKit pipeline:

```mermaid
flowchart LR
  U["User audio"] --> STT["AssemblyAI STT<br/>Universal 3.5 Pro Realtime"]
  STT --> TD["Turn detection"]
  TD --> LLM["LLM"]
  LLM --> TTS["TTS"]
  TTS --> U
```

Once you have an agent running, tune it for what matters most to your use case:

  
    Decide when the user is done speaking — modes, defaults, and entity tuning.
  
  
    Shorten the gap between the user finishing and the agent replying.
  
  
    Prompting, key terms, conversation context, and noise handling.
  
  
    Natural barge-in without false triggers from backchannels.
  

  For a standalone voice agent without LiveKit or an external LLM, see the [AssemblyAI Voice Agent API](/voice-agents/voice-agent-api), which handles STT, LLM routing, and TTS in a single WebSocket connection.

## Quickstart

Get a working, talking agent in a few minutes, then optimize from there.

Install the plugin and supporting packages (silero, codecs, dotenv) from PyPI:

```bash
pip install "livekit-agents[assemblyai,silero,codecs]~=1.6" \
    python-dotenv
```

If you plan to use LiveKit turn detection with `MultilingualModel()`, also install the turn detector plugin:

```bash
pip install "livekit-plugins-turn-detector~=1.0"
```

  Install the latest `livekit-agents` from PyPI. **Universal 3.5 Pro Realtime**, `agent_context`, and **Voice Focus** require a recent **1.6** release. Automatic conversation-context forwarding and `language_codes` require **1.6.6+**. Older versions won't recognize the `universal-3-5-pro` model and return a validation error.

Set your API keys in a `.env` file:

```env
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
ASSEMBLYAI_API_KEY=your_assemblyai_key
# Add API keys for your chosen LLM and TTS providers
```

  You can obtain an AssemblyAI API key by [signing up for a free
  account](https://www.assemblyai.com/dashboard/signup) and navigating to the [API
  Keys tab](https://www.assemblyai.com/dashboard/home) of the dashboard.

The following example uses `turn_detection="stt"` (recommended). Pay close attention to the comments for using with `MultilingualModel()`.

```python expandable
from dotenv import load_dotenv
from livekit import agents
from livekit.agents import AgentSession, Agent, TurnHandlingOptions
from livekit.plugins import (
    assemblyai,
    silero,
)
# For MultilingualModel, uncomment the following:
# from livekit.plugins.turn_detector.multilingual import MultilingualModel

load_dotenv()

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant.")

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    session = AgentSession(
        stt=assemblyai.STT(
            model="universal-3-5-pro",
            min_turn_silence=100,
            max_turn_silence=1000,  # When turn_detection="stt", override plugin default of 100.
            # If using MultilingualModel(), the plugin defaults (min: 100, max: 100) work well. Omit min_turn_silence and max_turn_silence above if preferred.
            vad_threshold=0.3,  # Match Silero's activation_threshold
            # continuous_partials is True by default in the LiveKit plugin — steady ~3s partials during long turns.
            # interruption_delay=0,  # Optional: faster first partial (~300ms effective). Default: 500 (~800ms effective).
        ),
        # llm=your_llm_plugin(),  # Add your LLM provider here
        # tts=your_tts_plugin(),  # Add your TTS provider here
        vad=silero.VAD.load(
            activation_threshold=0.3,  # Match AssemblyAI's internal VAD threshold
        ),
        turn_handling=TurnHandlingOptions(
            turn_detection="stt",
            # To use LiveKit's turn detection instead, replace the line above with:
            # turn_detection=MultilingualModel(),
            endpointing={"min_delay": 0},  # Avoid additive delay in STT mode
            # If using MultilingualModel(), set these instead:
            # endpointing={"min_delay": 0.5, "max_delay": 3.0},
        ),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
    )

    await session.generate_reply(
        instructions="Greet the user and offer your assistance."
    )

if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))
```

For a complete voice agent, you will also need to install LLM and TTS plugins for your chosen providers. See the [LiveKit plugins documentation](https://docs.livekit.io/agents/plugins/) for available options.

Start your agent in development mode:

```bash
python your_agent_file.py dev
```

Then test it in the LiveKit Playground:

1. Go to [agents-playground.livekit.io](https://agents-playground.livekit.io)
2. Connect to your LiveKit Cloud project (same credentials as your `.env`)
3. Click **Connect**: a room will be created, your agent will join, and you can start talking

## Parameters reference

### **Universal 3.5 Pro Realtime** parameters

These are the key parameters to tune for LiveKit when using **Universal 3.5 Pro Realtime**:

  The streaming model. Defaults to `"universal-3-5-pro"`, our recommended
  flagship model.

  Accuracy/latency preset: `"min_latency"`, `"balanced"`, or `"max_accuracy"`. Sets
  the defaults for mode-dependent fields (`min_turn_silence`, `vad_threshold`,
  `interruption_delay`, `continuous_partials`, `previous_context_n_turns`); any value
  you set explicitly still takes precedence. Leave unset for the server's default
  preset. Connect-time only — cannot be changed with `update_options`. Universal-3.5 Pro
  only. See [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency).

  List of terms to boost recognition for. Applied automatically alongside any
  contextual prompt.

  Contextual prompt — a natural-language description of what the audio is
  about (domain, scenario, or full details). Transcription behavior is built
  in and optimized automatically.

  Your agent's most recent spoken reply, up to 1750 characters, used as context
  for transcribing the next user turn. The 1750-character cap is enforced
  client-side — explicit values over the cap raise `ValueError` (both at
  construction and via `update_options`); only automatic forwarding truncates
  silently. Can be set at construction time and updated mid-stream with
  `update_options`. Universal-3.5 Pro only. See
  [Conversation context](#conversation-context).

  Steer transcription toward one or more expected languages instead of
  auto-detecting across all supported languages. Accepts a single code
  (`"es"`) or a list (`["en", "es"]`); entries are normalized to ISO 639-1
  (`"en-US"`, `"english"` → `"en"`). Up to 10 codes. Leave unset for the
  default multilingual behavior. Universal-3.5 Pro only. Can be updated
  mid-session via `update_options`; pass an empty list to clear steering.
  Requires `livekit-agents` **1.6.6+**.

  How many prior conversation entries are carried forward automatically. Range
  `0`–`100`; `0` disables carryover. Leave unset for the server default (5).
  Connect-time only — cannot be changed with `update_options`. Universal-3.5 Pro
  only.

  Milliseconds of silence before a speculative end-of-turn check. When the check
  fires, the model looks for terminal punctuation to decide whether the turn has
  ended.

  Maximum milliseconds of silence before the turn is forced to end, regardless
  of punctuation. **The LiveKit plugin defaults to `100`.** Set to `1000` when
  using `turn_detection="stt"`.

  AssemblyAI's internal Silero VAD threshold. **Universal 3.5 Pro Realtime**
  defaults to `0.3`, unlike Universal-Streaming's `0.4`. Align with LiveKit's
  Silero `activation_threshold` for consistent behavior.

  Server-side noise suppression that isolates the primary speaker. `"near-field"`
  for close-talking mics, `"far-field"` for distant capture. Connect-time only.
  Universal-3.5 Pro only. See [Voice focus](#voice-focus).

  How aggressively `voice_focus` suppresses background audio. `0.0`–`1.0`; higher
  is more aggressive. Connect-time only. Universal-3.5 Pro only.

  Whether to emit additional partial transcripts during long turns at a steady
  ~3 second cadence. When enabled (default on both the API and the LiveKit
  plugin), additional partials covering the full turn transcript are emitted
  approximately every 3 seconds while speech continues. When disabled, only one
  early partial is emitted near turn start. The first partial (at 750ms) is
  unaffected. Useful when downstream consumers (LLMs, UI, eager inference) need
  frequent updates during long, uninterrupted turns. See
  [Continuous partials](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  How soon the first partial transcript is emitted during a turn, in
  milliseconds. Range: `0`–`1000`. Lower values produce faster time to first
  token (TTFT) for barge-in and speculative inference; higher values produce
  more confident first partials. The server adds a minimum of 300ms on top of
  the configured value (`interruption_delay: 0` → ~300ms effective,
  `interruption_delay: 500` → ~800ms effective). See
  [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  **Universal 3.5 Pro Realtime** code-switches natively between supported
  languages. This parameter controls whether `language_code` and
  `language_confidence` are included in turn messages. Defaults to `true` in the
  LiveKit plugin, but `false` when using the API directly.

### General STT parameters

These parameters apply to all AssemblyAI streaming models and can remain the same between models:

  The sample rate of the audio stream.

  The encoding of the audio stream. Allowed values: `pcm_s16le`, `pcm_mulaw`.

### Legacy parameters

These parameters apply to the `universal-streaming-english` and `universal-streaming-multilingual` AssemblyAI streaming models, but **do not affect Universal 3.5 Pro Realtime**:

  Confidence threshold for end-of-turn detection. **Universal 3.5 Pro Realtime**
  uses punctuation-based turn detection instead.

  Whether to return formatted final transcripts. **Universal 3.5 Pro Realtime**
  always returns formatted transcripts, so this parameter no longer applies.

## Turn detection

In LiveKit, how your agent detects the end of a user's turn is controlled by the [`turn_detection`](https://docs.livekit.io/reference/agents/turn-handling-options/#turn_detection) parameter inside [`TurnHandlingOptions`](https://docs.livekit.io/reference/agents/turn-handling-options/), which is passed to `AgentSession` via the `turn_handling` argument.

**Universal 3.5 Pro Realtime** uses a **punctuation-based turn detection system**, which checks for terminal punctuation (`.` `?` `!`) after periods of silence rather than using a confidence score.

This means the `min_turn_silence` and `max_turn_silence` parameters you pass to AssemblyAI directly control when transcripts are emitted and when turns end. For more details on how this works, see [Configuring turn detection](/streaming/getting-started/transcribe-streaming-audio).

  When not explicitly provided, the default endpointing parameters for **Universal 3.5 Pro Realtime** differ on **LiveKit** versus using **AssemblyAI's API directly**:
  - LiveKit AssemblyAI plugin defaults:
    - `min_turn_silence=100`
    - `max_turn_silence=100`
  - AssemblyAI API defaults:
    - `min_turn_silence=100`
    - `max_turn_silence=1000`

  However, you can always override these by passing your own preferred values explicitly. Misconfiguring these parameters is the most common cause of poor performance — read the recommended values per mode below.

### Default parameter differences

**Universal 3.5 Pro Realtime's** endpointing is controlled by two AssemblyAI API parameters, `min_turn_silence` and `max_turn_silence`, that you pass to the STT plugin. These are separate from LiveKit's `endpointing.min_delay` and `endpointing.max_delay` (set inside `TurnHandlingOptions`).

| Parameter             | AssemblyAI API default | LiveKit plugin default | Description                                                                                                                                                         |
| --------------------- | ---------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `min_turn_silence`    | `100` ms               | `100` ms               | Silence before a speculative end-of-turn check. If terminal punctuation (`.` `?` `!`) is found, the turn ends. If not, a partial is emitted and the turn continues. |
| `max_turn_silence`    | **`1000`** ms          | **`100`** ms           | Maximum silence before forcing the turn to end, regardless of punctuation.                                                                                          |

The LiveKit plugin defaults are optimized for third-party turn detection models, where you want transcripts handed off as fast as possible. When using `turn_detection="stt"`, you should explicitly set `max_turn_silence=1000` if you'd like to mimic the behavior of streaming directly to the API without LiveKit.

  **Tuning endpointing parameters**

  **These are the default values used when no parameters are explicitly provided.** You will likely need to experiment with different values depending on your use case:

- **Increase `min_turn_silence`**: when brief pauses cause the speculative EOT check to fire too early, ending turns on terminal punctuation before the user has finished speaking.
- **Increase `max_turn_silence`**: when the forced turn end is cutting off users mid-thought or splitting entities like phone numbers across turns, a higher value lets the model wait longer before forcing the turn to end when the model is unsure.

See the [Entity splitting tradeoff](#entity-splitting-tradeoff) section for examples.

### STT-based Turn Detection (recommended)

With `turn_detection="stt"`, AssemblyAI's built-in punctuation-based turn detection determines when the user has finished speaking. AssemblyAI's `end_of_turn` signals are then used directly by LiveKit to commit the turn.

In this mode, we recommend explicitly setting `min_turn_silence=100` and `max_turn_silence=1000`. These are AssemblyAI's API defaults and provide a good balance of responsiveness and accuracy.

**The LiveKit plugin defaults to `min_turn_silence=100` and `max_turn_silence=100`, which might be too aggressive for STT-based turn detection.**

**Recommended starting parameters** (set on `assemblyai.STT()`, not on `AgentSession`):

| Parameter          | Default   | Description                                                          |
| ------------------ | --------- | -------------------------------------------------------------------- |
| `min_turn_silence` | `100` ms  | Silence duration before a speculative end-of-turn (EOT) check fires. |
| `max_turn_silence` | `1000` ms | Maximum silence before a turn is forced to end.                      |

**How it works:**

1. User speaks → audio streams to AssemblyAI
2. User pauses for `100ms` → AssemblyAI checks for terminal punctuation
3. If terminal punctuation (`.` `?` `!`) → turn ends immediately
4. If no terminal punctuation → partial emitted, turn continues waiting
5. If silence reaches `1000ms` → turn is forced to end regardless of punctuation

  **Endpointing `min_delay` is additive in STT mode**

  LiveKit's endpointing `min_delay` (default **0.5 seconds**) is applied **on top of** AssemblyAI's own endpointing. In STT mode, this delay starts *after* the STT end-of-speech signal, meaning it adds up to `500ms` of extra latency by default.

**Set `endpointing={"min_delay": 0}`** inside `TurnHandlingOptions` to avoid this. AssemblyAI's own endpointing parameters (`min_turn_silence` and `max_turn_silence`) already control the timing, so an additional delay on the LiveKit side is unnecessary latency. See [Latency](#latency) for the full picture.

```python
from livekit.agents import AgentSession, TurnHandlingOptions

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
        endpointing={"min_delay": 0},  # Avoid additive delay in STT mode
    ),
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,   # Silence (ms) before a speculative end-of-turn check
        max_turn_silence=1000,  # Max silence (ms) before forcing the turn to end
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,
    ),
)
```

### LiveKit turn detection (with `MultilingualModel()`)

As a third-party turn detection model, LiveKit's [turn detector](https://docs.livekit.io/agents/logic/turns/turn-detector/) runs on top of STT output to make turn decisions. AssemblyAI's role is then just to provide transcripts as quickly as possible, while the turn detection model decides when the user is actually done speaking.

Use `MultilingualModel()` rather than `EnglishModel()`, as **Universal 3.5 Pro Realtime** supports 18 languages. `MultilingualModel()` covers support for all of these languages.

The LiveKit plugin defaults of `min_turn_silence=100` and `max_turn_silence=100` work well here, as **`max_turn_silence` is brought down to match `min_turn_silence` so that transcripts are handed off to the turn detection model as fast as possible.**

**`MultilingualModel` parameters** (set inside `TurnHandlingOptions` → `endpointing`, not on the STT plugin):

| Parameter               | Default | Description                                                                                                                |
| ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
| `endpointing.min_delay` | `0.5` s | Time to wait before committing a turn when the model predicts a likely boundary.                                           |
| `endpointing.max_delay` | `3.0` s | Maximum time to wait when the model predicts the user will continue speaking. Has no effect without a turn detector model. |

**How it works:**

1. User speaks → audio streams to AssemblyAI
2. User pauses for `100ms` → AssemblyAI emits transcript (final and partial are the same) immediately
3. LiveKit's `MultilingualModel()` evaluates the transcript in conversational context
4. If the model predicts a likely turn boundary → waits `min_delay` (`0.5s`) then commits the turn
5. If the model predicts the user will continue → waits up to `max_delay` (`3.0s`) for more speech

```python
from livekit.agents import AgentSession, TurnHandlingOptions
from livekit.plugins.turn_detector.multilingual import MultilingualModel

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection=MultilingualModel(),
        endpointing={
            "min_delay": 0.5,  # Time (s) to wait before committing a turn when the model is confident
            "max_delay": 3.0,  # Max time (s) to wait when the model is not confident
        },
    ),
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,
    ),
)
```

### Other turn detection modes

- **`vad`**:

  - Detect end of turn from speech and silence data alone using [Silero VAD](https://docs.livekit.io/agents/logic/turns/vad/).
  - Turn boundaries are determined purely by voice activity without semantic context.
  - AssemblyAI's turn detection parameters still control when transcripts are emitted, but it is recommended to leave them at the plugin defaults (`min_turn_silence=100`, `max_turn_silence=100`) so transcripts arrive as quickly as possible.

- **`manual`**:
  - Disable automatic turn detection entirely.
  - You control turns explicitly using `session.commit_user_turn()`, `session.clear_user_turn()`, and `session.interrupt()`.
  - See the [manual turn control docs](https://docs.livekit.io/agents/logic/turns/#manual-turn-control) for details.

### VAD configuration

With `turn_detection="stt"`, AssemblyAI also sends `SpeechStarted` events that LiveKit uses for barge-in/interruption handling. `SpeechStarted` is only emitted when the model produces a transcript.

Silero VAD is not strictly required in this mode, but it is still recommended as Silero runs locally and it can be faster than waiting for AssemblyAI's `SpeechStarted` signal. LiveKit respects whichever signal arrives first, so Silero provides faster interruption while AssemblyAI's signal serves as a reliable backup.

With `MultilingualModel()`, Silero VAD is **required**, as it is the only source of `START_OF_SPEECH` events for interruption in this mode. AssemblyAI's `SpeechStarted` event is not used.

#### Threshold alignment

LiveKit's Silero VAD defaults to an `activation_threshold` of **0.5**. AssemblyAI's `vad_threshold` defaults to **0.3**. For best performance, we recommend setting both to **0.3**.

Both should be adjusted together to the same value to ensure accurate transcription and consistent barge-in thresholds.

When the thresholds are mismatched, you get a dead zone: if Silero is at `0.5` and AssemblyAI is at `0.3`, AssemblyAI will be actively transcribing speech that LiveKit hasn't detected yet, delaying interruption. Keeping them aligned eliminates this.

```python
session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        vad_threshold=0.3,         # AssemblyAI's internal VAD onset
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,  # Match AssemblyAI's threshold
    ),
)
```

If you're in a noisy environment and receiving false speech triggers, raise both `stt.vad_threshold` and `vad.activation_threshold` thresholds together.

### Entity splitting tradeoff

Lower `min_turn_silence` and `max_turn_silence` values produce faster transcripts but can split entities or utterances across turns. The two parameters affect this differently.

#### `min_turn_silence` too low

- Speculative check fires too early, splitting entities on punctuation.

- **Example:** _User spells out an email address with brief pauses between parts. The speculative check fires at 100ms of silence, and the model adds terminal punctuation to each segment, ending the turn prematurely._

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"It's John."                    → FINAL (100ms pause, check fires, period found → turn ends)
"Smith."                        → FINAL
"At gmail.com."                 → FINAL

# With (min_turn_silence=400, max_turn_silence=1000)
"It's john.smith@gmail.com."    → FINAL (single turn, properly formatted)
```

#### `max_turn_silence` too low

- Forced turn-end cuts off user mid-thought.

- **Example:** _User pauses longer than 1 second to think mid-sentence. The forced end fires at 1000ms, splitting the utterance into two turns regardless of punctuation._

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"I wanted to check on my order from..."  → FINAL (1000ms silence, forced end)
"last Tuesday, order number 4829."     → FINAL (new turn)

# With (min_turn_silence=100, max_turn_silence=2000)
"I wanted to check on my order from last Tuesday, order number 4829."  → FINAL (single turn)
```

**Universal 3.5 Pro Realtime's** formatting is significantly better when it has full context in a single turn. Email addresses, phone numbers, credit card numbers, and physical addresses all benefit from this.

  LLMs downstream can usually piece together split entities, but if your use
  case involves alphanumeric dictation or entity extraction, consider increasing
  `min_turn_silence` and `max_turn_silence` during those portions of the
  conversation. You can [update configuration
  mid-stream](#dynamic-configuration) to raise `max_turn_silence`
  temporarily (e.g., to `2000`–`4000` ms) when expecting entity input, then
  lower it again afterward.

Even when using third-party turn detection, you may want to increase `min_turn_silence` or `max_turn_silence` if users are likely to speak slowly or dictate entities. While this adds latency, it improves accuracy by giving the model more audio context before emitting a transcript and keeping the full entity complete within the same turn.

## Latency

A voice agent feels responsive when the gap between the user finishing and the agent replying is short.

Start with the **`mode` preset** — the highest-level dial for the accuracy/latency trade-off. It sets sensible defaults for the fine-grained levers below, so you can pick a target and tune from there:

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    mode="balanced",  # "min_latency" (fastest) · "balanced" · "max_accuracy" (best quality)
)
```

`mode` is set at construction time (it can't be changed mid-session) and influences the defaults of `min_turn_silence`, `vad_threshold`, `interruption_delay`, `continuous_partials`, and `previous_context_n_turns`. Any value you set explicitly still wins. Leave it unset to use the server's default preset. See [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency).

From there, fine-tune the individual levers:

- **Endpointing delay (STT mode).** LiveKit's `endpointing.min_delay` is applied *on top of* AssemblyAI's own end-of-turn timing and adds up to `500ms` by default. Set `endpointing={"min_delay": 0}` when using `turn_detection="stt"` — see [STT-based turn detection](#stt-based-turn-detection-recommended).
- **End-of-turn timing.** `min_turn_silence` (speculative check) and `max_turn_silence` (forced end) directly control how soon a turn ends. Lower is faster but risks splitting entities — see [Turn detection](#turn-detection).
- **Time to first partial.** `interruption_delay` controls how soon the first partial is emitted, which drives faster barge-in and speculative inference. The server adds a minimum of `300ms` on top of the configured value.

```python
# Tune first-partial timing for faster barge-in
stt.update_options(
    interruption_delay=0,  # ~300ms effective TTFT
)
```

- **Sample rate.** Use 16 kHz (`sample_rate=16000`). Higher rates don't improve accuracy and only add bandwidth.
- **Continuous partials.** `continuous_partials` (on by default) emits a partial every ~3 seconds during long turns. Leave it on for steady mid-turn updates, or disable it if you only need a single early partial.
- **Skip client-side preprocessing.** Don't run your own noise cancellation before audio reaches the model — the artifacts it introduces usually hurt accuracy more than the original noise. Use server-side [Voice Focus](#voice-focus) instead.

### Latency breakdown

| Stage                                      | Typical                                              | Controlled by           |
| ------------------------------------------ | ---------------------------------------------------- | ----------------------- |
| Network round trip                         | ~50 ms                                               | —                       |
| Speech-to-text                             | ~200–300 ms                                          | model                   |
| First partial (TTFT)                       | configured `interruption_delay` + ~300 ms server min | `interruption_delay`    |
| End of turn (terminal punctuation found)   | `min_turn_silence` (default 100 ms)                  | `min_turn_silence`      |
| End of turn (no punctuation, forced)       | up to `max_turn_silence`                             | `max_turn_silence`      |
| LiveKit endpointing (STT mode)             | + `min_delay` (set to `0`)                           | `endpointing.min_delay` |

## Accuracy

**Universal 3.5 Pro Realtime** is accurate out of the box. When you need more — domain vocabulary, proper nouns, noisy audio — reach for these levers. For entity-heavy dictation, also tune turn detection (see [Entity splitting tradeoff](#entity-splitting-tradeoff)), and note that the high-level [`mode` preset](#latency) shifts the overall accuracy/latency balance (use `max_accuracy` to favor quality).

### Prompting

  **Beta feature**

Prompting is considered a beta feature for **Universal 3.5 Pro Realtime**.

While it can be a powerful tool for improving accuracy in certain use cases, **we recommend starting without a `prompt` to first establish baseline performance.**

Once the baseline has been tested, you can add context to further optimize for your use case (e.g., language mix to expect (e.g., English and Hindi), use case or domain (e.g., medical, legal), etc.).

**Universal 3.5 Pro Realtime** supports a `prompt` parameter for [contextual prompting](/streaming/prompting-and-keyterms) — a description of what the audio is about. Transcription behavior (verbatim output, punctuation, turn detection) is built in and optimized automatically; the prompt carries context, not instructions.

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    prompt="Customer support call about an internet service outage.",
)
```

**Tips:**

- **Start with no prompt**: Universal 3.5 Pro Realtime delivers strong accuracy out of the box — only add context if domain-specific vocabulary is being misrecognized.
- **Describe the conversation**: domain, scenario, or full details — start broad, and add only details your application actually knows (see the [three context levels](/streaming/prompting-and-keyterms#contextual-prompting)).
- **Include known names and identifiers**: caller name, account or order IDs, products — detailed context helps the model spell them correctly.

### Key terms

Instead of `prompt`, use `keyterms_prompt` to boost recognition of specific names, brands, or domain terms:

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    keyterms_prompt=["AssemblyAI", "LiveKit", "Universal 3.5 Pro Realtime"],
)
```

### Language selection

By default, **Universal 3.5 Pro Realtime** auto-detects and code-switches across all supported languages. If you know which languages a caller will speak, `language_codes` steers transcription toward them for better accuracy on short or ambiguous turns. It can also be updated mid-session (e.g. when a caller switches languages) via `update_options`. Requires `livekit-agents` **1.6.6+**.

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    language_codes=["en", "es"],  # single code or list, up to 10; ISO 639-1
)
```

### Conversation context

Give the model both sides of the dialog so it transcribes the next user turn more accurately. **Universal 3.5 Pro Realtime** keeps a short, per-session memory of the conversation from two sources:

- **The agent half** — what your agent just said, forwarded automatically into `agent_context` by `AgentSession` on `livekit-agents` 1.6.6+ (no wiring needed).
- **The user half** — prior STT-finalized user turns, carried forward automatically (no configuration needed).

With the agent's question in context, the model can anticipate the answer, sharpen entity recognition, and disambiguate similar-sounding words. For example, after your agent asks `"What's your email address?"`, `agent_context` lets the model produce `"user@assemblyai.com"` instead of `"user at assemblyai dot com"`. This has the biggest impact on short replies (`"yes"`, `"7pm"`, single names) and spelled-out entities. See [Conversation context](/streaming/universal-3-5-pro/context-carryover) for the full reference.

| Parameter                  | Type    | Description                                                                                                                                                                                                    |
| -------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_context`            | string  | Your agent's most recent spoken reply (what your TTS just said), up to 1750 characters. Set at construction time to seed an opening greeting; updated automatically after each agent reply on 1.6.6+.           |
| `previous_context_n_turns` | integer | How many prior conversation entries are carried forward automatically. Connect-time only. Range `0`–`100`; `0` disables carryover; leave unset for the server default (5).                                     |

  `agent_context` and `previous_context_n_turns` are supported only on Universal-3.5 Pro (`universal-3-5-pro`). `previous_context_n_turns` is set at construction time and cannot be changed with `update_options()`.

#### Automatic on 1.6.6+

On `livekit-agents` **1.6.6+**, `AgentSession` forwards each assistant reply into `agent_context` automatically — no wiring needed. Replies longer than the 1750-character limit are truncated automatically, keeping the tail (the end of the reply — usually the question — has the most biasing value).

To opt out, use the session-level toggle:

```python
session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    stt_context_options={"forward_chat_context": False},  # default: True
)
```

  While automatic forwarding is on, any `agent_context` you set manually is
  replaced by the next assistant reply. To manage `agent_context` yourself
  (e.g. custom trimming or filtering), set `forward_chat_context: False` and
  push it via `update_options(agent_context=...)` after each reply.

  To seed the model with your agent's opening greeting, pass `agent_context`
  on `assemblyai.STT(...)` at construction time — automatic forwarding takes
  over from the first reply onward.

If you're on a version of `livekit-agents` that predates **1.6.6**, or you want to manage `agent_context` yourself, wire it via the `conversation_item_added` event. LiveKit emits it for every conversation item; filter for the assistant's messages and forward the (trimmed) text to the STT plugin via `update_options(agent_context=...)`. No reconnect is required.

A complete runnable example is on GitHub: [including-agent-context](https://github.com/dlange-aai/assemblyai-livekit-examples/tree/main/including-agent-context).

```python
from livekit.agents import AgentSession, ConversationItemAddedEvent
from livekit.plugins import assemblyai

AGENT_CONTEXT_MAX_CHARS = 1750

session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
    ),
    stt_context_options={"forward_chat_context": False},  # Disable automatic forwarding on 1.6.6+
    # llm=your_llm_plugin(),
    # tts=your_tts_plugin(),
)

@session.on("conversation_item_added")
def _on_conversation_item_added(ev: ConversationItemAddedEvent) -> None:
    # Only forward the agent's own replies as context for the next user turn.
    if ev.item.type != "message" or ev.item.role != "assistant":
        return

    agent_stt = session.stt
    if not isinstance(agent_stt, assemblyai.STT):
        return

    spoken = ev.item.text_content
    if not spoken:
        return

    # Push the latest reply; earlier turns are carried forward automatically.
    agent_stt.update_options(agent_context=spoken[-AGENT_CONTEXT_MAX_CHARS:])
```

#### LiveKit Inference

LiveKit Inference (`stt="assemblyai/universal-3-5-pro"`) receives the same automatic conversation-context forwarding by default, with the same `stt_context_options={"forward_chat_context": False}` opt-out. `language_codes` is plugin-only for now.

### Voice focus

Voice Focus isolates the primary speaker and suppresses background noise — chatter, keyboard clicks, fan hum, room echo — **server-side, before audio reaches the model**. Use it instead of client-side noise cancellation, which tends to introduce artifacts that hurt accuracy more than the noise itself.

| Parameter               | Type   | Description                                                                                                                                       |
| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `voice_focus`           | string | `"near-field"` for headsets, handsets, and other close-talking mics; `"far-field"` for conference rooms, laptop mics, and other distant capture. |
| `voice_focus_threshold` | float  | Optional. `0.0`–`1.0`; higher values suppress background audio more aggressively.                                                                |

Both are connection-time parameters on the Universal-3.5 Pro and cannot be changed with `update_options()`. See [Voice Focus](/streaming/voice-focus) for details.

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    voice_focus="far-field",      # "near-field" for close-talking mics
    voice_focus_threshold=0.5,    # Optional: 0.0–1.0, higher = more aggressive
)
```

## Interruption handling

In voice agent conversations, users often produce backchannel utterances ("mhm", "yeah", "um", "okay") while the agent is speaking. These short fillers can trigger LiveKit's interruption logic, causing the agent to stop mid-sentence even though the user didn't intend to interrupt.

LiveKit Cloud users should reach for adaptive interruption handling first. Self-hosted deployments can combine the two custom filters described below.

### Recommended: Adaptive interruption handling

LiveKit Agents v1.5.0 introduced [adaptive interruption handling](https://docs.livekit.io/agents/logic/turns/adaptive-interruption-handling/), a server-side model that classifies overlapping speech as a real interrupt or a backchannel. On a false interrupt the agent's TTS resumes from where it left off. No re-generation is needed. The feature is recommended for LiveKit Cloud users and is included on all Cloud plans.

```python
session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    # llm=your_llm_plugin(),
    # tts=your_tts_plugin(),
    vad=silero.VAD.load(),
    interruption={"mode": "adaptive"},  # default in v1.5.0+
)
```

See LiveKit's [adaptive interruption handling docs](https://docs.livekit.io/agents/logic/turns/adaptive-interruption-handling/) for full requirements and behavior.

### Self-hosted alternative

If you're self-hosting LiveKit, can't use adaptive interruption handling, or need explicit control over which utterances count as filler, the two complementary filters below provide strong guardrails for interruption and barge-in handling. They have non-overlapping failure modes: the backchannel filter stops known fillers before they enter the pipeline, while the buffer-clearing filter catches any short utterance (including unknown fillers or stutters) that slips past. Running both provides the strongest coverage. A working reference implementation is available on [GitHub](https://github.com/AssemblyAI-Solutions/livekit-interruption-filters).

The backchannel filter intercepts STT events at the `stt_node` level, before they reach LiveKit's `audio_recognition`. It checks each transcript event for known disfluencies and backchannels ("um", "mhm", "yeah", "okay", etc.) and drops the event entirely. Because the event never enters the pipeline, none of the downstream orchestration (interrupt gates, end-of-turn detection, and preemptive LLM generation) ever reacts to it.

The filter is implemented as a mixin class that wraps `Agent.stt_node`:

```python
from __future__ import annotations

import logging
import string
import time
from collections.abc import AsyncIterable

from livekit import rtc
from livekit.agents import Agent, stt
from livekit.agents.voice import ModelSettings

# "yes" / "no" deliberately omitted - in a booking flow a bare "yes"
# is a real confirmation. Edit for your domain.
BACKCHANNELS = frozenset({
    "mhm", "mm", "mmhm", "mmhmm",
    "uh", "uhhuh", "huh",
    "um", "umm", "uhm",
    "er", "erm",
    "hmm", "hm",
    "ah", "oh",
    "yeah", "yep", "yup",
    "okay", "ok",
    "right", "alright", "gotcha",
})

_TRANSCRIPT_TYPES = {
    stt.SpeechEventType.INTERIM_TRANSCRIPT,
    stt.SpeechEventType.PREFLIGHT_TRANSCRIPT,
    stt.SpeechEventType.FINAL_TRANSCRIPT,
}

_PUNCT_STRIP = str.maketrans("", "", string.punctuation)

log = logging.getLogger("backchannel_stt_filter")

def _is_all_backchannel(text: str) -> bool:
    """Return True only when every token is a known backchannel."""
    tokens = text.lower().translate(_PUNCT_STRIP).split()
    return bool(tokens) and all(tok in BACKCHANNELS for tok in tokens)

class BackchannelSTTFilterMixin:
    """Drop backchannel-only transcripts while the agent is speaking."""

    _FILTER_GRACE_S: float = 1.0
    _last_speaking_at: float = 0.0

    async def stt_node(
        self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings
    ):
        async for ev in Agent.default.stt_node(self, audio, model_settings):
            if self._should_drop(ev):
                text = ev.alternatives[0].text if ev.alternatives else ""
                log.info(
                    "event_filtered transcript=%r ev_type=%s agent_state=%s",
                    text, ev.type, self.session.agent_state,
                )
                continue
            yield ev

    def _should_drop(self, ev: stt.SpeechEvent) -> bool:
        now = time.monotonic()
        if self.session.agent_state == "speaking":
            self._last_speaking_at = now
        elif now - self._last_speaking_at > self._FILTER_GRACE_S:
            return False

        if ev.type not in _TRANSCRIPT_TYPES:
            return False

        text = ev.alternatives[0].text if ev.alternatives else ""
        return _is_all_backchannel(text)
```

**How it works:**

1. While the agent is speaking (plus a 1-second grace window after speech ends), the filter inspects each STT transcript event
2. It strips punctuation and checks whether every token in the transcript matches the `BACKCHANNELS` set
3. Pure-filler transcripts like "mhm" or "yeah okay" are dropped. They never reach LiveKit's pipeline
4. Utterances with any non-filler token (e.g., "yeah I want the suite") always pass through

  The `BACKCHANNELS` set is domain-customizable. "yes" and "no" are deliberately excluded because they often represent genuine confirmations in booking or IVR scenarios. Edit the set for your use case.

LiveKit accumulates committed `FINAL` transcripts in a private buffer (`_audio_transcript`) across the user's uncommitted turn. The `_interrupt_by_audio_activity` method checks the running word count against `min_words` to decide whether to pause TTS.

Without intervention, two consecutive short fillers like "yeah" + "um" sum to two words and trip the interrupt gate, even though each utterance on its own is below threshold.

This filter listens on the `user_input_transcribed` event and wipes the buffer whenever the agent is speaking and the user input falls below the configured `min_words` threshold. Each short utterance is evaluated independently rather than against the accumulated total.

  This filter requires `interruption.min_words` to be set to `2` or higher. Without it, the word-count gate is disabled and the filter has no effect.

```python
from __future__ import annotations

import logging

from livekit.agents import AgentSession

log = logging.getLogger("short_utterance_buffer_filter")

def install_short_utterance_filter(session: AgentSession) -> None:
    """Clear transcript buffers when a short utterance arrives during agent speech."""

    @session.on("user_input_transcribed")
    def _on_user_input_transcribed(ev) -> None:
        word_count = len(ev.transcript.split())
        min_words = session.options.interruption["min_words"]

        if session.agent_state != "speaking":
            return

        if word_count >= min_words:
            return

        activity = getattr(session, "_activity", None)
        recognition = getattr(activity, "_audio_recognition", None) if activity else None
        if recognition is None:
            return

        # Wipe all three transcript buffers so short utterances
        # don't accumulate past the interrupt threshold.
        recognition._audio_transcript = ""
        recognition._audio_interim_transcript = ""
        recognition._audio_preflight_transcript = ""

        # Best-effort: abort any in-flight preemptive LLM call
        # triggered by this short utterance.
        cancel = getattr(activity, "_cancel_preemptive_generation", None)
        if callable(cancel):
            try:
                cancel()
            except Exception:
                log.debug("_cancel_preemptive_generation failed", exc_info=True)

        log.info(
            "buffer_cleared transcript=%r words=%d is_final=%s",
            ev.transcript, word_count, ev.is_final,
        )
```

  This filter accesses LiveKit private APIs (`_audio_transcript`, `_audio_interim_transcript`, `_audio_preflight_transcript`, `_cancel_preemptive_generation`). Pin your `livekit-agents` version to avoid breakage on minor updates. Tested with `livekit-agents>=1.5`.

Three changes are needed:

**1. Mix in the backchannel filter on your agent class.** Place it before `Agent` in the class bases so its `stt_node` runs first:

```python
from filters.backchannel_stt import BackchannelSTTFilterMixin

class MyAgent(BackchannelSTTFilterMixin, Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant.")
```

**2. Configure the session with `interruption.min_words >= 2`.** This enables the word-count gate that the buffer-clearing filter depends on:

```python
session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    # llm=your_llm_plugin(),
    # tts=your_tts_plugin(),
    vad=None,  # Recommended: disable VAD so only STT drives interruption
    turn_handling={
        "turn_detection": "stt",
        "endpointing": {"min_delay": 1.0, "max_delay": 4.0},
        "interruption": {
            "enabled": True,
            "resume_false_interruption": True,
            "false_interruption_timeout": 1.5,
            "min_words": 2,  # Required for the buffer-clearing filter
        },
    },
)
```

**3. Install the buffer-clearing filter on the session**, then start it with your agent:

```python
from filters.short_utterance_buffer import install_short_utterance_filter

install_short_utterance_filter(session)

await session.start(room=ctx.room, agent=MyAgent())
```

  Setting `vad=None` with `turn_detection="stt"` is the recommended setup for both filters. This ensures only STT-based signals drive interruption, avoiding timing races from a competing VAD interrupt path. If you need VAD for faster barge-in, both filters still work: set `interruption.min_words` to `2` and ensure Silero's `activation_threshold` matches `vad_threshold`.

**Expected behavior** with both filters active and `min_words=2`:

| User utterance (during agent speech) | Backchannel filter | Buffer-clearing filter | Result |
| ------------------------------------ | ------------------ | ---------------------- | ------ |
| "mhm"                               | Dropped            | -                      | Agent continues |
| "um"                                 | Dropped            | -                      | Agent continues |
| "yeah yeah"                          | Dropped            | -                      | Agent continues |
| Unknown short word                   | Passes through     | Buffer cleared         | Agent continues |
| "yeah I'd like the suite"            | Passes through     | Passes through         | Agent interrupts |
| "suite please"                       | Passes through     | Passes through         | Agent interrupts |
| "mhm" (1+ second after agent stops) | Passes through     | -                      | Agent responds |

## Dynamic configuration

You can update `prompt`, `keyterms_prompt`, `agent_context`, `language_codes`, `min_turn_silence`, `max_turn_silence`, `continuous_partials`, and `interruption_delay` during an active session using `update_options` — no reconnect required. This lets you adapt to the conversation stage: boost names while collecting caller details, widen silence windows while a user dictates an email, then tighten them again.

```python
# Update one or more options mid-stream
stt.update_options(
    max_turn_silence=3000,  # Increase for entity dictation
)

# Later, reset to default
stt.update_options(
    max_turn_silence=1000,
)
```

| Conversation stage                          | Adjustment                                                                              |
| ------------------------------------------- | --------------------------------------------------------------------------------------- |
| Caller identification (names, account IDs)  | Boost terms with `update_options(keyterms_prompt=[...])`                                 |
| Entity dictation (email, phone, address)    | Raise `max_turn_silence` to ~`2000`–`4000` ms, then lower it again afterward            |
| After each agent reply                      | Handled automatically on 1.6.6+ (see [Conversation context](#conversation-context))     |
| Caller switches language                    | `update_options(language_codes=["es"])`                                                  |
| Faster barge-in                             | Lower `interruption_delay` (see [Latency](#latency))                                     |

For more information, see [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream).

## Troubleshooting

| Issue                                     | Cause                                                     | Solution                                                                                             |
| ----------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Extra latency with `turn_detection="stt"` | LiveKit's endpointing `min_delay` is additive in STT mode | Set `endpointing={"min_delay": 0}` inside `TurnHandlingOptions` on `AgentSession`                    |
| No interruption handling                  | Missing VAD                                               | Ensure `vad=silero.VAD` is set, with `activation_threshold` equal to `vad_threshold` (default `0.3`) |
| Turn over-segmentation                    | `min_turn_silence` too low                                | Increase from `100` to `200`–`500`                                                                   |
| Entities split across turns               | `max_turn_silence` too low                                | Increase `max_turn_silence` (e.g., `1500`–`3500`)                                                    |
| Latency on non-terminal utterances        | `max_turn_silence` too high                               | Lower `max_turn_silence`                                                                             |
| Mis-heard names, brands, or jargon        | No vocabulary hints                                       | Add `keyterms_prompt`, or supply `prompt`/`agent_context` for context                               |
| Poor accuracy in noisy audio              | Background noise or room echo                             | Enable `voice_focus` (`near-field` or `far-field`)                                                   |

## Migrating from another STT provider

To balance accuracy, latency, turn-taking, and interruption handling, map your current setup to AssemblyAI using the questions below. Each answer points to the settings that reproduce — and usually improve on — your current behavior.

### How are you detecting end-of-turn today?

| Today | Recommended on AssemblyAI |
| --- | --- |
| Your STT provider's own end-of-turn model (e.g. Deepgram endpointing) | `turn_detection="stt"` with `min_turn_silence=100`, `max_turn_silence=1000`, `endpointing={"min_delay": 0}`. AssemblyAI's punctuation-based end-of-turn replaces it. |
| Silence / VAD only | Prefer `turn_detection="stt"` for semantic endpointing, or `turn_detection="vad"` to stay silence-only. Align Silero `activation_threshold` and `vad_threshold` at `0.3`. |
| LiveKit's turn-detector model | `turn_detection=MultilingualModel()`, keep the plugin defaults (`min_turn_silence=100`, `max_turn_silence=100`), set `endpointing={"min_delay": 0.5, "max_delay": 3.0}`, and keep Silero VAD (required). |

### Which model and settings are you migrating from?

| What you pass today | AssemblyAI equivalent |
| --- | --- |
| Current model (Deepgram, ElevenLabs, etc.) | `model="universal-3-5-pro"` (recommended flagship) |
| Overall accuracy/latency tuning | `mode="min_latency"` / `"balanced"` / `"max_accuracy"` — a one-line starting point before fine-tuning |
| Endpointing / silence thresholds | `min_turn_silence` (speculative end-of-turn) and `max_turn_silence` (forced end) — see [Turn detection](#turn-detection) |
| Custom vocabulary / keywords | `keyterms_prompt=[...]`; broader domain context → `prompt` |
| Formatting / punctuation toggles | On by default — formatted transcripts always (`format_turns` does not apply) |
| Language(s) | 18 languages with code-switching; pin one or more languages via `language_codes`; `language_detection` adds language codes |

### VAD, interruptions, and infrastructure

| Today | Recommended |
| --- | --- |
| Silero VAD | Keep it; set `activation_threshold=0.3` to match `vad_threshold` |
| Adaptive vs. VAD-based barge-in | LiveKit Cloud → [adaptive interruption handling](#recommended-adaptive-interruption-handling); self-hosted → the [self-hosted filters](#self-hosted-alternative) |
| Telephony / SIP routing | `sample_rate=8000` and `encoding="pcm_mulaw"` for 8 kHz telephony |
| Client-side noise cancellation | Drop it; use server-side [Voice Focus](#voice-focus) instead |
| LiveKit Cloud vs. self-hosting | Cloud unlocks adaptive interruption; self-hosting uses the filters and version pinning |

### Parameter cheat sheet

If you're coming from AssemblyAI's standard streaming model:

| Change                             | From                                       | To                                                                                       |
| ---------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------- |
| Model                              | `assemblyai.STT()`                         | `assemblyai.STT(model="universal-3-5-pro")`                                                      |
| Turn detection                     | `turn_detection="stt"` or `EnglishModel()` | `turn_detection="stt"` or `MultilingualModel()`                                          |
| VAD                                | Optional                                   | Set `vad=silero.VAD.load()` to match `vad_threshold`                                     |
| `min_turn_silence`                 | `400` (old default)                        | `100` (new default)                                                                      |
| `max_turn_silence`                 | `1280` (old default)                       | `1000` (API default) or `100` (with 3rd-party turn detector)                             |
| `end_of_turn_confidence_threshold` | Configurable                               | **Not applicable**: **Universal 3.5 Pro Realtime** uses punctuation-based turn detection |
| `endpointing.min_delay` (formerly `min_endpointing_delay`) | Default `0.5`                              | Set to `0` inside `TurnHandlingOptions` when using `turn_detection="stt"`                |

Migrating a production deployment? [Talk to our team](https://www.assemblyai.com/contact/sales).

---

<a id="universal-3-5-pro-realtime-on-pipecat-docs-voice-agents-pipecat-universal-3-5-pro-mdx"></a>

# Universal 3.5 Pro Realtime on Pipecat

URL: https://www.assemblyai.com/docs/voice-agents/pipecat-universal-3-5-pro
Source: docs/voice-agents/pipecat-universal-3-5-pro.mdx
Navigation: Overview > Use cases & integrations > Integrations > Voice agent orchestrators
Description: Integrate AssemblyAI's Universal 3.5 Pro Realtime speech-to-text model into a Pipecat voice agent

## Overview

This guide covers integrating AssemblyAI's **Universal 3.5 Pro Realtime** speech-to-text model into a [Pipecat](https://docs.pipecat.ai/) voice agent. Everything here applies equally to **Universal-3.5 Pro Streaming** (`universal-3-5-pro`) — both belong to the same U3 Pro family and share every parameter in this guide, so you can swap the `model` string without changing anything else.

  **Universal 3.5 Pro Realtime is our flagship next-generation streaming model for voice agents** — multilingual and promptable, with [conversation context](#conversation-context) and [voice focus](#voice-focus).

  Available on **Pipecat 1.4.0+** — set `model="universal-3-5-pro"`.

AssemblyAI provides the speech-to-text and (optionally) the turn detection in your Pipecat pipeline:

```mermaid
flowchart LR
  U["User audio"] --> STT["AssemblyAI STT<br/>Universal 3.5 Pro Realtime"]
  STT --> TD["Turn detection<br/>Pipecat (VAD + Smart Turn)<br/>or AssemblyAI"]
  TD --> LLM["LLM"]
  LLM --> TTS["TTS"]
  TTS --> U
```

Once you have an agent running, tune it for what matters most to your use case:

  
    Decide when the user is done speaking — the two Pipecat modes, defaults, and entity tuning.
  
  
    Shorten the gap between the user finishing and the agent replying.
  
  
    Prompting, key terms, conversation context, and noise handling.
  
  
    Natural barge-in while the agent is speaking.
  

      <path d="M45.7718 70.7701C50.4477 69.0096 55.7252 70.3307 59.0204 74.0864L101.936 123.001H230.064L272.98 74.0864C276.275 70.3307 281.552 69.0096 286.228 70.7701C290.904 72.5306 294 77.0042 294 82.0005V190H332V214H270V113.873L244.52 142.915C242.242 145.512 238.955 147.001 235.5 147.001H96.5C93.0452 147.001 89.7581 145.512 87.4796 142.915L62 113.873V214H0V190H38V82.0005C38 77.0042 41.0958 72.5306 45.7718 70.7701Z" fill="currentColor" />
      <path d="M270 238.001H332V262.001H270V238.001Z" fill="currentColor" />
      <path d="M0 238.001H62V262.001H0V238.001Z" fill="currentColor" />
      <path d="M128 198.001C128 206.837 120.837 214.001 112 214.001C103.163 214.001 96 206.837 96 198.001C96 189.164 103.163 182.001 112 182.001C120.837 182.001 128 189.164 128 198.001Z" fill="currentColor" />
      <path d="M236 198.001C236 206.837 228.837 214.001 220 214.001C211.163 214.001 204 206.837 204 198.001C204 189.164 211.163 182.001 220 182.001C228.837 182.001 236 189.164 236 198.001Z" fill="currentColor" />
    </svg>
  }
  href="https://docs.pipecat.ai/server/services/stt/assemblyai"
>
  View Pipecat's AssemblyAI STT plugin reference.

  For a standalone voice agent without Pipecat or an external LLM, see the [AssemblyAI Voice Agent API](/voice-agents/voice-agent-api), which handles STT, LLM routing, and TTS in a single WebSocket connection.

## Quickstart

Get a working, talking agent in a few minutes, then optimize from there.

Install Pipecat with the AssemblyAI, LLM, and TTS extras you need:

```bash
pip install "pipecat-ai[assemblyai,openai,cartesia]" python-dotenv
```

**What's included:**

- `assemblyai`: AssemblyAI U3 Pro STT service
- `openai`: OpenAI LLM service (used in the example)
- `cartesia`: Cartesia TTS service (used in the example)

  The example uses OpenAI and Cartesia, but you can use any LLM or TTS supported
  by Pipecat — just swap the extras (e.g.,
  `pipecat-ai[assemblyai,anthropic,elevenlabs]`).

  Universal 3.5 Pro Realtime, automatic [conversation context](#conversation-context),
  and [Voice Focus](#voice-focus) require **`pipecat-ai` 1.4.0+**. Older versions
  won't recognize the `universal-3-5-pro` model.

Set your API keys in a `.env` file:

```env
ASSEMBLYAI_API_KEY=your_assemblyai_key
OPENAI_API_KEY=your_openai_key
CARTESIA_API_KEY=your_cartesia_key
```

  You can obtain an AssemblyAI API key by [signing up for a free
  account](https://www.assemblyai.com/dashboard/signup) and navigating to the [API
  Keys tab](https://www.assemblyai.com/dashboard/home) of the dashboard.

The example below uses Pipecat-controlled turn detection (the default). Pay attention to the comments for switching to AssemblyAI's built-in turn detection, and note that the **assistant aggregator** at the end of the pipeline is what enables automatic [conversation context](#conversation-context).

```python expandable
import os

from dotenv import load_dotenv
from loguru import logger

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMRunFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.base_transport import BaseTransport, TransportParams
from pipecat.transports.daily.transport import DailyParams
from pipecat.workers.runner import WorkerRunner

load_dotenv()

transport_params = {
    "daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
    "webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),
}

async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
    stt = AssemblyAISTTService(
        api_key=os.environ["ASSEMBLYAI_API_KEY"],
        settings=AssemblyAISTTService.Settings(
            model="universal-3-5-pro",
            min_turn_silence=100,
            # max_turn_silence is auto-synced to min_turn_silence in Pipecat mode.
            # vad_threshold=0.3,            # Align with your local VAD's threshold
            # continuous_partials=True,     # Default — steady ~3s partials during long turns
            # interruption_delay=0,         # Optional: faster first partial (~300ms effective)
        ),
        vad_force_turn_endpoint=True,  # Pipecat mode (default).
        # Set False to use AssemblyAI's built-in turn detection (universal-3-5-pro / universal-3-5-pro only):
        # vad_force_turn_endpoint=False,
    )

    llm = OpenAILLMService(api_key=os.environ["OPENAI_API_KEY"])
    tts = CartesiaTTSService(api_key=os.environ["CARTESIA_API_KEY"])

    context = LLMContext()
    user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
        context,
        user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
    )

    pipeline = Pipeline(
        [
            transport.input(),     # Transport user input
            stt,                   # STT
            user_aggregator,       # User responses
            llm,                   # LLM
            tts,                   # TTS
            transport.output(),    # Transport bot output
            assistant_aggregator,  # Assistant responses → automatic conversation context
        ]
    )

    worker = PipelineWorker(pipeline, params=PipelineParams(enable_metrics=True))

    @transport.event_handler("on_client_connected")
    async def on_client_connected(transport, client):
        context.add_message(
            {"role": "system", "content": "You are a helpful voice assistant. Keep replies brief and speakable."}
        )
        await worker.queue_frames([LLMRunFrame()])

    runner = WorkerRunner(handle_sigint=runner_args.handle_sigint)
    await runner.add_workers(worker)
    await runner.run()

async def bot(runner_args: RunnerArguments):
    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport, runner_args)

if __name__ == "__main__":
    from pipecat.runner.run import main

    main()
```

  Two complete, runnable examples live in the Pipecat repo:
  [voice-assemblyai.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-assemblyai.py)
  (Pipecat turn detection) and
  [voice-assemblyai-turn-detection.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-assemblyai-turn-detection.py)
  (AssemblyAI's built-in turn detection).

Run the agent directly with local audio:

```bash
python your_agent.py
```

Speak into your microphone after hearing the greeting. For WebRTC or Daily testing, see [Running your agent](#running-your-agent).

## Parameters reference

### **Universal 3.5 Pro Realtime** parameters

These are the key parameters to tune. Set them inside `AssemblyAISTTService.Settings(...)`. They apply to the whole U3 Pro family (`universal-3-5-pro` and `universal-3-5-pro`).

  The streaming model. `"universal-3-5-pro"` is the recommended flagship model;
  the plugin currently defaults to `"universal-3-5-pro"`, so set `model` explicitly. Both
  belong to the U3 Pro family and share every parameter below.

  Accuracy/latency preset: `"min_latency"`, `"balanced"`, or `"max_accuracy"`.
  Sets sensible defaults for mode-dependent fields; any value you set explicitly
  still takes precedence. The server defaults to `"balanced"`. Construction-time
  only. U3 Pro family only. See [Optimizing accuracy and
  latency](/streaming/getting-started/optimizing-accuracy-and-latency).

  List of terms to boost recognition for. Used on its own, your terms are
  appended to the default prompt automatically. Can't be set in the same request
  as `prompt` — see [Key terms](#key-terms) to combine boosting with a custom
  prompt.

  Contextual prompt — a natural-language description of what the audio is about
  (domain, scenario, or full details). Can't be set in the same request as
  `keyterms_prompt`; fold the terms into the prompt text instead (see
  [Key terms](#key-terms)). **Prompting is currently a beta feature**: see
  [Prompting](/streaming/prompting-and-keyterms) for more information.

  Context carryover seed — your agent's most recent spoken reply, up to ~1500
  characters, used to transcribe the next user turn more accurately. Set it at
  construction time to seed an opening greeting; later turns are fed
  automatically. U3 Pro family only. See [Conversation
  context](#conversation-context).

  How many prior conversation entries are carried forward automatically. Range
  `0`–`100`; `0` disables carryover entirely (including the automatic
  `agent_context` feed). Construction-time only; leave unset for the server
  default (`5` on `universal-3-5-pro`; `3` on older `u3-rt-pro`). U3 Pro family only.

  Milliseconds of silence before a speculative end-of-turn check. When the check
  fires, the model looks for terminal punctuation (`.` `?` `!`) to decide whether
  the turn has ended. (Formerly `min_end_of_turn_silence_when_confident`,
  deprecated but still supported with a warning.)

  Maximum silence before the turn is forced to end, regardless of punctuation.
  **Auto-synced to `min_turn_silence` in Pipecat mode**; respected as configured
  in AssemblyAI's built-in turn detection mode.

  AssemblyAI's internal VAD threshold (`0.0`–`1.0`) for classifying audio frames
  as silence. Align with your local VAD's activation threshold to avoid a "dead
  zone" where AssemblyAI transcribes speech your VAD hasn't detected yet.

  Server-side noise suppression that isolates the primary speaker.
  `"near-field"` for close-talking mics, `"far-field"` for distant capture.
  Construction-time only. U3 Pro family only. See [Voice focus](#voice-focus).

  How aggressively `voice_focus` suppresses background audio. `0.0`–`1.0`; higher
  is more aggressive. Only takes effect when `voice_focus` is set.
  Construction-time only. U3 Pro family only.

  Whether to emit additional partial transcripts during long turns at a steady
  ~3 second cadence. When enabled (default on both the API and this plugin),
  additional partials covering the full turn transcript are emitted
  approximately every 3 seconds while speech continues. When disabled, only one
  early partial is emitted near turn start. The first partial (at 750ms) is
  unaffected. Useful when downstream consumers (LLMs, UI, eager inference) need
  frequent updates during long, uninterrupted turns. See
  [Continuous partials](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  How soon the first partial transcript is emitted during a turn, in
  milliseconds. Range: `0`–`1000`. Lower values produce faster time to first
  token (TTFT) for barge-in and speculative inference; higher values produce
  more confident first partials. The server adds a minimum of 300ms on top of
  the configured value (`interruption_delay=0` → ~300ms effective,
  `interruption_delay=500` → ~800ms effective). See
  [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  **Universal 3.5 Pro Realtime** code-switches natively between supported
  languages. This parameter controls whether `language_code` and
  `language_confidence` are included in turn messages.

  Enable speaker diarization. See [Speaker diarization](#speaker-diarization).

### General parameters

These apply across models and Pipecat setups. `api_key`, `vad_force_turn_endpoint`, `should_interrupt`, and `speaker_format` are passed directly to `AssemblyAISTTService(...)`, not inside `Settings`.

  Your AssemblyAI API key.

  `True` for Pipecat mode (VAD + Smart Turn controls turns); `False` for
  AssemblyAI's built-in turn detection (`universal-3-5-pro` / `universal-3-5-pro` only).
  See [Turn detection](#turn-detection).

  Whether the user starting to speak interrupts the bot. Only applies in
  AssemblyAI's built-in turn detection mode (`vad_force_turn_endpoint=False`).

  Template string for formatting speaker labels (e.g., `"[{speaker}] {text}"`).
  Used with `speaker_labels`.

  The sample rate of the audio stream.

  The encoding of the audio stream. Allowed values: `pcm_s16le`, `pcm_mulaw`.

### Legacy parameters

These apply to the `universal-streaming-english` and `universal-streaming-multilingual` models, but **do not affect Universal 3.5 Pro Realtime or `universal-3-5-pro`**:

  Confidence threshold for end-of-turn detection. The U3 Pro family uses
  punctuation-based turn detection instead, so this parameter has no effect.

  Whether to return formatted final transcripts. The U3 Pro family always
  returns formatted transcripts, so this parameter no longer applies.

## Turn detection

In Pipecat, you choose **which component decides when the user is done speaking** with the `vad_force_turn_endpoint` flag on `AssemblyAISTTService`. The U3 Pro family uses a **punctuation-based** end-of-turn system: after a period of silence, the model checks for terminal punctuation (`.` `?` `!`) rather than a confidence score. For more on how this works, see [Configuring turn detection](/streaming/getting-started/transcribe-streaming-audio).

  The `vad_force_turn_endpoint` parameter controls which turn detection mode is
  used. It defaults to `True` (Pipecat mode), which sends a `ForceEndpoint`
  message to AssemblyAI when the local VAD detects silence. Set it to `False` to
  use AssemblyAI's built-in turn detection instead. Choosing the right mode is
  critical for balancing responsiveness and turn accuracy in your voice agent.

### Pipecat mode (default, recommended)

**When to use:** Most voice agent applications requiring responsive interruptions.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        min_turn_silence=100,
    ),
    vad_force_turn_endpoint=True,  # Default (Pipecat mode)
)
```

**How it works:**

- VAD + the Smart Turn analyzer control when the user is done speaking.
- A `ForceEndpoint` message is sent to AssemblyAI on VAD silence detection.
- `max_turn_silence` is **automatically synchronized** with `min_turn_silence`.
- Best for low-latency, responsive voice agents.

### AssemblyAI's built-in turn detection

**When to use:** When you want AssemblyAI's punctuation-based turn detection to control turn endings, configured through the settings below.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,  # Now respected independently
    ),
    vad_force_turn_endpoint=False,  # AssemblyAI's built-in turn detection
)
```

**How it works:**

1. User speaks → audio streams to AssemblyAI.
2. User pauses for `min_turn_silence` (e.g., `100ms`) → the model checks for terminal punctuation.
3. If terminal punctuation (`.` `?` `!`) is found → the turn ends immediately.
4. If not → a partial is emitted and the turn continues waiting.
5. If silence reaches `max_turn_silence` (e.g., `1000ms`) → the turn is forced to end regardless.

In this mode all timing parameters are respected as configured, the service emits `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame`, and `SpeechStarted` events drive fast barge-in. Only available with `universal-3-5-pro` / `universal-3-5-pro` (other models require Pipecat mode).

### Entity splitting tradeoff

Lower `min_turn_silence` and `max_turn_silence` values produce faster transcripts but can split entities or utterances across turns. The two parameters affect this differently.

#### `min_turn_silence` too low

The speculative check fires too early, splitting entities on punctuation:

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"It's John."                    → FINAL (100ms pause, check fires, period found → turn ends)
"Smith."                        → FINAL
"At gmail.com."                 → FINAL

# With (min_turn_silence=400, max_turn_silence=1000)
"It's john.smith@gmail.com."    → FINAL (single turn, properly formatted)
```

#### `max_turn_silence` too low

The forced turn-end cuts off the user mid-thought:

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"I wanted to check on my order from..."  → FINAL (1000ms silence, forced end)
"last Tuesday, order number 4829."       → FINAL (new turn)

# With (min_turn_silence=100, max_turn_silence=2000)
"I wanted to check on my order from last Tuesday, order number 4829."  → FINAL (single turn)
```

  **Universal 3.5 Pro Realtime's** formatting is significantly better when it has
  full context in a single turn — email addresses, phone numbers, credit card
  numbers, and physical addresses all benefit. If your use case involves
  alphanumeric dictation, raise `max_turn_silence` during those portions of the
  conversation (e.g., to `2000`–`4000` ms) using [dynamic
  configuration](#dynamic-configuration), then lower it again afterward. In
  Pipecat mode, raise `min_turn_silence` (which `max_turn_silence` follows) for
  the same effect.

## Latency

A voice agent feels responsive when the gap between the user finishing and the agent replying is short.

Start with the **`mode` preset** — the highest-level dial for the accuracy/latency trade-off. It sets sensible defaults for the fine-grained levers below, so you can pick a target and tune from there:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        mode="balanced",  # "min_latency" (fastest) · "balanced" · "max_accuracy" (best quality)
    ),
)
```

`mode` is set at construction time (it can't be changed mid-session) and influences the defaults of the levers below. Any value you set explicitly still wins. Leave it unset to use the server's default preset. See [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency).

From there, fine-tune the individual levers:

- **End-of-turn timing.** `min_turn_silence` (speculative check) and `max_turn_silence` (forced end) directly control how soon a turn ends. Lower is faster but risks splitting entities — see [Turn detection](#turn-detection).
- **Time to first partial.** `interruption_delay` controls how soon the first partial is emitted, which drives faster barge-in and speculative inference. The server adds a minimum of `300ms` on top of the configured value.
- **Sample rate.** Use 16 kHz (`sample_rate=16000`). Higher rates don't improve accuracy and only add bandwidth.
- **Continuous partials.** `continuous_partials` (on by default) emits a partial every ~3 seconds during long turns. Leave it on for steady mid-turn updates, or disable it if you only need a single early partial.
- **Skip client-side preprocessing.** Don't run your own noise cancellation before audio reaches the model — the artifacts it introduces usually hurt accuracy more than the original noise. Use server-side [Voice Focus](#voice-focus) instead.

### Latency breakdown

| Stage                                    | Typical                                              | Controlled by        |
| ---------------------------------------- | ---------------------------------------------------- | -------------------- |
| Network round trip                       | ~50 ms                                               | —                    |
| Speech-to-text                           | ~200–300 ms                                          | model                |
| First partial (TTFT)                     | configured `interruption_delay` + ~300 ms server min | `interruption_delay` |
| End of turn (terminal punctuation found) | `min_turn_silence` (default 100 ms)                  | `min_turn_silence`   |
| End of turn (no punctuation, forced)     | up to `max_turn_silence`                             | `max_turn_silence`   |

## Accuracy

**Universal 3.5 Pro Realtime** is accurate out of the box. When you need more — domain vocabulary, proper nouns, noisy audio — reach for these levers. For entity-heavy dictation, also tune turn detection (see [Entity splitting tradeoff](#entity-splitting-tradeoff)), and note that the high-level [`mode` preset](#latency) shifts the overall accuracy/latency balance (use `max_accuracy` to favor quality).

### Prompting

  **Beta feature**

  Prompting is considered a beta feature for **Universal 3.5 Pro Realtime**.

  While it can be a powerful tool for improving accuracy in certain use cases,
  **we recommend starting without a `prompt` to first establish baseline
  performance.** Once the baseline has been tested, you can add context to
  further optimize for your use case (e.g., language mix to expect, use case or
  domain).

**Universal 3.5 Pro Realtime** supports a `prompt` parameter for [contextual prompting](/streaming/prompting-and-keyterms) — a description of what the audio is about. Transcription behavior (verbatim output, punctuation, turn detection) is built in and optimized automatically; the prompt carries context, not instructions.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        prompt="Customer support call about an internet service outage.",
    ),
)
```

### Key terms

Use `keyterms_prompt` to boost recognition of specific names, brands, or domain terms. On its own, your terms are appended to the default prompt automatically — so you get boosting and prompting together:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        keyterms_prompt=["Xiomara", "Saoirse", "Pipecat", "AssemblyAI"],
    ),
)
```

  You can't pass `prompt` and `keyterms_prompt` in the same request — doing so
  raises a validation error. You don't have to give up term boosting to use a
  contextual prompt, though. Either:

  - Pass **`keyterms_prompt` on its own** — your terms are appended to the
    default prompt automatically, or
  - Fold the terms into a custom **`prompt`**, e.g. end it with
    `"Make sure to boost the words Xiomara, Saoirse, Pipecat in the audio."`

### Conversation context

Give the model both sides of the dialog so it transcribes the next user turn more accurately. **Universal 3.5 Pro Realtime** keeps a short, per-session memory of the conversation from two sources:

- **The agent half** — what your agent just said.
- **The user half** — prior STT-finalized user turns.

With the agent's question in context, the model can anticipate the answer, sharpen entity recognition, and disambiguate similar-sounding words. For example, after your agent asks `"What's your email address?"`, the model can produce `"user@assemblyai.com"` instead of `"user at assemblyai dot com"`. This has the biggest impact on short replies (`"yes"`, `"7pm"`, single names) and spelled-out entities. See [Conversation context](/streaming/universal-3-5-pro/context-carryover) for the full reference.

  **In Pipecat, conversation context is automatic — no event wiring required.**
  As long as your pipeline includes the standard LLM context aggregator (the
  `assistant_aggregator` from `LLMContextAggregatorPair`), Pipecat broadcasts an
  `LLMContextAssistantTurnFrame` when each bot turn completes, and
  `AssemblyAISTTService` feeds that reply to the model as `agent_context`
  automatically. Just use a U3 Pro family model on `pipecat-ai` 1.4.0+.

| Parameter                  | Type | Description                                                                                                                                                              |
| -------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `agent_context`            | str  | Your agent's most recent spoken reply, up to ~1500 characters. Set it at construction time to seed an opening greeting; subsequent replies are fed automatically.        |
| `previous_context_n_turns` | int  | How many prior conversation entries are carried forward automatically. Range `0`–`100`; `0` disables carryover entirely. Construction-time only; server default is `5` on `universal-3-5-pro` (`3` on older `u3-rt-pro`). |

#### Seeding the opening greeting

The automatic feed kicks in once your agent completes its first turn. To give the model context for the user's *very first* reply (the answer to your greeting), set `agent_context` at construction time:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        # Seed the opening line; later turns are fed automatically by the aggregator.
        agent_context="Hi! Thanks for calling Acme. What's the email on your account?",
        # previous_context_n_turns=5,  # Default on universal-3-5-pro. Set 0 to disable carryover entirely.
    ),
)
```

#### Manual control with `update_agent_context()`

If your pipeline doesn't use the standard LLM context aggregator, or you want explicit control over what the model sees, push the agent's reply yourself. This is a **live update — no reconnect required**:

```python
# Whenever your agent finishes speaking:
await stt.update_agent_context("Your account is past due. Would you like to pay now?")
```

  `agent_context`, `previous_context_n_turns`, and `update_agent_context()` are
  supported only on the U3 Pro family (`universal-3-5-pro`, `universal-3-5-pro`). Values
  are clipped to ~1500 characters and re-seeded automatically on reconnect.
  Setting `previous_context_n_turns=0` disables the automatic feed as well.

### Voice focus

Voice Focus isolates the primary speaker and suppresses background noise — chatter, keyboard clicks, fan hum, room echo — **server-side, before audio reaches the model**. Use it instead of client-side noise cancellation, which tends to introduce artifacts that hurt accuracy more than the noise itself.

| Parameter               | Type   | Description                                                                                                                                       |
| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `voice_focus`           | str    | `"near-field"` for headsets, handsets, and other close-talking mics; `"far-field"` for conference rooms, laptop mics, and other distant capture. |
| `voice_focus_threshold` | float  | Optional. `0.0`–`1.0`; higher values suppress background audio more aggressively.                                                                |

Both are construction-time parameters on the U3 Pro family. See [Voice Focus](/streaming/voice-focus) for details.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        voice_focus="far-field",    # "near-field" for close-talking mics
        voice_focus_threshold=0.5,  # Optional: 0.0–1.0, higher = more aggressive
    ),
)
```

## Interruption handling

Barge-in — the user interrupting while the agent is speaking — is handled by Pipecat, and the signals that drive it depend on your turn detection mode.

- **Pipecat mode (`vad_force_turn_endpoint=True`).** Pipecat's local VAD and the Smart Turn analyzer detect the user starting to speak and interrupt the bot's TTS. AssemblyAI also emits `SpeechStarted` events as a backstop.
- **AssemblyAI's built-in turn detection (`vad_force_turn_endpoint=False`).** The service emits `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame` and uses AssemblyAI's `SpeechStarted` events for fast barge-in. Set `should_interrupt=False` (constructor argument) to disable barge-in entirely in this mode.

```json
{"type": "SpeechStarted", "timestamp": 14400, "confidence": 0.79}
```

On detection, Pipecat stops TTS playback and switches to listening. To reduce false interruptions from short backchannels (`"mhm"`, `"yeah"`, `"okay"`), keep your VAD threshold aligned with `vad_threshold` and lean on Pipecat's Smart Turn analyzer, which evaluates whether speech is a genuine turn rather than a filler.

## Dynamic configuration

Update settings mid-conversation by queueing an `STTUpdateSettingsFrame` with a settings delta — adapt to the conversation stage as it unfolds. See [stt-assemblyai.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/update-settings/stt/stt-assemblyai.py) for a complete working example.

```python expandable
from pipecat.frames.frames import STTUpdateSettingsFrame
from pipecat.services.assemblyai.stt import AssemblyAISTTService

# Update keyterms during the conversation
await worker.queue_frame(
    STTUpdateSettingsFrame(
        delta=AssemblyAISTTService.Settings(
            keyterms_prompt=["NewName", "NewCompany"],
        )
    )
)

# Widen the silence window during entity dictation
await worker.queue_frame(
    STTUpdateSettingsFrame(
        delta=AssemblyAISTTService.Settings(
            min_turn_silence=200,
            max_turn_silence=3000,  # Respected in AssemblyAI's built-in turn detection mode
        )
    )
)
```

  **`agent_context` is the only setting applied live.** Changing any other
  setting via `STTUpdateSettingsFrame` reconnects the AssemblyAI session to
  apply it (a brief interruption). To push conversation context without a
  reconnect, use the dedicated `stt.update_agent_context(...)` method — see
  [Conversation context](#conversation-context).

| Conversation stage                         | Adjustment                                                                       |
| ------------------------------------------ | -------------------------------------------------------------------------------- |
| Caller identification (names, account IDs) | Boost terms with `keyterms_prompt`                                               |
| Entity dictation (email, phone, address)   | Raise `max_turn_silence` to ~`2000`–`4000` ms, then lower it again afterward     |
| After each agent reply                     | Automatic — or push `agent_context` via `update_agent_context()`                 |
| Faster barge-in                            | Lower `interruption_delay`                                                       |

For more information, see [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream).

## Speaker diarization

Identify different speakers in multi-party conversations.

### Basic diarization

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        speaker_labels=True,
    ),
)
```

Speaker labels (e.g., `"A"`, `"B"`, `"C"`) are included in final transcripts.

### With custom formatting

Format transcripts with speaker labels for LLM context:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        speaker_labels=True,
    ),
    speaker_format="<{speaker}>{text}</{speaker}>",
)
```

**Format options:**

| Style    | Format string                   |
| -------- | ------------------------------- |
| XML      | `<{speaker}>{text}</{speaker}>` |
| Markdown | `**{speaker}**: {text}`         |
| Bracket  | `[{speaker}] {text}`            |

## Running your agent

### Development mode (local audio)

```bash
python your_agent.py
```

Speak into your microphone after hearing the greeting.

### Production with Daily

For production deployments, use the [Daily transport](https://docs.pipecat.ai/server/services/transport/daily) for WebRTC-based real-time audio/video. Your agent joins a Daily room as a participant and handles audio I/O through Daily's infrastructure.

### Telephony with Telnyx

When bridging phone calls through Pipecat (e.g., via Telnyx), the audio is 8 kHz, not 16 kHz. Match the transport sample rates:

```python
transport = TelnyxTransport(
    # ...
    audio_in_sample_rate=8000,
    audio_out_sample_rate=8000,
)
```

## Troubleshooting

| Issue                                | Cause                                                | Solution                                                                                          |
| ------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `universal-3-5-pro` not recognized   | `pipecat-ai` older than 1.4.0                        | Upgrade: `pip install -U "pipecat-ai[assemblyai]"`                                                |
| Turn over-segmentation               | `min_turn_silence` too low                           | Increase from `100` to `200`–`500`                                                                |
| Entities split across turns          | `max_turn_silence` too low (AssemblyAI mode)         | Increase `max_turn_silence` (e.g., `1500`–`3500`); in Pipecat mode, raise `min_turn_silence`      |
| Latency on non-terminal utterances   | `max_turn_silence` too high                          | Lower `max_turn_silence`                                                                          |
| Conversation context has no effect   | Non-U3-Pro model, or `previous_context_n_turns=0`    | Use a U3 Pro family model and leave `previous_context_n_turns` unset (or > 0)                      |
| Mid-session setting change drops audio | Reconnect on a non-`agent_context` setting change  | Expected — only `agent_context` updates live; use `update_agent_context()` for context            |
| Mis-heard names, brands, or jargon   | No vocabulary hints                                  | Add `keyterms_prompt`, or supply `prompt`/`agent_context` for context                             |
| Poor accuracy in noisy audio         | Background noise or room echo                        | Enable `voice_focus` (`near-field` or `far-field`)                                                |

## Migrating from another STT provider

To balance accuracy, latency, turn-taking, and interruption handling, map your current setup to AssemblyAI using the questions below.

### How are you detecting end-of-turn today?

| Today                                               | Recommended on AssemblyAI                                                                                                                |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Your STT provider's own end-of-turn model           | AssemblyAI's built-in turn detection: `vad_force_turn_endpoint=False` with `min_turn_silence=100`, `max_turn_silence=1000`.               |
| Silence / VAD only, with your own turn logic        | Pipecat mode (`vad_force_turn_endpoint=True`, default). VAD + Smart Turn decide turns; AssemblyAI returns finals ASAP.                    |
| You want the framework to own turn-taking           | Pipecat mode (default) — Pipecat's Smart Turn analyzer makes the turn decision.                                                          |

### Which model and settings are you migrating from?

| What you pass today                        | AssemblyAI equivalent                                                                                  |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| Current model (Deepgram, ElevenLabs, etc.) | `model="universal-3-5-pro"` (recommended flagship) or `"universal-3-5-pro"`                                     |
| Overall accuracy/latency tuning            | `mode="min_latency"` / `"balanced"` / `"max_accuracy"` — a one-line starting point before fine-tuning  |
| Endpointing / silence thresholds           | `min_turn_silence` (speculative end-of-turn) and `max_turn_silence` (forced end)                       |
| Custom vocabulary / keywords               | `keyterms_prompt=[...]`; broader domain context → `prompt`                                             |
| Provider-side conversation context         | Automatic — include the LLM context aggregator; seed greetings via `agent_context`                     |
| Formatting / punctuation toggles           | On by default — formatted transcripts always (`format_turns` does not apply)                           |
| Telephony / SIP routing                    | `sample_rate=8000` and `encoding="pcm_mulaw"` for 8 kHz telephony                                      |
| Client-side noise cancellation             | Drop it; use server-side [Voice Focus](#voice-focus) instead                                           |

Migrating a production deployment? [Talk to our team](https://www.assemblyai.com/contact/sales).

## Speech model comparison

Interested in using a different model?

| Feature                            | U3 Pro family <br/>(`universal-3-5-pro`, `universal-3-5-pro`) | universal-streaming-english | universal-streaming-multilingual |
| ---------------------------------- | ----------------------------------------------------- | --------------------------- | -------------------------------- |
| **_Turn Detection Modes_**         |                                                       |                             |                                  |
| Pipecat mode (VAD + Smart Turn)    | ✅                                                    | ✅                          | ✅                               |
| AssemblyAI turn detection mode     | ✅                                                    | ❌                          | ❌                               |
| **_Turn Detection Parameters_**    |                                                       |                             |                                  |
| `min_turn_silence`                 | ✅                                                    | ✅                          | ✅                               |
| `max_turn_silence`                 | ✅                                                    | ✅                          | ✅                               |
| `end_of_turn_confidence_threshold` | ❌                                                    | ✅ (1.0)                    | ✅ (1.0)                         |
| `continuous_partials`              | ✅                                                    | ❌                          | ❌                               |
| `interruption_delay`               | ✅                                                    | ❌                          | ❌                               |
| **_Advanced Features_**            |                                                       |                             |                                  |
| Keyterms boosting                  | ✅                                                    | ✅                          | ✅                               |
| Custom prompting (beta)            | ✅                                                    | ❌                          | ❌                               |
| Conversation context (carryover)   | ✅                                                    | ❌                          | ❌                               |
| Voice Focus                        | ✅                                                    | ❌                          | ❌                               |
| Speaker diarization                | ✅                                                    | ✅                          | ✅                               |
| Dynamic parameter updates          | ✅                                                    | ✅                          | ✅                               |
| **_Language Support_**             |                                                       |                             |                                  |
| Multilingual code switching        | ✅                                                    | ❌                          | ✅                               |
| Language detection                 | ✅                                                    | ❌                          | ✅                               |

**Legend:**

- ✅ Fully supported and recommended
- ❌ Not supported / Not used

  **The U3 Pro family is recommended** for all new voice agent implementations.
  The universal-streaming models are maintained for backward compatibility but
  lack the optimizations and features specifically designed for real-time
  conversational AI.

  The `end_of_turn_confidence_threshold` parameter is **not used** with the U3
  Pro family (it won't affect behavior). For universal-streaming models, Pipecat
  automatically sets it to `1.0` in Pipecat mode to disable semantic turn
  detection and ensure fast responses. You don't need to configure this
  parameter manually.

---

<a id="zapier-integration-with-assemblyai-docs-integrations-zapier-mdx"></a>

# Zapier Integration with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/zapier
Source: docs/integrations/zapier.mdx
Navigation: Overview > Use cases & integrations > Integrations > No-code tools
Description: Transcribe audio in Zapier using the AssemblyAI app.

Zapier is a workflow automation tool that lets you integrate various services together without requiring coding knowledge.
You can use our AI models to process audio data by transcribing it with speech recognition models and analyzing it with speech understanding models.
You can supply audio to the AssemblyAI app and connect the output of our models to other services in your Zaps.

## Quickstart

In your Zap editor, add an action, search for `AssemblyAI` and select the AssemblyAI app.

![Change action Zapier screen, with AssemblyAI in search box.](/assets/img/integrations/zapier/1-change-action.png)

Next, configure the action.
In the **App & event** tab, select **Transcribe** for the **Event** dropdown, then click **Continue**.

![App & event Zapier tab with Event field set to "Transcribe".](/assets/img/integrations/zapier/2-app-event.png)

Then, in the **Account** tab, click **Sign in** which will open a separate window.
In the window, enter your AssemblyAI API key in **API Key** field, and click **Yes, Continue to AssemblyAI**.
Back in the Zap editor, click **Continue**.

![Account Zapier tab where you are prompted to Connect AssemblyAI with a Sign in button.](/assets/img/integrations/zapier/3-account.png)

In the **Action** tab, enter the URL of the audio or video file you want to transcribe in the **Audio URL** field.
The URL has to be publicly accessible. Click **Continue**.

![Action Zapier tab where you are prompted to enter the Audio URL to transcribe using AssemblyAI.](/assets/img/integrations/zapier/4-action.png)

Finally, you can test the action. You can use all the fields returned by the action in subsequent steps.

  All AssemblyAI actions return sample data during testing instead of running
  the action. This makes it easier to build your Zaps, however, you have to test
  using normal Zap runs to verify everything is working correctly. Learn more
  about why we [return sample data during testing
  below](#testing-with-sample-data).

![Test Zapier tab where you can see the output of the AssemblyAI Transcribe action.](/assets/img/integrations/zapier/5-test.png)

## Zapier Actions

### Transcribe

Transcribe an audio file and wait until the transcript has completed or failed.
Configure the `Audio URL` field with the URL of the audio file you want to transcribe.
The `Audio URL` must be accessible by AssemblyAI's servers.

If you don't want to wait until the transcript is ready, change the `Wait until Transcript is Ready` parameter to `False`.

### Get Transcript

Retrieves a transcript by its ID.

### Get Subtitles

Export the transcript as SRT or VTT subtitles.

You can only invoke this action after the transcript is completed.

### Get Sentences

Retrieve the sentences of the transcript by its ID.

You can only invoke this action after the transcript is completed.

### Get Paragraphs

Retrieve the paragraphs of the transcript by its ID.

You can only invoke this action after the transcript is completed.

## Testing with sample data

A transcript goes through multiple phases to transcribe audio, reflected by different statuses.
The initial status is typically `processing`, and the final status is either `completed` or `error`. A transcript may also have a `queued` status if the job is waiting to be processed (for example, when you've exceeded your rate limit).
**During a normal Zap run**, the Transcribe event will wait until the transcript status is `completed`, and throw an error if the status is `error`.
Unfortunately, this is not the case during testing.

Because of a Zapier platform limitation, **during testing**, the Transcribe event will return a transcript before it has reached the `completed` status.
A transcript that does not have the `completed` status cannot be used in subsequent tests.

This way you can easily test using sample data, but you still have to use normal Zap runs to verify the end-to-end functionality.

## Additional resources

You can learn more about using Zapier with AssemblyAI in these resources:

- [How to generate subtitles for your videos using the AssemblyAI app for Zapier](https://www.assemblyai.com/blog/generate-subtitles-with-zapier)
- [How to get started with AssemblyAI on Zapier](https://help.zapier.com/hc/en-us/articles/16411509681933-How-to-get-started-with-AssemblyAI-on-Zapier)
- [AssemblyAI app on Zapier](https://zapier.com/apps/assemblyai/integrations)

---

<a id="integrate-make-with-assemblyai-docs-integrations-make-mdx"></a>

# Integrate Make with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/make
Source: docs/integrations/make.mdx
Navigation: Overview > Use cases & integrations > Integrations > No-code tools
Description: Use our Make (formerly Integromat) app to use AssemblyAI's speech AI in your Make scenarios.

[Make](https://make.com/) (formerly Integromat) is a workflow automation tool that lets you integrate various services together without requiring coding knowledge.

With the AssemblyAI app for Make, you can use our AI models to process audio data by transcribing it with speech recognition models, analyzing it with speech understanding models, and building generative features on top of it with LLMs.
You can supply audio to the AssemblyAI app and connect the output of our models to other services in your Make scenarios.

## Quickstart

Create or edit a scenario in Make.
Add a new module, search for AssemblyAI, and select the module that you want to use.

![Search for AssemblyAI modules in Make](/assets/img/integrations/make/search-module.png)

Select the module that you want to use.

![Add an AssemblyAI module in Make](/assets/img/integrations/make/add-module.png)

Create a new connection or select an existing one.
In **AssemblyAI API Key**, enter the API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home). In **Environment**, select the [region you would like to use](/pre-recorded-audio/select-the-region). You must select either US or EU. Click **Save**.

![Create a connection to AssemblyAI in Make](/assets/img/integrations/make/create-connection.png)

Finally, configure your AssemblyAI module. Continue reading to learn more about all the available modules.

![Configure an AssemblyAI module in Make](/assets/img/integrations/make/configure-module.png)

## AssemblyAI app modules

The AssemblyAI app for Make provides the following modules:

### Files

#### Upload a File

Upload an audio file to AssemblyAI so you can transcribe it.
You can pass the `Upload URL` output field to the `Audio URL` input field of [Transcribe an Audio File](#transcribe-an-audio-file) module.

### Transcripts

#### Transcribe an Audio File

Transcribe an audio file and wait until the transcript has completed or failed.
Configure the `Audio URL` field with the URL of the audio file you want to transcribe.
The `Audio URL` must be accessible by AssemblyAI's servers.
If you don't have a publicly accessible URL, you can use the [Upload a File](#upload-a-file) module to upload the audio file to AssemblyAI.

If you don't want to wait until the transcript is ready, change the `Wait until Transcript is Ready` parameter to `No` under **Show advanced settings**.

  Configure your desired [Speech Understanding models](/speech-understanding)
  when you create the transcript. The results of the models will be included in
  the transcript output.

#### Wait until Transcript is Ready

Wait for an existing transcript to be ready.
This module will complete when the status of the transcript changes to "completed" or "error".

#### Watch for Transcript Ready Notification

Create a webhook URL to receive a notification when a transcript is ready.
When the transcript is ready, the webhook will be invoked with the transcript status and ID.
The status will be "completed" or "error".

#### Get a Transcript

Retrieve a transcript by ID.

#### Get Paragraphs of a Transcript

Retrieve the paragraphs of a transcript.

You can only invoke this module after the transcript is completed.

#### Get Sentences of a Transcript

Retrieve the sentences of a transcript.

You can only invoke this module after the transcript is completed.

#### Get Subtitles for a Transcript

Create SRT or VTT subtitles for a transcript.

You can only invoke this module after the transcript is completed.

#### Get Redacted Audio of a Transcript

First, you need to configure PII audio redaction using these fields when you create the transcript:

- `Redact PII`: `Yes`
- `Redact PII Audio`: `Yes`
- `Redact PII Policies`: Configure at least one PII policy

Then, you can use this module to retrieve the redacted audio of the transcript.

You can only invoke this module after the transcript is completed.

#### Search for Words in a Transcript

Search for words in a transcript.

You can only invoke this module after the transcript is completed.

#### List Transcripts

Paginate over all transcripts.

#### Delete a Transcript

Delete a transcript by ID.
Deleting a transcript does not delete the transcript resource itself, but removes the data from the resource and marks it as deleted.

  You can only invoke this module after the transcript status is "completed" or
  "error".

### Other modules

#### Apply LLM Gateway to Transcripts

Send a message to [LLM Gateway](/llm-gateway/chat-completions).
Configure the `Model` field with the model you want to use, and the `Role` and `Content` fields to build your message.
See [Available models](/llm-gateway/available-models) for the list of supported models, and [Cloud Endpoints and Data Residency](/llm-gateway/cloud-endpoints-and-data-residency) for which models are available in each region.

To apply LLM Gateway to a transcript, map the `Text` output field from the [Transcribe an Audio File](#transcribe-an-audio-file) module into the `Content` field.

![Apply LLM Gateway to Transcripts module in Make](/assets/img/integrations/make/llmgw-module.png)

#### Make an API Call

Make your own REST API HTTP requests to the AssemblyAI API using your existing connection.

## Additional resources

You can learn more about using Make with AssemblyAI in these resources:

- [Iterate Over Speaker Labels in Make](/pre-recorded-audio/guides/make-speaker-labels)
- [Redact PII in Audio with Make and AssemblyAI](https://www.assemblyai.com/blog/redact-pii-audio-with-make/)
- [Make Integration page for AssemblyAI](https://www.make.com/en/integrations/assembly-ai)
- [AssemblyAI Make App Invitation Link](https://us1.make.com/app/invite/f21437a4d43b63efc8ee9aec385d9f10)

---

<a id="the-assemblyai-n8n-integration-docs-integrations-n-8-n-mdx"></a>

# The AssemblyAI n8n Integration

URL: https://www.assemblyai.com/docs/integrations/n-8-n
Source: docs/integrations/n-8-n.mdx
Navigation: Overview > Use cases & integrations > Integrations > No-code tools
Description: Integrate AssemblyAI with 1000+ apps and services using n8n's automation platform.

Unlock the full potential of AssemblyAI and [n8n's](https://n8n.io/) automation platform by connecting AssemblyAI's speech-to-text and speech understanding capabilities with over 1,000 apps, data sources, services, and n8n's built-in AI features.

The [AssemblyAI n8n integration](https://n8n.io/integrations/assemblyai/) is built and maintained by AssemblyAI (verified by n8n).

## Overview

This comprehensive tutorial walks you through building a complete AssemblyAI workflow within n8n Cloud that:

1. Watches for changes to a Google Drive folder
2. Once an audio file is added, submits a transcription request to AssemblyAI with the audio
3. Polls for the transcription process to complete
4. Once complete, processes the transcript into a text file with just the transcript text and speaker labels
5. Saves the processed transcript `.txt` file output to a Google Drive folder
6. Deletes the transcript from AssemblyAI's servers

  The [Google Drive n8n integration](https://n8n.io/integrations/google-drive/)
  is used in this example for file storage and triggering, but you can adapt the
  workflow to use other services like Dropbox, Amazon S3, FTP/SFTP, or webhooks
  based on your needs. For all available integrations, see the [n8n integrations
  page](https://n8n.io/integrations/).

## Prerequisites

Before you begin, you'll need:

- An [AssemblyAI API key](https://www.assemblyai.com/dashboard/home)
  - Signing up for an AssemblyAI account is free! An account on the free tier can transcribe up to **\$50 in transcription** and will be able to process up to **five files in parallel**. If you upgrade your account before using \$50 in transcription any unused amount will be retained on your account.
- An n8n account, either via [n8n Cloud](https://app.n8n.cloud/register) (recommended) or a self-hosted instance.
- A [Google Cloud account](https://console.cloud.google.com/) for Google Drive API access
  - A folder in Google Drive to monitor for new audio files (`/audio_files`)
  - A folder in Google Drive to save the formatted `.txt` transcript output (`/transcripts`)

## Credentials

<img src="/assets/img/integrations/n8n/credentials.png" alt="n8n credentials page for configuring Google Drive and AssemblyAI credentials" />

Before building your workflow, you'll need to configure credentials for both Google Drive and AssemblyAI in n8n.

### Google Drive OAuth Credentials

1. Go to your n8n credentials page
2. Click **Add Credential** and search for **"Google Drive OAuth2 API"**
3. Follow the setup process to connect your Google Drive account with Google Cloud Console:
   - [Video tutorial: Setting up Google Drive credentials](https://www.youtube.com/watch?v=FBGtpWMTppw&t=244s)
   - [n8n documentation: Google OAuth setup](https://docs.n8n.io/integrations/builtin/credentials/google/oauth-single-service/)
4. Save the credential

### AssemblyAI Credentials

1. Go to your n8n credentials page
2. Click **Add Credential** and search for **"AssemblyAI API"**
3. Enter your [AssemblyAI API key](https://www.assemblyai.com/dashboard/home)
4. Save the credential

Once both credentials are configured, you can use them throughout your workflow.

## Instructions

### Step 1: Set Up Your Workflow and Trigger

1. Log in to your n8n account (either [n8n Cloud](https://app.n8n.cloud/login) or your self-hosted instance)
2. Click the **+** icon or **New Workflow** to create a new workflow
3. Give your workflow a descriptive name (e.g., "AssemblyAI Transcription Pipeline")
4. Install the AssemblyAI node:
   - Click the **+** icon to open the nodes panel
   - Search for "AssemblyAI"
   - Click the **Install** button next to the AssemblyAI node

<img src="/assets/img/integrations/n8n/google_trigger.png" alt="n8n Google Drive trigger node configuration for detecting new audio files" />

The first step is to decide how n8n will know when your audio file is ready to transcribe. This is your workflow trigger. For this example, we'll use **Google Drive** to automatically trigger the workflow when a new audio file is added to a folder.

1. **Link your Google Drive account** with the credentials you created in the [Credentials](#credentials) step.
2. Add a **Google Drive** trigger node to your workflow
3. Select the trigger event: `On changes involving a specific folder`
4. Configure the trigger parameters:
   - **Credential to connect with:** Select your Google Drive credentials
   - **Poll Times:** Set Mode to Every Minute
   - **Trigger On:** Changes Involving a Specific Folder
   - **Folder:** From list + Select the folder in Google Drive to monitor
   - **Watch For:** File Created

When a new audio file is added to the specified Google Drive folder, the workflow will run.

  **Running the workflow**

  To test the workflow, you can manually trigger it at any time by clicking the orange **Execute Workflow** button in the n8n editor to populate the nodes with sample data.

For the Google Drive trigger to work as intended, you will need to deploy this workflow (to n8n cloud) and then add a new audio file to the monitored folder in Google Drive (`/audio_files`). This is handled in a [later step](/integrations/n-8-n#step-8-deploy).

For more information on supported audio file types, size limits, and using Google Drive with AssemblyAI, see:

- [Can I submit files stored in Google Drive?](/faq/can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive)
- [What Is the Recommended File Type for Using Your API?](/faq/what-is-the-recommended-file-type-for-using-your-api)
- [What types of audio URLs can I use with the API?](/faq/what-types-of-audio-urls-can-i-use-with-the-api)
- [Are there any limits on file size or duration?](/faq/are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api)

  **Important Google Drive Considerations**

  For the sake of simplicity, this tutorial uses a public Google Drive link, so you must ensure your file is **100 MB or less** and the Google Drive **folder settings are set to Public** (anyone with the link can view).

<img src="/assets/img/integrations/n8n/public_folder.png" alt="Google Drive folder sharing settings set to public access" />

If you'd like to transcribe private folders/files greater than 100MB, you'll need to add a Google Drive "Download file" node to download the binary data first, then upload it to AssemblyAI using the AssemblyAI "Upload a file" node.

### Step 2: Upload Audio File (Optional)

After the Google Drive trigger fires, you have the file's `webContentLink` available at `{{ $json.webContentLink }}`.

#### For files under 100MB

You can **skip this step** and use the public Google Drive link directly in [Step 3](/integrations/n-8-n#step-3-submit-transcription-request) without uploading. AssemblyAI accepts [all Google Drive files below 100MB](/faq/can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive).

#### For files over 100MB

Use the **AssemblyAI Upload** node:

1. Add a **Google Drive** "Download File" node and configure it to download the file by ID (you can get the file ID from the trigger response)
2. Add an **AssemblyAI** "Upload a file" node
3. Configure the credential with your [AssemblyAI API key](#prerequisites)
4. The upload will return an `upload_url` to use in the next step

### Step 3: Submit Transcription Request

<img src="/assets/img/integrations/n8n/create_transcript_node.png" alt="n8n AssemblyAI Create a transcription node added to the workflow" />

Now submit the transcription request with your desired features enabled.

Using the + sign linked to the Google Drive trigger, add an **AssemblyAI** `Create a transcription` node.

- **Credential to connect with**: Select your AssemblyAI Account credential (if you haven't set this up yet, see [Credentials](#credentials))
- **Resource**: Transcript
- **Operation**: Create
- **Audio URL**: `{{ $json.webContentLink }}`
  - You can also drag and drop the `webContentLink` field from the Google Drive Trigger response into this field

In **Additional Fields**, enable the following features for this tutorial:

- **Speaker Labels**: `true` (identifies different speakers through diarization)
- **Language Detection**: `true` (automatically detects the language, also known as ALD)

<img src="/assets/img/integrations/n8n/create_transcript_params.png" alt="Create a transcription node parameters with Speaker Labels and Language Detection enabled" />

You can explore all available features, such as sentiment analysis, entity detection, pii redaction, and more in our [API Reference](/api-reference/transcripts/submit).

The response will contain a `transcript_id` that you'll use to poll for completion.

### Step 4: Poll for Transcription Completion

<img src="/assets/img/integrations/n8n/polling.png" alt="n8n workflow polling loop that checks transcription status until completion" />

Since transcription is asynchronous, the transcription request returns immediately. You need to poll the API until the transcript is ready.

Add a `Wait` node and configure the wait parameters:

- **Resume**: After Time Interval
- **Wait Amount**: 3.00
- **Wait Unit**: Seconds

This will wait 3 seconds before checking the transcript's status.

Add an **AssemblyAI** `Get a transcription` node after the `Wait` node and configure the node parameters:

- **Credential to connect with**: Select your AssemblyAI Account credential (see [Credentials](#credentials))
- **Resource**: Transcript
- **Operation**: Get
- **Transcript ID**: `{{ $json.id }}`
  - You can drag and drop the `id` field from the `Create a transcription` response into this field

<img src="/assets/img/integrations/n8n/get_transcript_params.png" alt="AssemblyAI Get a transcription node parameters with Transcript ID field" />

Add a `Switch` node after the `Get a transcription` node and configure the switch to check the transcript's [status](/api-reference/transcripts/submit#response.body.status):

Add 4 routing rules, each with:

- **Value 1**: `{{ $json.status }}`
  - Or drag the `status` field from the `Get a transcription` response
- **Condition**: "is equal to"
- **Value 2**: Set one of these for each rule:
  1. `queued`
  2. `processing`
  3. `error`
  4. `completed`

  Most transcripts go directly to `processing`. The `queued` status is only returned when a job is actually waiting to be processed, such as when you've exceeded your rate limit. The routing rule for `queued` is included so the workflow handles that case correctly when it occurs.

<img src="/assets/img/integrations/n8n/status_switch.png" alt="n8n Switch node with routing rules for queued, processing, error, and completed statuses" />

Handle each switch node output based on the transcript status:

- **Output 0 (queued)**: Connect back to the **Wait** node to continue polling
- **Output 1 (processing)**: Connect back to the **Wait** node to continue polling
- **Output 2 (error)**: Add a **Stop and Error** node to halt the workflow
  - Note: You may want to add logging or resubmit the transcript instead
- **Output 3 (completed)**: Continue to the next step to process the completed transcript

By connecting the `processing` and `queued` outputs back to the `Get a transcription`, you create a polling loop that continues checking (every 3 seconds) until the transcript is complete or encounters an error.

When the transcript is complete, the response will contain the [full transcript data](/api-reference/transcripts/submit#response).

  For this tutorial, we are only using the `Create a transcription`, `Get a
  transcription`, and `Delete a transcription` actions from the AssemblyAI node.
  However, n8n's AssemblyAI integration supports **all available endpoints**,
  including uploading files, retrieving redacted audio, getting sentences and
  paragraphs, LLM Gateway, and more.

### Step 5: Process Transcript Output into a Human-Readable Transcript

<img src="/assets/img/integrations/n8n/python_formatting.png" alt="n8n Python Code node formatting the transcript with speaker labels" />

Once you have the completed transcript, we want to manipulate the data into a more usable format.

To pull out the transcript text with speaker labels:

Add a `Code` node (Python) to your workflow.

Use this code to format utterances into a readable transcript:

```python
transcript = _('Get a transcription').first().json
formatted_text = ''

if 'utterances' in transcript and transcript['utterances']:
    # Format with speaker labels
    for utterance in transcript['utterances']:
        formatted_text += f"Speaker {utterance['speaker']}: {utterance['text']}\n\n"
else:
    # Use plain text if no speaker labels
    formatted_text = transcript['text']

return {'formattedText': formatted_text}
```

This will output a `formattedText` field containing the transcript with speaker labels.

### Step 6: Save to Google Drive

To save the formatted transcript back to Google Drive:

<img src="/assets/img/integrations/n8n/upload_file.png" alt="n8n Convert to File node configured to output a text file" />

Add a `Convert to File` node and configure it:

- **Operation**: Convert to Text File
- **Text Input Field**: formattedText
- **Put Output File in Field**: data

<img src="/assets/img/integrations/n8n/upload_drive.png" alt="n8n Google Drive Upload file node configured to save the transcript" />

Add a Google Drive `Upload file` node and configure it:

- **Credential to connect with**: Select your Google Drive credential
- **Resource**: File
- **Operation**: Upload
- **Input Data Field Name**: data
- **File Name**: `{{ $('Get a transcription').item.json.id }}` (or drag and drop from input data)
- **Parent Drive**: From list > My Drive
- **Parent Folder**: From list > transcripts

The formatted transcript will be saved as a `.txt` file in your specified Google Drive folder:

<img src="/assets/img/integrations/n8n/google_transcript.png" alt="Formatted transcript text file saved in the Google Drive transcripts folder" />

### Step 7: Delete Transcript (Optional)

<img src="/assets/img/integrations/n8n/delete.png" alt="n8n AssemblyAI Delete a transcription node at the end of the workflow" />

Once you're done processing the transcript, as it is now saved in Google Drive, you can **optionally** delete it from AssemblyAI's servers.

Add an AssemblyAI `Delete a transcription` node at the end of your workflow.

Configure the node parameters:

- **Credential to connect with**: Select your AssemblyAI Account credential
- **Resource**: Transcript
- **Operation**: Delete
- **Transcript ID**: `{{ $('Get a transcription').item.json.id }}`
  - You can drag and drop the `id` field from the `Get a transcription` node response into this field

The transcript will be permanently deleted from AssemblyAI's servers.

### Step 8: Deploy

Once the transcript has been deleted, the workflow is complete! At this point, it should look like this:

<img src="/assets/img/integrations/n8n/full_tutorial.png" alt="Complete n8n AssemblyAI transcription workflow with all connected nodes" />

Now you can deploy the workflow to n8n Cloud by navigating back to the n8n Cloud **Overview** page, locating your workflow, and moving the slider icon to the `Active` position.

<img src="/assets/img/integrations/n8n/deploy.png" alt="n8n Cloud Overview page with the workflow slider set to Active" />

And we're done! Add a file to `/audio_files` in Google Drive, and within a few seconds to a few minutes (depending on the `Poll Times` set for the [Google Drive trigger](/integrations/n-8-n#choose-your-trigger) and the duration of the audio file), you should see the transcript appear in the `/transcripts` Google Drive folder.

## Conclusion

In this tutorial, you built a complete AssemblyAI transcription workflow in n8n that automatically processes audio files from Google Drive, submits them to AssemblyAI for transcription, polls for completion, formats the transcript with speaker labels, saves the output back to Google Drive as a `.txt` file, and finally deletes the transcript from AssemblyAI's servers.

This is just one simple idea, but the possibilities are endless! You can customize this workflow further by adding additional AssemblyAI features and products like sentiment analysis, entity detection, and LLM Gateway (see the **"What can you do with AssemblyAI?"** section of [this page](https://n8n.io/integrations/assemblyai/) for all available actions), or by integrating with other services like Slack, OpenAI, Supabase, and the hundreds of other [official n8n integrations](https://n8n.io/integrations/).

## Additional Resources

- [AssemblyAI n8n Integration Page](https://n8n.io/integrations/assemblyai/)
- [AssemblyAI n8n Integration GitHub Repo](https://github.com/gsharp-aai/n8n-nodes-assemblyai)

## Need some help?

If you get stuck, think something is broken or missing from our n8n integration, or just have some questions, we'd love to help you out! Contact our support team directly via support@assemblyai.com or open a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="the-postman-collection-for-the-assemblyai-api-docs-integrations-postman-mdx"></a>

# The Postman collection for the AssemblyAI API

URL: https://www.assemblyai.com/docs/integrations/postman
Source: docs/integrations/postman.mdx
Navigation: Overview > Use cases & integrations > Integrations > No-code tools
Description: Use the AssemblyAI API Postman collection to experiment with our API.

Postman is a user-friendly tool for testing API endpoints. The AssemblyAI API Postman collection contains all the HTTP requests you can make to the AssemblyAI API, so you don't need to write them yourself.

## Quickstart

Open the [AssemblyAI API collection](https://assembly.ai/postman) in Postman and click the **Fork** button. This will create a copy of the collection that you can edit.

![Click Fork on the AssemblyAI API collection](/assets/img/integrations/postman/1-fork-collection.png)

Fill out the form and click **Fork Collection**.

Next, click on the **Variables** tab and configure the `apiKey` variable with your AssemblyAI API key.
You can find your AssemblyAI API key in the [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).

![Configure AssemblyAI API key in Postman collection](/assets/img/integrations/postman/2-configure-variables.png)

Let's upload an audio file:

1. Open the **Files > Upload a media file** request
2. Switch to the **Body** tab
3. Change the dropdown from **none** to **binary**
4. Select an audio file of your choosing, or [download this sample audio file](https://assembly.ai/nbc.mp3)
5. Click the **Send** button

![Send upload audio file to AssemblYAI API HTTP request](/assets/img/integrations/postman/3-upload-file-request.png)

Inspect the **Body** of the response and copy the `upload_url` value.

![The response for uploading an audio file to AssemblYAI API](/assets/img/integrations/postman/4-upload-file-response.png)

Now that the audio file is uploaded, you can transcribe the audio file.

1. Open the **Transcripts > Transcribe audio** request
2. Switch to the **Body** tab
3. Find the `audio_url` property and update it to the `upload_url` value from the previous request.
4. Remove all the other properties. Optionally, you can leave any property that you do want to use.
5. Click the **Send** button

![Create a transcript HTTP request](/assets/img/integrations/postman/5-create-transcript-request.png)

Inspect the **Body** of the response and copy the `id` value.

![Create a transcript HTTP response](/assets/img/integrations/postman/6-create-transcript-response.png)

The transcription job will take longer depending on the duration of the file.
You need to check the `status` property to check if a transcript is ready.
The `status` typically goes from `processing` to `completed`. The `status` can also be `queued` if the job is waiting to be processed (for example, when you've exceeded your rate limit), in which case it will move to `processing` once a slot is available.
The `status` can also become `error` at any point. If an error occurs, you can find the error message under the `error` property.

1. Open the **Transcripts > Get transcript** request
2. Find the `transcript_id` under **Path variables** and update the value with the `id` value from the previous request.
3. Remove all the other properties. Optionally, you can leave any property that you do want to use.
4. Click the **Send** button

![Get a transcript HTTP request and response](/assets/img/integrations/postman/7-get-transcript.png)

Inspect the **Body** of the response to check if the `status` is `completed` or `error`.
If not, resend the request until it is `completed` or `error`.

Now that you have a completed transcript, you can send these other HTTP requests with your current transcript ID:

- Transcripts
  - Get subtitles for transcript
  - Get sentences in transcript
  - Get paragraphs in transcript
  - Search words in transcript
  - Get redacted audio (if PII audio redaction is enabled)

---

<a id="build-a-zoom-real-time-transcription-bot-with-recall-ai-docs-integrations-recall-mdx"></a>

# Build a Zoom Real-time transcription bot with Recall.ai

URL: https://www.assemblyai.com/docs/integrations/recall
Source: docs/integrations/recall.mdx
Navigation: Overview > Use cases & integrations > Integrations > Meeting transcriber tools
Description: Build a Zoom Real-time transcription bot with Recall.ai documentation.

A real-time transcription bot that integrates [Recall.ai](https://www.recall.ai/assemblyai) with [AssemblyAI](https://www.assemblyai.com) to provide live transcription of Zoom meetings.

## Quickstart

```bash
# 1. Clone and install
git clone https://github.com/AssemblyAI/assemblyai-recallai-zoom-bot.git
cd assemblyai-recallai-zoom-bot
npm install

# 2. Run ngrok and copy the ngrok URL
# ngrok http 8000

# 3. Configure your .env file and edit it with your API keys and ngrok URL
cp .env.example .env

# 4a. Open a new terminal and run
# node webhook.js

# 4b. Open another terminal and run
# node zoomBot.js
```

## Prerequisites

- [Node.js](https://nodejs.org/en/) (v14 or higher)
- [ngrok](https://ngrok.com) - [Installation guide](https://ngrok.com/download)
- Recall.ai API key with AssemblyAI configured

## Step-by-step

Follow this step-by-step guide to set up and run the transcription bot.

### Step 1: Get your API keys

**1.1 Choose your Recall.ai region and get your API Key:**

| Region           | Dashboard                                                    | RECALL_REGION    |
| ---------------- | ------------------------------------------------------------ | ---------------- |
| US Pay-as-you-go | [us-west-2.recall.ai](https://us-west-2.recall.ai)           | `us-west-2`      |
| US Monthly plan  | [us-east-1.recall.ai](https://us-east-1.recall.ai)           | `us-east-1`      |
| EU               | [eu-central-1.recall.ai](https://eu-central-1.recall.ai)     | `eu-central-1`   |
| Japan            | [ap-northeast-1.recall.ai](https://ap-northeast-1.recall.ai) | `ap-northeast-1` |

**1.2 Configure AssemblyAI in your Recall.ai dashboard:**

- Get an AssemblyAI API key from [assemblyai.com](https://www.assemblyai.com)
- In your Recall.ai dashboard (same region as step 1.1), navigate to the transcription providers section
- Add your AssemblyAI API key to enable AssemblyAI as a transcript provider

  This step is required for the bot to work with AssemblyAI transcription!

### Step 2: Clone the example repo and install dependencies

Open a new terminal and run the following commands:

```bash
git clone https://github.com/AssemblyAI/assemblyai-recallai-zoom-bot.git
cd assemblyai-recallai-zoom-bot
npm install
```

### Step 3: Configure environment

Set up your environment variables using the example `.env` file in the repo:

```bash
cp .env.example .env
```

Edit `.env` with your Recall values.

```bash
RECALL_API_KEY=your_recall_api_key
RECALL_REGION=us-west-2
```

Set `RECALL_REGION` to according to your Recall region in Step 1.1

### Step 4: Start ngrok tunnel

In a new terminal, start ngrok to create a public URL for your webhook:

```bash
ngrok http 8000
```

The output should look like this:

```
ngrok by @inconshreveable

Session Status                online
Account                       your-account@email.com
Version                       2.3.40
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    https://abc123.ngrok.io -> http://localhost:8000
Forwarding                    http://abc123.ngrok.io -> http://localhost:8000
```

Copy the https URL (e.g., `https://abc123.ngrok.io`) - you'll need this for the next step.

### Step 5: Update Webhook URL

Edit your `.env` file and set `WEBHOOK_URL` to your ngrok URL:

```bash
WEBHOOK_URL=https://abc123.ngrok.io
```

Use the exact URL from ngrok output (no trailing slash)

### Step 6: Start webhook server

In another terminal, start the webhook server (receives transcripts):

```bash
node webhook.js
```

### Step 7: Start the bot CLI

In another terminal, start the bot CLI (manages meeting connection):

```bash
node zoomBot.js
```

Once you run this command, you'll see this output:

```
[BOT] Starting Recall.ai bot for Zoom → AssemblyAI integration
[BOT] Configured for region: us-west-2
[BOT] API endpoint: https://us-west-2.recall.ai/api/v1
[BOT] Starting application...
[BOT] Validating configuration...
[BOT] ✓ Configuration valid
[BOT] ✓ Region: us-west-2
[BOT] ✓ Webhook: https://abc123.ngrok.io
[BOT] Ready to join meeting and start transcription
What is your meeting URL?:
```

### Step 8: Join meeting and start transcription

Enter your Zoom meeting URL when prompted:

```
What is your meeting URL?: https://zoom.us/j/123456789
```

And your terminal will look like this:

```
[BOT] Creating bot for meeting: https://zoom.us/j/123456789
[BOT] Webhook endpoint: https://abc123.ngrok.io/meeting_transcript
[API] POST /bot - Creating bot with AssemblyAI integration
[API] Bot created successfully
[BOT] Bot ID: bot_abc123
[BOT] Status: joining_call
[BOT] ✓ Bot deployed successfully
[BOT] Bot is joining meeting and will start sending transcripts to webhook
[BOT] Check Terminal 2 for real-time transcripts
Type "STOP" to end transcription:
```

In your second terminal (running `webhook.js`), you should see a meeting transcript of your participants:

```
[WEBHOOK] Server running on port 8000
[WEBHOOK] Ready to receive transcripts from Recall.ai
[WEBHOOK] Integration: Zoom → Recall.ai → AssemblyAI → This webhook
[TRANSCRIPT] PARTIAL - John Doe: Hello everyone
[TRANSCRIPT] PARTIAL - John Doe: Hello everyone, welcome to
[TRANSCRIPT] FINAL - John Doe: Hello everyone, welcome to today's meeting.
[TRANSCRIPT] FINAL - Jane Smith: Thanks for joining, let's get started with the agenda.
```

### Step 9: Stop transcription

To stop transcribing, type "STOP" to end transcription on your third terminal running `node zoomBot.js`:

```
Type "STOP" to end transcription: STOP
```

## Troubleshooting

### Environment Variable Errors

#### `RECALL_API_KEY not found in .env file`

Solution: Make sure you copied `.env.example` to `.env` and added your API key.

#### `RECALL_REGION must be one of: us-west-2, us-east-1, eu-central-1, ap-northeast-1`

Solution: Check your `.env` file and ensure `RECALL_REGION` matches where you got your API key

### Bot Creation Errors

#### AssemblyAI not configured

```bash
Failed to create bot: { recording_config: { transcript: { provider: [Object] } } }
```

Solution: Configure AssemblyAI in your Recall.ai dashboard (Step 1.2)

### Webhook Issues

#### No transcripts appearing

```bash
[WEBHOOK] Server running on port 8000
[WEBHOOK] Ready to receive transcripts from Recall.ai
[WEBHOOK] Integration: Zoom → Recall.ai → AssemblyAI → This webhook
(no transcript output)
```

Solution:

- Verify `WEBHOOK_URL` in `.env` matches your ngrok URL exactly
- Ensure ngrok is still running (it may timeout after inactivity)
- Check that the webhook server was started before the bot

#### ngrok connection refused

```bash
Failed to complete tunnel connection
```

Solution:

- Restart ngrok: `ngrok http 8000`
- Update `WEBHOOK_URL` in `.env` with the new ngrok URL
- Ensure port 8000 is available

### Common network issues

#### Timeout connecting to Recall.ai

```bash
timeout of 10000ms exceeded
```

Solution:

- Check your internet connection
- Verify your `RECALL_REGION` is correct
- Try again after a few seconds

#### Bot appears to join but no transcripts

Solution:

- Ensure people are speaking in the meeting
- Check that meeting participants have unmuted their microphones
- Verify AssemblyAI is properly configured in Recall.ai dashboard

**Stuck?** Contact our support team at support@assemblyai.com or create a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="transcribe-your-zoom-meetings-docs-integrations-zoom-rtms-mdx"></a>

# Transcribe Your Zoom Meetings

URL: https://www.assemblyai.com/docs/integrations/zoom-rtms
Source: docs/integrations/zoom-rtms.mdx
Navigation: Overview > Use cases & integrations > Integrations > Meeting transcriber tools
Description: Transcribe Your Zoom Meetings documentation.

This guide creates a Node.js service that captures audio from Zoom Real-Time Media Streams (RTMS) and provides both real-time and asynchronous transcription using AssemblyAI.

  **Zoom RTMS Documentation**

  For complete Zoom RTMS documentation, visit
  https://developers.zoom.us/docs/rtms/

## Features

- **Real-time Transcription**: Live transcription during meetings using AssemblyAI's streaming API
- **Asynchronous Transcription**: Complete post-meeting transcription with advanced features
- **Flexible Audio Modes**:
  - Mixed stream (all participants combined)
  - Individual participant streams transcribed
- **Multichannel Audio Support**: Separate channels for different participants
- **Configurable Processing**: Enable/disable real-time or async transcription independently

## Setup

### Prerequisites

- Node.js 16+
- FFmpeg installed on your system
- Zoom RTMS Developer Preview access
- AssemblyAI API key
- ngrok (for local development and testing)

### Installation

1. **Clone the example repository and install dependencies**:

```bash
git clone https://github.com/zkleb-aai/assemblyai-zoom-rtms.git
cd assemblyai-zoom-rtms
npm install
```

2. **Configure environment variables**:

```bash
cp .env.example .env
```

Fill in your `.env` file:

```env
# Zoom Configuration
ZM_CLIENT_ID=your_zoom_client_id
ZM_CLIENT_SECRET=your_zoom_client_secret
ZOOM_SECRET_TOKEN=your_webhook_secret_token

# AssemblyAI Configuration
ASSEMBLYAI_API_KEY=your_assemblyai_api_key

# Service Configuration
PORT=8080
REALTIME_ENABLED=true
REALTIME_MODE=mixed
ASYNC_ENABLED=true
AUDIO_CHANNELS=mono
AUDIO_SAMPLE_RATE=16000
TARGET_CHUNK_DURATION_MS=100
```

### Local development with ngrok

For testing and development, you can use ngrok to expose your local server to the internet:

1. **Install ngrok**: Download from [ngrok.com](https://ngrok.com/) or install via package manager:

   ```bash
   # macOS
   brew install ngrok

   # Windows (chocolatey)
   choco install ngrok

   # Or download directly from ngrok.com
   ```

2. **Start your local server**:

   ```bash
   npm start
   ```

3. **In a separate terminal, start ngrok**:

   ```bash
   ngrok http 8080
   ```

4. **Copy the ngrok URL**: ngrok will display a forwarding URL like:

   ```
   Forwarding    https://example-abc123.ngrok-free.app -> http://localhost:8080
   ```

5. **Use the ngrok URL in your Zoom app webhook configuration**:
   ```
   https://example-abc123.ngrok-free.app/webhook
   ```

### Configuration options

#### Real-time transcription

- `REALTIME_ENABLED`: Enable/disable live transcription (default: `true`)
- `REALTIME_MODE`:
  - `mixed`: Single stream with all participants combined
  - `individual`: Separate streams per participant

#### Audio settings

- `AUDIO_CHANNELS`: `mono` or `multichannel`
- `AUDIO_SAMPLE_RATE`: Audio sample rate in Hz (default: `16000`)
- `TARGET_CHUNK_DURATION_MS`: Audio chunk duration for streaming (default: `100`)

#### Async transcription

- `ASYNC_ENABLED`: Enable/disable post-meeting transcription (default: `true`)

## Usage

### Start the service

```bash
npm start
```

The service will start on the configured port (default: 8080) and display:

```
🎧 Zoom RTMS to AssemblyAI Transcription Service
📋 Configuration:
   Real-time: ✅ (mixed)
   Audio: mono @ 16000Hz
   Async: ✅
🚀 Server running on port 8080
📡 Webhook endpoint: http://localhost:8080/webhook
```

### Configure Zoom webhook

1. In your Zoom App configuration, set the webhook endpoint to:

   ```
   # For production
   https://your-domain.com/webhook

   # For local development with ngrok
   https://example-abc123.ngrok-free.app/webhook
   ```

2. Subscribe to these events:
   - `meeting.rtms_started`
   - `meeting.rtms_stopped`

### Testing with ngrok

When using ngrok for testing:

1. **Keep ngrok running**: The ngrok tunnel must remain active during testing
2. **Update webhook URL**: If you restart ngrok, you'll get a new URL that needs to be updated in your Zoom app configuration
3. **Monitor ngrok logs**: ngrok shows incoming webhook requests in its terminal output
4. **Free tier limitations**: The free ngrok tier has some limitations; consider upgrading for heavy testing

### Real-time output

During meetings, you'll see live transcription:

```
🚀 AssemblyAI session started: [abc12345]
🎙️ [abc12345] Hello everyone, welcome to the meeting
📝 [abc12345] FINAL: Hello everyone, welcome to the meeting.
```

### Post-meeting files

After each meeting, the service generates:

- `transcript_[meeting_uuid].json` - Full AssemblyAI response with metadata
- `transcript_[meeting_uuid].txt` - Plain text transcript

## Advanced configuration

### AssemblyAI features

Modify the `ASYNC_CONFIG` object in the code to enable additional features:

```javascript
const ASYNC_CONFIG = {
  speaker_labels: true, // Speaker identification
  auto_chapters: true, // Automatic chapter detection
  sentiment_analysis: true, // Sentiment analysis
  entity_detection: true, // Named entity recognition
  redact_pii: true, // PII redaction
  summarization: true, // Auto-summarization
  auto_highlights: true, // Key highlights
};
```

See [AssemblyAI's API documentation](/api-reference/transcripts/submit) for all available options.

### Audio processing modes

#### Mixed mode (default)

- Single audio stream combining all participants
- Most efficient for general transcription
- Best for meetings with clear speakers

#### Individual mode

- Separate transcription stream per participant
- Better speaker attribution
- Higher resource usage

#### Multichannel audio

- Separate audio channels for different participants
- Enables advanced speaker separation
- Requires `AUDIO_CHANNELS=multichannel`

## API endpoints

### `POST` /webhook

Handles Zoom RTMS webhook events:

- URL validation
- Meeting start/stop events
- Automatic RTMS connection setup

## Error handling

The service includes comprehensive error handling:

- Automatic reconnection for dropped connections
- Graceful cleanup on meeting end
- Audio buffer flushing to prevent data loss
- Temporary file cleanup

## Monitoring

### Real-time logs

- Connection status updates
- Audio processing statistics
- Transcription progress
- Error notifications

### Example log output

```
📡 Connecting to Zoom signaling for meeting abc123
✅ Zoom signaling connected for meeting abc123
🎵 Connecting to Zoom media for meeting abc123
✅ Zoom media connected for meeting abc123
🚀 Started audio streaming for meeting abc123
🎵 [abc12345] 100 chunks, 32768 bytes, 10.2s
📝 [abc12345] FINAL: This is the final transcription.
```

### Development workflow

1. Start your local server: `npm start`
2. Start ngrok in another terminal: `ngrok http 8080`
3. Update your Zoom app webhook URL with the ngrok URL
4. Test with Zoom meetings
5. Monitor logs in both your app and ngrok terminals

---

<a id="integrate-telnyx-with-assemblyai-docs-integrations-telnyx-mdx"></a>

# Integrate Telnyx with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/telnyx
Source: docs/integrations/telnyx.mdx
Navigation: Overview > Use cases & integrations > Integrations > Telephony tools
Description: Build voice agents with Telnyx and AssemblyAI using Pipecat or LiveKit.

[Telnyx](https://telnyx.com/) is a global connectivity platform that provides programmable voice, messaging, and wireless services. By combining Telnyx with AssemblyAI, you can build real-time voice agents with industry-leading speech recognition accuracy and advanced turn detection.

This guide shows you how to integrate Telnyx with AssemblyAI using two popular voice agent orchestrators: LiveKit and Pipecat.

## LiveKit Telephony Integration

[LiveKit](https://livekit.io/) is a real-time communication platform for building voice, video, and data applications. You can integrate Telnyx SIP trunking with LiveKit to enable phone calls to your voice agents that use AssemblyAI for speech recognition.

### How it works

Telnyx SIP trunks bridge phone calls into LiveKit rooms as special SIP participants. Your LiveKit agent (configured with AssemblyAI STT) connects to the room and interacts with the caller. The flow is:

1. **Phone call** → Telnyx SIP trunk
2. **Telnyx** → LiveKit room (creates SIP participant)
3. **LiveKit agent** (with AssemblyAI STT) → joins room and handles conversation

### Before you begin

- **Telnyx account**: Create an account and purchase a phone number at [telnyx.com](https://telnyx.com)
- **LiveKit project**: Get your SIP URI from your [LiveKit project settings](https://cloud.livekit.io)
- **LiveKit CLI**: Install the [LiveKit CLI](https://docs.livekit.io/home/cli/cli-setup/) on your computer
- **Environment variables**: Configure `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET`, and `ASSEMBLYAI_API_KEY`

### Step 1: Configure Telnyx SIP connection

Configure your Telnyx SIP connection to route calls to LiveKit. Follow the detailed setup guide:

[Telnyx LiveKit SIP Configuration Guide](https://developers.telnyx.com/docs/voice/sip-trunking/livekit-configuration-guide)

Key steps:

1. Create a SIP connection in Telnyx Mission Control Portal
2. Set connection type to FQDN and provide your LiveKit SIP URI
3. Configure outbound call authentication (username/password)
4. Assign your phone number(s) to the SIP connection

### Step 2: Configure LiveKit SIP trunks

Create inbound and outbound SIP trunks in LiveKit using the LiveKit CLI.

#### Inbound trunk

Create `inboundTrunk.json`:

```json
{
  "trunk": {
    "name": "Telnyx Inbound Trunk",
    "numbers": ["YOUR_TELNYX_NUMBER"]
  }
}
```

Create the trunk:

```bash
lk sip inbound create inboundTrunk.json
```

Save the returned `SIPTrunkID` for the next step.

#### Dispatch rule

Create `dispatchRule.json` to route incoming calls to your agent:

```json
{
  "name": "Agent Dispatch Rule",
  "trunk_ids": ["YOUR_TRUNK_ID"],
  "rule": {
    "dispatchRuleIndividual": {
      "roomPrefix": "call-"
    }
  },
  "roomConfig": {
    "agents": [
      {
        "agentName": "my-telephony-agent"
      }
    ]
  }
}
```

Create the dispatch rule:

```bash
lk sip dispatch create dispatchRule.json
```

This automatically dispatches your agent to each incoming call in a new room.

#### Outbound trunk (optional)

For outbound calling, create `outboundTrunk.json`:

```json
{
  "trunk": {
    "name": "Telnyx Outbound Trunk",
    "address": "sip.telnyx.com",
    "numbers": ["YOUR_TELNYX_NUMBER"],
    "auth_username": "YOUR_OUTBOUND_USER",
    "auth_password": "YOUR_OUTBOUND_PASS"
  }
}
```

Create the trunk:

```bash
lk sip outbound create outboundTrunk.json
```

### Step 3: Build your LiveKit agent with AssemblyAI

Create a LiveKit agent that uses AssemblyAI for speech recognition. Once your SIP trunks and dispatch rules are configured, no special telephony code is required—the agent simply joins the room when a call comes in.

#### Prerequisites

Setup and activate a virtual environment:

```bash
python -m venv venv
source venv/bin/activate
```

Install the required dependencies:

```bash
pip install livekit-agents livekit-plugins-assemblyai livekit-plugins-openai livekit-plugins-silero livekit-plugins-rime
```

Download model files:

```bash
python agent.py download-files
```

#### Agent code

Here's a minimal agent using AssemblyAI STT:

```python expandable
import asyncio
from dotenv import load_dotenv
from livekit import rtc
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm
from livekit.agents.voice_assistant import VoiceAssistant
from livekit.plugins import assemblyai, openai, silero, rime

load_dotenv()

async def entrypoint(ctx: JobContext):
    initial_ctx = llm.ChatContext().append(
        role="system",
        text=(
            "You are a helpful voice assistant. Your interface with users will be voice. "
            "Use short and concise responses, avoiding unpronounceable punctuation."
        ),
    )

    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

    assistant = VoiceAssistant(
        vad=silero.VAD.load(),
        stt=assemblyai.STT(),
        llm=openai.LLM(),
        tts=rime.TTS(),
        chat_ctx=initial_ctx,
    )

    assistant.start(ctx.room)

    await asyncio.sleep(1)
    await assistant.say("Hello! How can I help you today?", allow_interruptions=True)

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```

#### Environment variables

Set the following environment variables:

```bash
export LIVEKIT_URL="wss://your-livekit-server"
export LIVEKIT_API_KEY="YOUR_LIVEKIT_API_KEY"
export LIVEKIT_API_SECRET="YOUR_LIVEKIT_API_SECRET"
export ASSEMBLYAI_API_KEY="YOUR_ASSEMBLYAI_API_KEY"
export OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
export RIME_API_KEY="YOUR_RIME_API_KEY"
```

#### Run the agent

Start your agent:

```bash
python agent.py dev
```

Now dial your Telnyx phone number to test the integration. The call will be routed through Telnyx → LiveKit → your agent with AssemblyAI speech recognition.

### Testing outbound calls

To test an outbound call, create `sipParticipant.json`:

```json
{
  "sip_trunk_id": "YOUR_OUTBOUND_TRUNK_ID",
  "sip_call_to": "+15105551234",
  "room_name": "test-outbound-call",
  "participant_identity": "outbound-test",
  "participant_name": "Test Call"
}
```

Place the call:

```bash
lk sip participant create sipParticipant.json
```

### Troubleshooting

**Call connects but agent never speaks**: Verify your dispatch rule includes `roomConfig.agents` with the correct `agentName`. Without this, the agent won't be automatically dispatched to the room.

**Audio quality issues**: Check your Telnyx SIP connection settings and ensure proper codec configuration. See the [Telnyx troubleshooting guide](https://developers.telnyx.com/docs/voice/sip-trunking/livekit-configuration-guide#troubleshooting).

## Pipecat Integration

[Pipecat](https://github.com/pipecat-ai/pipecat) is an open-source framework for building voice and multimodal conversational AI agents. You can integrate Telnyx Media Streaming with Pipecat to enable phone calls to your voice agents that use AssemblyAI for speech recognition.

This guide covers both dial-in (users call your number) and dial-out (your bot calls users) functionality.

### Prerequisites

- **Telnyx account**: Create an account and purchase phone numbers at [telnyx.com](https://telnyx.com)
- **Public server or tunnel**: For dial-out, you'll need a public-facing server or tunneling service like [ngrok](https://ngrok.com/)
- **API keys**: Get API keys for AssemblyAI, OpenAI (or other LLM), and your preferred TTS service

Install the required dependencies:

```bash
pip install pipecat-ai python-dotenv loguru
```

Set up your environment variables:

```bash
TELNYX_API_KEY=your_telnyx_api_key
ASSEMBLYAI_API_KEY=your_assemblyai_api_key
OPENAI_API_KEY=your_openai_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key  # or other TTS provider
```

### How it works

**Dial-in flow**:

1. User calls your Telnyx number
2. Telnyx executes your TeXML application which establishes a WebSocket connection
3. Telnyx opens a WebSocket to your server with real-time audio and call metadata
4. Your bot processes the audio using the Pipecat pipeline with AssemblyAI STT
5. The bot responds with audio sent back to Telnyx over WebSocket
6. Telnyx plays the audio to the caller in real-time

**Dial-out flow**:

1. Your application triggers a dial-out via API
2. Server initiates a Telnyx call using the Call Control API
3. Telnyx establishes the call and opens a WebSocket connection
4. Your bot joins the WebSocket and sets up the pipeline
5. The recipient answers and is connected to your bot
6. The bot handles the conversation with real-time audio streaming

### Dial-in Setup

#### Step 1: Create TeXML application

Telnyx uses TeXML (Telnyx Extensible Markup Language) to control call flow. Create a TeXML application that establishes a WebSocket connection to your bot:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>
```

The `bidirectionalMode="rtp"` parameter enables real-time audio streaming in both directions.

**Custom data with query parameters**: You can pass custom data to your bot by adding query parameters to the WebSocket URL:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws?user_id=12345&session_type=support" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>
```

#### Step 2: Configure Telnyx phone number

1. Go to the [Telnyx Portal](https://portal.telnyx.com)
2. Navigate to Voice → Programmable Voice → TeXML Applications
3. Create a new TeXML Application with your WebSocket URL
4. Assign the TeXML Application to your phone number

#### Step 3: Build your bot

Here's a complete dial-in bot using AssemblyAI for speech recognition:

```python expandable
import os
from dotenv import load_dotenv
from loguru import logger

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.runner.utils import parse_telephony_websocket
from pipecat.serializers.telnyx import TelnyxFrameSerializer
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.deepgram.tts import DeepgramTTSService
from pipecat.transports.websocket.fastapi import (
    FastAPIWebsocketTransport,
    FastAPIWebsocketParams,
)

load_dotenv()

async def run_bot(websocket):
    """Run the voice agent bot with AssemblyAI STT."""

    # Parse Telnyx WebSocket data - automatically extracts call information
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Extract call information (automatically provided by Telnyx)
    stream_id = call_data["stream_id"]
    call_control_id = call_data["call_control_id"]
    outbound_encoding = call_data["outbound_encoding"]
    from_number = call_data["from"]  # Caller's number
    to_number = call_data["to"]      # Your Telnyx number

    logger.info(f"Incoming call from {from_number} to {to_number}")

    # Create Telnyx serializer with call details
    serializer = TelnyxFrameSerializer(
        stream_id=stream_id,
        call_control_id=call_control_id,
        api_key=os.getenv("TELNYX_API_KEY"),
    )

    # Configure WebSocket transport
    transport = FastAPIWebsocketTransport(
        websocket=websocket,
        params=FastAPIWebsocketParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            add_wav_header=False,
            vad_analyzer=SileroVADAnalyzer(),
            serializer=serializer,
        ),
    )

    # Configure AI services
    stt = AssemblyAISTTService(api_key=os.getenv("ASSEMBLYAI_API_KEY"))
    llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-5-mini")
    tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"))

    # Customize bot behavior based on call information
    messages = [
        {
            "role": "system",
            "content": f"You are a helpful voice assistant. The caller is calling from {from_number}. Keep responses concise and conversational."
        }
    ]

    context = OpenAILLMContext(messages)
    context_aggregator = llm.create_context_aggregator(context)

    # Build pipeline
    pipeline = Pipeline([
        transport.input(),
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
    ])

    task = PipelineTask(
        pipeline,
        params=PipelineParams(
            allow_interruptions=True,
            audio_in_sample_rate=8000,
            audio_out_sample_rate=8000,
        ),
    )

    @transport.event_handler("on_client_disconnected")
    async def on_client_disconnected(transport, client):
        logger.info("Call ended")
        await task.queue_frame(EndFrame())

    # Run the pipeline
    runner = PipelineRunner()
    await runner.run(task)
```

See the [complete dial-in example](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/inbound) for full implementation details including server setup.

### Dial-out Setup

Dial-out allows your bot to initiate calls to phone numbers using Telnyx's outbound calling capabilities.

#### How dial-out works

1. Your application triggers a dial-out (via API call or user action)
2. Server initiates a Telnyx call using the Call Control API
3. Telnyx establishes the call and opens a WebSocket connection
4. Your bot joins the WebSocket and sets up the pipeline
5. The recipient answers and is connected to your bot
6. The bot handles the conversation with real-time audio streaming

#### Bot configuration for dial-out

The dial-out bot configuration is similar to dial-in. Telnyx automatically provides call information in the WebSocket messages:

```python
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Extract call information
stream_id = call_data["stream_id"]
call_control_id = call_data["call_control_id"]
from_number = call_data["from"]  # Your Telnyx number
to_number = call_data["to"]      # Target number you're calling

# Customize bot behavior for outbound calls
greeting = f"Hi! This is an automated call from {from_number}. How are you today?"
```

The transport and pipeline configuration are identical to dial-in. See the [complete dial-out example](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/outbound) for full server implementation with outbound call creation.

### Key Features

**Audio format**: Telnyx Media Streaming uses 8kHz mono audio with 16-bit PCM encoding. Configure your pipeline accordingly:

```python
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
        allow_interruptions=True,
    ),
)
```

**Automatic call termination**: When you provide Telnyx API credentials to the `TelnyxFrameSerializer`, it automatically ends calls when your pipeline ends:

```python
serializer = TelnyxFrameSerializer(
    stream_id=stream_id,
    call_control_id=call_control_id,
    api_key=os.getenv("TELNYX_API_KEY"),  # Enables auto-termination
)
```

**Built-in call information**: Unlike other providers, Telnyx automatically includes caller information (to/from numbers) in the WebSocket messages, eliminating the need for custom webhook servers in basic dial-in scenarios.

### Advanced Configuration

For production use cases, you can customize AssemblyAI's turn detection and add keyterms for improved accuracy:

```python
from pipecat.services.assemblyai.stt import AssemblyAISTTService, AssemblyAIConnectionParams

# Configure AssemblyAI with custom parameters
stt_params = AssemblyAIConnectionParams(
    sample_rate=8000,
    end_of_turn_confidence_threshold=0.4,
    min_turn_silence=400,
    max_turn_silence=1280,
    keyterms_prompt=["NPI", "TIN", "CMS", "PTAN", "CPT", "CDT", "DOB", "SSN"]
)

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    connection_params=stt_params,
)
```

**Turn detection options**: AssemblyAI has built-in VAD and turn detection. You can either:

- Use AssemblyAI's turn detection (recommended for best accuracy)
- Use Silero VAD by including `vad_analyzer=SileroVADAnalyzer()` in the transport params

**Keyterms**: Add domain-specific terms to improve recognition accuracy for specialized vocabulary.

## Resources

- [LiveKit Documentation](https://docs.livekit.io/)
- [LiveKit Telephony Quickstart](https://docs.livekit.io/agents/start/telephony/)
- [Telnyx SIP Trunk Configuration Guide](https://developers.telnyx.com/docs/voice/sip-trunking/livekit-configuration-guide)
- [Telnyx LiveKit SIP Setup](https://developers.telnyx.com/docs/voice/sip-trunking/livekit-configuration-guide)
- [Pipecat Documentation](https://docs.pipecat.ai/)
- [Pipecat Telnyx WebSocket Guide](https://docs.pipecat.ai/guides/telephony/telnyx-websockets)
- [Pipecat Telnyx Examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot)

---

<a id="integrate-twilio-with-assemblyai-docs-integrations-twilio-mdx"></a>

# Integrate Twilio with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/twilio
Source: docs/integrations/twilio.mdx
Navigation: Overview > Use cases & integrations > Integrations > Telephony tools
Description: Transcribe Twilio Voice data using AssemblyAI.

Twilio is a programmable communication platform for voice, messaging, and email.
By combining Twilio with AssemblyAI, you can transcribe voice calls in [real-time](/streaming/getting-started/transcribe-streaming-audio), and voice recordings and voice messages [asynchronously](/pre-recorded-audio).
Combine transcription with our [speech understanding models](/speech-understanding/summarization) to analyze the calls and messages.

Blog posts:

- [Transcribe a phone call in real-time using Python with AssemblyAI and Twilio](https://www.assemblyai.com/blog/transcribe-phone-call-real-time-python/)
- [Transcribe Phone Calls in Real-Time using Node.js with AssemblyAI, and Twilio](https://www.twilio.com/en-us/blog/phone-call-transcription-assemblyai-twilio-node)
- [Transcribe phone calls in real-time using C# .NET with AssemblyAI and Twilio](https://www.twilio.com/en-us/blog/transcribe-phone-calls-real-time-csharp-assemblyai-twilio)
- [Transcribe phone calls in real-time in Go with Twilio and AssemblyAI](https://www.assemblyai.com/blog/transcribe-phone-calls-in-realtime-in-go-with-twilio-and-assemblyai/)
- [Answer Questions about Twilio Voice Recordings with AssemblyAI and LangChain.js](https://www.twilio.com/en-us/blog/qa-voice-recordings-assemblyai-langchain-js)
- [Using Django & AssemblyAI for More Accurate Twilio Call Transcriptions](https://www.fullstackpython.com/blog/django-accurate-twilio-voice-transcriptions.html)

Videos:

- [Transcribe Twilio Phone Calls in Real-Time with AssemblyAI | JavaScript WebSockets Tutorial](https://youtu.be/3XmtJgWcOT0)

---

<a id="transcribe-your-amazon-connect-recordings-docs-integrations-amazon-connect-mdx"></a>

# Transcribe Your Amazon Connect Recordings

URL: https://www.assemblyai.com/docs/integrations/amazon-connect
Source: docs/integrations/amazon-connect.mdx
Navigation: Overview > Use cases & integrations > Integrations > Telephony tools
Description: Transcribe Your Amazon Connect Recordings documentation.

This guide walks through the process of setting up a transcription pipeline for Amazon Connect recordings using AssemblyAI.

## Get Started

Before we begin, make sure you have:

- An AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.
- An AWS account.
- An [Amazon Connect instance](https://docs.aws.amazon.com/connect/latest/adminguide/amazon-connect-instances.html).

## Step-by-Step Instructions

In the AWS console, navigate to the **Amazon Connect** services page. Select your instance and then click into the **Data Storage** section. On this page, find the subsection named **Call Recordings** and note the S3 bucket path where your call recordings are stored, you'll need this for later.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep1.png" alt="Amazon Connect Data Storage page showing the Call Recordings S3 bucket path" />

Navigate to the **Lambda** services page, and create a new function. Set the runtime to **Python 3.13**. In the **Change default execution role** section, choose the option to create a new role with basic Lambda permissions. Assign a function name and then click **Create function**.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep2.png" alt="AWS Lambda create function page with the Python 3.13 runtime selected" />

In this new function, scroll down to the **Code Source** section and paste the following code into `lambda_function.py`.

```python expandable
import json
import os
import boto3
import http.client
import time
from urllib.parse import unquote_plus
import logging

# Configure logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Configuration settings

# See config parameters here: https://www.assemblyai.com/docs/api-reference/transcripts/submit

ASSEMBLYAI_CONFIG = {
# 'language_code': 'en_us',
# 'multichannel': True,
# 'redact_pii': True,
}

# Initialize AWS services

s3_client = boto3.client('s3')

def get_presigned_url(bucket, key, expiration=3600):
    """Generate a presigned URL for the S3 object"""

    logger.info({
        "message": "Generating presigned URL",
        "bucket": bucket,
        "key": key,
        "expiration": expiration
    })

    s3_client_with_config = boto3.client(
        's3',
        config=boto3.session.Config(signature_version='s3v4')
    )

    return s3_client_with_config.generate_presigned_url(
        'get_object',
        Params={'Bucket': bucket, 'Key': key},
        ExpiresIn=expiration
    )

def delete_transcript_from_assemblyai(transcript_id, api_key):
    """
    Delete transcript data from AssemblyAI's database using their DELETE endpoint.

    Args:
    transcript_id (str): The AssemblyAI transcript ID to delete
    api_key (str): The AssemblyAI API key

    Returns:
    bool: True if deletion was successful, False otherwise
    """

    headers = {
        "authorization": api_key,
        "content-type": "application/json"
    }

    conn = http.client.HTTPSConnection("api.assemblyai.com")

    try: # Send DELETE request to AssemblyAI API
        conn.request("DELETE", f"/v2/transcript/{transcript_id}", headers=headers)
        response = conn.getresponse()

        # Check if deletion was successful (HTTP 200)
        if response.status == 200:
            response_data = json.loads(response.read().decode())
            logger.info(f"Successfully deleted transcript {transcript_id} from AssemblyAI")
            return True
        else:
            error_message = response.read().decode()
            logger.error(f"Failed to delete transcript {transcript_id}: HTTP {response.status} - {error_message}")
            return False

    except Exception as e:
        logger.info(f"Error deleting transcript {transcript_id}: {str(e)}")
        return False
    finally:
        conn.close()

def transcribe_audio(audio_url, api_key):
    """Transcribe audio using AssemblyAI API with http.client"""
    logger.info({"message": "Starting audio transcription"})

    headers = {
        "authorization": api_key,
        "content-type": "application/json"
    }

    conn = http.client.HTTPSConnection("api.assemblyai.com")

    # Submit the audio file for transcription with config parameters

    request_data = {"audio_url": audio_url}

    # Add all configuration settings

    request_data.update(ASSEMBLYAI_CONFIG)

    json_data = json.dumps(request_data)
    conn.request("POST", "/v2/transcript", json_data, headers)
    response = conn.getresponse()

    if response.status != 200:
        raise Exception(f"Failed to submit audio for transcription: {response.read().decode()}")

    response_data = json.loads(response.read().decode())
    transcript_id = response_data['id']
    logger.info({"message": "Audio submitted for transcription", "transcript_id": transcript_id})

    # Poll for transcription completion

    while True:
        conn = http.client.HTTPSConnection("api.assemblyai.com")
        conn.request("GET", f"/v2/transcript/{transcript_id}", headers=headers)
        polling_response = conn.getresponse()
        polling_data = json.loads(polling_response.read().decode())

        if polling_data['status'] == 'completed':
            conn.close()
            logger.info({"message": "Transcription completed successfully"})
            return polling_data  # Return full JSON response instead of just text
        elif polling_data['status'] == 'error':
            conn.close()
            raise Exception(f"Transcription failed: {polling_data['error']}")

        conn.close()
        time.sleep(3)

def lambda_handler(event, context):
    """Lambda function to handle S3 events and process audio files"""
    try: # Get the AssemblyAI API key from environment variables
        api_key = os.environ.get('ASSEMBLYAI_API_KEY')
        if not api_key:
            raise ValueError("ASSEMBLYAI_API_KEY environment variable is not set")

        # Process each record in the S3 event
        for record in event.get('Records', []):
            # Get the S3 bucket and key
            bucket = record['s3']['bucket']['name']
            key = unquote_plus(record['s3']['object']['key'])

            # Generate a presigned URL for the audio file
            audio_url = get_presigned_url(bucket, key)

            # Get the full transcript JSON from AssemblyAI
            transcript_data = transcribe_audio(audio_url, api_key)

            # Prepare the transcript key - maintaining path structure but changing directory and extension
            transcript_key = key.replace('/CallRecordings/', '/AssemblyAITranscripts/', 1).replace('.wav', '.json')

            # Convert the JSON data to a string
            transcript_json_str = json.dumps(transcript_data, indent=2)

            # Upload the transcript JSON to the same bucket but in transcripts directory
            s3_client.put_object(
                Bucket=bucket,  # Use the same bucket
                Key=transcript_key,
                Body=transcript_json_str,
                ContentType='application/json'
            )
            logger.info({"message": "Transcript uploaded to transcript bucket successfully.", "key": transcript_key})

            # Uncomment the following line to delete transcript data from AssemblyAI after saving to S3
            # https://www.assemblyai.com/docs/api-reference/transcripts/delete
            # delete_transcript_from_assemblyai(transcript_data['id'], api_key)

        return {
            "statusCode": 200,
            "body": json.dumps({
                "message": "Audio file(s) processed successfully",
                "detail": "Transcripts have been stored in the AssemblyAITranscripts directory"
            })
        }

    except Exception as e:
        print(f"Error: {str(e)}")

        return {
            "statusCode": 500,
            "body": json.dumps({
                "message": "Error processing audio file(s)",
                "error": str(e)
            })
        }
```

At the top of the lambda function, you can edit the config to enable features for your transcripts. To see all available parameters, check out our [API reference](/api-reference/transcripts/submit).

```python
ASSEMBLYAI_CONFIG = {
   # 'language_code': 'en_us',
   # 'multichannel': True,
   # 'redact_pii': True,
}
```

If you would like to delete transcripts from AssemblyAI after completion, you can uncomment line **166** to enable the `delete_transcript_from_assemblyai` function. This ensures the transcript data is only saved on your S3 database and not stored on AssemblyAI's database.

Once you have finished editing the lambda function, click **Deploy** to save your changes.

On the same page, navigate to the **Configuration** section, under **General configuration** adjust the timeout to 15min 0sec and click **Save**. The processing times for transcription will be a lot shorter, but this ensures plenty of time for the function to complete.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep3.png" alt="Lambda General configuration with the timeout set to 15 minutes" />

Now from this page, on the left side panel click **Environment variables**. Click edit and then add an environment variable, `ASSEMBLYAI_API_KEY`, and set the value to your AssemblyAI API key. Then click **Save**.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep4.png" alt="Lambda Environment variables page adding the ASSEMBLYAI_API_KEY variable" />

Now, navigate to the **IAM** services page. On the left side panel under Access Management click **Roles** and search for your Lambda function role (it's structure should look like `function_name-role-id`). Click into the role and then in the **Permissions policies** section click the dropdown for **Add permissions** and then select **Attach policies**.

From this page, find the policy named `AmazonS3FullAccess` and click **Add permissions**.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep5.png" alt="IAM role permissions page attaching the AmazonS3FullAccess policy" />

Now, navigate to the **S3** services page and click into the general purpose bucket where your Amazon Connect recordings are stored. Browse to the **Properties** tab and then scroll down to **Event notifications**. Click **Create event notification**. Give the event a name and then in the prefix section, insert the folder path we noted from Step 1 to ensure the event is triggered for the correct folder.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep6.png" alt="S3 Create event notification form with the Call Recordings folder prefix" />

Then in the **Event types** section, select **All object create events**.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep7.png" alt="S3 event notification Event types with All object create events selected" />

Then scroll down to the **Destination** section, set the destination as **Lambda function** and then select the Lambda function we created in Step 2. Then click **Save changes**.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep8.png" alt="S3 event notification Destination set to the Lambda function" />

To finalise the integration, we'll need to set the recording behaviour from within your AWS Contact Flows. Navigate to your Amazon Connect instance access URL and sign in to your Admin account. In the left side panel, navigate to the **Routing** section and then select **Flows**.

Choose a flow to test with, in this case we'll utilize the `Sample inbound flow (first contact experience)`. You should see the **Block Library** on the left hand side of the page. In this section, search for `Set recording and analytics behaviour` and then drag the block into your flow diagram and connect the arrows.

You can see in our example, we place the block right at the entry of the call flow:

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep9.png" alt="Amazon Connect flow with the Set recording and analytics behavior block at the call entry" />

After connecting this block, click the 3 vertical dots in the top right of the block and select **Edit settings**. Scroll down to the **Enable recording and analytics** subsection and expand the **Voice** section. Then select `On` and select `Agent and customer` (or whoever you'd like to record). Then click **Save**, click **Save** again in the top right and then click **Publish** to publish the flow.

<img src="/assets/img/integrations/AWS-Connect/AWSConnectStep10.png" alt="Set recording block settings enabling Voice recording for Agent and customer" />

With this new flow published, you should now receive recordings for your Amazon Connect calls that utilize that flow, and you should now receive AssemblyAI transcripts for those recordings!

The Amazon Connect Call Recordings are saved in the S3 bucket with this naming convention:
**/connect/\{instance-name\}/CallRecordings/\{YYYY\}/\{MM\}/\{DD\}/\{contact-id\}\_\{YYYYMMDDThh:mm\}\_UTC.wav**

The AssemblyAI Transcripts will be saved in the S3 bucket with this naming convention:
**/connect/\{instance-name\}/AssemblyAITranscripts/\{YYYY\}/\{MM\}/\{DD\}/\{contact-id\}\_\{YYYYMMDDThh:mm\}\_UTC.json**

To view the logs for this integration, navigate to the **CloudWatch** services page and under the **Logs** section, select **Log groups**. Select the log group that matches your Lambda to view the most recent log stream.

---

<a id="transcribe-genesys-cloud-recordings-with-assemblyai-docs-integrations-genesys-cloud-mdx"></a>

# Transcribe Genesys Cloud Recordings with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/genesys-cloud
Source: docs/integrations/genesys-cloud.mdx
Navigation: Overview > Use cases & integrations > Integrations > Telephony tools
Description: Transcribe Genesys Cloud Recordings with AssemblyAI documentation.

This guide walks through the process of setting up a transcription pipeline to send audio data from Genesys Cloud to AssemblyAI.

To accomplish this, we'll stream audio through Genesys's [AudioHook Monitor](https://appfoundry.genesys.com/filter/genesyscloud/listing/a3ff6a99-d866-4734-ab7a-16cff2e4308c) integration to a WebSocket server.
Upon call completion, the server will process this audio into a wav file and send it to AssemblyAI's Speech-to-text API for [pre-recorded audio](/pre-recorded-audio) transcription.

You can find all the necessary code for this guide in the [example GitHub repository](https://github.com/gsharp-aai/genesys-async-guide).

## Architecture Overview

Here's the general flow our app will follow:

```
+---------------------+     +--------------------+     +----------------------+
| 1. Genesys Cloud    |  →  | 2. WebSocket       |  →  | 3. Convert Raw       |
| (AudioHook Monitor) |     | Server             |     | Audio to WAV         |
+---------------------+     +--------------------+     +----------------------+
                                                                 ↓
+--------------------+     +---------------------+     +----------------------+
| 6. S3 Bucket       |  ←  | 5. AssemblyAI API   |  ←  | 4. Audio Upload (S3) |
| (Transcript Store) |     | (Transcription)     |     | (Trigger Lambda)     |
+--------------------+     +---------------------+     +----------------------+
```

## Getting started

Before we begin, make sure you have:

- An AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home).
- An AWS account, an [Access key](https://us-east-1.console.aws.amazon.com/iam/home#/security_credentials), and permissions to S3, Lambda, and CloudWatch.
- A [Genesys Cloud](https://www.genesys.com/genesys-cloud) account with the necessary permissions to create call flows, phone numbers, and routes.
- [ngrok](https://ngrok.com/downloads/mac-os) installed.

## Genesys AudioHook Monitor

In order to stream your voice calls to third party services outside of the Genesys Cloud platform, Genesys offers an official integration called [AudioHook Monitor](https://appfoundry.genesys.com/filter/genesyscloud/listing/a3ff6a99-d866-4734-ab7a-16cff2e4308c).

This integration allows you to specify the URL of a [WebSocket server](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) that implements the [AudioHook Protocol](https://developer.genesys.cloud/devapps/audiohook/), and once a connection has been established, Genesys will send both text (metadata messages/events encoded as JSON) and binary data (WebSocket frames containing the raw audio data in μ-law (Mu-law, PCMU) format).

An understanding of this integration and protocol is recommended before proceeding with this tutorial. Here are some helpful resources to get started:

- [Genesys App Foundry](https://appfoundry.genesys.com/filter/genesyscloud/listing/a3ff6a99-d866-4734-ab7a-16cff2e4308c)
- [AudioHook Monitor](https://help.mypurecloud.com/articles/about-audiohook-monitor/)
- [AudioHook Protocol](https://developer.genesys.cloud/devapps/audiohook/)
- [AudioHook Sample Service Repo](https://github.com/purecloudlabs/audiohook-reference-implementation#genesys-audiohook-sample-service)

## Step 1: Create a call flow in Genesys (optional)

You may already have an inbound call flow set up in Genesys (if so, skip to [Step 3](/integrations/genesys-cloud#step-3-create-a-s3-bucket)), but we'll create a simple one from scratch for the sake of this tutorial.

  **Adapting existing call flows**

  All you need to add is the **Audio Monitoring** step from the toolbox and make
  sure **Suppress recording for the entire flow** is unchecked in the flow's
  **Recording and Speech Recognition** settings.

Within the [Architect tool](https://apps.usw2.pure.cloud/architect/#/inboundcall/flows), click **Add** to create a new call flow. Enter a **Name** for your flow and click **Create Flow**. Select the newly created flow to open the drag and drop editor.

<img src="/assets/img/integrations/genesys/async/create_call_flow.png" alt="Create Flow dialog for adding a new inbound call flow in Genesys Architect" />

Create a **Reusable Task** from the bottom left of the left-side menu. From the Toolbox, search for **Audio Monitoring** and drag it just after the **Start** step of our flow. In the right-side menu for this option, make sure **Enable Monitoring** is enabled.

<img src="/assets/img/integrations/genesys/async/enable_call_monitoring.png" alt="Audio Monitoring step added after Start with Enable Monitoring turned on in Genesys Architect" />

Back in the Toolbox, search for **Transfer to User** and set that as the next step. In the right-hand menu, under **User** select a caller. Under **Pre-Transfer Audio** and **Failed Transfer Audio**, type your preferred messages.

<img src="/assets/img/integrations/genesys/async/transfer_to_user.png" alt="Transfer to User step configured with pre-transfer and failed transfer audio in Genesys Architect" />

Search for **Disconnect** in the toolbox and drag that as the step following **Failure**.

<img src="/assets/img/integrations/genesys/async/disconnect.png" alt="Disconnect step placed after the Failure path in the Genesys call flow" />

Search the Toolbox for **Jump to Reusable Task** and drag this tool to the Main Menu at the top of the left-side menu. Select a **DTMF** and **Speech Recognition** value (this will be used to transfer your call to the agent). Under **Task**, select the task you just created.

<img src="/assets/img/integrations/genesys/async/reusable_task.png" alt="Jump to Reusable Task tool added to the Main Menu with DTMF and speech recognition values" />

Under Settings in the left-side menu, navigate to the **Recording and Speech Recognition** section. Make sure **Suppress recording for the entire flow** is unchecked.

<img src="/assets/img/integrations/genesys/async/audio_settings.png" alt="Recording and Speech Recognition settings with Suppress recording for the entire flow unchecked" />

In the top navbar, make sure to click **Save** and then click **Publish** to have your changes take effect.

## Step 2: Setup a phone and routing for your flow (optional)

In the Genesys Cloud Admin section, navigate to the [Phones page](https://apps.usw2.pure.cloud/directory/#/admin/telephony/phone-management/phones) under the **Telephony section** and click **Add** to create a new phone. For **Person**, assign the User from your organization that you selected for the **Transfer to User** step in the [previous section](/integrations/genesys-cloud#step-1-create-a-call-flow-in-genesys-optional).

<img src="/assets/img/integrations/genesys/async/create_phone.png" alt="Creating a new phone on the Genesys Cloud Phone Management page" />
<img src="/assets/img/integrations/genesys/async/assign_phone.png" alt="Assigning the transfer-to user as the Person for the new phone in Genesys Cloud" />

Navigate to the [Number Management](https://apps.usw2.pure.cloud/directory/#/admin/telecom/numbers/numbers) page under the **Genesys Cloud Voice** section and select **Purchase Numbers**.
Enter an area code and click **Search**. Select a phone number from the list and click **Complete Purchase**.

<img src="/assets/img/integrations/genesys/async/purchase_numbers.png" alt="Number Management page with the Purchase Numbers option in Genesys Cloud Voice" />
<img src="/assets/img/integrations/genesys/async/purchase_numbers_search.png" alt="Searching for a phone number by area code before completing purchase in Genesys Cloud" />

Navigate to the [Call Routing](https://apps.usw2.pure.cloud/directory/#/admin/routing/ivrs) page under the **Routing** section and select **Add**. Under **What call flow should be used?** select your flow. For **Inbound Numbers**, type the number you purchased in the above step. Then click **Create**.

<img src="/assets/img/integrations/genesys/async/add_route.png" alt="Call Routing page with the Add option under the Routing section in Genesys Cloud" />
<img src="/assets/img/integrations/genesys/async/create_route.png" alt="Creating a call route mapping the purchased inbound number to the call flow" />

Under the **Telephony** section, navigate to the [External Trunks](https://apps.usw2.pure.cloud/directory/#/admin/telephony/trunks/external) page. Click **Create New**. Under **Caller ID**, the **Caller Address** will be an E.164 number and the phone number you created.

<img src="/assets/img/integrations/genesys/async/add_trunk.png" alt="Creating a new external trunk on the Genesys Cloud External Trunks page" />
<img src="/assets/img/integrations/genesys/async/name_trunk.png" alt="Setting the trunk Caller ID caller address to the purchased E.164 phone number" />

Under **SIP Access Control**, select **Allow All** (_note: this is only for development and testing purposes, please specify actual IPs in production_). Under the **Media** section, make sure you select **Record calls on this trunk**. Then click **Save External Trunk** (it may take a few moments for your trunk to be ready).

<img src="/assets/img/integrations/genesys/async/sip_access.png" alt="SIP Access Control set to Allow All in the external trunk settings" />
<img src="/assets/img/integrations/genesys/async/recording.png" alt="Media section with Record calls on this trunk enabled for the external trunk" />

## Step 3: Create a S3 bucket

After our Genesys call ends, store the audio file in S3.

Click **Create bucket**. Give your bucket a name like `your-audiohook-bucket`. Scroll down and click **Create bucket**.

<img src="/assets/img/integrations/genesys/async/audiohook_s3_bucket.png" alt="Creating an S3 bucket named your-audiohook-bucket in the AWS console" />

## Step 4: Create a WebSocket server

In this step, we'll set up a WebSocket server to receive messages and audio data from Genesys as they are sent.
Our server must respond to certain events (i.e. `open`, `close`, `ping`, `pause`, etc.) according to the AudioHook protocol.
Outside of these event messages, audio data is also transferred. We'll capture and temporarily store this audio locally until the connection is closed, at which point the server processes the audio to a `wav` file and uploads both the `wav` and `raw` audio files to a S3 bucket.

The AudioHook Monitor will send requests to a WebSocket URL we specify when setting up the integration in [Step 5](/integrations/genesys-cloud#step-5-setting-up-audiohook-monitor).
When first enabled, AudioHook Monitor will do a quick verification step to ensure that the WebSocket server has implemented the AudioHook protocol correctly.

For this example, the server is written in JavaScript ([Express](https://expressjs.com/)) and hosted locally. We'll use [ngrok](https://ngrok.com/) to create a secure tunnel that exposes it to the internet with a public URL so that Genesys can make a connection.
However, the server can be implemented using your preferred programming language and deployed in whatever environment you choose, provided both support WebSocket TLS connections for secure bidirectional text and binary message exchange.

  **Server implementation**

  This server is a method to get up and running quickly for development and
  testing purposes without the complexity of a production deployment. How you
  implement this in practice will vary widely depending on your traffic volume,
  scaling needs, reliability requirements, security concerns, etc.

Clone this [example repo](https://github.com/gsharp-aai/genesys-async-guide) of a WebSocket server that implements the AudioHook protocol.
Follow the `README` instructions to download the necessary dependencies and start the server.
Make sure to look over `server.js` to get an understanding of how the requests from Genesys are received, processed, and responded to, as well as how the audio is stored, converted, and uploaded to our S3 bucket.

Make sure to create a `.env` file and set the variables:

```env
PORT=3000
AWS_REGION=us-east-1 # Region of your S3 bucket
AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID # Found under IAM > Security Credentials
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY # Found under IAM > Security Credentials
S3_BUCKET=your-audiohook-bucket # Name of your S3 bucket
S3_KEY_PREFIX=calls/ # The file structure you want your bucket to follow
API_KEY=YOUR_API_KEY # Used for authenticating messages from Genesys
RECORDINGS_DIR=./recordings # Temp file storage location
```

  `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` can be found on your account's
  [IAM > Security Credentials
  page](https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials).
  `API_KEY` is explained further in the [next
  step](/integrations/genesys-cloud#step-5-setting-up-audiohook-monitor), but it can be anything you
  want to verify that the requests are actually originating from Genesys.

Download [ngrok](https://ngrok.com/). Assuming your server is running on `port 3000`, run `ngrok http 3000 --inspect=false` in your terminal.
From the resulting terminal output, note the forwarding url that should look something like this: `https://<id>.ngrok-free.app`. This is our WebSocket server URL that we'll provide to the AudioHook Monitor in the next step.

<img src="/assets/img/integrations/genesys/async/ngrok_connection.png" alt="ngrok terminal output showing the forwarding URL for the local WebSocket server" />

## Step 5: Setting up AudioHook Monitor

In the Genesys Cloud Admin section, navigate to the [Integrations](https://apps.usw2.pure.cloud/directory/#/admin/integrations/apps) page. Add a new integration via the plus sign in the top right corner.

<img src="/assets/img/integrations/genesys/async/new_integration.png" alt="Adding a new integration from the Genesys Cloud Integrations page" />

Search for **AudioHook Monitor** and install.

<img src="/assets/img/integrations/genesys/async/install.png" alt="Installing the AudioHook Monitor integration from the Genesys Cloud integration catalog" />

Navigate to the AudioHook Monitor's **Configuration** tab. Under the **Properties section**, make sure both channels are selected and the **Connection URI** is set to the ngrok url from the [previous step](/integrations/genesys-cloud#step-4-create-a-websocket-server). For the ngrok url, replace `https` with `wss`.

<img src="/assets/img/integrations/genesys/async/server_url.png" alt="AudioHook Monitor Configuration tab with the wss Connection URI and both channels selected" />

In the **Configuration** tab, navigate to the **Credentials** section, and click **Configure**. Here you can set an API key to a value our server will use to verify that requests originated from Genesys. This is done via the `X-API-KEY` request header. Our server will compare this key to the value we set in our `.env` for `API_KEY`, so make sure they match. Click **Save**.

<img src="/assets/img/integrations/genesys/async/configure.png" alt="Setting the X-API-KEY credential in the AudioHook Monitor Credentials configuration" />

Back on the **Integrations** page, click the toggle button under the **Status** column to activate your AudioHook. Genesys will attempt to verify our server is correctly configured according to the AudioHook protocol. If it is unable to do so, a red error will show with the reason for the failed connection. If it succeeds, the connection will toggle to Active.

<img src="/assets/img/integrations/genesys/async/audiohook_success.png" alt="AudioHook Monitor integration toggled to Active status after successful verification" />

## Step 6: Set up your AssemblyAI API call

Navigate to the [Lambda](https://us-east-1.console.aws.amazon.com/lambda/home) services page, and create a new function. Set the runtime to `Node.js 22.x`. In the **Change default execution role** section, choose the option to create a **new role with basic Lambda permissions**. Assign a function name and then click **Create function**.

<img src="/assets/img/integrations/genesys/async/new_lambda.png" alt="Creating a new Node.js Lambda function with a basic execution role in the AWS console" />

In this new function, scroll down to the **Code Source** section and paste the following code into `index.js`:

```javascript expandable
// Import required AWS SDK modules
import { S3 } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { GetObjectCommand } from "@aws-sdk/client-s3";

// Configure logging
const logger = {
  info: (data) => console.log(JSON.stringify(data)),
  error: (data) => console.error(JSON.stringify(data)),
};

// Configuration settings for AssemblyAI
// See config parameters here: https://www.assemblyai.com/docs/api-reference/transcripts/submit
const ASSEMBLYAI_CONFIG = {
  multichannel: true, // Using multichannel here as we told Genesys to send us multichannel audio.
};

// Initialize AWS S3 client
const s3Client = new S3();

/**
 * Generate a presigned URL for the S3 object
 * @param {string} bucket - S3 bucket name
 * @param {string} key - S3 object key
 * @param {number} expiration - URL expiration time in seconds
 * @returns {Promise<string>} Presigned URL
 */
const getPresignedUrl = async (bucket, key, expiration = 3600) => {
  logger.info({
    message: "Generating presigned URL",
    bucket: bucket,
    key: key,
    expiration: expiration,
  });

  const command = new GetObjectCommand({
    Bucket: bucket,
    Key: key,
  });

  return getSignedUrl(s3Client, command, { expiresIn: expiration });
};

/**
 * Delete transcript data from AssemblyAI's database
 * @param {string} transcriptId - The AssemblyAI transcript ID to delete
 * @param {string} apiKey - The AssemblyAI API key
 * @returns {Promise<boolean>} True if deletion was successful, False otherwise
 */
const deleteTranscriptFromAssemblyAI = async (transcriptId, apiKey) => {
  try {
    const response = await fetch(
      `https://api.assemblyai.com/v2/transcript/${transcriptId}`,
      {
        method: "DELETE",
        headers: {
          authorization: apiKey,
          "content-type": "application/json",
        },
      }
    );

    if (response.ok) {
      logger.info(
        `Successfully deleted transcript ${transcriptId} from AssemblyAI`
      );
      return true;
    } else {
      const errorData = await response.text();
      logger.error(
        `Failed to delete transcript ${transcriptId}: HTTP ${response.status} - ${errorData}`
      );
      return false;
    }
  } catch (error) {
    logger.error(`Error deleting transcript ${transcriptId}: ${error.message}`);
    return false;
  }
};

/**
 * Submit audio for transcription
 * @param {object} requestData - Request data including audio URL and config
 * @param {string} apiKey - AssemblyAI API key
 * @returns {Promise<string>} Transcript ID
 */
const submitTranscriptionRequest = async (requestData, apiKey) => {
  const response = await fetch("https://api.assemblyai.com/v2/transcript", {
    method: "POST",
    headers: {
      authorization: apiKey,
      "content-type": "application/json",
    },
    body: JSON.stringify(requestData),
  });

  if (!response.ok) {
    const errorText = await response.text();
    throw new Error(`Failed to submit audio for transcription: ${errorText}`);
  }

  const responseData = await response.json();
  const transcriptId = responseData.id;

  logger.info({
    message: "Audio submitted for transcription",
    transcript_id: transcriptId,
  });

  return transcriptId;
};

/**
 * Poll for transcription completion
 * @param {string} transcriptId - Transcript ID
 * @param {string} apiKey - AssemblyAI API key
 * @returns {Promise<object>} Transcription data
 */
const pollTranscriptionStatus = async (transcriptId, apiKey) => {
  const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

  // Keep polling until we get a completion or error
  while (true) {
    const response = await fetch(
      `https://api.assemblyai.com/v2/transcript/${transcriptId}`,
      {
        method: "GET",
        headers: {
          authorization: apiKey,
          "content-type": "application/json",
        },
      }
    );

    if (!response.ok) {
      const errorText = await response.text();
      throw new Error(`Failed to poll transcription status: ${errorText}`);
    }

    const pollingData = await response.json();

    if (pollingData.status === "completed") {
      logger.info({ message: "Transcription completed successfully" });
      return pollingData;
    } else if (pollingData.status === "error") {
      throw new Error(`Transcription failed: ${pollingData.error}`);
    }

    // Wait before polling again
    await sleep(3000);
  }
};

/**
 * Transcribe audio using AssemblyAI API
 * @param {string} audioUrl - URL of the audio file
 * @param {string} apiKey - AssemblyAI API key
 * @returns {Promise<object>} Transcription data
 */
const transcribeAudio = async (audioUrl, apiKey) => {
  logger.info({ message: "Starting audio transcription" });

  // Prepare request data with config parameters
  const requestData = { audio_url: audioUrl, ...ASSEMBLYAI_CONFIG };

  // Submit the audio file for transcription
  const transcriptId = await submitTranscriptionRequest(requestData, apiKey);

  // Poll for transcription completion
  return await pollTranscriptionStatus(transcriptId, apiKey);
};

/**
 * Lambda function handler
 * @param {object} event - S3 event
 * @param {object} context - Lambda context
 * @returns {Promise<object>} Response
 */
export const handler = async (event, context) => {
  try {
    // Get the AssemblyAI API key from environment variables
    const apiKey = process.env.ASSEMBLYAI_API_KEY;
    if (!apiKey) {
      throw new Error("ASSEMBLYAI_API_KEY environment variable is not set");
    }

    // Process each record in the S3 event
    const records = event.Records || [];

    for (const record of records) {
      // Get the S3 bucket and key
      const bucket = record.s3.bucket.name;
      const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " "));

      // Generate a presigned URL for the audio file
      const audioUrl = await getPresignedUrl(bucket, key);

      // Get the full transcript JSON from AssemblyAI
      const transcriptData = await transcribeAudio(audioUrl, apiKey);

      // Prepare the transcript key - maintaining path structure but changing directory and extension
      const transcriptKey = key
        .replace("audio", "transcripts", 1)
        .replace(".wav", ".json");

      // Convert the JSON data to a string
      const transcriptJsonStr = JSON.stringify(transcriptData, null, 2);

      // Upload the transcript JSON to the same bucket but in transcripts directory
      await s3Client.putObject({
        Bucket: bucket, // Use the same bucket
        Key: transcriptKey, // Store under the /transcripts directory
        Body: transcriptJsonStr,
        ContentType: "application/json",
      });

      logger.info({
        message: "Transcript uploaded to transcript bucket successfully.",
        key: transcriptKey,
      });

      // Uncomment the following line to delete transcript data from AssemblyAI after saving to S3
      // https://www.assemblyai.com/docs/api-reference/transcripts/delete
      // await deleteTranscriptFromAssemblyAI(transcriptData.id, apiKey);
    }

    return {
      statusCode: 200,
      body: JSON.stringify({
        message: "Audio file(s) processed successfully",
        detail:
          "Transcripts have been stored in the AssemblyAITranscripts directory",
      }),
    };
  } catch (error) {
    console.error(`Error: ${error.message}`);
    return {
      statusCode: 500,
      body: JSON.stringify({
        message: "Error processing audio file(s)",
        error: error.message,
      }),
    };
  }
};
```

At the top of the Lambda function, you can edit the config to enable features for your transcripts. Since our call is two channels, we'll want to set `multichannel` to `true`. To see all available parameters, check out our [API reference](/api-reference/transcripts/submit).

```javascript
ASSEMBLYAI_CONFIG = {
  multichannel: true,
  // 'language_code': 'en_us',
  // 'redact_pii': true
  // etc.
};
```

  If you would like to delete transcripts from AssemblyAI after completion, you
  can uncomment `line 212` to enable the `deleteTranscriptFromAssemblyAI`
  function. This ensures the transcript data is only saved to your S3 bucket and
  not stored on AssemblyAI's database.

Once you have finished editing the Lambda function, click **Deploy** to save your changes.

<img src="/assets/img/integrations/genesys/async/lambda_example.png" alt="Lambda code source editor with the transcription handler pasted into index.js" />

On the same page, navigate to the **Configuration** section. Under **General configuration**, click **Edit**, and then adjust **Timeout** to `15min 0sec` and click **Save**. The processing times for transcription will be much shorter, but this ensures the function will have plenty of time to run.

<img src="/assets/img/integrations/genesys/async/configuration.png" alt="Lambda Configuration tab General configuration section in the AWS console" />
<img src="/assets/img/integrations/genesys/async/set_timeout.png" alt="Setting the Lambda function timeout to 15 minutes in general configuration" />

On the left side panel, click **Environment variables**. Click **Edit**. Add an environment variable, `ASSEMBLYAI_API_KEY`, and set the value to your AssemblyAI [API key](https://www.assemblyai.com/dashboard/home). Then click **Save**.

<img src="/assets/img/integrations/genesys/async/lambda_api_key.png" alt="Adding the ASSEMBLYAI_API_KEY environment variable to the Lambda function" />

Now, navigate to the [IAM](https://us-east-1.console.aws.amazon.com/iam/) services page. On the left side panel under **Access Management**, click **Roles** and search for your Lambda function's role (its structure should look like `<function_name>-<role_id>`). Click the role and then in the **Permissions policies** section click the dropdown for **Add permissions** and then select **Attach policies.**

<img src="/assets/img/integrations/genesys/async/attach_policies.png" alt="Attaching policies to the Lambda function's IAM role in the AWS console" />

From this page, find the policies named `AmazonS3FullAccess` and `CloudWatchEventsFullAccess`. Click **Add permissions** for both.

<img src="/assets/img/integrations/genesys/async/adding_permissions.png" alt="Adding AmazonS3FullAccess and CloudWatchEventsFullAccess permissions to the IAM role" />

`CloudWatchEventsFullAccess` is optional, but helpful for debugging purposes. Once your Lambda runs, it should output all logs to [CloudWatch](https://us-east-1.console.aws.amazon.com/cloudwatch) under a Log group `/aws/lambda/<your-lambda-fn>`

Now, navigate to the [S3](https://us-east-1.console.aws.amazon.com/s3) services page and click into the general purpose bucket where your Genesys recordings are stored. Browse to the **Properties** tab and then scroll down to **Event notifications**. Click **Create event notification**.

<img src="/assets/img/integrations/genesys/async/create_event_notif.png" alt="Creating an S3 event notification from the bucket Properties tab in the AWS console" />

Give the event a name and then in the **Prefix** section enter `calls/` (or whatever `S3_KEY_PREFIX` is set to), and in the **Suffix** section enter `.wav`. This will ensure the event is triggered once our `wav` file has been uploaded. In the **Event types** section, select **All object create events**.

<img src="/assets/img/integrations/genesys/async/creation_of_event.png" alt="Configuring the S3 event notification with calls/ prefix, .wav suffix, and object create events" />

Scroll down to the **Destination** section, set the destination as **Lambda function** and then select the Lambda function we created in [Step 6](/integrations/genesys-cloud#step-6-set-up-your-assemblyai-api-call). Then click **Save changes**.

<img src="/assets/img/integrations/genesys/async/adding_to_lambda.png" alt="Setting the S3 event notification destination to the Lambda function" />

## Step 7: Transcribe your first call

To test everything is working, call the phone number you linked to this flow in [Step 2](/integrations/genesys-cloud#step-2-setup-a-phone-and-routing-for-your-flow-optional). Referring to the example flow above, press the DTMF value on the key pad or say the Speech Recognition value. Once transferred, your WebSocket server should start to receive data and output to console:

```bash
{
  version: '2',
  id: '<id>',
  type: 'ping',
  seq: 4,
  position: 'PT8.2S',
  parameters: { rtt: 'PT0.035392266S' },
  serverseq: 3
}
Received binary audio data: 3200 bytes
Received binary audio data: 3200 bytes
...
Processed 146KB of audio data so far
```

Once the call has ended, you should see the following server logs:

```bash expandable
{
  version: '2',
  id: '<id>',
  type: 'close',
  seq: 5,
  position: 'PT10.2S',
  parameters: { reason: 'end' },
  serverseq: 4
}
Handling close message
Closing file stream
Converting raw audio to WAV: '<wav file name>'

# Skipping ffmpeg output for brevity...

Uploading recording '<raw file>' to S3
Successfully uploaded raw recording to S3: '<raw file>'
Successfully uploaded WAV recording to S3: '<wav file>'
Sent closed response, seq=5
WebSocket closed for session '<session_id>': code=1000, reason=Session Ended
Cleaning up session '<session_id>'
Deleted local raw recording file: '<raw_file>'
Deleted local WAV recording file: '<wav_file>'
```

  To view the logs for this Lambda function, navigate to the [CloudWatch](https://us-east-1.console.aws.amazon.com/cloudwatch) services page and under the Logs section, select **Log groups**.
  Select the log group that matches your Lambda to view the most recent log stream. This can be very useful for debugging purposes if you run into any issues.

  <img src="/assets/img/integrations/genesys/async/cloudwatch_logs.png" alt="CloudWatch Log groups page for viewing the Lambda function's log streams" />

Head to your S3 bucket. Within the `/calls` directory, files will be stored under a unique identifier with the following structure:

`your-audiohook-bucket/calls/<timestamp>_<call_id>_<speaker_id>/<file_type>`

with audio files (both `raw` and `wav`) under `/audio` and transcript responses under `/transcripts`.

<img src="/assets/img/integrations/genesys/async/s3_bucket_success.png" alt="S3 bucket showing stored call recordings and transcript files after a successful run" />

  The `raw` file can be nice to have for conversions to other formats in the
  future, but this step can be omitted to save on storage costs.

**Success!** You have successfully integrated AssemblyAI with Genesys Cloud via AudioHook Monitor. If you run into any issues or have further questions, please reach out to our [Support team](https://www.assemblyai.com/contact/support).

## Other considerations

### Supported audio formats

- Audio is sent as binary WebSocket frames containing the raw audio data in the negotiated format. Currently, only μ-law (Mu-law, PCMU) is [supported](https://developer.genesys.cloud/devapps/audiohook/session-walkthrough#audio-streaming).
- Before being uploaded to S3, the audio is converted to `wav` format using [ffmpeg](https://ffmpeg.org/). As a lossless format, `wav` generally results in high transcription accuracy, but is not required. A full list of [file formats supported by AssemblyAI's API](/faq/what-audio-and-video-file-types-are-supported-by-your-api) can be found in our FAQ.

### Multichannel

- As mentioned in [Step 6](/integrations/genesys-cloud#step-6-set-up-your-assemblyai-api-call), the `multichannel` parameter should be enabled as the files are stereo utilizing one channel for each participant. When possible, multiple channels are recommended by AssemblyAI for the most accurate transcription results.
- If single channel is preferred, you can simplify the approach to only send a single channel with both speakers (via AudioHook) and adjust your server code to be single channel.

---

<a id="u0001f99c-u0001f517-langchain-integration-with-assemblyai-docs-integrations-langchain-mdx"></a>

# \U0001F99C️\U0001F517 LangChain Integration with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/langchain
Source: docs/integrations/langchain.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools > LangChain
Description: Transcribe audio in LangChain using the built-in integration with AssemblyAI.

## AssemblyAI LangChain Integration

AssemblyAI has a LangChain integration for both Python and JavaScript.
Learn more about our language-specific integrations:

  
  

## New to LangChain?

[LangChain](https://www.langchain.com) is an open-source framework for developing applications with [Large Language Models (LLMs)](https://www.assemblyai.com/blog/introduction-large-language-models-generative-ai/) and other AI technologies.
LangChain has a set of pre-built components that you can use to load data and apply LLMs to your data.
However, LLMs only operate on textual data and don't understand speech in audio and video files.
To apply LLMs to speech, you first need to transcribe the audio to text, which is what the AssemblyAI integration helps you with.

---

<a id="u0001f99c-u0001f517-langchain-python-integration-with-assemblyai-docs-integrations-langchain-python-mdx"></a>

# \U0001F99C️\U0001F517 LangChain Python Integration with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/langchain/python
Source: docs/integrations/langchain/python.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools > LangChain
Description: Transcribe audio in LangChain Python using the built-in integration with AssemblyAI.

To apply LLMs to speech, you first need to transcribe the audio to text, which is what the AssemblyAI integration for LangChain helps you with.

Looking for the LangChain JavaScript integration?<br />
[Go to the LangChain.JS integration](/integrations/langchain/js).

## Quickstart

Install [the AssemblyAI package](https://github.com/langchain-ai/langchain) and [the AssemblyAI Python SDK](https://github.com/AssemblyAI/assemblyai-python-sdk):

```bash
pip install langchain
pip install assemblyai
```

Set your AssemblyAI API key as an environment variable named `ASSEMBLYAI_API_KEY`. You can [get a free AssemblyAI API key from the AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).

```bash
# Mac/Linux:
export ASSEMBLYAI_API_KEY=YOUR_API_KEY

# Windows:
set ASSEMBLYAI_API_KEY=YOUR_API_KEY
```

Import the `AssemblyAIAudioTranscriptLoader` from `langchain.document_loaders`.

```python
from langchain.document_loaders import AssemblyAIAudioTranscriptLoader
```

1. Pass the local file path or URL as the `file_path` argument of the `AssemblyAIAudioTranscriptLoader`.
2. Call the `load` method to get the transcript as LangChain documents.

```python
audio_file = "https://assembly.ai/sports_injuries.mp3"
# or a local file path: audio_file = "./sports_injuries.mp3"

loader = AssemblyAIAudioTranscriptLoader(file_path=audio_file)

docs = loader.load()
```

The `load` method returns an array of documents, but by default, there's only one document in the array with the full transcript.

The transcribed text is available in the `page_content` attribute:

```python
docs[0].page_content
# Load time, a new president and new congressional makeup. Same old ...
```

The `metadata` contains the full JSON response with more meta information:

```python
{
  'language_code': <LanguageCode.en_us: 'en_us'>,
  'audio_url': 'https://assembly.ai/nbc.mp3',
  'punctuate': True,
  'format_text': True,
  ...
}
```

## Transcript formats

You can specify the `transcript_format` argument to load the transcript in different formats.

Depending on the format, `load_data()` returns either one or more documents. These are the different `TranscriptFormat` options:

- `TEXT`: One document with the transcription text
- `SENTENCES`: Multiple documents, splits the transcription by each sentence
- `PARAGRAPHS`: Multiple documents, splits the transcription by each paragraph
- `SUBTITLES_SRT`: One document with the transcript exported in SRT subtitles format
- `SUBTITLES_VTT`: One document with the transcript exported in VTT subtitles format

```python
import assemblyai as aai
from langchain.document_loaders import AssemblyAIAudioTranscriptLoader
from langchain.document_loaders.assemblyai import TranscriptFormat

loader = AssemblyAIAudioTranscriptLoader(
    file_path="./your_file.mp3",
    transcript_format=TranscriptFormat.SENTENCES,
)

docs = loader.load()
```

## Transcription config

You can also specify the `config` argument to use different transcript features and speech understanding models.
Here's an example of using the `config` argument to enable speaker labels, auto chapters, and entity detection:

```python
import assemblyai as aai
from langchain.document_loaders import AssemblyAIAudioTranscriptLoader

config = aai.TranscriptionConfig(
    speaker_labels=True, auto_chapters=True, entity_detection=True
)

loader = AssemblyAIAudioTranscriptLoader(file_path="./your_file.mp3", config=config)
```

  For the full list of options, see [Transcript API
  reference](/api-reference/transcripts/submit#request).

## Pass the AssemblyAI API key as an argument

Instead of configuring the AssemblyAI API key as the `ASSEMBLYAI_API_KEY` environment variable,
you can also pass it as the `api_key` argument.

```python
loader = AssemblyAIAudioTranscriptLoader(
    file_path="./your_file.mp3", api_key="<YOUR_API_KEY>"
)
```

## Additional resources

You can learn more about using LangChain with AssemblyAI in these resources.

- [LangChain docs for the AssemblyAI document loader](https://python.langchain.com/docs/integrations/document_loaders/assemblyai)
- [How to use audio data in LangChain with Python](https://www.assemblyai.com/blog/load-audio-langchain-python/)
- [Retrieval Augmented Generation on audio data with LangChain and Chroma](https://www.assemblyai.com/blog/retrieval-augmented-generation-audio-langchain/)
- [Build LangChain Audio Apps with Python in 5 Minutes](https://www.youtube.com/watch?v=7w7ysaDz2W4)
- [How to use LangChain for RAG over audio files](https://www.youtube.com/watch?v=l9YJrLg61ac)
- [AssemblyAI Python SDK](https://github.com/AssemblyAI/assemblyai-python-sdk)

---

<a id="u0001f99c-u0001f517-langchain-javascript-integration-with-assemblyai-docs-integrations-langchain-js-mdx"></a>

# \U0001F99C️\U0001F517 LangChain JavaScript Integration with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/langchain/js
Source: docs/integrations/langchain/js.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools > LangChain
Description: Transcribe audio in LangChain.JS using the built-in integration with AssemblyAI.

To apply LLMs to speech, you first need to transcribe the audio to text, which is what the AssemblyAI integration for LangChain helps you with.

Looking for the Python integration?<br />
[Go to the LangChain Python integration](/integrations/langchain/python).

## Quickstart

Add the [AssemblyAI SDK](https://github.com/AssemblyAI/assemblyai-node-sdk) to your project:

  

```bash
npm install langchain @langchain/community
```

  
  

```bash
yarn add langchain @langchain/community
```

  
  

```bash
pnpm add langchain @langchain/community
```

  
  

```bash
bun add langchain @langchain/community
```

  

To use the loaders, you need an [AssemblyAI account](https://www.assemblyai.com/dashboard/signup) and get your AssemblyAI API key from the [dashboard](https://www.assemblyai.com/dashboard/home).
Configure the API key as the `ASSEMBLYAI_API_KEY` environment variable or the `apiKey` options parameter.

```javascript expandable
import {
  AudioTranscriptLoader,
  // AudioTranscriptParagraphsLoader,
  // AudioTranscriptSentencesLoader
} from "@langchain/community/document_loaders/web/assemblyai";

// You can also use a local file path and the loader will upload it to AssemblyAI for you.
const audioUrl = "https://assembly.ai/espn.m4a";

// Use `AudioTranscriptParagraphsLoader` or `AudioTranscriptSentencesLoader` for splitting the transcript into paragraphs or sentences
const loader = new AudioTranscriptLoader(
  {
    audio: audioUrl,
    // any other parameters as documented here: https://www.assemblyai.com/docs/api-reference/transcript#create-a-transcript
  },
  {
    apiKey: "<ASSEMBLYAI_API_KEY>", // or set the `ASSEMBLYAI_API_KEY` env variable
  }
);
const docs = await loader.load();
console.dir(docs, { depth: Infinity });
```

  - You can use the `AudioTranscriptParagraphsLoader` or
  `AudioTranscriptSentencesLoader` to split the transcript into paragraphs or
  sentences. - If the `audio_file` is a local file path, the loader will upload
  it to AssemblyAI for you. - The `audio_file` can also be a video file. See the
  [list of supported file types in the FAQ
  doc](/faq/what-audio-and-video-file-types-are-supported-by-your-api). -
  If you don't pass in the `apiKey` option, the loader will use the
  `ASSEMBLYAI_API_KEY` environment variable. - You can add more properties in
  addition to `audio`. Find the full list of request parameters in the
  [AssemblyAI API docs](/api-reference/overview).

<br />

You can also use the `AudioSubtitleLoader` to get `srt` or `vtt` subtitles as a document.

```javascript
import { AudioSubtitleLoader } from "@langchain/community/document_loaders/web/assemblyai";

// You can also use a local file path and the loader will upload it to AssemblyAI for you.
const audioUrl = "https://assembly.ai/espn.m4a";

const loader = new AudioSubtitleLoader(
  {
    audio: audioUrl,
    // any other parameters as documented here: https://www.assemblyai.com/docs/api-reference/transcript#create-a-transcript
  },
  "srt", // srt or vtt
  {
    apiKey: "<ASSEMBLYAI_API_KEY>", // or set the `ASSEMBLYAI_API_KEY` env variable
  }
);

const docs = await loader.load();
console.dir(docs, { depth: Infinity });
```

## Additional resources

You can learn more about using LangChain with AssemblyAI in these resources:

- [The LangChain docs for the AssemblyAI document loader](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/assemblyai_audio_transcription)
- [How to integrate spoken audio into LangChain.js using AssemblyAI](https://www.assemblyai.com/blog/integrate-audio-langchainjs/)
- [Integrate Audio into LangChain.js apps in 5 Minutes](https://www.youtube.com/watch?v=hNpUSaYZIzs)
- [AssemblyAI JavaScript SDK](https://github.com/AssemblyAI/assemblyai-node-sdk)

---

<a id="vercel-ai-sdk-integration-with-assemblyai-docs-integrations-vercel-ai-sdk-mdx"></a>

# ▲ Vercel AI SDK Integration with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/vercel-ai-sdk
Source: docs/integrations/vercel-ai-sdk.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Transcribe audio with the Vercel AI SDK using the official @ai-sdk/assemblyai provider.

The [AI SDK](https://ai-sdk.dev) gives you a single, unified `transcribe()` API that works across speech-to-text providers. The [`@ai-sdk/assemblyai`](https://ai-sdk.dev/providers/ai-sdk-providers/assemblyai) provider — maintained by Vercel in the [`vercel/ai`](https://github.com/vercel/ai) repo — plugs AssemblyAI's speech models into that API, so you can transcribe audio in any TypeScript or JavaScript project without calling the AssemblyAI API directly.

## Quickstart

Install the AI SDK core package along with the AssemblyAI provider:

  

```bash
npm install ai @ai-sdk/assemblyai
```

  
  

```bash
yarn add ai @ai-sdk/assemblyai
```

  
  

```bash
pnpm add ai @ai-sdk/assemblyai
```

  
  

```bash
bun add ai @ai-sdk/assemblyai
```

  

You'll need an [AssemblyAI account](https://www.assemblyai.com/dashboard/signup) and an API key from your [dashboard](https://www.assemblyai.com/dashboard/home). The provider reads it from the `ASSEMBLYAI_API_KEY` environment variable:

```bash
export ASSEMBLYAI_API_KEY="<YOUR_API_KEY>"
```

Then transcribe a local file with `transcribe()`:

```typescript
import { transcribe } from 'ai';
import { assemblyai } from '@ai-sdk/assemblyai';
import { readFile } from 'node:fs/promises';

const result = await transcribe({
  model: assemblyai.transcription('universal-3-5-pro'),
  audio: await readFile('audio.mp3'),
});

console.log(result.text);
```

  `transcribe` is a stable export as of AI SDK v7, which is what `npm install ai`
  installs today. The examples on this page target v7.

  **Using AI SDK v6?** The examples on this page require v7. AI SDK v6 (and v5) export
  only `experimental_transcribe` — `import { transcribe } from 'ai'` throws on those
  versions — and the latest provider (`@ai-sdk/assemblyai` 3.x) is built for v7, so it
  won't work against v6. If your project is on v6 (for example, an existing project or
  a registry policy that hasn't picked up v7 yet), install the v6-compatible packages
  and alias the experimental export:

  ```bash
  npm install ai@ai-v6 @ai-sdk/assemblyai@ai-v6
  ```

  ```typescript
  import { experimental_transcribe as transcribe } from 'ai';
  ```

  Everything else on this page — the `providerOptions`, model IDs, and result shape —
  is the same.

## Configuring the provider

The default `assemblyai` instance reads your key from `ASSEMBLYAI_API_KEY`. To pass the key explicitly — for example, from a secret manager or in a multi-tenant setup — or to add custom headers or a custom `fetch` implementation, create your own provider instance with `createAssemblyAI`:

```typescript
import { createAssemblyAI } from '@ai-sdk/assemblyai';

const assemblyai = createAssemblyAI({
  apiKey: process.env.ASSEMBLYAI_API_KEY,
});

const result = await transcribe({
  model: assemblyai.transcription('universal-3-5-pro'),
  audio: await readFile('audio.mp3'),
});
```

## Choosing a speech model

`universal-3-5-pro` is the recommended default. `universal-2` is also supported. Pass the model ID to `assemblyai.transcription()`:

```typescript
const model = assemblyai.transcription('universal-3-5-pro');
```

For a breakdown of each model's accuracy, latency, and language support, see [Select the speech model](/pre-recorded-audio/select-the-speech-model).

## Using AssemblyAI features

AssemblyAI's diarization and audio-intelligence features are enabled through `providerOptions.assemblyai`. Option keys are the **camelCase** equivalents of the API's snake_case parameters (for example, `speaker_labels` → `speakerLabels`, `redact_pii` → `redactPii`):

```typescript
import { transcribe } from 'ai';
import {
  assemblyai,
  type AssemblyAITranscriptionModelOptions,
} from '@ai-sdk/assemblyai';
import { readFile } from 'node:fs/promises';

const result = await transcribe({
  model: assemblyai.transcription('universal-3-5-pro'),
  audio: await readFile('meeting.mp3'),
  providerOptions: {
    assemblyai: {
      speakerLabels: true,
      keytermsPrompt: ['AssemblyAI', 'Universal'],
      redactPii: true,
      redactPiiPolicies: ['person_name', 'email_address'],
    } satisfies AssemblyAITranscriptionModelOptions,
  },
});
```

Annotating the options object with `satisfies AssemblyAITranscriptionModelOptions` gives you editor autocomplete and type-checking for the full option set, so you don't have to memorize parameter names or hunt through docs.

  The table below reflects `@ai-sdk/assemblyai` 3.0.5. For the always-current list —
  including any options added in newer releases — see the
  [`@ai-sdk/assemblyai` provider reference](https://ai-sdk.dev/providers/ai-sdk-providers/assemblyai),
  which lives alongside the provider code and is the canonical source.

### Transcription options

Every option below is optional and passed under `providerOptions.assemblyai`. Types and behavior follow the AssemblyAI transcription API.

| Option | Type | Description |
| --- | --- | --- |
| `audioEndAt` | number | End time of the audio, in milliseconds. |
| `audioStartFrom` | number | Start time of the audio, in milliseconds. |
| `autoChapters` | boolean | Automatically generate chapters for the transcript. |
| `autoHighlights` | boolean | Automatically generate highlights for the transcript. |
| `boostParam` | enum | Boost level for `wordBoost`. Allowed values: `low`, `default`, `high`. **Deprecated** — applies only to the deprecated `wordBoost`; use `keytermsPrompt` instead. |
| `contentSafety` | boolean | Enable content safety filtering. |
| `contentSafetyConfidence` | number | Confidence threshold for content safety filtering (25-100). |
| `customSpelling` | array of objects | Custom spelling rules. Each object has `from` (array of strings) and `to` (string). |
| `disfluencies` | boolean | Include disfluencies (um, uh, etc.) in the transcript. |
| `domain` | string | Enable a domain-specific model for specialized terminology. Currently supports `medical-v1` (Medical Mode). |
| `entityDetection` | boolean | Detect entities in the transcript. |
| `filterProfanity` | boolean | Filter profanity in the transcript. |
| `formatText` | boolean | Apply text formatting to the transcript. |
| `iabCategories` | boolean | Include IAB categories in the transcript. |
| `keytermsPrompt` | array of strings | Domain-specific keyterms to boost recognition for (max 6 words per phrase). Replaces `wordBoost` for newer models — supported by `universal-3-pro`, `universal-3-5-pro`, and `slam-1` (and `universal-2` when enabled). |
| `languageCode` | string | Language code for the audio. Supports numerous ISO-639-1 and ISO-639-3 codes. |
| `languageConfidenceThreshold` | number | Confidence threshold for language detection. |
| `languageDetection` | boolean | Enable automatic language detection. |
| `languageDetectionOptions` | object | Options for language detection: `expectedLanguages` (array of strings), `fallbackLanguage` (string), `codeSwitching` (boolean), `codeSwitchingConfidenceThreshold` (number, 0-1). |
| `multichannel` | boolean | Process multiple audio channels separately. |
| `prompt` | string | Natural-language context (up to 1,500 words) to steer the model. Only supported by `universal-3-pro`, `universal-3-5-pro`, and `slam-1`. |
| `punctuate` | boolean | Add punctuation to the transcript. |
| `redactPii` | boolean | Redact personally identifiable information (PII). |
| `redactPiiAudio` | boolean | Redact PII in the audio file. |
| `redactPiiAudioOptions` | object | Options for PII-redacted audio: `returnRedactedNoSpeechAudio` (boolean), `overrideAudioRedactionMethod` (`silence`). Requires `redactPiiAudio`. |
| `redactPiiAudioQuality` | enum | Quality of the redacted audio file. Allowed values: `mp3`, `wav`. |
| `redactPiiPolicies` | array of enums | Which types of information to redact (e.g. `person_name`, `phone_number`). |
| `redactPiiReturnUnredacted` | boolean | Return the original unredacted transcript alongside the redacted one. Requires `redactPii`. |
| `redactPiiSub` | enum | Substitution method for redacted PII. Allowed values: `entity_name`, `hash`. |
| `redactStaticEntities` | object | Map of user-defined labels to exact terms to redact, e.g. `{ INTERNAL_TOOL: ['Bearclaw'] }`. Applied on top of standard PII redaction. Requires `redactPii`. |
| `removeAudioTags` | enum | Remove inline annotations from rich transcripts. Allowed values: `all`, `speaker`. Universal-3 Pro models. |
| `sentimentAnalysis` | boolean | Perform sentiment analysis on the transcript. |
| `speakerLabels` | boolean | Label different speakers in the transcript (diarization). |
| `speakerOptions` | object | Diarization options: `minSpeakersExpected` (number), `maxSpeakersExpected` (number). |
| `speakersExpected` | number | Expected number of speakers in the audio. |
| `speechThreshold` | number | Threshold for speech detection (0-1). |
| `summarization` | boolean | Generate a summary of the transcript. |
| `summaryModel` | enum | Summarization model. Allowed values: `informative`, `conversational`, `catchy`. |
| `summaryType` | enum | Type of summary. Allowed values: `bullets`, `bullets_verbose`, `gist`, `headline`, `paragraph`. |
| `temperature` | number | Sampling temperature (0-1) controlling randomness. Universal-3 Pro models. |
| `webhookAuthHeaderName` | string | Name of the authentication header for webhook requests. |
| `webhookAuthHeaderValue` | string | Value of the authentication header for webhook requests. |
| `webhookUrl` | string | URL to send webhook notifications to. |
| `wordBoost` | array of strings | Words to boost in the transcript. **Deprecated** — rejected by `universal-3-pro`, `universal-3-5-pro`, and `slam-1` (works only on `universal-2`/`best`); use `keytermsPrompt` instead. |

## Getting the full results back

`transcribe()` returns a provider-agnostic result, but AssemblyAI's richer output — utterances, entities, sentiment, and more — is preserved, so you don't lose anything by going through the AI SDK:

- **Top-level fields** like `result.text`, `result.segments`, and `result.durationInSeconds` are normalized across providers.
- **`result.providerMetadata.assemblyai`** holds AssemblyAI-specific results such as `utterances`, `entities`, and `sentimentAnalysisResults`.
- **`result.response.body`** is the complete, raw AssemblyAI transcript object.

```typescript
const result = await transcribe({
  model: assemblyai.transcription('universal-3-5-pro'),
  audio: await readFile('meeting.mp3'),
  providerOptions: {
    assemblyai: { speakerLabels: true, sentimentAnalysis: true },
  },
});

// Normalized, provider-agnostic fields
console.log(result.text);
console.log(result.segments);

// AssemblyAI-specific results
const { utterances, sentimentAnalysisResults } =
  result.providerMetadata.assemblyai;

// The complete, raw AssemblyAI transcript
const rawTranscript = result.response.body;
```

  Timestamps use **different units depending on where you read them.** The top-level
  `result.segments` are in **seconds** (the AI SDK's normalized format), while every
  timing inside `result.providerMetadata.assemblyai` and `result.response.body` is in
  **milliseconds** (AssemblyAI's native format). Convert accordingly when you combine
  the two.

## Additional resources

- [AssemblyAI provider reference on ai-sdk.dev](https://ai-sdk.dev/providers/ai-sdk-providers/assemblyai) — the canonical, complete list of provider options
- [`vercel/ai` on GitHub](https://github.com/vercel/ai) — the AI SDK source and where the provider is maintained
- [AssemblyAI API reference](/api-reference/overview) — the underlying transcription API and every parameter it accepts

---

<a id="integrate-power-automate-with-assemblyai-docs-integrations-power-automate-mdx"></a>

# Integrate Power Automate with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/power-automate
Source: docs/integrations/power-automate.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Use our Power Automate & Azure Logic Apps connector to use AssemblyAI's speech AI in your flows.

[Microsoft Power Automate](https://www.microsoft.com/en-us/power-platform/products/power-automate) is a low-code workflow automation platform with a rich collection of connectors to Microsoft's first-party services and third-party services. [Azure Logic Apps](https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-overview) is the equivalent service built for developers and IT pros.

The AssemblyAI connector makes our API available to both Microsoft Power Automate and Azure Logic Apps.
With the connector, you can use AssemblyAI to transcribe audio data with speech recognition models, analyze the data with speech understanding models, and build generative features on top of it with LLMs.
You can supply audio to the AssemblyAI connector and connect the output of our models to other services in your flows.

## Quickstart

Create or edit a flow in Power Automate.
Add a new action, search for AssemblyAI, and select the action that you want to use.

![Search for AssemblyAI actions in Power Automate](/assets/img/integrations/power-automate/search-action.png)

You will be prompted to create a connection to AssemblyAI. Give your connection a name and enter the API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home), and click **Create new**.

![Create a connection to AssemblyAI in Power Automate](/assets/img/integrations/power-automate/create-connection.png)

Finally, configure your AssemblyAI action. Continue reading to learn more about all the available actions.

![Configure an AssemblyAI action in Power Automate](/assets/img/integrations/power-automate/configure-action.png)

## Upload a File

To transcribe an audio file using AssemblyAI, the file needs to be accessible to AssemblyAI.
If your audio file is already accessible via a URL, you can use your existing URL.

Otherwise, you can use the `Upload a Media File` action to upload a file to AssemblyAI.
You will get back a URL for your file which can only be used to transcribe using your API key.
Once you transcribe the file, the file will be removed from AssemblyAI's servers.

## Transcribe Audio

To transcribe your audio, configure the `Audio URL` parameter using your audio file URL.
Then, configure the additional parameters to enable more [Speech Recognition](/pre-recorded-audio) features and [Speech Understanding](/speech-understanding) models.

The result of the Transcribe Audio action is a transcript that will start being processed immediately.
To get the completed transcript, you have two options:

1. [Handle the Transcript Ready Webhook](#handle-the-transcript-ready-webhook)
2. [Poll the Transcript Status](#poll-the-transcript-status)

### Handle the Transcript Ready Webhook

If you don't want to handle the webhook using Logic Apps or Power Automate, configure the `Webhook URL` parameter in your `Transcribe Audio` action, and implement your webhook following [AssemblyAI's webhook documentation](/pre-recorded-audio/webhooks#handle-webhook-deliveries).

To handle the webhook using Logic Apps or Power Automate, follow these steps:

Create a separate Logic App or Power Automate Flow.

Configure `When an HTTP request is received` as the trigger:

- Set `Who Can Trigger The Flow?` to `Anyone`
- Set `Request Body JSON Schema` to:
  ```json
  {
    "type": "object",
    "properties": {
      "transcript_id": {
        "type": "string"
      },
      "status": {
        "type": "string"
      }
    }
  }
  ```
- Set `Method` to `POST`

Add an AssemblyAI `Get Transcript` action, passing in the `transcript_id` from the trigger to the `Transcript ID` parameter.

Before doing anything else, you should check whether the `Status` is `completed` or `error`.
Add a `Condition` action that checks if the `Status` from the `Get Transcript` output is `error`:

- In the `True` branch, add a `Terminate` action
  - Set the `Status` to `Failed`
  - Set the `Code` to `Transcript Error`
  - Pass the `Error` from the `Get Transcript` output to the `Message` parameter.
- You can leave the `False` branch empty.

Now you can add any action after the `Condition` knowing the transcript status is `completed`,
and you can retrieve any of the output properties of the `Get Transcript` action.

Save your Logic App or Flow. The `HTTP URL` will be generated for the `When an HTTP request is received` trigger.
Copy the `HTTP URL` and head back to your original Logic App or Flow.

In your original Logic App or Flow, update the `Transcribe Audio` action.
Paste the `HTTP URL` you copied previously into the `Webhook URL` parameter, and save.

When the transcript status becomes `completed` or `error`, AssemblyAI will send an HTTP POST request to the webhook URL,
which will be handled by your other Logic App or Flow.

As an alternative to using the webhook, you can poll the transcript status as explained in the next section.

### Poll the Transcript Status

You can poll the transcript status using the following steps:

Add an `Initialize variable` action

- Set `Name` to `transcript_status`
- Set `Type` to `String`
- Store the `Status` from the `Transcribe Audio` output into the `Value` parameter

Add a `Do until` action

- Configure the `Loop Until` parameter with the following Fx code:
  ```plaintext
  or(equals(variables('transcript_status'), 'completed'), equals(variables('transcript_status'), 'error'))
  ```
  This code checks whether the `transcript_status` variable is `completed` or `error`.
- Configure the `Count` parameter to `86400`
- Configure the `Timeout` parameter to `PT24H`

Inside the `Do until` action, add the following actions:

- Add a `Delay` action that waits for one second
- Add a `Get Transcript` action and pass the `ID` from the `Transcribe Audio` output to the `Transcript ID` parameter
- Add a `Set variable` action
  - Set `Name` to `transcript_status`
  - Pass the `Status` of the `Get Transcript` output to the `Value` parameter

The `Do until` loop will continue until the transcript is completed, or an error occurred.

Add another `Get Transcript` action, like before, but add it after the `Do until` loop so its output becomes available outside the scope of the `Do until` action.

Before doing anything else, you should check whether the transcript `Status` is `completed` or `error`.
Add a `Condition` action that checks if the `transcript_status` is `error`:

- In the `True` branch, add a `Terminate` action
  - Set `Status` to `Failed`
  - Set `Code` to `Transcript Error`
  - Pass the `Error` from the `Get Transcript` output to the `Message` parameter.
- You can leave the `False` branch empty.

Now you can add any action after the `Condition` knowing the transcript status is `completed`,
and you can retrieve any of the output properties of the `Get Transcript` action.

## Connector actions

The AssemblyAI app for Power Automate provides the following actions:

### Files

#### Upload a Media File

Upload a media file to AssemblyAI's servers.
You can pass the `Upload URL` output field to the `Audio URL` input field of [Transcribe an Audio File](#transcribe-audio) action.

### Transcripts

#### Transcribe Audio

Create a transcript from a media file that is accessible via a URL.
Configure the `Audio URL` field with the URL of the audio file you want to transcribe.
The `Audio URL` must be accessible by AssemblyAI's servers.
If you don't have a publicly accessible URL, you can use the [Upload a File](#upload-a-file) action to upload the audio file to AssemblyAI.

  **Wait until transcript is ready**

The output of this action is a transcript that is not yet `completed`. Learn [how to wait until the transcript is ready here](#transcribe-audio).

  Configure your desired [Speech Understanding models](/speech-understanding)
  when you create the transcript. The results of the models will be included in
  the transcript output when the transcript is completed.

#### Get Transcript

Get the transcript resource. The transcript is ready when the `status` is `completed`.

#### Get Paragraphs in Transcript

Get the transcript split by paragraphs. The API semantically segments your transcript into paragraphs to create more reader-friendly transcripts.

You can only invoke this action after the transcript is completed.

#### Get Sentences in Transcript

Get the transcript split by sentences. The API semantically segments the transcript into sentences to create more reader-friendly transcripts.

You can only invoke this action after the transcript is completed.

#### Get Subtitles for Transcript

Get the transcript resource. The transcript is ready when the `status` is `completed`.

You can only invoke this action after the transcript is completed.

#### Get Redacted Audio

First, you need to configure PII audio redaction using these fields when you create the transcript:

- `Redact PII`: `Yes`
- `Redact PII Audio`: `Yes`
- `Redact PII Policies`: Configure at least one PII policy

Then, you can use this action to retrieve the redacted audio of the transcript.

You can only invoke this action after the transcript is completed.

#### Search Words in Transcript

Search through the transcript for keywords. You can search for individual words, numbers, or phrases containing up to five words or numbers.

You can only invoke this action after the transcript is completed.

#### List Transcripts

Get the transcript resource. The transcript is ready when the `status` is `completed`.

#### Delete Transcript

Delete the transcript. Deleting does not delete the resource itself, but removes the data from the resource and marks it as deleted.

  You can only invoke this action after the transcript status is `completed` or
  `error`.

## Additional resources

You can learn more about using Power Automate with AssemblyAI in these resources:

- [Power Automate & Logic Apps docs by Microsoft](https://learn.microsoft.com/en-us/connectors/assemblyai/)

---

<a id="semantic-kernel-integration-for-assemblyai-docs-integrations-semantic-kernel-mdx"></a>

# Semantic Kernel Integration for AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/semantic-kernel
Source: docs/integrations/semantic-kernel.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Transcribe audio in Semantic Kernel for C# .NET using the built-in integration with AssemblyAI.

Semantic Kernel is an SDK for multiple programming languages to develop applications with [Large Language Models (LLMs)](https://www.assemblyai.com/blog/introduction-large-language-models-generative-ai/#what-are-language-models).
However, LLMs only operate on textual data and don't understand what is said in audio files.
With the [AssemblyAI integration for Semantic Kernel](https://github.com/AssemblyAI/assemblyai-semantic-kernel), you can use AssemblyAI's transcription models using the `TranscribePlugin` to transcribe your audio and video files.

## Quickstart

Add the [AssemblyAI.SemanticKernel NuGet package](https://www.nuget.org/packages/AssemblyAI.SemanticKernel) to your project.

  

```bash
dotnet add package AssemblyAI.SemanticKernel
```

  
  

```powershell
Install-Package AssemblyAI.SemanticKernel
```

  

Next, register the `TranscriptPlugin` into your kernel:

```csharp
using AssemblyAI.SemanticKernel;
using Microsoft.SemanticKernel;

// Build your kernel
var kernel = Kernel.CreateBuilder();

// Get AssemblyAI API key from env variables, or much better, from .NET configuration
string apiKey = Environment.GetEnvironmentVariable("ASSEMBLYAI_API_KEY")
  ?? throw new Exception("ASSEMBLYAI_API_KEY env variable not configured.");

kernel.ImportPluginFromObject(
    new TranscriptPlugin(apiKey: apiKey)
);
```

## Usage

Get the `Transcribe` function from the transcript plugin and invoke it with the context variables.

```csharp
var result = await kernel.InvokeAsync(
    nameof(TranscriptPlugin),
    TranscriptPlugin.TranscribeFunctionName,
    new KernelArguments
    {
        ["INPUT"] = "https://assembly.ai/espn.m4a"
    }
);
Console.WriteLine(result.GetValue<string>());
```

You can get the transcript using `result.GetValue<string>()`.

You can also upload local audio and video file. To do this:

- Set the `TranscriptPlugin.AllowFileSystemAccess` property to `true`.
- Configure the `INPUT` variable with a local file path.

```csharp
kernel.ImportPluginFromObject(
    new TranscriptPlugin(apiKey: apiKey)
    {
        AllowFileSystemAccess = true
    }
);
var result = await kernel.InvokeAsync(
    nameof(TranscriptPlugin),
    TranscriptPlugin.TranscribeFunctionName,
    new KernelArguments
    {
        ["INPUT"] = "https://assembly.ai/espn.m4a"
    }
);
Console.WriteLine(result.GetValue<string>());
```

You can also invoke the function from within a semantic function like this.

```csharp
string prompt = """
                Here is a transcript:
                {{TranscriptPlugin.Transcribe "https://assembly.ai/espn.m4a"}}
                ---
                Summarize the transcript.
                """;
var result = await kernel.InvokePromptAsync(prompt);
Console.WriteLine(result.GetValue<string>());
```

## Additional resources

You can learn more about using Semantic Kernel with AssemblyAI in these resources:

- [Ask .NET Rocks! questions with Semantic Kernel, GPT, and Chroma DB](https://www.assemblyai.com/blog/ask-dotnetrocks-questions-semantic-kernel/)
- [AssemblyAI integration for Semantic Kernel GitHub repository](https://github.com/AssemblyAI/assemblyai-semantic-kernel)

---

<a id="integrate-activepieces-with-assemblyai-docs-integrations-activepieces-mdx"></a>

# Integrate Activepieces with AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/activepieces
Source: docs/integrations/activepieces.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Add Speech AI to your Activepieces flows with the AssemblyAI piece.

[Activepieces](https://www.activepieces.com/) is an open-source, no-code automation platform that enables users to streamline workflows by connecting various applications and automating tasks.

With the AssemblyAI piece for Activepieces, you can use AssemblyAI to transcribe audio data with speech recognition models, analyze the data with speech understanding models, and build generative features on top of it with LLMs.
You can supply audio to the AssemblyAI piece and connect the output of any of AssemblyAI's models to other services in your Activepieces flow.

## Quickstart

Create or edit a flow in Activepiece. Add a trigger of your choosing, and then click the plus-icon to add a new action.
Search for AssemblyAI, click on the AssemblyAI piece, and select the action that you want to use.

![Add an AssemblyAI piece action](/assets/img/integrations/activepieces/add-action.png)

Create a new connection or select an existing one.

![Select a connection to AssemblyAI in Activepieces](/assets/img/integrations/activepieces/select-api-key.png)

In the **API Key** field, enter the API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home), and click **Save**.

![Configure your connection to AssemblyAI in Activepieces](/assets/img/integrations/activepieces/configure-api-key.png)

Finally, configure your AssemblyAI action. Continue reading to learn more about all the available action.

## AssemblyAI actions

The AssemblyAI piece for Activepieces provides the following actions:

### Files

#### Upload File

Upload an audio file to AssemblyAI so you can transcribe it.
You can pass the `Upload URL` output field to the `Audio URL` input field of Transcribe an Audio File module.

### Transcripts

#### Transcribe

Transcribe an audio file and wait until the transcript has completed or failed.
Configure the `Audio URL` field with the URL of the audio file you want to transcribe.
The `Audio URL` must be accessible by AssemblyAI's servers.
If you don't have a publicly accessible URL, you can use the Upload a File module to upload the audio file to AssemblyAI.

If you don't want to wait until the transcript is ready, uncheck the `Wait until transcript is ready` parameter.

  Configure your desired [Speech Understanding models](/speech-understanding)
  when you create the transcript. The results of the models will be included in
  the transcript output.

#### Get Transcript

Retrieve a transcript by ID.

#### Get Transcript Paragraphs

Retrieve the paragraphs of a transcript.

You can only invoke this module after the transcript is completed.

#### Get Transcript Sentences

Retrieve the sentences of a transcript.

You can only invoke this module after the transcript is completed.

#### Get Transcript Subtitles

Create SRT or VTT subtitles for a transcript.

You can only invoke this module after the transcript is completed.

#### Get Transcript Redacted Audio

First, you need to configure PII audio redaction using these fields when you create the transcript:

- `Redact PII`: `Checked`
- `Redact PII Audio`: `Checked`
- `Redact PII Policies`: Configure at least one PII policy

Then, you can use this module to retrieve the redacted audio of the transcript.

You can only invoke this module after the transcript is completed.

#### Search words in transcript

Search for words in a transcript.

You can only invoke this module after the transcript is completed.

#### List transcripts

Paginate over all transcripts.

#### Delete transcript

Delete a transcript by ID.
Deleting a transcript doesn't delete the transcript resource itself, but removes the data from the resource and marks it as deleted.

  You can only invoke this module after the transcript status is "completed" or
  "error".

### Other actions

#### Custom API Call

Make your own REST API HTTP requests to the AssemblyAI API using your existing connection.

## Additional resources

You can learn more about using Activepieces with AssemblyAI in these resources:

- [AssemblyAI Integrations on Activepieces](https://www.activepieces.com/pieces/assemblyai)
- [npmjs page for @activepieces/piece-assemblyai](https://www.npmjs.com/package/@activepieces/piece-assemblyai)

---

<a id="haystack-integration-for-assemblyai-docs-integrations-haystack-mdx"></a>

# Haystack Integration for AssemblyAI

URL: https://www.assemblyai.com/docs/integrations/haystack
Source: docs/integrations/haystack.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Transcribe, summarize and diarize audio in a Haystack pipeline with Python using the integration with AssemblyAI.

[Haystack (2.x)](https://github.com/deepset-ai/haystack) is an open-source Python framework for building custom LLM applications. The Haystack Integration for AssemblyAI seamlessly integrates with Haystack to use audio files in LLM pipelines.

On top of audio transcription the AssemblyAITranscriber offers summarization and speaker diarization. This makes it possible to not only convert audio to text but also obtain concise summaries and identify speakers in a conversation.

## Quickstart

Install the [assemblyai-haystack package](https://pypi.org/project/assemblyai-haystack/) using pip. This package installs and uses the AssemblyAI Python SDK and Haystack 2.0. You can find more information about the SDK at the [AssemblyAI Python SDK GitHub repository](https://github.com/AssemblyAI/assemblyai-python-sdk).

```bash
pip install assemblyai-haystack
```

## Usage

Add an `AssemblyAITranscriber` component and initialize it by passing your AssemblyAI API key. Once the pipeline is ready to run, make sure to pass at least the `file_path` argument to the `run` function. The `file_path` can be a `URL` or a local file path. In the `run` function, you can also specify whether you want summarization and speaker diarization results.

```python expandable
import os

from assemblyai_haystack.transcriber import AssemblyAITranscriber
from haystack import Pipeline
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore

ASSEMBLYAI_API_KEY = os.environ.get("ASSEMBLYAI_API_KEY")

document_store = InMemoryDocumentStore()
file_url = "https://assembly.ai/wildfires.mp3"

indexing = Pipeline()
indexing.add_component("transcriber", AssemblyAITranscriber(api_key=ASSEMBLYAI_API_KEY))
indexing.add_component("writer", DocumentWriter(document_store))
indexing.connect("transcriber.transcription", "writer.documents")
indexing.run(
    {
        "transcriber": {
            "file_path": file_url,
            "summarization": None,
            "speaker_labels": None,
        }
    }
)

print("Indexed Document Count:", document_store.count_documents())
```

  Calling `indexing.run()` blocks until the transcription is finished.

The results of the transcription, summarization and speaker diarization are returned in separate document lists:

- `transcription`
- `summarization`
- `speaker_labels`

When `AssemblyAITranscriber` is used in a Haystack pipeline, transcription happens by default. In the metadata of the transcription, you will also get the `ID` of the transcription and the `URL` of your audio file.

A bullet point summary of what is being discussed will be returned if `summarization` is set to `TRUE`. The transcription divided into utterances of speakers will be returned if `speaker_labels` is set to `TRUE`.

The output of the `AssemblyAITranscriber` is a Haystack document. When all features are turned on, the created document looks like this:

```python expandable
{
"transcription":
  [Document(
    id=bdf3eb20f6440cf4b15fa4fa3176eeb72bf0139a3ad4c76741724132907a5daa,
    content: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skyli...",
    meta: {
      'transcript_id': '2335cc07-1fbf-48ba-9855-7db3eeeb80f4',
      'audio_url': "https://assembly.ai/wildfires.mp3"
      }
    )
  ],

"summarization":
  [Document(
    id=f88864d9229b30013d5248156e74d5bfd4435e73aadb0c0ce79040be10a4f308,
    content: "- Smoke from hundreds of wildfires in Canada is triggering air quality alerts...")],

"speaker_labels":
  [Document(
    id=a7e222bc6a965ab1032401a6fa22da2e774294ce049b9d228acbb8b100ea2ecf,
    content: "Smoke from hundreds of wildfires in Canada is triggering air quality...",
    meta: {
      'speaker': 'A'
      }
    ),

  Document(
    id=711a1888af58601e6392490a5e4ca4c10958f93a52d8f0734869c54573ea76f5,
    content: "Well, there's a couple of things. The season has been pretty dry already...",
    meta: {
      'speaker': 'B'
      }
    ),

  Document(
    id=8fc78631d420e2e6127b8bdff2830f693febb91ed1566b9a84527cf023023d9e,
    content: "So what is it in this haze that makes it harmful?",
    meta: {
      'speaker': 'A'
      }
    ),

    ...
]}
```

## Additional resources

You can learn more about using Haystack with AssemblyAI in these resources:

- [Announcing the AssemblyAI Integration for Haystack](https://www.assemblyai.com/blog/announcing-the-assemblyai-integration-for-haystack/)
- [AssemblyAI integration for Haystack GitHub repository](https://github.com/AssemblyAI/assemblyai-haystack)

---

<a id="cloudflare-docs-nav-links-cloudflare-mdx"></a>

# Cloudflare

URL: https://www.assemblyai.com/docs/nav-links/cloudflare
Source: docs/nav-links/cloudflare.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Open Cloudflare.

---

<a id="relay-app-docs-nav-links-relay-app-mdx"></a>

# Relay.app

URL: https://www.assemblyai.com/docs/nav-links/relay-app
Source: docs/nav-links/relay-app.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Open Relay.app.

---

<a id="bubble-by-knowcode-docs-nav-links-bubble-by-knowcode-mdx"></a>

# Bubble by Knowcode

URL: https://www.assemblyai.com/docs/nav-links/bubble-by-knowcode
Source: docs/nav-links/bubble-by-knowcode.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Open Bubble by Knowcode.

---

<a id="pipedream-docs-nav-links-pipedream-mdx"></a>

# Pipedream

URL: https://www.assemblyai.com/docs/nav-links/pipedream
Source: docs/nav-links/pipedream.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Open Pipedream.

---

<a id="drupal-docs-nav-links-drupal-mdx"></a>

# Drupal

URL: https://www.assemblyai.com/docs/nav-links/drupal
Source: docs/nav-links/drupal.mdx
Navigation: Overview > Use cases & integrations > Integrations > Community-maintained tools
Description: Open Drupal.

---

<a id="trust-center-docs-nav-links-trust-center-mdx"></a>

# Trust center

URL: https://www.assemblyai.com/docs/nav-links/trust-center
Source: docs/nav-links/trust-center.mdx
Navigation: Overview > Trust & security
Description: Open Trust center.

---

<a id="security-overview-docs-nav-links-security-overview-mdx"></a>

# Security overview

URL: https://www.assemblyai.com/docs/nav-links/security-overview
Source: docs/nav-links/security-overview.mdx
Navigation: Overview > Trust & security
Description: Open Security overview.

---

<a id="data-controls-docs-data-controls-mdx"></a>

# Data Controls

URL: https://www.assemblyai.com/docs/data-controls
Source: docs/data-controls.mdx
Navigation: Overview > Trust & security
Description: Manage how AssemblyAI retains and uses your data — opt out of the model improvement program, set a time-to-live for audio and transcripts, and sign a BAA, all self-serve from the dashboard.

The [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the AssemblyAI dashboard lets you manage how AssemblyAI retains and uses your data. From a single page you can opt out of the model improvement program, set a time-to-live (TTL) for your audio and transcripts, and review and sign a Business Associate Agreement (BAA).

The page always reflects the **current state of your account** so you can see, at a glance, your model-improvement opt-out status, your active data-retention (TTL) setting, and whether a BAA is in place.

  All three of these controls — opt out, TTL, and BAA — are available **self-serve at no additional cost**. You can manage them directly from the dashboard without contacting sales or support.

## Who can access Data Controls

  You must be on a **paid plan** to access Data Controls. Free users cannot opt out of the model improvement program, set a TTL, or sign a BAA. To access these settings, [upgrade to a paid plan](https://www.assemblyai.com/dashboard/pricing).

Only members with the **Owner** or **Admin** role can view and change Data Controls settings. Members with the **Reader** role do not have access to this page. For more on roles and permissions, see [Account Management](/account-management#roles).

## Opt out of the model improvement program

Toggle **Opt Out of Data Sharing for Model Improvement Program** to control whether AssemblyAI may use your Customer Data to train its models. When you opt out, at this time:

- AssemblyAI will not use your Customer Data to train its artificial intelligence and machine learning models.
- AssemblyAI will not use your Customer Data to perform benchmarking.
- AssemblyAI will not use your Deidentified Data to train its artificial intelligence and machine learning models.

Opt-out changes are forward-looking only and apply to subsequent new requests.

For full details on how model training works, see [Data retention and model training](/data-retention-and-model-training#model-training).

## Set a time-to-live for audio and transcripts

The **time-to-live (TTL)** mechanism controls how long AssemblyAI retains your audio and transcripts in the asynchronous production environment. When you set a TTL, AssemblyAI begins the deletion process for your audio and transcripts at the set TTL time. **This TTL also applies to the inputs and outputs of LLM Gateway.** Certain metadata is stored for logging and billing purposes.

You can choose a preset (1 day, 3 days, 7 days, or 30 days) or set a **Custom** value. Any changes are applied to subsequent new requests.

For more detail, including information on potential TTL deletion lag times, see [Data retention and model training](/data-retention-and-model-training#data-retention).

## Business Associate Agreement (BAA)

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). If you need a BAA in place to process PHI, you can review and sign AssemblyAI's standard online BAA terms directly from the Data Controls page.

The page shows your current BAA status (for example, **Not signed**). To execute a BAA, select **Review & sign BAA** and complete the online signing flow.

  You must have an upgraded (paid) account to see the option to initiate a BAA in the dashboard. If you're on the free plan, the BAA controls are hidden — [upgrade](https://www.assemblyai.com/dashboard/pricing) first.

### How signing a BAA affects your opt-out status

  Once you have signed a BAA, you are **automatically opted out** of the model improvement program upon signature, and the **opt-out toggle can no longer be changed**. AssemblyAI does not use files submitted under a BAA for model training.

Signing a BAA also changes your default data retention behavior. For BAA-specific retention timelines, see the [asynchronous production environment](/data-retention-and-model-training#asynchronous-production-environment) and [LLM Gateway](/data-retention-and-model-training#llm-gateway-production-environment) retention tables.

## Related resources

  
    Learn how AssemblyAI handles data retention, encryption, model training, and compliance.
  
  
    Understand roles and permissions, including who can access Data Controls.

---

<a id="data-retention-and-model-training-docs-data-retention-and-model-training-mdx"></a>

# Data retention and model training

URL: https://www.assemblyai.com/docs/data-retention-and-model-training
Source: docs/data-retention-and-model-training.mdx
Navigation: Overview > Trust & security
Description: Learn about how AssemblyAI handles data retention, encryption, model training, and compliance.

## Model training

We consider model training critical to providing you with the most accurate models and services that we can. Only certain files submitted to the API, as permitted by the applicable contract, are used for model training. These files undergo a redaction process designed to redact personally identifiable information before any remaining data is used for model training. We will not use files you submit for model training if you are subject to a Business Associate Addendum, are utilizing [our European servers](/pre-recorded-audio/select-the-region), or if you have opted out from model training. You can find more information on if and how to opt out [here](/faq/how-to-opt-out-of-data-sharing-for-our-model-improvement-program).

## LLM Gateway model training

AssemblyAI has opted out of data training with all LLM Gateway providers.

Please note this is separate from whether AssemblyAI may train our models with your data. You can find more information on if and how to opt out of data sharing for our model improvement program [here](/faq/how-to-opt-out-of-data-sharing-for-our-model-improvement-program).

## Encryption

Data at rest is encrypted with AES 128 or AES-256, and data in transit uses TLS 1.2+. AssemblyAI posts SSL scans quarterly to its [Trust Center](https://app.vanta.com/assemblyai/trust/7n80syl8zln1bn1qm3x8eg) to verify the use of TLS with modern ciphersuites to its service.

### Async

For transcription of pre-recorded audio, AssemblyAI supports the following TLS versions and cipher suites:

**Supported TLS versions:**

- TLS 1.3
- TLS 1.2

**Supported cipher suites:**

- TLS_AES_128_GCM_SHA256
- TLS_AES_256_GCM_SHA384
- TLS_CHACHA20_POLY1305_SHA256
- ECDHE-ECDSA-AES128-GCM-SHA256
- ECDHE-RSA-AES128-GCM-SHA256
- ECDHE-ECDSA-AES256-GCM-SHA384
- ECDHE-RSA-AES256-GCM-SHA384

### Streaming

For transcription of streaming audio, AssemblyAI supports the following TLS version and cipher suites:

**Supported TLS versions:**

- TLS 1.3

**Supported cipher suites:**

- TLS_AES_128_GCM_SHA256
- TLS_AES_256_GCM_SHA384
- TLS_CHACHA20_POLY1305_SHA256

Ensure your client or application is configured to use one of the supported TLS versions and cipher suites when connecting to AssemblyAI services.

## GDPR compliance

We have designed our products with GDPR principles top of mind but also understand that privacy compliance is a moving target. As privacy requirements continue to evolve (rapidly), we are constantly working to assess and improve our practices. You can read more about our privacy practices in our Privacy Policy [here](https://www.assemblyai.com/legal/privacy-policy), and Data Processing Addendum [here](https://www.assemblyai.com/legal/data-processing-addendum).

## SOC2 certification

We have both SOC2 Type 1 and Type 2 certifications. You can find more information on this on our [Trust Center](https://www.assemblyai.com/trust). We also have a great blog post on the subject, which you can find [here](https://www.assemblyai.com/blog/assemblyai-obtains-soc2-type-2-compliance-for-2022-2023/).

## Data retention

### Streaming production environment

If you are opted out of model training, we offer zero data retention of audio and transcripts for our Streaming product. Certain metadata about the transcript is stored and maintained for logging and billing purposes.

The model training environment differs from the production environment. You can find more information on model training in our [Model Training section](#model-training). If you would like to opt out of model training, please see our [Opt-Out FAQ](/faq/how-to-opt-out-of-data-sharing-for-our-model-improvement-program).

### Asynchronous production environment

| Artifact Type\*                                                                                                       | Time-To-Live Configured                                                                                                                                                                   | BAA Executed                                                                                                                                                                                   | No TTL or BAA                                                         | Customer-Initiated Deletion Request               |
| --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------- |
| **Customer-Uploaded Audio Files**                                                                                     | Deletion process begins in AWS at TTL expiration, which can be as low as one (1) hour, subject to AWS TTL processing times\*\*<br/><br/>If TTL is longer than 24 hours, deletion process will automatically begin at 24 hours and is at most 48 hours. | Default deletion process begins at 72 hours (or can be set to as low as 1-hour), subject to processing times due to the AWS TTL\*\*                                                          | Deletion process begins at 24 hours and is at most 48 hours.          | Deleted when customer initiates deletion request. |
| **Final Transcription Artifact**                                                                                      | Deletion process begins in AWS at TTL expiration, which can be as low as one (1) hour, subject to AWS TTL processing times\*\*                                                            | Default deletion process begins at 72 hours (or can be set to as low as 1-hour), subject to processing times due to the AWS TTL\*\*                                                            | Deletion process begins at 30 days and is automatic thereafter.       | Deleted when customer initiates deletion request. |
| **Customer-Provided URL Audio Reference**                                                                             | Linked to the lifecycle of a Final Transcription Artifact: Deletion process begins in AWS at TTL expiration, which can be as low as one (1) hour, subject to AWS TTL processing times\*\* | Linked to the lifecycle of a Final Transcription Artifact: Default deletion process begins at 72 hours (or can be set to as low as 1-hour), subject to processing times due to the AWS TTL\*\* | Linked to the lifecycle of a Final Transcription Artifact: Deletion process begins at 30 days and is automatic thereafter. | Deleted when customer initiates deletion request. |
| **Intermediate Artifact**                                                                                             | Linked to the lifecycle of a Final Transcription Artifact: Deletion process begins in AWS at TTL expiration, which can be as low as one (1) hour, subject to AWS TTL processing times\*\*<br/><br/>If TTL is longer than 48 hours, deletion process will automatically begin at 48 hours and is at most 72 hours. | Linked to the lifecycle of a Final Transcription Artifact: Default deletion process begins at 72 hours (or can be set to as low as 1-hour), subject to processing times due to the AWS TTL\*\* | Deletion process begins at 48 hours and is at most 72 hours.          | Deleted when customer initiates deletion request. |
| **Transcription Request Text Inputs** (this includes content inputs such as key terms prompts, word boost list, etc.) | Linked to the lifecycle of a Final Transcription Artifact: Deletion process begins in AWS at TTL expiration, which can be as low as one (1) hour, subject to AWS TTL processing times\*\* | Linked to the lifecycle of a Final Transcription Artifact: Default deletion process begins at 72 hours (or can be set to as low as 1-hour), subject to processing times due to the AWS TTL\*\* | Linked to the lifecycle of a Final Transcription Artifact: Deletion process begins at 30 days and is automatic thereafter. | Deleted when customer initiates deletion request. |

\*Certain metadata is stored for logging and billing purposes.

\*\*The minimum TTL that AssemblyAI may set for Final Transcription Artifacts in the asynchronous production environment is 1 (one) hour. The TTL mechanism that AssemblyAI uses is through Amazon Web Services's ("AWS") DynamoDB TTL mechanism (the "AWS TTL"). The deletion process begins in AWS at TTL expiration, but is subject to AWS TTL processing times. In practice, these deletion events typically take place anywhere from a few minutes to a few hours after the deletion process begins in AWS, depending on circumstances, including server location. However, we have seen lag times anywhere from 2-3 hours to a few days. Once the artifact is deleted in AWS, AssemblyAI processes this deletion almost immediately. See [here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html) for more information about AWS's TTL mechanism.

  To learn how to delete a transcript via the API, see [Delete transcripts](/pre-recorded-audio/delete-transcripts).

#### Confirming deletion

Should you wish to confirm a file has been deleted, or in case you did not store the `transcript_id` when the transcription request was made, you can get a list of all transcripts. You can make a GET request to `https://api.assemblyai.com/v2/transcript` which will return a list of all transcripts created or specify a `transcript_id` to review a single transcript.

The model training environment differs from the production environment. You can find more information on model training in our [Model Training section](#model-training). If you would like to opt out of model training, please see our [Opt-Out FAQ](/faq/how-to-opt-out-of-data-sharing-for-our-model-improvement-program).

### LLM Gateway production environment

- If you have an executed BAA and use either Anthropic or Google inference models, we offer zero data retention for LLM Gateway inputs and outputs. Certain metadata is stored for logging and billing purposes.

- If you have a designated TTL on your LLM Gateway account, we delete inputs and outputs on an hourly basis. Certain metadata is stored for logging and billing purposes.

- If a customer initiates a deletion request, inputs and outputs are deleted at the time of the request. Certain metadata is stored for logging and billing purposes. For deletion of speech understanding requests, please see below.

- For speech understanding requests, such as translation, speaker ID, or custom formatting, the retention is linked to the life of an asynchronous Final Transcription Artifact, noted above.

#### Provider-specific retention policies

**Anthropic Claude models**

We use Anthropic models through Amazon Bedrock. Amazon Bedrock doesn't store or log your prompts and completions. Amazon Bedrock doesn't use your prompts and completions to train any AWS models and doesn't distribute them to third parties. See [here](https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html) for more information on Amazon Bedrock data protection policies.

If Amazon Bedrock fails, for non-EU customers we may send your request to the Anthropic API, where we have 0-day retention configured. Please see Anthropic's commercial terms [here](https://www.anthropic.com/legal/commercial-terms).

**OpenAI GPT models**

OpenAI models have ZDR (zero data retention). See OpenAI's policy [here](https://developers.openai.com/api/docs/guides/your-data#zero-data-retention) for more information on how OpenAI defines ZDR.

For OpenAI open-weight models (gpt-oss-120b, gpt-oss-20b), we use these models through Amazon Bedrock. Amazon Bedrock doesn't store or log your prompts and completions. Amazon Bedrock doesn't use your prompts and completions to train any AWS models and doesn't distribute them to third parties. See [here](https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html) for more information on Amazon Bedrock data protection policies.

**Google Gemini models**

Google Gemini models have ZDR (zero data retention). See Google's policy [here](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/vertex-ai-zero-data-retention) for more information on how Google defines ZDR.

#### Opt-in providers

The following providers are opt-in only. To use them with Speech Understanding or the LLM Gateway, opt in from the [Data Controls](/data-controls) page in your dashboard. The providers listed above are enabled by default when you use Speech Understanding or the LLM Gateway with those models by name.

**Together AI**

Together AI has ZDR (zero data retention) enabled. See Together AI's [privacy policy](https://www.together.ai/privacy) for more information.

**Fireworks AI**

Fireworks AI has [ZDR by default](https://docs.fireworks.ai/guides/security_compliance/data_handling#zero-data-retention). See Fireworks AI's [data handling documentation](https://docs.fireworks.ai/guides/security_compliance/data_handling#zero-data-retention) for more information.

**DigitalOcean**

See DigitalOcean's [inference data privacy documentation](https://docs.digitalocean.com/products/inference/details/data-privacy/) for more information.

AssemblyAI has opted out of model training with all LLM Gateway providers. Please note this is separate from whether AssemblyAI may train our models with your data, and the model training environment differs from the production environment. You can find more information on model training in our [Model Training section](#model-training). If you would like to opt out of model training, please see our [Opt-Out FAQ](/faq/how-to-opt-out-of-data-sharing-for-our-model-improvement-program).

---

<a id="api-reference-docs-api-reference-overview-mdx"></a>

# API Reference

URL: https://www.assemblyai.com/docs/api-reference/overview
Source: docs/api-reference/overview.mdx
Navigation: API Reference > Overview
Description: Choose an API to view its full reference.

Pick a product to jump to its full API reference. Each product tab has its own reference section with all endpoints, request and response schemas, and example calls.

  
    Get clean, customizable transcripts in 99 languages with industry-leading accuracy and natural language prompting.
  
  
    Real-time transcription your notetaker, agents, and captions can depend on.
  
  
    Send an audio file in a single HTTP request, get a transcript back in milliseconds. No polling, no session management.
  
  
    One OpenAI-compatible API for every frontier model — with automatic fallbacks, zero markup, and zero data retention.
  
  
    Stream audio in, get audio back. We handle the rest so you can focus on your product.

---

<a id="quickstart-docs-pre-recorded-audio-getting-started-transcribe-an-audio-file-mdx"></a>

# Quickstart

URL: https://www.assemblyai.com/docs/pre-recorded-audio/getting-started/transcribe-an-audio-file
Source: docs/pre-recorded-audio/getting-started/transcribe-an-audio-file.mdx
Navigation: Pre-recorded STT > Getting started
Description: Learn how to transcribe and analyze an audio file.

## Overview

By the end of this guide, you'll have a working script that transcribes an audio file in a single SDK call. Build it with an AI coding agent, or write it yourself — both are below.

Prefer to try it first? Transcribe audio without writing any code in the [AssemblyAI Playground](https://www.assemblyai.com/dashboard/playground/transcript).

## Before you begin

You'll need:

- **An API key** — grab one from [your dashboard](https://www.assemblyai.com/dashboard/home). Every example below reads it from an environment variable, so set it once:

  ```bash
  export ASSEMBLYAI_API_KEY=<your-key>
  ```

- **Python 3.8+ or Node.js 18+**, depending on which SDK you use.

**Building with an AI coding agent?** Wire it up to AssemblyAI's live docs (MCP server) and the AssemblyAI skill so it writes correct, up-to-date code instead of relying on stale training data:

```bash
claude mcp add --transport http --scope user assemblyai-docs https://assemblyai.com/docs/mcp
npx skills add AssemblyAI/assemblyai-skill --global
```

Then describe what you want to build. To get the same result as the steps below, paste:

```text
Use the AssemblyAI Python SDK to transcribe https://assembly.ai/wildfires.mp3 and print the transcript text.
```

## Transcribe your first file

Prefer to write it yourself? Follow these steps to transcribe our hosted sample file. The SDK uploads, submits, and polls for you in a single call.

### Step 1: Install the SDK

```bash
pip install assemblyai
```

```bash
npm install assemblyai
```

### Step 2: Run your first transcription

Save this as `transcribe.py` (Python) or `transcribe.js` (JavaScript):

```python
import os

from assemblyai.prerecorded.v2 import Transcriber

transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])

transcript = transcriber.transcribe("https://assembly.ai/wildfires.mp3")
print(transcript.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const transcript = await client.transcripts.transcribe({
  audio: "https://assembly.ai/wildfires.mp3",
});
console.log(transcript.text);
```

Then run it — `python transcribe.py` or `node transcribe.js`. You'll see the transcript printed:

```text
Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US...
```

That's the whole first call. From here you can add options — speaker labels, language detection, or a local file — see the [complete example](#complete-example) to combine them, or use the [HTTP API directly](#using-the-http-api-directly) if you're not using an SDK.

## Customize your request

The call above works with no extra configuration. Add capabilities by setting options on the same request — combine as many as you need (the [complete example](#complete-example) sets several at once).

### Transcribe a local file

Pass a file path instead of a URL; the SDK uploads it for you.

```python
transcript = transcriber.transcribe("./example.mp3")
```

```javascript
const transcript = await client.transcripts.transcribe({
  audio: "./example.mp3",
});
```

### Identify speakers

Enable [Speaker Diarization](/pre-recorded-audio/label-speakers) to split the transcript by speaker. Each labeled segment (an *utterance*) has a speaker ID and its text.

```python
from assemblyai.prerecorded.v2 import TranscriptionConfig

config = TranscriptionConfig(speaker_labels=True)
transcript = transcriber.transcribe("https://assembly.ai/wildfires.mp3", config=config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")
```

```javascript
const transcript = await client.transcripts.transcribe({
  audio: "https://assembly.ai/wildfires.mp3",
  speaker_labels: true,
});

for (const utterance of transcript.utterances) {
  console.log(`Speaker ${utterance.speaker}: ${utterance.text}`);
}
```

### Detect the language automatically

Use [Automatic Language Detection](/pre-recorded-audio/language-detection) to detect the dominant spoken language. The `language_detection=True` option is used in the complete example below.

## Complete example

Here's the complete, runnable script — the call above plus options and error handling:

```python expandable
import os

from assemblyai import TranscriptStatus
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# Use a publicly-accessible URL
audio_file = "https://assembly.ai/wildfires.mp3"

# Or use a local file:
# audio_file = "./example.mp3"

config = TranscriptionConfig(
    language_detection=True,
    speaker_labels=True,
)

transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
transcript = transcriber.transcribe(audio_file, config=config)

if transcript.status == TranscriptStatus.error:
    raise RuntimeError(f"Transcription failed: {transcript.error}")

# Log transcript.id for every request (not just errors), with a timestamp and API region.
# It's required to fetch results, retry, or delete the transcript later, and it's the first
# thing support@assemblyai.com asks for. Delete: /pre-recorded-audio/delete-transcripts
# Troubleshooting: /pre-recorded-audio/guides/common_errors_and_solutions

print(f"\nFull Transcript:\n\n{transcript.text}")

# Optionally print speaker diarization results
# for utterance in transcript.utterances:
#     print(f"Speaker {utterance.speaker}: {utterance.text}")
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

const baseUrl = "https://api.assemblyai.com";

const client = new AssemblyAI({
  apiKey: process.env.ASSEMBLYAI_API_KEY,
  baseUrl: baseUrl,
});

// Use a publicly-accessible URL
const audioFile = "https://assembly.ai/wildfires.mp3";

// Or use a local file:
// const audioFile = "./example.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  speaker_labels: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }

  // Log transcript.id for every request (not just errors), with a timestamp and API region.
  // It's required to fetch results, retry, or delete the transcript later, and it's the first
  // thing support@assemblyai.com asks for. Delete: /pre-recorded-audio/delete-transcripts
  // Troubleshooting: /pre-recorded-audio/guides/common_errors_and_solutions

  console.log(`\nFull Transcript:\n\n${transcript.text}`);

  // Optionally print speaker diarization results
  // for (const utterance of transcript.utterances) {
  //   console.log(`Speaker ${utterance.speaker}: ${utterance.text}`);
  // }
};

run();
```

## What you get back

A completed transcript includes the full `text` plus metadata, and per-speaker `utterances` when you enable `speaker_labels`. The SDK exposes these as attributes (`transcript.text`, `transcript.utterances[0].speaker`); the raw API returns the same fields as JSON:

```json
{
  "id": "106993b6-ac12-45d0-b74a-1bbd923e755d",
  "status": "completed",
  "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts...",
  "language_code": "en",
  "audio_duration": 282,
  "confidence": 0.95,
  "utterances": [
    {
      "speaker": "A",
      "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts...",
      "confidence": 0.97,
      "start": 100,
      "end": 26560,
      "words": [
        { "text": "Smoke", "start": 100, "end": 640, "confidence": 0.9, "speaker": "A" }
      ]
    }
  ]
}
```

`start` and `end` are in milliseconds. Persist `id` to fetch, retry, or delete the transcript later. See the [transcript API reference](/api-reference/transcripts/get) for the complete field list.

## Using the HTTP API directly

Not using an SDK? The same flow works over plain HTTP — authenticate with your key in the `authorization` header (no `Bearer` prefix), submit to `POST /v2/transcript`, then poll (repeatedly call `GET /v2/transcript/{id}`) until the status is `completed`. The SDKs above do all of this for you, including uploading local files and polling.

All three examples read your key from the same `ASSEMBLYAI_API_KEY` environment variable you set in [Before you begin](#before-you-begin). The cURL example also needs [`jq`](https://jqlang.github.io/jq/) (`brew install jq`); the Python example needs the `requests` library (`pip install requests`); the JavaScript example needs Node.js 18+ (built-in `fetch`).

Submit the file, poll until the status is `completed`, then print the text. (The variable is named `state` because zsh reserves `status`.)

```bash expandable
id=$(curl -s -X POST https://api.assemblyai.com/v2/transcript \
  -H "authorization: $ASSEMBLYAI_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "language_detection": true,
    "speaker_labels": true
  }' | jq -r .id)

while true; do
  state=$(curl -s https://api.assemblyai.com/v2/transcript/$id \
    -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r .status)
  [ "$state" = "completed" ] && break
  [ "$state" = "error" ] && { echo "Transcription failed"; break; }
  sleep 3
done

curl -s https://api.assemblyai.com/v2/transcript/$id \
  -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r .text
```

To transcribe a local file, upload it first and use the returned `upload_url` as the `audio_url`:

```bash
curl -s -X POST https://api.assemblyai.com/v2/upload \
  -H "authorization: $ASSEMBLYAI_API_KEY" \
  --data-binary @./example.mp3 | jq -r .upload_url
```

  The file must be streamed as raw bytes with `curl --data-binary @<file>` (note the `@`). Using `-d`/`--data`, or passing a JSON body or a file-path string, will return a successful `upload_url` but then fail downstream at transcription with a `Transcoding failed. File type application/json` or `text/plain` error. See [Troubleshoot Common Errors](/pre-recorded-audio/guides/common_errors_and_solutions) for details.

```python expandable
import os
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}

# Use a publicly-accessible URL
audio_file = "https://assembly.ai/wildfires.mp3"

# Or upload a local file:
# with open("./example.mp3", "rb") as f:
#     response = requests.post(base_url + "/v2/upload", headers=headers, data=f)
#     if response.status_code != 200:
#         print(f"Error: {response.status_code}, Response: {response.text}")
#         response.raise_for_status()
#     upload_json = response.json()
#     audio_file = upload_json["upload_url"]

data = {
    "audio_url": audio_file,
    "language_detection": True,
    "speaker_labels": True
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_json = response.json()
transcript_id = transcript_json["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(f"\nFull Transcript:\n\n{transcript['text']}")

        # Optionally print speaker diarization results
        # for utterance in transcript['utterances']:
        #     print(f"Speaker {utterance['speaker']}: {utterance['text']}")
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: process.env.ASSEMBLYAI_API_KEY,
};

async function transcribe() {
  // Use a publicly-accessible URL
  const audioFile = "https://assembly.ai/wildfires.mp3";

  // Or upload a local file:
  // import fs from "fs-extra";
  // const audioData = await fs.readFile("./example.mp3");
  // const uploadRes = await fetch(`${baseUrl}/v2/upload`, {
  //   method: "POST",
  //   headers,
  //   body: audioData,
  // });
  // if (!uploadRes.ok) throw new Error(`Error: ${uploadRes.status}`);
  // const uploadResponse = await uploadRes.json();
  // const audioFile = uploadResponse.upload_url;

  const data = {
    audio_url: audioFile,
    language_detection: true,
    speaker_labels: true,
  };

  let res = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(data),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptResponse = await res.json();
  const transcriptId = transcriptResponse.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  while (true) {
    res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    const transcript = await res.json();

    if (transcript.status === "completed") {
      console.log(`\nFull Transcript:\n\n${transcript.text}`);

      // Optionally print speaker diarization results
      // for (const utterance of transcript.utterances) {
      //   console.log(`Speaker ${utterance.speaker}: ${utterance.text}`);
      // }
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

transcribe();
```

## Limits

- **File size:** up to 5 GB per request (`/v2/transcript`); local files uploaded via `/v2/upload` up to 2.2 GB.
- **Duration:** 160 ms to 10 hours per file.
- **Formats:** most common audio and video formats — submit your file as-is, no transcoding needed.
- **Rate limit:** default 5 parallel jobs on free accounts, 200 on paid. Check yours on the [rate limits page](https://www.assemblyai.com/dashboard/home).

## Next steps

Now that you have transcribed your first audio file:

- Explore [our Speech Understanding features](https://www.assemblyai.com/products/speech-understanding) for more ways to analyze your audio data
- Learn more about searching, summarizing, or asking questions on your transcript with [our LLM Gateway feature](/llm-gateway/quickstart)
- Find out how to use [webhooks](/pre-recorded-audio/webhooks) to get notified when your transcripts are ready

For more information, check out the full [API reference documentation](/).

## Need some help?

If you get stuck, or have any other questions, we'd love to help you out. Contact our support team at support@assemblyai.com or create a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="model-selection-docs-pre-recorded-audio-select-the-speech-model-mdx"></a>

# Model selection

URL: https://www.assemblyai.com/docs/pre-recorded-audio/select-the-speech-model
Source: docs/pre-recorded-audio/select-the-speech-model.mdx
Navigation: Pre-recorded STT > Getting started
Description: Model selection documentation.

The `speech_models` parameter lets you specify which model(s) to use for transcription. If omitted, it defaults to `["universal-3-5-pro", "universal-2"]`. With this default, `universal-3-5-pro` handles its 18 supported languages; for all other languages, the request will automatically fall back to `universal-2`.

## Available models

| Name                | Parameter                           | Description                                                                          | Best for                                                                                                          |
| ------------------- | ----------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- |
| **Universal-3.5 Pro** <span class="recommended-pill">Recommended</span> | `speech_models=['universal-3-5-pro']` | Our highest accuracy, fastest model with 18-language support, native code switching, and contextual prompting. | Highest-accuracy transcription, post-call analytics, meeting notetakers, medical transcription, domain-specific accuracy via prompting |
| **Universal-2**     | `speech_models=['universal-2']`     | Our accurate, cost-effective model with support across 99 languages.      | High-volume batch transcription, 99-language coverage, price-sensitive workloads, fallback for unsupported U3 Pro languages |

| Name                | Parameter                            | Description                                                                          | Best for                                                                                                          |
| ------------------- | ------------------------------------ | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- |
| **Universal-3.5 Pro** <span class="recommended-pill">Recommended</span> | `speech_models: ['universal-3-5-pro']` | Our highest accuracy, fastest model with 18-language support, native code switching, and contextual prompting. | Highest-accuracy transcription, post-call analytics, meeting notetakers, medical transcription, domain-specific accuracy via prompting |
| **Universal-2**     | `speech_models: ['universal-2']`     | Our accurate, cost-effective model with support across 99 languages.      | High-volume batch transcription, 99-language coverage, price-sensitive workloads, fallback for unsupported U3 Pro languages |

| Name                | API Parameter                          | Description                                                                          | Best for                                                                                                          |
| ------------------- | -------------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- |
| **Universal-3.5 Pro** <span class="recommended-pill">Recommended</span> | `"speech_models": ["universal-3-5-pro"]` | Our highest accuracy, fastest model with 18-language support, native code switching, and contextual prompting. | Highest-accuracy transcription, post-call analytics, meeting notetakers, medical transcription, domain-specific accuracy via prompting |
| **Universal-2**     | `"speech_models":["universal-2"]`      | Our accurate, cost-effective model with support across 99 languages.      | High-volume batch transcription, 99-language coverage, price-sensitive workloads, fallback for unsupported U3 Pro languages |

## Set the model

You can change the model by setting the `speech_models` in the POST request body:

```python highlight={12}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

data = {
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speech_models": ["universal-3-5-pro"],
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(transcription_result['text'])
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

You can change the model by setting `speech_models` in the transcription config:

```python highlight={7} 
from assemblyai import TranscriptStatus
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    language_detection=True
)
transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == TranscriptStatus.error:
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

You can change the model by setting the `speech_models` in the POST request body:

```javascript highlight={12}  expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assembly.ai/wildfires.mp3",
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

You can change the model by setting the `speech_models` in the transcript parameters:

```javascript highlight={12}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }

  console.log(transcript.text);
};

run();
```

## Identify the model used

After transcription completes, you can check which model was actually used to process your request by reading the `speech_model_used` field. This is useful when you provide multiple models in the `speech_models` array, as the system may fall back to a different model depending on language support.

```python  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

data = {
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speech_models": ["universal-3-5-pro"],
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Model used: {transcription_result['speech_model_used']}")
    print(transcription_result['text'])
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)
```

```python 
from assemblyai import TranscriptStatus
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    language_detection=True
)
transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == TranscriptStatus.error:
    raise RuntimeError(f"Transcription failed: {transcript.error}")

print(f"Model used: {transcript.json_response['speech_model_used']}")
print(transcript.text)
```

```javascript  expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assembly.ai/wildfires.mp3",
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(`Model used: ${transcriptionResult.speech_model_used}`);
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

```javascript  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }

  console.log(`Model used: ${transcript.speech_model_used}`);
  console.log(transcript.text);
};

run();
```

## Complete example

Here is the full working code that demonstrates model selection with error handling:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speech_models": ["universal-3-5-pro"],
    "language_detection": True
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_json = response.json()
transcript_id = transcript_json["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(f"Model used: {transcript['speech_model_used']}")
        print(f"\nTranscript:\n\n{transcript['text']}")
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```python
from assemblyai import TranscriptStatus
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    language_detection=True,
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == TranscriptStatus.error:
    raise RuntimeError(f"Transcription failed: {transcript.error}")

print(f"Model used: {transcript.json_response['speech_model_used']}")
print(f"\nTranscript:\n\n{transcript.text}")
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

async function transcribe() {
  const audioFile = "https://assembly.ai/wildfires.mp3";

  const data = {
    audio_url: audioFile,
    speech_models: ["universal-3-5-pro"],
    language_detection: true,
  };

  let res = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(data),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptResponse = await res.json();
  const transcriptId = transcriptResponse.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  while (true) {
    res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    const transcript = await res.json();

    if (transcript.status === "completed") {
      console.log(`Model used: ${transcript.speech_model_used}`);
      console.log(`\nTranscript:\n\n${transcript.text}`);
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

transcribe();
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }

  console.log(`Model used: ${transcript.speech_model_used}`);
  console.log(`\nTranscript:\n\n${transcript.text}`);
};

run();
```

---

<a id="prompting-and-keyterms-docs-pre-recorded-audio-universal-3-5-pro-prompting-mdx"></a>

# Prompting and Keyterms

URL: https://www.assemblyai.com/docs/pre-recorded-audio/universal-3-5-pro/prompting
Source: docs/pre-recorded-audio/universal-3-5-pro/prompting.mdx
Navigation: Pre-recorded STT > Features
Description: Prompting and Keyterms documentation.

Universal-3.5 Pro is highly accurate out of the box, but for challenging audio like short clips with limited context, noisy environments, or audio with niche references, you can give the model information about your audio to improve transcription accuracy.

There are two ways to give the model information about your audio:

- **Contextual prompting** (`prompt`) — a natural-language description of what the audio is about: the domain, the scenario, or the full details of the conversation.
- **Keyterms prompting** (`keyterms_prompt`) — an explicit list of terms you want the model to recognize accurately.

## Contextual prompting

Use the `prompt` parameter to provide context about your audio — describe what is being transcribed, not how to transcribe it. Formatting and behavioral instructions are ignored. The model stays grounded in the audio, so irrelevant context won't cause hallucinated words.

For example, this is a 2-second clip from a League of Legends pro interview:

Without prompt:

```txt
And so look who I've been a dear.
```

With prompt:

```txt
In solo queue, I ban Azir.
```

```python {11} expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assembly.ai/prompt-8",
    "language_detection": True,
    "prompt": "League of Legends roles"
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_response = response.json()
transcript_id = transcript_response["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript["text"])
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```python {8} expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "https://assembly.ai/prompt-8"

config = TranscriptionConfig(
  language_detection=True,
  prompt="League of Legends roles",
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcript = transcriber.transcribe(audio_file, config)

print(transcript.text)
```

```javascript {10} expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assembly.ai/prompt-8",
  language_detection: true,
  prompt: "League of Legends roles",
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

```javascript {13} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assembly.ai/prompt-8";

const params = {
  audio: audioFile,
  language_detection: true,
  prompt: "League of Legends roles",
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log(transcript.text);
};

run();
```

### Prompting guide

Contextual prompts work at three levels of specificity. Use the least specific level that covers your use case, and add detail when your audio contains uncommon names or terms the model can't otherwise know.

| Level | Length | What it contains | Example |
| --- | --- | --- | --- |
| **Domain** | 2–5 words | The domain only | `Medical consultation call.` |
| **Scenario** | 5–15 words | What the conversation is about | `Cardiology consultation about chest pain symptoms.` |
| **Detailed** | 20–50 words | Full description, including names, products, or identifiers | `Cardiology consultation between Dr. Smith and an elderly patient regarding recurring chest pain, ECG results, and medication adjustment for hypertension.` |

Guidelines for writing contextual prompts:

- Write plain, complete sentences that describe the audio
- Keep it to one short block of text. Don't pack lists of keywords into the contextual prompt

## Keyterms prompting

Keyterms prompting allows you to provide up to 1,000 words or phrases (maximum 6 words per phrase) using the `keyterms_prompt` parameter to improve transcription accuracy for those terms and related variations or contextually similar phrases.

Here is an example showing how you can use keyterms prompting to improve transcription accuracy for a name with distinctive spelling and formatting.

Without keyterms prompting:

```txt
Hi, this is Kelly Byrne Donahue
```

With keyterms prompting:

```txt
Hi, this is Kelly Byrne-Donoghue
```

```python {10} expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assemblyaiassets.com/audios/keyterms_prompting.wav",
    "language_detection": True,
    "keyterms_prompt": ["Kelly Byrne-Donoghue"]
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_response = response.json()
transcript_id = transcript_response["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript["text"])
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```javascript {11} expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assemblyaiassets.com/audios/keyterms_prompting.wav",
  language_detection: true,
  keyterms_prompt: ["Kelly Byrne-Donoghue"],
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

```python {7}
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "https://assemblyaiassets.com/audios/keyterms_prompting.wav"

config = TranscriptionConfig(
    language_detection=True,
    keyterms_prompt=["Kelly Byrne-Donoghue"]
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

print(transcript.text)
```

```javascript {12}
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assemblyaiassets.com/audios/keyterms_prompting.wav";

const params = {
  audio: audioFile,
  language_detection: true,
  keyterms_prompt: ["Kelly Byrne-Donoghue"],
};

const transcript = await client.transcripts.transcribe(params);

console.log(transcript.text);
```

  **Keyword count limits**

While we support up to 1000 key words and phrases, actual capacity may be lower due to internal tokenization and implementation constraints.
Key points to remember:
- Each word in a multi-word phrase counts towards the 1000 keyword limit
- Capitalization affects capacity (uppercase tokens consume more than lowercase)
- Longer words consume more capacity than shorter words

For optimal results, use shorter phrases when possible and be mindful of your total token count when approaching the keyword limit.

## Need help?

If you'd like help building or optimizing a prompt for your audio, our team can help: open a live chat or email us via the widget in the bottom-right corner ([contact info](https://www.assemblyai.com/contact/support)).

---

<a id="medical-mode-docs-pre-recorded-audio-medical-mode-mdx"></a>

# Medical Mode

URL: https://www.assemblyai.com/docs/pre-recorded-audio/medical-mode
Source: docs/pre-recorded-audio/medical-mode.mdx
Navigation: Pre-recorded STT > Features
Description: Improve transcription accuracy for medical terminology in pre-recorded audio

Medical Mode is an add-on that enhances transcription accuracy for medical terminology — including medication names, procedures, conditions, and dosages. It is optimized for medical entity recognition to correct terms that other models frequently get wrong.

Medical Mode can be used with all of our Pre-recorded STT models.

Enable Medical Mode by setting the `domain` parameter to `"medical-v1"`. No changes to your existing pipeline are required.

## Quickstart

To enable Medical Mode, set `domain` to `"medical-v1"` in the POST request body:

```python {11}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assembly.ai/lispro",
    "language_detection": True,
    "domain": "medical-v1"
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_response = response.json()
transcript_id = transcript_response["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript["text"])
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

To enable Medical Mode, set `domain` to `"medical-v1"` in the transcription config.

```python {12} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# You can use a local filepath:
# audio_file = "./example.mp3"

# Or use a publicly-accessible URL:
audio_file = "https://assembly.ai/lispro"

config = TranscriptionConfig(
    language_detection=True,
    domain="medical-v1",
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcript = transcriber.transcribe(audio_file, config)

print(transcript.text)
```

To enable Medical Mode, set `domain` to `"medical-v1"` in the POST request body:

```javascript {10}  expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assembly.ai/lispro",
  language_detection: true,
  domain: "medical-v1",
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

To enable Medical Mode, set `domain` to `"medical-v1"` in the transcription config.

```javascript {17}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// You can use a local filepath:
// const audioFile = "./example.mp3"

// Or use a publicly-accessible URL:
const audioFile = "https://assembly.ai/lispro";

const params = {
  audio: audioFile,
  language_detection: true,
  domain: "medical-v1",
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log(transcript.text);
};

run();
```

### Example output

Without Medical Mode:

```plain
I have here insulin to be used for both prandial mealtime and sliding scale is
insulin lisprohumalog subcutaneously.
```

With Medical Mode, lisprohumalog is updated to Lispro (Humalog) - following the standard medical convention of writing the generic name first, with the brand name in parentheses.

```plain
I have here insulin to be used for both prandial mealtime and sliding scale is
insulin Lispro (Humalog) subcutaneously.
```

## Use cases

Medical Mode is designed for healthcare AI applications where accurate medical terminology is critical:

- **Ambient clinical documentation** — Capture medication names, dosages, and clinical terms correctly in real-time scribing workflows.
- **AI-powered clinical notes** — Generate clean transcripts for downstream LLMs producing SOAP notes, discharge summaries, and referral letters.
- **Front-office automation** — Handle drug names, provider names, and clinic-specific terminology in scheduling calls, insurance verification, and voice agents.
- **Multi-speaker clinical conversations** — Combine with [Speaker Diarization](/pre-recorded-audio/label-speakers) for provider/patient separation in telehealth, therapy documentation, and clinical settings.

## Combine with other features

Medical Mode works alongside other transcription features. You can combine it with:

- [Speaker Diarization](/pre-recorded-audio/label-speakers) to identify who said what in clinical conversations
- [Keyterms Prompting](/pre-recorded-audio/universal-3-5-pro/prompting#keyterms-prompting) to further boost accuracy for specific medical terms unique to your use case
- [PII Redaction](/guardrails/redact-pii-from-transcripts) to redact sensitive patient information from transcripts

```python
data = {
    "audio_url": "<YOUR_AUDIO_URL>",
    "language_detection": True,
    "domain": "medical-v1",
    "speaker_labels": True,
    "keyterms_prompt": ["Lisinopril", "Metformin", "Humalog"]
}
```

```python
config = TranscriptionConfig(
    language_detection=True,
    domain="medical-v1",
    speaker_labels=True,
    keyterms_prompt=["Lisinopril", "Metformin", "Humalog"],
)
```

```javascript
const data = {
  audio_url: "<YOUR_AUDIO_URL>",
  language_detection: true,
  domain: "medical-v1",
  speaker_labels: true,
  keyterms_prompt: ["Lisinopril", "Metformin", "Humalog"],
};
```

```javascript
const params = {
  audio: audioFile,
  language_detection: true,
  domain: "medical-v1",
  speaker_labels: true,
  keyterms_prompt: ["Lisinopril", "Metformin", "Humalog"],
};
```

Medical Mode supports English, Spanish, German, and French. If you use Medical Mode with an unsupported language, the API ignores the `domain` parameter and returns a warning indicating that Medical Mode was not applied: `"Skipped medical-v1 domain correction because the language is not supported"`

Your transcript is still returned using standard transcription, and you will not be charged for Medical Mode.

## HIPAA compliance

AssemblyAI offers a Business Associate Agreement (BAA) for customers who need to process Protected Health Information (PHI). AssemblyAI is SOC 2 Type 2, ISO 27001:2022, and PCI DSS v4.0 certified. Medical Mode does not change existing data handling or retention policies.

Paid customers can review and sign our standard online BAA self-serve, at no additional cost, from the [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the dashboard. You must have an upgraded (paid) account to see the BAA option. See [Data Controls](/data-controls#business-associate-agreement-baa) for details, or [contact sales](https://www.assemblyai.com/contact-sales) for enterprise pricing.

---

<a id="speaker-diarization-docs-pre-recorded-audio-label-speakers-mdx"></a>

# Speaker Diarization

URL: https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers
Source: docs/pre-recorded-audio/label-speakers.mdx
Navigation: Pre-recorded STT > Features
Description: Add speaker labels to your transcript

## Overview

Speaker diarization identifies individual speakers in your audio and labels each segment of the transcript with the speaker. When enabled, the transcript is returned as a list of utterances, where each utterance represents an uninterrupted segment of speech from a single speaker.

Accuracy improves the more each speaker talks as the model accumulates embedding context. For best results, each speaker should have at least 30 seconds of continuous speech.

### Quickstart

To enable Speaker Diarization, set `speaker_labels` to `True` in the POST request body:

```python {21}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "speaker_labels": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

for utterance in transcription_result['utterances']:
  print(f"Speaker {utterance['speaker']}: {utterance['text']}")
```

To enable Speaker Diarization, set `speaker_labels` to `True` in the transcription config.

```python {14}  expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# You can use a local filepath:
# audio_file = "./example.mp3"

# Or use a publicly-accessible URL:
audio_file = (
    "https://assembly.ai/wildfires.mp3"
)

config = TranscriptionConfig(
  language_detection=True,
  speaker_labels=True,
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcript = transcriber.transcribe(audio_file, config)

for utterance in transcript.utterances:
  print(f"Speaker {utterance.speaker}: {utterance.text}")
```

To enable Speaker Diarization, set `speaker_labels` to `true` in the POST request body:

```javascript {26}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./audio/audio.mp3";
const audioData = await fs.readFile(path);

let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();

const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  speaker_labels: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    for (const utterance of transcriptionResult.utterances) {
      console.log(`Speaker ${utterance.speaker}: ${utterance.text}`);
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

To enable Speaker Diarization, set `speaker_labels` to `true` in the transcription config.

```javascript {17}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// You can use a local filepath:
// const audioFile = "./example.mp3"

// Or use a publicly-accessible URL:
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  speaker_labels: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  for (const utterance of transcript.utterances ?? []) {
    console.log(`Speaker ${utterance.speaker}: ${utterance.text}`);
  }
};

run();
```

### Configuration

You can constrain the number of speakers with `speakers_expected`, or with `min_speakers_expected`/`max_speakers_expected`. These are hard boundaries on the number of speaker labels, not hints: `max_speakers_expected` is a strict cap — if more people speak than that, the additional speakers are merged into existing labels — and `min_speakers_expected` is a strict floor.

- If you know the **exact** number of speakers, set `speakers_expected`.
- If you have a rough idea, use `min_speakers_expected` and `max_speakers_expected` to set a range. Set `max_speakers_expected` a little higher than the number of speakers you expect so the model has room to identify any additional speakers. Setting it too high can cause the model to over-split and return more speaker labels than are actually present.

<table>
  <colgroup>
    <col style={{ width: "22%" }} />
    <col style={{ width: "10%" }} />
    <col style={{ width: "28%" }} />
    <col style={{ width: "40%" }} />
  </colgroup>
  <thead>
    <tr>
      <th>Key</th>
      <th>Type</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>`speaker_labels`</td>
      <td>boolean</td>
      <td>`false`</td>
      <td>Enable Speaker Diarization.</td>
    </tr>
    <tr>
      <td>`speakers_expected`</td>
      <td>number</td>
      <td>—</td>
      <td>Set the exact number of speakers.</td>
    </tr>
    <tr>
      <td>`speaker_options`</td>
      <td>object</td>
      <td>—</td>
      <td>Set range of possible speakers.</td>
    </tr>
    <tr>
      <td>`speaker_options.min_speakers_expected`</td>
      <td>number</td>
      <td>—</td>
      <td>A hard lower limit on the number of speaker labels. The model won't return fewer speakers than this.</td>
    </tr>
    <tr>
      <td>`speaker_options.max_speakers_expected`</td>
      <td>number</td>
      <td>**0–2 minutes**: —<br/>**2–10 minutes**: 10 speakers<br/>**10+ minutes**: 30 speakers</td>
      <td>A hard upper limit on the number of speaker labels. If more people speak than this, the additional speakers are merged into existing labels. Give the model a little headroom above the number of speakers you expect; setting it too high can cause over-splitting and return more speakers than are actually present.</td>
    </tr>
  </tbody>
</table>

  Only set `speakers_expected` when you are certain of the exact speaker count. If you're unsure, use `min_speakers_expected` and `max_speakers_expected` to describe a range instead — providing an incorrect exact count can negatively affect diarization accuracy.

### Reading the response

When diarization is enabled, the transcript includes an `utterances` array in place of a single text block. Each object in the array represents one uninterrupted segment of speech from a single speaker.

The `utterances` array contains objects with the following fields:

| Field         | Type   | Description                                                                                            |
| ------------- | ------ | ------------------------------------------------------------------------------------------------------ |
| `speaker`     | string | The speaker label, assigned as sequential letters such as A, B, and C.                                 |
| `text`        | string | The transcribed text for the utterance.                                                                |
| `start`       | number | The start time of the utterance in milliseconds.                                                       |
| `end`         | number | The end time of the utterance in milliseconds.                                                         |
| `confidence`  | number | A score between 0 and 1 indicating the model's confidence in the transcribed text.                     |
| `words`       | array  | A word-level breakdown of the utterance.                                                               |

Each object in the `words` array contains the following fields:

| Field        | Type   | Description                                                                         |
| ------------ | ------ | ----------------------------------------------------------------------------------- |
| `text`       | string | The transcribed word.                                                               |
| `speaker`    | string | The speaker label for the word.                                                     |
| `start`      | number | The start time of the word in milliseconds.                                         |
| `end`        | number | The end time of the word in milliseconds.                                           |
| `confidence` | number | A score between 0 and 1 indicating the model's confidence in the word.              |

### Identify speakers by name

Speaker Diarization assigns generic labels like "Speaker A" and "Speaker B" to each speaker. If you want to replace these labels with actual names or roles, you can use Speaker Identification to transform your transcript.

**Before Speaker Identification:**

```txt
Speaker A: Good morning, and welcome to the show.
Speaker B: Thanks for having me.
```

**After Speaker Identification:**

```txt
Michel Martin: Good morning, and welcome to the show.
Peter DeCarlo: Thanks for having me.
```

The following example shows how to transcribe audio with Speaker Diarization and then apply Speaker Identification to replace the generic speaker labels with actual names.

```python maxLines=30 expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

audio_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker diarization and speaker identification
data = {
    "audio_url": audio_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": ["Michel Martin", "Peter DeCarlo"]
            }
        }
    }
}

# Submit the transcription request
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription results
while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

# Print utterances with identified speaker names
for utterance in transcript["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")
```

```javascript maxLines=30 expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
  "content-type": "application/json",
};

const audioUrl = "https://assembly.ai/wildfires.mp3";

// Configure transcript with speaker diarization and speaker identification
const data = {
  audio_url: audioUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      speaker_identification: {
        speaker_type: "name",
        known_values: ["Michel Martin", "Peter DeCarlo"],
      },
    },
  },
};

async function main() {
  // Submit the transcription request
  const response = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: headers,
    body: JSON.stringify(data),
  });
  if (!response.ok) throw new Error(`Error: ${response.status}`);

  const { id: transcriptId } = await response.json();
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for transcription results
  while (true) {
    const pollingResponse = await fetch(pollingEndpoint, { headers });
    if (!pollingResponse.ok) throw new Error(`Error: ${pollingResponse.status}`);
    const transcript = await pollingResponse.json();

    if (transcript.status === "completed") {
      // Print utterances with identified speaker names
      for (const utterance of transcript.utterances) {
        console.log(`${utterance.speaker}: ${utterance.text}`);
      }
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

main().catch(console.error);
```

For more details on Speaker Identification, including how to identify speakers by role and how to apply it to existing transcripts, see the [Speaker Identification guide](/speech-understanding/speaker-identification).

### Best practices for accurate diarization

Follow these tips to get the best results from Speaker Diarization:

- **Ensure sufficient speech per speaker.** Each speaker should speak for at least 30 seconds uninterrupted. The model may struggle to create separate clusters for speakers who only contribute short phrases like "Yeah", "Right", or "Sounds good".
- **Minimize cross-talk.** Overlapping speech between speakers can reduce diarization accuracy. Where possible, ensure speakers take turns.
- **Reduce background noise.** Background noise, echoes, or playback of recorded audio during a conversation can interfere with speaker separation.
- **Use `speaker_options` instead of `speakers_expected` when uncertain.** Only use `speakers_expected` when you are confident about the exact number of speakers. If this number is incorrect, the model may produce random splits of single-speaker segments or merge multiple speakers into one. It's generally recommended to use `min_speakers_expected` and set `max_speakers_expected` slightly higher (e.g., `min_speakers_expected` + 2) to allow flexibility.
- **Avoid setting `max_speakers_expected` too high.** Setting the maximum too high may reduce accuracy, causing sentences from the same speaker to be split across multiple speaker labels.
- **Be aware of speaker similarity.** If speakers sound similar, the model may have difficulty distinguishing between them.

---

<a id="multichannel-transcription-docs-pre-recorded-audio-transcribe-multiple-audio-channels-mdx"></a>

# Multichannel Transcription

URL: https://www.assemblyai.com/docs/pre-recorded-audio/transcribe-multiple-audio-channels
Source: docs/pre-recorded-audio/transcribe-multiple-audio-channels.mdx
Navigation: Pre-recorded STT > Features
Description: Multichannel Transcription documentation.

If you have a multichannel audio file with multiple speakers, you can transcribe each of them separately.

The response includes an `audio_channels` property with the number of different channels, and an additional `utterances` property, containing a list of turn-by-turn utterances.

Each utterance contains channel information, starting at 1.

Additionally, each word in the `words` array contains the channel identifier.

## Quickstart

```python title="Python SDK" {8} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True,
  multichannel=True
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

for utterance in transcript.utterances:
  print(f"Channel {utterance.speaker}: {utterance.text}")
```

```python title="Python" {20}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "multichannel": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

for utterance in transcription_result['utterances']:
  print(f"Channel {utterance['speaker']}: {utterance['text']}")
```

```javascript title="JavaScript SDK" {13}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  multichannel: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  for (const utterance of transcript.utterances ?? []) {
    console.log(`Channel ${utterance.speaker}: ${utterance.text}`);
  }
};

run();
```

```javascript title="JavaScript" {21}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  multichannel: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    for (const utterance of transcriptionResult.utterances) {
      console.log(`Channel ${utterance.speaker}: ${utterance.text}`);
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  Multichannel audio increases the transcription time by approximately 40%.

## Per-channel diarization

If you have a multichannel audio file where individual channels may contain multiple speakers, you can combine `multichannel` and `speaker_labels` to perform diarization within each channel.

  When using `multichannel` with `speaker_labels`, the `speaker_options` parameters (`min_speakers_expected` and `max_speakers_expected`) are applied **per channel**, not globally across the entire file. For example, setting `min_speakers_expected: 5` and `max_speakers_expected: 7` on a 5-channel file means the model will find 5–7 speakers on _each_ channel, resulting in 25–35 total speakers. Adjust your speaker options accordingly when using multichannel transcription.

When both parameters are enabled:

- Channels are labeled numerically (1, 2, 3, etc.)
- Speakers within each channel are labeled alphabetically (A, B, C, etc.)
- The combined speaker label format is `{channel}{speaker}` (e.g., "1A", "1B", "2A")

For example, if channel 1 has two speakers and channel 2 has one speaker, the labels would be:

- First speaker on channel 1: `1A`
- Second speaker on channel 1: `1B`
- First speaker on channel 2: `2A`

```python title="Python SDK" {8-9} expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True,
  multichannel=True,
  speaker_labels=True
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

for utterance in transcript.utterances:
  print(f"Speaker {utterance.speaker}: {utterance.text}")
```

```python title="Python" {20-21} expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "multichannel": True,
    "speaker_labels": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

for utterance in transcription_result['utterances']:
  print(f"Speaker {utterance['speaker']}: {utterance['text']}")
```

```javascript title="JavaScript SDK" {13-14} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  multichannel: true,
  speaker_labels: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  for (const utterance of transcript.utterances ?? []) {
    console.log(`Speaker ${utterance.speaker}: ${utterance.text}`);
  }
};

run();
```

```javascript title="JavaScript" {21-22} expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  multichannel: true,
  speaker_labels: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    for (const utterance of transcriptionResult.utterances) {
      console.log(`Speaker ${utterance.speaker}: ${utterance.text}`);
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

---

<a id="code-switching-docs-pre-recorded-audio-code-switching-mdx"></a>

# Code Switching

URL: https://www.assemblyai.com/docs/pre-recorded-audio/code-switching
Source: docs/pre-recorded-audio/code-switching.mdx
Navigation: Pre-recorded STT > Features
Description: Code Switching documentation.

Transcribe audio containing multiple languages with code switching detection. This feature enables accurate transcription of conversations where speakers naturally switch between languages during conversations.

## Universal-3.5 Pro (Recommended)

Universal-3.5 Pro natively handles code switching across 18 languages. Set `language_detection` as `True` and the model follows speakers as they shift mid-sentence between languages like English and Spanish, French, Hindi or Mandarin, preserving exactly what was said without translating everything into a single language.

See Universal-3.5 Pro code switching in action.

### **English \<\> French**

```txt
I said something like, j'ai dit à mes étudiants que it's time to really pay attention to what the idea of code-switching is.
```

### **English \<\> Hindi**

```txt
मेरा रुकने का तो बहुत मन है, but I have an exam to give tomorrow.
```

### **English \<\> Mandarin**

```txt
But this sentence, 我父母不工作了, you can see the 了 at the end of the sentence indicates that the situation now, my parents don't work, is different from what it was before.
```

### Quickstart

```python title="Python SDK" {7}  expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local-file.mp3"
audio_file = "https://assembly.ai/code-switching-3"

config = TranscriptionConfig(
    language_detection=True,
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" {10}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assembly.ai/code-switching-3",
    "language_detection": True,
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_response = response.json()
transcript_id = transcript_response["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript["text"])
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```javascript title="JavaScript SDK" {14}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// You can use a local filepath:
// const audioFile = "./local-file.mp3";

// Or use a publicly-accessible URL:
const audioFile = "https://assembly.ai/code-switching-3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    console.error(`Transcription failed: ${transcript.error}`);
    process.exit(1);
  }

  console.log(`\nFull Transcript:\n\n${transcript.text}\n`);
};

run();
```

```javascript title="JavaScript" {10}  expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assembly.ai/code-switching-3",
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

## Universal-2

While Universal-2 supports code switching, we recommend upgrading to Universal-3.5 Pro for best results.

To enable code switching on Universal-2, set `speech_models` to `universal-2`, `language_detection` to `true`, and `code_switching` to `true` inside the `language_detection_options` parameter.

### Quickstart

```python title="Python SDK" {11}  expandable
from assemblyai import LanguageDetectionOptions
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "./bilingual-audio.mp3"
# audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    speech_models=["universal-2"],
    language_detection=True,
    language_detection_options=LanguageDetectionOptions(
      code_switching=True
    )
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)

```

```python title="Python" {22}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./bilingual-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url,
    "speech_models": ["universal-2"],
    "language_detection": True,
    "language_detection_options": {
        "code_switching": True
    },
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)
```

```javascript title="JavaScript SDK" {18}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// You can use a local filepath:
const audioFile = "./bilingual-audio.mp3";

// Or use a publicly-accessible URL:
// const audioFile = "<AUDIO_URL>";

const params = {
  audio: audioFile,
  speech_models: ["universal-2"],
  language_detection: true,
  language_detection_options: {
    code_switching: true,
  },
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    console.error(`Transcription failed: ${transcript.error}`);
    process.exit(1);
  }

  console.log(`\nFull Transcript:\n\n${transcript.text}\n`);
};

run();
```

```javascript title="JavaScript" {22}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./bilingual-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl,
  speech_models: ["universal-2"],
  language_detection: true,
  language_detection_options: {
    code_switching: true,
  },
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

### Example API Response

When enabling code switching with automatic language detection, the two detected language codes with the highest confidence and their confidence will be included in the transcript JSON.

```json
"language_detection_results": {
      "code_switching_languages": [
           {"language": "en", "confidence": 0.8},
           {"language": "es", "confidence": 0.7}
     ]
}
```

### Manually Setting Language Codes

To manually set the [language codes](/pre-recorded-audio/code-switching), you can use the `language_codes` parameter. A max of two language codes can be set and one code must be `"en"`. For example, if your file contains both English and Spanish, it would be `"language_codes": ["en", "es"]`.

```python title="Python SDK" {8} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "./bilingual-audio.mp3"
# audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  speech_models=["universal-2"],
  language_codes=["en", "es"]  # English-Spanish code switching
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" {20}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./bilingual-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url,
    "speech_models": ["universal-2"],
    "language_codes": ["en", "es"]  # English-Spanish code switching
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)
```

```javascript title="JavaScript SDK" {16}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// You can use a local filepath:
const audioFile = "./bilingual-audio.mp3";

// Or use a publicly-accessible URL:
// const audioFile = "<AUDIO_URL>";

const params = {
  audio: audioFile,
  speech_models: ["universal-2"],
  language_codes: ["en", "es"], // English-Spanish code switching
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    console.error(`Transcription failed: ${transcript.error}`);
    process.exit(1);
  }

  console.log(`\nFull Transcript:\n\n${transcript.text}\n`);
};

run();
```

```javascript title="JavaScript" {20}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./bilingual-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl,
  speech_models: ["universal-2"],
  language_codes: ["en", "es"], // English-Spanish code switching
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

### Code Switching Confidence Threshold

The `code_switching_confidence_threshold` parameter controls how the model routes transcription when multiple languages are detected.
When code switching is enabled, the model detects up to two languages per audio file and assigns each a confidence score.

  **Code Switching Routing Behavior**

This parameter controls routing, not rejection. Audio is always transcribed, even if confidence scores do not meet this threshold. To return an error instead of a low-confidence transcription, you can use [language_confidence_threshold](/pre-recorded-audio/language-detection#set-a-language-confidence-threshold) alongside this parameter.

The threshold determines which language is used for transcription using the following logic:
- If the non-English language's confidence score **meets or exceeds** the threshold, the audio is routed to that non-English language model.
- If the non-English language's confidence score **falls below** the threshold, the audio is routed to whichever language has the **highest overall confidence**, which may be English or non-English.
- If both detected languages are non-English, the audio is always routed to whichever has the higher confidence score, regardless of the threshold.

  **Code Switching Default**

By default, the `code_switching_confidence_threshold` parameter is set to `0.3`. If you would like to disable this, make sure to set this parameter to `0`.
Setting `code_switching_confidence_threshold` to `0` means the non-English language is always used for routing, even with very low confidence.

```python title="Python SDK" {12}  expandable
from assemblyai import LanguageDetectionOptions
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

audio_file = "./bilingual-audio.mp3"
# audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    speech_models=["universal-2"],
    language_detection=True,
    language_detection_options=LanguageDetectionOptions(
      code_switching=True,
      code_switching_confidence_threshold=0.5 # Optional parameter - this is set to 0.3 by default
    )
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)

```

```python title="Python" {23}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./bilingual-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url,
    "speech_models": ["universal-2"],
    "language_detection": True,
    "language_detection_options": {
        "code_switching": True,
        "code_switching_confidence_threshold": 0.5 # Optional parameter - this is set to 0.3 by default
    },
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)
```

```javascript title="JavaScript SDK" {19}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// You can use a local filepath:
const audioFile = "./bilingual-audio.mp3";

// Or use a publicly-accessible URL:
// const audioFile = "<AUDIO_URL>";

const params = {
  audio: audioFile,
  speech_models: ["universal-2"],
  language_detection: true,
  language_detection_options: {
    code_switching: true,
    code_switching_confidence_threshold: 0.5, // Optional parameter - this is set to 0.3 by default
  },
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    console.error(`Transcription failed: ${transcript.error}`);
    process.exit(1);
  }

  console.log(`\nFull Transcript:\n\n${transcript.text}\n`);
};

run();
```

```javascript title="JavaScript" {23}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./bilingual-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl,
  speech_models: ["universal-2"],
  language_detection: true,
  language_detection_options: {
    code_switching: true,
    code_switching_confidence_threshold: 0.5, // Optional parameter - this is set to 0.3 by default
  },
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

## Support for 99 languages

Universal-3.5 Pro supports 18 languages, and for anything outside that set, the system automatically falls back to Universal-2, giving you coverage across 99 languages total without any extra configuration.

| Model | Supported languages |
| --- | --- |
| `universal-3-5-pro` | Global English, Australian English, British English, US English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Finnish, Hebrew, Hindi, Japanese, Mandarin, Norwegian, Swedish, Turkish, Vietnamese |
| `universal-2` | Global English, Australian English, British English, US English, Spanish, French, German, Italian, Portuguese, Dutch, Hindi, Japanese, Chinese, Finnish, Korean, Polish, Russian, Turkish, Ukrainian, Vietnamese, Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani, Bashkir, Basque, Belarusian, Bengali, Bosnian, Breton, Bulgarian, Burmese, Catalan, Croatian, Czech, Danish, Estonian, Faroese, Galician, Georgian, Greek, Gujarati, Haitian, Hausa, Hawaiian, Hebrew, Hungarian, Icelandic, Indonesian, Javanese, Kannada, Kazakh, Khmer, Lao, Latin, Latvian, Lingala, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian, Norwegian Nynorsk, Occitan, Panjabi, Pashto, Persian, Romanian, Sanskrit, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Sundanese, Swahili, Swedish, Swiss German, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Tibetan, Turkmen, Urdu, Uzbek, Welsh |

---

<a id="automatic-language-detection-docs-pre-recorded-audio-language-detection-mdx"></a>

# Automatic Language Detection

URL: https://www.assemblyai.com/docs/pre-recorded-audio/language-detection
Source: docs/pre-recorded-audio/language-detection.mdx
Navigation: Pre-recorded STT > Features > Languages
Description: Detect the dominant language and automatically route your request to the best available speech model for that language.

Automatic language detection identifies the dominant language in your audio and routes the request to the best available model based on the detected language and the models you specify in `speech_models`. You can check which model processed your request using the `speech_model_used` field in the response.

For best results, your audio should contain at least 15–90 seconds of spoken audio.

## Quickstart

```python title="Python SDK" for="python-sdk" highlight={13} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    language_detection=True
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

print(transcript.text)
print(transcript.json_response["language_code"])
```

```python title="Python" for="python" highlight={20}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    print(f"Language Code: {transcription_result['language_code']}")
    print(f"Text: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" for="javascript-sdk" highlight={13}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
  console.log(transcript.language_code);
};

run();
```

```javascript title="JavaScript" for="javascript" highlight={21}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    console.log(transcriptionResult.language_code);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

## Set a list of expected languages

If you're confident the audio is in one of a few languages, provide that list via `language_detection_options.expected_languages`. Detection is restricted to these candidates and the model will choose the language with the highest confidence from this list. This can eliminate scenarios where Automatic Language Detection selects an unexpected language for transcription.

- Use our [language codes](/pre-recorded-audio/supported-languages) (e.g., `"en"`, `"es"`, `"fr"`).
- If `expected_languages` is not specified, it is set to `["all"]` by default.

```python title="Python SDK" for="python-sdk" highlight={13}  expandable
from assemblyai import LanguageDetectionOptions
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

options = LanguageDetectionOptions(
    expected_languages=["en", "es", "fr", "de"],
    fallback_language="auto"
)

config = TranscriptionConfig(
    language_detection=True,
    language_detection_options=options
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

print(transcript.text)
print(transcript.json_response["language_code"])
```

```python title="Python" for="python" highlight={22}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "language_detection_options": {
      "expected_languages": ["en", "es", "fr", "de"],
      "fallback_language": "auto"
  }
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    print(f"Language Code: {transcription_result['language_code']}")
    print(f"Text: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" for="javascript-sdk" highlight={15}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  language_detection_options: {
    expected_languages: ["en", "es", "fr", "de"],
    fallback_language: "auto",
  },
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
  console.log(transcript.language_code);
};

run();
```

```javascript title="JavaScript" for="javascript" highlight={22}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  language_detection_options: {
    expected_languages: ["en", "es", "fr", "de"],
    fallback_language: "auto",
  },
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    console.log(transcriptionResult.language_code);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

## Choose a fallback language

Control what language transcription should fall back to when detection cannot confidently select a language from the `expected_languages` list.

- Set `language_detection_options.fallback_language` to a specific language code (e.g., `"en"`).
- `fallback_language` must be one of the language codes in `expected_languages` or `"auto"`.
- When `fallback_language` is unspecified, it is set to `"auto"` by default. This tells our model to choose the fallback language from `expected_languages` with the highest confidence score.

```python title="Python SDK" for="python-sdk" highlight={14}  expandable
from assemblyai import LanguageDetectionOptions
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

options = LanguageDetectionOptions(
    expected_languages=["en", "es", "fr", "de"],
    fallback_language="auto"
)

config = TranscriptionConfig(
    language_detection=True,
    language_detection_options=options
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

print(transcript.text)
print(transcript.json_response["language_code"])
```

```python title="Python" for="python" highlight={22}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "language_detection_options": {
      "expected_languages": ["en", "es", "fr", "de"],
      "fallback_language": "auto"
  }
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    print(f"Language Code: {transcription_result['language_code']}")
    print(f"Text: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" for="javascript-sdk" highlight={15}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  language_detection_options: {
    expected_languages: ["en", "es", "fr", "de"],
    fallback_language: "auto",
  },
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
  console.log(transcript.language_code);
};

run();
```

```javascript title="JavaScript" for="javascript" highlight={22}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  language_detection_options: {
    expected_languages: ["en", "es", "fr", "de"],
    fallback_language: "auto",
  },
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    console.log(transcriptionResult.language_code);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

## Choose a locale

When English is detected by automatic language detection, the transcript is rendered in the default US English spelling. Set your preferred English locale to render it in a regional spelling variant instead.

- Set `language_detection_options.localization` to a locale code. The supported values are `en_au` (Australian English) and `en_uk` (British English).
- When English is detected, the transcript uses that locale's spelling — for example `colour`, `behaviour`, and `organise`.
- `language_code` in the response returns the region-aware code (such as `en_au`) instead of the base `en`.

```python title="Python" for="python" highlight={21}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "language_detection_options": {
      "localization": ["en_au"]
  }
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(transcription_result['text'])
    print(transcription_result['language_code'])
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)
```

```javascript title="JavaScript" for="javascript" highlight={24}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  language_detection_options: {
    localization: ["en_au"],
  },
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    console.log(transcriptionResult.language_code);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  **Things to know**

  - **Only applies to English.** If the detected language isn't English, transcription proceeds as normal in the detected language. The locale is applied only when the detected language is English; otherwise the request is unaffected.
  - **English only.** `en_au` and `en_uk` are the only supported locales today. Base and default-region codes such as `en` and `en_us` aren't localization variants and return a `400`.
  - **One locale per base language.** Passing both `en_au` and `en_uk` returns a `400`.
  - **Requires automatic language detection.** `localization` only applies when `language_detection` is `true`. If you already know the language, set `language_code` to `en_au` or `en_uk` directly instead.
  - **Spelling only.** Localization affects spelling, not vocabulary or grammar.

## Confidence score

If language detection is enabled, the API returns a confidence score for the detected language. The score ranges from 0.0 (low confidence) to 1.0 (high confidence).

```python title="Python SDK" for="python-sdk" highlight={7,13} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    language_detection=True
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

print(transcript.text)
print(transcript.json_response["language_confidence"])
```

```python title="Python" for="python" highlight={33}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    print(f"Language Confidence: {transcription_result['language_confidence']}")
    print(f"Text: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" for="javascript-sdk" highlight={19}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
  console.log(transcript.language_confidence);
};

run();
```

```javascript title="JavaScript" for="javascript" highlight={36}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    console.log(transcriptionResult.language_confidence);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

## Set a language confidence threshold

You can set the confidence threshold that must be reached if language detection is enabled. An error will be returned
if the language confidence is below this threshold. Valid values are in the range [0,1] inclusive.

```python title="Python SDK" for="python-sdk" highlight={7,13} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    language_detection=True,
    language_confidence_threshold=0.8
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")
else:
  print(transcript.json_response["language_confidence"])
  print(transcript.text)
```

```python title="Python" for="python" highlight={20}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "language_confidence_threshold": 0.8
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    print(f"Text: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" for="javascript-sdk" highlight={13}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  language_confidence_threshold: 0.8,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }

  console.log(transcript.text);
  console.log(transcript.language_confidence);
};

run();
```

```javascript title="JavaScript" for="javascript" highlight={21}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  language_confidence_threshold: 0.8,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    console.log(transcriptionResult.language_confidence);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  If the `language_confidence_threshold` you specify is not met you will receive
  an error message like `detected language 'bg', confidence 0.2949, is below the
  requested confidence threshold value of '0.4'`.

## Troubleshooting

### Accented speech detected as the wrong language

Automatic Language Detection uses Whisper-based language identification, which can sometimes misidentify heavily accented speech as a different language. For example, English spoken with a strong accent may be detected as Finnish, Latvian, Latin, or Arabic.

When this happens, the model might not just return a wrong language label -- it might also **transcribe the audio in the incorrectly detected language**. This effectively translates the speech rather than transcribing it, producing output in a language the speaker wasn't using.

  The exact transcription behavior can vary depending on the detected language and speech model used.

### Recommended mitigations

**Use `expected_languages` to constrain detection (most effective).** If you know which languages your audio may contain, set `expected_languages` to only those languages. This prevents the model from selecting an unexpected language entirely.

For example, if your application processes interviews in English, Spanish, and French:

```json
{
  "language_detection": true,
  "language_detection_options": {
    "expected_languages": ["en", "es", "fr"],
    "fallback_language": "en"
  }
}
```

Setting `fallback_language` to your most common language (e.g., `"en"`) ensures that if the model can't confidently choose between the expected languages, it defaults to the language most likely to produce a useful transcript.

**Use `language_confidence_threshold` to reject low-confidence detections.** Setting a threshold (e.g., `0.7`) causes the API to return an error instead of a transcript when confidence is low. This helps catch some misdetections, but not cases where the model is confidently wrong.

**Monitor `language_confidence` in responses.** Log the `language_code` and `language_confidence` fields from your transcript responses. Unexpected language codes or unusual confidence patterns can help you identify misdetection issues early and decide whether to retry with `expected_languages` or flag the transcript for review.

---

<a id="set-language-manually-docs-pre-recorded-audio-set-language-manually-mdx"></a>

# Set Language Manually

URL: https://www.assemblyai.com/docs/pre-recorded-audio/set-language-manually
Source: docs/pre-recorded-audio/set-language-manually.mdx
Navigation: Pre-recorded STT > Features > Languages
Description: Specify a dominant language via language_code to route your request to the best available speech model for that language.

If you already know the dominant language of your audio file, you can use the `language_code` parameter to specify the language rather than using [Automatic language detection](/pre-recorded-audio/language-detection).

## Quickstart

```python title="Python SDK" highlight={7} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    language_code="es"
)

transcript = Transcriber(api_key="<YOUR_API_KEY>", config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" highlight={19}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_code": "es"
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" highlight={12}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_code: "es",
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={20}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_code: "es",
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

See the [Supported languages](/pre-recorded-audio/supported-languages) page for all supported languages and their codes.

---

<a id="supported-languages-docs-pre-recorded-audio-supported-languages-mdx"></a>

# Supported Languages

URL: https://www.assemblyai.com/docs/pre-recorded-audio/supported-languages
Source: docs/pre-recorded-audio/supported-languages.mdx
Navigation: Pre-recorded STT > Features > Languages
Description: Supported languages for pre-recorded audio speech-to-text models.

AssemblyAI supports a wide range of languages across our speech-to-text models for pre-recorded audio. The available languages vary by model. Check out the [Models](/getting-started/models) page to learn more about our different models and how to choose the best one for your use case. See our [Model selection](/pre-recorded-audio/select-the-speech-model) page for more details on specifying a model in your request.

## Universal-3.5 Pro

Universal-3.5 Pro supports the following 18 languages. To automatically fall back to Universal-2 for anything outside this set, set `speech_models` to `["universal-3-5-pro", "universal-2"]` and set `language_detection` to `true`.

### Regional dialects and variants

Universal-3.5 Pro goes beyond standard language support with deep understanding of regional dialects and local variants. Whether your audio features Quebecois French, Mexican Spanish, or Brazilian Portuguese, the model accurately captures speech as it's naturally spoken — including colloquial expressions, local vocabulary, and accent-specific pronunciation patterns.

  **Dialect support**

You do not need to specify a dialect code to get accurate dialect transcription. Universal-3.5 Pro automatically recognizes regional speech patterns when using the base language code (e.g., `fr` for all French dialects, `es` for all Spanish dialects).

| Dialect / Variant | Description |
| --- | --- |
| American English | Standard US English, including regional variants (Southern, Midwestern, Northeastern) |
| British English | UK English, including Received Pronunciation and regional accents |
| Australian English | Australian English with local expressions and pronunciation |

| Dialect / Variant | Description |
| --- | --- |
| Castilian Spanish | Standard Peninsular Spanish as spoken in central and northern Spain |
| Mexican Spanish | Mexican Spanish with local vocabulary and pronunciation |
| Argentine Spanish | Rioplatense Spanish with distinctive *voseo* and pronunciation |
| Colombian Spanish | Colombian Spanish with regional speech patterns |
| Chilean Spanish | Chilean Spanish with rapid speech patterns and local slang |
| Caribbean Spanish | Cuban, Dominican, and Puerto Rican Spanish dialects |
| Spanglish | English-Spanish code-mixing common in US bilingual communities |

| Dialect / Variant | Description |
| --- | --- |
| Metropolitan French | Standard Parisian French |
| Canadian French (Quebecois) | Quebec French with distinctive vocabulary, pronunciation, and expressions |
| Belgian French | Belgian French with local vocabulary and pronunciation |

| Dialect / Variant | Description |
| --- | --- |
| Brazilian Portuguese | Brazilian Portuguese with local vocabulary, pronunciation, and expressions |
| European Portuguese | Standard Lisbon Portuguese with Iberian pronunciation |

| Dialect / Variant | Description |
| --- | --- |
| Standard Italian | Standard Italian based on Tuscan-influenced speech |

## Universal-2

Universal-2 supports 99 languages. Pass the corresponding `language_code` in your transcription request to specify the language.

### Accuracy metrics

The following groups Universal-2 languages by transcription accuracy, measured by Word Error Rate (WER).

English, Spanish, French, German, Indonesian, Italian, Japanese, Dutch, Polish, Portuguese, Russian, Swedish, Turkish, Ukrainian, Catalan

10% to ≤25% WER)">
Arabic, Azerbaijani, Bulgarian, Bosnian, Mandarin Chinese, Czech, Danish,
Greek, Estonian, Finnish, Galician, Hebrew, Hindi, Croatian, Hungarian, Korean,
Macedonian, Malay, Norwegian, Romanian, Slovak, Swiss German,
Tagalog, Thai, Urdu, Vietnamese

25% to ≤50% WER)">
Afrikaans, Belarusian, Welsh, Persian (Farsi), Armenian, Icelandic,
Kazakh, Lithuanian, Latvian, Maori, Marathi, Slovenian, Swahili, Tamil

50% WER)">
Amharic, Assamese, Bengali, Gujarati, Hausa, Javanese, Georgian, Khmer,
Kannada, Luxembourgish, Lingala, Lao, Malayalam, Mongolian, Maltese, Burmese,
Nepali, Occitan, Punjabi, Pashto, Sindhi, Shona, Somali, Serbian, Telugu,
Tajik, Uzbek, Yoruba

## Unsupported feature behavior

Not all features are available for every language. If you enable a feature that isn't supported for the language of your audio, the API's behavior depends on how the language was specified:

- **With `language_code` (manual):** The API rejects the request and returns an error, such as `"The following models are not available in this language: speaker_labels"`. This lets you catch configuration issues before processing.
- **With `language_detection` (automatic):** The request completes normally, but any features that aren't supported for the detected language are silently omitted from the response. The transcription itself still succeeds. This is because the language isn't known until after the request is submitted, so the API can't validate feature compatibility upfront.

To avoid unexpected results when using Automatic Language Detection, use the `language_code` and `language_confidence` fields in the response to verify the detected language and handle cases where a feature may not have been applied.

---

<a id="custom-spelling-docs-pre-recorded-audio-correct-spelling-of-terms-mdx"></a>

# Custom Spelling

URL: https://www.assemblyai.com/docs/pre-recorded-audio/correct-spelling-of-terms
Source: docs/pre-recorded-audio/correct-spelling-of-terms.mdx
Navigation: Pre-recorded STT > Features
Description: Custom Spelling documentation.

Custom Spelling lets you customize how words are spelled or formatted in the transcript.

To use Custom Spelling, include `custom_spelling` in your transcription parameters. The parameter should be a list of dictionaries, with each dictionary specifying a mapping from a word or phrase to a new spelling or format of a word.

```python {20}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "custom_spelling": [
      {
        "from": ["Decarlo"],
        "to": "DeCarlo"
      },
      {
        "from": ["SQL"],
        "to": "Sequel"
      }
    ]
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

  

To use Custom Spelling, pass a dictionary to `set_custom_spelling()` on the transcription config. Each key-value pair specifies a mapping from a word or phrase to a new spelling or format of a word. The key specifies the new spelling or format, and the corresponding value is the word or phrase you want to replace.

```python {9}  expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)
config.set_custom_spelling(
  {
    "Gettleman": ["gettleman"],
    "SQL": ["Sequel"],
  }
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

  

To use Custom Spelling, include `custom_spelling` in your transcription parameters. The parameter should be an array of objects, with each object specifying a mapping from a word or phrase to a new spelling or format of a word.

```javascript {23}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);

let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();

const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  custom_spelling: [
    {
      from: ["Decarlo"],
      to: "DeCarlo",
    },
    {
      from: ["SQL"],
      to: "Sequel",
    },
  ],
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

To use Custom Spelling, include `custom_spelling` in your transcription parameters. The parameter should be an array of objects, with each object specifying a mapping from a word or phrase to a new spelling or format of a word.

```javascript {13}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  custom_spelling: [
    {
      from: ["Decarlo"],
      to: "DeCarlo",
    },
    {
      from: ["Sequel"],
      to: "SQL",
    },
  ],
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

  

The value in the `to` key is case-sensitive, but the value in the `from` key isn't. Additionally, the `to` key must only contain one word, while the `from` key can contain multiple words.

---

<a id="filler-words-docs-pre-recorded-audio-include-filler-words-mdx"></a>

# Filler Words

URL: https://www.assemblyai.com/docs/pre-recorded-audio/include-filler-words
Source: docs/pre-recorded-audio/include-filler-words.mdx
Navigation: Pre-recorded STT > Features
Description: Filler Words documentation.

The following filler words are removed by default:

- "um"
- "uh"
- "hmm"
- "mhm"
- "uh-huh"
- "ah"
- "huh"
- "hm"
- "m"

If you want to keep filler words in the transcript, you can set the `disfluencies` to `true` in the transcription config.

```python title="Python SDK" highlight={6} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    language_detection=True,
    disfluencies=True
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" highlight={19}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "disfluencies": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" highlight={12}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  disfluencies: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={19}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  disfluencies: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

---

<a id="word-search-docs-pre-recorded-audio-search-for-words-in-transcript-mdx"></a>

# Word Search

URL: https://www.assemblyai.com/docs/pre-recorded-audio/search-for-words-in-transcript
Source: docs/pre-recorded-audio/search-for-words-in-transcript.mdx
Navigation: Pre-recorded STT > Features
Description: Word Search documentation.

You can search through a completed transcript for a specific set of keywords, which is useful for quickly finding relevant information.

The parameter can be a list of words, numbers, or phrases up to five words.

```python title="Python SDK"  expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

# Set the words you want to search for
words = ["foo", "bar", "foo bar", "42"]

matches = transcript.word_search(words)

for match in matches:
  print(f"Found '{match.text}' {match.count} times in the transcript")
```

```python title="Python"  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

words = ['foo', 'bar', 'foo bar', '42']

word_search_endpoint = f"{base_url}/v2/transcript/{transcript_id}/word-search?words={','.join(words)}"

response_json = requests.get(word_search_endpoint, headers=headers).json()

for match in response_json['matches']:
  print(f"Found '{match['text']}' {match['count']} times in the transcript")
```

```javascript title="JavaScript SDK"  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  // Set the words you want to search for.
  const words = ["foo", "bar", "foo bar", "42"];

  const { matches } = await client.transcripts.wordSearch(transcript.id, words);

  for (const match of matches) {
    console.log(`Found '${match.text}' ${match.count} times in the transcript`);
  }
};

run();
```

```javascript title="JavaScript"  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcript = await res.json();

const transcriptId = transcript.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

const words = ["foo", "bar", "foo bar", "42"];

res = await fetch(`${baseUrl}/v2/transcript/${transcriptId}/word-search?words=${words.join(",")}`, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

for (const match of response.matches) {
  console.log(`Found '${match.text}' ${match.count} times in the transcript`);
}
```

---

<a id="set-the-start-and-end-of-the-transcript-docs-pre-recorded-audio-set-the-start-and-end-of-the-transcript-mdx"></a>

# Set the Start and End of the Transcript

URL: https://www.assemblyai.com/docs/pre-recorded-audio/set-the-start-and-end-of-the-transcript
Source: docs/pre-recorded-audio/set-the-start-and-end-of-the-transcript.mdx
Navigation: Pre-recorded STT > Features
Description: Set the Start and End of the Transcript documentation.

If you only want to transcribe a portion of your file, you can set the `audio_start_from` and the `audio_end_at` parameters in your transcription config.

```python title="Python SDK" highlight={7-8} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  audio_start_from=5000,  # The start time of the transcription in milliseconds
  audio_end_at=15000  # The end time of the transcription in milliseconds
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" highlight={19,20}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "audio_start_from": 5000, # The start time of the transcription in milliseconds
    "audio_end_at": 15000 # The end time of the transcription in milliseconds
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" highlight={12-13}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  audio_start_from: 5000, // The start time of the transcription in milliseconds
  audio_end_at: 15000, // The end time of the transcription in milliseconds
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={19-20}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  audio_start_from: 5000, // The start time of the transcription in milliseconds
  audio_end_at: 15000, // The end time of the transcription in milliseconds
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

---

<a id="transcript-status-docs-pre-recorded-audio-check-transcript-status-mdx"></a>

# Transcript Status

URL: https://www.assemblyai.com/docs/pre-recorded-audio/check-transcript-status
Source: docs/pre-recorded-audio/check-transcript-status.mdx
Navigation: Pre-recorded STT > Features > Transcription operations
Description: Transcript Status documentation.

After you've submitted a file for transcription, your transcript has one of the following statuses:

| Status       | Description                                        |
| ------------ | -------------------------------------------------- |
| `processing` | The audio file is being processed.                 |
| `queued`     | The audio file is waiting to be processed. This status is only returned when a job is actually queued, such as when you've exceeded your rate limit. Otherwise, transcripts go directly to `processing`. |
| `completed`  | The transcription has completed successfully.      |
| `error`      | An error occurred while processing the audio file. |

## Handling errors

If the transcription fails, the status of the transcript is `error`, and the transcript includes an `error` property explaining what went wrong.

```python title="Python SDK" highlight={11-13} 
from assemblyai import TranscriptStatus
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcript = transcriber.transcribe(audio_file, config)

if transcript.status == TranscriptStatus.error:
    print(f"Transcription failed: {transcript.error}")
    exit(1)

print(transcript.text)
```

```python title="Python" highlight={33-34}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript: {transcription_result['text']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)
```

```javascript title="JavaScript SDK" highlight={16-19} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    console.error(`Transcription failed: ${transcript.error}`);
    process.exit(1);
  }

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={35-36}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

A transcription may fail for various reasons:

- Unsupported file format
- Missing audio in file
- Unreachable audio URL

If a transcription fails due to a server error, we recommend that you resubmit the file for transcription to allow another server to process the audio.

  
A "400 Bad Request" error typically indicates that there's a problem with the formatting or content of the API request. Double-check the syntax of your request and ensure that all required parameters are included as described in the [API reference](/pre-recorded-audio/api-reference/transcripts/submit). If the issue persists, contact our support team for assistance.

---

<a id="transcript-export-options-docs-pre-recorded-audio-export-transcripts-as-srt-vtt-or-text-mdx"></a>

# Transcript export options

URL: https://www.assemblyai.com/docs/pre-recorded-audio/export-transcripts-as-srt-vtt-or-text
Source: docs/pre-recorded-audio/export-transcripts-as-srt-vtt-or-text.mdx
Navigation: Pre-recorded STT > Features > Transcription operations
Description: Transcript export options documentation.

This page explains the different ways you can export and format your transcript data, including SRT/VTT caption files, paragraphs and sentences, and word-level timestamps.

The plain-text transcript returned in the `text` field is raw transcript text only — it does not include timestamps or speaker labels. To build a custom text export in a format such as `[Speaker] timecode -> text`, enable [speaker labels](/pre-recorded-audio/label-speakers) and construct the output from the `utterances` array (which contains `speaker`, `start`, `end`, and `text` for each utterance). See the [Timestamped transcripts](/guides/timestamped-transcripts) guide and the [Create subtitles with speaker labels](/pre-recorded-audio/guides/speaker_labelled_subtitles) cookbook for examples.

## Export SRT or VTT caption files

You can export completed transcripts in SRT or VTT format, which can be used for subtitles and closed captions in videos.

You can also customize the maximum number of characters per caption by specifying the `chars_per_caption` parameter.

```python title="Python SDK" highlight={13-16}  expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

srt = transcript.export_subtitles_srt(
  # Optional: Customize the maximum number of characters per caption
  chars_per_caption=32
  )

with open(f"transcript_{transcript.id}.srt", "w") as srt_file:
  srt_file.write(srt)

# vtt = transcript.export_subtitles_vtt()

# with open(f"transcript_{transcript_id}.vtt", "w") as vtt_file:
#   vtt_file.write(vtt)

```

```python title="Python" highlight={40}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

# chars_per_caption is optional
srt_response = requests.get(f"{polling_endpoint}/srt?chars_per_caption=32", headers=headers)

with open(f"transcript_{transcript_id}.srt", "w") as srt_file:
  srt_file.write(srt_response.text)

# vtt_response = requests.get(f"{polling_endpoint}/vtt", headers=headers)

# with open(f"transcript_{transcript_id}.vtt", "w") as vtt_file:
#   vtt_file.write(vtt_response.text)
```

```javascript title="JavaScript SDK" highlight={18}  expandable
import { AssemblyAI } from "assemblyai";
import fs from "fs";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  let srt = await client.transcripts.subtitles(transcript.id, "srt", 32);
  fs.writeFileSync(`transcript_${transcript.id}.srt`, srt);

  // let vtt = await client.transcripts.subtitles(transcript.id, 'vtt', 32)
  // fs.writeFileSync(`transcript_${transcript.id}.vtt`, vtt)
};

run();
```

```javascript title="JavaScript" highlight={43}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

const srtEndpoint = `${baseUrl}/v2/transcript/${transcriptId}/srt?chars_per_caption=32`;
res = await fetch(srtEndpoint, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const srt = await res.text();

fs.writeFileSync(`transcript_${transcriptId}.srt`, srt);

// const vttEndpoint = `${baseUrl}/v2/transcript/${transcriptId}/vtt?chars_per_caption=32`
// const vtt = await fetch(vttEndpoint, { headers }).then(res => res.text())

// fs.writeFileSync(`transcript_${transcriptId}.vtt`, vtt)
```

## Export paragraphs

You can retrieve transcripts that are automatically segmented into paragraphs. The text of the transcript is broken down by paragraphs, along with additional metadata.

```python title="Python SDK" highlight={13-16}  expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

paragraphs = transcript.get_paragraphs()
for paragraph in paragraphs:
  print(paragraph.text)
  print()
```

```python title="Python" highlight={39-42}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

paragraphs = requests.get(polling_endpoint + '/paragraphs', headers=headers).json()['paragraphs']
for paragraph in paragraphs:
  print(paragraph['text'])
  print()
```

```javascript title="JavaScript SDK" highlight={15-18}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  const { paragraphs } = await client.transcripts.paragraphs(transcript.id);
  for (const paragraph of paragraphs) {
    console.log(paragraph.text);
  }
};

run();
```

```javascript title="JavaScript" highlight={42-50}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

const paragraphsEndpoint = `${baseUrl}/v2/transcript/${transcriptId}/paragraphs`;

res = await fetch(paragraphsEndpoint, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const paragraphsResponse = await res.json();
const paragraphs = paragraphsResponse.paragraphs;

for (const paragraph of paragraphs) {
  console.log(paragraph.text);
  console.log();
}
```

## Export sentences

You can retrieve transcripts that are automatically segmented into sentences, for a more reader-friendly experience. The text of the transcript is broken down by sentences, along with additional metadata.

```python title="Python SDK" highlight={13-16}  expandable
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

sentences = transcript.get_sentences()
for sentence in sentences:
  print(sentence.text)
  print()
```

```python title="Python" highlight={39-42}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

sentences = requests.get(polling_endpoint + '/sentences', headers=headers).json()['sentences']
for sentence in sentences:
  print(sentence['text'])
  print()
```

```javascript title="JavaScript SDK" highlight={15-19}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  const { sentences } = await client.transcripts.sentences(transcript.id);
  for (const sentence of sentences) {
    console.log(sentence.text);
  }
};

run();
```

```javascript title="JavaScript" highlight={44-50}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

const sentencesEndpoint = `${baseUrl}/v2/transcript/${transcriptId}/sentences`;

res = await fetch(sentencesEndpoint, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const sentencesResponse = await res.json();
const sentences = sentencesResponse.sentences;

for (const sentence of sentences) {
  console.log(sentence.text);
  console.log();
}
```

The response is an array of objects, each representing a sentence or a paragraph in the transcript. See the [API reference](/api-reference/transcripts/get-sentences) for more info.

## Word-level timestamps

The response also includes an array with information about each word:

```python title="Python SDK" highlight={10-11} 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcript = transcriber.transcribe(audio_file, config)

for word in transcript.words:
  print(f"Word: {word.text}, Start: {word.start}, End: {word.end}, Confidence: {word.confidence}")
```

```python title="Python" highlight={30-31}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    for word in transcription_result['words']:
      print(f"Word: {word['text']}, Start: {word['start']}, End: {word['end']}, Confidence: {word['confidence']}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" highlight={21}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);

  // Print word-level details
  for (const word of transcript.words) {
    console.log(
      `Word: ${word.text}, Start: ${word.start}, End: ${word.end}, Confidence: ${word.confidence}`
    );
  }
};

run();
```

```javascript title="JavaScript" highlight={35-38}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    // Print word-level details
    for (const word of transcriptionResult.words) {
      console.log(
        `Word: ${word.text}, Start: ${word.start}, End: ${word.end}, Confidence: ${word.confidence}`
      );
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

### API Reference

```js expandable
{
    words: [
      {
        text: "Smoke",
        start: 240,
        end: 640,
        confidence: 0.70473,
        speaker: null,
      },
      {
        text: "from",
        start: 680,
        end: 968,
        confidence: 0.99967,
        speaker: null,
      },
      {
        text: "hundreds",
        start: 1024,
        end: 1416,
        confidence: 0.99795,
        speaker: null,
      },
      {
        text: "of",
        start: 1448,
        end: 1592,
        confidence: 0.99926,
        speaker: null,
      },
      {
        text: "wildfires",
        start: 1616,
        end: 2248,
        confidence: 0.99838,
        speaker: null,
      },
      {
        text: "in",
        start: 2264,
        end: 2440,
        confidence: 0.99782,
        speaker: null,
      },
      {
        text: "Canada",
        start: 2480,
        end: 2968,
        confidence: 0.99977,
        speaker: null,
      },
    ],
  
}
```

## Additional resources

  
    Learn how to create caption files that include speaker identification.

---

<a id="delete-transcripts-docs-pre-recorded-audio-delete-transcripts-mdx"></a>

# Delete Transcripts

URL: https://www.assemblyai.com/docs/pre-recorded-audio/delete-transcripts
Source: docs/pre-recorded-audio/delete-transcripts.mdx
Navigation: Pre-recorded STT > Features > Transcription operations
Description: Delete Transcripts documentation.

You can remove the data from the transcript and mark it as deleted.

```python title="Python SDK" highlight={22}  expandable
import assemblyai
from assemblyai.prerecorded.v2 import Transcriber, Transcript, TranscriptionConfig

# `Transcript.delete_by_id()` and `Transcript.get_by_id()` use the default client, which reads the global settings.
assemblyai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
  language_detection=True
)

transcriber = Transcriber(config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)

Transcript.delete_by_id(transcript.id)

transcript = Transcript.get_by_id(transcript.id)
print(transcript.text)
```

```python title="Python" highlight={39}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

response = requests.delete(polling_endpoint, headers=headers)

print(response.json()['text'])
```

```javascript title="JavaScript SDK" highlight={18}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);

  const res = await client.transcripts.delete(transcript.id);

  console.log(res);
};

run();
```

```javascript title="JavaScript" highlight={42-44}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

res = await fetch(pollingEndpoint, {
  method: "DELETE",
  headers,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const deleteResponse = await res.json();

console.log(deleteResponse.text);
```

  **Data Retention**

  See [here](/data-retention-and-model-training) for more information on
  how long AssemblyAI retains data in the production environment.

---

<a id="upload-a-media-file-docs-pre-recorded-audio-api-reference-files-upload-mdx"></a>

# Upload a media file

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/files/upload
Source: docs/pre-recorded-audio/api-reference/files/upload.mdx
Navigation: Pre-recorded STT > API reference
Description: Upload a media file to AssemblyAI.

Metadata:
- openapi: ../../../openapi.yaml POST /v2/upload

---

<a id="submit-a-transcript-docs-pre-recorded-audio-api-reference-transcripts-submit-mdx"></a>

# Submit a transcript

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/submit
Source: docs/pre-recorded-audio/api-reference/transcripts/submit.mdx
Navigation: Pre-recorded STT > API reference
Description: Create a transcription request.

Metadata:
- openapi: ../../../openapi.yaml POST /v2/transcript

---

<a id="get-a-transcript-docs-pre-recorded-audio-api-reference-transcripts-get-mdx"></a>

# Get a transcript

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/get
Source: docs/pre-recorded-audio/api-reference/transcripts/get.mdx
Navigation: Pre-recorded STT > API reference
Description: Retrieve a transcript by ID.

Metadata:
- openapi: ../../../openapi.yaml GET /v2/transcript/{transcript_id}

---

<a id="get-transcript-sentences-docs-pre-recorded-audio-api-reference-transcripts-get-sentences-mdx"></a>

# Get transcript sentences

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/get-sentences
Source: docs/pre-recorded-audio/api-reference/transcripts/get-sentences.mdx
Navigation: Pre-recorded STT > API reference
Description: Retrieve transcript sentences.

Metadata:
- openapi: ../../../openapi.yaml GET /v2/transcript/{transcript_id}/sentences

---

<a id="get-transcript-paragraphs-docs-pre-recorded-audio-api-reference-transcripts-get-paragraphs-mdx"></a>

# Get transcript paragraphs

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/get-paragraphs
Source: docs/pre-recorded-audio/api-reference/transcripts/get-paragraphs.mdx
Navigation: Pre-recorded STT > API reference
Description: Retrieve transcript paragraphs.

Metadata:
- openapi: ../../../openapi.yaml GET /v2/transcript/{transcript_id}/paragraphs

---

<a id="get-subtitles-docs-pre-recorded-audio-api-reference-transcripts-get-subtitles-mdx"></a>

# Get subtitles

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/get-subtitles
Source: docs/pre-recorded-audio/api-reference/transcripts/get-subtitles.mdx
Navigation: Pre-recorded STT > API reference
Description: Export subtitles for a transcript.

Metadata:
- openapi: ../../../openapi.yaml GET /v2/transcript/{transcript_id}/{subtitle_format}

---

<a id="get-redacted-audio-docs-pre-recorded-audio-api-reference-transcripts-get-redacted-audio-mdx"></a>

# Get redacted audio

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/get-redacted-audio
Source: docs/pre-recorded-audio/api-reference/transcripts/get-redacted-audio.mdx
Navigation: Pre-recorded STT > API reference
Description: Retrieve redacted audio for a transcript.

Metadata:
- openapi: ../../../openapi.yaml GET /v2/transcript/{transcript_id}/redacted-audio

---

<a id="word-search-docs-pre-recorded-audio-api-reference-transcripts-word-search-mdx"></a>

# Word search

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/word-search
Source: docs/pre-recorded-audio/api-reference/transcripts/word-search.mdx
Navigation: Pre-recorded STT > API reference
Description: Search a transcript for words or phrases.

Metadata:
- openapi: ../../../openapi.yaml GET /v2/transcript/{transcript_id}/word-search

---

<a id="list-transcripts-docs-pre-recorded-audio-api-reference-transcripts-list-mdx"></a>

# List transcripts

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/list
Source: docs/pre-recorded-audio/api-reference/transcripts/list.mdx
Navigation: Pre-recorded STT > API reference
Description: List transcripts.

Metadata:
- openapi: ../../../openapi.yaml GET /v2/transcript

---

<a id="delete-a-transcript-docs-pre-recorded-audio-api-reference-transcripts-delete-mdx"></a>

# Delete a transcript

URL: https://www.assemblyai.com/docs/pre-recorded-audio/api-reference/transcripts/delete
Source: docs/pre-recorded-audio/api-reference/transcripts/delete.mdx
Navigation: Pre-recorded STT > API reference
Description: Delete a transcript.

Metadata:
- openapi: ../../../openapi.yaml DELETE /v2/transcript/{transcript_id}

---

<a id="webhooks-for-pre-recorded-audio-docs-pre-recorded-audio-webhooks-mdx"></a>

# Webhooks for pre-recorded audio

URL: https://www.assemblyai.com/docs/pre-recorded-audio/webhooks
Source: docs/pre-recorded-audio/webhooks.mdx
Navigation: Pre-recorded STT > Advanced
Description: Get notified when a pre-recorded audio transcription is ready.

Webhooks are custom HTTP callbacks that you can define to get notified when your pre-recorded audio transcripts are ready.

  This guide covers webhooks for [pre-recorded audio
  transcription](/pre-recorded-audio). For webhooks with
  streaming audio, see [Webhooks for streaming
  speech-to-text](/streaming/webhooks).

To use webhooks, you need to set up your own webhook receiver to handle webhook deliveries.

## Create a webhook for a transcription

  **Don't have a webhook endpoint yet?**

  Create a test webhook endpoint with [webhook.site](https://webhook.site) to
  test your webhook integration.

To create a webhook, set the `webhook_url` parameter when you create a new transcription. The URL must be accessible from AssemblyAI's servers.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url,
    "webhook_url": "https://example.com/webhook"
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)
```

To create a webhook, use `set_webhook()` on the transcription config. The URL must be accessible from AssemblyAI's servers.

Use `submit()` instead of `transcribe()` to create a transcription without waiting for it to complete.

```python
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig().set_webhook("https://example.com/webhook")

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcriber.submit(audio_file, config)
```

To create a webhook, set the `webhook_url` parameter when you create a new transcription. The URL must be accessible from AssemblyAI's servers.

```javascript expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl,
  webhook_url: "https://example.com/webhook",
};

const url = `${baseUrl}/v2/transcript`;
const response = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!response.ok) throw new Error(`Error: ${response.status}`);
```

To create a webhook, include the `webhook_url` parameter when you create a new transcription. The URL must be accessible from AssemblyAI's servers.

Use `submit()` instead of `transcribe()` to create a transcription without waiting for it to complete.

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  webhook_url: "https://example.com/webhook",
};

const run = async () => {
  const transcript = await client.transcripts.submit(params);
};

run();
```

## Handle webhook deliveries

When the transcript is ready, AssemblyAI will send a `POST` HTTP request to the URL that you specified.

Your webhook endpoint must return a 2xx HTTP status code within 10 seconds to indicate successful receipt. If a 2xx status is not received within 10 seconds, AssemblyAI will retry the webhook call up to a total of 10 attempts. If at any point your endpoint returns a 4xx status code, the webhook call is considered failed and will not be retried.

  **Static Webhook IP addresses**

AssemblyAI sends all webhook deliveries from fixed IP addresses:

| Region | IP Address     |
| ------ | -------------- |
| US     | `44.238.19.20` |
| EU     | `54.220.25.36` |

### Delivery payload

The webhook delivery payload contains a JSON object with the following properties:

```json
{
  "transcript_id": "5552493-16d8-42d8-8feb-c2a16b56f6e8",
  "status": "completed"
}
```

| Key             | Type   | Description                                                  |
| --------------- | ------ | ------------------------------------------------------------ |
| `transcript_id` | string | The ID of the transcript.                                    |
| `status`        | string | The status of the transcript. Either `completed` or `error`. |

The webhook payload only contains the `transcript_id` and `status`. It doesn't include the transcript text or error details. If the `status` is `"error"`, make a `GET /v2/transcript/{transcript_id}` request to retrieve the error message from the `error` field in the response. See [Retrieve a transcript with the transcript ID](#retrieve-a-transcript-with-the-transcript-id) below.

### Webhook status code

When you retrieve the transcript using the transcript ID, the JSON response will include a `webhook_status_code` field. This field contains the HTTP status code that AssemblyAI received when calling your webhook endpoint, allowing you to verify that the webhook was delivered successfully.

### Retrieve a transcript with the transcript ID

```python
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

transcript_id = "<TRANSCRIPT_ID>"

polling_endpoint = f"https://api.assemblyai.com/v2/transcript/{transcript_id}"

transcription_result = requests.get(polling_endpoint, headers=headers).json()

if transcription_result['status'] == 'completed':
  print(f"Transcript ID: {transcript_id}")
elif transcription_result['status'] == 'error':
  raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

```

```python
import assemblyai
from assemblyai.prerecorded.v2 import Transcript

# `Transcript.get_by_id()` uses the default client, which reads the global settings.
assemblyai.settings.api_key = "<YOUR_API_KEY>"

transcript = Transcript.get_by_id("<TRANSCRIPT_ID>")

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)

```

```javascript
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const transcriptId = "<TRANSCRIPT_ID>";
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let res = await fetch(pollingEndpoint, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptionResult = await res.json();

if (transcriptionResult.status === "completed") {
  console.log(transcriptionResult.text);
} else if (transcriptionResult.status === "error") {
  throw new Error(`Transcription failed: ${transcriptionResult.error}`);
}
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const transcript = await client.transcripts.get("<TRANSCRIPT_ID>");

if (transcript.status === "error") {
  throw new Error(`Transcription failed: ${transcript.error}`);
}

console.log(transcript.text);
```

## Authenticate webhook deliveries

You can authenticate webhook deliveries from AssemblyAI by including a custom HTTP header in the request.

To add an authentication header, include the auth header name and value in `set_webhook()`.

```python {2}
config = TranscriptionConfig().set_webhook(
    "https://example.com/webhook", "X-My-Webhook-Secret", "secret-value"
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcriber.submit(audio_url, config)
```

To add an authentication header, include the `webhook_auth_header_name` and `webhook_auth_header_value` parameters.

```javascript {5-6}
client.transcripts.submit({
  audio:
    'https://assembly.ai/wildfires.mp3',
  webhook_url: 'https://example.com/webhook',
  webhook_auth_header_name: "X-My-Webhook-Secret",
  webhook_auth_header_value: "secret-value"
})
```

To add an authentication header, include the auth header name and value in `set_webhook()`.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url,
    "webhook_url": "https://example.com/webhook",
    "webhook_auth_header_name": "X-My-Webhook-Secret",
    "webhook_auth_header_value": "secret-value"
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)
```

To add an authentication header, include the auth header name and value in `set_webhook()`.

```python
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig().set_webhook("https://example.com/webhook", "X-My-Webhook-Secret", "secret-value")

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcriber.submit(audio_file, config)
```

To add an authentication header, include the `webhook_auth_header_name` and `webhook_auth_header_value` parameters.

```javascript expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl,
  webhook_url: "https://example.com/webhook",
  webhook_auth_header_name: "X-My-Webhook-Secret",
  webhook_auth_header_value: "secret-value",
};

const url = `${baseUrl}/v2/transcript`;
const response = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!response.ok) throw new Error(`Error: ${response.status}`);
```

To add an authentication header, include the `webhook_auth_header_name` and `webhook_auth_header_value` parameters.

```javascript expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  webhook_url: "https://example.com/webhook",
  webhook_auth_header_name: "X-My-Webhook-Secret",
  webhook_auth_header_value: "secret-value",
};

const run = async () => {
  const transcript = await client.transcripts.submit(params);
};

run();
```

## Add metadata to webhook deliveries

To associate metadata for a specific transcription request, you can add your own query parameters to the webhook URL.

```plain
https://example.com/webhook?customer_id=1234&order_id=5678
```

Now, when you receive the webhook delivery, you'll know the customer who requested it.

## Failed webhook deliveries

Webhook deliveries can fail for multiple reasons. For example, if your server is down or takes more than 10 seconds to respond.

If a webhook delivery fails, AssemblyAI will attempt to redeliver it up to 10 times, waiting 10 seconds between each attempt. If all attempts fail, AssemblyAI considers the delivery as permanently failed.

---

<a id="benchmarks-docs-pre-recorded-audio-benchmarks-mdx"></a>

# Benchmarks

URL: https://www.assemblyai.com/docs/pre-recorded-audio/benchmarks
Source: docs/pre-recorded-audio/benchmarks.mdx
Navigation: Pre-recorded STT > Advanced
Description: Industry-leading accuracy for pre-recorded speech-to-text.

Benchmarks are an important first step before running your own [evaluation](/pre-recorded-audio/evaluations). Below are the current benchmarks for our pre-recorded models so you can assess performance across accuracy, latency, and error rates.

  Public benchmarks can be misleading due to overfitting and benchmark gaming.
  We strongly recommend [running your own evaluation](/pre-recorded-audio/evaluations) on your
  audio data to identify the best model for your use case.

For the full interactive benchmark experience with competitive comparisons, visit [assemblyai.com/benchmarks](https://www.assemblyai.com/benchmarks).

## Word error rate (WER)

Word Error Rate (WER) is the classical metric for speech-to-text accuracy. It counts substitutions, deletions, and insertions against a reference transcript, divided by the total word count in the ground truth. AssemblyAI Universal-3.5 Pro achieves a mean WER of **5.6%** (median **4.9%**) on English benchmarks.

WER weights every word equally, so a misrecognized filler word counts the same as a misrecognized email address or medication name. For production voice workflows, we recommend pairing WER with [Missed entity rate](#missed-entity-rate), which measures accuracy on the high-stakes entities — names, emails, phone numbers, and medical terms — that actually drive end-user outcomes.

### English benchmarks

Most recent update: **January 2026**.

| Dataset                                                                                    | Universal-3.5 Pro WER (%)            | Universal-2 WER (%)                | Relative gain vs Universal-2          |
| ------------------------------------------------------------------------------------------ | ---------------------------------- | ---------------------------------- | ------------------------------------- |
| **Overall Performance**                                                                    | **Mean: 5.6%** \| **Median: 4.9%** | **Mean: 6.1%** \| **Median: 6.5%** | **Mean: 8.2%** \| **Median: 24.6%**   |
| [commonvoice](https://github.com/common-voice/cv-dataset)                                  | 4.87%                              | 6.48%                              | 24.8%                        |
| [earnings21](https://huggingface.co/datasets/distil-whisper/earnings21)                    | 8.80%                              | 9.37%                              | 6.1%                         |
| [librispeech_test_clean](https://huggingface.co/datasets/AudioLLMs/librispeech_test_clean) | 1.52%                              | 1.68%                              | 9.5%                         |
| [librispeech_test_other](https://huggingface.co/datasets/openslr/librispeech_asr)          | 2.69%                              | 3.00%                              | 10.3%                        |
| [meanwhile](https://huggingface.co/datasets/distil-whisper/meanwhile)                      | 4.22%                              | 4.41%                              | 4.3%                         |
| [tedlium](https://huggingface.co/datasets/lmms-lab/tedlium)                                | 6.77%                              | 7.30%                              | 7.3%                         |
| [rev16](https://huggingface.co/datasets/distil-whisper/rev16)                              | 10.29%                             | 10.32%                             | 0.3%                         |

### Multilingual benchmarks

Most recent update: **January 2026**. Dataset: [FLEURS](https://huggingface.co/datasets/google/fleurs).

| Language Code | Language   | Universal-3.5 Pro WER (%) | Universal-2 WER (%) | Relative gain vs Universal-2 |
| ------------- | ---------- | ----------------------- | ------------------- | ---------------------------- |
| **Average**   | **All**    | **4.58%**               | **7.42%**           | **38.3%**                    |
| de            | German     | 4.88%                   | 6.22%               | 21.5%                        |
| en            | English    | -                       | 4.38%               | -                            |
| es            | Spanish    | 3.98%                   | 4.56%               | 12.7%                        |
| fi            | Finnish    | -                       | 10.10%              | -                            |
| fr            | French     | 4.98%                   | 7.56%               | 34.1%                        |
| hi            | Hindi      | -                       | 7.38%               | -                            |
| it            | Italian    | 3.69%                   | 4.75%               | 22.3%                        |
| ja            | Japanese   | -                       | 7.79%               | -                            |
| ko            | Korean     | -                       | 14.54%              | -                            |
| nl            | Dutch      | -                       | 7.79%               | -                            |
| pl            | Polish     | -                       | 6.63%               | -                            |
| pt            | Portuguese | 5.39%                   | 5.98%               | 9.9%                         |
| ru            | Russian    | -                       | 5.80%               | -                            |
| tr            | Turkish    | -                       | 8.12%               | -                            |
| uk            | Ukrainian  | -                       | 7.42%               | -                            |
| vi            | Vietnamese | -                       | 9.75%               | -                            |

## Missed entity rate

For production voice workflows, the actual words that matter most are entities — names, organizations, emails, phone numbers, and medical terms. The Missed Entity Rate (MER) measures how often a model fails to correctly transcribe these high-stakes terms. See [Missed Entity Rate](/pre-recorded-audio/evaluations#missed-entity-rate-mer) for the full definition.

Universal-3.5 Pro delivers relative gains over Universal-2 across every entity category we track for voice workflows, with the largest improvements on emails, locations, and medical terms.

| Entity type         | Universal-3.5 Pro MER (%) | Universal-2 MER (%) | Relative gain vs Universal-2 |
| ------------------- | ----------------------- | ------------------- | ---------------------------- |
| Medical terms       | 13.15%                  | 18.43%              | 28.6%                        |
| Locations           | 8.61%                   | 12.40%              | 30.6%                        |
| Job titles          | 9.03%                   | 9.86%               | 8.4%                         |
| Organization names  | 17.02%                  | 20.96%              | 18.8%                        |
| Email addresses     | 33.76%                  | 53.81%              | 37.3%                        |
| Phone numbers       | 13.14%                  | 14.69%              | 10.6%                        |
| Credit card numbers | 21.83%                  | 25.07%              | 12.9%                        |

## Hallucinations and consecutive errors

Hallucinations are a critical concern in production STT systems. AssemblyAI reduces hallucinations by 30% compared to Whisper, across three error categories:

- **Fabrications** — words inserted that were never spoken
- **Omissions** — spoken words that are missing from the transcript
- **Hallucinations** — extended sequences of fabricated content

## Benchmark challenges

Models are often trained on publicly available datasets — sometimes the very same datasets used for evaluation. When this happens, the model becomes overfit to the evaluation set and will show artificially strong performance on standard WER tests. This makes WER potentially misleading, as real-world performance on unseen audio will be significantly worse.

## External benchmarks

For third-party benchmarks, we recommend the [Hugging Face ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard). Note that many models listed require self-hosting and lack production features like speaker diarization and automatic language detection.

## Methodology

Our benchmarks are evaluated across **250+ hours of audio data**, **80,000+ audio files**, and **26 datasets**. We apply standard text normalization before calculating metrics. For full details on our methodology, visit [assemblyai.com/benchmarks](https://www.assemblyai.com/benchmarks).

## Run your own benchmark

We'd be happy to help. AssemblyAI has a benchmarking tool to help you run a custom evaluation against your real audio files. [Contact us](https://www.assemblyai.com/contact/support) for more information.

You can also run your own benchmarks following the [Hugging Face framework](https://github.com/huggingface/open_asr_leaderboard) which provides a GitHub repo with full instructions.

---

<a id="evaluating-pre-recorded-stt-models-docs-pre-recorded-audio-evaluations-mdx"></a>

# Evaluating Pre-recorded STT models

URL: https://www.assemblyai.com/docs/pre-recorded-audio/evaluations
Source: docs/pre-recorded-audio/evaluations.mdx
Navigation: Pre-recorded STT > Advanced
Description: Evaluating Pre-recorded STT models documentation.

## Introduction

The high level objective of a pre-recorded STT model evaluation is to answer the question: _Which Speech-to-text model is the best for my product?_

This guide provides a step-by-step framework for evaluating and benchmarking pre-recorded Speech-to-text models, with specific guidance for evaluating Universal-3.5 Pro and its prompting capabilities.

  Need help with evaluations or prompt optimization? [Contact our Sales
  team](https://www.assemblyai.com/contact/sales) — we can help you design an
  evaluation, optimize prompts for your audio, and benchmark against your ground
  truth data.

## Evaluation metrics

### Traditional metrics

#### Word Error Rate (WER)

$$
WER = \frac{S + D + I}{N}
$$

This formula takes the number of Substitutions (S), Deletions (D), and Insertions (I), and divides their sum by the Total Number of Words in the ground truth transcript (N).

  While WER calculation may seem simple, it requires a methodical granular
  approach and reliable reference data. Word Error Rate can tell you how
  “different” the automatic transcription was compared to the human
  transcription, and *generally*, this is a reliable metric to determine how
  “good” a transcription is. For more info on WER as a metric, read Dylan Fox’s
  [blog post on Word Error Rate](https://www.assemblyai.com/blog/word-error-rate).

#### Concatenated minimum-Permutation Word Error Rate (cpWER)

$$
\text{cpWER} = \frac{S_{\text{spk}} + D + I}{N}
$$

cpWER is similar to WER, but it also measures the number of errors a speech recognition model makes where words with incorrectly-ascribed speakers are considered to be incorrect. The primary difference from standard WER is how _S_ is calculated: $S_{\text{spk}}$ counts both word substitutions and correctly transcribed words that are assigned to the wrong speaker. A correct word with an incorrect speaker label counts as a substitution error, thereby penalizing both transcription and speaker diarization mistakes.

#### Formatted WER (F-WER)

F-WER is similar to WER but F-WER does not apply text normalization, so all formatting differences are accounted for, in addition to word differences when computing the WER. Therefore, F-WER is always higher than or equal to WER.

#### Sentence Error Rate (SER)

$$
\text{SER} = \frac{N_{\text{err}}}{N_{\text{sent}}}
$$

The Sentence Error Rate (SER) is the ratio of the number of sentences with one or more errors to the total number of sentences.

#### Diarization Error Rate (DER)

$$
DER = \frac{false alarm + missed detection + confusion}{total}
$$

This formula takes the duration of non-speech incorrectly classified as speech (false alarm), the duration of speech incorrectly classified as non-speech (missed detection), the duration of speaker confusion (confusion), and divides the sum over the total speech duration.

#### Missed Entity Rate (MER)

$$
\text{MER} = 1 - \frac{N_{\text{rec}}}{N_{\text{total}}}
$$

Fundamentally, MER is a negative recall rate computed for specified target entities. It is defined as the number of correctly transcribed entities relative to their total occurrence count. It accounts for multiple occurrences of the same entity and their positions within the hypothesis transcription. Our Research team proposes this as the best metric to measure the effectiveness of word boost.

For a simpler approach when evaluating high-stakes entities (such as credit card numbers, names, or dosages), consider a binary pass/fail metric per file: score each file as 1 if the model captured the target entity correctly, or 0 if it did not. Run this across 100 or more files for statistical reliability. This is especially useful when you need a quick signal on entity accuracy at scale without computing full MER breakdowns.

### Metrics for Universal-3.5 Pro

Universal-3.5 Pro is significantly more capable than prior models, and traditional WER alone may not fully capture its performance. The following metrics provide a more complete picture.

#### Semantic WER

Traditional WER treats every difference between the model output and a reference transcript as an error—even when the difference is semantically equivalent. Semantic WER corrects this by normalizing equivalent words and phrases before calculating WER, so that differences like `dr.` vs `doctor` or `1300` vs `thirteen hundred` aren't counted as errors.

##### Rule-based normalization

At its simplest, Semantic WER is a preprocessing step. Before running standard WER, apply find-and-replace rules to both the reference and hypothesis transcripts:

- **Number formats**: `1300` → `thirteen hundred`, `$5` → `five dollars`
- **Abbreviations and titles**: `dr.` → `doctor`, `mr.` → `mister`, `govt` → `government`
- **Contractions**: `gonna` → `going to`, `can't` → `cannot`
- **Variant spellings**: `grey` → `gray`, `cancelled` → `canceled`
- **Filler words**: Remove `um`, `uh`, `you know` from both sides (or keep both—just be consistent)

This alone eliminates a significant portion of false errors and can be implemented in a few lines of Python. No model inference required.

##### LLM-based scoring

For cases where simple rules can't capture the nuance—was an omission meaningful? Is a proper noun misspelling close enough?—an LLM can perform word-level alignment and classify each difference by severity:

- **No penalty**: Semantically equivalent forms (number formats, contractions, variant spellings)
- **Minor penalty**: Single-character misspellings, minor grammatical markers
- **Major penalty**: Incorrect substitutions, meaning-altering errors, significant omissions or additions of content words

These approaches are particularly valuable for Universal-3.5 Pro because the model often transcribes audio more accurately than human transcribers, producing differences that are correct but would be penalized by traditional WER. For an implementation of Semantic WER using Bayesian optimization, see [prompt-seeker](https://github.com/AssemblyAI-Solutions/prompt-seeker).

#### LASER score (LLM-based ASR Evaluation Rubric)

LASER is a published LLM-based evaluation metric ([Parulekar & Jyothi, EMNLP 2025](https://aclanthology.org/2025.emnlp-main.1257/)) that uses an LLM prompt with detailed examples to classify ASR errors and compute a score:

$$
\text{LASER} = 1 - \frac{\text{Total Penalty}}{\text{Reference Word Count}}
$$

The LLM aligns each word in the ASR output against the reference transcription and assigns a penalty per word pair:

- **No penalty (0)**: Acceptable variations including numerical format differences, abbreviations, compound word splits, transliterations, alternate spellings, proper noun variants, and colloquial terms
- **Minor penalty (0.5)**: Small spelling errors (single character) or minor grammatical errors (gender, tense, number markers) that preserve sentence meaning
- **Major penalty (1.0)**: Incorrect word substitutions, significant omissions or additions, and reordering that changes meaning

LASER provides structured per-error feedback alongside the score. This makes it useful for prompt optimization workflows where you need to understand _why_ a prompt performed poorly, not just _how much_ error there was. For an implementation of LASER scoring, see [aai-cli](https://github.com/alexkroman/aai-cli).

### Why new metrics matter

Traditional WER treats every difference between the model output and human transcription as an error. Universal-3.5 Pro’s contextual awareness means it will often transcribe words that human transcribers missed entirely. In traditional WER, these show up as **insertions** (penalized errors), even though the model is correct. This makes WER an unreliable metric when used alone — your evaluation is only as good as your ground truth labels.

WER is only as good as your ground truth labels. Human transcriptions contain systematic errors — missed filler words, incorrect proper nouns, simplified speech patterns, and translated code-switching. When Universal-3.5 Pro transcribes audio more accurately than the human label, those improvements show up as WER _errors_.

Before reporting WER, manually audit at least 20 insertions to determine what percentage are true errors versus ground truth omissions. In our testing, the majority of insertions were cases where Universal-3.5 Pro correctly transcribed audio that the human transcriber missed.

This is why [Artificial Analysis](https://artificialanalysis.ai/speech-to-text), an independent AI benchmarking organization, had to create proprietary evaluation datasets with manually corrected ground truths when building their [Speech-to-Text leaderboard](https://artificialanalysis.ai/speech-to-text). Existing public datasets contain systematic human transcription errors that penalize models which are actually more accurate.

## The evaluation process

This section provides a step-by-step guide on how to run an evaluation. The evaluation process should closely match your production environment, including the files you intend to transcribe, the model you intend to use, and the settings applied to those models.

### Step 1: Prepare your evaluation dataset

Ensure that the files you use to benchmark are representative of the files you plan to use in production. For example, if you plan to transcribe meetings, gather a set of meeting recordings. If you plan to transcribe phone calls, focus on finding phone calls that match your customer base’s language and region.

We recommend using at least 25 files that are representative of your use case. Length is less important than diversity of audio conditions — a good evaluation set covers the range of speakers, accents, audio quality, and vocabulary your model will encounter in production.

Then, gather human-labeled data to act as your source of ground truth. Ground truth is accurately transcribed audio data that will serve as the "correct answer" for our benchmark. Human-labeled data can be purchased from an external vendor or created manually.

  Open-source audio corpora (for example, datasets on [Hugging Face](https://huggingface.co/datasets?task_categories=automatic-speech-recognition)) can serve as a starting point for building ground truth, but they require review and correction before use in production evaluations. These datasets contain the same systematic human transcription errors described below — missing filler words, incorrect proper nouns, and simplified speech patterns — and should be audited against your actual audio before benchmarking.

#### Ground truth quality

The quality of your ground truth data directly affects the reliability of your evaluation. With Universal-3.5 Pro, this is more important than ever because the model frequently outperforms human transcribers.

Common issues with ground truth data:

- **Missing filler words**: Human transcribers often omit `um`, `uh`, `like`, and other disfluencies
- **Incorrect proper nouns**: Rare names, technical terms, and domain vocabulary are often misspelled
- **Simplified speech patterns**: Human transcribers tend to “clean up” speech, missing repetitions, false starts, and self-corrections
- **Code-switching errors**: Multilingual segments are frequently translated to English rather than transcribed as spoken

Before running evaluations, audit a sample of your ground truth files by listening to the audio and comparing. If your ground truth contains systematic errors, your WER numbers will be misleading. To inspect and correct issues in your ground truth files, use the [Truth File Corrector](https://www.assemblyai.com/dashboard/home) in the AssemblyAI Dashboard (found at the bottom of the left sidebar), which lets you listen back to audio and fix human transcription errors by clicking through differences.

See [this article](https://www.assemblyai.com/blog/new-word-error-rate-wer-benchmark) to learn more about why your word error rate (WER) benchmark might be lying to you.

#### Dataset diversity

A prompt that performs well overall may underperform on specific audio types. Include a diverse mix of audio in your evaluation set and track per-dataset breakdowns:

| Audio type            | Characteristics                         | Typical WER range                      |
| --------------------- | --------------------------------------- | -------------------------------------- |
| Earnings calls        | Clean English, formal vocabulary        | Low                                    |
| Meeting recordings    | Multi-speaker, informal                 | Moderate                               |
| Code-switching audio  | Mixed languages (e.g., English/Spanish) | Higher (normalization affects scoring) |
| Medical consultations | Clinical vocabulary, accented speech    | Moderate                               |
| Phone calls           | Compression artifacts, background noise | Moderate to high                       |

### Step 2: Establish a baseline

Before optimizing prompts, measure your baseline performance by transcribing your evaluation set with Universal-3.5 Pro and **no custom prompt**. The built-in default is already applied when `prompt` is omitted, and it outperforms most custom prompts. Record both WER and Semantic WER so you can track improvements as you layer instructions on top.

### Step 3: Transcribe and evaluate with prompts

Transcribe your files using [AssemblyAI’s API](/pre-recorded-audio/getting-started/transcribe-an-audio-file) with Universal-3.5 Pro and your candidate prompts.

When crafting evaluation prompts, use the [prompting guide](/pre-recorded-audio/universal-3-5-pro/prompting) as a reference. Key principles:

- **Use authoritative language**: The model responds better to `Mandatory:`, `Required:`, and `Always:` than soft language like `try to` or `please`
- **Be specific about speech patterns**: Enumerate what you want preserved (disfluencies, filler words, hesitations, repetitions, stutters, false starts, colloquialisms)
- **Give instructions, not just context**: `This is a doctor-patient visit. Prioritize accurately transcribing medications and diseases wherever possible.` is far more effective than `This is a doctor-patient visit.`
- **Start with fewer instructions, add one at a time**: Every added instruction risks conflicting with another. Add a single instruction, evaluate it against your dataset, and only then add the next.

### Step 4: Text normalization

Before calculating WER metrics, both reference (ground truth) and hypothesis (model generated) texts need to be normalized to ensure a fair comparison.

This accounts for differences in:

- Punctuation and capitalization
- Number formatting (e.g., “twenty-one” vs. “21”)
- Contractions and abbreviations
- Other stylistic variations that don’t affect meaning

Normalization can be done with a library like [Whisper Normalizer](https://pypi.org/project/whisper-normalizer/).

  If you are prompting Universal-3.5 Pro to include `[unclear]` or `[masked]` tags
  for uncertain audio, ensure your normalizer strips these tags before computing
  WER. Otherwise, they will be counted as insertions.

### Step 5: Compare and calculate

Calculate the error rates using the formulas above or consider using a library like [jiwer](https://github.com/jitsi/jiwer). For Semantic WER, apply text normalization replacements before calculating WER. For LASER scoring, use an LLM-based evaluator (see [Open-source tools](#open-source-tools) below).

When reviewing results:

- Check per-dataset breakdowns, not just aggregate WER
- Audit insertions manually by listening to the audio
- Compare both traditional WER and Semantic WER to get a full picture
- Track which prompt components improve which audio types

### Qualitative analysis

Quantitative metrics don’t capture everything. Qualitative analysis helps you identify differences between STT providers that metrics might miss — for example, how certain key terms are transcribed can make or break a transcript, even if the rest of the transcript has a lower overall error rate.

Qualitative analysis is also useful for tie-breaking when benchmarking metrics don’t clearly favor one model over another. Since you’re comparing models against each other, ground truth files aren’t required.

**Side-by-side comparison**: Have users compare and pick their preferred transcript between two **formatted** outputs from different STT providers. Tools like [Diffchecker](https://www.diffchecker.com/) or any side-by-side interface work well for this.

**LLM as judge**: An LLM can automatically identify differences between two transcriptions and pick a winner. However, be cautious: an LLM judge can be misled by outputs that _look_ correct but contain subtle errors (such as translated code-switching segments that read well in English but don’t reflect what was actually spoken). Always pair LLM-based judgments with spot-checking against the actual audio.

**A/B testing in production**: Serve transcripts from different providers to users and collect feedback. You can ask users to score transcripts directly, or track indirect signals like the number of support ticket complaints about transcription quality.

### Domain-specific evaluation considerations

WER is not always the right primary metric. Some domains prioritize output qualities that traditional accuracy metrics do not capture:

- **Medical scribes**: Customers often evaluate based on **user preference rate** and **readability** — whether clinicians prefer the transcript output for generating clinical notes. Formatting quality, medical terminology accuracy, and structured output can matter more than raw WER. See the [Medical Scribe guides](/medical-scribe-best-practices) for domain-specific evaluation guidance.
- **Legal transcription**: Verbatim accuracy including disfluencies and speaker attribution may be more important than clean, readable output.
- **Media and entertainment**: Proper noun accuracy for names, places, and brands can outweigh overall WER.

When running evaluations for domain-specific use cases, define your success criteria before choosing metrics. If your end users care about readability and preference, include qualitative evaluation (side-by-side comparisons, user preference scoring) alongside quantitative metrics.

## Iterating on prompts

Finding the optimal prompt for your use case is an iterative process. There are two main approaches:

### Manual iteration

1. Start with the [default system prompt](/pre-recorded-audio/universal-3-5-pro/prompting#best-all-around-default) or one of the reference prompts below
2. Transcribe a representative sample of your audio
3. Review the output against your ground truth, focusing on the types of errors that matter most for your use case
4. Adjust the prompt to address specific error patterns
5. Re-evaluate and compare

### Automated optimization

For large-scale prompt optimization, consider using one of the open-source tools described below. These tools systematically test prompt component combinations and score them against your evaluation data, converging on the best prompt for your specific audio.

### Reference prompts for evaluation

Use these prompts directly from the [prompting guide](/pre-recorded-audio/universal-3-5-pro/prompting#recommended-prompts) as your evaluation prompts.

#### Evaluation prompt

Start with the built-in default ([Best all around](/pre-recorded-audio/universal-3-5-pro/prompting#best-all-around-default)). Omit the `prompt` parameter to use it — you don’t need to set it explicitly:

```text
Transcribe with context and proper nouns preserved, where speech is
present in the audio. Each language as spoken. English as English.
Non-native speakers.
```

For maximum verbatim capture and multilingual code-switching, use the [Verbatim with multilingual support](/pre-recorded-audio/universal-3-5-pro/prompting#verbatim-with-multilingual-support) prompt instead. The trade-off is that the model may occasionally hallucinate disfluencies or language switches that don’t exist in the audio.

#### Comparison prompt (for identifying model uncertainty)

This is the [Handling unclear audio with `[unclear]`](/pre-recorded-audio/universal-3-5-pro/prompting#handling-unclear-audio-with-unclear) prompt. Run it alongside the evaluation prompt on the same audio and diff the outputs to find where the model is guessing:

```text
Always: Transcribe speech exactly as heard. If uncertain or audio is
unclear, mark as [unclear]. After the first output, review the transcript
again. Pay close attention to hallucinations, misspellings, or errors,
and revise them like a computer performing spell and grammar checks.
Ensure words and phrases make grammatical sense in sentences.
```

By comparing the two outputs, you can identify exactly which segments the model is least confident about. This is useful for:

- Evaluating how the model handles unclear or noisy audio
- Finding segments where the model’s guesses may be incorrect
- Prioritizing which audio segments to manually review
- Understanding whether WER differences are coming from genuine errors or uncertain segments

### What works and what doesn’t

The authoritative list lives in the prompting guide — see [What works / what to avoid](/pre-recorded-audio/universal-3-5-pro/prompting#what-works--what-to-avoid). The same rules apply when building evaluation prompts: lead with `Transcribe…`, use authoritative language (`Required:`, `Mandatory:`, `Always:`), describe the _pattern_ to watch for, and add instructions one at a time.

  Listing specific word examples in your prompt causes hallucinations. The model
  becomes over-eager to insert those exact words into the transcript, even when
  they weren’t spoken. For example, `Pharmaceutical accuracy required
  (omeprazole over omeprizole, metformin over metforman)` will cause the model
  to hallucinate those drug names. Instead, describe the _pattern_ of entities
  to prioritize: `Pharmaceutical accuracy required across all medications and
  drug names`. If you already know the specific terms, use [keyterms
  prompting](/pre-recorded-audio/universal-3-5-pro/prompting#keyterms-prompting) instead — it’s
  optimized for term boosting and more reliable than describing terms in a
  free-form prompt. See the [prompting
  guide](/pre-recorded-audio/universal-3-5-pro/prompting#entity-accuracy-and-spelling) for
  more details.

## Open-source tools

### aai-cli

[aai-cli](https://github.com/alexkroman/aai-cli) is a command-line tool for evaluating and optimizing transcription prompts. It supports:

- **Prompt evaluation**: Score a prompt against datasets from Hugging Face or your own audio files using WER and LASER metrics
- **Prompt optimization**: Automatically iterate on prompts using DSPy GEPA with LASER feedback, where an LLM reflects on transcription errors and proposes improved prompts
- **Dataset discovery**: Search and load audio datasets from Hugging Face for benchmarking

```bash
# Evaluate a prompt
aai eval --prompt "Transcribe verbatim." --max-samples 50

# Optimize a prompt
aai optimize --starting-prompt "Transcribe verbatim." --iterations 5 --samples 50
```

### prompt-seeker

[prompt-seeker](https://github.com/AssemblyAI-Solutions/prompt-seeker) uses Bayesian optimization (Optuna TPE) to systematically find the best transcription prompt by testing component combinations across diverse audio datasets and scoring with Semantic WER. It supports:

- **Component-based optimization**: Modular prompt pieces (language, disfluency, punctuation, etc.) are tested in combinations
- **Meta-optimization**: An LLM designs new component spaces between optimization rounds based on accumulated findings
- **Per-dataset analysis**: Breakdown of what works for each audio type in your evaluation set

```bash
# Run optimization (50 trials across your data)
uv run python -m prompt_seeker.cli optimize \
  --datasets "my_calls:50" \
  --trials 50 -c 20

# Run the meta-optimizer (Claude designs rounds autonomously)
uv run python -m prompt_seeker.cli meta-optimize \
  --datasets "my_calls:50" \
  --rounds 3 --trials 50 -c 10
```

Both tools require ground truth transcriptions for scoring. If you don’t have ground truth yet, transcribe a sample of your audio manually and use that as your starting point.

## Conclusion

Evaluating Universal-3.5 Pro requires going beyond traditional WER. The model’s contextual awareness and prompting capabilities mean that evaluation is as much about finding the right prompt as it is about measuring accuracy. Use Semantic WER or LASER alongside traditional WER, audit your ground truth data carefully, and iterate on prompts systematically to find the best configuration for your audio.

---

<a id="cloud-endpoints-and-data-residency-docs-pre-recorded-audio-select-the-region-mdx"></a>

# Cloud Endpoints and Data Residency

URL: https://www.assemblyai.com/docs/pre-recorded-audio/select-the-region
Source: docs/pre-recorded-audio/select-the-region.mdx
Navigation: Pre-recorded STT > Advanced
Description: Cloud Endpoints and Data Residency documentation.

Choose the endpoint that best fits your application's requirements—whether that's using the default US region or ensuring your audio data stays within the European Union.

## Default Endpoint

The default endpoint (`api.assemblyai.com`) processes your pre-recorded audio transcription requests in the US region. If you don't specify a base URL, this is the endpoint used for the request.

## EU Data Residency

The EU endpoint (`api.eu.assemblyai.com`) guarantees your data never leaves the European Union. This is designed for organizations with strict data residency and governance requirements—your audio and transcription data will remain entirely within the EU.

## Endpoints

| Endpoint     | Base URL                | Description          |
| ------------ | ----------------------- | -------------------- |
| US (default) | `api.assemblyai.com`    | Data stays in the US |
| EU           | `api.eu.assemblyai.com` | Data stays in the EU |

## Which endpoint should I use?

- **No data residency requirements?** Use the **default endpoint**. No configuration change is needed.
- **Need EU data residency?** Use the **EU endpoint** to ensure your audio and transcription data stays within the European Union.

## How to use it

Update your base URL to your preferred endpoint. Select an endpoint tab below to see examples for each.

The US endpoint is the default. If you're using the SDKs without overriding the base URL, you're already using this endpoint. No configuration change is required.

```python title="Python SDK" 
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

# No base_url override needed — US is the default
# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>", config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" highlight={4}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK"  expandable
import { AssemblyAI } from "assemblyai";

// No baseUrl override needed — US is the default
const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro", "universal-2"],
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={4}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  speech_models: ["universal-3-5-pro", "universal-2"],
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

Use the EU endpoint to keep your audio and transcription data within the European Union.

```python title="Python SDK" highlight={7} 
from assemblyai import Client, Settings
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

client = Client(
    settings=Settings(
        api_key="<YOUR_API_KEY>",
        base_url="https://api.eu.assemblyai.com",
    )
)

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
)

transcriber = Transcriber(client=client, config=config)
transcript = transcriber.transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" highlight={4}  expandable
import requests
import time

base_url = "https://api.eu.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = f"https://api.eu.assemblyai.com/v2/transcript/{transcript_id}"

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" highlight={5}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
  baseUrl: "https://api.eu.assemblyai.com",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro", "universal-2"],
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={4}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.eu.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  speech_models: ["universal-3-5-pro", "universal-2"],
  language_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

---

<a id="rate-limits-docs-pre-recorded-audio-rate-limits-mdx"></a>

# Rate limits

URL: https://www.assemblyai.com/docs/pre-recorded-audio/rate-limits
Source: docs/pre-recorded-audio/rate-limits.mdx
Navigation: Pre-recorded STT > Advanced
Description: Rate limits documentation.

When you submit audio files to the `/v2/transcript` endpoint, AssemblyAI processes them in parallel up to your account's rate limit. The rate limit is the maximum number of transcription jobs that can be actively processing at the same time.

## Default limits

| Account type | Rate limit (parallel transcriptions) |
| ------------ | ------------------------------------ |
| Free         | 5                                    |
| Paid         | 200+                                 |

  **Need a higher rate limit?**

Our services are infinitely scalable and we offer custom rate limits that scale to support any workload at no additional cost. If you need a higher rate limit, please either [contact our Sales team](https://www.assemblyai.com/contact) or send an email to [support@assemblyai.com](mailto:support@assemblyai.com).

## What happens when you hit the limit

If you submit a transcription that would exceed your rate limit, it is added to a queue. Queued transcriptions are processed automatically in FIFO (first-in, first-out) order as previously submitted transcriptions complete.

If you exceed your rate limit, you will receive an email stating that your transcripts have been throttled. You will only receive this email once per day.

![Email notifying user that their transcripts have been throttled due to exceeding the rate limit](/assets/pages/guides/limit.png)

If your account balance goes below zero, your rate limit will be reduced to 1.

## HTTP rate limits

In addition to the parallel-job rate limit, there is an HTTP rate limit for the API that restricts accounts to a maximum of 20,000 requests per five minutes. This limit counts all requests across endpoints — submissions (POST) and polling (GET) combined — and exceeding it returns an HTTP `403`. To stay under it, use [webhooks](/pre-recorded-audio/webhooks) instead of frequent polling, or widen and jitter your polling interval. See [Polling without exceeding the rate limit](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale#polling-without-exceeding-the-rate-limit) for a worked example.

## Check your limit

You can view your current rate limit on the [Rate Limits page](https://www.assemblyai.com/dashboard/home) of your dashboard.

With the current version of multi-project support, rate limiting is applied at the account level, not at the project level. This means that the rate limits for each API key mirror the rate limits for the account.

For example, if an account has a rate limit of 200, each API key for that account will be able to process up to 200 requests in parallel.

## Related pages

- [Real-time STT rate limits](/streaming/rate-limits)
- [Account Management](/account-management)
- [Transcript status](/pre-recorded-audio/check-transcript-status)

---

<a id="migration-guide-deepgram-to-assemblyai-docs-pre-recorded-audio-migration-guides-dg-to-aai-mdx"></a>

# Migration guide: Deepgram to AssemblyAI

URL: https://www.assemblyai.com/docs/pre-recorded-audio/migration-guides/dg_to_aai
Source: docs/pre-recorded-audio/migration-guides/dg_to_aai.mdx
Navigation: Pre-recorded STT > Guides > Migration guides
Description: Migration guide: Deepgram to AssemblyAI documentation.

This guide walks through the process of migrating from Deepgram to AssemblyAI for transcribing pre-recorded audio.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Side-By-Side Code Comparison

Below is a side-by-side comparison of a basic snippet to transcribe a **local file** by Deepgram and AssemblyAI:

```python expandable
from deepgram import (
     DeepgramClient,
     PrerecordedOptions,
     FileSource,
)

API_KEY = "YOUR_DG_API_KEY"

AUDIO_FILE = "./example.wav"

def main():
    try:
        deepgram = DeepgramClient(API_KEY)

        with open(AUDIO_FILE, "rb") as file:
            buffer_data = file.read()

            payload: FileSource = {
                "buffer": buffer_data,
            }

            options = PrerecordedOptions(
                model="nova-2",
                smart_format=True,
                diarize=True
            )

            response = deepgram.listen.prerecorded.v("1").transcribe_file(payload, options)

            print(response.to_json(indent=4))

    except Exception as e:
        print(f"Exception: {e}")

if name == "main":
    main()
```

```python expandable
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()

audio_file = "./example.wav"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,
)
transcript = transcriber.transcribe(audio_file, config)

if transcript.status == aai.TranscriptStatus.error:
    print(f"Transcription failed: {transcript.error}")
    exit(1)

print(transcript.text)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")
```

Below is a side-by-side comparison of a basic snippet to transcribe a **publicly-accessible URL** by Deepgram and AssemblyAI:

```python expandable
from deepgram import (
    DeepgramClient,
    PrerecordedOptions
)

API_KEY = "YOUR_DG_API_KEY"

AUDIO_URL = {
    "url": "https://dpgr.am/spacewalk.wav"
}

def main():
    try:
        deepgram = DeepgramClient(API_KEY)

        options = PrerecordedOptions(
            model="nova-2",
            smart_format=True,
            diarize=True
        )

        response = deepgram.listen.prerecorded.v("1").transcribe_url(AUDIO_URL, options)

        print(response.to_json(indent=4))

    except Exception as e:
        print(f"Exception: {e}")

if name == "main":
    main()
```

```python expandable
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()

audio_file = (
    "https://assembly.ai/sports_injuries.mp3"
)

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,
)
transcript = transcriber.transcribe(audio_file, config)

if transcript.status == aai.TranscriptStatus.error:
    print(f"Transcription failed: {transcript.error}")
    exit(1)

print(transcript.text)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")
```

Here are helpful things to know about our `transcribe` method:

- The SDK handles polling under the hood
- Transcript is directly accessible via `transcript.text`
- English is the default language. We recommend specifying `speech_models=["universal-3-5-pro", "universal-2"]` for the highest accuracy
- We have a [cookbook for error handling common errors](/pre-recorded-audio/guides/common_errors_and_solutions) when using our API.

## Installation

```python
from deepgram import (
    DeepgramClient,
    PrerecordedOptions,
    FileSource,
)

API_KEY = "YOUR_DG_API_KEY"
deepgram = DeepgramClient(API_KEY)
```

```python
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()
```

When migrating from Deepgram to AssemblyAI, you'll first need to handle authentication and SDK setup:

Get your API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home) \
To follow this guide, install AssemblyAI's Python SDK by typing this code into your terminal: \
 `pip install assemblyai`

Things to know:

- Store your API key securely in an environment variable
- API key authentication works the same across all AssemblyAI SDKs

## Audio File Sources

```python expandable
# Local Files
AUDIO_FILE = "example.wav"
with open(AUDIO_FILE, "rb") as file:
    buffer_data = file.read()

payload: FileSource = {
    "buffer": buffer_data,
}

options = PrerecordedOptions(
    smart_format=True,
    summarize="v2",
)

file_response = deepgram.listen.rest.v("1").transcribe_file(payload, options)

json = file_response.to_json()

#Public URLs
AUDIO_URL = {
    "url": "https://static.deepgram.com/examples/Bueller-Life-moves-pretty-fast.wav"
}

options = PrerecordedOptions(
    smart_format=True,
    summarize="v2"
)

url_response = deepgram.listen.rest.v("1").transcribe_url(AUDIO_URL, options)

json = url_response.to_json()
```

```python
transcriber = aai.Transcriber()

# Local Files
transcript = transcriber.transcribe("./audio.mp3")

# Public URLs
transcript = transcriber.transcribe("https://example.com/audio.mp3")

# S3 files (using pre-signed URLs)
s3_client = boto3.client('s3')
presigned_url = s3_client.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'audio.mp3'},
    ExpiresIn=3600
)

transcript = transcriber.transcribe(presigned_url)
```

Here are helpful things to know when migrating your audio input handling:

- There's no need to specify the audio format to AssemblyAI - it's auto-detected. AssemblyAI accepts almost every audio/video file type: [here is a full list of all our supported file types](/faq/what-audio-and-video-file-types-are-supported-by-your-api)
- Our SDK handles file upload and transcription automatically in one step
- For S3 files, you'll need to generate pre-signed URLs ([see example in cookbook](/pre-recorded-audio/guides/transcribe_from_s3))

## Adding Features

```python
options = PrerecordedOptions(
   model="nova-2",
   smart_format=True,
   diarize=True,
   detect_entities=True
)

response = deepgram.listen.prerecorded.v("1").transcribe_url(AUDIO_URL, options)

```

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True, # Speaker diarization
    auto_chapters=True, # Auto chapter detection
    entity_detection=True, # Named entity detection
)

transcript = transcriber.transcribe(audio_url, config)

# Access speaker labels
for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")
```

Key differences:

- Use `aai.TranscriptionConfig` to specify any extra features that you wish to use
- The results for Speaker Diarization are stored in `transcript.utterances`. To see the full transcript response object, refer to our [API Reference](/api-reference).
- Check our [documentation](/speech-understanding/getting-started) for our full list of available features and their parameters

---

<a id="migration-guide-openai-to-assemblyai-docs-pre-recorded-audio-migration-guides-oai-to-aai-mdx"></a>

# Migration guide: OpenAI to AssemblyAI

URL: https://www.assemblyai.com/docs/pre-recorded-audio/migration-guides/oai_to_aai
Source: docs/pre-recorded-audio/migration-guides/oai_to_aai.mdx
Navigation: Pre-recorded STT > Guides > Migration guides
Description: Migration guide: OpenAI to AssemblyAI documentation.

This guide walks through the process of migrating from OpenAI to AssemblyAI for transcribing pre-recorded audio.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Side-By-Side Code Comparison

Below is a side-by-side comparison of a basic snippet to transcribe a **local file** by OpenAI and AssemblyAI:

```python
from openai import OpenAI

api_key = "YOUR_OPENAI_API_KEY"
client = OpenAI(api_key)

audio_file = open("./example.wav", "rb")

transcript = client.audio.transcriptions.create(
    model = "whisper-1",
    file = audio_file
)

print(transcript.text)
```

```python
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()

audio_file = "./example.wav"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
)
transcript = transcriber.transcribe(audio_file, config)

if transcript.status == aai.TranscriptStatus.error:
    print(f"Transcription failed: {transcript.error}")
    exit(1)

print(transcript.text)
```

Here are helpful things to know about our `transcribe` method:

- The SDK handles polling under the hood
- Transcript is directly accessible via `transcript.text`
- English is the default language. We recommend specifying `speech_models=["universal-3-5-pro", "universal-2"]` for the highest accuracy
- We have a [cookbook for error handling common errors](/pre-recorded-audio/guides/common_errors_and_solutions) when using our API.

## Installation

```python
from openai import OpenAI

api_key = "YOUR_OPENAI_API_KEY"
client = OpenAI(api_key)
```

```python
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()
```

When migrating from OpenAI to AssemblyAI, you'll first need to handle authentication and SDK setup:

Get your API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home) \
To follow this guide, install AssemblyAI's Python SDK by typing this code into your terminal: \
 `pip install assemblyai`

Things to know:

- Store your API key securely in an environment variable
- API key authentication works the same across all AssemblyAI SDKs

## Audio File Sources

```python
client = OpenAI()

# Local Files

audio_file = open("./example.wav", "rb")
transcript = client.audio.transcriptions.create(
    model = "whisper-1",
    file = audio_file
)

```

```python
transcriber = aai.Transcriber()

# Local Files
transcript = transcriber.transcribe("./audio.mp3")

# Public URLs
transcript = transcriber.transcribe("https://example.com/audio.mp3")
```

Here are helpful things to know when migrating your audio input handling:

- AssemblyAI natively supports transcribing publicly accessible audio URLs (for example, S3 URLs), the Whisper API only natively supports transcribing local files.
- There's no need to specify the audio format to AssemblyAI - it's auto-detected. AssemblyAI accepts almost every audio/video file type: [here is a full list of all our supported file types](/faq/what-audio-and-video-file-types-are-supported-by-your-api)
- The Whisper API only supports file sizes up to 25MB, AssemblyAI supports file sizes up to 5GB.

## Adding Features

```python
transcript = client.audio.transcriptions.create(
    file = audio_file,
    prompt = "INSERT_PROMPT", # Optional text to guide the model's style
    language = "en", # Set language code
    model = "whisper-1",
    response_format = "verbose_json",
    timestamp_granularities = ["word"]
)

# Access word-level timestamps

print(transcript.words)
```

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels = True, # Speaker diarization
    sentiment_analysis=True, # Sentiment Analysis
    entity_detection = True, # Named entity detection
)

transcript = transcriber.transcribe(audio_url, config)

# Access word-level timestamps
print(transcript.words)

# Access speaker labels
for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")
```

Key differences:

- OpenAI does not offer speech understanding features for their speech-to-text API
- Use `aai.TranscriptionConfig` to specify any extra features that you wish to use
- With AssemblyAI, timestamp granularity is word-level by default
- The results for Speaker Diarization are stored in `transcript.utterances`. To see the full transcript response object, refer to our [API Reference](/api-reference).
- Check our [documentation](/speech-understanding/getting-started) for our full list of available features and their parameters
- If you want to send a custom prompt to an LLM, you can use [LLM Gateway](/llm-gateway/quickstart) to apply the model to your transcribed audio files.

---

<a id="migration-guide-aws-transcribe-to-assemblyai-docs-pre-recorded-audio-migration-guides-aws-to-aai-mdx"></a>

# Migration guide: AWS Transcribe to AssemblyAI

URL: https://www.assemblyai.com/docs/pre-recorded-audio/migration-guides/aws_to_aai
Source: docs/pre-recorded-audio/migration-guides/aws_to_aai.mdx
Navigation: Pre-recorded STT > Guides > Migration guides
Description: Migration guide: AWS Transcribe to AssemblyAI documentation.

This guide walks through the process of migrating from AWS Transcribe to AssemblyAI for transcribing pre-recorded audio.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Side-by-side code comparison

Below is a side-by-side comparison of a basic snippet to transcribe a file by AWS Transcribe and AssemblyAI:

```python expandable
import time
import boto3

def transcribe_file(job_name, file_uri, transcribe_client):
    transcribe_client.start_transcription_job(
        TranscriptionJobName=job_name,
        Media={"MediaFileUri": file_uri},
        MediaFormat="wav",
        LanguageCode="en-US",
    )

    max_tries = 60
    while max_tries > 0:
        max_tries -= 1

        job = transcribe_client.get_transcription_job(
            TranscriptionJobName=job_name
        )

        job_status = job["TranscriptionJob"]["TranscriptionJobStatus"]

        if job_status in ["COMPLETED", "FAILED"]:
            print(f"Job {job_name} is {job_status}.")

        if job_status == "COMPLETED":
            print(
                f"Download the transcript from\n"
                f"\t{job['TranscriptionJob']['Transcript']['TranscriptFileUri']}."
            )
            break
        else:
            print(f"Waiting for {job_name}. Current status is {job_status}.")
            time.sleep(10)

def main():
    transcribe_client = boto3.client("transcribe")
    file_uri = "s3://test-transcribe/answer2.wav"
    transcribe_file("Example-job", file_uri, transcribe_client)

if name == "main":
    main()
```

```python expandable
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()

# You can use a local filepath:
# audio_file = "./example.mp3"
# Or use a publicly-accessible URL:
audio_file = (
    "https://assembly.ai/sports_injuries.mp3"
)

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,
)
transcript = transcriber.transcribe(audio_file, config)

if transcript.status == aai.TranscriptStatus.error:
    print(f"Transcription failed: {transcript.error}")
    exit(1)

print(transcript.text)
for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")
```

## Installation

```python
import boto3
import time

transcribe_client = boto3.client("transcribe")
```

```python
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()
```

When migrating from AWS to AssemblyAI, you'll first need to handle authentication and SDK setup:

Get your API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home)

Things to know:

- Store your API key securely in an environment variable
- API key authentication works the same across all AssemblyAI SDKs

## Audio File Sources

```python
def transcribe_file(job_name, file_uri, transcribe_client):
    transcribe_client.start_transcription_job(
        TranscriptionJobName=job_name,
        Media={"MediaFileUri": file_uri},
        MediaFormat="wav",
        LanguageCode="en-US",
    )
```

```python
transcriber = aai.Transcriber()

# Local files

transcript = transcriber.transcribe("./audio.mp3")

# Public URLs

transcript = transcriber.transcribe("https://example.com/audio.mp3")

# S3 files (using pre-signed URLs)

s3_client = boto3.client('s3')
presigned_url = s3_client.generate_presigned_url(
'get_object',
Params={'Bucket': 'my-bucket', 'Key': 'audio.mp3'},
ExpiresIn=3600
)
transcript = transcriber.transcribe(presigned_url)
```

Here are helpful things to know when migrating your audio input handling:

- There's no need to specify the audio format to AssemblyAI - it's auto-detected. AssemblyAI accepts almost every audio/video file type: [here is a full list of all our supported file types](/faq/what-audio-and-video-file-types-are-supported-by-your-api)
- Our SDK handles file upload and transcription automatically in one step
- For S3 files, you'll need to generate pre-signed URLs ([see example in cookbook](/pre-recorded-audio/guides/transcribe_from_s3))

## Basic Transcription

```python
while max_tries > 0:
    max_tries -= 1
    job = transcribe_client.get_transcription_job(
        TranscriptionJobName=job_name
    )
    job_status = job["TranscriptionJob"]["TranscriptionJobStatus"]
    if job_status in ["COMPLETED", "FAILED"]:
        break
    time.sleep(10)
```

```python
transcriber = aai.Transcriber()

# Local files

transcript = transcriber.transcribe("./audio.mp3")

# Public URLs

transcript = transcriber.transcribe("https://example.com/audio.mp3")

# S3 files (using pre-signed URLs)

s3_client = boto3.client('s3')
presigned_url = s3_client.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'audio.mp3'},
    ExpiresIn=3600
)
transcript = transcriber.transcribe(presigned_url)
```

Here are helpful things to know about our `transcribe` method:

- The SDK handles polling under the hood
- Transcript is directly accessible via `transcript.text`
- English is the default language. We recommend specifying `speech_models=["universal-3-5-pro", "universal-2"]` for the highest accuracy
- We have a [cookbook for error handling common errors](/pre-recorded-audio/guides/common_errors_and_solutions) when using our API.

## Adding Features

```python
transcribe_client.start_transcription_job(
    TranscriptionJobName=job_name,
    Media={"MediaFileUri": file_uri},
    Settings={
        "ShowSpeakerLabels": True,
        "MaxSpeakerLabels": 2
    }
)
```

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,           # Speaker diarization
    auto_chapters=True,           # Auto chapter detection
    entity_detection=True,        # Named entity detection
)

transcript = transcriber.transcribe(audio_file, config)

# Access speaker labels

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

```

Key differences:

- Use `aai.TranscriptionConfig` to specify any extra features that you wish to use
- The results for Speaker Diarization are stored in `transcript.utterances`. To see the full transcript response object, refer to our [API Reference](/api-reference).
- Check our [documentation](/speech-understanding/getting-started) for our full list of available features and their parameters

---

<a id="migration-guide-google-speech-to-text-to-assemblyai-docs-pre-recorded-audio-migration-guides-google-to-aai-mdx"></a>

# Migration guide: Google Speech-to-Text to AssemblyAI

URL: https://www.assemblyai.com/docs/pre-recorded-audio/migration-guides/google_to_aai
Source: docs/pre-recorded-audio/migration-guides/google_to_aai.mdx
Navigation: Pre-recorded STT > Guides > Migration guides
Description: Migration guide: Google Speech-to-Text to AssemblyAI documentation.

This guide walks through the process of migrating from Google Speech-to-Text (STT) to AssemblyAI for transcribing pre-recorded audio.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Side-by-side code comparison

Below is a side-by-side comparison of a basic snippet to transcribe a file by Google Speech-to-Text and AssemblyAI.

```python expandable
from google.cloud import speech

client = speech.SpeechClient()

audio = speech.RecognitionAudio(
    uri="gs://cloud-samples-tests/speech/Google_Gnome.wav"
)

config = speech.RecognitionConfig(
    encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
    sample_rate_hertz=16000,
    language_code="en-US",
    model="video", # Chosen model
)

operation = client.long_running_recognize(config=config, audio=audio)

print("Waiting for operation to complete...")
response = operation.result(timeout=90)

for i, result in enumerate(response.results):
    alternative = result.alternatives[0]
    print("-" * 20)
    print(f"First alternative of result {i}")
    print(f"Transcript: {alternative.transcript}")
```

```python expandable
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()

# You can use a local filepath:
# audio_file = "./example.mp3"
# Or use a publicly-accessible URL:
audio_file = (
    "https://assembly.ai/sports_injuries.mp3"
)

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
)
transcript = transcriber.transcribe(audio_file, config)

if transcript.status == aai.TranscriptStatus.error:
    print(f"Transcription failed: {transcript.error}")
    exit(1)

print(transcript.text)
```

## Installation

```python
from google.cloud import speech

client = speech.SpeechClient()
```

```python
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
transcriber = aai.Transcriber()
```

When migrating from Google Speech-to-Text to AssemblyAI, you'll first need to handle authentication and SDK setup:

Get your API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).
Things to know:

- Store your API key securely in an environment variable
- API key authentication works the same across all AssemblyAI SDKs

## Audio File Sources

```python
audio = speech.RecognitionAudio(uri="gs://cloud-samples-tests/speech/Google_Gnome.wav")

config = speech.RecognitionConfig(
    encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
    sample_rate_hertz=16000,
    language_code="en-US",
    model="video", # Chosen model
)

operation = client.long_running_recognize(config=config, audio=audio)
```

```python
transcriber = aai.Transcriber()

# Local files
transcript = transcriber.transcribe("./audio.mp3")

# Public URLs
transcript = transcriber.transcribe("https://example.com/audio.mp3")

# S3 files (using pre-signed URLs)
s3_client = boto3.client('s3')
presigned_url = s3_client.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'audio.mp3'},
    ExpiresIn=3600
)
transcript = transcriber.transcribe(presigned_url)
```

Here are helpful things to know when migrating your audio input handling:

- There's no need to specify the audio encoding format when using AssemblyAI - we have a transcoding pipeline under the hood which works on all [supported file types](/faq/what-audio-and-video-file-types-are-supported-by-your-api) so that you can get the most accurate transcription.
- You can submit a local file, URL, stream, buffer, blob, etc., directly to our transcriber. Check out some common ways you can host audio files [here](/pre-recorded-audio/guides/transcribe_from_s3).
- You can transcribe audio files that are up to 10 hours long and you can transcribe multiple files in parallel. The default amount of jobs you can transcribe at once is 200 while on the PAYG plan.

## Basic Transcription

```python
print("Waiting for operation to complete...")
response = operation.result(timeout=90)

for i, result in enumerate(response.results):
    alternative = result.alternatives[0]
    print("-" * 20)
    print(f"First alternative of result {i}")
    print(f"Transcript: {alternative.transcript}")
```

```python
transcriber = aai.Transcriber()

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
)
transcript = transcriber.transcribe(audio_file, config)

if transcript.status == aai.TranscriptStatus.error:
    print(f"Transcription failed: {transcript.error}")
else:
    print(transcript.text)
```

Here are helpful things to know about our `transcribe` method:

- The SDK handles polling under the hood.
- The full transcript is directly accessible via `transcript.text`.
- English is the default language. We recommend specifying `speech_models=["universal-3-5-pro", "universal-2"]` for the highest accuracy.
- We have a [cookbook for error handling common errors](/pre-recorded-audio/guides/common_errors_and_solutions) when using our API.

## Adding Features

```python
config = speech.RecognitionConfig(
    encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
    sample_rate_hertz=8000,
    language_code="en-US",
    enable_speaker_diarization=True,  # Speaker diarization
    diarization_speaker_count=2,  # Specify amount of speakers
    profanity_filter=True   # Remove profanity from transcript
)
```

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,  # Speaker diarization
    filter_profanity=True,  # Remove profanity from transcript
    speakers_expected=2,  # Specify amount of speakers in audio
)

transcript = transcriber.transcribe(audio_file, config)

# Access speaker labels

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")
```

Key differences:

- Use `aai.TranscriptionConfig` to specify any extra features that you wish to use.
- The results for Speaker Diarization are stored in `transcript.utterances`. To see the full transcript response object, refer to our [API Reference](/api-reference).
- Check our [documentation](/speech-understanding/getting-started) for our full list of available features and their parameters.

---

<a id="migration-guide-gladia-to-assemblyai-docs-pre-recorded-audio-migration-guides-gladia-to-aai-mdx"></a>

# Migration guide: Gladia to AssemblyAI

URL: https://www.assemblyai.com/docs/pre-recorded-audio/migration-guides/gladia_to_aai
Source: docs/pre-recorded-audio/migration-guides/gladia_to_aai.mdx
Navigation: Pre-recorded STT > Guides > Migration guides
Description: Migration guide: Gladia to AssemblyAI documentation.

This guide walks through the process of migrating from Gladia to AssemblyAI for transcribing pre-recorded audio.

## Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home).
If you'd prefer to use one of our official SDKs, check our [documentation for the full list of available SDKs](/pre-recorded-audio/guides/do-more-with-sdk).

  The [Gladia
  documentation](https://docs.gladia.io/chapters/pre-recorded-stt/getting-started)
  uses cURL commands to demonstrate API usage. In this guide, we will use Python
  code snippets to illustrate the same functionality across both APIs. If you
  prefer to use cURL, you can find the equivalent commands in the [AssemblyAI
  API Reference](/api-reference/overview).

## Side-by-side code comparison

Below is a side-by-side comparison of a basic snippet to transcribe pre-recorded audio with Gladia and AssemblyAI:

```python expandable
import requests
import time

base_url = "https://api.gladia.io"

headers = {
    "x-gladia-key": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
    files = {"audio": ("my-audio.mp3", f, "audio/mp3")}
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            files=files)

upload_url = response.json()["audio_url"]

data = {
    "audio_url": upload_url # You can also use a URL to an audio or video file on the web.
}

url = base_url + "/v2/pre-recorded"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id'] # You can also use response.json()['result_url'] to get the polling_endpoint directly.
polling_endpoint = url + "/" + transcript_id

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript['status'] == 'done':
        print(f"Full Transcript: {transcript['result']['transcription']['full_transcript']}")
        break

    elif transcript['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcript['error_code']}")

    else:
        time.sleep(3)
```

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = url + "/" + transcript_id

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript['status'] == 'completed':
        print(f"Full Transcript: {transcript['text']}")
        break

    elif transcript['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    else:
        time.sleep(3)
```

## Installation and authentication

```python
import requests
import time

base_url = "https://api.gladia.io"

headers = {
    "x-gladia-key": "<YOUR_API_KEY>"
}
```

```python
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}
```

When migrating from Gladia to AssemblyAI, you'll first need to handle authentication:

Get your API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).

Things to know:

- Store your API key securely in an environment variable.
- We support the ability to create [multiple API keys and projects](/account-management) to help you track and manage seperate environments.

  Gladia uses the `x-gladia-key` HTTP header for authentication, while
  AssemblyAI uses the `authorization` header.

## Audio file sources

You can provide either a locally stored audio file or a publicly accessible URL.

```python
# Local Files
with open("./my-audio.mp3", "rb") as f:
    files = {"audio": ("my-audio.mp3", f, "audio/mp3")}
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            files=files)

upload_url = response.json()["audio_url"]

data = {
    "audio_url": upload_url
}

#Public URLs
audio_file = "https://assembly.ai/sports_injuries.mp3"

data = {
    "audio_url": audio_file
}
```

```python expandable
# Local Files
with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url,
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
}

# Public URLs
audio_file = "https://assembly.ai/sports_injuries.mp3"

data = {
    "audio_url": audio_file,
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
}
```

## Basic transcription and polling the transcription status

Make a `POST` request to the [/v2/pre-recorded](https://docs.gladia.io/docs/api-reference/v2/pre-recorded/init) endpoint.

```python
url = base_url + "/v2/pre-recorded"
response = requests.post(url, json=data, headers=headers)
```

Every few seconds, make a `GET` request to the [/v2/pre-recorded/:transcript_id](https://docs.gladia.io/docs/api-reference/v2/pre-recorded/get) endpoint until the transcription status is `'done'`.

```python
transcript_id = response.json()['id'] # You can also use response.json()['result_url'] to get the polling_endpoint directly.
polling_endpoint = url + "/" + transcript_id

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript['status'] == 'done':
        print(f"Full Transcript: {transcript['result']['transcription']['full_transcript']}")
        break

    elif transcript['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcript['error_code']}")

    else:
        time.sleep(3)
```

Make a `POST` request to the [/v2/transcript](/api-reference/transcripts/submit) endpoint.

```python
url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)
```

Every few seconds, make a `GET` request to the [/v2/transcript/:transcript_id](/api-reference/transcripts/get) endpoint until the transcription status is `'completed'`.

```python
transcript_id = response.json()['id']
polling_endpoint = url + "/" + transcript_id

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript['status'] == 'completed':
        print(f"Full Transcript: {transcript['text']}")
        break

    elif transcript['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    else:
        time.sleep(3)
```

  **Transcription status**

Note that our APIs possible values for transcription status are `queued`, `processing`, `completed`, and `error`. Check out the [AssemblyAI API Reference](/api-reference/transcripts/get#response.body.status) for the full list of possible transcription status values.
If you'd rather not poll the API, you can use our [SDKs](/pre-recorded-audio/guides/do-more-with-sdk) which handle polling internally. Alternatively, you can also use [webhooks](/pre-recorded-audio/webhooks) to get notified when your transcript is complete.

Here are helpful things to know when migrating your audio input handling:

- Both AssemblyAI and Gladia allow you the option of uploading a local file or specifying a publicly accessible URL.
- There's no need to specify the audio format to AssemblyAI - it's auto-detected. AssemblyAI accepts almost every audio/video file type: [here is a full list of all our supported file types](/faq/what-audio-and-video-file-types-are-supported-by-your-api)
- For self-hosted or pre-signed URLs (i.e. S3), [see our example in this cookbook](/pre-recorded-audio/guides/transcribe_from_s3).

## Adding features

```python
data = {
    "audio_url": upload_url,
    "diarization": True,                # Speaker diarization
    "chapterization": True,             # Auto chapter detection
    "named_entity_recognition": True    # Named entity detection
}

# Access speaker labels
for utterance in transcript['result']['transcription']['utterances']:
    print(f"Speaker {utterance['speaker']}: {utterance['text']}")

# Access auto chapters
for chapter in transcript['result']['chapterization']['results']:
    print(f"{chapter['start']} - {chapter['end']}: {chapter['headline']}")

# Access named entities
for entity in transcript['result']['named_entity_recognition']['results']:
    print(entity['text'])
    print(entity['entity_type'])
    print(f"Timestamp: {entity['start']} - {entity['end']}\n")
```

```python expandable
data = {
    "audio_url": upload_url,
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
    "speaker_labels": True,     # Speaker diarization
    "auto_chapters": True,      # Auto chapter detection
    "entity_detection": True    # Named entity detection
}

# Access speaker labels
for utterance in transcript['utterances']:
    print(f"Speaker {utterance['speaker']}: {utterance['text']}")

# Access auto chapters
for chapter in transcript['chapters']:
    print(f"{chapter['start']} - {chapter['end']}: {chapter['headline']}")

# Access named entities
for entity in transcript['entities']:
    print(entity['text'])
    print(entity['entity_type'])
    print(f"Timestamp: {entity['start']} - {entity['end']}\n")
```

Key differences:

- Make sure to note any differences in parameters or response structure. If using [Speaker Diarization](/pre-recorded-audio/label-speakers), for example:
  - Parameters: AssemblyAI uses `speaker_labels`, while Gladia uses `diarization`.
  - Response: AssemblyAI uses `transcript.utterances`, while Gladia uses `transcript.result.transcription.utterances`.
- Make sure to review each API reference for the full list of parameters and response objects.
  - [AssemblyAI API Reference](/api-reference/transcripts/submit)
  - [Gladia API Reference](https://docs.gladia.io/docs/api-reference/v2/pre-recorded/init)

---

<a id="best-practices-for-building-meeting-notetakers-docs-meeting-notetaker-best-practices-mdx"></a>

# Best Practices for building Meeting Notetakers

URL: https://www.assemblyai.com/docs/meeting-notetaker-best-practices
Source: docs/meeting-notetaker-best-practices.mdx
Navigation: Pre-recorded STT > Guides > Tutorials
Description: Complete guide for building meeting notetakers with AssemblyAI

## Introduction

Building a robust meeting notetaker requires careful consideration of accuracy, latency, speaker identification, and real-time capabilities. This guide addresses common questions and provides practical solutions for both post-call and live meeting transcription scenarios.

## Why AssemblyAI for Meeting Notetakers?

AssemblyAI stands out as the premier choice for meeting notetakers with several key advantages:

### Industry-Leading Accuracy with Pre-recorded Audio

- **93.3%+ transcription accuracy** ensures reliable meeting documentation
- **2.9% speaker diarization error rate** for precise "who said what" attribution
- **Speech Understanding** integration for intelligent post-processing and insights
- **Keyterms prompt** allows providing meeting context to improve accuracy of transcription

### Streaming with Universal-3.5 Pro

As meeting notetakers evolve toward real-time capabilities, AssemblyAI's Universal-3.5 Pro Streaming model (`universal-3-5-pro`) offers significant benefits:

- **Speaker diarization** available for both pre-recorded and streaming transcription
- **Ultra-low latency (~300ms)** enables live transcription without delays
- **Format turns** feature provides structured, readable output in real-time
- **Keyterms prompt** allows providing meeting context to improve accuracy of transcription

### End-to-End Voice AI Platform

Unlike fragmented solutions, AssemblyAI provides a unified API for:

- Transcription with speaker diarization
- Automatic language detection and code switching
- Boosting accuracy via meeting context with keyterms prompt
- Speech Understanding tasks like speaker identification, translation, and transcript styling
- Post-processing workflows with custom prompting - from summarization to completely custom workflows
- Real-time and batch processing of pre-recorded audio in a single platform

## When Should I Use Pre-recorded vs Streaming for Meeting Notetakers?

Understanding when to use pre-recorded versus streaming speech-to-text is critical for building the right meeting notetaker.

### Pre-recorded Speech-to-text

**Post-call analysis** - Meeting already happened, you have the full recording

- **Highest accuracy needed** - Pre-recorded models have higher accuracy (93.3%+)
- **Speaker diarization is critical** - Pre-recorded has 2.9% speaker error rate
- **Broad language support** - Need any of 99+ languages
- **Advanced features required** - Summarization, sentiment analysis, entity detection, PII redaction, speaker identification
- **Batch processing** - Processing multiple recordings at once
- **Quality over speed** - Can wait seconds/minutes for perfect results

**Best for:** Zoom/Teams/Meet recording uploads, compliance, documentation, post-call summaries, searchable archives

### Streaming Speech-to-text

**Live meetings** - Transcribing as the meeting happens

You should use streaming when you need to display a live transcript of text to users as they are speaking. With Universal-3.5 Pro Streaming, accuracy is closer to pre-recorded, but pre-recorded will always be the most accurate option.

- **Real-time captions** - Displaying subtitles/captions to participants during calls
- **Immediate feedback** - Need transcription within ~300ms
- **Interactive features** - Live note-taking, real-time keyword detection, action item alerts
- **No recording available** - Processing live audio only

**Best for:** Live captions, real-time note-taking apps, accessibility features, live keyword alerts

**Streaming is billed per session**

Streaming is billed on the total duration that your WebSocket connection stays open, not on the amount of audio you send. For long-running meetings, make sure to terminate sessions when the meeting ends to avoid being billed for idle time. See [Billing and pricing](/billing-and-pricing) for details.

### Hybrid Approach (Recommended)

Many successful meeting notetakers use **both** pre-recorded and streaming speech-to-text:

1. **Streaming during the call** - Provide live captions and real-time notes to participants
2. **Pre-recorded after the call** - Generate high-quality transcript with speaker labels, summary, and insights

This gives users immediate value during meetings while providing comprehensive documentation afterward.

**Example workflow:**

- User joins meeting → Start streaming for live captions
- Meeting ends → Upload recording to pre-recorded API for final transcript with speaker names
- Generate meeting summary, action items, and searchable archive from pre-recorded transcript

## What Languages and Features for a Meeting Notetaker?

### Pre-Recorded Meetings

For post-call analysis, AssemblyAI supports:

**Languages**:

- 99 languages supported
- Automatic Language Detection to route to the most spoken language
- Code Switching to preserve changes in speech between languages

**Core Features**:

- Speaker diarization (1-10 speakers by default, expandable to any min/max)
- Multichannel audio support (each channel = one speaker)
- Automatic formatting, punctuation, and capitalization
- Keyterms prompting for boosting domain-specific terms

**Speech Understanding Models**:

- Summarization for meeting recaps
- Sentiment analysis for meeting tone assessment
- Entity detection for extracting key information
- Speaker identification to map generic labels to actual names/roles
- Translation between 86 languages

### Real-Time Streaming

For live meeting transcription:

**Languages**:

- English-only model (default)
- Multilingual model supporting English, Spanish, French, German, Portuguese, and Italian

### Streaming (Universal-3.5 Pro Streaming)

- Speaker diarization for identifying who is speaking
- Partial and final transcripts for responsive UI
- Format turns for structured, readable output
- Keyterms prompt for contextual accuracy

See the [Universal-3.5 Pro Streaming documentation](/streaming/getting-started/transcribe-streaming-audio) for full details.

## How Can I Get Started Building a Post-Call Meeting Notetaker?

Here's a complete example implementing pre-recorded transcription with all essential features:

```python expandable
import assemblyai as aai
import asyncio
from typing import Dict, List
from assemblyai.types import (
    SpeakerOptions,
    LanguageDetectionOptions,
    PIIRedactionPolicy,
    PIISubstitutionPolicy,
)

# Configure API key
aai.settings.api_key = "your_api_key_here"

async def transcribe_meeting_async(audio_source: str) -> Dict:
    """
    Asynchronously transcribe a meeting recording with full features

    Args:
        audio_source: Either a local file path or publicly accessible URL
    """
    # Configure comprehensive meeting analysis
    config = aai.TranscriptionConfig(
        # Speaker diarization
        speaker_labels=True,
        speakers_expected=None,  # Use if you know exact number from Zoom/Meet/Teams
        speaker_options=SpeakerOptions(
            min_speakers_expected=2,
            max_speakers_expected=10  # Set a bit higher than expected; too high can cause over-splitting
        ),
        multichannel=False,  # Set to True if audio has separate channel per speaker

        # Language detection
        language_detection=True,  # Auto-detect the most used language
        language_detection_options=LanguageDetectionOptions(
            code_switching=True,  # Preserve language switches
            code_switching_confidence_threshold=0.5,
        ),

        # Punctuation and formatting
        punctuate=True,
        format_text=True,

        # Boost accuracy of meeting-specific vocabulary
        keyterms_prompt=["quarterly", "KPI", "roadmap", "deliverables"],

        # Speech Understanding - commonly used models
        summarization=True,
        sentiment_analysis=True,
        entity_detection=True,
        redact_pii=True,
        redact_pii_policies=[
            PIIRedactionPolicy.person_name,
            PIIRedactionPolicy.organization,
            PIIRedactionPolicy.occupation,
        ],
        redact_pii_sub=PIISubstitutionPolicy.hash,
        redact_pii_audio=True
    )

    # Create transcriber
    transcriber = aai.Transcriber()

    try:
        # Submit transcription job
        transcript = await asyncio.to_thread(
            transcriber.transcribe,
            audio_source,
            config=config
        )

        # Check status
        if transcript.status == aai.TranscriptStatus.error:
            raise Exception(f"Transcription failed: {transcript.error}")

        # Process speaker-labeled utterances
        print("\n=== SPEAKER-LABELED TRANSCRIPT ===\n")

        for utterance in transcript.utterances:
            # Format timestamp
            start_time = utterance.start / 1000  # Convert to seconds
            end_time = utterance.end / 1000

            # Print formatted utterance
            print(f"[{start_time:.1f}s - {end_time:.1f}s] Speaker {utterance.speaker}:")
            print(f"  {utterance.text}")
            print(f"  Confidence: {utterance.confidence:.2%}\n")

        # Print summary data
        print("\n=== MEETING SUMMARY ===\n")
        print({
            "id": transcript.id,
            "status": transcript.status,
            "duration": transcript.audio_duration,
            "speaker_count": len(set(u.speaker for u in transcript.utterances)),
            "word_count": len(transcript.words) if transcript.words else 0,
            "detected_language": transcript.language_code if hasattr(transcript, 'language_code') else None,
            "summary": transcript.summary,
        })

        return {
            "transcript": transcript,
            "utterances": transcript.utterances,
            "summary": transcript.summary,
        }

    except Exception as e:
        print(f"Error during transcription: {e}")
        raise

async def main():
    """
    Example usage with error handling
    """
    # Use either local file OR URL (not both)
    audio_source = "https://assembly.ai/wildfires.mp3"  # Or "path/to/recording.mp3"

    try:
        result = await transcribe_meeting_async(audio_source)

        # Additional processing
        print(f"\nTotal speakers identified: {len(set(u.speaker for u in result['utterances']))}")
        print(f"Meeting duration: {result['transcript'].audio_duration} seconds")

    except Exception as e:
        print(f"Failed to process meeting: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```

## How Can I Get Started Building a During-Call Live Meeting Notetaker?

Here's a complete example for real-time streaming transcription with meeting-optimized settings:

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "your_api_key"

# Keyterms to improve recognition accuracy
KEYTERMS = [
    "Alice Johnson",
    "Bob Smith",
    "Carol Davis",
    "quarterly review",
    "action items",
    "follow up",
    "deadline",
    "budget"
]

# MEETING NOTETAKER CONFIGURATION (different from voice agents!)
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,  # ALWAYS TRUE for meetings - users need readable text

    # Meeting-optimized turn detection (wait longer than voice agents)
    # universal-3-5-pro defaults: min_turn_silence=100ms, max_turn_silence=1000ms
    "min_turn_silence": 560,  # Wait longer for natural pauses (voice agents use ~100ms)
    "max_turn_silence": 2000,  # Allow thinking pauses

    # Keyterms for accuracy - pass each term as a separate query parameter
    "keyterms_prompt": KEYTERMS,
}

API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS, doseq=True)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
transcript_buffer = []

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("=" * 80)
    print(f"[{datetime.now().strftime('%H:%M:%S')}] Meeting transcription started")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")
    print(f"Keyterms configured: {', '.join(KEYTERMS)}")
    print("=" * 80)
    print("\nSpeak into your microphone. Press Ctrl+C to stop.\n")

    def stream_audio():
        """Stream audio from microphone to WebSocket"""
        global stream
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                if not stop_event.is_set():
                    print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    """Handle incoming messages from AssemblyAI"""
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        # Uncomment to see full JSON for debugging:
        # print("=" * 80)
        # print(json.dumps(data, indent=2, ensure_ascii=False))
        # print("=" * 80)
        # print()

        if msg_type == "Begin":
            session_id = data.get("id", "N/A")
            print(f"[SESSION] Started - ID: {session_id}\n")

        elif msg_type == "Turn":
            end_of_turn = data.get("end_of_turn", False)
            transcript = data.get("transcript", "")
            turn_order = data.get("turn_order", 0)
            end_of_turn_confidence = data.get("end_of_turn_confidence", 0.0)

            # FOR MEETING NOTETAKERS: Show partials for responsive UI
            if not end_of_turn and transcript:
                print(f"\r[LIVE] {transcript}", end="", flush=True)

            # FOR MEETING NOTETAKERS: Use formatted finals for readable display
            # (Unlike voice agents which should use utterance for speed)
            if end_of_turn and transcript:
                timestamp = datetime.now().strftime('%H:%M:%S')
                print(f"\n[{timestamp}] {transcript}")
                print(f"           Turn: {turn_order} | Confidence: {end_of_turn_confidence:.2%}")

                # Detect action items
                transcript_lower = transcript.lower()
                if any(term in transcript_lower for term in ["action item", "follow up", "deadline", "assigned to", "todo"]):
                    print("           ⚠️  ACTION ITEM DETECTED!")

                # Store final transcript
                transcript_buffer.append({
                    "timestamp": timestamp,
                    "text": transcript,
                    "turn_order": turn_order,
                    "confidence": end_of_turn_confidence,
                    "type": "final"
                })
                print()

        elif msg_type == "Termination":
            audio_duration = data.get("audio_duration_seconds", 0)
            print(f"\n[SESSION] Terminated - Duration: {audio_duration}s")
            save_transcript()

        elif msg_type == "Error":
            error_msg = data.get("error", "Unknown error")
            print(f"\n[ERROR] {error_msg}")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\n[WEBSOCKET ERROR] {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\n[WEBSOCKET] Disconnected - Status: {close_status_code}, Message: {close_msg}")

    global stream, audio
    stop_event.set()

    # Clean up audio stream
    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def save_transcript():
    """Save the transcript to a file"""
    if not transcript_buffer:
        print("No transcript to save.")
        return

    filename = f"meeting_transcript_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"

    with open(filename, "w", encoding="utf-8") as f:
        f.write("Meeting Transcript\n")
        f.write(f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
        f.write(f"Keyterms: {', '.join(KEYTERMS)}\n")
        f.write("=" * 80 + "\n\n")

        for entry in transcript_buffer:
            f.write(f"[{entry['timestamp']}] {entry['text']}\n")
            f.write(f"Confidence: {entry['confidence']:.2%}\n\n")

    print(f"Transcript saved to: {filename}")

def run():
    """Main function to run the streaming transcription"""
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\n\nCtrl+C received. Stopping transcription...")
        stop_event.set()

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

  These settings wait longer before ending turns to accommodate natural
  conversation pauses and ensure readable formatted text for display. You can
  [tweak these settings](/streaming/getting-started/transcribe-streaming-audio) to get
  the best results for your notetaker.

## How Do I Handle Multichannel Meeting Audio?

Many meeting platforms (Zoom, Teams, Google Meet) can record each participant on separate audio channels. This dramatically improves speaker identification accuracy.

### For Pre-recorded Meetings

```python
config = aai.TranscriptionConfig(
    multichannel=True,  # Enable when each speaker is on different channel
    speaker_labels=False,  # Disable - channels already separate speakers
    # Other settings...
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file, config=config)

# Access per-channel transcripts
for channel, channel_transcript in enumerate(transcript.channels):
    print(f"\n=== Channel {channel} ===")
    print(channel_transcript.text)
```

**When to use multichannel:**

- Zoom local recordings with "Record separate audio file for each participant" enabled
- Professional podcast recordings with individual microphones
- Conference systems with dedicated channels per participant
- Phone calls with caller and callee on separate channels

**Benefits:**

- **Perfect speaker separation** - No diarization errors
- **No speaker confusion or overlap issues**
- **Faster processing time** - Diarization not needed
- **Higher accuracy** - Model processes clean single-speaker audio

**How to enable in meeting platforms:**

- **Zoom**: Settings → Recording → Advanced → "Record a separate audio file for each participant"
- **Teams**: Requires third-party recording solutions like [Recall.ai](https://www.recall.ai/)
- **Google Meet**: Requires third-party recording solutions like [Recall.ai](https://www.recall.ai/)

### For Streaming Meetings

For real-time multichannel audio, create separate streaming sessions per channel:

```python expandable
import asyncio
import websockets

class ChannelTranscriber:
    def __init__(self, channel_id: int, speaker_name: str):
        self.channel_id = channel_id
        self.speaker_name = speaker_name
        self.connection_params = {
            "sample_rate": 16000,
            "speech_model": "universal-3-5-pro",
            "format_turns": True,
        }

    async def transcribe_channel(self, audio_stream):
        """Transcribe a single audio channel"""
        url = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(self.connection_params)}"

        # If you're using `websockets` version 13.0 or later, use `additional_headers` parameter. For older versions (< 13.0), use `extra_headers` instead.
        async with websockets.connect(url, additional_headers={"Authorization": API_KEY}) as ws:
            # Send audio from this channel only
            async for audio_chunk in audio_stream:
                await ws.send(audio_chunk)

            # Receive transcripts
            async for message in ws:
                data = json.loads(message)
                if data.get("type") == "Turn" and data.get("end_of_turn"):
                    print(f"{self.speaker_name}: {data['transcript']}")

# Create transcriber for each channel
async def transcribe_multichannel_meeting(channel_audio_streams):
    transcribers = [
        ChannelTranscriber(0, "Alice"),
        ChannelTranscriber(1, "Bob"),
    ]

    # Run all channels concurrently
    await asyncio.gather(*[
        t.transcribe_channel(stream)
        for t, stream in zip(transcribers, channel_audio_streams)
    ])
```

See our [multichannel streaming guide](/streaming/label-speakers-and-separate-channels#multichannel-streaming-audio) for complete implementation details.

## How Should I Handle Pre-recorded Transcription in Production?

Choose the right approach based on your application's needs:

### Option 1: Simple Blocking Call

```python
# Simple blocking call
transcript = await asyncio.to_thread(transcriber.transcribe, audio_url, config=config)
```

**Pros:**

- Simple, straightforward code
- Good for low volume applications
- Easy to understand and debug

**Cons:**

- Ties up resources while waiting
- Not suitable for high volume
- Cannot process multiple files simultaneously

**Best for:** Personal projects, prototypes, low-traffic applications

### Option 2: Webhook Callbacks (Production Recommended)

```python
config = aai.TranscriptionConfig(
    webhook_url="https://your-app.com/webhooks/assemblyai",
    webhook_auth_header_name="X-Webhook-Secret",
    webhook_auth_header_value="your_secret_here",
    speaker_labels=True,
    summarization=True,
    # ... other config
)

# Submit job and return immediately (non-blocking)
transcript = transcriber.submit(audio_url, config=config)
print(f"Job submitted: {transcript.id}")
# Your app can continue processing other requests

# Your webhook receives results when ready (typically 15-30% of audio duration)
```

**Webhook handler example:**

```python expandable
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhooks/assemblyai", methods=["POST"])
def assemblyai_webhook():
    # Verify webhook authenticity
    if request.headers.get("X-Webhook-Secret") != "your_secret_here":
        return jsonify({"error": "Unauthorized"}), 401

    import requests as http_requests

    data = request.json
    transcript_id = data["transcript_id"]
    status = data["status"]

    if status == "completed":
        # Fetch the full transcript (webhook only sends transcript_id and status)
        transcript = http_requests.get(
            f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
            headers={"authorization": "your_api_key"}
        ).json()
        process_completed_meeting(transcript)
    elif status == "error":
        log_transcription_error(transcript_id)

    return jsonify({"received": True}), 200

def process_completed_meeting(transcript):
    """Process completed meeting transcript"""
    utterances = transcript["utterances"]
    summary = transcript["summary"]

    # Store in database
    save_to_database(transcript)

    # Notify user
    send_notification(transcript["id"])
```

**Pros:**

- Non-blocking - submit and forget
- Scales to high volume
- Process multiple files in parallel
- Automatic retry on failures
- Get notified when complete

**Best for:** Production apps, user-uploaded recordings, batch processing, SaaS products

### Option 3: Polling (Custom Workflows)

```python
# Submit job
transcript = transcriber.submit(audio_url, config=config)
print(f"Submitted: {transcript.id}")

# Poll for completion with progress tracking
while transcript.status not in [aai.TranscriptStatus.completed, aai.TranscriptStatus.error]:
    await asyncio.sleep(5)
    transcript = transcriber.get_transcript(transcript.id)

    # Optional: Show progress
    print(f"Status: {transcript.status}...")

if transcript.status == aai.TranscriptStatus.completed:
    process_transcript(transcript)
else:
    print(f"Error: {transcript.error}")
```

**Pros:**

- Full control over retry logic
- Can show progress to users
- Good for background jobs
- Works without webhook infrastructure

**Cons:**

- Must implement your own polling loop
- Ties up resources while polling
- More complex than webhooks

**Best for:** Background job processors, CLIs with progress bars, custom retry logic

### Comparison Table

| Method      | Blocking | Scalability | Complexity | Best For                     |
| ----------- | -------- | ----------- | ---------- | ---------------------------- |
| Blocking    | Yes      | Low         | Low        | Prototypes, low volume       |
| Webhooks    | No       | High        | Medium     | Production, high volume      |
| Polling     | Partial  | Medium      | Medium     | Background jobs, progress UI |

### Scaling Considerations

- **HTTP rate limit:** 20,000 requests per 5-minute window, counted across submissions (POST) and polling (GET) combined
- **Exceeding the limit:** returns a `403` response
- **Parallel transcriptions (rate limit):** 200+ for paid accounts (queued beyond that)
- **Ramp up gradually:** start at 10-50 parallel requests, double incrementally
- **Avoid the rate limit:** use [webhooks](/pre-recorded-audio/webhooks) or jittered, widened polling — see [Polling without exceeding the rate limit](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale#polling-without-exceeding-the-rate-limit)
- **Contact Sales** before large-scale rollouts

## How Do I Identify Speakers in My Recording?

Speaker diarization tells you **when** speakers change ("Speaker A", "Speaker B"), but **Speaker Identification** tells you **who** they are by name or role.

### Why Use Speaker Identification?

**Instead of:**

```
Speaker A: Let's review the Q3 numbers.
Speaker B: Revenue was up 15% this quarter.
Speaker A: Excellent work on that launch.
```

**You get:**

```
Sarah Chen: Let's review the Q3 numbers.
Michael Rodriguez: Revenue was up 15% this quarter.
Sarah Chen: Excellent work on that launch.
```

### How It Works

Speaker Identification uses AssemblyAI's Speech Understanding API to map generic speaker labels to actual names or roles that you provide:

```python expandable
import assemblyai as aai

aai.settings.api_key = "your_api_key"

# Step 1: Transcribe with speaker diarization
config = aai.TranscriptionConfig(
    speaker_labels=True,  # Must enable speaker diarization first
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "name",  # or "role"
                "known_values": ["Sarah Chen", "Michael Rodriguez", "Alex Kim"]
            }
        }
    }
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe("meeting_recording.mp3", config=config)

# Access results with identified speakers
for utterance in transcript.utterances:
    print(f"{utterance.speaker}: {utterance.text}")
```

### Identifying by Role Instead of Name

For customer service, sales calls, or scenarios where you don't know names:

```python
config = aai.TranscriptionConfig(
    speaker_labels=True,
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "known_values": ["Agent", "Customer"]  # or ["Interviewer", "Interviewee"]
            }
        }
    }
)
```

**Common role combinations:**

- `["Agent", "Customer"]` - Customer service calls
- `["Support", "Customer"]` - Technical support
- `["Interviewer", "Interviewee"]` - Interviews
- `["Host", "Guest"]` - Podcasts
- `["Doctor", "Patient"]` - Medical consultations (with HIPAA compliance)

### How to Get Speaker Names

**For platform recordings:**

1. **Zoom**: Extract participant names from Zoom API or meeting JSON
2. **Teams**: Get attendees from Microsoft Graph API
3. **Google Meet**: Use Google Calendar API to get participants

**Example with Zoom:**

```python
# Get participant names from Zoom meeting
zoom_participants = get_zoom_meeting_participants(meeting_id)
speaker_names = [p["name"] for p in zoom_participants]

# Use in speaker identification
config = aai.TranscriptionConfig(
    speaker_labels=True,
    speakers_expected=len(speaker_names),  # Exact number of speakers to detect
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": speaker_names
            }
        }
    }
)
```

### How Speaker Identification Works

**Speaker Identification Requirements:**

1. **Speaker diarization must be enabled** - Cannot identify speakers without diarization first
2. **Requires sufficient audio per speaker** - Each speaker needs enough speech for accurate matching
3. **Works best with distinct voices** - Similar voices may be confused
4. **Post-processing step** - Adds additional processing time after transcription

**Accuracy depends on:**

- Audio quality (clear, minimal background noise)
- Voice distinctiveness (different genders, accents, tones)
- Amount of speech per speaker (more = better)
- Number of speakers (fewer = more accurate)

### Alternative: Add Identification Later

You can add speaker identification to an existing transcript by posting to the Speech Understanding API with the `transcript_id`. This is useful when you get speaker names after the transcription completes, or when building iterative workflows where users confirm speaker identities.

```python expandable
import requests

# First, transcribe with speaker diarization
transcript = transcriber.transcribe(audio_url, config=aai.TranscriptionConfig(speaker_labels=True))

# Later, add speaker identification using the transcript ID
understanding_body = {
    "transcript_id": transcript.id,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": ["Sarah Chen", "Michael Rodriguez"]
            }
        }
    }
}

result = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers={"Authorization": aai.settings.api_key},
    json=understanding_body
).json()

# Access identified speakers from the response
for utterance in result["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")
```

This approach is useful when:

- You get speaker names after the transcription completes
- You want to try different name mappings
- Building iterative workflows where users confirm speaker identities

For complete API details, see our [Speaker Identification documentation](/speech-understanding/speaker-identification).

## How Do I Translate Between Languages in Meetings?

AssemblyAI supports translation between 86 languages, enabling you to transcribe meetings in one language and translate to another.

### When to Use Translation

**Common use cases:**

- Transcribe Spanish meeting → Translate to English for documentation
- Transcribe multilingual meeting → Translate all to common language
- Create translated meeting notes for international teams
- Provide translated summaries for stakeholders

### Basic Translation

Translation is a Speech Understanding feature. You enable it via the `speech_understanding` parameter with `target_languages`:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

# Configure transcription with translation
data = {
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "translation": {
                "target_languages": ["es", "de"],
                "formal": True
            }
        }
    }
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

print("--- Original Transcript ---")
print(transcript["text"][:200] + "...")

print("\n--- Translations ---")
for language_code, translated_text in transcript["translated_texts"].items():
    print(f"{language_code.upper()}:")
    print(translated_text[:200] + "...")
```

### Translation with Speaker Labels

For meetings where you need per-utterance translations with speaker attribution:

```python
data = {
    "audio_url": audio_url,
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "translation": {
                "target_languages": ["es"],
                "match_original_utterance": True,
                "formal": True
            }
        }
    }
}

for utterance in transcript["utterances"]:
    print(f"Speaker {utterance['speaker']}:")
    print(f"  Original: {utterance['text'][:100]}...")
    print(f"  Spanish: {utterance['translated_texts']['es'][:100]}...")
```

### Supported Language Pairs

AssemblyAI supports translation between **86 languages**, including:

**Popular combinations:**

- Spanish ↔ English
- French ↔ English
- German ↔ English
- Mandarin ↔ English
- Japanese ↔ English
- Portuguese ↔ English
- And all combinations between supported languages

### Translation Response Format

The response includes `translated_texts` as a dictionary keyed by language code:

```python
{
    "text": "Original transcript in source language",
    "translated_texts": {
        "es": "Translated transcript in Spanish",
        "de": "Translated transcript in German"
    },
    "utterances": [
        {
            "speaker": "A",
            "text": "Hello, how are you?",
            "translated_texts": {
                "es": "Hola, ¿cómo estás?"
            },
            "start": 0,
            "end": 1500
        }
    ]
}
```

For complete language support and translation details, see our [Translation documentation](/speech-understanding/translation).

## What Workflows Can I Build for My AI Meeting Notetaker?

Use these Speech Understanding and Guardrails features to transform raw transcripts into actionable insights.

### Summarization

`summarization: true`

**What it does:** Generates an abstractive recap of the conversation (not verbatim).  
**Output:** `summary` string (bullets/paragraph format).  
**Great for:** Meeting notes, call recaps, executive summaries.  
**Notes:** Condenses and rephrases; minor details may be omitted by design.

**Example:**

```python
config = aai.TranscriptionConfig(
    summarization=True,
    summary_type="bullets",  # or "bullets_verbose", "gist", "headline", "paragraph"
    summary_model="informative",  # or "conversational"
)
```

### Sentiment Analysis

`sentiment_analysis: true`

**What it does:** Scores per-utterance sentiment (positive / neutral / negative).  
**Output:** Array of `{ text, sentiment, confidence, start, end }`.  
**Great for:** Customer satisfaction tracking, coaching, churn prediction.  
**Notes:** Segment-level (not global mood); sarcasm and very short utterances are harder to classify.

**Example:**

```python
for utterance in transcript.sentiment_analysis_results:
    if utterance.sentiment == "NEGATIVE":
        print(f"Negative sentiment detected: {utterance.text}")
```

### Entity Detection

`entity_detection: true`

**What it does:** Extracts named entities (people, organizations, locations, products, etc.).  
**Output:** Array of `{ entity_type, text, start, end }`.  
**Great for:** Auto-tagging topics, tracking competitors mentioned, CRM enrichment.  
**Notes:** Operates on post-redaction text if PII redaction is enabled.

**Example:**

```python
# Extract all organizations mentioned
organizations = [
    entity.text for entity in transcript.entities
    if entity.entity_type == "organization"
]
print(f"Companies mentioned: {', '.join(organizations)}")
```

### Redact PII Text

`redact_pii: true`

**What it does:** Scans transcript for personally identifiable information and replaces matches per policy.  
**Output:** `text` with replacements; original `words` timing preserved.  
**Great for:** GDPR/CCPA compliance, safe sharing, SOC2 requirements.  
**Notes:** Runs **before** downstream features; they see the redacted text.

**Recommended policies for meetings:**

```python
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,      # Remove names
        PIIRedactionPolicy.email_address,    # Remove emails
        PIIRedactionPolicy.phone_number,     # Remove phone numbers
        PIIRedactionPolicy.organization,     # Remove company names
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,  # Stable hash tokens
)
```

**Why hash substitution?**

- Stable across the file (same value → same token)
- Maintains sentence structure for LLM processing
- Prevents reconstruction of original data

### Redact PII Audio

`redact_pii_audio: true`

**What it does:** Produces a second audio file where redacted portions are bleeped/silenced.  
**Output:** `redacted_audio_url` in the transcript response.  
**Great for:** External sharing, training materials, demos.  
**Notes:** Original audio is untouched; bleeped sections may sound choppy.

### Complete Example

```python expandable
config = aai.TranscriptionConfig(
    # Core transcription
    speaker_labels=True,

    # Speech Understanding
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,

    # PII protection
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.email_address,
        PIIRedactionPolicy.phone_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
    redact_pii_audio=True,
)

transcript = transcriber.transcribe(audio_url, config=config)

# Access all features
meeting_insights = {
    "summary": transcript.summary,
    "sentiment_trend": analyze_sentiment_trend(transcript.sentiment_analysis_results),
    "entities": extract_entities(transcript.entities),
    "safe_transcript": transcript.text,  # PII redacted
    "safe_audio": transcript.redacted_audio_url,  # PII bleeped
}
```

## How Do I Improve the Accuracy of My Notetaker?

**Best practices:**

- Include participant names for better speaker recognition
- Add company-specific jargon and acronyms
- Include product names and technical terms
- Keep individual terms under 50 characters
- Up to 200 terms per request (Universal-2) or 1000 terms (Universal-3.5 Pro)

### Using Keyterms Prompt for Pre-recorded Transcription

Keyterms prompting improves recognition accuracy for domain-specific vocabulary by up to 21%:

```python expandable
# Define domain-specific vocabulary
company_terms = [
    "AssemblyAI",
    "Universal-3.5 Pro",
    "Speech Understanding",
    "diarization"
]

participant_names = [
    "Dylan Fox",
    "Sarah Chen",
    "Michael Rodriguez"
]

technical_terms = [
    "API endpoint",
    "WebSocket",
    "latency metrics",
    "TTFT"
]

# Configure with keyterms prompt
config = aai.TranscriptionConfig(
    keyterms_prompt=company_terms + participant_names + technical_terms,
    speaker_labels=True,
    # ... other settings
)
```

### Using Keyterms Prompt for Streaming

```python expandable
# Streaming with contextual keyterms
keyterms = [
    # Participant names
    "Alice Johnson",
    "Bob Smith",

    # Meeting-specific vocabulary
    "Q4 objectives",
    "revenue targets",
    "customer acquisition",

    # Technical terms
    "API integration",
    "cloud migration"
]

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "keyterms_prompt": keyterms,
}
```

## How Do I Process the Response from the API?

### Processing Pre-recorded Responses

```python expandable
def process_transcript(transcript):
    """
    Extract and process all relevant data from pre-recorded transcript
    """
    # Basic transcript data
    meeting_data = {
        "id": transcript.id,
        "duration": transcript.audio_duration,
        "confidence": transcript.confidence,
        "full_text": transcript.text
    }

    # Process speaker utterances
    speakers = {}
    for utterance in transcript.utterances:
        speaker = utterance.speaker

        if speaker not in speakers:
            speakers[speaker] = {
                "utterances": [],
                "total_speaking_time": 0,
                "word_count": 0
            }

        speakers[speaker]["utterances"].append({
            "text": utterance.text,
            "start": utterance.start,
            "end": utterance.end,
            "confidence": utterance.confidence
        })

        # Calculate speaking time
        speakers[speaker]["total_speaking_time"] += (utterance.end - utterance.start) / 1000
        speakers[speaker]["word_count"] += len(utterance.text.split())

    meeting_data["speakers"] = speakers

    # Extract summary
    if transcript.summary:
        meeting_data["summary"] = transcript.summary

    # Calculate meeting statistics
    total_duration = transcript.audio_duration  # Already in seconds
    meeting_data["statistics"] = {
        "total_speakers": len(speakers),
        "total_words": sum(s["word_count"] for s in speakers.values()),
        "average_confidence": transcript.confidence,
        "speaking_distribution": {
            speaker: {
                "percentage": (data["total_speaking_time"] / total_duration) * 100,
                "minutes": data["total_speaking_time"] / 60
            }
            for speaker, data in speakers.items()
        }
    }

    return meeting_data

# Example usage
result = process_transcript(transcript)
print(f"Meeting had {result['statistics']['total_speakers']} speakers")
print(f"Speaker A spoke for {result['statistics']['speaking_distribution']['A']['minutes']:.1f} minutes")
```

### Processing Streaming Responses

```python expandable
class StreamingResponseProcessor:
    def __init__(self):
        self.partial_buffer = ""
        self.final_transcripts = []
        self.turn_metadata = []

    def process_message(self, message: dict):
        """
        Process real-time streaming messages
        """
        msg_type = message.get("type")

        if msg_type == "Begin":
            return {
                "event": "session_started",
                "session_id": message.get("id"),
                "expires_at": message.get("expires_at")
            }

        elif msg_type == "Turn":
            return self.process_turn(message)

        elif msg_type == "Termination":
            return {
                "event": "session_ended",
                "audio_duration": message.get("audio_duration_seconds"),
                "session_duration": message.get("session_duration_seconds")
            }

    def process_turn(self, data: dict):
        """Process turn messages"""
        is_final = data.get("end_of_turn")
        transcript = data.get("transcript", "")
        turn_order = data.get("turn_order")

        response = {
            "turn_order": turn_order,
            "is_final": is_final,
            "confidence": data.get("end_of_turn_confidence", 0)
        }

        # Handle partials (for live display)
        if not is_final and transcript:
            self.partial_buffer = transcript
            response["event"] = "partial"
            response["text"] = transcript

        # Handle finals (for storage)
        elif is_final:
            final_transcript = {
                "turn_order": turn_order,
                "text": transcript,
                "confidence": data.get("end_of_turn_confidence"),
                "timestamp": datetime.now().isoformat()
            }
            self.final_transcripts.append(final_transcript)
            response["event"] = "final"
            response["text"] = transcript

            # Clear partial buffer
            self.partial_buffer = ""

        return response

    def get_full_transcript(self):
        """
        Combine all final transcripts into complete meeting transcript
        """
        return {
            "full_text": " ".join(t["text"] for t in self.final_transcripts),
            "transcripts": self.final_transcripts,
            "total_turns": len(self.final_transcripts)
        }

# Example usage
processor = StreamingResponseProcessor()

# If you're using `websockets` version 13.0 or later, use `additional_headers` parameter. For older versions (< 13.0), use `extra_headers` instead.
async with websockets.connect(API_ENDPOINT, additional_headers=headers) as ws:
    async for message in ws:
        data = json.loads(message)
        result = processor.process_message(data)

        if result["event"] == "partial":
            # Update UI with live transcript
            update_live_caption(result["text"])

        elif result["event"] == "final":
            # Save final transcript
            save_transcript_segment(result)

# Get complete transcript when done
full_transcript = processor.get_full_transcript()
```

## Additional Resources

- [Universal Pre-recorded Documentation](/pre-recorded-audio)
- [Universal-3.5 Pro Streaming Documentation](/streaming)
- [Speaker Diarization Guide](/pre-recorded-audio/label-speakers)
- [Speaker Identification Guide](/speech-understanding/speaker-identification)
- [Translation Guide](/speech-understanding/translation)
- [Getting Started Guide](/pre-recorded-audio/getting-started/transcribe-an-audio-file)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)
- [Support](https://www.assemblyai.com/contact/support)

---

<a id="build-a-medical-scribe-docs-medical-scribe-best-practices-mdx"></a>

# Build a medical scribe

URL: https://www.assemblyai.com/docs/medical-scribe-best-practices
Source: docs/medical-scribe-best-practices.mdx
Navigation: Pre-recorded STT > Guides > Tutorials
Description: Build AI-powered medical scribes that transcribe patient encounters and generate clinical documentation.

AssemblyAI provides everything you need to build a medical scribe, from high-accuracy transcription with medical terminology support to HIPAA-compliant PII redaction and structured clinical note generation through the LLM Gateway.

Choose the guide that matches your clinical workflow:

  
    Transcribe recorded patient encounters with Universal-3.5 Pro. Includes Medical Mode, speaker diarization, entity detection, PII redaction, and SOAP note generation.
  
  
    Stream audio from a microphone during live encounters with Universal-3.5 Pro Streaming. Includes LLM Gateway post-processing, medical keyterms, and automatic SOAP note generation.
  

## Which approach should I use?

| Scenario | Recommended approach |
| --- | --- |
| Post-visit documentation with highest accuracy | [Post-visit medical scribe](/medical-scribe-best-practices/medical-scribe-post-visit) |
| Live encounter transcription during patient visits | [Real-time medical scribe](/medical-scribe-best-practices/medical-scribe-real-time) |
| Telemedicine or emergency department visits | [Real-time medical scribe](/medical-scribe-best-practices/medical-scribe-real-time) |
| Specialist consultations with complex terminology | [Post-visit medical scribe](/medical-scribe-best-practices/medical-scribe-post-visit) |
| Hybrid: real-time notes with post-visit verification | Use both guides together |

---

<a id="best-practices-for-building-contact-center-applications-docs-contact-center-best-practices-mdx"></a>

# Best Practices for building Contact Center Applications

URL: https://www.assemblyai.com/docs/contact-center-best-practices
Source: docs/contact-center-best-practices.mdx
Navigation: Pre-recorded STT > Guides > Tutorials
Description: Complete guide for building contact center applications with AssemblyAI

## Introduction

Building a contact center application requires careful consideration of accuracy, speaker separation, compliance, and scalability. This guide addresses common questions and provides practical solutions for both post-call analytics and real-time agent assist scenarios.

## Why AssemblyAI for contact centers?

AssemblyAI stands out as the premier choice for contact center applications with several key advantages:

### Industry-leading accuracy on telephony audio

- **Universal-3.5 Pro model** delivers best-in-class accuracy on 8kHz telephony audio
- **2.9% speaker diarization error rate** for precise agent vs. customer attribution
- **Multichannel support** for stereo call recordings where agent and customer are on separate channels
- **Keyterms prompt** allows providing call context to improve accuracy of company names, products, and compliance phrases

### Streaming with Universal-3.5 Pro

For real-time agent assist, AssemblyAI's Universal-3.5 Pro Streaming model (`universal-3-5-pro`) offers:

- **Low latency** enables live transcription during calls
- **Format turns** feature provides structured, readable output
- **Dynamic prompting** via `UpdateConfiguration` to update context mid-call
- **Dual-channel streaming** for separate agent and customer audio streams

### End-to-end voice AI platform

Unlike fragmented solutions, AssemblyAI provides a unified API for:

- Transcription with speaker diarization (agent vs. customer)
- Multichannel audio support for stereo call recordings
- PII redaction on both text and audio for HIPAA and PCI compliance
- Post-processing workflows with custom prompting - from call summaries to QA scoring
- Streaming and pre-recorded transcription in a single platform
- Compliance and security built for enterprise workloads (BAA, SOC2, ISO)

## When should I use pre-recorded vs streaming for contact centers?

Understanding when to use pre-recorded versus streaming is critical for contact center workflows.

### Pre-recorded Speech-to-text

**Post-call analytics** - Call already happened, you have the full recording

- **Highest accuracy needed** - Pre-recorded models have the highest accuracy
- **Speaker diarization is critical** - Pre-recorded has 2.9% speaker error rate
- **Multichannel recordings** - Most contact center recordings are stereo with agent and customer on separate channels
- **Compliance workflows** - Full PII redaction with audio de-identification
- **Post-call analytics** - Summarization, sentiment analysis, entity detection, QA scoring
- **Batch processing** - Processing large volumes of call recordings

**Best for:** QA scoring, compliance monitoring, coaching insights, post-call CRM updates, searchable call archives

### Streaming Speech-to-text

**Live calls** - Transcribing as the call happens

You should use streaming when you need to display a live transcript to agents during calls. With Universal-3.5 Pro Streaming, accuracy is closer to pre-recorded, but pre-recorded will always be the most accurate option.

- **Agent assist** - Live transcription visible to agents during calls
- **Real-time coaching** - Prompt agents with suggested responses or compliance reminders
- **Live compliance monitoring** - Detect compliance violations in real-time
- **No recording available** - Processing live audio only

**Best for:** Agent assist, real-time coaching, live compliance monitoring, live call transcription

### Hybrid approach (recommended)

Many contact center platforms use **both**:

1. **Streaming during the call** - Provide live transcription for agent assist and real-time coaching
2. **Pre-recorded after the call** - Generate high-quality transcript with speaker labels, summary, and analytics

**Example workflow:**

- Call begins → Start streaming for live agent assist
- Call ends → Upload recording to pre-recorded API for final transcript with speaker names
- Generate call summary, QA score, and compliance report from pre-recorded transcript
- Push results to CRM (e.g., Salesforce)

## What languages and features for a contact center application?

### Pre-recorded calls (Universal-3.5 Pro)

For post-call analytics, AssemblyAI supports:

**Languages**:

- 99 languages supported
- Automatic Language Detection to route to the most spoken language
- Code Switching to preserve changes in speech between languages

**Core Features**:

- Speaker diarization (agent-customer separation)
- Multichannel audio support - when agent and customer are on separate audio channels, enables perfect speaker separation without diarization
- Automatic formatting, punctuation, and capitalization
- Keyterms prompting for boosting domain-specific terms (up to 1000 terms for Universal-3.5 Pro)
- Natural language prompting (Universal-3.5 Pro) - up to 1,500 words to guide transcription behavior
- Speaker options with configurable min/max expected speakers for call transfers

**Speech Understanding**:

- Summarization for call recaps
- Sentiment analysis for customer satisfaction tracking
- Entity detection for extracting names, account numbers, and products
- Speaker identification to map generic labels to agent and customer names
- Translation between 86 languages

**Guardrails**:

- PII redaction on text and audio for HIPAA and PCI compliance

### Streaming (Universal-3.5 Pro Streaming)

For live call transcription, use **Universal-3.5 Pro Streaming** (`universal-3-5-pro`) for the highest streaming accuracy:

**Core Features**:

- Speaker diarization for identifying agent vs. customer
- Partial and final transcripts for responsive UI
- Format turns for structured, readable output
- Keyterms prompt for company names, products, and compliance phrases
- Dual-channel streaming for separate agent and customer audio

For more details, see the [Universal-3.5 Pro Streaming documentation](/streaming/getting-started/transcribe-streaming-audio).

## How can I get started building a post-call analytics pipeline?

Here's a complete example implementing pre-recorded transcription for contact center call analysis:

```python expandable
import assemblyai as aai
import asyncio
from typing import Dict, List
from assemblyai.types import (
    SpeakerOptions,
    PIIRedactionPolicy,
    PIISubstitutionPolicy,
)

# Configure API key
aai.settings.api_key = "your_api_key_here"

async def transcribe_call(audio_source: str, agent_name: str = None) -> Dict:
    """
    Transcribe a contact center call recording with full analytics

    Args:
        audio_source: Either a local file path or publicly accessible URL
        agent_name: Optional agent name for speaker identification
    """
    # Configure comprehensive call analysis
    config = aai.TranscriptionConfig(
        # Model selection
        speech_models=["universal-3-5-pro", "universal-2"],

        # Speaker diarization
        speaker_labels=True,
        speaker_options=SpeakerOptions(
            min_speakers_expected=2,  # Agent and customer
            max_speakers_expected=5   # Allow for call transfers - safe to keep high
        ),
        multichannel=False,  # Set to True if audio has separate channel per speaker

        # Language detection
        language_detection=True,

        # Boost accuracy of contact center vocabulary
        keyterms_prompt=[
            # Company-specific terms
            "Acme Corp", "Premium Support Plan",

            # Compliance phrases
            "recorded line", "calls are monitored and recorded",

            # Common contact center terms
            "account number", "case number", "ticket number",
            "escalation", "supervisor", "hold time",
        ],

        # Post-call analytics
        summarization=True,
        sentiment_analysis=True,
        entity_detection=True,

        # PII protection for compliance
        redact_pii=True,
        redact_pii_policies=[
            PIIRedactionPolicy.person_name,
            PIIRedactionPolicy.phone_number,
            PIIRedactionPolicy.email_address,
            PIIRedactionPolicy.account_number,
            PIIRedactionPolicy.us_social_security_number,
            PIIRedactionPolicy.credit_card_number,
            PIIRedactionPolicy.credit_card_cvv,
            PIIRedactionPolicy.credit_card_expiration,
            PIIRedactionPolicy.date_of_birth,
        ],
        redact_pii_sub=PIISubstitutionPolicy.hash,
        redact_pii_audio=True,
    )

    # Add speaker identification if agent name is known
    if agent_name:
        config.speech_understanding = {
            "request": {
                "speaker_identification": {
                    "speaker_type": "role",
                    "speakers": [
                        {"role": "Agent", "name": agent_name},
                        {"role": "Customer"}
                    ]
                }
            }
        }

    # Create transcriber
    transcriber = aai.Transcriber()

    try:
        # Submit transcription job
        transcript = await asyncio.to_thread(
            transcriber.transcribe,
            audio_source,
            config=config
        )

        # Check status
        if transcript.status == aai.TranscriptStatus.error:
            raise Exception(f"Transcription failed: {transcript.error}")

        # Process speaker-labeled utterances
        for utterance in transcript.utterances:
            start_time = utterance.start / 1000  # Convert ms to seconds
            end_time = utterance.end / 1000

            print(f"[{start_time:.1f}s - {end_time:.1f}s] {utterance.speaker}:")
            print(f"  {utterance.text}\n")

        return {
            "transcript": transcript,
            "utterances": transcript.utterances,
            "summary": transcript.summary,
            "sentiment": transcript.sentiment_analysis_results,
            "entities": transcript.entities,
            "redacted_audio_url": transcript.redacted_audio_url,
        }

    except Exception as e:
        print(f"Error during transcription: {e}")
        raise

async def main():
    audio_source = "https://your-storage.com/calls/call_recording.mp3"

    result = await transcribe_call(audio_source, agent_name="Sarah Johnson")

    print(f"\nCall duration: {result['transcript'].audio_duration} seconds")
    print(f"Summary: {result['summary']}")

if __name__ == "__main__":
    asyncio.run(main())
```

## How Do I Handle Multichannel Contact Center Audio?

Most contact center recordings are stereo with the agent on one channel and the customer on the other. Multichannel transcription gives you perfect speaker separation without diarization.

### Pre-recorded Multichannel

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    multichannel=True,  # Enable when agent and customer are on separate channels
    speaker_labels=False,  # Disable - channels already separate speakers

    # Still enable analytics
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,

    # PII redaction
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file, config=config)

# Channel 1 = Agent, Channel 2 = Customer (typical layout)
for utterance in transcript.utterances:
    role = "Agent" if utterance.channel == "1" else "Customer"
    print(f"{role}: {utterance.text}")
```

**When to use multichannel:**

- Call recordings from PBX systems with separate agent/customer channels
- Recordings from platforms like Genesys, Twilio, Five9, NICE, or Talkdesk
- Any stereo recording where each channel represents a different speaker

**Benefits:**

- **Perfect speaker separation** - No diarization errors
- **No speaker confusion or overlap issues**
- **Higher accuracy** - Model processes clean single-speaker audio per channel

### Streaming Multichannel

For real-time dual-channel transcription, create separate streaming sessions per channel:

```python expandable
import asyncio
import websockets
import json
from urllib.parse import urlencode

API_KEY = "your_api_key"

class ChannelTranscriber:
    def __init__(self, channel_id: int, role: str):
        self.channel_id = channel_id
        self.role = role
        self.connection_params = {
            "sample_rate": 8000,  # Telephony standard
            "speech_model": "universal-3-5-pro",
            "format_turns": True,
            "encoding": "pcm_mulaw",  # Common telephony encoding
        }

    async def transcribe_channel(self, audio_stream):
        url = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(self.connection_params, doseq=True)}"

        # If using websockets >= 13.0, use additional_headers. For < 13.0, use extra_headers.
        async with websockets.connect(url, additional_headers={"Authorization": API_KEY}) as ws:
            # Send and receive must run concurrently for real-time streaming
            async def send_audio():
                async for audio_chunk in audio_stream:
                    await ws.send(audio_chunk)

            async def receive_transcripts():
                async for message in ws:
                    data = json.loads(message)
                    if data.get("type") == "Turn" and data.get("end_of_turn"):
                        print(f"{self.role}: {data['transcript']}")

            await asyncio.gather(send_audio(), receive_transcripts())

# Create transcriber for each channel
async def transcribe_live_call(agent_audio_stream, customer_audio_stream):
    agent = ChannelTranscriber(0, "Agent")
    customer = ChannelTranscriber(1, "Customer")

    await asyncio.gather(
        agent.transcribe_channel(agent_audio_stream),
        customer.transcribe_channel(customer_audio_stream),
    )
```

See our [multichannel streaming guide](/streaming/label-speakers-and-separate-channels#multichannel-streaming-audio) for complete implementation details.

## How Can I Build a Real-Time Agent Assist?

Here's a complete example for real-time streaming transcription optimized for contact center agent assist:

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "your_api_key"

# Contact center keyterms
KEYTERMS = [
    # Company and product terms
    "Acme Corp",
    "Premium Support Plan",
    "Enterprise License",

    # Compliance phrases
    "recorded line",
    "calls are monitored",

    # Common contact center vocabulary
    "account number",
    "case number",
    "escalation",
    "supervisor",
]

# CONTACT CENTER CONFIGURATION
CONNECTION_PARAMS = {
    "sample_rate": 8000,  # Telephony standard (8kHz)
    "speech_model": "universal-3-5-pro",  # Universal-3.5 Pro Streaming for highest accuracy
    "format_turns": True,

    # Contact center turn detection
    # universal-3-5-pro defaults: min_turn_silence=100ms, max_turn_silence=1000ms
    "min_turn_silence": 400,  # Longer than default for natural call pauses
    "max_turn_silence": 1500,  # Longer for customers explaining issues

    # Keyterms for accuracy
    "keyterms_prompt": KEYTERMS,
}

API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS, doseq=True)}"

# Audio Configuration
FRAMES_PER_BUFFER = 400  # 50ms of audio at 8kHz
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
transcript_buffer = []

def on_open(ws):
    print("=" * 80)
    print(f"[{datetime.now().strftime('%H:%M:%S')}] Agent assist transcription started")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")
    print(f"Keyterms configured: {', '.join(KEYTERMS[:5])}...")
    print("=" * 80)

    def stream_audio():
        global stream
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                if not stop_event.is_set():
                    print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        if msg_type == "Begin":
            session_id = data.get("id", "N/A")
            print(f"[SESSION] Started - ID: {session_id}\n")

        elif msg_type == "Turn":
            end_of_turn = data.get("end_of_turn", False)
            transcript = data.get("transcript", "")
            turn_order = data.get("turn_order", 0)

            # Show partials for responsive agent UI
            if not end_of_turn and transcript:
                print(f"\r[LIVE] {transcript}", end="", flush=True)

            # Use formatted finals for agent display
            if end_of_turn and transcript:
                timestamp = datetime.now().strftime('%H:%M:%S')
                print(f"\n[{timestamp}] {transcript}")

                # Detect compliance keywords
                transcript_lower = transcript.lower()
                if any(term in transcript_lower for term in ["cancel", "refund", "complaint", "supervisor"]):
                    print("           ** ESCALATION KEYWORD DETECTED **")

                transcript_buffer.append({
                    "timestamp": timestamp,
                    "text": transcript,
                    "turn_order": turn_order,
                    "type": "final"
                })
                print()

        elif msg_type == "Termination":
            audio_duration = data.get("audio_duration_seconds", 0)
            print(f"\n[SESSION] Terminated - Duration: {audio_duration}s")

        elif msg_type == "Error":
            error_msg = data.get("error", "Unknown error")
            print(f"\n[ERROR] {error_msg}")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    print(f"\n[WEBSOCKET ERROR] {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    print(f"\n[WEBSOCKET] Disconnected - Status: {close_status_code}, Message: {close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def run():
    global audio, stream, ws_app

    audio = pyaudio.PyAudio()

    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
    except Exception as e:
        print(f"Error opening audio stream: {e}")
        if audio:
            audio.terminate()
        return

    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\n\nCtrl+C received. Stopping transcription...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## How Should I Handle Pre-recorded Transcription in Production?

### Webhook Callbacks (Recommended)

For high-volume contact center workloads, use webhooks instead of polling:

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    webhook_url="https://your-app.com/webhooks/assemblyai",
    webhook_auth_header_name="X-Webhook-Secret",
    webhook_auth_header_value="your_secret_here",
    speaker_labels=True,
    multichannel=True,
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
)

# Submit job and return immediately (non-blocking)
transcript = transcriber.submit(audio_url, config=config)
print(f"Job submitted: {transcript.id}")
# Your app continues processing other calls
```

**Webhook handler example:**

```python expandable
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhooks/assemblyai", methods=["POST"])
def assemblyai_webhook():
    if request.headers.get("X-Webhook-Secret") != "your_secret_here":
        return jsonify({"error": "Unauthorized"}), 401

    import requests as http_requests

    data = request.json
    transcript_id = data["transcript_id"]
    status = data["status"]

    if status == "completed":
        # Fetch the full transcript (webhook only sends transcript_id and status)
        transcript = http_requests.get(
            f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
            headers={"authorization": "your_api_key"}
        ).json()
        process_completed_call(transcript)
    elif status == "error":
        log_transcription_error(transcript_id)

    return jsonify({"received": True}), 200

def process_completed_call(transcript):
    """Process completed call transcript and push to CRM"""
    utterances = transcript["utterances"]
    summary = transcript["summary"]

    # Store in database
    save_to_database(transcript)

    # Push summary to CRM
    push_to_crm(transcript["id"], summary)

    # Run QA scoring
    qa_score = score_call_quality(utterances)
    save_qa_score(transcript["id"], qa_score)
```

### Scaling Considerations

- **HTTP rate limit:** 20,000 requests per 5-minute window, counted across submissions (POST) and polling (GET) combined
- **Exceeding the limit:** returns a `403` response
- **Parallel transcriptions (rate limit):** 200+ for paid accounts (queued beyond that)
- **Ramp up gradually:** start at 10-50 parallel requests, double incrementally
- **Avoid the rate limit:** use [webhooks](/pre-recorded-audio/webhooks) or jittered, widened polling — see [Polling without exceeding the rate limit](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale#polling-without-exceeding-the-rate-limit)
- **Contact Sales** before large-scale rollouts

## How Do I Handle PII and Compliance?

PII redaction is critical for contact center compliance (HIPAA, PCI-DSS, GDPR, CCPA).

### Recommended PII Configuration

```python expandable
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        # Customer identity
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.date_of_birth,
        PIIRedactionPolicy.us_social_security_number,

        # Contact information
        PIIRedactionPolicy.phone_number,
        PIIRedactionPolicy.email_address,
        PIIRedactionPolicy.location,

        # Financial information (PCI-DSS)
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.credit_card_cvv,
        PIIRedactionPolicy.credit_card_expiration,
        PIIRedactionPolicy.account_number,
        PIIRedactionPolicy.banking_information,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,  # Stable hash tokens
    redact_pii_audio=True,  # Create de-identified audio file
)
```

**Why hash substitution?**

- Stable across the file (same value = same token)
- Maintains sentence structure for downstream LLM processing
- Prevents reconstruction of original data

### HIPAA Compliance

- AssemblyAI provides a **Business Associate Agreement (BAA)** at no cost
- Paid customers can review and sign our standard online BAA self-serve from the [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the dashboard (Owner or Admin role required, upgraded account required)
- Use PII redaction with audio de-identification for full compliance

## How Do I Improve the Accuracy of My Contact Center Transcription?

### Prompting Best Practices

The most impactful lever for contact center accuracy is **prompting**. Use a structured prompt with a `Context:` field:

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],

    # Natural language prompt for transcription guidance
    prompt=(
        "Transcribe this audio with perfect punctuation and formatting. "
        "Preserve linguistic speech patterns including disfluencies, filler words, "
        "hesitations, repetitions, stutters, false starts, and colloquialisms. "
        "Transcribe in the original language mix (code-switching), preserving the "
        "words in the language they are spoken. Output plain transcript text only. "
        "Use a new line when the voice changes; each line contains only one "
        "person's words.\n\n"
        "Context: Acme Corp customer service call, recorded line, "
        "Agent: Sarah Johnson, calls are monitored and recorded"
    ),

    # Keyterms for proper nouns and domain vocabulary
    keyterms_prompt=[
        "Acme Corp",
        "Sarah Johnson",
        "Premium Support Plan",
        "Enterprise License",
        "recorded line",
        "calls are monitored and recorded",
    ],

    speaker_labels=True,
)
```

**Tips for effective prompting:**

- Use **positive instructions** ("transcribe verbatim") not negative ("do NOT summarize")
- **Start with fewer instructions, add one at a time** — every added instruction risks conflicting with another. Treat the older "3–6 instructions" guidance as an upper bound, not a target.
- **Layer instructions one by one** and test each against your call recordings to measure impact
- Dynamize the `Context:` line per call with known info: company name, agent name, compliance phrases
- Use **keyterms** for proper nouns and domain vocabulary (company names, product names, agent names)

### Using Keyterms for Pre-recorded Transcription

```python expandable
# Build keyterms dynamically per call
call_keyterms = [
    # Company terms (static)
    "Acme Corp",
    "Premium Support Plan",

    # Agent name (from routing system)
    agent_name,

    # Customer name (from CRM lookup)
    customer_name,

    # Account-specific terms
    "account ending in 4532",
]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    keyterms_prompt=call_keyterms,
    speaker_labels=True,
)
```

### Using Keyterms for Streaming

```python
# Streaming with contact center context
keyterms = [
    "Acme Corp",
    "Premium Support Plan",
    "Sarah Johnson",
    "recorded line",
]

CONNECTION_PARAMS = {
    "sample_rate": 8000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "encoding": "pcm_mulaw",
    "keyterms_prompt": keyterms,
}
```

## What Workflows Can I Build for My Contact Center Application?

Use these features to transform raw call transcripts into actionable insights.

### Summarization

`summarization: true`

**What it does:** Generates an abstractive recap of the call.
**Output:** `summary` string (bullets/paragraph format).
**Great for:** Post-call CRM updates, call recaps, supervisor review.

```python
config = aai.TranscriptionConfig(
    summarization=True,
    summary_type="bullets",  # or "bullets_verbose", "gist", "headline", "paragraph"
    summary_model="informative",  # or "conversational"
)
```

### Sentiment Analysis

`sentiment_analysis: true`

**What it does:** Scores per-utterance sentiment (positive / neutral / negative).
**Output:** Array of `{ text, sentiment, confidence, start, end }`.
**Great for:** Customer satisfaction tracking, escalation detection, QA scoring.

```python
# Analyze customer sentiment across a call
negative_count = 0
for result in transcript.sentiment_analysis_results:
    if result.sentiment == "NEGATIVE":
        negative_count += 1
        print(f"Negative at {result.start / 1000:.1f}s: {result.text}")

# Flag calls with high negative sentiment
if negative_count > 3:
    flag_for_supervisor_review(transcript.id)
```

### Entity Detection

`entity_detection: true`

**What it does:** Extracts named entities (people, organizations, locations, products, etc.).
**Output:** Array of `{ entity_type, text, start, end }`.
**Great for:** CRM enrichment, auto-tagging topics, competitor tracking.

```python
# Extract key entities from a call
organizations = [e.text for e in transcript.entities if e.entity_type == "organization"]
print(f"Companies mentioned: {', '.join(organizations)}")
```

### Speaker Identification

Map generic speaker labels to agent and customer names:

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    speaker_labels=True,
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "speakers": [
                    {"role": "Agent", "name": "Sarah Johnson", "description": "Customer service representative"},
                    {"role": "Customer"}
                ]
            }
        }
    }
)
```

### Translation

Translate call transcripts for international teams:

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,
    speech_understanding={
        "request": {
            "translation": {
                "target_languages": ["en"],  # Translate to English
                "match_original_utterance": True,  # Per-utterance translations
                "formal": True
            }
        }
    }
)
```

### Redact PII Text and Audio

```python
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
    redact_pii_audio=True,  # Generate de-identified audio
)

# After transcription
print(transcript.text)  # PII redacted in text
print(transcript.redacted_audio_url)  # PII bleeped in audio
```

## How Do I Process the Response from the API?

### Processing Pre-recorded Responses

```python expandable
def process_call_transcript(transcript):
    """
    Extract and process all relevant data from a pre-recorded call transcript
    """
    call_data = {
        "id": transcript.id,
        "duration": transcript.audio_duration,  # Already in seconds
        "confidence": transcript.confidence,
        "full_text": transcript.text,
    }

    # Process speaker utterances
    speakers = {}
    for utterance in transcript.utterances:
        speaker = utterance.speaker

        if speaker not in speakers:
            speakers[speaker] = {
                "utterances": [],
                "total_speaking_time": 0,
                "word_count": 0
            }

        speakers[speaker]["utterances"].append({
            "text": utterance.text,
            "start": utterance.start,
            "end": utterance.end,
        })

        speakers[speaker]["total_speaking_time"] += (utterance.end - utterance.start) / 1000
        speakers[speaker]["word_count"] += len(utterance.text.split())

    call_data["speakers"] = speakers

    # Extract summary
    if transcript.summary:
        call_data["summary"] = transcript.summary

    # Analyze sentiment
    if transcript.sentiment_analysis_results:
        sentiments = [r.sentiment for r in transcript.sentiment_analysis_results]
        call_data["sentiment_breakdown"] = {
            "positive": sentiments.count("POSITIVE"),
            "neutral": sentiments.count("NEUTRAL"),
            "negative": sentiments.count("NEGATIVE"),
        }

    # Calculate statistics
    total_duration = transcript.audio_duration
    call_data["statistics"] = {
        "total_speakers": len(speakers),
        "total_words": sum(s["word_count"] for s in speakers.values()),
        "speaking_distribution": {
            speaker: {
                "percentage": (data["total_speaking_time"] / total_duration) * 100,
                "minutes": data["total_speaking_time"] / 60,
            }
            for speaker, data in speakers.items()
        },
    }

    return call_data

result = process_call_transcript(transcript)
print(f"Call had {result['statistics']['total_speakers']} speakers")
print(f"Sentiment: {result.get('sentiment_breakdown', {})}")
```

## Additional Resources

- [Universal-3.5 Pro Pre-recorded Documentation](/pre-recorded-audio)
- [Universal-3.5 Pro Streaming Documentation](/streaming/getting-started/transcribe-streaming-audio)
- [Speaker Diarization Guide](/pre-recorded-audio/label-speakers)
- [Multichannel Streaming Guide](/streaming/label-speakers-and-separate-channels)
- [Speaker Identification Guide](/speech-understanding/speaker-identification)
- [Translation Guide](/speech-understanding/translation)
- [PII Redaction Guide](/guardrails/redact-pii-from-transcripts)
- [Getting Started Guide](/pre-recorded-audio/getting-started/transcribe-an-audio-file)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)
- [Support](https://www.assemblyai.com/contact/support)

---

<a id="get-youtube-video-transcripts-with-yt-dlp-docs-pre-recorded-audio-guides-transcribe-youtube-videos-mdx"></a>

# Get YouTube Video Transcripts with yt-dlp

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/transcribe_youtube_videos
Source: docs/pre-recorded-audio/guides/transcribe_youtube_videos.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Basic transcription workflows
Description: Get YouTube Video Transcripts with yt-dlp documentation.

In this guide, we'll show you how to transcribe YouTube videos.

For this, we use the [yt-dlp](https://github.com/yt-dlp/yt-dlp) library to download YouTube videos and then transcribe it with the AssemblyAI API.

`yt-dlp` is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork with additional features and fixes. It is better maintained and preferred over `youtube-dl` nowadays.

In this guide we'll show 2 different approaches:

- Option 1: Download video via CLI
- Option 2: Download video via code

Let's get started!

## Quickstart

```python expandable
import assemblyai as aai
import yt_dlp

def transcribe_youtube_video(video_url: str, api_key: str) -> str:
    """
    Transcribe a YouTube video given its URL.

    Args:
        video_url: The YouTube video URL to transcribe
        api_key: AssemblyAI API key

    Returns:
        The transcript text
    """
    # Configure yt-dlp options for audio extraction
    ydl_opts = {
        'format': 'm4a/bestaudio/best',
        'outtmpl': '%(id)s.%(ext)s',
        'postprocessors': [{
            'key': 'FFmpegExtractAudio',
            'preferredcodec': 'm4a',
        }]
    }

    # Download and extract audio
    with yt_dlp.YoutubeDL(ydl_opts) as ydl:
        ydl.download([video_url])
        # Get video ID from info dict
        info = ydl.extract_info(video_url, download=False)
        video_id = info['id']

    # Configure AssemblyAI
    aai.settings.api_key = api_key

    # Transcribe the downloaded audio file
    config = aai.TranscriptionConfig()
    transcriber = aai.Transcriber()
    transcript = transcriber.transcribe(f"{video_id}.m4a", config)

    return transcript.text

transcript_text = transcribe_youtube_video("https://www.youtube.com/watch?v=wtolixa9XTg", "YOUR-API-KEY")
print(transcript_text)
```

# Step-by-step guide

## Install Dependencies

Install [yt-dlp](https://github.com/yt-dlp/yt-dlp) and the [AssemblyAI Python SDK](https://github.com/AssemblyAI/assemblyai-python-sdk) via pip.

```bash
pip install -U yt-dlp
```

```bash
pip install assemblyai
```

## Option 1: Download video via CLI

In this approach we download the YouTube video via the command line and then transcribe it via the AssemblyAI API. We use the following video here:

- https://www.youtube.com/watch?v=wtolixa9XTg

To download it, use the `yt-dlp` command with the following options:

- `-f m4a/bestaudio`: The format should be the best audio version in m4a format.
- `-o "%(id)s.%(ext)s"`: The output name should be the id followed by the extension. In this example, the video gets saved to "wtolixa9XTg.m4a".
- `wtolixa9XTg`: the id of the video.

```bash
yt-dlp -f m4a/bestaudio -o "%(id)s.%(ext)s" https://www.youtube.com/watch?v=wtolixa9XTg
```

Next, set up the AssemblyAI SDK and trancribe the file. Replace `YOUR_API_KEY` with your own key. If you don't have one, you can [sign up here](https://assemblyai.com/dashboard/signup) for free.

Make sure that the path you pass to the `transcribe()` function corresponds to the saved filename.

```python
import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
transcript = transcriber.transcribe("wtolixa9XTg.m4a", config)
print(transcript.text)
```

## Option 2: Download video via code

In this approach we download the video with a Python script instead of the command line.

You can download the file with the following code:

```python
import yt_dlp

URLS = ['https://www.youtube.com/watch?v=wtolixa9XTg']

ydl_opts = {
    'format': 'm4a/bestaudio/best',  # The best audio version in m4a format
    'outtmpl': '%(id)s.%(ext)s',  # The output name should be the id followed by the extension
    'postprocessors': [{  # Extract audio using ffmpeg
        'key': 'FFmpegExtractAudio',
        'preferredcodec': 'm4a',
    }]
}

with yt_dlp.YoutubeDL(ydl_opts) as ydl:
    error_code = ydl.download(URLS)
```

After downloading, you can use the same code from option 1 to transcribe the file:

```python
import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
transcript = transcriber.transcribe("wtolixa9XTg.m4a", config)
```

---

<a id="build-a-ui-for-transcription-with-gradio-and-python-docs-pre-recorded-audio-guides-gradio-frontend-mdx"></a>

# Build a UI for Transcription with Gradio and Python

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/gradio-frontend
Source: docs/pre-recorded-audio/guides/gradio-frontend.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Basic transcription workflows
Description: Build a UI for Transcription with Gradio and Python documentation.

In this guide, we'll show you how to use the [Gradio](https://www.gradio.app/) library in Python to build a simple front-end that allows you to drag-and-drop files for transcription. If you've ever gotten tired of running your Python program from the command-line, now you can have a self-hosted UI for processing your transcripts!

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Installing Dependencies

Firstly, we'll need to get all of our dependencies installed. This demo depends on both the AssemblyAI Python SDK as well as Gradio, so we'll need to run the following command to get them both installed, or updated to their latest version if you've already gotten them installed.

```bash
pip install -U assemblyai gradio
```

## Creating Your Transcription Function

Now we'll move on to setting up a function that can handle transcription for us. This function will submit the filepath for the audio file you've uploaded via Gradio to our API, and will wait until the transcript has finished, or log an error to the console.

```python
import assemblyai

# Set your API key here, which can be found on your Dashboard as mentioned above.
assemblyai.settings.api_key = "API_KEY_HERE"

def transcribe(audio_path):
    config = assemblyai.TranscriptionConfig()
    transcriber = assemblyai.Transcriber()
    transcript = transcriber.transcribe(audio_path, config)

    if transcript.status == assemblyai.TranscriptStatus.error:
        print(f"Transcription failed: {transcript.error}")
```

## Setting Up our Gradio UI

Now we get to move on to setting up what our Gradio UI will look like. This project will be fairly simple, so it will only use a few components, but Gradio offers a wide range of components and ways to customize how they look to suit your needs. Check out their [documentation](https://www.gradio.app/docs) here for more detailed information.

We'll be using Gradio's Blocks API for a more custom way of setting up our UI. Specifically, we'll use their Markdown, File, Button, and Textbox components to create a title, enable file uploads, and submit files for transcription to render them to the screen.

Since we only want audio or video files to be submitted for transcription, we'll limit what can be uploaded with the `file_types` parameter.

```python
import gradio

with gradio.Blocks() as demo:
    gradio.Markdown("# AssemblyAI Transcription with Gradio")
    filepath = gradio.File(file_types=["audio", "video"])
    transcribe_button = gradio.Button(value="Transcribe")
    transcript = gradio.Textbox(value="", label="Transcript")
    transcribe_button.click(transcribe, inputs=[filepath], outputs=[transcript])

if __name__ == "__main__":
    demo.launch(debug=True, show_error=True)
```

Now you can run this code to deploy your Gradio app locally, with some helpful debug information in case you add more functionality. You can also get a publicly shareable link to your app by enabling `share=True` in your `.launch()` config. Also note that you should run this in a separate file outside of this Jupyter notebook so that your program doesn't time out. We hope this helps you create better interfaces for your integrations with our API, and please reach out to support@assemblyai.com if you have any questions we can help with!

---

<a id="detect-low-confidence-words-in-a-transcript-docs-pre-recorded-audio-guides-detecting-low-confidence-words-mdx"></a>

# Detect Low Confidence Words in a Transcript

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/detecting-low-confidence-words
Source: docs/pre-recorded-audio/guides/detecting-low-confidence-words.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Basic transcription workflows
Description: Detect Low Confidence Words in a Transcript documentation.

In this guide, we'll show you how to detect sentences that contain words with low confidence scores. Confidence scores represent how confident the model was in predicting the transcribed word. Detecting words with low confidence scores can be important for manually editing transcripts.
Each transcribed word will contain a corresponding confidence score between 0.0 (low confidence) and 1.0 (high confidence).
You can decide what your confidence threshold will be when implementing this logic in your application. For this guide, we will use a threshold of 0.4.

## Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can sign up for an account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home). This guide will use AssemblyAI's [JavaScript SDK](https://github.com/AssemblyAI/assemblyai-node-sdk). If you haven't already, install the SDK by following these [instructions](https://github.com/AssemblyAI/assemblyai-node-sdk#installation).

## Step-by-Step Instructions

Import the AssemblyAI package and create an AssemblyAI object with your API key:

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: process.env.ASSEMBLYAI_API_KEY,
});
```

Next create the transcript with your audio file, either via local audio file or URL (AssemblyAI's servers need to be able to access the URL, make sure the URL links to a downloadable file).

```javascript
const transcript = await client.transcripts.transcribe({
  audio_url: "./sample.mp4"
});
```

From there use the `id` from the transcript to request the transcript broken down into sentences.

```javascript
let { id } = transcript;
let { sentences } = await client.transcripts.sentences(id);
```

Set the confidence score threshold to a value of you choice (0.5 or less is a good start). In this guide, we'll use 0.4.

```javascript
let confidenceThreshold = 0.4;
```

Next, we will filter the sentences array down to just sentences that contain words with confidence scores of under 0.4.

```javascript
const sentencesWithLowConfidenceWords = (sentences, confidenceThreshold) => {
  return sentences.filter((sentence) => {
    const hasLowConfidenceWord = sentence.words.some(
      (word) => word.confidence < confidenceThreshold
    );
    return hasLowConfidenceWord;
  });
};

const filteredSentences = sentencesWithLowConfidenceWords(
  sentences,
  confidenceThreshold
);
```

Next we'll alter the `filteredSentences` array so that the `words` array for each sentence only contains the words with confidence scores under of 0.4.

```javascript
const filterScores = filteredSentences.map((item) => {
  return {
    ...item,
    words: item.words.filter((word) => word.confidence < confidenceThreshold),
  };
});
```

Finally, we'll display the final results. The final results will include the timestamp of the sentence that contains low confidence words, the sentence, the words that scored poorly, and their scores.

```javascript expandable
//This function is optional but can be used to format the timestamps from milleseconds to HH:MM:SS
const formatMilliseconds = (milliseconds) => {
  // Calculate hours, minutes, and seconds
  const hours = Math.floor(milliseconds / 3600000);
  const minutes = Math.floor((milliseconds % 3600000) / 60000);
  const seconds = Math.floor((milliseconds % 60000) / 1000);

  // Ensure the values are displayed with leading zeros if needed
  const formattedHours = hours.toString().padStart(2, "0");
  const formattedMinutes = minutes.toString().padStart(2, "0");
  const formattedSeconds = seconds.toString().padStart(2, "0");

  return `${formattedHours}:${formattedMinutes}:${formattedSeconds}`;
};

//Format the final results to contain the sentence, low confidence words, timestamps, and confidence scores.
const finalResults = filterScores.map((res) => {
  return `The following sentence at timestamp ${formatMilliseconds(res.start)} contained low confidence words: ${res.text} \n  Low confidence word(s) from this sentence: ${res.words
    .map((res) => {
      return `${res.text}[score: ${res.confidence}]`;
    })
    .join(", ")}}`;
});

console.log(finalResults);
```

The output will look something like this:

```
[
  'The following sentence at timestamp 00:04:34 contained low confidence words: I am contacting you first when I could just have phoned my bank and marked you as fraud in an instant. \n' +
    '  Low confidence word(s) from this sentence: marked[score: 0.33049]}',
  'The following sentence at timestamp 00:06:40 contained low confidence words: Sabitha, as much as I would like to help you, this is the best I can do for you. \n' +
    '  Low confidence word(s) from this sentence: Sabitha,[score: 0.22706]}',
  'The following sentence at timestamp 00:07:37 contained low confidence words: Thank you for calling Queston. \n' +
    '  Low confidence word(s) from this sentence: Queston.[score: 0.16557]}'
]
```

---

<a id="running-bulk-transcription-and-load-tests-at-scale-docs-pre-recorded-audio-guides-bulk-transcription-and-load-tests-at-scale-mdx"></a>

# Running Bulk Transcription and Load Tests at Scale

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale
Source: docs/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Batch transcription
Description: Best practices for submitting large volumes of audio files for transcription, including one-off load tests.

This guide applies to two closely related workloads:

- **Bulk transcription.** Submitting thousands of files in a batch — for example, a nightly backfill or a one-time migration.
- **Load testing.** Measuring turnaround time (TaT) and throughput before a production cutover.

The guidance is shared. If you're running a load test, also read the [load test subsection](#if-youre-running-a-load-test) of *Measure and verify*.

## Key recommendations

1. **Ramp.** Submit in 15-second windows. Start at 25 requests/window and grow ~8–9% per window until you reach your target sustained rate.
2. **Measure.** Use webhooks instead of polling. Record `submit_ts`, `complete_ts`, `audio_duration`, `model`, `features`, and `status` per request.
3. **Coordinate.** Recommended for runs above 200 requests/minute, required for large bulk uploads (tens of thousands of files), and for any EU workload.

The default rate limit for paid accounts is **200 parallel jobs**. If you need a higher limit, reach out to [support@assemblyai.com](mailto:support@assemblyai.com) — AssemblyAI offers custom rate limits at no additional cost.

## Before you begin

**Prerequisites**

- **Account balance.** If your balance hits zero mid-run, AssemblyAI drops your rate limit to 1 and invalidates your results. Bulk runs and load tests both incur standard transcription usage charges — fund your account for the full expected volume before you submit any requests.
- **Rate limit.** Check your current limit on the [Rate Limits page](https://www.assemblyai.com/dashboard/home) of your dashboard and size your target submission rate against it — see [Size your target rate](#size-your-target-rate).
- **Audio source.** Prefer pre-signed URLs (for example, from S3 or GCS) as your `audio_url`. Each `/v2/upload` call counts against your HTTP rate limit and adds latency proportional to file size — at bulk scale, uploads alone can exhaust your rate-limit budget. If local files are your only option, [upload them](/pre-recorded-audio/getting-started/transcribe-an-audio-file#using-the-http-api-directly) to `/v2/upload` first to get a hosted URL, and factor those uploads into your rate-limit planning.
- **Pre-signed URL expiration.** Set URL TTLs long enough to outlast your expected queue time plus turnaround time. One hour is a safe default for small-to-moderate runs; use two hours or more for runs that approach your rate limit or use less-common languages. URLs that expire while a job is queued or processing surface as `4xx` errors when AssemblyAI tries to retrieve the audio.
- **Completion tracking.** Configure [webhooks](/pre-recorded-audio/webhooks) (with a polling fallback) or a polling-only strategy before you start — you'll need a way to detect completion and record per-request timestamps.
- **Static-IP egress.** If your audio URLs are behind a strict S3 bucket policy, contact support to enable static-IP egress for retrieval. Webhook deliveries already come from fixed IPs documented on the [whitelisting FAQ](/faq/what-ip-address-should-i-whitelist-for-assemblyai).
- **Pricing reference.** See the [pricing page](https://www.assemblyai.com/pricing) for per-hour rates to plug into your cost estimate.

**Workload configuration**

Match these to your expected production traffic:

- **Region: US or EU.** The EU region handles less traffic than US and is more sensitive to load spikes. Coordinate with our team for any EU workload, regardless of size.
- **Traffic pattern.** Requests per minute at peak and whether traffic is steady or bursty.
- **Models and features.** Universal-3.5 Pro, Universal-2, speaker labels, PII redaction, summarization, and so on — each has different processing characteristics, and every feature you enable adds to TaT. Audit your request body and turn off what you don't need; a faster run is also a cheaper one.
- **Channels and speakers.** For call-center audio where agent and customer are on separate stereo channels, add `multichannel=true` to get per-channel utterances. For single-channel recordings with multiple speakers, use `speaker_labels=true` instead. See [Should I use Speaker Labels or Multi-channel?](/faq/should-i-use-speaker-labels-or-multi-channel) for guidance.
- **Language.** Some languages, such as Hindi, Swedish, and Hebrew, scale differently and may show longer TaT. Coordinate with our team if your workload is primarily in a less common language.
- **Audio format.** Format conversion and preprocessing run before transcription and contribute to overall TaT.
- **Cost estimate.** Sanity-check the expected spend before you submit: total audio hours × your per-hour rate. Bulk runs are billed the same as any other transcription, so a large backfill can produce a surprisingly large invoice if you haven't projected it.

## Pilot first

Before a full bulk run or load test, submit a pilot batch of 50–200 files using the exact configuration you plan to use at scale — same model, same features, same language, same webhook receiver, same error-handling logic. A pilot verifies that:

- The transcripts look right, and the model and feature set match what the downstream consumer expects.
- Your webhook receiver is reachable, verifying signatures, and writing results durably — or your polling loop is keeping up without hitting rate limits.
- Your retry logic handles `5xx` correctly and your dead-letter path captures `4xx` without silently dropping files.
- Your ramp and rate-limit controls behave as intended.

The most expensive bulk-run failures almost always come from discovering a configuration mistake — a wrong model, a feature flag left off, a webhook handler that drops results silently — after the whole batch has been billed. A pilot catches these while they're cheap to fix, and gives you a realistic mean TaT to plug into [Size your target rate](#size-your-target-rate).

## When to run

- **Small tests and moderate bulk runs** (well within your rate limit): US business hours (roughly 14:00–21:00 UTC) produce the most representative baseline latency, since throughput is highest during these periods.
- **Large runs** (200+ requests/minute): coordinate with our team before starting. Our team will pick a window and pre-scale for you.
- **EU region:** coordinate regardless of size.

## Ramp up gradually

The most common mistake — for both bulk uploads and load tests — is submitting all requests at once. A gradual ramp gives the pipeline time to scale ahead of your traffic, which is what produces the lowest and most consistent turnaround times. Submitting a large spike upfront typically results in higher TaT as capacity catches up.

**Recommended schedule (validated for Universal-3.5 Pro with speaker labels):**

- Divide your ramp into 15-second windows.
- Start at **25 requests per window**.
- Grow by **~8–9% per window** until you reach your target sustained rate.
- Don't pause mid-ramp. Stopping and restarting means ramping from the starting rate again, and you'll see higher latency when traffic resumes.
- Do not exceed your account's rate limit during the ramp.

If you're using different models or features, contact support for a tailored ramp plan — some components take longer to initialize and may need a slower ramp.

To compute a ramp for any target without a lookup table, use `rate_n = ceil(25 × 1.085ⁿ)` (capped at your target rate), where `n` is the window index starting at 0. For a fully worked 400 requests/minute schedule, see [Example ramp schedule](#example-ramp-schedule) at the bottom of this page.

After reaching your target rate, sustain it for at least 5–10 minutes. Your measurements are representative once p50 and p95 stay consistent over 2–3 consecutive minutes. If p50 is still falling, extend the sustain phase.

## Small runs (under 500 files)

If your total run is under 500 files, you don't need the ramp. Fire a bounded thread pool at your rate limit and submit in parallel. The ramp is for sustained high rates over several minutes.

## Size your target rate

Your rate limit caps how many jobs can be in progress at once, so your sustained submission rate needs to fit inside it. Pick a target rate that keeps the typical number of in-flight jobs comfortably below your limit:

1. Estimate mean turnaround time from your pilot run or the published [benchmarks](/pre-recorded-audio/benchmarks).
2. Multiply your target submission rate (requests per second) by that mean TaT to approximate the number of jobs that will be in flight at steady state.
3. Keep that number under ~80% of your rate limit. If it's higher, lower the target rate, shorten mean TaT (fewer features, shorter audio, a faster model), or [request a higher rate limit](#coordinate-with-our-team).

The 20% of headroom absorbs normal variation in audio duration, warm-up effects, and webhook-receive latency. Runs sized to the limit exactly will see TaT climb as the in-flight count bumps against the cap.

For planning, Universal-3.5 Pro English async typically completes a 5-minute file in about 9 seconds at p50 and 60 seconds at p95. Use that as your starting turnaround estimate before the pilot.

## Measure and verify

For every run — bulk or load test — track completion and record per-request metadata.

- **Use webhooks when possible.** You get a clean completion signal without polling overhead. See the [Webhooks documentation](/pre-recorded-audio/webhooks) for setup, retry behavior, and authentication.
- **Run a polling fallback alongside webhooks.** Webhooks can drop for many reasons — receiver downtime, signature mismatches, transient network failures. For every submitted job, record an expected completion deadline (around 2× mean TaT from your pilot) and `GET /v2/transcript/{id}` for any job whose webhook hasn't arrived by then. The fallback protects you from silent data loss without materially increasing your rate-limit usage.
- **Polling-only** every 1–2 seconds measures closer to actual completion but adds to your rate-limit budget. Use it when webhooks aren't available or when you need precise TaT during a load test.
- **Retry failures** with exponential backoff for `5xx` responses — see [Implement retry server error logic](/pre-recorded-audio/guides/retry-server-error). Investigate `4xx` responses; they indicate a client-side issue that retrying won't fix.

Record these fields per request:

- `submit_ts` — timestamp when `POST /v2/transcript` was sent
- `complete_ts` — timestamp when completion was detected
- `audio_duration` — length of the audio file, in seconds
- `model` — speech model used
- `features` — features enabled (e.g. `speaker_labels`, `auto_highlights`, `sentiment_analysis`)
- `status` — `completed` or `error`
- `id` — transcript ID, for debugging with support

Turnaround time = `complete_ts` − `submit_ts`.

Normalize TaT by audio duration to get the real-time factor: `RTF = turnaround_time ÷ audio_duration`. An RTF of 0.5 means the API processed the file in half its audio duration. RTF is the headline metric for comparing runs across regions, models, and audio-duration buckets — raw TaT varies too much with audio length to mean anything on its own.

### Polling without exceeding the rate limit

HTTP rate limits cap total API requests at **20,000 per 5 minutes** across all endpoints — submissions and polling combined. Exceeding this returns a `403` error.

If webhooks aren't an option, stay within that budget. As an illustration: at a sustained 33 requests/second submission rate with ~330 jobs in flight, submissions alone consume roughly 10,000 of your budget — so poll no more often than every 15 seconds to leave headroom. Scale these numbers to your own submission rate and in-flight job count:

| Polling interval | GETs/s at 330 in-flight | Total req/s (at 33 POST/s) | Within limit? |
| --- | --- | --- | --- |
| Every 3s | ~110 | ~143 | No |
| Every 5s | ~66 | ~99 | No |
| Every 10s | ~33 | ~66 | Yes |
| Every 15s | ~22 | ~55 | Yes |

When many jobs share the same polling interval they tend to cluster at the same second boundaries, spiking your rate-limit usage and occasionally returning `403` errors. Stagger each job's polling by ±25% of the interval (for example, `interval × random.uniform(0.75, 1.25)`) so GETs spread evenly across the window.

### If you're running a bulk job

Monitor for these signals during the run:

- **Healthy run:** TaT stays within ~20% of your first 5 minutes of sustained submissions, your completion queue drains steadily, and no errors arrive.
- **Diagnose, don't panic:** if something goes wrong, match the signal you see to the row in [Diagnosing problems](#diagnosing-problems) and respond accordingly — each signal has a different cause and fix.

### If you're running a load test

- **Separate ramp-phase from sustain-phase metrics.** Expect higher latency during ramp; use sustain-phase numbers as your benchmark.
- **Report percentiles, not averages.** Track p50, p75, p90, p95, p99, and max.
- **Normalize by audio duration.** Group results into duration buckets (0–5 min, 5–15 min, 15–30 min, 30–60 min) for meaningful comparison.
- **Pre-scaling caveat.** If our team pre-scaled for your test, your results reflect steady-state capacity — not cold-start or scale-up behavior.

## Diagnosing problems

| Signal | Meaning | Response |
| --- | --- | --- |
| `403` HTTP error | You've exceeded the 20,000 requests per 5 minutes HTTP rate limit (polling counts) | Increase polling interval or switch to webhooks; slow your submission rate |
| `4xx` submission error (other than 403) | Client-side issue (bad request, auth, invalid audio URL, etc.) | Inspect the response body and fix the request; retrying won't help |
| `5xx` submission error | Transient server-side issue | Retry with exponential backoff — see the [retry guide](/pre-recorded-audio/guides/retry-server-error) |
| TaT rises with no errors | Submission rate is outpacing available capacity | Slow the ramp or extend its duration; verify you're within your rate limit |

## Reference implementation

The AssemblyAI Python SDK's non-blocking `Transcriber.submit()` returns as soon as a transcript is queued, so you can drive the ramp yourself while using the SDK's `TranscriptionConfig` and exception classes. If you'd rather have the SDK handle both submission and polling for a smaller batch, see [Transcribe multiple files simultaneously](/pre-recorded-audio/guides/batch_transcription).

The following script ramps submissions to approximate the recommended schedule, retries transient errors, writes unrecoverable failures to a dead-letter log, and persists submitted `file → transcript_id` pairs so the run is resumable after a crash. The table above is hand-tuned to observed pipeline behavior, so the numbers this script produces may differ by one or two requests per window. Adjust `max_rate` to your target sustained rate.

```python expandable
import json
import logging
import math
import random
import time
from collections import deque
from pathlib import Path

import assemblyai as aai
from assemblyai.types import TranscriptError

logger = logging.getLogger(__name__)

aai.settings.api_key = "YOUR_API_KEY"
# For the EU region, point the SDK at the EU base URL:
# aai.settings.base_url = "https://api.eu.assemblyai.com"

STATE_PATH = Path("bulk_state.jsonl")       # Append-only submission log, for resume
FAILURES_PATH = Path("bulk_failures.jsonl") # Dead-letter log for unrecoverable submission errors

# Match your production configuration: model, features, language, etc.
config = aai.TranscriptionConfig(
    # For split-stereo call audio (agent on one channel, customer on the other):
    # multichannel=True,
    # For single-channel audio with multiple speakers:
    # speaker_labels=True,
)

transcriber = aai.Transcriber()

def append_jsonl(path, entry):
    with path.open("a") as f:
        f.write(json.dumps(entry) + "\n")

def already_submitted(path):
    """Return the set of file URLs successfully submitted in previous runs."""
    if not path.exists():
        return set()
    with path.open() as f:
        return {json.loads(line)["file"] for line in f if line.strip()}

def submit_file(file_url, max_retries=3):
    """Submit one file without waiting for completion. Retries transient errors with exponential backoff + jitter."""
    for attempt in range(max_retries + 1):
        try:
            transcript = transcriber.submit(file_url, config=config)
            return transcript.id
        except TranscriptError:
            if attempt == max_retries:
                raise
            time.sleep(2 ** attempt + random.random())

def submit_all(files, max_rate):
    """
    Submit files in 15-second windows, starting at 25 requests per window
    and growing ~8–9% per window until reaching max_rate.

    Files already recorded in STATE_PATH are skipped on restart. Failed
    submissions are written to FAILURES_PATH for inspection and manual replay.
    """
    done = already_submitted(STATE_PATH)
    remaining = deque(f for f in files if f not in done)

    rate = 25        # Starting requests per window
    window = 15      # Seconds per window
    growth = 1.085   # Per-window growth factor
    window_num = 0

    while remaining:
        window_num += 1
        batch_size = min(rate, len(remaining))

        for _ in range(batch_size):
            file = remaining.popleft()
            try:
                transcript_id = submit_file(file)
                append_jsonl(STATE_PATH, {"file": file, "id": transcript_id})
            except Exception as exc:
                logger.exception("Submission failed: %s", file)
                append_jsonl(FAILURES_PATH, {"file": file, "error": str(exc)})

        logger.info(
            "Window %d | Rate: %d/window | Submitted: %d | Remaining: %d",
            window_num, rate, batch_size, len(remaining),
        )

        rate = min(math.ceil(rate * growth), max_rate)

        if remaining:
            time.sleep(window)

# Usage:
# file_urls = ["https://example.com/audio1.mp3", ...]
# submit_all(file_urls, max_rate=100)  # Target: 100 requests per 15-second window
#
# Pick max_rate using the guidance in "Size your target rate".
# Use webhooks (with a polling fallback) to track completion; account for GETs
# in your rate-limit budget.
```

## Example ramp schedule

A fully worked schedule for ramping to 100 requests/window (400 requests/minute). Use it as a reference when building your own ramp, or as a starting point you can scale to a different target rate.

| Time window | Requests | Cumulative |
| --- | --- | --- |
| 0:00–0:15 | 25 | 25 |
| 0:15–0:30 | 27 | 52 |
| 0:30–0:45 | 29 | 81 |
| 0:45–1:00 | 31 | 112 |
| 1:00–1:15 | 33 | 145 |
| 1:15–1:30 | 35 | 180 |
| 1:30–1:45 | 38 | 218 |
| 1:45–2:00 | 41 | 259 |
| 2:00–2:15 | 44 | 303 |
| 2:15–2:30 | 47 | 350 |
| 2:30–2:45 | 51 | 401 |
| 2:45–3:00 | 55 | 456 |
| 3:00–3:15 | 59 | 515 |
| 3:15–3:30 | 64 | 579 |
| 3:30–3:45 | 69 | 648 |
| 3:45–4:00 | 75 | 723 |
| 4:00–4:15 | 81 | 804 |
| 4:15–4:30 | 88 | 892 |
| 4:30–4:45 | 95 | 987 |
| 4:45–5:00 | 100 | 1,087 |

## Coordinate with our team

Reach out to [support@assemblyai.com](mailto:support@assemblyai.com) or your account manager before you submit any requests if any of these apply:

- You plan to exceed **200 requests per minute**.
- You're running a **large one-time upload** (tens of thousands of files or more).
- You want **support available during the run** — for example, if you're running outside US business hours.
- You're using the **EU region**, regardless of size.

When you reach out, include:

- Expected request volume and ramp schedule, broken into 15-second windows
- Audio file durations and language breakdown
- Speech models and features you'll enable
- Whether audio is single-channel or multichannel
- Preferred run window (see [When to run](#when-to-run))

AssemblyAI can pre-scale pipeline components for your traffic, raise your rate limit, and monitor the run in real time. For recurring bulk workloads (for example, nightly batch jobs), we can set up persistent scaling.

## Related pages

- [Rate limits](/pre-recorded-audio/rate-limits)
- [Webhooks](/pre-recorded-audio/webhooks)
- [Benchmarks](/pre-recorded-audio/benchmarks)
- [Implement retry server error logic](/pre-recorded-audio/guides/retry-server-error)
- [Cloud endpoints and data residency](/pre-recorded-audio/select-the-region)

---

<a id="transcribe-multiple-files-simultaneously-using-the-javascript-sdk-docs-pre-recorded-audio-guides-sdk-node-batch-mdx"></a>

# Transcribe Multiple Files Simultaneously Using the JavaScript SDK

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/sdk-node-batch
Source: docs/pre-recorded-audio/guides/sdk-node-batch.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Batch transcription
Description: Transcribe Multiple Files Simultaneously Using the JavaScript SDK documentation.

In this guide, we'll show you how to transcribe multiple files simultaneously using the JavaScript SDK.

## Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can sign up for an account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home). This guide will use AssemblyAI's [JavaScript SDK](https://github.com/AssemblyAI/assemblyai-node-sdk). If you haven't already, install the SDK in your project by following these [instructions](https://github.com/AssemblyAI/assemblyai-node-sdk#installation).

## Step-by-Step Instructions

Set up your application folder structure by adding an audio folder which will house the files you would like to transcribe, a transcripts folder which will house your completed transcriptions, and a new `.js` file in the root of the project. Your file structure should look like this:

```
BatchApp
├───audio
│   ├───audio-1.mp3
│   └───audio-2.mp3
├───transcripts
├───batch.js
```

In the `batch.js` file import the AssemblyAI package, as well as the node fs and node path packages. Create an AssemblyAI object with your API key:

```
import { AssemblyAI } from "assemblyai";
import * as path from 'path';
import * as fs from 'fs';

const client = new AssemblyAI({
  apiKey: <Your API Key>,
});
```

Declare the variables `audioFolder`, `files`, `filePathArr`, and `transcriptsFolder`.

- `audioFolder` will be the relative path to the folder containing your audio files.
- `files` will read the files in the audio folder, and return them in an array.
- `filePathArr` will join the file names with the audio folder name to create the relative path to each individual file.
- `transcriptsFolder` will be the relative path to the folder containing your transcription files.

```
const audioFolder = './audio';
const files = await fs.promises.readdir(audioFolder);
const filePathArr = files.map(file => path.join(audioFolder, file));
const transcriptsFolder = './transcripts';
```

Next, we'll create a promise that will submit the file path for transcription. Make sure to add the parameters for the models you would like to run.

```
const getTranscript = (filePath) => new Promise((resolve, reject) => {
  client.transcripts.transcribe({
    audio: filePath,
    language_detection: true
  })
  .then(result => resolve(result))
  .catch(error => reject(error));
});
```

Next, we will create an async function that will call the `getTranscript` function and write the transcription text from each audio file to an individual text file in the transcripts folder.

``` expandable
const processFile = async (file) => {
  const getFileName = file.split('audio/'); //Separate the folder name and file name into substrings
  const fileName = getFileName[1]; //Grab the 2nd substring which is the file name
  const filePath = path.join(transcriptsFolder, `${fileName}.txt`); //Relative path for transcription text files.

  const transcript = await getTranscript(file); //Request the transcript
  const text = transcript.text; //Grab transcription text from the JSON response

  //Write the transcription text to a text file
  return new Promise((resolve, reject) => {
    fs.writeFile(filePath, text, err => {
      if (err) {
        reject(err);
        return;
      }

      resolve({
        ok: true,
        message: 'Text File created!'
      });
    });
  });
}
```

Next, we will create the run function. This function will:

- Create an array of unresolved promises with each promise requesting a transcript.
- Use `Promise.all` to iterate over the array of unresolved promises.

Then we'll call the run function

```
const run = async () => {
  const unresolvedPromises = filePathArr.map(processFile);
  await Promise.all(unresolvedPromises);
}

run()
```

Your final file will look like this:

``` expandable
import { AssemblyAI } from "assemblyai";
import * as path from 'path';
import * as fs from 'fs';

const client = new AssemblyAI({
  apiKey: <Your API>,
});

const audioFolder = './audio';
const files = await fs.promises.readdir(audioFolder);
const filePathArr = files.map(file => path.join(audioFolder, file));
const transcriptsFolder = './transcripts'

const getTranscript = (filePath) => new Promise((resolve, reject) => {
  client.transcripts.transcribe({
    audio: filePath,
    language_detection: true
  })
  .then(result => resolve(result))
  .catch(error => reject(error));
});

const processFile = async (file) => {
  const getFileName = file.split('audio/')
  const fileName = getFileName[1]
  const filePath = path.join(transcriptsFolder, `${fileName}.txt`);

  const transcript = await getTranscript(file);
  const text = transcript.text

  return new Promise((resolve, reject) => {
    fs.writeFile(filePath, text, err => {
      if (err) {
        reject(err);
        return;
      }

      resolve({
        ok: true,
        message: 'Text File created!'
      });
    });
  });
}

const run = async () => {
  const unresolvedPromises = filePathArr.map(processFile);
  await Promise.all(unresolvedPromises);
}

run()
```

If you have any questions, please feel free to reach out to our Support team at support@assemblyai.com.

---

<a id="transcribe-multiple-files-simultaneously-using-the-python-sdk-docs-pre-recorded-audio-guides-batch-transcription-mdx"></a>

# Transcribe Multiple Files Simultaneously Using the Python SDK

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/batch_transcription
Source: docs/pre-recorded-audio/guides/batch_transcription.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Batch transcription
Description: Transcribe Multiple Files Simultaneously Using the Python SDK documentation.

In this guide, we'll show you how to use the AssemblyAI API to transcribe multiple audio files at once. This guide focuses on demonstrating how to use the AssemblyAI Python SDK to achieve this.

## Quickstart

```python expandable
import assemblyai as aai
import threading
import os

aai.settings.api_key = "YOUR_API_KEY"
batch_folder = "audio"
transcription_result_folder = "transcripts"

config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()

def transcribe_audio(audio_file):
    transcriber = aai.Transcriber()
    transcript = transcriber.transcribe(os.path.join(batch_folder, audio_file), config)
    if transcript.status == "completed":
        with open(f"{transcription_result_folder}/{audio_file}.txt", "w") as f:
            f.write(transcript.text)
    elif transcript.status == "error":
        print("Error: ", transcript.error)

threads = []
for filename in os.listdir(batch_folder):
    thread = threading.Thread(target=transcribe_audio, args=(filename,))
    threads.append(thread)
    thread.start()

for thread in threads:
    thread.join()

print("All transcriptions are complete.")
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Step-by-Step Guide

Install the SDK.

```bash
pip install -U assemblyai
```

Import the `assemblyai `package and set the API key. Import `threading` and `OS` Python libraries that enable concurrent task processing and file path interactions respectively.

```python
import assemblyai as aai
import threading
import os

aai.settings.api_key = "YOUR_API_KEY"
```

Set the folders. The `batch` folder contains the audio files that you want to process and transcribe. The `transcription_result_folder` stores the .txt transcript files.

```python
batch_folder = "audio"
transcription_result_folder = "transcripts"
```

Create a `Transcriber` object.

```python
transcriber = aai.Transcriber()
```

Function to transcribe an audio file. Once the transcript is complete, a .txt file is generated to the `transcription_result_folder`. If there is an error with the transcription, it will not be processed to the results folder.

```python
def transcribe_audio(audio_file):
    config = aai.TranscriptionConfig()
    transcriber = aai.Transcriber()
    transcript = transcriber.transcribe(os.path.join(batch_folder, audio_file), config)
    if transcript.status == "completed":
        with open(f"{transcription_result_folder}/{audio_file}.txt", "w") as f:
            f.write(transcript.text)
    elif transcript.status == "error":
        print("Error: ", transcript.error)
```

Open threads to transcribe each file concurrently. Once all the threads are complete you will receive the "All transcriptions are complete" message in your terminal.

```python
threads = []
for filename in os.listdir(batch_folder):
    thread = threading.Thread(target=transcribe_audio, args=(filename,))
    threads.append(thread)
    thread.start()

for thread in threads:
    thread.join()

print("All transcriptions are complete.")
```

## Conclusion

This guide aims to demonstrate how to use AssemblyAI Python SDK to concurrently process multiple audio files at once. The output is transcript text files for each audio file in the specified folder.

Other integrations and features can be built on top of this main function. These include and are not limited to: exporting the file in different formats, adding Core Transcription or Speech Understanding features.

If you have any questions, please feel free to reach out to our Support team at [support@assemblyai.com](mailto:support@assemblyai.com).

---

<a id="transcribe-from-an-s3-bucket-docs-pre-recorded-audio-guides-transcribe-from-s3-mdx"></a>

# Transcribe from an S3 Bucket

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/transcribe_from_s3
Source: docs/pre-recorded-audio/guides/transcribe_from_s3.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Hosting audio files
Description: Transcribe from an S3 Bucket documentation.

AssemblyAI's Speech-to-Text APIs can be used with both local files and publicly accessible online files, but what if you want to transcribe an audio file that has restricted access? Luckily, you can do this with AssemblyAI too!

Read on to learn how you can transcribe an audio file stored in an AWS S3 bucket using AssemblyAI's APIs.

## Intro

In order to transcribe an audio file from an S3 bucket, AssemblyAI will need temporary access to the file. To provide this access, we'll generate a **presigned URL**, which is simply a URL that has temporary access rights baked-in.

The overall process looks like this:

1. Generate a presigned URL for the S3 audio file with [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html).
2. Pass this URL through to AssemblyAI's API with a POST request.
3. Wait until the transcription is complete, and then fetch it with a GET request.

## Prerequisites

First, you'll need an AssemblyAI account. You can [sign up for a free account](https://www.assemblyai.com/dashboard/signup) if you don't already have one.

Next, you'll need to **take note of your AssemblyAI API key**, which you can find on your [account dashboard](https://www.assemblyai.com/dashboard/home) after signing in. It will be on the left-hand side of the screen under _Your API Key_.

You'll need the value of this key later, so leave the browser window open or copy the value into a text file.

## AWS IAM User

Second, you'll need an AWS IAM user with `Programmatic` access and the `AmazonS3ReadOnlyAccess` permission. If you already have such an IAM user and you know its public and private keys, then you can move on to the next section. Otherwise, create one now as follows:

First, log into AWS as a root user or as another IAM user with the appropriate access, and then go to the [IAM Management Console](https://us-east-1.console.aws.amazon.com/iamv2/home?ref=assemblyai.com#/users) to add a new user.

<img src="/assets/pages/guides/cookbooks/images/transcribe_from_s3/img1.png" alt="AWS IAM Users list page with the Add users button highlighted" />

Set the user name you would like, and select _Programmatic access_ under _Select AWS access type:_

<img src="/assets/pages/guides/cookbooks/images/transcribe_from_s3/img2.png" alt="IAM Add user step setting the user name and selecting Access key - Programmatic access" />

Click _Next_, and then _Attach existing policies directly_. Copy and paste _AmazonS3ReadOnlyAccess_ into the _Filter policies_ search box, and then add this permission by clicking on the checkbox next to it:

<img src="/assets/pages/guides/cookbooks/images/transcribe_from_s3/img3.png" alt="IAM Set permissions step with Attach existing policies directly chosen and AmazonS3ReadOnlyAccess checked" />

Click _Next_ and add tags if you wish. Then click _Next_ and review the IAM user profile to ensure that everything looks copacetic before clicking _Create user_.

<img src="/assets/pages/guides/cookbooks/images/transcribe_from_s3/img4.png" alt="IAM Add user Review screen showing user details and the AmazonS3ReadOnlyAccess policy before creating" />

Finally, take note of the IAM user's _Access key ID_ and _Secret access key_. Again, **we will need these values later**, so copy them into a text file before moving on.

**Warning**
Make sure to copy the IAM user's _Secret access key_ and record it somewhere safe. Once you close the final window of the _Add user_ sequence, you will not be able to access this key again and will need to regenerate it if you forget/lose the original.

## Code

First, the necessary packages are installed.

```bash
pip install -U boto3 botocore
```

Then we can import them and set our relevant variable values. You'll need to edit these variables to be equivalent to the relevant values for your application:

1. `bucket_name` - The name of your AWS S3 bucket.
2. `object_name` - The name of the audio file in the S3 bucket that you want to transcribe.
3. `iam_access_id` - The access ID of the IAM user with programmatic access and S3 read permission.
4. `iam_secret_key` - The secret key of the IAM user.
5. `assembly_key` - Your AssemblyAI API key.

```python
import boto3
from botocore.exceptions import ClientError
import logging
import requests
import time

bucket_name = "<BUCKET_NAME>"
object_name = "<AUDIO_FILE_NAME>"

iam_access_id = "<IAM_ACCESS_ID>"
iam_secret_key = "<IAM_SECRET_KEY>"

assembly_key = "<ASSEMBLYAI_API_KEY>"
```

From here, we simply follow the sequence outlined in the introduction of this Colab:

1. Generate a presigned URL for the S3 audio file with [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html).

```python
# Create a low-level service client with the IAM credentials.
s3_client = boto3.client(
    "s3", aws_access_key_id=iam_access_id, aws_secret_access_key=iam_secret_key
)

# Generate a pre-signed URL for the audio file that expires after 30 minutes.
try:
    p_url = s3_client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": bucket_name, "Key": object_name},
        ExpiresIn=1800,
    )

except ClientError as e:
    logging.error(e)
```

2. Pass the presigned URL through to AssemblyAI's API with a POST request.

```python
# Use your AssemblyAI API Key for authorization.
headers = {"authorization": assembly_key, "content-type": "application/json"}

# Specify AssemblyAI's transcription API endpoint.
transcript_endpoint = "https://api.assemblyai.com/v2/transcript"

# Use the presigned URL as the `audio_url` in the POST request.
json = {"audio_url": p_url}

# Queue the audio file for transcription with a POST request.
post_response = requests.post(transcript_endpoint, json=json, headers=headers)
```

3. Wait until the transcription is complete, and then fetch it with a GET request.

```python
# Specify the endpoint of the transaction.
get_endpoint = transcript_endpoint + "/" + post_response.json()["id"]

# GET request the transcription.
get_response = requests.get(get_endpoint, headers=headers)

# If the transcription has not finished, wait util it has.
while get_response.json()["status"] != "completed":
    get_response = requests.get(get_endpoint, headers=headers)
    time.sleep(5)

# Once the transcription is complete, print it out.
print(get_response.json()["text"])
```

---

<a id="transcribe-google-drive-files-docs-pre-recorded-audio-guides-transcribing-google-drive-file-mdx"></a>

# Transcribe Google Drive Files

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/transcribing-google-drive-file
Source: docs/pre-recorded-audio/guides/transcribing-google-drive-file.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Hosting audio files
Description: Transcribe Google Drive Files documentation.

## **Step 1: Upload Your Audio File to Google Drive**

- **File Requirements**: Ensure your audio file is smaller than 100MB, as files larger than this cannot be directly downloaded from Google Drive links.
- **Uploading**: Log into your Google Drive account and upload the audio file you want to use.

## **Step 2: Make Your File Publicly Accessible**

- **Right-Click** on the uploaded file in Google Drive.
- Select **'Get Link'**.
- Change the setting from “Restricted” to “Anyone with the link”. This makes the file publicly accessible.

## **Step 3: Obtain the Downloadable URL**

- Click on `Copy link` to copy your shared link.
- Initially, the shared link will look something like this: https://drive.google.com/file/d/1YvY3gX-4ZwY7K4r3J0THKNTvvolB3D-S/view?usp=sharing.
- To make it a downloadable link, modify it to this format:  
   `https://drive.google.com/u/0/uc?id=FILE_ID&export=download`.
- **Example**: If your shared link is `https://drive.google.com/file/d/1YvY3gX-4ZwY7K4r3J0THKNTvvolB3D-S/view?usp=sharing`,  
   change it to `https://drive.google.com/u/0/uc?id=1YvY3gX-4ZwY7K4r3J0THKNTvvolB3D-S&export=download`.

<img src="/assets/pages/guides/cookbooks/core-transcription/google_drive_screenshot.png" alt="Google Drive Share dialog for audio.mp3 with General access set to Anyone with the link and a Copy link button" />

## **Step 4: Use the URL with AssemblyAI**

- Now, you can use this downloadable link in your AssemblyAI API request. This URL directly points to your audio file, allowing AssemblyAI to access and process it.

```python
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()

audio_url = (
"https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries.mp3"
)

transcript = transcriber.transcribe(audio_url, config)
```

## **Notes**

- **Security**: Ensure that sharing your audio file publicly complies with your privacy and security policies.
- If you prefer not to share your file publicly, you can [upload your file to our servers instead.](/api-reference/files/upload)
- **File Format**: Check that your audio file is in a format supported by AssemblyAI.

---

<a id="transcribe-github-files-docs-pre-recorded-audio-guides-transcribing-github-files-mdx"></a>

# Transcribe GitHub Files

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/transcribing-github-files
Source: docs/pre-recorded-audio/guides/transcribing-github-files.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Hosting audio files
Description: Transcribe GitHub Files documentation.

## Step 1: Upload Your Audio Files to a Public GitHub Repository

- **File Requirements**: GitHub has a file size limit of 100MB so ensure your audio files are 100MB in size or less. The files must be in a public repository otherwise you will receive an error saying the file is not publicly accessible. For a more secure way to host files check out our [Transcribing from an S3 Bucket Cookbook](/pre-recorded-audio/guides/transcribe_from_s3).

## Step 2: Obtain the Raw Audio URL from GitHub

1. Navigate to the repository that houses the audio file.
2. Click on the audio file. On the next page, right-click the "View raw" link and select "copy the link address" from the context menu.

Downloadable file URLs are formatted as `"https://github.com/<github-username>/<repo-name>/raw/<branch-name>/<file-name-and extension>"`

## Step 3: Add the Audio URL to your Request

POST `v2/transcript` endpoint

```JSON
{
    "audio_url":"https://github.com/user/audio-files/raw/main/audio.mp3"
}
```

Python SDK

```python
config = aai.TranscriptionConfig()
transcript = transcriber.transcribe("https://github.com/user/audio-files/raw/main/audio.mp3", config)
```

JavaScript SDK

```javascript
const transcript = await client.transcripts.transcribe({
  audio_url: "https://github.com/user/audio-files/raw/main/audio.mp3"
});
```

## **Resources**

[AssemblyAI's Supported File Types](/faq/what-audio-and-video-file-types-are-supported-by-your-api) <br/>
[Transcribe an Audio File](/pre-recorded-audio/getting-started/transcribe-an-audio-file)

---

<a id="iterate-over-speaker-labels-with-make-com-docs-pre-recorded-audio-guides-make-speaker-labels-mdx"></a>

# Iterate over Speaker Labels with Make.com

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/make-speaker-labels
Source: docs/pre-recorded-audio/guides/make-speaker-labels.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Speaker labels
Description: Iterate over Speaker Labels with Make.com documentation.

## Introduction

This is a quick guide on how to iterate over speaker labels in Make.com. This guide will return speaker labels as a readable format to a Google Doc. The end result will look like the two images below.

### Make.com Scenario:

![Image of full Make Sceanrio](/assets/img/cookbooks/makespeakers/make-scenario.png)

### Google Doc Transcript:

![Image of Final Transcript](/assets/img/cookbooks/makespeakers/make-final-transcript.png)

## Instructions

### Step 1: Transcribe the Audio

Create a scenario in Make.com. Add a new module. Search for and select the AssemblyAI app and select the "Transcribe an Audio File" module. Add an audio URL. Select speaker labels and other models you’d like to run.

![Transcribe Module](/assets/img/cookbooks/makespeakers/make-transcribe-audio.png)

### Click Run once to retrieve data.

![Transcribe Module](/assets/img/cookbooks/makespeakers/make-run.png)

### Step 2: Wait for Completion

Next, add the “Wait until Transcript is Ready” module.

![Wait for completion module](/assets/img/cookbooks/makespeakers/make-wait-for-completion.png)

### Select "ID" from the “Transcribe an Audio” module as input for the Transcript ID field.

![Returned data](/assets/img/cookbooks/makespeakers/make-get-id.png)

### Step 3: Get a Transcript

Next, add the “Get a Transcript” module and select "ID" from the “Transcribe an Audio” module as input for the Transcript ID field.

![Get transcript module data](/assets/img/cookbooks/makespeakers/make-get-transcript.png)

### Step 4: Create a Document

Search for and add the Google Docs app. From there choose the “Create a Document” module. Connect your Google account and name the Doc what you’d like. Add some filler content as well. Additionally, choose where you’d like the Doc to be located.

![Create Google doc module](/assets/img/cookbooks/makespeakers/make-create-doc.png)

### Step 5: Iterator Tool

Add the Iterator tool next. The speaker label data is in the utterances array. Select that array as input from the “Transcribe an Audio File" module. This tool will be used to perform a task for each utterance in the array. The next module will repeat its action for each utterance.

![Iterator module](/assets/img/cookbooks/makespeakers/make-iterator.png)

### Step 6: Write Speaker Labels Data to Google Doc

Add a module and choose the “Insert a Paragraph” module from Google Docs. Connect your Google account if it’s not already connected (you may have to reconnect it if you get a failed to load error). In the “Select a Document” drop-down, choose "by mapping". In the Document ID input field select document ID from the “Create a document” module. For appended text, you can follow the format below for a readable format.

![Insert paragrapgh module](/assets/img/cookbooks/makespeakers/make-insert-paragraph.png)

### Step 7: Run the Scenario.

Run the scenario and you should get a Google Doc in your Drive with the speaker labels included.

![Image of Final Transcript](/assets/img/cookbooks/makespeakers/make-final-transcript.png)

---

<a id="calculate-the-talk-listen-ratio-of-speakers-docs-pre-recorded-audio-guides-talk-listen-ratio-mdx"></a>

# Calculate the Talk / Listen Ratio of Speakers

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/talk-listen-ratio
Source: docs/pre-recorded-audio/guides/talk-listen-ratio.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Speaker labels
Description: Calculate the Talk / Listen Ratio of Speakers documentation.

This guide will show you how to use AssemblyAI's API to calculate the talk/listen ratio of speakers in a transcript. The following code uses the [Python SDK](https://github.com/AssemblyAI/assemblyai-python-sdk).

## Quickstart

```python expandable
import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"

def calculate_talk_listen_ratios(transcript):
    """
    :param transcript: AssemblyAI Transcript object
    :return: Dictionary with talk time, percentage, and talk-listen ratios for each speaker
    """
    # Ensure speaker labels were enabled
    if not transcript.utterances:
        raise ValueError("Speaker labels were not enabled for this transcript.")

    speaker_talk_time = {}
    total_time = 0

    for utterance in transcript.utterances:
        speaker = f"Speaker {utterance.speaker}"
        duration = utterance.end - utterance.start

        speaker_talk_time[speaker] = speaker_talk_time.get(speaker, 0) + duration
        total_time += duration

    # Calculate percentages and ratios
    result = {}
    for speaker, talk_time in speaker_talk_time.items():
        percentage = (talk_time / total_time) * 100
        result[speaker] = {
            "talk_time_ms": talk_time,
            "percentage": round(percentage, 2)
        }

    # Calculate talk-listen ratios for each speaker against all others
    for speaker in result.keys():
        other_speakers_time = sum(talk_time for spk, talk_time in speaker_talk_time.items() if spk != speaker)
        if other_speakers_time > 0:
            ratio = speaker_talk_time[speaker] / other_speakers_time
            result[speaker]["talk_listen_ratio"] = round(ratio, 2)
        else:
            result[speaker]["talk_listen_ratio"] = None  # Handle cases with only one speaker

    return result

transcriber = aai.Transcriber()
audio_url = ("YOUR_AUDIO_URL")
config = aai.TranscriptionConfig(speaker_labels=True)
transcript = transcriber.transcribe(audio_url, config)

talk_listen_stats = calculate_talk_listen_ratios(transcript)
print(talk_listen_stats)

```

## Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-Step Instructions

Install the SDK:

```bash
pip install assemblyai
```

Import the `assemblyai` package and set the API key.

```python
import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"
```

Define a function called `calculate_talk_listen_ratios`, which will calculate the talk-listen ratios for all speakers from a transcript with speaker labels. Speaker labels must be enabled for the ratios to be calculated.

```python expandable
def calculate_talk_listen_ratios(transcript):
    """
    :param transcript: AssemblyAI Transcript object
    :return: Dictionary with talk time, percentage, and talk-listen ratios for each speaker
    """
    # Ensure speaker labels were enabled
    if not transcript.utterances:
        raise ValueError("Speaker labels were not enabled for this transcript.")

    speaker_talk_time = {}
    total_time = 0

    for utterance in transcript.utterances:
        speaker = f"Speaker {utterance.speaker}"
        duration = utterance.end - utterance.start

        speaker_talk_time[speaker] = speaker_talk_time.get(speaker, 0) + duration
        total_time += duration

    # Calculate percentages and ratios
    result = {}
    for speaker, talk_time in speaker_talk_time.items():
        percentage = (talk_time / total_time) * 100
        result[speaker] = {
            "talk_time_ms": talk_time,
            "percentage": round(percentage, 2)
        }

    # Calculate talk-listen ratios for each speaker against all others
    for speaker in result.keys():
        other_speakers_time = sum(talk_time for spk, talk_time in speaker_talk_time.items() if spk != speaker)
        if other_speakers_time > 0:
            ratio = speaker_talk_time[speaker] / other_speakers_time
            result[speaker]["talk_listen_ratio"] = round(ratio, 2)
        else:
            result[speaker]["talk_listen_ratio"] = None  # Handle cases with only one speaker

    return result
```

Define a `transcriber`, an `audio_url` set to a link to the audio file (replace the example link that is provided with your own), and a `TranscriptionConfig` with `speaker_labels=True`. Then create a transcript which will be sent to the function `calculate_talk_listen_ratios` and print out the results.

```python
transcriber = aai.Transcriber()
audio_url = ("https://api.assemblyai-solutions.com/storage/v1/object/public/dual-channel-phone-data/Fisher_Call_Centre/audio05851.wav")
config = aai.TranscriptionConfig(speaker_labels=True)
transcript = transcriber.transcribe(audio_url, config)

talk_listen_stats = calculate_talk_listen_ratios(transcript)
print(talk_listen_stats)
```

Example output when using the above sample audio file:

```
{'Speaker A': {'talk_time_ms': 244196, 'percentage': 42.77, 'talk_listen_ratio': 0.75}, 'Speaker B': {'talk_time_ms': 326766, 'percentage': 57.23, 'talk_listen_ratio': 1.34}}
```

---

<a id="plot-a-speaker-timeline-with-matplotlib-docs-pre-recorded-audio-guides-speaker-timeline-mdx"></a>

# Plot A Speaker Timeline with Matplotlib

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/speaker_timeline
Source: docs/pre-recorded-audio/guides/speaker_timeline.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Speaker labels
Description: Plot A Speaker Timeline with Matplotlib documentation.

In this guide, we'll show you how to plot a speaker timeline with matplotlib, using results from the speaker diarization model.

## Quickstart

```python expandable
import assemblyai as aai
import matplotlib.pyplot as plt

aai.settings.api_key = "YOUR_API_KEY"

config = aai.TranscriptionConfig(speaker_labels=True)
transcriber = aai.Transcriber()
transcript = transcriber.transcribe("./my-audio.mp3", config)
utterances = transcript.utterances

def plot_speaker_timeline(utterances):
    fig, ax = plt.subplots(figsize=(12, 4))
    colors = ['b', 'g', 'r', 'c', 'm', 'y', 'k']
    speaker_colors = {}

    for utterance in utterances:
        start = utterance.start / 60000 # in minutes
        end = utterance.end / 60000 # in minutes
        speaker = utterance.speaker

        if speaker not in speaker_colors:
            speaker_colors[speaker] = colors[len(speaker_colors) % len(colors)] # set a colour for each new speaker

        ax.barh(speaker, end - start, left=start, color=speaker_colors[speaker], height=0.4) # create horizontal bar plot

    ax.set_xlabel('Time (mins)')
    ax.set_ylabel('Speakers')
    ax.set_title('Speaker Timeline')
    ax.grid(True, which='both', linestyle='--', linewidth=0.5)
    plt.show()

plot_speaker_timeline(utterances)
```

### Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

### Step-by-Step Instructions

Install the SDK.

```bash
pip install -U assemblyai
!pip install -U matplotlib
```

Import the `assemblyai` package and set the API key.

```python
import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"
```

Create a `TranscriptionConfig` object and set speaker labels to `True`.

```python
config = aai.TranscriptionConfig(speaker_labels=True)
```

Create a `Transcriber` object.

```python
transcriber = aai.Transcriber()
```

Use the Transcriber object's `transcribe` method and pass in the audio file's path and `config` object as parameters. The transcribe method saves the results of the transcription to the `Transcriber` object's `transcript` attribute.

```python
transcript = transcriber.transcribe("./my-audio.mp3", config)
```

 Alternatively, you can use an audio URL available on the internet. 

Extract the utterances from the transcript and set this to `utterances`.

```python
utterances = transcript.utterances
```

Import the `matplotlib.pyplot` library. Then use the following `plot_speaker_timeline` function which results in a plot image of the speaker timeline. This function extracts the `start` and `end` timestamps of each `utterance` per `speaker` and plots the data onto the horizontal bar chart. The X and Y axis are labelled accordingly.

```python expandable
import matplotlib.pyplot as plt

def plot_speaker_timeline(utterances):
    fig, ax = plt.subplots(figsize=(12, 4))
    colors = ['b', 'g', 'r', 'c', 'm', 'y', 'k']
    speaker_colors = {}

    for utterance in utterances:
        start = utterance.start / 60000 # in minutes
        end = utterance.end / 60000 # in minutes
        speaker = utterance.speaker

        if speaker not in speaker_colors:
            speaker_colors[speaker] = colors[len(speaker_colors) % len(colors)] # set a colour for each new speaker

        ax.barh(speaker, end - start, left=start, color=speaker_colors[speaker], height=0.4) # create horizontal bar plot

    ax.set_xlabel('Time (mins)')
    ax.set_ylabel('Speakers')
    ax.set_title('Speaker Timeline')
    ax.grid(True, which='both', linestyle='--', linewidth=0.5)
    plt.show()
```

Finally, call the `plot_speaker_timeline` function passing `utterances` as a parameter to see the plot image result.

```python
plot_speaker_timeline(utterances)
```

---

<a id="generate-custom-speaker-labels-with-pyannote-docs-pre-recorded-audio-guides-use-assemblyai-with-pyannote-to-generate-custom-speaker-labels-mdx"></a>

# Generate Custom Speaker Labels with Pyannote

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/Use_AssemblyAI_with_Pyannote_to_generate_custom_Speaker_Labels
Source: docs/pre-recorded-audio/guides/Use_AssemblyAI_with_Pyannote_to_generate_custom_Speaker_Labels.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Speaker labels
Description: Generate Custom Speaker Labels with Pyannote documentation.

In this guide, we'll show you how to generate Speaker Labels using Pyannote with an AssemblyAI transcript. This can be used to generate Speaker Labels for languages we currently do not support for speaker labelling.

## Quickstart

```python expandable
import os
import assemblyai as aai
from pyannote.audio import Pipeline
import torch
import pandas as pd
import numpy as np

# Assign your API keys
HUGGING_FACE_TOKEN = os.getenv("HF_TOKEN")
ASSEMBLYAI_API_KEY = os.getenv("ASSEMBLYAI_API_KEY")

# Authenticate with AssemblyAI
aai.settings.api_key = ASSEMBLYAI_API_KEY

def transcribe_audio(audio_file, language="en"):
    """
    Transcribe an audio file using AssemblyAI.

    Args:
        audio_file (str): Path to the audio file.
        language (str, optional): Language code for transcription. Defaults to "en".

    Returns:
        aai.Transcript: The transcription result.
    """

    transcriber = aai.Transcriber(config=aai.TranscriptionConfig(language_code=language))
    transcript = transcriber.transcribe(audio_file)
    print(f"Transcript ID: {transcript.id}")
    return transcript

def get_speaker_labels(audio_file, transcript: aai.Transcript):
    """
    Perform speaker diarization on an audio file and combine results with the transcript.

    Args:
        audio_file (str): Path to the audio file.
        transcript (aai.Transcript): The transcription result from AssemblyAI.

    Returns:
        str: A formatted string containing the transcript with speaker labels and timestamps.
    """
    # Initialize the speaker diarization pipeline with GPU support
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    pipeline = Pipeline.from_pretrained(
        "pyannote/speaker-diarization",
        use_auth_token=HUGGING_FACE_TOKEN,
    )

    if pipeline is None:
        raise ValueError("Failed to initialize the pipeline. Please check your authentication token and internet connection.")
    else:
        pipeline = pipeline.to(device)

    # Apply the pipeline to the audio file
    diarization = pipeline(audio_file)

    # Create a dictionary to store speaker segments
    speaker_segments = {}

    # Process diarization results
    for turn, _, speaker in diarization.itertracks(yield_label=True):
        start, end = turn.start, turn.end
        if speaker not in speaker_segments:
            speaker_segments[speaker] = []
        speaker_segments[speaker].append((start, end))

    # Convert speaker_segments to a DataFrame
    diarize_df = pd.DataFrame([(speaker, start, end)
                               for speaker, segments in speaker_segments.items()
                               for start, end in segments],
                              columns=['speaker', 'start', 'end'])

    # Assign speakers to transcript words
    for word in transcript.words:
        word_start = float(word.start) / 1000
        word_end = float(word.end) / 1000

        overlaps = diarize_df[
            (diarize_df['start'] <= word_end) & (diarize_df['end'] >= word_start)
        ].copy()

        if not overlaps.empty:
            overlaps['overlap'] = np.minimum(overlaps['end'], word_end) - np.maximum(overlaps['start'], word_start)
            word.speaker = overlaps.loc[overlaps['overlap'].idxmax(), 'speaker']
        else:
            word.speaker = "Unknown"

    full_transcript = ''

    # Update segment speakers based on the majority speaker of its words
    for segment in transcript.get_sentences():
        segment_start = float(segment.start) / 1000
        segment_end = float(segment.end) / 1000

        overlaps = diarize_df[
            (diarize_df['start'] <= segment_end) & (diarize_df['end'] >= segment_start)
        ].copy()

        if not overlaps.empty:
            overlaps['overlap'] = np.minimum(overlaps['end'], segment_end) - np.maximum(overlaps['start'], segment_start)
            segment.speaker = overlaps.loc[overlaps['overlap'].idxmax(), 'speaker']
            speaker_label = segment.speaker.replace('SPEAKER_', 'SPEAKER ')
            full_transcript += f'[{format_timestamp(segment_start)}] {speaker_label}: {segment.text}\n'
        else:
            segment.speaker = "Unknown"
            full_transcript += f'[{format_timestamp(segment_start)}] Unknown: {segment.text}\n'

    return full_transcript

def format_timestamp(seconds):
    """
    Convert seconds to a formatted timestamp string (HH:MM:SS).

    Args:
        seconds (float): Time in seconds.

    Returns:
        str: Formatted timestamp string.
    """
    hours, remainder = divmod(int(seconds), 3600)
    minutes, seconds = divmod(remainder, 60)
    return f"{hours:02d}:{minutes:02d}:{seconds:02d}"

audio_file = "audio.wav" # your local file path
transcript: aai.Transcript = transcribe_audio(audio_file, language="hr") # select a language code
transcript_with_speakers = get_speaker_labels(audio_file, transcript)
print(transcript_with_speakers)
```

### Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

You'll also need a HuggingFace account and API key. You can [sign up](https://huggingface.co/join) for a free account and get your API key [here](https://huggingface.co/settings/tokens). Create a **Read** type API token to ensure the necessary permissions are enabled.

  Browse to the
  [speaker-diarization](https://huggingface.co/pyannote/speaker-diarization) and
  [segmentation](https://huggingface.co/pyannote/segmentation) model pages and
  accept the **Gated Model** Terms & Conditions by entering your
  **Company/University**, **Website** and **Use Case** details in order to gain
  access to the use of these models.

### Step-by-Step Instructions

Install the necessary dependencies.

```bash
pip install assemblyai pyannote.audio torch pandas numpy
```

Import the necessary dependencies, assign your API keys and authenticate with AssemblyAI.

```python
import os
import assemblyai as aai
from pyannote.audio import Pipeline
import torch
import pandas as pd
import numpy as np

# Assign your API keys
HUGGING_FACE_TOKEN = os.getenv("HF_TOKEN")
ASSEMBLYAI_API_KEY = os.getenv("ASSEMBLYAI_API_KEY")

# Authenticate with AssemblyAI
aai.settings.api_key = ASSEMBLYAI_API_KEY
```

Create the `transcribe_audio` function, this will handle the transcription process with AssemblyAI.

```python
def transcribe_audio(audio_file, language="en"):
    """
    Transcribe an audio file using AssemblyAI.

    Args:
        audio_file (str): Path to the audio file.
        language (str, optional): Language code for transcription. Defaults to "en".

    Returns:
        aai.Transcript: The transcription result.
    """

    transcriber = aai.Transcriber(config=aai.TranscriptionConfig(language_code=language))
    transcript = transcriber.transcribe(audio_file)
    print(f"Transcript ID: {transcript.id}")
    return transcript
```

Create the `get_speaker_labels`function, this will handle the speaker diarization model processing to generate the custom speaker labels for the transcript.

Firstly, it initializes and applies the pipeline to the audio file.

Secondly, it processes the diarization results and converts the speaker segments into a DataFrame so we can compare the results with the transcript.

Lastly, the speaker segments are compared and assigned to the words and sentences of the transcript to create the speaker labelled transcript.

```python expandable
def get_speaker_labels(audio_file, transcript: aai.Transcript):
    """
    Perform speaker diarization on an audio file and combine results with the transcript.

    Args:
        audio_file (str): Path to the audio file.
        transcript (aai.Transcript): The transcription result from AssemblyAI.

    Returns:
        str: A formatted string containing the transcript with speaker labels and timestamps.
    """
    # Initialize the speaker diarization pipeline with GPU support
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    pipeline = Pipeline.from_pretrained(
        "pyannote/speaker-diarization",
        use_auth_token=HUGGING_FACE_TOKEN,
    )

    if pipeline is None:
        raise ValueError("Failed to initialize the pipeline. Please check your authentication token and internet connection.")
    else:
        pipeline = pipeline.to(device)

    # Apply the pipeline to the audio file
    diarization = pipeline(audio_file)

    # Create a dictionary to store speaker segments
    speaker_segments = {}

    # Process diarization results
    for turn, _, speaker in diarization.itertracks(yield_label=True):
        start, end = turn.start, turn.end
        if speaker not in speaker_segments:
            speaker_segments[speaker] = []
        speaker_segments[speaker].append((start, end))

    # Convert speaker_segments to a DataFrame
    diarize_df = pd.DataFrame([(speaker, start, end)
                               for speaker, segments in speaker_segments.items()
                               for start, end in segments],
                              columns=['speaker', 'start', 'end'])

    # Assign speakers to transcript words
    for word in transcript.words:
        word_start = float(word.start) / 1000
        word_end = float(word.end) / 1000

        overlaps = diarize_df[
            (diarize_df['start'] <= word_end) & (diarize_df['end'] >= word_start)
        ].copy()

        if not overlaps.empty:
            overlaps['overlap'] = np.minimum(overlaps['end'], word_end) - np.maximum(overlaps['start'], word_start)
            word.speaker = overlaps.loc[overlaps['overlap'].idxmax(), 'speaker']
        else:
            word.speaker = "Unknown"

    full_transcript = ''

    # Update segment speakers based on the majority speaker of its words
    for segment in transcript.get_sentences():
        segment_start = float(segment.start) / 1000
        segment_end = float(segment.end) / 1000

        overlaps = diarize_df[
            (diarize_df['start'] <= segment_end) & (diarize_df['end'] >= segment_start)
        ].copy()

        if not overlaps.empty:
            overlaps['overlap'] = np.minimum(overlaps['end'], segment_end) - np.maximum(overlaps['start'], segment_start)
            segment.speaker = overlaps.loc[overlaps['overlap'].idxmax(), 'speaker']
            speaker_label = segment.speaker.replace('SPEAKER_', 'SPEAKER ')
            full_transcript += f'[{format_timestamp(segment_start)}] {speaker_label}: {segment.text}\n'
        else:
            segment.speaker = "Unknown"
            full_transcript += f'[{format_timestamp(segment_start)}] Unknown: {segment.text}\n'

    return full_transcript
```

If you know the number of speakers in advance, you can use the `num_speakers` parameter to set the number of speakers:

```python
# Apply the pipeline to the audio file
diarization = pipeline(audio_file, num_speakers=4)
```

You can also provide upper/lower bands on the number of speakers using the `min_speakers` and `max_speakers` parameters:

```python
# Apply the pipeline to the audio file
diarization = pipeline(audio_file, min_speakers=2, max_speakers=5)
```

Create the `format_timestamp`, this will handle the timestamps conversion to improve the readability of the final speaker labelled transcript.

```python
def format_timestamp(seconds):
    """
    Convert seconds to a formatted timestamp string (HH:MM:SS).

    Args:
        seconds (float): Time in seconds.

    Returns:
        str: Formatted timestamp string.
    """
    hours, remainder = divmod(int(seconds), 3600)
    minutes, seconds = divmod(remainder, 60)
    return f"{hours:02d}:{minutes:02d}:{seconds:02d}"
```

Finally, select a local file and call the functions to generate and print your custom Speaker Labelled transcript.

```python
audio_file = "audio.wav" # your local file path
transcript: aai.Transcript = transcribe_audio(audio_file, language="hr") # select a language code
transcript_with_speakers = get_speaker_labels(audio_file, transcript)
print(transcript_with_speakers)
```

Here's an example speaker labelled output from a Croatian file:

```
[00:00:05] SPEAKER 04: Nalazimo se u Centro Zagreba, u parku Zrinjevac, gdje je kao što vidite jako ljepo, vreme je prekrasno, a danas ćemo ljude pitati što im se sviđa u Zagrebu ili što im se možda ne sviđa u Zagrebu.

[00:00:42] SPEAKER 04: Dobar dan, može jednokratko pitanje samo.

[00:00:46] SPEAKER 04: Može?

[00:00:48] SPEAKER 04: Evo lako, što vam se najviše sviđa u Zagrebu?

[00:00:50] SPEAKER 07: Što mi se najviše sviđa u Zagrebu?

[00:00:53] SPEAKER 07: E sad, teško pitanje, ali trenutno mi se najviše sviđa što nije klasična jesen, nego više prođeče u zraku.

[00:01:06] SPEAKER 07: Dobre.

[00:01:09] SPEAKER 07: Može sigurnost još uvijek s osišam sigurno u Zagrebu.

[00:01:13] SPEAKER 04: I po noći?

[00:01:15] SPEAKER 07: Pa po noći ne šetam baš toliko po noći, ali centar grada mi je dosta siguran, osvijetljen i to mi je okej.
```

---

<a id="use-speaker-diarization-with-async-chunking-docs-pre-recorded-audio-guides-speaker-diarization-with-async-chunking-mdx"></a>

# Use Speaker Diarization with Async Chunking

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/speaker-diarization-with-async-chunking
Source: docs/pre-recorded-audio/guides/speaker-diarization-with-async-chunking.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Speaker labels
Description: Use Speaker Diarization with Async Chunking documentation.

This guide uses AssemblyAI and [Nvidia's NeMo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_large) framework. We'll be using TitaNet, a state of the art open source model that is trained for speaker recognition tasks. TitaNet will allow us to generate audio embeddings for speakers, which can be used to identify semantic similarity matches between two speakers.

## Quickstart

```python expandable
import assemblyai as aai
import requests
import json
import time
import requests
import copy
from pydub import AudioSegment
import os
import nemo.collections.asr as nemo_asr
from pydub import AudioSegment

speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")

assemblyai_key = "YOUR_API_KEY"

headers = {
    "authorization": assemblyai_key
}

def get_transcript(transcript_id):
    polling_endpoint = f"https://api.assemblyai.com/v2/transcript/{transcript_id}"

    while True:
        transcription_result = requests.get(polling_endpoint, headers=headers).json()

        if transcription_result['status'] == 'completed':
            # print("Transcript ID:", transcript_id)
            return(transcription_result)
            break

        elif transcription_result['status'] == 'error':
            raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

        else:
            time.sleep(3)

def download_wav(presigned_url, output_filename):
    # Download the WAV file from the presigned URL
    response = requests.get(presigned_url)
    if response.status_code == 200:
        print("downloading...")
        with open(output_filename, 'wb') as f:
            f.write(response.content)
        print("successfully downloaded file:", output_filename)
    else:
        raise Exception("Failed to download file, status code: {}".format(response.status_code))

# Function to identify the longest monologue of each speaker from each clip
# you pass in the utterances and it returns the longest monologue from each speaker on that file
def find_longest_monologues(utterances):
    longest_monologues = {}
    current_monologue = {}
    last_speaker = None  # Track the last speaker to identify interruptions

    for utterance in utterances:
        speaker = utterance['speaker']
        start_time = utterance['start']
        end_time = utterance['end']

        if speaker not in current_monologue:
            current_monologue[speaker] = {"start": start_time, "end": end_time}
            longest_monologues[speaker] = []
        else:
            # Extend monologue only if it's the same speaker speaking continuously
            if current_monologue[speaker]["end"] == start_time and last_speaker == speaker:
                current_monologue[speaker]["end"] = end_time
            else:
                monologue_length = current_monologue[speaker]["end"] - current_monologue[speaker]["start"]
                new_entry = (monologue_length, copy.deepcopy(current_monologue[speaker]))

                if len(longest_monologues[speaker]) < 1 or monologue_length > min(longest_monologues[speaker], key=lambda x: x[0])[0]:
                    if len(longest_monologues[speaker]) == 1:
                        longest_monologues[speaker].remove(min(longest_monologues[speaker], key=lambda x: x[0]))

                    longest_monologues[speaker].append(new_entry)

                current_monologue[speaker] = {"start": start_time, "end": end_time}

        last_speaker = speaker  # Update the last speaker

    # Check the last monologue for each speaker
    for speaker, monologue in current_monologue.items():
        monologue_length = monologue["end"] - monologue["start"]
        new_entry = (monologue_length, monologue)
        if len(longest_monologues[speaker]) < 1 or monologue_length > min(longest_monologues[speaker], key=lambda x: x[0])[0]:
            if len(longest_monologues[speaker]) == 1:
                longest_monologues[speaker].remove(min(longest_monologues[speaker], key=lambda x: x[0]))
            longest_monologues[speaker].append(new_entry)

    return longest_monologues

# Create clips of each long monologue and embed the clip
# you pass in the file path and the longest monologue objects returned by the find_longest_monologues function.
# This function will create new audio file clips which contain only the longest monologue from each speaker
def clip_and_store_utterances(audio_file, longest_monologues):
    # Load the full conversation audio
    full_audio = AudioSegment.from_wav(audio_file)
    full_audio = full_audio.set_channels(1)

    utterance_clips = []

    for speaker, monologues in longest_monologues.items():
        for _, monologue in monologues:
            start_ms = monologue['start']
            end_ms = monologue['end']
            clip = full_audio[start_ms:end_ms]
            clip_filename = f"{speaker}_monologue_{start_ms}_{end_ms}.wav"
            clip.export(clip_filename, format="wav")

            utterance_clips.append({
                'clip_filename': clip_filename,
                'start': start_ms,
                'end': end_ms,
                'speaker': speaker
            })

    print("Total Number of Monologue Clips Found: ", len(utterance_clips))

    return utterance_clips

# This function uses NeMO to compare two files
def compare_embeddings(utterance_clip, reference_file):
    verification_result = speaker_model.verify_speakers(
        utterance_clip,
        reference_file
    )
    return verification_result

file_one = "YOUR_FILE_1"
file_two = "YOUR_FILE_2"
file_three = "YOUR_FILE_3"

download_wav(file_one, "testone.wav")
download_wav(file_two, "testtwo.wav")
download_wav(file_three, "testthree.wav")

# Store utterances from each clip, keyed by clip index
clip_utterances = {}

# Dictionary to track known speaker identities across all clips
# Maps current clip speaker labels to a unified speaker label
speaker_identity_map = {}

def process_clips(clip_transcript_ids, audio_files):
    global clip_utterances, speaker_identity_map

    # This will store the longest clip filenames for each speaker from the previous clips
    previous_speaker_clips = {}

    for clip_index, (transcript_id, audio_file) in enumerate(zip(clip_transcript_ids, audio_files)):
        transcript = get_transcript(transcript_id)
        utterances = transcript['utterances']
        clip_utterances[clip_index] = utterances # Store utterances for the current clip

        longest_monologues = find_longest_monologues(utterances)

        # Process the longest monologues for clipping and storing
        current_speaker_clips = {}
        for speaker, monologue_data in longest_monologues.items():
            clip_and_store_utterances(audio_file, {speaker: monologue_data})
            longest_clip = f"{speaker}_monologue_{monologue_data[0][1]['start']}_{monologue_data[0][1]['end']}.wav"
            current_speaker_clips[speaker] = longest_clip

        if clip_index == 0:
            speaker_identity_map = {speaker: speaker for speaker in longest_monologues.keys()}
            previous_speaker_clips = current_speaker_clips.copy()
        else:
            # Compare all new speakers against all base speakers from previous clips
            for new_speaker, new_clip in current_speaker_clips.items():
                for base_speaker, base_clip in previous_speaker_clips.items():
                    if compare_embeddings(new_clip, base_clip):
                        speaker_identity_map[new_speaker] = base_speaker
                        break
                else:
                    # If no match is found, assign a new label
                    new_label = chr(ord(max(speaker_identity_map.values(), key=lambda x: ord(x))) + 1)
                    speaker_identity_map[new_speaker] = new_label

            # Update the previous_speaker_clips for the next iteration
            previous_speaker_clips.update(current_speaker_clips)

        # Update utterances with the new speaker labels for the current clip
        for utterance in clip_utterances[clip_index]:
            original_speaker = utterance['speaker']
            # Update only if there's a change in speaker identity
            if original_speaker in speaker_identity_map:
                utterance['speaker'] = speaker_identity_map[original_speaker]

# Add your clip transcript IDs
clip_transcript_ids = [
    "YOUR_TRANSCRIPT_ID_1",
    "YOUR_TRANSCRIPT_ID_2",
    "YOUR_TRANSCRIPT_ID_3"
]

# Add filepaths to your downloaded files
audio_files = [
    "/testone.wav",
    "/testtwo.wav",
    "/testthree.wav"
]

process_clips(clip_transcript_ids, audio_files)

def display_transcript(transcript_data):
    for clip_index, utterances in transcript_data.items():
        print(f"Clip {clip_index + 1}:")
        for utterance in utterances:
            speaker = utterance['speaker']
            text = utterance['text']
            print(f"  Speaker {speaker}: {text}")
        print("\n")  # Add an extra newline for spacing between

display_transcript(clip_utterances)
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-step instructions

### Install Dependencies

```bash
pip install pytorch
pip install nemo_toolkit['all']
pip install ffmpeg
pip install assemblyai
```

### AssemblyAI Setup, Transcript Setup, and Load the Model Using NeMO

In this section, we'll import dependencies and add functions to transcribe and store transcript IDs if needed.

```python
import assemblyai as aai
import requests
import json
import time
import requests
import copy
from pydub import AudioSegment
import os
import nemo.collections.asr as nemo_asr

speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")

assemblyai_key = "YOUR_API_KEY"

headers = {
    "authorization": assemblyai_key
}
```

### Helper Functions

The function below requests a transcript based on a transcript ID.

```python
def get_transcript(transcript_id):
    polling_endpoint = f"https://api.assemblyai.com/v2/transcript/{transcript_id}"

    while True:
        transcription_result = requests.get(polling_endpoint, headers=headers).json()

        if transcription_result['status'] == 'completed':
            # print("Transcript ID:", transcript_id)
            return(transcription_result)
            break

        elif transcription_result['status'] == 'error':
            raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

        else:
            time.sleep(3)
```

Our main inference function will make use of these functions:

```python expandable
def download_wav(presigned_url, output_filename):
    # Download the WAV file from the presigned URL
    response = requests.get(presigned_url)
    if response.status_code == 200:
        print("downloading...")
        with open(output_filename, 'wb') as f:
            f.write(response.content)
        print("successfully downloaded file:", output_filename)
    else:
        raise Exception("Failed to download file, status code: {}".format(response.status_code))

# Function to identify the longest monologue of each speaker from each clip
# you pass in the utterances and it returns the longest monologue from each speaker on that file
def find_longest_monologues(utterances):
    longest_monologues = {}
    current_monologue = {}
    last_speaker = None  # Track the last speaker to identify interruptions

    for utterance in utterances:
        speaker = utterance['speaker']
        start_time = utterance['start']
        end_time = utterance['end']

        if speaker not in current_monologue:
            current_monologue[speaker] = {"start": start_time, "end": end_time}
            longest_monologues[speaker] = []
        else:
            # Extend monologue only if it's the same speaker speaking continuously
            if current_monologue[speaker]["end"] == start_time and last_speaker == speaker:
                current_monologue[speaker]["end"] = end_time
            else:
                monologue_length = current_monologue[speaker]["end"] - current_monologue[speaker]["start"]
                new_entry = (monologue_length, copy.deepcopy(current_monologue[speaker]))

                if len(longest_monologues[speaker]) < 1 or monologue_length > min(longest_monologues[speaker], key=lambda x: x[0])[0]:
                    if len(longest_monologues[speaker]) == 1:
                        longest_monologues[speaker].remove(min(longest_monologues[speaker], key=lambda x: x[0]))

                    longest_monologues[speaker].append(new_entry)

                current_monologue[speaker] = {"start": start_time, "end": end_time}

        last_speaker = speaker  # Update the last speaker

    # Check the last monologue for each speaker
    for speaker, monologue in current_monologue.items():
        monologue_length = monologue["end"] - monologue["start"]
        new_entry = (monologue_length, monologue)
        if len(longest_monologues[speaker]) < 1 or monologue_length > min(longest_monologues[speaker], key=lambda x: x[0])[0]:
            if len(longest_monologues[speaker]) == 1:
                longest_monologues[speaker].remove(min(longest_monologues[speaker], key=lambda x: x[0]))
            longest_monologues[speaker].append(new_entry)

    return longest_monologues

# Create clips of each long monologue and embed the clip
# you pass in the file path and the longest monologue objects returned by the find_longest_monologues function.
# This function will create new audio file clips which contain only the longest monologue from each speaker
def clip_and_store_utterances(audio_file, longest_monologues):
    # Load the full conversation audio
    full_audio = AudioSegment.from_wav(audio_file)
    full_audio = full_audio.set_channels(1)

    utterance_clips = []

    for speaker, monologues in longest_monologues.items():
        for _, monologue in monologues:
            start_ms = monologue['start']
            end_ms = monologue['end']
            clip = full_audio[start_ms:end_ms]
            clip_filename = f"{speaker}_monologue_{start_ms}_{end_ms}.wav"
            clip.export(clip_filename, format="wav")

            utterance_clips.append({
                'clip_filename': clip_filename,
                'start': start_ms,
                'end': end_ms,
                'speaker': speaker
            })

    print("Total Number of Monologue Clips Found: ", len(utterance_clips))

    return utterance_clips

# This function uses NeMO to compare two files
def compare_embeddings(utterance_clip, reference_file):
    verification_result = speaker_model.verify_speakers(
        utterance_clip,
        reference_file
    )
    return verification_result
```

### Inference

Add the links to the WAV file clips you have stored on your server.

```python
file_one = "YOUR_FILE_1"
file_two = "YOUR_FILE_2"
file_three = "YOUR_FILE_3"

download_wav(file_one, "testone.wav")
download_wav(file_two, "testtwo.wav")
download_wav(file_three, "testthree.wav")
```

```python expandable
from pydub import AudioSegment

# Store utterances from each clip, keyed by clip index
clip_utterances = {}

# Dictionary to track known speaker identities across all clips
# Maps current clip speaker labels to a unified speaker label
speaker_identity_map = {}

def process_clips(clip_transcript_ids, audio_files):
    global clip_utterances, speaker_identity_map

    # This will store the longest clip filenames for each speaker from the previous clips
    previous_speaker_clips = {}

    for clip_index, (transcript_id, audio_file) in enumerate(zip(clip_transcript_ids, audio_files)):
        transcript = get_transcript(transcript_id)
        utterances = transcript['utterances']
        clip_utterances[clip_index] = utterances # Store utterances for the current clip

        longest_monologues = find_longest_monologues(utterances)

        # Process the longest monologues for clipping and storing
        current_speaker_clips = {}
        for speaker, monologue_data in longest_monologues.items():
            clip_and_store_utterances(audio_file, {speaker: monologue_data})
            longest_clip = f"{speaker}_monologue_{monologue_data[0][1]['start']}_{monologue_data[0][1]['end']}.wav"
            current_speaker_clips[speaker] = longest_clip

        if clip_index == 0:
            speaker_identity_map = {speaker: speaker for speaker in longest_monologues.keys()}
            previous_speaker_clips = current_speaker_clips.copy()
        else:
            # Compare all new speakers against all base speakers from previous clips
            for new_speaker, new_clip in current_speaker_clips.items():
                for base_speaker, base_clip in previous_speaker_clips.items():
                    if compare_embeddings(new_clip, base_clip):
                        speaker_identity_map[new_speaker] = base_speaker
                        break
                else:
                    # If no match is found, assign a new label
                    new_label = chr(ord(max(speaker_identity_map.values(), key=lambda x: ord(x))) + 1)
                    speaker_identity_map[new_speaker] = new_label

            # Update the previous_speaker_clips for the next iteration
            previous_speaker_clips.update(current_speaker_clips)

        # Update utterances with the new speaker labels for the current clip
        for utterance in clip_utterances[clip_index]:
            original_speaker = utterance['speaker']
            # Update only if there's a change in speaker identity
            if original_speaker in speaker_identity_map:
                utterance['speaker'] = speaker_identity_map[original_speaker]

# Add your clip transcript IDs
clip_transcript_ids = [
    "YOUR_TRANSCRIPT_ID_1",
    "YOUR_TRANSCRIPT_ID_2",
    "YOUR_TRANSCRIPT_ID_3"
]

# Add filepaths to your downloaded files
audio_files = [
    "/testone.wav",
    "/testtwo.wav",
    "/testthree.wav"
]

process_clips(clip_transcript_ids, audio_files)
```

### New Clip Utterances

The clip utterances returned by the `process_clips` function will contain the corrected utterances, which can be seen by printing out the utterances or by using the `display_transcript` function.

```python
print(clip_utterances)
```

```python
def display_transcript(transcript_data):
    for clip_index, utterances in transcript_data.items():
        print(f"Clip {clip_index + 1}:")
        for utterance in utterances:
            speaker = utterance['speaker']
            text = utterance['text']
            print(f"  Speaker {speaker}: {text}")
        print("\n")  # Add an extra newline for spacing between

display_transcript(clip_utterances)
```

## Additional Resources

- [Async chunking](https://github.com/AssemblyAI-Solutions/async-chunking)
- [AsyncChunkPy: Near-Realtime Python Speech-to-Text App](https://github.com/AssemblyAI-Solutions/async-chunk-py)
- [Guide for Identifying speakers across multiple podcasts with AssemblyAI and TitaNet](https://docs.google.com/document/d/1xdOvY1LM2lGUNRZCf3kNQLSs5OCUfo74_MHDGm_jYc0/edit?tab=t.0#heading=h.59f375jz72wm)
- [Multi-Speaker Voice Identification and Diarization with AssemblyAI, Pinecone, and Nvidia's TitaNet Model](https://colab.research.google.com/drive/1vqpYcPLEjDjMJ9WvP8C1gvOIflBHs0VS?usp=sharing)

---

<a id="setup-a-speaker-identification-system-using-pinecone-nvidia-titanet-docs-pre-recorded-audio-guides-titanet-speaker-identification-mdx"></a>

# Setup A Speaker Identification System using Pinecone & Nvidia TitaNet

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/titanet-speaker-identification
Source: docs/pre-recorded-audio/guides/titanet-speaker-identification.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Speaker labels
Description: Setup A Speaker Identification System using Pinecone & Nvidia TitaNet documentation.

This guide will demonstrate how to build an advanced speaker recognition and diarization system that you can use to identify speakers across multiple audio files. It will use:

- AssemblyAI for transcription and initial diarization.
- Nvidia's TitaNet model for speaker embedding generation.
- Pinecone for efficient similarity search of speaker embeddings.

## Quickstart

```python expandable
from pinecone import Pinecone, ServerlessSpec
import assemblyai as aai
import requests
import os
from pydub import AudioSegment
import mimetypes
import wave
from nemo.collections.asr.models import EncDecSpeakerLabelModel
import torch
import numpy as np
import uuid
from sklearn.metrics.pairwise import cosine_similarity

# Obtain from your Pinecone dashboard.
pc = Pinecone(api_key="PINECONE_KEY_HERE")

# Obtain from your AssemblyAI dashboard.
aai.settings.api_key = "AAI_KEY_HERE"

def transcribe(file_url):
    config = aai.TranscriptionConfig(speaker_labels=True)  # Speaker labels must be enabled for this Cookbook.

    transcriber = aai.Transcriber(config=config)

    transcript = transcriber.transcribe(file_url)

    return transcript.json_response

def download_and_convert_to_wav(url, output_dir="./content/converted_audio"):
    # Create the output directory if it doesn't exist.
    os.makedirs(output_dir, exist_ok=True)

    # Extract filename from URL.
    filename = url.split("/")[-1].split("?")[0]
    base_filename, file_extension = os.path.splitext(filename)

    # Download the file.
    response = requests.get(url)
    if response.status_code == 200:
        # Determine the file type.
        content_type = response.headers.get("content-type")
        if content_type:
            guessed_extension = mimetypes.guess_extension(content_type)
            if guessed_extension:
                file_extension = guessed_extension

        # Save the downloaded file.
        downloaded_file = os.path.join(output_dir, filename)
        with open(downloaded_file, "wb") as f:
            f.write(response.content)

        # Generate the WAV file name.
        wav_filename = f"{base_filename}.wav"
        wav_file = os.path.join(output_dir, wav_filename)

        # Load the audio file.
        audio = AudioSegment.from_file(downloaded_file)

        # Convert to mono if it's stereo.
        if audio.channels > 1:
            print("Setting channels to 1.")
            audio = audio.set_channels(1)

        # Export as WAV.
        audio.export(wav_file, format="wav")
        print(f"File converted and saved as: {wav_file}")

        # Remove the original downloaded file if it's different from the WAV file.
        if downloaded_file != wav_file:
            os.remove(downloaded_file)

        # Ensure the WAV file is single channel.
        with wave.open(wav_file, "rb") as wf:
            n_channels = wf.getnchannels()
            if n_channels > 1:
                print(f"Converting {n_channels} channels to mono...")
                # Read the frames.
                frames = wf.readframes(wf.getnframes())
                # Get other parameters.
                params = wf.getparams()
                # Close the file.
                wf.close()
                # Convert to mono.
                mono_frames = b"".join([frames[i::n_channels] for i in range(n_channels)])
                # Write the mono WAV file.
                with wave.open(wav_file, "wb") as wf:
                    wf.setparams(
                        (1, params.sampwidth, params.framerate, params.nframes, params.comptype, params.compname)
                    )
                    wf.writeframes(mono_frames)
                print("Conversion to mono complete.")

        return wav_file
    else:
        print(f"Failed to download the file. Status code: {response.status_code}")
        return None

def add_speaker_embedding_to_pinecone(speaker_name, speaker_embedding, unique_id=None):
    # Ensure the embedding is a 1D numpy array.
    if isinstance(speaker_embedding, torch.Tensor):
        embedding_np = speaker_embedding.squeeze().cpu().numpy()
    elif isinstance(speaker_embedding, np.ndarray):
        embedding_np = speaker_embedding.squeeze()
    else:
        raise ValueError("Unsupported embedding type. Expected torch.Tensor or numpy.ndarray")

    # Ensure the embedding is the correct shape
    if embedding_np.shape != (192,):
        raise ValueError(f"Expected embedding of shape (192,), but got {embedding_np.shape}")

    # Convert to list for Pinecone
    embedding_list = embedding_np.tolist()

    # Generate a unique ID if not provided
    if unique_id is None:
        unique_id = f"speaker_{speaker_name}_{uuid.uuid4().hex[:8]}"

    # Create the metadata dictionary
    metadata = {"speaker_name": speaker_name}

    # Upsert the vector to Pinecone
    upsert_response = index.upsert(vectors=[(unique_id, embedding_list, metadata)])

    print(f"Upserted embedding for speaker {speaker_name} with ID {unique_id}")
    return unique_id

def find_closest_speaker(utterance_embedding, local_embeddings=None, local_only=False, threshold=0.5):
    def cosine_sim(a, b):
        return cosine_similarity(a.reshape(1, -1), b.reshape(1, -1))[0][0]

    best_match = {"speaker_name": "No match found", "score": 0}

    # Local embeddings processing.
    if local_embeddings is not None:
        for speaker_name, embedding in local_embeddings.items():
            score = cosine_sim(utterance_embedding, embedding)
            if score > best_match["score"]:
                print("Identified speaker " + speaker_name + " confidence " + str(score))
                best_match = {"speaker_name": speaker_name, "score": score}

    # Pinecone query (if not local_only and local_embeddings is empty or not provided)
    if not local_only and (local_embeddings is None or len(local_embeddings) == 0):
        results = index.query(vector=utterance_embedding.tolist(), top_k=1, include_metadata=True)
        if results["matches"]:
            pinecone_match = results["matches"][0]
            pinecone_score = pinecone_match["score"]
            if pinecone_score > best_match["score"]:
                best_match = {"speaker_name": pinecone_match["metadata"]["speaker_name"], "score": pinecone_score}

    # Check if the best match meets the threshold.
    if best_match["score"] < threshold:
        return "No match found", 0

    return best_match["speaker_name"], best_match["score"]

def identify_speakers_from_utterances(transcript, wav_file, min_utterance_length=5000, match_all_utterances=False):
    utterances = transcript["utterances"]
    speaker_model = EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")

    known_speakers = {}
    unknown_speakers = {}
    unknown_count = 0

    unknown_folder = "unknown_speaker_utterances"
    os.makedirs(unknown_folder, exist_ok=True)

    audio_file_name = os.path.basename(wav_file)
    full_audio = AudioSegment.from_wav(wav_file)

    def get_suitable_utterance(speaker, min_length):
        suitable_utterances = [
            u for u in utterances if u["speaker"] == speaker and (u["end"] - u["start"]) >= min_length
        ]
        if suitable_utterances:
            return max(suitable_utterances, key=lambda u: u["end"] - u["start"])
        return max((u for u in utterances if u["speaker"] == speaker), key=lambda u: u["end"] - u["start"])

    # First pass: Identify speakers.
    for speaker in set(u["speaker"] for u in utterances):
        if speaker not in known_speakers and speaker not in unknown_speakers:
            suitable_utterance = get_suitable_utterance(speaker, min_utterance_length)

            start_ms = suitable_utterance["start"]
            end_ms = suitable_utterance["end"]
            utterance_audio = full_audio[start_ms:end_ms]

            temp_wav = "temp_utterance.wav"
            utterance_audio.export(temp_wav, format="wav")
            embedding = speaker_model.get_embedding(temp_wav)
            os.remove(temp_wav)

            speaker_name, score = find_closest_speaker(embedding)
            print(f"Speaker: {speaker}, Closest match: {speaker_name}, Score: {score}")

            if score > 0.5:  # Adjust threshold as needed.
                known_speakers[speaker] = speaker_name
                print(f"Identified as known speaker: {speaker}")
            else:
                unknown_count += 1
                unknown_name = f"Unknown Speaker {chr(64 + unknown_count)}"
                unknown_wav = f"{unknown_folder}/unknown_speaker_{unknown_count}_from_{audio_file_name}"
                utterance_audio.export(unknown_wav, format="wav")
                unknown_speakers[speaker] = {
                    "name": unknown_name,
                    "wav_file": unknown_wav,
                    "duration": end_ms - start_ms,
                }
                print(f"New unknown speaker detected: {unknown_name} (Duration: {(end_ms - start_ms)/1000:.2f}s)")

    # Second pass: Replace speaker names.
    for utterance in utterances:
        if utterance["speaker"] in known_speakers:
            utterance["speaker"] in known_speakers[utterance["speaker"]]
        elif utterance["speaker"] in unknown_speakers:
            utterance["speaker"] = unknown_speakers[utterance["speaker"]]["name"]

    # Third pass: Match all utterances if requested.
    if match_all_utterances:
        print("Matching all utterances individually...")
        for utterance in utterances:
            start_ms = utterance["start"]
            end_ms = utterance["end"]
            utterance_audio = full_audio[start_ms:end_ms]

            temp_wav = "temp_utterance.wav"
            utterance_audio.export(temp_wav, format="wav")
            embedding = speaker_model.get_embedding(temp_wav)
            os.remove(temp_wav)

            new_speaker_name, score = find_closest_speaker(embedding)

            if score > 0.5 and new_speaker_name != utterance["speaker"]:
                print(f"Speaker change detected: '{utterance['speaker']}' -> '{new_speaker_name}' (Score: {score})")
                print(f"Utterance: {utterance['text'][:50]}...")
                utterance["speaker"] = new_speaker_name

    return utterances, unknown_speakers

pc.create_index(
    name="speaker-embeddings",
    dimension=192,  # Replace with model-specific dimensions - 192 is for TitaNet-Large.
    metric="cosine",  # Replace with your model metric.
    spec=ServerlessSpec(
        cloud="aws",
        region="us-east-1",
    ),
)

# Connect to our new index.
index = pc.Index("speaker-embeddings")

speaker_model = EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")

elon_fingerprint = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/musk_fingerprinting.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L211c2tfZmluZ2VycHJpbnRpbmcud2F2IiwiaWF0IjoxNzAwNDM4OTAwLCJleHAiOjE3MzE5NzQ5MDB9.O3QOJSBqFNb1sg4nurwSFA13xIPHyKuon3UfHcFYit0&t=2023-11-20T00%3A08%3A20.071Z"
)
altman_fingerprint = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/sam_altman_fingerprint.mp3?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L3NhbV9hbHRtYW5fZmluZ2VycHJpbnQubXAzIiwiaWF0IjoxNzAwNjY5NjI0LCJleHAiOjE3MzIyMDU2MjR9._1yuMGzBhFcHr7xv76160Hb_SC-mH_Wv3_qX-S7XsTU&t=2023-11-22T16%3A13%3A44.103Z"
)
lex_fingerprint = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/lex_fridman_fingerprint.mp3?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L2xleF9mcmlkbWFuX2ZpbmdlcnByaW50Lm1wMyIsImlhdCI6MTcwMDY2OTc4OSwiZXhwIjoxNzMyMjA1Nzg5fQ.PUWTOLIHl4dcrWjh2ZJ_2TBaxMXpcU-x6OcvUDe6ZXQ&t=2023-11-22T16%3A16%3A29.697Z"
)

known_speakers = {"Elon Musk": elon_fingerprint, "Sam Altman": altman_fingerprint, "Lex Fridman": lex_fingerprint}

# Upload the known speakers.
for speaker, audio_file in known_speakers.items():
    print("***")
    print(speaker)
    print(audio_file)
    embedding = speaker_model.get_embedding(audio_file)
    add_speaker_embedding_to_pinecone(speaker, embedding)

audio_file = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/musk_fingerprinting.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L211c2tfZmluZ2VycHJpbnRpbmcud2F2IiwiaWF0IjoxNzAwNDM4OTAwLCJleHAiOjE3MzE5NzQ5MDB9.O3QOJSBqFNb1sg4nurwSFA13xIPHyKuon3UfHcFYit0&t=2023-11-20T00%3A08%3A20.071Z"
)
utterance_embedding = speaker_model.get_embedding(audio_file)

results = index.query(vector=utterance_embedding.tolist(), top_k=3, include_metadata=True)
print(results)

# Example: Conversation Between Sam Altman and Elon Musk
transcript_obj = transcribe("https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/elon_altman_interview_clipped.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L2Vsb25fYWx0bWFuX2ludGVydmlld19jbGlwcGVkLndhdiIsImlhdCI6MTcwMDY5MDU3OSwiZXhwIjoxNzMyMjI2NTc5fQ.4qZHvVRGhNGttfcpcfXDcJkJe_tbkc_2Bvs4i51SNSE&t=2023-11-22T22%3A02%3A59.429Z")
wav_file = download_and_convert_to_wav("https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/elon_altman_interview_clipped.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L2Vsb25fYWx0bWFuX2ludGVydmlld19jbGlwcGVkLndhdiIsImlhdCI6MTcwMDY5MDU3OSwiZXhwIjoxNzMyMjI2NTc5fQ.4qZHvVRGhNGttfcpcfXDcJkJe_tbkc_2Bvs4i51SNSE&t=2023-11-22T22%3A02%3A59.429Z")
identified_utterances, unknown_speakers = identify_speakers_from_utterances(transcript_obj, wav_file)

for utterance in identified_utterances:
    print(f"{utterance['speaker']}: {utterance['text']}")
```

## Initial Setup

First, you'll need to [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/signup) and obtain your API key from your [account dashboard](https://www.assemblyai.com/dashboard/home). Then, [sign up for a Pinecone account](https://app.pinecone.io/?sessionType=signup) and obtain your API key from "API Keys" on the sidebar of your dashboard. Also note that any files you use for this Cookbook should be in WAV format. While not a requirement for AssemblyAI, TitaNet requires WAV format.

## Installing Dependencies

Now we'll need to install the necessary libraries and frameworks for this project. Please note that this process can take several minutes to complete.

```bash
pip install -U Cython torch nemo_toolkit ffmpeg pydub pinecone-client assemblyai hydra-core pytorch_lightning huggingface_hub==0.23.5 librosa transformers pandas inflect webdataset sentencepiece youtokentome pyannote-audio editdistance jiwer lhotse datasets
```

## Pinecone Setup

In this section, we'll import Pinecone, create a new index for our speaker embeddings, and connect to the index. Please enter your Pinecone API key in the placeholder below.

```python
from pinecone import Pinecone, ServerlessSpec

# Obtain from your Pinecone dashboard.
pc = Pinecone(api_key="PINECONE_KEY_HERE")

pc.create_index(
    name="speaker-embeddings",
    dimension=192,  # Replace with model-specific dimensions - 192 is for TitaNet-Large.
    metric="cosine",  # Replace with your model metric.
    spec=ServerlessSpec(
        cloud="aws",
        region="us-east-1",
    ),
)

# Connect to our new index.
index = pc.Index("speaker-embeddings")
```

## AssemblyAI Setup

Now we'll set up AssemblyAI for transcription and diarization. We'll import the necessary modules and create a function to transcribe our audio files with speaker labels enabled. Please enter your AssemblyAI API key in the cell below.

```python
import assemblyai as aai

aai.settings.api_key = "AAI_KEY_HERE"

def transcribe(file_url):
    config = aai.TranscriptionConfig(speaker_labels=True)  # Speaker labels must be enabled for this Cookbook.

    transcriber = aai.Transcriber(config=config)

    transcript = transcriber.transcribe(file_url)

    return transcript.json_response
```

We'll also need to create a `download_and_convert_to_wav` helper function. This function allows us to take file URLs, download them, then convert them to WAV format. If the URLs are already in WAV format, then they're just downloaded. The files must be in WAV format to work properly with the TitaNet.

```python expandable
import requests
import os
from pydub import AudioSegment
import mimetypes
import wave

def download_and_convert_to_wav(url, output_dir="./content/converted_audio"):
    # Create the output directory if it doesn't exist.
    os.makedirs(output_dir, exist_ok=True)

    # Extract filename from URL.
    filename = url.split("/")[-1].split("?")[0]
    base_filename, file_extension = os.path.splitext(filename)

    # Download the file.
    response = requests.get(url)
    if response.status_code == 200:
        # Determine the file type.
        content_type = response.headers.get("content-type")
        if content_type:
            guessed_extension = mimetypes.guess_extension(content_type)
            if guessed_extension:
                file_extension = guessed_extension

        # Save the downloaded file.
        downloaded_file = os.path.join(output_dir, filename)
        with open(downloaded_file, "wb") as f:
            f.write(response.content)

        # Generate the WAV file name.
        wav_filename = f"{base_filename}.wav"
        wav_file = os.path.join(output_dir, wav_filename)

        # Load the audio file.
        audio = AudioSegment.from_file(downloaded_file)

        # Convert to mono if it's stereo.
        if audio.channels > 1:
            print("Setting channels to 1.")
            audio = audio.set_channels(1)

        # Export as WAV.
        audio.export(wav_file, format="wav")
        print(f"File converted and saved as: {wav_file}")

        # Remove the original downloaded file if it's different from the WAV file.
        if downloaded_file != wav_file:
            os.remove(downloaded_file)

        # Ensure the WAV file is single channel.
        with wave.open(wav_file, "rb") as wf:
            n_channels = wf.getnchannels()
            if n_channels > 1:
                print(f"Converting {n_channels} channels to mono...")
                # Read the frames.
                frames = wf.readframes(wf.getnframes())
                # Get other parameters.
                params = wf.getparams()
                # Close the file.
                wf.close()
                # Convert to mono.
                mono_frames = b"".join([frames[i::n_channels] for i in range(n_channels)])
                # Write the mono WAV file.
                with wave.open(wav_file, "wb") as wf:
                    wf.setparams(
                        (1, params.sampwidth, params.framerate, params.nframes, params.comptype, params.compname)
                    )
                    wf.writeframes(mono_frames)
                print("Conversion to mono complete.")

        return wav_file
    else:
        print(f"Failed to download the file. Status code: {response.status_code}")
        return None
```

## NVIDIA's TitaNet Model Setup

Next we'll import `torch` and `nemo`, then connect to and load [NVIDIA's TitaNet model](https://huggingface.co/nvidia/speakerverification_en_titanet_large). This model allows us to generate speaker embeddings to create speaker fingerprints. It also enables the conversion of utterances into embeddings for comparison with our fingerprints.

```python
from nemo.collections.asr.models import EncDecSpeakerLabelModel

speaker_model = EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")
```

We'll now define an `add_speaker_embedding_to_pinecone` function to add our speaker embeddings to the Pinecone database.

```python expandable
import torch
import numpy as np
import uuid

def add_speaker_embedding_to_pinecone(speaker_name, speaker_embedding, unique_id=None):
    # Ensure the embedding is a 1D numpy array.
    if isinstance(speaker_embedding, torch.Tensor):
        embedding_np = speaker_embedding.squeeze().cpu().numpy()
    elif isinstance(speaker_embedding, np.ndarray):
        embedding_np = speaker_embedding.squeeze()
    else:
        raise ValueError("Unsupported embedding type. Expected torch.Tensor or numpy.ndarray")

    # Ensure the embedding is the correct shape
    if embedding_np.shape != (192,):
        raise ValueError(f"Expected embedding of shape (192,), but got {embedding_np.shape}")

    # Convert to list for Pinecone
    embedding_list = embedding_np.tolist()

    # Generate a unique ID if not provided
    if unique_id is None:
        unique_id = f"speaker_{speaker_name}_{uuid.uuid4().hex[:8]}"

    # Create the metadata dictionary
    metadata = {"speaker_name": speaker_name}

    # Upsert the vector to Pinecone
    upsert_response = index.upsert(vectors=[(unique_id, embedding_list, metadata)])

    print(f"Upserted embedding for speaker {speaker_name} with ID {unique_id}")
    return unique_id
```

## Add Thumbprints to our Pinecone Database

Below we'll use chunks of the speakers' conversations to generate speaker embeddings and add them to our vector database. Later on, we'll show how to take an audio file with speakers not in the vector database and obtain the data required to generate new speaker fingerprints to be uploaded to the Pinecone database.

```python
elon_fingerprint = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/musk_fingerprinting.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L211c2tfZmluZ2VycHJpbnRpbmcud2F2IiwiaWF0IjoxNzAwNDM4OTAwLCJleHAiOjE3MzE5NzQ5MDB9.O3QOJSBqFNb1sg4nurwSFA13xIPHyKuon3UfHcFYit0&t=2023-11-20T00%3A08%3A20.071Z"
)
altman_fingerprint = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/sam_altman_fingerprint.mp3?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L3NhbV9hbHRtYW5fZmluZ2VycHJpbnQubXAzIiwiaWF0IjoxNzAwNjY5NjI0LCJleHAiOjE3MzIyMDU2MjR9._1yuMGzBhFcHr7xv76160Hb_SC-mH_Wv3_qX-S7XsTU&t=2023-11-22T16%3A13%3A44.103Z"
)
lex_fingerprint = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/lex_fridman_fingerprint.mp3?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L2xleF9mcmlkbWFuX2ZpbmdlcnByaW50Lm1wMyIsImlhdCI6MTcwMDY2OTc4OSwiZXhwIjoxNzMyMjA1Nzg5fQ.PUWTOLIHl4dcrWjh2ZJ_2TBaxMXpcU-x6OcvUDe6ZXQ&t=2023-11-22T16%3A16%3A29.697Z"
)

known_speakers = {"Elon Musk": elon_fingerprint, "Sam Altman": altman_fingerprint, "Lex Fridman": lex_fingerprint}

# Upload the known speakers.
for speaker, audio_file in known_speakers.items():
    print("***")
    print(speaker)
    print(audio_file)
    embedding = speaker_model.get_embedding(audio_file)
    add_speaker_embedding_to_pinecone(speaker, embedding)
```

Now we can query our Pinecone database to ensure that our embeddings were uploaded successfully.

```python
audio_file = download_and_convert_to_wav(
    "https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/musk_fingerprinting.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L211c2tfZmluZ2VycHJpbnRpbmcud2F2IiwiaWF0IjoxNzAwNDM4OTAwLCJleHAiOjE3MzE5NzQ5MDB9.O3QOJSBqFNb1sg4nurwSFA13xIPHyKuon3UfHcFYit0&t=2023-11-20T00%3A08%3A20.071Z"
)
utterance_embedding = speaker_model.get_embedding(audio_file)

results = index.query(vector=utterance_embedding.tolist(), top_k=3, include_metadata=True)

print(results)
```

## Creating Functions to Find the Closest Speaker and Identify Speakers of Utterances

### Speaker Identification Function

The `find_closest_speaker` function is a crucial component of our speaker identification system. It compares a given utterance embedding to known speaker embeddings and identifies the closest match.

```python expandable
from sklearn.metrics.pairwise import cosine_similarity

def find_closest_speaker(utterance_embedding, local_embeddings=None, local_only=False, threshold=0.5):
    def cosine_sim(a, b):
        return cosine_similarity(a.reshape(1, -1), b.reshape(1, -1))[0][0]

    best_match = {"speaker_name": "No match found", "score": 0}

    # Local embeddings processing.
    if local_embeddings is not None:
        for speaker_name, embedding in local_embeddings.items():
            score = cosine_sim(utterance_embedding, embedding)
            if score > best_match["score"]:
                print("Identified speaker " + speaker_name + " confidence " + str(score))
                best_match = {"speaker_name": speaker_name, "score": score}

    # Pinecone query (if not local_only and local_embeddings is empty or not provided)
    if not local_only and (local_embeddings is None or len(local_embeddings) == 0):
        results = index.query(vector=utterance_embedding.tolist(), top_k=1, include_metadata=True)
        if results["matches"]:
            pinecone_match = results["matches"][0]
            pinecone_score = pinecone_match["score"]
            if pinecone_score > best_match["score"]:
                best_match = {"speaker_name": pinecone_match["metadata"]["speaker_name"], "score": pinecone_score}

    # Check if the best match meets the threshold.
    if best_match["score"] < threshold:
        return "No match found", 0

    return best_match["speaker_name"], best_match["score"]
```

### Speaker Identification from Utterances

The `identify_speakers_from_utterances` function is the core of our speaker identification system. It processes a transcript with utterances and identifies speakers, handling both known and unknown voices.

```python expandable
def identify_speakers_from_utterances(transcript, wav_file, min_utterance_length=5000, match_all_utterances=False):
    utterances = transcript["utterances"]
    speaker_model = EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")

    known_speakers = {}
    unknown_speakers = {}
    unknown_count = 0

    unknown_folder = "unknown_speaker_utterances"
    os.makedirs(unknown_folder, exist_ok=True)

    audio_file_name = os.path.basename(wav_file)
    full_audio = AudioSegment.from_wav(wav_file)

    def get_suitable_utterance(speaker, min_length):
        suitable_utterances = [
            u for u in utterances if u["speaker"] == speaker and (u["end"] - u["start"]) >= min_length
        ]
        if suitable_utterances:
            return max(suitable_utterances, key=lambda u: u["end"] - u["start"])
        return max((u for u in utterances if u["speaker"] == speaker), key=lambda u: u["end"] - u["start"])

    # First pass: Identify speakers.
    for speaker in set(u["speaker"] for u in utterances):
        if speaker not in known_speakers and speaker not in unknown_speakers:
            suitable_utterance = get_suitable_utterance(speaker, min_utterance_length)

            start_ms = suitable_utterance["start"]
            end_ms = suitable_utterance["end"]
            utterance_audio = full_audio[start_ms:end_ms]

            temp_wav = "temp_utterance.wav"
            utterance_audio.export(temp_wav, format="wav")
            embedding = speaker_model.get_embedding(temp_wav)
            os.remove(temp_wav)

            speaker_name, score = find_closest_speaker(embedding)
            print(f"Speaker: {speaker}, Closest match: {speaker_name}, Score: {score}")

            if score > 0.5:  # Adjust threshold as needed.
                known_speakers[speaker] = speaker_name
                print(f"Identified as known speaker: {speaker}")
            else:
                unknown_count += 1
                unknown_name = f"Unknown Speaker {chr(64 + unknown_count)}"
                unknown_wav = f"{unknown_folder}/unknown_speaker_{unknown_count}_from_{audio_file_name}"
                utterance_audio.export(unknown_wav, format="wav")
                unknown_speakers[speaker] = {
                    "name": unknown_name,
                    "wav_file": unknown_wav,
                    "duration": end_ms - start_ms,
                }
                print(f"New unknown speaker detected: {unknown_name} (Duration: {(end_ms - start_ms)/1000:.2f}s)")

    # Second pass: Replace speaker names.
    for utterance in utterances:
        if utterance["speaker"] in known_speakers:
            utterance["speaker"] in known_speakers[utterance["speaker"]]
        elif utterance["speaker"] in unknown_speakers:
            utterance["speaker"] = unknown_speakers[utterance["speaker"]]["name"]

    # Third pass: Match all utterances if requested.
    if match_all_utterances:
        print("Matching all utterances individually...")
        for utterance in utterances:
            start_ms = utterance["start"]
            end_ms = utterance["end"]
            utterance_audio = full_audio[start_ms:end_ms]

            temp_wav = "temp_utterance.wav"
            utterance_audio.export(temp_wav, format="wav")
            embedding = speaker_model.get_embedding(temp_wav)
            os.remove(temp_wav)

            new_speaker_name, score = find_closest_speaker(embedding)

            if score > 0.5 and new_speaker_name != utterance["speaker"]:
                print(f"Speaker change detected: '{utterance['speaker']}' -> '{new_speaker_name}' (Score: {score})")
                print(f"Utterance: {utterance['text'][:50]}...")
                utterance["speaker"] = new_speaker_name

    return utterances, unknown_speakers
```

## Examples: Speaker Identification and Diarization

To demonstrate the capabilities of our speaker identification and diarization system, we'll cover several examples. We'll start with a straightforward case and progressively move to more complex scenarios.

### Example 1: Conversation Between Sam Altman and Elon Musk

Our first example is a simple conversation between two well-known figures: Elon Musk and Sam Altman. This example will showcase how our system performs with clear, distinct voices in a controlled setting.

```python
transcript_obj = transcribe("https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/elon_altman_interview_clipped.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L2Vsb25fYWx0bWFuX2ludGVydmlld19jbGlwcGVkLndhdiIsImlhdCI6MTcwMDY5MDU3OSwiZXhwIjoxNzMyMjI2NTc5fQ.4qZHvVRGhNGttfcpcfXDcJkJe_tbkc_2Bvs4i51SNSE&t=2023-11-22T22%3A02%3A59.429Z")
wav_file = download_and_convert_to_wav("https://api.assemblyai-solutions.com/storage/v1/object/sign/sam_training_bucket/elon_altman_interview_clipped.wav?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJzYW1fdHJhaW5pbmdfYnVja2V0L2Vsb25fYWx0bWFuX2ludGVydmlld19jbGlwcGVkLndhdiIsImlhdCI6MTcwMDY5MDU3OSwiZXhwIjoxNzMyMjI2NTc5fQ.4qZHvVRGhNGttfcpcfXDcJkJe_tbkc_2Bvs4i51SNSE&t=2023-11-22T22%3A02%3A59.429Z")
identified_utterances, unknown_speakers = identify_speakers_from_utterances(transcript_obj, wav_file)

for utterance in identified_utterances:
    print(f"{utterance['speaker']}: {utterance['text']}")
```

---

<a id="use-automatic-language-detection-as-a-separate-step-from-transcription-docs-pre-recorded-audio-guides-automatic-language-detection-separate-mdx"></a>

# Use Automatic Language Detection as a Separate Step From Transcription

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/automatic-language-detection-separate
Source: docs/pre-recorded-audio/guides/automatic-language-detection-separate.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Automatic Language Detection
Description: Use Automatic Language Detection as a Separate Step From Transcription documentation.

In this guide, we'll show you how to perform automatic language detection separately from the transcription process. For the transcription, the file then gets then routed to either our [Universal-3.5 Pro or Universal-2](/pre-recorded-audio/select-the-speech-model) model class, depending on the supported language.

This workflow is designed to be cost-effective, slicing the first 60 seconds of audio and running it through Universal-2 ALD, which detects 99 languages, at a cost of $0.002 per transcript for this language detection workflow (not including the total transcription cost).

## Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Step-by-step instructions

Install the SDK:

```bash
pip install assemblyai
```

Import the `assemblyai` package and set your API key:

```python
import assemblyai as aai
aai.settings.api_key = "YOUR_API_KEY"
```

Create a set with all supported languages for _Universal_. You can find them in our [documentation here](/pre-recorded-audio/supported-languages).

```python expandable
supported_languages_for_universal = {
    "en",
    "en_au",
    "en_uk",
    "en_us",
    "es",
    "fr",
    "de",
    "it",
    "pt",
    "nl",
    "hi",
    "ja",
    "zh",
    "fi",
    "ko",
    "pl",
    "ru",
    "tr",
    "uk",
    "vi",
}
```

Define a `Transcriber`. Note that here we don't pass in a global `TranscriptionConfig`, but later apply different ones during the `transcribe()` call.

```python
transcriber = aai.Transcriber()
```

Define two helper functions:

- `detect_language()` performs language detection on the [first 60 seconds](/api-reference/transcripts/submit#request.body.audio_end_at) of the audio and returns the language code.
- `transcribe_file()` performs the transcription. For this, the identified language is applied and either Universal-3.5 Pro or Universal-2 is used depending on the supported language.

```python
def detect_language(audio_url):
    config = aai.TranscriptionConfig(
        audio_end_at=60000,  # first 60 seconds (in milliseconds)
        language_detection=True,
        speech_models=["universal-2"],
    )
    transcript = transcriber.transcribe(audio_url, config=config)
    return transcript.json_response["language_code"]

def transcribe_file(audio_url, language_code):
    config = aai.TranscriptionConfig(
        language_code=language_code,
        speech_models=(
            ["universal-3-5-pro", "universal-2"]
            if language_code in supported_languages_for_universal
            else ["universal-2"]
        ),
    )
    transcript = transcriber.transcribe(audio_url, config=config)
    return transcript
```

Test the code with different audio files. For each file, we apply both helper functions sequentially to first identify the language and then transcribe the file.

```python
audio_urls = [
    "https://storage.googleapis.com/aai-web-samples/public_benchmarking_portugese.mp3",
    "https://storage.googleapis.com/aai-web-samples/public_benchmarking_spanish.mp3",
    "https://storage.googleapis.com/aai-web-samples/slovenian_luka_doncic_interview.mp3",
    "https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries.mp3",
]

for audio_url in audio_urls:
    language_code = detect_language(audio_url)
    print("Identified language:", language_code)

    transcript = transcribe_file(audio_url, language_code)
    print("Transcript:", transcript.text[:100], "...")
```

---

<a id="route-to-default-language-if-language-confidence-is-low-docs-pre-recorded-audio-guides-automatic-language-detection-route-default-language-mdx"></a>

# Route to Default Language if Language Confidence is Low

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/automatic-language-detection-route-default-language
Source: docs/pre-recorded-audio/guides/automatic-language-detection-route-default-language.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Automatic Language Detection
Description: Route to Default Language if Language Confidence is Low documentation.

This guide will show you how to use AssemblyAI's API to resubmit a request using a default language if the Automatic Language Detection's `language_confidence` is below a certain threshold.

## Getting started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-step instructions

Install the SDK:

```bash
npm install assemblyai
```

```bash
pip install assemblyai
```

Import the `assemblyai` package and set the API key.

```js
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "YOUR_API_KEY",
});
```

```python
import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"
```

Define a `default_language`, which should be set to the [language code](/pre-recorded-audio/supported-languages) that will be used to rerun the transcript if language detection runs with low `language_confidence`.

```js
const default_language = "LANGUAGE_CODE";
```

```python
default_language = "LANGUAGE_CODE"
```

Define an `audio_url` that is set to a link to the audio file. Define and set the parameters `audio: audioUrl` and `language_detection: true`. We also need to define our `language_confidence_threshold`. For the purposes of this example, we'll set it to 0.8, representing 80% confidence.

If a transcript ends up with a `language_confidence` below this value, the transcript will error out and will return the transcript using the `default_language`.

```js
const audioUrl = "https://example.org/audio.mp3";

const params = {
  audio: audioUrl,
  language_detection: true,
  language_confidence_threshold: 0.8,
  speech_models: ["universal-3-5-pro", "universal-2"],
  // Add any other params
};
```

```python
transcriber = aai.Transcriber()
audio_url = ("https://example.org/audio.mp3")
config = aai.TranscriptionConfig(language_detection=True, language_confidence_threshold=0.8, speech_models=["universal-3-5-pro", "universal-2"])
transcript = transcriber.transcribe(audio_url, config)
```

You can handle the error safely by checking the error message and rerunning the transcript with the `language_code` set to the `default_language`.

The error handling flow works as follows:

1. If there is no error, the transcript ID and text are printed
2. If there is an error:
   - Check if it's related to `language_confidence` being below threshold
   - If so:
     - Print message about rerunning with default language
     - Create new transcript with `default_language` as the `language_code`
     - Print new transcript ID and text
   - If not:
     - Print the error message

When rerunning with the default language, the configuration is updated to:

- Turn off `language_detection`
- Remove `language_confidence_threshold`
- Set `language_code` to the `default_language`

You will not be charged for the first transcript if there is an error. You
will only be charged for the transcript that processes successfully.

```js expandable
const run = async (params) => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    if (
      transcript.error.includes(
        "below the requested confidence threshold value"
      )
    ) {
      console.log(
        `${transcript.error}. Running transcript again with language set to '${default_language}'.`
      );
      params = {
        ...params,
        language_detection: false,
        language_confidence_threshold: null,
        language_code: default_language,
      };
      run(params);
      return;
    }

    console.log(transcript.error);
    return;
  }

  console.log(`Transcript ID: ${transcript.id}`);
  console.log(transcript.text);
};

run(params);
```

```python
if transcript.error:
    if "below the requested confidence threshold value" in transcript.error:
        print(f"{transcript.error}. Running transcript again with language set to '{default_language}'.")
        new_config = aai.TranscriptionConfig(language_code=default_language, speech_models=["universal-3-5-pro", "universal-2"])
        transcript = transcriber.transcribe(audio_url, new_config)
        print(f"Transcript ID: {transcript.id}")
        print(transcript.text)
    else:
        print(transcript.error)
else:
    print(f"Transcript ID: {transcript.id}")
    print(transcript.text)
```

---

<a id="create-custom-length-subtitles-docs-pre-recorded-audio-guides-subtitle-creation-by-word-count-mdx"></a>

# Create Custom Length Subtitles

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/subtitle_creation_by_word_count
Source: docs/pre-recorded-audio/guides/subtitle_creation_by_word_count.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Subtitles
Description: Create Custom Length Subtitles documentation.

While our SRT/VTT endpoints do allow you to customize the maximum number of characters per caption using the chars_per_caption URL parameter in your API requests, there are some use-cases that require a custom number of words in each subtitle.

In this guide, we will demonstrate how to construct these subtitles yourself in Python!

## Quickstart

```python expandable
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"

config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()

transcript = transcriber.transcribe("./my-audio.mp3", config)

def second_to_timecode(x: float) -> str:
    hour, x = divmod(x, 3600)
    minute, x = divmod(x, 60)
    second, x = divmod(x, 1)
    millisecond = int(x * 1000.)

    return '%.2d:%.2d:%.2d,%.3d' % (hour, minute, second, millisecond)

def generate_subtitles_by_word_count(transcript, words_per_line):
  output = []
  subtitle_index = 1  # Start subtitle index at 1
  word_count = 0
  current_words = []

  for sentence in transcript.get_sentences():
    for word in sentence.words:
      current_words.append(word)
      word_count += 1
      if word_count >= words_per_line or word == sentence.words[-1]:
        start_time = second_to_timecode(current_words[0].start / 1000)
        end_time = second_to_timecode(current_words[-1].end / 1000)
        subtitle_text = " ".join([word.text for word in current_words])
        output.append(str(subtitle_index))
        output.append("%s --> %s" % (start_time, end_time))
        output.append(subtitle_text)
        output.append("")
        current_words = []  # Reset for the next subtitle
        word_count = 0  # Reset word count
        subtitle_index += 1

  return output

subs = generate_subtitles_by_word_count(transcript, 6)
with open(f"{transcript.id}.srt", 'w') as o:
    final = '\n'.join(subs)
    o.write(final)

print("SRT file generated.")
```

## Step-by-Step Instructions

```bash
pip install -U assemblyai

```

Create a `main.py` file and import the `assemblyai` package and set the API key.

```python
import assemblyai as aai

aai.settings.api_key = "YOUR-API-KEY"
```

Create a Transcriber object.

```python
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
```

Use the Transcriber object's transcribe method and pass in the audio file's path as a parameter. The transcribe method saves the results of the transcription to the Transcriber object's transcript attribute.

```python
transcript = transcriber.transcribe("./my-audio.mp3", config)
```

Alternatively, you can pass in the URL of the publicly accessible audio file on the internet.

```python
transcript = transcriber.transcribe("https://storage.googleapis.com/aai-docs-samples/espn.m4a", config)
```

Define a function that converts seconds to timecodes

```python
def second_to_timecode(x: float) -> str:
    hour, x = divmod(x, 3600)
    minute, x = divmod(x, 60)
    second, x = divmod(x, 1)
    millisecond = int(x * 1000.)

    return '%.2d:%.2d:%.2d,%.3d' % (hour, minute, second, millisecond)
```

Define a function that iterates through the transcripts object to construct a list according to the number of words per subtitle

```python expandable
def generate_subtitles_by_word_count(transcript, words_per_line):
  output = []
  subtitle_index = 1  # Start subtitle index at 1
  word_count = 0
  current_words = []

  for sentence in transcript.get_sentences():
    for word in sentence.words:
      current_words.append(word)
      word_count += 1
      if word_count >= words_per_line or word == sentence.words[-1]:
        start_time = second_to_timecode(current_words[0].start / 1000)
        end_time = second_to_timecode(current_words[-1].end / 1000)
        subtitle_text = " ".join([word.text for word in current_words])
        output.append(str(subtitle_index))
        output.append("%s --> %s" % (start_time, end_time))
        output.append(subtitle_text)
        output.append("")
        current_words = []  # Reset for the next subtitle
        word_count = 0  # Reset word count
        subtitle_index += 1

  return output
```

Generate your subtitle file

```python
subs = generate_subtitles_by_word_count(transcript, 6)
with open(f"{transcript.id}.srt", 'w') as o:
    final = '\n'.join(subs)
    o.write(final)

print("SRT file generated.")
```

Run your script.

```bash
python main.py
```

---

<a id="create-subtitles-with-speaker-labels-docs-pre-recorded-audio-guides-speaker-labelled-subtitles-mdx"></a>

# Create Subtitles with Speaker Labels

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/speaker_labelled_subtitles
Source: docs/pre-recorded-audio/guides/speaker_labelled_subtitles.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Subtitles
Description: Create Subtitles with Speaker Labels documentation.

## Quickstart

```python expandable
import assemblyai as aai

# SETTINGS
aai.settings.api_key = "YOUR-API-KEY"
filename = "YOUR-FILE-NAME"
transcriber = aai.Transcriber(config=aai.TranscriptionConfig(speaker_labels=True))
transcript = transcriber.transcribe(filename)

# Maximum number of words per subtitle
max_words_per_subtitle = 6

# Color assignments for speakers
speaker_colors = {
    "A": "red",
    "B": "orange",
    "C": "yellow",
    "D": "yellowgreen",
    "E": "green",
    "F": "lightskyblue",
    "G": "purple",
    "H": "mediumpurple",
    "I": "pink",
    "J": "brown",
}

# Process transcription segments
def process_segments(segments):
    srt_content = ""
    subtitle_index = 1
    for segment in segments:
        speaker = segment.speaker
        color = speaker_colors.get(speaker, "black")  # Default color is black

        # Split text into words and group into chunks
        words = segment.words
        for i in range(0, len(words), max_words_per_subtitle):
            chunk = words[i:i + max_words_per_subtitle]
            start_time = chunk[0].start  # -1 indicates continuation
            end_time = chunk[-1].end
            srt_content += create_subtitle(subtitle_index, start_time, end_time, chunk, color)
            subtitle_index += 1

    return srt_content

# Create a single subtitle
def create_subtitle(index, start_time, end_time, words, color):
    text = ""
    for word in words:
        text += word.text + ' '
    start_srt = format_time(start_time)
    end_srt = format_time(end_time)
    return f"{index}\n{start_srt} --> {end_srt}\n<font color=\"{color}\">{text}</font>\n\n"

# Format time in SRT style
def format_time(milliseconds):
    hours, remainder = divmod(milliseconds, 3600000)
    minutes, remainder = divmod(remainder, 60000)
    seconds, milliseconds = divmod(remainder, 1000)
    return f"{int(hours):02}:{int(minutes):02}:{int(seconds):02},{int(milliseconds):03}"

# Generate SRT content
sentences = transcript.get_sentences()
srt_content = process_segments(sentences)

# Save to SRT file
with open(filename + '.srt', 'w') as file:
    file.write(srt_content)

print(f"SRT file generated: {filename}.srt")
```

This Colab will demonstrate how to use AssemblyAI's [Speaker Diarization](/pre-recorded-audio/label-speakers) model together to format subtitles according to their respective speaker.

## Step-by-step guide

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://www.assemblyai.com/dashboard/signup) for an AssemblyAI account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home).

```bash
pip install assemblyai
```

First, we will configure our API key as well as our file to be transcribed. Then, we decide on a number of words we want to have per subtitle.

Lastly, we transcribe our file.

```python
import assemblyai as aai

# SETTINGS
aai.settings.api_key = "YOUR-API-KEY"
filename = "YOUR-FILE-NAME"
transcriber = aai.Transcriber(config=aai.TranscriptionConfig(speaker_labels=True))
transcript = transcriber.transcribe(filename)

# Maximum number of words per subtitle
max_words_per_subtitle = 6
```

## How the code works

`speaker_colors` is a dictionary that maps speaker identifiers (like "A", "B", "C", etc.) to specific colors. Each speaker in the transcription will be associated with a unique color in the subtitles.

When Speaker Diarization is enabled, sentences in our API response have a speaker code under the `speaker` key. We use the speaker code to determine the color of the subtitle text.

```python expandable
# Color assignments for speakers
speaker_colors = {
    "A": "red",
    "B": "orange",
    "C": "yellow",
    "D": "yellowgreen",
    "E": "green",
    "F": "lightskyblue",
    "G": "purple",
    "H": "mediumpurple",
    "I": "pink",
    "J": "brown",
}

# Process transcription segments
def process_segments(segments):
    srt_content = ""
    subtitle_index = 1
    for segment in segments:
        speaker = segment.speaker
        color = speaker_colors.get(speaker, "black")  # Default color is black

        # Split text into words and group into chunks
        words = segment.words
        for i in range(0, len(words), max_words_per_subtitle):
            chunk = words[i:i + max_words_per_subtitle]
            start_time = chunk[0].start  # -1 indicates continuation
            end_time = chunk[-1].end
            srt_content += create_subtitle(subtitle_index, start_time, end_time, chunk, color)
            subtitle_index += 1

    return srt_content

# Create a single subtitle
def create_subtitle(index, start_time, end_time, words, color):
    text = ""
    for word in words:
        text += word.text + ' '
    start_srt = format_time(start_time)
    end_srt = format_time(end_time)
    return f"{index}\n{start_srt} --> {end_srt}\n<font color=\"{color}\">{text}</font>\n\n"

# Format time in SRT style
def format_time(milliseconds):
    hours, remainder = divmod(milliseconds, 3600000)
    minutes, remainder = divmod(remainder, 60000)
    seconds, milliseconds = divmod(remainder, 1000)
    return f"{int(hours):02}:{int(minutes):02}:{int(seconds):02},{int(milliseconds):03}"
```

Our last step is to generate and save our subtitle file!

```python
# Generate SRT content
sentences = transcript.get_sentences()
srt_content = process_segments(sentences)

# Save to SRT file
with open(filename + '.srt', 'w') as file:
    file.write(srt_content)

print(f"SRT file generated: {filename}.srt")
```

---

<a id="generate-subtitles-for-videos-docs-pre-recorded-audio-guides-subtitles-mdx"></a>

# Generate Subtitles for Videos

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/subtitles
Source: docs/pre-recorded-audio/guides/subtitles.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Subtitles
Description: Generate Subtitles for Videos documentation.

You can export your completed transcripts in SRT or VTT format, which can be used for subtitles and closed captions in videos. Once your transcript status shows as completed, you can make a request to the appropriate endpoint to export your transcript in SRT or VTT format.

In this Colab, we'll walk through the process of generating subtitles for videos using the AssemblyAI API.

## SRT and VTT Subtitles Formats

### How do SRT Files Work?

SRT (SubRip Text) files are commonly used to store subtitles for videos. The format is plain text, and it contains the timing information for each subtitle along with the subtitle text itself.

Here's a breakdown of how the format works:

- Each subtitle entry consists of an index number, start time, end time, and text.
- The index number is a sequential number starting from 1.
- The start and end times are given in the format `hours:minutes:seconds,milliseconds` and are separated by `-->`.
- The text that follows the timing information is the subtitle text itself, and it may span multiple lines.
- Entries are separated by a blank line.

### How do VTT Formats Work?

WEBVTT (Web Video Text Tracks), which is a standard format for displaying timed text tracks (such as subtitles or captions) within HTML5 video.

The syntax is similar to SRT but has some differences:

- The file should begin with the header WEBVTT.
- Timing is done with a period (`.`) separating seconds and milliseconds instead of a comma (`,`).
- No blank lines are needed between entries.
- No index numbers are required.
- This format is supported by many modern browsers and can be ussed with the HTML5 `<track>` element to add subtitles to a `<video>` element.

If you're planning to upload this file to YouTube, you should be able to use it just like an SRT file. YouTube supports various subtitle formats, including WEBVTT.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

The entire source code of this guide can be viewed [here](https://github.com/AssemblyAI-Examples/docs-snippets/tree/main/subtitles).

## Step-by-Step Instructions

Install the SDK.

```bash
pip install -U assemblyai
```

Import the `assemblyai` package and set the API key.

```python
import assemblyai as aai

aai.settings.api_key = "YOUR_API_KEY"
```

Create a `Transcriber` object.

```python
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
```

Use the `Transcriber` object's `transcribe` method and pass in the audio file's path as a parameter. The `transcribe` method saves the results of the transcription to the `Transcriber` object's `transcript` attribute.

```python
transcript = transcriber.transcribe("./my-audio.mp3", config)
```

Alternatively, you can pass in the URL of the publicly accessible audio file on the internet.

```python
transcript = transcriber.transcribe("https://example.org/audio.mp3", config)
```

Export SRT subtitles with the `export_subtitles_srt` method.

```python
print(transcript.export_subtitles_srt())
```

Export VTT subtitles with the `export_subtitles_vtt` method.

```python
print(transcript.export_subtitles_vtt())
```

## Advanced Usage

You can also customize the maximum number of characters per caption using the `chars_per_caption` URL parameter in your API requests to either the SRT or VTT endpoints. For example, adding `?chars_per_caption=32` to the SRT endpoint URL ensures that each caption has no more than 32 characters.

[API/Model Reference](/getting-started/models)

## Conclusion

AssemblyAI can produce subtitles as both `.srt` and `.vtt` files. These are standard subtitle formats, and can be used with videos both on and off the web. For example, after generating your subtitle file, you can add it to a [Mux video](https://www.assemblyai.com/blog/how-to-add-subtitles-to-your-mux-videos-with-python/) using their platform, or you can use [ffmpeg](https://amiaopensource.github.io/ffmprovisr/#embed_subtitles) to embed it in a local video file. Subtitle formats contain plain text, so you can import these formatted captions to most video editors, or fine-tune them as needed.

## Related Cookbooks

[Custom Length Subtitles](/pre-recorded-audio/guides/subtitle_creation_by_word_count)  
[Translating Subtitles](/pre-recorded-audio/guides/translate_subtitles)

---

<a id="translate-an-assemblyai-subtitle-transcript-docs-pre-recorded-audio-guides-translate-subtitles-mdx"></a>

# Translate an AssemblyAI Subtitle Transcript

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/translate_subtitles
Source: docs/pre-recorded-audio/guides/translate_subtitles.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Subtitles
Description: Translate an AssemblyAI Subtitle Transcript documentation.

In this guide, we'll show you how to translate an AssemblyAI generated subtitle transcript. We will also be using the `Googletrans` Python library to implement the Google Translate API.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Step-by-Step Instructions

Install the relevant packages.

1. AssemblyAI SDK
2. Googletrans

```bash
pip install -U assemblyai googletrans
```

Import the `assemblyai` package and set the API key. Import `Translator` class from `googletrans` package.

```python
import assemblyai as aai
from googletrans import Translator

aai.settings.api_key = "YOUR_API_KEY"
```

Create a `Transcriber` object.

```python
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
```

Create a `Translator` object.

```python
translator = Translator()
```

Use the Transcriber object's `transcribe` method and pass in the audio file's path as a parameter. The transcribe method saves the results of the transcription to the `Transcriber` object's `transcript` attribute.

```python
transcript = transcriber.transcribe("./my-audio.mp3", config)
```

Alternatively, you can pass in a path to an audio file saved on the internet.

```python
transcript = transcriber.transcribe("https://example.org/audio.mp3", config)
```

Export SRT subtitles with the `export_subtitles_srt` method. Create a `subtitle_transcript` variable to translate in the next step.

```python
subtitle_transcript = transcript.export_subtitles_srt()
```

Translate subtitle transcript to target language

```python
translation = translator.translate(subtitle_transcript, dest='es')
```

Print results

```python
print(translation.text)
```

## Conclusion

Using the subtitles and transcripts generated by AssemblyAI, we can also generate translated alternatives using other translation APIs. The implementation logic will be similar with other solutions like DeepL, Yandex Translate and many more.

To translate to other languages, find the full list of Supported Languages in the **Further Documentations** section.

## Further Documentation

[Googletrans Library](https://pypi.org/project/googletrans/)

[Translation Supported Languages](https://py-googletrans.readthedocs.io/en/latest/#googletrans-languages)

---

<a id="schedule-a-delete-request-with-assemblyai-and-easycron-docs-pre-recorded-audio-guides-schedule-delete-mdx"></a>

# Schedule a DELETE request with AssemblyAI and EasyCron

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/schedule_delete
Source: docs/pre-recorded-audio/guides/schedule_delete.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Delete transcripts
Description: Schedule a DELETE request with AssemblyAI and EasyCron documentation.

This tutorial guides you through the process of scheduling a DELETE request for a transcript created with AssemblyAI's transcription service, utilizing EasyCron for scheduling.

## Quickstart

```python expandable
import assemblyai as aai
import requests
from datetime import datetime, timedelta, timezone

# Set up AssemblyAI API key
# In production, store this in an environment variable
aai.settings.api_key = "YOUR_ASSEMBLYAI_API_KEY"

# Get an EasyCron API key here: https://www.easycron.com/user/token
# Set up EasyCron API key
# In production, store this in an environment variable
EASYCRON_API_TOKEN = "YOUR_EASYCRON_API_KEY"

# Create transcriber instance and transcribe audio
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
transcript = transcriber.transcribe('https://assembly.ai/sports_injuries.mp3', config)

# Get the transcript ID
transcript_id = transcript.id

# Construct the URL for the DELETE request
url_to_call = f"https://api.assemblyai.com/v2/transcript/{transcript_id}"

# Calculate the time 24 hours from now for the cron expression
time_24_hours_from_now = datetime.now(timezone.utc) + timedelta(hours=24)
cron_minute = time_24_hours_from_now.minute
cron_hour = time_24_hours_from_now.hour
cron_day = time_24_hours_from_now.day
cron_month = time_24_hours_from_now.month
cron_year = time_24_hours_from_now.year

# Create the cron expression
cron_expression = f'{cron_minute} {cron_hour} {cron_day} {cron_month} * {cron_year}'

# EasyCron API endpoint for creating a new cron job
api_endpoint = 'https://www.easycron.com/rest/add'

# Data payload for EasyCron API
data = {
    'token': EASYCRON_API_TOKEN,
    'url': url_to_call,
    'cron_expression': cron_expression,
    'http_method': "DELETE",
    'http_headers': f'Authorization: {aai.settings.api_key}',
    'timezone': 'UTC'
}

# Make the request to EasyCron's API
response = requests.post(api_endpoint, data=data)

# Print the response from EasyCron
print("EasyCron API Response:")
print(response.text)
```

## Step-by-step guide

To get started, install the AssemblyAI Python SDK.

```bash
pip install "assemblyai"
```

Finally, import the `assemblyai` package and set your API token in the settings:

```python
import assemblyai as aai

# set the API key
aai.settings.api_key = f"{YOUR_API_KEY}"
```

```python
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()

# this is just an example file
transcript = transcriber.transcribe('https://assembly.ai/sports_injuries.mp3', config)
```

Store the transcript ID

```python
transcript_id = transcript.id
```

Using this, we now construct the URL that we want our `cron` job to call.

```python
url_to_call = f"https://api.assemblyai.com/v2/transcript/{transcript_id}"
```

## Using EasyCron's API to schedule a DELETE request

First, sign up for an account with EasyCron [here](https://www.easycron.com/cron-jobs). Locate your EasyCron API key [in your user dashboard.](https://www.easycron.com/user/token)

Then, insert your EasyCron API key into your code.

Next, we will use the datetime module to generate a `cron` expression for 24 hours from now (although you can configure the `timedelta` argument to be whatever you want)

```python
# In production, you should store your API keys in environment variables
EASYCRON_API_TOKEN = "YOUR_EASYCRON_API_KEY"

from datetime import datetime, timedelta, timezone

# Calculate the time 24 hours from now in EasyCron's expected format
# EasyCron uses a slightly different format, but for simplicity, we'll use the standard UTC format
# and convert this into a cron expression
time_24_hours_from_now = datetime.now(timezone.utc) + timedelta(hours=24)
cron_minute = time_24_hours_from_now.minute
cron_hour = time_24_hours_from_now.hour
cron_day = time_24_hours_from_now.day
cron_month = time_24_hours_from_now.month
cron_year = time_24_hours_from_now.year

cron_expression = f'{cron_minute} {cron_hour} {cron_day} {cron_month} * {cron_year}'
```

Now, we will schedule a `cron` job 24 hours from now with EasyCron to send a delete request to AssemblyAI for the transcript that we just generated.

```python
import requests

# EasyCron API endpoint for creating a new cron job
api_endpoint = 'https://www.easycron.com/rest/add'

# The data to be sent to EasyCron's API
data = {
    'token': EASYCRON_API_TOKEN,
    'url': url_to_call,
    'cron_expression': cron_expression,
    'http_method': "DELETE",
    'http_headers': f'Authorization: {AAI_API_TOKEN}',
    'timezone': 'UTC'
}

# Make the request to EasyCron's API
response = requests.post(api_endpoint, data=data)

# Print the response from EasyCron's API
print(response.text)
```

## Troubleshooting:

`Error 3: The cron expression you entered is invalid or it cannot be matched in a realitic future.`: Check that your [EasyCron account settings](https://www.easycron.com/user/edit) has UTC configured as the default timezone.

### Other resources

[AssemblyAI DELETE endpoint API reference](/api-reference/transcripts/delete)  
[EasyCron API reference](https://www.easycron.com/document/add)

---

<a id="troubleshoot-common-errors-docs-pre-recorded-audio-guides-common-errors-and-solutions-mdx"></a>

# Troubleshoot Common Errors

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/common_errors_and_solutions
Source: docs/pre-recorded-audio/guides/common_errors_and_solutions.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Error handling and audio file fixes
Description: Troubleshoot Common Errors documentation.

AssemblyAI's API always returns a response even when there is an error. This guide is designed to help you navigate and resolve common issues when implementing AssemblyAI.

## HTTP status codes

The AssemblyAI API uses HTTP response codes to indicate whether a request was successful.

The response codes generally fall into the following ranges:

- `2xx` indicates the request was successful.
- `4xx` indicates the request may have omitted a required parameter, or have invalid information.
- `5xx` indicates an error on AssemblyAI's end.

Below is a summary of the HTTP response codes you may encounter:

| Code          | Status            | Description                                       |
| ------------- | ----------------- | ------------------------------------------------- |
| 200           | OK                | Request was successful.                           |
| 400           | Bad request       | The request failed due to an invalid request.     |
| 401           | Unauthorized      | Missing or invalid API key.                       |
| 403           | Forbidden         | You don't have access to the requested resource.  |
| 404           | Not found         | The requested resource doesn't exist.             |
| 429           | Too many requests | Too many requests were sent to the API.           |
| 500, 503, 504 | Server error      | Something went wrong on AssemblyAI's end.         |

```json title="Response with error"
{
  "error": "Authentication error, API token missing/invalid"
}
```

  **API status**

  To stay up-to-date with any known service disruptions, subscribe to updates on
  the [Status](https://status.assemblyai.com) page.

## Understanding Errors with AssemblyAI

There are primarily two types of errors you might encounter when working with AssemblyAI:

- errors that occur when requesting a transcription (immediate HTTP `4xx` / `5xx`, message in the `error` field of the JSON body)
- errors that occur while the transcription is processing (the submit call returns `200`, but a later `GET /v2/transcript/{id}` shows `status: error` with the message in the `error` field)

The first category includes issues such as authentication errors, invalid request parameters, or server-related errors, which are typically flagged immediately when you attempt to initiate a request.

The second category, failed transcription jobs, pertains to errors that occur during the transcription process itself. These might be due to issues with the audio file, unsupported languages, or internal errors on our end.

## Handling Errors with AssemblyAI

### Error handling with AssemblyAI's SDKs (Recommended)

When using any of our SDKs, both types of errors are surfaced via the error key in a transcript object. For example in Python:

```python
if transcript.status == aai.TranscriptStatus.error:
    print(transcript.error)
```

### Error handling with HTTPS requests

For errors when making a request for a transcription, you will have to check the status code that we respond with. For errors that occur during the transcription process, you will need to access the "error" key in the JSON response. For other HTTP errors you can print the information from the response object. Here is an example you can use:

```python
if response.status_code != 200:
    try:
        print(response.json()['error'])
    except Exception:
        print(response.status_code, response.text, response.url)
```

## Common errors when making a request for a transcription

Unless otherwise noted, these errors are returned from `POST /v2/transcript` with the message in the `error` field of the JSON body.

### Missing `audio_url`

```json
# Status code: 400
{
"error": "Transcript creation error, audio_url not found"
}
```

This error occurs when the request body does not include the required `audio_url` field.

- **Solution**: Include an `audio_url` field that points to a publicly accessible audio file or to a URL returned by the [Upload endpoint](/api-reference/files/upload).

### Unsupported language code

```json
# Status code: 400
{
"error": "Language code <code> is not supported. See https://www.assemblyai.com/docs/concepts/supported-languages."
}
```

The `language_code` value you submitted isn't recognized.

- **Solution**: Check the [Supported languages](/pre-recorded-audio/supported-languages) page and submit a supported language code. You can also enable [Automatic Language Detection](/pre-recorded-audio/language-detection) instead of setting `language_code` manually.

### Invalid `speech_models` value

```json
# Status code: 400
{
"error": "\"speech_models\" must be a non-empty list containing one or more of: \"universal-3-pro\", \"universal-2\""
}
```

The `speech_models` parameter must be a non-empty list of supported model names.

Accounts on a feature flag for dated `universal-3-5-pro` variants may see additional model names in the list.

- **Solution**: Pass a non-empty list, for example `["universal-3-5-pro", "universal-2"]`. See [Select the speech model](/pre-recorded-audio/select-the-speech-model) for guidance.

### Deprecated `speech_model` (singular) parameter on async `/v2/transcript`

On the **async pre-recorded API**, `speech_model` no longer selects a model. Use **`speech_models`** (plural) instead.

- Passing a current model name (`universal-2`, `universal-3-5-pro`) to `speech_model` returns a **400 error** — deprecated.
- Passing a legacy name (`best`, `universal`) to `speech_model` returns **200**, but always routes to the default model.
- Omitting both `speech_model` and `speech_models` also routes to the default model.

This default-routing behavior is intentional — it keeps requests without an explicit current-model selection on the best model available, rather than pinned to an outdated one.

**Default model:** `speech_model` currently defaults to `universal-2`, the default will be updated to `universal-3-5-pro` on August 7. If you would like to explicitly pin a model, set the `speech_models` parameter. See [model selection](/pre-recorded-audio/select-the-speech-model) to compare models.

This entry applies to the async pre-recorded API (`POST /v2/transcript`) only. The Streaming API is a separate system that uses its own singular `speech_model` field with values like `universal-streaming-english` and `universal-3-5-pro` — see [Streaming model selection](/streaming/select-the-speech-model). Don't treat the singular `speech_model` field as universally deprecated.

### `language_detection` conflicts with `language_code`

```json
# Status code: 400
{
"error": "`language_detection` is not available when `language_code` is specified."
}
```

You cannot set both `language_detection: true` and a specific `language_code` in the same request.

The plural `language_codes` field returns a different (similarly worded) message.

- **Solution**: Either remove `language_code` and let [Automatic Language Detection](/pre-recorded-audio/language-detection) pick the language, or remove `language_detection` and keep `language_code`.

### Wrong-type validation (Boolean, etc.)

```json
# Status code: 400
{
"error": "`punctuate` must be a Boolean"
}
```

This is a generic wrong-type validation message. The field name changes depending on which parameter was submitted with the wrong type — any Boolean parameter sent as a non-Boolean value will trigger an equivalent message.

- **Solution**: Send the parameter with the correct JSON type (`true`/`false` for Booleans, an integer for integer fields, a string for string fields, etc.).

### Invalid `webhook_url`

```json
# Status code: 400
{
"error": "Invalid `webhook_url`, `webhook_url` should start with http."
}
```

The `webhook_url` field must be a valid URL beginning with `http://` or `https://`.

- **Solution**: Submit a fully-qualified webhook URL. See [Webhooks](/pre-recorded-audio/webhooks) for details.

### Missing `redact_pii_policies`

```json
# Status code: 400
{
"error": "You must explicitly define 'redact_pii_policies'"
}
```

This error fires when [PII Redaction](/guardrails/redact-pii-from-transcripts) is enabled but `redact_pii_policies` is missing.

This only applies to accounts created after `redact_pii_policies` became required. Older accounts auto-fill defaults and do not error.

- **Solution**: Provide an explicit list of policies, for example `"redact_pii_policies": ["person_name", "phone_number"]`. See [Redact PII](/guardrails/redact-pii-from-transcripts) for the full list of supported policy names.

### Unsupported policy name in `redact_pii_policies`

```json
# Status code: 400
{
"error": "Unsupported policy name in 'redact_pii_policies', supported policy names are [...]"
}
```

One of the values in `redact_pii_policies` isn't a recognized policy.

- **Solution**: Use only policies from the list in the error message and in the [Redact PII](/guardrails/redact-pii-from-transcripts) docs.

### Invalid `redact_pii_audio_quality`

```json
# Status code: 400
{
"error": "Invalid `redact_pii_audio_quality` value, choose 'mp3' or 'wav'"
}
```

`redact_pii_audio_quality` only accepts `"mp3"` or `"wav"`.

- **Solution**: Set `redact_pii_audio_quality` to `"mp3"` or `"wav"`.

### Malformed JSON body

```json
# Status code: 400
{
"error": "Error processing request, json data invalid"
}
```

The request body could not be parsed as JSON.

- **Solution**: Verify the body is valid JSON, the `Content-Type: application/json` header is set, and that no stray trailing commas, comments, or unescaped characters are present.

### Models are not supported for a particular language

```json
# Status code: 400
{
"error": "The following models are not available in this language: speaker_labels"
}
```

This error occurs when you set `language_code` explicitly and enable a feature that isn't supported for that language. The API validates feature compatibility upfront and rejects the request.

This error only applies when using `language_code`. If you use [Automatic Language Detection](/pre-recorded-audio/language-detection) instead, the request will complete successfully, but any unsupported features will be silently omitted from the response. See [Unsupported feature behavior](/pre-recorded-audio/supported-languages#unsupported-feature-behavior) for more details.

- **Solution**: Check the [Supported languages](/pre-recorded-audio/supported-languages) page to verify which features are available for your language before submitting a request.

### Insufficient Funds

```json
# Status code: 400
{
"error": "Your current account balance is negative. Please top up to continue using the API."
}
```

**Solution**:

- **Auto-pay**: Enable auto-pay in your account settings to automatically add funds when your balance runs low.
- **Check Funds**: Regularly check your account balance to ensure you have sufficient funds for your transcription requests.
- **Add Funds**: If needed, add funds to your account to continue using our services without interruption.

### Invalid API Token

```json
# Status code: 401
{
"error": "Authentication error, API token missing/invalid."
}
```

An invalid API token will prevent you from making successful requests.

**Solution**:

- Double-check that the API token you're using matches exactly with the token shown in your dashboard. Even a small typo can lead to authentication errors.

### Transcript ID not found

```json
# Status code: 400
{
"error": "Transcript lookup error, transcript id not found"
}
```

**Solution**:

- **Verify the endpoint and method that you are using**: Check that you are making a `POST` request to `http://api.assemblyai.com/v2/transcript`or a `GET` request to `http://api.assemblyai.com/v2/transcript/{transcript_id}` and not `http://api.assemblyai.com/v2/transcript/`
- **Token Verification**: Double-check that the API token you're using matches exactly with the token used to make the original request.
- If you're using Postman, ensure that `Encode URL automatically` under Settings is **disabled**.

### Server Errors

```json
# Status code: 500
{
"error": "Internal server error. Please retry your request. If the error persists, please contact support@assemblyai.com for more information."
}
```

Server errors rarely happen but can occasionally occur on our side.

**Solution**:

- **Retries**: Implement retries in your code for when our server returns a 500 code response. See [Implement Retry Server Error Logic](/pre-recorded-audio/guides/retry-server-error) for a worked example.
- **Automatic Retries**: Enable automatic retries for your transcription jobs under [Account > Settings](https://www.assemblyai.com/dashboard/home) on your dashboard. This ensures that if a job fails due to a temporary server issue, it will automatically be retried.
- **Check our Status page** to verify that we are not currently undergoing an incident
- **Reach out to Support**: Remember to provide the transcript ID, audio file used, and parameters used in your request or the full JSON response in your message. You can also email support@assemblyai.com for help!

## Errors on export, sub-resource, and upload endpoints

### Requesting captions, sentences, or paragraphs before the transcript completes

```json
# Status code: 400 — GET /v2/transcript/{id}/srt (or /vtt, /sentences, /paragraphs)
{
"error": "This transcript has a status of '<status>'. Transcripts must have a status of 'completed' before requesting captions."
}
```

The export endpoints can only be called after the transcript reaches `status: completed`. On the `/sentences` and `/paragraphs` endpoints the word `captions` in the message is replaced with `sentences` or `paragraphs` respectively.

- **Solution**: Poll `GET /v2/transcript/{id}` (or use a webhook) and only request captions/sentences/paragraphs once `status` is `completed`.

### `word-search` called without `words`

```json
# Status code: 400 — GET /v2/transcript/{id}/word-search
{
"error": "`words` is a required query parameter"
}
```

The `/word-search` endpoint requires the `words` query parameter.

- **Solution**: Pass a comma-separated list of search terms, for example `?words=foo,bar`. See the [Word Search](/api-reference/transcripts/word-search) reference.

### Redacted audio not available

```json
# Status code: 400 — GET /v2/transcript/{id}/redacted-audio
{
"error": "A redacted audio file for this transcript ID doesn't exist. Try submitting the transcript again with`redact_pii_audio`set to`true`."
}
```

The redacted audio file is only generated when [PII Audio Redaction](/guardrails/redact-pii-from-transcripts) was enabled at submission time.

- **Solution**: Resubmit the audio with `redact_pii: true` and `redact_pii_audio: true`. Set `redact_pii_policies` and (optionally) `redact_pii_audio_quality` to control what is redacted and the output format.

### Invalid subtitle format

```text
# Status code: 404 — GET /v2/transcript/{id}/<invalid-subtitle-format>
404: Not Found
```

When you request an unsupported subtitle format (anything other than `srt` or `vtt`), the API returns an HTML 404 page. The response body is the literal string `404: Not Found` — **not** JSON.

This is the only documented endpoint where a 404 does not come back as a JSON object with an `error` field. A "handled" 404 — for example, looking up an unknown transcript ID — still returns JSON in the usual shape.

- **Solution**: Use `srt` or `vtt`. See [Export SRT/VTT subtitles](/pre-recorded-audio/export-transcripts-as-srt-vtt-or-text).

### Empty upload body

```text
# Status code: 422 — POST /v2/upload
Upload failed, please try again. If you continue to have issues please reach out to support@assemblyai.com
```

Posting an empty body to `/v2/upload` returns a `422`. The response body is **plain text**, not JSON.

- **Solution**: Stream the file bytes as the request body. See the [Upload endpoint](/api-reference/files/upload) reference.

### Out-of-range `limit` when listing transcripts

```json
# Status code: 400 — GET /v2/transcript
{
"error": "'limit' must be greater than or equal to 1"
}
```

The `limit` query parameter on `GET /v2/transcript` must be `>= 1`.

- **Solution**: Send a positive integer for `limit`. See [List transcripts](/api-reference/transcripts/list).

## Common transcription processing errors

These errors come back from `GET /v2/transcript/{id}` after the submit call returned `200`. The transcript has `status: error` and the message is in the `error` field.

### Audio File URL Errors

#### Attempting to transcribe webpages or non-audio URLs

```json
{
  "status": "error",
  "audio_url": "https://www.youtube.com/watch?v=r8KTOBOMm0A",
  "error": "Transcoding failed. File type <mime> (<detail>) may be unsupported or an unexpected error occurred. See supported formats at https://www.assemblyai.com/docs/faq/what-audio-and-video-file-types-are-supported-by-your-api. If your file format is supported and you're still encountering this error, contact us at support@assemblyai.com.",
  ...
}
```

Our API requires a publicly accessible URL that points to an audio or video file. If the URL points to a webpage, an unsupported file type, or transcoding otherwise fails, this error is returned. To transcribe a YouTube video, [check out this Cookbook](/pre-recorded-audio/guides/transcribe_youtube_videos).

- **Solution**: Confirm the URL serves a [supported audio or video format](/faq/what-audio-and-video-file-types-are-supported-by-your-api). If the format is supported and you still see this error, contact [support@assemblyai.com](mailto:support@assemblyai.com).

#### No audio stream in the file

```json
{
  "status": "error",
  "error": "No audio stream found in the file. File type is <mime> (<detail>)."
}
```

The file downloaded successfully but contains no audio stream (for example, an image, a PDF, or a video file whose audio track has been stripped).

- **Solution**: Verify the file actually contains audio (for example with `ffprobe`). Re-encode or replace the file before resubmitting.

#### File exceeds the 5.5 GB maximum

```json
{
  "status": "error",
  "error": "<audio_url> length=<bytes> exceeds max_content_length=5500000000"
}
```

The file is larger than the 5.5 GB cap. The size is read from the `Content-Length` header before download, so this error fires almost immediately.

- **Solution**: Split the file into smaller chunks (e.g. with `ffmpeg`) or downsample/transcode before submitting. See [Are there any limits on file size or file duration?](/faq/are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api).

#### URL returns an empty body

```json
{
  "status": "error",
  "error": "No content found at <url>. Please make sure the file exists and is not empty."
}
```

The URL was reachable but returned an empty response body.

- **Solution**: Confirm the file is actually present at the URL and the server returns a non-zero `Content-Length`. Re-upload the file if necessary.

#### Uploaded file not found / region mismatch

```json
{
  "status": "error",
  "error": "File not found at <url>. Make sure the file was uploaded recently and that you're transcribing from the same region."
}
```

The URL returned by `POST /v2/upload` could not be resolved. This usually means the upload has been garbage-collected, or the upload was made to one region (e.g. US) and the transcript was submitted in another region (e.g. EU).

- **Solution**: Re-upload the file and submit the transcript using the same regional base URL. See [Do you offer servers in the EU?](/faq/do-you-offer-servers-in-the-eu) for the per-region endpoints.

#### Attempting to transcribe audio files that are not accessible

```json
{
  "status": "error",
  "audio_url": "https://foo.bar",
  "error": "Download error, unable to download https://foo.bar. Please make sure the file exists and is accessible from the internet."
}
```

**Solution**:

- **Public Access**: Verify that the audio file URL is publicly accessible. Our servers cannot transcribe audio from private or restricted URLs.
- **Google Drive URLs**: For audio stored on Google Drive, consult our [Google Drive Transcription Cookbook](/pre-recorded-audio/guides/transcribing-google-drive-file) to correctly format your URLs for access.
- **Direct Upload**: Utilize the [AssemblyAI Upload endpoint](/api-reference/files/upload) to upload files directly from your device, eliminating the need for a public URL.
- **AWS S3 Pre-signed URLs**: [This Cookbook](/pre-recorded-audio/guides/transcribe_from_s3) shows you how to use pre-signed URLs for AWS S3 storage to provide secure, temporary access for transcription without making your files public.

**Time-limited URLs (presigned URLs, CDN tokens, etc.) must be fresh when you submit them.** URLs from services like AWS S3 presigned URLs, Azure Blob Storage SAS tokens, GCP signed URLs, and social media CDNs (e.g., TikTok, Instagram) expire after a set duration. If the URL expires before AssemblyAI downloads the file, the origin server returns a `403` or `404` and you will see this download error.

To avoid this:
1. Generate the time-limited URL immediately before submitting your transcription request — do not cache or store URLs for later use.
2. Ensure the URL expiration window is long enough for the file to be downloaded (at least a few minutes).
3. If you cannot control the URL lifetime, use the [Upload endpoint](/api-reference/files/upload) to upload the file directly to AssemblyAI instead.

### Audio File Errors

#### Attempting to transcribe audio files that are too short

```json
{
  "status": "error",
  "audio_url": "https://foo.bar",
  "error": "Audio duration is too short."
}
```

The minimum audio duration for a file submitted to our API is 160ms.

**Solution**:

- **Add error handling for this error message**: When this error occurs, handle it safely by checking the error string and returning the error.
- **Add pre-submit checks for the duration of the audio file**: Prior to submitting a file for transcription, check the duration using a tool like soxi (part of the SoX package): `soxi -D audio.mp3`

### Language Detection Errors

#### No spoken audio detected

```json
{
  "status": "error",
  "error": "language_detection cannot be performed on files with no spoken audio."
}
```

[Automatic Language Detection](/pre-recorded-audio/language-detection) requires spoken audio to analyze. Files that are silent, music-only, or otherwise contain no speech can't be language-detected.

- **Solution**: Disable `language_detection` and set `language_code` explicitly for files where speech may be absent, or skip transcription for files that are known to be non-speech.

#### Detected language confidence below the threshold

```json
{
  "status": "error",
  "error": "detected language '<code>', confidence <c>, is below the requested confidence threshold value of '<threshold>'"
}
```

This fires when `language_detection` is enabled and the detected language's confidence is below the `language_confidence_threshold` you set. **Erroring is the default behavior** when the threshold isn't met.

- **Solution**: Set `language_detection_options.on_low_language_confidence` to `fallback` to fall back to a language you specify instead of erroring. Alternatively, lower the `language_confidence_threshold`. See [Automatic Language Detection](/pre-recorded-audio/language-detection) for details.

### Completed transcript that's empty or shorter than expected

The transcript comes back with `status: completed` and no `error`, but is empty or much shorter than you expected. Check that `audio_duration` in the response matches your source file — a file that was still uploading or not fully reachable when we fetched it transcribes without error, just truncated. For dual-channel audio, also rule out [duplicate or cancelling channels](/pre-recorded-audio/guides/identify_duplicate_channels).

---

<a id="implement-retry-server-error-logic-docs-pre-recorded-audio-guides-retry-server-error-mdx"></a>

# Implement Retry Server Error Logic

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/retry-server-error
Source: docs/pre-recorded-audio/guides/retry-server-error.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Error handling and audio file fixes
Description: Implement Retry Server Error Logic documentation.

In this guide, we'll show you how to setup automatic server error retry logic in your transcription process.

Server errors indicate a server-side issue during the transcription process. These rarely happen, but can occasionally occur on our side. If a transcription fails due to a server error, we recommend that you resubmit the file for transcription to allow another server to process the audio. If the issue persists, please reach out to our support team: support@assemblyai.com

This workflow is designed to automatically retry these transcripts if a server error is encountered.

  If your transcription fails due to a server error on our side, we will
  automatically retry the request up to three times. You can find this option in
  your [Account Settings](https://www.assemblyai.com/dashboard/home).

## Quickstart

```python expandable
import assemblyai as aai
import time

aai.settings.api_key = "YOUR_API_KEY"

SERVER_ERROR_MESSAGES = (
    "Internal server error. Please retry your request. If the error persists, please contact support@assemblyai.com for more information.",
    # Legacy message string, kept for backwards compatibility.
    "Server error, developers have been alerted.",
)

def is_server_error(transcript):
    return transcript.error is not None and any(
        transcript.error.startswith(msg.rstrip(".")) for msg in SERVER_ERROR_MESSAGES
    )

def handle_error_transcription(audio_url, transcriber, config, retries=1, wait_time=5):
    for attempt in range(retries + 1):
        transcript = transcriber.transcribe(audio_url, config)
        if is_server_error(transcript):
            if attempt < retries:
                print(f"Encountered a server error. Retrying in {wait_time} second(s)...")
                time.sleep(wait_time)
            else:
                print("Retry failed with a server error. Please contact AssemblyAI Support: support@assemblyai.com")
                return None
        elif transcript.status == aai.TranscriptStatus.error:
            print(f"Transcription failed: {transcript.error}")
            return None
        else:
            return transcript

audio_url = "YOUR_AUDIO_URL"
transcriber = aai.Transcriber()
config = aai.TranscriptionConfig()

transcript = handle_error_transcription(audio_url, transcriber, config, retries=1, wait_time=5)
if transcript:
    print(transcript.text)

```

## Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Step-by-step instructions

Install the SDK:

```bash
pip install assemblyai
```

Import the `assemblyai` and `time` package and set your API key:

```python
import assemblyai as aai
import time

aai.settings.api_key = "YOUR_API_KEY"
```

Create a function that handles errors that may occur during the transcription process. The default number of retires is 1. The default wait time before retranscribing is 5 seconds.

The current server-error string returned by the API is `Internal server error. Please retry your request. If the error persists, please contact support@assemblyai.com for more information.` The legacy `Server error, developers have been alerted` string is also matched so older retries keep working.

```python
SERVER_ERROR_MESSAGES = (
    "Internal server error. Please retry your request. If the error persists, please contact support@assemblyai.com for more information.",
    # Legacy message string, kept for backwards compatibility.
    "Server error, developers have been alerted.",
)

def is_server_error(transcript):
    return transcript.error is not None and any(
        transcript.error.startswith(msg.rstrip(".")) for msg in SERVER_ERROR_MESSAGES
    )

def handle_error_transcription(audio_url, transcriber, config, retries=1, wait_time=5):
    for attempt in range(retries + 1):
        transcript = transcriber.transcribe(audio_url, config)
        if is_server_error(transcript):
            if attempt < retries:
                print(f"Encountered a server error. Retrying in {wait_time} second(s)...")
                time.sleep(wait_time)
            else:
                print("Retry failed with a server error. Please contact AssemblyAI Support: support@assemblyai.com")
                return None
        elif transcript.status == aai.TranscriptStatus.error:
            print(f"Transcription failed: {transcript.error}")
            return None
        else:
            return transcript
```

Define the audio file that you want to transcribe.

```python
audio_url = "YOUR_AUDIO_URL"
```

Create a `Transcriber` object and specify features in `TranscriptionConfig`.

```python
transcriber = aai.Transcriber()
config = aai.TranscriptionConfig()
```

Call the function to handle transcription with error handling. Specify number of retries and wait time. Return the transcribed text if transcription is successful.

```python
transcript = handle_error_transcription(audio_url, transcriber, config, retries=1, wait_time=5)
if transcript:
    print(transcript.text)
```

---

<a id="implement-retry-upload-error-logic-docs-pre-recorded-audio-guides-retry-upload-error-mdx"></a>

# Implement Retry Upload Error Logic

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/retry-upload-error
Source: docs/pre-recorded-audio/guides/retry-upload-error.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Error handling and audio file fixes
Description: Implement Retry Upload Error Logic documentation.

In this guide, we'll show you how to set up automatic upload error retry logic in your transcription process.

Upload errors could be a result of a transient issue with our servers or they could be related to an issue with the file itself. Most likely the issue would be that the file is empty. Because the cause can be unclear at first, we recommend adding some retry logic to handle the rare occasions in which our upload service is experiencing performance issues. If the upload failure persists, you'll want to check whether the file is empty. If you're unclear on why the file is failing, please reach out to our support team at support@assemblyai.com.

This workflow is designed to automatically retry file uploads if an upload error is encountered.

## Quickstart

```python expandable
import assemblyai as aai
import time
from assemblyai.types import TranscriptError

aai.settings.api_key = "YOUR_API_KEY"

def transcribe_with_upload_retry(file_path, retries=3, delay=5):
    transcriber = aai.Transcriber()

    for attempt in range(retries):
        try:
            # Attempt to transcribe the file
            config = aai.TranscriptionConfig(speaker_labels=True)
            transcript = transcriber.transcribe(file_path, config)
            return transcript

        except TranscriptError as e:
            # Handle specific error if upload fails
            print(f"Attempt {attempt + 1} failed. {e}")

            # Retry if a TranscriptError occurs,
            if attempt + 1 < retries:
                print(f"Retrying in {delay} seconds...")
                time.sleep(delay)
            else:
                raise e  # Raise the error after max retries

    print("Max retries reached. Transcription failed.")
    return None

transcribe_with_upload_retry("YOUR_AUDIO_URL")

```

## Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Step-by-step instructions

Install the SDK:

```bash
pip install assemblyai
```

Import the `assemblyai` package and assemblyai's `TranscriptError`. Additionally import the and `time` package and set your API key:

```python
import assemblyai as aai
import time
from assemblyai.types import TranscriptError

aai.settings.api_key = "YOUR_API_KEY"
```

Create a function that retries upload failures. This example retries up to 3 times with a delay of 5 seconds each time.

```python expandable
def transcribe_with_upload_retry(file_path, retries=3, delay=5):
    transcriber = aai.Transcriber()

    for attempt in range(retries):
        try:
            # Attempt to transcribe the file
            config = aai.TranscriptionConfig(speaker_labels=True)
            transcript = transcriber.transcribe(file_path, config)
            return transcript

        except TranscriptError as e:
            # Handle specific error if upload fails
            print(f"Attempt {attempt + 1} failed. {e}")

            # Retry if a TranscriptError occurs,
            if attempt + 1 < retries:
                print(f"Retrying in {delay} seconds...")
                time.sleep(delay)
            else:
                raise e  # Raise the error after max retries

    print("Max retries reached. Transcription failed.")
    return None

transcribe_with_upload_retry("YOUR_AUDIO_URL")
```

---

<a id="identify-duplicate-dual-channel-files-docs-pre-recorded-audio-guides-identify-duplicate-channels-mdx"></a>

# Identify Duplicate Dual Channel Files

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/identify_duplicate_channels
Source: docs/pre-recorded-audio/guides/identify_duplicate_channels.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Error handling and audio file fixes
Description: Identify Duplicate Dual Channel Files documentation.

In this guide, you'll learn how to identify Dual Channel (Stereo) files that contain the same content on each channel. With AssemblyAI's API, we support Multi Channel files by transcribing each channel individually, then intelligently combining the results into one transcript. If a file contains the same content on both channels, then you may end up with duplicate content in your transcript.

To programmatically identify duplicate channels, you can hash the audio content found on each channel and compare the hashes to quickly determine whether you should remove a channel and then submit the file for more accurate results.

## Getting Started

To get started, you'll need to install a few prerequisites:

```bash
pip install -U pydub numpy audioop-lts assemblyai
```

## Helper Functions

Now we'll need to define a few helper functions before we set up our full workflow. The first one, `load_audio` will load any audio file, regardless of format, and return each channel as a NumPy array.

```python expandable
from pydub import AudioSegment
import numpy as np

def load_audio(file_path):
    # Load the audio file.
    audio = AudioSegment.from_file(file_path)

    # Ensure audio is stereo.
    if audio.channels != 2:
        raise ValueError("This function only supports stereo audio files.")

    # Convert audio data to raw samples.
    samples = np.array(audio.get_array_of_samples())

    # Reshape samples to separate channels.
    samples = samples.reshape((-1, audio.channels))

    # Extract left and right channels.
    left_channel = samples[:, 0]
    right_channel = samples[:, 1]

    return left_channel, right_channel
```

Our next function, `hash_audio_data` hashes each channel to return a SHA-256 hash.

```python
import hashlib

def hash_audio_data(data):
    hasher = hashlib.sha256()
    hasher.update(data.tobytes())
    return hasher.hexdigest()
```

Our next function, `compare_audio_channels` lets us know via a flag if the audio channels are the same, with 1 meaning that they are the same, and 0 meaning they're not.

```python
def compare_audio_channels(file_path):
    try:
        left_channel, right_channel = load_audio(file_path)

        # Compute hashes for both channels.
        left_hash = hash_audio_data(left_channel)
        right_hash = hash_audio_data(right_channel)

        # Compare hashes.
        if left_hash == right_hash:
            return True
        else:
            return False

    except Exception as e:
        print(f"An error occurred: {e}")
```

Based on the result of this function, we now need a way to remove the duplicate content to avoid duplicate transcription results. To do this, we'll define one final function, `convert_to_mono_if_duplicate` that removes one of the audio channels and creates a new mono file that we can submit to AssemblyAI, returning the path to it. If the file contains different content on each channel, however, this function will instead return the original file path.

```python expandable
import os

def convert_to_mono_if_duplicate(file_path):
    if compare_audio_channels(file_path):
        print("The audio content for this file is identical on both channels. Converting to mono...")

        # Load the audio file again with pydub.
        audio = AudioSegment.from_file(file_path)

        # Convert to mono by selecting one channel (since both are identical).
        mono_audio = audio.set_channels(1)

        # Save the new mono file.
        output_path = os.path.splitext(file_path)[0] + "_mono" + os.path.splitext(file_path)[1]
        mono_audio.export(output_path, format=os.path.splitext(file_path)[1][1:])

        print("File converted.\n")

        return output_path
    else:
        return file_path
```

## Combining with AssemblyAI

Now we can combine this workflow with AssemblyAI to programmatically catch files that could cause duplicate results and modify them before submitting them to AssemblyAI. To get started, you'll need an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

Now we'll create a workflow that checks if we should convert a file before using AssemblyAI's SDK to transcribe it.

```python expandable
import assemblyai as aai

aai.settings.api_key = "API_KEY"

original_path = "FILE_PATH"

updated_path = convert_to_mono_if_duplicate(original_path)

# Check if we modified the file and thus appended _mono to it.
if updated_path != original_path:
    # Use the default configuration, which will treat the file as mono, which it now is.
    config = aai.TranscriptionConfig()
    transcriber = aai.Transcriber(config=config)

    # Transcribe our new mono file.
    print(transcriber.transcribe(updated_path).text)
else:
    # Submit the file as Multi Channel if the content wasn't the same.
    config = aai.TranscriptionConfig(multichannel=True)

    # Load the config into our Transcriber.
    transcriber = aai.Transcriber(config=config)

    # Transcribe the stereo file.
    print(transcriber.transcribe(updated_path).text)
```

---

<a id="correct-audio-duration-discrepancies-with-multi-tool-validation-and-transcoding-docs-pre-recorded-audio-guides-audio-duration-fix-mdx"></a>

# Correct Audio Duration Discrepancies with Multi-Tool Validation and Transcoding

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/audio-duration-fix
Source: docs/pre-recorded-audio/guides/audio-duration-fix.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Error handling and audio file fixes
Description: Correct Audio Duration Discrepancies with Multi-Tool Validation and Transcoding documentation.

In this guide, you'll learn how to check the audio duration of a file using three different tools: `ffprobe`, `SoX`, and `MediaInfo`. This guide was created in response to customer feedback about transcription results showing incorrect audio durations. The issue was traced to audio files with corrupted metadata or problematic headers, leading to inaccurate duration data. If these tools report differing durations for the same file, transcription inconsistencies can arise. We will programmatically detect any duration mismatches and transcode the file to resolve them, typically resulting in a more accurate transcription.

## Quickstart

```python expandable
import assemblyai as aai
import subprocess

aai.settings.api_key = "YOUR_API_KEY"
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()

def get_duration_ffprobe(file_path):
    command = [
        'ffprobe', '-v', 'error', '-show_entries',
        'format=duration', '-of',
        'default=noprint_wrappers=1:nokey=1', file_path
    ]
    try:
        duration = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
        return float(duration.stdout.strip())
    except ValueError:
        print("Error: Unable to parse duration from ffprobe output.")
        return None

def get_duration_sox(file_path):
    command = ['soxi', '-D', file_path]
    try:
        duration = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
        return float(duration.stdout.strip())
    except ValueError:
        print("Error: Unable to parse duration from SoX output.")
        return None

def get_duration_mediainfo(file_path):
    command = ['mediainfo', '--Output=General;%Duration%', file_path]
    try:
        duration_ms = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
        duration_str = duration_ms.stdout.strip()
        # Check if the output is empty or not a valid number
        if duration_str:
            return float(duration_str) / 1000
        else:
            print("Error: MediaInfo returned empty or invalid duration")
            return None
    except ValueError:
        print("Error: Unable to parse duration from MediaInfo output.")
        return None

def check_audio_durations(file_path):
    #Check if audio durations differ among the three tools.
    ffprobe_duration = get_duration_ffprobe(file_path)
    sox_duration = get_duration_sox(file_path)
    mediainfo_duration = get_duration_mediainfo(file_path)

    # Print all retrieved durations
    print(f"ffprobe duration: {ffprobe_duration:.6f} seconds" if ffprobe_duration is not None else "ffprobe duration: Error retrieving duration")
    print(f"SoX duration: {sox_duration:.6f} seconds" if sox_duration is not None else "SoX duration: Error retrieving duration")
    print(f"MediaInfo duration: {mediainfo_duration:.6f} seconds" if mediainfo_duration is not None else "MediaInfo duration: Error retrieving duration")

    # Return durations for further checks
    return (ffprobe_duration, sox_duration, mediainfo_duration)

def transcribe(file):
    print("Executing transcription as audio durations are consistent.")
    transcript = transcriber.transcribe(file, config)
    print(transcript.text)

def transcode(input_file, output_file):
    #Transcode audio file to a 16kHz WAV file.
    print(f"Transcoding file {input_file} to {output_file}...")
    command = [
        'ffmpeg', '-i', input_file, '-ar', '16000', '-ac', '1', output_file
    ]
    try:
        subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        if os.path.exists(output_file):
            print(f"Transcoding complete. Output file: {output_file}")
        else:
            print("Error: Transcoding failed.")
    except subprocess.CalledProcessError as e:
        print("Warnings from ffmpeg")
        """Print errors or warnings from ffmpeg"""
        #print(e.stderr.decode())

def durations_are_consistent(durations, tolerance=0.01):
    #Check if durations are consistent within a given tolerance of 0.01 seconds.
    if None in durations:
        return False
    min_duration = min(durations)
    max_duration = max(durations)
    return (max_duration - min_duration) <= tolerance

def main(file_path):
    durations = check_audio_durations(file_path)

    if durations:
        if None in durations:
            print("Error: One or more duration values could not be retrieved.")
            transcoded_file = file_path.rsplit('.', 1)[0] + '_transcoded.wav'
            transcode(file_path, transcoded_file)
            new_durations = check_audio_durations(transcoded_file)
            if new_durations and durations_are_consistent(new_durations):
                transcribe(transcoded_file)
            else:
                print("Warning: The audio durations still differ or an error occurred with the transcoded file.")
        elif not durations_are_consistent(durations):
            print("Warning: The audio durations differ between tools.")
            transcoded_file = file_path.rsplit('.', 1)[0] + '_transcoded.wav'
            transcode(file_path, transcoded_file)
            new_durations = check_audio_durations(transcoded_file)
            if new_durations and durations_are_consistent(new_durations):
                transcribe(transcoded_file)
            else:
                print("Warning: The audio durations still differ or an error occurred with the transcoded file.")
        else:
            print("The audio durations are consistent.")
            transcribe(file_path)

audio_file="./audio.mp4"

if __name__ == "__main__":
    file_path = f"{audio_file}"
    main(file_path)
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-Step Instructions

Install the SDK:

```bash
pip install assemblyai
```

Import the `assemblyai` package along with `subprocess`, set your AssemblyAI API key, and initiate the transcriber.

```python
import assemblyai as aai
import subprocess

aai.settings.api_key = "YOUR_API_KEY"
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
```

For this cookbook you will need [`ffmpeg`](https://www.ffmpeg.org/), [`sox`](https://sourceforge.net/projects/sox/), and [`MediaInfo`](https://mediaarea.net/en/MediaInfo). We will use these tools to pull the duration from the audio. Matching audio duration is crucial because discrepancies may indicate issues with the audio file's metadata or headers. Such inconsistencies can lead to inaccurate transcription results, playback issues, or unexpected behaviour in media applications. By verifying that the duration is consistent across all three tools, we can detect potential problems early and correct any corrupted metadata or faulty headers before processing the audio further.

First, we will get the audio duration using `ffprobe`.

```python
def get_duration_ffprobe(file_path):
    command = [
        'ffprobe', '-v', 'error', '-show_entries',
        'format=duration', '-of',
        'default=noprint_wrappers=1:nokey=1', file_path
    ]
    try:
        duration = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
        return float(duration.stdout.strip())
    except ValueError:
        print("Error: Unable to parse duration from ffprobe output.")
        return None

```

Next, we will get the audio duration for the same file using `sox`.

```python
def get_duration_sox(file_path):
    command = ['soxi', '-D', file_path]
    try:
        duration = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
        return float(duration.stdout.strip())
    except ValueError:
        print("Error: Unable to parse duration from SoX output.")
        return None
```

Finally, we will get the audio duration for the same file using `MediaInfo`.

```python
def get_duration_mediainfo(file_path):
    command = ['mediainfo', '--Output=General;%Duration%', file_path]
    try:
        duration_ms = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
        duration_str = duration_ms.stdout.strip()
        # Check if the output is empty or not a valid number
        if duration_str:
            return float(duration_str) / 1000
        else:
            print("Error: MediaInfo returned empty or invalid duration")
            return None
    except ValueError:
        print("Error: Unable to parse duration from MediaInfo output.")
        return None

```

The following function will return the durations from the three tools and convert them to the same format.

```python
def check_audio_durations(file_path):
    #Check if audio durations differ among the three tools.
    ffprobe_duration = get_duration_ffprobe(file_path)
    sox_duration = get_duration_sox(file_path)
    mediainfo_duration = get_duration_mediainfo(file_path)

    # Print all retrieved durations
    print(f"ffprobe duration: {ffprobe_duration:.6f} seconds" if ffprobe_duration is not None else "ffprobe duration: Error retrieving duration")
    print(f"SoX duration: {sox_duration:.6f} seconds" if sox_duration is not None else "SoX duration: Error retrieving duration")
    print(f"MediaInfo duration: {mediainfo_duration:.6f} seconds" if mediainfo_duration is not None else "MediaInfo duration: Error retrieving duration")

    # Return durations for further checks
    return (ffprobe_duration, sox_duration, mediainfo_duration)
```

Define the `transcribe` function. This will run only when the duration is consistent among the three tools.

```python
def transcribe(file):
    print("Executing transcription as audio durations are consistent.")
    transcript = transcriber.transcribe(file, config)
    print(transcript.text)
```

Define the `transcode` function. We will run this if one or more durations differ. The output file will be a 16kHz WAV file as that is the format AssemblyAI models are trained on. When running the `ffmpeg` command, the transcode may fail or return warnings if there are issues with the input file's format, corrupted metadata, or unsupported codecs. These warnings tend to be verbose but you can print them for troubleshooting.

```python
def transcode(input_file, output_file):
    #Transcode audio file to a 16kHz WAV file.
    print(f"Transcoding file {input_file} to {output_file}...")
    command = [
        'ffmpeg', '-i', input_file, '-ar', '16000', '-ac', '1', output_file
    ]
    try:
        subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        if os.path.exists(output_file):
            print(f"Transcoding complete. Output file: {output_file}")
        else:
            print("Error: Transcoding failed.")
    except subprocess.CalledProcessError as e:
        print("Warnings from ffmpeg")
        """Print errors or warnings from ffmpeg"""
        #print(e.stderr.decode())
```

Define a function that will check if the durations are consistent. There may be small differences so it's best to allow a small tolerance. In this example the tolerance value will be 0.01 seconds.

```python
def durations_are_consistent(durations, tolerance=0.01):
    #Check if durations are consistent within a given tolerance of 0.01 seconds.
    if None in durations:
        return False
    min_duration = min(durations)
    max_duration = max(durations)
    return (max_duration - min_duration) <= tolerance
```

Finally, here is the order of operations for this program. This program will first check the duration of an audio file across different tools to ensure consistency. If any tool fails to retrieve a duration or if the durations differ, it transcodes the audio to a new 16kHz WAV file and checks the duration of the WAV file. If the durations are consistent in the transcoded file, the program proceeds to transcribe it. If inconsistencies remain after transcoding, it logs a warning to highlight the issue and will not transcribe the file.

```python expandable
def main(file_path):
    durations = check_audio_durations(file_path)

    if durations:
        if None in durations:
            print("Error: One or more duration values could not be retrieved.")
            transcoded_file = file_path.rsplit('.', 1)[0] + '_transcoded.wav'
            transcode(file_path, transcoded_file)
            new_durations = check_audio_durations(transcoded_file)
            if new_durations and durations_are_consistent(new_durations):
                transcribe(transcoded_file)
            else:
                print("Warning: The audio durations still differ or an error occurred with the transcoded file.")
        elif not durations_are_consistent(durations):
            print("Warning: The audio durations differ between tools.")
            transcoded_file = file_path.rsplit('.', 1)[0] + '_transcoded.wav'
            transcode(file_path, transcoded_file)
            new_durations = check_audio_durations(transcoded_file)
            if new_durations and durations_are_consistent(new_durations):
                transcribe(transcoded_file)
            else:
                print("Warning: The audio durations still differ or an error occurred with the transcoded file.")
        else:
            print("The audio durations are consistent.")
            transcribe(file_path)

audio_file="./audio/8950.mp4"

if __name__ == "__main__":
    file_path = f"{audio_file}"
    main(file_path)
```

If you continue to experience unexpected behaviour with your file, please contact our support team at support@assemblyai.com for assistance in diagnosing the issue.

---

<a id="audio-file-downsampling-recommendations-and-best-practices-docs-pre-recorded-audio-guides-downsampling-mdx"></a>

# Audio File Downsampling Recommendations and Best Practices

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/downsampling
Source: docs/pre-recorded-audio/guides/downsampling.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Error handling and audio file fixes
Description: Learn best practices for reducing audio file size while maintaining transcription quality with AssemblyAI

This tutorial will cover best practices when it comes to reducing audio file size while still ensuring proper processing by our API.

Though AssemblyAI recommends providing audio to our API in the highest quality and form closest to the original recording whenever possible, there may be times where this isn't possible due to outside constraints such as storage costs, processing speeds, transfer costs, etc.

  Reducing file size generally comes at the cost of transcription quality as
  audio data is being lost in the process which our ASR models use to
  transcribe. You should consider whether this tradeoff is acceptable for your
  specific use case.

## Sample rate, bit depth, and compression

**Sample Rate**: A higher sample rate captures more detail in the audio signal. For speech-to-text applications, 16 kHz is typically sufficient, as it accurately covers the frequency range of human speech without unnecessary overhead.

**Bit Depth**: Bit depth affects dynamic range, or the difference between the quietest and loudest sounds. A minimum of 16-bit is recommended for speech-to-text, offering a good balance between audio quality and file size.

**Compression**: Lossless compression preserves all audio data, ensuring maximum quality but resulting in larger files. Lossy compression reduces file size by removing some audio detail, which may impact transcription accuracy.

## Lossy vs lossless audio

First, determine which type of audio you are working with: Lossy or Lossless audio.

- **Lossless audio** is uncompressed, meaning it preserves all audio details, making it ideal for Speech-to-Text applications where accuracy is paramount. Common examples include wav and flac.

- **Lossy audio** is compressed, meaning some audio data is discarded to reduce file size. Common examples include mp3, aac, and m4a (sometimes). For a full list of supported audio formats, see [this page of our FAQ](/faq/what-audio-and-video-file-types-are-supported-by-your-api).

You may not necessarily need to convert your lossless audio to a lossy type, but formats like `mp3` are popular due to their efficient compression while still retaining relatively high quality. What's more important is understanding your desired sample rate and bit depth, then choosing the file format that works best for your use case. For a summary of the best formats and suitability for different Speech-to-Text use cases, see this [blog post](https://www.assemblyai.com/blog/best-audio-file-formats-for-speech-to-text).

  Simply converting a lossy audio type to a lossless audio type (i.e. mp3 to
  wav) will not add back any audio quality. The audio data is lost during
  compression, so this conversion will not improve transcription quality.

## Prerequisites

For this tutorial, you will need FFmpeg (which includes FFprobe).

FFmpeg is a command-line tool for converting, processing, and manipulating audio and video files between different formats, while ffprobe is a companion tool for analyzing and extracting detailed metadata information from media files without modifying them.

To download, go to [FFmpeg's official download page](https://ffmpeg.org/download.html).

If using homebrew, run the command: `brew install ffmpeg`

## Checking your audio

Before converting your audio, it's good to verify that the file's metadata accurately reflects the actual audio properties. Discrepancies between expected and actual metadata often indicate the audio has undergone previous processing steps, potentially compromising quality below anticipated levels.

```bash
ffprobe -v quiet -print_format json -show_format -show_streams FILE_PATH
```

What you want to pay attention to here are:

- `codec_name` - The name of the audio codec used to encode/decode the audio stream (e.g., mp3, aac, flac)
- `sample_rate` - The number of audio samples captured per second, measured in Hz (e.g., 44100 Hz for CD quality)
- `sample_fmt` - The format that defines how each audio sample is stored in memory (e.g., 16-bit integer, 32-bit float)
- `channels` - The number of separate audio channels in the stream (e.g., 1 for mono, 2 for stereo, etc.)
- `bits_per_sample` - The number of bits used to represent each individual audio sample, determining dynamic range and quality
- `bit_rate` - The amount of data used per second to represent the audio stream, measured in bits per second (bps) or kilobits per second (kbps)

Make sure they match what you would expect for the file type and recording method. For example, if I had what I believe to be a `wav`, but the metadata said the codec was `mp3` with a bitrate of 128 kbps, this would indicate the file was likely an mp3 that was incorrectly renamed with a .wav extension. Similarly, if you expected a high-quality music recording but saw a sample rate of 8000 Hz, this would suggest you're actually looking at a low-quality voice recording rather than the professional audio file you anticipated.

## Converting audio files

Once you've determined your audio file is being presented accurately, you'll want to use a command that resembles this:

```bash
ffmpeg -i input.wav -ar 16000 -ac 1 -ab 128k output.mp3
```

The above example can be broken down into:

- `-i input.wav` - Specifies the input file named "input.wav" that will be processed.
- `-ar 16000` - Sets the audio sample rate to 16,000 Hz (16 kHz) for the output file.
- `-ac 1` - Sets the audio channels to 1, converting the audio to mono (single channel). This should save on storage space, but if your file is stereo and you plan to enable `multichannel` for your request, set this to `2`.
- `-ab 128k` - Sets the audio bitrate to 128 kbps, common for mp3s.
- `output.mp3` - Specifies the output filename and format (mp3 file).

Note that the actual values and formats you use will vary between use cases, initial audio file quality, desired final file size, and other tradeoffs. It is recommended to experiment with different options to see what works best for your application.

## Converting video files

Though our API will accept certain video file formats, these files contain a large amount of video data that isn't useful during transcription. If you'd like to save on upload times and/or storage costs, you can also use `FFmpeg` to extract only the audio data:

```bash
ffmpeg -i input.mp4 -af aresample=async=1 -ac 1 -ar 48000 output.wav
```

Breaking down this command:

- `-i input.mp4` - Specifies the input file (an MP4 video file that contains audio to be extracted)
- `-af aresample=async=1` - Applies an audio filter (-af) that resamples the audio with async correction enabled to fix audio synchronization issues or timing inconsistencies. Optional: adds minimal processing overhead, but provides insurance against timing issues without affecting properly synced files.
- `-ac 1` - Sets the audio channels to 1, converting the audio to mono (single channel)
- `-ar 48000` - Sets the audio sample rate to 48,000 Hz (48 kHz) for professional audio quality
- `output.wav` - Specifies the output filename and format (uncompressed wav audio file)

  You can find a full list of FFmpeg conversion options
  [here](https://ffmpeg.org/ffmpeg.html#Options).

## Pre-transcription processing step

Ffmpeg is available in package form to introduce to your application's transcription flow programmatically. Instead of running these commands manually for each file:

```python expandable
import subprocess

def convert_audio(input_file, output_file):
    command = [
        'ffmpeg',
        '-i', input_file,
        '-ar', '16000',
        '-ac', '1',
        '-ab', '128k',
        output_file
    ]

    try:
        result = subprocess.run(command, check=True, capture_output=True, text=True)
        print("Conversion successful!")
        return True
    except subprocess.CalledProcessError as e:
        print(f"Error: {e.stderr}")
        return False

# Usage
convert_audio('input.wav', 'output.mp3')
```

```javascript expandable
import { spawn } from "child_process";

function convertAudio(inputFile, outputFile) {
  return new Promise((resolve, reject) => {
    const ffmpeg = spawn("ffmpeg", [
      "-i",
      inputFile,
      "-ar",
      "16000",
      "-ac",
      "1",
      "-ab",
      "128k",
      outputFile,
    ]);

    ffmpeg.on("close", (code) => {
      if (code === 0) {
        console.log("Conversion successful!");
        resolve();
      } else {
        reject(new Error(`FFmpeg exited with code ${code}`));
      }
    });

    ffmpeg.stderr.on("data", (data) => {
      console.error(`Error: ${data}`);
    });
  });
}

// Usage
convertAudio("input.wav", "output.mp3")
  .then(() => console.log("Done!"))
  .catch((err) => console.error(err));
```

Once this file is converted, provide it as the [audio_url](/api-reference/transcripts/submit#request.body.audio_url) in your transcription request.

  For troubleshooting purposes, it's useful to save both versions of your files
  when possible. For example, if you are only converting for transcription and
  discarding afterwards, you no longer have access to the file used during
  transcription.

## Conclusion

While numerous applications and services offer audio conversion capabilities, the methods outlined in this tutorial provide precise control over the conversion process and preserve the metadata essential for successful processing by AssemblyAI's transcription pipeline.

If you have any questions on downsampling or implementing a pre-processing flow to your application similar to the one described above, please contact our Support team at support@assemblyai.com.

---

<a id="translate-an-assemblyai-subtitle-transcript-docs-pre-recorded-audio-guides-translate-subtitles-mdx"></a>

# Translate an AssemblyAI Subtitle Transcript

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/translate_subtitles
Source: docs/pre-recorded-audio/guides/translate_subtitles.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Translation
Description: Translate an AssemblyAI Subtitle Transcript documentation.

In this guide, we'll show you how to translate an AssemblyAI generated subtitle transcript. We will also be using the `Googletrans` Python library to implement the Google Translate API.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Step-by-Step Instructions

Install the relevant packages.

1. AssemblyAI SDK
2. Googletrans

```bash
pip install -U assemblyai googletrans
```

Import the `assemblyai` package and set the API key. Import `Translator` class from `googletrans` package.

```python
import assemblyai as aai
from googletrans import Translator

aai.settings.api_key = "YOUR_API_KEY"
```

Create a `Transcriber` object.

```python
config = aai.TranscriptionConfig()
transcriber = aai.Transcriber()
```

Create a `Translator` object.

```python
translator = Translator()
```

Use the Transcriber object's `transcribe` method and pass in the audio file's path as a parameter. The transcribe method saves the results of the transcription to the `Transcriber` object's `transcript` attribute.

```python
transcript = transcriber.transcribe("./my-audio.mp3", config)
```

Alternatively, you can pass in a path to an audio file saved on the internet.

```python
transcript = transcriber.transcribe("https://example.org/audio.mp3", config)
```

Export SRT subtitles with the `export_subtitles_srt` method. Create a `subtitle_transcript` variable to translate in the next step.

```python
subtitle_transcript = transcript.export_subtitles_srt()
```

Translate subtitle transcript to target language

```python
translation = translator.translate(subtitle_transcript, dest='es')
```

Print results

```python
print(translation.text)
```

## Conclusion

Using the subtitles and transcripts generated by AssemblyAI, we can also generate translated alternatives using other translation APIs. The implementation logic will be similar with other solutions like DeepL, Yandex Translate and many more.

To translate to other languages, find the full list of Supported Languages in the **Further Documentations** section.

## Further Documentation

[Googletrans Library](https://pypi.org/project/googletrans/)

[Translation Supported Languages](https://py-googletrans.readthedocs.io/en/latest/#googletrans-languages)

---

<a id="translate-assemblyai-transcripts-into-other-languages-using-commercial-models-docs-pre-recorded-audio-guides-translate-transcripts-mdx"></a>

# Translate AssemblyAI Transcripts Into Other Languages Using Commercial Models

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/translate_transcripts
Source: docs/pre-recorded-audio/guides/translate_transcripts.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Translation
Description: Translate AssemblyAI Transcripts Into Other Languages Using Commercial Models documentation.

This Cookbook walks through how to translate AssemblyAI transcripts using a variety of commerical and open-source machine translation models.

Choosing a model depends on your use-case and preferences. Here are some considerations you may want to make when choosing a model to use for translation:

- Accuracy and Quality of Translation: you should compare the translations from each provider to see which translation you prefer
- Language Support: check the supported languages for [Google Translate](https://cloud.google.com/translate/docs/languages), [DeepL](https://www.deepl.com/docs-api/translate-text), and [python-translate](https://github.com/terryyin/translate-python) respectively
- Cost: while commercial models usually have a free-tier or trials, they will incur costs eventually

## Setup

To get started, paste your API token into the empty string below. If you don't already have an API token, you can get one for free [here](https://www.assemblyai.com/dashboard/signup).

```python
AAI_API_TOKEN = ""
```

**Make sure not to share this token with anyone** - it is a private key associated uniquely to your account.

Next, we'll install the AssemblyAI Python SDK.

```bash
pip install "assemblyai"
```

Finally, import the `assemblyai` package and set your API token in the settings:

```python
import assemblyai as aai

# set the API key
aai.settings.api_key = f"{AAI_API_TOKEN}"
```

```python
config = aai.TranscriptionConfig(language_detection=True)
transcriber = aai.Transcriber(config=config)

transcript = transcriber.transcribe('./my-audio.mp3')
```

Specify the target language for the translation

```python
to_lang = 'en'
```

Get detected language code from the AAI JSON response

```python
from_lang = transcript.json_response['language_code']
```

# Commercial Models

## Google Translate API

https://cloud.google.com/translate/docs/reference/libraries/v2/python

Note: you will need a GCP account as well as app credentials to make this API request.

```bash
pip install google-cloud-translate==2.0.1
```

Follow Google's docs on how to generate a credentials JSON file: https://cloud.google.com/docs/authentication/application-default-credentials

```python
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = './translate_creds.json'
```

```python expandable
from google.cloud import translate_v2

def translate(target: str, text: str) -> dict:
    """Translates text into the target language.

    Target must be an ISO 639-1 language code.
    See https://g.co/cloud/translate/v2/translate-reference#supported_languages
    """
    from google.cloud import translate_v2 as translate

    translate_client = translate.Client()

    if isinstance(text, bytes):
        text = text.decode("utf-8")

    # Text can also be a sequence of strings, in which case this method
    # will return a sequence of results for each text.
    result = translate_client.translate(text, target_language=target)

    print("Text: {}".format(result["input"]))
    print("Translation: {}".format(result["translatedText"]))
    print()

    return result

for sent in transcript.get_sentences():
  translate(to_lang, sent.text)
```

## DeepL API

https://www.deepl.com/en/docs-api

```bash
pip install deepl
```

You will need a DeepL account and API token, which can be found here: https://www.deepl.com/pro-api

```python
DEEPL_API_TOKEN = ''
```

```python
import deepl

def translate(text):
    translator = deepl.Translator(DEEPL_API_TOKEN)
    result = translator.translate_text(text, target_lang="EN-US") # Note: DeepL requires more formal language code
    return result.text

# Example usage
for sent in transcript.get_sentences():
  translated_text = translate(sent.text)
  print("Text: {}".format(sent.text))
  print("Translation: {}".format(translated_text))
  print()
```

# Open-Source Models

## `translate-python` Library

https://github.com/terryyin/translate-python

```bash
pip install translate
```

```python
from translate import Translator

def translate(text):
    translator = Translator(to_lang=to_lang, from_lang=from_lang)
    translation = translator.translate(text)
    return translation

for sent in transcript.get_sentences():
  translated_text = translate(sent.text)
  print("Text: {}".format(sent.text))
  print("Translation: {}".format(translated_text))
  print()
```

---

<a id="transform-chinese-transcripts-into-simplified-or-traditional-text-docs-pre-recorded-audio-guides-traditional-simplified-chinese-mdx"></a>

# Transform Chinese transcripts into Simplified or Traditional Text

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/traditional_simplified_chinese
Source: docs/pre-recorded-audio/guides/traditional_simplified_chinese.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Translation
Description: Transform Chinese transcripts into Simplified or Traditional Text documentation.

When transcribing Chinese audio, our models produce output that mixes both Simplified and Traditional Chinese characters. This happens because our models are typically trained on diverse datasets containing a mix of both writing systems.

This guide demonstrates a practical workaround for this using [OpenCC](https://github.com/BYVoid/OpenCC), an open-source Chinese conversion tool. We'll show you how to implement a post-processing step that can normalize your transcription output to either consistent Simplified Chinese or Traditional Chinese, depending on your needs.

While this guide uses Python, OpenCC is available across multiple programming languages.

# Quickstart

```python expandable
import assemblyai as aai
import opencc

aai.settings.api_key = "<YOUR-API-KEY>"

audio_file = "https://assembly.ai/chinese-interview.mp4"

config = aai.TranscriptionConfig(language_code="zh")

transcript = aai.Transcriber(config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

# t2s.json converts traditional characters to simplified
# use s2t.json to convert from simplified to traditional
converter = opencc.OpenCC('t2s.json')

simplified_transcript = converter.convert(transcript.text)

print(simplified_transcript)
```

# Step-by-step instructions

First, install the required packages:

1. AssemblyAI SDK
2. OpenCC

```bash
pip install -U assemblyai opencc
```

Import the necessary libraries and configure your API credentials:

```python
import assemblyai as aai
import opencc

aai.settings.api_key = "YOUR_API_KEY"
```

Specify your audio source and create a configuration for Chinese language transcription. Then submit your transcription request.

```python
audio_file = "https://assembly.ai/chinese-interview.mp4"

config = aai.TranscriptionConfig(language_code="zh")

transcript = aai.Transcriber(config=config).transcribe(audio_file)
```

Implement error handling to catch any transcription failures:

```python
if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")
```

Apply script conversion using OpenCC with the appropriate configuration:

```python
# Script conversion options:
# - 't2s.json': Traditional to Simplified
# - 's2t.json': Simplified to Traditional

# Create converter object with desired direction
converter = opencc.OpenCC('t2s.json') # For Traditional to Simplified

# Convert the transcript text
simplified_transcript = converter.convert(transcript.text)
```

Output or save your converted transcript:

```python
print(simplified_transcript)

# Optionally save to file
with open("converted_transcript.txt", "w", encoding="utf-8") as f:
    f.write(converted_transcript)
```

## Conclusion

This guide demonstrates how to solve the common challenge of mixed Chinese script systems in transcription outputs. By combining AssemblyAI's powerful speech recognition capabilities with OpenCC's script conversion tools, you can create a reliable pipeline for producing consistently formatted Chinese text from audio sources.

---

<a id="do-more-with-our-sdks-docs-pre-recorded-audio-guides-do-more-with-sdk-mdx"></a>

# Do More With Our SDKs

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/do-more-with-sdk
Source: docs/pre-recorded-audio/guides/do-more-with-sdk.mdx
Navigation: Pre-recorded STT > Guides > Tutorials > Do more with our SDKs
Description: Do More With Our SDKs documentation.

This guide will show you additional ways to make use of AssemblyAI's Python and JavaScript SDKs.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## How to Check and Update Your Version of the SDK

Sometimes errors are encountered because the version of the SDK you are using is not up to date. To see which version you are currently running, type this code in your terminal:

```bash
pip show assemblyai
```

If this version is not the same as the current version of the [Python SDK](https://github.com/AssemblyAI/assemblyai-python-sdk) then you can update your version by typing this code in your terminal:

```bash
pip install assemblyai --upgrade
```

```bash
npm info assemblyai version
```

If this version is not the same as the current version of the [JavaScript SDK](https://github.com/AssemblyAI/assemblyai-node-sdk) then you can update your version by typing this code in your terminal:

```bash
npm update assemblyai --save
```

## How to Catch and Log Errors

Catching and logging errors to the console is an easy way help you understand what is going wrong if the code does not run correctly.

Underneath the line of code where the transcript is created, `transcript = transcriber.transcribe(audio_url, config)`, add the following code to catch and log any errors to the terminal:

```python
if transcript.error:
    print(transcript.error)
```

Underneath the line of code where the transcript is created, `const transcript = await client.transcripts.transcribe(params)`, add the following code to catch and log any errors to the terminal:

```js
if (transcript.status === "error") {
  console.log(transcript.error);
}
```

## How to Log the Transcript JSON and Save it in a File

If using the error handling code above then add this below it, otherwise add it after the transcript is created, `transcript = transcriber.transcribe(audio_url, config)`:

```python
json_file = open('transcript.json', 'w', encoding='utf8')
json_str = json.dumps(transcript.json_response, ensure_ascii=False, indent=2)

json_file.write(json_str)
json_file.close()

print(json_str)
```

In order to write data to a file, first import the [fs](https://nodejs.org/api/fs.html) package:

```js
import fs from "fs";
```

If using the error handling code above then add this below it, otherwise add it after the transcript is created, `const transcript = await client.transcripts.transcribe(params)`:

```js
const transcriptJSON = JSON.stringify(transcript, null, "\t");

fs.writeFile("transcript.json", transcriptJSON, (err) => {
  if (err) throw err;
});

console.log(transcriptJSON);
```

## How to Log the Transcript ID and Retrieve a Previously Created Transcript

To log the transcript ID for a transcription, after the transcript is created and below any error handling, add the following code:

To log the transcript ID for a transcription, after the transcript is created and below any error handling, add the following code:

```python
print(transcript.id)
```

Use the following code to retrieve a previous transcript:

```python
transcript = aai.Transcript.get_by_id("<TRANSCRIPT_ID>")

print(transcript.text)
```

You can also retrieve multiple transcripts, which are then returned in a single `TranscriptGroup` object:

```python
transcript_group = aai.TranscriptGroup.get_by_ids(["<TRANSCRIPT_ID_1>", "<TRANSCRIPT_ID_2>"])

for transcript in transcript_group:
    print(transcript.text)
```

```js
console.log("Transcript ID: ", transcript.id)
```

To see a list of all previous transcriptions, use the following code:

```js
const allTranscriptsResponse = await fetch(
  "https://api.assemblyai.com/v2/transcript?limit=4",
  {
    method: "GET",
    headers: {
      Authorization: "<YOUR_API_KEY>",
    },
  }
);
const allTranscripts = await allTranscriptsResponse.json();
console.log(allTranscripts);
```

There are additional [query parameters](/api-reference/transcripts/list) that can be added to this request to limit the transcripts that are returned. The above example shows how to limit the number of returned transcripts to 4. These will be the four most recently created transcripts.

To get a specific transcript, use the following code:

```js
const transcriptResponse = await fetch(
  "https://api.assemblyai.com/v2/transcript/transcript_id",
  {
    method: "GET",
    headers: {
      Authorization: "<YOUR_API_KEY>",
    },
  }
);
const previousTranscript = await transcriptResponse.json();
console.log(previousTranscript);
```

Make sure when using the above code that you replace `transcript_id` in the url with the ID of the transcript you are looking to fetch.

---

<a id="quickstart-docs-streaming-getting-started-transcribe-streaming-audio-mdx"></a>

# Quickstart

URL: https://www.assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio
Source: docs/streaming/getting-started/transcribe-streaming-audio.mdx
Navigation: Real-time STT > Getting started
Description: Learn how to transcribe streaming audio.

## Overview

By the end of this guide, you'll have a working script that transcribes a live internet radio stream, printing each turn as it plays. No microphone required, so it runs anywhere, including a coding agent's sandbox. Build it with an AI coding agent, or write it yourself. Both are below.

Prefer to try it first? Transcribe audio without writing any code in the [AssemblyAI Playground](https://www.assemblyai.com/playground).

  **Streaming is billed per session**

  Streaming Speech-to-Text is billed on the total duration that your WebSocket connection stays open, not on the amount of audio you send. Always send a termination message when you're done with a stream — sessions that aren't closed auto-close after 3 hours and are billed for the full duration. See [Billing and pricing](/billing-and-pricing) for details.

## Before you begin

You'll need:

- **An API key** — grab one from [your dashboard](https://www.assemblyai.com/dashboard/home). Every example below reads it from an environment variable, so set it once:

  ```bash
  export ASSEMBLYAI_API_KEY=<your-key>
  ```

- **Python 3.8+ or Node.js 18+**, depending on which SDK you use.
- **Nothing else.** The examples pull a public AAC radio stream over HTTP, so no microphone or audio hardware is needed.

**Building with an AI coding agent?** Wire it up to AssemblyAI's live docs (MCP server) and the AssemblyAI skill so it writes correct, up-to-date code instead of relying on stale training data:

```bash
claude mcp add --transport http --scope user assemblyai-docs https://assemblyai.com/docs/mcp
npx skills add AssemblyAI/assemblyai-skill --global
```

Then describe what you want to build. To get the same result as the steps below, paste:

```text
Use the AssemblyAI Python SDK to transcribe a live AAC radio stream in real time and print each turn.
```

## Transcribe streaming audio

Prefer to write it yourself? Follow these steps to stream a live radio URL. The AssemblyAI SDK manages the WebSocket connection and session termination for you.

### Step 1: Install the SDK

```bash
pip install assemblyai requests
```

```bash
npm install assemblyai
```

### Step 2: Stream your first session

Save this as `transcribe.py` (Python) or `transcribe.js` (JavaScript). It streams about 25 seconds of a live radio URL, prints each turn, then closes the session:

```python expandable
import os
import time

import requests
from assemblyai.streaming.v3 import (
    BeginEvent,
    Encoding,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)

# A live AAC (ADTS) internet radio stream, so no microphone is needed.
STREAM_URL = "https://14123.live.streamtheworld.com/WBBRAMAAC.aac"
RUN_SECONDS = 25

def on_begin(client: RealTimeTranscriber, event: BeginEvent):
    print(f"Session started: {event.id}")
    print("Connected. Streaming live radio for ~25 seconds.")

def on_turn(client: RealTimeTranscriber, event: TurnEvent):
    if event.transcript:
        print(event.transcript)

def on_terminated(client: RealTimeTranscriber, event: TerminationEvent):
    print(f"Session terminated: {event.audio_duration_seconds}s of audio processed")

def on_error(client: RealTimeTranscriber, error: RealTimeError):
    print(f"Error: {error}")

def main():
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(terminate_timeout=30.0),
        api_key=os.environ["ASSEMBLYAI_API_KEY"],
    )

    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    # AAC is self-describing (ADTS headers carry the sample rate)
    client.connect(
        RealTimeParameters(speech_model="universal-3-5-pro", encoding=Encoding.aac)
    )

    # Pull the live radio stream and forward each chunk to the transcriber.
    response = requests.get(STREAM_URL, stream=True)
    deadline = time.time() + RUN_SECONDS
    try:
        for chunk in response.iter_content(chunk_size=4096):
            client.stream(chunk)
            if time.time() > deadline:
                break
    finally:
        response.close()
        # Terminate finalizes the open turn.
        # Keep the connection open long enough to receive the last final.
        client.disconnect(terminate=True)

if __name__ == "__main__":
    main()
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

// A live AAC (ADTS) internet radio stream, so no microphone is needed.
const STREAM_URL = "https://14123.live.streamtheworld.com/WBBRAMAAC.aac";
const RUN_MS = 25_000;

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const transcriber = client.streaming.transcriber({
  speechModel: "universal-3-5-pro",
  encoding: "aac",
});

transcriber.on("open", ({ id }) => console.log(`Session opened with ID: ${id}`));
transcriber.on("error", (error) => console.error("Error:", error));
transcriber.on("close", (code, reason) => console.log("Session closed:", code, reason));
transcriber.on("turn", (turn) => {
  if (turn.transcript) {
    console.log("Turn:", turn.transcript);
  }
});

const run = async () => {
  await transcriber.connect();
  console.log("Connected. Streaming live radio for ~25 seconds.");

  // Pull the live radio stream and forward each chunk to the transcriber.
  const response = await fetch(STREAM_URL);
  const reader = response.body.getReader();
  const deadline = Date.now() + RUN_MS;
  while (Date.now() < deadline) {
    const { value, done } = await reader.read();
    if (done) break;
    transcriber.sendAudio(Buffer.from(value));
  }
  await reader.cancel();

  // close() sends terminate, which finalizes the open turn.
  // Keep the connection open long enough to receive the last final.
  await transcriber.close();
};

run();
```

Then run it with `python transcribe.py` or `node transcribe.js`. Each turn prints as the audio plays. Terminating finalizes the open turn, so its final transcript prints just before the session ends after about 25 seconds. Because internet radio bursts audio faster than real time, the server may still be catching up when you stop sending, so the last final can arrive a few seconds later than a real-time source like a microphone would show it:

```text
Session started: 7f3a9c2e-...
markets are higher across the board this morning as investors weigh the latest earnings.
Session terminated: 25.0s of audio processed
```

That's a full real-time transcriber. Prefer raw WebSockets? See [Using the WebSocket API directly](#using-the-websocket-api-directly) below.

## What you get back

The transcriber emits JSON messages (the SDK surfaces them as `open` / `turn` / `close` events). The one you handle most is `Turn`, sent repeatedly as someone speaks — `end_of_turn: true` marks a finalized turn, and `transcript` is the text so far:

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": true,
  "turn_is_formatted": true,
  "end_of_turn_confidence": 1.0,
  "transcript": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts...",
  "words": [
    { "text": "Smoke", "start": 0, "end": 399, "confidence": 0.99, "word_is_final": true }
  ]
}
```

Live radio is continuous speech, so finalized turns arrive at natural pauses. Expect several partial updates before each `end_of_turn: true`.

You also receive a `Begin` message when the session opens (`{ "type": "Begin", "id": "...", "expires_at": ... }`) and a `Termination` message when it closes (`{ "type": "Termination", "audio_duration_seconds": 10, "session_duration_seconds": 12 }`). Word timings are in milliseconds. See the [message sequence breakdown](/streaming/getting-started/transcribe-streaming-audio) for the full event flow.

## Using the WebSocket API directly

Not using an SDK? Connect to the streaming WebSocket at `wss://streaming.assemblyai.com/v3/ws` directly. Authenticate with your key in the `Authorization` header (no `Bearer` prefix), and manage the connection, the audio source, the `Begin` / `Turn` / `Termination` messages, and session termination yourself. The SDK above does all of this for you. See the [message sequence breakdown](/streaming/getting-started/transcribe-streaming-audio) for the event flow and [endpoints and data zones](/streaming/endpoints-and-data-zones) for regional endpoints.

Both examples read your key from the same `ASSEMBLYAI_API_KEY` environment variable you set in [Before you begin](#before-you-begin).

  **Streaming from a browser?**

  Don't ship your API key to client-side code. Authenticate from the browser with a
  short-lived [temporary token](/streaming/authenticate-with-a-temporary-token) instead.

```bash
pip install requests websocket-client
```

```python expandable
import json
import os
import threading
import time
from urllib.parse import urlencode

import requests
import websocket

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
# A live AAC (ADTS) internet radio stream, so no microphone is needed.
STREAM_URL = "https://14123.live.streamtheworld.com/WBBRAMAAC.aac"
RUN_SECONDS = 25
# AAC is self-describing (ADTS headers carry the sample rate)
CONNECTION_PARAMS = {"speech_model": "universal-3-5-pro", "encoding": "aac"}
API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"

stop = threading.Event()

def on_open(ws):
    print("Connected. Streaming live radio for ~25 seconds.")

    def stream_audio():
        # Pull the live radio stream and forward each chunk as a binary frame.
        response = requests.get(STREAM_URL, stream=True)
        deadline = time.time() + RUN_SECONDS
        try:
            for chunk in response.iter_content(chunk_size=4096):
                if stop.is_set() or time.time() > deadline:
                    break
                ws.send(chunk, websocket.ABNF.OPCODE_BINARY)
        finally:
            response.close()
            if ws.sock and ws.sock.connected:
                # Terminate finalizes the open turn.
                # Keep the connection open long enough to receive the last final.
                ws.send(json.dumps({"type": "Terminate"}))

    threading.Thread(target=stream_audio, daemon=True).start()

def on_message(ws, message):
    data = json.loads(message)
    if data.get("type") == "Turn":
        print(data.get("transcript", ""), end="\n" if data.get("end_of_turn") else "\r")

def on_error(ws, error):
    # On a normal shutdown, websocket-client hands the server's close frame to
    # on_error; ignore it and let on_close report the disconnect. Real failures
    # arrive as exceptions, not close frames.
    if isinstance(error, websocket.ABNF) and error.opcode == websocket.ABNF.OPCODE_CLOSE:
        return
    print(f"\nError: {error}")
    stop.set()

def on_close(ws, status, msg):
    stop.set()
    print("\nDisconnected.")

def main():
    ws = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    ws_thread = threading.Thread(target=ws.run_forever, daemon=True)
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            ws_thread.join(0.1)
    except KeyboardInterrupt:
        stop.set()
        if ws.sock and ws.sock.connected:
            ws.send(json.dumps({"type": "Terminate"}))  # close the session
        ws.close()

if __name__ == "__main__":
    main()
```

```bash
npm install ws
```

```javascript expandable
const WebSocket = require("ws");
const querystring = require("querystring");

const API_KEY = process.env.ASSEMBLYAI_API_KEY;
// A live AAC (ADTS) internet radio stream, so no microphone is needed.
const STREAM_URL = "https://14123.live.streamtheworld.com/WBBRAMAAC.aac";
const RUN_MS = 25_000;
// AAC is self-describing (ADTS headers carry the sample rate)
const params = { speech_model: "universal-3-5-pro", encoding: "aac" };
const endpoint = `wss://streaming.assemblyai.com/v3/ws?${querystring.stringify(params)}`;

const ws = new WebSocket(endpoint, { headers: { Authorization: API_KEY } });

ws.on("open", async () => {
  console.log("Connected. Streaming live radio for ~25 seconds.");

  // Pull the live radio stream and forward each chunk as a binary frame.
  const response = await fetch(STREAM_URL);
  const reader = response.body.getReader();
  const deadline = Date.now() + RUN_MS;
  while (Date.now() < deadline && ws.readyState === WebSocket.OPEN) {
    const { value, done } = await reader.read();
    if (done) break;
    ws.send(value);
  }
  await reader.cancel();

  if (ws.readyState === WebSocket.OPEN) {
    // Terminate finalizes the open turn.
    // Keep the connection open long enough to receive the last final.
    ws.send(JSON.stringify({ type: "Terminate" }));
  }
});

ws.on("message", (message) => {
  const data = JSON.parse(message);
  if (data.type === "Turn") {
    process.stdout.write(data.end_of_turn ? `${data.transcript}\n` : `\r${data.transcript}`);
  }
});

ws.on("error", (error) => console.error("\nError:", error));
ws.on("close", () => {
  console.log("\nDisconnected.");
  process.exit();
});
```

## Limits

- **Session length:** a streaming session auto-closes after 3 hours.
- **Audio:** mono 16-bit PCM by default; set `sample_rate` to match your source. These examples use AAC (`encoding=aac`, ADTS framing). Opus is also accepted (`encoding=ogg_opus` for Ogg streams, `encoding=opus` for raw Opus packets). See [Sending audio](/streaming/message-sequence#sending-audio).
- **Rate limit:** new-session rate limits scale automatically with usage (default 5 for free accounts). Check yours on the [rate limits page](https://www.assemblyai.com/dashboard/home).

## Next steps

To learn more about Streaming Speech-to-Text, see the following resources:

- [Streaming Speech-to-Text overview](/streaming)
- [Message sequence breakdown](/streaming/getting-started/transcribe-streaming-audio) — understand the `Begin`, `Turn`, and `Termination` events
- [WebSocket API reference](/streaming/api-spec/streaming-websocket)

## Need some help?

If you get stuck, or have any other questions, we'd love to help you out. Contact our support team at support@assemblyai.com or create a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="model-selection-docs-streaming-select-the-speech-model-mdx"></a>

# Model selection

URL: https://www.assemblyai.com/docs/streaming/select-the-speech-model
Source: docs/streaming/select-the-speech-model.mdx
Navigation: Real-time STT > Getting started
Description: Model selection documentation.

The `speech_model` connection parameter lets you specify which model to use for streaming transcription. It's optional; if omitted, the session defaults to `universal-3-5-pro`.

## Available models

| Name                                      | Parameter                                            | Description                                                                                                                                             | Best for                                                                                                                            |
| ----------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Universal-3.5 Pro Streaming** <span class="recommended-pill">Recommended</span> | `"universal-3-5-pro"`                | The next generation of our flagship streaming model. More languages, improved prompting, and enhanced conversational context features.                                           | Real-time voice agents needing the latest accuracy, broad language support, and conversational context.        |
| **Universal-Streaming English**            | `"universal-streaming-english"`      | An English transcription model offering a good balance of speed and cost-effectiveness.                                                                 | Cost-effective English real-time transcription, English-only real-time apps                                                          |
| **Universal-Streaming Multilingual**      | `"universal-streaming-multilingual"` | A multilingual transcription model offering a good balance of speed and cost-effectiveness.                            | Cost-effective multilingual streaming across EN/ES/DE/FR/PT/IT                                                                      |

## Choosing a model

| Feature                         | Universal-3.5 Pro Streaming <span class="recommended-pill">Recommended</span>                                                                              | Universal-Streaming English                                              | Universal-Streaming Multilingual                                         |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------ |
| **Latency**                     | Fastest                                                                                                                                            | Fast                                                                     | Fast                                                                     |
| **Supported languages**         | EN, ES, DE, FR, PT, IT, TR, NL, SV, NO, DA, FI, HI, VI, AR, HE, JA, ZH                                                                         | EN                                                                       | EN, ES, DE, FR, PT, IT                                                   |
| **Partial transcripts**         | Yes                                                                                                                                                | Yes                                                                      | Yes                                                                      |
| **Multilingual**                | Native Code Switching                                                                                                                              | No                                                                       | Per Turn                                                                 |
| **Entity accuracy**             | Best                                                                                                                                               | Okay                                                                     | Okay                                                                     |
| **Disfluencies & filler words** | Yes                                                                                                                                                | No                                                                       | No                                                                       |
| **Language detection**          | Yes                                                                                                                                                | No                                                                       | Yes                                                                      |
| **Non-speech tags**             | No                                                                                                                                                 | No                                                                       | No                                                                       |
| **Customization**               | [Keyterms prompting](/streaming/prompting-and-keyterms) (known context) + [Native prompting](/streaming/prompting-and-keyterms) (unknown context) | [Keyterms prompting](/streaming/prompting-and-keyterms) (known context) | [Keyterms prompting](/streaming/prompting-and-keyterms) (known context) |

For detailed setup and configuration of Universal-3.5 Pro Streaming, see the [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio) page. For prompting guidance, see the [Prompting guide](/streaming/prompting-and-keyterms).

## Set the model

Set the `speech_model` connection parameter when you open the WebSocket.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",  # universal-streaming-english | universal-streaming-multilingual
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",  # universal-streaming-english | universal-streaming-multilingual
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro", // universal-streaming-english | universal-streaming-multilingual
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro", // universal-streaming-english | universal-streaming-multilingual
});
```

---

<a id="optimizing-accuracy-and-latency-docs-streaming-getting-started-optimizing-accuracy-and-latency-mdx"></a>

# Optimizing Accuracy and Latency

URL: https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency
Source: docs/streaming/getting-started/optimizing-accuracy-and-latency.mdx
Navigation: Real-time STT > Getting started
Description: Tune streaming models to balance transcription accuracy and latency.

## Mode

Universal-3.5 Pro Streaming workloads sit on a spectrum between two competing goals: returning transcripts as fast as possible, and returning the most accurate transcripts possible. To make this tradeoff explicit, Universal-3.5 Pro supports a **mode** connection parameter you can set when opening a streaming session.

| Mode | Value | When to use |
| --- | --- | --- |
| Min latency | `min_latency` | Lowest possible time-to-text. Best when responsiveness matters more than catching every word. |
| Balanced (default) | `balanced` | A middle ground between latency and accuracy. Best for voice agents and other interactive applications. |
| Max accuracy | `max_accuracy` | Highest transcription accuracy. Best for note-taking, scribes, and post-call analysis where a small added delay is acceptable. |

Set the `mode` connection parameter when you open the WebSocket.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "mode": "balanced",  # min_latency | max_accuracy
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        mode="balanced",  # min_latency | max_accuracy
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  mode: "balanced", // min_latency | max_accuracy
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  mode: "balanced", // min_latency | max_accuracy
});
```

## Language selection

By default, Universal-3.5 Pro Streaming is multilingual and code-switches natively across all supported languages with no configuration. When you know which languages a session will use, pass the `language_codes` connection parameter to steer the model toward them and improve language accuracy.

- For a **known subset**, pass the codes you expect (for example, `["en", "es"]`). The model still code-switches, but heavily biased to the languages you list.
- For a **monolingual session**, pass a single-element list (for example, `["es"]`).
- For **full multilingual**, omit `language_codes` to keep native code switching.

See [Multilingual transcription](/streaming/multilingual-transcription) for the full feature guide, including supported languages, language detection, and updating `language_codes` mid-stream.

## Turn detection

Universal-3.5 Pro Streaming detects the end of a turn using the transcript rather than silence alone. The `mode` preset sets the defaults, and every turn detection parameter can be overridden on the connection or updated mid-stream to tune endpointing for your use case.

- Raise `min_turn_silence` when brief pauses end turns too early, for example while a caller dictates a phone number. Raise `max_turn_silence` when you expect longer pauses within a turn.
- Lower `vad_threshold` when quiet speech is missed. When background noise causes false interruptions, raise `vad_threshold`, increase `interruption_delay`, or enable [Voice Focus](/streaming/voice-focus).

See [Turn detection](/streaming/turn-detection) for the full feature guide, including the turn lifecycle, entity capture, and bringing your own turn detection.

Universal Streaming uses **confidence-based turn detection**. The model predicts when speech naturally ends; if confidence exceeds `end_of_turn_confidence_threshold` and `min_turn_silence` has passed, the turn ends. Acoustic (silence-based) detection kicks in as a fallback after `max_turn_silence`.

| Parameter                          | Default   | Description                                                                                                |
| ---------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------- |
| `end_of_turn_confidence_threshold` | `0.4`     | Confidence threshold for semantic end-of-turn. Higher = more confident before ending; lower = ends faster. |
| `min_turn_silence`                 | `400` ms  | Silence required before a semantic end-of-turn fires.                                                      |
| `max_turn_silence`                 | `1280` ms | Maximum silence before forcing a turn to end via acoustic detection.                                       |
| `vad_threshold`                    | `0.4`     | Confidence threshold (0 to 1) for classifying audio frames as speech. Increase for noisy environments to reduce false speech detection. |

**Quick-start configurations**

Aggressive, for short, rapid back-and-forth (IVR replacements, order confirmations).

```javascript
const streamingConfig = {
  end_of_turn_confidence_threshold: 0.4,
  min_turn_silence: 160,
  max_turn_silence: 400,
};
```

Balanced, for most conversational voice agents (customer support).

```javascript
const streamingConfig = {
  end_of_turn_confidence_threshold: 0.4,
  min_turn_silence: 400,
  max_turn_silence: 1280,
};
```

Conservative, for reflective or complex speech (healthcare, sales, legal).

```javascript
const streamingConfig = {
  end_of_turn_confidence_threshold: 0.7,
  min_turn_silence: 800,
  max_turn_silence: 3600,
};
```

**Disabling turn detection**

If you're using your own VAD or turn detection model, send a `ForceEndpoint` event to force a turn boundary:

```python
ws.send(json.dumps({"type": "ForceEndpoint"}))
```

Or set `end_of_turn_confidence_threshold` to `1` (acoustic-only fallback) or `0` (silence-only). Setting it to `0` is **not recommended** unless you have a custom turn detection model running on top, because it forces a turn at every `min_turn_silence`-length pause and fragments mid-sentence thinking pauses.

---

<a id="turn-detection-docs-streaming-turn-detection-mdx"></a>

# Turn Detection

URL: https://www.assemblyai.com/docs/streaming/turn-detection
Source: docs/streaming/turn-detection.mdx
Navigation: Real-time STT > Features
Description: Turn Detection documentation.

Detect when a speaker has finished their turn.

Streaming transcription is delivered in **turns**. A turn is one continuous unit of speech belonging to a single speaker. Deciding when a turn has ended, called **endpointing**, determines how an interactive voice application feels. End turns too early and you cut speakers off mid-thought or split a complex entity like a phone number into multiple parts. End turns too late and your voice agent feels unresponsive.

## How turn detection works

Universal-3.5 Pro Streaming does not endpoint on silence alone. When a speaker pauses, the model evaluates what has been said so far to judge whether the speaker is finished or just pausing mid-thought.

Two silence thresholds control this behavior.

| Parameter | What it does | Default |
| --- | --- | --- |
| `min_turn_silence` | Silence in milliseconds before the model runs an end-of-turn check. If the turn reads as complete, the turn ends. Otherwise a partial is emitted and the turn stays open. | Set by `mode` (`min_latency: 128`, `balanced: 128`, `max_accuracy: 512`) |
| `max_turn_silence` | Maximum silence in milliseconds before the turn is forced to end, regardless of content. This is the fallback for turns that never read as complete. | Set by `mode` (`min_latency: 640`, `balanced: 1280`, `max_accuracy: 2560`) |

Lower values end turns faster. Higher values tolerate longer pauses without fragmenting the turn. Both parameters can be set on the connection and [updated mid-stream](#capturing-entities-like-phone-numbers), and their defaults come from the [`mode` preset](/streaming/getting-started/optimizing-accuracy-and-latency) you select.

**Speaker labels change the defaults**

When `speaker_labels` is enabled, the server applies diarization-tuned defaults instead (`min_turn_silence: 640`, `max_turn_silence: 768`, continuous partials disabled) so turns break cleanly at speaker changes.

## The lifecycle of a turn

Each turn produces a predictable message sequence of one `SpeechStarted` message, one or more partial `Turn` messages, and a final `Turn` message.

Every `Turn` message re-transcribes the full turn audio, so each message supersedes the previous one. Always replace what you have rendered rather than appending. This also means transcripts get more accurate as the turn progresses, since each pass gives the model more audio and more context to work with. For maximum accuracy, use the final transcript marked `end_of_turn: true`. Use partials where responsiveness matters more than accuracy.

### SpeechStarted

The server sends `SpeechStarted` once per turn, immediately before the turn's first `Turn` message. It carries the turn's start `timestamp` in milliseconds from the start of the audio stream, and the `confidence` of a new turn starting.

```json
{
  "type": "SpeechStarted",
  "timestamp": 1216,
  "confidence": 0.98
}
```

`SpeechStarted` is only emitted once the model produces an actual transcript, so background noise alone does not trigger it. That makes it a reliable barge-in signal for voice agents. When it arrives, stop your agent's TTS playback and start listening.

### Partials

While the speaker talks, the session emits `Turn` messages with `end_of_turn: false`.

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": false,
  "turn_is_formatted": false,
  "transcript": "my number is five five five",
  "end_of_turn_confidence": 0.0
}
```

Partials are emitted at three points.

- **Early partials** are emitted shortly after speech begins, timed by the `interruption_delay` parameter.
- **Pause partials** are emitted when a pause triggers an end-of-turn check but the turn reads as incomplete.
- **Continuous partials** are emitted roughly every 3 seconds during uninterrupted speech (`continuous_partials`, enabled by default), so long utterances keep updating even without pauses.

Partials are suited to live captions, barge-in handling, and speculative LLM prefetching, where you want text immediately and can tolerate it being revised.

### Finals

When the turn ends, the session emits a `Turn` message with `end_of_turn: true`. The final is fully formatted, with punctuation, casing, and numbers and entities rendered in written form.

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": true,
  "turn_is_formatted": true,
  "transcript": "My number is 555-0134.",
  "end_of_turn_confidence": 1.0
}
```

The final is the most accurate transcript of the turn. Send it to your LLM, store it, and act on it whenever accuracy matters.

## Tuning interruption sensitivity

`SpeechStarted` and early partials are what trigger interruptions in a voice agent, and two parameters control how eagerly they fire.

| Parameter | What it does | Default |
| --- | --- | --- |
| `vad_threshold` | Confidence between `0` and `1` required to classify an audio frame as speech. | `0.2` |
| `interruption_delay` | How long in milliseconds after speech begins the first partial is emitted, from `0` to `1000`. | Set by `mode` (`min_latency: 0`, `balanced: 500`, `max_accuracy: 500`) |

If quiet or distant speech is being missed, lower `vad_threshold` so softer audio counts as speech. If background noise or the agent's own TTS audio is triggering false interruptions, raise `vad_threshold` so the session requires stronger evidence of speech, increase `interruption_delay` so it waits longer before emitting the first partial, or enable [Voice Focus](/streaming/voice-focus) to suppress background audio before it reaches the model.

Like the silence thresholds, both can be set on the connection and updated mid-stream.

## Capturing entities like phone numbers

Speakers naturally pause between the groups of a phone number, credit card, or address, and each of those pauses can read as a completed thought, splitting one entity across several turns. When your application knows an entity is coming (for example, your voice agent just asked for a callback number), raise `min_turn_silence` mid-stream so those pauses don't end the turn. Afterward, set your `mode` again to restore the preset's defaults. Changes take effect immediately, without reconnecting.

```python
# Before capturing the entity
websocket.send('{"type": "UpdateConfiguration", "min_turn_silence": 1000}')

# After the response, re-apply your mode to restore its defaults
websocket.send('{"type": "UpdateConfiguration", "mode": "balanced"}')
```

```python
# Before capturing the entity
client.update_configuration(min_turn_silence=1000)

# After the response, re-apply your mode to restore its defaults
client.update_configuration(mode="balanced")
```

```javascript
// Before capturing the entity
websocket.send('{"type": "UpdateConfiguration", "min_turn_silence": 1000}');

// After the response, re-apply your mode to restore its defaults
websocket.send('{"type": "UpdateConfiguration", "mode": "balanced"}');
```

```javascript
// Before capturing the entity
transcriber.updateConfiguration({ minTurnSilence: 1000 });

// After the response, re-apply your mode to restore its defaults
transcriber.updateConfiguration({ mode: "balanced" });
```

If turns are still force-ended during very long pauses, raise `max_turn_silence` the same way. See [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream) for the full list of mid-stream parameters.

## Bring your own turn detection

If you run your own VAD or turn detection model, send a `ForceEndpoint` message to end the current turn immediately. The server responds with the turn's final `Turn` message right away, without waiting for silence.

```python
websocket.send('{"type": "ForceEndpoint"}')
```

```python
client.force_endpoint()
```

```javascript
websocket.send('{"type": "ForceEndpoint"}');
```

```javascript
transcriber.forceEndpoint();
```

## Going further

- [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency) covers the `mode` presets that set turn detection defaults, plus advanced tuning parameters, including turn detection on the Universal Streaming models.
- [Message sequence](/streaming/message-sequence) documents the complete WebSocket message contract, including turn ordering guarantees.
- [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream) lists everything you can change during a live session.

---

<a id="prompting-and-keyterms-docs-streaming-prompting-and-keyterms-mdx"></a>

# Prompting and Keyterms

URL: https://www.assemblyai.com/docs/streaming/prompting-and-keyterms
Source: docs/streaming/prompting-and-keyterms.mdx
Navigation: Real-time STT > Features
Description: Prompting and Keyterms documentation.

Use contextual prompting and keyterms to improve streaming transcription accuracy.

Streaming transcription supports two complementary ways to give the model information about your audio:

- **Contextual prompting** (`prompt`) — a natural-language description of what the audio is about: the domain, the scenario, or the full details of the conversation.
- **Keyterms prompting** (`keyterms_prompt`) — an explicit list of terms you want the model to recognize accurately.

Both improve recognition accuracy for your use case. Neither changes the output format.

## How prompting works

Universal-3.5 Pro is trained to use context about the audio — its domain, topic, or scenario — to better recognize the vocabulary that context makes likely. A call described as a cardiology consultation primes the model for medical terminology; a call described as an order-status check primes it for order IDs and product names.

The transcription instruction itself is built in and managed by AssemblyAI. You don't need to tell the model *how* to transcribe — verbatim behavior, punctuation, and formatting are already optimized for streaming and turn detection. The `prompt` parameter carries **context about your audio**, not instructions: formatting or behavioral commands in the prompt (such as punctuation rules) are not supported.

The model is trained to stay grounded in the audio: context that turns out to be irrelevant or only partially applicable does not cause the model to insert words that weren't spoken. This means you can safely send the same context with every session of a longer interaction — for example, the same call description for every segment of a conversation — even though only some of it applies at any given moment.

**Start with no prompt**

We strongly recommend testing with no `prompt` first. Universal-3.5 Pro is optimized out of the box, and context helps most when your audio contains domain-specific vocabulary the model is getting wrong. Add context when you see those errors, starting with the broadest level that describes your use case.

## Contextual prompting

Contextual prompts work at three levels of specificity. Use the least specific level that covers your use case, and add detail when your audio contains uncommon names or terms the model can't otherwise know.

| Level | Length | What it contains | Example |
| --- | --- | --- | --- |
| **Domain** | 2–5 words | The domain only | `Medical consultation call.` |
| **Scenario** | 5–15 words | What the conversation is about | `Cardiology consultation about chest pain symptoms.` |
| **Detailed** | 20–50 words | Full description, including names, products, or identifiers | `Cardiology consultation between Dr. Smith and an elderly patient regarding recurring chest pain, ECG results, and medication adjustment for hypertension.` |

**Domain context** tells the model what field the audio belongs to — medical, legal, technical support, food ordering. This is the safest starting point and is often enough to fix vocabulary errors.

**Scenario context** describes the specific situation. This is the right level for most applications that know what kind of call is taking place — appointment booking, billing inquiry, delivery complaint.

**Detailed context** describes everything your application already knows about the conversation: participant names, account or order identifiers, products, locations. This level is the most powerful when the audio contains proper nouns and identifiers — for example, a voice agent platform that already knows who is calling and why can pass that information so the model spells names and IDs correctly.

Guidelines for writing contextual prompts:

- Write plain, complete sentences that describe the audio — you are describing a recording, not commanding the model.
- Keep it to one short block of text. Don't pack lists of keywords into the contextual prompt — that's what [`keyterms_prompt`](#keyterms-prompting) is for.
- Specificity follows knowledge: only include details you actually know about the call. Wrong details won't corrupt the transcript, but they don't help either.

### Accuracy impact

We benchmarked contextual prompting on 20,000 real voice-agent calls at each of the three levels. Accuracy improves monotonically with context specificity, and the gains are largest for the entities each level describes. Word error rate (WER) and entity error rate (EER) below are relative reductions versus no prompt.

| Improvement vs. no prompt | Domain | Scenario | Detailed |
| --- | --- | --- | --- |
| Word error rate (WER) | −5% | −10% | −21% |
| Hallucinated words | −9% | −12% | −19% |
| Entity error rate — overall | −2% | −7% | −29% |
| Entity error rate — names | −5% | −16% | −49% |
| Entity error rate — places | −9% | −21% | −44% |
| Entity error rate — medical terms | −2% | −24% | −43% |

Two takeaways for choosing a level:

- **Scenario context is the practical default.** It needs only the kind of information any application already has — the type of call — and already cuts overall WER ~10% and entity errors on names and places by 16–21%.
- **Detailed context is the upper bound.** When your application can supply specifics it already knows (caller name, account or order IDs, products), entity accuracy improves dramatically — names nearly halve. This is the most powerful option for voice agents with access to customer context.

Across all levels, turn detection is unaffected. Hallucinated and fabricated words go *down* with more context — describing the audio makes the model less likely to invent content, not more.

### Specifying the language

**Prefer the `language_codes` parameter**

Language selection is best supported through the `language_codes` connection parameter — pass the list of languages you expect, or a single-element list for a monolingual session — see [Language selection](/streaming/getting-started/optimizing-accuracy-and-latency#language-selection). You can also specify the language in natural language as part of your contextual prompt.

State the language of the audio as part of your context. For example, knowing the language disambiguates the audio: the same sound could be transcribed "si" in a Spanish call but "C" in an English one.

```
Spanish customer support call about a billing inquiry.
```

For multilingual audio, name all expected languages:

```
Multilingual conversation in English, Spanish, and German.
```

## Keyterms prompting

Use the `keyterms_prompt` parameter to boost recognition of specific names, brands, or domain terms. Pass an array of terms you want the model to prioritize:

```
keyterms_prompt=["Keanu Reeves", "AssemblyAI", "Universal-2"]
```

Keyterms prompting is the right tool when you have an explicit vocabulary list — contact names, product catalogs, medical terms — rather than a description of the conversation. Like contextual prompting, it is trained to be robust to terms that don't end up being spoken, so you can pass your full list to every session.

**Start with no keyterms**

**We strongly recommend starting with no `keyterms_prompt` and then adding terms as needed** based on important words for your use case that you are consistently seeing the model struggle with.

Including a large number of terms or common terms that are well represented in the training data could lead to overcorrections and hallucinations.

**Limits:**

- You can include a maximum of **100 keyterms** per session.
- Each individual keyterm string must be **50 characters or less**.
- Keyterms longer than 50 characters are ignored; requests with more than 100 keyterms return an error.

**Best practices:**

- **Specify unique terminology.** Include proper names, company names, technical terms, or vocabulary specific to your domain that might not be commonly recognized.
- **Exact spelling and capitalization.** Provide keyterms with the precise spelling and capitalization you expect to see in the output transcript.
- **Avoid common words.** Do not include single, common English words (e.g., "information") as keyterms. The system is generally proficient with such words, and adding them as keyterms can be redundant.

## Quickstart

Set `prompt` and `keyterms_prompt` as connection parameters when you open the WebSocket. The example below uses both together, with `prompt` describing the conversation and `keyterms_prompt` enumerating specific terms.

```python
import json

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "prompt": "Cardiology consultation about chest pain symptoms.",
    "keyterms_prompt": json.dumps(["Dr. Smith", "AssemblyAI", "ECG"]),
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        prompt="Cardiology consultation about chest pain symptoms.",
        keyterms_prompt=["Dr. Smith", "AssemblyAI", "ECG"],
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  prompt: "Cardiology consultation about chest pain symptoms.",
  keyterms_prompt: JSON.stringify(["Dr. Smith", "AssemblyAI", "ECG"]),
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  prompt: "Cardiology consultation about chest pain symptoms.",
  keytermsPrompt: ["Dr. Smith", "AssemblyAI", "ECG"],
});
```

## Updating mid-stream

Both `prompt` and `keyterms_prompt` can be updated during an active streaming session without reconnecting. This is useful when your application learns more about the conversation as it progresses, or when a voice agent moves between conversation stages.

Send an `UpdateConfiguration` message with the new values:

```python
# Update the contextual prompt as the conversation evolves
websocket.send('{"type": "UpdateConfiguration", "prompt": "Now collecting payment details."}')

# Replace or establish a new set of keyterms
websocket.send('{"type": "UpdateConfiguration", "keyterms_prompt": ["Universal-3"]}')

# Remove keyterms and reset context biasing
websocket.send('{"type": "UpdateConfiguration", "keyterms_prompt": []}')
```

```python
# Update the contextual prompt as the conversation evolves
client.update_configuration(prompt="Now collecting payment details.")

# Replace or establish a new set of keyterms
client.update_configuration(keyterms_prompt=["Universal-3"])

# Remove keyterms and reset context biasing
client.update_configuration(keyterms_prompt=[])
```

```javascript
// Update the contextual prompt as the conversation evolves
websocket.send(
  '{"type": "UpdateConfiguration", "prompt": "Now collecting payment details."}'
);

// Replace or establish a new set of keyterms
websocket.send(
  '{"type": "UpdateConfiguration", "keyterms_prompt": ["Universal-3"]}'
);

// Remove keyterms and reset context biasing
websocket.send('{"type": "UpdateConfiguration", "keyterms_prompt": []}');
```

```javascript
// Update the contextual prompt as the conversation evolves
transcriber.updateConfiguration({ prompt: "Now collecting payment details." });

// Replace or establish a new set of keyterms
transcriber.updateConfiguration({ keytermsPrompt: ["Universal-3"] });

// Remove keyterms and reset context biasing
transcriber.updateConfiguration({ keytermsPrompt: [] });
```

Providing a new keyterms array completely replaces the existing set; sending an empty array `[]` removes all keyterms and resets context biasing to the default state. New values take effect immediately for subsequent audio processing.

See [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream) for the full list of parameters you can update mid-stream.

## What prompting is not

- **Prompting does not control output formatting.** Instructions in the prompt (punctuation rules, verbatim directives, formatting commands) are not supported — transcription behavior is managed internally and already optimized for streaming and turn detection.

## Looking for async prompting?

Prompting behavior differs between streaming and async (pre-recorded) use cases. This guide covers prompting for **streaming (real-time audio)**. If you're working with pre-recorded audio, see the [Prompting Guide (Async)](/pre-recorded-audio/universal-3-5-pro/prompting) and [Keyterms prompting (Async)](/pre-recorded-audio/universal-3-5-pro/prompting#keyterms-prompting).

---

<a id="conversation-context-docs-streaming-universal-3-5-pro-context-carryover-mdx"></a>

# Conversation Context

URL: https://www.assemblyai.com/docs/streaming/universal-3-5-pro/context-carryover
Source: docs/streaming/universal-3-5-pro/context-carryover.mdx
Navigation: Real-time STT > Features
Description: Conversation Context documentation.

Improve voice agent outcomes by giving Universal-3.5 Pro the full conversational context. Pass your voice agent's spoken replies into the session via the `agent_context` parameter and the model knows both sides of the dialog when transcribing the next user turn.

With the agent's side of the conversation in context, Universal-3.5 Pro can anticipate the kind of answer to expect, sharpen entity recognition, and disambiguate words that sound similar. For example, after your agent asks `"What's your email address?"` the model might transcribe the reply as `"user at assemblyai dot com"`. With that question passed in via `agent_context`, the model knows an email is coming and produces `"user@assemblyai.com"`.

The user side comes along for free: Universal-3.5 Pro Streaming automatically carries prior STT-finalized turns forward as context, so you don't need to configure anything for the user half of the conversation. `agent_context` is what fills in the agent half.

  **Set `agent_context` at connection time and after each agent reply**

  Pass `agent_context` as a connection-time query parameter to seed the model with your agent's opening greeting, then send `UpdateConfiguration` mid-stream after each subsequent agent reply. See [Passing your agent's reply as context](#passing-your-agents-reply-as-context) for the full pattern.

## How it works

During a streaming session, Universal-3.5 Pro Streaming keeps a short memory of the conversation. Two sources feed that memory: `agent_context` values you push in (the agent half) and prior finalized user turns (the user half, carried forward automatically). The model uses both when transcribing the next user turn.

This means:

- Context is **per-session**. Closing the WebSocket clears it, and a new session starts fresh.
- Only `agent_context` values and **finalized** user turns (`end_of_turn: true`) are carried forward, not partials.

### Defaults

| Behavior | Default |
| --- | --- |
| Auto-carry of prior user turns | Enabled |
| Number of prior entries carried | 5 |
| Maximum context size | 1750 characters |

Older entries are dropped first as new ones come in, so the most recent conversation is always preserved.

  The number of prior entries carried is the current default for `universal-3-5-pro`. Earlier `u3-rt-pro` sessions carry 3.

## Passing your agent's reply as context

Pass your voice agent's spoken reply (what your TTS just said) via the `agent_context` parameter. There are two ways to set it:

- **At connection time:** pass `agent_context` as a query parameter on the WebSocket URL. Use this to seed the model with your agent's opening greeting before the user has said anything.
- **Mid-stream:** send an `UpdateConfiguration` message with the `agent_context` field after each subsequent agent reply.

Both forms let the model know the question the user is about to answer, which is especially important for short replies (`"yes"`, `"7pm"`, `"that's all"`).

The user side is handled for you: prior STT-finalized turns are automatically carried forward as context, so you only need to manage the agent half explicitly.

### Setting an opening greeting at connection time

When you open the WebSocket, pass `agent_context` alongside your other connection parameters. The first user turn will be transcribed with the greeting already in the model's context.

```python {2}
params = {
    "agent_context": "Welcome to the Krusty Krab, home of the Krabby Patty, may I take your order?",
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
}
```

```python {5}
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        agent_context="Welcome to the Krusty Krab, home of the Krabby Patty, may I take your order?",
    )
)
```

```javascript {4-5}
const params = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  agent_context:
    "Welcome to the Krusty Krab, home of the Krabby Patty, may I take your order?",
};
```

```javascript {4-5}
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  agentContext:
    "Welcome to the Krusty Krab, home of the Krabby Patty, may I take your order?",
});

await transcriber.connect();
```

### Updating agent context mid-stream

A typical voice agent loop looks like this:

1. User speaks → Universal-3.5 Pro Streaming emits a final turn.
2. Your agent runs an LLM step and generates a reply.
3. Your TTS speaks the reply to the user.
4. User responds → next turn.

During step 3, send the agent's reply text to the streaming session so the model knows what question the user will be answering next turn.

```python
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "agent_context": "Sure, what date would you like to book?",
}))
```

```python
client.update_configuration(
    agent_context="Sure, what date would you like to book?",
)
```

```javascript
ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  agent_context: "Sure, what date would you like to book?",
}));
```

```javascript
transcriber.updateConfiguration({
  agent_context: "Sure, what date would you like to book?",
});
```

## Limits

- **Universal-3.5 Pro only.** `agent_context` is supported on `speech_model: "universal-3-5-pro"`. If you set it at connection time on any other model, the session is rejected; if you send it mid-stream on another model, it's stripped with a warning.
- **Per-value cap: 1750 characters.** Trim long agent replies down to the substantive question before sending.

## When conversation context helps most

`agent_context` has the largest impact on:

- **Voice agents:** short user responses to agent questions (`"yes"`, `"no"`, `"that's all"`, dates, times, single names).
- **Spelled-out entities:** emails, account IDs, addresses, and similar inputs read aloud after the agent has just asked for them. Setting `agent_context` to the agent's prompt (e.g. `"What's your email address?"`) primes the model for what's coming.
- **Disambiguation:** words that sound similar but only one fits the conversation (`"fleas"` vs `"please"`, `"to"` vs `"two"` vs `"too"`).
- **Entity recall:** names, products, or terms that were established earlier in the conversation.

It has less impact on long, self-contained turns where the audio already provides enough context on its own.

## Interactions with other parameters

**`prompt`.** Conversation context layers on top of the model's built-in transcription instruction and any [contextual `prompt`](/streaming/prompting-and-keyterms) you provide. The two stack cleanly.

**`keyterms_prompt`.** Use `keyterms_prompt` alongside `agent_context` as needed; the two don't conflict.

**Multilingual sessions.** Carrying prior turns biases the model toward the languages already seen in the conversation. For sessions that mix three or more languages, this can occasionally push the model toward translating rather than transcribing. If you see drift, set a single transcription language in your `prompt` (see [Specifying the transcription language](/streaming/getting-started/transcribe-streaming-audio)).

---

<a id="multilingual-transcription-docs-streaming-multilingual-transcription-mdx"></a>

# Multilingual Transcription

URL: https://www.assemblyai.com/docs/streaming/multilingual-transcription
Source: docs/streaming/multilingual-transcription.mdx
Navigation: Real-time STT > Features
Description: Multilingual Transcription documentation.

Transcribe live audio in any supported language, or several at once, with native code switching.

Universal-3.5 Pro Streaming is multilingual by default. It follows speakers as they shift mid-sentence between languages like English and Spanish, French, Hindi or Mandarin, preserving exactly what was said without translating everything into a single language. No configuration is required.

Hear what code switching sounds like, and how it's transcribed.

##### **English \<\> French**

```txt
I said something like, j'ai dit à mes étudiants que it's time to really pay attention to what the idea of code-switching is.
```

##### **English \<\> Hindi**

```txt
मेरा रुकने का तो बहुत मन है, but I have an exam to give tomorrow.
```

##### **English \<\> Mandarin**

```txt
But this sentence, 我父母不工作了, you can see the 了 at the end of the sentence indicates that the situation now, my parents don't work, is different from what it was before.
```

## Quickstart

Code switching is on by default. Open a streaming session with no language parameters and the model code-switches natively across all supported languages.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
});
```

## Steering toward known languages

When you know which languages a session will use, pass the `language_codes` connection parameter to steer the model toward them and improve language accuracy.

- For a **known subset**, pass the codes you expect (for example, `["en", "es"]`). The model still code-switches, but heavily biased to the languages you list.
- For a **monolingual session**, pass a single-element list (for example, `["es"]`).
- For **full multilingual**, omit `language_codes` to keep native code switching across all supported languages.

Set the `language_codes` connection parameter when you open the WebSocket.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "language_codes": ["en", "es"],
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        language_codes=["en", "es"],
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  language_codes: ["en", "es"],
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  languageCodes: ["en", "es"],
});
```

**Describing languages in a contextual prompt**

You can also name the expected languages in natural language as part of a contextual prompt (for example, `Multilingual conversation in English, Spanish, and German.`), but `language_codes` is the best-supported way to select languages. See [Prompting and keyterms](/streaming/prompting-and-keyterms#specifying-the-language).

## Supported languages

Universal-3.5 Pro Streaming code-switches across 18 languages.

| Language   | Code | Language   | Code |
| ---------- | ---- | ---------- | ---- |
| English    | `en` | Danish     | `da` |
| Spanish    | `es` | Finnish    | `fi` |
| French     | `fr` | Hindi      | `hi` |
| German     | `de` | Vietnamese | `vi` |
| Italian    | `it` | Arabic     | `ar` |
| Portuguese | `pt` | Hebrew     | `he` |
| Turkish    | `tr` | Japanese   | `ja` |
| Dutch      | `nl` | Mandarin   | `zh` |
| Swedish    | `sv` | Norwegian  | `no` |

## Detecting the spoken language

Set the `language_detection` connection parameter to `true` to have each turn report which language was spoken. Turn messages then include `language_code` and `language_confidence` (a float between `0` and `1`) fields, populated when an utterance is complete or the turn is final.

```json
{
  "type": "Turn",
  "end_of_turn": true,
  "transcript": "¿Puedo pagar la factura por teléfono?",
  "language_code": "es",
  "language_confidence": 0.92
}
```

`language_detection` only controls reporting and doesn't change how the model transcribes. It's available on Universal-3.5 Pro Streaming and Universal-Streaming Multilingual.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "language_detection": True,
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        language_detection=True,
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  language_detection: true,
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  languageDetection: true,
});
```

## Updating languages mid-stream

`language_codes` can be updated during an active session without reconnecting. This is useful when a voice agent detects the caller's language, or when a conversation is handed off to a speaker of another language. Send an `UpdateConfiguration` message with the new values. The change takes effect from the next turn.

```python
# Steer transcription toward new languages
websocket.send('{"type": "UpdateConfiguration", "language_codes": ["en", "fr"]}')

# Clear steering and return to native code switching
websocket.send('{"type": "UpdateConfiguration", "language_codes": []}')
```

```python
# Steer transcription toward new languages
client.update_configuration(language_codes=["en", "fr"])

# Clear steering and return to native code switching
client.update_configuration(language_codes=[])
```

```javascript
// Steer transcription toward new languages
websocket.send(
  '{"type": "UpdateConfiguration", "language_codes": ["en", "fr"]}'
);

// Clear steering and return to native code switching
websocket.send('{"type": "UpdateConfiguration", "language_codes": []}');
```

```javascript
// Steer transcription toward new languages
transcriber.updateConfiguration({ languageCodes: ["en", "fr"] });

// Clear steering and return to native code switching
transcriber.updateConfiguration({ languageCodes: [] });
```

See [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream) for the full list of parameters you can update mid-stream.

## Other streaming models

Universal-3.5 Pro Streaming is the only streaming model with native (mid-sentence) code switching. The other streaming models handle languages differently.

| Model                            | Languages                              | Code switching |
| -------------------------------- | -------------------------------------- | -------------- |
| Universal-3.5 Pro Streaming      | 18 languages (see above)               | Native, mid-sentence |
| Universal-Streaming Multilingual | `en`, `es`, `de`, `fr`, `pt`, `it`     | Per turn       |
| Universal-Streaming English      | `en` only                              | No             |

See [Select the speech model](/streaming/select-the-speech-model) for a full model comparison.

## Looking for pre-recorded code switching?

This guide covers code switching for **streaming (real-time audio)**. If you're working with pre-recorded audio, see [Code Switching (Pre-recorded)](/pre-recorded-audio/code-switching).

---

<a id="medical-mode-docs-streaming-medical-mode-mdx"></a>

# Medical Mode

URL: https://www.assemblyai.com/docs/streaming/medical-mode
Source: docs/streaming/medical-mode.mdx
Navigation: Real-time STT > Features
Description: Medical Mode documentation.

Medical Mode is an add-on that enhances streaming transcription accuracy for medical terminology — including medication names, procedures, conditions, and dosages. It is optimized for medical entity recognition to correct terms that other models frequently get wrong.

Medical Mode can be used with all of our Real-time STT models.

Enable Medical Mode by setting the `domain` connection parameter to `"medical-v1"`. No other changes to your existing pipeline are required.

## Quickstart

Set `domain` to `"medical-v1"` as a connection parameter when you open the WebSocket.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        domain="medical-v1",
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  domain: "medical-v1",
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  domain: "medical-v1",
});
```

### Example output

Without Medical Mode:

```plain
I have here insulin to be used for both prandial mealtime and sliding scale is
insulin lisprohumalog subcutaneously.
```

With Medical Mode, lisprohumalog is updated to Lispro (Humalog) - following the standard medical convention of writing the generic name first, with the brand name in parentheses.

```plain
I have here insulin to be used for both prandial mealtime and sliding scale is
insulin Lispro (Humalog) subcutaneously.
```

## Use cases

Medical Mode is designed for healthcare AI applications where accurate medical terminology is critical:

- **Ambient clinical documentation** — Capture medication names, dosages, and clinical terms correctly during live patient encounters.
- **Real-time medical scribes** — Deliver accurate transcripts to clinicians during or immediately after a consult.
- **Front-office voice agents** — Handle drug names, provider names, and clinic-specific terminology in scheduling calls and insurance verification.
- **Medical contact centers** — Transcribe calls with correct medical vocabulary for downstream processing and quality assurance.

## Combine with other features

Medical Mode works alongside other streaming features. You can combine it with:

- [Streaming Diarization](/streaming/label-speakers-and-separate-channels) to identify who said what in clinical conversations
- [Keyterms Prompting](/streaming/prompting-and-keyterms) to further boost accuracy for specific medical terms unique to your use case

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "speaker_labels": "true",
    "keyterms_prompt": json.dumps(["Lisinopril", "Metformin", "Humalog"])
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        domain="medical-v1",
        speaker_labels=True,
        keyterms_prompt=["Lisinopril", "Metformin", "Humalog"],
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  domain: "medical-v1",
  speaker_labels: true,
  keyterms_prompt: JSON.stringify(["Lisinopril", "Metformin", "Humalog"]),
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  domain: "medical-v1",
  speakerLabels: true,
  keytermsPrompt: ["Lisinopril", "Metformin", "Humalog"],
});
```

## Configuration for medical audio

Medical conversations — such as clinical dictation, patient encounters, and ambient scribes — have different speech patterns than typical voice agent interactions. Clinicians often pause mid-sentence to think, review a chart, or formulate a diagnosis. The default turn detection settings are optimized for fast-paced voice agent dialogues and can incorrectly fragment these natural pauses into separate turns.

To prevent premature turn boundaries in medical audio, increase the silence thresholds:

```javascript
const streamingConfig = {
  min_turn_silence: 800,
  max_turn_silence: 3600,
};
```

| Parameter | Default | Recommended for Medical | Why |
| --- | --- | --- | --- |
| `min_turn_silence` | `100` ms (U3 Pro) / `400` ms (Universal Streaming) | `800` ms | Gives clinicians time to pause mid-sentence without triggering a speculative end-of-turn check. |
| `max_turn_silence` | `1000` ms (U3 Pro) / `1280` ms (Universal Streaming) | `3600` ms | Allows extended pauses for chart review or thinking without forcing a turn boundary. |

These values match the [Conservative quick start configuration](/streaming/getting-started/transcribe-streaming-audio) on the turn detection page. You can further adjust them based on your specific workflow — for example, a real-time medical scribe may benefit from a lower `max_turn_silence` (around `2000` ms) than a dictation application.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "domain": "medical-v1",
    "min_turn_silence": 800,
    "max_turn_silence": 3600,
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        domain="medical-v1",
        min_turn_silence=800,
        max_turn_silence=3600,
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  domain: "medical-v1",
  min_turn_silence: 800,
  max_turn_silence: 3600,
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  domain: "medical-v1",
  minTurnSilence: 800,
  maxTurnSilence: 3600,
});
```

  **Avoid setting end_of_turn_confidence_threshold to 0**

If you are using a Universal Streaming model (not U3 Pro), do not set `end_of_turn_confidence_threshold` to `0`. This completely disables semantic turn detection and forces a turn boundary at every silence, which is especially harmful for medical audio where mid-sentence pauses are common. See [Turn detection](/streaming/getting-started/transcribe-streaming-audio) for details.

## HIPAA compliance

AssemblyAI offers a Business Associate Agreement (BAA) for customers who need to process Protected Health Information (PHI). AssemblyAI is SOC 2 Type 2, ISO 27001:2022, and PCI DSS v4.0 certified. Medical Mode does not change existing data handling or retention policies.

Paid customers can review and sign our standard online BAA self-serve, at no additional cost, from the [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the dashboard. You must have an upgraded (paid) account to see the BAA option. See [Data Controls](/data-controls#business-associate-agreement-baa) for details, or [contact sales](https://www.assemblyai.com/contact-sales) for enterprise pricing.

---

<a id="voice-focus-docs-streaming-voice-focus-mdx"></a>

# Voice Focus

URL: https://www.assemblyai.com/docs/streaming/voice-focus
Source: docs/streaming/voice-focus.mdx
Navigation: Real-time STT > Features
Description: Voice Focus documentation.

Isolate the primary voice and suppress background noise on streaming transcription.

Hear the speaker, not the room. Voice Focus isolates the primary voice and suppresses background chatter, keyboard clicks, fan hum, and room echo before the audio reaches the transcription model.

## Variants

Pick the variant based on how close the speaker is to the microphone.

| Variant     | Value         | When to use                                                                           |
| ----------- | ------------- | ------------------------------------------------------------------------------------- |
| Near field  | `near-field`  | Headsets, handsets, and other close-talking microphones.                              |
| Far field   | `far-field`   | Conference rooms, drive-thru speakers, laptop mics, and other distant capture setups. |

## Tuning parameters

Tune how aggressively each variant suppresses background audio with the following parameter.

| Parameter               | Type  | Default | Description                                                                                                                                          |
| ----------------------- | ----- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `voice_focus_threshold` | float | `0.7`   | Optional. A float between `0.0` and `1.0` that controls how aggressively background audio is suppressed. Higher values are more aggressive. Defaults to `0.7` for both variants. |

## Quickstart

Set the `voice_focus` connection parameter when you open the WebSocket. See [Tuning parameters](#tuning-parameters) to adjust how aggressively background audio is suppressed.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "voice_focus": "near-field",
    # "voice_focus_threshold": 0.7,  # Optional, defaults to 0.7. Higher is more aggressive.
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        voice_focus="near-field",
        # voice_focus_threshold=0.7,  # Optional, defaults to 0.7. Higher is more aggressive.
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  voice_focus: "near-field",
  // voice_focus_threshold: 0.7, // Optional, defaults to 0.7. Higher is more aggressive.
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  voiceFocus: "near-field",
  // voiceFocusThreshold: 0.7, // Optional, defaults to 0.7. Higher is more aggressive.
});
```

---

<a id="streaming-diarization-and-multichannel-docs-streaming-label-speakers-and-separate-channels-mdx"></a>

# Streaming Diarization and Multichannel

URL: https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels
Source: docs/streaming/label-speakers-and-separate-channels.mdx
Navigation: Real-time STT > Features
Description: Streaming Diarization and Multichannel documentation.

Identify and label individual speakers in real time, or transcribe multichannel audio using the Streaming API.

## Overview

Streaming Diarization lets you identify and label individual speakers in real
time directly from the Streaming API. Each `Turn` event includes a `speaker_label`
field (e.g. `A`, `B`) indicating the dominant speaker for that turn. Each final
word in the `words` array also carries a `speaker` field, enabling mid-turn speaker
change detection. Speaker accuracy improves over the course of a session as the model
accumulates embedding context — so the longer the conversation, the better the labels.

  **Already using AssemblyAI streaming?**

  You can enable Streaming Diarization by adding `speaker_labels: true` to your
  connection parameters. No other changes are required — the `speaker_label` field
  will appear on every `Turn` event, and each final word in the `words` array will
  include a `speaker` field automatically.

### Quickstart

Enable Streaming Diarization by setting `speaker_labels` to `true` when you open the WebSocket.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "speaker_labels": True,
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        speaker_labels=True,
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  speaker_labels: true,
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  speakerLabels: true,
});
```

### Configuration

Enable Streaming Diarization by adding `speaker_labels: true` to your connection
parameters. You can optionally cap the number of speakers with `max_speakers`.
This is a hard limit, not a hint: if more people speak than the value you set, the
additional speakers are merged into the closest existing speaker label rather than
given a new one. If you're unsure of the exact speaker count, set `max_speakers` a
little higher than what you expect so the model has room to identify any additional
speakers. Setting it too high, though, can cause the model to over-split — assigning
new speaker labels to segments that actually belong to an existing speaker.

| Parameter        | Type    | Default | Description                                                                                                                                                        |
| ---------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `speaker_labels` | boolean | `false` | Set to `true` to enable real-time speaker diarization.                                                                                                             |
| `max_speakers`   | integer | —       | Optional. Hard cap on the number of speaker labels (1–10). If more people speak than this value, the additional speakers are merged into the closest existing label. Give the model a little headroom above the number of speakers you expect; setting it too high can cause over-splitting. |

  Diarization is supported on all streaming models: `universal-3-5-pro`,
  `universal-streaming-english`, and `universal-streaming-multilingual`. You do
  not need to change your speech model to use it — just add
  `speaker_labels: true`.

### Reading speaker labels

When diarization is enabled, every `Turn` event includes a `speaker_label` field
reflecting the dominant speaker for that turn.

```json
{
  "type": "Turn",
  "transcript": "Good morning, thanks for joining the call.",
  "speaker_label": "A",
  "end_of_turn": true,
  "turn_is_formatted": true
}
```

#### Word-level speaker labels

Each final word in the `words` array also carries a `speaker` field. This allows
you to detect speaker changes within a single turn — for example, a turn where one
speaker finishes another's sentence, or where a brief interjection appears mid-turn.

```json expandable
{
  "type": "Turn",
  "transcript": "Yeah. Different things the way they are said here than in contrast to the way they'd be said in other countries. Yeah. Your Colombian Spanish won't work. Uh, no. And she said she could— when I—",
  "speaker_label": "A",
  "end_of_turn": true,
  "words": [
    { "text": "Yeah.",      "speaker": "PENDING", "word_is_final": true, "start":    0, "end":   96, "confidence": 0.204160 },
    { "text": "Different",  "speaker": "A",       "word_is_final": true, "start":  145, "end":  290, "confidence": 0.844642 },
    { "text": "things",     "speaker": "A",       "word_is_final": true, "start":  306, "end":  516, "confidence": 0.998971 },
    { "text": "the",        "speaker": "A",       "word_is_final": true, "start":  581, "end":  613, "confidence": 0.807043 },
    { "text": "way",        "speaker": "A",       "word_is_final": true, "start":  662, "end":  694, "confidence": 0.999722 },
    { "text": "they",       "speaker": "A",       "word_is_final": true, "start":  840, "end":  920, "confidence": 0.995030 },
    { "text": "are",        "speaker": "A",       "word_is_final": true, "start":  985, "end": 1017, "confidence": 0.867126 },
    { "text": "said",       "speaker": "A",       "word_is_final": true, "start": 1082, "end": 1260, "confidence": 0.990436 },
    { "text": "here",       "speaker": "A",       "word_is_final": true, "start": 1308, "end": 1502, "confidence": 0.999566 },
    { "text": "than",       "speaker": "A",       "word_is_final": true, "start": 1873, "end": 2003, "confidence": 0.618784 },
    { "text": "in",         "speaker": "A",       "word_is_final": true, "start": 2116, "end": 2229, "confidence": 0.955239 },
    { "text": "contrast",   "speaker": "A",       "word_is_final": true, "start": 2277, "end": 2568, "confidence": 0.998716 },
    { "text": "to",         "speaker": "A",       "word_is_final": true, "start": 2617, "end": 2714, "confidence": 0.992622 },
    { "text": "the",        "speaker": "A",       "word_is_final": true, "start": 2778, "end": 2810, "confidence": 0.996170 },
    { "text": "way",        "speaker": "A",       "word_is_final": true, "start": 2859, "end": 2956, "confidence": 0.999566 },
    { "text": "they'd",     "speaker": "A",       "word_is_final": true, "start": 3020, "end": 3214, "confidence": 0.844162 },
    { "text": "be",         "speaker": "A",       "word_is_final": true, "start": 3263, "end": 3295, "confidence": 0.998969 },
    { "text": "said",       "speaker": "A",       "word_is_final": true, "start": 3424, "end": 3602, "confidence": 0.994370 },
    { "text": "in",         "speaker": "A",       "word_is_final": true, "start": 3650, "end": 3683, "confidence": 0.999225 },
    { "text": "other",      "speaker": "A",       "word_is_final": true, "start": 3747, "end": 3861, "confidence": 0.999323 },
    { "text": "countries.", "speaker": "A",       "word_is_final": true, "start": 3974, "end": 4281, "confidence": 0.868172 },
    { "text": "Yeah.",      "speaker": "PENDING", "word_is_final": true, "start": 4458, "end": 4636, "confidence": 0.656062 },
    { "text": "Your",       "speaker": "B",       "word_is_final": true, "start": 5040, "end": 5088, "confidence": 0.856220 },
    { "text": "Colombian",  "speaker": "B",       "word_is_final": true, "start": 5121, "end": 5638, "confidence": 0.962598 },
    { "text": "Spanish",    "speaker": "B",       "word_is_final": true, "start": 5638, "end": 6090, "confidence": 0.999557 },
    { "text": "won't",      "speaker": "B",       "word_is_final": true, "start": 6154, "end": 6284, "confidence": 0.999431 },
    { "text": "work.",      "speaker": "B",       "word_is_final": true, "start": 6332, "end": 6445, "confidence": 0.589761 },
    { "text": "Uh,",        "speaker": "A",       "word_is_final": true, "start": 6736, "end": 6752, "confidence": 0.343677 },
    { "text": "no.",        "speaker": "A",       "word_is_final": true, "start": 7673, "end": 7867, "confidence": 0.728975 },
    { "text": "And",        "speaker": "A",       "word_is_final": true, "start": 8820, "end": 8869, "confidence": 0.891464 },
    { "text": "she",        "speaker": "A",       "word_is_final": true, "start": 8901, "end": 9014, "confidence": 0.992945 },
    { "text": "said",       "speaker": "A",       "word_is_final": true, "start": 9079, "end": 9240, "confidence": 0.999052 },
    { "text": "she",        "speaker": "A",       "word_is_final": true, "start": 9305, "end": 9402, "confidence": 0.902266 },
    { "text": "could—",     "speaker": "A",       "word_is_final": true, "start": 9466, "end": 9579, "confidence": 0.605757 },
    { "text": "when",       "speaker": "A",       "word_is_final": true, "start": 9644, "end": 9757, "confidence": 0.827706 },
    { "text": "I—",         "speaker": "A",       "word_is_final": true, "start": 9870, "end": 9886, "confidence": 0.437470 }
  ]
}
```

A few things to keep in mind when consuming `speaker`:

- **Final words only.** The `speaker` field only appears on words where `word_is_final: true`. Non-final (in-progress) words never carry it.
- **`speaker` can be absent on individual words.** If the field is missing from a word entirely, treat that word as unattributed and fall back to the turn-level `speaker_label` if you need a label. Absent means the field is omitted from the JSON — it will never be `null`.
- **`PENDING` at word level** means the model couldn't confidently attribute that word to any specific speaker — common for short backchannels ("uh huh", "yeah") or brief low-quality audio segments. It is not an ambiguity flag between two known speakers; words in a confidently-attributed stretch carry the speaker's letter, not `PENDING`.

If a turn contains less than approximately 1 second of audio, the turn-level `speaker_label`
will be set to `"PENDING"`. This is because the model needs at least ~1 second of
audio to generate a reliable diarization embedding — without enough audio, embeddings
may be inaccurate and could lead to a single speaker being labeled as multiple
speakers. Labeling short turns as `"PENDING"` ensures that speaker labels remain
as accurate as possible.

```json
{
  "type": "Turn",
  "transcript": "Hello?",
  "speaker_label": "PENDING",
  "end_of_turn": true,
  "turn_is_formatted": true
}
```

Your application should handle this case gracefully.

A typical multi-speaker exchange looks like this:

```
[A] Good morning, thanks for joining the call.
[B] Good morning. Happy to be here.
[A] So let's start with a quick overview of the project timeline.
[B] Sure. We're currently on track for the March deadline.
[A] Great. And how's the team handling the workload?
[C] It's been busy, but manageable. We brought on two new engineers last week.
```

### How speaker accuracy improves over time

Streaming Diarization builds a speaker profile incrementally as audio flows in.
In practice this means:

- **Early in a session**, speaker assignments may be less stable, especially if
  the first few turns are short.
- **As the session progresses**, the model accumulates richer speaker embeddings
  and assignments become more consistent.

For long-form use cases (call center, clinical scribe, meeting transcription),
the model will settle into accurate, stable labels well before the end of the
conversation.

### Revised speaker labels

During a live session, Streaming Diarization assigns `speaker_label` values in
real time as each Turn is emitted. These labels can shift as the session
progresses and more audio becomes available. Early turns in particular may
be reassigned as the model builds a clearer picture of each speaker.

When the session ends, the server performs a final refinement pass with full
visibility into the entire conversation. Any turns whose speaker labels changed
are sent back as a single `SpeakerRevision` message. Turns that were already
correct are omitted.

Use the revised labels whenever you need the highest-quality speaker attribution
for the final transcript — for example, when persisting a meeting transcript,
generating a post-call summary, or feeding text into a downstream LLM that
benefits from accurate speaker turns.

  The end-of-session refinement adds approximately 400ms of latency. Any
  `SpeakerRevision` messages arrive before the `Termination` message and do
  not affect the real-time `speaker_label` values delivered during the session.

#### Message shape

A single `SpeakerRevision` message is sent at the end of the session containing
a `revisions` array. Each item corrects one turn; only turns whose speaker
assignments changed are included. The `turn_order` field in each item matches
the `turn_order` of the original Turn message being revised.

```json
{
  "type": "SpeakerRevision",
  "revisions": [
    {
      "turn_order": 3,
      "speaker_label": "B",
      "words": [
        { "text": "Hello",  "start": 1200, "end": 1450, "speaker": "B" },
        { "text": "there.", "start": 1450, "end": 1780, "speaker": "B" }
      ]
    },
    {
      "turn_order": 7,
      "speaker_label": "A",
      "words": [
        { "text": "Got it.", "start": 4100, "end": 4520, "speaker": "A" }
      ]
    }
  ]
}
```

| Field                       | Type                | Description                                                              |
| --------------------------- | ------------------- | ------------------------------------------------------------------------ |
| `type`                      | `"SpeakerRevision"` | Message type identifier.                                                 |
| `revisions`                 | `Revision[]`        | Array of turn corrections. Only turns whose labels changed are included. |
| `revisions[].turn_order`    | integer             | Matches the `turn_order` of the original `Turn` being corrected.         |
| `revisions[].speaker_label` | string \| null      | Corrected turn-level speaker label.                                      |
| `revisions[].words`         | `Word[]`            | Words with corrected per-word `speaker` assignments.                     |

  Text content and word timestamps are never changed. Only speaker
  assignments are revised.

#### How to handle it

Match each `turn_order` in `revisions` against the turn you already received,
then replace its `speaker_label` and per-word `speaker` values.

```python
def on_message(ws, message):
    data = json.loads(message)
    if data.get("type") == "SpeakerRevision":
        for revision in data.get("revisions", []):
            turn = turns_by_order[revision["turn_order"]]
            turn["speaker_label"] = revision["speaker_label"]
            for word, revised_word in zip(turn["words"], revision["words"]):
                word["speaker"] = revised_word["speaker"]
```

```python
from assemblyai.streaming.v3 import RealTimeEvents, SpeakerRevisionEvent

def on_speaker_revision(self, event: SpeakerRevisionEvent):
    for revision in event.revisions:
        turn = turns_by_order[revision.turn_order]
        turn.speaker_label = revision.speaker_label
        for word, revised_word in zip(turn.words, revision.words):
            word.speaker = revised_word.speaker

client.on(RealTimeEvents.SpeakerRevision, on_speaker_revision)
```

```javascript
ws.on("message", (data) => {
  const msg = JSON.parse(data);
  if (msg.type === "SpeakerRevision") {
    for (const revision of msg.revisions) {
      const turn = turnsByOrder[revision.turn_order];
      turn.speakerLabel = revision.speaker_label;
      revision.words.forEach((revisedWord, i) => {
        if (turn.words[i]) turn.words[i].speaker = revisedWord.speaker;
      });
    }
  }
});
```

#### When it is sent

`SpeakerRevision` messages are only sent at the end of a stream, after the
`Terminate` signal. They are never emitted mid-session. A given session may
produce zero or many revisions. Only turns whose speaker assignments changed
are included.

If the session ends unexpectedly (network drop, error closure), revisions may
not be delivered. Always handle this gracefully and fall back to the live
labels you received during the session.

### Known limitations

Real-time diarization is an inherently harder problem than diarization for async transcription on pre-recorded audio.
The following limitations apply to the current beta:

- **Short utterances** — Turns with less than ~1 second of audio are labeled
  as `"PENDING"` because there is insufficient audio to generate a reliable
  speaker embedding. This prevents inaccurate embeddings from causing a single
  speaker to be split across multiple labels.
- **Overlapping speech** — When two speakers talk simultaneously, the model
  cannot split the audio and will assign the turn to a single speaker.
  Performance degrades with frequent cross-talk.
- **Session start accuracy** — The first 1–2 turns of a session may be
  misassigned because the model has not yet built up speaker profiles. This
  self-corrects quickly in practice.
- **Noisy environments** — Background noise and microphone bleed between
  speakers can reduce embedding quality and lead to more frequent
  misassignments.

  For the best results, use a microphone setup that minimizes cross-talk and
  background noise, and ensure each speaker produces at least a few complete
  sentences before you rely on per-turn labels for downstream processing.

---

## Multichannel streaming audio

To transcribe multichannel streaming audio, we recommend creating a separate session for each channel. This approach allows you to maintain clear speaker separation and get accurate diarized transcriptions for conversations, phone calls, or interviews where speakers are recorded on two different channels.

The following code example demonstrates how to transcribe a dual-channel audio file with diarized, speaker-separated transcripts. This same approach can be applied to any multi-channel audio stream, including those with more than two channels.

        
            

            Firstly, install the required dependencies.

```bash
pip install websocket-client numpy pyaudio
```

            
            
            Use this complete script to transcribe dual-channel audio with speaker separation:

```python expandable
import websocket
import json
import threading
import numpy as np
import wave
import time
import pyaudio
from urllib.parse import urlencode

# Configuration
YOUR_API_KEY = "<YOUR_API_KEY>"
AUDIO_FILE_PATH = "<DUAL_CHANNEL_AUDIO_FILE_PATH>"
API_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_PARAMS = {
    "sample_rate": 8000,
    "format_turns": "true",
    "end_of_turn_confidence_threshold": 0.4,
    "min_turn_silence": 160,
    "max_turn_silence": 400,
}
# Build API endpoint with URL encoding
API_ENDPOINT = f"{API_BASE_URL}?{urlencode(API_PARAMS)}"

class ChannelTranscriber:
    def __init__(self, channel_id, channel_name):
        self.channel_id = channel_id
        self.channel_name = channel_name
        self.ws_app = None
        self.audio_data = []
        self.current_turn_line = None
        self.line_count = 0

    def load_audio_channel(self):
        """Extract single channel from dual-channel audio file."""
        with wave.open(AUDIO_FILE_PATH, 'rb') as wf:
            frames = wf.readframes(wf.getnframes())
            audio_array = np.frombuffer(frames, dtype=np.int16)

            if wf.getnchannels() == 2:
                audio_array = audio_array.reshape(-1, 2)
                channel_audio = audio_array[:, self.channel_id]

                # Split into chunks for streaming
                FRAMES_PER_BUFFER = 400  # 50ms chunks
                for i in range(0, len(channel_audio), FRAMES_PER_BUFFER):
                    chunk = channel_audio[i:i+FRAMES_PER_BUFFER]
                    if len(chunk) < FRAMES_PER_BUFFER:
                        chunk = np.pad(chunk, (0, FRAMES_PER_BUFFER - len(chunk)), 'constant')
                    self.audio_data.append(chunk.astype(np.int16).tobytes())

    def on_open(self, ws):
        """Stream audio data when connection opens."""
        def stream_audio():
            for chunk in self.audio_data:
                ws.send(chunk, websocket.ABNF.OPCODE_BINARY)
                time.sleep(0.05)  # 50ms intervals

            # Send termination message
            terminate_message = {"type": "Terminate"}
            ws.send(json.dumps(terminate_message))

        threading.Thread(target=stream_audio, daemon=True).start()

    def clear_current_line(self):
        if self.current_turn_line is not None:
            print("\r" + " " * 100 + "\r", end="", flush=True)

    def print_partial_transcript(self, words):
        self.clear_current_line()
        # Build transcript from individual words
        word_texts = [word.get('text', '') for word in words]
        transcript = ' '.join(word_texts)
        partial_text = f"{self.channel_name}: {transcript}"
        print(partial_text, end="", flush=True)
        self.current_turn_line = len(partial_text)

    def print_final_transcript(self, transcript):
        self.clear_current_line()
        final_text = f"{self.channel_name}: {transcript}"
        print(final_text, flush=True)
        self.current_turn_line = None
        self.line_count += 1

    def on_message(self, ws, message):
        """Handle transcription results."""
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Turn":
            transcript = data.get('transcript', '').strip()
            words = data.get('words', [])

            if transcript or words:
                if data.get('end_of_turn'):
                    self.print_final_transcript(transcript)
                else:
                    self.print_partial_transcript(words)

    def start_transcription(self):
        self.load_audio_channel()

        self.ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=self.on_open,
            on_message=self.on_message,
        )

        thread = threading.Thread(target=self.ws_app.run_forever, daemon=True)
        thread.start()
        return thread

def play_audio_file():
    try:
        with wave.open(AUDIO_FILE_PATH, 'rb') as wf:
            p = pyaudio.PyAudio()

            stream = p.open(
                format=p.get_format_from_width(wf.getsampwidth()),
                channels=wf.getnchannels(),
                rate=wf.getframerate(),
                output=True
            )

            print(f"Playing audio: {AUDIO_FILE_PATH}")

            # Play audio in chunks
            chunk_size = 1024
            data = wf.readframes(chunk_size)

            while data:
                stream.write(data)
                data = wf.readframes(chunk_size)

            stream.stop_stream()
            stream.close()
            p.terminate()

            print("Audio playback finished")

    except Exception as e:
        print(f"Error playing audio: {e}")

def transcribe_multichannel():
    # Create transcribers for each channel
    transcriber_1 = ChannelTranscriber(0, "Speaker 1")
    transcriber_2 = ChannelTranscriber(1, "Speaker 2")

    # Start audio playback
    audio_thread = threading.Thread(target=play_audio_file, daemon=True)
    audio_thread.start()

    # Start both transcriptions
    thread_1 = transcriber_1.start_transcription()
    thread_2 = transcriber_2.start_transcription()

    # Wait for completion
    thread_1.join()
    thread_2.join()
    audio_thread.join()

if __name__ == "__main__":
    transcribe_multichannel()
```

            
        
    

        
            

            Install the required dependencies.

```bash
pip install assemblyai numpy pyaudio
```

            
            
            Use this complete script to transcribe dual-channel audio with speaker separation:

```python expandable
import logging
from typing import Type
import threading
import time
import wave
import numpy as np
import pyaudio

from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    TerminationEvent,
    TurnEvent,
)

# Configuration
API_KEY = "<YOUR_API_KEY>"
AUDIO_FILE_PATH = "<DUAL_CHANNEL_AUDIO_FILE_PATH>"

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class ChannelTranscriber:
    def __init__(self, channel_id, channel_name, sample_rate):
        self.channel_id = channel_id
        self.channel_name = channel_name
        self.sample_rate = sample_rate
        self.client = None
        self.audio_data = []
        self.current_turn_line = None
        self.line_count = 0
        self.streaming_done = threading.Event()

    def load_audio_channel(self):
        """Extract single channel from dual-channel audio file."""
        with wave.open(AUDIO_FILE_PATH, 'rb') as wf:
            frames = wf.readframes(wf.getnframes())
            audio_array = np.frombuffer(frames, dtype=np.int16)

            if wf.getnchannels() == 2:
                audio_array = audio_array.reshape(-1, 2)
                channel_audio = audio_array[:, self.channel_id]

                # Split into chunks for streaming
                FRAMES_PER_BUFFER = 400  # 50ms chunks
                for i in range(0, len(channel_audio), FRAMES_PER_BUFFER):
                    chunk = channel_audio[i:i+FRAMES_PER_BUFFER]
                    if len(chunk) < FRAMES_PER_BUFFER:
                        chunk = np.pad(chunk, (0, FRAMES_PER_BUFFER - len(chunk)), 'constant')
                    self.audio_data.append(chunk.astype(np.int16).tobytes())

    def clear_current_line(self):
        if self.current_turn_line is not None:
            print("\r" + " " * 100 + "\r", end="", flush=True)

    def print_partial_transcript(self, words):
        self.clear_current_line()
        # Build transcript from individual words
        word_texts = [word.text for word in words]
        transcript = ' '.join(word_texts)
        partial_text = f"{self.channel_name}: {transcript}"
        print(partial_text, end="", flush=True)
        self.current_turn_line = len(partial_text)

    def print_final_transcript(self, transcript):
        self.clear_current_line()
        final_text = f"{self.channel_name}: {transcript}"
        print(final_text, flush=True)
        self.current_turn_line = None
        self.line_count += 1

    def on_begin(self, client: Type[RealTimeTranscriber], event: BeginEvent):
        """Called when the streaming session begins."""
        pass  # Session started

    def on_turn(self, client: Type[RealTimeTranscriber], event: TurnEvent):
        """Called when a turn is received."""
        transcript = event.transcript.strip() if event.transcript else ''
        words = event.words if event.words else []

        if transcript or words:
            if event.end_of_turn:
                self.print_final_transcript(transcript)
            else:
                self.print_partial_transcript(words)

    def on_terminated(self, client: Type[RealTimeTranscriber], event: TerminationEvent):
        """Called when the session is terminated."""
        self.clear_current_line()
        self.streaming_done.set()

    def on_error(self, client: Type[RealTimeTranscriber], error: RealTimeError):
        """Called when an error occurs."""
        print(f"\n{self.channel_name}: Error: {error}")
        self.streaming_done.set()

    def start_transcription(self):
        """Start the transcription for this channel."""
        self.load_audio_channel()

        # Create streaming client
        self.client = RealTimeTranscriber(api_key=API_KEY)

        # Register event handlers
        self.client.on(RealTimeEvents.Begin, self.on_begin)
        self.client.on(RealTimeEvents.Turn, self.on_turn)
        self.client.on(RealTimeEvents.Termination, self.on_terminated)
        self.client.on(RealTimeEvents.Error, self.on_error)

        # Connect to streaming service with turn detection configuration
        self.client.connect(
            RealTimeParameters(
                sample_rate=self.sample_rate,
                format_turns=True,
                end_of_turn_confidence_threshold=0.4,
                min_turn_silence=160,
                max_turn_silence=400,
            )
        )

        # Create audio generator
        def audio_generator():
            for chunk in self.audio_data:
                yield chunk
                time.sleep(0.05)  # 50ms intervals

        try:
            # Stream audio
            self.client.stream(audio_generator())
        finally:
            # Disconnect
            self.client.disconnect(terminate=True)
            self.streaming_done.set()

    def start_transcription_thread(self):
        """Start transcription in a separate thread."""
        thread = threading.Thread(target=self.start_transcription, daemon=True)
        thread.start()
        return thread

def play_audio_file():
    try:
        with wave.open(AUDIO_FILE_PATH, 'rb') as wf:
            p = pyaudio.PyAudio()

            stream = p.open(
                format=p.get_format_from_width(wf.getsampwidth()),
                channels=wf.getnchannels(),
                rate=wf.getframerate(),
                output=True
            )

            print(f"Playing audio: {AUDIO_FILE_PATH}")

            # Play audio in chunks
            chunk_size = 1024
            data = wf.readframes(chunk_size)

            while data:
                stream.write(data)
                data = wf.readframes(chunk_size)

            stream.stop_stream()
            stream.close()
            p.terminate()

            print("Audio playback finished")

    except Exception as e:
        print(f"Error playing audio: {e}")

def transcribe_multichannel():
    # Get sample rate from file
    with wave.open(AUDIO_FILE_PATH, 'rb') as wf:
        sample_rate = wf.getframerate()

    # Create transcribers for each channel
    transcriber_1 = ChannelTranscriber(0, "Speaker 1", sample_rate)
    transcriber_2 = ChannelTranscriber(1, "Speaker 2", sample_rate)

    # Start audio playback
    audio_thread = threading.Thread(target=play_audio_file, daemon=True)
    audio_thread.start()

    # Start both transcriptions
    thread_1 = transcriber_1.start_transcription_thread()
    thread_2 = transcriber_2.start_transcription_thread()

    # Wait for completion
    thread_1.join()
    thread_2.join()
    audio_thread.join()

if __name__ == "__main__":
    transcribe_multichannel()
```

            
        
    

        
            

            Firstly, install the required dependencies.

```bash
npm install ws
```

            
            
            Use this complete script to transcribe dual-channel audio with speaker separation:

```javascript expandable
const WebSocket = require("ws");
const fs = require("fs");
const { spawn } = require("child_process");

// Configuration
const YOUR_API_KEY = "<YOUR_API_KEY>";
const AUDIO_FILE_PATH = "<DUAL_CHANNEL_AUDIO_FILE_PATH>";
const API_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_PARAMS = {
  sample_rate: 8000,
  format_turns: "true",
  end_of_turn_confidence_threshold: 0.4,
  min_turn_silence: 160,
  max_turn_silence: 400,
};

// Build API endpoint with URL encoding
const queryString = new URLSearchParams(API_PARAMS).toString();
const API_ENDPOINT = `${API_BASE_URL}?${queryString}`;

// Simple WAV file parser
class SimpleWavParser {
  constructor(filePath) {
    this.buffer = fs.readFileSync(filePath);
    this.parseHeader();
  }

  parseHeader() {
    // Read WAV header
    this.channels = this.buffer.readUInt16LE(22);
    this.sampleRate = this.buffer.readUInt32LE(24);
    this.bitsPerSample = this.buffer.readUInt16LE(34);

    // Find data chunk
    let dataOffset = 12;
    while (dataOffset < this.buffer.length - 8) {
      const chunkId = this.buffer.toString("ascii", dataOffset, dataOffset + 4);
      const chunkSize = this.buffer.readUInt32LE(dataOffset + 4);

      if (chunkId === "data") {
        this.dataStart = dataOffset + 8;
        this.dataSize = chunkSize;
        break;
      }

      dataOffset += 8 + chunkSize;
    }
  }

  getChannelData(channelIndex) {
    if (this.channels !== 2) {
      throw new Error("Audio file is not stereo");
    }

    const bytesPerSample = this.bitsPerSample / 8;
    const samplesPerChannel = this.dataSize / (bytesPerSample * this.channels);
    const channelData = [];

    // Extract samples for the specified channel
    for (let i = 0; i < samplesPerChannel; i++) {
      const sampleOffset =
        this.dataStart + (i * this.channels + channelIndex) * bytesPerSample;

      if (this.bitsPerSample === 16) {
        const sample = this.buffer.readInt16LE(sampleOffset);
        channelData.push(sample);
      } else if (this.bitsPerSample === 8) {
        const sample = this.buffer.readUInt8(sampleOffset) - 128;
        channelData.push(sample * 256); // Convert to 16-bit range
      }
    }

    return channelData;
  }
}

class ChannelTranscriber {
  constructor(channelId, channelName) {
    this.channelId = channelId;
    this.channelName = channelName;
    this.ws = null;
    this.audioData = [];
    this.currentTurnLine = null;
    this.lineCount = 0;
    this.isConnected = false;
  }

  loadAudioChannel() {
    try {
      const wavParser = new SimpleWavParser(AUDIO_FILE_PATH);
      const channelSamples = wavParser.getChannelData(this.channelId);

      // Split into chunks for streaming (50ms chunks at 8000Hz = 400 samples)
      const FRAMES_PER_BUFFER = 400;

      for (let i = 0; i < channelSamples.length; i += FRAMES_PER_BUFFER) {
        const chunkArray = new Int16Array(FRAMES_PER_BUFFER);

        // Copy samples and pad if necessary
        for (let j = 0; j < FRAMES_PER_BUFFER; j++) {
          if (i + j < channelSamples.length) {
            chunkArray[j] = channelSamples[i + j];
          } else {
            chunkArray[j] = 0; // Pad with silence
          }
        }

        // Convert to Buffer (Little Endian)
        const buffer = Buffer.from(chunkArray.buffer);
        this.audioData.push(buffer);
      }
    } catch (error) {
      throw error;
    }
  }

  clearCurrentLine() {
    if (this.currentTurnLine !== null) {
      process.stdout.write("\r" + " ".repeat(100) + "\r");
    }
  }

  printPartialTranscript(words) {
    this.clearCurrentLine();
    // Build transcript from individual words
    const wordTexts = words.map((word) => word.text || "");
    const transcript = wordTexts.join(" ");
    const partialText = `${this.channelName}: ${transcript}`;
    process.stdout.write(partialText);
    this.currentTurnLine = partialText.length;
  }

  printFinalTranscript(transcript) {
    this.clearCurrentLine();
    const finalText = `${this.channelName}: ${transcript}`;
    console.log(finalText);
    this.currentTurnLine = null;
    this.lineCount++;
  }

  async streamAudio() {
    // Wait a bit for connection to stabilize
    await new Promise((resolve) => setTimeout(resolve, 100));

    for (const chunk of this.audioData) {
      if (this.ws.readyState === WebSocket.OPEN) {
        this.ws.send(chunk, { binary: true });
        await new Promise((resolve) => setTimeout(resolve, 50)); // 50ms intervals
      } else {
        break;
      }
    }

    // Send termination message
    if (this.ws.readyState === WebSocket.OPEN) {
      const terminateMessage = { type: "Terminate" };
      this.ws.send(JSON.stringify(terminateMessage));
    }
  }

  startTranscription() {
    return new Promise((resolve, reject) => {
      try {
        this.loadAudioChannel();
      } catch (error) {
        reject(error);
        return;
      }

      this.ws = new WebSocket(API_ENDPOINT, {
        headers: {
          Authorization: YOUR_API_KEY,
        },
      });

      this.ws.on("open", () => {
        this.isConnected = true;
        // Start streaming audio
        this.streamAudio().catch((error) => {});
      });

      this.ws.on("message", (data) => {
        try {
          const message = JSON.parse(data.toString());
          const msgType = message.type;

          if (msgType === "Turn") {
            const transcript = (message.transcript || "").trim();
            const words = message.words || [];

            if (transcript || words.length > 0) {
              if (message.end_of_turn) {
                this.printFinalTranscript(transcript);
              } else {
                this.printPartialTranscript(words);
              }
            }
          } else if (msgType === "error") {
            console.error(`\n${this.channelName}: API Error:`, message.error);
          }
        } catch (error) {
          // Silently ignore parse errors
        }
      });

      this.ws.on("close", (code, reason) => {
        this.clearCurrentLine();
        if (code !== 1000 && code !== 1001) {
          console.log(`\n${this.channelName}: Connection closed unexpectedly`);
        }
        this.isConnected = false;
        resolve();
      });

      this.ws.on("error", (error) => {
        console.error(`\n${this.channelName} WebSocket error:`, error.message);
        this.isConnected = false;
        reject(error);
      });
    });
  }

  close() {
    if (this.ws && this.isConnected) {
      this.ws.close();
    }
  }
}

function playAudioFile() {
  return new Promise((resolve) => {
    console.log(`Playing audio: ${AUDIO_FILE_PATH}`);

    // Use platform-specific audio player
    let command;
    let args;

    if (process.platform === "darwin") {
      // macOS
      command = "afplay";
      args = [AUDIO_FILE_PATH];
    } else if (process.platform === "win32") {
      // Windows - using PowerShell
      command = "powershell";
      args = [
        "-c",
        `(New-Object Media.SoundPlayer '${AUDIO_FILE_PATH}').PlaySync()`,
      ];
    } else {
      // Linux - try aplay
      command = "aplay";
      args = [AUDIO_FILE_PATH];
    }

    try {
      const player = spawn(command, args, {
        stdio: ["ignore", "ignore", "ignore"], // Suppress all output from player
      });

      player.on("close", (code) => {
        if (code === 0) {
          console.log("Audio playback finished");
        }
        resolve();
      });

      player.on("error", (error) => {
        // Silently continue without audio
        resolve();
      });
    } catch (error) {
      resolve();
    }
  });
}

async function transcribeMultichannel() {
  const transcriber1 = new ChannelTranscriber(0, "Speaker 1");
  const transcriber2 = new ChannelTranscriber(1, "Speaker 2");

  try {
    // Verify API key is set
    if (YOUR_API_KEY === "<YOUR_API_KEY>") {
      console.error("ERROR: Please set YOUR_API_KEY before running");
      process.exit(1);
    }

    // Verify file exists
    if (!fs.existsSync(AUDIO_FILE_PATH)) {
      console.error(`ERROR: Audio file not found: ${AUDIO_FILE_PATH}`);
      process.exit(1);
    }

    // Start audio playback (non-blocking)
    const audioPromise = playAudioFile();

    // Start both transcriptions
    const transcriptionPromises = [
      transcriber1.startTranscription(),
      transcriber2.startTranscription(),
    ];

    // Wait for all to complete
    await Promise.all([...transcriptionPromises, audioPromise]);
  } catch (error) {
    console.error("\nError during transcription:", error.message);

    // Clean up
    transcriber1.close();
    transcriber2.close();

    process.exit(1);
  }
}

// Handle graceful shutdown
process.on("SIGINT", () => {
  console.log("\n"); // Clean line break before exit
  process.exit(0);
});

// Main execution
if (require.main === module) {
  transcribeMultichannel();
}
```

            
        
    

        
            

            Firstly, install the required dependencies.

```bash
npm install assemblyai
```

            
            
            Use this complete script to transcribe dual-channel audio with speaker separation:

```javascript expandable
import { AssemblyAI } from "assemblyai";
import fs from "fs";
import { spawn } from "child_process";
import { Readable } from "stream";

// Configuration
const YOUR_API_KEY = "<YOUR_API_KEY>";
const AUDIO_FILE_PATH = "<DUAL_CHANNEL_AUDIO_FILE_PATH>";

// Simple WAV file parser
class SimpleWavParser {
  constructor(filePath) {
    this.buffer = fs.readFileSync(filePath);
    this.parseHeader();
  }

  parseHeader() {
    // Read WAV header
    this.channels = this.buffer.readUInt16LE(22);
    this.sampleRate = this.buffer.readUInt32LE(24);
    this.bitsPerSample = this.buffer.readUInt16LE(34);

    // Find data chunk
    let dataOffset = 12;
    while (dataOffset < this.buffer.length - 8) {
      const chunkId = this.buffer.toString("ascii", dataOffset, dataOffset + 4);
      const chunkSize = this.buffer.readUInt32LE(dataOffset + 4);

      if (chunkId === "data") {
        this.dataStart = dataOffset + 8;
        this.dataSize = chunkSize;
        break;
      }

      dataOffset += 8 + chunkSize;
    }
  }

  getChannelData(channelIndex) {
    if (this.channels !== 2) {
      throw new Error("Audio file is not stereo");
    }

    const bytesPerSample = this.bitsPerSample / 8;
    const samplesPerChannel = this.dataSize / (bytesPerSample * this.channels);
    const channelData = [];

    // Extract samples for the specified channel
    for (let i = 0; i < samplesPerChannel; i++) {
      const sampleOffset =
        this.dataStart + (i * this.channels + channelIndex) * bytesPerSample;

      if (this.bitsPerSample === 16) {
        const sample = this.buffer.readInt16LE(sampleOffset);
        channelData.push(sample);
      } else if (this.bitsPerSample === 8) {
        const sample = this.buffer.readUInt8(sampleOffset) - 128;
        channelData.push(sample * 256); // Convert to 16-bit range
      }
    }

    return channelData;
  }
}

class ChannelTranscriber {
  constructor(client, channelId, channelName, sampleRate) {
    this.client = client;
    this.channelId = channelId;
    this.channelName = channelName;
    this.sampleRate = sampleRate;
    this.transcriber = null;
    this.audioData = [];
    this.currentTurnLine = null;
    this.lineCount = 0;
  }

  loadAudioChannel() {
    try {
      const wavParser = new SimpleWavParser(AUDIO_FILE_PATH);
      const channelSamples = wavParser.getChannelData(this.channelId);

      // Split into chunks for streaming (50ms chunks)
      const FRAMES_PER_BUFFER = Math.floor(this.sampleRate * 0.05); // 50ms

      for (let i = 0; i < channelSamples.length; i += FRAMES_PER_BUFFER) {
        const chunkArray = new Int16Array(FRAMES_PER_BUFFER);

        // Copy samples and pad if necessary
        for (let j = 0; j < FRAMES_PER_BUFFER; j++) {
          if (i + j < channelSamples.length) {
            chunkArray[j] = channelSamples[i + j];
          } else {
            chunkArray[j] = 0; // Pad with silence
          }
        }

        // Convert to Buffer (Little Endian)
        const buffer = Buffer.from(chunkArray.buffer);
        this.audioData.push(buffer);
      }
    } catch (error) {
      throw error;
    }
  }

  clearCurrentLine() {
    if (this.currentTurnLine !== null) {
      process.stdout.write("\r" + " ".repeat(100) + "\r");
    }
  }

  printPartialTranscript(words) {
    this.clearCurrentLine();
    // Build transcript from individual words
    const wordTexts = words.map((word) => word.text || "");
    const transcript = wordTexts.join(" ");
    const partialText = `${this.channelName}: ${transcript}`;
    process.stdout.write(partialText);
    this.currentTurnLine = partialText.length;
  }

  printFinalTranscript(transcript) {
    this.clearCurrentLine();
    const finalText = `${this.channelName}: ${transcript}`;
    console.log(finalText);
    this.currentTurnLine = null;
    this.lineCount++;
  }

  async startTranscription() {
    try {
      this.loadAudioChannel();
    } catch (error) {
      throw error;
    }

    const turnDetectionConfig = {
      endOfTurnConfidenceThreshold: 0.4,
      minEndOfTurnSilenceWhenConfident: 160,
      maxTurnSilence: 400,
    };

    // Create transcriber with SDK
    this.transcriber = this.client.streaming.transcriber({
      sampleRate: this.sampleRate,
      formatTurns: true,
      ...turnDetectionConfig,
    });

    // Set up event handlers
    this.transcriber.on("open", ({ id }) => {
      // Session opened
    });

    this.transcriber.on("error", (error) => {
      console.error(`\n${this.channelName}: Error:`, error);
    });

    this.transcriber.on("close", (code, reason) => {
      this.clearCurrentLine();
      if (code !== 1000 && code !== 1001) {
        console.log(`\n${this.channelName}: Connection closed unexpectedly`);
      }
    });

    this.transcriber.on("turn", (turn) => {
      const transcript = (turn.transcript || "").trim();
      const words = turn.words || [];

      if (transcript || words.length > 0) {
        if (turn.end_of_turn) {
          this.printFinalTranscript(transcript);
        } else {
          this.printPartialTranscript(words);
        }
      }
    });

    // Connect to the streaming service
    await this.transcriber.connect();

    // Create a readable stream from audio chunks
    const audioStream = new Readable({
      async read() {
        // This will be controlled by our manual push below
      },
    });

    // Pipe audio stream to transcriber
    Readable.toWeb(audioStream).pipeTo(this.transcriber.stream());

    // Stream audio data
    for (const chunk of this.audioData) {
      audioStream.push(chunk);
      await new Promise((resolve) => setTimeout(resolve, 50)); // 50ms intervals
    }

    // Signal end of stream
    audioStream.push(null);

    // Wait a bit for final transcripts
    await new Promise((resolve) => setTimeout(resolve, 1000));

    // Close the transcriber
    await this.transcriber.close();
  }

  async close() {
    if (this.transcriber) {
      await this.transcriber.close();
    }
  }
}

function playAudioFile() {
  return new Promise((resolve) => {
    console.log(`Playing audio: ${AUDIO_FILE_PATH}`);

    // Use platform-specific audio player
    let command;
    let args;

    if (process.platform === "darwin") {
      // macOS
      command = "afplay";
      args = [AUDIO_FILE_PATH];
    } else if (process.platform === "win32") {
      // Windows - using PowerShell
      command = "powershell";
      args = [
        "-c",
        `(New-Object Media.SoundPlayer '${AUDIO_FILE_PATH}').PlaySync()`,
      ];
    } else {
      // Linux - try aplay
      command = "aplay";
      args = [AUDIO_FILE_PATH];
    }

    try {
      const player = spawn(command, args, {
        stdio: ["ignore", "ignore", "ignore"], // Suppress all output from player
      });

      player.on("close", (code) => {
        if (code === 0) {
          console.log("Audio playback finished");
        }
        resolve();
      });

      player.on("error", (error) => {
        // Silently continue without audio
        resolve();
      });
    } catch (error) {
      resolve();
    }
  });
}

async function transcribeMultichannel() {
  // Verify API key is set
  if (YOUR_API_KEY === "<YOUR_API_KEY>") {
    console.error("ERROR: Please set YOUR_API_KEY before running");
    process.exit(1);
  }

  // Verify file exists
  if (!fs.existsSync(AUDIO_FILE_PATH)) {
    console.error(`ERROR: Audio file not found: ${AUDIO_FILE_PATH}`);
    process.exit(1);
  }

  // Get sample rate from file
  const wavParser = new SimpleWavParser(AUDIO_FILE_PATH);
  const sampleRate = wavParser.sampleRate;

  // Create SDK client
  const client = new AssemblyAI({
    apiKey: YOUR_API_KEY,
  });

  const transcriber1 = new ChannelTranscriber(
    client,
    0,
    "Speaker 1",
    sampleRate
  );
  const transcriber2 = new ChannelTranscriber(
    client,
    1,
    "Speaker 2",
    sampleRate
  );

  try {
    // Start audio playback (non-blocking)
    const audioPromise = playAudioFile();

    // Start both transcriptions
    const transcriptionPromises = [
      transcriber1.startTranscription(),
      transcriber2.startTranscription(),
    ];

    // Wait for all to complete
    await Promise.all([...transcriptionPromises, audioPromise]);
  } catch (error) {
    console.error("\nError during transcription:", error.message);

    // Clean up
    await transcriber1.close();
    await transcriber2.close();

    process.exit(1);
  }
}

// Handle graceful shutdown
process.on("SIGINT", () => {
  console.log("\n"); // Clean line break before exit
  process.exit(0);
});

// Main execution
transcribeMultichannel();
```

            
        
    

  **Configure turn detection for your use case**

The examples above use turn detection settings optimized for short responses and rapid back-and-forth conversations. To optimize for your specific audio scenario, you can adjust the turn detection parameters.

For configuration examples tailored to different use cases, refer to our [Configuration examples](/streaming/getting-started/transcribe-streaming-audio).

Modify the turn detection parameters in `API_PARAMS`:
```python
API_PARAMS = {
    "sample_rate": 8000,
    "format_turns": "true",
    "end_of_turn_confidence_threshold": 0.4,
    "min_turn_silence": 160,
    "max_turn_silence": 400,
}
```

Modify the `RealTimeParameters` in the `start_transcription` method:
```python
# Connect to streaming service with turn detection configuration
self.client.connect(
    RealTimeParameters(
        sample_rate=self.sample_rate,
        format_turns=True,
        end_of_turn_confidence_threshold=0.4,
        min_turn_silence=160,
        max_turn_silence=400,
    )
)
```

Modify the turn detection parameters in `API_PARAMS`:
```javascript
const API_PARAMS = {
    sample_rate: 8000,
    format_turns: 'true',
    end_of_turn_confidence_threshold: 0.4,
    min_turn_silence: 160,
    max_turn_silence: 400,
};
````

Modify the turn detection configuration object:
```javascript
const turnDetectionConfig = {
  endOfTurnConfidenceThreshold: 0.4,
  minEndOfTurnSilenceWhenConfident: 160,
  maxTurnSilence: 400
};

// Create transcriber with SDK
this.transcriber = this.client.streaming.transcriber({
sampleRate: this.sampleRate,
formatTurns: true,
...turnDetectionConfig
});

````

---

<a id="pii-redaction-docs-streaming-pii-redaction-mdx"></a>

# PII Redaction

URL: https://www.assemblyai.com/docs/streaming/pii-redaction
Source: docs/streaming/pii-redaction.mdx
Navigation: Real-time STT > Features
Description: PII Redaction documentation.

Automatically detect and redact personally identifiable information from streaming transcripts in real time.

## Overview

Streaming PII Redaction lets you automatically detect and remove personally identifiable information from your streaming transcripts in real time. When enabled, the API redacts PII in **final turns only** before sending them to the client.

  **Final turns only**

  PII redaction only applies to final turns. When `redact_pii` is `true`,
  `include_partial_turns` defaults to `false` automatically so no unredacted
  text reaches the client. Only set `include_partial_turns` to `true` if you
  explicitly want partial (non-final) turns, which will contain unredacted PII
  alongside the redacted final turns.

When you enable PII redaction, your final turns will look like this:

- With `hash` substitution: `Hi, my name is ####!`
- With `entity_name` substitution: `Hi, my name is [PERSON_NAME]!`

  **Pre-recorded PII redaction**

  For PII redaction on pre-recorded audio, including generating redacted audio files, see [Redact PII from transcripts](/guardrails/redact-pii-from-transcripts).

## Connection parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `redact_pii` | boolean | Yes | `false` | Enable PII text redaction. Only applies to final turns. |
| `redact_pii_policies` | array | No | All | PII entity types to redact. Over the raw WebSocket, pass a JSON-encoded array of policy names (e.g. `["person_name","phone_number"]`). The SDKs accept a native list/array. If omitted and `redact_pii` is `true`, all detected PII is redacted. See [PII policies](/guardrails/redact-pii-from-transcripts#pii-policies) for the full list. |
| `redact_pii_sub` | string | No | `hash` | Replacement scheme. `hash` replaces PII with `#` characters, `entity_name` replaces with `[ENTITY_TYPE]`. |
| `include_partial_turns` | boolean | No | `false` when `redact_pii` is `true`, otherwise `true` | Whether to include partial (non-final) turns. Defaults to `false` automatically when PII redaction is enabled, so no unredacted text reaches the client. Set to `true` only if you explicitly want to receive partial turns, which will contain unredacted PII. |

## Quickstart

Enable PII redaction by setting `redact_pii` to `true` when you open the WebSocket. Optionally pass `redact_pii_policies` to limit which entity types are redacted, and `redact_pii_sub` to choose the replacement scheme.

```python
import json

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "redact_pii": True,
    "redact_pii_policies": json.dumps(["person_name", "phone_number", "email_address"]),
    "redact_pii_sub": "entity_name",
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        redact_pii=True,
        redact_pii_policies=["person_name", "phone_number", "email_address"],
        redact_pii_sub="entity_name",
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  redact_pii: true,
  redact_pii_policies: JSON.stringify([
    "person_name",
    "phone_number",
    "email_address",
  ]),
  redact_pii_sub: "entity_name",
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  redactPii: true,
  redactPiiPolicies: ["person_name", "phone_number", "email_address"],
  redactPiiSub: "entity_name",
});
```

### Example output

With `entity_name` substitution:

```plain
Hi, my name is [PERSON_NAME] and you can reach me at [PHONE_NUMBER] or [EMAIL_ADDRESS].
```

With `hash` substitution:

```plain
Hi, my name is #### and you can reach me at ###-###-#### or ####@#####.###.
```

## Supported PII policies

Streaming PII redaction supports the same policies as pre-recorded PII redaction, including `person_name`, `phone_number`, `email_address`, `credit_card_number`, `us_social_security_number`, `date_of_birth`, and more.

For the full list of available policies, see [PII policies](/guardrails/redact-pii-from-transcripts#pii-policies).

## Troubleshooting

  PII redaction only applies to **final turns**. If you're seeing PII, you
  likely set `include_partial_turns` to `true`, which returns unredacted
  partial turns alongside redacted finals. Remove that override (or set it to
  `false`) to only receive redacted final turns — this is the default when
  `redact_pii` is enabled.

  Audio redaction is not available for streaming. To generate a redacted audio
  file, use [pre-recorded PII redaction](/guardrails/redact-pii-from-transcripts#create-redacted-audio-files)
  with the `redact_pii_audio` parameter.

---

<a id="filter-profanity-docs-streaming-filter-profanity-from-transcripts-mdx"></a>

# Filter profanity

URL: https://www.assemblyai.com/docs/streaming/filter-profanity-from-transcripts
Source: docs/streaming/filter-profanity-from-transcripts.mdx
Navigation: Real-time STT > Features
Description: Filter profanity documentation.

Automatically filter profanity from streaming transcripts in real time.

## Overview

Streaming profanity filtering lets you automatically mask profane words in your streaming transcripts in real time. When enabled, the API replaces profane words with asterisks in both **partial** and **final** turns before sending them to the client.

The mask uses the first letter of the word followed by `n - 1` asterisks (for example, `shit` becomes `s***`). Apostrophes, capitalization, and surrounding punctuation are preserved (for example, `shit's` becomes `s***'s`).

Profanity filtering supports all streaming models: `universal-3-5-pro`, `universal-streaming-english`, and `universal-streaming-multilingual`. It also works alongside other features such as `format_turns` and PII redaction.

  **Pre-recorded profanity filtering**

  For profanity filtering on pre-recorded audio, see [Filter profanity from transcripts](/guardrails/filter-profanity-from-transcripts).

## Connection parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `filter_profanity` | boolean | No | `false` | Enable real-time profanity filtering. When `true`, profane words in both partial and final turns are masked with asterisks (first letter preserved). The server accepts the truthy strings `true`, `1`, and `yes`. Invalid values cause the WebSocket to close with code `3006`. |
| `include_partial_turns` | boolean | No | `true` | When `false`, the API only sends final turns. Useful with `filter_profanity: true` if you display partials directly to end-users and want to avoid any unmasked profanity flashing during word completion. |

## Quickstart

Set `filter_profanity` to `true` when you open the WebSocket.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "filter_profanity": True,
}
```

```python
client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        filter_profanity=True,
    )
)
```

```javascript
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  speech_model: "universal-3-5-pro",
  filter_profanity: true,
};
```

```javascript
const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
  filterProfanity: true,
});
```

  **Suppress unmasked partials with include_partial_turns=false**

  Profanity filtering applies to both partial and final turns, but during word-completion an unmasked partial can briefly appear before the model resolves the word and applies the mask. If your application surfaces partials directly to end-users (for example a live caption stream or voice-agent UI), set `include_partial_turns: false` on the connection to suppress all partial turns and only receive masked finals. The default is `true` (partials enabled), so this requires an explicit opt-out.

### Example output

With `filter_profanity=true`, a final turn might look like:

```plain
s*** is what you say when you stub your toe.
```

The mask preserves word length, apostrophes, and surrounding punctuation, so a word like `shit's` is returned as `s***'s` and `motherfucker` becomes `m***********`.

## Troubleshooting

  The streaming filter targets the same word list as pre-recorded profanity
  filtering and only masks words on that list. Some words you might consider
  profane, such as `crap` and `damn`, are intentionally not masked and pass
  through unchanged. If you need stricter filtering, apply your own
  post-processing on top of the masked transcript.

  Profanity masking applies during word classification, so an unmasked partial
  can briefly appear before the word is fully recognized and masked. If your
  UI surfaces partials directly to users, set `include_partial_turns: false`
  on the connection. Final turns are always masked.

---

<a id="message-sequence-docs-streaming-message-sequence-mdx"></a>

# Message Sequence

URL: https://www.assemblyai.com/docs/streaming/message-sequence
Source: docs/streaming/message-sequence.mdx
Navigation: Real-time STT > API reference
Description: Canonical reference for the streaming WebSocket protocol. Covers every client and server message and the order they appear in.

This page is the canonical reference for the streaming WebSocket protocol: every message the client sends, every message the server emits, and the order they appear in. For the field-level schema of each message, see the [Streaming API reference](/streaming/api-spec/streaming-websocket).

## Sequence diagram

```mermaid
sequenceDiagram
  participant C as Client
  participant S as Server
  C->>S: WebSocket connect (token, speech_model, sample_rate, ...)
  S->>C: Begin
  loop while streaming
    C->>S: binary audio frames (raw PCM, ~50 ms)
    S->>C: SpeechStarted
    S->>C: Turn (end_of_turn: false) xN (partials)
    S->>C: Turn (end_of_turn: true)
  end
  C->>S: Terminate
  S-->>C: in-flight Turn(s)
  S->>C: Termination
  S->>C: close (1000)
```

## Messages at a glance

**Client → server**

| Message | Frame type | Purpose |
| --- | --- | --- |
| Audio | **Binary** | Raw audio bytes (PCM, no JSON or base64 wrapper). Send ~50 ms chunks. |
| `UpdateConfiguration` | Text (JSON) | Change transcription settings mid-session. |
| `ForceEndpoint` | Text (JSON) | Immediately end the current turn. |
| `KeepAlive` | Text (JSON) | Reset the inactivity timer (only needed with `inactivity_timeout`). |
| `Terminate` | Text (JSON) | End the session. |

**Server → client**

| Message | When | Purpose |
| --- | --- | --- |
| `Begin` | Once, on connect | Session ID, expiration time, and the configuration the server actually applied. |
| `SpeechStarted` | Per turn (Universal-3.5 Pro only) | Speech detected; precedes the turn's first `Turn` message. |
| `Turn` | Repeatedly | Partial and final transcripts; see the walkthrough below. |
| `SpeakerRevision` | If `speaker_labels=true` | Revised speaker labels for earlier turns. |
| `Heartbeat` | Every 5s if `session_heartbeat=true` | Liveness ping with audio-received and real-time-factor stats. |
| `Termination` | Once, last message | Session totals; nothing follows it. |
| `Error` | On failure, before close | Error code and detail; the connection then closes. |

## Session initialization

When the session begins, you receive a `Begin` message with the session ID, expiration time, and the configuration the server applied to your session.

```json
{
  "type": "Begin",
  "id": "3207b601-2054-48df-ba77-8784dfcf9fb8",
  "expires_at": 1772570132,
  "configuration": {
    "model": "universal-3-5-pro",
    "mode": "balanced",
    "api_version": "2025-05-12"
  }
}
```

`expires_at` is a Unix timestamp (seconds). When it is reached, the server closes the session with error `3008`.

  Always check that `configuration.model` matches the `speech_model` you requested. Unrecognized or misspelled query parameters are ignored rather than rejected, so this echo is the fastest way to catch a typo (for example, `speechModel` instead of `speech_model`).

## Sending audio

Audio is sent as **binary WebSocket frames** containing audio bytes in the encoding and sample rate you set at connection time (default: 16 kHz, 16-bit mono PCM). Do not wrap audio in JSON and do not base64-encode it. A text frame that isn't valid JSON is rejected and closes the session.

Compressed audio is supported via Opus and AAC. Opus uses roughly 8× less bandwidth than 16 kHz PCM.

- `encoding=ogg_opus` — an Ogg-encapsulated Opus byte stream (the format produced by ffmpeg, gstreamer, opusenc, and browser MediaRecorder). Binary frames can be arbitrary chunks of the stream; no page or packet alignment is required.
- `encoding=opus` — raw Opus packets without a container. Each binary frame must contain exactly one Opus packet.
- `encoding=aac` — an AAC stream in ADTS framing. Raw AAC packets and MP4/M4A containers are not supported. Binary frames can be arbitrary chunks of the stream; the server buffers partial frames.

For AAC and both Opus encodings, `sample_rate` is ignored because each stream describes its own sample rate.

For PCM, send audio in chunks of roughly 50 ms (800 samples at 16 kHz); for Opus, one 20 ms packet (or page) per frame is typical. You can send audio faster than real time (for example when streaming from a file), but throughput is throttled at ~1.25× real time. If more than five minutes of audio is buffered ahead of processing, the session closes with error `3007`.

```python
import websocket

# audio_chunk is raw PCM bytes (e.g. 1600 bytes = 800 samples ≈ 50 ms at 16 kHz mono)
ws.send(audio_chunk, websocket.ABNF.OPCODE_BINARY)
```

```python
# audio_iterator yields raw PCM byte chunks; the SDK handles binary framing.
client.stream(audio_iterator)
```

```javascript
// audioChunk is a Buffer/Uint8Array of raw PCM bytes
ws.send(audioChunk);
```

```javascript
// Pipe any readable byte stream into transcriber.stream(); the SDK handles framing.
Readable.toWeb(audioSource).pipeTo(transcriber.stream());
```

## Speech started

Immediately before the **first `Turn` message of each turn**, the server sends a `SpeechStarted` message. The `timestamp` field is the start of the turn, in milliseconds relative to the beginning of the audio stream. The `confidence` field is the average word confidence of the initial transcript.

```json
{
  "type": "SpeechStarted",
  "timestamp": 1216,
  "confidence": 0.987654
}
```

  Universal Streaming skips `SpeechStarted` and goes straight to the first `Turn`.

## Partial transcript

As the speaker is talking, the server emits one or more `Turn` messages with `end_of_turn: false`. These are partial transcripts.

```json expandable
{
  "turn_order": 0,
  "turn_is_formatted": false,
  "end_of_turn": false,
  "transcript": "My name is",
  "end_of_turn_confidence": 0,
  "words": [
    {
      "start": 1216,
      "end": 1627,
      "text": "My",
      "confidence": 0.956314,
      "word_is_final": false
    },
    {
      "start": 1668,
      "end": 2490,
      "text": "name",
      "confidence": 0.999393,
      "word_is_final": false
    },
    {
      "start": 2531,
      "end": 3067,
      "text": "is",
      "confidence": 0.753325,
      "word_is_final": false
    }
  ],
  "utterance": "",
  "type": "Turn"
}
```

The cadence and shape of partials depends on the model. See [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio) and [Universal Streaming](/streaming/getting-started/transcribe-streaming-audio) for the details of how each model produces partials.

## End of turn

When the turn ends, the server emits a `Turn` message with `end_of_turn: true` and `turn_is_formatted: true`. This is the last message for this `turn_order` and the transcript is fully formatted.

```json expandable
{
  "turn_order": 0,
  "turn_is_formatted": true,
  "end_of_turn": true,
  "transcript": "My name is Sonny.",
  "end_of_turn_confidence": 1,
  "words": [
    {
      "start": 1216,
      "end": 1635,
      "text": "My",
      "confidence": 0.956583,
      "word_is_final": true
    },
    {
      "start": 1676,
      "end": 2515,
      "text": "name",
      "confidence": 0.999199,
      "word_is_final": true
    },
    {
      "start": 2556,
      "end": 2975,
      "text": "is",
      "confidence": 0.999535,
      "word_is_final": true
    },
    {
      "start": 3016,
      "end": 4155,
      "text": "Sonny.",
      "confidence": 0.316031,
      "word_is_final": true
    }
  ],
  "utterance": "My name is Sonny.",
  "type": "Turn"
}
```

  **Universal Streaming differs.** By default (`format_turns=false`), the final arrives with `turn_is_formatted: false`. With `format_turns=true`, you receive two `end_of_turn: true` messages for the same `turn_order`: the unformatted final first, then a formatted final right after. Treat a turn as complete only when both `end_of_turn` and `turn_is_formatted` are `true`, or you'll process every turn twice.

Default (`format_turns=false`) — single unformatted final:

```json expandable
{
  "turn_order": 0,
  "turn_is_formatted": false,
  "end_of_turn": true,
  "transcript": "my name is sonny",
  "end_of_turn_confidence": 0.812345,
  "words": [
    { "start": 1216, "end": 1635, "text": "my",    "confidence": 0.956583, "word_is_final": true },
    { "start": 1676, "end": 2515, "text": "name",  "confidence": 0.999199, "word_is_final": true },
    { "start": 2556, "end": 2975, "text": "is",    "confidence": 0.999535, "word_is_final": true },
    { "start": 3016, "end": 4155, "text": "sonny", "confidence": 0.316031, "word_is_final": true }
  ],
  "type": "Turn"
}
```

With `format_turns=true` — the unformatted final above, then a formatted final right after:

```json expandable
{
  "turn_order": 0,
  "turn_is_formatted": true,
  "end_of_turn": true,
  "transcript": "My name is Sonny.",
  "end_of_turn_confidence": 0.812345,
  "words": [
    { "start": 1216, "end": 1635, "text": "My",     "confidence": 0.956583, "word_is_final": true },
    { "start": 1676, "end": 2515, "text": "name",   "confidence": 0.999199, "word_is_final": true },
    { "start": 2556, "end": 2975, "text": "is",     "confidence": 0.999535, "word_is_final": true },
    { "start": 3016, "end": 4155, "text": "Sonny.", "confidence": 0.316031, "word_is_final": true }
  ],
  "type": "Turn"
}
```

## Forcing an endpoint

To end the current turn immediately, for example when your own VAD or a push-to-talk button decides the user is done, send a `ForceEndpoint` message:

```python
ws.send(json.dumps({"type": "ForceEndpoint"}))
```

```python
client.force_endpoint()
```

```javascript
ws.send(JSON.stringify({ type: "ForceEndpoint" }));
```

```javascript
transcriber.forceEndpoint();
```

The server responds with the `end_of_turn: true` message(s) for the current turn right away, without waiting for silence or punctuation.

## Updating configuration mid-session

You can change transcription settings at any point after `Begin` without reconnecting. Only the fields you include are changed:

```python
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "end_of_turn_confidence_threshold": 0.8,
    "min_end_of_turn_silence_when_confident": 400,
    "max_turn_silence": 1500,
    "keyterms_prompt": ["Keanu Reeves", "AssemblyAI"],
}))
```

```python
client.update_configuration(
    end_of_turn_confidence_threshold=0.8,
    min_end_of_turn_silence_when_confident=400,
    max_turn_silence=1500,
    keyterms_prompt=["Keanu Reeves", "AssemblyAI"],
)
```

```javascript
ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  end_of_turn_confidence_threshold: 0.8,
  min_end_of_turn_silence_when_confident: 400,
  max_turn_silence: 1500,
  keyterms_prompt: ["Keanu Reeves", "AssemblyAI"],
}));
```

```javascript
transcriber.updateConfiguration({
  endOfTurnConfidenceThreshold: 0.8,
  minEndOfTurnSilenceWhenConfident: 400,
  maxTurnSilence: 1500,
  keytermsPrompt: ["Keanu Reeves", "AssemblyAI"],
});
```

There is no acknowledgement message; the new settings apply to audio processed after the update. They do not retroactively affect the turn in progress. Some fields are model-specific (for example `mode`, `prompt`, and `agent_context` are Universal-3.5 Pro only). See [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream) for the full list.

## Keep alive

**`KeepAlive` messages are not required.** By default, sessions remain open until [explicitly terminated](#session-termination) or until the 3-hour maximum session duration is reached.

`KeepAlive` is only relevant if you have configured the `inactivity_timeout` connection parameter, which closes the session after a period of no audio or messages being sent. If you are using `inactivity_timeout` and want to keep the session open during periods where no audio is being sent, send a `KeepAlive` message to reset the inactivity timer:

```python
ws.send(json.dumps({"type": "KeepAlive"}))
```

```python
client.keep_alive()
```

```javascript
ws.send(JSON.stringify({ type: "KeepAlive" }));
```

```javascript
transcriber.keepAlive();
```

If the inactivity timeout elapses, the session closes with error `3006` and the message `Session terminated due to inactivity: No messages received for N seconds`.

  **Ordering guarantees.** `turn_order` is monotonically increasing, and all messages for a turn are delivered before the first message of the next turn. Within a turn, each `Turn` message supersedes the previous one. Render the latest `transcript`; do not append. A turn is complete on the message where both `end_of_turn` and `turn_is_formatted` are `true`.

## Heartbeat

When you set `session_heartbeat=true` at connection time, or toggle it on with `UpdateConfiguration`, the server sends a `Heartbeat` message every 5 seconds of wall-clock time for the life of the session, during both speech and silence. Use it to confirm the session is alive and to watch how the client's audio delivery compares to real time.

```json
{
  "type": "Heartbeat",
  "total_audio_received_ms": 45000,
  "total_duration_ms": 45205,
  "realtime_factor": 0.9964,
  "max_speech_probability": 0.999954
}
```

- `total_audio_received_ms` is the total audio received since the session started.
- `total_duration_ms` is the wall-clock time since the session started.
- `realtime_factor` is the ratio of audio received to wall-clock time over the window since the previous heartbeat. It sits around 1.0 when the client streams in real time and drops below 1.0 when the client is under-delivering audio.
- `max_speech_probability` is the maximum voice-activity speech probability observed since the previous heartbeat. It runs near 1.0 during speech and near 0.0 during silence.

## Session termination

To end a session, send a `Terminate` message:

```python
ws.send(json.dumps({"type": "Terminate"}))
```

```python
client.disconnect(terminate=True)
```

```javascript
ws.send(JSON.stringify({ type: "Terminate" }));
```

```javascript
await transcriber.close();
```

After you send `Terminate`, **keep reading from the WebSocket until you receive the `Termination` message**. The server first flushes any in-flight messages, which can include:

- the final (and formatted) `Turn` for audio you already sent;
- on Universal Streaming, a closing `Turn` with an empty `transcript` for an unfinished turn;
- a `SpeakerRevision` if `speaker_labels=true`. The end-of-session refinement adds approximately 400 ms of latency at session close. See [Revised speaker labels](/streaming/label-speakers-and-separate-channels#revised-speaker-labels) for the message schema and consumption guidance.

Closing the socket as soon as you send `Terminate` silently discards your last transcript.

The `Termination` message is always the last message:

```json
{
  "type": "Termination",
  "audio_duration_seconds": 13,
  "session_duration_seconds": 14
}
```

- `audio_duration_seconds` is the total audio processed in the session.
- `session_duration_seconds` is the total wall-clock time the connection was open. **This is the duration the session is billed on.**

After `Termination`, no further messages are sent and the server closes the connection with code `1000`.

  Always terminate sessions explicitly. Streaming is [billed per session](/billing-and-pricing#streaming-speech-to-text-billing). Sessions that are not terminated remain open and continue to accrue charges until the server auto-closes them after 3 hours (error code `3008`). See [Common errors](/streaming/common-session-errors-and-closures) for more details.

## Errors

If the session fails at any point, the server sends an `Error` message as a text frame and then closes the connection:

```json
{
  "type": "Error",
  "error_code": 3007,
  "error": "Audio transmission rate exceeded: too much audio buffered"
}
```

See [Common errors](/streaming/common-session-errors-and-closures) for the full list of error codes and how to handle reconnection.

---

<a id="streaming-websocket-api-docs-streaming-api-spec-streaming-websocket-mdx"></a>

# Streaming WebSocket API

URL: https://www.assemblyai.com/docs/streaming/api-spec/streaming-websocket
Source: docs/streaming/api-spec/streaming-websocket.mdx
Navigation: Real-time STT > API reference
Description: Full AsyncAPI reference for the Real-time STT WebSocket API.

Metadata:
- asyncapi: /api-reference/specs/streaming.yaml streaming

---

<a id="generate-streaming-token-docs-streaming-api-spec-generate-streaming-token-mdx"></a>

# Generate streaming token

URL: https://www.assemblyai.com/docs/streaming/api-spec/generate-streaming-token
Source: docs/streaming/api-spec/generate-streaming-token.mdx
Navigation: Real-time STT > API reference
Description: Generate a temporary token for Real-time STT.

Metadata:
- openapi: /api-reference/specs/streaming-token.yaml GET /v3/token

---

<a id="authenticate-with-a-temporary-token-docs-streaming-authenticate-with-a-temporary-token-mdx"></a>

# Authenticate with a temporary token

URL: https://www.assemblyai.com/docs/streaming/authenticate-with-a-temporary-token
Source: docs/streaming/authenticate-with-a-temporary-token.mdx
Navigation: Real-time STT > Advanced
Description: Authenticate with a temporary token documentation.

## Authenticate with a temporary token

If you need to authenticate on the client, you can avoid exposing your API key by using temporary authentication tokens.
You should generate this token on your server and pass it to the client.

View the endpoint reference.

To generate a temporary token, make a `POST` request to the temporary [token endpoint](/api-reference/streaming-api/generate-streaming-token).

Use the `expires_in_seconds` parameter to specify the duration for which the token will remain valid. Optionally, use the `max_session_duration_seconds` parameter to specify the desired maximum duration for the session initialized using this token.

```python
import requests
from urllib.parse import urlencode

def create_temporary_token():
    url = "https://streaming.assemblyai.com/v3/token"
    response = requests.get(
        f"{url}?{urlencode({'expires_in_seconds': 60})}",
        headers={"Authorization": "<YOUR_API_KEY>"},
    )
    data = response.json()
    return data.get("token")
```

To generate a temporary token, call `RealTimeTranscriber.create_temporary_token()`.

Use the `expires_in_seconds` parameter to specify the duration for which the token will remain valid. Optionally, use the `max_session_duration_seconds` parameter to specify the desired maximum duration for the session started using this token.

```python
from assemblyai.streaming.v3 import RealTimeTranscriber

def create_temporary_token():
    client = RealTimeTranscriber(api_key="<YOUR_API_KEY>")

    return client.create_temporary_token(expires_in_seconds=60)
```

To generate a temporary token, make a `POST` request to the temporary [token endpoint](/api-reference/streaming-api/generate-streaming-token).

Use the `expires_in_seconds` parameter to specify the duration for which the token will remain valid. Optionally, use the `max_session_duration_seconds` parameter to specify the desired maximum duration for the session started using this token.

```js
async function createTemporaryToken() {
  const url = new URL("https://streaming.assemblyai.com/v3/token");
  url.search = new URLSearchParams({
    expires_in_seconds: 60,
  }).toString();

  const response = await fetch(url, {
    headers: {
      Authorization: "<YOUR_API_KEY>",
    },
  });

  const data = await response.json();
  return data.token;
}
```

To generate a temporary token, call `client.streaming.createTemporaryToken()`.

Use the `expires_in_seconds` parameter to specify the duration for which the token will remain valid. Optionally, use the `max_session_duration_seconds` parameter to specify the desired maximum duration for the session started using this token.

```js
import { AssemblyAI } from "assemblyai";

async function createTemporaryToken() {
  const client = new AssemblyAI({ apiKey: "<YOUR_API_KEY>" });
  return client.streaming.createTemporaryToken({
    expires_in_seconds: 60,
  });
}
```

  `expires_in_seconds` must be a value between `1` and `600` seconds. If
  specified, `max_session_duration_seconds` must be a value between `60` and
  `10800` seconds (defaults to maximum session duration of 3 hours).

The client should retrieve the token from the server and use the token to authenticate the transcriber.

  Each token has a one-time use restriction and can only be used for a single
  session. Any usage associated with a temporary token will be attributed to the
  API key that generated it.

To use it, specify the `token` parameter as a query parameter in the WebSocket URL.

```python
params_w_token = {**CONNECTION_PARAMS, "speech_model": "universal-3-5-pro", "token": token}
ws_app = websocket.WebSocketApp(
    f'{API_ENDPOINT_BASE_URL}?{urlencode(params_w_token)}',
    on_open=on_open,
    on_message=on_message,
    on_error=on_error,
    on_close=on_close,
)
```

To use it, specify the `token` parameter when initializing the `RealTimeTranscriber`.

```python
from assemblyai.streaming.v3 import (
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
)

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(token=token)
)

client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
    )
)
```

To use it, specify the `token` parameter as a query parameter in the WebSocket URL.

```js
const token = await getToken(); // Implement getToken to retrieve token from server
const paramsWithToken = { ...CONNECTION_PARAMS, speech_model: "universal-3-5-pro", token };
ws = new WebSocket(
  `${API_ENDPOINT_BASE_URL}?${querystring.stringify(paramsWithToken)}`
);
```

To use it, specify the `token` parameter when initializing the `StreamingTranscriber`.

```js
import { StreamingTranscriber } from "assemblyai";

const token = await getToken(); // Implement getToken to retrieve token from server
const rt = new StreamingTranscriber({
  token,
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
});
```

---

<a id="webhooks-for-streaming-speech-to-text-docs-streaming-webhooks-mdx"></a>

# Webhooks for streaming speech-to-text

URL: https://www.assemblyai.com/docs/streaming/webhooks
Source: docs/streaming/webhooks.mdx
Navigation: Real-time STT > Advanced
Description: Get notified when a streaming transcription session ends.

Webhooks allow you to receive the complete transcript via HTTP callback when a streaming session ends. This is in addition to the real-time WebSocket responses you receive during the session, such as partial and finalized turns. These WebSocket messages are delivered continuously as audio is processed, while the webhook is sent once after the session terminates and contains only the finalized turns.

  This guide covers webhooks for [streaming audio
  transcription](/streaming). For webhooks with pre-recorded audio, see
  [Webhooks for pre-recorded audio](/pre-recorded-audio/webhooks).

## Configure webhooks for a streaming session

To use webhooks with streaming speech-to-text, add the following parameters to your WebSocket connection URL:

| Parameter                   | Required | Description                                                               |
| --------------------------- | -------- | ------------------------------------------------------------------------- |
| `webhook_url`               | Yes      | The URL to send the transcript to when the session ends.                  |
| `webhook_auth_header_name`  | No       | The name of the authentication header to include in the webhook request.  |
| `webhook_auth_header_value` | No       | The value of the authentication header to include in the webhook request. |

  **Don't have a webhook endpoint yet?**

  Create a test webhook endpoint with [webhook.site](https://webhook.site) to
  test your webhook integration.

### Example WebSocket URL with webhook parameters

Add the webhook parameters as query parameters to the WebSocket URL:

```
wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&webhook_url=https://example.com/webhook
```

To include authentication:

```
wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&webhook_url=https://example.com/webhook&webhook_auth_header_name=X-Webhook-Secret&webhook_auth_header_value=secret-value
```

```python expandable
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "<YOUR_API_KEY>"

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,
    # Webhook parameters
    "webhook_url": "https://example.com/webhook",
    "webhook_auth_header_name": "X-Webhook-Secret",  # Optional
    "webhook_auth_header_value": "secret-value",  # Optional
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    """Called when a message is received from the WebSocket."""
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        if msg_type == "Begin":
            session_id = data.get("id")
            expires_at = data.get("expires_at")
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get("transcript", "")
            if data.get('end_of_turn'):
                print("\r" + " " * 80 + "\r", end="")
                print(transcript)
            else:
                print(f"\r{transcript}", end="")
        elif msg_type == "Termination":
            audio_duration = data.get("audio_duration_seconds", 0)
            session_duration = data.get("session_duration_seconds", 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def run():
    global audio, stream, ws_app

    audio = pyaudio.PyAudio()

    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(2)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete.")

if __name__ == "__main__":
    run()
```

```python expandable
import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    TerminationEvent,
    TurnEvent,
)
from typing import Type

api_key = "<YOUR_API_KEY>"

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()

def on_begin(self: Type[RealTimeTranscriber], event: BeginEvent):
    print(f"Session started: {event}")
    print("Connected. Speak into your microphone; press Ctrl+C to stop.")

def on_turn(self: Type[RealTimeTranscriber], event: TurnEvent):
    print(f"{event.transcript} ({event.end_of_turn})")

def on_terminated(self: Type[RealTimeTranscriber], event: TerminationEvent):
    print(f"Session terminated: {event.audio_duration_seconds} seconds of audio processed")

def on_error(self: Type[RealTimeTranscriber], error: RealTimeError):
    print(f"Error occurred: {error}")

def main():
    client = RealTimeTranscriber(api_key=api_key)

    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    client.connect(
        RealTimeParameters(
            sample_rate=16000,
            format_turns=True,
            # Webhook parameters
            webhook_url="https://example.com/webhook",
            webhook_auth_header_name="X-Webhook-Secret",  # Optional
            webhook_auth_header_value="secret-value",  # Optional
        )
    )

    try:
        client.stream(microphone_stream())
    finally:
        client.disconnect(terminate=True)

if __name__ == "__main__":
    main()
```

```javascript expandable
const WebSocket = require("ws");

const API_KEY = "<YOUR_API_KEY>";

const connectionParams = new URLSearchParams({
  sample_rate: 16000,
  format_turns: true,
  // Webhook parameters
  webhook_url: "https://example.com/webhook",
  webhook_auth_header_name: "X-Webhook-Secret", // Optional
  webhook_auth_header_value: "secret-value", // Optional
});

const API_ENDPOINT = `wss://streaming.assemblyai.com/v3/ws?${connectionParams.toString()}`;

const ws = new WebSocket(API_ENDPOINT, {
  headers: {
    Authorization: API_KEY,
  },
});

ws.on("open", () => {
  console.log("WebSocket connection opened.");
  console.log(`Connected to: ${API_ENDPOINT}`);

  // Start streaming audio data here
  // For example, using a microphone input library
});

ws.on("message", (data) => {
  try {
    const message = JSON.parse(data);
    const msgType = message.type;

    if (msgType === "Begin") {
      const sessionId = message.id;
      const expiresAt = new Date(message.expires_at * 1000);
      console.log(`\nSession began: ID=${sessionId}, ExpiresAt=${expiresAt}`);
    } else if (msgType === "Turn") {
      const transcript = message.transcript || "";
      if (message.end_of_turn) {
        process.stdout.write("\r" + " ".repeat(80) + "\r");
        console.log(transcript);
      } else {
        process.stdout.write(`\r${transcript}`);
      }
    } else if (msgType === "Termination") {
      const audioDuration = message.audio_duration_seconds || 0;
      const sessionDuration = message.session_duration_seconds || 0;
      console.log(
        `\nSession Terminated: Audio Duration=${audioDuration}s, Session Duration=${sessionDuration}s`
      );
    }
  } catch (e) {
    console.error("Error handling message:", e);
  }
});

ws.on("error", (error) => {
  console.error("WebSocket Error:", error);
});

ws.on("close", (code, reason) => {
  console.log(`WebSocket Disconnected: Status=${code}, Msg=${reason}`);
});

// To gracefully close the session, send a Terminate message
function terminateSession() {
  if (ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: "Terminate" }));
  }
}
```

```javascript expandable
import { Readable } from "stream";
import { AssemblyAI } from "assemblyai";
import recorder from "node-record-lpcm16";

const run = async () => {
  const client = new AssemblyAI({
    apiKey: "<YOUR_API_KEY>",
  });

  const transcriber = client.streaming.transcriber({
    sampleRate: 16_000,
    formatTurns: true,
    // Webhook parameters
    webhookUrl: "https://example.com/webhook",
    webhookAuthHeaderName: "X-Webhook-Secret", // Optional
    webhookAuthHeaderValue: "secret-value", // Optional
  });

  transcriber.on("open", ({ id }) => {
    console.log(`Session opened with ID: ${id}`);
  });

  transcriber.on("error", (error) => {
    console.error("Error:", error);
  });

  transcriber.on("close", (code, reason) =>
    console.log("Session closed:", code, reason)
  );

  transcriber.on("turn", (turn) => {
    if (!turn.transcript) {
      return;
    }
    console.log("Turn:", turn.transcript);
  });

  try {
    console.log("Connecting to streaming transcript service");
    await transcriber.connect();

    console.log("Starting recording");
    const recording = recorder.record({
      channels: 1,
      sampleRate: 16_000,
      audioType: "wav",
    });

    Readable.toWeb(recording.stream()).pipeTo(transcriber.stream());

    process.on("SIGINT", async function () {
      console.log();
      console.log("Stopping recording");
      recording.stop();

      console.log("Closing streaming transcript connection");
      await transcriber.close();

      process.exit();
    });
  } catch (error) {
    console.error(error);
  }
};

run();
```

## Handle webhook deliveries

When the streaming session ends, AssemblyAI sends a `POST` HTTP request to the URL you specified. The webhook contains the complete transcript from the session.

Your webhook endpoint must return a 2xx HTTP status code within 10 seconds to indicate successful receipt. If a 2xx status is not received within 10 seconds, AssemblyAI will retry the webhook call up to a total of 10 attempts. If at any point your endpoint returns a 4xx status code, the webhook call is considered failed and will not be retried.

  **Static Webhook IP addresses**

AssemblyAI sends all webhook deliveries from fixed IP addresses:

| Region | IP Address     |
| ------ | -------------- |
| US     | `44.238.19.20` |
| EU     | `54.220.25.36` |

### Delivery payload

The webhook delivery payload contains the complete transcript from the streaming session as a JSON object. The payload includes the session ID and an array of messages containing all the transcript turns.

```json expandable
{
  "session_id": "273e79fd-99e9-4e1d-91da-90f56a132d01",
  "messages": [
    {
      "turn_order": 0,
      "turn_is_formatted": true,
      "end_of_turn": true,
      "transcript": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US Skylines from Maine to Maryland to Minnesota are gray and smoggy, and in some places the air.",
      "end_of_turn_confidence": 0.5005,
      "words": [
        {
          "start": 4880,
          "end": 5040,
          "text": "Smoke",
          "confidence": 0.76054,
          "word_is_final": true
        },
        {
          "start": 5280,
          "end": 5360,
          "text": "from",
          "confidence": 0.761065,
          "word_is_final": true
        }
      ],
      "utterance": "",
      "type": "Turn"
    }
  ]
}
```

| Key                                 | Type    | Description                                                    |
| ----------------------------------- | ------- | -------------------------------------------------------------- |
| `session_id`                        | string  | The unique identifier for the streaming session.               |
| `messages`                          | array   | An array of transcript turn objects from the session.          |
| `messages[].turn_order`             | integer | The order of the turn in the session (0-indexed).              |
| `messages[].turn_is_formatted`      | boolean | Whether the transcript has been formatted.                     |
| `messages[].end_of_turn`            | boolean | Whether this message represents the end of a turn.             |
| `messages[].transcript`             | string  | The transcribed text for this turn.                            |
| `messages[].end_of_turn_confidence` | number  | Confidence score for the end of turn detection.                |
| `messages[].words`                  | array   | Word-level details including timestamps and confidence scores. |
| `messages[].type`                   | string  | The message type, typically "Turn".                            |

## Authenticate webhook deliveries

To secure your webhook endpoint, you can include custom authentication headers in the webhook request. When configuring your streaming session, provide the `webhook_auth_header_name` and `webhook_auth_header_value` parameters.

AssemblyAI will include this header in the webhook request, allowing you to verify that the request came from AssemblyAI.

```
webhook_auth_header_name=X-Webhook-Secret&webhook_auth_header_value=secret-value
```

In your webhook receiver, verify the header value matches what you configured:

```python
auth_header = request.headers.get("X-Webhook-Secret")
if auth_header != "secret-value":
    return "Unauthorized", 401
```

## Best practices

When implementing webhooks for streaming speech-to-text, consider the following best practices:

1. **Always verify authentication**: If you configure an authentication header, always verify it in your webhook receiver to ensure requests are from AssemblyAI.

2. **Respond quickly**: Return a response from your webhook endpoint as quickly as possible. If you need to perform time-consuming processing, do it asynchronously after returning the response.

3. **Handle failures gracefully**: Your webhook endpoint should handle errors gracefully and return appropriate HTTP status codes.

4. **Use HTTPS**: Always use HTTPS for your webhook URL to ensure the transcript data is encrypted in transit.

5. **Log webhook deliveries**: Keep logs of webhook deliveries for debugging and auditing purposes.

---

<a id="updating-configuration-mid-stream-docs-streaming-updating-configuration-mid-stream-mdx"></a>

# Updating Configuration Mid-Stream

URL: https://www.assemblyai.com/docs/streaming/updating-configuration-mid-stream
Source: docs/streaming/updating-configuration-mid-stream.mdx
Navigation: Real-time STT > Advanced
Description: Update streaming session parameters without reconnecting using the UpdateConfiguration message.

Send an `UpdateConfiguration` message during an active streaming session to change session parameters without reconnecting. Every field is optional — include only the parameters you want to change.

```json
{
  "type": "UpdateConfiguration",
  "keyterms_prompt": ["account number", "routing number"],
  "max_turn_silence": 5000
}
```

## Supported fields

### Mode

| Field | Type | Models |
| --- | --- | --- |
| `mode` | `"min_latency"` \| `"balanced"` \| `"max_accuracy"` | U3 Pro |

Switch between accuracy/latency trade-off modes mid-session. See [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency).

### Turn detection

| Field | Type | Models |
| --- | --- | --- |
| `min_turn_silence` | int (ms) | U3 Pro, Universal Streaming |
| `max_turn_silence` | int (ms) | U3 Pro, Universal Streaming |
| `end_of_turn_confidence_threshold` | float (0–1) | Universal Streaming |
| `interruption_delay` | int (ms) | U3 Pro |

- `min_turn_silence` — Minimum silence in ms before an end-of-turn check fires. Lower values produce faster turn endings; higher values reduce entity splitting on numbers and proper nouns.
- `max_turn_silence` — Maximum silence in ms before forcing a turn to end, regardless of confidence. Increase for moments where you expect a longer pause (caller reading out a number, address, etc.).
- `end_of_turn_confidence_threshold` — Confidence threshold for ending a turn on Universal Streaming.
- `interruption_delay` — How quickly the first partial is emitted on U3 Pro. Lower values produce faster TTFT for aggressive barge-in; higher values produce more confident first partials. See [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio).

### VAD

| Field | Type | Models |
| --- | --- | --- |
| `vad_threshold` | float (0–1) | U3 Pro, Universal Streaming |

- `vad_threshold` — Confidence threshold for classifying audio frames as speech. Increase for noisy environments to reduce false speech detection.

### Prompting and context

| Field | Type | Models |
| --- | --- | --- |
| `prompt` | string | U3 Pro |
| `keyterms_prompt` | string[] | U3 Pro, Universal Streaming |
| `agent_context` | string | U3 Pro |
| `language_codes` | string[] | U3 Pro |

- `prompt` — Update the contextual prompt mid-stream as your application learns more about the conversation. See [Prompting guide](/streaming/prompting-and-keyterms).
- `keyterms_prompt` — Replace the current keyterms list. The most effective way to improve recognition accuracy mid-stream — dynamically swap the list as your voice agent moves between conversation stages. See [Keyterms prompting](/streaming/prompting-and-keyterms).
- `agent_context` — Pass your voice agent's most recent spoken reply so the model has it as context for the next user turn. See [Context carryover](/streaming/universal-3-5-pro/context-carryover#passing-your-agents-reply-as-context).
- `language_codes` — Update the set of languages transcription is steered toward; takes effect from the next turn. Pass an empty list to clear steering and return to native code-switching.

### Partials

| Field | Type | Models |
| --- | --- | --- |
| `continuous_partials` | bool | U3 Pro |

- `continuous_partials` — Toggle steady-cadence partial emission on or off mid-session.

## Example

Set the keyterms for a caller-identification stage of a voice agent flow:

```python
websocket.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["Kelly Byrne-Donoghue", "date of birth", "January", "February"],
}))
```

```python
client.update_configuration(
    keyterms_prompt=["Kelly Byrne-Donoghue", "date of birth", "January", "February"],
)
```

```javascript
ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  keyterms_prompt: ["Kelly Byrne-Donoghue", "date of birth", "January", "February"],
}));
```

```javascript
transcriber.updateConfiguration({
  keytermsPrompt: ["Kelly Byrne-Donoghue", "date of birth", "January", "February"],
});
```

## Behavior notes

- `UpdateConfiguration` is a delta. Fields you omit keep their current values.
- `mode`, `prompt`, `agent_context`, `interruption_delay`, and `continuous_partials` are U3 Pro only.

---

<a id="common-session-errors-and-closures-docs-streaming-common-session-errors-and-closures-mdx"></a>

# Common session errors and closures

URL: https://www.assemblyai.com/docs/streaming/common-session-errors-and-closures
Source: docs/streaming/common-session-errors-and-closures.mdx
Navigation: Real-time STT > Advanced
Description: Common session errors and closures documentation.

## Common session errors and closures

In WebSocket based connections, closures and errors represent different ways a connection can terminate.
A closure is a normal, expected termination initiated by either the client or the server, whereas errors are terminations resulting from an unexpected problem like network issues, protocol mismatches, timeouts, or server-side issues.
In the event of an error, the `on_error` callback is triggered just prior to `on_close`. **If an error is not encountered, then only `on_close` is called.**

When a session closes, the `on_close` callback receives a status code and reason detailing why the connection ended.
This information is useful when attempting to debug issues or handle certain closure scenarios programmatically.
The below table lists some of the common reasons for a session closure along with their corresponding codes and descriptions.

### The `Error` message

When a session ends because of an error (rather than a normal closure), the server sends an `Error` text frame immediately before it closes the WebSocket. The frame carries the close code and a human-readable message:

```json
{
  "type": "Error",
  "error_code": 3008,
  "error": "Session Expired: Maximum session duration exceeded"
}
```

| Field        | Type    | Description                                                                        |
| ------------ | ------- | ---------------------------------------------------------------------------------- |
| `type`       | string  | Always `"Error"`.                                                                  |
| `error_code` | integer | The error code. The WebSocket then closes with this same value as its close code.  |
| `error`      | string  | Human-readable description of what went wrong.                                     |

  The WebSocket close-frame reason is limited to 123 bytes. For longer errors, the close reason reads `See Error message for details` and the full text is delivered in the `Error` frame above. Read the `Error` message rather than relying on the close reason.

### Close codes

| Code   | Reason                                                                         | Description                                                                                                                                                                                                                                                     |
| ------ | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `3008` | Session Expired: Maximum session duration exceeded                             | Session exceeded 3 hour limit (or max session duration set by [temporary token](/streaming/authenticate-with-a-temporary-token)). You are billed for the full session duration. Always [terminate sessions](/streaming/getting-started/transcribe-streaming-audio) explicitly to control costs. |
| `3007` | Input duration violation: `<time>` ms. Expected between 50 and 1000 ms         | Audio chunk size less than 50ms or greater than 1000ms.                                                                                                                                                                                                         |
| `3006` | Invalid Message Type: `<message>`                                              | Unsupported [message type](/streaming/common-session-errors-and-closures).                                                                                                                                                                                 |
| `3006` | Invalid JSON: `<json>`                                                         | Message contains invalid JSON.                                                                                                                                                                                                                                  |
| `3006` | Invalid Message: `<message>`                                                   | Message is not valid (i.e. `'[]'`).                                                                                                                                                                                                                             |
| `3006` | Session terminated due to inactivity: No messages received for `<n>` seconds   | The [`inactivity_timeout`](/streaming/message-sequence#keep-alive) you configured elapsed with no audio or messages. Send a [`KeepAlive`](/streaming/message-sequence#keep-alive) message to reset the timer if you need to hold the session open during silence. |
| `3007` | Audio Transmission Rate Exceeded: Received `<time>` sec. audio in `<time>` sec | Audio sent faster than real-time.                                                                                                                                                                                                                               |
| `3005` | Session Cancelled: An error occurred                                           | Unknown server error.                                                                                                                                                                                                                                           |
| `3009` | Unauthorized Connection: Too many concurrent sessions                          | Streaming rate limit exceeded. For more on rate limits, see your [Account's Rate Limits](https://www.assemblyai.com/dashboard/home) and [how streaming rate limits work](/streaming/rate-limits). |
| `1008` | Unauthorized Connection: Missing Authorization header                          | Missing or invalid API token. Your API tokens can be found on the [API Keys page](https://www.assemblyai.com/dashboard/home) of your account dashboard.                                                                                                     |
| `1008` | Unauthorized Connection: `<reason>`                                            | Account related issue (insufficient account balance, account temporarily disabled, etc.).                                                                                                                                                                       |
| `1011` | Internal error                                                                 | An unexpected server-side error occurred while establishing the connection (for example, during authentication). Retry the connection; if the problem persists, contact support@assemblyai.com. |
| `410`  | Deprecated endpoint                                                            | The V2 streaming API has been retired. Use this endpoint instead: [`wss://streaming.assemblyai.com/v3/ws`](/streaming/getting-started/transcribe-streaming-audio). |

  Code `3005` is still used as a catch-all for server-side errors not covered by
  the more specific codes above.

  **Handling closed sessions**

  A common way to handle a closure such as `3008 - Session Expired: Maximum session duration exceeded` is to parse the status code and reason in the `on_close` callback. If a specific code and reason are detected, you can then take appropriate action, such as opening a new session or logging useful debugging information.

Note that the `on_error` callback is not triggered in this case, as the session closes for a known reason and not due to encountering an error.

If you believe your session received an error or closed due to a reason not listed above, please reach out to support@assemblyai.com with the [session id](/streaming/api-spec/streaming-websocket#receive.receiveSessionBegins.id) and any further details.

---

<a id="benchmarks-docs-streaming-benchmarks-mdx"></a>

# Benchmarks

URL: https://www.assemblyai.com/docs/streaming/benchmarks
Source: docs/streaming/benchmarks.mdx
Navigation: Real-time STT > Advanced
Description: Industry-leading accuracy for streaming speech-to-text.

Benchmarks are an important first step before running your own [evaluation](/streaming/evaluations/voice-agents). Below are the current benchmarks for our streaming models so you can assess performance across accuracy, latency, and error rates.

  Public benchmarks can be misleading due to overfitting and benchmark gaming.
  We strongly recommend [running your own evaluation](/streaming/evaluations/voice-agents) on your
  audio data to identify the best model for your use case.

For the full interactive benchmark experience with competitive comparisons, visit [assemblyai.com/benchmarks](https://www.assemblyai.com/benchmarks).

## Word error rate (WER)

Word Error Rate (WER) is the classical metric for speech-to-text accuracy. It counts substitutions, deletions, and insertions against a reference transcript, divided by the total word count in the ground truth.

WER weights every word equally, so a misrecognized filler word counts the same as a misrecognized email or phone number. For voice agents, we recommend pairing WER with [Missed entity rate](#missed-entity-rate), which measures accuracy on the high-stakes entities — names, emails, phone numbers, and medical terms — that break agent conversion in production.

### English benchmarks

Most recent update: **March 2026**.

| Dataset                                                                                    | Universal-3.5 Pro WER (%)            | Universal Streaming WER (%)        | Relative gain vs Universal Streaming   |
| ------------------------------------------------------------------------------------------ | ---------------------------------- | ---------------------------------- | -------------------------------------- |
| **Overall Performance**                                                                    | **Mean: 6.3%** \| **Median: 6.1%** | **Mean: 8.6%** \| **Median: 7.8%** | **Mean: 26.7%** \| **Median: 21.8%**   |
| [commonvoice](https://github.com/common-voice/cv-dataset)                                  | 6.11%                              | 11.81%                             | 48.3%                                |
| [earnings21](https://huggingface.co/datasets/distil-whisper/earnings21)                    | 9.25%                              | 12.37%                             | 25.2%                                |
| [librispeech_test_clean](https://huggingface.co/datasets/AudioLLMs/librispeech_test_clean) | 1.78%                              | 2.71%                              | 34.3%                                |
| [librispeech_test_other](https://huggingface.co/datasets/openslr/librispeech_asr)          | 3.11%                              | 5.82%                              | 46.6%                                |
| [meanwhile](https://huggingface.co/datasets/distil-whisper/meanwhile)                      | 5.74%                              | 6.73%                              | 14.7%                                |
| [tedlium](https://huggingface.co/datasets/lmms-lab/tedlium)                                | 7.50%                              | 7.81%                              | 4.0%                                 |
| [rev16](https://huggingface.co/datasets/distil-whisper/rev16)                              | 10.86%                             | 12.99%                             | 16.4%                                |

### Multilingual benchmarks

Most recent update: **March 2026**.

| Language Code | Language   | Universal-3.5 Pro WER (%) | Universal Streaming WER (%) | Relative gain vs Universal Streaming |
| ------------- | ---------- | ----------------------- | --------------------------- | ------------------------------------ |
| **Average**   | **All**    | **8.49%**               | **11.74%**                  | **27.7%**                            |
| de            | German     | 11.79%                  | 13.99%                      | 15.7%                                |
| en            | English    | 8.43%                   | 12.94%                      | 34.9%                                |
| es            | Spanish    | 7.63%                   | 9.81%                       | 22.2%                                |
| fr            | French     | 9.59%                   | 16.53%                      | 42.0%                                |
| it            | Italian    | 5.60%                   | 7.36%                       | 23.9%                                |
| pt            | Portuguese | 7.88%                   | 9.83%                       | 19.8%                                |

## Missed entity rate

For voice agents, the words that break conversion are entities — emails, phone numbers, organizations, names, and medical terms. The Missed Entity Rate (MER) measures how often a model fails to correctly transcribe these high-stakes terms. See [Missed Entity Rate](/pre-recorded-audio/evaluations#missed-entity-rate-mer) for the full definition.

Universal-3.5 Pro Streaming delivers double-digit relative gains over Universal Streaming across most categories, with the biggest improvements on emails, locations, and medical terms.

| Entity type        | Universal-3.5 Pro MER (%) | Universal Streaming MER (%) | Relative gain vs Universal Streaming |
| ------------------ | ----------------------- | --------------------------- | ------------------------------------ |
| Job titles         | 8.74%                   | 10.13%                      | 13.7%                                |
| Dates and times    | 8.30%                   | 9.91%                       | 16.2%                                |
| Locations          | 9.22%                   | 12.99%                      | 29.0%                                |
| Medical terms      | 14.78%                  | 19.61%                      | 24.6%                                |
| Organization names | 17.06%                  | 21.41%                      | 20.3%                                |
| Phone numbers      | 34.79%                  | 37.11%                      | 6.3%                                 |
| Email addresses    | 59.64%                  | 89.09%                      | 33.1%                                |

Source: [AssemblyAI Streaming product page](https://www.assemblyai.com/products/streaming-speech-to-text).

## Latency

Most recent update: **May 2026**.

These figures are measured using in-domain voice agent conversation data from our internal evaluation set. In production, actual latency will vary depending on audio characteristics, silence duration, and turn detection configuration. Latency claims referenced elsewhere in our documentation may reflect different evaluation conditions or measurement approaches.

If you have questions about benchmarking methodology or need help interpreting these numbers for your use case, reach out to our team at [support@assemblyai.com](mailto:support@assemblyai.com).

### Emission latency

Time from when a word is spoken to when that word is returned by the API. Best metric for use cases that consume partial transcripts in real time.

| Model                            | P50 (ms) | P90 (ms) |
| -------------------------------- | -------- | -------- |
| Universal Streaming English      | 317      | 597      |
| Universal Streaming Multilingual | 303      | 535      |
| Universal-3.5 Pro Streaming        | N/A      | N/A      |

Universal-3.5 Pro Streaming emits segment-level partials rather than word-by-word, so the standard emission-latency metric doesn't apply.

### Time to Complete Turn (TTCT)

Time from end of speaker turn to receipt of the finalized transcript. Best metric for voice agent responsiveness.

| Model                       | P50 (ms) | P90 (ms) |
| --------------------------- | -------- | -------- |
| Universal Streaming English | 649      | 990      |
| Universal-3.5 Pro Streaming   | 568      | 829      |

  Time to First Token (TTFT) is a useful metric for LLMs but is not well suited to measuring real-time STT performance. For real-time STT, use emission latency (for partial transcript speed) and TTCT (for voice agent responsiveness).

## Latency gaming

In streaming, speed is critical. To achieve lower TTFT (time to first token) metrics, some providers emit tokens before any audio is actually spoken. These early tokens are hallucinations designed to game the benchmark, making TTFT a misleading measure of actual latency.

## External benchmarks

For third-party streaming benchmarks, we recommend the [Coval Speech-to-Text Playground](https://benchmarks.coval.ai/?tab=stt).

## Methodology

Our benchmarks are evaluated across **250+ hours of audio data**, **80,000+ audio files**, and **26 datasets**. We apply standard text normalization before calculating metrics. For full details on our methodology, visit [assemblyai.com/benchmarks](https://www.assemblyai.com/benchmarks).

## Run your own benchmark

We'd be happy to help. AssemblyAI has a benchmarking tool to help you run a custom evaluation against your real audio files. [Contact us](https://www.assemblyai.com/contact/support) for more information.

You can also run your own benchmarks following the [Hugging Face framework](https://github.com/huggingface/open_asr_leaderboard) which provides a GitHub repo with full instructions.

---

<a id="evaluating-real-time-stt-models-for-voice-agents-docs-streaming-evaluations-voice-agents-mdx"></a>

# Evaluating Real-time STT models for Voice Agents

URL: https://www.assemblyai.com/docs/streaming/evaluations/voice-agents
Source: docs/streaming/evaluations/voice-agents.mdx
Navigation: Real-time STT > Advanced
Description: Evaluating Real-time STT models for Voice Agents documentation.

## Introduction

The high level objective of a real-time STT model evaluation is to answer the question: _Which streaming Speech-to-text model is the best for my voice agent?_

For voice agents, the STT component is the "ears" of the system — transcription errors propagate into the LLM and response logic, so even small accuracy gaps compound in impact. At scale, these differences have real financial consequences: a 10% difference in entity accuracy (for example, credit card numbers) across millions of calls can translate to significant revenue loss. A rigorous evaluation ensures you choose the model that drives the best outcomes.

This guide will provide a step-by-step framework for evaluating and benchmarking streaming Speech-to-text models to help you select the best fit for your voice agent.

  Need help evaluating our Speech-to-text products? [Contact our Sales
  team](https://www.assemblyai.com/contact/sales) to request for an evaluation.

## Common evaluation metrics

### Time to First Token (TTFT) / Time to First Byte (TTFB)

$$
TTFT = t_{first\_token} - t_{stream\_start}
$$

This measures the time from when the audio stream begins (including model startup/initialization) to when the very first token is returned by the model.

  TTFT is borrowed from LLM evaluation and is less informative for real-time STT than other metrics. Because some providers emit tokens before any audio is actually spoken to game this metric, TTFT can be misleading. For real-time STT, prefer emission latency (the time from when a word is spoken to when that word is returned, best for use cases that consume partial transcriptions in real time) or [TTCT](#time-to-complete-transcript-ttct--transcription-delay) (for voice agents that act on finalized turns).

  For Universal-3.5 Pro Streaming, the `interruption_delay` parameter directly controls TTFT by configuring how soon the first partial transcript is emitted. Lower values (e.g. `0`) produce faster TTFT (~300ms effective) at the cost of less audio context; higher values (e.g. `500`–`1000`) are slower but more confident. See [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio) for configuration details.

### Time to Complete Transcript (TTCT) / Transcription Delay

$$
TTCT = t_{final\_token} - t_{utterance\_end}
$$

This measures the latency between when a user finishes speaking (end of speech detected) and when the complete transcription for that utterance is received from the STT model. This metric is crucial for understanding overall streaming model latency performance.

  TTCT is measured on a per utterance basis, and a single user turn may contain
  multiple utterances. This metric is crucial for minimizing overall voice agent
  latency since it represents how soon you'll be able to send STT outputs
  downstream to the LLM.

### End-of-Turn Finalization Latency / Endpointing Latency

$$
Latency = t_{turn\_boundary} - t_{speech\_end}
$$

This measures the time from when the user actually finishes speaking (end of speech detected) to when the system recognizes and signals the end of their conversational turn. This includes both speech detection latency and any additional processing to determine turn completion.

  End-of-Turn Finalization Latency should measured from the actual environment
  the user will use when the turn ends. AssemblyAI provides native endpointing
  in our models, but you can also use our outputs with other providers and their
  End-of-Turn models like
  [LiveKit](/voice-agents/livekit-universal-3-5-pro),
  [Pipecat](/voice-agents/pipecat-universal-3-5-pro),
  or [Vapi](https://vapi.ai).

### End-of-Turn Detection Accuracy

$$
EOT Accuracy = \frac{TP_{eos}}{TP_{eos} + FP_{eos} + FN_{eos}}
$$

This measures how accurately the model detects when a user has finished speaking, considering true positives (correct detections), false positives (premature cutoffs), and false negatives (missed endpoints).

### Word Error Rate (WER)

$$
WER = \frac{S + D + I}{N}
$$

This formula takes the number of Substitutions (S), Deletions (D), and Insertions (I), and divides their sum by the Total Number of Words in the ground truth transcript (N).

  For streaming models, it's important to measure both partial WER (accuracy of
  interim results) and final WER (accuracy after all corrections). The delta
  between these indicates the model's self-correction capability. While WER
  calculation may seem simple, it requires a methodical granular approach and
  reliable reference data.

### Semantic WER

Traditional WER treats every difference between the model output and a reference transcript as an error—even when the difference is semantically equivalent. Semantic WER corrects this by normalizing equivalent words and phrases before calculating WER, so that differences like `dr.` vs `doctor` or `1300` vs `thirteen hundred` aren't counted as errors.

#### Rule-based normalization

At its simplest, Semantic WER is a preprocessing step. Before running standard WER, apply find-and-replace rules to both the reference and hypothesis transcripts:

- **Number formats**: `1300` → `thirteen hundred`, `$5` → `five dollars`
- **Abbreviations and titles**: `dr.` → `doctor`, `mr.` → `mister`, `govt` → `government`
- **Contractions**: `gonna` → `going to`, `can't` → `cannot`
- **Variant spellings**: `grey` → `gray`, `cancelled` → `canceled`
- **Filler words**: Remove `um`, `uh`, `you know` from both sides (or keep both—just be consistent)

This alone eliminates a significant portion of false errors and can be implemented in a few lines of Python. No model inference required.

#### LLM-based scoring

For cases where simple rules can't capture the nuance—was an omission meaningful? Is a proper noun misspelling close enough?—an LLM can perform word-level alignment and classify each difference by severity:

- **No penalty**: Semantically equivalent forms (number formats, contractions, variant spellings)
- **Minor penalty**: Single-character misspellings, minor grammatical markers
- **Major penalty**: Incorrect substitutions, meaning-altering errors, significant omissions or additions of content words

These approaches are particularly valuable for voice agents because traditional WER penalizes transcription variants that would never trip up an LLM. If a user says "I'd like thirteen hundred units" and the model transcribes "1300 units", traditional WER counts that as an error — but the LLM processes both identically.

For implementations of Semantic WER, see [prompt-seeker](https://github.com/AssemblyAI-Solutions/prompt-seeker) and [aai-cli](https://github.com/alexkroman/aai-cli) in the [pre-recorded evaluation guide](/pre-recorded-audio/evaluations#open-source-tools).

### LASER score (LLM-based ASR Evaluation Rubric)

LASER is a published LLM-based evaluation metric ([Parulekar & Jyothi, EMNLP 2025](https://aclanthology.org/2025.emnlp-main.1257/)) that uses an LLM prompt with detailed examples to classify ASR errors and compute a score:

$$
\text{LASER} = 1 - \frac{\text{Total Penalty}}{\text{Reference Word Count}}
$$

The LLM aligns each word in the ASR output against the reference transcription and assigns a penalty per word pair:

- **No penalty (0)**: Acceptable variations including numerical format differences, abbreviations, compound word splits, transliterations, alternate spellings, proper noun variants, and colloquial terms
- **Minor penalty (0.5)**: Small spelling errors (single character) or minor grammatical errors (gender, tense, number markers) that preserve sentence meaning
- **Major penalty (1.0)**: Incorrect word substitutions, significant omissions or additions, and reordering that changes meaning

LASER provides structured per-error feedback alongside the score, making it useful for understanding _why_ a transcript scored poorly, not just how much error there was. For an implementation of LASER scoring, see [aai-cli](https://github.com/alexkroman/aai-cli).

  Accuracy metrics like WER, Semantic WER, and LASER work the same way for streaming as they do for pre-recorded audio. In both cases, you're comparing a final transcript against ground truth. The difference is only in how the transcript was generated (real-time vs batch), not in how you evaluate it. For detailed evaluation methodology and tooling, see the [pre-recorded evaluation guide](/pre-recorded-audio/evaluations).

### Ground truth quality

The quality of your ground truth data directly affects the reliability of your evaluation. Modern STT models frequently outperform human transcribers, which means errors in your ground truth files will show up as false negatives in your metrics.

Common issues with ground truth data:

- **Missing filler words**: Human transcribers often omit `um`, `uh`, `like`, and other disfluencies
- **Incorrect proper nouns**: Rare names, technical terms, and domain vocabulary are often misspelled
- **Simplified speech patterns**: Human transcribers tend to "clean up" speech, missing repetitions, false starts, and self-corrections
- **Code-switching errors**: Multilingual segments are frequently translated to English rather than transcribed as spoken

Before running evaluations, audit a sample of your ground truth files by listening to the audio and comparing. To inspect and correct issues in your ground truth files, use the [Truth File Corrector](https://www.assemblyai.com/dashboard/home) in the AssemblyAI Dashboard (found at the bottom of the left sidebar), which lets you listen back to audio and fix human transcription errors by clicking through differences.

  WER is only as good as your ground truth labels. Human transcriptions contain systematic errors — missed filler words, incorrect proper nouns, simplified speech patterns, and translated code-switching. When your STT model transcribes audio more accurately than the human label, those improvements show up as WER _errors_. Before reporting WER, manually audit at least 20 insertions to determine what percentage are true errors versus ground truth omissions.

  If you see an unexpected spike in insertion rates — especially when evaluating a newer or higher-accuracy model — audit your ground truth before concluding the model is worse. High insertion counts often indicate the model is correctly transcribing words that the human transcriber missed.

## The evaluation process

This section will be the core of the documentation, providing a step-by-step guide on how to run an evaluation.

For that reason, the evaluation process should closely match your production environment - including the streaming conditions you intend to transcribe, the model you intend to use, and the settings applied to those models.

### Step 1: Set up your benchmarking environment

When benchmarking a voice agent, you can decide to benchmark the audio directly against the provider's API and/or set up a live testing environment.

Benchmarks with real files against the API are best for measuring overall accuracy and latency metrics, like WER and TTFB. These will give you a good idea of the model's performance. See our section on [Pre-recorded evaluation benchmarks](/pre-recorded-audio/evaluations#the-evaluation-process) for how to do this part.

Voice agents are complex and other metrics like TTCT and End-of-Turn Finalization Latency often depend on additional factors like the end user's audio device and environment. We highly recommend you run live side by side evals with your voice agent hooked up to multiple real-time STT providers to experience what your user will feel for themselves.

### Step 2: Run your test scenarios

If you have out of the box test scenarios, you can run these through the API and capture the metrics above.

If you don't have scenarios, you can make these up based on expected customer behaviors and measure the side by side results across different providers. For example, if you are building a drive-through ordering system, create simulated test scenarios to represent different user orders, pacing, tonality, background noise, accents, etc.

If you are unsure how to proceed here, it might be worth checking out [BlueJay](https://getbluejay.ai/), [Coval](https://www.coval.dev/), [Hamming](https://hamming.ai/), who all help with evaluating and measuring performance of voice agents.

### Step 3: Compare the results

It is highly unlikely you will find a single real-time STT model that wins in all of the metrics outlined above. Ultimately your goal should be to compare for your use case which of these metrics helps your agent drive the best end user outcome.

For this case, you might consider:

- Are you replacing humans with your voice agent? It is likely TTCT and Endpointing Latency matter most since these metrics best simulate human behavior.
- Are you working with domain specific words like medical? While WER is important, it's most important the LLM in your voice agent understands the user. This requires simulating full test scenarios outside of just metrics.
- Are you showing transcript text to your end users (like subtitles)? Perhaps WER is most important to end user perception of quality and accuracy.

See [this article](https://www.assemblyai.com/blog/new-word-error-rate-wer-benchmark) to learn more about why your word error rate (WER) benchmark might be lying to you.

## Vibes vs metrics

While metrics provide a useful quantitative evaluation of a streaming Speech-to-text model, voice agents are complex and are not made up solely of STT models. For this, we recommend doing a "vibe-eval".

### Why do a vibe-eval?

Vibe-evals are useful to determine the qualitative difference between STT providers that affect the end outcome of your voice agent. For example, how errors in transcript may or may not trip up a voice agent as the LLM may fix some of the issues.

Vibe-evals are also good for tie-breaking instances where the benchmarking metrics don't lean in favour of one model over the other.

Another benefit of doing a vibe-evals is that truth files don't have to be sourced for them since Speech-to-text models are being compared against each other in a real voice agent.

### How to do a vibe-eval?

**Side-by-side comparison:** Compare formatted transcription outputs from different STT providers. Tools like [Diffchecker](https://www.diffchecker.com/) or any side-by-side interface work well for this. Since you're comparing models against each other, ground truth files aren't required.

**LLM as a judge:** An LLM can automatically identify differences between two transcriptions and pick a winner. However, be cautious: an LLM judge can be misled by outputs that look correct but contain subtle errors (such as translated code-switching segments that read well in English but don't reflect what was actually spoken). Always pair LLM-based judgments with spot-checking against the actual audio. For a structured LLM-based scoring approach, see [LASER](#laser-score-llm-based-asr-evaluation-rubric) above and [aai-cli](https://github.com/alexkroman/aai-cli) for an implementation.

**Live A/B testing:** Set up your voice agent with different STT providers using integrations like [LiveKit](/voice-agents/livekit-universal-3-5-pro) where you can swap out providers. Serve transcripts from different providers and collect feedback — ask users to score agent interactions directly, or track indirect signals like support ticket complaints about the agent misunderstanding users.

## Conclusion

We hope that this short guide was helpful in shaping your evaluation methodology.

Have more questions about evaluating our Speech-to-text models? [Contact our sales team](https://www.assemblyai.com/contact/sales) and we can help.

---

<a id="streaming-endpoints-and-data-zones-docs-streaming-endpoints-and-data-zones-mdx"></a>

# Streaming Endpoints and Data Zones

URL: https://www.assemblyai.com/docs/streaming/endpoints-and-data-zones
Source: docs/streaming/endpoints-and-data-zones.mdx
Navigation: Real-time STT > Advanced
Description: Streaming Endpoints and Data Zones documentation.

Choose the endpoint that best fits your application's requirements—whether that's achieving the lowest possible latency or ensuring your audio data stays within a specific geographic region.

## Edge Routing

The default endpoint (`streaming.assemblyai.com`) automatically routes requests to the nearest available region, minimizing latency for real-time transcription. With infrastructure in Oregon, Virginia, and Ireland, this endpoint delivers best-in-class streaming performance regardless of where your users are located.

## Data Zone Routing

Data Zone endpoints guarantee your data never leaves the specified region. This is designed for organizations with strict data residency and governance requirements—your audio and transcription data will remain entirely within the US or EU, respectively.

## Endpoints

| Endpoint               | WebSocket URL                             | Description                    |
| ---------------------- | ----------------------------------------- | ------------------------------ |
| Edge Routing (default) | `wss://streaming.assemblyai.com/v3/ws`    | Lowest latency, nearest region |
| US data residency      | `wss://streaming.us.assemblyai.com/v3/ws` | Data stays in the US           |
| EU data residency      | `wss://streaming.eu.assemblyai.com/v3/ws` | Data stays in the EU           |

## Which endpoint should I use?

- **Optimizing for latency?** Use **Edge Routing** (default). No configuration change is needed — it automatically routes to the nearest region for the fastest response.
- **Need data residency?** Use a **Data Zone** endpoint. Choose US or EU to ensure your audio and transcription data stays within that region.

## How to use it

Update your WebSocket connection URL to your preferred endpoint. Select an endpoint tab below to see examples for each.

Edge Routing is the default. If you're using the SDKs without overriding the base URL, you're already using this endpoint. No configuration change is required.

```python title="Python SDK"  expandable
import pyaudio
from assemblyai.streaming.v3 import (
    RealTimeParameters,
    RealTimeTranscriber,
)

api_key = "<YOUR_API_KEY>"

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()

# No api_host override needed — Edge Routing is the default
client = RealTimeTranscriber(api_key=api_key)

client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
    )
)
client.stream(microphone_stream())
```

```python title="Python" highlight={6} 
import websocket
from urllib.parse import urlencode

YOUR_API_KEY = "<YOUR_API_KEY>"
CONNECTION_PARAMS = {"sample_rate": 16000, "speech_model": "universal-3-5-pro"}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

ws = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": YOUR_API_KEY},
)

ws.run_forever()
```

```javascript title="JavaScript SDK" 
import { AssemblyAI } from "assemblyai";

// No streamingBaseUrl override needed — Edge Routing is the default
const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
});

await transcriber.connect();
```

```javascript title="JavaScript" highlight={6} 
const WebSocket = require("ws");
const querystring = require("querystring");

const YOUR_API_KEY = "<YOUR_API_KEY>";
const CONNECTION_PARAMS = { sample_rate: 16000, speech_model: "universal-3-5-pro" };
const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

const ws = new WebSocket(API_ENDPOINT, {
  headers: {
    Authorization: YOUR_API_KEY,
  },
});
```

Use the US Data Zone endpoint to keep your audio and transcription data within the United States.

```python title="Python SDK" highlight={31}  expandable
import pyaudio
from assemblyai.streaming.v3 import (
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
)

api_key = "<YOUR_API_KEY>"

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(api_host="streaming.us.assemblyai.com"),
    api_key=api_key,
)

client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
    )
)
client.stream(microphone_stream())
```

```python title="Python" highlight={6} 
import websocket
from urllib.parse import urlencode

YOUR_API_KEY = "<YOUR_API_KEY>"
CONNECTION_PARAMS = {"sample_rate": 16000, "speech_model": "universal-3-5-pro"}
API_ENDPOINT_BASE_URL = "wss://streaming.us.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

ws = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": YOUR_API_KEY},
)

ws.run_forever()
```

```javascript title="JavaScript SDK" highlight={5} 
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
  streamingBaseUrl: "wss://streaming.us.assemblyai.com",
});

const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
});

await transcriber.connect();
```

```javascript title="JavaScript" highlight={6} 
const WebSocket = require("ws");
const querystring = require("querystring");

const YOUR_API_KEY = "<YOUR_API_KEY>";
const CONNECTION_PARAMS = { sample_rate: 16000, speech_model: "universal-3-5-pro" };
const API_ENDPOINT_BASE_URL = "wss://streaming.us.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

const ws = new WebSocket(API_ENDPOINT, {
  headers: {
    Authorization: YOUR_API_KEY,
  },
});
```

Use the EU Data Zone endpoint to keep your audio and transcription data within the European Union.

```python title="Python SDK" highlight={31}  expandable
import pyaudio
from assemblyai.streaming.v3 import (
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
)

api_key = "<YOUR_API_KEY>"

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(api_host="streaming.eu.assemblyai.com"),
    api_key=api_key,
)

client.connect(
    RealTimeParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
    )
)
client.stream(microphone_stream())
```

```python title="Python" highlight={6} 
import websocket
from urllib.parse import urlencode

YOUR_API_KEY = "<YOUR_API_KEY>"
CONNECTION_PARAMS = {"sample_rate": 16000, "speech_model": "universal-3-5-pro"}
API_ENDPOINT_BASE_URL = "wss://streaming.eu.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

ws = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": YOUR_API_KEY},
)

ws.run_forever()
```

```javascript title="JavaScript SDK" highlight={5} 
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
  streamingBaseUrl: "wss://streaming.eu.assemblyai.com",
});

const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  speechModel: "universal-3-5-pro",
});

await transcriber.connect();
```

```javascript title="JavaScript" highlight={6} 
const WebSocket = require("ws");
const querystring = require("querystring");

const YOUR_API_KEY = "<YOUR_API_KEY>";
const CONNECTION_PARAMS = { sample_rate: 16000, speech_model: "universal-3-5-pro" };
const API_ENDPOINT_BASE_URL = "wss://streaming.eu.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

const ws = new WebSocket(API_ENDPOINT, {
  headers: {
    Authorization: YOUR_API_KEY,
  },
});
```

---

<a id="rate-limits-docs-streaming-rate-limits-mdx"></a>

# Rate limits

URL: https://www.assemblyai.com/docs/streaming/rate-limits
Source: docs/streaming/rate-limits.mdx
Navigation: Real-time STT > Advanced
Description: Rate limits documentation.

AssemblyAI's Real-time STT API features unlimited, automatic scaling rate limits for paid accounts. Unlike pre-recorded transcription, there is no hard cap on the total number of open streaming sessions. Instead, the constraint is on the number of **new streaming sessions** that can be opened per minute.

## Default limits

| Account type | Starting rate limit (new sessions/min) |
| ------------ | -------------------------------------- |
| Free         | 5                                      |
| Paid         | 100+                                   |

  **Need a higher rate limit?**

Our services are infinitely scalable and we offer custom rate limits that scale to support any workload at no additional cost. If you need a higher rate limit, please either [contact our Sales team](https://www.assemblyai.com/contact) or send an email to [support@assemblyai.com](mailto:support@assemblyai.com).

## Auto-scaling

Anytime you are utilizing **70% or more of your current limit**, the number of new streams able to be opened over the next minute will automatically increase by 10%.

For example, assuming you max out your available new sessions rate limit each minute for 5 minutes:

| Minute | New sessions/min              | Total open streams |
| ------ | ----------------------------- | ------------------ |
| 1      | 100 (default)                 | 100                |
| 2      | 110 (100 x 1.10)             | 210                |
| 3      | 121 (110 x 1.10)             | 331                |
| 4      | 133 (121 x 1.10)             | 464                |
| 5      | 146 (133 x 1.10)             | 610                |

At the start of the 6th minute, there would be 610 total open streams. Over the next 60 seconds, a maximum of 161 (146 x 1.10) new streams would be able to be opened.

This usage pattern has no ceiling and can continue to scale up indefinitely to whatever your application requires.

## Scale-down behavior

Based on the rate at which you are opening streams in comparison to your current limit, your new sessions per minute limit adjusts as follows:

- **70% or more** -- Scale up by 10%.
- **50-69%** -- Unchanged.
- **Less than 50%** -- Begins to scale back down to your account's starting limit.

If your usage has been low for a period of time, the scale-down may impact your ability to immediately open a large number of new streams, even if you have a high number of open sessions.

## What happens when you hit the limit

If you exceed your current new sessions per minute limit, you will receive a WebSocket closure with code `1008` and the message `Unauthorized connection: Too many concurrent sessions`.

You can check your current limit on the [Rate Limits page](https://www.assemblyai.com/dashboard/home) of your dashboard.

If you are receiving this error unexpectedly, verify that all of your sessions are being terminated properly. Sessions that are not properly closed continue to count against your rate limit.

## Properly terminating sessions

To ensure sessions are cleaned up correctly:

- If you are using the [WebSocket API](/streaming/api-spec/streaming-websocket) directly, send a `Terminate` message (`{"type": "Terminate"}`) to close the session.

Failing to terminate sessions properly can cause your open-session count to remain artificially high, leading to unexpected rate limit errors when opening new sessions.

  **This also affects your bill**

  Streaming is [billed per session](/billing-and-pricing#streaming-speech-to-text-billing) — the total time your WebSocket stays open, not the amount of audio you send. Sessions that aren't terminated auto-close after 3 hours and are billed for the full duration.

## Check your limit

You can view your current streaming rate limit on the [Rate Limits page](https://www.assemblyai.com/dashboard/home) of your dashboard.

With the current version of multi-project support, rate limiting is applied at the account level, not at the project level. This means that the rate limits for each API key mirror the rate limits for the account.

## Related pages

- [Pre-Recorded STT rate limits](/pre-recorded-audio/rate-limits)
- [Account Management](/account-management)
- [Common session errors and closures](/streaming/common-session-errors-and-closures)

---

<a id="self-hosted-streaming-docs-streaming-self-hosted-streaming-mdx"></a>

# Self-Hosted Streaming

URL: https://www.assemblyai.com/docs/streaming/self-hosted-streaming
Source: docs/streaming/self-hosted-streaming.mdx
Navigation: Real-time STT > Advanced
Description: Deploy AssemblyAI's streaming transcription solution within your own infrastructure

The **AssemblyAI Self-Hosted Streaming Solution** provides a secure, low-latency real-time transcription service that can be deployed within your own infrastructure. Audio, transcripts, and PII never leave your network — only license validation and usage metadata are transmitted back to AssemblyAI.

Self-hosted streaming requires an upfront commercial commitment of $20,000. [Contact our sales team](https://www.assemblyai.com/contact/sales) to discuss your needs and learn more about our self-hosted offering.

The deployment instructions, Compose files, nginx configuration, and example clients are maintained in the public [`streaming-self-hosting-stack`](https://github.com/AssemblyAI/streaming-self-hosting-stack) repository. This page covers what self-hosted streaming is, what you need to run it, and how the stack is shaped. **Go to the repo for the actual setup steps.**

## What you can self-host

Self-hosted streaming ships as **two separate stacks**. Each stack serves one model family, runs from its own Docker Compose file, and uses its own GPU. You pick the stack that matches the model you want to serve — they are not designed to run side by side.

| Stack | Model(s) served | Compose file | Best for |
|-------|-----------------|--------------|----------|
| **Universal Streaming** | Universal Streaming English + Multilingual | [`streaming/docker-compose.english-multilang.yml`](https://github.com/AssemblyAI/streaming-self-hosting-stack/blob/main/streaming/docker-compose.english-multilang.yml) | English and multilingual transcription workloads, telephony, captioning |
| **Universal-3.5 Pro Streaming** | Universal-3.5 Pro | [`streaming/docker-compose.universal-3-5-pro.yml`](https://github.com/AssemblyAI/streaming-self-hosting-stack/blob/main/streaming/docker-compose.universal-3-5-pro.yml) | Voice agents — short utterances, low end-of-turn latency, continuous partials |

For product capabilities and accuracy details, see the [Universal Streaming](/streaming) and [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio) overview pages.

## Core principle

- **Complete data isolation.** Audio, transcripts, and PII stay inside your infrastructure. The only outbound traffic is license validation and (for usage-based contracts) usage metadata to `https://usage-tracker.assemblyai.com`.

## System requirements

### Hardware

- **Universal Streaming.** NVIDIA T4 or newer per ASR container. We recommend at least 4 CPU and 16 GB RAM per ASR container.
- **Universal-3.5 Pro Streaming.** NVIDIA L4, A10, A100, L40S, H100, or equivalent with **at least 24 GB VRAM**. The container also bundles ~14 GB of model weights, so plan disk accordingly. T4 GPUs are **not** sufficient for U3.5 Pro.

### Software

- **Operating system.** Linux
- **Container runtime.** Docker and Docker Compose (v2 — the `docker compose` command, not `docker-compose`)
- **NVIDIA Container Toolkit.** Required for Docker to access the GPU
- **AWS credentials.** AssemblyAI provisions a scoped AWS access key for your team so you can pull container images from our private ECR registry

## Architecture

Both stacks share the same gateway, load balancer, and license proxy — they only differ in the ASR backend.

### Shared services (both stacks)

- **`streaming-api`** — Gateway WebSocket service that clients connect to. Handles session lifecycle, audio framing, and routing to the ASR backend.
- **`license-and-usage-proxy`** — Validates the license file at startup and reports usage metadata (for usage-based contracts).
- **`streaming-asr-lb`** — `nginx:alpine` load balancer that routes ASR gRPC requests to the right backend based on the `X-Model-Version` header.

### Universal Streaming stack

Adds two ASR backends:

- **`streaming-asr-english`** — English speech recognition.
- **`streaming-asr-multilang`** — Multilingual speech recognition.

### Universal-3.5 Pro Streaming stack

Adds a single ASR backend:

- **`streaming-asr-u3pro`** — Universal-3.5 Pro speech recognition. Available as of v0.6.0.

### Connection flow

```
WebSocket client ──→ streaming-api:8080
                          │
                          ├─ License validation ──→ license-and-usage-proxy:8080
                          │                              │
                          ├─ Usage reporting ────────────┴──→ https://usage-tracker.assemblyai.com
                          │                                  (usage-based billing only)
                          │
                          └─ ASR (gRPC) ──────────→ streaming-asr-lb:80
                                                       │
                                                       └─ Header-based routing (X-Model-Version):
                                                          ├── en-default → streaming-asr-english:50051 (gRPC)
                                                          ├── ml-default → streaming-asr-multilang:50051 (gRPC)
                                                          └── u3-pro     → streaming-asr-u3pro:50051 (gRPC)
```

The load balancer only forwards to backends that are actually deployed in the running stack — Universal Streaming routes for `en-default` and `ml-default`, U3.5 Pro routes for `u3-pro`.

## Getting started

Follow the [upstream repo's README](https://github.com/AssemblyAI/streaming-self-hosting-stack#setup-instructions) for the actual setup steps. At a high level:

1. **Get credentials and a license file** from your AssemblyAI representative — an AWS access key scoped to ECR, and a `license.jwt` file. The same license file works for both stacks.
2. **Install Docker, Docker Compose, and the NVIDIA Container Toolkit.** See the README's [setup section](https://github.com/AssemblyAI/streaming-self-hosting-stack#setup-instructions) for verification commands.
3. **Authenticate to ECR** with the provided AWS credentials.
4. **Pick a stack and configure `.env`** with the image references from the repo's [`.env.example`](https://github.com/AssemblyAI/streaming-self-hosting-stack/blob/main/streaming/.env.example).
5. **Start the stack** with `docker compose up -d` (Universal Streaming) or `docker compose -f docker-compose.u3pro.yml up -d` (Universal-3.5 Pro Streaming).

Universal Streaming ASR containers take roughly **2 minutes** to become ready and log `Ready to serve!`. The Universal-3.5 Pro Streaming ASR container takes roughly **5 minutes** and logs `U3Pro ASR Server ready!`. Health checks may report `unhealthy` during startup — that is expected.

## Running a test client

The [repo](https://github.com/AssemblyAI/streaming-self-hosting-stack) ships an example Python client under [`streaming/example`](https://github.com/AssemblyAI/streaming-self-hosting-stack/tree/main/streaming/example) that streams a pre-recorded WAV file to the WebSocket endpoint. It supports all three speech models via the `--speech-model` flag:

- `universal-streaming-english` — Universal Streaming, English
- `universal-streaming-multilingual` — Universal Streaming, multilingual
- `universal-3-5-pro` — Universal-3.5 Pro Streaming

The client routes to the correct ASR backend automatically via the `X-Model-Version` header. Make sure the value you pass matches a backend deployed in the stack you started.

## Switching between stacks

The two stacks listen on the same ports (`streaming-api` on `8080`, ASR load balancer on the gRPC backend), so they cannot run simultaneously. To switch:

```bash
# Stop the running stack first.
docker compose down                                # if Universal Streaming is running
docker compose -f docker-compose.u3pro.yml down    # if Universal-3.5 Pro Streaming is running

# Update .env for the new stack (image vars differ), then start.
docker compose up -d                               # Universal Streaming
docker compose -f docker-compose.u3pro.yml up -d   # Universal-3.5 Pro Streaming
```

## Production deployment

Per-service deployment strategy, resource sizing, autoscaling thresholds, health-check tuning, and the `license-and-usage-proxy` `/v1/status` endpoint reference all live in the repo's [Production Deployment Recommendations](https://github.com/AssemblyAI/streaming-self-hosting-stack#production-deployment-recommendations) section.

## Release notes and changelog

Release notes for the self-hosted stack — including per-version model improvements, API additions, and breaking changes — live in the repo's [README changelog](https://github.com/AssemblyAI/streaming-self-hosting-stack#changelog). Tagged releases are visible on the [Releases page](https://github.com/AssemblyAI/streaming-self-hosting-stack/releases).

## Support

For deployment questions, image access, license issues, or to report bugs, contact your AssemblyAI representative.

---

<a id="universal-3-5-pro-realtime-on-livekit-docs-voice-agents-livekit-universal-3-5-pro-mdx"></a>

# Universal 3.5 Pro Realtime on LiveKit

URL: https://www.assemblyai.com/docs/voice-agents/livekit-universal-3-5-pro
Source: docs/voice-agents/livekit-universal-3-5-pro.mdx
Navigation: Real-time STT > Integrations
Description: Integrate AssemblyAI's Universal 3.5 Pro Realtime speech-to-text model into a LiveKit voice agent

## Overview

This guide covers integrating AssemblyAI's **Universal 3.5 Pro Realtime** speech-to-text model into a LiveKit voice agent using the [Agents framework](https://docs.livekit.io/agents/).

  **Universal 3.5 Pro Realtime is our flagship next-generation streaming model for voice agents** — multilingual and promptable, with [conversation context](#conversation-context) and [voice focus](#voice-focus).

  Available on **`livekit-agents` 1.6+** — set `model="universal-3-5-pro"`.

AssemblyAI provides the speech-to-text and turn detection in your LiveKit pipeline:

```mermaid
flowchart LR
  U["User audio"] --> STT["AssemblyAI STT<br/>Universal 3.5 Pro Realtime"]
  STT --> TD["Turn detection"]
  TD --> LLM["LLM"]
  LLM --> TTS["TTS"]
  TTS --> U
```

Once you have an agent running, tune it for what matters most to your use case:

  
    Decide when the user is done speaking — modes, defaults, and entity tuning.
  
  
    Shorten the gap between the user finishing and the agent replying.
  
  
    Prompting, key terms, conversation context, and noise handling.
  
  
    Natural barge-in without false triggers from backchannels.
  

  For a standalone voice agent without LiveKit or an external LLM, see the [AssemblyAI Voice Agent API](/voice-agents/voice-agent-api), which handles STT, LLM routing, and TTS in a single WebSocket connection.

## Quickstart

Get a working, talking agent in a few minutes, then optimize from there.

Install the plugin and supporting packages (silero, codecs, dotenv) from PyPI:

```bash
pip install "livekit-agents[assemblyai,silero,codecs]~=1.6" \
    python-dotenv
```

If you plan to use LiveKit turn detection with `MultilingualModel()`, also install the turn detector plugin:

```bash
pip install "livekit-plugins-turn-detector~=1.0"
```

  Install the latest `livekit-agents` from PyPI. **Universal 3.5 Pro Realtime**, `agent_context`, and **Voice Focus** require a recent **1.6** release. Automatic conversation-context forwarding and `language_codes` require **1.6.6+**. Older versions won't recognize the `universal-3-5-pro` model and return a validation error.

Set your API keys in a `.env` file:

```env
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
ASSEMBLYAI_API_KEY=your_assemblyai_key
# Add API keys for your chosen LLM and TTS providers
```

  You can obtain an AssemblyAI API key by [signing up for a free
  account](https://www.assemblyai.com/dashboard/signup) and navigating to the [API
  Keys tab](https://www.assemblyai.com/dashboard/home) of the dashboard.

The following example uses `turn_detection="stt"` (recommended). Pay close attention to the comments for using with `MultilingualModel()`.

```python expandable
from dotenv import load_dotenv
from livekit import agents
from livekit.agents import AgentSession, Agent, TurnHandlingOptions
from livekit.plugins import (
    assemblyai,
    silero,
)
# For MultilingualModel, uncomment the following:
# from livekit.plugins.turn_detector.multilingual import MultilingualModel

load_dotenv()

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant.")

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    session = AgentSession(
        stt=assemblyai.STT(
            model="universal-3-5-pro",
            min_turn_silence=100,
            max_turn_silence=1000,  # When turn_detection="stt", override plugin default of 100.
            # If using MultilingualModel(), the plugin defaults (min: 100, max: 100) work well. Omit min_turn_silence and max_turn_silence above if preferred.
            vad_threshold=0.3,  # Match Silero's activation_threshold
            # continuous_partials is True by default in the LiveKit plugin — steady ~3s partials during long turns.
            # interruption_delay=0,  # Optional: faster first partial (~300ms effective). Default: 500 (~800ms effective).
        ),
        # llm=your_llm_plugin(),  # Add your LLM provider here
        # tts=your_tts_plugin(),  # Add your TTS provider here
        vad=silero.VAD.load(
            activation_threshold=0.3,  # Match AssemblyAI's internal VAD threshold
        ),
        turn_handling=TurnHandlingOptions(
            turn_detection="stt",
            # To use LiveKit's turn detection instead, replace the line above with:
            # turn_detection=MultilingualModel(),
            endpointing={"min_delay": 0},  # Avoid additive delay in STT mode
            # If using MultilingualModel(), set these instead:
            # endpointing={"min_delay": 0.5, "max_delay": 3.0},
        ),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
    )

    await session.generate_reply(
        instructions="Greet the user and offer your assistance."
    )

if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))
```

For a complete voice agent, you will also need to install LLM and TTS plugins for your chosen providers. See the [LiveKit plugins documentation](https://docs.livekit.io/agents/plugins/) for available options.

Start your agent in development mode:

```bash
python your_agent_file.py dev
```

Then test it in the LiveKit Playground:

1. Go to [agents-playground.livekit.io](https://agents-playground.livekit.io)
2. Connect to your LiveKit Cloud project (same credentials as your `.env`)
3. Click **Connect**: a room will be created, your agent will join, and you can start talking

## Parameters reference

### **Universal 3.5 Pro Realtime** parameters

These are the key parameters to tune for LiveKit when using **Universal 3.5 Pro Realtime**:

  The streaming model. Defaults to `"universal-3-5-pro"`, our recommended
  flagship model.

  Accuracy/latency preset: `"min_latency"`, `"balanced"`, or `"max_accuracy"`. Sets
  the defaults for mode-dependent fields (`min_turn_silence`, `vad_threshold`,
  `interruption_delay`, `continuous_partials`, `previous_context_n_turns`); any value
  you set explicitly still takes precedence. Leave unset for the server's default
  preset. Connect-time only — cannot be changed with `update_options`. Universal-3.5 Pro
  only. See [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency).

  List of terms to boost recognition for. Applied automatically alongside any
  contextual prompt.

  Contextual prompt — a natural-language description of what the audio is
  about (domain, scenario, or full details). Transcription behavior is built
  in and optimized automatically.

  Your agent's most recent spoken reply, up to 1750 characters, used as context
  for transcribing the next user turn. The 1750-character cap is enforced
  client-side — explicit values over the cap raise `ValueError` (both at
  construction and via `update_options`); only automatic forwarding truncates
  silently. Can be set at construction time and updated mid-stream with
  `update_options`. Universal-3.5 Pro only. See
  [Conversation context](#conversation-context).

  Steer transcription toward one or more expected languages instead of
  auto-detecting across all supported languages. Accepts a single code
  (`"es"`) or a list (`["en", "es"]`); entries are normalized to ISO 639-1
  (`"en-US"`, `"english"` → `"en"`). Up to 10 codes. Leave unset for the
  default multilingual behavior. Universal-3.5 Pro only. Can be updated
  mid-session via `update_options`; pass an empty list to clear steering.
  Requires `livekit-agents` **1.6.6+**.

  How many prior conversation entries are carried forward automatically. Range
  `0`–`100`; `0` disables carryover. Leave unset for the server default (5).
  Connect-time only — cannot be changed with `update_options`. Universal-3.5 Pro
  only.

  Milliseconds of silence before a speculative end-of-turn check. When the check
  fires, the model looks for terminal punctuation to decide whether the turn has
  ended.

  Maximum milliseconds of silence before the turn is forced to end, regardless
  of punctuation. **The LiveKit plugin defaults to `100`.** Set to `1000` when
  using `turn_detection="stt"`.

  AssemblyAI's internal Silero VAD threshold. **Universal 3.5 Pro Realtime**
  defaults to `0.3`, unlike Universal-Streaming's `0.4`. Align with LiveKit's
  Silero `activation_threshold` for consistent behavior.

  Server-side noise suppression that isolates the primary speaker. `"near-field"`
  for close-talking mics, `"far-field"` for distant capture. Connect-time only.
  Universal-3.5 Pro only. See [Voice focus](#voice-focus).

  How aggressively `voice_focus` suppresses background audio. `0.0`–`1.0`; higher
  is more aggressive. Connect-time only. Universal-3.5 Pro only.

  Whether to emit additional partial transcripts during long turns at a steady
  ~3 second cadence. When enabled (default on both the API and the LiveKit
  plugin), additional partials covering the full turn transcript are emitted
  approximately every 3 seconds while speech continues. When disabled, only one
  early partial is emitted near turn start. The first partial (at 750ms) is
  unaffected. Useful when downstream consumers (LLMs, UI, eager inference) need
  frequent updates during long, uninterrupted turns. See
  [Continuous partials](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  How soon the first partial transcript is emitted during a turn, in
  milliseconds. Range: `0`–`1000`. Lower values produce faster time to first
  token (TTFT) for barge-in and speculative inference; higher values produce
  more confident first partials. The server adds a minimum of 300ms on top of
  the configured value (`interruption_delay: 0` → ~300ms effective,
  `interruption_delay: 500` → ~800ms effective). See
  [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  **Universal 3.5 Pro Realtime** code-switches natively between supported
  languages. This parameter controls whether `language_code` and
  `language_confidence` are included in turn messages. Defaults to `true` in the
  LiveKit plugin, but `false` when using the API directly.

### General STT parameters

These parameters apply to all AssemblyAI streaming models and can remain the same between models:

  The sample rate of the audio stream.

  The encoding of the audio stream. Allowed values: `pcm_s16le`, `pcm_mulaw`.

### Legacy parameters

These parameters apply to the `universal-streaming-english` and `universal-streaming-multilingual` AssemblyAI streaming models, but **do not affect Universal 3.5 Pro Realtime**:

  Confidence threshold for end-of-turn detection. **Universal 3.5 Pro Realtime**
  uses punctuation-based turn detection instead.

  Whether to return formatted final transcripts. **Universal 3.5 Pro Realtime**
  always returns formatted transcripts, so this parameter no longer applies.

## Turn detection

In LiveKit, how your agent detects the end of a user's turn is controlled by the [`turn_detection`](https://docs.livekit.io/reference/agents/turn-handling-options/#turn_detection) parameter inside [`TurnHandlingOptions`](https://docs.livekit.io/reference/agents/turn-handling-options/), which is passed to `AgentSession` via the `turn_handling` argument.

**Universal 3.5 Pro Realtime** uses a **punctuation-based turn detection system**, which checks for terminal punctuation (`.` `?` `!`) after periods of silence rather than using a confidence score.

This means the `min_turn_silence` and `max_turn_silence` parameters you pass to AssemblyAI directly control when transcripts are emitted and when turns end. For more details on how this works, see [Configuring turn detection](/streaming/getting-started/transcribe-streaming-audio).

  When not explicitly provided, the default endpointing parameters for **Universal 3.5 Pro Realtime** differ on **LiveKit** versus using **AssemblyAI's API directly**:
  - LiveKit AssemblyAI plugin defaults:
    - `min_turn_silence=100`
    - `max_turn_silence=100`
  - AssemblyAI API defaults:
    - `min_turn_silence=100`
    - `max_turn_silence=1000`

  However, you can always override these by passing your own preferred values explicitly. Misconfiguring these parameters is the most common cause of poor performance — read the recommended values per mode below.

### Default parameter differences

**Universal 3.5 Pro Realtime's** endpointing is controlled by two AssemblyAI API parameters, `min_turn_silence` and `max_turn_silence`, that you pass to the STT plugin. These are separate from LiveKit's `endpointing.min_delay` and `endpointing.max_delay` (set inside `TurnHandlingOptions`).

| Parameter             | AssemblyAI API default | LiveKit plugin default | Description                                                                                                                                                         |
| --------------------- | ---------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `min_turn_silence`    | `100` ms               | `100` ms               | Silence before a speculative end-of-turn check. If terminal punctuation (`.` `?` `!`) is found, the turn ends. If not, a partial is emitted and the turn continues. |
| `max_turn_silence`    | **`1000`** ms          | **`100`** ms           | Maximum silence before forcing the turn to end, regardless of punctuation.                                                                                          |

The LiveKit plugin defaults are optimized for third-party turn detection models, where you want transcripts handed off as fast as possible. When using `turn_detection="stt"`, you should explicitly set `max_turn_silence=1000` if you'd like to mimic the behavior of streaming directly to the API without LiveKit.

  **Tuning endpointing parameters**

  **These are the default values used when no parameters are explicitly provided.** You will likely need to experiment with different values depending on your use case:

- **Increase `min_turn_silence`**: when brief pauses cause the speculative EOT check to fire too early, ending turns on terminal punctuation before the user has finished speaking.
- **Increase `max_turn_silence`**: when the forced turn end is cutting off users mid-thought or splitting entities like phone numbers across turns, a higher value lets the model wait longer before forcing the turn to end when the model is unsure.

See the [Entity splitting tradeoff](#entity-splitting-tradeoff) section for examples.

### STT-based Turn Detection (recommended)

With `turn_detection="stt"`, AssemblyAI's built-in punctuation-based turn detection determines when the user has finished speaking. AssemblyAI's `end_of_turn` signals are then used directly by LiveKit to commit the turn.

In this mode, we recommend explicitly setting `min_turn_silence=100` and `max_turn_silence=1000`. These are AssemblyAI's API defaults and provide a good balance of responsiveness and accuracy.

**The LiveKit plugin defaults to `min_turn_silence=100` and `max_turn_silence=100`, which might be too aggressive for STT-based turn detection.**

**Recommended starting parameters** (set on `assemblyai.STT()`, not on `AgentSession`):

| Parameter          | Default   | Description                                                          |
| ------------------ | --------- | -------------------------------------------------------------------- |
| `min_turn_silence` | `100` ms  | Silence duration before a speculative end-of-turn (EOT) check fires. |
| `max_turn_silence` | `1000` ms | Maximum silence before a turn is forced to end.                      |

**How it works:**

1. User speaks → audio streams to AssemblyAI
2. User pauses for `100ms` → AssemblyAI checks for terminal punctuation
3. If terminal punctuation (`.` `?` `!`) → turn ends immediately
4. If no terminal punctuation → partial emitted, turn continues waiting
5. If silence reaches `1000ms` → turn is forced to end regardless of punctuation

  **Endpointing `min_delay` is additive in STT mode**

  LiveKit's endpointing `min_delay` (default **0.5 seconds**) is applied **on top of** AssemblyAI's own endpointing. In STT mode, this delay starts *after* the STT end-of-speech signal, meaning it adds up to `500ms` of extra latency by default.

**Set `endpointing={"min_delay": 0}`** inside `TurnHandlingOptions` to avoid this. AssemblyAI's own endpointing parameters (`min_turn_silence` and `max_turn_silence`) already control the timing, so an additional delay on the LiveKit side is unnecessary latency. See [Latency](#latency) for the full picture.

```python
from livekit.agents import AgentSession, TurnHandlingOptions

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
        endpointing={"min_delay": 0},  # Avoid additive delay in STT mode
    ),
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,   # Silence (ms) before a speculative end-of-turn check
        max_turn_silence=1000,  # Max silence (ms) before forcing the turn to end
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,
    ),
)
```

### LiveKit turn detection (with `MultilingualModel()`)

As a third-party turn detection model, LiveKit's [turn detector](https://docs.livekit.io/agents/logic/turns/turn-detector/) runs on top of STT output to make turn decisions. AssemblyAI's role is then just to provide transcripts as quickly as possible, while the turn detection model decides when the user is actually done speaking.

Use `MultilingualModel()` rather than `EnglishModel()`, as **Universal 3.5 Pro Realtime** supports 18 languages. `MultilingualModel()` covers support for all of these languages.

The LiveKit plugin defaults of `min_turn_silence=100` and `max_turn_silence=100` work well here, as **`max_turn_silence` is brought down to match `min_turn_silence` so that transcripts are handed off to the turn detection model as fast as possible.**

**`MultilingualModel` parameters** (set inside `TurnHandlingOptions` → `endpointing`, not on the STT plugin):

| Parameter               | Default | Description                                                                                                                |
| ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
| `endpointing.min_delay` | `0.5` s | Time to wait before committing a turn when the model predicts a likely boundary.                                           |
| `endpointing.max_delay` | `3.0` s | Maximum time to wait when the model predicts the user will continue speaking. Has no effect without a turn detector model. |

**How it works:**

1. User speaks → audio streams to AssemblyAI
2. User pauses for `100ms` → AssemblyAI emits transcript (final and partial are the same) immediately
3. LiveKit's `MultilingualModel()` evaluates the transcript in conversational context
4. If the model predicts a likely turn boundary → waits `min_delay` (`0.5s`) then commits the turn
5. If the model predicts the user will continue → waits up to `max_delay` (`3.0s`) for more speech

```python
from livekit.agents import AgentSession, TurnHandlingOptions
from livekit.plugins.turn_detector.multilingual import MultilingualModel

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection=MultilingualModel(),
        endpointing={
            "min_delay": 0.5,  # Time (s) to wait before committing a turn when the model is confident
            "max_delay": 3.0,  # Max time (s) to wait when the model is not confident
        },
    ),
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,
    ),
)
```

### Other turn detection modes

- **`vad`**:

  - Detect end of turn from speech and silence data alone using [Silero VAD](https://docs.livekit.io/agents/logic/turns/vad/).
  - Turn boundaries are determined purely by voice activity without semantic context.
  - AssemblyAI's turn detection parameters still control when transcripts are emitted, but it is recommended to leave them at the plugin defaults (`min_turn_silence=100`, `max_turn_silence=100`) so transcripts arrive as quickly as possible.

- **`manual`**:
  - Disable automatic turn detection entirely.
  - You control turns explicitly using `session.commit_user_turn()`, `session.clear_user_turn()`, and `session.interrupt()`.
  - See the [manual turn control docs](https://docs.livekit.io/agents/logic/turns/#manual-turn-control) for details.

### VAD configuration

With `turn_detection="stt"`, AssemblyAI also sends `SpeechStarted` events that LiveKit uses for barge-in/interruption handling. `SpeechStarted` is only emitted when the model produces a transcript.

Silero VAD is not strictly required in this mode, but it is still recommended as Silero runs locally and it can be faster than waiting for AssemblyAI's `SpeechStarted` signal. LiveKit respects whichever signal arrives first, so Silero provides faster interruption while AssemblyAI's signal serves as a reliable backup.

With `MultilingualModel()`, Silero VAD is **required**, as it is the only source of `START_OF_SPEECH` events for interruption in this mode. AssemblyAI's `SpeechStarted` event is not used.

#### Threshold alignment

LiveKit's Silero VAD defaults to an `activation_threshold` of **0.5**. AssemblyAI's `vad_threshold` defaults to **0.3**. For best performance, we recommend setting both to **0.3**.

Both should be adjusted together to the same value to ensure accurate transcription and consistent barge-in thresholds.

When the thresholds are mismatched, you get a dead zone: if Silero is at `0.5` and AssemblyAI is at `0.3`, AssemblyAI will be actively transcribing speech that LiveKit hasn't detected yet, delaying interruption. Keeping them aligned eliminates this.

```python
session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        vad_threshold=0.3,         # AssemblyAI's internal VAD onset
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,  # Match AssemblyAI's threshold
    ),
)
```

If you're in a noisy environment and receiving false speech triggers, raise both `stt.vad_threshold` and `vad.activation_threshold` thresholds together.

### Entity splitting tradeoff

Lower `min_turn_silence` and `max_turn_silence` values produce faster transcripts but can split entities or utterances across turns. The two parameters affect this differently.

#### `min_turn_silence` too low

- Speculative check fires too early, splitting entities on punctuation.

- **Example:** _User spells out an email address with brief pauses between parts. The speculative check fires at 100ms of silence, and the model adds terminal punctuation to each segment, ending the turn prematurely._

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"It's John."                    → FINAL (100ms pause, check fires, period found → turn ends)
"Smith."                        → FINAL
"At gmail.com."                 → FINAL

# With (min_turn_silence=400, max_turn_silence=1000)
"It's john.smith@gmail.com."    → FINAL (single turn, properly formatted)
```

#### `max_turn_silence` too low

- Forced turn-end cuts off user mid-thought.

- **Example:** _User pauses longer than 1 second to think mid-sentence. The forced end fires at 1000ms, splitting the utterance into two turns regardless of punctuation._

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"I wanted to check on my order from..."  → FINAL (1000ms silence, forced end)
"last Tuesday, order number 4829."     → FINAL (new turn)

# With (min_turn_silence=100, max_turn_silence=2000)
"I wanted to check on my order from last Tuesday, order number 4829."  → FINAL (single turn)
```

**Universal 3.5 Pro Realtime's** formatting is significantly better when it has full context in a single turn. Email addresses, phone numbers, credit card numbers, and physical addresses all benefit from this.

  LLMs downstream can usually piece together split entities, but if your use
  case involves alphanumeric dictation or entity extraction, consider increasing
  `min_turn_silence` and `max_turn_silence` during those portions of the
  conversation. You can [update configuration
  mid-stream](#dynamic-configuration) to raise `max_turn_silence`
  temporarily (e.g., to `2000`–`4000` ms) when expecting entity input, then
  lower it again afterward.

Even when using third-party turn detection, you may want to increase `min_turn_silence` or `max_turn_silence` if users are likely to speak slowly or dictate entities. While this adds latency, it improves accuracy by giving the model more audio context before emitting a transcript and keeping the full entity complete within the same turn.

## Latency

A voice agent feels responsive when the gap between the user finishing and the agent replying is short.

Start with the **`mode` preset** — the highest-level dial for the accuracy/latency trade-off. It sets sensible defaults for the fine-grained levers below, so you can pick a target and tune from there:

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    mode="balanced",  # "min_latency" (fastest) · "balanced" · "max_accuracy" (best quality)
)
```

`mode` is set at construction time (it can't be changed mid-session) and influences the defaults of `min_turn_silence`, `vad_threshold`, `interruption_delay`, `continuous_partials`, and `previous_context_n_turns`. Any value you set explicitly still wins. Leave it unset to use the server's default preset. See [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency).

From there, fine-tune the individual levers:

- **Endpointing delay (STT mode).** LiveKit's `endpointing.min_delay` is applied *on top of* AssemblyAI's own end-of-turn timing and adds up to `500ms` by default. Set `endpointing={"min_delay": 0}` when using `turn_detection="stt"` — see [STT-based turn detection](#stt-based-turn-detection-recommended).
- **End-of-turn timing.** `min_turn_silence` (speculative check) and `max_turn_silence` (forced end) directly control how soon a turn ends. Lower is faster but risks splitting entities — see [Turn detection](#turn-detection).
- **Time to first partial.** `interruption_delay` controls how soon the first partial is emitted, which drives faster barge-in and speculative inference. The server adds a minimum of `300ms` on top of the configured value.

```python
# Tune first-partial timing for faster barge-in
stt.update_options(
    interruption_delay=0,  # ~300ms effective TTFT
)
```

- **Sample rate.** Use 16 kHz (`sample_rate=16000`). Higher rates don't improve accuracy and only add bandwidth.
- **Continuous partials.** `continuous_partials` (on by default) emits a partial every ~3 seconds during long turns. Leave it on for steady mid-turn updates, or disable it if you only need a single early partial.
- **Skip client-side preprocessing.** Don't run your own noise cancellation before audio reaches the model — the artifacts it introduces usually hurt accuracy more than the original noise. Use server-side [Voice Focus](#voice-focus) instead.

### Latency breakdown

| Stage                                      | Typical                                              | Controlled by           |
| ------------------------------------------ | ---------------------------------------------------- | ----------------------- |
| Network round trip                         | ~50 ms                                               | —                       |
| Speech-to-text                             | ~200–300 ms                                          | model                   |
| First partial (TTFT)                       | configured `interruption_delay` + ~300 ms server min | `interruption_delay`    |
| End of turn (terminal punctuation found)   | `min_turn_silence` (default 100 ms)                  | `min_turn_silence`      |
| End of turn (no punctuation, forced)       | up to `max_turn_silence`                             | `max_turn_silence`      |
| LiveKit endpointing (STT mode)             | + `min_delay` (set to `0`)                           | `endpointing.min_delay` |

## Accuracy

**Universal 3.5 Pro Realtime** is accurate out of the box. When you need more — domain vocabulary, proper nouns, noisy audio — reach for these levers. For entity-heavy dictation, also tune turn detection (see [Entity splitting tradeoff](#entity-splitting-tradeoff)), and note that the high-level [`mode` preset](#latency) shifts the overall accuracy/latency balance (use `max_accuracy` to favor quality).

### Prompting

  **Beta feature**

Prompting is considered a beta feature for **Universal 3.5 Pro Realtime**.

While it can be a powerful tool for improving accuracy in certain use cases, **we recommend starting without a `prompt` to first establish baseline performance.**

Once the baseline has been tested, you can add context to further optimize for your use case (e.g., language mix to expect (e.g., English and Hindi), use case or domain (e.g., medical, legal), etc.).

**Universal 3.5 Pro Realtime** supports a `prompt` parameter for [contextual prompting](/streaming/prompting-and-keyterms) — a description of what the audio is about. Transcription behavior (verbatim output, punctuation, turn detection) is built in and optimized automatically; the prompt carries context, not instructions.

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    prompt="Customer support call about an internet service outage.",
)
```

**Tips:**

- **Start with no prompt**: Universal 3.5 Pro Realtime delivers strong accuracy out of the box — only add context if domain-specific vocabulary is being misrecognized.
- **Describe the conversation**: domain, scenario, or full details — start broad, and add only details your application actually knows (see the [three context levels](/streaming/prompting-and-keyterms#contextual-prompting)).
- **Include known names and identifiers**: caller name, account or order IDs, products — detailed context helps the model spell them correctly.

### Key terms

Instead of `prompt`, use `keyterms_prompt` to boost recognition of specific names, brands, or domain terms:

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    keyterms_prompt=["AssemblyAI", "LiveKit", "Universal 3.5 Pro Realtime"],
)
```

### Language selection

By default, **Universal 3.5 Pro Realtime** auto-detects and code-switches across all supported languages. If you know which languages a caller will speak, `language_codes` steers transcription toward them for better accuracy on short or ambiguous turns. It can also be updated mid-session (e.g. when a caller switches languages) via `update_options`. Requires `livekit-agents` **1.6.6+**.

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    language_codes=["en", "es"],  # single code or list, up to 10; ISO 639-1
)
```

### Conversation context

Give the model both sides of the dialog so it transcribes the next user turn more accurately. **Universal 3.5 Pro Realtime** keeps a short, per-session memory of the conversation from two sources:

- **The agent half** — what your agent just said, forwarded automatically into `agent_context` by `AgentSession` on `livekit-agents` 1.6.6+ (no wiring needed).
- **The user half** — prior STT-finalized user turns, carried forward automatically (no configuration needed).

With the agent's question in context, the model can anticipate the answer, sharpen entity recognition, and disambiguate similar-sounding words. For example, after your agent asks `"What's your email address?"`, `agent_context` lets the model produce `"user@assemblyai.com"` instead of `"user at assemblyai dot com"`. This has the biggest impact on short replies (`"yes"`, `"7pm"`, single names) and spelled-out entities. See [Conversation context](/streaming/universal-3-5-pro/context-carryover) for the full reference.

| Parameter                  | Type    | Description                                                                                                                                                                                                    |
| -------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_context`            | string  | Your agent's most recent spoken reply (what your TTS just said), up to 1750 characters. Set at construction time to seed an opening greeting; updated automatically after each agent reply on 1.6.6+.           |
| `previous_context_n_turns` | integer | How many prior conversation entries are carried forward automatically. Connect-time only. Range `0`–`100`; `0` disables carryover; leave unset for the server default (5).                                     |

  `agent_context` and `previous_context_n_turns` are supported only on Universal-3.5 Pro (`universal-3-5-pro`). `previous_context_n_turns` is set at construction time and cannot be changed with `update_options()`.

#### Automatic on 1.6.6+

On `livekit-agents` **1.6.6+**, `AgentSession` forwards each assistant reply into `agent_context` automatically — no wiring needed. Replies longer than the 1750-character limit are truncated automatically, keeping the tail (the end of the reply — usually the question — has the most biasing value).

To opt out, use the session-level toggle:

```python
session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    stt_context_options={"forward_chat_context": False},  # default: True
)
```

  While automatic forwarding is on, any `agent_context` you set manually is
  replaced by the next assistant reply. To manage `agent_context` yourself
  (e.g. custom trimming or filtering), set `forward_chat_context: False` and
  push it via `update_options(agent_context=...)` after each reply.

  To seed the model with your agent's opening greeting, pass `agent_context`
  on `assemblyai.STT(...)` at construction time — automatic forwarding takes
  over from the first reply onward.

If you're on a version of `livekit-agents` that predates **1.6.6**, or you want to manage `agent_context` yourself, wire it via the `conversation_item_added` event. LiveKit emits it for every conversation item; filter for the assistant's messages and forward the (trimmed) text to the STT plugin via `update_options(agent_context=...)`. No reconnect is required.

A complete runnable example is on GitHub: [including-agent-context](https://github.com/dlange-aai/assemblyai-livekit-examples/tree/main/including-agent-context).

```python
from livekit.agents import AgentSession, ConversationItemAddedEvent
from livekit.plugins import assemblyai

AGENT_CONTEXT_MAX_CHARS = 1750

session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
    ),
    stt_context_options={"forward_chat_context": False},  # Disable automatic forwarding on 1.6.6+
    # llm=your_llm_plugin(),
    # tts=your_tts_plugin(),
)

@session.on("conversation_item_added")
def _on_conversation_item_added(ev: ConversationItemAddedEvent) -> None:
    # Only forward the agent's own replies as context for the next user turn.
    if ev.item.type != "message" or ev.item.role != "assistant":
        return

    agent_stt = session.stt
    if not isinstance(agent_stt, assemblyai.STT):
        return

    spoken = ev.item.text_content
    if not spoken:
        return

    # Push the latest reply; earlier turns are carried forward automatically.
    agent_stt.update_options(agent_context=spoken[-AGENT_CONTEXT_MAX_CHARS:])
```

#### LiveKit Inference

LiveKit Inference (`stt="assemblyai/universal-3-5-pro"`) receives the same automatic conversation-context forwarding by default, with the same `stt_context_options={"forward_chat_context": False}` opt-out. `language_codes` is plugin-only for now.

### Voice focus

Voice Focus isolates the primary speaker and suppresses background noise — chatter, keyboard clicks, fan hum, room echo — **server-side, before audio reaches the model**. Use it instead of client-side noise cancellation, which tends to introduce artifacts that hurt accuracy more than the noise itself.

| Parameter               | Type   | Description                                                                                                                                       |
| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `voice_focus`           | string | `"near-field"` for headsets, handsets, and other close-talking mics; `"far-field"` for conference rooms, laptop mics, and other distant capture. |
| `voice_focus_threshold` | float  | Optional. `0.0`–`1.0`; higher values suppress background audio more aggressively.                                                                |

Both are connection-time parameters on the Universal-3.5 Pro and cannot be changed with `update_options()`. See [Voice Focus](/streaming/voice-focus) for details.

```python
stt=assemblyai.STT(
    model="universal-3-5-pro",
    voice_focus="far-field",      # "near-field" for close-talking mics
    voice_focus_threshold=0.5,    # Optional: 0.0–1.0, higher = more aggressive
)
```

## Interruption handling

In voice agent conversations, users often produce backchannel utterances ("mhm", "yeah", "um", "okay") while the agent is speaking. These short fillers can trigger LiveKit's interruption logic, causing the agent to stop mid-sentence even though the user didn't intend to interrupt.

LiveKit Cloud users should reach for adaptive interruption handling first. Self-hosted deployments can combine the two custom filters described below.

### Recommended: Adaptive interruption handling

LiveKit Agents v1.5.0 introduced [adaptive interruption handling](https://docs.livekit.io/agents/logic/turns/adaptive-interruption-handling/), a server-side model that classifies overlapping speech as a real interrupt or a backchannel. On a false interrupt the agent's TTS resumes from where it left off. No re-generation is needed. The feature is recommended for LiveKit Cloud users and is included on all Cloud plans.

```python
session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    # llm=your_llm_plugin(),
    # tts=your_tts_plugin(),
    vad=silero.VAD.load(),
    interruption={"mode": "adaptive"},  # default in v1.5.0+
)
```

See LiveKit's [adaptive interruption handling docs](https://docs.livekit.io/agents/logic/turns/adaptive-interruption-handling/) for full requirements and behavior.

### Self-hosted alternative

If you're self-hosting LiveKit, can't use adaptive interruption handling, or need explicit control over which utterances count as filler, the two complementary filters below provide strong guardrails for interruption and barge-in handling. They have non-overlapping failure modes: the backchannel filter stops known fillers before they enter the pipeline, while the buffer-clearing filter catches any short utterance (including unknown fillers or stutters) that slips past. Running both provides the strongest coverage. A working reference implementation is available on [GitHub](https://github.com/AssemblyAI-Solutions/livekit-interruption-filters).

The backchannel filter intercepts STT events at the `stt_node` level, before they reach LiveKit's `audio_recognition`. It checks each transcript event for known disfluencies and backchannels ("um", "mhm", "yeah", "okay", etc.) and drops the event entirely. Because the event never enters the pipeline, none of the downstream orchestration (interrupt gates, end-of-turn detection, and preemptive LLM generation) ever reacts to it.

The filter is implemented as a mixin class that wraps `Agent.stt_node`:

```python
from __future__ import annotations

import logging
import string
import time
from collections.abc import AsyncIterable

from livekit import rtc
from livekit.agents import Agent, stt
from livekit.agents.voice import ModelSettings

# "yes" / "no" deliberately omitted - in a booking flow a bare "yes"
# is a real confirmation. Edit for your domain.
BACKCHANNELS = frozenset({
    "mhm", "mm", "mmhm", "mmhmm",
    "uh", "uhhuh", "huh",
    "um", "umm", "uhm",
    "er", "erm",
    "hmm", "hm",
    "ah", "oh",
    "yeah", "yep", "yup",
    "okay", "ok",
    "right", "alright", "gotcha",
})

_TRANSCRIPT_TYPES = {
    stt.SpeechEventType.INTERIM_TRANSCRIPT,
    stt.SpeechEventType.PREFLIGHT_TRANSCRIPT,
    stt.SpeechEventType.FINAL_TRANSCRIPT,
}

_PUNCT_STRIP = str.maketrans("", "", string.punctuation)

log = logging.getLogger("backchannel_stt_filter")

def _is_all_backchannel(text: str) -> bool:
    """Return True only when every token is a known backchannel."""
    tokens = text.lower().translate(_PUNCT_STRIP).split()
    return bool(tokens) and all(tok in BACKCHANNELS for tok in tokens)

class BackchannelSTTFilterMixin:
    """Drop backchannel-only transcripts while the agent is speaking."""

    _FILTER_GRACE_S: float = 1.0
    _last_speaking_at: float = 0.0

    async def stt_node(
        self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings
    ):
        async for ev in Agent.default.stt_node(self, audio, model_settings):
            if self._should_drop(ev):
                text = ev.alternatives[0].text if ev.alternatives else ""
                log.info(
                    "event_filtered transcript=%r ev_type=%s agent_state=%s",
                    text, ev.type, self.session.agent_state,
                )
                continue
            yield ev

    def _should_drop(self, ev: stt.SpeechEvent) -> bool:
        now = time.monotonic()
        if self.session.agent_state == "speaking":
            self._last_speaking_at = now
        elif now - self._last_speaking_at > self._FILTER_GRACE_S:
            return False

        if ev.type not in _TRANSCRIPT_TYPES:
            return False

        text = ev.alternatives[0].text if ev.alternatives else ""
        return _is_all_backchannel(text)
```

**How it works:**

1. While the agent is speaking (plus a 1-second grace window after speech ends), the filter inspects each STT transcript event
2. It strips punctuation and checks whether every token in the transcript matches the `BACKCHANNELS` set
3. Pure-filler transcripts like "mhm" or "yeah okay" are dropped. They never reach LiveKit's pipeline
4. Utterances with any non-filler token (e.g., "yeah I want the suite") always pass through

  The `BACKCHANNELS` set is domain-customizable. "yes" and "no" are deliberately excluded because they often represent genuine confirmations in booking or IVR scenarios. Edit the set for your use case.

LiveKit accumulates committed `FINAL` transcripts in a private buffer (`_audio_transcript`) across the user's uncommitted turn. The `_interrupt_by_audio_activity` method checks the running word count against `min_words` to decide whether to pause TTS.

Without intervention, two consecutive short fillers like "yeah" + "um" sum to two words and trip the interrupt gate, even though each utterance on its own is below threshold.

This filter listens on the `user_input_transcribed` event and wipes the buffer whenever the agent is speaking and the user input falls below the configured `min_words` threshold. Each short utterance is evaluated independently rather than against the accumulated total.

  This filter requires `interruption.min_words` to be set to `2` or higher. Without it, the word-count gate is disabled and the filter has no effect.

```python
from __future__ import annotations

import logging

from livekit.agents import AgentSession

log = logging.getLogger("short_utterance_buffer_filter")

def install_short_utterance_filter(session: AgentSession) -> None:
    """Clear transcript buffers when a short utterance arrives during agent speech."""

    @session.on("user_input_transcribed")
    def _on_user_input_transcribed(ev) -> None:
        word_count = len(ev.transcript.split())
        min_words = session.options.interruption["min_words"]

        if session.agent_state != "speaking":
            return

        if word_count >= min_words:
            return

        activity = getattr(session, "_activity", None)
        recognition = getattr(activity, "_audio_recognition", None) if activity else None
        if recognition is None:
            return

        # Wipe all three transcript buffers so short utterances
        # don't accumulate past the interrupt threshold.
        recognition._audio_transcript = ""
        recognition._audio_interim_transcript = ""
        recognition._audio_preflight_transcript = ""

        # Best-effort: abort any in-flight preemptive LLM call
        # triggered by this short utterance.
        cancel = getattr(activity, "_cancel_preemptive_generation", None)
        if callable(cancel):
            try:
                cancel()
            except Exception:
                log.debug("_cancel_preemptive_generation failed", exc_info=True)

        log.info(
            "buffer_cleared transcript=%r words=%d is_final=%s",
            ev.transcript, word_count, ev.is_final,
        )
```

  This filter accesses LiveKit private APIs (`_audio_transcript`, `_audio_interim_transcript`, `_audio_preflight_transcript`, `_cancel_preemptive_generation`). Pin your `livekit-agents` version to avoid breakage on minor updates. Tested with `livekit-agents>=1.5`.

Three changes are needed:

**1. Mix in the backchannel filter on your agent class.** Place it before `Agent` in the class bases so its `stt_node` runs first:

```python
from filters.backchannel_stt import BackchannelSTTFilterMixin

class MyAgent(BackchannelSTTFilterMixin, Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant.")
```

**2. Configure the session with `interruption.min_words >= 2`.** This enables the word-count gate that the buffer-clearing filter depends on:

```python
session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    # llm=your_llm_plugin(),
    # tts=your_tts_plugin(),
    vad=None,  # Recommended: disable VAD so only STT drives interruption
    turn_handling={
        "turn_detection": "stt",
        "endpointing": {"min_delay": 1.0, "max_delay": 4.0},
        "interruption": {
            "enabled": True,
            "resume_false_interruption": True,
            "false_interruption_timeout": 1.5,
            "min_words": 2,  # Required for the buffer-clearing filter
        },
    },
)
```

**3. Install the buffer-clearing filter on the session**, then start it with your agent:

```python
from filters.short_utterance_buffer import install_short_utterance_filter

install_short_utterance_filter(session)

await session.start(room=ctx.room, agent=MyAgent())
```

  Setting `vad=None` with `turn_detection="stt"` is the recommended setup for both filters. This ensures only STT-based signals drive interruption, avoiding timing races from a competing VAD interrupt path. If you need VAD for faster barge-in, both filters still work: set `interruption.min_words` to `2` and ensure Silero's `activation_threshold` matches `vad_threshold`.

**Expected behavior** with both filters active and `min_words=2`:

| User utterance (during agent speech) | Backchannel filter | Buffer-clearing filter | Result |
| ------------------------------------ | ------------------ | ---------------------- | ------ |
| "mhm"                               | Dropped            | -                      | Agent continues |
| "um"                                 | Dropped            | -                      | Agent continues |
| "yeah yeah"                          | Dropped            | -                      | Agent continues |
| Unknown short word                   | Passes through     | Buffer cleared         | Agent continues |
| "yeah I'd like the suite"            | Passes through     | Passes through         | Agent interrupts |
| "suite please"                       | Passes through     | Passes through         | Agent interrupts |
| "mhm" (1+ second after agent stops) | Passes through     | -                      | Agent responds |

## Dynamic configuration

You can update `prompt`, `keyterms_prompt`, `agent_context`, `language_codes`, `min_turn_silence`, `max_turn_silence`, `continuous_partials`, and `interruption_delay` during an active session using `update_options` — no reconnect required. This lets you adapt to the conversation stage: boost names while collecting caller details, widen silence windows while a user dictates an email, then tighten them again.

```python
# Update one or more options mid-stream
stt.update_options(
    max_turn_silence=3000,  # Increase for entity dictation
)

# Later, reset to default
stt.update_options(
    max_turn_silence=1000,
)
```

| Conversation stage                          | Adjustment                                                                              |
| ------------------------------------------- | --------------------------------------------------------------------------------------- |
| Caller identification (names, account IDs)  | Boost terms with `update_options(keyterms_prompt=[...])`                                 |
| Entity dictation (email, phone, address)    | Raise `max_turn_silence` to ~`2000`–`4000` ms, then lower it again afterward            |
| After each agent reply                      | Handled automatically on 1.6.6+ (see [Conversation context](#conversation-context))     |
| Caller switches language                    | `update_options(language_codes=["es"])`                                                  |
| Faster barge-in                             | Lower `interruption_delay` (see [Latency](#latency))                                     |

For more information, see [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream).

## Troubleshooting

| Issue                                     | Cause                                                     | Solution                                                                                             |
| ----------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Extra latency with `turn_detection="stt"` | LiveKit's endpointing `min_delay` is additive in STT mode | Set `endpointing={"min_delay": 0}` inside `TurnHandlingOptions` on `AgentSession`                    |
| No interruption handling                  | Missing VAD                                               | Ensure `vad=silero.VAD` is set, with `activation_threshold` equal to `vad_threshold` (default `0.3`) |
| Turn over-segmentation                    | `min_turn_silence` too low                                | Increase from `100` to `200`–`500`                                                                   |
| Entities split across turns               | `max_turn_silence` too low                                | Increase `max_turn_silence` (e.g., `1500`–`3500`)                                                    |
| Latency on non-terminal utterances        | `max_turn_silence` too high                               | Lower `max_turn_silence`                                                                             |
| Mis-heard names, brands, or jargon        | No vocabulary hints                                       | Add `keyterms_prompt`, or supply `prompt`/`agent_context` for context                               |
| Poor accuracy in noisy audio              | Background noise or room echo                             | Enable `voice_focus` (`near-field` or `far-field`)                                                   |

## Migrating from another STT provider

To balance accuracy, latency, turn-taking, and interruption handling, map your current setup to AssemblyAI using the questions below. Each answer points to the settings that reproduce — and usually improve on — your current behavior.

### How are you detecting end-of-turn today?

| Today | Recommended on AssemblyAI |
| --- | --- |
| Your STT provider's own end-of-turn model (e.g. Deepgram endpointing) | `turn_detection="stt"` with `min_turn_silence=100`, `max_turn_silence=1000`, `endpointing={"min_delay": 0}`. AssemblyAI's punctuation-based end-of-turn replaces it. |
| Silence / VAD only | Prefer `turn_detection="stt"` for semantic endpointing, or `turn_detection="vad"` to stay silence-only. Align Silero `activation_threshold` and `vad_threshold` at `0.3`. |
| LiveKit's turn-detector model | `turn_detection=MultilingualModel()`, keep the plugin defaults (`min_turn_silence=100`, `max_turn_silence=100`), set `endpointing={"min_delay": 0.5, "max_delay": 3.0}`, and keep Silero VAD (required). |

### Which model and settings are you migrating from?

| What you pass today | AssemblyAI equivalent |
| --- | --- |
| Current model (Deepgram, ElevenLabs, etc.) | `model="universal-3-5-pro"` (recommended flagship) |
| Overall accuracy/latency tuning | `mode="min_latency"` / `"balanced"` / `"max_accuracy"` — a one-line starting point before fine-tuning |
| Endpointing / silence thresholds | `min_turn_silence` (speculative end-of-turn) and `max_turn_silence` (forced end) — see [Turn detection](#turn-detection) |
| Custom vocabulary / keywords | `keyterms_prompt=[...]`; broader domain context → `prompt` |
| Formatting / punctuation toggles | On by default — formatted transcripts always (`format_turns` does not apply) |
| Language(s) | 18 languages with code-switching; pin one or more languages via `language_codes`; `language_detection` adds language codes |

### VAD, interruptions, and infrastructure

| Today | Recommended |
| --- | --- |
| Silero VAD | Keep it; set `activation_threshold=0.3` to match `vad_threshold` |
| Adaptive vs. VAD-based barge-in | LiveKit Cloud → [adaptive interruption handling](#recommended-adaptive-interruption-handling); self-hosted → the [self-hosted filters](#self-hosted-alternative) |
| Telephony / SIP routing | `sample_rate=8000` and `encoding="pcm_mulaw"` for 8 kHz telephony |
| Client-side noise cancellation | Drop it; use server-side [Voice Focus](#voice-focus) instead |
| LiveKit Cloud vs. self-hosting | Cloud unlocks adaptive interruption; self-hosting uses the filters and version pinning |

### Parameter cheat sheet

If you're coming from AssemblyAI's standard streaming model:

| Change                             | From                                       | To                                                                                       |
| ---------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------- |
| Model                              | `assemblyai.STT()`                         | `assemblyai.STT(model="universal-3-5-pro")`                                                      |
| Turn detection                     | `turn_detection="stt"` or `EnglishModel()` | `turn_detection="stt"` or `MultilingualModel()`                                          |
| VAD                                | Optional                                   | Set `vad=silero.VAD.load()` to match `vad_threshold`                                     |
| `min_turn_silence`                 | `400` (old default)                        | `100` (new default)                                                                      |
| `max_turn_silence`                 | `1280` (old default)                       | `1000` (API default) or `100` (with 3rd-party turn detector)                             |
| `end_of_turn_confidence_threshold` | Configurable                               | **Not applicable**: **Universal 3.5 Pro Realtime** uses punctuation-based turn detection |
| `endpointing.min_delay` (formerly `min_endpointing_delay`) | Default `0.5`                              | Set to `0` inside `TurnHandlingOptions` when using `turn_detection="stt"`                |

Migrating a production deployment? [Talk to our team](https://www.assemblyai.com/contact/sales).

---

<a id="universal-3-5-pro-realtime-on-pipecat-docs-voice-agents-pipecat-universal-3-5-pro-mdx"></a>

# Universal 3.5 Pro Realtime on Pipecat

URL: https://www.assemblyai.com/docs/voice-agents/pipecat-universal-3-5-pro
Source: docs/voice-agents/pipecat-universal-3-5-pro.mdx
Navigation: Real-time STT > Integrations
Description: Integrate AssemblyAI's Universal 3.5 Pro Realtime speech-to-text model into a Pipecat voice agent

## Overview

This guide covers integrating AssemblyAI's **Universal 3.5 Pro Realtime** speech-to-text model into a [Pipecat](https://docs.pipecat.ai/) voice agent. Everything here applies equally to **Universal-3.5 Pro Streaming** (`universal-3-5-pro`) — both belong to the same U3 Pro family and share every parameter in this guide, so you can swap the `model` string without changing anything else.

  **Universal 3.5 Pro Realtime is our flagship next-generation streaming model for voice agents** — multilingual and promptable, with [conversation context](#conversation-context) and [voice focus](#voice-focus).

  Available on **Pipecat 1.4.0+** — set `model="universal-3-5-pro"`.

AssemblyAI provides the speech-to-text and (optionally) the turn detection in your Pipecat pipeline:

```mermaid
flowchart LR
  U["User audio"] --> STT["AssemblyAI STT<br/>Universal 3.5 Pro Realtime"]
  STT --> TD["Turn detection<br/>Pipecat (VAD + Smart Turn)<br/>or AssemblyAI"]
  TD --> LLM["LLM"]
  LLM --> TTS["TTS"]
  TTS --> U
```

Once you have an agent running, tune it for what matters most to your use case:

  
    Decide when the user is done speaking — the two Pipecat modes, defaults, and entity tuning.
  
  
    Shorten the gap between the user finishing and the agent replying.
  
  
    Prompting, key terms, conversation context, and noise handling.
  
  
    Natural barge-in while the agent is speaking.
  

      <path d="M45.7718 70.7701C50.4477 69.0096 55.7252 70.3307 59.0204 74.0864L101.936 123.001H230.064L272.98 74.0864C276.275 70.3307 281.552 69.0096 286.228 70.7701C290.904 72.5306 294 77.0042 294 82.0005V190H332V214H270V113.873L244.52 142.915C242.242 145.512 238.955 147.001 235.5 147.001H96.5C93.0452 147.001 89.7581 145.512 87.4796 142.915L62 113.873V214H0V190H38V82.0005C38 77.0042 41.0958 72.5306 45.7718 70.7701Z" fill="currentColor" />
      <path d="M270 238.001H332V262.001H270V238.001Z" fill="currentColor" />
      <path d="M0 238.001H62V262.001H0V238.001Z" fill="currentColor" />
      <path d="M128 198.001C128 206.837 120.837 214.001 112 214.001C103.163 214.001 96 206.837 96 198.001C96 189.164 103.163 182.001 112 182.001C120.837 182.001 128 189.164 128 198.001Z" fill="currentColor" />
      <path d="M236 198.001C236 206.837 228.837 214.001 220 214.001C211.163 214.001 204 206.837 204 198.001C204 189.164 211.163 182.001 220 182.001C228.837 182.001 236 189.164 236 198.001Z" fill="currentColor" />
    </svg>
  }
  href="https://docs.pipecat.ai/server/services/stt/assemblyai"
>
  View Pipecat's AssemblyAI STT plugin reference.

  For a standalone voice agent without Pipecat or an external LLM, see the [AssemblyAI Voice Agent API](/voice-agents/voice-agent-api), which handles STT, LLM routing, and TTS in a single WebSocket connection.

## Quickstart

Get a working, talking agent in a few minutes, then optimize from there.

Install Pipecat with the AssemblyAI, LLM, and TTS extras you need:

```bash
pip install "pipecat-ai[assemblyai,openai,cartesia]" python-dotenv
```

**What's included:**

- `assemblyai`: AssemblyAI U3 Pro STT service
- `openai`: OpenAI LLM service (used in the example)
- `cartesia`: Cartesia TTS service (used in the example)

  The example uses OpenAI and Cartesia, but you can use any LLM or TTS supported
  by Pipecat — just swap the extras (e.g.,
  `pipecat-ai[assemblyai,anthropic,elevenlabs]`).

  Universal 3.5 Pro Realtime, automatic [conversation context](#conversation-context),
  and [Voice Focus](#voice-focus) require **`pipecat-ai` 1.4.0+**. Older versions
  won't recognize the `universal-3-5-pro` model.

Set your API keys in a `.env` file:

```env
ASSEMBLYAI_API_KEY=your_assemblyai_key
OPENAI_API_KEY=your_openai_key
CARTESIA_API_KEY=your_cartesia_key
```

  You can obtain an AssemblyAI API key by [signing up for a free
  account](https://www.assemblyai.com/dashboard/signup) and navigating to the [API
  Keys tab](https://www.assemblyai.com/dashboard/home) of the dashboard.

The example below uses Pipecat-controlled turn detection (the default). Pay attention to the comments for switching to AssemblyAI's built-in turn detection, and note that the **assistant aggregator** at the end of the pipeline is what enables automatic [conversation context](#conversation-context).

```python expandable
import os

from dotenv import load_dotenv
from loguru import logger

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMRunFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.base_transport import BaseTransport, TransportParams
from pipecat.transports.daily.transport import DailyParams
from pipecat.workers.runner import WorkerRunner

load_dotenv()

transport_params = {
    "daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
    "webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),
}

async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
    stt = AssemblyAISTTService(
        api_key=os.environ["ASSEMBLYAI_API_KEY"],
        settings=AssemblyAISTTService.Settings(
            model="universal-3-5-pro",
            min_turn_silence=100,
            # max_turn_silence is auto-synced to min_turn_silence in Pipecat mode.
            # vad_threshold=0.3,            # Align with your local VAD's threshold
            # continuous_partials=True,     # Default — steady ~3s partials during long turns
            # interruption_delay=0,         # Optional: faster first partial (~300ms effective)
        ),
        vad_force_turn_endpoint=True,  # Pipecat mode (default).
        # Set False to use AssemblyAI's built-in turn detection (universal-3-5-pro / universal-3-5-pro only):
        # vad_force_turn_endpoint=False,
    )

    llm = OpenAILLMService(api_key=os.environ["OPENAI_API_KEY"])
    tts = CartesiaTTSService(api_key=os.environ["CARTESIA_API_KEY"])

    context = LLMContext()
    user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
        context,
        user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
    )

    pipeline = Pipeline(
        [
            transport.input(),     # Transport user input
            stt,                   # STT
            user_aggregator,       # User responses
            llm,                   # LLM
            tts,                   # TTS
            transport.output(),    # Transport bot output
            assistant_aggregator,  # Assistant responses → automatic conversation context
        ]
    )

    worker = PipelineWorker(pipeline, params=PipelineParams(enable_metrics=True))

    @transport.event_handler("on_client_connected")
    async def on_client_connected(transport, client):
        context.add_message(
            {"role": "system", "content": "You are a helpful voice assistant. Keep replies brief and speakable."}
        )
        await worker.queue_frames([LLMRunFrame()])

    runner = WorkerRunner(handle_sigint=runner_args.handle_sigint)
    await runner.add_workers(worker)
    await runner.run()

async def bot(runner_args: RunnerArguments):
    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport, runner_args)

if __name__ == "__main__":
    from pipecat.runner.run import main

    main()
```

  Two complete, runnable examples live in the Pipecat repo:
  [voice-assemblyai.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-assemblyai.py)
  (Pipecat turn detection) and
  [voice-assemblyai-turn-detection.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-assemblyai-turn-detection.py)
  (AssemblyAI's built-in turn detection).

Run the agent directly with local audio:

```bash
python your_agent.py
```

Speak into your microphone after hearing the greeting. For WebRTC or Daily testing, see [Running your agent](#running-your-agent).

## Parameters reference

### **Universal 3.5 Pro Realtime** parameters

These are the key parameters to tune. Set them inside `AssemblyAISTTService.Settings(...)`. They apply to the whole U3 Pro family (`universal-3-5-pro` and `universal-3-5-pro`).

  The streaming model. `"universal-3-5-pro"` is the recommended flagship model;
  the plugin currently defaults to `"universal-3-5-pro"`, so set `model` explicitly. Both
  belong to the U3 Pro family and share every parameter below.

  Accuracy/latency preset: `"min_latency"`, `"balanced"`, or `"max_accuracy"`.
  Sets sensible defaults for mode-dependent fields; any value you set explicitly
  still takes precedence. The server defaults to `"balanced"`. Construction-time
  only. U3 Pro family only. See [Optimizing accuracy and
  latency](/streaming/getting-started/optimizing-accuracy-and-latency).

  List of terms to boost recognition for. Used on its own, your terms are
  appended to the default prompt automatically. Can't be set in the same request
  as `prompt` — see [Key terms](#key-terms) to combine boosting with a custom
  prompt.

  Contextual prompt — a natural-language description of what the audio is about
  (domain, scenario, or full details). Can't be set in the same request as
  `keyterms_prompt`; fold the terms into the prompt text instead (see
  [Key terms](#key-terms)). **Prompting is currently a beta feature**: see
  [Prompting](/streaming/prompting-and-keyterms) for more information.

  Context carryover seed — your agent's most recent spoken reply, up to ~1500
  characters, used to transcribe the next user turn more accurately. Set it at
  construction time to seed an opening greeting; later turns are fed
  automatically. U3 Pro family only. See [Conversation
  context](#conversation-context).

  How many prior conversation entries are carried forward automatically. Range
  `0`–`100`; `0` disables carryover entirely (including the automatic
  `agent_context` feed). Construction-time only; leave unset for the server
  default (`5` on `universal-3-5-pro`; `3` on older `u3-rt-pro`). U3 Pro family only.

  Milliseconds of silence before a speculative end-of-turn check. When the check
  fires, the model looks for terminal punctuation (`.` `?` `!`) to decide whether
  the turn has ended. (Formerly `min_end_of_turn_silence_when_confident`,
  deprecated but still supported with a warning.)

  Maximum silence before the turn is forced to end, regardless of punctuation.
  **Auto-synced to `min_turn_silence` in Pipecat mode**; respected as configured
  in AssemblyAI's built-in turn detection mode.

  AssemblyAI's internal VAD threshold (`0.0`–`1.0`) for classifying audio frames
  as silence. Align with your local VAD's activation threshold to avoid a "dead
  zone" where AssemblyAI transcribes speech your VAD hasn't detected yet.

  Server-side noise suppression that isolates the primary speaker.
  `"near-field"` for close-talking mics, `"far-field"` for distant capture.
  Construction-time only. U3 Pro family only. See [Voice focus](#voice-focus).

  How aggressively `voice_focus` suppresses background audio. `0.0`–`1.0`; higher
  is more aggressive. Only takes effect when `voice_focus` is set.
  Construction-time only. U3 Pro family only.

  Whether to emit additional partial transcripts during long turns at a steady
  ~3 second cadence. When enabled (default on both the API and this plugin),
  additional partials covering the full turn transcript are emitted
  approximately every 3 seconds while speech continues. When disabled, only one
  early partial is emitted near turn start. The first partial (at 750ms) is
  unaffected. Useful when downstream consumers (LLMs, UI, eager inference) need
  frequent updates during long, uninterrupted turns. See
  [Continuous partials](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  How soon the first partial transcript is emitted during a turn, in
  milliseconds. Range: `0`–`1000`. Lower values produce faster time to first
  token (TTFT) for barge-in and speculative inference; higher values produce
  more confident first partials. The server adds a minimum of 300ms on top of
  the configured value (`interruption_delay=0` → ~300ms effective,
  `interruption_delay=500` → ~800ms effective). See
  [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio)
  for details.

  **Universal 3.5 Pro Realtime** code-switches natively between supported
  languages. This parameter controls whether `language_code` and
  `language_confidence` are included in turn messages.

  Enable speaker diarization. See [Speaker diarization](#speaker-diarization).

### General parameters

These apply across models and Pipecat setups. `api_key`, `vad_force_turn_endpoint`, `should_interrupt`, and `speaker_format` are passed directly to `AssemblyAISTTService(...)`, not inside `Settings`.

  Your AssemblyAI API key.

  `True` for Pipecat mode (VAD + Smart Turn controls turns); `False` for
  AssemblyAI's built-in turn detection (`universal-3-5-pro` / `universal-3-5-pro` only).
  See [Turn detection](#turn-detection).

  Whether the user starting to speak interrupts the bot. Only applies in
  AssemblyAI's built-in turn detection mode (`vad_force_turn_endpoint=False`).

  Template string for formatting speaker labels (e.g., `"[{speaker}] {text}"`).
  Used with `speaker_labels`.

  The sample rate of the audio stream.

  The encoding of the audio stream. Allowed values: `pcm_s16le`, `pcm_mulaw`.

### Legacy parameters

These apply to the `universal-streaming-english` and `universal-streaming-multilingual` models, but **do not affect Universal 3.5 Pro Realtime or `universal-3-5-pro`**:

  Confidence threshold for end-of-turn detection. The U3 Pro family uses
  punctuation-based turn detection instead, so this parameter has no effect.

  Whether to return formatted final transcripts. The U3 Pro family always
  returns formatted transcripts, so this parameter no longer applies.

## Turn detection

In Pipecat, you choose **which component decides when the user is done speaking** with the `vad_force_turn_endpoint` flag on `AssemblyAISTTService`. The U3 Pro family uses a **punctuation-based** end-of-turn system: after a period of silence, the model checks for terminal punctuation (`.` `?` `!`) rather than a confidence score. For more on how this works, see [Configuring turn detection](/streaming/getting-started/transcribe-streaming-audio).

  The `vad_force_turn_endpoint` parameter controls which turn detection mode is
  used. It defaults to `True` (Pipecat mode), which sends a `ForceEndpoint`
  message to AssemblyAI when the local VAD detects silence. Set it to `False` to
  use AssemblyAI's built-in turn detection instead. Choosing the right mode is
  critical for balancing responsiveness and turn accuracy in your voice agent.

### Pipecat mode (default, recommended)

**When to use:** Most voice agent applications requiring responsive interruptions.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        min_turn_silence=100,
    ),
    vad_force_turn_endpoint=True,  # Default (Pipecat mode)
)
```

**How it works:**

- VAD + the Smart Turn analyzer control when the user is done speaking.
- A `ForceEndpoint` message is sent to AssemblyAI on VAD silence detection.
- `max_turn_silence` is **automatically synchronized** with `min_turn_silence`.
- Best for low-latency, responsive voice agents.

### AssemblyAI's built-in turn detection

**When to use:** When you want AssemblyAI's punctuation-based turn detection to control turn endings, configured through the settings below.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,  # Now respected independently
    ),
    vad_force_turn_endpoint=False,  # AssemblyAI's built-in turn detection
)
```

**How it works:**

1. User speaks → audio streams to AssemblyAI.
2. User pauses for `min_turn_silence` (e.g., `100ms`) → the model checks for terminal punctuation.
3. If terminal punctuation (`.` `?` `!`) is found → the turn ends immediately.
4. If not → a partial is emitted and the turn continues waiting.
5. If silence reaches `max_turn_silence` (e.g., `1000ms`) → the turn is forced to end regardless.

In this mode all timing parameters are respected as configured, the service emits `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame`, and `SpeechStarted` events drive fast barge-in. Only available with `universal-3-5-pro` / `universal-3-5-pro` (other models require Pipecat mode).

### Entity splitting tradeoff

Lower `min_turn_silence` and `max_turn_silence` values produce faster transcripts but can split entities or utterances across turns. The two parameters affect this differently.

#### `min_turn_silence` too low

The speculative check fires too early, splitting entities on punctuation:

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"It's John."                    → FINAL (100ms pause, check fires, period found → turn ends)
"Smith."                        → FINAL
"At gmail.com."                 → FINAL

# With (min_turn_silence=400, max_turn_silence=1000)
"It's john.smith@gmail.com."    → FINAL (single turn, properly formatted)
```

#### `max_turn_silence` too low

The forced turn-end cuts off the user mid-thought:

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"I wanted to check on my order from..."  → FINAL (1000ms silence, forced end)
"last Tuesday, order number 4829."       → FINAL (new turn)

# With (min_turn_silence=100, max_turn_silence=2000)
"I wanted to check on my order from last Tuesday, order number 4829."  → FINAL (single turn)
```

  **Universal 3.5 Pro Realtime's** formatting is significantly better when it has
  full context in a single turn — email addresses, phone numbers, credit card
  numbers, and physical addresses all benefit. If your use case involves
  alphanumeric dictation, raise `max_turn_silence` during those portions of the
  conversation (e.g., to `2000`–`4000` ms) using [dynamic
  configuration](#dynamic-configuration), then lower it again afterward. In
  Pipecat mode, raise `min_turn_silence` (which `max_turn_silence` follows) for
  the same effect.

## Latency

A voice agent feels responsive when the gap between the user finishing and the agent replying is short.

Start with the **`mode` preset** — the highest-level dial for the accuracy/latency trade-off. It sets sensible defaults for the fine-grained levers below, so you can pick a target and tune from there:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        mode="balanced",  # "min_latency" (fastest) · "balanced" · "max_accuracy" (best quality)
    ),
)
```

`mode` is set at construction time (it can't be changed mid-session) and influences the defaults of the levers below. Any value you set explicitly still wins. Leave it unset to use the server's default preset. See [Optimizing accuracy and latency](/streaming/getting-started/optimizing-accuracy-and-latency).

From there, fine-tune the individual levers:

- **End-of-turn timing.** `min_turn_silence` (speculative check) and `max_turn_silence` (forced end) directly control how soon a turn ends. Lower is faster but risks splitting entities — see [Turn detection](#turn-detection).
- **Time to first partial.** `interruption_delay` controls how soon the first partial is emitted, which drives faster barge-in and speculative inference. The server adds a minimum of `300ms` on top of the configured value.
- **Sample rate.** Use 16 kHz (`sample_rate=16000`). Higher rates don't improve accuracy and only add bandwidth.
- **Continuous partials.** `continuous_partials` (on by default) emits a partial every ~3 seconds during long turns. Leave it on for steady mid-turn updates, or disable it if you only need a single early partial.
- **Skip client-side preprocessing.** Don't run your own noise cancellation before audio reaches the model — the artifacts it introduces usually hurt accuracy more than the original noise. Use server-side [Voice Focus](#voice-focus) instead.

### Latency breakdown

| Stage                                    | Typical                                              | Controlled by        |
| ---------------------------------------- | ---------------------------------------------------- | -------------------- |
| Network round trip                       | ~50 ms                                               | —                    |
| Speech-to-text                           | ~200–300 ms                                          | model                |
| First partial (TTFT)                     | configured `interruption_delay` + ~300 ms server min | `interruption_delay` |
| End of turn (terminal punctuation found) | `min_turn_silence` (default 100 ms)                  | `min_turn_silence`   |
| End of turn (no punctuation, forced)     | up to `max_turn_silence`                             | `max_turn_silence`   |

## Accuracy

**Universal 3.5 Pro Realtime** is accurate out of the box. When you need more — domain vocabulary, proper nouns, noisy audio — reach for these levers. For entity-heavy dictation, also tune turn detection (see [Entity splitting tradeoff](#entity-splitting-tradeoff)), and note that the high-level [`mode` preset](#latency) shifts the overall accuracy/latency balance (use `max_accuracy` to favor quality).

### Prompting

  **Beta feature**

  Prompting is considered a beta feature for **Universal 3.5 Pro Realtime**.

  While it can be a powerful tool for improving accuracy in certain use cases,
  **we recommend starting without a `prompt` to first establish baseline
  performance.** Once the baseline has been tested, you can add context to
  further optimize for your use case (e.g., language mix to expect, use case or
  domain).

**Universal 3.5 Pro Realtime** supports a `prompt` parameter for [contextual prompting](/streaming/prompting-and-keyterms) — a description of what the audio is about. Transcription behavior (verbatim output, punctuation, turn detection) is built in and optimized automatically; the prompt carries context, not instructions.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        prompt="Customer support call about an internet service outage.",
    ),
)
```

### Key terms

Use `keyterms_prompt` to boost recognition of specific names, brands, or domain terms. On its own, your terms are appended to the default prompt automatically — so you get boosting and prompting together:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        keyterms_prompt=["Xiomara", "Saoirse", "Pipecat", "AssemblyAI"],
    ),
)
```

  You can't pass `prompt` and `keyterms_prompt` in the same request — doing so
  raises a validation error. You don't have to give up term boosting to use a
  contextual prompt, though. Either:

  - Pass **`keyterms_prompt` on its own** — your terms are appended to the
    default prompt automatically, or
  - Fold the terms into a custom **`prompt`**, e.g. end it with
    `"Make sure to boost the words Xiomara, Saoirse, Pipecat in the audio."`

### Conversation context

Give the model both sides of the dialog so it transcribes the next user turn more accurately. **Universal 3.5 Pro Realtime** keeps a short, per-session memory of the conversation from two sources:

- **The agent half** — what your agent just said.
- **The user half** — prior STT-finalized user turns.

With the agent's question in context, the model can anticipate the answer, sharpen entity recognition, and disambiguate similar-sounding words. For example, after your agent asks `"What's your email address?"`, the model can produce `"user@assemblyai.com"` instead of `"user at assemblyai dot com"`. This has the biggest impact on short replies (`"yes"`, `"7pm"`, single names) and spelled-out entities. See [Conversation context](/streaming/universal-3-5-pro/context-carryover) for the full reference.

  **In Pipecat, conversation context is automatic — no event wiring required.**
  As long as your pipeline includes the standard LLM context aggregator (the
  `assistant_aggregator` from `LLMContextAggregatorPair`), Pipecat broadcasts an
  `LLMContextAssistantTurnFrame` when each bot turn completes, and
  `AssemblyAISTTService` feeds that reply to the model as `agent_context`
  automatically. Just use a U3 Pro family model on `pipecat-ai` 1.4.0+.

| Parameter                  | Type | Description                                                                                                                                                              |
| -------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `agent_context`            | str  | Your agent's most recent spoken reply, up to ~1500 characters. Set it at construction time to seed an opening greeting; subsequent replies are fed automatically.        |
| `previous_context_n_turns` | int  | How many prior conversation entries are carried forward automatically. Range `0`–`100`; `0` disables carryover entirely. Construction-time only; server default is `5` on `universal-3-5-pro` (`3` on older `u3-rt-pro`). |

#### Seeding the opening greeting

The automatic feed kicks in once your agent completes its first turn. To give the model context for the user's *very first* reply (the answer to your greeting), set `agent_context` at construction time:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        # Seed the opening line; later turns are fed automatically by the aggregator.
        agent_context="Hi! Thanks for calling Acme. What's the email on your account?",
        # previous_context_n_turns=5,  # Default on universal-3-5-pro. Set 0 to disable carryover entirely.
    ),
)
```

#### Manual control with `update_agent_context()`

If your pipeline doesn't use the standard LLM context aggregator, or you want explicit control over what the model sees, push the agent's reply yourself. This is a **live update — no reconnect required**:

```python
# Whenever your agent finishes speaking:
await stt.update_agent_context("Your account is past due. Would you like to pay now?")
```

  `agent_context`, `previous_context_n_turns`, and `update_agent_context()` are
  supported only on the U3 Pro family (`universal-3-5-pro`, `universal-3-5-pro`). Values
  are clipped to ~1500 characters and re-seeded automatically on reconnect.
  Setting `previous_context_n_turns=0` disables the automatic feed as well.

### Voice focus

Voice Focus isolates the primary speaker and suppresses background noise — chatter, keyboard clicks, fan hum, room echo — **server-side, before audio reaches the model**. Use it instead of client-side noise cancellation, which tends to introduce artifacts that hurt accuracy more than the noise itself.

| Parameter               | Type   | Description                                                                                                                                       |
| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `voice_focus`           | str    | `"near-field"` for headsets, handsets, and other close-talking mics; `"far-field"` for conference rooms, laptop mics, and other distant capture. |
| `voice_focus_threshold` | float  | Optional. `0.0`–`1.0`; higher values suppress background audio more aggressively.                                                                |

Both are construction-time parameters on the U3 Pro family. See [Voice Focus](/streaming/voice-focus) for details.

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        voice_focus="far-field",    # "near-field" for close-talking mics
        voice_focus_threshold=0.5,  # Optional: 0.0–1.0, higher = more aggressive
    ),
)
```

## Interruption handling

Barge-in — the user interrupting while the agent is speaking — is handled by Pipecat, and the signals that drive it depend on your turn detection mode.

- **Pipecat mode (`vad_force_turn_endpoint=True`).** Pipecat's local VAD and the Smart Turn analyzer detect the user starting to speak and interrupt the bot's TTS. AssemblyAI also emits `SpeechStarted` events as a backstop.
- **AssemblyAI's built-in turn detection (`vad_force_turn_endpoint=False`).** The service emits `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame` and uses AssemblyAI's `SpeechStarted` events for fast barge-in. Set `should_interrupt=False` (constructor argument) to disable barge-in entirely in this mode.

```json
{"type": "SpeechStarted", "timestamp": 14400, "confidence": 0.79}
```

On detection, Pipecat stops TTS playback and switches to listening. To reduce false interruptions from short backchannels (`"mhm"`, `"yeah"`, `"okay"`), keep your VAD threshold aligned with `vad_threshold` and lean on Pipecat's Smart Turn analyzer, which evaluates whether speech is a genuine turn rather than a filler.

## Dynamic configuration

Update settings mid-conversation by queueing an `STTUpdateSettingsFrame` with a settings delta — adapt to the conversation stage as it unfolds. See [stt-assemblyai.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/update-settings/stt/stt-assemblyai.py) for a complete working example.

```python expandable
from pipecat.frames.frames import STTUpdateSettingsFrame
from pipecat.services.assemblyai.stt import AssemblyAISTTService

# Update keyterms during the conversation
await worker.queue_frame(
    STTUpdateSettingsFrame(
        delta=AssemblyAISTTService.Settings(
            keyterms_prompt=["NewName", "NewCompany"],
        )
    )
)

# Widen the silence window during entity dictation
await worker.queue_frame(
    STTUpdateSettingsFrame(
        delta=AssemblyAISTTService.Settings(
            min_turn_silence=200,
            max_turn_silence=3000,  # Respected in AssemblyAI's built-in turn detection mode
        )
    )
)
```

  **`agent_context` is the only setting applied live.** Changing any other
  setting via `STTUpdateSettingsFrame` reconnects the AssemblyAI session to
  apply it (a brief interruption). To push conversation context without a
  reconnect, use the dedicated `stt.update_agent_context(...)` method — see
  [Conversation context](#conversation-context).

| Conversation stage                         | Adjustment                                                                       |
| ------------------------------------------ | -------------------------------------------------------------------------------- |
| Caller identification (names, account IDs) | Boost terms with `keyterms_prompt`                                               |
| Entity dictation (email, phone, address)   | Raise `max_turn_silence` to ~`2000`–`4000` ms, then lower it again afterward     |
| After each agent reply                     | Automatic — or push `agent_context` via `update_agent_context()`                 |
| Faster barge-in                            | Lower `interruption_delay`                                                       |

For more information, see [Updating configuration mid-stream](/streaming/updating-configuration-mid-stream).

## Speaker diarization

Identify different speakers in multi-party conversations.

### Basic diarization

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        speaker_labels=True,
    ),
)
```

Speaker labels (e.g., `"A"`, `"B"`, `"C"`) are included in final transcripts.

### With custom formatting

Format transcripts with speaker labels for LLM context:

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        speaker_labels=True,
    ),
    speaker_format="<{speaker}>{text}</{speaker}>",
)
```

**Format options:**

| Style    | Format string                   |
| -------- | ------------------------------- |
| XML      | `<{speaker}>{text}</{speaker}>` |
| Markdown | `**{speaker}**: {text}`         |
| Bracket  | `[{speaker}] {text}`            |

## Running your agent

### Development mode (local audio)

```bash
python your_agent.py
```

Speak into your microphone after hearing the greeting.

### Production with Daily

For production deployments, use the [Daily transport](https://docs.pipecat.ai/server/services/transport/daily) for WebRTC-based real-time audio/video. Your agent joins a Daily room as a participant and handles audio I/O through Daily's infrastructure.

### Telephony with Telnyx

When bridging phone calls through Pipecat (e.g., via Telnyx), the audio is 8 kHz, not 16 kHz. Match the transport sample rates:

```python
transport = TelnyxTransport(
    # ...
    audio_in_sample_rate=8000,
    audio_out_sample_rate=8000,
)
```

## Troubleshooting

| Issue                                | Cause                                                | Solution                                                                                          |
| ------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `universal-3-5-pro` not recognized   | `pipecat-ai` older than 1.4.0                        | Upgrade: `pip install -U "pipecat-ai[assemblyai]"`                                                |
| Turn over-segmentation               | `min_turn_silence` too low                           | Increase from `100` to `200`–`500`                                                                |
| Entities split across turns          | `max_turn_silence` too low (AssemblyAI mode)         | Increase `max_turn_silence` (e.g., `1500`–`3500`); in Pipecat mode, raise `min_turn_silence`      |
| Latency on non-terminal utterances   | `max_turn_silence` too high                          | Lower `max_turn_silence`                                                                          |
| Conversation context has no effect   | Non-U3-Pro model, or `previous_context_n_turns=0`    | Use a U3 Pro family model and leave `previous_context_n_turns` unset (or > 0)                      |
| Mid-session setting change drops audio | Reconnect on a non-`agent_context` setting change  | Expected — only `agent_context` updates live; use `update_agent_context()` for context            |
| Mis-heard names, brands, or jargon   | No vocabulary hints                                  | Add `keyterms_prompt`, or supply `prompt`/`agent_context` for context                             |
| Poor accuracy in noisy audio         | Background noise or room echo                        | Enable `voice_focus` (`near-field` or `far-field`)                                                |

## Migrating from another STT provider

To balance accuracy, latency, turn-taking, and interruption handling, map your current setup to AssemblyAI using the questions below.

### How are you detecting end-of-turn today?

| Today                                               | Recommended on AssemblyAI                                                                                                                |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Your STT provider's own end-of-turn model           | AssemblyAI's built-in turn detection: `vad_force_turn_endpoint=False` with `min_turn_silence=100`, `max_turn_silence=1000`.               |
| Silence / VAD only, with your own turn logic        | Pipecat mode (`vad_force_turn_endpoint=True`, default). VAD + Smart Turn decide turns; AssemblyAI returns finals ASAP.                    |
| You want the framework to own turn-taking           | Pipecat mode (default) — Pipecat's Smart Turn analyzer makes the turn decision.                                                          |

### Which model and settings are you migrating from?

| What you pass today                        | AssemblyAI equivalent                                                                                  |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| Current model (Deepgram, ElevenLabs, etc.) | `model="universal-3-5-pro"` (recommended flagship) or `"universal-3-5-pro"`                                     |
| Overall accuracy/latency tuning            | `mode="min_latency"` / `"balanced"` / `"max_accuracy"` — a one-line starting point before fine-tuning  |
| Endpointing / silence thresholds           | `min_turn_silence` (speculative end-of-turn) and `max_turn_silence` (forced end)                       |
| Custom vocabulary / keywords               | `keyterms_prompt=[...]`; broader domain context → `prompt`                                             |
| Provider-side conversation context         | Automatic — include the LLM context aggregator; seed greetings via `agent_context`                     |
| Formatting / punctuation toggles           | On by default — formatted transcripts always (`format_turns` does not apply)                           |
| Telephony / SIP routing                    | `sample_rate=8000` and `encoding="pcm_mulaw"` for 8 kHz telephony                                      |
| Client-side noise cancellation             | Drop it; use server-side [Voice Focus](#voice-focus) instead                                           |

Migrating a production deployment? [Talk to our team](https://www.assemblyai.com/contact/sales).

## Speech model comparison

Interested in using a different model?

| Feature                            | U3 Pro family <br/>(`universal-3-5-pro`, `universal-3-5-pro`) | universal-streaming-english | universal-streaming-multilingual |
| ---------------------------------- | ----------------------------------------------------- | --------------------------- | -------------------------------- |
| **_Turn Detection Modes_**         |                                                       |                             |                                  |
| Pipecat mode (VAD + Smart Turn)    | ✅                                                    | ✅                          | ✅                               |
| AssemblyAI turn detection mode     | ✅                                                    | ❌                          | ❌                               |
| **_Turn Detection Parameters_**    |                                                       |                             |                                  |
| `min_turn_silence`                 | ✅                                                    | ✅                          | ✅                               |
| `max_turn_silence`                 | ✅                                                    | ✅                          | ✅                               |
| `end_of_turn_confidence_threshold` | ❌                                                    | ✅ (1.0)                    | ✅ (1.0)                         |
| `continuous_partials`              | ✅                                                    | ❌                          | ❌                               |
| `interruption_delay`               | ✅                                                    | ❌                          | ❌                               |
| **_Advanced Features_**            |                                                       |                             |                                  |
| Keyterms boosting                  | ✅                                                    | ✅                          | ✅                               |
| Custom prompting (beta)            | ✅                                                    | ❌                          | ❌                               |
| Conversation context (carryover)   | ✅                                                    | ❌                          | ❌                               |
| Voice Focus                        | ✅                                                    | ❌                          | ❌                               |
| Speaker diarization                | ✅                                                    | ✅                          | ✅                               |
| Dynamic parameter updates          | ✅                                                    | ✅                          | ✅                               |
| **_Language Support_**             |                                                       |                             |                                  |
| Multilingual code switching        | ✅                                                    | ❌                          | ✅                               |
| Language detection                 | ✅                                                    | ❌                          | ✅                               |

**Legend:**

- ✅ Fully supported and recommended
- ❌ Not supported / Not used

  **The U3 Pro family is recommended** for all new voice agent implementations.
  The universal-streaming models are maintained for backward compatibility but
  lack the optimizations and features specifically designed for real-time
  conversational AI.

  The `end_of_turn_confidence_threshold` parameter is **not used** with the U3
  Pro family (it won't affect behavior). For universal-streaming models, Pipecat
  automatically sets it to `1.0` in Pipecat mode to disable semantic turn
  detection and ensure fast responses. You don't need to configure this
  parameter manually.

---

<a id="streaming-migration-guide-universal-streaming-to-universal-3-5-pro-streaming-docs-streaming-migration-guides-universal-to-universal-3-5-pro-streaming-mdx"></a>

# Streaming Migration Guide: Universal Streaming to Universal-3.5 Pro Streaming

URL: https://www.assemblyai.com/docs/streaming/migration-guides/universal-to-universal-3-5-pro-streaming
Source: docs/streaming/migration-guides/universal-to-universal-3-5-pro-streaming.mdx
Navigation: Real-time STT > Guides > Migration guides
Description: Streaming Migration Guide: Universal Streaming to Universal-3.5 Pro Streaming documentation.

This guide walks through the process of upgrading from Universal Streaming to Universal-3.5 Pro Streaming for real-time audio transcription.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home).

## Quick upgrade

If you're already using Universal Streaming, you can quickly test Universal-3.5 Pro Streaming by switching the `speech_model` parameter to `"universal-3-5-pro"` and removing `format_turns` (formatting is always on in U3.5 Pro). Just update the connection params and start streaming.

```python
# Before (Universal Streaming)
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,
}

# After (Universal-3.5 Pro Streaming)
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
}
```

  **That's it for a quick test.** But there are important behavioral differences
  in turn detection, partials, and formatting that may require updates to your
  message handling logic. Read on for the full migration details.

## Why upgrade

Universal-3.5 Pro Streaming delivers:

- **Exceptional entity accuracy** — credit card numbers, phone numbers, email addresses, physical addresses, and names captured correctly at streaming speed
- **Promptable model** — contextual prompting via `prompt` (describe what the audio is about), plus domain-term boosting via `keyterms_prompt` (up to 100 terms)
- **Better turn detection** — punctuation-based system that waits when speakers pause mid-thought and responds when they're done
- **Native multilingual code-switching** — English, Spanish, German, French, Portuguese, Italian in a single model
- **Sub-300ms latency** — fast time to complete transcript
- **Mid-stream configuration** — update keyterms, prompts, and silence parameters without dropping the connection

For full details, see [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio).

## What changes

This table covers the key parameter, behavior, and response field differences. Use it as a migration checklist.

| What                                    | Universal Streaming                                                                                                                                             | Universal-3.5 Pro Streaming                                                     | Action Required                                                                                              |
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| **`speech_model`**                      | Not required (defaults to English)                                                                                                                              | `"universal-3-5-pro"`                                                                 | Add `speech_model: "universal-3-5-pro"` to connection params                                                         |
| **`format_turns`**                      | `false` by default; set `true` for formatted transcripts                                                                                                        | Always on (not a parameter)                                                   | Remove `format_turns` from connection params                                                                 |
| **Turn detection**                      | Confidence-based (`end_of_turn_confidence_threshold`, default `0.4` — **officially deprecated**)                                                                | Punctuation-based (`min_turn_silence` + terminal punctuation)                 | Remove `end_of_turn_confidence_threshold` (deprecated); tune `min_turn_silence` / `max_turn_silence` instead |
| **`min_turn_silence`**                  | `400` ms (minimum silence before checking confidence)                                                                                                           | `100` ms (silence before speculative EOT check)                               | Review and adjust if you tuned this value                                                                    |
| **`max_turn_silence`**                  | `1280` ms                                                                                                                                                       | `1000` ms                                                                     | Review and adjust if you tuned this value                                                                    |
| **`end_of_turn` / `turn_is_formatted`** | The model has built in formatting so `turn_is_formatted` is `true` on all turns including partials — do not use it as a turn-end signal. Use `end_of_turn: true` to detect when a turn has completed. | Always the same value — one end-of-turn transcript per turn, always formatted | Simplify: just check `end_of_turn: true` for the final formatted transcript                                  |
| **Partials**                            | Emitted frequently during speech (unformatted on English model, formatted on multilingual model)                                                                | Early partial at ~750ms, silence-based partials, plus continuous partials every ~3s during long turns (`continuous_partials` enabled by default) | Expect stable, fully-transcribed partials rather than word-by-word updates                                   |
| **`prompt`**                            | Not supported                                                                                                                                                   | Supported — contextual prompting (describe the audio)                                 | New capability (optional)                                                                                    |
| **`keyterms_prompt`**                   | Supported (connection-time only; not updatable mid-stream)                                                                                                      | Supported; can be used together with `prompt`; updatable mid-stream           | No change needed; new: can combine with `prompt` and update via `UpdateConfiguration`                        |
| **`UpdateConfiguration`**               | Turn detection params only (`end_of_turn_confidence_threshold`, `min_turn_silence`, `max_turn_silence`)                                                         | `prompt`, `keyterms_prompt`, `min_turn_silence`, `max_turn_silence`, `agent_context` | Update any mid-stream config logic to use new fields                                                         |
| **`ForceEndpoint`**                     | Supported                                                                                                                                                       | Supported                                                                     | No change needed                                                                                             |
| **`language`**                          | `"en"` or `"multi"` (**officially deprecated**)                                                                                                                 | Not a parameter (native code-switching)                                       | Remove `language` param; use `language_codes` to bias toward the languages you expect (a single-element list for one language) |
| **`vad_threshold`**                     | `0.4` (default)                                                                                                                                                 | `0.3` (default)                                                               | Review and adjust if you tuned this value — lower default means higher noise sensitivity                    |
| **`language_detection`**                | Supported (`true`/`false`, default `false`) with multilingual model                                                                                             | Supported — automatic with code-switching                                     | Remove if set; U3.5 Pro detects language automatically                                                        |
| **Languages**                           | English default; multilingual requires `speech_model: "universal-streaming-multilingual"`                                                                       | Native multilingual code switching (6 languages) in a single model            | Remove multilingual model switching; optionally pass `language_codes` to bias toward the languages you expect |

Sources: [U3.5 Pro docs](/streaming/getting-started/transcribe-streaming-audio), [Universal docs](/streaming/getting-started/transcribe-streaming-audio), [Turn detection docs](/streaming/getting-started/transcribe-streaming-audio), [API Reference](/streaming/api-spec/streaming-websocket)

## Side-by-side code

Full working Python examples side by side using raw `websocket-client`.

    

    ```python expandable
    import pyaudio
    import websocket
    import json
    import threading
    import time
    from urllib.parse import urlencode

    YOUR_API_KEY = "<YOUR_API_KEY>"

    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "format_turns": True,
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    FRAMES_PER_BUFFER = 800
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()

    def on_open(ws):
        print("WebSocket connection opened.")
        def stream_audio():
            global stream
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    break

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = True
        audio_thread.start()

    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get("type")

            if msg_type == "Begin":
                print(f"Session began: ID={data.get('id')}")
            elif msg_type == "Turn":
                transcript = data.get("transcript", "")
                if data.get("end_of_turn"):
                    print(f"\r{' ' * 80}\r{transcript}")
                else:
                    print(f"\r{transcript}", end="")
            elif msg_type == "Termination":
                print(f"\nSession terminated: {data.get('audio_duration_seconds', 0)}s of audio")
        except Exception as e:
            print(f"Error handling message: {e}")

    def on_error(ws, error):
        print(f"\nWebSocket Error: {error}")
        stop_event.set()

    def on_close(ws, close_status_code, close_msg):
        print(f"\nWebSocket Disconnected: Status={close_status_code}")
        global stream, audio
        stop_event.set()
        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
        if audio:
            audio.terminate()

    def run():
        global audio, stream, ws_app

        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Speak into your microphone. Press Ctrl+C to stop.")

        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nStopping...")
            stop_event.set()
            if ws_app and ws_app.sock and ws_app.sock.connected:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

    if __name__ == "__main__":
        run()
    ```
    

    

    ```python expandable
    import pyaudio
    import websocket
    import json
    import threading
    import time
    from urllib.parse import urlencode

    YOUR_API_KEY = "<YOUR_API_KEY>"

    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "speech_model": "universal-3-5-pro",
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    FRAMES_PER_BUFFER = 800
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()

    def on_open(ws):
        print("WebSocket connection opened.")
        def stream_audio():
            global stream
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    break

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = True
        audio_thread.start()

    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get("type")

            if msg_type == "Begin":
                print(f"Session began: ID={data.get('id')}")
            elif msg_type == "Turn":
                transcript = data.get("transcript", "")
                end_of_turn = data.get("end_of_turn", False)
                if end_of_turn:
                    print(f"\r{' ' * 80}\r{transcript}")
                else:
                    print(f"\r{transcript}", end="")
            elif msg_type == "Termination":
                print(f"\nSession terminated: {data.get('audio_duration_seconds', 0)}s of audio")
        except Exception as e:
            print(f"Error handling message: {e}")

    def on_error(ws, error):
        print(f"\nWebSocket Error: {error}")
        stop_event.set()

    def on_close(ws, close_status_code, close_msg):
        print(f"\nWebSocket Disconnected: Status={close_status_code}")
        global stream, audio
        stop_event.set()
        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
        if audio:
            audio.terminate()

    def run():
        global audio, stream, ws_app

        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Speak into your microphone. Press Ctrl+C to stop.")

        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nStopping...")
            stop_event.set()
            if ws_app and ws_app.sock and ws_app.sock.connected:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

    if __name__ == "__main__":
        run()
    ```
    

## Turn detection

This is the most significant behavioral difference between the two models.

**Universal Streaming** uses a **confidence-based** system combining semantic and acoustic detection ([source](/streaming/getting-started/transcribe-streaming-audio)):

| Parameter                          | Default   | Description                                                                       |
| ---------------------------------- | --------- | --------------------------------------------------------------------------------- |
| `end_of_turn_confidence_threshold` | `0.4`     | Confidence threshold (0.0-1.0) to trigger end of turn (**officially deprecated**) |
| `min_turn_silence`                 | `400` ms  | Minimum silence before checking confidence                                        |
| `max_turn_silence`                 | `1280` ms | Maximum silence before forcing end of turn                                        |

The model evaluates `end_of_turn_confidence` during silence. If the score exceeds `end_of_turn_confidence_threshold` after `min_turn_silence`, the turn ends. Otherwise, the turn is forced to end after `max_turn_silence`.

**Universal-3.5 Pro** uses a **punctuation-based** system ([source](/streaming/getting-started/transcribe-streaming-audio)):

| Parameter          | Default   | Description                                          |
| ------------------ | --------- | ---------------------------------------------------- |
| `min_turn_silence` | `100` ms  | Silence before a speculative end-of-turn check fires |
| `max_turn_silence` | `1000` ms | Maximum silence before a turn is forced to end       |

When silence reaches `min_turn_silence`, the model transcribes the audio and checks for terminal punctuation (`.` `?` `!`):

- **Terminal punctuation found** — the turn ends (`end_of_turn: true`)
- **No terminal punctuation** — a partial is emitted (`end_of_turn: false`) and the turn continues
- **Silence reaches `max_turn_silence`** — the turn is forced to end regardless of punctuation

  **`end_of_turn_confidence_threshold` does not exist on Universal-3.5 Pro** (it
  was never part of the U3.5 Pro API — not deprecated, just absent). It is
  officially deprecated on Universal Streaming. Remove this parameter and
  configure `min_turn_silence` and `max_turn_silence` instead. For configuration
  guidance, see [Configuring Turn
  Detection](/streaming/getting-started/transcribe-streaming-audio).

## New capabilities

These features are new or enhanced in Universal-3.5 Pro. For full details, see [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio).

### Prompting

Universal-3.5 Pro supports a `prompt` parameter for contextual prompting — a natural-language description of what the audio is about (domain, scenario, or full details). Transcription behavior itself is built in and optimized for streaming and turn detection. See the [Prompting Guide](/streaming/prompting-and-keyterms) for details.

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "prompt": "Customer support call about an internet service outage.",
}
```

  **Start with no prompt.** Universal-3.5 Pro is optimized out of the box. Add
  context when domain-specific vocabulary is being misrecognized, starting with
  the broadest description that fits your use case.

### Keyterms prompting

Boost recognition of specific names, brands, or domain terms. Maximum 100 keyterms, each 50 characters or less. See [Keyterms Prompting](/streaming/prompting-and-keyterms) for details.

```python
import json

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "keyterms_prompt": json.dumps(["Keanu Reeves", "AssemblyAI", "Universal-3"]),
}
```

  **`prompt` and `keyterms_prompt` can be used together.** Use `prompt` to
  describe the conversation and `keyterms_prompt` to enumerate the specific
  terms that matter — they are complementary.

### Mid-stream configuration updates

Update `prompt`, `keyterms_prompt`, `min_turn_silence`, and `max_turn_silence` during an active session without reconnecting. See [Updating configuration mid-stream](/streaming/getting-started/transcribe-streaming-audio) for details.

```python
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["cardiology", "echocardiogram", "Dr. Patel"],
    "max_turn_silence": 5000
}))
```

## Force turn end

`ForceEndpoint` is supported on both Universal Streaming and Universal-3.5 Pro — no migration changes needed. Force the current turn to end immediately based on external signals. See [Forcing a turn endpoint](/streaming/getting-started/transcribe-streaming-audio) for details.

```python
ws.send(json.dumps({"type": "ForceEndpoint"}))
```

## Language support

**Universal Streaming** transcribes English by default. For multilingual support, use `speech_model: "universal-streaming-multilingual"`. ([Source](/streaming/getting-started/transcribe-streaming-audio))

**Universal-3.5 Pro** natively code-switches between 18 languages in a single model — no separate multilingual model needed: English, Spanish, German, French, Portuguese, Italian, Turkish, Dutch, Swedish, Norwegian, Danish, Finnish, Hindi, Vietnamese, Arabic, Hebrew, Japanese, and Mandarin. It also supports automatic language detection, returning `language_code` and `language_confidence` fields in Turn messages. To bias toward the languages you expect, pass the `language_codes` connection parameter with a list of codes (a single-element list for one language) ([Language selection](/streaming/getting-started/optimizing-accuracy-and-latency#language-selection)). See [Supported languages](/streaming/getting-started/transcribe-streaming-audio) for the full list.

**Language Detection:** Universal Streaming supports the `language_detection` connection parameter (`true`/`false`, default `false`) with the multilingual model. When enabled, Turn messages include `language_code` and `language_confidence` fields. Universal-3.5 Pro also supports language detection with code-switching — see [Supported languages](/streaming/getting-started/transcribe-streaming-audio) for details.

## Resources

- [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio)
- [Universal Streaming](/streaming/getting-started/transcribe-streaming-audio)
- [Turn Detection (Universal)](/streaming/getting-started/transcribe-streaming-audio)
- [Prompting Guide (Streaming)](/streaming/prompting-and-keyterms)
- [Keyterms Prompting](/streaming/prompting-and-keyterms)
- [Supported Languages](/streaming/getting-started/transcribe-streaming-audio)
- [Streaming API Reference](/streaming/api-spec/streaming-websocket)

---

<a id="streaming-migration-guide-deepgram-to-assemblyai-docs-streaming-migration-guides-dg-to-aai-streaming-mdx"></a>

# Streaming Migration Guide: Deepgram to AssemblyAI

URL: https://www.assemblyai.com/docs/streaming/migration-guides/dg_to_aai_streaming
Source: docs/streaming/migration-guides/dg_to_aai_streaming.mdx
Navigation: Real-time STT > Guides > Migration guides
Description: Streaming Migration Guide: Deepgram to AssemblyAI documentation.

This guide walks through the process of migrating from Deepgram to AssemblyAI for transcribing streaming audio.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Side-By-Side Code Comparison

Below is a side-by-side comparison of a basic snippet to transcribe live audio by Deepgram and AssemblyAI using a microphone:

    

    ```python expandable
    import json, threading, time, websocket, pyaudio, signal
    from urllib.parse import urlencode

    DG_KEY = "YOUR_DG_API_KEY"

    PARAMS = {
        "model": "nova-3",
        "encoding": "linear16",
        "sample_rate": "16000",
        "channels": "1",
        "punctuate": "true",
        "interim_results": "true",
    }

    EP = f"wss://api.deepgram.com/v1/listen?{urlencode(PARAMS)}"

    FRAMES      = 800
    CHANNELS    = 1
    FORMAT      = pyaudio.paInt16
    SAMPLE_RATE = int(PARAMS["sample_rate"])

    audio = stream = ws_app = None
    audio_thread = ws_thread = None
    stop_event = threading.Event()

    # ── WebSocket callbacks ───────────────────────────────────────────────
    def on_open(ws):
        def mic_loop():
            while not stop_event.is_set():
                ws.send(stream.read(FRAMES, exception_on_overflow=False),
                        websocket.ABNF.OPCODE_BINARY)
        global audio_thread
        audio_thread = threading.Thread(target=mic_loop, daemon=True)
        audio_thread.start()

    def on_message(ws, msg):
        global final_metadata
        d = json.loads(msg)
        if d.get("type") == "Results":
            txt = d["channel"]["alternatives"][0]["transcript"]
            if d["is_final"]:
                print(" " * 80, end="\r")
                print(txt)
            else:
                print(txt, end="\r")
        elif d.get("type") == "Metadata":
            final_metadata = d  # save the metadata for later

    def on_error(ws, error):
        print(f"\nWebSocket error: {error}")
        stop_event.set()

    def on_close(ws, *args):
        stop_event.set()
        if final_metadata:
            print("\nFinal Metadata from Deepgram:")
            print(json.dumps(final_metadata, indent=2))
        else:
            print("\nNo metadata received before close.")

        def graceful_shutdown(signum, frame):
            print("\nCtrl+C received → shutting down …")
            stop_event.set()

            # Send CloseStream before closing WebSocket
            if ws_app and ws_app.sock and ws_app.sock.connected:
                try:
                    ws_app.send(json.dumps({"type": "CloseStream"}))
                    time.sleep(0.5)  # Allow time for final metadata to come through
                except Exception as e:
                    print("Error sending CloseStream:", e)
                ws_app.close()

            if ws_thread and ws_thread.is_alive():
                ws_thread.join(timeout=2.0)

    signal.signal(signal.SIGINT, graceful_shutdown)

    def run():
        global audio, stream, ws_app, ws_thread

        # 1. open microphone
        audio = pyaudio.PyAudio()
        try:
            stream = audio.open(format=FORMAT, channels=CHANNELS,
                                rate=SAMPLE_RATE, input=True,
                                frames_per_buffer=FRAMES)
            print("Microphone stream opened. Press Ctrl+C to stop.")
        except Exception as e:
            print(f"Error opening mic: {e}")
            audio.terminate()
            return

        # 2. create WebSocket
        ws_app = websocket.WebSocketApp(
            EP,
            header={"Authorization": f"Token {DG_KEY}"},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        # 3. start WS thread (only once!)
        ws_thread = threading.Thread(target=ws_app.run_forever, daemon=True)
        ws_thread.start()

        # 4. block main thread until WS thread ends
        ws_thread.join()

        # 5. cleanup
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

    if __name__ == "__main__":
        run()
    ```
    

    

    ```python expandable
    import pyaudio
    import websocket
    import json
    import threading
    import time
    from urllib.parse import urlencode
    from datetime import datetime

    # --- Configuration ---
    YOUR_API_KEY = "YOUR-API-KEY"  # Replace with your actual API key

    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "speech_model": "universal-3-5-pro",
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    # Audio Configuration
    FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    # Global variables for audio stream and websocket
    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()  # To signal the audio thread to stop

    # --- WebSocket Event Handlers ---

    def on_open(ws):
        """Called when the WebSocket connection is established."""
        print("WebSocket connection opened.")
        print(f"Connected to: {API_ENDPOINT}")

        # Start sending audio data in a separate thread
        def stream_audio():
            global stream
            print("Starting audio streaming...")
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    # Send audio data as binary message
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    # If stream read fails, likely means it's closed, stop the loop
                    break
            print("Audio streaming stopped.")

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = (
            True  # Allow main thread to exit even if this thread is running
        )
        audio_thread.start()

    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get('type')

            if msg_type == "Begin":
                session_id = data.get('id')
                expires_at = data.get('expires_at')
                print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
            elif msg_type == "Turn":
                transcript = data.get('transcript', '')
                end_of_turn = data.get('end_of_turn', False)

                # Print final end-of-turn transcript
                if end_of_turn:
                    print('\r' + ' ' * 80 + '\r', end='')
                    print(transcript)
                else:
                    print(f"\r{transcript}", end='')
            elif msg_type == "Termination":
                audio_duration = data.get('audio_duration_seconds', 0)
                session_duration = data.get('session_duration_seconds', 0)
                print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
        except json.JSONDecodeError as e:
            print(f"Error decoding message: {e}")
        except Exception as e:
            print(f"Error handling message: {e}")

    def on_error(ws, error):
        """Called when a WebSocket error occurs."""
        print(f"\nWebSocket Error: {error}")
        # Attempt to signal stop on error
        stop_event.set()

    def on_close(ws, close_status_code, close_msg):
        """Called when the WebSocket connection is closed."""
        print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
        # Ensure audio resources are released
        global stream, audio
        stop_event.set()  # Signal audio thread just in case it's still running

        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
            stream = None
        if audio:
            audio.terminate()
            audio = None
        # Try to join the audio thread to ensure clean exit
        if audio_thread and audio_thread.is_alive():
            audio_thread.join(timeout=1.0)

    # --- Main Execution ---
    def run():
        global audio, stream, ws_app

        # Initialize PyAudio
        audio = pyaudio.PyAudio()

        # Open microphone stream
        try:
            stream = audio.open(
                input=True,
                frames_per_buffer=FRAMES_PER_BUFFER,
                channels=CHANNELS,
                format=FORMAT,
                rate=SAMPLE_RATE,
            )
            print("Microphone stream opened successfully.")
            print("Speak into your microphone. Press Ctrl+C to stop.")
        except Exception as e:
            print(f"Error opening microphone stream: {e}")
            if audio:
                audio.terminate()
            return  # Exit if microphone cannot be opened

        # Create WebSocketApp
        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            # Keep main thread alive until interrupted
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nCtrl+C received. Stopping...")
            stop_event.set()  # Signal audio thread to stop

            # Send termination message to the server
            if ws_app and ws_app.sock and ws_app.sock.connected:
                try:
                    terminate_message = {"type": "Terminate"}
                    print(f"Sending termination message: {json.dumps(terminate_message)}")
                    ws_app.send(json.dumps(terminate_message))
                    # Give a moment for messages to process before forceful close
                    time.sleep(5)
                except Exception as e:
                    print(f"Error sending termination message: {e}")

            # Close the WebSocket connection (will trigger on_close)
            if ws_app:
                ws_app.close()

            # Wait for WebSocket thread to finish
            ws_thread.join(timeout=2.0)

        except Exception as e:
            print(f"\nAn unexpected error occurred: {e}")
            stop_event.set()
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

        finally:
            # Final cleanup (already handled in on_close, but good as a fallback)
            if stream and stream.is_active():
                stream.stop_stream()
            if stream:
                stream.close()
            if audio:
                audio.terminate()
            print("Cleanup complete. Exiting.")

    if __name__ == "__main__":
        run()
    ```
    

## Authentication

    

    ```python
    import json, threading, time, websocket, pyaudio, signal
    from urllib.parse import urlencode

    DG_KEY = "YOUR_DG_API_KEY"
    ```
    
    

        ```python
        import pyaudio
        import websocket
        import json
        import threading
        import time
        from urllib.parse import urlencode
        from datetime import datetime

        YOUR_API_KEY = "YOUR-API-KEY"
        ```
    

When migrating from Deepgram to AssemblyAI, you'll first need to handle authentication:

Get your API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home)

  **Protect Your API Key**

  For improved security, store your API key as an environment variable.

## Connection Parameters & Microphone Setup

    
        ```python
        PARAMS = {
            "model": "nova-3",
            "encoding": "linear16",
            "sample_rate": "16000",
            "channels": "1",
            "punctuate": "true",
            "interim_results": "true",
        }

        EP = f"wss://api.deepgram.com/v1/listen?{urlencode(PARAMS)}"

        FRAMES = 800
        CHANNELS = 1
        FORMAT = pyaudio.paInt16
        SAMPLE_RATE = int(PARAMS["sample_rate"])

        audio = stream = ws_app = None
        audio_thread = ws_thread = None
        stop_event = threading.Event()
        ```
    

    

        ```python
        CONNECTION_PARAMS = {
            "sample_rate": 16000,
            "speech_model": "universal-3-5-pro",
        }
        API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
        API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

        # Audio Configuration
        FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
        SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
        CHANNELS = 1
        FORMAT = pyaudio.paInt16

        # Global variables for audio stream and websocket
        audio = None
        stream = None
        ws_app = None
        audio_thread = None
        stop_event = threading.Event()  # To signal the audio thread to stop
        ```
    

Here are helpful things to know about connecting our streaming model:

- **Universal-3.5 Pro model** — Connect to `wss://streaming.assemblyai.com/v3/ws` with `speech_model=universal-3-5-pro` to use our latest, highest-accuracy streaming model. Unlike Deepgram’s `model="nova-3"`, you use a single `speech_model` parameter.

- **Built-in formatting** — Universal-3.5 Pro always returns formatted transcripts with smart punctuation & casing, similar to Deepgram’s `punctuate=true`. No extra parameter is needed.

- **Partials are always on — like Deepgram’s `interim_results=true`** — AssemblyAI streams interim results automatically. Universal-3.5 Pro emits partials during periods of silence, with at most one partial per silence period.

## Opening the WebSocket

    
        ```python
        def on_open(ws):
            def mic_loop():
                while not stop_event.is_set():
                    ws.send(stream.read(FRAMES, exception_on_overflow=False),
                            websocket.ABNF.OPCODE_BINARY)
            global audio_thread
            audio_thread = threading.Thread(target=mic_loop, daemon=True)
            audio_thread.start()
        ```
    
    

    ```python expandable
    def on_open(ws):
        """Called when the WebSocket connection is established."""
        print("WebSocket connection opened.")
        print(f"Connected to: {API_ENDPOINT}")

        # Start sending audio data in a separate thread
        def stream_audio():
            global stream
            print("Starting audio streaming...")
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    # Send audio data as binary message
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    # If stream read fails, likely means it's closed, stop the loop
                    break
            print("Audio streaming stopped.")

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = (
            True  # Allow main thread to exit even if this thread is running
        )
        audio_thread.start()
    ```
    

    **Tip:** Adding error-handling and log lines (as in the AssemblyAI snippet) lets you see exactly when the socket opens, audio starts, or a read fails—catching issues early saving time debugging silent failures.

## Receiving Messsages from the WebSocket

    
        ```python
        def on_message(ws, msg):
            global final_metadata
            d = json.loads(msg)
            if d.get("type") == "Results":
                txt = d["channel"]["alternatives"][0]["transcript"]
                if d["is_final"]:
                    print(" " * 80, end="\r")
                    print(txt)
                else:
                    print(txt, end="\r")
            elif d.get("type") == "Metadata":
                final_metadata = d  # save the metadata for later
        ```
    
    

    ```python expandable
    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get(‘type’)

            if msg_type == "Begin":
                session_id = data.get(‘id’)
                expires_at = data.get(‘expires_at’)
                print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
            elif msg_type == "Turn":
                transcript = data.get(‘transcript’, ‘’)
                end_of_turn = data.get(‘end_of_turn’, False)

                # Print final end-of-turn transcript
                if end_of_turn:
                    print(‘\r’ + ‘ ‘ * 80 + ‘\r’, end=’’)
                    print(transcript)
                else:
                    print(f"\r{transcript}", end=’’)
            elif msg_type == "Termination":
                audio_duration = data.get(‘audio_duration_seconds’, 0)
                session_duration = data.get(‘session_duration_seconds’, 0)
                print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
        except json.JSONDecodeError as e:
            print(f"Error decoding message: {e}")
        except Exception as e:
            print(f"Error handling message: {e}")
    ```
    

Helpful things to know about AssemblyAI’s message payloads:

- **Clear message types** – Instead of checking `is_final`, you’ll receive explicit `"Begin"`, `"Turn"`, and `"Termination"` events, making your logic simpler and more readable.

- **Session metadata up-front** – The first `"Begin"` message delivers a `session_id` and expiry timestamp. You can log or surface these for tracing or billing.

- **End-of-turn detection** – Each `"Turn"` object includes an `end_of_turn` boolean. When `end_of_turn` is `true`, the transcript is a final, formatted result. When `false`, it is a partial transcript. Universal-3.5 Pro always returns formatted transcripts with smart punctuation & casing built in.

## Handling Errors

    
        ```python
        def on_error(ws, error):
            print(f"\nWebSocket error: {error}")
            stop_event.set()
        ```
    
    
        ```python
        def on_error(ws, error):
            """Called when a WebSocket error occurs."""
            print(f'\nWebSocket Error: {error}')

            # Attempt to signal stop on error
            stop_event.set()
        ```
    

Capture and log any errors emitted by the WebSocket connection to streamline troubleshooting and maintain smooth operation.

## Closing the WebSocket

    
        ```python
        def on_close(ws, *args):
            stop_event.set()
            if final_metadata:
                print("\nFinal Metadata from Deepgram:")
                print(json.dumps(final_metadata, indent=2))
            else:
                print("\nNo metadata received before close.")
        ```
    
    

    ```python
    def on_close(ws, close_status_code, close_msg):
        """Called when the WebSocket connection is closed."""
        print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
        # Ensure audio resources are released
        global stream, audio
        stop_event.set()  # Signal audio thread just in case it's still running

        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
            stream = None
        if audio:
            audio.terminate()
            audio = None
        # Try to join the audio thread to ensure clean exit
        if audio_thread and audio_thread.is_alive():
            audio_thread.join(timeout=1.0)
        ```
    

Helpful things to know about AssemblyAI’s WebSocket Closure:

- **Connection diagnostics on tap** - If the socket closes unexpectedly, AssemblyAI supplies both a status code and a reason message (close_status_code, close_msg), so you know immediately whether the server timed out, refused auth, or encountered another error.

- **Metadata arrives at session start, not at close** - Deepgram sends its final metadata only when the socket closes. AssemblyAI delivers session information up front in the initial "Begin" message, so you can log IDs and expiry times right away.

## Opening the Microphone Stream & Creating a WebSocket

    
        
            ```python expandable
            global audio, stream, ws_app, ws_thread

            # 1. open microphone
            audio = pyaudio.PyAudio()
            try:
                stream = audio.open(format=FORMAT, channels=CHANNELS,
                                    rate=SAMPLE_RATE, input=True,
                                    frames_per_buffer=FRAMES)
                print("Microphone stream opened. Press Ctrl+C to stop.")
            except Exception as e:
                print(f"Error opening mic: {e}")
                audio.terminate()
                return

            # 2. create WebSocket
            ws_app = websocket.WebSocketApp(
                EP,
                header={"Authorization": f"Token {DG_KEY}"},
                on_open=on_open,
                on_message=on_message,
                on_error=on_error,
                on_close=on_close,
            )

            ws_thread = threading.Thread(target=ws_app.run_forever, daemon=True)
            ws_thread.start()
            ws_thread.join()
        ```
    
    
    ```python expandable
    global audio, stream, ws_app
    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )
    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()
    ```
    

## Session Shutdown

    
        ```python expandable
        def graceful_shutdown(signum, frame):
            print("\nCtrl+C received → shutting down …")
            stop_event.set()

            # Send CloseStream before closing WebSocket
            if ws_app and ws_app.sock and ws_app.sock.connected:
                try:
                    ws_app.send(json.dumps({"type": "CloseStream"}))
                    time.sleep(0.5)  # Allow time for final metadata to come through
                except Exception as e:
                    print("Error sending CloseStream:", e)
                ws_app.close()

            if ws_thread and ws_thread.is_alive():
                ws_thread.join(timeout=2.0)

        signal.signal(signal.SIGINT, graceful_shutdown)

        def run():
            # Open microphone and create WebSocket

            # cleanup
            if stream and stream.is_active():
                stream.stop_stream()
            if stream:
                stream.close()
            if audio:
                audio.terminate()
            print("Cleanup complete. Exiting.")
        ```
    
    
    ```python expandable
    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")
    ```
    

Helpful things to know about AssemblyAI’s shutdown:

- **JSON payload difference** - When closing the stream with AssemblyAI, your JSON payload will be `{"type: "Terminate" }` instead of `{"type: "CloseStream" }`

- **No metadata race condition** - Because AssemblyAI already provided session info at "Begin" and doesn’t append extra data at shutdown, you don’t have to sleep (time.sleep(0.5)) to wait for “final metadata” before closing—making the exit faster and less error-prone.

## Resources

For additional information about using AssemblyAI's Streaming Speech-To-Text API you can also refer to:

- [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio)
- [Streaming Guide](/streaming/getting-started/transcribe-streaming-audio)

---

<a id="migration-guide-speechmatics-to-assemblyai-docs-streaming-migration-guides-speechmatics-to-aai-streaming-mdx"></a>

# Migration guide: Speechmatics to AssemblyAI

URL: https://www.assemblyai.com/docs/streaming/migration-guides/speechmatics_to_aai_streaming
Source: docs/streaming/migration-guides/speechmatics_to_aai_streaming.mdx
Navigation: Real-time STT > Guides > Migration guides
Description: Migration guide: Speechmatics to AssemblyAI documentation.

This guide walks through the process of migrating from **Speechmatics** to **AssemblyAI** for streaming Speech-to-text.

## Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Side-by-side code comparison

Below is a side-by-side comparison of a basic Python code snippet to transcribe streaming audio by Speechmatics and AssemblyAI.

```python expandable
import pyaudio
import websocket
import json
import threading
import time

# --- Configuration ---

YOUR_API_KEY = "YOUR-API-KEY" # Replace with your actual API key

CONNECTION_PARAMS = {
    "language": "en",
    "enable_partials": True,
    "max_delay": 2.0
}
API_ENDPOINT = "wss://eu2.rt.speechmatics.com/v2/en"

# Audio Configuration

FRAMES_PER_BUFFER = 1024 # Chunk size
SAMPLE_RATE = None # Will be set based on device capabilities
CHANNELS = 1
FORMAT = pyaudio.paFloat32 # Speechmatics uses float32 format

# Global variables for audio stream and websocket

audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event() # To signal the audio thread to stop
audio_seq_no = 0 # Track number of audio chunks sent

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    # Send StartRecognition message
    start_message = {
        "message": "StartRecognition",
        "audio_format": {
            "type": "raw",
            "encoding": "pcm_f32le",
            "sample_rate": SAMPLE_RATE
        },
        "transcription_config": {
            "language": CONNECTION_PARAMS["language"],
            "enable_partials": CONNECTION_PARAMS["enable_partials"],
            "max_delay": CONNECTION_PARAMS["max_delay"]
        }
    }
    ws.send(json.dumps(start_message))

def on_message(ws, message):
    global audio_seq_no

    try:
        data = json.loads(message)
        msg_type = data.get('message')

        if msg_type == "RecognitionStarted":
            session_id = data.get('id')
            print(f"\nSession began: ID={session_id}")

            # Start sending audio data in a separate thread
            def stream_audio():
                global audio_seq_no, stream
                print("Starting audio streaming...")
                while not stop_event.is_set():
                    try:
                        audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                        # Send audio data as binary message
                        ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                        audio_seq_no += 1
                    except Exception as e:
                        print(f"Error streaming audio: {e}")
                        # If stream read fails, likely means it's closed, stop the loop
                        break
                print("Audio streaming stopped.")

            global audio_thread
            audio_thread = threading.Thread(target=stream_audio)
            audio_thread.daemon = (
                True  # Allow main thread to exit even if this thread is running
            )
            audio_thread.start()

        elif msg_type == "AddPartialTranscript":
            transcript = data.get('metadata', {}).get('transcript', '')
            if transcript:
                print(f"\r{transcript}", end='')

        elif msg_type == "AddTranscript":
            transcript = data.get('metadata', {}).get('transcript', '')
            if transcript:
                # Clear previous line for final messages
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)

        elif msg_type == "EndOfTranscript":
            print("\nSession Terminated: Transcription complete")

        elif msg_type == "Error":
            error_type = data.get('type')
            reason = data.get('reason')
            print(f"\nWebSocket Error: {error_type} - {reason}")
            stop_event.set()

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")  # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---

def run():
    global audio, stream, ws_app, SAMPLE_RATE

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Get default input device (can alter to specify specific device)
    default_device = audio.get_default_input_device_info()
    device_index = default_device['index']
    SAMPLE_RATE = int(audio.get_device_info_by_index(device_index)['defaultSampleRate'])

    print(f"Using microphone: {default_device['name']}")

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
            input_device_index=device_index
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": f"Bearer {YOUR_API_KEY}"},  # Speechmatics uses Bearer token
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=lambda: ws_app.run_forever(ping_interval=30, ping_timeout=10))
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send EndOfStream message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                end_message = {
                    "message": "EndOfStream",
                    "last_seq_no": audio_seq_no
                }
                print(f"Sending termination message: {json.dumps(end_message)}")
                ws_app.send(json.dumps(end_message))
                # Give a moment for messages to process before forceful close
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()

````

```python expandable
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "YOUR-API-KEY"  # Replace with your actual API key

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# --- WebSocket Event Handlers ---
def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            end_of_turn = data.get('end_of_turn', False)

            # Print final end-of-turn transcript
            if end_of_turn:
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
````

## Step 1: Install dependencies

Install the required Python packages.

```bash
pip install websocket-client pyaudio
```

Install the required Python packages:

```bash
pip install websocket-client pyaudio
```

## Step 2: Configure the API key

In this step, you'll configure your API key to authenticate your requests.

  Navigate to [API Keys](https://portal.speechmatics.com/settings/api-keys) in your account settings and copy your API key.

  Navigate to [API Keys](https://www.assemblyai.com/dashboard/home) in your account dashboard and copy your API key.

Store your API key in a variable. Replace `<YOUR_API_KEY>` with your copied API key.

```python
import pyaudio
import websocket
import json
import threading
import time

YOUR_API_KEY = "YOUR-API-KEY"
```

Store your API key in a variable. Replace `<YOUR_API_KEY>` with your copied API key.

```python
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

YOUR_API_KEY = "YOUR-API-KEY"
```

 

  
  

```python
import requests

def generate_temp_token(api_key, ttl=60):
    """Generate a temporary authentication token that expires after the specified time."""
    url = "https://mp.speechmatics.com/v1/api_keys?type=rt"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {api_key}"
    }
    payload = {
        "ttl": ttl
    }

    response = requests.post(url, json=payload, headers=headers)
    data = response.json()
    return data.get("key_value")
```

  
    **Token usage**

    Instead of authorizing your request with `YOUR_API_KEY` (via request header), use the temporary token generated by this function when establishing the WebSocket connection.
    ```python
      API_ENDPOINT= f"wss://eu2.rt.speechmatics.com/v2?jwt={generate_temp_token(api_key)}"
      ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
      )
    ```
  
  
  
  ```python
  import requests
  from urllib.parse import urlencode

def generate_temp_token(api_key, expires_in_seconds=60):
    """Generate a temporary authentication token that expires after the specified time."""
    url = "https://streaming.assemblyai.com/v3/token"

    response = requests.get(
        f"{url}?{urlencode({'expires_in_seconds': expires_in_seconds})}",
        headers={"Authorization": api_key}
    )
    data = response.json()
    return data.get("token")

````

  **Token usage**

  Instead of authorizing your request with `YOUR_API_KEY` (via request header), use the temporary token generated by this function when establishing the WebSocket connection.
  ```python
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}&token={generate_temp_token(api_key)}"
    ws_app = websocket.WebSocketApp(
      API_ENDPOINT,
      on_open=on_open,
      on_message=on_message,
      on_error=on_error,
      on_close=on_close,
    )
  ```

## Step 3: Set up audio configuration

Configure the audio settings for your microphone stream.

```python expandable
import pyaudio

# Audio Configuration
FRAMES_PER_BUFFER = 1024  # Chunk size
SAMPLE_RATE = None  # Will be set based on device capabilities
CHANNELS = 1
FORMAT = pyaudio.paFloat32  # Speechmatics uses float32 format

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop
audio_seq_no = 0  # Track number of audio chunks sent

def run():
  global audio, stream, ws_app, SAMPLE_RATE

  # Initialize PyAudio
  audio = pyaudio.PyAudio()

  # Get default input device (can alter to specify specific device)
  default_device = audio.get_default_input_device_info()
  device_index = default_device['index']
  SAMPLE_RATE = int(audio.get_device_info_by_index(device_index)['defaultSampleRate'])

  print(f"Using microphone: {default_device['name']}")

  # Open microphone stream
  try:
      stream = audio.open(
          input=True,
          frames_per_buffer=FRAMES_PER_BUFFER,
          channels=CHANNELS,
          format=FORMAT,
          rate=SAMPLE_RATE,
          input_device_index=device_index
      )
      print("Microphone stream opened successfully.")
      print("Speak into your microphone. Press Ctrl+C to stop.")
  except Exception as e:
      print(f"Error opening microphone stream: {e}")
      if audio:
          audio.terminate()
      return  # Exit if microphone cannot be opened
````

  **Sample rate**

**Speechmatics** recommends using a `16 kHz` sample rate for speech audio. Anything higher will be downsampled server-side.

```python expandable
import pyaudio

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
}

# Audio Configuration

FRAMES_PER_BUFFER = 800 # 50ms of audio (0.05s \* 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket

audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event() # To signal the audio thread to stop

def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

````

  **Sample rate**

Using a sample rate of `16 kHz` and encoding of `pcm_s16le` is recommended, as our STT model operates at a `16 kHz` sample rate.
If the incoming audio uses a different rate, we perform additional sampling rate conversion under the hood, which might marginally increase latency.

  **Audio data format**

If you want to stream data from elsewhere, make sure that your audio data is in the following format:

- Single-channel
- PCM16 (default) or Mu-law encoding
- A sample rate that matches the value of the `sample_rate` parameter (16 kHz is recommended)
- 50 milliseconds of audio per message (larger chunk sizes are workable, but may result in latency fluctuations)

## Step 4: Create event handlers

In this step, you’ll set up callback functions that handle the different events.

Create functions to handle the events from the real-time service.

```python expandable
import json

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    # Send StartRecognition message
    start_message = {
        "message": "StartRecognition",
        "audio_format": {
            "type": "raw",
            "encoding": "pcm_f32le",
            "sample_rate": SAMPLE_RATE
        },
        "transcription_config": {
            "language": CONNECTION_PARAMS["language"],
            "enable_partials": CONNECTION_PARAMS["enable_partials"],
            "max_delay": CONNECTION_PARAMS["max_delay"]
        }
    }
    ws.send(json.dumps(start_message))

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
````

```python expandable
import threading

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")  # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

````

  **Connection configuration**

**Speechmatics** requires a **handshake** where the connection configuration is specified before audio is streamed. **AssemblyAI** allows you to configure the connection via query parameters in the URL and start streaming audio immediately.

The **Speechmatics** handshake begins when `on_open` sends a `StartRecognition` message to configure the session. Audio streaming only starts after the `RecognitionStarted` message type is parsed and confirmed in the `on_message` callback.

Create another function to handle transcripts.

**Speechmatics** has separate partial (`AddPartialTranscript`) and final (`AddTranscript`) transcripts. The terminate session message is `EndOfTranscript`.

**AssemblyAI** instead uses a `Turn` object with an `end_of_turn` boolean flag to indicate finality. The terminate session message is `Termination`.
For more on the Turn object, see **Streaming** [Core concepts](/streaming/getting-started/transcribe-streaming-audio) section.

```python expandable
def on_message(ws, message):
    global audio_seq_no

    try:
        data = json.loads(message)
        msg_type = data.get('message')

        if msg_type == "RecognitionStarted":
            session_id = data.get('id')
            print(f"\nSession began: ID={session_id}")

            # Start sending audio data in a separate thread
            def stream_audio():
                global audio_seq_no, stream
                print("Starting audio streaming...")
                while not stop_event.is_set():
                    try:
                        audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                        # Send audio data as binary message
                        ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                        audio_seq_no += 1
                    except Exception as e:
                        print(f"Error streaming audio: {e}")
                        # If stream read fails, likely means it's closed, stop the loop
                        break
                print("Audio streaming stopped.")

            global audio_thread
            audio_thread = threading.Thread(target=stream_audio)
            audio_thread.daemon = (
                True  # Allow main thread to exit even if this thread is running
            )
            audio_thread.start()

        elif msg_type == "AddPartialTranscript":
            transcript = data.get('metadata', {}).get('transcript', '')
            if transcript:
                print(f"\r{transcript}", end='')

        elif msg_type == "AddTranscript":
            transcript = data.get('metadata', {}).get('transcript', '')
            if transcript:
                # Clear previous line for final messages
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)

        elif msg_type == "EndOfTranscript":
            print("\nSession Terminated: Transcription complete")

        elif msg_type == "Error":
            error_type = data.get('type')
            reason = data.get('reason')
            print(f"\nWebSocket Error: {error_type} - {reason}")
            stop_event.set()

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")
````

```python expandable
import json
from datetime import datetime

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            end_of_turn = data.get('end_of_turn', False)

            # Print final end-of-turn transcript
            if end_of_turn:
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

````

  **Transcript message structure**

  Please note the difference in transcript message structure below:
  ```python
  # Speechmatics
  {
    "message": "AddPartialTranscript",
    "metadata": {
      "transcript": "hello world"
    },
    # Other transcript data...
  }

  # AssemblyAI
  {
    "type": "Turn",
    "transcript": "hello world",
    "end_of_turn": false,
    # Other transcript data...
  }
````

## Step 5: Connect and start transcription

To stream audio, establish a connection to the API via [WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API).

Create a WebSocket connection to the Realtime service.
```python
def run():
    global audio, stream, ws_app, SAMPLE_RATE
    # Skipping audio/microphone setup code...

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": f"Bearer {YOUR_API_KEY}"},  # Speechmatics uses Bearer token
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=lambda: ws_app.run_forever(ping_interval=30, ping_timeout=10))
    ws_thread.daemon = True
    ws_thread.start()

````

Create and run a WebSocket connection to the Realtime service.

```python expandable
import websocket
import threading

def run():
    global audio, stream, ws_app
    # Skipping audio/microphone setup code...

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()
````

  **Authorization **

  Note that while both services use an `Authorization` header to authenticate
  the WebSocket connection, **Speechmatics** uses a `Bearer` prefix, while
  **AssemblyAI** does not.

## Step 6: Close the connection

Keep the main thread alive until interrupted, handle keyboard interrupts and thrown exceptions, and clean up upon closing of the WebSocket connection.

```python expandable
def run():
    global audio, stream, ws_app, SAMPLE_RATE
    # Skipping audio/microphone setup and WebSocket connection code...

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send EndOfStream message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                end_message = {
                    "message": "EndOfStream",
                    "last_seq_no": audio_seq_no
                }
                print(f"Sending termination message: {json.dumps(end_message)}")
                ws_app.send(json.dumps(end_message))
                # Give a moment for messages to process before forceful close
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")
```

```python expandable
def run():
    global audio, stream, ws_app
    # Skipping audio/microphone setup and WebSocket connection code...

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

````

The connection will close automatically when you press `Ctrl+C`. In both cases, the `on_close` handler will clean up the audio resources.

## Step 7: Execute the main function

Finally, run the main function to start the main execution.

```python
if __name__ == "__main__":
    run()
````

```python
if __name__ == "__main__":
    run()
```

## Next steps

To learn more about both Streaming APIs, their key differences, and how to best migrate, see the following resources:

**AssemblyAI**

- [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio)
- [Streaming Speech-to-Text](/streaming/getting-started/transcribe-streaming-audio)
- [Streaming API reference](/streaming/api-spec/streaming-websocket)

**Speechmatics**

- [Transcribe in Real-Time](https://docs.speechmatics.com/introduction/rt-guide)
- [Using Microphone Input](https://docs.speechmatics.com/tutorials/using-mic)
- [Real-Time API Reference](https://docs.speechmatics.com/rt-api-ref)

## Need some help?

If you get stuck or have any other questions, contact our support team at `support@assemblyai.com` or create a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="streaming-migration-guide-gladia-to-assemblyai-docs-streaming-migration-guides-gladia-to-aai-streaming-mdx"></a>

# Streaming Migration Guide: Gladia to AssemblyAI

URL: https://www.assemblyai.com/docs/streaming/migration-guides/gladia-to-aai-streaming
Source: docs/streaming/migration-guides/gladia-to-aai-streaming.mdx
Navigation: Real-time STT > Guides > Migration guides
Description: Streaming Migration Guide: Gladia to AssemblyAI documentation.

This guide walks through the process of migrating from Gladia to AssemblyAI for transcribing streaming audio.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/home).

## Side-By-Side Code Comparison

Below is a side-by-side comparison of a basic snippet to transcribe live audio by Gladia and AssemblyAI using a microphone:

    

    ```python expandable
    import asyncio
    import base64
    import json
    import signal
    from datetime import time
    import pyaudio
    import requests
    from websockets.asyncio.client import connect
    from websockets.exceptions import ConnectionClosedOK

    # Constants
    GLADIA_API_KEY = "<YOUR_GLADIA_API_KEY>"
    GLADIA_API_URL = "https://api.gladia.io"

    # Audio configuration
    CHANNELS = 1
    FORMAT = pyaudio.paInt16
    FRAMES_PER_BUFFER = 3200
    SAMPLE_RATE = 16_000

    async def main():
        # Initialize the session
        config = {
            "encoding": "wav/pcm",
            "sample_rate": SAMPLE_RATE,
            "bit_depth": 16,
            "channels": CHANNELS,
            "language_config": {
                "languages": ["en"],
            },
        }

        # Start the live session
        response = requests.post(
            f"{GLADIA_API_URL}/v2/live",
            headers={"X-Gladia-Key": GLADIA_API_KEY},
            json=config,
            timeout=3,
        )

        if not response.ok:
            print(f"{response.status_code}: {response.text or response.reason}")
            exit(response.status_code)

        session_data = response.json()

        # Connect to the websocket
        async with connect(session_data["url"]) as websocket:
            print("\n################ Begin session ################\n")

            # Handle Ctrl+C to stop recording
            loop = asyncio.get_running_loop()
            loop.add_signal_handler(
                signal.SIGINT,
                lambda: loop.create_task(stop_recording(websocket)),
            )

            # Create tasks for sending audio and receiving transcripts
            send_audio_task = asyncio.create_task(send_audio(websocket))
            receive_transcript_task = asyncio.create_task(receive_transcript(websocket))

            await asyncio.wait([send_audio_task, receive_transcript_task])

    async def stop_recording(websocket):
        print(">>>>> Ending the recording…")
        await websocket.send(json.dumps({"type": "stop_recording"}))
        await asyncio.sleep(0)

    async def send_audio(websocket):
        # Initialize PyAudio
        p = pyaudio.PyAudio()

        # Open audio stream
        stream = p.open(
            format=FORMAT,
            channels=CHANNELS,
            rate=SAMPLE_RATE,
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
        )

        # Send audio chunks
        while True:
            data = stream.read(FRAMES_PER_BUFFER)
            data = base64.b64encode(data).decode("utf-8")
            json_data = json.dumps({"type": "audio_chunk", "data": {"chunk": str(data)}})
            try:
                await websocket.send(json_data)
                await asyncio.sleep(0.1)  # Send audio every 100ms
            except ConnectionClosedOK:
                return

    async def receive_transcript(websocket):
        # Process incoming messages
        async for message in websocket:
            content = json.loads(message)

            # Print transcripts
            if content["type"] == "transcript" and content["data"]["is_final"]:
                text = content["data"]["utterance"]["text"].strip()
                print(f"Final: {text}")

            # Print final results
            if content["type"] == "post_final_transcript":
                print("\n################ End of session ################\n")
                print(json.dumps(content, indent=2, ensure_ascii=False))

    if __name__ == "__main__":
        asyncio.run(main())
    ```

    
    

    ```python expandable
    import pyaudio
    import websocket
    import json
    import threading
    import time
    from urllib.parse import urlencode
    from datetime import datetime

    # --- Configuration ---
    YOUR_API_KEY = "YOUR-API-KEY"  # Replace with your actual API key

    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "speech_model": "universal-3-5-pro",
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    # Audio Configuration
    FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    # Global variables for audio stream and websocket
    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()  # To signal the audio thread to stop

    # --- WebSocket Event Handlers ---

    def on_open(ws):
        """Called when the WebSocket connection is established."""
        print("WebSocket connection opened.")
        print(f"Connected to: {API_ENDPOINT}")

        # Start sending audio data in a separate thread
        def stream_audio():
            global stream
            print("Starting audio streaming...")
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    # Send audio data as binary message
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    # If stream read fails, likely means it's closed, stop the loop
                    break
            print("Audio streaming stopped.")

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = (
            True  # Allow main thread to exit even if this thread is running
        )
        audio_thread.start()

    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get('type')

            if msg_type == "Begin":
                session_id = data.get('id')
                expires_at = data.get('expires_at')
                print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
            elif msg_type == "Turn":
                transcript = data.get('transcript', '')
                end_of_turn = data.get('end_of_turn', False)

                # Print final end-of-turn transcript
                if end_of_turn:
                    print('\r' + ' ' * 80 + '\r', end='')
                    print(transcript)
                else:
                    print(f"\r{transcript}", end='')
            elif msg_type == "Termination":
                audio_duration = data.get('audio_duration_seconds', 0)
                session_duration = data.get('session_duration_seconds', 0)
                print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
        except json.JSONDecodeError as e:
            print(f"Error decoding message: {e}")
        except Exception as e:
            print(f"Error handling message: {e}")

    def on_error(ws, error):
        """Called when a WebSocket error occurs."""
        print(f"\nWebSocket Error: {error}")
        # Attempt to signal stop on error
        stop_event.set()

    def on_close(ws, close_status_code, close_msg):
        """Called when the WebSocket connection is closed."""
        print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
        # Ensure audio resources are released
        global stream, audio
        stop_event.set()  # Signal audio thread just in case it's still running

        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
            stream = None
        if audio:
            audio.terminate()
            audio = None
        # Try to join the audio thread to ensure clean exit
        if audio_thread and audio_thread.is_alive():
            audio_thread.join(timeout=1.0)

    # --- Main Execution ---
    def run():
        global audio, stream, ws_app

        # Initialize PyAudio
        audio = pyaudio.PyAudio()

        # Open microphone stream
        try:
            stream = audio.open(
                input=True,
                frames_per_buffer=FRAMES_PER_BUFFER,
                channels=CHANNELS,
                format=FORMAT,
                rate=SAMPLE_RATE,
            )
            print("Microphone stream opened successfully.")
            print("Speak into your microphone. Press Ctrl+C to stop.")
        except Exception as e:
            print(f"Error opening microphone stream: {e}")
            if audio:
                audio.terminate()
            return  # Exit if microphone cannot be opened

        # Create WebSocketApp
        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            # Keep main thread alive until interrupted
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nCtrl+C received. Stopping...")
            stop_event.set()  # Signal audio thread to stop

            # Send termination message to the server
            if ws_app and ws_app.sock and ws_app.sock.connected:
                try:
                    terminate_message = {"type": "Terminate"}
                    print(f"Sending termination message: {json.dumps(terminate_message)}")
                    ws_app.send(json.dumps(terminate_message))
                    # Give a moment for messages to process before forceful close
                    time.sleep(5)
                except Exception as e:
                    print(f"Error sending termination message: {e}")

            # Close the WebSocket connection (will trigger on_close)
            if ws_app:
                ws_app.close()

            # Wait for WebSocket thread to finish
            ws_thread.join(timeout=2.0)

        except Exception as e:
            print(f"\nAn unexpected error occurred: {e}")
            stop_event.set()
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

        finally:
            # Final cleanup (already handled in on_close, but good as a fallback)
            if stream and stream.is_active():
                stream.stop_stream()
            if stream:
                stream.close()
            if audio:
                audio.terminate()
            print("Cleanup complete. Exiting.")

    if __name__ == "__main__":
        run()

    ```

    

## Authentication

    

    ```python
    import asyncio
    import base64
    import json
    import signal
    from datetime import time
    import pyaudio
    import requests
    from websockets.asyncio.client import connect
    from websockets.exceptions import ConnectionClosedOK

    GLADIA_API_KEY = "<YOUR_GLADIA_API_KEY>"
    ```
    
    

        ```python
        import pyaudio
        import websocket
        import json
        import threading
        import time
        from urllib.parse import urlencode
        from datetime import datetime

        YOUR_API_KEY = "YOUR-API-KEY"
        ```
    

  **Protect Your API Key**

  For improved security, store your API key as an environment variable.

## Connection Parameters & Microphone Setup

    
        ```python
        GLADIA_API_URL = "https://api.gladia.io"

        CHANNELS = 1
        FORMAT = pyaudio.paInt16
        FRAMES_PER_BUFFER = 3200
        SAMPLE_RATE = 16_000

        config = {
            "encoding": "wav/pcm",
            "sample_rate": SAMPLE_RATE,
            "bit_depth": 16,
            "channels": CHANNELS,
            "language_config": {
                "languages": ["en"]
            },
        }
        ```
    

    

        ```python
        CONNECTION_PARAMS = {
            "sample_rate": 16000,
            "speech_model": "universal-3-5-pro",
        }
        API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
        API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

        # Audio Configuration
        FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
        SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
        CHANNELS = 1
        FORMAT = pyaudio.paInt16

        # Global variables for audio stream and websocket
        audio = None
        stream = None
        ws_app = None
        audio_thread = None
        stop_event = threading.Event()  # To signal the audio thread to stop
        ```
    

Helpful information about our streaming model:

- **Universal-3.5 Pro Model** — Connect to `wss://streaming.assemblyai.com/v3/ws` with `speech_model=universal-3-5-pro` to use our latest, highest-accuracy streaming model — Universal-3.5 Pro.

- **Built-in Formatting** — Universal-3.5 Pro always returns formatted transcripts with smart punctuation & casing. No extra parameter is needed.

- **Partial Transcripts** — AssemblyAI streams interim results automatically. Universal-3.5 Pro emits partials during periods of silence, with at most one partial per silence period.

## Open Microphone Stream & Create WebSocket

    
        
            ```python expandable

            # Start the live session
            response = requests.post(
                f"{GLADIA_API_URL}/v2/live",
                headers={"X-Gladia-Key": GLADIA_API_KEY},
                json=config,
                timeout=3,
            )

            if not response.ok:
                print(f"{response.status_code}: {response.text or response.reason}")
                exit(response.status_code)

            session_data = response.json()

            async def send_audio(websocket):
                # Initialize PyAudio
                p = pyaudio.PyAudio()

                # Open audio stream
                stream = p.open(
                    format=FORMAT,
                    channels=CHANNELS,
                    rate=SAMPLE_RATE,
                    input=True,
                    frames_per_buffer=FRAMES_PER_BUFFER,
                )

                # Send audio chunks
                while True:
                    data = stream.read(FRAMES_PER_BUFFER)
                    data = base64.b64encode(data).decode("utf-8")
                    json_data = json.dumps({"type": "audio_chunk", "data": {"chunk": str(data)}})
                    try:
                        await websocket.send(json_data)
                        await asyncio.sleep(0.1)  # Send audio every 100ms
                    except ConnectionClosedOK:
                        return
        ```
    
    
    ```python expandable
    global audio, stream, ws_app
    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )
    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    ```
    
      **Websocket Error Handling**

        Capture and log any errors emitted by the WebSocket connection to streamline troubleshooting and maintain smooth operation.
        ```python
        def on_error(ws, error):
            """Called when a WebSocket error occurs."""
            print(f'\nWebSocket Error: {error}')
            # Attempt to signal stop on error
            stop_event.set()
        ```
    
    

## Open WebSocket

    
        ```python
        # Connect to Websocket
        async with connect(session_data["url"]) as websocket:
            print("\n################ Begin session ################\n")
            
            # Create tasks for sending audio and receiving transcripts
            send_audio_task = asyncio.create_task(send_audio(websocket))
            receive_transcript_task = asyncio.create_task(receive_transcript(websocket))
            
            await asyncio.wait([send_audio_task, receive_transcript_task])
        ```
    
    

    ```python expandable
    def on_open(ws):
        """Called when the WebSocket connection is established."""
        print("WebSocket connection opened.")
        print(f"Connected to: {API_ENDPOINT}")

        # Start sending audio data in a separate thread
        def stream_audio():
            global stream
            print("Starting audio streaming...")
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    # Send audio data as binary message
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    # If stream read fails, likely means it's closed, stop the loop
                    break
            print("Audio streaming stopped.")

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = (
            True  # Allow main thread to exit even if this thread is running
        )
        audio_thread.start()
    ```
    

## Receive Messsages from WebSocket

    
        ```python
        async def receive_transcript(websocket):
            # Process incoming messages
            async for message in websocket:
                content = json.loads(message)
                
                # Print transcripts
                if content["type"] == "transcript" and content["data"]["is_final"]:
                    text = content["data"]["utterance"]["text"].strip()
                    print(f"Final: {text}")
                    
                # Print final results
                if content["type"] == "post_final_transcript":
                    print("\n################ End of session ################\n")
                    print(json.dumps(content, indent=2, ensure_ascii=False))
        ```
    
    

    ```python expandable
    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get('type')

            if msg_type == "Begin":
                session_id = data.get('id')
                expires_at = data.get('expires_at')
                print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
            elif msg_type == "Turn":
                transcript = data.get('transcript', '')
                end_of_turn = data.get('end_of_turn', False)

                # Print final end-of-turn transcript
                if end_of_turn:
                    print('\r' + ' ' * 80 + '\r', end='')
                    print(transcript)
                else:
                    print(f"\r{transcript}", end='')
            elif msg_type == "Termination":
                audio_duration = data.get('audio_duration_seconds', 0)
                session_duration = data.get('session_duration_seconds', 0)
                print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
        except json.JSONDecodeError as e:
            print(f"Error decoding message: {e}")
        except Exception as e:
            print(f"Error handling message: {e}")
    ```
    

Helpful information about AssemblyAI’s message payloads:

- **Clear Message Types** – Instead of checking `is_final`, you’ll receive explicit `"Begin"`, `"Turn"`, and `"Termination"` events, making your logic simpler and more readable.

- **Session Metadata Up-Front** – The first `"Begin"` message delivers a `session_id` and expiry timestamp so you can immediately log or surface these for tracing or billing.

- **End-of-Turn Detection** – Each `"Turn"` object includes an `end_of_turn` boolean. When `end_of_turn` is `true`, the transcript is a final, formatted result. When `false`, it is a partial transcript. Universal-3.5 Pro always returns formatted transcripts with smart punctuation & casing built in.

## Close the WebSocket

    
        ```python
        async def stop_recording(websocket):
            print(">>>>> Ending the recording…")
            await websocket.send(json.dumps({"type": "stop_recording"}))
            await asyncio.sleep(0)
        ```
    
    

    ```python
    def on_close(ws, close_status_code, close_msg):
        """Called when the WebSocket connection is closed."""
        print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
        # Ensure audio resources are released
        global stream, audio
        stop_event.set()  # Signal audio thread just in case it's still running

        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
            stream = None
        if audio:
            audio.terminate()
            audio = None
        # Try to join the audio thread to ensure clean exit
        if audio_thread and audio_thread.is_alive():
            audio_thread.join(timeout=1.0)
        ```
    

Helpful information about AssemblyAI’s WebSocket Closure:

- **Connection Diagnostics** - If the socket closes unexpectedly, AssemblyAI supplies both a status code and a reason message (close_status_code, close_msg), so you know immediately whether the server timed out, refused authentication, or encountered a different error.

## Session Shutdown

    
        ```python
        async with connect(session_data["url"]) as websocket:
            ...
            # Handle Ctrl+C to stop recording
            loop = asyncio.get_running_loop()
            loop.add_signal_handler(
                signal.SIGINT,
                lambda: loop.create_task(stop_recording(websocket)),
            )
        ```
    
    
    ```python expandable
    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")
    ```
    

Helpful information to know about AssemblyAI’s shutdown:

- **JSON Payload Difference** - When closing the stream with AssemblyAI, your JSON payload will be `{ "type": "Terminate" }` instead of `{ "type": "stop_recording" }`.

- **No Metadata Race Condition** - AssemblyAI provides session info at "Begin" and doesn’t append extra data at shutdown, making the exit faster and less error-prone.

## Resources

For additional information about using AssemblyAI's Streaming Speech-To-Text API you can also refer to:

- [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio)
- [Streaming Guide](/streaming/getting-started/transcribe-streaming-audio)

---

<a id="best-practices-for-building-meeting-notetakers-docs-meeting-notetaker-best-practices-mdx"></a>

# Best Practices for building Meeting Notetakers

URL: https://www.assemblyai.com/docs/meeting-notetaker-best-practices
Source: docs/meeting-notetaker-best-practices.mdx
Navigation: Real-time STT > Guides > Tutorials
Description: Complete guide for building meeting notetakers with AssemblyAI

## Introduction

Building a robust meeting notetaker requires careful consideration of accuracy, latency, speaker identification, and real-time capabilities. This guide addresses common questions and provides practical solutions for both post-call and live meeting transcription scenarios.

## Why AssemblyAI for Meeting Notetakers?

AssemblyAI stands out as the premier choice for meeting notetakers with several key advantages:

### Industry-Leading Accuracy with Pre-recorded Audio

- **93.3%+ transcription accuracy** ensures reliable meeting documentation
- **2.9% speaker diarization error rate** for precise "who said what" attribution
- **Speech Understanding** integration for intelligent post-processing and insights
- **Keyterms prompt** allows providing meeting context to improve accuracy of transcription

### Streaming with Universal-3.5 Pro

As meeting notetakers evolve toward real-time capabilities, AssemblyAI's Universal-3.5 Pro Streaming model (`universal-3-5-pro`) offers significant benefits:

- **Speaker diarization** available for both pre-recorded and streaming transcription
- **Ultra-low latency (~300ms)** enables live transcription without delays
- **Format turns** feature provides structured, readable output in real-time
- **Keyterms prompt** allows providing meeting context to improve accuracy of transcription

### End-to-End Voice AI Platform

Unlike fragmented solutions, AssemblyAI provides a unified API for:

- Transcription with speaker diarization
- Automatic language detection and code switching
- Boosting accuracy via meeting context with keyterms prompt
- Speech Understanding tasks like speaker identification, translation, and transcript styling
- Post-processing workflows with custom prompting - from summarization to completely custom workflows
- Real-time and batch processing of pre-recorded audio in a single platform

## When Should I Use Pre-recorded vs Streaming for Meeting Notetakers?

Understanding when to use pre-recorded versus streaming speech-to-text is critical for building the right meeting notetaker.

### Pre-recorded Speech-to-text

**Post-call analysis** - Meeting already happened, you have the full recording

- **Highest accuracy needed** - Pre-recorded models have higher accuracy (93.3%+)
- **Speaker diarization is critical** - Pre-recorded has 2.9% speaker error rate
- **Broad language support** - Need any of 99+ languages
- **Advanced features required** - Summarization, sentiment analysis, entity detection, PII redaction, speaker identification
- **Batch processing** - Processing multiple recordings at once
- **Quality over speed** - Can wait seconds/minutes for perfect results

**Best for:** Zoom/Teams/Meet recording uploads, compliance, documentation, post-call summaries, searchable archives

### Streaming Speech-to-text

**Live meetings** - Transcribing as the meeting happens

You should use streaming when you need to display a live transcript of text to users as they are speaking. With Universal-3.5 Pro Streaming, accuracy is closer to pre-recorded, but pre-recorded will always be the most accurate option.

- **Real-time captions** - Displaying subtitles/captions to participants during calls
- **Immediate feedback** - Need transcription within ~300ms
- **Interactive features** - Live note-taking, real-time keyword detection, action item alerts
- **No recording available** - Processing live audio only

**Best for:** Live captions, real-time note-taking apps, accessibility features, live keyword alerts

**Streaming is billed per session**

Streaming is billed on the total duration that your WebSocket connection stays open, not on the amount of audio you send. For long-running meetings, make sure to terminate sessions when the meeting ends to avoid being billed for idle time. See [Billing and pricing](/billing-and-pricing) for details.

### Hybrid Approach (Recommended)

Many successful meeting notetakers use **both** pre-recorded and streaming speech-to-text:

1. **Streaming during the call** - Provide live captions and real-time notes to participants
2. **Pre-recorded after the call** - Generate high-quality transcript with speaker labels, summary, and insights

This gives users immediate value during meetings while providing comprehensive documentation afterward.

**Example workflow:**

- User joins meeting → Start streaming for live captions
- Meeting ends → Upload recording to pre-recorded API for final transcript with speaker names
- Generate meeting summary, action items, and searchable archive from pre-recorded transcript

## What Languages and Features for a Meeting Notetaker?

### Pre-Recorded Meetings

For post-call analysis, AssemblyAI supports:

**Languages**:

- 99 languages supported
- Automatic Language Detection to route to the most spoken language
- Code Switching to preserve changes in speech between languages

**Core Features**:

- Speaker diarization (1-10 speakers by default, expandable to any min/max)
- Multichannel audio support (each channel = one speaker)
- Automatic formatting, punctuation, and capitalization
- Keyterms prompting for boosting domain-specific terms

**Speech Understanding Models**:

- Summarization for meeting recaps
- Sentiment analysis for meeting tone assessment
- Entity detection for extracting key information
- Speaker identification to map generic labels to actual names/roles
- Translation between 86 languages

### Real-Time Streaming

For live meeting transcription:

**Languages**:

- English-only model (default)
- Multilingual model supporting English, Spanish, French, German, Portuguese, and Italian

### Streaming (Universal-3.5 Pro Streaming)

- Speaker diarization for identifying who is speaking
- Partial and final transcripts for responsive UI
- Format turns for structured, readable output
- Keyterms prompt for contextual accuracy

See the [Universal-3.5 Pro Streaming documentation](/streaming/getting-started/transcribe-streaming-audio) for full details.

## How Can I Get Started Building a Post-Call Meeting Notetaker?

Here's a complete example implementing pre-recorded transcription with all essential features:

```python expandable
import assemblyai as aai
import asyncio
from typing import Dict, List
from assemblyai.types import (
    SpeakerOptions,
    LanguageDetectionOptions,
    PIIRedactionPolicy,
    PIISubstitutionPolicy,
)

# Configure API key
aai.settings.api_key = "your_api_key_here"

async def transcribe_meeting_async(audio_source: str) -> Dict:
    """
    Asynchronously transcribe a meeting recording with full features

    Args:
        audio_source: Either a local file path or publicly accessible URL
    """
    # Configure comprehensive meeting analysis
    config = aai.TranscriptionConfig(
        # Speaker diarization
        speaker_labels=True,
        speakers_expected=None,  # Use if you know exact number from Zoom/Meet/Teams
        speaker_options=SpeakerOptions(
            min_speakers_expected=2,
            max_speakers_expected=10  # Set a bit higher than expected; too high can cause over-splitting
        ),
        multichannel=False,  # Set to True if audio has separate channel per speaker

        # Language detection
        language_detection=True,  # Auto-detect the most used language
        language_detection_options=LanguageDetectionOptions(
            code_switching=True,  # Preserve language switches
            code_switching_confidence_threshold=0.5,
        ),

        # Punctuation and formatting
        punctuate=True,
        format_text=True,

        # Boost accuracy of meeting-specific vocabulary
        keyterms_prompt=["quarterly", "KPI", "roadmap", "deliverables"],

        # Speech Understanding - commonly used models
        summarization=True,
        sentiment_analysis=True,
        entity_detection=True,
        redact_pii=True,
        redact_pii_policies=[
            PIIRedactionPolicy.person_name,
            PIIRedactionPolicy.organization,
            PIIRedactionPolicy.occupation,
        ],
        redact_pii_sub=PIISubstitutionPolicy.hash,
        redact_pii_audio=True
    )

    # Create transcriber
    transcriber = aai.Transcriber()

    try:
        # Submit transcription job
        transcript = await asyncio.to_thread(
            transcriber.transcribe,
            audio_source,
            config=config
        )

        # Check status
        if transcript.status == aai.TranscriptStatus.error:
            raise Exception(f"Transcription failed: {transcript.error}")

        # Process speaker-labeled utterances
        print("\n=== SPEAKER-LABELED TRANSCRIPT ===\n")

        for utterance in transcript.utterances:
            # Format timestamp
            start_time = utterance.start / 1000  # Convert to seconds
            end_time = utterance.end / 1000

            # Print formatted utterance
            print(f"[{start_time:.1f}s - {end_time:.1f}s] Speaker {utterance.speaker}:")
            print(f"  {utterance.text}")
            print(f"  Confidence: {utterance.confidence:.2%}\n")

        # Print summary data
        print("\n=== MEETING SUMMARY ===\n")
        print({
            "id": transcript.id,
            "status": transcript.status,
            "duration": transcript.audio_duration,
            "speaker_count": len(set(u.speaker for u in transcript.utterances)),
            "word_count": len(transcript.words) if transcript.words else 0,
            "detected_language": transcript.language_code if hasattr(transcript, 'language_code') else None,
            "summary": transcript.summary,
        })

        return {
            "transcript": transcript,
            "utterances": transcript.utterances,
            "summary": transcript.summary,
        }

    except Exception as e:
        print(f"Error during transcription: {e}")
        raise

async def main():
    """
    Example usage with error handling
    """
    # Use either local file OR URL (not both)
    audio_source = "https://assembly.ai/wildfires.mp3"  # Or "path/to/recording.mp3"

    try:
        result = await transcribe_meeting_async(audio_source)

        # Additional processing
        print(f"\nTotal speakers identified: {len(set(u.speaker for u in result['utterances']))}")
        print(f"Meeting duration: {result['transcript'].audio_duration} seconds")

    except Exception as e:
        print(f"Failed to process meeting: {e}")

if __name__ == "__main__":
    asyncio.run(main())
```

## How Can I Get Started Building a During-Call Live Meeting Notetaker?

Here's a complete example for real-time streaming transcription with meeting-optimized settings:

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "your_api_key"

# Keyterms to improve recognition accuracy
KEYTERMS = [
    "Alice Johnson",
    "Bob Smith",
    "Carol Davis",
    "quarterly review",
    "action items",
    "follow up",
    "deadline",
    "budget"
]

# MEETING NOTETAKER CONFIGURATION (different from voice agents!)
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,  # ALWAYS TRUE for meetings - users need readable text

    # Meeting-optimized turn detection (wait longer than voice agents)
    # universal-3-5-pro defaults: min_turn_silence=100ms, max_turn_silence=1000ms
    "min_turn_silence": 560,  # Wait longer for natural pauses (voice agents use ~100ms)
    "max_turn_silence": 2000,  # Allow thinking pauses

    # Keyterms for accuracy - pass each term as a separate query parameter
    "keyterms_prompt": KEYTERMS,
}

API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS, doseq=True)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
transcript_buffer = []

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("=" * 80)
    print(f"[{datetime.now().strftime('%H:%M:%S')}] Meeting transcription started")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")
    print(f"Keyterms configured: {', '.join(KEYTERMS)}")
    print("=" * 80)
    print("\nSpeak into your microphone. Press Ctrl+C to stop.\n")

    def stream_audio():
        """Stream audio from microphone to WebSocket"""
        global stream
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                if not stop_event.is_set():
                    print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    """Handle incoming messages from AssemblyAI"""
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        # Uncomment to see full JSON for debugging:
        # print("=" * 80)
        # print(json.dumps(data, indent=2, ensure_ascii=False))
        # print("=" * 80)
        # print()

        if msg_type == "Begin":
            session_id = data.get("id", "N/A")
            print(f"[SESSION] Started - ID: {session_id}\n")

        elif msg_type == "Turn":
            end_of_turn = data.get("end_of_turn", False)
            transcript = data.get("transcript", "")
            turn_order = data.get("turn_order", 0)
            end_of_turn_confidence = data.get("end_of_turn_confidence", 0.0)

            # FOR MEETING NOTETAKERS: Show partials for responsive UI
            if not end_of_turn and transcript:
                print(f"\r[LIVE] {transcript}", end="", flush=True)

            # FOR MEETING NOTETAKERS: Use formatted finals for readable display
            # (Unlike voice agents which should use utterance for speed)
            if end_of_turn and transcript:
                timestamp = datetime.now().strftime('%H:%M:%S')
                print(f"\n[{timestamp}] {transcript}")
                print(f"           Turn: {turn_order} | Confidence: {end_of_turn_confidence:.2%}")

                # Detect action items
                transcript_lower = transcript.lower()
                if any(term in transcript_lower for term in ["action item", "follow up", "deadline", "assigned to", "todo"]):
                    print("           ⚠️  ACTION ITEM DETECTED!")

                # Store final transcript
                transcript_buffer.append({
                    "timestamp": timestamp,
                    "text": transcript,
                    "turn_order": turn_order,
                    "confidence": end_of_turn_confidence,
                    "type": "final"
                })
                print()

        elif msg_type == "Termination":
            audio_duration = data.get("audio_duration_seconds", 0)
            print(f"\n[SESSION] Terminated - Duration: {audio_duration}s")
            save_transcript()

        elif msg_type == "Error":
            error_msg = data.get("error", "Unknown error")
            print(f"\n[ERROR] {error_msg}")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\n[WEBSOCKET ERROR] {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\n[WEBSOCKET] Disconnected - Status: {close_status_code}, Message: {close_msg}")

    global stream, audio
    stop_event.set()

    # Clean up audio stream
    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def save_transcript():
    """Save the transcript to a file"""
    if not transcript_buffer:
        print("No transcript to save.")
        return

    filename = f"meeting_transcript_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"

    with open(filename, "w", encoding="utf-8") as f:
        f.write("Meeting Transcript\n")
        f.write(f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
        f.write(f"Keyterms: {', '.join(KEYTERMS)}\n")
        f.write("=" * 80 + "\n\n")

        for entry in transcript_buffer:
            f.write(f"[{entry['timestamp']}] {entry['text']}\n")
            f.write(f"Confidence: {entry['confidence']:.2%}\n\n")

    print(f"Transcript saved to: {filename}")

def run():
    """Main function to run the streaming transcription"""
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\n\nCtrl+C received. Stopping transcription...")
        stop_event.set()

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

  These settings wait longer before ending turns to accommodate natural
  conversation pauses and ensure readable formatted text for display. You can
  [tweak these settings](/streaming/getting-started/transcribe-streaming-audio) to get
  the best results for your notetaker.

## How Do I Handle Multichannel Meeting Audio?

Many meeting platforms (Zoom, Teams, Google Meet) can record each participant on separate audio channels. This dramatically improves speaker identification accuracy.

### For Pre-recorded Meetings

```python
config = aai.TranscriptionConfig(
    multichannel=True,  # Enable when each speaker is on different channel
    speaker_labels=False,  # Disable - channels already separate speakers
    # Other settings...
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file, config=config)

# Access per-channel transcripts
for channel, channel_transcript in enumerate(transcript.channels):
    print(f"\n=== Channel {channel} ===")
    print(channel_transcript.text)
```

**When to use multichannel:**

- Zoom local recordings with "Record separate audio file for each participant" enabled
- Professional podcast recordings with individual microphones
- Conference systems with dedicated channels per participant
- Phone calls with caller and callee on separate channels

**Benefits:**

- **Perfect speaker separation** - No diarization errors
- **No speaker confusion or overlap issues**
- **Faster processing time** - Diarization not needed
- **Higher accuracy** - Model processes clean single-speaker audio

**How to enable in meeting platforms:**

- **Zoom**: Settings → Recording → Advanced → "Record a separate audio file for each participant"
- **Teams**: Requires third-party recording solutions like [Recall.ai](https://www.recall.ai/)
- **Google Meet**: Requires third-party recording solutions like [Recall.ai](https://www.recall.ai/)

### For Streaming Meetings

For real-time multichannel audio, create separate streaming sessions per channel:

```python expandable
import asyncio
import websockets

class ChannelTranscriber:
    def __init__(self, channel_id: int, speaker_name: str):
        self.channel_id = channel_id
        self.speaker_name = speaker_name
        self.connection_params = {
            "sample_rate": 16000,
            "speech_model": "universal-3-5-pro",
            "format_turns": True,
        }

    async def transcribe_channel(self, audio_stream):
        """Transcribe a single audio channel"""
        url = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(self.connection_params)}"

        # If you're using `websockets` version 13.0 or later, use `additional_headers` parameter. For older versions (< 13.0), use `extra_headers` instead.
        async with websockets.connect(url, additional_headers={"Authorization": API_KEY}) as ws:
            # Send audio from this channel only
            async for audio_chunk in audio_stream:
                await ws.send(audio_chunk)

            # Receive transcripts
            async for message in ws:
                data = json.loads(message)
                if data.get("type") == "Turn" and data.get("end_of_turn"):
                    print(f"{self.speaker_name}: {data['transcript']}")

# Create transcriber for each channel
async def transcribe_multichannel_meeting(channel_audio_streams):
    transcribers = [
        ChannelTranscriber(0, "Alice"),
        ChannelTranscriber(1, "Bob"),
    ]

    # Run all channels concurrently
    await asyncio.gather(*[
        t.transcribe_channel(stream)
        for t, stream in zip(transcribers, channel_audio_streams)
    ])
```

See our [multichannel streaming guide](/streaming/label-speakers-and-separate-channels#multichannel-streaming-audio) for complete implementation details.

## How Should I Handle Pre-recorded Transcription in Production?

Choose the right approach based on your application's needs:

### Option 1: Simple Blocking Call

```python
# Simple blocking call
transcript = await asyncio.to_thread(transcriber.transcribe, audio_url, config=config)
```

**Pros:**

- Simple, straightforward code
- Good for low volume applications
- Easy to understand and debug

**Cons:**

- Ties up resources while waiting
- Not suitable for high volume
- Cannot process multiple files simultaneously

**Best for:** Personal projects, prototypes, low-traffic applications

### Option 2: Webhook Callbacks (Production Recommended)

```python
config = aai.TranscriptionConfig(
    webhook_url="https://your-app.com/webhooks/assemblyai",
    webhook_auth_header_name="X-Webhook-Secret",
    webhook_auth_header_value="your_secret_here",
    speaker_labels=True,
    summarization=True,
    # ... other config
)

# Submit job and return immediately (non-blocking)
transcript = transcriber.submit(audio_url, config=config)
print(f"Job submitted: {transcript.id}")
# Your app can continue processing other requests

# Your webhook receives results when ready (typically 15-30% of audio duration)
```

**Webhook handler example:**

```python expandable
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhooks/assemblyai", methods=["POST"])
def assemblyai_webhook():
    # Verify webhook authenticity
    if request.headers.get("X-Webhook-Secret") != "your_secret_here":
        return jsonify({"error": "Unauthorized"}), 401

    import requests as http_requests

    data = request.json
    transcript_id = data["transcript_id"]
    status = data["status"]

    if status == "completed":
        # Fetch the full transcript (webhook only sends transcript_id and status)
        transcript = http_requests.get(
            f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
            headers={"authorization": "your_api_key"}
        ).json()
        process_completed_meeting(transcript)
    elif status == "error":
        log_transcription_error(transcript_id)

    return jsonify({"received": True}), 200

def process_completed_meeting(transcript):
    """Process completed meeting transcript"""
    utterances = transcript["utterances"]
    summary = transcript["summary"]

    # Store in database
    save_to_database(transcript)

    # Notify user
    send_notification(transcript["id"])
```

**Pros:**

- Non-blocking - submit and forget
- Scales to high volume
- Process multiple files in parallel
- Automatic retry on failures
- Get notified when complete

**Best for:** Production apps, user-uploaded recordings, batch processing, SaaS products

### Option 3: Polling (Custom Workflows)

```python
# Submit job
transcript = transcriber.submit(audio_url, config=config)
print(f"Submitted: {transcript.id}")

# Poll for completion with progress tracking
while transcript.status not in [aai.TranscriptStatus.completed, aai.TranscriptStatus.error]:
    await asyncio.sleep(5)
    transcript = transcriber.get_transcript(transcript.id)

    # Optional: Show progress
    print(f"Status: {transcript.status}...")

if transcript.status == aai.TranscriptStatus.completed:
    process_transcript(transcript)
else:
    print(f"Error: {transcript.error}")
```

**Pros:**

- Full control over retry logic
- Can show progress to users
- Good for background jobs
- Works without webhook infrastructure

**Cons:**

- Must implement your own polling loop
- Ties up resources while polling
- More complex than webhooks

**Best for:** Background job processors, CLIs with progress bars, custom retry logic

### Comparison Table

| Method      | Blocking | Scalability | Complexity | Best For                     |
| ----------- | -------- | ----------- | ---------- | ---------------------------- |
| Blocking    | Yes      | Low         | Low        | Prototypes, low volume       |
| Webhooks    | No       | High        | Medium     | Production, high volume      |
| Polling     | Partial  | Medium      | Medium     | Background jobs, progress UI |

### Scaling Considerations

- **HTTP rate limit:** 20,000 requests per 5-minute window, counted across submissions (POST) and polling (GET) combined
- **Exceeding the limit:** returns a `403` response
- **Parallel transcriptions (rate limit):** 200+ for paid accounts (queued beyond that)
- **Ramp up gradually:** start at 10-50 parallel requests, double incrementally
- **Avoid the rate limit:** use [webhooks](/pre-recorded-audio/webhooks) or jittered, widened polling — see [Polling without exceeding the rate limit](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale#polling-without-exceeding-the-rate-limit)
- **Contact Sales** before large-scale rollouts

## How Do I Identify Speakers in My Recording?

Speaker diarization tells you **when** speakers change ("Speaker A", "Speaker B"), but **Speaker Identification** tells you **who** they are by name or role.

### Why Use Speaker Identification?

**Instead of:**

```
Speaker A: Let's review the Q3 numbers.
Speaker B: Revenue was up 15% this quarter.
Speaker A: Excellent work on that launch.
```

**You get:**

```
Sarah Chen: Let's review the Q3 numbers.
Michael Rodriguez: Revenue was up 15% this quarter.
Sarah Chen: Excellent work on that launch.
```

### How It Works

Speaker Identification uses AssemblyAI's Speech Understanding API to map generic speaker labels to actual names or roles that you provide:

```python expandable
import assemblyai as aai

aai.settings.api_key = "your_api_key"

# Step 1: Transcribe with speaker diarization
config = aai.TranscriptionConfig(
    speaker_labels=True,  # Must enable speaker diarization first
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "name",  # or "role"
                "known_values": ["Sarah Chen", "Michael Rodriguez", "Alex Kim"]
            }
        }
    }
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe("meeting_recording.mp3", config=config)

# Access results with identified speakers
for utterance in transcript.utterances:
    print(f"{utterance.speaker}: {utterance.text}")
```

### Identifying by Role Instead of Name

For customer service, sales calls, or scenarios where you don't know names:

```python
config = aai.TranscriptionConfig(
    speaker_labels=True,
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "known_values": ["Agent", "Customer"]  # or ["Interviewer", "Interviewee"]
            }
        }
    }
)
```

**Common role combinations:**

- `["Agent", "Customer"]` - Customer service calls
- `["Support", "Customer"]` - Technical support
- `["Interviewer", "Interviewee"]` - Interviews
- `["Host", "Guest"]` - Podcasts
- `["Doctor", "Patient"]` - Medical consultations (with HIPAA compliance)

### How to Get Speaker Names

**For platform recordings:**

1. **Zoom**: Extract participant names from Zoom API or meeting JSON
2. **Teams**: Get attendees from Microsoft Graph API
3. **Google Meet**: Use Google Calendar API to get participants

**Example with Zoom:**

```python
# Get participant names from Zoom meeting
zoom_participants = get_zoom_meeting_participants(meeting_id)
speaker_names = [p["name"] for p in zoom_participants]

# Use in speaker identification
config = aai.TranscriptionConfig(
    speaker_labels=True,
    speakers_expected=len(speaker_names),  # Exact number of speakers to detect
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": speaker_names
            }
        }
    }
)
```

### How Speaker Identification Works

**Speaker Identification Requirements:**

1. **Speaker diarization must be enabled** - Cannot identify speakers without diarization first
2. **Requires sufficient audio per speaker** - Each speaker needs enough speech for accurate matching
3. **Works best with distinct voices** - Similar voices may be confused
4. **Post-processing step** - Adds additional processing time after transcription

**Accuracy depends on:**

- Audio quality (clear, minimal background noise)
- Voice distinctiveness (different genders, accents, tones)
- Amount of speech per speaker (more = better)
- Number of speakers (fewer = more accurate)

### Alternative: Add Identification Later

You can add speaker identification to an existing transcript by posting to the Speech Understanding API with the `transcript_id`. This is useful when you get speaker names after the transcription completes, or when building iterative workflows where users confirm speaker identities.

```python expandable
import requests

# First, transcribe with speaker diarization
transcript = transcriber.transcribe(audio_url, config=aai.TranscriptionConfig(speaker_labels=True))

# Later, add speaker identification using the transcript ID
understanding_body = {
    "transcript_id": transcript.id,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "known_values": ["Sarah Chen", "Michael Rodriguez"]
            }
        }
    }
}

result = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers={"Authorization": aai.settings.api_key},
    json=understanding_body
).json()

# Access identified speakers from the response
for utterance in result["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")
```

This approach is useful when:

- You get speaker names after the transcription completes
- You want to try different name mappings
- Building iterative workflows where users confirm speaker identities

For complete API details, see our [Speaker Identification documentation](/speech-understanding/speaker-identification).

## How Do I Translate Between Languages in Meetings?

AssemblyAI supports translation between 86 languages, enabling you to transcribe meetings in one language and translate to another.

### When to Use Translation

**Common use cases:**

- Transcribe Spanish meeting → Translate to English for documentation
- Transcribe multilingual meeting → Translate all to common language
- Create translated meeting notes for international teams
- Provide translated summaries for stakeholders

### Basic Translation

Translation is a Speech Understanding feature. You enable it via the `speech_understanding` parameter with `target_languages`:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

# Configure transcription with translation
data = {
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "translation": {
                "target_languages": ["es", "de"],
                "formal": True
            }
        }
    }
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

print("--- Original Transcript ---")
print(transcript["text"][:200] + "...")

print("\n--- Translations ---")
for language_code, translated_text in transcript["translated_texts"].items():
    print(f"{language_code.upper()}:")
    print(translated_text[:200] + "...")
```

### Translation with Speaker Labels

For meetings where you need per-utterance translations with speaker attribution:

```python
data = {
    "audio_url": audio_url,
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "translation": {
                "target_languages": ["es"],
                "match_original_utterance": True,
                "formal": True
            }
        }
    }
}

for utterance in transcript["utterances"]:
    print(f"Speaker {utterance['speaker']}:")
    print(f"  Original: {utterance['text'][:100]}...")
    print(f"  Spanish: {utterance['translated_texts']['es'][:100]}...")
```

### Supported Language Pairs

AssemblyAI supports translation between **86 languages**, including:

**Popular combinations:**

- Spanish ↔ English
- French ↔ English
- German ↔ English
- Mandarin ↔ English
- Japanese ↔ English
- Portuguese ↔ English
- And all combinations between supported languages

### Translation Response Format

The response includes `translated_texts` as a dictionary keyed by language code:

```python
{
    "text": "Original transcript in source language",
    "translated_texts": {
        "es": "Translated transcript in Spanish",
        "de": "Translated transcript in German"
    },
    "utterances": [
        {
            "speaker": "A",
            "text": "Hello, how are you?",
            "translated_texts": {
                "es": "Hola, ¿cómo estás?"
            },
            "start": 0,
            "end": 1500
        }
    ]
}
```

For complete language support and translation details, see our [Translation documentation](/speech-understanding/translation).

## What Workflows Can I Build for My AI Meeting Notetaker?

Use these Speech Understanding and Guardrails features to transform raw transcripts into actionable insights.

### Summarization

`summarization: true`

**What it does:** Generates an abstractive recap of the conversation (not verbatim).  
**Output:** `summary` string (bullets/paragraph format).  
**Great for:** Meeting notes, call recaps, executive summaries.  
**Notes:** Condenses and rephrases; minor details may be omitted by design.

**Example:**

```python
config = aai.TranscriptionConfig(
    summarization=True,
    summary_type="bullets",  # or "bullets_verbose", "gist", "headline", "paragraph"
    summary_model="informative",  # or "conversational"
)
```

### Sentiment Analysis

`sentiment_analysis: true`

**What it does:** Scores per-utterance sentiment (positive / neutral / negative).  
**Output:** Array of `{ text, sentiment, confidence, start, end }`.  
**Great for:** Customer satisfaction tracking, coaching, churn prediction.  
**Notes:** Segment-level (not global mood); sarcasm and very short utterances are harder to classify.

**Example:**

```python
for utterance in transcript.sentiment_analysis_results:
    if utterance.sentiment == "NEGATIVE":
        print(f"Negative sentiment detected: {utterance.text}")
```

### Entity Detection

`entity_detection: true`

**What it does:** Extracts named entities (people, organizations, locations, products, etc.).  
**Output:** Array of `{ entity_type, text, start, end }`.  
**Great for:** Auto-tagging topics, tracking competitors mentioned, CRM enrichment.  
**Notes:** Operates on post-redaction text if PII redaction is enabled.

**Example:**

```python
# Extract all organizations mentioned
organizations = [
    entity.text for entity in transcript.entities
    if entity.entity_type == "organization"
]
print(f"Companies mentioned: {', '.join(organizations)}")
```

### Redact PII Text

`redact_pii: true`

**What it does:** Scans transcript for personally identifiable information and replaces matches per policy.  
**Output:** `text` with replacements; original `words` timing preserved.  
**Great for:** GDPR/CCPA compliance, safe sharing, SOC2 requirements.  
**Notes:** Runs **before** downstream features; they see the redacted text.

**Recommended policies for meetings:**

```python
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,      # Remove names
        PIIRedactionPolicy.email_address,    # Remove emails
        PIIRedactionPolicy.phone_number,     # Remove phone numbers
        PIIRedactionPolicy.organization,     # Remove company names
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,  # Stable hash tokens
)
```

**Why hash substitution?**

- Stable across the file (same value → same token)
- Maintains sentence structure for LLM processing
- Prevents reconstruction of original data

### Redact PII Audio

`redact_pii_audio: true`

**What it does:** Produces a second audio file where redacted portions are bleeped/silenced.  
**Output:** `redacted_audio_url` in the transcript response.  
**Great for:** External sharing, training materials, demos.  
**Notes:** Original audio is untouched; bleeped sections may sound choppy.

### Complete Example

```python expandable
config = aai.TranscriptionConfig(
    # Core transcription
    speaker_labels=True,

    # Speech Understanding
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,

    # PII protection
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.email_address,
        PIIRedactionPolicy.phone_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
    redact_pii_audio=True,
)

transcript = transcriber.transcribe(audio_url, config=config)

# Access all features
meeting_insights = {
    "summary": transcript.summary,
    "sentiment_trend": analyze_sentiment_trend(transcript.sentiment_analysis_results),
    "entities": extract_entities(transcript.entities),
    "safe_transcript": transcript.text,  # PII redacted
    "safe_audio": transcript.redacted_audio_url,  # PII bleeped
}
```

## How Do I Improve the Accuracy of My Notetaker?

**Best practices:**

- Include participant names for better speaker recognition
- Add company-specific jargon and acronyms
- Include product names and technical terms
- Keep individual terms under 50 characters
- Up to 200 terms per request (Universal-2) or 1000 terms (Universal-3.5 Pro)

### Using Keyterms Prompt for Pre-recorded Transcription

Keyterms prompting improves recognition accuracy for domain-specific vocabulary by up to 21%:

```python expandable
# Define domain-specific vocabulary
company_terms = [
    "AssemblyAI",
    "Universal-3.5 Pro",
    "Speech Understanding",
    "diarization"
]

participant_names = [
    "Dylan Fox",
    "Sarah Chen",
    "Michael Rodriguez"
]

technical_terms = [
    "API endpoint",
    "WebSocket",
    "latency metrics",
    "TTFT"
]

# Configure with keyterms prompt
config = aai.TranscriptionConfig(
    keyterms_prompt=company_terms + participant_names + technical_terms,
    speaker_labels=True,
    # ... other settings
)
```

### Using Keyterms Prompt for Streaming

```python expandable
# Streaming with contextual keyterms
keyterms = [
    # Participant names
    "Alice Johnson",
    "Bob Smith",

    # Meeting-specific vocabulary
    "Q4 objectives",
    "revenue targets",
    "customer acquisition",

    # Technical terms
    "API integration",
    "cloud migration"
]

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "keyterms_prompt": keyterms,
}
```

## How Do I Process the Response from the API?

### Processing Pre-recorded Responses

```python expandable
def process_transcript(transcript):
    """
    Extract and process all relevant data from pre-recorded transcript
    """
    # Basic transcript data
    meeting_data = {
        "id": transcript.id,
        "duration": transcript.audio_duration,
        "confidence": transcript.confidence,
        "full_text": transcript.text
    }

    # Process speaker utterances
    speakers = {}
    for utterance in transcript.utterances:
        speaker = utterance.speaker

        if speaker not in speakers:
            speakers[speaker] = {
                "utterances": [],
                "total_speaking_time": 0,
                "word_count": 0
            }

        speakers[speaker]["utterances"].append({
            "text": utterance.text,
            "start": utterance.start,
            "end": utterance.end,
            "confidence": utterance.confidence
        })

        # Calculate speaking time
        speakers[speaker]["total_speaking_time"] += (utterance.end - utterance.start) / 1000
        speakers[speaker]["word_count"] += len(utterance.text.split())

    meeting_data["speakers"] = speakers

    # Extract summary
    if transcript.summary:
        meeting_data["summary"] = transcript.summary

    # Calculate meeting statistics
    total_duration = transcript.audio_duration  # Already in seconds
    meeting_data["statistics"] = {
        "total_speakers": len(speakers),
        "total_words": sum(s["word_count"] for s in speakers.values()),
        "average_confidence": transcript.confidence,
        "speaking_distribution": {
            speaker: {
                "percentage": (data["total_speaking_time"] / total_duration) * 100,
                "minutes": data["total_speaking_time"] / 60
            }
            for speaker, data in speakers.items()
        }
    }

    return meeting_data

# Example usage
result = process_transcript(transcript)
print(f"Meeting had {result['statistics']['total_speakers']} speakers")
print(f"Speaker A spoke for {result['statistics']['speaking_distribution']['A']['minutes']:.1f} minutes")
```

### Processing Streaming Responses

```python expandable
class StreamingResponseProcessor:
    def __init__(self):
        self.partial_buffer = ""
        self.final_transcripts = []
        self.turn_metadata = []

    def process_message(self, message: dict):
        """
        Process real-time streaming messages
        """
        msg_type = message.get("type")

        if msg_type == "Begin":
            return {
                "event": "session_started",
                "session_id": message.get("id"),
                "expires_at": message.get("expires_at")
            }

        elif msg_type == "Turn":
            return self.process_turn(message)

        elif msg_type == "Termination":
            return {
                "event": "session_ended",
                "audio_duration": message.get("audio_duration_seconds"),
                "session_duration": message.get("session_duration_seconds")
            }

    def process_turn(self, data: dict):
        """Process turn messages"""
        is_final = data.get("end_of_turn")
        transcript = data.get("transcript", "")
        turn_order = data.get("turn_order")

        response = {
            "turn_order": turn_order,
            "is_final": is_final,
            "confidence": data.get("end_of_turn_confidence", 0)
        }

        # Handle partials (for live display)
        if not is_final and transcript:
            self.partial_buffer = transcript
            response["event"] = "partial"
            response["text"] = transcript

        # Handle finals (for storage)
        elif is_final:
            final_transcript = {
                "turn_order": turn_order,
                "text": transcript,
                "confidence": data.get("end_of_turn_confidence"),
                "timestamp": datetime.now().isoformat()
            }
            self.final_transcripts.append(final_transcript)
            response["event"] = "final"
            response["text"] = transcript

            # Clear partial buffer
            self.partial_buffer = ""

        return response

    def get_full_transcript(self):
        """
        Combine all final transcripts into complete meeting transcript
        """
        return {
            "full_text": " ".join(t["text"] for t in self.final_transcripts),
            "transcripts": self.final_transcripts,
            "total_turns": len(self.final_transcripts)
        }

# Example usage
processor = StreamingResponseProcessor()

# If you're using `websockets` version 13.0 or later, use `additional_headers` parameter. For older versions (< 13.0), use `extra_headers` instead.
async with websockets.connect(API_ENDPOINT, additional_headers=headers) as ws:
    async for message in ws:
        data = json.loads(message)
        result = processor.process_message(data)

        if result["event"] == "partial":
            # Update UI with live transcript
            update_live_caption(result["text"])

        elif result["event"] == "final":
            # Save final transcript
            save_transcript_segment(result)

# Get complete transcript when done
full_transcript = processor.get_full_transcript()
```

## Additional Resources

- [Universal Pre-recorded Documentation](/pre-recorded-audio)
- [Universal-3.5 Pro Streaming Documentation](/streaming)
- [Speaker Diarization Guide](/pre-recorded-audio/label-speakers)
- [Speaker Identification Guide](/speech-understanding/speaker-identification)
- [Translation Guide](/speech-understanding/translation)
- [Getting Started Guide](/pre-recorded-audio/getting-started/transcribe-an-audio-file)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)
- [Support](https://www.assemblyai.com/contact/support)

---

<a id="build-a-medical-scribe-docs-medical-scribe-best-practices-mdx"></a>

# Build a medical scribe

URL: https://www.assemblyai.com/docs/medical-scribe-best-practices
Source: docs/medical-scribe-best-practices.mdx
Navigation: Real-time STT > Guides > Tutorials
Description: Build AI-powered medical scribes that transcribe patient encounters and generate clinical documentation.

AssemblyAI provides everything you need to build a medical scribe, from high-accuracy transcription with medical terminology support to HIPAA-compliant PII redaction and structured clinical note generation through the LLM Gateway.

Choose the guide that matches your clinical workflow:

  
    Transcribe recorded patient encounters with Universal-3.5 Pro. Includes Medical Mode, speaker diarization, entity detection, PII redaction, and SOAP note generation.
  
  
    Stream audio from a microphone during live encounters with Universal-3.5 Pro Streaming. Includes LLM Gateway post-processing, medical keyterms, and automatic SOAP note generation.
  

## Which approach should I use?

| Scenario | Recommended approach |
| --- | --- |
| Post-visit documentation with highest accuracy | [Post-visit medical scribe](/medical-scribe-best-practices/medical-scribe-post-visit) |
| Live encounter transcription during patient visits | [Real-time medical scribe](/medical-scribe-best-practices/medical-scribe-real-time) |
| Telemedicine or emergency department visits | [Real-time medical scribe](/medical-scribe-best-practices/medical-scribe-real-time) |
| Specialist consultations with complex terminology | [Post-visit medical scribe](/medical-scribe-best-practices/medical-scribe-post-visit) |
| Hybrid: real-time notes with post-visit verification | Use both guides together |

---

<a id="best-practices-for-building-voice-agents-docs-voice-agents-best-practices-mdx"></a>

# Best Practices for Building Voice Agents

URL: https://www.assemblyai.com/docs/voice-agents/best-practices
Source: docs/voice-agents/best-practices.mdx
Navigation: Real-time STT > Guides > Tutorials
Description: Complete guide for building voice agents with AssemblyAI's Universal-3 Pro Streaming

## Introduction

AssemblyAI's **Universal-3.5 Pro Streaming** is the most accurate real-time speech-to-text model designed for voice agents. It delivers formatted, immutable transcripts with sub-300ms latency, exceptional entity accuracy, native multilingual code switching, and a fully promptable interface, all optimized for conversational AI workflows.

The STT component is the "ears" of your voice agent. Transcription errors propagate into the LLM and response logic, so even small accuracy gaps compound in impact. Choosing and configuring the right STT model is one of the highest-leverage decisions you can make when building a voice agent. For guidance on how to evaluate and compare STT models for your use case, see the [streaming evaluation guide](/streaming/evaluations/voice-agents).

## Why Universal-3.5 Pro Streaming for Voice Agents?

Voice agents need speed, accuracy, and natural turn-taking. Universal-3.5 Pro Streaming is purpose-built for this:

**Sub-300ms latency with formatted output**

- Immutable transcripts arrive fully formatted (punctuation, capitalization), no waiting for a separate formatting step
- Every final transcript is ready for immediate LLM processing

**Exceptional entity accuracy**

- Credit card numbers, phone numbers, email addresses, physical addresses, and names are transcribed with high accuracy
- Short utterances like "yes", "no", "mmhmm" are handled reliably

**Punctuation-based turn detection**

- Turn boundaries are determined by terminal punctuation (`.` `?` `!`) combined with silence thresholds
- Configurable `min_turn_silence` and `max_turn_silence` parameters let you tune responsiveness vs. accuracy
- No confidence-score guessing. The model understands when a sentence is complete

**Fully promptable**

- Contextual `prompt` parameter for describing what the audio is about
- Dynamic prompting mid-session via `UpdateConfiguration`. Adapt the model to each stage of the conversation
- `keyterms_prompt` for boosting recognition of specific names, brands, and domain terms

**Native multilingual support**

- Supports English, Spanish, French, German, Italian, and Portuguese
- Automatic code-switching between languages within a single session
- Language-specific prompting for improved accuracy

## What Languages Does Universal-3.5 Pro Streaming Support?

Universal-3.5 Pro Streaming supports 18 languages with automatic code-switching:

- English
- Spanish
- German
- French
- Portuguese
- Italian
- Turkish
- Dutch
- Swedish
- Norwegian
- Danish
- Finnish
- Hindi
- Vietnamese
- Arabic
- Hebrew
- Japanese
- Mandarin

The model handles code-switching natively. Speakers can switch between supported languages mid-conversation without any configuration changes. Accuracy improves when you specify the expected language in the prompt. See [Supported languages](/streaming/getting-started/transcribe-streaming-audio) for the full language list and regional dialect reference.

To bias the model toward a specific language, pass the `language_code` connection parameter — see [Language selection](/streaming/getting-started/optimizing-accuracy-and-latency#language-selection). For multilingual conversations, no configuration is needed — code-switching is native.

## How Do I Get Started?

### Complete voice agent stack

AssemblyAI provides a speech-to-speech [Voice Agent API](/voice-agents/voice-agent-api) that abstracts away the complexity of a full voice agent stack: managed STT, LLM, turn detection, and TTS in a single endpoint.

For a cascading architecture, AssemblyAI has the best speech-to-text model. For a complete stack, you need:

1. **Speech-to-Text (STT):** AssemblyAI Universal-3.5 Pro Streaming
2. **Large Language Model (LLM):** OpenAI, Anthropic, Google, etc.
3. **Text-to-Speech (TTS):** Rime, Cartesia, ElevenLabs, etc.
4. **Orchestration:** LiveKit, Pipecat, or custom build

### Pre-built integrations

**LiveKit Agents (recommended)**

LiveKit provides the fastest path to a working voice agent with AssemblyAI. See [Universal-3.5 Pro Streaming on LiveKit](/voice-agents/livekit-universal-3-5-pro) for a full guide.

```python
from livekit.agents import AgentSession
from livekit.plugins import assemblyai, silero

session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(
        activation_threshold=0.3,
    ),
    turn_detection="stt",
    min_endpointing_delay=0,
)
```

**Pipecat by Daily**

Pipecat is an open-source framework for conversational AI with maximum customizability. See [Universal-3.5 Pro Streaming on Pipecat](/voice-agents/pipecat-universal-3-5-pro) for a full guide.

```python
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.assemblyai.models import AssemblyAIConnectionParams

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    connection_params=AssemblyAIConnectionParams(
        speech_model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
    ),
    vad_force_turn_endpoint=False,  # Use AssemblyAI's built-in turn detection
)
```

### Direct WebSocket connection

For custom builds, connect directly to the WebSocket API:

```python expandable
import json
import pyaudio
import websocket
import threading
import time
from urllib.parse import urlencode

API_KEY = "YOUR_API_KEY"
SAMPLE_RATE = 16000

CONNECTION_PARAMS = {
    "sample_rate": SAMPLE_RATE,
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 1000,
}

API_ENDPOINT_BASE = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE}?{urlencode(CONNECTION_PARAMS)}"

def on_message(ws, message):
    data = json.loads(message)

    if data.get("type") == "Turn":
        transcript = data.get("transcript", "")
        end_of_turn = data.get("end_of_turn", False)

        if end_of_turn:
            # Final transcript - send to LLM
            print(f"Final: {transcript}")
        else:
            # Partial - can start pre-emptive LLM generation
            print(f"Partial: {transcript}")

    elif data.get("type") == "SpeechStarted":
        # User started speaking - handle barge-in
        print("Speech detected, interrupt agent if speaking")

ws = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": API_KEY},
    on_message=on_message,
)
```

## How Does Turn Detection Work?

Universal-3.5 Pro Streaming uses a **punctuation-based turn detection system** controlled by two parameters:

| Parameter          | Default    | Description                                                              |
| ------------------ | ---------- | ------------------------------------------------------------------------ |
| `min_turn_silence` | `100` ms   | Silence before a speculative end-of-turn check fires.                    |
| `max_turn_silence` | `1000` ms  | Maximum silence before forcing the turn to end.                          |

**How it works:**

1. User speaks → audio streams to AssemblyAI
2. User pauses for `min_turn_silence` → model checks for terminal punctuation (`.` `?` `!`)
3. If terminal punctuation found → turn ends immediately with `end_of_turn: true`
4. If no terminal punctuation → partial emitted with `end_of_turn: false`, turn continues
5. If silence reaches `max_turn_silence` → turn forced to end regardless of punctuation

This is different from the legacy Universal-Streaming models, which used a confidence-based `end_of_turn_confidence_threshold`. Universal-3.5 Pro Streaming does not use that parameter. Turn decisions are based on punctuation after silence thresholds.

### Configuration presets

```python
# Fast - quick confirmations, IVR, yes/no questions
fast_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 800,
}

# Balanced - most voice agent conversations (recommended)
balanced_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 1000,
}

# Patient - entity dictation, complex instructions
patient_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 200,
    "max_turn_silence": 2000,
}
```

### Entity splitting tradeoff

Lower silence values produce faster transcripts but can split entities across turns:

```text
# With (min_turn_silence=100, max_turn_silence=1000)
"It's John."                    → turn ends (period found after 100ms pause)
"Smith."                        → new turn
"At gmail.com."                 → new turn

# With (min_turn_silence=400, max_turn_silence=2000)
"It's john.smith@gmail.com."    → single turn (properly formatted)
```

For voice agents, the downstream LLM can usually piece together split entities. But if your use case involves entity extraction or alphanumeric dictation, increase `min_turn_silence` and `max_turn_silence` during those portions of the conversation using [dynamic configuration updates](#how-do-i-update-configuration-mid-session).

## How Do I Handle Barge-In and Interruptions?

### SpeechStarted events

Universal-3.5 Pro Streaming emits `SpeechStarted` events when speech has been detected. `SpeechStarted` is only emitted when the model produces a transcript. This makes it a reliable signal for barge-in handling:

```json
{
  "type": "SpeechStarted",
  "timestamp": 14400,
  "confidence": 0.79
}
```

When you receive a `SpeechStarted` event:
1. Stop TTS playback immediately
2. Switch the agent back to listening mode
3. Wait for the user's full turn to complete before responding

### Filtering backchannels

Users often produce short backchannel utterances ("mhm", "yeah", "um", "okay") while the agent is speaking. Treating every `SpeechStarted` event as a barge-in causes the agent to stop mid-sentence on these fillers, even though the user didn't intend to interrupt.

The fix is to gate barge-in on each `Turn` event during agent speech: suppress the interrupt when the transcript is short or every token is a known backchannel. Implementation depends on your stack:

- **Voice Agent API**: semantic interruption classification is built in. See [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions).
- **LiveKit**: LiveKit Cloud users should enable [adaptive interruption handling](/voice-agents/livekit-universal-3-5-pro#interruption-handling). Self-hosted deployments use the two custom filters in the same section.
- **Direct WebSocket**: see [Interruption handling](/voice-agents/universal-3-5-pro-streaming-api#interruption-handling) on the Universal-3.5 Pro Streaming API page for the combined word-count + filler-word filter.

### VAD threshold alignment

Universal-3.5 Pro Streaming includes an internal Silero VAD controlled by the `vad_threshold` parameter (default `0.3`). If you're also running a local VAD (common in LiveKit and Pipecat), align the thresholds to avoid a dead zone where one detects speech but the other doesn't:

```python
# Both thresholds aligned at 0.3
stt = assemblyai.STT(
    model="universal-3-5-pro",
    vad_threshold=0.3,
)
vad = silero.VAD.load(
    activation_threshold=0.3,
)
```

If you're in a noisy environment and getting false speech triggers, raise both thresholds together.

## How Can I Use Prompting to Improve Accuracy?

### The prompt parameter

Universal-3.5 Pro Streaming supports a `prompt` parameter for [contextual prompting](/streaming/prompting-and-keyterms) — a natural-language description of what the audio is about. Transcription behavior itself (verbatim output, punctuation, turn detection) is built in and managed automatically; the prompt carries context, not instructions.

  **Beta feature**

Prompting is a beta feature. We recommend starting without a prompt to establish baseline performance, then adding context to optimize for your use case.

```python
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "prompt": "AI voice agent call with a customer about an internet service outage."
}
```

**Tips for effective prompts:**

- **Describe the conversation**: domain, scenario, or full details — start broad, and add only details your application actually knows (see the [three context levels](/streaming/prompting-and-keyterms#contextual-prompting))
- **Include names and identifiers you already know**: caller name, account or order IDs, products — detailed context helps the model spell them correctly
- **Specify language**: use the `language_code` connection parameter for monolingual sessions ([Language selection](/streaming/getting-started/optimizing-accuracy-and-latency#language-selection))

### Keyterms prompting

Use `keyterms_prompt` to boost recognition of specific names, brands, or domain terms, up to 100 terms per session:

```python
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "keyterms_prompt": json.dumps([
        "AssemblyAI",
        "LiveKit",
        "Dr. Rodriguez",
        "Lisinopril",
        "iPhone 15 Pro",
    ])
}
```

**Best practices for keyterms:**

- Include proper names, product names, technical terms, and domain-specific jargon
- Include terms up to 50 characters each
- Don't include common English words, single letters, or generic phrases
- Don't exceed 100 terms total

For detailed guidance, see [Keyterms prompting](/streaming/prompting-and-keyterms).

## How Do I Update Configuration Mid-Session?

You can update `prompt`, `keyterms_prompt`, `min_turn_silence`, and `max_turn_silence` during an active session using `UpdateConfiguration`. This is one of Universal-3.5 Pro Streaming's most powerful features for voice agents.

### Dynamic keyterms by conversation stage

As your voice agent moves through different stages, update keyterms to match what the user is likely to say:

```python
# Caller identification stage
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["Kelly Byrne-Donoghue", "date of birth", "January", "February"]
}))

# Medical intake stage
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["cardiology", "echocardiogram", "Dr. Patel", "metoprolol"]
}))

# Payment stage - also increase max_turn_silence for credit card dictation
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "keyterms_prompt": ["Visa", "Mastercard", "American Express"],
    "max_turn_silence": 3000
}))
```

### Dynamic prompting

You can also update the transcription prompt mid-session. This is especially powerful when paired with tool calls in your LLM:

- If your agent asks a yes/no question, prompt the model to anticipate short responses
- If your agent asks for a phone number or email, prompt it to expect those formats
- If you present a list of options, boost those options in the prompt

```python
# After asking "Would you like to confirm your appointment?"
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "prompt": "User is responding yes or no to a confirmation question. Expect short responses."
}))

# After asking "What's your phone number?"
ws.send(json.dumps({
    "type": "UpdateConfiguration",
    "prompt": "User is dictating a phone number. Expect digits and formatting.",
    "max_turn_silence": 3000
}))
```

## How Do I Use Speaker Diarization?

Streaming Diarization identifies and labels individual speakers in real time. Each `Turn` event includes a `speaker_label` field (e.g., `"A"`, `"B"`) indicating which speaker produced that transcript.

Enable it by adding `speaker_labels: true` to your connection parameters:

```python
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "speaker_labels": True,
}
```

Speaker accuracy improves over the course of a session as the model accumulates embedding context.

**With LiveKit:**

```python
stt = assemblyai.STT(
    model="universal-3-5-pro",
    speaker_labels=True,
)
```

**With Pipecat** (including custom formatting):

```python
stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    connection_params=AssemblyAIConnectionParams(
        speech_model="universal-3-5-pro",
        speaker_labels=True,
    ),
    speaker_format="[{speaker}] {text}",
)
```

For more details, see [Streaming Diarization and Multichannel](/streaming/label-speakers-and-separate-channels).

## How Do I Optimize for Latency?

### Key optimizations

**1. Use the right silence thresholds**

Start with `min_turn_silence=100` and `max_turn_silence=1000`. Only increase if you're seeing entity splitting issues.

**2. Tune `interruption_delay` for faster TTFT**

The `interruption_delay` parameter controls how soon the first partial is emitted. Set `interruption_delay=0` for the fastest possible time to first token (~300ms effective). The default of 500ms produces a first partial at ~800ms. See [Tuning early partial timing](/streaming/getting-started/transcribe-streaming-audio) for details.

**3. Eliminate additive delays in your orchestrator**

In LiveKit with `turn_detection="stt"`, set `min_endpointing_delay=0`. LiveKit's default 0.5s delay is additive on top of AssemblyAI's own endpointing.

**4. Use 16kHz sample rate**

This balances audio quality and bandwidth. Higher sample rates don't improve accuracy.

**5. Align VAD thresholds**

Mismatched VAD thresholds between your local VAD and AssemblyAI create a dead zone that delays interruption. Set both to `0.3`.

**6. Skip unnecessary features**

Only enable `speaker_labels` if you need diarization. Only use `keyterms_prompt` if you have domain-specific terms. Each feature adds marginal processing overhead.

### Latency breakdown

| Stage | Typical latency | Notes |
| ----- | --------------- | ----- |
| Audio to AssemblyAI | ~50ms | Network dependent |
| Speech-to-text | ~200-300ms | Sub-300ms P50 |
| `min_turn_silence` check | 100ms+ | Configurable |
| `max_turn_silence` fallback | 1000ms+ | Only if no terminal punctuation |

## How Does the Message Sequence Work?

Universal-3.5 Pro Streaming sends messages in a specific sequence. Here's what a typical conversation looks like:

**1. Session begins**

```json
{
  "type": "Begin",
  "id": "session-id",
  "expires_at": 1759796682
}
```

**2. Speech detected**

```json
{
  "type": "SpeechStarted",
  "timestamp": 1200,
  "confidence": 0.85
}
```

**3. Early partial** (emitted after 750ms of continuous speech)

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": false,
  "turn_is_formatted": false,
  "transcript": "Yeah my credit card..."
}
```

**4. Silence-based partial** (speaker pauses, no terminal punctuation)

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": false,
  "turn_is_formatted": false,
  "transcript": "Yeah my credit card number is--"
}
```

**5. Final transcript** (terminal punctuation found, or `max_turn_silence` reached)

```json
{
  "type": "Turn",
  "turn_order": 0,
  "end_of_turn": true,
  "turn_is_formatted": true,
  "transcript": "Yeah, my credit card number is 8888-8888-8888-8888.",
  "speaker_label": "A"
}
```

For Universal-3.5 Pro Streaming, `end_of_turn` and `turn_is_formatted` always have the same value. You can reliably use `end_of_turn: true` to detect a formatted, final transcript.

**6. Session termination**

```json
{
  "type": "Termination",
  "audio_duration_seconds": 45.2
}
```

For the complete message reference, see [Message sequence](/streaming/getting-started/transcribe-streaming-audio).

## How Can I Improve Accuracy?

### Keyterms prompting

The single most effective way to improve accuracy on domain-specific terms. Keyterms are especially useful for improving recognition of proper nouns, product names, and technical jargon spoken with accents or in noisy environments. See [How Can I Use Prompting to Improve Accuracy?](#how-can-i-use-prompting-to-improve-accuracy) above.

### Dynamic configuration updates

Update keyterms and prompts mid-session based on conversation context. See [How Do I Update Configuration Mid-Session?](#how-do-i-update-configuration-mid-session) above.

### Tune silence thresholds

If entities are splitting across turns, increase `min_turn_silence` (for punctuation-triggered splits) or `max_turn_silence` (for forced timeout splits). You can do this dynamically mid-session for specific conversation stages like entity dictation.

### Noise handling

Universal-3.5 Pro Streaming handles background noise well out of the box. Avoid adding noise cancellation as a preprocessing step. The artifacts it introduces typically cause more harm than the background noise itself.

For telephony environments with low-quality audio (such as 8 kHz mulaw), you can prompt the model to tag genuinely unclear segments as `[unclear]` rather than forcing a guess. This helps you identify audio segments that no model (or human) can reliably transcribe, and prevents inaccurate guesses from entering your downstream pipeline.

## Scaling and Rate Limits

Universal-3.5 Pro Streaming provides unlimited parallel streams:

- No hard caps on simultaneous connections
- No overage fees for spike traffic
- Automatic scaling from 5 to 50,000+ streams

**Rate limits:**

- Free users: 5 new streams per minute
- Pay-as-you-go: 100 new streams per minute
- When using 70%+ of your limit, capacity automatically increases 10% every 60 seconds

These limits are designed to never interfere with legitimate applications. Your baseline limit is guaranteed and never decreases, so you can scale smoothly without artificial barriers.

---

## Evaluating your voice agent

Benchmark scores are a useful starting point, but they don't tell the full story. To determine which STT model works best for your voice agent in production:

- **Run A/B tests at scale.** Swap models in your agent pipeline, compare core outcome metrics (task completion rate, booking rate, resolution rate), and let real user behavior determine the winner.
- **Optimize for outcomes, not benchmarks.** The question is not which model has the best WER. It is which model drives users to complete their goal most reliably.
- **Simulate real scenarios early.** Before you have real users, manually test the full agent flow under realistic conditions. This surfaces issues that isolated STT benchmarks will miss.

For a complete evaluation framework including accuracy metrics, latency metrics, and ground truth best practices, see the [streaming evaluation guide](/streaming/evaluations/voice-agents).

## Additional Resources

- [Streaming evaluation guide](/streaming/evaluations/voice-agents)
- [Universal-3.5 Pro Streaming overview](/streaming/getting-started/transcribe-streaming-audio)
- [Turn detection guide](/streaming/getting-started/transcribe-streaming-audio)
- [Prompting guide](/streaming/prompting-and-keyterms)
- [Keyterms prompting](/streaming/prompting-and-keyterms)
- [Message sequence](/streaming/getting-started/transcribe-streaming-audio)
- [Streaming diarization and multichannel](/streaming/label-speakers-and-separate-channels)
- [Supported languages](/streaming/getting-started/transcribe-streaming-audio)
- [LiveKit integration](/voice-agents/livekit-universal-3-5-pro)
- [Pipecat integration](/voice-agents/pipecat-universal-3-5-pro)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)

---

<a id="best-practices-for-building-contact-center-applications-docs-contact-center-best-practices-mdx"></a>

# Best Practices for building Contact Center Applications

URL: https://www.assemblyai.com/docs/contact-center-best-practices
Source: docs/contact-center-best-practices.mdx
Navigation: Real-time STT > Guides > Tutorials
Description: Complete guide for building contact center applications with AssemblyAI

## Introduction

Building a contact center application requires careful consideration of accuracy, speaker separation, compliance, and scalability. This guide addresses common questions and provides practical solutions for both post-call analytics and real-time agent assist scenarios.

## Why AssemblyAI for contact centers?

AssemblyAI stands out as the premier choice for contact center applications with several key advantages:

### Industry-leading accuracy on telephony audio

- **Universal-3.5 Pro model** delivers best-in-class accuracy on 8kHz telephony audio
- **2.9% speaker diarization error rate** for precise agent vs. customer attribution
- **Multichannel support** for stereo call recordings where agent and customer are on separate channels
- **Keyterms prompt** allows providing call context to improve accuracy of company names, products, and compliance phrases

### Streaming with Universal-3.5 Pro

For real-time agent assist, AssemblyAI's Universal-3.5 Pro Streaming model (`universal-3-5-pro`) offers:

- **Low latency** enables live transcription during calls
- **Format turns** feature provides structured, readable output
- **Dynamic prompting** via `UpdateConfiguration` to update context mid-call
- **Dual-channel streaming** for separate agent and customer audio streams

### End-to-end voice AI platform

Unlike fragmented solutions, AssemblyAI provides a unified API for:

- Transcription with speaker diarization (agent vs. customer)
- Multichannel audio support for stereo call recordings
- PII redaction on both text and audio for HIPAA and PCI compliance
- Post-processing workflows with custom prompting - from call summaries to QA scoring
- Streaming and pre-recorded transcription in a single platform
- Compliance and security built for enterprise workloads (BAA, SOC2, ISO)

## When should I use pre-recorded vs streaming for contact centers?

Understanding when to use pre-recorded versus streaming is critical for contact center workflows.

### Pre-recorded Speech-to-text

**Post-call analytics** - Call already happened, you have the full recording

- **Highest accuracy needed** - Pre-recorded models have the highest accuracy
- **Speaker diarization is critical** - Pre-recorded has 2.9% speaker error rate
- **Multichannel recordings** - Most contact center recordings are stereo with agent and customer on separate channels
- **Compliance workflows** - Full PII redaction with audio de-identification
- **Post-call analytics** - Summarization, sentiment analysis, entity detection, QA scoring
- **Batch processing** - Processing large volumes of call recordings

**Best for:** QA scoring, compliance monitoring, coaching insights, post-call CRM updates, searchable call archives

### Streaming Speech-to-text

**Live calls** - Transcribing as the call happens

You should use streaming when you need to display a live transcript to agents during calls. With Universal-3.5 Pro Streaming, accuracy is closer to pre-recorded, but pre-recorded will always be the most accurate option.

- **Agent assist** - Live transcription visible to agents during calls
- **Real-time coaching** - Prompt agents with suggested responses or compliance reminders
- **Live compliance monitoring** - Detect compliance violations in real-time
- **No recording available** - Processing live audio only

**Best for:** Agent assist, real-time coaching, live compliance monitoring, live call transcription

### Hybrid approach (recommended)

Many contact center platforms use **both**:

1. **Streaming during the call** - Provide live transcription for agent assist and real-time coaching
2. **Pre-recorded after the call** - Generate high-quality transcript with speaker labels, summary, and analytics

**Example workflow:**

- Call begins → Start streaming for live agent assist
- Call ends → Upload recording to pre-recorded API for final transcript with speaker names
- Generate call summary, QA score, and compliance report from pre-recorded transcript
- Push results to CRM (e.g., Salesforce)

## What languages and features for a contact center application?

### Pre-recorded calls (Universal-3.5 Pro)

For post-call analytics, AssemblyAI supports:

**Languages**:

- 99 languages supported
- Automatic Language Detection to route to the most spoken language
- Code Switching to preserve changes in speech between languages

**Core Features**:

- Speaker diarization (agent-customer separation)
- Multichannel audio support - when agent and customer are on separate audio channels, enables perfect speaker separation without diarization
- Automatic formatting, punctuation, and capitalization
- Keyterms prompting for boosting domain-specific terms (up to 1000 terms for Universal-3.5 Pro)
- Natural language prompting (Universal-3.5 Pro) - up to 1,500 words to guide transcription behavior
- Speaker options with configurable min/max expected speakers for call transfers

**Speech Understanding**:

- Summarization for call recaps
- Sentiment analysis for customer satisfaction tracking
- Entity detection for extracting names, account numbers, and products
- Speaker identification to map generic labels to agent and customer names
- Translation between 86 languages

**Guardrails**:

- PII redaction on text and audio for HIPAA and PCI compliance

### Streaming (Universal-3.5 Pro Streaming)

For live call transcription, use **Universal-3.5 Pro Streaming** (`universal-3-5-pro`) for the highest streaming accuracy:

**Core Features**:

- Speaker diarization for identifying agent vs. customer
- Partial and final transcripts for responsive UI
- Format turns for structured, readable output
- Keyterms prompt for company names, products, and compliance phrases
- Dual-channel streaming for separate agent and customer audio

For more details, see the [Universal-3.5 Pro Streaming documentation](/streaming/getting-started/transcribe-streaming-audio).

## How can I get started building a post-call analytics pipeline?

Here's a complete example implementing pre-recorded transcription for contact center call analysis:

```python expandable
import assemblyai as aai
import asyncio
from typing import Dict, List
from assemblyai.types import (
    SpeakerOptions,
    PIIRedactionPolicy,
    PIISubstitutionPolicy,
)

# Configure API key
aai.settings.api_key = "your_api_key_here"

async def transcribe_call(audio_source: str, agent_name: str = None) -> Dict:
    """
    Transcribe a contact center call recording with full analytics

    Args:
        audio_source: Either a local file path or publicly accessible URL
        agent_name: Optional agent name for speaker identification
    """
    # Configure comprehensive call analysis
    config = aai.TranscriptionConfig(
        # Model selection
        speech_models=["universal-3-5-pro", "universal-2"],

        # Speaker diarization
        speaker_labels=True,
        speaker_options=SpeakerOptions(
            min_speakers_expected=2,  # Agent and customer
            max_speakers_expected=5   # Allow for call transfers - safe to keep high
        ),
        multichannel=False,  # Set to True if audio has separate channel per speaker

        # Language detection
        language_detection=True,

        # Boost accuracy of contact center vocabulary
        keyterms_prompt=[
            # Company-specific terms
            "Acme Corp", "Premium Support Plan",

            # Compliance phrases
            "recorded line", "calls are monitored and recorded",

            # Common contact center terms
            "account number", "case number", "ticket number",
            "escalation", "supervisor", "hold time",
        ],

        # Post-call analytics
        summarization=True,
        sentiment_analysis=True,
        entity_detection=True,

        # PII protection for compliance
        redact_pii=True,
        redact_pii_policies=[
            PIIRedactionPolicy.person_name,
            PIIRedactionPolicy.phone_number,
            PIIRedactionPolicy.email_address,
            PIIRedactionPolicy.account_number,
            PIIRedactionPolicy.us_social_security_number,
            PIIRedactionPolicy.credit_card_number,
            PIIRedactionPolicy.credit_card_cvv,
            PIIRedactionPolicy.credit_card_expiration,
            PIIRedactionPolicy.date_of_birth,
        ],
        redact_pii_sub=PIISubstitutionPolicy.hash,
        redact_pii_audio=True,
    )

    # Add speaker identification if agent name is known
    if agent_name:
        config.speech_understanding = {
            "request": {
                "speaker_identification": {
                    "speaker_type": "role",
                    "speakers": [
                        {"role": "Agent", "name": agent_name},
                        {"role": "Customer"}
                    ]
                }
            }
        }

    # Create transcriber
    transcriber = aai.Transcriber()

    try:
        # Submit transcription job
        transcript = await asyncio.to_thread(
            transcriber.transcribe,
            audio_source,
            config=config
        )

        # Check status
        if transcript.status == aai.TranscriptStatus.error:
            raise Exception(f"Transcription failed: {transcript.error}")

        # Process speaker-labeled utterances
        for utterance in transcript.utterances:
            start_time = utterance.start / 1000  # Convert ms to seconds
            end_time = utterance.end / 1000

            print(f"[{start_time:.1f}s - {end_time:.1f}s] {utterance.speaker}:")
            print(f"  {utterance.text}\n")

        return {
            "transcript": transcript,
            "utterances": transcript.utterances,
            "summary": transcript.summary,
            "sentiment": transcript.sentiment_analysis_results,
            "entities": transcript.entities,
            "redacted_audio_url": transcript.redacted_audio_url,
        }

    except Exception as e:
        print(f"Error during transcription: {e}")
        raise

async def main():
    audio_source = "https://your-storage.com/calls/call_recording.mp3"

    result = await transcribe_call(audio_source, agent_name="Sarah Johnson")

    print(f"\nCall duration: {result['transcript'].audio_duration} seconds")
    print(f"Summary: {result['summary']}")

if __name__ == "__main__":
    asyncio.run(main())
```

## How Do I Handle Multichannel Contact Center Audio?

Most contact center recordings are stereo with the agent on one channel and the customer on the other. Multichannel transcription gives you perfect speaker separation without diarization.

### Pre-recorded Multichannel

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    multichannel=True,  # Enable when agent and customer are on separate channels
    speaker_labels=False,  # Disable - channels already separate speakers

    # Still enable analytics
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,

    # PII redaction
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file, config=config)

# Channel 1 = Agent, Channel 2 = Customer (typical layout)
for utterance in transcript.utterances:
    role = "Agent" if utterance.channel == "1" else "Customer"
    print(f"{role}: {utterance.text}")
```

**When to use multichannel:**

- Call recordings from PBX systems with separate agent/customer channels
- Recordings from platforms like Genesys, Twilio, Five9, NICE, or Talkdesk
- Any stereo recording where each channel represents a different speaker

**Benefits:**

- **Perfect speaker separation** - No diarization errors
- **No speaker confusion or overlap issues**
- **Higher accuracy** - Model processes clean single-speaker audio per channel

### Streaming Multichannel

For real-time dual-channel transcription, create separate streaming sessions per channel:

```python expandable
import asyncio
import websockets
import json
from urllib.parse import urlencode

API_KEY = "your_api_key"

class ChannelTranscriber:
    def __init__(self, channel_id: int, role: str):
        self.channel_id = channel_id
        self.role = role
        self.connection_params = {
            "sample_rate": 8000,  # Telephony standard
            "speech_model": "universal-3-5-pro",
            "format_turns": True,
            "encoding": "pcm_mulaw",  # Common telephony encoding
        }

    async def transcribe_channel(self, audio_stream):
        url = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(self.connection_params, doseq=True)}"

        # If using websockets >= 13.0, use additional_headers. For < 13.0, use extra_headers.
        async with websockets.connect(url, additional_headers={"Authorization": API_KEY}) as ws:
            # Send and receive must run concurrently for real-time streaming
            async def send_audio():
                async for audio_chunk in audio_stream:
                    await ws.send(audio_chunk)

            async def receive_transcripts():
                async for message in ws:
                    data = json.loads(message)
                    if data.get("type") == "Turn" and data.get("end_of_turn"):
                        print(f"{self.role}: {data['transcript']}")

            await asyncio.gather(send_audio(), receive_transcripts())

# Create transcriber for each channel
async def transcribe_live_call(agent_audio_stream, customer_audio_stream):
    agent = ChannelTranscriber(0, "Agent")
    customer = ChannelTranscriber(1, "Customer")

    await asyncio.gather(
        agent.transcribe_channel(agent_audio_stream),
        customer.transcribe_channel(customer_audio_stream),
    )
```

See our [multichannel streaming guide](/streaming/label-speakers-and-separate-channels#multichannel-streaming-audio) for complete implementation details.

## How Can I Build a Real-Time Agent Assist?

Here's a complete example for real-time streaming transcription optimized for contact center agent assist:

```python expandable
# pip install pyaudio websocket-client
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "your_api_key"

# Contact center keyterms
KEYTERMS = [
    # Company and product terms
    "Acme Corp",
    "Premium Support Plan",
    "Enterprise License",

    # Compliance phrases
    "recorded line",
    "calls are monitored",

    # Common contact center vocabulary
    "account number",
    "case number",
    "escalation",
    "supervisor",
]

# CONTACT CENTER CONFIGURATION
CONNECTION_PARAMS = {
    "sample_rate": 8000,  # Telephony standard (8kHz)
    "speech_model": "universal-3-5-pro",  # Universal-3.5 Pro Streaming for highest accuracy
    "format_turns": True,

    # Contact center turn detection
    # universal-3-5-pro defaults: min_turn_silence=100ms, max_turn_silence=1000ms
    "min_turn_silence": 400,  # Longer than default for natural call pauses
    "max_turn_silence": 1500,  # Longer for customers explaining issues

    # Keyterms for accuracy
    "keyterms_prompt": KEYTERMS,
}

API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS, doseq=True)}"

# Audio Configuration
FRAMES_PER_BUFFER = 400  # 50ms of audio at 8kHz
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
transcript_buffer = []

def on_open(ws):
    print("=" * 80)
    print(f"[{datetime.now().strftime('%H:%M:%S')}] Agent assist transcription started")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")
    print(f"Keyterms configured: {', '.join(KEYTERMS[:5])}...")
    print("=" * 80)

    def stream_audio():
        global stream
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                if not stop_event.is_set():
                    print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        if msg_type == "Begin":
            session_id = data.get("id", "N/A")
            print(f"[SESSION] Started - ID: {session_id}\n")

        elif msg_type == "Turn":
            end_of_turn = data.get("end_of_turn", False)
            transcript = data.get("transcript", "")
            turn_order = data.get("turn_order", 0)

            # Show partials for responsive agent UI
            if not end_of_turn and transcript:
                print(f"\r[LIVE] {transcript}", end="", flush=True)

            # Use formatted finals for agent display
            if end_of_turn and transcript:
                timestamp = datetime.now().strftime('%H:%M:%S')
                print(f"\n[{timestamp}] {transcript}")

                # Detect compliance keywords
                transcript_lower = transcript.lower()
                if any(term in transcript_lower for term in ["cancel", "refund", "complaint", "supervisor"]):
                    print("           ** ESCALATION KEYWORD DETECTED **")

                transcript_buffer.append({
                    "timestamp": timestamp,
                    "text": transcript,
                    "turn_order": turn_order,
                    "type": "final"
                })
                print()

        elif msg_type == "Termination":
            audio_duration = data.get("audio_duration_seconds", 0)
            print(f"\n[SESSION] Terminated - Duration: {audio_duration}s")

        elif msg_type == "Error":
            error_msg = data.get("error", "Unknown error")
            print(f"\n[ERROR] {error_msg}")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    print(f"\n[WEBSOCKET ERROR] {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    print(f"\n[WEBSOCKET] Disconnected - Status: {close_status_code}, Message: {close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def run():
    global audio, stream, ws_app

    audio = pyaudio.PyAudio()

    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
    except Exception as e:
        print(f"Error opening audio stream: {e}")
        if audio:
            audio.terminate()
        return

    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\n\nCtrl+C received. Stopping transcription...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## How Should I Handle Pre-recorded Transcription in Production?

### Webhook Callbacks (Recommended)

For high-volume contact center workloads, use webhooks instead of polling:

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    webhook_url="https://your-app.com/webhooks/assemblyai",
    webhook_auth_header_name="X-Webhook-Secret",
    webhook_auth_header_value="your_secret_here",
    speaker_labels=True,
    multichannel=True,
    summarization=True,
    sentiment_analysis=True,
    entity_detection=True,
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
)

# Submit job and return immediately (non-blocking)
transcript = transcriber.submit(audio_url, config=config)
print(f"Job submitted: {transcript.id}")
# Your app continues processing other calls
```

**Webhook handler example:**

```python expandable
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhooks/assemblyai", methods=["POST"])
def assemblyai_webhook():
    if request.headers.get("X-Webhook-Secret") != "your_secret_here":
        return jsonify({"error": "Unauthorized"}), 401

    import requests as http_requests

    data = request.json
    transcript_id = data["transcript_id"]
    status = data["status"]

    if status == "completed":
        # Fetch the full transcript (webhook only sends transcript_id and status)
        transcript = http_requests.get(
            f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
            headers={"authorization": "your_api_key"}
        ).json()
        process_completed_call(transcript)
    elif status == "error":
        log_transcription_error(transcript_id)

    return jsonify({"received": True}), 200

def process_completed_call(transcript):
    """Process completed call transcript and push to CRM"""
    utterances = transcript["utterances"]
    summary = transcript["summary"]

    # Store in database
    save_to_database(transcript)

    # Push summary to CRM
    push_to_crm(transcript["id"], summary)

    # Run QA scoring
    qa_score = score_call_quality(utterances)
    save_qa_score(transcript["id"], qa_score)
```

### Scaling Considerations

- **HTTP rate limit:** 20,000 requests per 5-minute window, counted across submissions (POST) and polling (GET) combined
- **Exceeding the limit:** returns a `403` response
- **Parallel transcriptions (rate limit):** 200+ for paid accounts (queued beyond that)
- **Ramp up gradually:** start at 10-50 parallel requests, double incrementally
- **Avoid the rate limit:** use [webhooks](/pre-recorded-audio/webhooks) or jittered, widened polling — see [Polling without exceeding the rate limit](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale#polling-without-exceeding-the-rate-limit)
- **Contact Sales** before large-scale rollouts

## How Do I Handle PII and Compliance?

PII redaction is critical for contact center compliance (HIPAA, PCI-DSS, GDPR, CCPA).

### Recommended PII Configuration

```python expandable
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        # Customer identity
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.date_of_birth,
        PIIRedactionPolicy.us_social_security_number,

        # Contact information
        PIIRedactionPolicy.phone_number,
        PIIRedactionPolicy.email_address,
        PIIRedactionPolicy.location,

        # Financial information (PCI-DSS)
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.credit_card_cvv,
        PIIRedactionPolicy.credit_card_expiration,
        PIIRedactionPolicy.account_number,
        PIIRedactionPolicy.banking_information,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,  # Stable hash tokens
    redact_pii_audio=True,  # Create de-identified audio file
)
```

**Why hash substitution?**

- Stable across the file (same value = same token)
- Maintains sentence structure for downstream LLM processing
- Prevents reconstruction of original data

### HIPAA Compliance

- AssemblyAI provides a **Business Associate Agreement (BAA)** at no cost
- Paid customers can review and sign our standard online BAA self-serve from the [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the dashboard (Owner or Admin role required, upgraded account required)
- Use PII redaction with audio de-identification for full compliance

## How Do I Improve the Accuracy of My Contact Center Transcription?

### Prompting Best Practices

The most impactful lever for contact center accuracy is **prompting**. Use a structured prompt with a `Context:` field:

```python expandable
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],

    # Natural language prompt for transcription guidance
    prompt=(
        "Transcribe this audio with perfect punctuation and formatting. "
        "Preserve linguistic speech patterns including disfluencies, filler words, "
        "hesitations, repetitions, stutters, false starts, and colloquialisms. "
        "Transcribe in the original language mix (code-switching), preserving the "
        "words in the language they are spoken. Output plain transcript text only. "
        "Use a new line when the voice changes; each line contains only one "
        "person's words.\n\n"
        "Context: Acme Corp customer service call, recorded line, "
        "Agent: Sarah Johnson, calls are monitored and recorded"
    ),

    # Keyterms for proper nouns and domain vocabulary
    keyterms_prompt=[
        "Acme Corp",
        "Sarah Johnson",
        "Premium Support Plan",
        "Enterprise License",
        "recorded line",
        "calls are monitored and recorded",
    ],

    speaker_labels=True,
)
```

**Tips for effective prompting:**

- Use **positive instructions** ("transcribe verbatim") not negative ("do NOT summarize")
- **Start with fewer instructions, add one at a time** — every added instruction risks conflicting with another. Treat the older "3–6 instructions" guidance as an upper bound, not a target.
- **Layer instructions one by one** and test each against your call recordings to measure impact
- Dynamize the `Context:` line per call with known info: company name, agent name, compliance phrases
- Use **keyterms** for proper nouns and domain vocabulary (company names, product names, agent names)

### Using Keyterms for Pre-recorded Transcription

```python expandable
# Build keyterms dynamically per call
call_keyterms = [
    # Company terms (static)
    "Acme Corp",
    "Premium Support Plan",

    # Agent name (from routing system)
    agent_name,

    # Customer name (from CRM lookup)
    customer_name,

    # Account-specific terms
    "account ending in 4532",
]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    keyterms_prompt=call_keyterms,
    speaker_labels=True,
)
```

### Using Keyterms for Streaming

```python
# Streaming with contact center context
keyterms = [
    "Acme Corp",
    "Premium Support Plan",
    "Sarah Johnson",
    "recorded line",
]

CONNECTION_PARAMS = {
    "sample_rate": 8000,
    "speech_model": "universal-3-5-pro",
    "format_turns": True,
    "encoding": "pcm_mulaw",
    "keyterms_prompt": keyterms,
}
```

## What Workflows Can I Build for My Contact Center Application?

Use these features to transform raw call transcripts into actionable insights.

### Summarization

`summarization: true`

**What it does:** Generates an abstractive recap of the call.
**Output:** `summary` string (bullets/paragraph format).
**Great for:** Post-call CRM updates, call recaps, supervisor review.

```python
config = aai.TranscriptionConfig(
    summarization=True,
    summary_type="bullets",  # or "bullets_verbose", "gist", "headline", "paragraph"
    summary_model="informative",  # or "conversational"
)
```

### Sentiment Analysis

`sentiment_analysis: true`

**What it does:** Scores per-utterance sentiment (positive / neutral / negative).
**Output:** Array of `{ text, sentiment, confidence, start, end }`.
**Great for:** Customer satisfaction tracking, escalation detection, QA scoring.

```python
# Analyze customer sentiment across a call
negative_count = 0
for result in transcript.sentiment_analysis_results:
    if result.sentiment == "NEGATIVE":
        negative_count += 1
        print(f"Negative at {result.start / 1000:.1f}s: {result.text}")

# Flag calls with high negative sentiment
if negative_count > 3:
    flag_for_supervisor_review(transcript.id)
```

### Entity Detection

`entity_detection: true`

**What it does:** Extracts named entities (people, organizations, locations, products, etc.).
**Output:** Array of `{ entity_type, text, start, end }`.
**Great for:** CRM enrichment, auto-tagging topics, competitor tracking.

```python
# Extract key entities from a call
organizations = [e.text for e in transcript.entities if e.entity_type == "organization"]
print(f"Companies mentioned: {', '.join(organizations)}")
```

### Speaker Identification

Map generic speaker labels to agent and customer names:

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    speaker_labels=True,
    speech_understanding={
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "speakers": [
                    {"role": "Agent", "name": "Sarah Johnson", "description": "Customer service representative"},
                    {"role": "Customer"}
                ]
            }
        }
    }
)
```

### Translation

Translate call transcripts for international teams:

```python
config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
    speaker_labels=True,
    speech_understanding={
        "request": {
            "translation": {
                "target_languages": ["en"],  # Translate to English
                "match_original_utterance": True,  # Per-utterance translations
                "formal": True
            }
        }
    }
)
```

### Redact PII Text and Audio

```python
config = aai.TranscriptionConfig(
    redact_pii=True,
    redact_pii_policies=[
        PIIRedactionPolicy.person_name,
        PIIRedactionPolicy.credit_card_number,
        PIIRedactionPolicy.us_social_security_number,
        PIIRedactionPolicy.account_number,
    ],
    redact_pii_sub=PIISubstitutionPolicy.hash,
    redact_pii_audio=True,  # Generate de-identified audio
)

# After transcription
print(transcript.text)  # PII redacted in text
print(transcript.redacted_audio_url)  # PII bleeped in audio
```

## How Do I Process the Response from the API?

### Processing Pre-recorded Responses

```python expandable
def process_call_transcript(transcript):
    """
    Extract and process all relevant data from a pre-recorded call transcript
    """
    call_data = {
        "id": transcript.id,
        "duration": transcript.audio_duration,  # Already in seconds
        "confidence": transcript.confidence,
        "full_text": transcript.text,
    }

    # Process speaker utterances
    speakers = {}
    for utterance in transcript.utterances:
        speaker = utterance.speaker

        if speaker not in speakers:
            speakers[speaker] = {
                "utterances": [],
                "total_speaking_time": 0,
                "word_count": 0
            }

        speakers[speaker]["utterances"].append({
            "text": utterance.text,
            "start": utterance.start,
            "end": utterance.end,
        })

        speakers[speaker]["total_speaking_time"] += (utterance.end - utterance.start) / 1000
        speakers[speaker]["word_count"] += len(utterance.text.split())

    call_data["speakers"] = speakers

    # Extract summary
    if transcript.summary:
        call_data["summary"] = transcript.summary

    # Analyze sentiment
    if transcript.sentiment_analysis_results:
        sentiments = [r.sentiment for r in transcript.sentiment_analysis_results]
        call_data["sentiment_breakdown"] = {
            "positive": sentiments.count("POSITIVE"),
            "neutral": sentiments.count("NEUTRAL"),
            "negative": sentiments.count("NEGATIVE"),
        }

    # Calculate statistics
    total_duration = transcript.audio_duration
    call_data["statistics"] = {
        "total_speakers": len(speakers),
        "total_words": sum(s["word_count"] for s in speakers.values()),
        "speaking_distribution": {
            speaker: {
                "percentage": (data["total_speaking_time"] / total_duration) * 100,
                "minutes": data["total_speaking_time"] / 60,
            }
            for speaker, data in speakers.items()
        },
    }

    return call_data

result = process_call_transcript(transcript)
print(f"Call had {result['statistics']['total_speakers']} speakers")
print(f"Sentiment: {result.get('sentiment_breakdown', {})}")
```

## Additional Resources

- [Universal-3.5 Pro Pre-recorded Documentation](/pre-recorded-audio)
- [Universal-3.5 Pro Streaming Documentation](/streaming/getting-started/transcribe-streaming-audio)
- [Speaker Diarization Guide](/pre-recorded-audio/label-speakers)
- [Multichannel Streaming Guide](/streaming/label-speakers-and-separate-channels)
- [Speaker Identification Guide](/speech-understanding/speaker-identification)
- [Translation Guide](/speech-understanding/translation)
- [PII Redaction Guide](/guardrails/redact-pii-from-transcripts)
- [Getting Started Guide](/pre-recorded-audio/getting-started/transcribe-an-audio-file)
- [API Playground](https://www.assemblyai.com/playground/streaming)
- [Changelog](https://www.assemblyai.com/changelog)
- [Support](https://www.assemblyai.com/contact/support)

---

<a id="stream-a-pre-recorded-file-in-real-time-docs-streaming-guides-stream-prerecorded-file-realtime-mdx"></a>

# Stream a pre-recorded file in real time

URL: https://www.assemblyai.com/docs/streaming/guides/stream_prerecorded_file_realtime
Source: docs/streaming/guides/stream_prerecorded_file_realtime.mdx
Navigation: Real-time STT > Guides > Tutorials > Basic streaming workflows
Description: Stream a pre-recorded file in real time documentation.

When you stream a pre-recorded audio file to the Streaming API, you need to send audio at the same pace it was recorded. If you send audio faster than real time, the server receives more data than it can process in sequence, which can cause degraded transcription accuracy, unexpected session closures, or other errors.

This guide shows you how to pace audio correctly so that the server processes it as if a person were speaking into a live microphone.

## Why real-time pacing matters

The Streaming API is designed for live audio. It expects audio to arrive at roughly the same rate it was originally spoken. When you stream a pre-recorded file without any pacing, your code reads and sends the entire file in seconds, even if the recording is minutes long. This causes problems:

- **Unexpected session behavior** &mdash; Sending audio faster than real time can overwhelm the connection and cause the server to close the session or return errors.
- **Inaccurate results** &mdash; The speech model is optimized for real-time input. Audio that arrives too quickly may not be processed the same way as live speech, potentially affecting transcription quality.
- **Unreliable benchmarks** &mdash; If you're evaluating transcription quality, faster-than-real-time streaming produces results that don't reflect production conditions where audio arrives at normal speed.

If you only need a transcript and don't need real-time results, use the [pre-recorded transcription API](/pre-recorded-audio/getting-started/transcribe-an-audio-file) instead. It processes audio as fast as possible and is optimized for batch workloads.

## Before you begin

To complete this guide, you need:

- An AssemblyAI API key. [Sign up](https://assemblyai.com/dashboard/signup) and get your key from the [dashboard](https://www.assemblyai.com/dashboard/home).
- [Python 3.8+](https://www.python.org/) or [Node.js 18+](https://nodejs.org/).
- A WAV audio file (mono, 16-bit PCM). If your file is in a different format, see [Prepare your audio file](#prepare-your-audio-file).

## Quickstart

```python expandable
import websocket
import json
import threading
import time
import wave
import os
from urllib.parse import urlencode

# --- Configuration ---
ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AUDIO_FILE = "audio.wav"
CHUNK_DURATION = 0.1  # Send 100ms of audio per chunk
SAMPLE_RATE = 16000   # Must match your audio file's sample rate

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": SAMPLE_RATE,
}
API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"

ws_app = None
audio_thread = None
stop_event = threading.Event()

def on_open(ws):
    print("Connected. Streaming audio at real-time speed...")

    def stream_file():
        with wave.open(AUDIO_FILE, "rb") as wf:
            frames_per_chunk = int(wf.getframerate() * CHUNK_DURATION)
            start_time = time.monotonic()
            chunks_sent = 0

            while not stop_event.is_set():
                frames = wf.readframes(frames_per_chunk)
                if not frames:
                    break

                ws.send(frames, websocket.ABNF.OPCODE_BINARY)
                chunks_sent += 1

                # Wall-clock pacing: sleep until the next chunk is due
                next_chunk_time = start_time + (chunks_sent * CHUNK_DURATION)
                sleep_duration = next_chunk_time - time.monotonic()
                if sleep_duration > 0:
                    time.sleep(sleep_duration)

        print("Finished sending audio. Waiting for final transcripts...")
        try:
            ws.send(json.dumps({"type": "Terminate"}))
        except Exception:
            pass

    global audio_thread
    audio_thread = threading.Thread(target=stream_file, daemon=True)
    audio_thread.start()

def on_message(ws, message):
    data = json.loads(message)

    if data["type"] == "Begin":
        print(f"Session ID: {data['id']}")
    elif data["type"] == "Turn":
        transcript = data.get("transcript", "")
        if not transcript:
            return
        if data.get("end_of_turn"):
            print(f"[Final]: {transcript}")
        else:
            print(f"[Partial]: {transcript}")
    elif data["type"] == "Termination":
        print(f"Done. Processed {data.get('audio_duration_seconds', 0)}s of audio.")

def on_error(ws, error):
    print(f"Error: {error}")
    stop_event.set()

def on_close(ws, status_code, msg):
    print(f"Disconnected (status={status_code})")
    stop_event.set()

ws_app = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": ASSEMBLYAI_API_KEY},
    on_open=on_open,
    on_message=on_message,
    on_error=on_error,
    on_close=on_close,
)

ws_thread = threading.Thread(target=ws_app.run_forever, daemon=True)
ws_thread.start()

try:
    while ws_thread.is_alive():
        time.sleep(0.1)
except KeyboardInterrupt:
    print("\nStopping...")
    stop_event.set()
    if ws_app and ws_app.sock and ws_app.sock.connected:
        try:
            ws_app.send(json.dumps({"type": "Terminate"}))
            time.sleep(2)
        except Exception:
            pass
    if ws_app:
        ws_app.close()
    ws_thread.join(timeout=2.0)
```

```javascript expandable
import WebSocket from "ws";
import { readFileSync } from "fs";
import { stringify } from "querystring";

// --- Configuration ---
const ASSEMBLYAI_API_KEY = process.env.ASSEMBLYAI_API_KEY;
const AUDIO_FILE = "audio.wav";
const CHUNK_DURATION = 0.1; // Send 100ms of audio per chunk
const SAMPLE_RATE = 16000;  // Must match your audio file's sample rate

const CONNECTION_PARAMS = {
  speech_model: "universal-3-5-pro",
  sample_rate: SAMPLE_RATE,
};
const API_ENDPOINT = `wss://streaming.assemblyai.com/v3/ws?${stringify(CONNECTION_PARAMS)}`;

function readWavAudioData(filePath) {
  const fileBuffer = readFileSync(filePath);

  // Verify RIFF/WAVE header
  if (fileBuffer.toString("ascii", 0, 4) !== "RIFF" ||
      fileBuffer.toString("ascii", 8, 12) !== "WAVE") {
    throw new Error("Not a valid WAV file");
  }

  // Find the "data" chunk
  let offset = 12;
  while (offset < fileBuffer.length - 8) {
    const chunkId = fileBuffer.toString("ascii", offset, offset + 4);
    const chunkSize = fileBuffer.readUInt32LE(offset + 4);
    if (chunkId === "data") {
      return fileBuffer.subarray(offset + 8, offset + 8 + chunkSize);
    }
    offset += 8 + chunkSize;
  }
  throw new Error("No data chunk found in WAV file");
}

function run() {
  const ws = new WebSocket(API_ENDPOINT, {
    headers: { Authorization: ASSEMBLYAI_API_KEY },
  });

  ws.on("open", () => {
    console.log("Connected. Streaming audio at real-time speed...");

    const audioData = readWavAudioData(AUDIO_FILE);

    const bytesPerSample = 2; // 16-bit PCM
    const bytesPerChunk = Math.floor(SAMPLE_RATE * CHUNK_DURATION) * bytesPerSample;
    let offset = 0;
    let chunksSent = 0;
    const startTime = Date.now();

    function sendNextChunk() {
      if (offset >= audioData.length) {
        console.log("Finished sending audio. Waiting for final transcripts...");
        ws.send(JSON.stringify({ type: "Terminate" }));
        return;
      }

      const chunk = audioData.subarray(offset, offset + bytesPerChunk);
      ws.send(chunk);
      offset += bytesPerChunk;
      chunksSent++;

      // Wall-clock pacing: schedule the next chunk at the correct time
      const nextChunkTime = startTime + chunksSent * CHUNK_DURATION * 1000;
      const delay = nextChunkTime - Date.now();
      setTimeout(sendNextChunk, Math.max(0, delay));
    }

    sendNextChunk();
  });

  ws.on("message", (message) => {
    const data = JSON.parse(message);

    if (data.type === "Begin") {
      console.log(`Session ID: ${data.id}`);
    } else if (data.type === "Turn") {
      const transcript = data.transcript || "";
      if (!transcript) return;
      if (data.end_of_turn) {
        console.log(`[Final]: ${transcript}`);
      } else {
        console.log(`[Partial]: ${transcript}`);
      }
    } else if (data.type === "Termination") {
      console.log(`Done. Processed ${data.audio_duration_seconds || 0}s of audio.`);
    }
  });

  ws.on("error", (error) => console.error(`Error: ${error}`));

  ws.on("close", (code, reason) => {
    console.log(`Disconnected (status=${code})`);
  });
}

run();
```

## Step-by-step guide

### Install dependencies

```bash
pip install websocket-client
```

```bash
npm install ws
```

### Prepare your audio file

The Streaming API accepts raw audio samples. WAV is the simplest format to work with because it contains uncompressed PCM data that you can read directly.

Your audio file must be:
- **Mono** (single channel)
- **16-bit PCM** encoding
- A **sample rate** that matches the `sample_rate` connection parameter

If your file doesn't meet these requirements, convert it with [FFmpeg](https://ffmpeg.org/):

```bash
ffmpeg -i input.mp3 -ar 16000 -ac 1 -sample_fmt s16 output.wav
```

To check your file's properties:

```bash
ffprobe -v quiet -print_format json -show_streams audio.wav
```

### Configure the connection

Set your API key and match the `sample_rate` parameter to your audio file:

```python
import websocket
import json
import threading
import time
import wave
import os
from urllib.parse import urlencode

ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AUDIO_FILE = "audio.wav"
CHUNK_DURATION = 0.1  # 100ms per chunk
SAMPLE_RATE = 16000

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": SAMPLE_RATE,
}
API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"
```

```javascript
import WebSocket from "ws";
import { readFileSync } from "fs";
import { stringify } from "querystring";

const ASSEMBLYAI_API_KEY = process.env.ASSEMBLYAI_API_KEY;
const AUDIO_FILE = "audio.wav";
const CHUNK_DURATION = 0.1; // 100ms per chunk
const SAMPLE_RATE = 16000;

const CONNECTION_PARAMS = {
  speech_model: "universal-3-5-pro",
  sample_rate: SAMPLE_RATE,
};
const API_ENDPOINT = `wss://streaming.assemblyai.com/v3/ws?${stringify(CONNECTION_PARAMS)}`;
```

### Implement wall-clock pacing

The key to simulating real-time audio is **wall-clock pacing**. Instead of calling `sleep` for a fixed duration after each chunk (which accumulates drift from processing time), track elapsed time from the start and sleep only until the next chunk is due.

Here's the difference:

**Naive approach (not recommended)** &mdash; Fixed sleep after each send. Processing time adds up, so audio arrives progressively later than real time:

```python
# Don't do this for benchmarking
while True:
    frames = wav_file.readframes(frames_per_chunk)
    ws.send(frames)
    time.sleep(chunk_duration)  # Drift accumulates over time
```

**Wall-clock approach (recommended)** &mdash; Calculate when each chunk should be sent based on the start time. This self-corrects any drift:

```python
start_time = time.monotonic()
chunks_sent = 0

while not stop_event.is_set():
    frames = wav_file.readframes(frames_per_chunk)
    if not frames:
        break

    ws.send(frames, websocket.ABNF.OPCODE_BINARY)
    chunks_sent += 1

    # Sleep until the next chunk is due
    next_chunk_time = start_time + (chunks_sent * CHUNK_DURATION)
    sleep_duration = next_chunk_time - time.monotonic()
    if sleep_duration > 0:
        time.sleep(sleep_duration)
```

```javascript expandable
const audioData = readFileSync(AUDIO_FILE); // Raw PCM bytes from WAV data chunk
const bytesPerSample = 2; // 16-bit PCM
const bytesPerChunk = Math.floor(SAMPLE_RATE * CHUNK_DURATION) * bytesPerSample;
let offset = 0;
let chunksSent = 0;
const startTime = Date.now();

function sendNextChunk() {
  if (offset >= audioData.length) {
    ws.send(JSON.stringify({ type: "Terminate" }));
    return;
  }

  const chunk = audioData.subarray(offset, offset + bytesPerChunk);
  ws.send(chunk);
  offset += bytesPerChunk;
  chunksSent++;

  // Schedule the next chunk at the correct wall-clock time
  const nextChunkTime = startTime + chunksSent * CHUNK_DURATION * 1000;
  const delay = nextChunkTime - Date.now();
  setTimeout(sendNextChunk, Math.max(0, delay));
}
```

This approach uses `time.monotonic()` (Python) or `Date.now()` (JavaScript) to track elapsed time from the start of streaming. Each chunk is scheduled based on its position in the file, not relative to the previous chunk. If one iteration takes longer than expected, the next chunk is sent sooner to catch up &mdash; keeping the overall pace at real time.

### End the session

After you send all audio, send a `Terminate` message so the server can flush its buffers and return any remaining transcripts:

```python
ws.send(json.dumps({"type": "Terminate"}))
```

```javascript
ws.send(JSON.stringify({ type: "Terminate" }));
```

The server responds with a `Termination` message that includes the total audio duration processed. Wait for this message before closing the WebSocket connection so you don't miss any final transcripts.

## Choosing a chunk duration

The `CHUNK_DURATION` value controls how much audio you send in each message. Common values:

- **100ms** (`0.1`) &mdash; Good default. Balances network overhead with smooth pacing.
- **50ms** (`0.05`) &mdash; More closely simulates microphone input. Use this if you want behavior closest to a live mic stream.
- **200ms** (`0.2`) &mdash; Fewer network calls, slightly less real-time feel. Acceptable for most benchmarks.

Smaller chunks send more WebSocket messages but more closely approximate continuous microphone input. For benchmarking, 100ms is a good starting point.

## Common mistakes

| Mistake | Impact | Fix |
|---|---|---|
| No pacing at all | Audio arrives in seconds; session may close or return errors | Add wall-clock pacing as shown above |
| Naive fixed sleep | Drift accumulates over a long file; audio arrives late | Use wall-clock pacing with `time.monotonic()` or `Date.now()` |
| Wrong sample rate | Server interprets audio at the wrong speed | Match `sample_rate` to your file. Check with `ffprobe` |
| Sending stereo audio | Only the first channel is used, or the session errors | Convert to mono: `ffmpeg -i input.wav -ac 1 output.wav` |
| Not sending `Terminate` | Server waits for more audio until the session times out, so you miss final transcripts | Always send `{"type": "Terminate"}` after the last audio chunk |

## Next steps

- [Transcribe audio files with Streaming](/streaming/guides/streaming_transcribe_audio_file) &mdash; Full example with audio playback and transcript saving.
- [Evaluate Streaming transcription accuracy with WER](/streaming/guides/evaluate_streaming_wer) &mdash; Benchmark your streaming transcription quality.
- [Common session errors and closures](/streaming/common-session-errors-and-closures) &mdash; Troubleshoot session disconnects.

---

<a id="transcribe-system-audio-in-real-time-macos-docs-streaming-guides-transcribe-system-audio-mdx"></a>

# Transcribe System Audio in Real-Time (macOS)

URL: https://www.assemblyai.com/docs/streaming/guides/transcribe_system_audio
Source: docs/streaming/guides/transcribe_system_audio.mdx
Navigation: Real-time STT > Guides > Tutorials > Basic streaming workflows
Description: Transcribe System Audio in Real-Time (macOS) documentation.

This guide solves the challenge of transcribing system audio, which is can be used for transcribing media content or online calls. By using virtual audio devices, you'll learn how to easily pipe system audio to AssemblyAI's transcription API on both Mac and Windows.

The key to success lies in creating a virtual input device that captures your speaker output and converts it into an input stream. This approach allows you to bypass the limitations of direct system audio access.

**For Mac Users:** We recommend using [BlackHole](https://github.com/ExistentialAudio/BlackHole), a free open-source tool available through Homebrew. BlackHole creates a virtual audio device that can route your system audio to AssemblyAI's API seamlessly.

**For Windows Users:** [Virtual Audio Cable (VAC)](https://vac.muzychenko.net/en/) is a popular option. While we don't provide specific Windows instructions in this guide, VAC offers similar functionality to BlackHole for the Windows environment.

## Quickstart

```python expandable
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "YOUR_API_KEY"  # Replace with your actual API key

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# WAV recording variables
recorded_frames = []  # Store audio frames for WAV file
recording_lock = threading.Lock()  # Thread-safe access to recorded_frames

# --- BlackHole Device Detection ---

def get_blackhole_device_index():
    """Find BlackHole audio device index."""
    p = pyaudio.PyAudio()
    blackhole_index = None

    print("Available audio devices:")

    for i in range(p.get_device_count()):
        dev_info = p.get_device_info_by_index(i)
        print(f"  {i}: {dev_info['name']} (inputs: {dev_info['maxInputChannels']})")

        if str(dev_info['name']).startswith('BlackHole') and dev_info['maxInputChannels'] > 0:
            blackhole_index = i
            print(f"  -> Found BlackHole device at index {i}")

    p.terminate()
    return blackhole_index

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                # Store audio data for WAV recording
                with recording_lock:
                    recorded_frames.append(audio_data)

                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Find BlackHole device
    blackhole_index = get_blackhole_device_index()

    if blackhole_index is None:
        print("Error: BlackHole audio device not found!")
        print("Please install BlackHole from https://existential.audio/blackhole/")
        return

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open Blackhole audio stream
    try:
        stream = audio.open(
            input=True,
            input_device_index=blackhole_index,  # Use BlackHole device
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )

        print(f"BlackHole audio stream opened successfully (device index: {blackhole_index}).")
        print("Now capturing system audio through BlackHole. Press Ctrl+C to stop.")
        print("Make sure audio is routed through BlackHole for transcription.")

    except Exception as e:
        print(f"Error opening Blackhole audio stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if blackhole cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## Step-By-Step Guide

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) and get your API key from your dashboard.

### Install/import Packages & Set API Key

Install the package pyaudio.

```bash
pip install pyaudio
```

Import packages and set your API key.

```python
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

YOUR_API_KEY = "YOUR-API-KEY"  # Replace with your actual API key
```

### Audio Configuration & Global Variables

Set all of your audio configurations and global variables. Make sure that you have the parameter `speech_model` set to `"universal-3-5-pro"`.

```python {3} expandable
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# WAV recording variables
recorded_frames = []  # Store audio frames for WAV file
recording_lock = threading.Lock()  # Thread-safe access to recorded_frames
```

### Define Function to Find Blackhole Audio Device Index

Define a function called `get_blackhole_device_index`, which retrieves the device index for your BlackHole virtual input device.

```python
def get_blackhole_device_index():
    """Find BlackHole audio device index."""
    p = pyaudio.PyAudio()
    blackhole_index = None

    print("Available audio devices:")

    for i in range(p.get_device_count()):
        dev_info = p.get_device_info_by_index(i)
        print(f"  {i}: {dev_info['name']} (inputs: {dev_info['maxInputChannels']})")

        if str(dev_info['name']).startswith('BlackHole') and dev_info['maxInputChannels'] > 0:
            blackhole_index = i
            print(f"  -> Found BlackHole device at index {i}")

    p.terminate()
    return blackhole_index
```

### Websocket Event Handlers

#### Open Websocket

```python expandable
def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                # Store audio data for WAV recording
                with recording_lock:
                    recorded_frames.append(audio_data)

                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()
```

#### Handle Websocket Messages

```python expandable
def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")
```

#### Close Websocket

```python
def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
```

#### Websocket Error Handling

```python
def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()
```

### Begin Real-time STT Transcription

Make sure to find the Blackhole device index and to set it to the `input_device_index`.

```python {5,19} expandable
def run():
    global audio, stream, ws_app

    # Find BlackHole device index
    blackhole_index = get_blackhole_device_index()

    if blackhole_index is None:
        print("Error: BlackHole audio device not found!")
        print("Please install BlackHole from https://existential.audio/blackhole/")
        return

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open Blackhole audio stream
    try:
        stream = audio.open(
            input=True,
            input_device_index=blackhole_index,  # Use BlackHole device
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )

        print(f"BlackHole audio stream opened successfully (device index: {blackhole_index}).")
        print("Now capturing system audio through BlackHole. Press Ctrl+C to stop.")
        print("Make sure audio is routed through BlackHole for transcription.")

    except Exception as e:
        print(f"Error opening Blackhole audio stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if blackhole cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

You can press Ctrl+C to stop the transcription.

## Troubleshooting

- You need to select BlackHole as your system output device for the audio to be piped correctly

- If you still need to hear the audio, you can create a multi-output device on Mac that sends audio to both BlackHole and your speakers/headphones
  Here's how to set it up:
  Open "Audio MIDI Setup" (you can find this by searching in Spotlight).
  Click the "+" button in the bottom left corner and choose "Create Multi-Output Device".
  In the list on the right, check both your regular output (e.g., "MacBook Pro Speakers") and "BlackHole 2ch".
  Optionally, rename this new device to something like "BlackHole + Speakers". You may need to modify your script to search for this new device.

---

<a id="terminate-streaming-session-after-inactivity-docs-streaming-guides-terminate-realtime-programmatically-mdx"></a>

# Terminate Streaming Session After Inactivity

URL: https://www.assemblyai.com/docs/streaming/guides/terminate_realtime_programmatically
Source: docs/streaming/guides/terminate_realtime_programmatically.mdx
Navigation: Real-time STT > Guides > Tutorials > Basic streaming workflows
Description: Terminate Streaming Session After Inactivity documentation.

An often-overlooked aspect of implementing AssemblyAI's Streaming Speech-to-Text (STT) service is efficiently terminating transcription sessions. In this cookbook, you will learn how to terminate a Streaming session after any fixed duration of silence.

For the full code, refer to this [GitHub gist.](https://gist.github.com/m-ods/133f7c7e4c08abf97ae53dc7577cadf7)

## Quickstart

```python expandable
import pyaudio
import websocket
import json
import os
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": 16000,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()

# Silence tracking
last_transcript_received = datetime.now()
terminated = False

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    global last_transcript_received, terminated

    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if terminated and msg_type != "Termination":
            return

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')

            # Update timestamp if meaningful speech received
            if transcript.strip():
                last_transcript_received = datetime.now()

            # Check for silence timeout
            silence_duration = (datetime.now() - last_transcript_received).total_seconds()
            if silence_duration > 5:
                print("No transcription received in 5 seconds. Terminating session...")
                try:
                    ws.send(json.dumps({"type": "Terminate"}))
                except Exception:
                    pass
                terminated = True
                stop_event.set()
                return

        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": ASSEMBLYAI_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-step instructions

First, install the required packages.

```bash
pip install websocket-client pyaudio
```

Import packages and set your API key.

```python
import pyaudio
import websocket
import json
import os
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
```

### Audio configuration and global variables

```python
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": 16000,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
```

## Implementing Speech Activity Checks

Our Streaming API emits a Turn Event each time speech is processed. You can use this behavior to detect inactivity and automatically terminate the session.

We track the timestamp of the most recent non-empty transcript. On every Turn Event, we:

- Update the timestamp if meaningful speech is received

- Check how many seconds have passed since the last valid transcript

- If that exceeds your timeout (e.g. 5 seconds), terminate the session

### Key Variables

```python
last_transcript_received = datetime.now()
terminated = False
```

These are updated on every Turn event.

### WebSocket event handlers

#### Open WebSocket

```python expandable
def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()
```

#### Handle WebSocket messages with silence detection

```python expandable
def on_message(ws, message):
    global last_transcript_received, terminated

    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if terminated and msg_type != "Termination":
            return

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')

            # Update timestamp if meaningful speech received
            if transcript.strip():
                last_transcript_received = datetime.now()

            # Check for silence timeout
            silence_duration = (datetime.now() - last_transcript_received).total_seconds()
            if silence_duration > 5:
                print("No transcription received in 5 seconds. Terminating session...")
                try:
                    ws.send(json.dumps({"type": "Terminate"}))
                except Exception:
                    pass
                terminated = True
                stop_event.set()
                return

        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")
```

This pattern ensures sessions are cleanly terminated after inactivity.

#### WebSocket error and close handlers

```python expandable
def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
```

### Begin real-time STT transcription

```python expandable
def run():
    global audio, stream, ws_app

    audio = pyaudio.PyAudio()

    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": ASSEMBLYAI_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## What You'll Observe

- Live transcription continues as long as there's speech

- After 5 seconds of silence, the session ends automatically

You can change the timeout value to suit your needs by modifying the `silence_duration > 5` check.

---

<a id="migrating-from-streaming-v2-to-streaming-v3-python-docs-streaming-guides-v2-to-v3-migration-mdx"></a>

# Migrating from Streaming v2 to Streaming v3 (Python)

URL: https://www.assemblyai.com/docs/streaming/guides/v2_to_v3_migration
Source: docs/streaming/guides/v2_to_v3_migration.mdx
Navigation: Real-time STT > Guides > Tutorials > Basic streaming workflows
Description: Migrating from Streaming v2 to Streaming v3 (Python) documentation.

This cookbook guides you through migrating from AssemblyAI's legacy Real-time STT model (v2) to our latest Universal Real-time STT model (v3), which provides ultra-low latency for faster transcription, intelligent endpointing for more natural speech detection, and improved accuracy across various audio conditions.

Check out this [blog post](https://www.assemblyai.com/blog/introducing-universal-streaming) to learn more about this new model!

## Overview of changes

The migration involves several key improvements:

- **API Version**: Upgrade from v2 (`/v2/realtime/ws`) to v3 (`/v3/ws`)
- **Enhanced Error Handling**: Robust cleanup and resource management
- **Improved Threading**: Better control over audio streaming threads
- **Modern Message Format**: Updated message types and structure
- **Configuration Options**: More flexible connection parameters
- **Graceful Shutdown**: Proper termination handling

You can follow the step-by-step guide below to make changes to your existing code but here is what your code should look like in the end:

```python expandable
import pyaudio
import websocket
import json
import threading
import time
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "YOUR-API-KEY"  # Replace with your actual API key

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,  # Request formatted final transcripts
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

For more information on our Universal Streaming feature, see [this section](/streaming) of our official documentation.

## Step-by-step migration guide

### 1. Update API endpoint and configuration

**Before (v2):**

```python
ws = websocket.WebSocketApp(
    f'wss://api.assemblyai.com/v2/realtime/ws?sample_rate={SAMPLE_RATE}',
    header={'Authorization': YOUR_API_KEY},
    on_message=on_message,
    on_open=on_open,
    on_error=on_error,
    on_close=on_close
)
```

**After (v3):**

```python
CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,  # Request formatted final transcripts
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

ws_app = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": YOUR_API_KEY},
    # ...
)
```

**Key Changes:**

- New base URL: `streaming.assemblyai.com` instead of `api.assemblyai.com`
- Version upgrade: `/v3/ws` instead of `/v2/realtime/ws`
- Configuration via URL parameters using `urlencode()`
- Added `format_turns` option for better transcript formatting

### 2. Improve audio configuration

**Before (v2):**

```python
FRAMES_PER_BUFFER = 3200  # 200ms of audio
SAMPLE_RATE = 16000
CHANNELS = 1
FORMAT = pyaudio.paInt16
```

**After (v3):**

```python
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16
```

**Key Changes:**

- Reduced buffer size from 200ms to 50ms for lower latency
- Sample rate now references the configuration parameter
- Added detailed comments explaining the calculations

### 3. Enhance thread management

**Before (v2):**

```python
def on_open(ws):
    def stream_audio():
        while True:
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f'\nError streaming audio: {e}')
                break

    audio_thread = Thread(target=stream_audio, daemon=True)
    audio_thread.start()
```

**After (v3):**

```python
# Global variables for better resource management
stop_event = threading.Event()
audio_thread = None

def on_open(ws):
    def stream_audio():
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()
```

**Key Changes:**

- Added `threading.Event()` for controlled thread termination
- Global `audio_thread` variable for better lifecycle management
- Condition-based loop (`while not stop_event.is_set()`) instead of infinite loop
- Improved error handling and logging

### 4. Update message handling

**Before (v2):**

```python expandable
def on_message(ws, message):
    try:
        msg = json.loads(message)
        msg_type = msg.get('message_type')

        if msg_type == 'SessionBegins':
            session_id = msg.get('session_id')
            print("Session ID:", session_id)
            return

        text = msg.get('text', '')
        if not text:
            return

        if msg_type == 'PartialTranscript':
            print(text, end='\r')
        elif msg_type == 'FinalTranscript':
            print(text, end='\r\n')
        elif msg_type == 'error':
            print(f'\nError: {msg.get("error", "Unknown error")}')
    except Exception as e:
        print(f'\nError handling message: {e}')
```

**After (v3):**

```python expandable
def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")
```

**Key Changes:**

- Message types renamed: `SessionBegins` → `Begin`, `PartialTranscript`/`FinalTranscript` → `Turn`
- Field names updated: `message_type` → `type`, `session_id` → `id`, `text` → `transcript`
- Added session expiration timestamp handling
- Improved transcript formatting with `turn_is_formatted` flag
- Added `Termination` message handling with session statistics
- Enhanced error handling with specific `JSONDecodeError` catch

### 5. Implement robust resource management

**Before (v2):**

```python
def on_close(ws, status, msg):
    stream.stop_stream()
    stream.close()
    audio.terminate()
    print('\nDisconnected')

# Global audio resources (potential for memory leaks)
audio = pyaudio.PyAudio()
stream = audio.open(...)
```

**After (v3):**

```python
def on_close(ws, close_status_code, close_msg):
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    global stream, audio
    stop_event.set()  # Signal audio thread to stop

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Ensure audio thread cleanup
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

def on_error(ws, error):
    print(f"\nWebSocket Error: {error}")
    stop_event.set()  # Signal threads to stop on error
```

**Key Changes:**

- Added thread stop signaling via `stop_event.set()`
- Conditional resource cleanup with null checks
- Proper thread joining with timeout
- Resource nullification to prevent reuse
- Enhanced error handling in `on_error`

### 6. Add graceful shutdown handling

**Before (v2):**

```python
try:
    ws.run_forever()
except Exception as e:
    print(f'\nError: {e}')
```

**After (v3):**

```python expandable
def run():
    # ... initialization code ...

    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        # Send termination message
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                ws_app.send(json.dumps(terminate_message))
                time.sleep(5)  # Allow message processing
            except Exception as e:
                print(f"Error sending termination message: {e}")

        ws_app.close()
        ws_thread.join(timeout=2.0)
    finally:
        # Final cleanup
        # ... resource cleanup code ...
        pass
```

**Key Changes:**

- WebSocket runs in separate thread for better control
- Proper KeyboardInterrupt handling
- Graceful termination message sending
- Thread joining with timeouts
- Comprehensive cleanup in `finally` block

  Streaming is billed on the total duration that your WebSocket connection stays open (session duration), not on the amount of audio sent. Always terminate sessions explicitly to avoid being billed for idle time — unterminated sessions auto-close after 3 hours and are billed for the full duration. See [Streaming Speech-to-Text billing](/billing-and-pricing#streaming-speech-to-text-billing) for details.

### 7. Improve error handling and logging

**Before (v2):**

- Basic error printing
- Limited context in error messages
- No resource cleanup on errors

**After (v3):**

- Detailed error context and timestamps
- Proper exception type handling
- Resource cleanup on all error paths
- Connection status checking before operations

## Migration checklist

- [ ] Update API endpoint from v2 to v3
- [ ] Change base URL to `streaming.assemblyai.com`
- [ ] Update message type handling (`Begin`, `Turn`, `Termination`)
- [ ] Implement `threading.Event()` for thread control
- [ ] Add proper resource cleanup in all code paths
- [ ] Update field names in message parsing
- [ ] Add graceful shutdown with termination messages
- [ ] Implement timeout-based thread joining
- [ ] Add detailed error logging with context
- [ ] Test KeyboardInterrupt handling
- [ ] Verify audio resource cleanup
- [ ] Test connection failure scenarios

## Testing your migration

1. **Basic Functionality**: Verify transcription works with simple speech
2. **Error Handling**: Test with invalid API keys or network issues
3. **Graceful Shutdown**: Test Ctrl+C interruption
4. **Resource Cleanup**: Monitor for memory leaks during extended use
5. **Thread Management**: Verify proper thread termination
6. **Message Formatting**: Test with `format_turns` enabled/disabled

## Common migration issues

### Issue: "WebSocket connection failed"

**Solution**: Verify you're using the new v3 endpoint URL and proper authentication header format.

### Issue: "Message type not recognized"

**Solution**: Update message type handling from old names (`SessionBegins`, `PartialTranscript`) to new ones (`Begin`, `Turn`).

### Issue: "Audio thread won't stop"

**Solution**: Ensure you're using `threading.Event()` and calling `stop_event.set()` in error handlers.

### Issue: "Resource leak warnings"

**Solution**: Verify all audio resources are properly cleaned up in `on_close` and `finally` blocks.

## Benefits of migration

- **Improved Reliability**: Better error handling and recovery
- **Lower Latency**: Reduced buffer sizes for faster response
- **Enhanced Features**: Formatted transcripts and session statistics
- **Better Resource Management**: Proper cleanup prevents memory leaks
- **Graceful Shutdown**: Clean termination with proper cleanup
- **Modern Architecture**: Improved threading and event handling

## Conclusion

This migration provides a more robust, maintainable, and feature-rich streaming transcription implementation. The enhanced error handling, resource management, and modern API features make it suitable for production use cases where reliability and performance are critical.

---

<a id="migrating-from-streaming-v2-to-streaming-v3-javascript-docs-streaming-guides-v2-to-v3-migration-js-mdx"></a>

# Migrating from Streaming v2 to Streaming v3 (JavaScript)

URL: https://www.assemblyai.com/docs/streaming/guides/v2_to_v3_migration_js
Source: docs/streaming/guides/v2_to_v3_migration_js.mdx
Navigation: Real-time STT > Guides > Tutorials > Basic streaming workflows
Description: Migrating from Streaming v2 to Streaming v3 (JavaScript) documentation.

This cookbook guides you through migrating from AssemblyAI's legacy Real-time STT model (v2) to our latest Universal Real-time STT model (v3), which provides ultra-low latency for faster transcription, intelligent endpointing for more natural speech detection, and improved accuracy across various audio conditions.

Check out this [blog post](https://www.assemblyai.com/blog/introducing-universal-streaming) to learn more about this new model!

## Overview of changes

The migration involves several key improvements:

- **API Version**: Upgrade from v2 (`/v2/realtime/ws`) to v3 (`/v3/ws`)
- **Enhanced Error Handling**: Robust cleanup and resource management
- **Modern Message Format**: Updated message types and structure
- **Configuration Options**: More flexible connection parameters
- **Graceful Shutdown**: Proper termination handling

You can follow the step-by-step guide below to make changes to your existing code but here is what your code should look like in the end:

```javascript expandable
const WebSocket = require("ws");
const mic = require("mic");
const querystring = require("querystring");
const fs = require("fs");

// --- Configuration ---
const YOUR_API_KEY = "YOUR-API-KEY"; // Replace with your actual API key
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  format_turns: true, // Request formatted final transcripts
};
const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

// Audio Configuration
const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;
const CHANNELS = 1;

// Global variables
let micInstance = null;
let micInputStream = null;
let ws = null;
let stopRequested = false;

// WAV recording variables
let recordedFrames = []; // Store audio frames for WAV file

// --- Helper functions ---
function clearLine() {
  process.stdout.write("\r" + " ".repeat(80) + "\r");
}

function formatTimestamp(timestamp) {
  return new Date(timestamp * 1000).toISOString();
}

function createWavHeader(sampleRate, channels, dataLength) {
  const buffer = Buffer.alloc(44);

  // RIFF header
  buffer.write("RIFF", 0);
  buffer.writeUInt32LE(36 + dataLength, 4);
  buffer.write("WAVE", 8);

  // fmt chunk
  buffer.write("fmt ", 12);
  buffer.writeUInt32LE(16, 16); // fmt chunk size
  buffer.writeUInt16LE(1, 20); // PCM format
  buffer.writeUInt16LE(channels, 22);
  buffer.writeUInt32LE(sampleRate, 24);
  buffer.writeUInt32LE(sampleRate * channels * 2, 28); // byte rate
  buffer.writeUInt16LE(channels * 2, 32); // block align
  buffer.writeUInt16LE(16, 34); // bits per sample

  // data chunk
  buffer.write("data", 36);
  buffer.writeUInt32LE(dataLength, 40);

  return buffer;
}

function saveWavFile() {
  if (recordedFrames.length === 0) {
    console.log("No audio data recorded.");
    return;
  }

  // Generate filename with timestamp
  const timestamp = new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19);
  const filename = `recorded_audio_${timestamp}.wav`;

  try {
    // Combine all recorded frames
    const audioData = Buffer.concat(recordedFrames);
    const dataLength = audioData.length;

    // Create WAV header
    const wavHeader = createWavHeader(SAMPLE_RATE, CHANNELS, dataLength);

    // Write WAV file
    const wavFile = Buffer.concat([wavHeader, audioData]);
    fs.writeFileSync(filename, wavFile);

    console.log(`Audio saved to: ${filename}`);
    console.log(
      `Duration: ${(dataLength / (SAMPLE_RATE * CHANNELS * 2)).toFixed(2)} seconds`
    );
  } catch (error) {
    console.error(`Error saving WAV file: ${error}`);
  }
}

// --- Main function ---
async function run() {
  console.log("Starting AssemblyAI streaming transcription...");
  console.log("Audio will be saved to a WAV file when the session ends.");

  // Initialize WebSocket connection
  ws = new WebSocket(API_ENDPOINT, {
    headers: {
      Authorization: YOUR_API_KEY,
    },
  });

  // Setup WebSocket event handlers
  ws.on("open", () => {
    console.log("WebSocket connection opened.");
    console.log(`Connected to: ${API_ENDPOINT}`);
    // Start the microphone
    startMicrophone();
  });

  ws.on("message", (message) => {
    try {
      const data = JSON.parse(message);
      const msgType = data.type;

      if (msgType === "Begin") {
        const sessionId = data.id;
        const expiresAt = data.expires_at;
        console.log(
          `\nSession began: ID=${sessionId}, ExpiresAt=${formatTimestamp(expiresAt)}`
        );
      } else if (msgType === "Turn") {
        const transcript = data.transcript || "";
        if (data.end_of_turn) {
          clearLine();
          console.log(transcript);
        } else {
          process.stdout.write(`\r${transcript}`);
        }
      } else if (msgType === "Termination") {
        const audioDuration = data.audio_duration_seconds;
        const sessionDuration = data.session_duration_seconds;
        console.log(
          `\nSession Terminated: Audio Duration=${audioDuration}s, Session Duration=${sessionDuration}s`
        );
      }
    } catch (error) {
      console.error(`\nError handling message: ${error}`);
      console.error(`Message data: ${message}`);
    }
  });

  ws.on("error", (error) => {
    console.error(`\nWebSocket Error: ${error}`);
    cleanup();
  });

  ws.on("close", (code, reason) => {
    console.log(`\nWebSocket Disconnected: Status=${code}, Msg=${reason}`);
    cleanup();
  });

  // Handle process termination
  setupTerminationHandlers();
}

function startMicrophone() {
  try {
    micInstance = mic({
      rate: SAMPLE_RATE.toString(),
      channels: CHANNELS.toString(),
      debug: false,
      exitOnSilence: 6, // This won't actually exit, just a parameter for mic
    });

    micInputStream = micInstance.getAudioStream();

    micInputStream.on("data", (data) => {
      if (ws && ws.readyState === WebSocket.OPEN && !stopRequested) {
        // Store audio data for WAV recording
        recordedFrames.push(Buffer.from(data));

        // Send audio data to WebSocket
        ws.send(data);
      }
    });

    micInputStream.on("error", (err) => {
      console.error(`Microphone Error: ${err}`);
      cleanup();
    });

    micInstance.start();
    console.log("Microphone stream opened successfully.");
    console.log("Speak into your microphone. Press Ctrl+C to stop.");
  } catch (error) {
    console.error(`Error opening microphone stream: ${error}`);
    cleanup();
  }
}

function cleanup() {
  stopRequested = true;

  // Save recorded audio to WAV file
  saveWavFile();

  // Stop microphone if it's running
  if (micInstance) {
    try {
      micInstance.stop();
    } catch (error) {
      console.error(`Error stopping microphone: ${error}`);
    }
    micInstance = null;
  }

  // Close WebSocket connection if it's open
  if (ws && [WebSocket.OPEN, WebSocket.CONNECTING].includes(ws.readyState)) {
    try {
      // Send termination message if possible
      if (ws.readyState === WebSocket.OPEN) {
        const terminateMessage = { type: "Terminate" };
        console.log(
          `Sending termination message: ${JSON.stringify(terminateMessage)}`
        );
        ws.send(JSON.stringify(terminateMessage));
      }
      ws.close();
    } catch (error) {
      console.error(`Error closing WebSocket: ${error}`);
    }
    ws = null;
  }

  console.log("Cleanup complete.");
}

function setupTerminationHandlers() {
  // Handle Ctrl+C and other termination signals
  process.on("SIGINT", () => {
    console.log("\nCtrl+C received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  process.on("SIGTERM", () => {
    console.log("\nTermination signal received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  // Handle uncaught exceptions
  process.on("uncaughtException", (error) => {
    console.error(`\nUncaught exception: ${error}`);
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(1), 1000);
  });
}

// Start the application
run();
```

For more information on our Universal Streaming feature, see [this section](/streaming) of our official documentation.

## Step-by-step migration guide

### 1. Update API endpoint and configuration

**Before (v2):**

```javascript
const API_KEY = "<YOUR_API_KEY>";
const SAMPLE_RATE = 16000; // 16kHz sample rate

const ws = new WebSocket(
  `wss://api.assemblyai.com/v2/realtime/ws?sample_rate=${SAMPLE_RATE}`,
  {
    headers: {
      Authorization: API_KEY,
    },
  }
);
```

**After (v3):**

```javascript
// --- Configuration ---
const YOUR_API_KEY = "YOUR-API-KEY"; // Replace with your actual API key
const CONNECTION_PARAMS = {
  sample_rate: 16000,
  format_turns: true, // Request formatted final transcripts
};
const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

// Initialize WebSocket connection
ws = new WebSocket(API_ENDPOINT, {
  headers: {
    Authorization: YOUR_API_KEY,
  },
});
```

**Key Changes:**

- New base URL: `streaming.assemblyai.com` instead of `api.assemblyai.com`
- Version upgrade: `/v3/ws` instead of `/v2/realtime/ws`
- Configuration via URL parameters using `querystring`
- Added `format_turns` option for better transcript formatting

### 2. Audio configuration

**Before (v2):**

```javascript
const SAMPLE_RATE = 16000;
const CHANNELS = 1;
```

**After (v3):**

```javascript
const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;
const CHANNELS = 1;
```

**Key Changes:**

- Sample rate now references the configuration parameter

### 3. Update message handling schema

**Before (v2):**

```javascript expandable
ws.on("message", (message) => {
  try {
    const msg = JSON.parse(message);
    const msgType = msg.message_type;

    if (msgType === "SessionBegins") {
      const sessionId = msg.session_id;
      console.log("Session ID:", sessionId);
      return;
    }

    const text = msg.text || "";
    if (!text) {
      return;
    }

    if (msgType === "PartialTranscript") {
      console.log("Partial:", text);
    } else if (msgType === "FinalTranscript") {
      console.log("Final:", text);
    } else if (msgType === "error") {
      console.error("Error:", msg.error);
    }
  } catch (error) {
    console.error("Error handling message:", error);
  }
});
```

**After (v3):**

```javascript expandable
ws.on("message", (message) => {
  try {
    const data = JSON.parse(message);
    const msgType = data.type;
    if (msgType === "Begin") {
      const sessionId = data.id;
      const expiresAt = data.expires_at;
      console.log(
        `\nSession began: ID=${sessionId}, ExpiresAt=${formatTimestamp(expiresAt)}`
      );
    } else if (msgType === "Turn") {
      const transcript = data.transcript || "";
      if (data.end_of_turn) {
        clearLine();
        console.log(transcript);
      } else {
        process.stdout.write(`\r${transcript}`);
      }
    } else if (msgType === "Termination") {
      const audioDuration = data.audio_duration_seconds;
      const sessionDuration = data.session_duration_seconds;
      console.log(
        `\nSession Terminated: Audio Duration=${audioDuration}s, Session Duration=${sessionDuration}s`
      );
    }
  } catch (error) {
    console.error(`\nError handling message: ${error}`);
    console.error(`Message data: ${message}`);
  }
});
```

**Key Changes:**

- Message types renamed: `SessionBegins` → `Begin`, `PartialTranscript`/`FinalTranscript` → `Turn`
- Field names updated: `message_type` → `type`, `session_id` → `id`, `text` → `transcript`
- Added session expiration timestamp handling (`expires_at`)
- New transcript formatting with `turn_is_formatted` flag
- Added turn tracking with `turn_order` and `end_of_turn` fields
- New confidence scoring with `end_of_turn_confidence`
- Added `Termination` message with session statistics
- Error handling moved from message-based to WebSocket events

### 4. Add graceful shutdown handling and improve error handling and logging

**Before (v2):**

```javascript expandable
ws.on("close", (code, reason) => onClose(ws, code, reason));

function onClose(ws, code, reason) {
  if (recording) {
    recording.end();
  }
  console.log("Disconnected");
}

process.on("SIGINT", async function () {
  console.log();
  console.log("Stopping recording");
  if (recording) {
    recording.end();
  }
  console.log("Closing real-time transcript connection");
  if (ws.readyState === WebSocket.OPEN) {
    ws.close();
  }
  process.exit();
});
```

**After (v3):**

```javascript expandable
ws.on("close", (code, reason) => {
  console.log(`\nWebSocket Disconnected: Status=${code}, Msg=${reason}`);
  cleanup();
});

function cleanup() {
  stopRequested = true;
  // Save recorded audio to WAV file
  saveWavFile();
  // Stop microphone if it's running
  if (micInstance) {
    try {
      micInstance.stop();
    } catch (error) {
      console.error(`Error stopping microphone: ${error}`);
    }
    micInstance = null;
  }
  // Close WebSocket connection if it's open
  if (ws && [WebSocket.OPEN, WebSocket.CONNECTING].includes(ws.readyState)) {
    try {
      // Send termination message if possible
      if (ws.readyState === WebSocket.OPEN) {
        const terminateMessage = { type: "Terminate" };
        console.log(
          `Sending termination message: ${JSON.stringify(terminateMessage)}`
        );
        ws.send(JSON.stringify(terminateMessage));
      }
      ws.close();
    } catch (error) {
      console.error(`Error closing WebSocket: ${error}`);
    }
    ws = null;
  }
  console.log("Cleanup complete.");
}

function setupTerminationHandlers() {
  // Handle Ctrl+C and other termination signals
  process.on("SIGINT", () => {
    console.log("\nCtrl+C received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });
  process.on("SIGTERM", () => {
    console.log("\nTermination signal received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });
  // Handle uncaught exceptions
  process.on("uncaughtException", (error) => {
    console.error(`\nUncaught exception: ${error}`);
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(1), 1000);
  });
}
```

**Key Changes:**

- Proper KeyboardInterrupt handling
- Graceful termination message sending
- Detailed error context and timestamps
- Proper exception type handling
- Resource cleanup on all error paths
- Connection status checking before operations

  Streaming is billed on the total duration that your WebSocket connection stays open (session duration), not on the amount of audio sent. Always terminate sessions explicitly to avoid being billed for idle time — unterminated sessions auto-close after 3 hours and are billed for the full duration. See [Streaming Speech-to-Text billing](/billing-and-pricing#streaming-speech-to-text-billing) for details.

## Migration checklist

- [ ] Update API endpoint from v2 to v3
- [ ] Update message type handling (`Begin`, `Turn`, `Termination`)
- [ ] Add proper resource cleanup in all code paths
- [ ] Update field names in message parsing
- [ ] Add graceful shutdown with termination messages
- [ ] Add detailed error logging with context
- [ ] Test KeyboardInterrupt handling
- [ ] Verify audio resource cleanup
- [ ] Test connection failure scenarios

## Testing your migration

1. **Basic Functionality**: Verify transcription works with simple speech
2. **Error Handling**: Test with invalid API keys or network issues
3. **Graceful Shutdown**: Test Ctrl+C interruption
4. **Resource Cleanup**: Monitor for memory leaks during extended use
5. **Message Formatting**: Test with `format_turns` enabled/disabled

## Common migration issues

### Issue: "WebSocket connection failed"

**Solution**: Verify you're using the new v3 endpoint URL and proper authentication header format.

### Issue: "Message type not recognized"

**Solution**: Update message type handling from old names (`SessionBegins`, `PartialTranscript`) to new ones (`Begin`, `Turn`).

## Benefits of migration

- **Improved Reliability**: Better error handling and recovery
- **Lower Latency**: Reduced buffer sizes for faster response
- **Enhanced Features**: Formatted transcripts and session statistics
- **Better Resource Management**: Proper cleanup prevents memory leaks
- **Graceful Shutdown**: Clean termination with proper cleanup

## Conclusion

This migration provides a more robust, maintainable, and feature-rich streaming transcription implementation. The enhanced error handling, resource management, and modern API features make it suitable for production use cases where reliability and performance are critical.

---

<a id="next-js-example-using-real-time-stt-docs-nav-links-next-js-example-using-streaming-stt-mdx"></a>

# Next.js example using Real-time STT

URL: https://www.assemblyai.com/docs/nav-links/next-js-example-using-streaming-stt
Source: docs/nav-links/next-js-example-using-streaming-stt.mdx
Navigation: Real-time STT > Guides > Tutorials > Streaming for front-end applications
Description: Open Next.js example using Real-time STT.

---

<a id="vanilla-javascript-example-using-real-time-stt-docs-nav-links-vanilla-javascript-example-using-streaming-stt-mdx"></a>

# Vanilla Javascript example using Real-time STT

URL: https://www.assemblyai.com/docs/nav-links/vanilla-javascript-example-using-streaming-stt
Source: docs/nav-links/vanilla-javascript-example-using-streaming-stt.mdx
Navigation: Real-time STT > Guides > Tutorials > Streaming for front-end applications
Description: Open Vanilla Javascript example using Real-time STT.

---

<a id="apply-llm-gateway-to-streaming-docs-streaming-guides-real-time-llm-gateway-mdx"></a>

# Apply LLM Gateway to Streaming

URL: https://www.assemblyai.com/docs/streaming/guides/real_time_llm_gateway
Source: docs/streaming/guides/real_time_llm_gateway.mdx
Navigation: Real-time STT > Guides > Tutorials > Streaming with LLM Gateway
Description: Learn how to analyze streaming audio transcripts using LLM Gateway.

## Overview

A Large Language Model (LLM) is a machine learning model that uses natural language processing (NLP) to generate text. [LLM Gateway](/llm-gateway/quickstart) is a unified API that provides access to 25+ models from Claude, GPT, Gemini, and more through a single interface. You can use LLM Gateway to analyze streaming audio transcripts in real time, for example to summarize a live conversation or extract action items as they happen.

By the end of this tutorial, you'll be able to use LLM Gateway to analyze a streaming audio transcript from your microphone.

Here's the full sample code for what you'll build in this tutorial:

```python expandable
import pyaudio
import websocket
import json
import threading
import time
import wave
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "<YOUR_API_KEY>"  # Replace with your actual API key

# LLM Gateway Configuration
PROMPT = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"
LLM_GATEWAY_CONFIG = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": PROMPT}
    ],
    "max_tokens": 4000
}

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,  # Request formatted final transcripts
    "speech_model": "universal-3-5-pro",
    "llm_gateway": json.dumps(LLM_GATEWAY_CONFIG)  # LLM Gateway configuration
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# WAV recording variables
recorded_frames = []  # Store audio frames for WAV file
recording_lock = threading.Lock()  # Thread-safe access to recorded_frames

def save_wav_file():
    """Save recorded audio frames to a WAV file."""
    if not recorded_frames:
        print("No audio data recorded.")
        return

    # Generate filename with timestamp
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"recorded_audio_{timestamp}.wav"

    try:
        with wave.open(filename, 'wb') as wf:
            wf.setnchannels(CHANNELS)
            wf.setsampwidth(2)  # 16-bit = 2 bytes
            wf.setframerate(SAMPLE_RATE)

            # Write all recorded frames
            with recording_lock:
                wf.writeframes(b''.join(recorded_frames))

        print(f"Audio saved to: {filename}")
        print(f"Duration: {len(recorded_frames) * FRAMES_PER_BUFFER / SAMPLE_RATE:.2f} seconds")

    except Exception as e:
        print(f"Error saving WAV file: {e}")

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                # Store audio data for WAV recording
                with recording_lock:
                    recorded_frames.append(audio_data)

                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"Session started: {session_id}")
        elif msg_type == "Turn":
            end_of_turn = data.get('end_of_turn', False)

            if end_of_turn:
                transcript = data.get('transcript', '')
                print(f"\nTranscript:\n{transcript}\n")
        elif msg_type == "LLMGatewayResponse":
            # Extract the LLM response content
            llm_data = data.get('data', {})
            llm_content = llm_data.get("choices", [{}])[0].get("message", {}).get("content", "")
            print(f"LLM Response:\n{llm_content}\n")
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"Session terminated: {audio_duration} seconds of audio processed")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    # Save recorded audio to WAV file
    save_wav_file()

    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
        print("Audio will be saved to a WAV file when the session ends.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

```python expandable
import logging
from typing import Type

import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    LLMGatewayResponseEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TurnEvent,
    TerminationEvent,
)
from assemblyai.streaming.v3.models import LLMGatewayConfig, LLMGatewayMessage

api_key = "<YOUR_API_KEY>"
prompt = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()

def on_begin(self: Type[RealTimeTranscriber], event: BeginEvent):
    print(f"Session started: {event.id}")

def on_turn(self: Type[RealTimeTranscriber], event: TurnEvent):
    if event.end_of_turn:
        print(f"\nTranscript:\n{event.transcript}\n")

def on_llm_response(self: Type[RealTimeTranscriber], event: LLMGatewayResponseEvent):
    # Extract the actual LLM response content from the data
    llm_content = event.data.get("choices", [{}])[0].get("message", {}).get("content", "")
    print(f"LLM Response:\n{llm_content}\n")

def on_terminated(self: Type[RealTimeTranscriber], event: TerminationEvent):
    print(
        f"Session terminated: {event.audio_duration_seconds} seconds of audio processed"
    )

def on_error(self: Type[RealTimeTranscriber], error: RealTimeError):
    print(f"Error occurred: {error}")

def main():
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(
            api_host="streaming.assemblyai.com",
        ),
        api_key=api_key,
    )

    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.LLMGatewayResponse, on_llm_response)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    client.connect(
        RealTimeParameters(
            sample_rate=16000,
            speech_model="universal-3-5-pro",
            format_turns=True,
            llm_gateway=LLMGatewayConfig(
                model="claude-sonnet-4-6",
                messages=[
                    LLMGatewayMessage(role="user", content=prompt)
                ],
                max_tokens=4000
            )
        )
    )

    try:
        client.stream(microphone_stream())
    finally:
        client.disconnect(terminate=True)

if __name__ == "__main__":
    main()
```

```javascript expandable
const WebSocket = require("ws");
const mic = require("mic");
const querystring = require("querystring");
const fs = require("fs");

// --- Configuration ---
const YOUR_API_KEY = "<YOUR_API_KEY>"; // Replace with your actual API key

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};

const CONNECTION_PARAMS = {
  sample_rate: 16000,
  format_turns: true, // Request formatted final transcripts
  speech_model: "universal-3-5-pro",
  llm_gateway: JSON.stringify(LLM_GATEWAY_CONFIG), // LLM Gateway configuration
};
const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

// Audio Configuration
const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;
const CHANNELS = 1;

// Global variables
let micInstance = null;
let micInputStream = null;
let ws = null;
let stopRequested = false;

// WAV recording variables
let recordedFrames = []; // Store audio frames for WAV file

// --- Helper functions ---
function formatTimestamp(timestamp) {
  return new Date(timestamp * 1000).toISOString();
}

function createWavHeader(sampleRate, channels, dataLength) {
  const buffer = Buffer.alloc(44);

  // RIFF header
  buffer.write("RIFF", 0);
  buffer.writeUInt32LE(36 + dataLength, 4);
  buffer.write("WAVE", 8);

  // fmt chunk
  buffer.write("fmt ", 12);
  buffer.writeUInt32LE(16, 16); // fmt chunk size
  buffer.writeUInt16LE(1, 20); // PCM format
  buffer.writeUInt16LE(channels, 22);
  buffer.writeUInt32LE(sampleRate, 24);
  buffer.writeUInt32LE(sampleRate * channels * 2, 28); // byte rate
  buffer.writeUInt16LE(channels * 2, 32); // block align
  buffer.writeUInt16LE(16, 34); // bits per sample

  // data chunk
  buffer.write("data", 36);
  buffer.writeUInt32LE(dataLength, 40);

  return buffer;
}

function saveWavFile() {
  if (recordedFrames.length === 0) {
    console.log("No audio data recorded.");
    return;
  }

  // Generate filename with timestamp
  const timestamp = new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19);
  const filename = `recorded_audio_${timestamp}.wav`;

  try {
    // Combine all recorded frames
    const audioData = Buffer.concat(recordedFrames);
    const dataLength = audioData.length;

    // Create WAV header
    const wavHeader = createWavHeader(SAMPLE_RATE, CHANNELS, dataLength);

    // Write WAV file
    const wavFile = Buffer.concat([wavHeader, audioData]);
    fs.writeFileSync(filename, wavFile);

    console.log(`Audio saved to: ${filename}`);
    console.log(
      `Duration: ${(dataLength / (SAMPLE_RATE * CHANNELS * 2)).toFixed(2)} seconds`
    );
  } catch (error) {
    console.error(`Error saving WAV file: ${error}`);
  }
}

// --- Main function ---
async function run() {
  console.log(
    "Starting AssemblyAI streaming transcription with LLM Gateway..."
  );
  console.log("Audio will be saved to a WAV file when the session ends.");

  // Initialize WebSocket connection
  ws = new WebSocket(API_ENDPOINT, {
    headers: {
      Authorization: YOUR_API_KEY,
    },
  });

  // Setup WebSocket event handlers
  ws.on("open", () => {
    console.log("WebSocket connection opened.");
    console.log(`Connected to: ${API_ENDPOINT_BASE_URL}`);
    // Start the microphone
    startMicrophone();
  });

  ws.on("message", (message) => {
    try {
      const data = JSON.parse(message);
      const msgType = data.type;

      if (msgType === "Begin") {
        const sessionId = data.id;
        console.log(`Session started: ${sessionId}`);
      } else if (msgType === "Turn") {
        const endOfTurn = data.end_of_turn;

        if (endOfTurn) {
          const transcript = data.transcript || "";
          console.log(`\nTranscript:\n${transcript}\n`);
        }
      } else if (msgType === "LLMGatewayResponse") {
        // Extract the LLM response content
        const llmData = data.data || {};
        const llmContent = llmData.choices?.[0]?.message?.content || "";
        console.log(`LLM Response:\n${llmContent}\n`);
      } else if (msgType === "Termination") {
        const audioDuration = data.audio_duration_seconds;
        console.log(
          `Session terminated: ${audioDuration} seconds of audio processed`
        );
      }
    } catch (error) {
      console.error(`\nError handling message: ${error}`);
      console.error(`Message data: ${message}`);
    }
  });

  ws.on("error", (error) => {
    console.error(`\nWebSocket Error: ${error}`);
    cleanup();
  });

  ws.on("close", (code, reason) => {
    console.log(`\nWebSocket Disconnected: Status=${code}, Msg=${reason}`);
    cleanup();
  });

  // Handle process termination
  setupTerminationHandlers();
}

function startMicrophone() {
  try {
    micInstance = mic({
      rate: SAMPLE_RATE.toString(),
      channels: CHANNELS.toString(),
      debug: false,
      exitOnSilence: 6, // This won't actually exit, just a parameter for mic
    });

    micInputStream = micInstance.getAudioStream();

    micInputStream.on("data", (data) => {
      if (ws && ws.readyState === WebSocket.OPEN && !stopRequested) {
        // Store audio data for WAV recording
        recordedFrames.push(Buffer.from(data));

        // Send audio data to WebSocket
        ws.send(data);
      }
    });

    micInputStream.on("error", (err) => {
      console.error(`Microphone Error: ${err}`);
      cleanup();
    });

    micInstance.start();
    console.log("Microphone stream opened successfully.");
    console.log("Speak into your microphone. Press Ctrl+C to stop.");
  } catch (error) {
    console.error(`Error opening microphone stream: ${error}`);
    cleanup();
  }
}

function cleanup() {
  stopRequested = true;

  // Save recorded audio to WAV file
  saveWavFile();

  // Stop microphone if it's running
  if (micInstance) {
    try {
      micInstance.stop();
    } catch (error) {
      console.error(`Error stopping microphone: ${error}`);
    }
    micInstance = null;
  }

  // Close WebSocket connection if it's open
  if (ws && [WebSocket.OPEN, WebSocket.CONNECTING].includes(ws.readyState)) {
    try {
      // Send termination message if possible
      if (ws.readyState === WebSocket.OPEN) {
        const terminateMessage = { type: "Terminate" };
        console.log(
          `Sending termination message: ${JSON.stringify(terminateMessage)}`
        );
        ws.send(JSON.stringify(terminateMessage));
      }
      ws.close();
    } catch (error) {
      console.error(`Error closing WebSocket: ${error}`);
    }
    ws = null;
  }

  console.log("Cleanup complete.");
}

function setupTerminationHandlers() {
  // Handle Ctrl+C and other termination signals
  process.on("SIGINT", () => {
    console.log("\nCtrl+C received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  process.on("SIGTERM", () => {
    console.log("\nTermination signal received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  // Handle uncaught exceptions
  process.on("uncaughtException", (error) => {
    console.error(`\nUncaught exception: ${error}`);
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(1), 1000);
  });
}

// Start the application
run();
```

```javascript expandable
import { AssemblyAI } from "assemblyai";
import recorder from "node-record-lpcm16";
import { Readable } from "stream";

// --- Configuration ---
const API_KEY = "<YOUR_API_KEY>";

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};

const run = async () => {
  const client = new AssemblyAI({
    apiKey: API_KEY,
  });

  const transcriber = client.streaming.transcriber({
    sampleRate: 16000,
    formatTurns: true,
    speechModel: "universal-3-5-pro",
    llmGateway: LLM_GATEWAY_CONFIG,
  });

  // Session started event
  transcriber.on("open", ({ id }) => {
    console.log(`Session started: ${id}`);
  });

  // Error event
  transcriber.on("error", (error) => {
    console.error(`\nError: ${error}`);
  });

  // Close event
  transcriber.on("close", (code, reason) => {
    console.log("Session closed:", code, reason || "");
  });

  // Turn event - displays transcript when turn ends
  transcriber.on("turn", (turn) => {
    if (turn.end_of_turn) {
      console.log(`\nTranscript:\n${turn.transcript}\n`);
    }
  });

  // LLM Gateway Response event - displays LLM summary
  transcriber.on("llmGatewayResponse", (response) => {
    const llmContent = response.data?.choices?.[0]?.message?.content || "";
    console.log(`\nLLM Response:\n${llmContent}\n`);
  });

  // Termination event
  transcriber.on("termination", (event) => {
    console.log(
      `Session terminated: ${event.audio_duration_seconds} seconds of audio processed`
    );
  });

  try {
    await transcriber.connect();

    console.log("Connecting to streaming transcript service");
    console.log("Starting recording");

    const recording = recorder.record({
      channels: 1,
      sampleRate: 16000,
      audioType: "wav", // Linear PCM
    });

    Readable.toWeb(recording.stream()).pipeTo(transcriber.stream());

    // Stop recording and close connection using Ctrl-C
    process.on("SIGINT", async function () {
      console.log();
      console.log("Stopping recording");
      recording.stop();

      console.log("Closing streaming transcript connection");
      await transcriber.close();

      process.exit();
    });
  } catch (error) {
    console.error(`Error: ${error}`);
  }
};

run();
```

## Before you begin

To complete this tutorial, you need:

- [Python](https://www.python.org/) or [Node](https://nodejs.org/en) installed.
- An <a href="https://www.assemblyai.com/dashboard/signup" target="_blank">AssemblyAI account with a credit card set up</a>.
- A microphone connected to your computer.
- Basic understanding of how to [Transcribe streaming audio](/streaming/getting-started/transcribe-streaming-audio).

## Step 1: Install prerequisites

Install the required packages via pip:

```bash
pip install pyaudio websocket-client
```

Install the AssemblyAI Python SDK and a recording package via pip:

```bash
pip install assemblyai pyaudio
```

Install the required packages via NPM:

```bash
npm install ws mic
```

Install the AssemblyAI JavaScript SDK and a recording package via NPM:

```bash
npm install assemblyai node-record-lpcm16
```

## Step 2: Connect to Universal Streaming

In this step, you'll set up a connection to the Universal Streaming API with the `llm_gateway` parameter. This parameter configures LLM Gateway to process your streaming transcripts.

For more information about streaming transcription, see [Transcribe streaming audio](/streaming/getting-started/transcribe-streaming-audio).

```python expandable
import pyaudio
import websocket
import json
import threading
import time
import wave
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "<YOUR_API_KEY>"  # Replace with your actual API key

# LLM Gateway Configuration
PROMPT = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"
LLM_GATEWAY_CONFIG = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": PROMPT}
    ],
    "max_tokens": 4000
}

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,  # Request formatted final transcripts
    "speech_model": "universal-3-5-pro",
    "llm_gateway": json.dumps(LLM_GATEWAY_CONFIG)  # LLM Gateway configuration
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# WAV recording variables
recorded_frames = []  # Store audio frames for WAV file
recording_lock = threading.Lock()  # Thread-safe access to recorded_frames
```

```python expandable
import logging
from typing import Type

import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    LLMGatewayResponseEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TurnEvent,
    TerminationEvent,
)
from assemblyai.streaming.v3.models import LLMGatewayConfig, LLMGatewayMessage

api_key = "<YOUR_API_KEY>"
prompt = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()
```

```javascript expandable
const WebSocket = require("ws");
const mic = require("mic");
const querystring = require("querystring");
const fs = require("fs");

// --- Configuration ---
const YOUR_API_KEY = "<YOUR_API_KEY>"; // Replace with your actual API key

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};

const CONNECTION_PARAMS = {
  sample_rate: 16000,
  format_turns: true, // Request formatted final transcripts
  speech_model: "universal-3-5-pro",
  llm_gateway: JSON.stringify(LLM_GATEWAY_CONFIG), // LLM Gateway configuration
};
const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

// Audio Configuration
const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;
const CHANNELS = 1;

// Global variables
let micInstance = null;
let micInputStream = null;
let ws = null;
let stopRequested = false;

// WAV recording variables
let recordedFrames = []; // Store audio frames for WAV file
```

```javascript
import { AssemblyAI } from "assemblyai";
import recorder from "node-record-lpcm16";
import { Readable } from "stream";

// --- Configuration ---
const API_KEY = "<YOUR_API_KEY>";

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};
```

The `llm_gateway` parameterisa JSON-stringified object that follows the same interface as the [LLM Gateway chat completions API](/llm-gateway/chat-completions). It accepts the following fields:

| Key          | Type   | Description                                                                            |
| ------------ | ------ | -------------------------------------------------------------------------------------- |
| `model`      | string | The model to use. See [Available models](/llm-gateway/available-models). |
| `messages`   | array  | An array of message objects. The `content` field contains your prompt.                 |
| `max_tokens` | number | The maximum number of tokens to generate.                                              |

## Step 3: Stream audio and analyze with LLM Gateway

In this step, you'll stream audio from your microphone, collect the transcribed text from completed turns, and then send the accumulated transcript to LLM Gateway for analysis when the session ends.

Set up the event handlers to stream audio and collect transcripts from completed turns.

```python expandable
def save_wav_file():
    """Save recorded audio frames to a WAV file."""
    if not recorded_frames:
        print("No audio data recorded.")
        return

    # Generate filename with timestamp
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"recorded_audio_{timestamp}.wav"

    try:
        with wave.open(filename, 'wb') as wf:
            wf.setnchannels(CHANNELS)
            wf.setsampwidth(2)  # 16-bit = 2 bytes
            wf.setframerate(SAMPLE_RATE)

            # Write all recorded frames
            with recording_lock:
                wf.writeframes(b''.join(recorded_frames))

        print(f"Audio saved to: {filename}")
        print(f"Duration: {len(recorded_frames) * FRAMES_PER_BUFFER / SAMPLE_RATE:.2f} seconds")

    except Exception as e:
        print(f"Error saving WAV file: {e}")

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                # Store audio data for WAV recording
                with recording_lock:
                    recorded_frames.append(audio_data)

                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"Session started: {session_id}")
        elif msg_type == "Turn":
            end_of_turn = data.get('end_of_turn', False)

            if end_of_turn:
                transcript = data.get('transcript', '')
                print(f"\nTranscript:\n{transcript}\n")
        elif msg_type == "LLMGatewayResponse":
            # Extract the LLM response content
            llm_data = data.get('data', {})
            llm_content = llm_data.get("choices", [{}])[0].get("message", {}).get("content", "")
            print(f"LLM Response:\n{llm_content}\n")
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"Session terminated: {audio_duration} seconds of audio processed")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    # Save recorded audio to WAV file
    save_wav_file()

    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
```

```python
def on_begin(self: Type[RealTimeTranscriber], event: BeginEvent):
    print(f"Session started: {event.id}")

def on_turn(self: Type[RealTimeTranscriber], event: TurnEvent):
    if event.end_of_turn:
        print(f"\nTranscript:\n{event.transcript}\n")

def on_llm_response(self: Type[RealTimeTranscriber], event: LLMGatewayResponseEvent):
    # Extract the actual LLM response content from the data
    llm_content = event.data.get("choices", [{}])[0].get("message", {}).get("content", "")
    print(f"LLM Response:\n{llm_content}\n")

def on_terminated(self: Type[RealTimeTranscriber], event: TerminationEvent):
    print(
        f"Session terminated: {event.audio_duration_seconds} seconds of audio processed"
    )

def on_error(self: Type[RealTimeTranscriber], error: RealTimeError):
    print(f"Error occurred: {error}")
```

```javascript expandable
// --- Helper functions ---
function formatTimestamp(timestamp) {
  return new Date(timestamp * 1000).toISOString();
}

function createWavHeader(sampleRate, channels, dataLength) {
  const buffer = Buffer.alloc(44);

  // RIFF header
  buffer.write("RIFF", 0);
  buffer.writeUInt32LE(36 + dataLength, 4);
  buffer.write("WAVE", 8);

  // fmt chunk
  buffer.write("fmt ", 12);
  buffer.writeUInt32LE(16, 16); // fmt chunk size
  buffer.writeUInt16LE(1, 20); // PCM format
  buffer.writeUInt16LE(channels, 22);
  buffer.writeUInt32LE(sampleRate, 24);
  buffer.writeUInt32LE(sampleRate * channels * 2, 28); // byte rate
  buffer.writeUInt16LE(channels * 2, 32); // block align
  buffer.writeUInt16LE(16, 34); // bits per sample

  // data chunk
  buffer.write("data", 36);
  buffer.writeUInt32LE(dataLength, 40);

  return buffer;
}

function saveWavFile() {
  if (recordedFrames.length === 0) {
    console.log("No audio data recorded.");
    return;
  }

  // Generate filename with timestamp
  const timestamp = new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19);
  const filename = `recorded_audio_${timestamp}.wav`;

  try {
    // Combine all recorded frames
    const audioData = Buffer.concat(recordedFrames);
    const dataLength = audioData.length;

    // Create WAV header
    const wavHeader = createWavHeader(SAMPLE_RATE, CHANNELS, dataLength);

    // Write WAV file
    const wavFile = Buffer.concat([wavHeader, audioData]);
    fs.writeFileSync(filename, wavFile);

    console.log(`Audio saved to: ${filename}`);
    console.log(
      `Duration: ${(dataLength / (SAMPLE_RATE * CHANNELS * 2)).toFixed(2)} seconds`
    );
  } catch (error) {
    console.error(`Error saving WAV file: ${error}`);
  }
}

function startMicrophone() {
  try {
    micInstance = mic({
      rate: SAMPLE_RATE.toString(),
      channels: CHANNELS.toString(),
      debug: false,
      exitOnSilence: 6, // This won't actually exit, just a parameter for mic
    });

    micInputStream = micInstance.getAudioStream();

    micInputStream.on("data", (data) => {
      if (ws && ws.readyState === WebSocket.OPEN && !stopRequested) {
        // Store audio data for WAV recording
        recordedFrames.push(Buffer.from(data));

        // Send audio data to WebSocket
        ws.send(data);
      }
    });

    micInputStream.on("error", (err) => {
      console.error(`Microphone Error: ${err}`);
      cleanup();
    });

    micInstance.start();
    console.log("Microphone stream opened successfully.");
    console.log("Speak into your microphone. Press Ctrl+C to stop.");
  } catch (error) {
    console.error(`Error opening microphone stream: ${error}`);
    cleanup();
  }
}
```

```javascript expandable
// Session started event
transcriber.on("open", ({ id }) => {
  console.log(`Session started: ${id}`);
});

// Error event
transcriber.on("error", (error) => {
  console.error(`\nError: ${error}`);
});

// Close event
transcriber.on("close", (code, reason) => {
  console.log("Session closed:", code, reason || "");
});

// Turn event - displays transcript when turn ends
transcriber.on("turn", (turn) => {
  if (turn.end_of_turn) {
    console.log(`\nTranscript:\n${turn.transcript}\n`);
  }
});

// LLM Gateway Response event - displays LLM summary
transcriber.on("llmGatewayResponse", (response) => {
  const llmContent = response.data?.choices?.[0]?.message?.content || "";
  console.log(`\nLLM Response:\n${llmContent}\n`);
});

// Termination event
transcriber.on("termination", (event) => {
  console.log(
    `Session terminated: ${event.audio_duration_seconds} seconds of audio processed`
  );
});
```

Define a function to send the accumulated transcript to LLM Gateway for analysis. This function uses the [LLM Gateway chat completions API](/llm-gateway/chat-completions) to process the transcript with your prompt.

When using the raw WebSocket approach with `llm_gateway` in the connection parameters, LLM Gateway responses are received as `LLMGatewayResponse` messages through the WebSocket, handled by the `on_message` callback registered in the previous step. No separate API call is needed.

When using the Python SDK with `LLMGatewayConfig`, analysis responses are received automatically through the `LLMGatewayResponseEvent` event handler registered in the previous step. No separate API call is needed.

When using the raw WebSocket approach with `llm_gateway` in the connection parameters, LLM Gateway responses are received as `LLMGatewayResponse` messages through the WebSocket, handled by the `on_message` callback in the `run()` function. No separate API call is needed.

When using the JavaScript SDK with `llmGateway` in the transcriber options, LLM Gateway responses are received automatically through the `llmGatewayResponse` event handler registered in the previous step. No separate API call is needed.

Run the streaming session and analyze the transcript with LLM Gateway when the session ends.

```python expandable
# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
        print("Audio will be saved to a WAV file when the session ends.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

```python expandable
def main():
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(
            api_host="streaming.assemblyai.com",
        ),
        api_key=api_key,
    )

    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.LLMGatewayResponse, on_llm_response)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    client.connect(
        RealTimeParameters(
            sample_rate=16000,
            speech_model="universal-3-5-pro",
            format_turns=True,
            llm_gateway=LLMGatewayConfig(
                model="claude-sonnet-4-6",
                messages=[
                    LLMGatewayMessage(role="user", content=prompt)
                ],
                max_tokens=4000
            )
        )
    )

    try:
        client.stream(microphone_stream())
    finally:
        client.disconnect(terminate=True)

if __name__ == "__main__":
    main()
```

```javascript expandable
// --- Main function ---
async function run() {
  console.log(
    "Starting AssemblyAI streaming transcription with LLM Gateway..."
  );
  console.log("Audio will be saved to a WAV file when the session ends.");

  // Initialize WebSocket connection
  ws = new WebSocket(API_ENDPOINT, {
    headers: {
      Authorization: YOUR_API_KEY,
    },
  });

  // Setup WebSocket event handlers
  ws.on("open", () => {
    console.log("WebSocket connection opened.");
    console.log(`Connected to: ${API_ENDPOINT_BASE_URL}`);
    // Start the microphone
    startMicrophone();
  });

  ws.on("message", (message) => {
    try {
      const data = JSON.parse(message);
      const msgType = data.type;

      if (msgType === "Begin") {
        const sessionId = data.id;
        console.log(`Session started: ${sessionId}`);
      } else if (msgType === "Turn") {
        const endOfTurn = data.end_of_turn;

        if (endOfTurn) {
          const transcript = data.transcript || "";
          console.log(`\nTranscript:\n${transcript}\n`);
        }
      } else if (msgType === "LLMGatewayResponse") {
        // Extract the LLM response content
        const llmData = data.data || {};
        const llmContent = llmData.choices?.[0]?.message?.content || "";
        console.log(`LLM Response:\n${llmContent}\n`);
      } else if (msgType === "Termination") {
        const audioDuration = data.audio_duration_seconds;
        console.log(
          `Session terminated: ${audioDuration} seconds of audio processed`
        );
      }
    } catch (error) {
      console.error(`\nError handling message: ${error}`);
      console.error(`Message data: ${message}`);
    }
  });

  ws.on("error", (error) => {
    console.error(`\nWebSocket Error: ${error}`);
    cleanup();
  });

  ws.on("close", (code, reason) => {
    console.log(`\nWebSocket Disconnected: Status=${code}, Msg=${reason}`);
    cleanup();
  });

  // Handle process termination
  setupTerminationHandlers();
}

function cleanup() {
  stopRequested = true;

  // Save recorded audio to WAV file
  saveWavFile();

  // Stop microphone if it's running
  if (micInstance) {
    try {
      micInstance.stop();
    } catch (error) {
      console.error(`Error stopping microphone: ${error}`);
    }
    micInstance = null;
  }

  // Close WebSocket connection if it's open
  if (ws && [WebSocket.OPEN, WebSocket.CONNECTING].includes(ws.readyState)) {
    try {
      // Send termination message if possible
      if (ws.readyState === WebSocket.OPEN) {
        const terminateMessage = { type: "Terminate" };
        console.log(
          `Sending termination message: ${JSON.stringify(terminateMessage)}`
        );
        ws.send(JSON.stringify(terminateMessage));
      }
      ws.close();
    } catch (error) {
      console.error(`Error closing WebSocket: ${error}`);
    }
    ws = null;
  }

  console.log("Cleanup complete.");
}

function setupTerminationHandlers() {
  // Handle Ctrl+C and other termination signals
  process.on("SIGINT", () => {
    console.log("\nCtrl+C received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  process.on("SIGTERM", () => {
    console.log("\nTermination signal received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  // Handle uncaught exceptions
  process.on("uncaughtException", (error) => {
    console.error(`\nUncaught exception: ${error}`);
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(1), 1000);
  });
}

// Start the application
run();
```

```javascript expandable
const run = async () => {
  const client = new AssemblyAI({
    apiKey: API_KEY,
  });

  const transcriber = client.streaming.transcriber({
    sampleRate: 16000,
    formatTurns: true,
    speechModel: "universal-3-5-pro",
    llmGateway: LLM_GATEWAY_CONFIG,
  });

  // ... (event handlers from Step 1)

  try {
    await transcriber.connect();

    console.log("Connecting to streaming transcript service");
    console.log("Starting recording");

    const recording = recorder.record({
      channels: 1,
      sampleRate: 16000,
      audioType: "wav", // Linear PCM
    });

    Readable.toWeb(recording.stream()).pipeTo(transcriber.stream());

    // Stop recording and close connection using Ctrl-C
    process.on("SIGINT", async function () {
      console.log();
      console.log("Stopping recording");
      recording.stop();

      console.log("Closing streaming transcript connection");
      await transcriber.close();

      process.exit();
    });
  } catch (error) {
    console.error(`Error: ${error}`);
  }
};

run();
```

The output will look something like this:

```
Session started: de5d9927-73a6-4be8-b52d-b4c07be37e6b

Transcript: Hi, my name is Sonny.

Transcript: I am a voice agent.

Stopping...

Session terminated: 12s of audio processed

Analyzing conversation with LLM Gateway...
The speaker introduces themselves as Sonny and identifies as a voice agent.
```

  Want to make your LLM requests more resilient? Use [fallback models](/llm-gateway/fallback) to automatically switch to a backup model if your primary model is unavailable.

## Next steps

In this tutorial, you've learned how to analyze streaming audio transcripts using LLM Gateway. The type of output depends on your prompt, so try exploring different prompts to see how they affect the output. Here are a few more prompts to try:

- "Provide an analysis of the transcript and offer areas to improve with exact quotes."
- "What's the main take-away from the transcript?"
- "Generate a set of action items from this transcript."

To learn more about LLM Gateway and streaming, see the following resources:

- [LLM Gateway Overview](/llm-gateway/quickstart)
- [Basic Chat Completions](/llm-gateway/chat-completions)
- [Model Selection](/streaming)
- [Transcribe streaming audio](/streaming/getting-started/transcribe-streaming-audio)

## Need some help?

If you get stuck, or have any other questions, we'd love to help you out. Contact our support team at support@assemblyai.com or create a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="translate-real-time-stt-transcripts-with-llm-gateway-docs-streaming-guides-real-time-translation-mdx"></a>

# Translate Real-time STT Transcripts with LLM Gateway

URL: https://www.assemblyai.com/docs/streaming/guides/real_time_translation
Source: docs/streaming/guides/real_time_translation.mdx
Navigation: Real-time STT > Guides > Tutorials > Streaming with LLM Gateway
Description: Translate Real-time STT Transcripts with LLM Gateway documentation.

In this guide, you'll learn how to implement real-time translation of final transcripts using AssemblyAI's Streaming API and LLM Gateway.

## Quickstart

    

    ```python expandable
    import pyaudio
    import websocket
    import json
    import threading
    import time
    import requests
    from urllib.parse import urlencode

    YOUR_API_KEY = "YOUR_API_KEY"  # Replace with your actual API key

    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "speech_model": "universal-3-5-pro",
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    FRAMES_PER_BUFFER = 800
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()

    def translate_text(text):
        """Called when translating final transcripts."""
        headers = {
            "authorization": YOUR_API_KEY
        }

        llm_gateway_data = {
            "model": "gemini-2.5-flash-lite",
            "messages": [
                {"role": "user", "content": f"Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: {text}"}
            ],
            "max_tokens": 1000
        }

        result = requests.post(
            "https://llm-gateway.assemblyai.com/v1/chat/completions",
            headers=headers,
            json=llm_gateway_data
        )
        return result.json()["choices"][0]["message"]["content"]

    def on_open(ws):
        print("WebSocket connection opened.")
        def stream_audio():
            global stream
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    break

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = True
        audio_thread.start()

    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get("type")

            if msg_type == "Begin":
                print(f"Session began: ID={data.get('id')}")
            elif msg_type == "Turn":
                transcript = data.get("transcript", "")
                if data.get("end_of_turn"):
                    print(f"\r{' ' * 80}\r", end="")
                    print(translate_text(transcript))
                else:
                    print(f"\r{transcript}", end="")
            elif msg_type == "Termination":
                print(f"\nSession terminated: {data.get('audio_duration_seconds', 0)}s of audio")
        except Exception as e:
            print(f"Error handling message: {e}")

    def on_error(ws, error):
        print(f"\nWebSocket Error: {error}")
        stop_event.set()

    def on_close(ws, close_status_code, close_msg):
        print(f"\nWebSocket Disconnected: Status={close_status_code}")
        global stream, audio
        stop_event.set()
        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
        if audio:
            audio.terminate()

    def run():
        global audio, stream, ws_app

        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Speak into your microphone. Press Ctrl+C to stop.")

        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nStopping...")
            stop_event.set()
            if ws_app and ws_app.sock and ws_app.sock.connected:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

    if __name__ == "__main__":
        run()
    ```

    
    

    ```javascript expandable
    import WebSocket from "ws";
    import mic from "mic";

    const YOUR_API_KEY = "YOUR_API_KEY";
    const CONNECTION_PARAMS = {
      sample_rate: 16000,
      speech_model: "universal-3-5-pro",
    };
    const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
    const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${new URLSearchParams(CONNECTION_PARAMS).toString()}`;
    const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;

    let micInstance = null;
    let ws = null;

    async function translateText(text) {
      const response = await fetch(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        {
          method: "POST",
          headers: {
            Authorization: YOUR_API_KEY,
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
            model: "gemini-2.5-flash-lite",
            messages: [
              { role: "user", content: `Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: ${text}` },
            ],
            max_tokens: 1000,
          }),
        }
      );
      const data = await response.json();
      return data.choices[0].message.content;
    }

    function run() {
      ws = new WebSocket(API_ENDPOINT, {
        headers: { Authorization: YOUR_API_KEY },
      });

      ws.on("open", () => {
        console.log("WebSocket connection opened.");

        micInstance = mic({
          rate: String(SAMPLE_RATE),
          channels: "1",
          bitwidth: "16",
          encoding: "signed-integer",
          endian: "little",
        });

        const micInputStream = micInstance.getAudioStream();
        micInputStream.on("data", (data) => {
          if (ws.readyState === WebSocket.OPEN) {
            ws.send(data);
          }
        });

        micInstance.start();
        console.log("Speak into your microphone. Press Ctrl+C to stop.");
      });

      ws.on("message", async (data) => {
        try {
          const msg = JSON.parse(data);
          if (msg.type === "Begin") {
            console.log(`Session began: ID=${msg.id}`);
          } else if (msg.type === "Turn") {
            const transcript = msg.transcript || "";
            if (msg.end_of_turn) {
              process.stdout.write("\r" + " ".repeat(80) + "\r");
              const translated = await translateText(transcript);
              console.log(translated);
            } else {
              process.stdout.write(`\r${transcript}`);
            }
          } else if (msg.type === "Termination") {
            console.log(
              `\nSession terminated: ${msg.audio_duration_seconds}s of audio`
            );
          }
        } catch (e) {
          console.error("Error handling message:", e);
        }
      });

      ws.on("error", (error) => {
        console.error("WebSocket error:", error);
      });

      ws.on("close", (code) => {
        console.log(`WebSocket closed: ${code}`);
        if (micInstance) micInstance.stop();
      });

      process.on("SIGINT", () => {
        console.log("\nStopping...");
        if (micInstance) micInstance.stop();
        if (ws && ws.readyState === WebSocket.OPEN) {
          ws.send(JSON.stringify({ type: "Terminate" }));
          setTimeout(() => {
            ws.close();
            process.exit(0);
          }, 2000);
        } else {
          process.exit(0);
        }
      });
    }

    run();
    ```

    

## Step-by-Step Instructions

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) and get your API key from your dashboard.

### Install Dependencies

    

    ```bash
    pip install websocket-client pyaudio requests
    ```

    
    

    ```bash
    npm install ws mic
    ```

    

### Import Packages & Set API Key

    

    ```python
    import pyaudio
    import websocket
    import json
    import threading
    import time
    import requests
    from urllib.parse import urlencode

    YOUR_API_KEY = "YOUR_API_KEY"  # Replace with your actual API key
    ```

    
    

    ```javascript
    import WebSocket from "ws";
    import mic from "mic";

    const YOUR_API_KEY = "YOUR_API_KEY";
    ```

    

### Audio Configuration & Global Variables

Set all of your audio configurations and global variables.

    

    ```python
    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "speech_model": "universal-3-5-pro",
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    FRAMES_PER_BUFFER = 800
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()
    ```

    
    

    ```javascript
    const CONNECTION_PARAMS = {
      sample_rate: 16000,
      speech_model: "universal-3-5-pro",
    };
    const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
    const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${new URLSearchParams(CONNECTION_PARAMS).toString()}`;
    const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;

    let micInstance = null;
    let ws = null;
    ```

    

### Define Translate Text Function

Define a function called `translate_text` (Python) or `translateText` (JavaScript), which uses LLM Gateway to translate the English final transcripts into another language. This example is translating the text into Spanish. To set this to a different language, just replace "Spanish" in the prompt with your language of choice.

    

    ```python
    def translate_text(text):
        """Called when translating final transcripts."""
        headers = {
            "authorization": YOUR_API_KEY
        }

        llm_gateway_data = {
            "model": "gemini-2.5-flash-lite",
            "messages": [
                {"role": "user", "content": f"Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: {text}"}
            ],
            "max_tokens": 1000
        }

        result = requests.post(
            "https://llm-gateway.assemblyai.com/v1/chat/completions",
            headers=headers,
            json=llm_gateway_data
        )
        return result.json()["choices"][0]["message"]["content"]
    ```

    
    

    ```javascript expandable
    async function translateText(text) {
      const response = await fetch(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        {
          method: "POST",
          headers: {
            Authorization: YOUR_API_KEY,
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
            model: "gemini-2.5-flash-lite",
            messages: [
              { role: "user", content: `Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: ${text}` },
            ],
            max_tokens: 1000,
          }),
        }
      );
      const data = await response.json();
      return data.choices[0].message.content;
    }
    ```

    

### Websocket Event Handlers

#### Open Websocket

    

    ```python
    def on_open(ws):
        print("WebSocket connection opened.")
        def stream_audio():
            global stream
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    break

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = True
        audio_thread.start()
    ```

    
    

    ```javascript expandable
    ws.on("open", () => {
      console.log("WebSocket connection opened.");

      micInstance = mic({
        rate: String(SAMPLE_RATE),
        channels: "1",
        bitwidth: "16",
        encoding: "signed-integer",
        endian: "little",
      });

      const micInputStream = micInstance.getAudioStream();
      micInputStream.on("data", (data) => {
        if (ws.readyState === WebSocket.OPEN) {
          ws.send(data);
        }
      });

      micInstance.start();
      console.log("Speak into your microphone. Press Ctrl+C to stop.");
    });
    ```

    

#### Handle Websocket Messages

In this function, use the previously defined `translate_text` / `translateText` to translate all final transcripts.

    

    ```python {10-12}
    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get("type")

            if msg_type == "Begin":
                print(f"Session began: ID={data.get('id')}")
            elif msg_type == "Turn":
                transcript = data.get("transcript", "")
                if data.get("end_of_turn"):
                    print(f"\r{' ' * 80}\r", end="")
                    print(translate_text(transcript))
                else:
                    print(f"\r{transcript}", end="")
            elif msg_type == "Termination":
                print(f"\nSession terminated: {data.get('audio_duration_seconds', 0)}s of audio")
        except Exception as e:
            print(f"Error handling message: {e}")
    ```

    
    

    ```javascript {8-11} expandable
    ws.on("message", async (data) => {
      try {
        const msg = JSON.parse(data);
        if (msg.type === "Begin") {
          console.log(`Session began: ID=${msg.id}`);
        } else if (msg.type === "Turn") {
          const transcript = msg.transcript || "";
          if (msg.end_of_turn) {
            process.stdout.write("\r" + " ".repeat(80) + "\r");
            const translated = await translateText(transcript);
            console.log(translated);
          } else {
            process.stdout.write(`\r${transcript}`);
          }
        } else if (msg.type === "Termination") {
          console.log(
            `\nSession terminated: ${msg.audio_duration_seconds}s of audio`
          );
        }
      } catch (e) {
        console.error("Error handling message:", e);
      }
    });
    ```

    

#### Close Websocket

    

    ```python
    def on_close(ws, close_status_code, close_msg):
        print(f"\nWebSocket Disconnected: Status={close_status_code}")
        global stream, audio
        stop_event.set()
        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
        if audio:
            audio.terminate()
    ```

    
    

    ```javascript
    ws.on("close", (code) => {
      console.log(`WebSocket closed: ${code}`);
      if (micInstance) micInstance.stop();
    });
    ```

    

#### Websocket Error Handling

    

    ```python
    def on_error(ws, error):
        print(f"\nWebSocket Error: {error}")
        stop_event.set()
    ```

    
    

    ```javascript
    ws.on("error", (error) => {
      console.error("WebSocket error:", error);
    });
    ```

    

### Begin Real-time STT Transcription

    

    ```python expandable
    def run():
        global audio, stream, ws_app

        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Speak into your microphone. Press Ctrl+C to stop.")

        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nStopping...")
            stop_event.set()
            if ws_app and ws_app.sock and ws_app.sock.connected:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

    if __name__ == "__main__":
        run()
    ```

    
    

    ```javascript expandable
    function run() {
      ws = new WebSocket(API_ENDPOINT, {
        headers: { Authorization: YOUR_API_KEY },
      });

      // ... event handlers defined above ...

      process.on("SIGINT", () => {
        console.log("\nStopping...");
        if (micInstance) micInstance.stop();
        if (ws && ws.readyState === WebSocket.OPEN) {
          ws.send(JSON.stringify({ type: "Terminate" }));
          setTimeout(() => {
            ws.close();
            process.exit(0);
          }, 2000);
        } else {
          process.exit(0);
        }
      });
    }

    run();
    ```

---

<a id="apply-noise-reduction-to-audio-for-streaming-speech-to-text-docs-streaming-guides-noise-reduction-streaming-mdx"></a>

# Apply Noise Reduction to Audio for Streaming Speech-to-Text

URL: https://www.assemblyai.com/docs/streaming/guides/noise_reduction_streaming
Source: docs/streaming/guides/noise_reduction_streaming.mdx
Navigation: Real-time STT > Guides > Tutorials > Use-case specific streaming workflows
Description: Apply Noise Reduction to Audio for Streaming Speech-to-Text documentation.

This guide demonstrates how to implement a noise reduction system for real-time audio transcription using AssemblyAI's Real-time STT and the `noisereduce` library. You'll learn how to create a custom audio pipeline that preprocesses incoming audio to remove background noise before it reaches the transcription service.

This solution is particularly valuable for:

- Voice assistants operating in noisy environments
- Customer service applications processing calls
- Meeting transcription tools
- Voice-enabled applications requiring high accuracy

The implementation uses Python and combines proven audio processing techniques with AssemblyAI's powerful transcription capabilities. While our example focuses on microphone input, the principles can be applied to any real-time audio stream.

## Quickstart

```python expandable
import pyaudio
import websocket
import json
import os
import threading
import time
import numpy as np
import noisereduce as nr
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": 16000,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Noise reduction configuration
NOISE_BUFFER_SECONDS = 0.5
NOISE_BUFFER_SIZE = int(SAMPLE_RATE * NOISE_BUFFER_SECONDS)

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    def stream_audio():
        global stream
        print("Starting audio streaming with noise reduction...")
        buffer = np.array([], dtype=np.int16)
        overlap = 1024
        has_overlap = False

        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                audio_array = np.frombuffer(audio_data, dtype=np.int16)
                buffer = np.append(buffer, audio_array)

                if len(buffer) >= NOISE_BUFFER_SIZE:
                    # Apply noise reduction
                    float_audio = buffer.astype(np.float32) / 32768.0
                    denoised = nr.reduce_noise(
                        y=float_audio,
                        sr=SAMPLE_RATE,
                        prop_decrease=0.75,
                        n_fft=1024,
                    )
                    int_audio = (denoised * 32768.0).astype(np.int16)

                    # Send only the non-overlapping portion to avoid duplicate audio
                    if has_overlap:
                        ws.send(int_audio[overlap:].tobytes(), websocket.ABNF.OPCODE_BINARY)
                    else:
                        ws.send(int_audio.tobytes(), websocket.ABNF.OPCODE_BINARY)
                        has_overlap = True

                    # Keep some overlap for continuity
                    buffer = buffer[-overlap:]
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
        print("Audio will be noise-reduced before transcription.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": ASSEMBLYAI_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## Step-by-step guide

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard. Please note that Streaming Speech-to-text is available for upgraded accounts only. If you're on the free plan, you'll need to upgrade your account by adding a credit card.

### Install and import packages

Install the required packages:

```bash
pip install websocket-client pyaudio noisereduce numpy
```

Import packages and set your API key.

```python
import pyaudio
import websocket
import json
import os
import threading
import time
import numpy as np
import noisereduce as nr
from urllib.parse import urlencode
from datetime import datetime

ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
```

**Make sure not to share this token with anyone** - it is a private key associated uniquely to your account.

### Audio configuration and global variables

Set all of your audio configurations and global variables. The `NOISE_BUFFER_SIZE` controls how much audio is buffered before applying noise reduction — 0.5 seconds provides a good balance between latency and noise reduction quality.

```python expandable
CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": 16000,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Noise reduction configuration
NOISE_BUFFER_SECONDS = 0.5
NOISE_BUFFER_SIZE = int(SAMPLE_RATE * NOISE_BUFFER_SECONDS)

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
```

### WebSocket event handlers

#### Open WebSocket

When the connection opens, we start a background thread that reads audio from the microphone, buffers it, applies noise reduction using `noisereduce`, and sends the denoised audio to AssemblyAI.

The noise reduction works by:
1. Accumulating raw audio samples into a buffer
2. Once the buffer reaches 0.5 seconds, converting to float and applying `nr.reduce_noise()`
3. Converting back to int16 and sending over the WebSocket
4. Keeping the last 1024 samples as overlap for continuity, and only sending the non-overlapping portion to avoid duplicate audio

```python expandable
def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    def stream_audio():
        global stream
        print("Starting audio streaming with noise reduction...")
        buffer = np.array([], dtype=np.int16)
        overlap = 1024
        has_overlap = False

        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                audio_array = np.frombuffer(audio_data, dtype=np.int16)
                buffer = np.append(buffer, audio_array)

                if len(buffer) >= NOISE_BUFFER_SIZE:
                    # Apply noise reduction
                    float_audio = buffer.astype(np.float32) / 32768.0
                    denoised = nr.reduce_noise(
                        y=float_audio,
                        sr=SAMPLE_RATE,
                        prop_decrease=0.75,
                        n_fft=1024,
                    )
                    int_audio = (denoised * 32768.0).astype(np.int16)

                    # Send only the non-overlapping portion to avoid duplicate audio
                    if has_overlap:
                        ws.send(int_audio[overlap:].tobytes(), websocket.ABNF.OPCODE_BINARY)
                    else:
                        ws.send(int_audio.tobytes(), websocket.ABNF.OPCODE_BINARY)
                        has_overlap = True

                    # Keep some overlap for continuity
                    buffer = buffer[-overlap:]
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()
```

#### Handle WebSocket messages

```python expandable
def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}, ExpiresAt={datetime.fromtimestamp(expires_at)}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(transcript)
            else:
                print(f"\r{transcript}", end='')
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio Duration={audio_duration}s, Session Duration={session_duration}s")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")
```

#### WebSocket error and close handlers

```python expandable
def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
```

### Begin real-time STT transcription

Open the microphone, connect the WebSocket, and start streaming noise-reduced audio.

```python expandable
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
        print("Audio will be noise-reduced before transcription.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": ASSEMBLYAI_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

You can press Ctrl+C to stop the transcription.

---

<a id="transcribe-audio-files-with-streaming-docs-streaming-guides-streaming-transcribe-audio-file-mdx"></a>

# Transcribe audio files with Streaming

URL: https://www.assemblyai.com/docs/streaming/guides/streaming_transcribe_audio_file
Source: docs/streaming/guides/streaming_transcribe_audio_file.mdx
Navigation: Real-time STT > Guides > Tutorials > Use-case specific streaming workflows
Description: Transcribe audio files with Streaming documentation.

This guide shows you how to transcribe **WAV audio files** with varying sample rates using our Streaming API.

If you're streaming a pre-recorded file for benchmarking or testing, see [Stream a pre-recorded file in real time](/streaming/guides/stream_prerecorded_file_realtime) for wall-clock pacing that more closely simulates live microphone input.

## Quickstart

Here is the complete Python script to transcribe a WAV audio file using the Streaming API.

```python expandable
import websocket
import json
import threading
import time
import wave
import sys
import os
from urllib.parse import urlencode
from pathlib import Path

# --- Configuration ---
ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AUDIO_FILE = "audio.wav"  # Path to your audio file
SAMPLE_RATE = 48000  # Change to match the sample rate of your audio file
SAVE_TRANSCRIPT_TO_FILE = True  # Set to False to disable saving transcript to file
PLAY_AUDIO = True  # Set to False to disable audio playback

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": SAMPLE_RATE,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Global variables
ws_app = None
audio_thread = None
stop_event = threading.Event()

# Track session data for output file
session_data = {
    "session_id": None,
    "audio_file": AUDIO_FILE,
    "audio_duration_seconds": None,
    "turns": []
}

# --- Helper Functions ---

def validate_audio_file(filepath, sample_rate):
    """Validate audio file before streaming."""
    file_ext = Path(filepath).suffix.lower()
    if file_ext != ".wav":
        print(f"Error: Only WAV files are supported. Got: {file_ext}", file=sys.stderr)
        print(f"Convert your file to WAV using: ffmpeg -i {filepath} -ar {sample_rate} -ac 1 output.wav", file=sys.stderr)
        sys.exit(1)

    with wave.open(filepath, 'rb') as wav_file:
        if wav_file.getnchannels() != 1:
            print("Error: Only mono audio is supported", file=sys.stderr)
            print(f"Convert your file to mono using: ffmpeg -i {filepath} -ar {sample_rate} -ac 1 output.wav", file=sys.stderr)
            sys.exit(1)

        file_sample_rate = wav_file.getframerate()
        if file_sample_rate != sample_rate:
            print(f"Error: File sample rate ({file_sample_rate}) doesn't match expected rate ({sample_rate})", file=sys.stderr)
            print(f"Either update SAMPLE_RATE to {file_sample_rate}, or convert your file using: ffmpeg -i {filepath} -ar {sample_rate} -ac 1 output.wav", file=sys.stderr)
            sys.exit(1)

def save_transcript():
    """Save the transcript to a file in the same directory as the script."""
    audio_name = Path(session_data["audio_file"]).stem
    session_id = session_data["session_id"] or "unknown"
    output_file = f"{audio_name}_{session_id}.txt"

    with open(output_file, "w") as f:
        f.write(f"AssemblyAI Session ID: {session_data['session_id']}\n")
        f.write(f"Audio file: {session_data['audio_file']}\n")
        f.write(f"Audio duration: {session_data['audio_duration_seconds']} seconds\n")
        f.write("See all available parameters and defaults at https://www.assemblyai.com/docs/streaming/api-spec/streaming-websocket#request.query\n\n")
        f.write("\nTranscription Output\n")
        for i, turn in enumerate(session_data["turns"], 1):
            f.write(f"[Turn #{i}]: {turn}\n")

    print(f"Transcript saved to {output_file}")

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    def stream_file():
        chunk_duration = 0.05  # 50ms chunks
        audio_player = None

        if PLAY_AUDIO:
            try:
                import pyaudio
                p = pyaudio.PyAudio()
                with wave.open(AUDIO_FILE, 'rb') as wav_file:
                    audio_player = p.open(
                        format=p.get_format_from_width(wav_file.getsampwidth()),
                        channels=wav_file.getnchannels(),
                        rate=wav_file.getframerate(),
                        output=True
                    )
            except ImportError:
                print("Warning: pyaudio not installed. Audio playback disabled.", file=sys.stderr)
                print("Install with: pip install pyaudio", file=sys.stderr)

        try:
            with wave.open(AUDIO_FILE, 'rb') as wav_file:
                frames_per_chunk = int(SAMPLE_RATE * chunk_duration)

                while not stop_event.is_set():
                    frames = wav_file.readframes(frames_per_chunk)
                    if not frames:
                        break

                    if audio_player:
                        audio_player.write(frames)
                    else:
                        time.sleep(chunk_duration)

                    ws.send(frames, websocket.ABNF.OPCODE_BINARY)
        finally:
            if audio_player:
                audio_player.stop_stream()
                audio_player.close()
                p.terminate()

        # All audio sent - terminate the session
        print("File streaming complete. Waiting for final transcripts...")
        try:
            ws.send(json.dumps({"type": "Terminate"}))
        except Exception:
            pass

    global audio_thread
    audio_thread = threading.Thread(target=stream_file)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_data["session_id"] = data.get('id')
            print(f"Session ID: {data.get('id')}\n")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if not transcript:
                return

            if data.get('end_of_turn'):
                print(f"[Final]: {transcript}\n")
                session_data["turns"].append(transcript)
            else:
                print(f"[Partial]: {transcript}")
        elif msg_type == "Termination":
            session_data["audio_duration_seconds"] = data.get('audio_duration_seconds', 0)
            print(f"Session terminated: {data.get('audio_duration_seconds', 0)} seconds of audio processed")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    stop_event.set()

    if SAVE_TRANSCRIPT_TO_FILE:
        save_transcript()

    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global ws_app

    # Validate audio file before connecting
    validate_audio_file(AUDIO_FILE, SAMPLE_RATE)

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": ASSEMBLYAI_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

## Step-by-step guide

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home).

### Install and import packages

Install the required packages. PyAudio is optional — only needed for audio playback during streaming.

```bash
pip install websocket-client
pip install pyaudio
```

Import packages.

```python
import websocket
import json
import os
import threading
import time
import wave
import sys
from urllib.parse import urlencode
from pathlib import Path
```

### Configure settings

Set your `ASSEMBLYAI_API_KEY` environment variable.

Set `AUDIO_FILE` to the relative or absolute path of your audio file, and set `SAMPLE_RATE` to match your file's sample rate.

```python expandable
ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AUDIO_FILE = "audio.wav"  # Path to your audio file
SAMPLE_RATE = 48000  # Change to match the sample rate of your audio file
SAVE_TRANSCRIPT_TO_FILE = True  # Set to False to disable saving transcript to file
PLAY_AUDIO = True  # Set to False to disable audio playback

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": SAMPLE_RATE,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Global variables
ws_app = None
audio_thread = None
stop_event = threading.Event()

# Track session data for output file
session_data = {
    "session_id": None,
    "audio_file": AUDIO_FILE,
    "audio_duration_seconds": None,
    "turns": []
}
```

### Helper functions

The following helper functions are used to validate audio files and save the transcript output:

- `validate_audio_file()` - Validates that the audio file is a mono WAV file with the expected sample rate.
- `save_transcript()` - Saves the transcript to a text file after the session ends.

```python expandable
def validate_audio_file(filepath, sample_rate):
    """Validate audio file before streaming."""
    file_ext = Path(filepath).suffix.lower()
    if file_ext != ".wav":
        print(f"Error: Only WAV files are supported. Got: {file_ext}", file=sys.stderr)
        print(f"Convert your file to WAV using: ffmpeg -i {filepath} -ar {sample_rate} -ac 1 output.wav", file=sys.stderr)
        sys.exit(1)

    with wave.open(filepath, 'rb') as wav_file:
        if wav_file.getnchannels() != 1:
            print("Error: Only mono audio is supported", file=sys.stderr)
            print(f"Convert your file to mono using: ffmpeg -i {filepath} -ar {sample_rate} -ac 1 output.wav", file=sys.stderr)
            sys.exit(1)

        file_sample_rate = wav_file.getframerate()
        if file_sample_rate != sample_rate:
            print(f"Error: File sample rate ({file_sample_rate}) doesn't match expected rate ({sample_rate})", file=sys.stderr)
            print(f"Either update SAMPLE_RATE to {file_sample_rate}, or convert your file using: ffmpeg -i {filepath} -ar {sample_rate} -ac 1 output.wav", file=sys.stderr)
            sys.exit(1)

def save_transcript():
    """Save the transcript to a file in the same directory as the script."""
    audio_name = Path(session_data["audio_file"]).stem
    session_id = session_data["session_id"] or "unknown"
    output_file = f"{audio_name}_{session_id}.txt"

    with open(output_file, "w") as f:
        f.write(f"AssemblyAI Session ID: {session_data['session_id']}\n")
        f.write(f"Audio file: {session_data['audio_file']}\n")
        f.write(f"Audio duration: {session_data['audio_duration_seconds']} seconds\n")
        f.write("See all available parameters and defaults at https://www.assemblyai.com/docs/streaming/api-spec/streaming-websocket#request.query\n\n")
        f.write("\nTranscription Output\n")
        for i, turn in enumerate(session_data["turns"], 1):
            f.write(f"[Turn #{i}]: {turn}\n")

    print(f"Transcript saved to {output_file}")
```

### WebSocket event handlers

#### Open WebSocket and stream audio file

When the connection opens, we start a background thread that reads the WAV file in 50ms chunks and sends them over the WebSocket. If `PLAY_AUDIO` is enabled, the audio is also played through your speakers.

```python expandable
def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT}")

    def stream_file():
        chunk_duration = 0.05  # 50ms chunks
        audio_player = None

        if PLAY_AUDIO:
            try:
                import pyaudio
                p = pyaudio.PyAudio()
                with wave.open(AUDIO_FILE, 'rb') as wav_file:
                    audio_player = p.open(
                        format=p.get_format_from_width(wav_file.getsampwidth()),
                        channels=wav_file.getnchannels(),
                        rate=wav_file.getframerate(),
                        output=True
                    )
            except ImportError:
                print("Warning: pyaudio not installed. Audio playback disabled.", file=sys.stderr)
                print("Install with: pip install pyaudio", file=sys.stderr)

        try:
            with wave.open(AUDIO_FILE, 'rb') as wav_file:
                frames_per_chunk = int(SAMPLE_RATE * chunk_duration)

                while not stop_event.is_set():
                    frames = wav_file.readframes(frames_per_chunk)
                    if not frames:
                        break

                    if audio_player:
                        audio_player.write(frames)
                    else:
                        time.sleep(chunk_duration)

                    ws.send(frames, websocket.ABNF.OPCODE_BINARY)
        finally:
            if audio_player:
                audio_player.stop_stream()
                audio_player.close()
                p.terminate()

        # All audio sent - terminate the session
        print("File streaming complete. Waiting for final transcripts...")
        try:
            ws.send(json.dumps({"type": "Terminate"}))
        except Exception:
            pass

    global audio_thread
    audio_thread = threading.Thread(target=stream_file)
    audio_thread.daemon = True
    audio_thread.start()
```

#### Handle WebSocket messages

```python expandable
def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_data["session_id"] = data.get('id')
            print(f"Session ID: {data.get('id')}\n")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if not transcript:
                return

            if data.get('end_of_turn'):
                print(f"[Final]: {transcript}\n")
                session_data["turns"].append(transcript)
            else:
                print(f"[Partial]: {transcript}")
        elif msg_type == "Termination":
            session_data["audio_duration_seconds"] = data.get('audio_duration_seconds', 0)
            print(f"Session terminated: {data.get('audio_duration_seconds', 0)} seconds of audio processed")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")
```

#### WebSocket error and close handlers

```python
def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")
    stop_event.set()

    if SAVE_TRANSCRIPT_TO_FILE:
        save_transcript()

    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
```

### Connect and stream the file

```python expandable
def run():
    global ws_app

    # Validate audio file before connecting
    validate_audio_file(AUDIO_FILE, SAMPLE_RATE)

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": ASSEMBLYAI_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

The session will terminate once the file is finished streaming. If `SAVE_TRANSCRIPT_TO_FILE` is enabled (default), the transcript will be saved to `{audio_filename}_{session_id}.txt` in the current working directory.

  The `AUDIO_FILE` path can be either relative (e.g., `audio.wav`) or absolute
  (e.g., `/path/to/audio.wav`).

## Example output

Here's an example of what the console output looks like when streaming an audio file:

```console expandable
WebSocket connection opened.
Connected to: wss://streaming.assemblyai.com/v3/ws?speech_model=universal-3-5-pro&sample_rate=48000

Session ID: f37d7c4e-6be9-47ed-b6fc-7600fc78e34d

[Partial]: the
[Partial]: the quick
[Partial]: the quick brown
[Partial]: the quick brown fox
[Partial]: the quick brown fox jumps
[Partial]: the quick brown fox jumps over
[Partial]: the quick brown fox jumps over the
[Partial]: the quick brown fox jumps over the lazy
[Partial]: The quick brown fox jumps over the lazy dog
[Final]: The quick brown fox jumps over the lazy dog.

[Partial]: It
[Partial]: It is
[Partial]: It is a
[Partial]: It is a common
[Partial]: It is a common typing
[Partial]: It is a common typing test
[Final]: It is a common typing test.

File streaming complete. Waiting for final transcripts...
Session terminated: 7.52 seconds of audio processed

WebSocket Disconnected: Status=1000, Msg=None
Transcript saved to audio_f37d7c4e-6be9-47ed-b6fc-7600fc78e34d.txt
Cleanup complete. Exiting.
```

The output shows:

- **Partial transcripts**: Real-time updates as words are recognized
- **Final**: The complete turn with proper capitalization and punctuation

---

<a id="evaluate-streaming-transcription-accuracy-with-wer-docs-streaming-guides-evaluate-streaming-wer-mdx"></a>

# Evaluate Streaming transcription accuracy with WER

URL: https://www.assemblyai.com/docs/streaming/guides/evaluate_streaming_wer
Source: docs/streaming/guides/evaluate_streaming_wer.mdx
Navigation: Real-time STT > Guides > Tutorials > Use-case specific streaming workflows
Description: Evaluate Streaming transcription accuracy with WER documentation.

Learn how to evaluate the accuracy of your AssemblyAI streaming transcripts using Word Error Rate (WER), the industry-standard metric for measuring speech-to-text performance. This guide walks you through setting up a complete benchmarking workflow to measure how well your streaming implementation performs against a reference transcript.

## Quickstart

```python expandable
# pip install websocket-client jiwer whisper-normalizer
import websocket
import json
import os
import threading
import time
import wave
from urllib.parse import urlencode
import jiwer
from whisper_normalizer.basic import BasicTextNormalizer
from whisper_normalizer.english import EnglishTextNormalizer

# --- Configuration ---
ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AUDIO_FILE = "audio.wav"  # Path to your audio file
SAMPLE_RATE = 48000  # Change to match the sample rate of your audio file

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": SAMPLE_RATE,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Global variables
ws_app = None
audio_thread = None
stop_event = threading.Event()
assembly_streaming_transcript = ""

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")

    def stream_file():
        chunk_duration = 0.1

        with wave.open(AUDIO_FILE, 'rb') as wav_file:
            if wav_file.getnchannels() != 1:
                raise ValueError("Only mono audio is supported")

            file_sample_rate = wav_file.getframerate()
            if file_sample_rate != SAMPLE_RATE:
                print(f"Warning: File sample rate ({file_sample_rate}) doesn't match expected rate ({SAMPLE_RATE})")

            frames_per_chunk = int(file_sample_rate * chunk_duration)

            while not stop_event.is_set():
                frames = wav_file.readframes(frames_per_chunk)
                if not frames:
                    break
                ws.send(frames, websocket.ABNF.OPCODE_BINARY)

        print("File streaming complete. Waiting for final transcripts...")
        try:
            ws.send(json.dumps({"type": "Terminate"}))
        except Exception:
            pass

    global audio_thread
    audio_thread = threading.Thread(target=stream_file)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    global assembly_streaming_transcript
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            print(f"Session ID: {data.get('id')}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                assembly_streaming_transcript += transcript + " "
            print(transcript)
        elif msg_type == "Termination":
            print(f"Session terminated: {data.get('audio_duration_seconds', 0)} seconds of audio processed")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}")
    stop_event.set()
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---

ws_app = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": ASSEMBLYAI_API_KEY},
    on_open=on_open,
    on_message=on_message,
    on_error=on_error,
    on_close=on_close,
)

ws_thread = threading.Thread(target=ws_app.run_forever)
ws_thread.daemon = True
ws_thread.start()

try:
    while ws_thread.is_alive():
        time.sleep(0.1)
except KeyboardInterrupt:
    print("\nStopping...")
    stop_event.set()
    if ws_app:
        ws_app.close()
    ws_thread.join(timeout=2.0)

# --- Evaluate collected transcripts ---

reference_transcript = "AssemblyAI is a deep learning company that builds powerful APIs to help you transcribe and understand audio. The most common use case for the API is to automatically convert prerecorded audio and video files as well as real time audio streams into text transcriptions. Our APIs convert audio and video into text using powerful deep learning models that we research and develop end to end in house. Millions of podcasts, zoom recordings, phone calls or video files are being transcribed with Assembly AI every single day. But where Assembly AI really excels is with helping you understand your data. So let's say we transcribe Joe Biden's State of the Union using Assembly AI's API. With our Auto Chapters feature, you can generate time coded summaries of the key moments of your audio file. For example, with the State of the Union address we get chapter summaries like this. Auto Chapters automatically segments your audio or video files into chapters and provides a summary for each of these chapters. With Sentiment Analysis, we can classify what's being spoken in your audio files as either positive, negative or neutral. So for example, in the State of the Union address we see that this sentence was classified as positive, whereas this sentence was classified as negative. Content Safety Detection can flag sensitive content as it is spoken like hate speech, profanity, violence or weapons. For example, in Biden's State of the Union address, content safety detection flagged parts of his speech as being about weapons. This feature is especially useful for automatic content moderation and brand safety use cases. With Auto Highlights, you can automatically identify important words and phrases that are being spoken in your data owned by the State of the Union address. AssemblyAI's API detected these words and phrases as being important. Lastly, with entity detection you can identify entities that are spoken in your audio like organization names or person names. In Biden's speech, these were the entities that were detected. This is just a preview of the most popular features of AssemblyAI's API. If you want a full list of features, go check out our API documentation linked in the description below. And if you ever need some support, our team of developers is here to help. Everyday developers are using these features to build really exciting applications. From meeting summarizers to brand safety or contextual targeting platforms to full blown conversational intelligence tools. We can't wait to see what you build with AssemblyAI."

# Initialize normalizer
normalizer = EnglishTextNormalizer()
# For Spanish and other languages
# normalizer = BasicTextNormalizer()

def calculate_wer(reference, hypothesis, language='en'):
    # Normalize both texts
    normalized_reference = normalizer(reference)
    print("Reference: " + reference)
    print("Normalized Reference: " + normalized_reference + "\n")

    normalized_hypothesis = normalizer(hypothesis)
    print("Hypothesis: " + hypothesis)
    print("Normalized Hypothesis: " + normalized_hypothesis + "\n")

    # Calculate WER
    wer = jiwer.wer(normalized_reference, normalized_hypothesis)

    return wer * 100  # Return as percentage

wer_score = calculate_wer(reference_transcript, assembly_streaming_transcript.strip())
print(f"Final WER: {wer_score:.2f}%")
```

## Step-by-step implementation

1. **Install the required dependencies**

```bash
pip install websocket-client jiwer whisper-normalizer
```

2. **Import the necessary libraries**

```python
# pip install websocket-client jiwer whisper-normalizer
import websocket
import json
import os
import threading
import time
import wave
from urllib.parse import urlencode
import jiwer
from whisper_normalizer.basic import BasicTextNormalizer
from whisper_normalizer.english import EnglishTextNormalizer
```

3. **Set up configuration and transcript collection**

   Configure your API key, audio file settings, and create a global variable to store streaming transcripts. Your streaming session will append to this variable as it processes audio, and you'll use it for WER analysis.

```python
ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
AUDIO_FILE = "audio.wav"  # Path to your audio file
SAMPLE_RATE = 48000  # Change to match the sample rate of your audio file

CONNECTION_PARAMS = {
    "speech_model": "universal-3-5-pro",
    "sample_rate": SAMPLE_RATE,
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Global variables
ws_app = None
audio_thread = None
stop_event = threading.Event()
assembly_streaming_transcript = ""
```

4. **Configure streaming audio processing**

   Stream your audio file to the AssemblyAI endpoint. The `on_message` function captures final transcripts and appends them to your collection variable.

```python expandable
def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")

    def stream_file():
        chunk_duration = 0.1

        with wave.open(AUDIO_FILE, 'rb') as wav_file:
            if wav_file.getnchannels() != 1:
                raise ValueError("Only mono audio is supported")

            file_sample_rate = wav_file.getframerate()
            if file_sample_rate != SAMPLE_RATE:
                print(f"Warning: File sample rate ({file_sample_rate}) doesn't match expected rate ({SAMPLE_RATE})")

            frames_per_chunk = int(file_sample_rate * chunk_duration)

            while not stop_event.is_set():
                frames = wav_file.readframes(frames_per_chunk)
                if not frames:
                    break
                ws.send(frames, websocket.ABNF.OPCODE_BINARY)

        print("File streaming complete. Waiting for final transcripts...")
        try:
            ws.send(json.dumps({"type": "Terminate"}))
        except Exception:
            pass

    global audio_thread
    audio_thread = threading.Thread(target=stream_file)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    global assembly_streaming_transcript
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            print(f"Session ID: {data.get('id')}")
        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                assembly_streaming_transcript += transcript + " "
            print(transcript)
        elif msg_type == "Termination":
            print(f"Session terminated: {data.get('audio_duration_seconds', 0)} seconds of audio processed")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}")
    stop_event.set()
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# Connect and stream
ws_app = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": ASSEMBLYAI_API_KEY},
    on_open=on_open,
    on_message=on_message,
    on_error=on_error,
    on_close=on_close,
)

ws_thread = threading.Thread(target=ws_app.run_forever)
ws_thread.daemon = True
ws_thread.start()

try:
    while ws_thread.is_alive():
        time.sleep(0.1)
except KeyboardInterrupt:
    print("\nStopping...")
    stop_event.set()
    if ws_app:
        ws_app.close()
    ws_thread.join(timeout=2.0)
```

5. **Prepare your reference transcript**
   Define the ground truth transcript for comparison. This serves as your accuracy benchmark for the WER calculation.

  **Pro tip:** Create a high-quality reference transcript by first transcribing
  your audio file with AssemblyAI's Universal-3.5 Pro model, then manually
  reviewing and correcting any errors to achieve 100% accuracy.

  Ground truth quality directly affects WER results. Human transcriptions often contain systematic errors — missing filler words, incorrect proper nouns, and simplified speech patterns. If your reference transcript has errors, your WER score will be misleading. For detailed guidance on auditing ground truth files, see the [streaming evaluation guide](/streaming/evaluations/voice-agents#ground-truth-quality).

```python
# Evaluate collected transcripts
reference_transcript = "AssemblyAI is a deep learning company that builds powerful APIs to help you transcribe and understand audio. The most common use case for the API is to automatically convert prerecorded audio and video files as well as real time audio streams into text transcriptions. Our APIs convert audio and video into text using powerful deep learning models that we research and develop end to end in house. Millions of podcasts, zoom recordings, phone calls or video files are being transcribed with Assembly AI every single day. But where Assembly AI really excels is with helping you understand your data. So let's say we transcribe Joe Biden's State of the Union using Assembly AI's API. With our Auto Chapters feature, you can generate time coded summaries of the key moments of your audio file. For example, with the State of the Union address we get chapter summaries like this. Auto Chapters automatically segments your audio or video files into chapters and provides a summary for each of these chapters. With Sentiment Analysis, we can classify what's being spoken in your audio files as either positive, negative or neutral. So for example, in the State of the Union address we see that this sentence was classified as positive, whereas this sentence was classified as negative. Content Safety Detection can flag sensitive content as it is spoken like hate speech, profanity, violence or weapons. For example, in Biden's State of the Union address, content safety detection flagged parts of his speech as being about weapons. This feature is especially useful for automatic content moderation and brand safety use cases. With Auto Highlights, you can automatically identify important words and phrases that are being spoken in your data owned by the State of the Union address. AssemblyAI's API detected these words and phrases as being important. Lastly, with entity detection you can identify entities that are spoken in your audio like organization names or person names. In Biden's speech, these were the entities that were detected. This is just a preview of the most popular features of AssemblyAI's API. If you want a full list of features, go check out our API documentation linked in the description below. And if you ever need some support, our team of developers is here to help. Everyday developers are using these features to build really exciting applications. From meeting summarizers to brand safety or contextual targeting platforms to full blown conversational intelligence tools. We can't wait to see what you build with AssemblyAI."
```

6. **Initialize text normalization**
   Set up the normalizer and create your WER calculation function to ensure consistent text formatting before comparison.

```python
# Initialize normalizers
normalizer = EnglishTextNormalizer()
# For Spanish and other languages
# normalizer = BasicTextNormalizer()

def calculate_wer(reference, hypothesis, language='en'):
    # Normalize both texts
    normalized_reference = normalizer(reference)
    print("Reference: " + reference)
    print("Normalized Reference: " + normalized_reference + "\n")

    normalized_hypothesis = normalizer(hypothesis)
    print("Hypothesis: " + hypothesis)
    print("Normalized Hypothesis: " + normalized_hypothesis + "\n")

    # Calculate WER
    wer = jiwer.wer(normalized_reference, normalized_hypothesis)

    return wer * 100  # Return as percentage
```

7. **Calculate your WER score**
   Run the final calculation to measure transcription accuracy.

```python
wer_score = calculate_wer(reference_transcript, assembly_streaming_transcript.strip())
print(f"Final WER: {wer_score:.2f}%")
```

## Next steps

WER is a useful starting point, but it treats all errors equally — trivial formatting differences are penalized the same as critical errors like wrong names or hallucinated words. Consider complementing your WER analysis with [Semantic WER](/streaming/evaluations/voice-agents#semantic-wer), which normalizes equivalent words and phrases before calculating WER so that differences like `dr.` vs `doctor` or `1300` vs `thirteen hundred` aren't counted as errors. For a complete evaluation framework, see the [streaming evaluation guide](/streaming/evaluations/voice-agents).

---

<a id="determine-optimal-turn-detection-settings-from-historical-audio-analysis-docs-streaming-guides-turn-detection-improvement-using-async-mdx"></a>

# Determine Optimal Turn Detection Settings from Historical Audio Analysis

URL: https://www.assemblyai.com/docs/streaming/guides/turn_detection_improvement_using_async
Source: docs/streaming/guides/turn_detection_improvement_using_async.mdx
Navigation: Real-time STT > Guides > Tutorials > Use-case specific streaming workflows
Description: Determine Optimal Turn Detection Settings from Historical Audio Analysis documentation.

This guide shows how to analyze utterance gaps from multiple pre-recorded audio files to automatically determine optimal turn detection settings for real-time streaming transcription. It processes an entire folder, aggregates gap statistics across all recordings, and configures the WebSocket with parameters tailored to your specific conversation patterns.

## Quickstart

```python expandable
import requests
import time
import json
import pyaudio
import websocket
import threading
from urllib.parse import urlencode
from datetime import datetime
import os
from pathlib import Path

YOUR_API_KEY = "<YOUR_API_KEY>"  # Replace with your API key
AUDIO_FOLDER_PATH = "<YOUR_AUDIO_FILE_FOLDER>"  # Folder containing audio files

# Audio Configuration
SAMPLE_RATE = 16000
CHANNELS = 1
FORMAT = pyaudio.paInt16
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
recorded_frames = []
recording_lock = threading.Lock()

# Store the optimized configuration
OPTIMIZED_CONFIG = {}

def get_audio_files(folder_path):
    """
    Gets all audio files from the specified folder.
    Supports all formats accepted by AssemblyAI's API
    """
    audio_extensions = {'.aac', '.ac3', '.aif', '.aiff', '.alac', '.amr', '.ape',
    '.au', '.dss', '.flac', '.m4a', '.m4b', '.m4p', '.mp3',
    '.mpga', '.ogg', '.oga', '.mogg', '.opus', '.qcp', '.tta',
    '.voc', '.wav', '.wv', '.webm', '.MTS', '.M2TS', '.TS',
    '.mov', '.mp4', '.m4v'}
    folder = Path(folder_path)

    if not folder.exists():
        raise FileNotFoundError(f"Folder not found: {folder_path}")

    audio_files = [
        str(f) for f in folder.iterdir()
        if f.is_file() and f.suffix.lower() in audio_extensions
    ]

    if not audio_files:
        raise ValueError(f"No audio files found in {folder_path}")

    return sorted(audio_files)

def analyze_single_file(audio_file, api_key, file_index, total_files):
    """
    Analyzes a single audio file and returns gap statistics.
    """
    print("\n" + "=" * 70)
    print(f"ANALYZING FILE {file_index}/{total_files}: {Path(audio_file).name}")
    print("=" * 70)

    base_url = "https://api.assemblyai.com"
    headers = {"authorization": api_key}

    # Upload audio file
    print(f"\nUploading audio file...")

    if audio_file.startswith("http"):
        upload_url = audio_file
        print("Using provided URL")
    else:
        with open(audio_file, "rb") as f:
            response = requests.post(
                base_url + "/v2/upload",
                headers=headers,
                data=f
            )
        upload_url = response.json()["upload_url"]
        print(f"Upload complete")

    # Enable Speaker Labels
    data = {
        "audio_url": upload_url,
        "speaker_labels": True,
        # "language_detection": True # Enable automatic language detection if your files are in different languages
    }

    response = requests.post(
        base_url + "/v2/transcript",
        json=data,
        headers=headers
    )
    transcript_id = response.json()['id']
    print(f"Transcript ID: {transcript_id}")

    # Poll for completion
    print("\nWaiting for transcription to complete...")
    polling_endpoint = base_url + "/v2/transcript/" + transcript_id

    while True:
        transcription_result = requests.get(polling_endpoint, headers=headers).json()

        if transcription_result['status'] == 'completed':
            print("Transcription completed!")
            break
        elif transcription_result['status'] == 'error':
            print(f"Transcription failed: {transcription_result['error']}")
            return None
        else:
            time.sleep(3)

    # Calculate gaps
    utterances = transcription_result['utterances']

    if len(utterances) < 2:
        print("⚠ Not enough utterances to analyze gaps (need at least 2)")
        return None

    gaps = []
    for i in range(len(utterances) - 1):
        current_end = utterances[i]['end']
        next_start = utterances[i + 1]['start']
        gap = next_start - current_end

        if gap > 0:
            gaps.append(gap)

    if not gaps:
        print("⚠ No gaps found between utterances (all speech overlaps)")
        return None

    # Calculate statistics
    stats = {
        'filename': Path(audio_file).name,
        'average_gap_ms': sum(gaps) / len(gaps),
        'min_gap_ms': min(gaps),
        'max_gap_ms': max(gaps),
        'median_gap_ms': sorted(gaps)[len(gaps) // 2],
        'total_utterances': len(utterances),
        'total_gaps': len(gaps),
        'all_gaps': gaps
    }

    print(f"\nResults for {stats['filename']}:")
    print(f"   Total utterances:  {stats['total_utterances']}")
    print(f"   Total gaps:        {stats['total_gaps']}")
    print(f"   Average gap:       {stats['average_gap_ms']:.0f} ms")
    print(f"   Median gap:        {stats['median_gap_ms']:.0f} ms")

    # Save transcript JSON to file
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    safe_filename = Path(audio_file).stem.replace(' ', '_')
    json_filename = f"transcript_{safe_filename}_{timestamp}.json"

    try:
        with open(json_filename, 'w', encoding='utf-8') as f:
            json.dump(transcription_result, f, indent=2, ensure_ascii=False)
        print(f"   Transcript saved:  {json_filename}")
    except Exception as e:
        print(f"   Error saving transcript: {e}")

    return stats

def analyze_multiple_files(folder_path, api_key):
    """
    Analyzes all audio files in a folder and returns aggregated statistics.
    """
    print("=" * 70)
    print("MULTI-FILE UTTERANCE GAP ANALYSIS")
    print("=" * 70)

    audio_files = get_audio_files(folder_path)
    total_files = len(audio_files)

    print(f"\nFound {total_files} audio file(s) in: {folder_path}")
    for i, file in enumerate(audio_files, 1):
        print(f"   {i}. {Path(file).name}")

    # Analyze each file
    all_file_stats = []
    all_gaps = []

    for i, audio_file in enumerate(audio_files, 1):
        try:
            stats = analyze_single_file(audio_file, api_key, i, total_files)
            if stats:
                all_file_stats.append(stats)
                all_gaps.extend(stats['all_gaps'])
        except Exception as e:
            print(f"\n✗ Error analyzing {Path(audio_file).name}: {str(e)}")
            continue

    if not all_file_stats:
        print("\n✗ No files were successfully analyzed")
        return None

    # Calculate aggregated statistics
    print("\n" + "=" * 70)
    print("AGGREGATED GAP ANALYSIS RESULTS")
    print("=" * 70)

    aggregated_stats = {
        'total_files_analyzed': len(all_file_stats),
        'total_utterances': sum(s['total_utterances'] for s in all_file_stats),
        'total_gaps': sum(s['total_gaps'] for s in all_file_stats),
        'overall_average_gap_ms': sum(all_gaps) / len(all_gaps),
        'overall_median_gap_ms': sorted(all_gaps)[len(all_gaps) // 2],
        'overall_min_gap_ms': min(all_gaps),
        'overall_max_gap_ms': max(all_gaps),
        'file_averages': [s['average_gap_ms'] for s in all_file_stats],
        'file_stats': all_file_stats
    }

    print(f"\nFiles successfully analyzed:  {aggregated_stats['total_files_analyzed']}/{total_files}")
    print(f"Total utterances (all files): {aggregated_stats['total_utterances']}")
    print(f"Total gaps analyzed:          {aggregated_stats['total_gaps']}")
    print(f"\nOverall average gap:          {aggregated_stats['overall_average_gap_ms']:.0f} ms ({aggregated_stats['overall_average_gap_ms']/1000:.2f} seconds)")
    print(f"Overall median gap:           {aggregated_stats['overall_median_gap_ms']:.0f} ms")
    print(f"Overall minimum gap:          {aggregated_stats['overall_min_gap_ms']:.0f} ms")
    print(f"Overall maximum gap:          {aggregated_stats['overall_max_gap_ms']:.0f} ms")

    # Show per-file breakdown
    print(f"\nPer-file average gaps:")
    for stat in all_file_stats:
        print(f"   • {stat['filename']:<40} {stat['average_gap_ms']:>6.0f} ms")

    # Calculate variability
    avg_of_file_averages = sum(aggregated_stats['file_averages']) / len(aggregated_stats['file_averages'])
    variability_ratio = aggregated_stats['overall_max_gap_ms'] / aggregated_stats['overall_average_gap_ms']

    print(f"\nAverage of file averages:     {avg_of_file_averages:.0f} ms")
    print(f"Variability ratio:            {variability_ratio:.2f}x")

    if variability_ratio > 3:
        print("└─> HIGH variability - mixed conversation patterns across files")
    elif variability_ratio > 2:
        print("└─> MODERATE variability - some pattern variation")
    else:
        print("└─> LOW variability - consistent conversation rhythm")

    # Save aggregated results
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    summary_filename = f"aggregated_analysis_{timestamp}.json"

    try:
        summary_data = {
            'analysis_date': datetime.now().isoformat(),
            'folder_path': folder_path,
            'aggregated_statistics': {
                'total_files_analyzed': aggregated_stats['total_files_analyzed'],
                'total_utterances': aggregated_stats['total_utterances'],
                'total_gaps': aggregated_stats['total_gaps'],
                'overall_average_gap_ms': aggregated_stats['overall_average_gap_ms'],
                'overall_median_gap_ms': aggregated_stats['overall_median_gap_ms'],
                'overall_min_gap_ms': aggregated_stats['overall_min_gap_ms'],
                'overall_max_gap_ms': aggregated_stats['overall_max_gap_ms'],
                'variability_ratio': variability_ratio
            },
            'per_file_results': [
                {
                    'filename': s['filename'],
                    'average_gap_ms': s['average_gap_ms'],
                    'median_gap_ms': s['median_gap_ms'],
                    'total_utterances': s['total_utterances'],
                    'total_gaps': s['total_gaps']
                }
                for s in all_file_stats
            ]
        }

        with open(summary_filename, 'w', encoding='utf-8') as f:
            json.dump(summary_data, f, indent=2, ensure_ascii=False)
        print(f"\nAggregated analysis saved to: {summary_filename}")
    except Exception as e:
        print(f"\nError saving aggregated analysis: {e}")

    return aggregated_stats

def determine_streaming_config(aggregated_stats):
    """
    Determines optimal Universal-Streaming configuration based on aggregated gap analysis.
    Returns WebSocket connection parameters.
    """
    if aggregated_stats is None:
        print("\nUsing default balanced configuration (no gap data available)")
        return {
            'name': 'Balanced (Default)',
            'min_turn_silence': 400,
            'max_turn_silence': 1280,
            'description': 'Standard configuration for general use'
        }

    print("\n" + "=" * 70)
    print("DETERMINING OPTIMAL STREAMING CONFIGURATION")
    print("=" * 70)

    avg_gap = aggregated_stats['overall_average_gap_ms']
    num_files = aggregated_stats['total_files_analyzed']

    print(f"\nBased on analysis of {num_files} file(s)")
    print(f"Overall average gap: {avg_gap:.0f} ms")

    # Determine configuration based on average gap
    if avg_gap < 500:
        config = {
            'name': 'Aggressive',
            'min_turn_silence': 160,
            'max_turn_silence': 400,
            'description': 'Fast-paced conversation with quick turn-taking'
        }
        use_cases = "IVR systems, order confirmations, yes/no queries, retail support"
    elif avg_gap < 1000:
        config = {
            'name': 'Balanced',
            'min_turn_silence': 400,
            'max_turn_silence': 1280,
            'description': 'Natural conversation pacing'
        }
        use_cases = "General customer support, consultations, standard voice agents"
    else:
        config = {
            'name': 'Conservative',
            'min_turn_silence': 800,
            'max_turn_silence': 3600,
            'description': 'Thoughtful, complex speech with longer pauses'
        }
        use_cases = "Technical support, healthcare, legal consultations, troubleshooting"

    print(f"\nSelected Configuration: {config['name']}")
    print(f"   Reasoning: Average gap of {avg_gap:.0f}ms indicates {config['description']}")
    print(f"\nConfiguration Parameters:")
    print(f"   • min_turn_silence: {config['min_turn_silence']} ms")
    print(f"   • max_turn_silence: {config['max_turn_silence']} ms")
    print(f"\nRecommended use cases: {use_cases}")

    return config

#  WEBSOCKET HANDLERS WITH OPTIMIZED SETTINGS

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Using optimized {OPTIMIZED_CONFIG['name']} configuration")

    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                with recording_lock:
                    recorded_frames.append(audio_data)

                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}")
            print(f"   Expires at: {datetime.fromtimestamp(expires_at)}")
            print(f"   Configuration: {OPTIMIZED_CONFIG['name']}")
            print("\nSpeak now... (Press Ctrl+C to stop)\n")

        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(f"FINAL: {transcript}")
            else:
                print(f"\r  partial: {transcript}", end='')

        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio={audio_duration}s, Session={session_duration}s")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# RUN STREAMING WITH OPTIMIZED CONFIGURATION

def run_streaming(config):
    """
    Runs the streaming transcription with optimized turn detection settings.
    """
    global audio, stream, ws_app, OPTIMIZED_CONFIG

    OPTIMIZED_CONFIG = config

    print("\n" + "=" * 70)
    print("STARTING REAL-TIME STREAMING")
    print("=" * 70)

    # Build connection parameters with optimized settings
    CONNECTION_PARAMS = {
        "sample_rate": SAMPLE_RATE,
        "format_turns": True,
        "min_turn_silence": str(config['min_turn_silence']),
        "max_turn_silence": str(config['max_turn_silence'])
    }

    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    print(f"\nWebSocket Endpoint: {API_ENDPOINT_BASE_URL}")
    print(f"\nApplied Configuration:")
    for key, value in CONNECTION_PARAMS.items():
        print(f"   • {key}: {value}")

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("\nMicrophone stream opened successfully.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message...")
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

# MAIN WORKFLOW

def main():
    """
    Main workflow: Analyze multiple files -> Configure -> Run Streaming
    """

    try:
        # Step 1: Analyze all audio files in folder
        aggregated_stats = analyze_multiple_files(AUDIO_FOLDER_PATH, YOUR_API_KEY)

        # Step 2: Determine optimal configuration based on aggregated data
        streaming_config = determine_streaming_config(aggregated_stats)

        # Step 3: Run streaming with optimized settings
        run_streaming(streaming_config)

    except Exception as e:
        print(f"\nError in workflow: {str(e)}")
        raise

# EXECUTION

if __name__ == "__main__":
    main()
```

## Step-By-Step Guide

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) and get your API key from your dashboard.

1. **Install All Required Packages**

```shell
pip install requests pyaudio websocket-client
```

2. **Configuration and Global Variables**

Set up API credentials, file paths, audio parameters (16kHz sample rate, mono channel), and initialize global variables for managing WebSocket connections and audio streaming threads.

```python expandable
import requests
import time
import json
import pyaudio
import websocket
import threading
from urllib.parse import urlencode
from datetime import datetime
import os
from pathlib import Path

YOUR_API_KEY = "<YOUR_API_KEY>"  # Replace with your API key
AUDIO_FOLDER_PATH = "<YOUR_AUDIO_FILE_FOLDER>"  # Folder containing audio files

# Audio Configuration
SAMPLE_RATE = 16000
CHANNELS = 1
FORMAT = pyaudio.paInt16
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()
recorded_frames = []
recording_lock = threading.Lock()

# Store the optimized configuration
OPTIMIZED_CONFIG = {}
```

3. **Define get_audio_files() Function**

This function scans a specified folder for audio/video files with supported extensions and returns a sorted list of file paths for batch processing.

```python
def get_audio_files(folder_path):
    audio_extensions = {'.aac', '.ac3', '.aif', '.aiff', '.alac', '.amr', '.ape',
    '.au', '.dss', '.flac', '.m4a', '.m4b', '.m4p', '.mp3',
    '.mpga', '.ogg', '.oga', '.mogg', '.opus', '.qcp', '.tta',
    '.voc', '.wav', '.wv', '.webm', '.MTS', '.M2TS', '.TS',
    '.mov', '.mp4', '.m4v'}
    folder = Path(folder_path)

    if not folder.exists():
        raise FileNotFoundError(f"Folder not found: {folder_path}")

    audio_files = [
        str(f) for f in folder.iterdir()
        if f.is_file() and f.suffix.lower() in audio_extensions
    ]

    if not audio_files:
        raise ValueError(f"No audio files found in {folder_path}")

    return sorted(audio_files)
```

4. **Define `analyze_single_file()` Function**

This function uploads an audio file to AssemblyAI, requests transcription with speaker labels enabled, polls until completion, then calculates gap statistics between utterances (average, median, min, max) and saves the transcript JSON.

```python expandable
def analyze_single_file(audio_file, api_key, file_index, total_files):
    print("\n" + "=" * 70)
    print(f"ANALYZING FILE {file_index}/{total_files}: {Path(audio_file).name}")
    print("=" * 70)

    base_url = "https://api.assemblyai.com"
    headers = {"authorization": api_key}

    # Upload audio file
    print(f"\nUploading audio file...")

    if audio_file.startswith("http"):
        upload_url = audio_file
        print("Using provided URL")
    else:
        with open(audio_file, "rb") as f:
            response = requests.post(
                base_url + "/v2/upload",
                headers=headers,
                data=f
            )
        upload_url = response.json()["upload_url"]
        print(f"Upload complete")

    # Enable Speaker Labels
    data = {
        "audio_url": upload_url,
        "speaker_labels": True,
        # "language_detection": True # Enable automatic language detection if your files are in different languages
    }

    response = requests.post(
        base_url + "/v2/transcript",
        json=data,
        headers=headers
    )
    transcript_id = response.json()['id']
    print(f"Transcript ID: {transcript_id}")

    # Poll for completion
    print("\nWaiting for transcription to complete...")
    polling_endpoint = base_url + "/v2/transcript/" + transcript_id

    while True:
        transcription_result = requests.get(polling_endpoint, headers=headers).json()

        if transcription_result['status'] == 'completed':
            print("Transcription completed!")
            break
        elif transcription_result['status'] == 'error':
            print(f"Transcription failed: {transcription_result['error']}")
            return None
        else:
            time.sleep(3)

    # Calculate gaps
    utterances = transcription_result['utterances']

    if len(utterances) < 2:
        print("⚠ Not enough utterances to analyze gaps (need at least 2)")
        return None

    gaps = []
    for i in range(len(utterances) - 1):
        current_end = utterances[i]['end']
        next_start = utterances[i + 1]['start']
        gap = next_start - current_end

        if gap > 0:
            gaps.append(gap)

    if not gaps:
        print("⚠ No gaps found between utterances (all speech overlaps)")
        return None

    # Calculate statistics
    stats = {
        'filename': Path(audio_file).name,
        'average_gap_ms': sum(gaps) / len(gaps),
        'min_gap_ms': min(gaps),
        'max_gap_ms': max(gaps),
        'median_gap_ms': sorted(gaps)[len(gaps) // 2],
        'total_utterances': len(utterances),
        'total_gaps': len(gaps),
        'all_gaps': gaps
    }

    print(f"\nResults for {stats['filename']}:")
    print(f"   Total utterances:  {stats['total_utterances']}")
    print(f"   Total gaps:        {stats['total_gaps']}")
    print(f"   Average gap:       {stats['average_gap_ms']:.0f} ms")
    print(f"   Median gap:        {stats['median_gap_ms']:.0f} ms")

    # Save transcript JSON to file
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    safe_filename = Path(audio_file).stem.replace(' ', '_')
    json_filename = f"transcript_{safe_filename}_{timestamp}.json"

    try:
        with open(json_filename, 'w', encoding='utf-8') as f:
            json.dump(transcription_result, f, indent=2, ensure_ascii=False)
        print(f"   Transcript saved:  {json_filename}")
    except Exception as e:
        print(f"   Error saving transcript: {e}")

    return stats
```

5. **Define `analyze_multiple_files()` Function**

This function orchestrates the analysis of all files in a folder by calling `analyze_single_file()` for each, aggregates all gap data across files, calculates overall statistics, displays per-file breakdowns, and saves a comprehensive summary JSON.

```python expandable
def analyze_multiple_files(folder_path, api_key):
    print("=" * 70)
    print("MULTI-FILE UTTERANCE GAP ANALYSIS")
    print("=" * 70)

    audio_files = get_audio_files(folder_path)
    total_files = len(audio_files)

    print(f"\nFound {total_files} audio file(s) in: {folder_path}")
    for i, file in enumerate(audio_files, 1):
        print(f"   {i}. {Path(file).name}")

    # Analyze each file
    all_file_stats = []
    all_gaps = []

    for i, audio_file in enumerate(audio_files, 1):
        try:
            stats = analyze_single_file(audio_file, api_key, i, total_files)
            if stats:
                all_file_stats.append(stats)
                all_gaps.extend(stats['all_gaps'])
        except Exception as e:
            print(f"\n✗ Error analyzing {Path(audio_file).name}: {str(e)}")
            continue

    if not all_file_stats:
        print("\n✗ No files were successfully analyzed")
        return None

    # Calculate aggregated statistics
    print("\n" + "=" * 70)
    print("AGGREGATED GAP ANALYSIS RESULTS")
    print("=" * 70)

    aggregated_stats = {
        'total_files_analyzed': len(all_file_stats),
        'total_utterances': sum(s['total_utterances'] for s in all_file_stats),
        'total_gaps': sum(s['total_gaps'] for s in all_file_stats),
        'overall_average_gap_ms': sum(all_gaps) / len(all_gaps),
        'overall_median_gap_ms': sorted(all_gaps)[len(all_gaps) // 2],
        'overall_min_gap_ms': min(all_gaps),
        'overall_max_gap_ms': max(all_gaps),
        'file_averages': [s['average_gap_ms'] for s in all_file_stats],
        'file_stats': all_file_stats
    }

    print(f"\nFiles successfully analyzed:  {aggregated_stats['total_files_analyzed']}/{total_files}")
    print(f"Total utterances (all files): {aggregated_stats['total_utterances']}")
    print(f"Total gaps analyzed:          {aggregated_stats['total_gaps']}")
    print(f"\nOverall average gap:          {aggregated_stats['overall_average_gap_ms']:.0f} ms ({aggregated_stats['overall_average_gap_ms']/1000:.2f} seconds)")
    print(f"Overall median gap:           {aggregated_stats['overall_median_gap_ms']:.0f} ms")
    print(f"Overall minimum gap:          {aggregated_stats['overall_min_gap_ms']:.0f} ms")
    print(f"Overall maximum gap:          {aggregated_stats['overall_max_gap_ms']:.0f} ms")

    # Show per-file breakdown
    print(f"\nPer-file average gaps:")
    for stat in all_file_stats:
        print(f"   • {stat['filename']:<40} {stat['average_gap_ms']:>6.0f} ms")

    # Calculate variability
    avg_of_file_averages = sum(aggregated_stats['file_averages']) / len(aggregated_stats['file_averages'])
    variability_ratio = aggregated_stats['overall_max_gap_ms'] / aggregated_stats['overall_average_gap_ms']

    print(f"\nAverage of file averages:     {avg_of_file_averages:.0f} ms")
    print(f"Variability ratio:            {variability_ratio:.2f}x")

    if variability_ratio > 3:
        print("└─> HIGH variability - mixed conversation patterns across files")
    elif variability_ratio > 2:
        print("└─> MODERATE variability - some pattern variation")
    else:
        print("└─> LOW variability - consistent conversation rhythm")

    # Save aggregated results
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    summary_filename = f"aggregated_analysis_{timestamp}.json"

    try:
        summary_data = {
            'analysis_date': datetime.now().isoformat(),
            'folder_path': folder_path,
            'aggregated_statistics': {
                'total_files_analyzed': aggregated_stats['total_files_analyzed'],
                'total_utterances': aggregated_stats['total_utterances'],
                'total_gaps': aggregated_stats['total_gaps'],
                'overall_average_gap_ms': aggregated_stats['overall_average_gap_ms'],
                'overall_median_gap_ms': aggregated_stats['overall_median_gap_ms'],
                'overall_min_gap_ms': aggregated_stats['overall_min_gap_ms'],
                'overall_max_gap_ms': aggregated_stats['overall_max_gap_ms'],
                'variability_ratio': variability_ratio
            },
            'per_file_results': [
                {
                    'filename': s['filename'],
                    'average_gap_ms': s['average_gap_ms'],
                    'median_gap_ms': s['median_gap_ms'],
                    'total_utterances': s['total_utterances'],
                    'total_gaps': s['total_gaps']
                }
                for s in all_file_stats
            ]
        }

        with open(summary_filename, 'w', encoding='utf-8') as f:
            json.dump(summary_data, f, indent=2, ensure_ascii=False)
        print(f"\nAggregated analysis saved to: {summary_filename}")
    except Exception as e:
        print(f"\nError saving aggregated analysis: {e}")

    return aggregated_stats
```

6. **Define `determine_streaming_config()` Function**

This function takes aggregated gap statistics and selects one of three preset configurations with optimized turn detection parameters for different conversation styles.

```python expandable
def determine_streaming_config(aggregated_stats):
    if aggregated_stats is None:
        print("\nUsing default balanced configuration (no gap data available)")
        return {
            'name': 'Balanced (Default)',
            'min_turn_silence': 400,
            'max_turn_silence': 1280,
            'description': 'Standard configuration for general use'
        }

    print("\n" + "=" * 70)
    print("DETERMINING OPTIMAL STREAMING CONFIGURATION")
    print("=" * 70)

    avg_gap = aggregated_stats['overall_average_gap_ms']
    num_files = aggregated_stats['total_files_analyzed']

    print(f"\nBased on analysis of {num_files} file(s)")
    print(f"Overall average gap: {avg_gap:.0f} ms")

    # Determine configuration based on average gap
    if avg_gap < 500:
        config = {
            'name': 'Aggressive',
            'min_turn_silence': 160,
            'max_turn_silence': 400,
            'description': 'Fast-paced conversation with quick turn-taking'
        }
        use_cases = "IVR systems, order confirmations, yes/no queries, retail support"
    elif avg_gap < 1000:
        config = {
            'name': 'Balanced',
            'min_turn_silence': 400,
            'max_turn_silence': 1280,
            'description': 'Natural conversation pacing'
        }
        use_cases = "General customer support, consultations, standard voice agents"
    else:
        config = {
            'name': 'Conservative',
            'min_turn_silence': 800,
            'max_turn_silence': 3600,
            'description': 'Thoughtful, complex speech with longer pauses'
        }
        use_cases = "Technical support, healthcare, legal consultations, troubleshooting"

    print(f"\nSelected Configuration: {config['name']}")
    print(f"   Reasoning: Average gap of {avg_gap:.0f}ms indicates {config['description']}")
    print(f"\nConfiguration Parameters:")
    print(f"   • min_turn_silence: {config['min_turn_silence']} ms")
    print(f"   • max_turn_silence: {config['max_turn_silence']} ms")
    print(f"\nRecommended use cases: {use_cases}")

    return config
```

7. **Create WebSocket Event Handlers (`on_open`, `on_message`, `on_error`, `on_close`)**

These functions manage the real-time streaming connection lifecycle: `on_open` starts the audio streaming thread, `on_message` processes transcription results (partial and final turns), and the close/error handlers clean up resources.

```python expandable
def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Using optimized {OPTIMIZED_CONFIG['name']} configuration")

    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                with recording_lock:
                    recorded_frames.append(audio_data)

                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = True
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"\nSession began: ID={session_id}")
            print(f"   Expires at: {datetime.fromtimestamp(expires_at)}")
            print(f"   Configuration: {OPTIMIZED_CONFIG['name']}")
            print("\nSpeak now... (Press Ctrl+C to stop)\n")

        elif msg_type == "Turn":
            transcript = data.get('transcript', '')
            if data.get('end_of_turn'):
                print('\r' + ' ' * 80 + '\r', end='')
                print(f"FINAL: {transcript}")
            else:
                print(f"\r  partial: {transcript}", end='')

        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"\nSession Terminated: Audio={audio_duration}s, Session={session_duration}s")

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    global stream, audio
    stop_event.set()

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
```

8. **Define `run_streaming()` Function**

This function initializes PyAudio to capture microphone input, establishes a WebSocket connection with the optimized configuration parameters, and streams audio in real-time while displaying transcription results until the user stops with Ctrl+C.

```python expandable
def run_streaming(config):
    global audio, stream, ws_app, OPTIMIZED_CONFIG

    OPTIMIZED_CONFIG = config

    print("\n" + "=" * 70)
    print("STARTING REAL-TIME STREAMING")
    print("=" * 70)

    # Build connection parameters with optimized settings
    CONNECTION_PARAMS = {
        "sample_rate": SAMPLE_RATE,
        "format_turns": True,
        "min_turn_silence": str(config['min_turn_silence']),
        "max_turn_silence": str(config['max_turn_silence'])
    }

    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    print(f"\nWebSocket Endpoint: {API_ENDPOINT_BASE_URL}")
    print(f"\nApplied Configuration:")
    for key, value in CONNECTION_PARAMS.items():
        print(f"   • {key}: {value}")

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("\nMicrophone stream opened successfully.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()

        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message...")
                ws_app.send(json.dumps(terminate_message))
                time.sleep(1)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        if ws_app:
            ws_app.close()

        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")
```

9. **Define `main()` Workflow**

Execute the three-step process: analyze all audio files in the folder, determine the best streaming configuration based on aggregated utterance gaps, then launch real-time streaming with the optimized settings.

```python
def main():
    try:
        # Step 1: Analyze all audio files in folder
        aggregated_stats = analyze_multiple_files(AUDIO_FOLDER_PATH, YOUR_API_KEY)

        # Step 2: Determine optimal configuration based on aggregated data
        streaming_config = determine_streaming_config(aggregated_stats)

        # Step 3: Run streaming with optimized settings
        run_streaming(streaming_config)

    except Exception as e:
        print(f"\nError in workflow: {str(e)}")
        raise

if __name__ == "__main__":
    main()
```

---

<a id="transcribe-a-short-audio-file-docs-sync-stt-getting-started-transcribe-a-short-audio-file-mdx"></a>

# Transcribe a short audio file

URL: https://www.assemblyai.com/docs/sync-stt/getting-started/transcribe-a-short-audio-file
Source: docs/sync-stt/getting-started/transcribe-a-short-audio-file.mdx
Navigation: Sync STT > Getting started
Description: Learn how to transcribe an audio file synchronously with the Sync API.

## Overview

Send an audio file in a single call, get a transcript back in milliseconds. No polling, no session management. The SDKs wrap the whole round trip in one method:

```python
import os

from assemblyai.sync.v1 import SyncTranscriber

transcriber = SyncTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
result = transcriber.transcribe("./sample.wav")
print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("./sample.wav");
console.log(result.text);
```

The Sync API uses our flagship Universal-3.5 Pro model and accepts audio from 80 milliseconds up to 120 seconds. It transcribes with state-of-the-art accuracy in [19 languages](/sync-stt/language-selection#supported-languages).

Not using an SDK? The same flow works over plain HTTP — see [Using the HTTP API directly](#using-the-http-api-directly).

  **When to use Sync STT**

  Sync STT is ideal for pre-recorded audio clips under 2 minutes where you need
  an immediate response without polling — for example, voice message transcription,
  short call recordings, or voice agent pipelines that handle turn detection
  externally and submit completed utterances for transcription. For audio longer
  than 120 seconds, use [Pre-recorded STT](/pre-recorded-audio/getting-started/transcribe-an-audio-file).
  For live microphone audio, use [Real-time STT](/streaming/getting-started/transcribe-streaming-audio).

## Before you begin

To complete this guide, you need:

- **An API key** — browse to [API Keys](https://www.assemblyai.com/dashboard/home) in your dashboard and copy your key. Every example below reads it from an environment variable, so set it once:

  ```bash
  export ASSEMBLYAI_API_KEY=<your-key>
  ```

- **An audio file** in WAV format (or raw PCM S16LE), between 80 milliseconds and 120 seconds long.
- **Python 3.8+** for the Python SDK, or **Node.js 18+** for the JavaScript SDK. The HTTP examples at the bottom of this page need Python 3.8+, Node.js 18+, or cURL.

## Transcribe your first file

### Step 1: Install the SDK

```bash
pip install -U assemblyai
```

```bash
npm install assemblyai
```

### Step 2: Run your first transcription

Save this as `transcribe.py` (Python) or `transcribe.mjs` (JavaScript), next to a `sample.wav` file:

```python
import os

from assemblyai.sync.v1 import SyncTranscriber

transcriber = SyncTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
result = transcriber.transcribe("./sample.wav")
print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("./sample.wav");
console.log(result.text);
```

Then run it — `python transcribe.py` or `node transcribe.mjs`. You'll see the transcript printed:

```text
Hi, I'm calling about my Best Buy order...
```

`transcribe()` accepts a local file path or raw audio bytes — an open binary file object in Python, or a `Uint8Array`, `Blob`, or readable stream in JavaScript — but not a URL. The Sync API has no URL ingestion; for remote files, download them first or use [Pre-recorded STT](/pre-recorded-audio/getting-started/transcribe-an-audio-file).

## Customize your request

The call above works with no extra configuration. Pass a config to change how the audio is transcribed — in Python a `SyncTranscriptionConfig`, either as the transcriber's default (`SyncTranscriber(config=...)`) or per call (`transcribe(data, config=...)`); in JavaScript, a plain object as the second argument to `transcribe()`.

### Select a model

`model` selects the sync speech model and defaults to `universal-3-5-pro`. Set it explicitly to pin the model:

```python
from assemblyai.sync.v1 import SyncTranscriptionConfig

config = SyncTranscriptionConfig(model="universal-3-5-pro")
result = transcriber.transcribe("./sample.wav", config=config)
```

```javascript
const result = await client.sync.transcribe("./sample.wav", {
  model: "universal-3-5-pro",
});
```

### Choose a language

Pass [`language_codes`](/sync-stt/language-selection) as a list of ISO 639-1 codes — one code for monolingual audio, several for multilingual audio:

```python
from assemblyai.sync.v1 import SyncTranscriptionConfig

config = SyncTranscriptionConfig(language_codes=["es"])
result = transcriber.transcribe("./sample.wav", config=config)
```

```javascript
const result = await client.sync.transcribe("./sample.wav", {
  language_codes: ["es"],
});
```

### Get word timestamps

Per-word timings are opt-in. Set [`timestamps=True`](/sync-stt/word-timestamps) to compute `start`/`end` for every word, at a small latency cost. Without it, words carry `text` and `confidence` only:

```python
from assemblyai.sync.v1 import SyncTranscriptionConfig

config = SyncTranscriptionConfig(timestamps=True)
result = transcriber.transcribe("./sample.wav", config=config)

for word in result.words:
    print(word.text, word.start, word.end)  # milliseconds
```

```javascript
const result = await client.sync.transcribe("./sample.wav", {
  timestamps: true,
});

for (const word of result.words) {
  console.log(word.text, word.start, word.end); // milliseconds
}
```

### Transcribe raw PCM audio

WAV files carry their sample rate and channel count in the file header. Raw PCM (S16LE little-endian) doesn't, so set `sample_rate` and `channels` on the config — both are required, and setting either one routes the audio as raw PCM:

```python
from assemblyai.sync.v1 import SyncTranscriptionConfig

config = SyncTranscriptionConfig(sample_rate=16000, channels=1)

with open("./sample.pcm", "rb") as f:
    result = transcriber.transcribe(f.read(), config=config)

print(result.text)
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("./sample.pcm");

const result = await client.sync.transcribe(audio, {
  sample_rate: 16000,
  channels: 1,
});

console.log(result.text);
```

## Complete example

Here's the complete, runnable script — the call above plus options and error handling:

```python expandable
import os

from assemblyai.sync.v1 import (
    SyncTranscriber,
    SyncTranscriptError,
    SyncTranscriptionConfig,
)

config = SyncTranscriptionConfig(
    model="universal-3-5-pro",
    language_codes=["en"],
    timestamps=True,
)

with SyncTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"]) as transcriber:
    try:
        result = transcriber.transcribe("./sample.wav", config=config)
    except SyncTranscriptError as error:
        # error_code is machine-readable (bad_audio, audio_too_large,
        # capacity_exceeded, inference_timeout, ...); retry_after is set on
        # 429/503 responses. See /sync-stt/error-handling
        raise RuntimeError(
            f"Transcription failed ({error.status_code}/{error.error_code}): {error}"
        ) from error

    # Record session_id for every request, not just errors — it's the first
    # thing support@assemblyai.com asks for.
    print(f"session_id: {result.session_id}")
    print(f"\nFull Transcript:\n\n{result.text}")

    for word in result.words:
        print(f"{word.text} ({word.confidence:.2f}) {word.start}-{word.end}ms")
```

```javascript expandable
import { AssemblyAI, SyncTranscriptError } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

try {
  const result = await client.sync.transcribe("./sample.wav", {
    model: "universal-3-5-pro",
    language_codes: ["en"],
    timestamps: true,
  });

  // Record session_id for every request, not just errors — it's the first
  // thing support@assemblyai.com asks for.
  console.log(`session_id: ${result.session_id}`);
  console.log(`\nFull Transcript:\n\n${result.text}`);

  for (const word of result.words) {
    console.log(
      `${word.text} (${word.confidence.toFixed(2)}) ${word.start}-${word.end}ms`
    );
  }
} catch (error) {
  if (error instanceof SyncTranscriptError) {
    // errorCode is machine-readable (bad_audio, audio_too_large,
    // capacity_exceeded, inference_timeout, ...); retryAfter is set on
    // 429/503 responses. See /sync-stt/error-handling
    throw new Error(
      `Transcription failed (${error.status}/${error.errorCode}): ${error.message}`
    );
  }
  throw error;
}
```

  Calling the Sync API repeatedly? `warm()` opens the connection ahead of time
  so your next `transcribe()` doesn't pay the DNS, TCP, and TLS handshake on the
  critical path. See [Connection pre-warming](/sync-stt/connection-pre-warming).

## What you get back

`transcribe()` returns a `SyncTranscriptResponse` with the transcript and word-level details:

| Attribute | Description |
|-------|-------------|
| `result.text` | Full transcript of the audio. |
| `result.words` | Per-word `text` and `confidence`, plus `start`/`end` timestamps (in milliseconds) when the request sets [`timestamps=True`](/sync-stt/word-timestamps). Otherwise `start`/`end` are `None`. |
| `result.confidence` | Overall transcript confidence score (0–1). |
| `result.audio_duration_ms` | Duration of the submitted audio in milliseconds. |
| `result.session_id` | Server-generated UUID. Include this when contacting support. |
| `result.request_time_ms` | End-to-end server-side processing time for the request, in milliseconds. |

Failed requests raise `SyncTranscriptError`, which carries the HTTP status, a machine-readable error code (for example `bad_audio`, `audio_too_large`, `capacity_exceeded`, `inference_timeout`), and a retry-after value in seconds for `429`/`503` responses — named `status_code`/`error_code`/`retry_after` in the Python SDK and `status`/`errorCode`/`retryAfter` in the JavaScript SDK. See [Error handling](/sync-stt/error-handling) for status codes and retry guidance.

## Using the HTTP API directly

If you prefer to call the Sync API over HTTP without an SDK, post the audio to `POST https://sync.assemblyai.com/transcribe` and read the transcript out of the JSON response.

The Sync API accepts `multipart/form-data` with an `audio` part. For WAV files, set the part's Content-Type to `audio/wav`. Authenticate with your key in the `Authorization` header (no `Bearer` prefix), and send the `X-AAI-Model: universal-3-5-pro` header — it is required on every request.

Replace `<YOUR_API_KEY>` in the request header:

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav'
```

The response JSON is printed to stdout. Pipe to `jq` to extract fields:

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav' \
  | jq '.text'
```

Install the `requests` library if you haven't already:

```bash
pip install requests
```

Send the audio file and parse the response:

```python expandable
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
    },
    timeout=60,
)
response.raise_for_status()
result = response.json()
print(result["text"])
print(result["session_id"])
```

```javascript expandable
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

if (!response.ok) {
  const error = await response.json();
  throw new Error(error.message || error.detail);
}

const result = await response.json();
console.log(result.text);
console.log(result.session_id);
```

### Response shape

A successful request returns a JSON object with the transcript and word-level details:

```json
{
  "text": "Hi, I'm calling about my Best Buy order...",
  "words": [
    { "text": "Hi",  "confidence": 0.91 },
    { "text": "I'm", "confidence": 0.88 }
  ],
  "confidence": 0.87,
  "audio_duration_ms": 101567,
  "session_id": "eb92c4ff-4bbb-429f-9b99-7279d7fe738f",
  "request_time_ms": 243.7
}
```

| Field | Description |
|-------|-------------|
| `text` | Full transcript of the audio. |
| `words` | Per-word `confidence`, plus `start`/`end` timestamps (in milliseconds) when the request enables [word timestamps](/sync-stt/word-timestamps). |
| `confidence` | Overall transcript confidence score (0–1). |
| `audio_duration_ms` | Duration of the submitted audio in milliseconds. |
| `session_id` | Server-generated UUID. Include this when contacting support. |
| `request_time_ms` | End-to-end server-side processing time for the request, in milliseconds. |

### Sending raw PCM audio

When your audio is raw PCM (S16LE little-endian), set the `audio` part Content-Type to `audio/pcm` and include `sample_rate` and `channels` in the `config` part:

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.pcm;type=audio/pcm' \
  -F 'config={"sample_rate":16000,"channels":1};type=application/json'
```

```python
import json
import requests

with open("sample.pcm", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.pcm", audio, "audio/pcm"),
        "config": (
            None,
            json.dumps({"sample_rate": 16000, "channels": 1}),
            "application/json",
        ),
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("sample.pcm");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/pcm" }), "sample.pcm");
form.append(
  "config",
  new Blob(
    [JSON.stringify({ sample_rate: 16000, channels: 1 })],
    { type: "application/json" }
  )
);

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

## Next steps

- [Prompting and keyterms](/sync-stt/prompting-and-keyterms) — improve accuracy with contextual prompts and keyterm biasing
- [Conversation context](/sync-stt/conversation-context) — pass prior dialogue for multi-turn continuity
- [Language selection](/sync-stt/language-selection) — transcribe in any of the 19 supported languages
- [Word timestamps](/sync-stt/word-timestamps) — get per-word `start`/`end` timings
- [Audio requirements](/sync-stt/audio-requirements) — duration, size, and format constraints
- [Connection pre-warming](/sync-stt/connection-pre-warming) — cut the handshake off the critical path
- [Error handling](/sync-stt/error-handling) — status codes and retry guidance
- [Cloud endpoints & data residency](/sync-stt/endpoints-and-data-zones) — use the EU endpoint for data residency
- [API reference](/api-reference/sync-api/transcribe) — full endpoint documentation

## Need help?

If you get stuck, contact our support team at [support@assemblyai.com](mailto:support@assemblyai.com) or create a [support ticket](https://www.assemblyai.com/contact/support). Include the `session_id` from the response to help us look up your request.

---

<a id="prompting-and-keyterms-docs-sync-stt-prompting-and-keyterms-mdx"></a>

# Prompting and Keyterms

URL: https://www.assemblyai.com/docs/sync-stt/prompting-and-keyterms
Source: docs/sync-stt/prompting-and-keyterms.mdx
Navigation: Sync STT > Features
Description: Prompting and Keyterms documentation.

Use contextual prompting and keyterms to improve Sync STT transcription accuracy.

The Sync API supports two complementary ways to give the model information about your audio, both passed in the optional `config` JSON part alongside the `audio` part:

- **Contextual prompting** (`prompt`) — a natural-language description of what the audio is about: the domain, the scenario, or details of the conversation.
- **Keyterms** (`keyterms_prompt`) — an explicit list of terms you want the model to recognize accurately.

Both improve recognition accuracy for your use case. Neither changes the output format.

## Contextual prompting

Universal-3.5 Pro is trained to use context about the audio — its domain, topic, or scenario — to better recognize the vocabulary that context makes likely. A clip described as a cardiology consultation primes the model for medical terminology; one described as an order-status check primes it for order IDs and product names.

Write the `prompt` as a **description of your audio**, not as instructions to the model. Transcription behavior — punctuation, formatting, verbatim style — is already optimized out of the box, and a managed default prompt is applied when you don't provide one.

- Maximum length: **4096 characters**
- When omitted, a managed default prompt is applied
- `language_code` is **ignored** when a custom `prompt` is set — state the language as part of your description if you need both (see [Specifying the language](#specifying-the-language))

**Start with no prompt**

We strongly recommend testing with no `prompt` first. Universal-3.5 Pro is optimized out of the box, and context helps most when your audio contains domain-specific vocabulary the model is getting wrong. Add context when you see those errors, starting with the broadest description that covers your use case.

### Choosing a level of specificity

Contextual prompts work at three levels of specificity. Use the least specific level that covers your use case, and add detail when your audio contains uncommon names or terms the model can't otherwise know.

| Level | Length | What it contains | Example |
| --- | --- | --- | --- |
| **Domain** | 2–5 words | The domain only | `Medical consultation call.` |
| **Scenario** | 5–15 words | What the conversation is about | `Cardiology consultation about chest pain symptoms.` |
| **Detailed** | 20–50 words | Full description, including names, products, or identifiers | `Cardiology consultation between Dr. Smith and an elderly patient regarding recurring chest pain, ECG results, and medication adjustment for hypertension.` |

**Domain context** tells the model what field the audio belongs to — medical, legal, technical support, food ordering. This is the safest starting point and is often enough to fix vocabulary errors.

**Scenario context** describes the specific situation. This is the right level for most applications that know what kind of conversation is taking place — appointment booking, billing inquiry, delivery complaint.

**Detailed context** describes everything your application already knows about the conversation: participant names, account or order identifiers, products, locations. This level is the most powerful when the audio contains proper nouns and identifiers — for example, a voice agent platform that already knows who is calling and why can pass that information so the model spells names and IDs correctly.

Accuracy improves monotonically with context specificity — benchmarked on 20,000 real voice-agent calls, scenario context cuts word error rate ~10% and detailed context ~21% versus no prompt, with the largest gains on names and identifiers. See [Accuracy impact](/streaming/prompting-and-keyterms#accuracy-impact) for the full results.

Guidelines for writing contextual prompts:

- Write plain, complete sentences that describe the audio — you are describing a recording, not commanding the model.
- Include specifics your application already knows — participant names, products, order identifiers — when the audio contains proper nouns the model can't otherwise know.
- Keep it to one short block of text. Don't pack lists of keywords into the prompt — that's what [`keyterms_prompt`](#keyterms-prompting) is for.
- The model stays grounded in the audio: context that turns out to be irrelevant does not cause it to insert words that weren't spoken, so you can safely send the same description with every request of a longer interaction.

### Specifying the language

A custom `prompt` replaces the managed default, including any language steering from `language_code`. When you use a custom prompt with non-English audio, state the language as part of your description:

```
Spanish customer support call about a billing inquiry.
```

For multilingual audio, name all expected languages:

```
Multilingual conversation in English, Spanish, and German.
```

If you don't need a custom prompt, prefer the `language_code` config field — see [Language selection](/sync-stt/language-selection).

```python
from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(
    prompt="Cardiology consultation about chest pain symptoms.",
)

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("sample.wav", {
  prompt: "Cardiology consultation about chest pain symptoms.",
});
console.log(result.text);
```

```python
import json
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
        "config": (
            None,
            json.dumps(
                {"prompt": "Cardiology consultation about chest pain symptoms."}
            ),
            "application/json",
        ),
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");
form.append(
  "config",
  new Blob(
    [
      JSON.stringify({
        prompt: "Cardiology consultation about chest pain symptoms.",
      }),
    ],
    { type: "application/json" }
  )
);

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav' \
  -F 'config={"prompt":"Cardiology consultation about chest pain symptoms."};type=application/json'
```

## Keyterms prompting

The `keyterms_prompt` field biases the model toward specific tokens — useful for proper nouns, product names, technical jargon, or any term the model might otherwise mishear.

- Value: an array of strings
- Maximum total length: **2048 characters** across all terms
- Whitespace is stripped and empty terms are ignored

Keyterms are the right tool when you have an explicit vocabulary list — contact names, product catalogs, medical terms — rather than a description of the conversation. Like contextual prompting, keyterms biasing is robust to terms that don't end up being spoken, so you can pass your full list with every request.

**Start with no keyterms**

**We strongly recommend starting with no `keyterms_prompt` and then adding terms as needed** based on important words for your use case that you are consistently seeing the model struggle with.

Including a large number of terms or common terms that are well represented in the training data could lead to overcorrections and hallucinations.

**Best practices:**

- **Specify unique terminology.** Include proper names, company names, technical terms, or vocabulary specific to your domain that might not be commonly recognized.
- **Exact spelling and capitalization.** Provide keyterms with the precise spelling and capitalization you expect to see in the output transcript.
- **Avoid common words.** Do not include single, common English words (e.g., "information") as keyterms. The system is generally proficient with such words, and adding them as keyterms can be redundant.

```python
from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(
    keyterms_prompt=["AssemblyAI", "LeMUR", "U3-Pro"],
)

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("sample.wav", {
  keyterms_prompt: ["AssemblyAI", "LeMUR", "U3-Pro"],
});
console.log(result.text);
```

```python
import json
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
        "config": (
            None,
            json.dumps({"keyterms_prompt": ["AssemblyAI", "LeMUR", "U3-Pro"]}),
            "application/json",
        ),
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");
form.append(
  "config",
  new Blob(
    [JSON.stringify({ keyterms_prompt: ["AssemblyAI", "LeMUR", "U3-Pro"] })],
    { type: "application/json" }
  )
);

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav' \
  -F 'config={"keyterms_prompt":["AssemblyAI","LeMUR","U3-Pro"]};type=application/json'
```

## Using prompt and keyterms together

You can combine `prompt` and `keyterms_prompt` in the same `config` part, with `prompt` describing the conversation and `keyterms_prompt` enumerating specific terms:

```python
from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(
    prompt="Cardiology consultation about chest pain symptoms.",
    keyterms_prompt=["Dr. Smith", "ECG", "hypertension"],
)

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("sample.wav", {
  prompt: "Cardiology consultation about chest pain symptoms.",
  keyterms_prompt: ["Dr. Smith", "ECG", "hypertension"],
});
console.log(result.text);
```

```python
import json
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

config = {
    "prompt": "Cardiology consultation about chest pain symptoms.",
    "keyterms_prompt": ["Dr. Smith", "ECG", "hypertension"],
}

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
        "config": (None, json.dumps(config), "application/json"),
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const config = {
  prompt: "Cardiology consultation about chest pain symptoms.",
  keyterms_prompt: ["Dr. Smith", "ECG", "hypertension"],
};

const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");
form.append(
  "config",
  new Blob([JSON.stringify(config)], { type: "application/json" })
);

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav' \
  -F 'config={"prompt":"Cardiology consultation about chest pain symptoms.","keyterms_prompt":["Dr. Smith","ECG","hypertension"]};type=application/json'
```

## Related features

- **Conversation context** — supply the dialogue that preceded the current clip with the `conversation_context` field, so the model transcribes multi-turn conversations with better continuity and proper-noun consistency. See [Conversation context](/sync-stt/conversation-context).
- **Language selection** — steer the default prompt toward one or more languages with `language_code`. See [Language selection](/sync-stt/language-selection).

## Looking for streaming or async prompting?

This guide covers prompting for **Sync STT (short pre-recorded clips)**. For live audio, see [Prompting and keyterms (Real-time)](/streaming/prompting-and-keyterms). For longer pre-recorded audio, see the [Prompting Guide (Async)](/pre-recorded-audio/universal-3-5-pro/prompting).

---

<a id="conversation-context-docs-sync-stt-conversation-context-mdx"></a>

# Conversation context

URL: https://www.assemblyai.com/docs/sync-stt/conversation-context
Source: docs/sync-stt/conversation-context.mdx
Navigation: Sync STT > Features
Description: Improve multi-turn transcription accuracy by passing prior dialogue with each request.

The `conversation_context` field lets you supply dialogue that preceded the audio provided in the current request. It gives the model continuity across a multi-turn conversation (for example, a human speaking with a voice agent) to boost transcription accuracy, while still transcribing only the current turn.

Include turns from either side of the conversation as separate list items in chronological order (oldest first, most recent last), where a single string is treated as one turn.

- Value: an array of strings (one per turn) or a single string (treated as one turn)
- Maximum: **100 turns / 4096 characters total** — context over either cap is trimmed, not rejected
- If the context exceeds a cap or the model's context window, the oldest turns (those at the start of the list) are dropped first
- Entries carry no speaker labels — include the agent's replies and the caller's utterances as separate entries

The example below passes the three previous turns of the conversation, where the current audio being transcribed is the caller responding with their order number.

```python
from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(
    conversation_context=[
        "Hi, thank you for calling. How can I help?",
        "I'd like to check on the status of my order.",
        "Got it. Do you know your order ID number?",
    ],
)

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("sample.wav", {
  conversation_context: [
    "Hi, thank you for calling. How can I help?",
    "I'd like to check on the status of my order.",
    "Got it. Do you know your order ID number?",
  ],
});
console.log(result.text);
```

```python
import json
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

config = {
    "conversation_context": [
        "Hi, thank you for calling. How can I help?",
        "I'd like to check on the status of my order.",
        "Got it. Do you know your order ID number?",
    ],
}

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
        "config": (None, json.dumps(config), "application/json"),
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const config = {
  conversation_context: [
    "Hi, thank you for calling. How can I help?",
    "I'd like to check on the status of my order.",
    "Got it. Do you know your order ID number?",
  ],
};

const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");
form.append(
  "config",
  new Blob([JSON.stringify(config)], { type: "application/json" })
);

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav' \
  -F 'config={"conversation_context":["Hi, thank you for calling. How can I help?","I'"'"'d like to check on the status of my order.","Got it. Do you know your order ID number?"]};type=application/json'
```

## Combining with other features

`conversation_context` composes with [prompting and keyterms](/sync-stt/prompting-and-keyterms) in the same `config` part — describe the call with `prompt`, list explicit vocabulary with `keyterms_prompt`, and pass the preceding dialogue with `conversation_context`.

## Looking for streaming context?

For live audio, Real-time STT carries context across turns automatically within a session, and voice agents can interleave their replies with the `agent_context` parameter. See [Context carryover (Real-time)](/streaming/universal-3-5-pro/context-carryover).

---

<a id="language-selection-docs-sync-stt-language-selection-mdx"></a>

# Language selection

URL: https://www.assemblyai.com/docs/sync-stt/language-selection
Source: docs/sync-stt/language-selection.mdx
Navigation: Sync STT > Features
Description: Steer Sync STT transcription toward one or more languages with the language_code field.

Pass a `language_code` in the `config` part to steer transcription toward a specific language. For multilingual or code-switching audio, pass an array of codes instead of a single code.

- Accepts a single ISO 639-1 code (`"es"`) or a list of codes (`["en", "es"]`)
- Defaults to English (`en`) when omitted
- Ignored when a custom `prompt` is set — state the language as part of your contextual prompt if you use both (see [Specifying the language](/sync-stt/prompting-and-keyterms#specifying-the-language))

In the SDKs the equivalent field is `language_codes` (on `SyncTranscriptionConfig` in Python, on the config object in JavaScript), which always takes a list — pass a single-element list such as `["es"]` for monolingual audio.

```python
from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(language_codes=["es"])

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("sample.wav", {
  language_codes: ["es"],
});
console.log(result.text);
```

```python
import json
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
        "config": (None, json.dumps({"language_code": "es"}), "application/json"),
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");
form.append(
  "config",
  new Blob([JSON.stringify({ language_code: "es" })], {
    type: "application/json",
  })
);

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav' \
  -F 'config={"language_code":"es"};type=application/json'
```

For multilingual or code-switching audio, pass `language_code` as an array listing every expected language:

```json
{ "language_code": ["en", "es"] }
```

With the SDKs, list the same codes: `SyncTranscriptionConfig(language_codes=["en", "es"])` in Python, or `{ language_codes: ["en", "es"] }` in JavaScript.

## Supported languages

| Code | Language | Code | Language |
|------|----------|------|----------|
| `en` | English | `da` | Danish |
| `es` | Spanish | `fi` | Finnish |
| `de` | German | `hi` | Hindi |
| `fr` | French | `vi` | Vietnamese |
| `it` | Italian | `ar` | Arabic |
| `pt` | Portuguese | `he` | Hebrew |
| `tr` | Turkish | `ja` | Japanese |
| `nl` | Dutch | `ur` | Urdu |
| `sv` | Swedish | `zh` | Mandarin |
| `no` | Norwegian | | |

---

<a id="word-timestamps-docs-sync-stt-word-timestamps-mdx"></a>

# Word timestamps

URL: https://www.assemblyai.com/docs/sync-stt/word-timestamps
Source: docs/sync-stt/word-timestamps.mdx
Navigation: Sync STT > Features
Description: Get exact per-word start and end times with your synchronous transcript.

The `timestamps` field controls whether each word in the response carries `start` and `end` times (in milliseconds, relative to the start of the audio).

- Value: a boolean, default `false`
- `false` (default): no timestamps are computed, and `start`/`end` are omitted from the `words` objects (`None` on each `SyncWord` in the Python SDK) — there is no extra processing cost
- `true`: exact timings are computed for each word, at an additional latency cost

Timings are exact or absent, never approximate: any word that can't be aligned has `start`/`end` omitted rather than estimated. Timestamp accuracy is best for English audio.

```python
from assemblyai.sync.v1 import SyncTranscriber, SyncTranscriptionConfig

config = SyncTranscriptionConfig(timestamps=True)

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>", config=config)
result = transcriber.transcribe("sample.wav")
for word in result.words:
    print(f"{word.start} - {word.end}  {word.text}")
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("sample.wav", {
  timestamps: true,
});
for (const word of result.words) {
  console.log(`${word.start} - ${word.end}  ${word.text}`);
}
```

```python
import json
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

config = {"timestamps": True}

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
        "config": (None, json.dumps(config), "application/json"),
    },
    timeout=60,
)
response.raise_for_status()
for word in response.json()["words"]:
    print(f'{word["start"]:>6} - {word["end"]:>6}  {word["text"]}')
```

```javascript
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const config = { timestamps: true };

const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");
form.append(
  "config",
  new Blob([JSON.stringify(config)], { type: "application/json" })
);

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
for (const word of result.words) {
  console.log(`${word.start} - ${word.end}  ${word.text}`);
}
```

```bash
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav' \
  -F 'config={"timestamps":true};type=application/json'
```

With `timestamps` enabled, each entry in `words` includes its timing:

```json
{
  "text": "Hi, I'm calling about my Best Buy order...",
  "words": [
    { "text": "Hi",  "start": 0,   "end": 200, "confidence": 0.91 },
    { "text": "I'm", "start": 220, "end": 320, "confidence": 0.88 }
  ],
  "confidence": 0.87,
  "audio_duration_ms": 101567,
  "session_id": "eb92c4ff-4bbb-429f-9b99-7279d7fe738f"
}
```

## Combining with other features

`timestamps` composes with the other `config` fields — [prompting and keyterms](/sync-stt/prompting-and-keyterms), [conversation context](/sync-stt/conversation-context), and [language selection](/sync-stt/language-selection) — in the same request.

---

<a id="transcribe-a-short-audio-file-docs-api-reference-sync-api-transcribe-mdx"></a>

# Transcribe a short audio file

URL: https://www.assemblyai.com/docs/api-reference/sync-api/transcribe
Source: docs/api-reference/sync-api/transcribe.mdx
Navigation: Sync STT > API reference
Description: Synchronously transcribe an audio file up to 120 seconds.

Metadata:
- openapi: ../specs/sync-api.yaml POST /transcribe

---

<a id="pre-warm-a-connection-docs-api-reference-sync-api-warm-mdx"></a>

# Pre-warm a connection

URL: https://www.assemblyai.com/docs/api-reference/sync-api/warm
Source: docs/api-reference/sync-api/warm.mdx
Navigation: Sync STT > API reference
Description: Establish the HTTPS connection ahead of a /transcribe request to take the TLS handshake off the critical path.

Metadata:
- openapi: ../specs/sync-api.yaml GET /warm

---

<a id="audio-requirements-docs-sync-stt-audio-requirements-mdx"></a>

# Audio requirements

URL: https://www.assemblyai.com/docs/sync-stt/audio-requirements
Source: docs/sync-stt/audio-requirements.mdx
Navigation: Sync STT > Advanced
Description: Duration, size, format, and sample-rate constraints for Sync STT audio.

The Sync API accepts short audio clips with the following constraints:

| Constraint | Value |
|------------|-------|
| Minimum duration | 80 ms |
| Maximum duration | 120 s |
| Maximum file size | 40 MB |
| Sample width | 16-bit only |
| Channels | Mono or stereo (stereo is down-mixed internally) |
| Sample rates | 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz |
| Formats | WAV (`audio/wav`) or raw PCM S16LE (`audio/pcm`) |

For WAV, the sample rate and channel count are read from the file header. For raw PCM, pass `sample_rate` and `channels` in the `config` part — see [Sending raw PCM audio](/sync-stt/getting-started/transcribe-a-short-audio-file#sending-raw-pcm-audio).

Audio that violates a constraint is rejected with a `400`, `413`, or `415` — see [Error handling](/sync-stt/error-handling) for the full error table.

For audio longer than 120 seconds, use [Pre-recorded STT](/pre-recorded-audio/getting-started/transcribe-an-audio-file). For live audio, use [Real-time STT](/streaming/getting-started/transcribe-streaming-audio).

---

<a id="error-handling-docs-sync-stt-error-handling-mdx"></a>

# Error handling

URL: https://www.assemblyai.com/docs/sync-stt/error-handling
Source: docs/sync-stt/error-handling.mdx
Navigation: Sync STT > Advanced
Description: Sync STT error responses, status codes, and retry guidance.

Errors return a JSON body with either an `error_code` field (audio, capacity, and inference errors) or a `detail` field (auth and rate-limit errors).

```json
{ "error_code": "audio_too_large", "message": "audio duration 130000 ms exceeds limit 120000 ms" }
```

| HTTP | `error_code` | Cause |
|------|-------------|-------|
| 400 | `bad_audio` | Malformed WAV, misaligned PCM, or missing `sample_rate`/`channels` for PCM |
| 400 | `audio_too_short` | Audio below 80 ms |
| 400 | `bad_request` | Missing `audio` part, invalid config JSON, or field limits exceeded |
| 401 | — | Missing or invalid API key |
| 413 | `audio_too_large` | Duration > 120 s or file > 40 MB |
| 415 | `unsupported_media_type` | Unsupported format, non-16-bit audio, or unsupported sample rate |
| 429 | — | Rate limit exceeded — retry after `Retry-After` header |
| 503 | `capacity_exceeded` | Server at rate limit cap — retry after `Retry-After: 1` |
| 503 | `service_unavailable` | Model still loading (cold start) |
| 504 | `inference_timeout` | Request exceeded the 30 s deadline |
| 500 | `inference_error` | Internal model error |

## Retry guidance

- **429 and 503** are transient — retry after the `Retry-After` header (or a short backoff when absent).
- **400, 413, and 415** indicate a problem with the request itself — fix the audio or config before retrying. See [Audio requirements](/sync-stt/audio-requirements) for the constraints.
- **500 and 504** are safe to retry once; persistent failures warrant a support ticket.

## Need help?

If you get stuck, contact our support team at [support@assemblyai.com](mailto:support@assemblyai.com) or create a [support ticket](https://www.assemblyai.com/contact/support). Include the `session_id` from the response (or the failing request's timestamp and endpoint if no response was returned) to help us look up your request.

---

<a id="cloud-endpoints-data-residency-docs-sync-stt-endpoints-and-data-zones-mdx"></a>

# Cloud endpoints & data residency

URL: https://www.assemblyai.com/docs/sync-stt/endpoints-and-data-zones
Source: docs/sync-stt/endpoints-and-data-zones.mdx
Navigation: Sync STT > Advanced
Description: AssemblyAI Sync STT endpoints for global routing and US/EU data residency.

Choose the endpoint that best fits your application's requirements — whether that's using the default endpoint to route to the nearest data center, or ensuring your audio data stays within a given data residency zone such as EU or US.

## Endpoints

| Endpoint | URL | Region |
|----------|-----|--------|
| Global (default) | `https://sync.assemblyai.com/transcribe` | Nearest region |
| US data residency | `https://sync.us.assemblyai.com/transcribe` | us-west-2, us-east-1 |
| EU data residency | `https://sync.eu.assemblyai.com/transcribe` | eu-north-1 |

## Which endpoint should I use?

- **Default?** Use the **Global endpoint** (`sync.assemblyai.com`). It will route your request to the region with the fastest response time for you.
- **Need US data residency?** Use the **US endpoint** (`sync.us.assemblyai.com`). Your audio and transcription data will remain within the US.
- **Need EU data residency?** Use the **EU endpoint** (`sync.eu.assemblyai.com`). Your audio and transcription data will remain within the EU.

## How to use it

Replace the base URL in your request with the appropriate endpoint. The request format, headers, and parameters are identical for all endpoints. In the Python SDK the base URL comes from the `sync_base_url` setting; in the JavaScript SDK it's the `syncBaseUrl` client option. Both default to the global endpoint.

```python title="Global (default)"
from assemblyai.sync.v1 import SyncTranscriber

transcriber = SyncTranscriber(api_key="<YOUR_API_KEY>")
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```python title="US data residency"
from assemblyai import Client, Settings
from assemblyai.sync.v1 import SyncTranscriber

client = Client(
    settings=Settings(
        api_key="<YOUR_API_KEY>",
        sync_base_url="https://sync.us.assemblyai.com",
    ),
)

transcriber = SyncTranscriber(client=client)
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```python title="EU data residency"
from assemblyai import Client, Settings
from assemblyai.sync.v1 import SyncTranscriber

client = Client(
    settings=Settings(
        api_key="<YOUR_API_KEY>",
        sync_base_url="https://sync.eu.assemblyai.com",
    ),
)

transcriber = SyncTranscriber(client=client)
result = transcriber.transcribe("sample.wav")
print(result.text)
```

```javascript title="Global (default)"
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const result = await client.sync.transcribe("sample.wav");
console.log(result.text);
```

```javascript title="US data residency"
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: process.env.ASSEMBLYAI_API_KEY,
  syncBaseUrl: "https://sync.us.assemblyai.com",
});

const result = await client.sync.transcribe("sample.wav");
console.log(result.text);
```

```javascript title="EU data residency"
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: process.env.ASSEMBLYAI_API_KEY,
  syncBaseUrl: "https://sync.eu.assemblyai.com",
});

const result = await client.sync.transcribe("sample.wav");
console.log(result.text);
```

```python title="Global (default)"
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
    },
    timeout=60,
)
response.raise_for_status()
result = response.json()
print(result["text"])
```

```python title="US data residency"
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.us.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
    },
    timeout=60,
)
response.raise_for_status()
result = response.json()
print(result["text"])
```

```python title="EU data residency"
import requests

with open("sample.wav", "rb") as f:
    audio = f.read()

response = requests.post(
    "https://sync.eu.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
    },
    timeout=60,
)
response.raise_for_status()
result = response.json()
print(result["text"])
```

```javascript title="Global (default)"
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");

const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```javascript title="US data residency"
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");

const response = await fetch("https://sync.us.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```javascript title="EU data residency"
import { readFileSync } from "fs";

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");

const response = await fetch("https://sync.eu.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```bash title="Global (default)"
curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav'
```

```bash title="US data residency"
curl -X POST https://sync.us.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav'
```

```bash title="EU data residency"
curl -X POST https://sync.eu.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav'
```

---

<a id="connection-pre-warming-docs-sync-stt-connection-pre-warming-mdx"></a>

# Connection pre-warming

URL: https://www.assemblyai.com/docs/sync-stt/connection-pre-warming
Source: docs/sync-stt/connection-pre-warming.mdx
Navigation: Sync STT > Advanced
Description: Use the /warm endpoint to take the TLS handshake off the critical path of a synchronous transcription request.

Sync STT is a single request/response: your application sends audio to `/transcribe` and gets the transcript back in the same HTTP exchange. That makes connection setup part of your latency budget — if the HTTPS connection isn't open yet, the very first thing your request does is negotiate one, and your audio doesn't start uploading until that negotiation finishes.

The `GET /warm` endpoint exists to take that setup off the critical path. Calling it as soon as you know audio is coming — typically the moment recording starts — establishes the connection in the background, so the eventual `/transcribe` request starts uploading audio immediately.

## What a cold request pays for

Before an HTTPS client can send the first byte of a request to a host it isn't connected to, three things happen in sequence:

1. **DNS resolution** — the client resolves `sync.assemblyai.com` to an IP address. Usually fast (often cached), but a cold resolver lookup adds a round trip of its own.
2. **TCP handshake** — the client and server exchange SYN / SYN-ACK / ACK to open the socket. This costs **one network round trip**.
3. **TLS handshake** — the encrypted session is negotiated on top of the socket. The client sends a `ClientHello` with its supported cipher suites and key share; the server responds with its certificate and its own key share; the client verifies the certificate chain and both sides derive the session keys. On TLS 1.3 this costs **one additional round trip** (TLS 1.2 needs two).

Only after all three complete does the actual `POST /transcribe` — headers, then audio — go on the wire. None of this work depends on your audio; it's pure connection plumbing, but it sits in front of every cold request.

The cost scales directly with your network distance to the endpoint: each step is measured in round trips, so the farther you are from the serving region, the more the handshake costs. For a client near the serving region the total is typically a few tens of milliseconds; for a distant or intercontinental client it can add well over 100 ms — a meaningful fraction of the transcription itself for short clips.

## How `/warm` works

`GET /warm` is an unauthenticated no-op:

```json
{ "warm": "toasty" }
```

The response body is irrelevant — the value of the call is entirely on the wire. Making any request forces your HTTP client to resolve DNS, open the TCP socket, and complete the TLS handshake. The resulting connection goes into your client's connection pool, and a `/transcribe` request sent shortly afterward through the **same client** reuses it as-is: no DNS, no TCP handshake, no TLS handshake — the request headers and audio bytes start flowing immediately.

Include the same `X-AAI-Model` header you'll use for transcription, so the warmed path matches the one your `/transcribe` request will take. No API key is required.

## Why "right before" matters

Pooled connections don't live forever:

- **Client-side pool expiry** — HTTP clients evict idle connections from their pool, some aggressively (for example, `httpx` drops idle connections after 5 seconds by default).
- **Server-side idle timeout** — the endpoint closes connections that stay idle for a few minutes.

If the connection has been evicted or closed by the time you call `/transcribe`, the request silently opens a fresh one and pays the full handshake anyway — the warm-up bought nothing.

So the ideal moment to call `/warm` is when you *know* audio is coming but don't have it yet — in a voice application, that's the moment the user starts speaking. The handshake then runs concurrently with recording, and by the time the utterance ends the connection is open and fresh. `/warm` is idempotent and extremely cheap, so calling it again to refresh an aging connection is fine.

Pre-warming only helps if the `/warm` and `/transcribe` requests share a connection pool. Use the same client object (a `requests.Session` in Python, the same process in Node.js) for both calls, and the same base URL — a connection warmed against one [endpoint](/sync-stt/endpoints-and-data-zones) doesn't help a request sent to another.

## How to use it

```python
import assemblyai as aai
from assemblyai.sync.v1 import SyncTranscriber

# Hold the idle connection long enough to cover the whole recording.
aai.settings.keepalive_expiry = 120

with SyncTranscriber(api_key="<YOUR_API_KEY>") as transcriber:
    # Call as soon as you know audio is coming (e.g. recording starts).
    # The handshake completes while the audio is still being captured.
    transcriber.warm()

    # ... recording finishes ...

    # Same transcriber -> reuses the warmed connection; no handshake here.
    result = transcriber.transcribe("sample.wav")
    print(result.text)
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

// Call as soon as you know audio is coming (e.g. recording starts).
// The handshake completes while the audio is still being captured.
// warm() resolves to a boolean and never throws — it returns false if the
// connection couldn't be opened.
await client.sync.warm();

// ... recording finishes ...

// Same client -> reuses the warmed connection; no handshake here.
const result = await client.sync.transcribe("sample.wav");
console.log(result.text);
```

```python
import requests

session = requests.Session()

# Call as soon as you know audio is coming (e.g. recording starts).
# The handshake completes while the audio is still being captured.
session.get(
    "https://sync.assemblyai.com/warm",
    headers={"X-AAI-Model": "universal-3-5-pro"},
    timeout=10,
)

# ... recording finishes ...

with open("sample.wav", "rb") as f:
    audio = f.read()

# Same session -> reuses the warmed connection; no handshake here.
response = session.post(
    "https://sync.assemblyai.com/transcribe",
    headers={
        "Authorization": "<YOUR_API_KEY>",
        "X-AAI-Model": "universal-3-5-pro",
    },
    files={
        "audio": ("sample.wav", audio, "audio/wav"),
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["text"])
```

```javascript
import { readFileSync } from "fs";

// Call as soon as you know audio is coming (e.g. recording starts).
// Node's fetch keeps the connection alive in its pool by default.
await fetch("https://sync.assemblyai.com/warm", {
  headers: { "X-AAI-Model": "universal-3-5-pro" },
});

// ... recording finishes ...

const audio = readFileSync("sample.wav");
const form = new FormData();
form.append("audio", new Blob([audio], { type: "audio/wav" }), "sample.wav");

// Same process -> reuses the warmed connection; no handshake here.
const response = await fetch("https://sync.assemblyai.com/transcribe", {
  method: "POST",
  headers: {
    Authorization: "<YOUR_API_KEY>",
    "X-AAI-Model": "universal-3-5-pro",
  },
  body: form,
});

const result = await response.json();
console.log(result.text);
```

```bash
# Each curl process opens its own connection, so a separate warm-up call
# doesn't help a later invocation. Chain both requests in one invocation
# with --next to share the connection:
curl https://sync.assemblyai.com/warm \
  -H 'X-AAI-Model: universal-3-5-pro' \
  --next \
  -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: <YOUR_API_KEY>' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav'
```

The SDKs expose this directly — `SyncTranscriber.warm()` in Python, `client.sync.warm()` in JavaScript. Call it when recording starts, then call `transcribe()` as usual on the same transcriber or client.

---

<a id="voice-agent-api-docs-voice-agents-voice-agent-api-mdx"></a>

# Voice Agent API

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api
Source: docs/voice-agents/voice-agent-api.mdx
Navigation: Voice Agent API > Getting started
Description: Real-time voice agents that talk and listen. Configure an agent once, then deploy it to a browser, your own app, or the phone.

Talk to a voice agent in your browser in four commands, then put it on a phone number.

  
    Python 3.9+, standard library only.
  
  
    Node 18+, no dependencies.
  

## Step 1: Clone

```bash Python
git clone https://github.com/AssemblyAI/voice-agent-starter-python
cd voice-agent-starter-python
cp .env.example .env
```

```bash JavaScript
git clone https://github.com/AssemblyAI/voice-agent-starter-js
cd voice-agent-starter-js
cp .env.example .env
```

## Step 2: Add your key

Paste your [API key](https://www.assemblyai.com/dashboard/api-keys) into `.env`:

```bash .env
ASSEMBLYAI_API_KEY=<your-key>
```

## Step 3: Publish the starter agent

```bash Python
python publish.py
```

```bash JavaScript
npm run publish
```

Publishes the starter agent to your account and saves its ID for the next step.

## Step 4: Talk to it

```bash Python
python deployment/browser/server.py
```

```bash JavaScript
npm start
```

Open `http://localhost:3000` in Chrome or Edge and start talking.

## Put it on a phone number

Twilio passes calls to AssemblyAI over SIP, so there's no media server or webhook to run. Buy a Twilio number, then add your credentials and a trunk domain you invent:

```bash .env
TWILIO_ACCOUNT_SID=AC...                          # console.twilio.com, top of the page
TWILIO_AUTH_TOKEN=your_token_here                 # same place, hidden until you click it
TWILIO_PHONE_NUMBER=+1...                         # a number already in your account, E.164
TWILIO_TRUNK_DOMAIN=acme-agent.pstn.twilio.com    # a name you invent, unique across Twilio
```

```bash Python
python deployment/telephony/connect.py
```

```bash JavaScript
npm run phone
```

That creates the SIP trunk on your domain, routes it to AssemblyAI, attaches your number to it, and binds your agent. Every step checks for existing state first, so it's safe to re-run.

Call the number.

## Next steps

Every file in `agents/` is the request body for [`POST /v1/agents`](/voice-agents/voice-agent-api/api-spec/create-agent), sent unchanged. The starter agent is four fields:

```json agents/minimal.jsonc
{
  "name": "Minimal agent",
  "system_prompt": "You are a friendly assistant on a voice call. Keep every reply to one or two short sentences. Answer what was asked, lead with the answer, and skip the preamble. If you don't know something, say so. No exclamation marks.",
  "voice": { "voice_id": "anna" },
  "greeting": "Hey, what can I do for you?"
}
```

Copy the closest file to start your own (`cp agents/http-tools.jsonc agents/my-agent.jsonc`). Every field is commented. Publish again and both the browser tab and the phone number pick up the change on the next call.

Nine agents ship with each starter. Set `AGENT=` to pick one:

| `AGENT=` | Demonstrates | Requires |
| --- | --- | --- |
| `minimal` | The three required fields, and the defaults applied to the rest | |
| `keyterms` | Biasing transcription toward names and jargon | |
| `turn-taking` | Silence thresholds and interruption handling | |
| `byo-llm` | Claude through the AssemblyAI gateway, or your own endpoint | |
| `http-tools` | Tools that AssemblyAI calls on the agent's behalf | |
| `exa-search` | Web search during a call | `EXA_API_KEY` |
| `airtable-crm` | Reading a caller record and writing one back | `AIRTABLE_*` |
| `cal-booking` | Checking availability, then booking a slot | `CAL_*` |
| `dtmf` | PCI compliance: card entry on the keypad, never in the transcript, the logs, or the model | `DTMF_WEBHOOK_URL` |

```bash Python
AGENT=cal-booking python publish.py
python deployment/browser/server.py
```

```bash JavaScript
AGENT=cal-booking npm run publish
npm start
```

Values written as `${VAR}` in an agent file are substituted at publish time from `.env`, or from `agents/<name>.env` for credentials only one agent uses. Both are gitignored, so the JSON is safe to commit.

Deploy the [Python](https://render.com/deploy?repo=https://github.com/AssemblyAI/voice-agent-starter-python) or [JavaScript](https://render.com/deploy?repo=https://github.com/AssemblyAI/voice-agent-starter-js) starter to Render. It prompts for `ASSEMBLYAI_API_KEY` and nothing else.

Anyone with the URL can start sessions billed to that key.

  
    Create a reusable agent with one REST call, then update, list, and delete it.
  
  
    Shape how it sounds and behaves: prompt, voice, greeting, audio, turn detection, keyterms, tools.
  
  
    Connect by `agent_id` over the API, from a browser, or to a phone number with Twilio.
  

Or jump straight to a topic:

- [Add tools](/voice-agents/voice-agent-api/tools/overview): server-side HTTP tools and client-side function tools
- [Connect your own LLM](/voice-agents/voice-agent-api/connect-your-own-llm): point the agent at an OpenAI-compatible model
- [Manage agents (REST)](/voice-agents/voice-agent-api/manage-agents): every endpoint, field, and validation rule
- [Browser integration](/voice-agents/voice-agent-api/browser-integration): mint tokens and build your own client
- [Phone agents over SIP](/voice-agents/voice-agent-api/connect-to-twilio): the phone step above in full, including a Twilio CLI path
- [Events reference](/voice-agents/voice-agent-api/events-reference): every WebSocket event with full payloads
- [Build with AI coding tools](/voice-agents/voice-agent-api/build-with-ai-tools): point Claude Code, Cursor, or v0 at these docs
- [Troubleshooting](/voice-agents/voice-agent-api/troubleshooting): symptom-to-fix table and support logging

No code at all? Talk to an agent in the [Voice Agent playground](https://www.assemblyai.com/dashboard/playground/voice-agent).

---

<a id="build-with-ai-coding-tools-docs-voice-agents-voice-agent-api-build-with-ai-tools-mdx"></a>

# Build with AI coding tools

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/build-with-ai-tools
Source: docs/voice-agents/voice-agent-api/build-with-ai-tools.mdx
Navigation: Voice Agent API > Getting started
Description: Point your AI coding assistant at AssemblyAI's live docs so it writes correct, current Voice Agent code instead of guessing from stale training data.

Building your voice agent with Claude Code, Cursor, Windsurf, Codex, or a prompt-based builder like Lovable or v0? Give the tool access to AssemblyAI's live docs and a system prompt that encodes the non-obvious gotchas, and it writes code that runs the first time instead of hallucinating fields.

## Connect the docs

Point your AI coding agent at AssemblyAI's live docs so it writes correct, current code instead of guessing from stale training data.

**Claude Code, Cursor, Windsurf, Codex, or any MCP client** — add the AssemblyAI docs MCP server, `https://mcp.assemblyai.com/docs`. For Claude Code:

```bash
claude mcp add --transport http --scope user assemblyai-docs https://mcp.assemblyai.com/docs
npx skills add AssemblyAI/assemblyai-skill --global
```

**Lovable, v0, and other prompt-based builders** — link the docs in your prompt, e.g. `https://www.assemblyai.com/docs/voice-agents/voice-agent-api`.

Then describe what you want. To build the browser agent from the [Quickstart](/voice-agents/voice-agent-api), paste:

```text
Using the AssemblyAI Voice Agent API, create an agent, then give me a single-page browser app that connects to it by agent ID and lets me talk to it, with echo cancellation on.
```

## Give it a system prompt

Drop the prompt below into your assistant's system prompt or rules file. It encodes the non-obvious gotchas the API reference doesn't lead with, points the assistant at the right reference pages for everything else, and gives it sensible defaults for audio, turn detection, and tool design.

````markdown
# Voice Agent API: AI Assistant System Prompt

> Use this as the system prompt for your AI coding assistant (Claude Code, Cursor, Windsurf, etc.) when building with AssemblyAI's Voice Agent API. It encodes the non-obvious gotchas the API reference doesn't emphasize and points your assistant to the right docs pages for everything else.

## Role

You are an expert pair-programmer helping me build a real-time voice agent using **AssemblyAI's Voice Agent API**. Optimize for code that runs, with the smallest set of features that solves my problem.

**Default to a browser app** unless I tell you otherwise. Browsers give you AEC (acoustic echo cancellation) for free, which solves the single biggest source of broken voice agents: the agent hearing its own TTS and interrupting itself. Twilio phone agents (natively supported) and native mobile clients are also valid; if I'm going that route, plan for AEC server-side or require headphones.

**The docs are the source of truth.** Don't re-derive things from memory. When you need a payload, error code, voice ID, or config field that isn't in this prompt, WebFetch the relevant page from the docs map at the bottom. This prompt only encodes the gotchas and opinionated defaults that the reference docs don't make obvious; everything else, look up.

## Seven non-obvious things about this API

1. **Audio is PCM16 mono at 24 kHz, base64-encoded.** In the browser, force this with `new AudioContext({ sampleRate: 24000 })` so nothing resamples. Default to Chrome/Edge. Safari ignores the constructor's `sampleRate` and needs manual resampling.

2. **Don't send `input.audio` before `session.ready`.** Buffer or drop early frames.

3. **`greeting` and `output.voice` are immutable after `session.ready`. `system_prompt`, `input.turn_detection`, `input.keyterms`, and `tools` are mutable.** Send another `session.update` with only the fields you're changing.

4. **Tool result: send it the moment your tool returns.** No buffering, no waiting on `reply.done`, no special timing dance. The agent fills the gap with a transition phrase while your tool runs; as soon as you ship `tool.result` the agent generates its next reply using the result. The `arguments` on `tool.call` is already a parsed object. The `result` on `tool.result` must be `JSON.stringify(value)`, not an object. Always echo the original `call_id`. Envelopes for reference:

   ```
   → { type:"tool.call",   call_id:"c_123", name:"get_weather", arguments:{ location:"London" } }
   ← (run your tool)
   → { type:"tool.result", call_id:"c_123", result:"{\"temp_c\":22}" }
   ```

5. **On barge-in (`reply.done` with `status: "interrupted"`), flush the audio buffer immediately.** Stop the current `AudioBufferSourceNode`, clear the queue, reset `nextStartTime` to `audioCtx.currentTime`. Otherwise the user hears another second of stale TTS after they interrupt. Bonus: flushing on `input.speech.started` (not waiting for `reply.done`) makes barge-in feel ~300 ms snappier.

6. **In the browser, mint a short-lived token server-side** and pass it as `?token=...` on the WebSocket URL. Never expose the raw API key in client-side code.

7. **Send `session.end` before closing the socket on intentional disconnects.** If you just close the WebSocket, the server holds the session for 30 seconds so you can `session.resume` — and that window is billable. `session.end` short-circuits the grace window, emits a final `session.ended`, and closes the socket. Skip it only when the disconnect is unintentional and you want the option to resume.

## Browser audio: exact `getUserMedia` constraints

```js
navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,    // ON. Stops self-interruption loops.
    noiseSuppression: false,   // OFF. Voice Focus runs server-side; double-stacking hurts ASR.
    autoGainControl: true,     // ON. Gentle volume normalization.
  },
});
```

The non-obvious one is `noiseSuppression: false`. Browser noise suppression and AssemblyAI's server-side Voice Focus are independent passes; running both eats real speech and degrades recognition in noisy rooms. Trust the server.

## Turn detection: recommended defaults

The factory defaults cut users off too fast in real conversation. Start with:

```js
session.update({ input: { turn_detection: {
  vad_threshold: 0.5, min_silence: 1400, max_silence: 4000, interrupt_response: true
}}});
```

Erring 200-400 ms long is barely perceptible; erring short feels rude. For dictation or list-reading where pauses are structural, push `min_silence` to 1800-2200. Set `interrupt_response: false` for read-aloud / monologue agents only.

### Adaptive pattern: slow down after the agent asks a question

When `transcript.agent` ends in `?`, bump silence thresholds so the user has time to think, then revert on the next `transcript.user`:

```js
let baseline = { vad_threshold: 0.5, min_silence: 1400, max_silence: 4000, interrupt_response: true };
let waitingForAnswer = false;
const setTD = td => ws.send(JSON.stringify({ type:"session.update", session:{ input:{ turn_detection: td }}}));

ws.onmessage = (ev) => {
  const m = JSON.parse(ev.data);
  if (m.type === "transcript.agent" && /\?\s*$/.test(m.text || "")) {
    waitingForAnswer = true;
    setTD({ ...baseline, min_silence: 2200, max_silence: 6000 });
  }
  if (m.type === "transcript.user" && waitingForAnswer) {
    waitingForAnswer = false;
    setTD(baseline);
  }
};
```

Same idea applies for other "thinking moments": after the agent reads a long menu, after "take your time", or after a tool result the user needs to react to.

## Tools: when, and when not

**Use a tool when:** the agent needs external data or to take an external action, AND the result must influence what it says next. Pattern is: agent decides → tool runs → result fed back → agent speaks an informed reply.

**Do NOT use a tool for:**

- **Logging or analytics.** You already get every word via `transcript.user` and `transcript.agent`. Log those directly. A `log_event` tool just adds an LLM round-trip.
- **Extraction, summarization, classification of what was said.** Don't make the agent call `extract_order` mid-turn. Collect the transcript events and run a single AssemblyAI [LLM Gateway](/llm-gateway/quickstart) call against the finished (or in-progress) transcript when you actually need the structured output. The voice loop stays fast and you get to use a bigger model for the extraction step.
- **Persona or state changes the *client* can decide.** Prefer a `session.update` from your code (on a UI button, keyword, or transcript regex) over a `change_persona` tool the LLM has to remember to call.

Every extra tool is a chance for the agent to call it at the wrong moment. Ship with the smallest set that earns its keep.

### Writing tool descriptions

Treat `description` and each parameter `description` as code, not docs:

- One sentence per tool. Lead with the action verb + trigger condition: *"Get the current weather for a city. Use when the user asks about weather or conditions in a specific place."*
- Spell out the return shape and units.
- Give each parameter an example value: *"location: city only, no country, e.g. 'London'."*
- Use `enum` aggressively on string params; removes "model invented a category" bugs.
- If a description needs more than 3 sentences, the tool is doing too much. Split it or shrink it.

### Pair `keyterms` with any lookup tool

If you have a `lookup_company` tool, push the candidate company names into `input.keyterms` so ASR doesn't mangle "Anthropic" into "anthrop pick" before the tool ever sees it. Same for menus, contact lists, drug names, song titles. `keyterms` is mutable; narrow it as scope narrows.

## Voice prompt writing: what's different from chat

- **No markdown.** TTS reads asterisks and bullets literally.
- Front-load the most important rule. Long prompts dilute attention.
- Define identity ("You are X") rather than listing behaviors.
- Give explicit permissions: "Have opinions. Crack jokes if it fits."
- List exact phrases to avoid ("Great question", "Happy to help") instead of saying "be casual."
- Round numbers when speaking: "around 2 in the afternoon," not "2:14 PM."
- No exclamation marks. No decision trees.
- Keep it short to start. Persona is iterated by ear, not by writing more words.

Full guide: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/prompting-guide

## Getting a browser app running

1. Clone the voice agent starter, in [Python](https://github.com/AssemblyAI/voice-agent-starter-python) or [JavaScript](https://github.com/AssemblyAI/voice-agent-starter-js). Both ship an `AGENTS.md` your coding tool reads for repo conventions, and neither has dependencies to install.
2. Put `ASSEMBLYAI_API_KEY` in `.env`, publish an agent (`python publish.py` or `npm run publish`), and start the browser server (`python deployment/browser/server.py` or `npm start`).
3. Open `http://localhost:3000` (localhost counts as a secure context, so the mic works). Edit the agent's JSON file in `agents/`, publish again, and start another call.

Want a single self-contained HTML file instead, with no clone? [Browser integration](/voice-agents/voice-agent-api/browser-integration) has one, plus the server-side token endpoint it needs.

### Minimum-viable playback + flush (if you're not starting from the starter)

```js
const RATE = 24000;
const audioCtx = new AudioContext({ sampleRate: RATE });
let nextStartTime = 0;
const liveSources = new Set();

function playReplyAudio(b64) {
  const raw = atob(b64);
  const pcm = new Int16Array(raw.length / 2);
  for (let i = 0; i < pcm.length; i++) pcm[i] = raw.charCodeAt(i*2) | (raw.charCodeAt(i*2+1) << 8);
  const buf = audioCtx.createBuffer(1, pcm.length, RATE);
  const ch = buf.getChannelData(0);
  for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;
  const src = audioCtx.createBufferSource();
  src.buffer = buf; src.connect(audioCtx.destination);
  const startAt = Math.max(audioCtx.currentTime, nextStartTime);
  src.start(startAt);
  src.onended = () => liveSources.delete(src);
  liveSources.add(src);
  nextStartTime = startAt + buf.duration;
}

function flushPlayback() {
  for (const s of liveSources) { try { s.onended = null; s.stop(0); s.disconnect(); } catch {} }
  liveSources.clear();
  nextStartTime = audioCtx.currentTime;
}
// reply.audio: playReplyAudio(msg.data)
// reply.done w/ status==="interrupted" OR input.speech.started: flushPlayback()
```

## Docs map: where to look for what

When you need something not covered above, WebFetch the right page rather than guessing:

- Full LLM-friendly dump (the firehose): https://www.assemblyai.com/docs/voice-agents/voice-agent-api/llms-full.txt
- Every event payload, every field: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/events-reference
- Every config field, mutability rules: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/session-configuration
- Tool schema, MCP integration: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/overview
- Voice IDs (English + multilingual): https://www.assemblyai.com/docs/voice-agents/voice-agent-api/voices
- Token endpoint, browser auth: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/browser-integration
- Twilio phone agents: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/connect-to-twilio
- Error codes and common failures: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/troubleshooting
- LLM Gateway (transcript extraction / summarization): https://www.assemblyai.com/docs/llm-gateway/quickstart
- LLM Gateway over transcripts (recipe): https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files
- Structured JSON extraction from dialogue: https://www.assemblyai.com/docs/guides/dialogue-data

## Common errors at a glance

The three you'll hit first (full list at the troubleshooting URL above):

- `UNAUTHORIZED` (WebSocket close 1008): bad API key, or token expired before you connected. Mint a fresh token right before opening the socket.
- `invalid_audio`: the `audio` field failed base64 decode or PCM16 conversion. Usually means wrong sample rate, WAV header included, or float32 instead of int16.
- `invalid_format`: message was structurally bad (malformed JSON, missing `type`, missing `audio`). Usually a serialization bug, not an audio bug.

## When in doubt

Ask me one focused question rather than guessing. If audio is off (pitch, echo, latency), it's almost always one of three things: sample rate, AEC, or the interrupt-flush. Check those three first. For anything else, the docs map above is the source of truth.
````

---

<a id="supported-languages-docs-voice-agents-voice-agent-api-supported-languages-mdx"></a>

# Supported languages

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/supported-languages
Source: docs/voice-agents/voice-agent-api/supported-languages.mdx
Navigation: Voice Agent API > Getting started
Description: Input and output languages supported by the Voice Agent API.

The Voice Agent API's input (speech recognition) and output (speech synthesis) cover different sets of languages.

## Input languages

The Voice Agent API uses Universal-3.5 Pro Streaming for speech recognition. It supports the following input languages with native code-switching:

| Language   | Flag |
| ---------- | ---- |
| English    | 🇺🇸 |
| Spanish    | 🇪🇸 |
| German     | 🇩🇪 |
| French     | 🇫🇷 |
| Portuguese | 🇵🇹 |
| Italian    | 🇮🇹 |
| Turkish    | 🇹🇷 |
| Dutch      | 🇳🇱 |
| Swedish    | 🇸🇪 |
| Norwegian  | 🇳🇴 |
| Danish     | 🇩🇰 |
| Finnish    | 🇫🇮 |
| Hindi      | 🇮🇳 |
| Vietnamese | 🇻🇳 |
| Arabic     | 🇸🇦 |
| Hebrew     | 🇮🇱 |
| Japanese   | 🇯🇵 |
| Chinese    | 🇨🇳 |

For more on the underlying streaming model, see [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio).

## Output languages

The Voice Agent API officially supports the following output languages, each backed by at least one [voice](/voice-agents/voice-agent-api/voices) with a matching primary or native accent:

| Language   | Flag | Recommended voices                                                                                    |
| ---------- | ---- | ----------------------------------------------------------------------------------------------------- |
| English    | 🇺🇸 🇬🇧 | `alba`, `eve`, `george`, `jane`, `jean`, `mary`, `michael` (🇺🇸), `anna`, `charles`, `paul`, `vera` (🇬🇧) |
| Italian    | 🇮🇹 | `giovanni`                                                                                            |
| Spanish    | 🇪🇸 | `lola`                                                                                                |
| German     | 🇩🇪 | `juergen`                                                                                             |
| Portuguese | 🇵🇹 | `rafael`                                                                                              |
| French     | 🇫🇷 | `estelle`                                                                                             |

See the [Voices](/voice-agents/voice-agent-api/voices) page for voice descriptions.

### Coming soon

Native-accent voices for the following languages are on the roadmap. If you'd like early access or want to register interest, contact [support@assemblyai.com](mailto:support@assemblyai.com).

| Language   | Flag |
| ---------- | ---- |
| Hindi      | 🇮🇳 |
| Turkish    | 🇹🇷 |
| Dutch      | 🇳🇱 |
| Swedish    | 🇸🇪 |
| Norwegian  | 🇳🇴 |
| Danish     | 🇩🇰 |
| Finnish    | 🇫🇮 |
| Vietnamese | 🇻🇳 |
| Arabic     | 🇸🇦 |
| Hebrew     | 🇮🇱 |
| Japanese   | 🇯🇵 |
| Chinese    | 🇨🇳 |

---

<a id="create-an-agent-docs-voice-agents-voice-agent-api-create-agent-mdx"></a>

# Create an agent

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/create-agent
Source: docs/voice-agents/voice-agent-api/create-agent.mdx
Navigation: Voice Agent API > Create & manage agents
Description: Create a reusable voice agent with a single REST call.

Your agent — its prompt, voice, and tools — is one JSON object AssemblyAI stores for you. Create it once, then [deploy](/voice-agents/voice-agent-api/deploy) it by referencing its `agent_id` from your server, a browser, or a phone call.

**Configure once, deploy anywhere.** You can also configure an agent inline at connect-time over the WebSocket via [`session.update`](/voice-agents/voice-agent-api/session-configuration), which is handy for one-off or fully dynamic agents. But for anything you reuse, a stored agent is simpler: the config lives on the server, secrets stay off the client, and HTTP tools run server-side.

## Create an agent

`POST https://agents.assemblyai.com/v1/agents` with your API key in the `Authorization` header. Only `name`, `system_prompt`, and `voice` are required:

```bash cURL
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep responses under two sentences.",
    "greeting": "Hi, how can I help?",
    "voice": { "voice_id": "alba" }
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent. Keep responses under two sentences.",
        "greeting": "Hi, how can I help?",
        "voice": {"voice_id": "alba"},
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent. Keep responses under two sentences.",
    greeting: "Hi, how can I help?",
    voice: { voice_id: "alba" },
  }),
});
const data = await res.json();
console.log(data);
```

The response includes a generated `id`, which is what you deploy with:

```json
{ "id": "7ad24396-b822-4dca-871a-be9cc4781cf9", "name": "Support Assistant", "...": "..." }
```

Use that id to [deploy your agent](/voice-agents/voice-agent-api/deploy): bind it by `agent_id` over the WebSocket, from a browser, or on a phone number.

## A complete agent

An agent is one JSON object. Here's **every field** in a single create call — copy it and delete what you don't need. Only `name`, `system_prompt`, and `voice` are required; everything else has a sensible default. Use Python or Node to skip cURL's quote-escaping when you edit longer fields like `system_prompt`:

```bash cURL expandable
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "greeting": "Hi, how can I help?",
    "voice": { "voice_id": "alba" },
    "input": {
      "format": { "encoding": "audio/pcm", "sample_rate": 24000 },
      "keyterms": ["AssemblyAI", "Universal-3"],
      "turn_detection": { "vad_threshold": 0.5, "min_silence": 1000, "max_silence": 3000, "interrupt_response": true }
    },
    "output": {
      "voice": "alba",
      "format": { "encoding": "audio/pcm", "sample_rate": 24000 },
      "volume": 100
    },
    "tools": [
      {
        "name": "get_weather",
        "description": "Get current weather for a location.",
        "parameters": {
          "type": "object",
          "properties": { "latitude": { "type": "number" }, "longitude": { "type": "number" } },
          "required": ["latitude", "longitude"]
        },
        "http": { "url": "https://api.example.com/weather", "http_method": "GET" }
      }
    ],
    "llm": [
      { "base_url": "https://llm-gateway.assemblyai.com/v1", "model": "claude-sonnet-4-6", "api_key": "'"$ASSEMBLYAI_API_KEY"'" }
    ]
  }'
```

```python Python expandable
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "greeting": "Hi, how can I help?",
        "voice": {"voice_id": "alba"},
        "input": {
            "format": {"encoding": "audio/pcm", "sample_rate": 24000},
            "keyterms": ["AssemblyAI", "Universal-3"],
            "turn_detection": {
                "vad_threshold": 0.5, "min_silence": 1000,
                "max_silence": 3000, "interrupt_response": True,
            },
        },
        "output": {
            "voice": "alba",
            "format": {"encoding": "audio/pcm", "sample_rate": 24000},
            "volume": 100,
        },
        "tools": [
            {
                "name": "get_weather",
                "description": "Get current weather for a location.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "latitude": {"type": "number"},
                        "longitude": {"type": "number"},
                    },
                    "required": ["latitude", "longitude"],
                },
                "http": {"url": "https://api.example.com/weather", "http_method": "GET"},
            }
        ],
        "llm": [
            {
                "base_url": "https://llm-gateway.assemblyai.com/v1",
                "model": "claude-sonnet-4-6",
                "api_key": os.environ["ASSEMBLYAI_API_KEY"],
            }
        ],
    },
)
resp.raise_for_status()
print(resp.json()["id"])  # the agent_id you connect with
```

```javascript Node.js expandable
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    greeting: "Hi, how can I help?",
    voice: { voice_id: "alba" },
    input: {
      format: { encoding: "audio/pcm", sample_rate: 24000 },
      keyterms: ["AssemblyAI", "Universal-3"],
      turn_detection: {
        vad_threshold: 0.5, min_silence: 1000,
        max_silence: 3000, interrupt_response: true,
      },
    },
    output: {
      voice: "alba",
      format: { encoding: "audio/pcm", sample_rate: 24000 },
      volume: 100,
    },
    tools: [
      {
        name: "get_weather",
        description: "Get current weather for a location.",
        parameters: {
          type: "object",
          properties: { latitude: { type: "number" }, longitude: { type: "number" } },
          required: ["latitude", "longitude"],
        },
        http: { url: "https://api.example.com/weather", http_method: "GET" },
      },
    ],
    llm: [
      {
        base_url: "https://llm-gateway.assemblyai.com/v1",
        model: "claude-sonnet-4-6",
        api_key: process.env.ASSEMBLYAI_API_KEY,
      },
    ],
  }),
});
const agent = await res.json();
console.log(agent.id); // the agent_id you connect with
```

Defaults if omitted: `input`/`output` use PCM at 24 kHz, `tools` and `llm` are empty (managed model), and `greeting` is none (the agent listens first). Go deeper on each field:

- `system_prompt` — [Prompting guide](/voice-agents/voice-agent-api/prompting-guide)
- `voice` / `output.volume` — [Voices](/voice-agents/voice-agent-api/voices), [Volume](/voice-agents/voice-agent-api/volume)
- `greeting` — [Greeting](/voice-agents/voice-agent-api/greeting)
- `input`/`output.format` — [Audio format](/voice-agents/voice-agent-api/audio-format#encodings)
- `input.turn_detection` — [Turn detection](/voice-agents/voice-agent-api/turn-detection-and-interruptions)
- `input.keyterms` — [Keyterms](/voice-agents/voice-agent-api/transcription-prompt#key-terms)
- `tools` — [Tools](/voice-agents/voice-agent-api/tools/overview)
- `llm` — [Connect your own LLM](/voice-agents/voice-agent-api/connect-your-own-llm)

For exact types, defaults, and validation, see the [create-agent API reference](/api-reference/voice-agent-api/create-agent) (generated from the spec, always in sync).

## Manage and deploy

- **Update** with `PUT /v1/agents/{id}` (send only the fields that change); also list, retrieve, and delete. See the [Manage agents reference](/voice-agents/voice-agent-api/manage-agents).
- **Deploy** by `agent_id` over the API, a browser, or a phone number. See [Deploy your agent](/voice-agents/voice-agent-api/deploy).

## Next steps

  
    Server-side HTTP tools and client-side function tools.
  
  
    Connect by `agent_id` over the API, a browser, or a phone number.
  
  
    Write system prompts that sound human and follow instructions.
  
  
    Every endpoint, field, and validation rule.

---

<a id="manage-agents-docs-voice-agents-voice-agent-api-manage-agents-mdx"></a>

# Manage agents

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/manage-agents
Source: docs/voice-agents/voice-agent-api/manage-agents.mdx
Navigation: Voice Agent API > Create & manage agents
Description: REST reference for creating and managing reusable voice agents. Endpoints, request and response schemas, the HTTP tool schema, and validation rules.

The Voice Agent API includes a REST API for creating and managing reusable voice agents. An agent stores its system prompt, greeting, voice, and tools server-side, so you can [deploy](/voice-agents/voice-agent-api/deploy) it across channels by referencing its `id`.

For a guided walkthrough, see [Create an agent](/voice-agents/voice-agent-api/create-agent).

**Base URL:** `https://agents.assemblyai.com`

## Authentication

Every request requires your AssemblyAI API key in the `Authorization` header. The raw key works directly; a `Bearer ` prefix is accepted and stripped:

```
Authorization: <YOUR_API_KEY>
```

| Response | Meaning |
| -------- | ------- |
| `401 {"detail": "Unauthorized"}` | Key missing, malformed, or not entitled for Voice Agents. |
| `401 {"detail": "Missing authorization header"}` | No `Authorization` header sent. |

## Endpoints

| Method   | Path                | Description                  | Success |
| -------- | ------------------- | ---------------------------- | ------- |
| `POST`   | `/v1/agents`        | Create an agent              | `201`   |
| `GET`    | `/v1/agents`        | List your agents             | `200`   |
| `GET`    | `/v1/agents/{id}`   | Retrieve one agent           | `200`   |
| `PUT`    | `/v1/agents/{id}`   | Update an agent              | `200`   |
| `DELETE` | `/v1/agents/{id}`   | Delete an agent              | `204`   |

### Create an agent

`POST /v1/agents`

```bash cURL
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent.",
    "greeting": "Hi, how can I help?",
    "voice": { "voice_id": "alba" }
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent.",
        "greeting": "Hi, how can I help?",
        "voice": {"voice_id": "alba"},
    },
)
resp.raise_for_status()
print(resp.json()["id"])
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent.",
    greeting: "Hi, how can I help?",
    voice: { voice_id: "alba" },
  }),
});
const data = await res.json();
console.log(data);
```

Returns `201` with the full [agent record](#agent-object).

### List agents

`GET /v1/agents` returns a lightweight array of `id`, `name`, and timestamps, not full configurations.

```bash cURL
curl https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY"
```

```python Python
# pip install requests
import os
import requests

resp = requests.get(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "GET",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
  },
});
const data = await res.json();
console.log(data);
```

### Retrieve an agent

`GET /v1/agents/{id}` returns the full [agent record](#agent-object). Tool header **values** are masked as `"***"` (see [HTTP tool config](#http-tool-config)).

### Update an agent

`PUT /v1/agents/{id}` accepts the same body as create, but **every field is optional**. Send only what changes. Returns `200` with the updated record.

```bash cURL
curl -X PUT https://agents.assemblyai.com/v1/agents/$AGENT_ID \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "greeting": "Thanks for calling Acme. What can I do for you?" }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.put(
    f"https://agents.assemblyai.com/v1/agents/{os.environ['AGENT_ID']}",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={"greeting": "Thanks for calling Acme. What can I do for you?"},
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch(`https://agents.assemblyai.com/v1/agents/${process.env.AGENT_ID}`, {
  method: "PUT",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ greeting: "Thanks for calling Acme. What can I do for you?" }),
});
const data = await res.json();
console.log(data);
```

### Delete an agent

`DELETE /v1/agents/{id}` returns `204` with no body.

## Request body

`POST /v1/agents` takes the agent as one JSON object; `PUT /v1/agents/{id}` takes the same body with every field optional (only `name`, `system_prompt`, and `voice` are required). For a copy-ready body with **every field populated**, see [A complete agent](/voice-agents/voice-agent-api/create-agent#a-complete-agent). For exact field types, defaults, and validation, see the [create-agent API reference](/api-reference/voice-agent-api/create-agent).

## Tool object

| Field             | Type                                | Required | Default         | Notes |
| ----------------- | ----------------------------------- | -------- | --------------- | ----- |
| `name`            | string                              | Yes       | - | Tool name the model calls. |
| `description`     | string                              | Yes       | - | When to call it. The model's main signal. |
| `parameters`      | [Parameters](#parameters-json-schema) | No     | `null`          | A JSON Schema describing the tool's arguments. |
| `http`            | [`HttpToolConfig`](#http-tool-config) | No     | `null`          | The outbound request. Omit for a client-handled tool. |
| `timeout_seconds` | integer                             | No       | `120`           | Range `1`–`300`. |
| `execution_mode`  | `"interactive"` \| `"hold"`         | No       | `"interactive"` | How the agent behaves while the tool runs. See [tool calling](/voice-agents/voice-agent-api/tools/overview#execution-modes). |

### Parameters (JSON Schema)

`parameters` is a standard **JSON Schema** object describing the tool's arguments. The same shape for HTTP and client-side tools:

```json
{
  "type": "object",
  "properties": {
    "order_id": {
      "type": "string",
      "description": "The customer's order ID.",
      "examples": ["AB-12345", "ZZ-90001"],
      "pattern": "[A-Z]{2}-\\d{5}"
    }
  },
  "required": ["order_id"]
}
```

Each property accepts standard JSON-Schema keywords. Beyond `type` and `description`, these sharpen tool-calling and turn-detection accuracy:

| Keyword     | Type     | Notes |
| ----------- | -------- | ----- |
| `enum`      | string[] | Restrict to a fixed set of allowed values. |
| `examples`  | array    | Concrete example values. Improves recognition and turn-taking. |
| `pattern`   | string   | Regex the value must match to be accepted. |
| `format`    | string   | A named JSON-Schema format (`email`, `date-time`, …). |

If you omit these, the agent infers the expected shape from each property's `description` at runtime. See [Parameter hints](/voice-agents/voice-agent-api/tools/overview#parameter-hints-improve-accuracy) for guidance and examples.

### HttpToolConfig

Write shape (create / update):

```json
{
  "url": "https://api.example.com/orders",
  "http_method": "POST",
  "headers": [
    { "name": "Authorization", "value": "Bearer xyz" }
  ]
}
```

Read shape (get / list):

```json
{
  "url": "https://api.example.com/orders",
  "http_method": "POST",
  "headers": [
    { "name": "Authorization", "last_set_at": "2026-06-20T10:00:00Z" }
  ]
}
```

| Field         | Type                            | Required | Default  | Notes |
| ------------- | ------------------------------- | -------- | -------- | ----- |
| `url`         | string                          | Yes      | -        | Must be `https` and a public host. Max 2048 chars. |
| `http_method` | HTTP method                     | No       | `"POST"` | `GET`, `POST`, `PUT`, `PATCH`, or `DELETE`. |
| `headers`     | HttpToolHeader[] \| null        | No       | `null`   | Sent on every call. Values are encrypted at rest and **write-only**. |

**HttpToolHeader (write):** `{ name, value?, remove? }`. Provide `value` to set or rotate the header. Provide name only (no `value`) to keep the stored value unchanged (useful for round-tripping on update). Set `remove: true` to delete it. Sending both `value` and `remove: true` is rejected.

**HttpToolHeader (read):** `{ name, last_set_at }`. Values are never returned.

**How arguments are sent:** `GET`/`DELETE` → query string (stringified, `null` dropped); `POST`/`PUT`/`PATCH` → JSON body. Query params already in `url` are merged with the model's arguments.

**Rotating a secret:** send `{ "name": "Authorization", "value": "Bearer <new>" }` in the headers list of a `PUT /v1/agents/{id}`. Any other headers you want to keep, round-trip as name-only entries.

**Server-side execution constraints.** AssemblyAI makes the request on your behalf and enforces: `https` only; public hosts only (private/loopback/link-local/CGNAT IPs blocked, including obfuscated literals); redirects not followed (a `3xx` is returned to the model as an error); response body capped at 8 KiB before being fed to the model; per-call timeout from `timeout_seconds`.

## Agent object

The full record returned by create, get, and update:

```json expandable
{
  "id": "7ad24396-b822-4dca-871a-be9cc4781cf9",
  "name": "Weather Buddy",
  "system_prompt": "You are Weather Buddy...",
  "greeting": "Hey there, which city would you like the weather for?",
  "tools": [
    {
      "id": "f29a0fd4-6f8b-4acb-8a74-e60f417f6dbb",
      "name": "get_weather",
      "description": "Fetch current weather for a geographic point...",
      "http": {
        "url": "https://api.open-meteo.com/v1/forecast?current=temperature_2m&temperature_unit=fahrenheit",
        "http_method": "GET",
        "headers": {}
      },
      "parameters": {
        "type": "object",
        "properties": {
          "latitude":  { "type": "number", "description": "Latitude in decimal degrees." },
          "longitude": { "type": "number", "description": "Longitude in decimal degrees." }
        },
        "required": ["latitude", "longitude"]
      },
      "timeout_seconds": 30,
      "execution_mode": "interactive"
    }
  ],
  "voice": { "voice_id": "alba" },
  "input":  { "type": "audio", "format": { "encoding": "audio/pcm", "sample_rate": 24000 }, "keyterms": null, "turn_detection": null },
  "output": { "type": "audio", "voice": "alba", "format": { "encoding": "audio/pcm", "sample_rate": 24000 }, "volume": null },
  "created_at": "2026-06-08T12:04:27.607110Z",
  "updated_at": "2026-06-08T12:04:27.607113Z"
}
```

Tools gain a generated `id`. Header values are masked as `"***"` (here the tool has no headers, so `headers` is `{}`).

## Validation errors

| Status | When | Example |
| ------ | ---- | ------- |
| `400`  | A field fails a domain rule | `Invalid voice 'xyz'. Must be one of: ...`, `tools[0].http.url: webhook URL must use https://`, `'tools[0].timeout_seconds' must be between 1s and 300s` |
| `401`  | Auth failed | `{"detail": "Unauthorized"}` |
| `422`  | Malformed JSON or wrong types | Standard schema-validation body naming the offending field. |

## Deploy a stored agent

Once created, connect by `agent_id`. Over the WebSocket, bind to the agent in your first `session.update`:

```json
{
  "type": "session.update",
  "session": { "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9" }
}
```

`agent_id` is mutually exclusive with inline session fields (`system_prompt`, `greeting`, `tools`, `input`, `output`). See [Deploy your agent](/voice-agents/voice-agent-api/deploy).

---

<a id="create-an-agent-docs-voice-agents-voice-agent-api-api-spec-create-agent-mdx"></a>

# Create an agent

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/create-agent
Source: docs/voice-agents/voice-agent-api/api-spec/create-agent.mdx
Navigation: Voice Agent API > Create & manage agents > Endpoints
Description: Create a reusable voice agent.

Metadata:
- openapi: /api-reference/specs/agents.yaml POST /v1/agents

---

<a id="list-agents-docs-voice-agents-voice-agent-api-api-spec-list-agents-mdx"></a>

# List agents

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/list-agents
Source: docs/voice-agents/voice-agent-api/api-spec/list-agents.mdx
Navigation: Voice Agent API > Create & manage agents > Endpoints
Description: List all voice agents in your account.

Metadata:
- openapi: /api-reference/specs/agents.yaml GET /v1/agents

---

<a id="retrieve-an-agent-docs-voice-agents-voice-agent-api-api-spec-get-agent-mdx"></a>

# Retrieve an agent

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/get-agent
Source: docs/voice-agents/voice-agent-api/api-spec/get-agent.mdx
Navigation: Voice Agent API > Create & manage agents > Endpoints
Description: Retrieve a stored voice agent by ID.

Metadata:
- openapi: /api-reference/specs/agents.yaml GET /v1/agents/{agent_id}

---

<a id="update-an-agent-docs-voice-agents-voice-agent-api-api-spec-update-agent-mdx"></a>

# Update an agent

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/update-agent
Source: docs/voice-agents/voice-agent-api/api-spec/update-agent.mdx
Navigation: Voice Agent API > Create & manage agents > Endpoints
Description: Update the configuration of a stored voice agent.

Metadata:
- openapi: /api-reference/specs/agents.yaml PUT /v1/agents/{agent_id}

---

<a id="delete-an-agent-docs-voice-agents-voice-agent-api-api-spec-delete-agent-mdx"></a>

# Delete an agent

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/delete-agent
Source: docs/voice-agents/voice-agent-api/api-spec/delete-agent.mdx
Navigation: Voice Agent API > Create & manage agents > Endpoints
Description: Delete a stored voice agent.

Metadata:
- openapi: /api-reference/specs/agents.yaml DELETE /v1/agents/{agent_id}

---

<a id="connect-your-own-llm-docs-voice-agents-voice-agent-api-connect-your-own-llm-mdx"></a>

# Connect your own LLM

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/connect-your-own-llm
Source: docs/voice-agents/voice-agent-api/connect-your-own-llm.mdx
Navigation: Voice Agent API > Agent behavior
Description: Point a voice agent at your own OpenAI-compatible LLM endpoint instead of AssemblyAI's managed model.

By default, a voice agent uses AssemblyAI's **managed** conversational model — you don't configure anything. To run the agent on a different model, set the `llm` field on the agent to your own **OpenAI-compatible** chat-completions endpoint. AssemblyAI calls that endpoint at runtime to generate every reply.

**When to use this.** Reach for a custom LLM when you need a specific model, your own fine-tune, or your own provider account and billing. If you just want a different frontier model without managing an endpoint, point `llm` at the [LLM Gateway](#use-the-llm-gateway) instead.

## Connect a model

Add an `llm` array to a [create](/voice-agents/voice-agent-api/create-agent) or [update](/voice-agents/voice-agent-api/manage-agents) request. Each entry needs a `base_url`, a `model`, and an `api_key`:

```bash cURL expandable
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "voice": { "voice_id": "alba" },
    "llm": [
      {
        "base_url": "https://api.openai.com/v1",
        "model": "gpt-5-mini",
        "api_key": "sk-..."
      }
    ]
  }'
```

```python Python expandable
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "voice": {"voice_id": "alba"},
        "llm": [
            {
                "base_url": "https://api.openai.com/v1",
                "model": "gpt-5-mini",
                "api_key": "sk-...",
            }
        ],
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js expandable
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    voice: { voice_id: "alba" },
    llm: [
      {
        base_url: "https://api.openai.com/v1",
        model: "gpt-5-mini",
        api_key: "sk-...",
      },
    ],
  }),
});
const data = await res.json();
console.log(data);
```

| Field      | Type   | Required | Notes                                                                                     |
| ---------- | ------ | -------- | ----------------------------------------------------------------------------------------- |
| `base_url` | string | Yes      | HTTPS base URL of the OpenAI-compatible endpoint. Must be `https` and a public host. The agent calls `POST {base_url}/chat/completions`. |
| `model`    | string | Yes      | Model name sent in the chat-completions request body.                                     |
| `api_key`  | string | Yes      | Key for your endpoint. **Write-only** — encrypted at rest and never returned in any response. |

## Update or rotate the model

Send a new `llm` array on `PUT /v1/agents/{id}`. Include `api_key` to rotate the key; the whole `llm` entry is replaced:

```bash cURL
curl -X PUT https://agents.assemblyai.com/v1/agents/$AGENT_ID \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "llm": [
      { "base_url": "https://api.openai.com/v1", "model": "gpt-4.1", "api_key": "sk-new..." }
    ]
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.put(
    f"https://agents.assemblyai.com/v1/agents/{os.environ['AGENT_ID']}",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "llm": [
            {"base_url": "https://api.openai.com/v1", "model": "gpt-4.1", "api_key": "sk-new..."}
        ]
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch(`https://agents.assemblyai.com/v1/agents/${process.env.AGENT_ID}`, {
  method: "PUT",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    llm: [
      { base_url: "https://api.openai.com/v1", model: "gpt-4.1", api_key: "sk-new..." },
    ],
  }),
});
const data = await res.json();
console.log(data);
```

To switch the agent **back to the managed model**, send `"llm": []`.

## Use the LLM Gateway

You don't need your own provider account to use a frontier model. Point `base_url` at AssemblyAI's [LLM Gateway](/llm-gateway/quickstart) and pass your AssemblyAI API key — you get Claude, GPT, Gemini, and more through one endpoint, billed on your AssemblyAI account:

```bash cURL expandable
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Gateway Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "voice": { "voice_id": "alba" },
    "llm": [
      {
        "base_url": "https://llm-gateway.assemblyai.com/v1",
        "model": "claude-sonnet-4-6",
        "api_key": "'"$ASSEMBLYAI_API_KEY"'"
      }
    ]
  }'
```

```python Python expandable
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Gateway Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "voice": {"voice_id": "alba"},
        "llm": [
            {
                "base_url": "https://llm-gateway.assemblyai.com/v1",
                "model": "claude-sonnet-4-6",
                "api_key": os.environ["ASSEMBLYAI_API_KEY"],
            }
        ],
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js expandable
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Gateway Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    voice: { voice_id: "alba" },
    llm: [
      {
        base_url: "https://llm-gateway.assemblyai.com/v1",
        model: "claude-sonnet-4-6",
        api_key: process.env.ASSEMBLYAI_API_KEY,
      },
    ],
  }),
});
const data = await res.json();
console.log(data);
```

See [Available models](/llm-gateway/available-models) for the full list. Use the EU host `https://llm-gateway.eu.assemblyai.com/v1` for EU workloads.

## Requirements & behavior

- **OpenAI-compatible.** The endpoint must accept `POST /chat/completions` in the OpenAI schema.
- **Streaming.** Realtime voice needs token streaming, so the model must support streamed chat completions.
- **One config.** `llm` is a list, but only a single entry is accepted today (fallbacks aren't supported yet).
- **HTTPS + public host.** Non-`https` URLs and private/loopback hosts are rejected.
- **Reads mask the key.** `GET`/list responses return only `base_url` and `model` — never `api_key`.

Latency and reliability now depend on your endpoint. A slow or rate-limited model shows up directly as reply latency in the conversation. See [Best practices](/voice-agents/best-practices) for tuning.

---

<a id="prompting-guide-docs-voice-agents-voice-agent-api-prompting-guide-mdx"></a>

# Prompting guide

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/prompting-guide
Source: docs/voice-agents/voice-agent-api/prompting-guide.mdx
Navigation: Voice Agent API > Agent behavior
Description: Patterns for writing system prompts that improve instruction following, conversationality, and voice output quality.

Set your agent's `system_prompt` via [`session.update`](/voice-agents/voice-agent-api/session-configuration#system-prompt). The patterns below are tested against real voice agent conversations.

---

## Make instructions stick

### Front-load your most important rule

Put your most critical instruction at the top and reinforce it. Long prompts dilute attention. If you bury a key rule in the middle, the model deprioritizes it.

```text
BE SHORT. This is the most important rule. Keep every response under two sentences.

You are a customer support agent for Acme Corp...
```

### Use negative instructions with exact phrasings

Listing the exact phrases you don't want works better than vague positive instructions like "be casual". The model pattern-matches against concrete strings.

```text
Never say "certainly", "absolutely", "happy to help", or "great question".
```

### Pair bad examples with good ones

Show what you don't want next to what you do want. The contrast teaches the rule.

```text
When the user describes their project, don't give a feature tour:
  Bad: "You could build A, B, or C. What problem are you trying to solve?"
  Good: "Yeah, like a receptionist."
```

### Add self-check heuristics

Give the model a check it can run before responding. Abstract rules like "be brief" don't land. A concrete heuristic does.

```text
If your reply has a comma, ask yourself if it could just stop at the comma.
If your reply is more than 15 words, shorten it.
```

### Match example length to desired output length

If your prompt examples are paragraphs, the model outputs paragraphs. Keep example outputs as terse as the real responses you want.

### Iterate against real transcripts

Fix specific failures from real conversations. Quote the failing output and show the correction:

```text
When the user asks "what does this do?", don't say:
  "This is a powerful tool that enables you to streamline your workflow
   by providing real-time insights and actionable data across multiple
   dimensions of your business operations."
Say:
  "It transcribes your calls and pulls out the key points."
```

Speculative rules ("what if the user asks about X") add noise without improving quality. Iterate against logs, not imagination.

---

## Sound human

### Give the agent an identity

Identity statements shape tone better than behavioral lists. Tell the model _who it is_, not just _what to do_.

```text
You're a person on a call, not a feature tour. You're not auditioning,
you're having a conversation.
```

### Use permission language

Safety training makes models default to formal, cautious responses. Explicit permissions unlock natural behavior that "be friendly" never will.

```text
Have opinions. You can crack jokes. You can be a little dry when someone
wastes your time. You don't need to hedge everything.
```

### Mirror the user's length and energy

The model's default is to talk more than the user. Instruct it to match.

```text
Match the user's length. When they talk in clipped phrases, you do the same.
If they give you one word, reply with a few words. If they go deep, go deep.
```

### Define engagement modes

A single behavioral playbook produces the same response shape regardless of context. Define separate modes so the agent reads the room.

```text
When the user is engaged and asking detailed questions, give thorough answers.
When the user gives short or distracted responses, keep it brief and check in.
When someone is clearly just messing around, you can be playful or cut it short.
```

### Ban bot tells

List the specific phrases that make agents sound like chatbots and ban them.

```text
Never say:
- "Great question!"
- "That's an interesting point."
- "Want me to walk you through that?"
- "What are you building today?"
- "I'd be happy to help with that."
```

### Ground temporal and situational context

Inject session-specific information into the prompt at runtime. Users ask "what time is it?" and the model should not hallucinate.

```text
Current date and time: {datetime_utc}
Your voice: {voice_name}
User's name: {user_name}
```

---

## Define capabilities explicitly

Without clear boundaries, the model invents capabilities to please users or denies things it can actually do.

### List what the agent can and cannot do

```text
Things you CAN do:
- Look up order status (use the get_order tool)
- Answer questions about our pricing and plans
- Switch languages mid-conversation

Things you CANNOT do:
- Process refunds (transfer to a human agent)
- Access the user's account settings
- Look anything up on the internet
```

### Pin verified facts

List every factual claim the model is allowed to make. For anything else, direct users to documentation or a human.

```text
Facts you can state:
- Our API supports 6 languages
- Latency is under 300ms
- Plans start at $0 with pay-as-you-go

Do NOT make up statistics, pricing, or technical specs beyond what's listed above.
If you're unsure, say "I'd want to double-check that, let me point you to our docs."
```

---

## Optimize for voice output

TTS engines read formatting characters literally. Formatting that works in chat sounds broken when spoken aloud.

### Tell the model why formatting rules exist

Give a concrete example of the failure mode so it understands the constraint.

```text
No markdown formatting. If you write **alba**, the user hears
"asterisk asterisk alba asterisk asterisk". No bullets, no bold,
no headers. Plain conversational sentences only.
```

### Spell out how to read literals

URLs, field names, and code identifiers need explicit substitution rules.

```text
When reading URLs:
- Say "dot" for periods, "slash" for slashes
- Drop the protocol ("assemblyai dot com", not "https colon slash slash")
- Spell out short paths ("slash docs slash getting started")

When reading code or field names:
- Say "underscore" for underscores
- Spell out abbreviations ("API" as "A-P-I")
```

### Round numbers and times

Voice users don't need precision. Use natural approximations.

```text
Round times: say "around 2 in the afternoon UTC", not "2:34:17 PM".
Round large numbers: say "about 10 thousand", not "9,847".
Dates: say "April 30th", not "2026-04-30".
```

---

## General prompt hygiene

### Write policies, not decision trees

A policy is a general rule the model applies across situations. A decision tree is a set of fragile conditionals the model will misinterpret.

```text
Policy (good):
  "If the user shares an email address, read it back to confirm."

Decision tree (bad):
  "If the user shares an email, check if it's a gmail address, then
   ask if it's their work email, then confirm the spelling of..."
```

### Trim aggressively when adding new sections

Long prompts drown their best rules. Every time you add a new instruction, look for something to remove.

### Test by reading aloud

Read the agent's outputs out loud. Visual scanning misses rhythm problems and unnatural phrasing that users notice immediately on a phone call.

---

## Putting it together

A well-structured voice agent prompt typically follows this order:

1. **Identity and most important rule**
2. **Tone and conversational style** (permissions, mirroring, bot-tell bans)
3. **Capabilities and facts** (what it can and cannot do, pinned facts)
4. **Tool usage instructions** (when to call each tool, what to say while waiting)
5. **Voice formatting rules** (no markdown, reading literals, rounding)
6. **Engagement modes** (how to behave in different conversational contexts)

Your prompt works together with other session configuration. Use [`keyterms`](/voice-agents/voice-agent-api/session-configuration#key-terms) to boost recognition of domain-specific words, and configure [turn detection](/voice-agents/voice-agent-api/session-configuration#turn-detection) thresholds to match the conversational pace your prompt encourages.

---

<a id="greeting-docs-voice-agents-voice-agent-api-greeting-mdx"></a>

# Greeting

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/greeting
Source: docs/voice-agents/voice-agent-api/greeting.mdx
Navigation: Voice Agent API > Agent behavior
Description: Set the exact words a voice agent speaks when a conversation starts, or omit it to listen first.

The `greeting` is what the agent says at the start of the conversation, spoken aloud. It lives on the agent, set when you [create](/voice-agents/voice-agent-api/create-agent) or [update](/voice-agents/voice-agent-api/manage-agents) it, or inline over the WebSocket via [`session.update`](/voice-agents/voice-agent-api/session-configuration). If omitted, the agent waits silently for the user to speak first.

## Set a greeting

Pass `greeting` as a top-level field alongside the required `name`, `system_prompt`, and `voice`:

```bash cURL
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "voice": { "voice_id": "alba" },
    "greeting": "Hey, this is Riley from Acme support. What'\''s going on?"
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "voice": {"voice_id": "alba"},
        "greeting": "Hey, this is Riley from Acme support. What's going on?",
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    voice: { voice_id: "alba" },
    greeting: "Hey, this is Riley from Acme support. What's going on?",
  }),
});
const data = await res.json();
console.log(data);
```

| Field      | Type           | Required | Notes                                                          |
| ---------- | -------------- | -------- | -------------------------------------------------------------- |
| `greeting` | string \| null | No       | Spoken on connect; sent straight to TTS. Omit to listen first. |

The greeting is sent **straight to the TTS engine**. It is not run through the LLM first. Whatever string you put here is exactly what the user hears, word for word.

This has two consequences:
- Don't write a meta-greeting like _"Greet the user warmly and ask how you can help"_. The TTS will literally speak that sentence. Write the exact words you want spoken.
- The system prompt's tone/persona rules do not get applied to the greeting. Match the tone yourself.

Omit `greeting` entirely to have the agent listen first — for an inbound call where the caller speaks first, or when an out-of-band channel (like an IVR menu) already handled the greeting.

## Change the greeting

On a stored agent, change the greeting by updating the agent:

```bash cURL
curl -X PUT https://agents.assemblyai.com/v1/agents/$AGENT_ID \
  -H "Authorization: $ASSEMBLYAI_API_KEY" -H "Content-Type: application/json" \
  -d '{ "greeting": "Thanks for calling Acme. What can I do for you?" }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.put(
    f"https://agents.assemblyai.com/v1/agents/{os.environ['AGENT_ID']}",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={"greeting": "Thanks for calling Acme. What can I do for you?"},
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch(
  `https://agents.assemblyai.com/v1/agents/${process.env.AGENT_ID}`,
  {
    method: "PUT",
    headers: {
      Authorization: process.env.ASSEMBLYAI_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      greeting: "Thanks for calling Acme. What can I do for you?",
    }),
  },
);
const data = await res.json();
console.log(data);
```

When you configure an agent **inline** via [`session.update`](/voice-agents/voice-agent-api/session-configuration#mutability-after-session-ready), the greeting is **immutable** after `session.ready`. Set it on your first `session.update`; trying to change it mid-session returns `immutable_field`. This caveat applies only to the inline path, not to updating a stored agent.

---

<a id="tools-docs-voice-agents-voice-agent-api-tools-overview-mdx"></a>

# Tools

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/overview
Source: docs/voice-agents/voice-agent-api/tools/overview.mdx
Navigation: Voice Agent API > Agent behavior > Tools
Description: Give your agent tools to fetch data and take actions, plus the parameter hints, execution modes, and patterns that make tool calls and turn-taking reliable.

Tools let your agent take actions and fetch live data. There are two kinds, and most agents use the first:

  
    Defined on your [stored agent](/voice-agents/voice-agent-api/create-agent). You give a URL and a parameter list; **AssemblyAI makes the request for you** and feeds the result to the model. Your client does nothing.
  
  
    Declared inline in `session.tools`. The agent emits a `tool.call`; **your code runs the logic** and sends back a `tool.result`. Use when the tool needs local state or custom logic.
  

This page covers what's common to both: how parameter hints sharpen accuracy, how to get the agent to call your tools, [execution modes](#execution-modes), [progressive reveal](#progressive-tool-reveal), and [per-agent patterns](#patterns-by-agent-type).

## Parameter hints improve accuracy

`parameters` is a **JSON Schema** object: `{ "type": "object", "properties": { … }, "required": [ … ] }`, the same on both tool types. Beyond `type` and `description`, each property accepts standard JSON-Schema keywords that describe the **shape** of the value. They do two jobs:

- **Tool-calling accuracy.** A spoken value that doesn't fit the shape is rejected before the tool runs, and the agent re-asks for just that value instead of calling the tool with garbage.
- **Turn-detection accuracy.** Knowing what a complete value looks like lets the agent tell whether the user has finished speaking it. It waits for all ten digits of a phone number instead of cutting in after "my number is four one five…".

These keywords describe the **value your tool receives** — the value the agent produces from the caller's speech, not the words the caller says out loud. A caller might say "four one five, five five five…" and the agent produces `+14155552671`. Your `pattern` and `examples` describe that produced value.

The agent's cleanup of spoken input is **best-effort, not guaranteed**. It's reliable for well-known shapes (phone numbers, emails, dates), but for long or free-form digit sequences — card numbers, account numbers, claim IDs — the value can reach your tool **as it was spoken**: with spaces between digits (`"4 2 4 2 4 2 …"`) when the caller reads them one at a time, or with digits missing if speech recognition dropped some. Your `pattern` is matched against that value as-is, so write it to tolerate the forms the value can actually arrive in (see [spoken digit sequences](#spoken-digit-sequences) below). A pattern that only accepts the tidy form (`^\\d{16}$`) will reject the spaced form and trap the caller in a re-ask loop.

| Keyword     | What it does | Good values |
| ----------- | ------------ | ----------- |
| `enum`      | Restrict the value to a fixed set. The model picks one; anything else is rejected. | `["billing", "sales", "support"]` |
| `examples`  | Sample values in the exact form your tool receives them. Give 2–4 realistic ones. | `["+14155552671", "+442071838750"]` |
| `pattern`   | A regex the final value must match (Python `re`, matched against the whole value). Escape backslashes in JSON. | `"\\+[1-9]\\d{1,14}"` |
| `format`    | A named JSON-Schema format the value should conform to. | `"email"`, `"date-time"`, `"date"` |

If you omit these, the agent **infers** the expected shape from the property's `description` at runtime. Setting them explicitly is an override for when you want tight, predictable validation. A precise `description` is still what matters most; the hints sharpen it.

### How to choose each keyword

**`enum`**: use it whenever the value is one of a known, small set. Removes "the model invented a category" bugs entirely.

```json
{ "department": { "type": "string", "description": "Which team to route to.",
  "enum": ["billing", "sales", "support"] } }
```

**`examples`**: use these for any free-form value. Two to four realistic examples beat a long description. Every example should be a value the tool would actually accept (and match `pattern` if you set one).

```json
{ "airport_code": { "type": "string", "description": "IATA airport code.",
  "examples": ["SFO", "LHR", "JFK"] } }
```

**`pattern`**: use this when the value has a strict format. It's a Python regular expression matched against the **whole** value (so you don't need `^` or `$`). In JSON you have to escape backslashes, so write `\\d`, not `\d`. The value is matched **as the agent produces it** — for well-known shapes that's the cleaned-up form, but for spoken digit sequences it may still carry the spaces or pauses from speech (see [below](#spoken-digit-sequences)), so the pattern must accept those. If the value doesn't match, the agent re-asks for it.

| Value | `pattern` | `format` | `examples` |
| ----- | --------- | -------- | ---------- |
| US ZIP | `\\d{5}(-\\d{4})?` | - | `["94103", "10001-2201"]` |
| E.164 phone | `\\+[1-9]\\d{1,14}` | - | `["+14155552671"]` |
| Order ID | `[A-Z]{2}-\\d{5}` | - | `["AB-12345"]` |
| ISO date | `\\d{4}-\\d{2}-\\d{2}` | `date` | `["2026-06-09"]` |
| Email | - | `email` | `["alex@acme.com"]` |

`\\d` is any digit, `[A-Z]` is one uppercase letter, `{2}` is "exactly two", `{1,14}` is "one to fourteen", and `?` makes the part before it optional. So `[A-Z]{2}-\\d{5}` matches an order ID like `AB-12345`.

Keep patterns **tight but not brittle**. Too loose (`.*`) gives no benefit; too strict rejects legitimate values and traps the user in a re-ask loop. Make every value in `examples` match the pattern; if you can't, loosen the pattern, not the examples.

**`format`**: a standard JSON-Schema named format (`email`, `date-time`, `uri`, …). Use it for well-known value types instead of hand-rolling a `pattern`.

### Spoken digit sequences

Long digit sequences — card numbers, account numbers, claim IDs — are the most common source of re-ask loops, because callers rarely say them cleanly. They read them one digit at a time or in groups, so the value often reaches your tool **with spaces in it**: `"4 2 4 2 4 2 4 2 4 2 4 2 4 2 4 2"`, `"4242 4242 4242 4242"`, or `"4242424242424242"` — all for the same card. A `pattern` written only for the tidy form rejects the spaced ones and the caller gets stuck repeating themselves.

Write the `pattern` to allow interior spaces, bound the **digit** count (not the character count), and give `examples` that include the spaced form so both validation and turn-detection expect it:

```json
{
  "card_number": {
    "type": "string",
    "description": "The card number as spoken. May contain spaces between digits when the caller reads them one at a time; 13–19 digits once spaces are removed.",
    "pattern": " *([0-9] *){13,19}",
    "examples": [
      "4242424242424242",
      "4242 4242 4242 4242",
      "4 2 4 2 4 2 4 2 4 2 4 2 4 2 4 2"
    ]
  }
}
```

Then strip non-digits in your tool handler before you use the value — e.g. `re.sub(r"\D", "", card_number)` — and run your own length/checksum validation there. Keep the `pattern` as a shape hint that admits real spoken input, not as your final validation.

This tolerates a **complete** number that was spoken with spaces. It does not (and should not) accept a genuinely **incomplete** one — if speech recognition only captured 12 of 16 digits, that value is rejected and the agent re-asks, which is correct. For long numbers, prompt the caller to read them at a steady pace rather than one digit at a time with long pauses.

### Worked example

A tool that books a callback, with phone constrained by `pattern` + `examples` and time-of-day by `enum`:

```bash cURL expandable
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Callback Assistant",
    "system_prompt": "You schedule callbacks. Collect the phone number and preferred time of day, then call schedule_callback.",
    "voice": { "voice_id": "alba" },
    "tools": [
      {
        "name": "schedule_callback",
        "description": "Schedule a callback to a phone number at a chosen time.",
        "parameters": {
          "type": "object",
          "properties": {
            "phone": {
              "type": "string",
              "description": "The number to call back, including country code.",
              "examples": ["+14155552671", "+442071838750"],
              "pattern": "\\+[1-9]\\d{1,14}"
            },
            "window": {
              "type": "string",
              "description": "Preferred time of day for the callback.",
              "enum": ["morning", "afternoon", "evening"]
            }
          },
          "required": ["phone", "window"]
        },
        "http": { "url": "https://api.example.com/callbacks", "http_method": "POST" }
      }
    ]
  }'
```

```python Python expandable
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Callback Assistant",
        "system_prompt": "You schedule callbacks. Collect the phone number and preferred time of day, then call schedule_callback.",
        "voice": {"voice_id": "alba"},
        "tools": [
            {
                "name": "schedule_callback",
                "description": "Schedule a callback to a phone number at a chosen time.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "phone": {
                            "type": "string",
                            "description": "The number to call back, including country code.",
                            "examples": ["+14155552671", "+442071838750"],
                            "pattern": "\\+[1-9]\\d{1,14}",
                        },
                        "window": {
                            "type": "string",
                            "description": "Preferred time of day for the callback.",
                            "enum": ["morning", "afternoon", "evening"],
                        },
                    },
                    "required": ["phone", "window"],
                },
                "http": {"url": "https://api.example.com/callbacks", "http_method": "POST"},
            }
        ],
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js expandable
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Callback Assistant",
    system_prompt: "You schedule callbacks. Collect the phone number and preferred time of day, then call schedule_callback.",
    voice: { voice_id: "alba" },
    tools: [
      {
        name: "schedule_callback",
        description: "Schedule a callback to a phone number at a chosen time.",
        parameters: {
          type: "object",
          properties: {
            phone: {
              type: "string",
              description: "The number to call back, including country code.",
              examples: ["+14155552671", "+442071838750"],
              pattern: "\\+[1-9]\\d{1,14}",
            },
            window: {
              type: "string",
              description: "Preferred time of day for the callback.",
              enum: ["morning", "afternoon", "evening"],
            },
          },
          required: ["phone", "window"],
        },
        http: { url: "https://api.example.com/callbacks", http_method: "POST" },
      },
    ],
  }),
});
const data = await res.json();
console.log(data);
```

With these set, if the user says "call me at four one five, five five five…" and trails off, the agent waits (the value doesn't yet match `\+[1-9]\d{1,14}`) instead of cutting in. If STT garbles a digit, the agent re-asks for the phone number specifically rather than firing the tool with a bad value.

## Getting the agent to call your tools

In rough order of impact:

1. **Strong tool descriptions.** Treat `description` as "when should I reach for this?", not "what does this do?". Name the trigger ("Call this when the user asks about X"), name the anti-trigger ("Do not call this for Y"), and mention any precondition. Most "tool never fires" failures trace here.
2. **Strong parameter hints.** Vague params produce missing or invented values, which the validator rejects (or worse, your tool runs on garbage). Lead with format, add an example, use `enum` for fixed sets. See [Parameter hints](#parameter-hints-improve-accuracy).
3. **Default-to-call wording in `system_prompt`.** *"When in doubt, call the tool. A wasted call is fine. Answering wrong from memory is not."* Don't stack exceptions.
4. **Few-shot examples in `system_prompt`** are the strongest behavioural signal:
   > User: "Where's my order?"
   > You: [call search_orders] "Looks like it's out for delivery today."
5. **Keep tool sets small (≤10 per phase).** Past that, selection accuracy drops. See [Progressive tool reveal](#progressive-tool-reveal).

## Execution modes

Set `execution_mode` per tool to choose how the agent waits. This applies to both HTTP and function tools. The sequence diagrams below show the client-side `tool.call`/`tool.result` flow; for HTTP tools that round trip happens server-side, but the conversational behavior (interactive keeps talking, hold goes quiet) is identical.

| Use `"interactive"` for...                          | Use `"hold"` for...                                  |
| --------------------------------------------------- | ---------------------------------------------------- |
| DB lookups, REST calls, short calculations          | Phone transfers, escalations                         |
| Returns under ~5 seconds                            | Long-running ops (>10s, async jobs)                  |
| Transition phrase ("let me check") feels natural    | Sensitive flows (payment auth, identity verification) |

**Default to interactive.** Two common mis-uses:

- ❌ Wrapping a slow DB query in hold "to be safe". Agent goes mute, user thinks the call dropped. Use interactive with a longer `timeout_seconds`.
- ❌ Using interactive for a 30-second human transfer. Agent fills with small-talk; user gets suspicious.

### Interactive

```text
server                                 client
  │  reply.started                       │
  │ ───────────────────────────────────► │
  │  reply.audio  ("let me check that")  │
  │ ───────────────────────────────────► │
  │  tool.call                           │  client accumulates result
  │ ───────────────────────────────────► │  (does NOT send tool.result yet)
  │  reply.done                          │
  │ ───────────────────────────────────► │  client drains pending results:
  │                                      │  tool.result
  │ ◄─────────────────────────────────── │
  │  reply.started                       │  agent delivers answer
  │ ───────────────────────────────────► │
  │  reply.audio  ("it's 22°C and sunny")│
  │ ───────────────────────────────────► │
  │  reply.done                          │
  │ ───────────────────────────────────► │
```

### Hold

While the tool is in flight:

1. Agent stays silent (no `reply.started`).
2. User speech doesn't trigger replies. Utterances are added to context but the agent doesn't respond until you send `tool.result` or `reply.create`.
3. `tool.result` **auto-fires** the next reply. Don't also send `reply.create` after.

```json
{
  "type": "session.update",
  "session": {
    "tools": [
      {
        "type": "function",
        "name": "transfer_call",
        "description": "Transfer the call to a human agent. Takes 15–30 seconds.",
        "parameters": {"type": "object", "properties": {"department": {"type": "string"}}, "required": ["department"]},
        "execution_mode": "hold",
        "timeout_seconds": 60
      }
    ]
  }
}
```

```text
server                                 client
  │  tool.call (hold)                    │
  │ ───────────────────────────────────► │  kick off long-running op
  │                                      │  (agent silent, no reply.started)
  │                                      │
  │                                      │  reply.create { instructions: ... }
  │ ◄─────────────────────────────────── │  ── optional status update
  │  reply.started → reply.audio → done  │
  │ ───────────────────────────────────► │
  │                                      │  (op completes)
  │                                      │  tool.result
  │ ◄─────────────────────────────────── │
  │  reply.started                       │  auto-fired by tool.result
  │ ───────────────────────────────────► │
  │  reply.audio  ("all set...")         │
  │ ───────────────────────────────────► │
  │  reply.done                          │
  │ ───────────────────────────────────► │
```

During hold, the server does **not** emit `transcript.user.delta` or `transcript.user` in real time. Transcripts flush once the hold ends (`tool.result` or `reply.create`). Live captioning pauses during the hold; nothing is dropped.

### Status updates during hold

Send [`reply.create`](/voice-agents/voice-agent-api/events-reference#reply-create) with optional `instructions` to make the agent speak mid-hold without ending it:

```python
await ws.send(json.dumps({
    "type": "reply.create",
    "instructions": "Let the customer know you're still working on the transfer."
}))
```

The hold continues until you send the matching `tool.result`.

## Progressive tool reveal

For multi-step workflows (*lookup → estimate → commit*), don't register all tools upfront. After each successful `tool.result`, send `session.update` adding the next phase's tools, **and** update `system_prompt` to match.

Why: a tool that isn't in the current list can't be called, so the model can't fabricate a commit before the prerequisite step has run. Smaller per-phase tool sets also raise selection accuracy.

### Worked example: taxi booking

```text
session state              tools exposed
─────────────────────────  ──────────────────────────────────────────
session start              [lookup_postcode]
                                  │ user gives pickup postcode
                                  ▼
                           ⚙ lookup_postcode("SW1A 1AA") → ✓
─────────────────────────  ──────────────────────────────────────────
tier 2 unlocked            [lookup_postcode, estimate_fare]
                                  │ user gives dropoff
                                  ▼
                           ⚙ estimate_fare(...) → ✓
─────────────────────────  ──────────────────────────────────────────
tier 3 unlocked            [lookup_postcode, estimate_fare, book_ride,
                            get_booking, track_driver, cancel_ride]
                                  │ user confirms + name
                                  ▼
                           ⚙ book_ride(name="Alex", ...) → ✓
```

Until `lookup_postcode` returns a real postcode, the model has no `book_ride` tool. It can verbally promise a booking; it can't create one.

### Client-side wiring

```python
TIER_1_TOOLS = [lookup_postcode]
TIER_2_TOOLS = [lookup_postcode, estimate_fare]
TIER_3_TOOLS = [lookup_postcode, estimate_fare, book_ride,
                get_booking, track_driver, cancel_ride]

tier_2_unlocked = tier_3_unlocked = False

async def maybe_unlock_next_tier(tool_name, result):
    global tier_2_unlocked, tier_3_unlocked
    if result.get("error"):
        return

    if not tier_2_unlocked and tool_name == "lookup_postcode" and result.get("postcode"):
        tier_2_unlocked = True
        await ws.send(json.dumps({"type": "session.update",
                                  "session": {"tools": TIER_2_TOOLS, "system_prompt": TIER_2_PROMPT}}))
    elif not tier_3_unlocked and tool_name == "estimate_fare" and result.get("estimated_fare"):
        tier_3_unlocked = True
        await ws.send(json.dumps({"type": "session.update",
                                  "session": {"tools": TIER_3_TOOLS, "system_prompt": TIER_3_PROMPT}}))
```

**Update tools AND `system_prompt` together.** Tool-only gating where the prompt still references a now-hidden tool can underperform not gating at all. The model hunts for a tool the prompt promised and stalls or improvises when it can't find it. Strip or rewrite every prompt sentence that names a tool whose visibility changed.

### Per-call state machine

The strongest form: every successful tool call is a state transition; each state owns a narrow prompt + small tool list.

| State            | System prompt focus                                          | Tools                            |
| ---------------- | ------------------------------------------------------------ | -------------------------------- |
| `s0_greet`       | "Get pickup postcode. Nothing else."                         | `lookup_postcode`                |
| `s2_quoting`     | "Call `estimate_fare`. Filler only; no fare numbers."       | `estimate_fare`                  |
| `s4_have_name`   | "Call `book_ride` with captured pickup, dropoff, name."      | `book_ride`                      |
| `s5_booked`      | "Read back confirmation. Offer track/cancel."                | `get_booking`, `track_driver`, `cancel_ride` |

### Escape hatches

Real users go off-script. Two patterns, used together:

- **Transition tools**: `revise_pickup`, `revise_dropoff`, `restart`, `end_call` exposed in every state. Model picks the right escape; orchestrator rolls state back.
- **`respond_freely`**: a no-op tool in every state for tangential questions ("are you a real person?"). Model calls it instead of leaving the state.

### Anti-fabrication clause

Gating makes hallucinations harmless (no real booking happens) but doesn't suppress the spoken claim. Pair with prompt wording:

```text
NEVER quote a fare, distance, time, confirmation number, name, or ETA unless
those exact values came from a tool result in this conversation. If you
haven't seen a tool result, you do NOT have these values. Don't estimate
them. Don't guess. Don't say "around" a number.
```

## Patterns by agent type

### Customer support

```
s0:  [lookup_ticket]                                           ← always
s1:  [lookup_ticket, escalate_to_human (hold), close_ticket]   ← after lookup
any: [respond_freely, end_call]
```

Prompt focus: "Use `lookup_ticket` for any ticket question. Only `escalate_to_human` after checking the ticket. Don't promise outcomes you can't verify."

### Booking / reservations

```
s0:  [check_availability, cancel_reservation]                          ← always
s1a: [check_availability, create_reservation, cancel_reservation]      ← if available
s1b: [check_availability, add_to_waitlist, cancel_reservation]         ← if full
```

Prompt focus: "Confirm party size, date, time. Call `check_availability`. If open, offer it and book. If not, offer the next two times or the waitlist."

The next tool depends on the prior result. Don't expose both `create_reservation` and `add_to_waitlist` simultaneously. The model picks the wrong one ~30% of the time.

### Banking / account

```
s0:  [verify_identity (hold)]                                  ← gatekeeper
s1:  [get_balance, list_recent_transactions]                   ← after auth
s2:  [start_transfer, dispute_charge (hold), close_account]    ← actions
any: [end_call]
```

Prompt focus: "Before sharing any account info, call `verify_identity`. Never quote a balance or transaction you haven't fetched. Never promise a dispute outcome; only the system can."

The anti-fabrication clause matters most here. A bank agent inventing a balance is a P0.

## Debugging

### Tool never fires

- Description too vague. Name the user phrases that should trigger it.
- System prompt missing "default to calling" wording.
- Too many tools (>10). Drop or split via [progressive reveal](#progressive-tool-reveal).
- Add a few-shot example to the system prompt. This is the strongest signal.

### Wrong arguments

- Parameter description missing format/example. Add `(e.g. 2026-04-30)`, or set `examples`/`pattern`. See [Parameter hints](#parameter-hints-improve-accuracy).
- Free-text where you want fixed buckets. Use `enum`.
- User says the value multiple ways. Normalise in the description.

### Agent keeps saying it "didn't catch" a number (re-ask loop)

Almost always a `pattern` that's too strict for spoken input, on a long digit sequence (card, account, claim ID). The caller reads the digits one at a time, so the value arrives with spaces (`"4 2 4 2 …"`), your pattern only accepts the tidy form, and every attempt is silently rejected — the caller just hears "I didn't catch that" on repeat. Widen the pattern to allow interior spaces and add a spaced `example`. See [Spoken digit sequences](#spoken-digit-sequences).

### Agent invents a result

Most common cause: the model is being asked to do something *after* a tool result without having actually called the tool. Two fixes, used together:

1. **Progressive reveal**: gate the commit tool behind the read tool.
2. **Anti-fabrication clause** in the prompt (see above).

### Tool fires repeatedly

- `tool.result` arriving while `last_event` is `reply.started`. Make sure your handler flushes on `reply.done`.
- Tool slower than `timeout_seconds`. Agent gets internal timeout, user tries again. Bump the timeout.

### Tool fires when it shouldn't

Description is too broad. Add explicit anti-triggers:

```
**Use this for**: weather questions.
**Do not call for**: general chit-chat, scheduling, or any non-weather topic.
```

---

<a id="http-tools-docs-voice-agents-voice-agent-api-tools-http-tools-mdx"></a>

# HTTP tools

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/http-tools
Source: docs/voice-agents/voice-agent-api/tools/http-tools.mdx
Navigation: Voice Agent API > Agent behavior > Tools
Description: Server-side tools. Give your agent a URL and a parameter list, and AssemblyAI makes the request for you when the model calls the tool.

The simplest way to give an agent a tool. Add an `http` block to a tool on your [stored agent](/voice-agents/voice-agent-api/create-agent), and AssemblyAI calls the endpoint whenever the model invokes the tool. There's **no `tool.call`/`tool.result` round trip in your client**; your app just streams audio.

Add the tool in the `tools` array when you create the agent (or send the same array on `PUT /v1/agents/{id}` to add it to an existing one):

```bash cURL expandable
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Weather Assistant",
    "system_prompt": "You are a friendly assistant. When asked about weather, call get_weather and read back the result.",
    "voice": { "voice_id": "alba" },
    "tools": [
      {
        "name": "get_weather",
        "description": "Get current weather for a location. Use this whenever the user asks about weather, temperature, or conditions.",
        "parameters": {
          "type": "object",
          "properties": {
            "latitude":  { "type": "number", "description": "Latitude in decimal degrees, e.g. 48.85." },
            "longitude": { "type": "number", "description": "Longitude in decimal degrees, e.g. 2.35." }
          },
          "required": ["latitude", "longitude"]
        },
        "execution_mode": "interactive",
        "timeout_seconds": 30,
        "http": {
          "url": "https://api.example.com/weather",
          "http_method": "GET",
          "headers": [
            { "name": "Authorization", "value": "Bearer <secret>" }
          ]
        }
      }
    ]
  }'
```

```python Python expandable
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Weather Assistant",
        "system_prompt": "You are a friendly assistant. When asked about weather, call get_weather and read back the result.",
        "voice": {"voice_id": "alba"},
        "tools": [
            {
                "name": "get_weather",
                "description": "Get current weather for a location. Use this whenever the user asks about weather, temperature, or conditions.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "latitude": {"type": "number", "description": "Latitude in decimal degrees, e.g. 48.85."},
                        "longitude": {"type": "number", "description": "Longitude in decimal degrees, e.g. 2.35."},
                    },
                    "required": ["latitude", "longitude"],
                },
                "execution_mode": "interactive",
                "timeout_seconds": 30,
                "http": {
                    "url": "https://api.example.com/weather",
                    "http_method": "GET",
                    "headers": [
                        {"name": "Authorization", "value": "Bearer <secret>"}
                    ],
                },
            }
        ],
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js expandable
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Weather Assistant",
    system_prompt: "You are a friendly assistant. When asked about weather, call get_weather and read back the result.",
    voice: { voice_id: "alba" },
    tools: [
      {
        name: "get_weather",
        description: "Get current weather for a location. Use this whenever the user asks about weather, temperature, or conditions.",
        parameters: {
          type: "object",
          properties: {
            latitude: { type: "number", description: "Latitude in decimal degrees, e.g. 48.85." },
            longitude: { type: "number", description: "Longitude in decimal degrees, e.g. 2.35." },
          },
          required: ["latitude", "longitude"],
        },
        execution_mode: "interactive",
        timeout_seconds: 30,
        http: {
          url: "https://api.example.com/weather",
          http_method: "GET",
          headers: [{ name: "Authorization", value: "Bearer <secret>" }],
        },
      },
    ],
  }),
});
const data = await res.json();
console.log(data);
```

`parameters` is a **JSON Schema** object (the same format as [function tools](/voice-agents/voice-agent-api/tools/client-side-tools)), so the [accuracy hints](/voice-agents/voice-agent-api/tools/overview#parameter-hints-improve-accuracy) (`enum`, `examples`, `pattern`, `format`) work here too.

`headers` is a list of `{ name, value }` entries. On update, each entry can set, keep, or remove a header:

| Intent | Shape |
|---|---|
| Set or rotate the value | `{ "name": "Authorization", "value": "Bearer <new-secret>" }` |
| Keep the stored value (read responses don't include it) | `{ "name": "Authorization" }` |
| Remove the header | `{ "name": "Authorization", "remove": true }` |

## How arguments map to the request

The model's arguments are placed into the request based on the HTTP method:

| Method                 | Where arguments go                                               |
| ---------------------- | ---------------------------------------------------------------- |
| `GET`, `DELETE`        | **Query string.** Each value stringified; `null` values dropped. |
| `POST`, `PUT`, `PATCH` | **JSON body.** Native types preserved (numbers, nested objects). |

Query params already in the tool `url` are **merged** with the model's arguments, so you can pin fixed params in the URL and let the model fill the dynamic ones. The response body (capped at 8 KiB) is fed back to the model as the tool result.

## Securing and constraining HTTP tools

- **Auth headers are write-only.** Values in `http.headers` are encrypted at rest and never returned. Reads include only the header `name` and the time it was last set. Put API keys and bearer tokens here.
- **`https` and public hosts only.** Requests to private, loopback, or link-local addresses are blocked; redirects are not followed.
- **Errors reach the model as text.** A timeout or non-2xx returns a short message the model can recover from, so write your API to return useful error bodies, and tell the agent in its `system_prompt` how to handle a failure (for example, apologize and ask the caller to try again, or offer an alternative).

See the [Manage agents reference](/voice-agents/voice-agent-api/manage-agents#http-tool-config) for the full schema.

## Make them more accurate

Everything on the [Tools overview](/voice-agents/voice-agent-api/tools/overview) applies to HTTP tools:

- [Parameter hints](/voice-agents/voice-agent-api/tools/overview#parameter-hints-improve-accuracy): `enum`, `examples`, `pattern`, `format` per property.
- [Execution modes](/voice-agents/voice-agent-api/tools/overview#execution-modes): `interactive` vs `hold`.
- [Getting the agent to call your tools](/voice-agents/voice-agent-api/tools/overview#getting-the-agent-to-call-your-tools) and [per-agent patterns](/voice-agents/voice-agent-api/tools/overview#patterns-by-agent-type).

---

<a id="client-side-tools-docs-voice-agents-voice-agent-api-tools-client-side-tools-mdx"></a>

# Client-side tools

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/client-side-tools
Source: docs/voice-agents/voice-agent-api/tools/client-side-tools.mdx
Navigation: Voice Agent API > Agent behavior > Tools
Description: Function tools your own code runs. Declare them inline, handle the tool.call, and return a tool.result without breaking turn-taking.

Declare tools inline in `session.tools` when the logic must run in your own code: local state, in-memory lookups, or anything you don't want to expose as an HTTP endpoint. The agent emits a [`tool.call`](/voice-agents/voice-agent-api/events-reference#tool-call); you run the tool and reply with a [`tool.result`](/voice-agents/voice-agent-api/events-reference#tool-result) when [`reply.done`](/voice-agents/voice-agent-api/events-reference#reply-done) is the latest event you've received.

If your tool just calls an HTTP API, prefer a server-side [HTTP tool](/voice-agents/voice-agent-api/tools/http-tools): AssemblyAI makes the request for you and your client never handles the round trip.

## Quick start

```python
import asyncio, json, websockets

URL = "wss://agents.assemblyai.com/v1/ws"
TOOLS = [{
    "type": "function",
    "name": "get_weather",
    "description": "Get current weather for any city. Use this whenever the user asks about weather, temperature, or conditions. Prefer calling this over guessing.",
    "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string", "description": "City name (e.g. London)"}},
        "required": ["city"],
    },
}]

async def main():
    async with websockets.connect(URL, extra_headers={"Authorization": "Bearer YOUR_KEY"}) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a weather assistant. Call get_weather for weather questions. When in doubt, call the tool.",
                "greeting": "Hi! Ask me about the weather.",
                "tools": TOOLS,
                "output": {"type": "audio", "voice": "alba"},
            },
        }))

        last_event, pending = None, []

        async def flush_if_idle():
            if last_event != "reply.done" or not pending:
                return
            for t in pending:
                await ws.send(json.dumps({"type": "tool.result", "call_id": t["call_id"],
                                          "result": json.dumps(t["result"])}))
            pending.clear()

        async for raw in ws:
            event = json.loads(raw); t = event.get("type")
            if t == "tool.call" and event["name"] == "get_weather":
                pending.append({"call_id": event["call_id"], "result": {"temp_c": 22, "description": "Sunny"}})
                await flush_if_idle()
            elif t in ("reply.started", "input.speech.started"):
                last_event = t
            elif t == "reply.done":
                last_event = t
                if event.get("status") == "interrupted":
                    pending.clear()
                else:
                    await flush_if_idle()

asyncio.run(main())
```

## Defining function tools

Function tools use a JSON Schema for `parameters` and carry `"type": "function"`. [HTTP tools](/voice-agents/voice-agent-api/tools/http-tools) use the same JSON-Schema `parameters`; `description`, `execution_mode`, and `timeout_seconds` work the same way for both.

```json expandable
{
  "type": "session.update",
  "session": {
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get current weather for any city. Use this whenever the user asks about weather.",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string", "description": "City name, e.g. London"}},
          "required": ["location"]
        },
        "execution_mode": "interactive",
        "timeout_seconds": 120
      }
    ]
  }
}
```

| Field             | Type    | Default         | Notes                                                                  |
| ----------------- | ------- | --------------- | ---------------------------------------------------------------------- |
| `type`            | string  | (required)      | Always `"function"`.                                                   |
| `name`            | string  | (required)      | snake_case, verb-noun. Referenced by `tool.call`.                      |
| `description`     | string  | `""`            | The model's main signal for when to call. See [Getting the agent to call your tools](/voice-agents/voice-agent-api/tools/overview#getting-the-agent-to-call-your-tools). |
| `parameters`      | object  | `{}`            | JSON Schema. See below.                                                |
| `execution_mode`  | string  | `"interactive"` | `"interactive"` or `"hold"`. See [Execution modes](/voice-agents/voice-agent-api/tools/overview#execution-modes). |
| `timeout_seconds` | number  | `120`           | 1–300. On timeout the agent apologises; the session continues.         |

`session.tools` updates **replace** the previous array (not merge). See [Progressive tool reveal](/voice-agents/voice-agent-api/tools/overview#progressive-tool-reveal).

### Parameters

```json
{
  "type": "object",
  "properties": {
    "phone_number": {
      "type": "string",
      "description": "E.164 format (e.g. +14155551234). Strip spaces, parentheses, dashes."
    },
    "department": {
      "type": "string",
      "enum": ["billing", "sales", "support"],
      "description": "Pick the closest match (e.g. 'tech help' → support)."
    }
  },
  "required": ["phone_number", "department"]
}
```

- Lead with format ("E.164", "ISO-8601 date", "lowercase").
- Always include an example.
- Use `enum` for fixed sets.
- `required` only for fields the tool truly can't function without; otherwise the model interrogates the user.

The same [accuracy hints](/voice-agents/voice-agent-api/tools/overview#parameter-hints-improve-accuracy) (`enum`, `examples`, `pattern`, `format`) can go on each property here.

`parameters` is **not** validated at `session.update` time. Malformed schemas (missing `type: "object"`, broken `enum`) are accepted silently and break tool calling at runtime. Validate locally.

## Returning tool results

**Send `tool.result` when `reply.done` is the latest event you've received.** Not earlier (agent is still mid-transition-phrase), not later (a new turn has started).

```python
last_event: str | None = None
pending_tools: list[dict] = []

async def flush_if_idle():
    if last_event != "reply.done" or not pending_tools:
        return
    for tool in pending_tools:
        await ws.send(json.dumps({
            "type": "tool.result",
            "call_id": tool["call_id"],
            "result": json.dumps(tool["result"]),   # JSON string
        }))
    pending_tools.clear()

# In your event loop:
if t == "tool.call":
    result = run_tool(event["name"], event["arguments"])
    pending_tools.append({"call_id": event["call_id"], "result": result})
    await flush_if_idle()                # may already be idle if reply.done fired first

elif t in ("reply.started", "input.speech.started"):
    last_event = t                        # turn in flight, hold results

elif t == "reply.done":
    last_event = t
    if event.get("status") == "interrupted":
        pending_tools.clear()             # agent moved on, drop stale results
    else:
        await flush_if_idle()
```

Two non-obvious bits:

- Call `flush_if_idle()` from the `tool.call` handler. Your tool may return *after* `reply.done` already fired.
- Update `last_event` on `reply.started` / `input.speech.started` so results that become available mid-turn are held until that turn ends.

### Errors that help the agent recover

The `error` field is read verbatim by the model. Weak errors cause guessing loops; specific errors get clean recoveries.

Weak (agent re-asks for everything):

```json
{ "error": "Lookup failed." }
```

Strong (agent re-asks only for the field that failed):

```json
{ "error": "Could not resolve DROPOFF 'Central train station'. Pickup resolved ('SW1A 1AA'). Ask the user for a UK postcode for the dropoff." }
```

Patterns: name the failing field, say what *did* work so the agent doesn't re-ask for it, tell the agent what to ask for next.

## Going further

The shared mechanics live on the [Tools overview](/voice-agents/voice-agent-api/tools/overview): [execution modes](/voice-agents/voice-agent-api/tools/overview#execution-modes) (`interactive` vs `hold`), [progressive tool reveal](/voice-agents/voice-agent-api/tools/overview#progressive-tool-reveal), [per-agent patterns](/voice-agents/voice-agent-api/tools/overview#patterns-by-agent-type), and [debugging](/voice-agents/voice-agent-api/tools/overview#debugging).

---

<a id="voices-docs-voice-agents-voice-agent-api-voices-mdx"></a>

# Voices

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/voices
Source: docs/voice-agents/voice-agent-api/voices.mdx
Navigation: Voice Agent API > Conversational experience
Description: Pick a voice that matches your agent's personality and language.

Pick any voice ID from the tables below and set it as `voice` when you create the agent:

```bash cURL
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "voice": { "voice_id": "anna" }
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "voice": {"voice_id": "anna"},
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    voice: { voice_id: "anna" },
  }),
});
const data = await res.json();
console.log(data);
```

Configuring inline over the WebSocket instead of on a stored agent? Set it on `session.output.voice` in a [`session.update`](/voice-agents/voice-agent-api/events-reference#session-update) before `session.ready`:

```json
{
  "type": "session.update",
  "session": {
    "output": { "voice": "anna" }
  }
}
```

The voice is **immutable once the session is established**, so it can't be changed mid-conversation. Pick it before the agent connects.

## Language support

The agent recognizes far more input languages than it speaks. For the authoritative, up-to-date matrix, see [Supported languages](/voice-agents/voice-agent-api/supported-languages).

In short:

- **Input (recognized):** 18 languages, with native code-switching. Because the agent recognizes more languages than it speaks, this enables translation use cases where a caller speaks one language and the agent responds in another.
- **Output (spoken):** 6 officially supported languages: 🇺🇸🇬🇧 English, 🇮🇹 Italian, 🇪🇸 Spanish, 🇩🇪 German, 🇵🇹 Portuguese, and 🇫🇷 French. Each is backed by at least one voice below with a matching accent. More output languages are [on the roadmap](/voice-agents/voice-agent-api/supported-languages#coming-soon).

## Choose a voice by accent

Each voice has a primary accent:

- For **English** (the right pick for an English agent), choose from [English voices](#english-voices). Most have a 🇺🇸 American accent; `anna`, `charles`, `paul`, and `vera` have a 🇬🇧 British accent.
- For a **native accent** in Italian, Spanish, German, Portuguese, or French, choose the matching [language-specific voice](#language-specific-voices).

## English voices

These voices have an English accent and are the recommended pick for English agents.

| Voice      | Accent  |
| ---------- | ------- |
| `alba`     | 🇺🇸      |
| `eve`      | 🇺🇸      |
| `george`   | 🇺🇸      |
| `jane`     | 🇺🇸      |
| `jean`     | 🇺🇸      |
| `mary`     | 🇺🇸      |
| `michael`  | 🇺🇸      |
| `anna`     | 🇬🇧      |
| `charles`  | 🇬🇧      |
| `paul`     | 🇬🇧      |
| `vera`     | 🇬🇧      |

## Language-specific voices

These voices have a native accent in a specific non-English language and code-switch naturally between that language and English.

| Voice       | Native accent    |
| ----------- | ---------------- |
| `giovanni`  | 🇮🇹 Italian       |
| `lola`      | 🇪🇸 Spanish       |
| `juergen`   | 🇩🇪 German        |
| `rafael`    | 🇵🇹 Portuguese    |
| `estelle`   | 🇫🇷 French        |

---

<a id="output-volume-docs-voice-agents-voice-agent-api-volume-mdx"></a>

# Output volume

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/volume
Source: docs/voice-agents/voice-agent-api/volume.mdx
Navigation: Voice Agent API > Conversational experience
Description: Set how loud a voice agent's speech plays back.

Control how loud the agent's speech plays with `output.volume`. Set it when you [create](/voice-agents/voice-agent-api/create-agent) or [update](/voice-agents/voice-agent-api/manage-agents) the agent, or inline over the WebSocket via [`session.update`](/voice-agents/voice-agent-api/session-configuration).

Accepts a number from `0` (silent) to `100` (loudest). If omitted, the voice plays at its native level.

```bash cURL {3}
curl -X PUT https://agents.assemblyai.com/v1/agents/$AGENT_ID \
  -H "Authorization: $ASSEMBLYAI_API_KEY" -H "Content-Type: application/json" \
  -d '{ "output": { "volume": 60 } }'
```

```python Python {8}
# pip install requests
import os
import requests

resp = requests.put(
    f"https://agents.assemblyai.com/v1/agents/{os.environ['AGENT_ID']}",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={"output": {"volume": 60}},
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js {10}
// Node 18+ has fetch built in
const res = await fetch(
  `https://agents.assemblyai.com/v1/agents/${process.env.AGENT_ID}`,
  {
    method: "PUT",
    headers: {
      Authorization: process.env.ASSEMBLYAI_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ output: { volume: 60 } }),
  },
);
const data = await res.json();
console.log(data);
```

| Field           | Type          | Required | Notes                                                  |
| --------------- | ------------- | -------- | ------------------------------------------------------ |
| `output.volume` | number \| null | No       | `0` (silent) to `100` (loudest). `null` plays at native level. |

Unlike `output.voice` and `output.format`, which are **immutable** after `session.ready`, `output.volume` can be changed mid-session via [`session.update`](/voice-agents/voice-agent-api/session-configuration#mutability-after-session-ready). The new value applies to subsequent `reply.audio` chunks.

---

<a id="turn-detection-and-interruptions-docs-voice-agents-voice-agent-api-turn-detection-and-interruptions-mdx"></a>

# Turn detection and interruptions

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/turn-detection-and-interruptions
Source: docs/voice-agents/voice-agent-api/turn-detection-and-interruptions.mdx
Navigation: Voice Agent API > Conversational experience
Description: How the Voice Agent API decides when the user has finished speaking and when they're interrupting. On by default, no tuning required.

Turn detection and barge-in are on by default and based on what the user actually said, not just silence or volume. There's nothing to wire up.

**Leave it on default.** With no `turn_detection` config, the agent adapts to each speaker's pace and automatically slows down to capture values your tools need, like a phone number or email. The way to make turn-taking better is **good tool descriptions**, not VAD knobs. Tune the raw thresholds only as a last resort.

## What you get by default

- **Semantic end-of-turn.** The agent decides you're done from the meaning of what you said, so it won't cut you off mid-thought or sit on a long pause once you've clearly finished.
- **Smart waiting for tool values.** When a tool parameter expects a phone number, email, date, or other entity, the agent waits for the whole value before ending your turn, instead of jumping in after the first pause.
- **Adaptive pacing.** If a speaker pauses a lot, the agent gives them more room; if they're crisp, it replies faster. This gets better over the call.
- **Semantic barge-in.** Back-channels like "uh-huh" or "makes sense" don't interrupt; "wait, stop" does.

A normal user turn emits [`input.speech.started`](/voice-agents/voice-agent-api/events-reference#input-speech-started) → [`transcript.user.delta`](/voice-agents/voice-agent-api/events-reference#transcript-user-delta) (partials) → [`input.speech.stopped`](/voice-agents/voice-agent-api/events-reference#input-speech-stopped) → [`transcript.user`](/voice-agents/voice-agent-api/events-reference#transcript-user) (final) → [`reply.started`](/voice-agents/voice-agent-api/events-reference#reply-started). You never signal end-of-turn yourself.

## Get better turn-taking with tool descriptions

The agent waits for a complete value when it knows one is coming. It learns that from your tool's parameters, so describe the important ones well. No turn-detection config involved; just add the tool when you create the agent:

```bash cURL expandable
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Account Assistant",
    "system_prompt": "You verify callers. Ask for their phone number, then call verify_account.",
    "voice": { "voice_id": "alba" },
    "tools": [
      {
        "name": "verify_account",
        "description": "Look up a customer by phone number.",
        "parameters": {
          "type": "object",
          "properties": {
            "phone": {
              "type": "string",
              "description": "The caller'\''s phone number, including country code.",
              "examples": ["+14155552671"]
            }
          },
          "required": ["phone"]
        }
      }
    ]
  }'
```

```python Python expandable
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Account Assistant",
        "system_prompt": "You verify callers. Ask for their phone number, then call verify_account.",
        "voice": {"voice_id": "alba"},
        "tools": [
            {
                "name": "verify_account",
                "description": "Look up a customer by phone number.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "phone": {
                            "type": "string",
                            "description": "The caller's phone number, including country code.",
                            "examples": ["+14155552671"],
                        }
                    },
                    "required": ["phone"],
                },
            }
        ],
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js expandable
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Account Assistant",
    system_prompt: "You verify callers. Ask for their phone number, then call verify_account.",
    voice: { voice_id: "alba" },
    tools: [
      {
        name: "verify_account",
        description: "Look up a customer by phone number.",
        parameters: {
          type: "object",
          properties: {
            phone: {
              type: "string",
              description: "The caller's phone number, including country code.",
              examples: ["+14155552671"],
            },
          },
          required: ["phone"],
        },
      },
    ],
  }),
});
const data = await res.json();
console.log(data);
```

With this, when the agent asks for the phone number it waits for all the digits instead of cutting in after "my number is four one five…". A clear `description` (plus optional `examples` or `pattern`) is what drives it. See [parameter hints](/voice-agents/voice-agent-api/tools/overview#parameter-hints-improve-accuracy).

## Interruptions

When the user truly interrupts, the server emits:

- [`reply.done`](/voice-agents/voice-agent-api/events-reference#reply-done) with `status: "interrupted"`
- [`transcript.agent`](/voice-agents/voice-agent-api/events-reference#transcript-agent) with `interrupted: true` and `text` trimmed to what the user actually heard.

On that signal, stop and clear your queued audio so the user doesn't keep hearing stale speech:

```javascript
ws.onmessage = ({ data }) => {
  const m = JSON.parse(data);
  if (m.type === "input.speech.started") flushPlayback();   // snappiest barge-in
  if (m.type === "reply.done" && m.status === "interrupted") flushPlayback();
};
```

See [Handling interruptions](/voice-agents/voice-agent-api/audio-format#handling-interruptions) for the full audio-flush pattern.

## Transcription mode

Turn detection sits on top of the transcript, and the transcript comes from a speech-to-text model whose speed-versus-accuracy balance you set with `input.transcription_mode`. The mode also presets how long the model waits in silence before ending a turn, so it is the cleanest single knob for that tradeoff. Reach for it before the raw thresholds below.

| Mode           | Behavior                                                                    |
| -------------- | --------------------------------------------------------------------------- |
| `min_latency`  | Ends turns fastest and is least patient in silence.                         |
| `balanced`     | Default. A middle ground.                                                   |
| `max_accuracy` | Most accurate transcript; waits longest to confirm the end of a turn.       |

Set it on the stored agent when you create it:

```bash cURL {8}
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent.",
    "voice": { "voice_id": "alba" },
    "input": { "transcription_mode": "max_accuracy" }
  }'
```

```python Python {12}
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent.",
        "voice": {"voice_id": "alba"},
        "input": {"transcription_mode": "max_accuracy"},
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js {12}
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent.",
    voice: { voice_id: "alba" },
    input: { transcription_mode: "max_accuracy" },
  }),
});
const data = await res.json();
console.log(data);
```

`transcription_mode` is updatable mid-session, so switch it by call stage: run `balanced` or `min_latency` for normal back-and-forth, switch to `max_accuracy` while capturing something you can't get wrong (an order ID, account number, or a name being spelled out), then switch back.

```json
// Entering an entity-capture step
{ "type": "session.update", "session": { "input": { "transcription_mode": "max_accuracy" } } }

// Value captured
{ "type": "session.update", "session": { "input": { "transcription_mode": "min_latency" } } }
```

## Override the defaults (rarely needed)

If you must tune sensitivity, set the VAD knobs in `input.turn_detection`. All fields are optional; send only what you change. Set them on the stored agent when you create it:

```bash cURL
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "voice": { "voice_id": "alba" },
    "input": {
      "turn_detection": {
        "vad_threshold": 0.5,
        "min_silence": 1000,
        "max_silence": 3000,
        "interrupt_response": true
      }
    }
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "voice": {"voice_id": "alba"},
        "input": {
            "turn_detection": {
                "vad_threshold": 0.5,
                "min_silence": 1000,
                "max_silence": 3000,
                "interrupt_response": True,
            }
        },
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    voice: { voice_id: "alba" },
    input: {
      turn_detection: {
        vad_threshold: 0.5,
        min_silence: 1000,
        max_silence: 3000,
        interrupt_response: true,
      },
    },
  }),
});
const data = await res.json();
console.log(data);
```

Configuring inline over the WebSocket instead? Set the same [`turn_detection`](/voice-agents/voice-agent-api/session-configuration#turn-detection) object under `session.input` in a `session.update`:

```json
{
  "type": "session.update",
  "session": {
    "input": {
      "turn_detection": {
        "vad_threshold": 0.5,
        "min_silence": 1000,
        "max_silence": 3000,
        "interrupt_response": true
      }
    }
  }
}
```

| Field                | Default   | Description |
| -------------------- | --------- | ----------- |
| `vad_threshold`      | `0.5`     | Speech detection sensitivity (`0.0` to `1.0`). Lower is more sensitive. |
| `min_silence`        | adaptive  | Minimum silence for a confident end-of-turn, in ms. Left unset, the agent paces this adaptively. |
| `max_silence`        | adaptive  | Maximum silence before forcing end-of-turn, in ms. Left unset, the agent paces this adaptively. |
| `interrupt_response` | `true`    | Set `false` to disable barge-in entirely. |
| `interruption_delay` | per mode  | How long after the user starts speaking, in ms (`0` to `1000`), before a barge-in can interrupt the agent. Follows the [transcription mode](#transcription-mode) (`0` for `min_latency`, `500` for `balanced` and `max_accuracy`). Raise it so brief back-channels like "mm-hmm" don't cut the agent off. |

Setting `min_silence` or `max_silence` turns off the adaptive pacing and entity-aware waiting described above for the rest of the session. Prefer leaving them unset.

If the agent keeps interrupting itself, the mic is picking up its own TTS. Use headphones or a [browser-based client](/voice-agents/voice-agent-api/browser-integration) (which has echo cancellation). See [Troubleshooting](/voice-agents/voice-agent-api/troubleshooting#agent-interrupts-itself-echo--feedback-loop).

---

<a id="add-transcription-context-docs-voice-agents-voice-agent-api-transcription-prompt-mdx"></a>

# Add transcription context

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/transcription-prompt
Source: docs/voice-agents/voice-agent-api/transcription-prompt.mdx
Navigation: Voice Agent API > Conversational experience
Description: Improve a voice agent's transcription accuracy for domain terms with a contextual prompt and a key terms list.

Two levers make the agent's **speech-to-text** transcribe the caller more accurately: a free-form **transcription prompt** that describes the call, and a **key terms** list of exact words to boost. Use them together: the prompt sets the scene, and key terms nail specific names and codes. Both live under `input`, set when you [create](/voice-agents/voice-agent-api/create-agent) or [update](/voice-agents/voice-agent-api/manage-agents) the agent, or inline via [`session.update`](/voice-agents/voice-agent-api/session-configuration). For how both work under the hood, see [Prompting and keyterms](/streaming/prompting-and-keyterms) in the Streaming docs.

## Contextual prompt

`input.transcription_prompt` tells the speech-to-text model what the call is about so it resolves jargon, product names, and homophones in what the caller says. Describe the domain and scenario in plain language.

This is **not** the same as `system_prompt`. `system_prompt` shapes what the **agent says**; `transcription_prompt` shapes how the caller's speech is **transcribed**. Give it context ("this is a pharmacy support call"), not instructions ("use formal punctuation"); behavioral commands are ignored.

```bash cURL {9}
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Pharmacy Assistant",
    "system_prompt": "You help patients with prescription refills.",
    "voice": { "voice_id": "alba" },
    "input": {
      "transcription_prompt": "A pharmacy support call. Callers mention medication names, dosages in mg, insurance carriers, and pharmacy locations."
    }
  }'
```

```python Python {13}
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Pharmacy Assistant",
        "system_prompt": "You help patients with prescription refills.",
        "voice": {"voice_id": "alba"},
        "input": {
            "transcription_prompt": "A pharmacy support call. Callers mention medication names, dosages in mg, insurance carriers, and pharmacy locations."
        },
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js {13}
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Pharmacy Assistant",
    system_prompt: "You help patients with prescription refills.",
    voice: { voice_id: "alba" },
    input: {
      transcription_prompt:
        "A pharmacy support call. Callers mention medication names, dosages in mg, insurance carriers, and pharmacy locations.",
    },
  }),
});
const data = await res.json();
console.log(data);
```

| Field                        | Type   | Required | Notes                                                             |
| ---------------------------- | ------ | -------- | ---------------------------------------------------------------- |
| `input.transcription_prompt` | string | No       | Free-form context about the call, up to 1750 characters. Updatable mid-session. |

## Key terms

`input.keyterms` is an explicit list of rare or domain-specific words to boost, like a person's name, a product, or an acronym. It works as a word boost, biasing the speech recognition model toward each term. Reach for it when you have specific strings to nail, and use the prompt above for the surrounding context.

```bash cURL {8}
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "voice": { "voice_id": "alba" },
    "input": { "keyterms": ["AssemblyAI", "Lemur", "Ozempic"] }
  }'
```

```python Python {12}
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Support Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "voice": {"voice_id": "alba"},
        "input": {"keyterms": ["AssemblyAI", "Lemur", "Ozempic"]},
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js {12}
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Support Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    voice: { voice_id: "alba" },
    input: { keyterms: ["AssemblyAI", "Lemur", "Ozempic"] },
  }),
});
const data = await res.json();
console.log(data);
```

| Field            | Type              | Required | Notes                                                                  |
| ---------------- | ----------------- | -------- | ---------------------------------------------------------------------- |
| `input.keyterms` | string[] \| null  | No       | Up to 100 transcription-bias strings. `null` or `[]` clears the boost. |

**Add:**

- Brand, product, and feature names that aren't in everyday English (`"AssemblyAI"`, `"Lemur"`, `"Ozempic"`).
- Proper nouns specific to this caller: their full name, their account holder's name, the agent's name if it's unusual.
- Domain jargon the model might otherwise transcribe as a common-word homophone (`"hemochromatosis"`, `"polysomnography"`).
- Acronyms you want spelled in full (`"PCI DSS"`, `"FedRAMP"`).

**Don't add:**

- Common English words. Each entry boosts that string, and adding common words at the same weight as your rare terms dilutes the boost.
- Whole sentences or phrases. The boost is per-term, not per-phrase.
- Punctuation, formatting, or instructions. The list is transcription hints, not prompt context.

Replace the list any time by sending a new `keyterms` array; it takes effect on the next user utterance. Refresh it when the conversation enters a new domain, for example when a support call moves from billing to technical.

---

<a id="steer-toward-known-languages-docs-voice-agents-voice-agent-api-language-selection-mdx"></a>

# Steer toward known languages

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/language-selection
Source: docs/voice-agents/voice-agent-api/language-selection.mdx
Navigation: Voice Agent API > Conversational experience
Description: Steer a voice agent's speech-to-text toward the languages you expect with language_codes, or leave it on automatic multilingual detection.

The agent transcribes any [supported language](/voice-agents/voice-agent-api/supported-languages) automatically and code-switches natively, so a caller can move between, say, English and Spanish mid-sentence with no configuration. When you know which language(s) a line serves, set `input.language_codes` to steer transcription toward them for a bit more accuracy. Omit it for automatic detection. Set it when you [create](/voice-agents/voice-agent-api/create-agent) or [update](/voice-agents/voice-agent-api/manage-agents) the agent.

- **Global or mixed-language line**: omit `language_codes`. The agent detects and code-switches on its own.
- **Region-specific line**: pin the language(s) you expect. A Spanish support line set to `["es"]` won't drift into look-alike transcriptions of another language.

```bash cURL {8}
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Soporte",
    "system_prompt": "Eres un agente de soporte amable. Responde en español.",
    "voice": { "voice_id": "lola" },
    "input": { "language_codes": ["es"] }
  }'
```

```python Python {12}
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Soporte",
        "system_prompt": "Eres un agente de soporte amable. Responde en español.",
        "voice": {"voice_id": "lola"},
        "input": {"language_codes": ["es"]},
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js {12}
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Soporte",
    system_prompt: "Eres un agente de soporte amable. Responde en español.",
    voice: { voice_id: "lola" },
    input: { language_codes: ["es"] },
  }),
});
const data = await res.json();
console.log(data);
```

| Field                  | Type     | Required | Notes                                                                     |
| ---------------------- | -------- | -------- | ------------------------------------------------------------------------- |
| `input.language_codes` | string[] | No       | Languages to steer toward. Omit (or leave empty) for automatic detection. Pass one code to pin a single language, or several to constrain to a set. |

## Supported language codes

Universal-3.5 Pro Streaming code-switches across all 18 languages below. Pass any of these codes in `language_codes`:

| Language   | Code | Language   | Code |
| ---------- | ---- | ---------- | ---- |
| English    | `en` | Danish     | `da` |
| Spanish    | `es` | Finnish    | `fi` |
| French     | `fr` | Hindi      | `hi` |
| German     | `de` | Vietnamese | `vi` |
| Italian    | `it` | Arabic     | `ar` |
| Portuguese | `pt` | Hebrew     | `he` |
| Turkish    | `tr` | Japanese   | `ja` |
| Dutch      | `nl` | Mandarin   | `zh` |
| Swedish    | `sv` | Norwegian  | `no` |

Set the input language and a matching [voice](/voice-agents/voice-agent-api/voices) together: transcribing Spanish while replying in an English voice is jarring. The example above pairs `["es"]` with the Spanish voice `lola`.

`language_codes` is applied when the speech-to-text connection opens; a mid-session change takes effect on the next reconnect, not the current turn. For how native code-switching works, see [Multilingual transcription](/streaming/multilingual-transcription).

---

<a id="isolate-the-caller-s-voice-docs-voice-agents-voice-agent-api-noise-suppression-mdx"></a>

# Isolate the caller's voice

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/noise-suppression
Source: docs/voice-agents/voice-agent-api/noise-suppression.mdx
Navigation: Voice Agent API > Conversational experience
Description: Isolate the caller's voice and suppress background noise before it reaches the agent's speech-to-text, with voice_focus.

Real calls are noisy: keyboard clacks, café chatter, a car engine, room echo. That noise makes the agent mishear words and, worse, can trip false turn-taking and interruptions. `input.voice_focus` isolates the primary speaker and suppresses background audio before it reaches transcription, so the agent responds to the caller and not the room. Set it when you [create](/voice-agents/voice-agent-api/create-agent) or [update](/voice-agents/voice-agent-api/manage-agents) the agent.

| Model         | Value        | Use for                                                                    |
| ------------- | ------------ | -------------------------------------------------------------------------- |
| Near field    | `near-field` | Headsets, handsets, and other close-talking mics. Default.                 |
| Far field     | `far-field`  | Phone speakerphone, conference rooms, drive-thru, laptop and car mics.     |

Match the model to how the caller is captured: a browser agent on a headset wants `near-field`; a phone or in-room agent wants `far-field`.

```bash cURL {8}
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Drive-thru Assistant",
    "system_prompt": "You take drive-thru orders.",
    "voice": { "voice_id": "alba" },
    "input": { "voice_focus": "far-field", "voice_focus_threshold": 0.8 }
  }'
```

```python Python {12}
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Drive-thru Assistant",
        "system_prompt": "You take drive-thru orders.",
        "voice": {"voice_id": "alba"},
        "input": {"voice_focus": "far-field", "voice_focus_threshold": 0.8},
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js {12}
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Drive-thru Assistant",
    system_prompt: "You take drive-thru orders.",
    voice: { voice_id: "alba" },
    input: { voice_focus: "far-field", voice_focus_threshold: 0.8 },
  }),
});
const data = await res.json();
console.log(data);
```

| Field                         | Type   | Required | Notes                                                                       |
| ----------------------------- | ------ | -------- | --------------------------------------------------------------------------- |
| `input.voice_focus`           | string | No       | `near-field` (default) or `far-field`.                                      |
| `input.voice_focus_threshold` | number | No       | `0.0` to `1.0`, default `0.7`. Higher suppresses more aggressively. Requires `voice_focus`. |

Voice Focus is applied when the speech-to-text connection opens, so set it at agent-create or connect time. For how the models and threshold behave under the hood, see [Voice Focus](/streaming/voice-focus) in the Streaming docs.

---

<a id="deploy-your-agent-docs-voice-agents-voice-agent-api-deploy-mdx"></a>

# Deploy your agent

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/deploy
Source: docs/voice-agents/voice-agent-api/deploy.mdx
Navigation: Voice Agent API > Deploy
Description: Deploy your agent documentation.

One agent, bound by its `agent_id`, answers wherever you point it. Pick where.

  
    A web app, widget, or in-product voice. Your server mints a short-lived token so no API key reaches the page.
  
  
    An inbound number people call. Twilio hands the call straight to AssemblyAI over SIP.
  

|                  | Browser                                        | Phone                                          |
| ---------------- | ---------------------------------------------- | ---------------------------------------------- |
| **Best for**     | Web apps, widgets, demos                       | Support lines, reception, order taking         |
| **You provide**  | A server endpoint that mints tokens            | A Twilio account and a number you own          |
| **You run**      | Your web app                                   | Nothing                                        |
| **Auth**         | Short-lived token, per session                 | API key, server-side at setup                  |
| **Audio**        | The browser captures and plays it              | Twilio and AssemblyAI handle it end to end     |
| **Echo**         | Handled by the browser, no headphones needed   | Handled by the carrier                         |
| **Setup**        | Minutes                                        | About 10 minutes, mostly Twilio config         |

Want both? Publish the agent once and point both at the same `agent_id`. The [Quickstart](/voice-agents/voice-agent-api) does exactly that in five commands.

## Whichever you pick

Both open the same WebSocket, `wss://agents.assemblyai.com/v1/ws`, and bind the same way. Send one `session.update` with the `agent_id` and nothing else, and the stored prompt, voice, and tools load server-side:

```json
{
  "type": "session.update",
  "session": { "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9" }
}
```

Wait for [`session.ready`](/voice-agents/voice-agent-api/events-reference#session-ready) before streaming audio, and finish with [`session.end`](/voice-agents/voice-agent-api/events-reference#session-end). Skipping `session.end` leaves the session in a 30-second grace window that you pay for. See [Ending the session cleanly](/voice-agents/voice-agent-api/browser-integration#6-ending-the-session-cleanly).

`agent_id` is **mutually exclusive** with inline session fields. Binding to a stored agent and also sending `system_prompt`, `greeting`, `tools`, `input`, or `output` is rejected. To configure per session instead, send those inline and omit `agent_id`. See [Inline configuration](/voice-agents/voice-agent-api/session-configuration).

## Neither of these

On another telephony provider, or embedding in something we don't have a guide for? You connect the same way with your API key in the `Authorization` header, and move the audio yourself. [Stream audio](/voice-agents/voice-agent-api/audio-format) has the encodings and the send, play, and barge-in handling.

A raw API key is fine for servers and trusted backends. **Never ship it in browser or mobile client code.** Use the [browser token flow](/voice-agents/voice-agent-api/browser-integration) for anything client-side.

---

<a id="browser-integration-docs-voice-agents-voice-agent-api-browser-integration-mdx"></a>

# Browser integration

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/browser-integration
Source: docs/voice-agents/voice-agent-api/browser-integration.mdx
Navigation: Voice Agent API > Deploy
Description: Connect browser-based apps to the Voice Agent API using a temporary token.

Connect a browser to your [stored agent](/voice-agents/voice-agent-api/create-agent) in three steps:

1. Your server calls [`GET /v1/token`](/api-reference/voice-agent-api/generate-voice-agent-token) with your API key to mint a short-lived temporary token.
2. Your browser opens the WebSocket with `?token=<token>`, no API key exposed.
3. The browser sends one `session.update` with your `agent_id`; the agent's stored prompt, voice, and tools load automatically.

Your API key never leaves your server. Each token is single-use, it starts exactly one session, and all usage is attributed to the key that generated it.

This page connects to a **stored agent** by `agent_id`, the recommended path. If you'd rather configure the agent inline per session instead of creating one, send `system_prompt` / `greeting` / `output` in the `session.update` and omit `agent_id`. The two are mutually exclusive. See [Inline configuration](/voice-agents/voice-agent-api/session-configuration).

Browsers provide built-in acoustic echo cancellation through `getUserMedia`, so browser-based clients work hands-free without headphones. If you're developing on a laptop, the browser integration is the recommended starting point.

## 1. Generate a token on your server

Call [`GET /v1/token`](/api-reference/voice-agent-api/generate-voice-agent-token) with your API key in the `Authorization` header. Pick an `expires_in_seconds` short enough to limit replay risk (60–300s is a good default) and an optional `max_session_duration_seconds` to cap the session length.

These two parameters control different things and are easy to confuse:

- **`expires_in_seconds`** is the *token redemption window*: how long the client has to use this token to open a WebSocket. If the window elapses before the WebSocket is opened, the server returns a [`session.error`](/voice-agents/voice-agent-api/events-reference#session-error) with code `unauthorized` on the first frame instead of `session.ready`. Once a `session.ready` has been received, this value no longer applies.
- **`max_session_duration_seconds`** is the *session duration cap*: how long the resulting voice agent session is allowed to run after the WebSocket is open.

View the endpoint reference.

```javascript expandable
// server/routes/voice-token.js
import express from "express";

const router = express.Router();

router.get("/voice-token", async (_req, res) => {
  const url = new URL("https://agents.assemblyai.com/v1/token");
  url.searchParams.set("expires_in_seconds", "300");
  url.searchParams.set("max_session_duration_seconds", "8640");

  const response = await fetch(url, {
    headers: { Authorization: `Bearer ${process.env.ASSEMBLYAI_API_KEY}` },
  });

  if (!response.ok) {
    return res.status(response.status).send(await response.text());
  }

  const { token } = await response.json();
  res.json({ token });
});

export default router;
```

`expires_in_seconds` must be between 1 and 600. `max_session_duration_seconds` must be between 60 and 10800 (defaults to 10800, the 3-hour maximum session duration).

Tokens are **single-use**: fetch a fresh one immediately before every connection, including reconnects via [`session.resume`](/voice-agents/voice-agent-api/events-reference#session-resume). End sessions cleanly with [`session.end`](/voice-agents/voice-agent-api/events-reference#session-end) so you don't pay for the 30-second resume grace window.

## 2. Connect from the browser with the token

Fetch the token from your server, open the WebSocket with `?token=<token>` (no `Authorization` header needed), and bind to your agent by `agent_id`:

```javascript expandable
// browser/voice-agent.js
const AGENT_ID = "7ad24396-b822-4dca-871a-be9cc4781cf9"; // from POST /v1/agents

const { token } = await fetch("/api/voice-token").then((r) => r.json());

const wsUrl = new URL("wss://agents.assemblyai.com/v1/ws");
wsUrl.searchParams.set("token", token);
const ws = new WebSocket(wsUrl);

ws.addEventListener("open", () => {
  ws.send(
    JSON.stringify({
      type: "session.update",
      session: { agent_id: AGENT_ID },
    }),
  );
});

ws.addEventListener("message", (event) => {
  const message = JSON.parse(event.data);
  // Handle session.ready, reply.audio, transcript.*, tool.call, etc.
  console.log(message);
});
```

## 3. Browser client

### The full app

The [Python](https://github.com/AssemblyAI/voice-agent-starter-python) and [JavaScript](https://github.com/AssemblyAI/voice-agent-starter-js) starters ship a browser client with the token endpoint from step 1 already wired up, plus a live event log of every frame in both directions:

```bash Python
python deployment/browser/server.py
```

```bash JavaScript
npm start
```

It handles the parts that are easy to get wrong: capture and playback each run in their own `AudioContext` with a resampling worklet, so a browser that refuses to open a context at 24 kHz still sounds right. Start there if you want something working before you start cutting.

### The lite version

A minimal client that captures microphone audio, streams it to the Voice Agent API, and plays back the agent's response. Two files: an HTML page and an AudioWorklet processor.

AudioWorklet processors load from a URL, so this needs two files. Serve them locally with `npx serve .`.

Create `pcm-processor.js` in the same directory as your HTML file:

```javascript
// pcm-processor.js - AudioWorklet that captures PCM16 from the mic
class PCMProcessor extends AudioWorkletProcessor {
  process(inputs) {
    const input = inputs[0]?.[0];
    if (input) {
      // Convert Float32 [-1, 1] to Int16
      const pcm16 = new Int16Array(input.length);
      for (let i = 0; i < input.length; i++) {
        pcm16[i] = Math.max(-32768, Math.min(32767, Math.round(input[i] * 32767)));
      }
      this.port.postMessage(pcm16.buffer, [pcm16.buffer]);
    }
    return true;
  }
}

registerProcessor("pcm-processor", PCMProcessor);
```

Then create your HTML file:

```html expandable
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Voice Agent</title>
</head>
<body>
  <button id="start">Start conversation</button>
  <pre id="log"></pre>
  <script>
    const log = (msg) => { document.getElementById("log").textContent += msg + "\n"; };

    // Your stored agent's ID, from POST /v1/agents
    const AGENT_ID = "7ad24396-b822-4dca-871a-be9cc4781cf9";

    document.getElementById("start").addEventListener("click", async () => {
      // 1. Get token from your server (see step 1 above)
      const { token } = await fetch("/api/voice-token").then((r) => r.json());

      // 2. Force AudioContext to 24 kHz - avoids manual resampling on both
      //    capture and playback in Chromium and Firefox. Safari ignores this
      //    option (see Browser compatibility below) and runs at the hardware
      //    rate, so production code should resample inside the worklet.
      const audioCtx = new AudioContext({ sampleRate: 24000 });
      await audioCtx.audioWorklet.addModule("pcm-processor.js");

      // 3. Capture mic audio with echo cancellation enabled
      const stream = await navigator.mediaDevices.getUserMedia({
        audio: { echoCancellation: true, sampleRate: 24000 },
      });
      const source = audioCtx.createMediaStreamSource(stream);
      const worklet = new AudioWorkletNode(audioCtx, "pcm-processor");

      // 4. Connect WebSocket
      const wsUrl = new URL("wss://agents.assemblyai.com/v1/ws");
      wsUrl.searchParams.set("token", token);
      const ws = new WebSocket(wsUrl);

      let ready = false;
      let playbackTime = audioCtx.currentTime;

      // Send mic audio to the server once the session is ready
      worklet.port.onmessage = (e) => {
        if (ready && ws.readyState === WebSocket.OPEN) {
          const b64 = btoa(String.fromCharCode(...new Uint8Array(e.data)));
          ws.send(JSON.stringify({ type: "input.audio", audio: b64 }));
        }
      };
      source.connect(worklet).connect(audioCtx.destination);

      ws.addEventListener("open", () => {
        ws.send(JSON.stringify({
          type: "session.update",
          session: { agent_id: AGENT_ID },
        }));
      });

      ws.addEventListener("message", (event) => {
        const msg = JSON.parse(event.data);

        if (msg.type === "session.ready") {
          ready = true;
          log("Session ready, start speaking");
        } else if (msg.type === "reply.audio") {
          // Decode base64 PCM16 and schedule playback
          const raw = atob(msg.data);
          const pcm16 = new Int16Array(raw.length / 2);
          for (let i = 0; i < pcm16.length; i++) {
            pcm16[i] = raw.charCodeAt(i * 2) | (raw.charCodeAt(i * 2 + 1) << 8);
          }
          const float32 = new Float32Array(pcm16.length);
          for (let i = 0; i < pcm16.length; i++) {
            float32[i] = pcm16[i] / 32768;
          }
          const buffer = audioCtx.createBuffer(1, float32.length, 24000);
          buffer.getChannelData(0).set(float32);
          const src = audioCtx.createBufferSource();
          src.buffer = buffer;
          src.connect(audioCtx.destination);
          const now = audioCtx.currentTime;
          playbackTime = Math.max(playbackTime, now);
          src.start(playbackTime);
          playbackTime += buffer.duration;
        } else if (msg.type === "reply.done" && msg.status === "interrupted") {
          // Reset playback schedule to avoid stale audio
          playbackTime = audioCtx.currentTime;
        } else if (msg.type === "transcript.user") {
          log("You: " + msg.text);
        } else if (msg.type === "transcript.agent") {
          log("Agent: " + msg.text);
        } else if (msg.type === "session.error" || msg.type === "error") {
          log("Error: " + msg.message);
        }
      });

      ws.addEventListener("close", () => log("Connection closed"));
    });
  </script>
</body>
</html>
```

The key line is `new AudioContext({ sampleRate: 24000 })`. Most browsers default to the device sample rate (usually 48 kHz), so without this you'd need to manually resample both mic input and playback output. Forcing 24 kHz on the context avoids this entirely. **Safari ignores this option** and runs at the hardware rate. See [Browser compatibility](#5-browser-compatibility) for a Safari-safe pipeline.

## 4. Audio capture and echo cancellation

Ask for the microphone with `echoCancellation` on and `noiseSuppression` off:

```javascript
const stream = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: false },
});
```

**Echo cancellation on.** When the agent's speech plays through the speakers, the microphone picks it up and sends it back, and the agent interrupts itself. Every response gets cut short with `status: "interrupted"`. The browser's built-in acoustic echo cancellation removes speaker output from the mic signal, which is why a browser client needs no headphones.

**Noise suppression off.** The server already cleans up the input, and stacking a second denoising layer on top introduces artifacts that hurt transcription accuracy more than the original noise did. Same goes for RNNoise, Krisp, BVC, and similar. To tune how aggressively the server isolates the caller from background noise, use `input.voice_focus` instead, see [Isolate the caller's voice](/voice-agents/voice-agent-api/noise-suppression).

## 5. Browser compatibility

The client above works as-is on Chromium-based browsers (Chrome, Edge, Brave, Arc) and Firefox. Safari has a known quirk that produces silently garbled audio if you don't account for it.

| Browser            | `AudioContext({ sampleRate })` honored | Recommended pipeline                                                                                                  |
| ------------------ | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Chrome / Edge      | Yes                                    | Use the quickstart as-is.                                                                                             |
| Firefox            | Yes                                    | Use the quickstart as-is.                                                                                             |
| Safari (desktop, iOS) | **No**, runs at hardware rate (typically 48 kHz) | Let `AudioContext` use its default rate and resample to/from 24 kHz inside the worklet (capture) and before playback. |

### Safari: resample inside the worklet

Safari ignores the `sampleRate` constructor option, so an `AudioContext({ sampleRate: 24000 })` will silently run at 48 kHz on most Macs. Sending those samples to the Voice Agent API as if they were 24 kHz produces audio that sounds chipmunked or garbled.

Detect the actual context rate at runtime, send it into the worklet, and resample there:

```javascript
// browser/voice-agent.js (Safari-safe context)
const audioCtx = new AudioContext(); // let Safari pick its hardware rate
await audioCtx.audioWorklet.addModule("pcm-processor.js");
const worklet = new AudioWorkletNode(audioCtx, "pcm-processor", {
  processorOptions: { inputSampleRate: audioCtx.sampleRate, targetSampleRate: 24000 },
});
```

```javascript
// pcm-processor.js: linear resample to 24 kHz before posting PCM16
class PCMProcessor extends AudioWorkletProcessor {
  constructor(options) {
    super();
    const { inputSampleRate, targetSampleRate } = options.processorOptions;
    this.ratio = inputSampleRate / targetSampleRate;
  }
  process(inputs) {
    const input = inputs[0]?.[0];
    if (!input) return true;
    const outLength = Math.floor(input.length / this.ratio);
    const pcm16 = new Int16Array(outLength);
    for (let i = 0; i < outLength; i++) {
      const sample = input[Math.floor(i * this.ratio)] ?? 0;
      pcm16[i] = Math.max(-32768, Math.min(32767, Math.round(sample * 32767)));
    }
    this.port.postMessage(pcm16.buffer, [pcm16.buffer]);
    return true;
  }
}
registerProcessor("pcm-processor", PCMProcessor);
```

For playback, `createBuffer(1, length, 24000)` works on all current browsers, since the context resamples on output. Linear interpolation is good enough for speech.

### Cross-browser checklist

- **User gesture required.** All major browsers gate `getUserMedia` and `AudioContext` startup behind a user gesture (Safari is strictest). Start audio inside a `click` or `touchstart` handler and call `await audioCtx.resume()` before connecting nodes.
- **HTTPS or `localhost`.** `getUserMedia` only works on secure origins.
- **Echo cancellation.** Pass `echoCancellation: true` to `getUserMedia` so the agent's TTS playing through the speakers doesn't get re-captured by the mic.
- **Audio output sink.** On iOS Safari, set the [`<audio playsinline>`](https://developer.mozilla.org/docs/Web/HTML/Element/video#playsinline) attribute or route through an `AudioContext` destination. Autoplay and full-screen behavior differ from desktop.

## 6. Ending the session cleanly

  Do not just close the WebSocket. Send [`session.end`](/voice-agents/voice-agent-api/events-reference#session-end) first, then close. A bare `ws.close()` (or the browser tearing the socket down on navigation) leaves the session in the 30-second [`session.resume`](/voice-agents/voice-agent-api/events-reference#session-resume) grace window, and that window is billable.

Wire `session.end` to your explicit end-call control, wait for [`session.ended`](/voice-agents/voice-agent-api/events-reference#session-ended), and clean up there:

```js
function endCall() {
  if (ws && ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: "session.end" }));
    // Server emits session.ended, then closes the socket. Clean up on that event.
  } else {
    cleanup();
  }
}

function cleanup() {
  ws?.close();
  stream?.getTracks().forEach((t) => t.stop());
  audioCtx?.close();
  log("Call ended");
}

// Inside your existing message handler:
//   else if (msg.type === "session.ended") cleanup();
```

Cover tab close and navigation too. `pagehide` fires on both and is more reliable than `beforeunload` on mobile Safari:

```js
window.addEventListener("pagehide", () => {
  if (ws && ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({ type: "session.end" }));
  }
});
```

  Send `session.end` synchronously inside the `pagehide` handler. Anything async (like `await fetch`) will not finish before the socket is torn down.

Server clients follow the same pattern on their shutdown signal (`SIGINT`, `SIGTERM`, or your own hangup handler). See [Unexpected billing after the call ended](/voice-agents/voice-agent-api/troubleshooting#unexpected-billing-after-the-call-ended) if your sessions look about 30 seconds longer than the call itself.

---

<a id="set-up-an-inbound-phone-agent-via-sip-docs-voice-agents-voice-agent-api-connect-to-twilio-mdx"></a>

# Set up an inbound phone agent via SIP

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/connect-to-twilio
Source: docs/voice-agents/voice-agent-api/connect-to-twilio.mdx
Navigation: Voice Agent API > Deploy
Description: Point a Twilio number you already own at AssemblyAI over a SIP trunk and attach a voice agent. Inbound only.

Twilio passes inbound calls to AssemblyAI over SIP, so there's no media server, audio bridge, or webhook to run.

```
Caller → your Twilio number → SIP trunk → AssemblyAI → your agent
```

**You need:**

- A [Twilio account](https://console.twilio.com) (Account SID + Auth Token from the Console home page) with a phone number you already own
- An [AssemblyAI API key](https://www.assemblyai.com/dashboard/api-keys)
- An `agent_id`. Don't have one? [Create an agent](/voice-agents/voice-agent-api/create-agent) first.
- The [Twilio CLI](https://www.twilio.com/docs/twilio-cli/quickstart) installed and logged in (`twilio login`), plus `curl`, `jq`, and `uuidgen`

Prefer clicking? You can do every Twilio step in the [Console](https://console.twilio.com) under **Voice → Manage → Elastic SIP Trunking**. The CLI just makes it scriptable.

Five steps, three on Twilio and two on AssemblyAI:

1. Create a SIP trunk
2. Route its origination to `sip:sip.assemblyai.com`
3. Attach your number to the trunk
4. Register the number with AssemblyAI
5. Bind your agent to it

## 1. Set your details

Save these in a `.env` file in your working directory:

```bash .env icon=gear
AAI_API_KEY="your-assemblyai-api-key"
AGENT_ID="7ad24396-b822-4dca-871a-be9cc4781cf9"    # the agent that answers
NUMBER="+1..."                                     # your Twilio number, E.164
TRUNK_DOMAIN="my-company-agent.pstn.twilio.com"    # you invent this
```

Load it into your shell:

```bash
set -a && source .env && set +a
```

`TRUNK_DOMAIN` doesn't exist yet. You're naming the trunk you create in the next step, and it must end in `.pstn.twilio.com` and be unique across Twilio, so include your company name.

## 2. Point Twilio at AssemblyAI

Creates the trunk, points its origination at `sip:sip.assemblyai.com`, and attaches your number to the trunk.

```bash setup-trunk.sh icon=terminal expandable
#!/usr/bin/env bash
set -euo pipefail

# 1. Create the trunk
TRUNK_SID=$(twilio api:trunking:v1:trunks:create \
  --friendly-name "AssemblyAI voice agent" \
  --domain-name "$TRUNK_DOMAIN" -o json | jq -r '.[0].sid')
echo "Trunk: $TRUNK_SID"

# 2. Route incoming calls to AssemblyAI
twilio api:trunking:v1:trunks:origination-urls:create \
  --trunk-sid "$TRUNK_SID" --friendly-name "AssemblyAI SIP" \
  --sip-url "sip:sip.assemblyai.com" --priority 1 --weight 1 --enabled >/dev/null

# 3. Attach your number to the trunk
NUMBER_SID=$(twilio api:core:incoming-phone-numbers:list \
  --phone-number "$NUMBER" -o json | jq -r '.[0].sid')
twilio api:trunking:v1:trunks:phone-numbers:create \
  --trunk-sid "$TRUNK_SID" --phone-number-sid "$NUMBER_SID" >/dev/null
echo "Number $NUMBER attached to trunk"
```

The trunk now controls the number. Any Voice webhook set on the number itself no longer applies.

Reusing a trunk that already has an origination URL? Delete the old one before running this, or calls may go to the wrong place:

```bash
twilio api:trunking:v1:trunks:origination-urls:list --trunk-sid "$TRUNK_SID"
twilio api:trunking:v1:trunks:origination-urls:remove --trunk-sid "$TRUNK_SID" --sid "OUxxxx"
```

## 3. Attach your agent to the number

Registers the number with AssemblyAI against your trunk, binds `AGENT_ID` to it, and prints the result.

```bash register-number.sh icon=phone-arrow-down-left expandable
#!/usr/bin/env bash
set -euo pipefail

AAI="https://agents.us.assemblyai.com/v1"
AUTH=(-H "Authorization: Bearer $AAI_API_KEY" -H "Content-Type: application/json")

# 4. Register the number
curl -fsS -X POST "$AAI/phone-numbers/import" "${AUTH[@]}" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d "{\"phone_number\":\"$NUMBER\",\"termination_uri\":\"$TRUNK_DOMAIN\"}" >/dev/null

# 5. Bind the agent
curl -fsS -X PUT "$AAI/phone-numbers/$NUMBER/agent" "${AUTH[@]}" \
  -d "{\"agent_id\":\"$AGENT_ID\"}" >/dev/null

# Verify
curl -fsS "$AAI/phone-numbers/$NUMBER" "${AUTH[@]}" | jq
```

The final `jq` output should show your `agent_id` with `"type": "imported"`.

To point the number at a different agent later, re-run only the `PUT`. Twilio needs no changes, and editing the agent itself changes nothing here: the number is bound to the ID, so updates to that agent take effect on the next call.

## 4. Call it

Ring the number. You should hear the greeting, then have a real-time conversation with the agent.

Twilio bills the inbound minutes and AssemblyAI bills the session, so a live number draws on both accounts.

## If it does not work

| Symptom | Cause |
| --- | --- |
| Nothing happens when you call | The number isn't on the trunk, the origination URL isn't exactly `sip:sip.assemblyai.com` and enabled, or no agent is attached. A number with a Voice webhook still set in the Console is not going through the trunk. |
| Call connects, then silence | The origination URL is wrong or disabled. |
| `is attached to a different trunk` | Detach it in the Twilio Console, then re-run. |
| `Twilio POST /v1/Trunks failed (400)` | The SIP domain is taken or malformed. Choose another `*.pstn.twilio.com`. |
| `404 agent_not_found` | Wrong `AGENT_ID`. |
| `409` number already registered | Already imported. Safe to ignore on a re-run. |
| `422 phone_number_has_no_agent` | Re-run the `PUT` in step 3. |
| Twilio command not recognised | Names change between CLI versions. Run `twilio api:trunking:v1 --help`, or use the Console. |

## Want it scripted?

The [Python](https://github.com/AssemblyAI/voice-agent-starter-python) and [JavaScript](https://github.com/AssemblyAI/voice-agent-starter-js) starters run all five steps as one idempotent command, publishing the agent along the way:

```bash Python
python deployment/telephony/connect.py
```

```bash JavaScript
npm run phone
```

## Next steps

- [Configure your agent](/voice-agents/voice-agent-api/session-configuration): Customize the system prompt, greeting, and turn detection.
- [Choose a voice](/voice-agents/voice-agent-api/voices): Pick a voice for your agent.
- [Add tools to your agent](/voice-agents/voice-agent-api/tools/overview): Give your agent the ability to call functions.

---

<a id="audio-format-docs-voice-agents-voice-agent-api-audio-format-mdx"></a>

# Audio format

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/audio-format
Source: docs/voice-agents/voice-agent-api/audio-format.mdx
Navigation: Voice Agent API > Deploy
Description: Audio format documentation.

All audio is **base64-encoded and mono**. Input and output are configured independently and can differ.

Browsers and Twilio handle capture, playback, and echo for you. This page is for clients that move the bytes themselves.

## Encodings

| Encoding       | Sample rate  | Bit depth | Best for                              |
| -------------- | ------------ | --------- | ------------------------------------- |
| `audio/pcm`    | 24,000 Hz    | 16-bit signed integer (little-endian) | Default. Browsers and apps |
| `audio/pcmu`   | 8,000 Hz     | 8-bit μ-law | Telephony (G.711 μ-law)             |
| `audio/pcma`   | 8,000 Hz     | 8-bit A-law | Telephony (G.711 A-law)             |

Both default to `audio/pcm` at 24 kHz. Change it for telephony, where 8 kHz G.711 matches the phone network and avoids resampling.

Set it on the agent when you [create](/voice-agents/voice-agent-api/create-agent) or [update](/voice-agents/voice-agent-api/manage-agents) it:

```bash cURL
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Telephony Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
    "voice": { "voice_id": "alba" },
    "input":  { "format": { "encoding": "audio/pcm" } },
    "output": { "format": { "encoding": "audio/pcmu" } }
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/agents",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "name": "Telephony Assistant",
        "system_prompt": "You are a friendly support agent. Keep replies under two sentences.",
        "voice": {"voice_id": "alba"},
        "input": {"format": {"encoding": "audio/pcm"}},
        "output": {"format": {"encoding": "audio/pcmu"}},
    },
)
resp.raise_for_status()
print(resp.json())
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/agents", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Telephony Assistant",
    system_prompt: "You are a friendly support agent. Keep replies under two sentences.",
    voice: { voice_id: "alba" },
    input: { format: { encoding: "audio/pcm" } },
    output: { format: { encoding: "audio/pcmu" } },
  }),
});
const data = await res.json();
console.log(data);
```

Or inline, in a [`session.update`](/voice-agents/voice-agent-api/events-reference#session-update):

```json
{
  "type": "session.update",
  "session": {
    "input": { "format": { "encoding": "audio/pcm" } },
    "output": { "format": { "encoding": "audio/pcmu" } }
  }
}
```

| Field                   | Type    | Required | Notes                                                        |
| ----------------------- | ------- | -------- | ------------------------------------------------------------ |
| `input.format.encoding` | string  | No       | `audio/pcm`, `audio/pcmu`, or `audio/pcma`. Default `audio/pcm`. |
| `output.format.encoding`| string  | No       | Same values as input. Default `audio/pcm`.                   |
| `format.sample_rate`    | integer | No       | Hz. Determined by the encoding if omitted.                   |

Inline, `output.format` is [immutable after `session.ready`](/voice-agents/voice-agent-api/session-configuration#mutability-after-session-ready), so set it on your first update. Volume is separate, see [Output volume](/voice-agents/voice-agent-api/volume).

## Sending audio

Send [`input.audio`](/voice-agents/voice-agent-api/events-reference#input-audio) events continuously, each a base64 chunk in the configured encoding. Chunk size doesn't matter; ~50 ms works well.

```python
import base64

async def send_audio():
    while True:
        chunk = await mic_queue.get()
        await ws.send(json.dumps({
            "type": "input.audio",
            "audio": base64.b64encode(chunk).decode()
        }))
```

Three rules:

- **Wait for [`session.ready`](/voice-agents/voice-agent-api/events-reference#session-ready)** before the first chunk.
- **Send at real time, not faster.** Frames beyond about one second of audio per second of wall clock are dropped, not buffered, and transcription comes back incomplete. Pace pre-recorded test clips with `asyncio.sleep`.
- **Send raw mic audio.** The server denoises already, and a second layer (RNNoise, Krisp, BVC) adds artifacts that cost more accuracy than the noise did. To tune it, use `input.voice_focus`, see [Isolate the caller's voice](/voice-agents/voice-agent-api/noise-suppression).

Without echo cancellation the agent hears itself and interrupts itself, cutting every reply short with `status: "interrupted"`. Native audio APIs (PortAudio, `sounddevice`) have none, so use headphones. Browsers and carriers handle it for you.

## Playing output audio

Write each [`reply.audio`](/voice-agents/voice-agent-api/events-reference#reply-audio) chunk straight into an output buffer and let the OS drain it. The buffer absorbs network jitter, so late messages don't create gaps:

```python
SAMPLE_RATE = 24000  # 8000 for audio/pcmu or audio/pcma

with sd.OutputStream(samplerate=SAMPLE_RATE, channels=1, dtype="int16") as speaker:
    if event["type"] == "reply.audio":
        pcm = np.frombuffer(base64.b64decode(event["data"]), dtype=np.int16)
        speaker.write(pcm)
```

Don't schedule playback with `sleep`. Sleep durations aren't exact, so the playback clock drifts from the hardware clock and you get pops and gaps.

```python
# ❌ Don't do this
while True:
    play(get_next_chunk())
    await asyncio.sleep(0.020)  # drift accumulates → audio artifacts
```

## Handling interruptions

On barge-in the server stops generating and emits [`reply.done`](/voice-agents/voice-agent-api/events-reference#reply-done) with `status: "interrupted"`, plus [`transcript.agent`](/voice-agents/voice-agent-api/events-reference#transcript-agent) with `interrupted: true` and the text trimmed to what was actually spoken.

Flush your queued audio so the user doesn't hear stale speech, then restart the stream:

```python
if event["type"] == "reply.done" and event.get("status") == "interrupted":
    speaker.abort()  # discard buffered audio
    speaker.start()  # ready for the next reply
```

| Platform           | Flush approach                                                                    |
| ------------------ | --------------------------------------------------------------------------------- |
| **Python** (sounddevice) | `speaker.abort()` then `speaker.start()`                                    |
| **Web** (AudioContext)   | Disconnect the source node, create a new `AudioBufferSourceNode`, reconnect  |
| **iOS** (AVAudioEngine)  | `playerNode.stop()` then `playerNode.play()`                                 |
| **Android** (AudioTrack) | `audioTrack.pause()`, `audioTrack.flush()`, then `audioTrack.play()`         |

Barge-in is semantic: "uh-huh" won't interrupt, "wait, stop" will. See [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions).

---

<a id="recordings-transcripts-docs-voice-agents-voice-agent-api-session-history-mdx"></a>

# Recordings & transcripts

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/session-history
Source: docs/voice-agents/voice-agent-api/session-history.mdx
Navigation: Voice Agent API > After a session
Description: List voice agent sessions and retrieve each one's recording, conversation timeline, and metadata, with examples for parsing transcripts and playing back audio.

Every connection to a voice agent is recorded as a **session**. The Sessions API lists past sessions, retrieves any one with its recording and full conversation timeline, and deletes them. It is what you build a call-history view, transcript review, QA workflow, or usage dashboard on.

Send your API key in the `Authorization` header (a `Bearer ` prefix is accepted and stripped). The base URL is `https://agents.assemblyai.com`. For the field-level schema of each endpoint, see the [Sessions API reference](/api-reference/voice-agent-api/list-sessions).

## Find completed sessions

Poll `GET /v1/sessions` and process the ones whose `status` is `completed`, fetching each and downloading its artifacts. Keep the last `session_id` you handled (or the newest `created_at`) so each poll only picks up new sessions.

You also know a session's `session_id` the moment it connects, since it arrives over the WebSocket in `session.ready`. Store it live and pull the recording and transcript once the session ends.

## List sessions

`GET /v1/sessions` returns a page of sessions, newest first, without artifacts. Narrow the results with `status` and `agent_id`, size the page with `limit` (1 to 200, default 50), and page forward with `cursor`.

```bash cURL
curl "https://agents.assemblyai.com/v1/sessions?limit=50&agent_id=$AGENT_ID" \
  -H "Authorization: $ASSEMBLYAI_API_KEY"
```

```python Python
# pip install requests
import os
import requests

resp = requests.get(
    "https://agents.assemblyai.com/v1/sessions",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    params={"limit": 50, "agent_id": os.environ["AGENT_ID"]},
)
resp.raise_for_status()
print(resp.json()["sessions"])
```

```javascript Node.js
// Node 18+ has fetch built in
const url = new URL("https://agents.assemblyai.com/v1/sessions");
url.searchParams.set("limit", "50");
url.searchParams.set("agent_id", process.env.AGENT_ID);

const res = await fetch(url, {
  headers: { Authorization: process.env.ASSEMBLYAI_API_KEY },
});
const data = await res.json();
console.log(data.sessions);
```

```json
{
  "sessions": [
    {
      "id": "sess_9a648a2ab75747a9a597ba046ced3e13",
      "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9",
      "status": "completed",
      "public_close_reason": "client_end",
      "duration_seconds": 42.6,
      "created_at": "2026-07-14T18:04:27.607110Z",
      "ended_at": "2026-07-14T18:05:10.204981Z"
    }
  ],
  "has_more": true,
  "response_metadata": { "next_cursor": "c2Vzc185YTY0OGEyYWI3NTc0N2E5" }
}
```

To pull the full history, keep passing `next_cursor` back as `cursor` until `has_more` is `false`:

```python Python
import os
import requests

BASE = "https://agents.assemblyai.com"
HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

def all_sessions(**filters):
    cursor = None
    while True:
        resp = requests.get(
            f"{BASE}/v1/sessions",
            headers=HEADERS,
            params={"limit": 200, "cursor": cursor, **filters},
        )
        resp.raise_for_status()
        page = resp.json()
        yield from page["sessions"]
        cursor = page["response_metadata"]["next_cursor"]
        if not page["has_more"] or not cursor:
            break

for session in all_sessions(agent_id=os.environ["AGENT_ID"]):
    print(session["id"], session["status"], session["duration_seconds"])
```

```javascript Node.js
const BASE = "https://agents.assemblyai.com";
const HEADERS = { Authorization: process.env.ASSEMBLYAI_API_KEY };

async function* allSessions(filters = {}) {
  let cursor = "";
  do {
    const url = new URL(`${BASE}/v1/sessions`);
    url.searchParams.set("limit", "200");
    if (cursor) url.searchParams.set("cursor", cursor);
    for (const [k, v] of Object.entries(filters)) url.searchParams.set(k, v);

    const res = await fetch(url, { headers: HEADERS });
    const page = await res.json();
    yield* page.sessions;
    cursor = page.response_metadata.next_cursor;
    if (!page.has_more || !cursor) break;
  } while (true);
}

for await (const s of allSessions({ agent_id: process.env.AGENT_ID })) {
  console.log(s.id, s.status, s.duration_seconds);
}
```

## Retrieve a session

`GET /v1/sessions/{id}` returns the full session. On top of the list metadata it adds two things: `config`, the session's configuration as it ran (system prompt, input/output, tools, LLM), and `artifacts`, each a short-lived **pre-signed download URL**.

```bash cURL
curl "https://agents.assemblyai.com/v1/sessions/$SESSION_ID" \
  -H "Authorization: $ASSEMBLYAI_API_KEY"
```

```python Python
import os
import requests

resp = requests.get(
    f"https://agents.assemblyai.com/v1/sessions/{os.environ['SESSION_ID']}",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
)
resp.raise_for_status()
session = resp.json()
print(session["status"], session["artifacts"])
```

```javascript Node.js
const res = await fetch(
  `https://agents.assemblyai.com/v1/sessions/${process.env.SESSION_ID}`,
  { headers: { Authorization: process.env.ASSEMBLYAI_API_KEY } },
);
const session = await res.json();
console.log(session.status, session.artifacts);
```

```json expandable
{
  "id": "sess_9a648a2ab75747a9a597ba046ced3e13",
  "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9",
  "status": "completed",
  "public_close_reason": "client_end",
  "duration_seconds": 42.6,
  "config": {
    "system_prompt": "You are a friendly support agent.",
    "greeting": "Hi, how can I help?",
    "input": { "type": "audio", "format": { "encoding": "audio/pcm", "sample_rate": 24000 }, "keyterms": [], "turn_detection": null },
    "output": { "type": "audio", "voice": "alba", "format": { "encoding": "audio/pcm", "sample_rate": 24000 } },
    "tools": [],
    "llm": []
  },
  "created_at": "2026-07-14T18:04:27.607110Z",
  "ended_at": "2026-07-14T18:05:10.204981Z",
  "artifacts": [
    { "type": "audio", "url": "https://s3.amazonaws.com/.../audio.ogg?X-Amz-Signature=...", "content_type": "audio/ogg" },
    { "type": "timeline", "url": "https://s3.amazonaws.com/.../timeline.json?X-Amz-Signature=...", "content_type": "application/json" },
    { "type": "metadata", "url": "https://s3.amazonaws.com/.../metadata.json?X-Amz-Signature=...", "content_type": "application/json" }
  ]
}
```

The three artifacts appear once the session completes; the array is empty while it is still active. They are the **audio** recording (OGG/Opus), the **timeline** (the conversation as JSON), and **metadata** (recording details). Download each straight from its `url` with no `Authorization` header, since the signature already authorizes the request.

Artifact URLs expire after a short TTL. Store the `session_id`, not the URL, and re-fetch the session for fresh links right before you download.

## Parse the conversation timeline

The `timeline` artifact is the whole conversation, turn by turn:

```json expandable
{
  "session_id": "sess_9a648a2ab75747a9a597ba046ced3e13",
  "started_at_unix_ms": 1768413867285,
  "turns": [
    {
      "turn_id": "resp_c5f07b9645b649fdb31bfbdc790d224b",
      "item_id": "msg_7ff94d121e0a473f9bfb53216294538a",
      "status": "completed",
      "trigger": "greeting",
      "user_transcript": null,
      "agent_text": "Hi, how can I help?",
      "agent_reply_started_at_ms": 1768413868332,
      "agent_reply_ended_at_ms": 1768413871652,
      "time_to_first_audio_ms": 1013
    },
    {
      "turn_id": "resp_86122ef6e8264c1e94e1bd88ed1f9fbc",
      "item_id": "msg_66d56698eb0e4b3a96ec9bd494db44b8",
      "status": "completed",
      "trigger": "user_speech",
      "user_transcript": "What's the weather in Paris?",
      "user_confidence": 0.98,
      "agent_text": "It's 22 degrees and sunny in Paris.",
      "agent_reply_started_at_ms": 1768413878530,
      "time_to_first_audio_ms": 1457,
      "tool_calls": [
        {
          "call_id": "call_abc",
          "name": "get_weather",
          "arguments": { "latitude": 48.85, "longitude": 2.35 },
          "result": "{\"temp_c\": 22, \"condition\": \"sunny\"}",
          "dispatched_at_ms": 1768413876500,
          "result_received_at_ms": 1768413876920,
          "duration_ms": 420,
          "is_error": false
        }
      ]
    }
  ]
}
```

Each turn pairs what the user said (`user_transcript`, `null` on agent-initiated turns like the greeting) with what the agent replied (`agent_text`), and carries a `tool_calls` array **only on turns where a tool actually fired**. `turn_id` is the model response ID and `status` is the turn outcome (`completed` or `interrupted`). Timestamps ending `_at_ms` are absolute Unix epoch milliseconds; durations ending `_ms` (like `time_to_first_audio_ms` and `duration_ms`) are relative spans.

A turn can also have **no** `user_transcript` and **no** `agent_text`: an interruption before the agent spoke, or a brief noise that opened a turn without a resolved transcript. Skip those when you build a transcript rather than assuming every turn has text (the code below does).

Empty arrays are dropped from the JSON: a session where no one spoke has **no** `turns` key at all, and a turn with no tools has **no** `tool_calls`. Default them when you read: `timeline.get("turns", [])` in Python, `timeline.turns ?? []` in JS.

To render or store the conversation, download the timeline and flatten its turns into an ordered list of messages:

```python Python
import os
import requests

BASE = "https://agents.assemblyai.com"
HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

def artifact_url(session, kind):
    return next((a["url"] for a in session["artifacts"] if a["type"] == kind), None)

def load_transcript(session_id):
    session = requests.get(
        f"{BASE}/v1/sessions/{session_id}", headers=HEADERS
    ).json()

    url = artifact_url(session, "timeline")
    if url is None:
        return []  # session still active, or no recording

    timeline = requests.get(url).json()  # pre-signed URL: no auth header

    messages = []
    for turn in timeline.get("turns", []):  # `turns` is omitted when empty
        if turn.get("user_transcript"):
            messages.append({"role": "user", "text": turn["user_transcript"]})
        for call in turn.get("tool_calls", []):
            messages.append({
                "role": "tool",
                "name": call["name"],
                "arguments": call.get("arguments"),
                "result": call.get("result"),
                "error": call["is_error"] or call.get("timed_out", False),
            })
        if turn.get("agent_text"):
            messages.append({"role": "agent", "text": turn["agent_text"]})
    return messages

for m in load_transcript(os.environ["SESSION_ID"]):
    if m["role"] == "tool":
        flag = " (error)" if m["error"] else ""
        print(f"  ↳ {m['name']}({m['arguments']}) → {m['result']}{flag}")
    else:
        print(f"{m['role']}: {m['text']}")
```

```javascript Node.js
const BASE = "https://agents.assemblyai.com";
const HEADERS = { Authorization: process.env.ASSEMBLYAI_API_KEY };

const artifactUrl = (session, kind) =>
  session.artifacts.find((a) => a.type === kind)?.url ?? null;

async function loadTranscript(sessionId) {
  const session = await (
    await fetch(`${BASE}/v1/sessions/${sessionId}`, { headers: HEADERS })
  ).json();

  const url = artifactUrl(session, "timeline");
  if (!url) return []; // session still active, or no recording

  const timeline = await (await fetch(url)).json(); // pre-signed: no auth header

  const messages = [];
  for (const turn of timeline.turns ?? []) {  // `turns` is omitted when empty
    if (turn.user_transcript)
      messages.push({ role: "user", text: turn.user_transcript });
    for (const call of turn.tool_calls ?? []) {
      messages.push({
        role: "tool",
        name: call.name,
        arguments: call.arguments,
        result: call.result,
        error: call.is_error || call.timed_out,
      });
    }
    if (turn.agent_text)
      messages.push({ role: "agent", text: turn.agent_text });
  }
  return messages;
}

for (const m of await loadTranscript(process.env.SESSION_ID)) {
  if (m.role === "tool") {
    console.log(
      `  ↳ ${m.name}(${JSON.stringify(m.arguments)}) → ${m.result}` +
        (m.error ? " (error)" : ""),
    );
  } else {
    console.log(`${m.role}: ${m.text}`);
  }
}
```

```text
agent: Hi, how can I help?
user: What's the weather in Paris?
  ↳ get_weather({"latitude": 48.85, "longitude": 2.35}) → {"temp_c": 22, "condition": "sunny"}
agent: It's 22 degrees and sunny in Paris.
```

## Work with the recording

The `audio` artifact is an OGG/Opus file; the `metadata` artifact describes it:

```json
{
  "session_id": "sess_9a648a2ab75747a9a597ba046ced3e13",
  "started_at": "2026-07-14T18:04:27.607110Z",
  "ended_at": "2026-07-14T18:05:10.204981Z",
  "format": "ogg/opus",
  "channels": 2,
  "channel_layout": "stereo (left=user, right=agent)",
  "sample_rate": 24000,
  "file": "sess_9a648a2ab75747a9a597ba046ced3e13/recording/audio.ogg"
}
```

The recording is stereo with the **user on the left channel and the agent on the right**, so you can split channels for diarized playback or per-speaker analysis. To save it locally:

```python Python
import os
import requests

BASE = "https://agents.assemblyai.com"
HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

session = requests.get(
    f"{BASE}/v1/sessions/{os.environ['SESSION_ID']}", headers=HEADERS
).json()

audio = next(a for a in session["artifacts"] if a["type"] == "audio")
meta = next(a for a in session["artifacts"] if a["type"] == "metadata")

with requests.get(audio["url"], stream=True) as r:  # pre-signed: no auth header
    r.raise_for_status()
    with open("recording.ogg", "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

print(requests.get(meta["url"]).json())
```

```javascript Node.js
import { writeFile } from "node:fs/promises";

const BASE = "https://agents.assemblyai.com";
const HEADERS = { Authorization: process.env.ASSEMBLYAI_API_KEY };

const session = await (
  await fetch(`${BASE}/v1/sessions/${process.env.SESSION_ID}`, {
    headers: HEADERS,
  })
).json();

const audio = session.artifacts.find((a) => a.type === "audio");
const meta = session.artifacts.find((a) => a.type === "metadata");

const buf = Buffer.from(await (await fetch(audio.url)).arrayBuffer());
await writeFile("recording.ogg", buf);

console.log(await (await fetch(meta.url)).json());
```

To play it back in your app, keep the API key server-side: expose a small endpoint that returns a fresh audio URL, then point an `<audio>` element at it.

```javascript Server (Express)
import express from "express";

const app = express();

app.get("/api/sessions/:id/audio", async (req, res) => {
  const r = await fetch(
    `https://agents.assemblyai.com/v1/sessions/${req.params.id}`,
    { headers: { Authorization: process.env.ASSEMBLYAI_API_KEY } },
  );
  if (!r.ok) return res.status(r.status).end();

  const session = await r.json();
  const audio = session.artifacts.find((a) => a.type === "audio");
  // The URL is short-lived, so the client requests it right before playback.
  res.json({ url: audio?.url ?? null });
});

app.listen(3000);
```

```tsx React
import { useEffect, useState } from "react";

export function SessionRecording({ sessionId }: { sessionId: string }) {
  const [url, setUrl] = useState<string | null>(null);

  useEffect(() => {
    fetch(`/api/sessions/${sessionId}/audio`)
      .then((r) => r.json())
      .then((d) => setUrl(d.url));
  }, [sessionId]);

  if (!url) return <p>Loading recording…</p>;

  return (
    <audio controls preload="metadata" src={url}>
      Your browser does not support the audio element.
    </audio>
  );
}
```

OGG/Opus plays natively in Chrome, Firefox, and Edge; Safari support is limited. To support Safari, transcode server-side first, for example `ffmpeg -i recording.ogg recording.m4a`.

## Delete a session

`DELETE /v1/sessions/{id}` soft-deletes a session and returns `204`. It stops appearing in `GET /v1/sessions` and its artifacts become inaccessible.

```bash cURL
curl -X DELETE "https://agents.assemblyai.com/v1/sessions/$SESSION_ID" \
  -H "Authorization: $ASSEMBLYAI_API_KEY"
```

---

<a id="list-sessions-docs-voice-agents-voice-agent-api-api-spec-list-sessions-mdx"></a>

# List sessions

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/list-sessions
Source: docs/voice-agents/voice-agent-api/api-spec/list-sessions.mdx
Navigation: Voice Agent API > After a session > Endpoints
Description: List your voice agent sessions, newest first.

Metadata:
- openapi: /api-reference/specs/agents.yaml GET /v1/sessions

---

<a id="retrieve-a-session-docs-voice-agents-voice-agent-api-api-spec-get-session-mdx"></a>

# Retrieve a session

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/get-session
Source: docs/voice-agents/voice-agent-api/api-spec/get-session.mdx
Navigation: Voice Agent API > After a session > Endpoints
Description: Retrieve a voice agent session by ID, including its recording, timeline, and metadata artifacts.

Metadata:
- openapi: /api-reference/specs/agents.yaml GET /v1/sessions/{session_id}

---

<a id="delete-a-session-docs-voice-agents-voice-agent-api-api-spec-delete-session-mdx"></a>

# Delete a session

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/delete-session
Source: docs/voice-agents/voice-agent-api/api-spec/delete-session.mdx
Navigation: Voice Agent API > After a session > Endpoints
Description: Soft-delete a voice agent session.

Metadata:
- openapi: /api-reference/specs/agents.yaml DELETE /v1/sessions/{session_id}

---

<a id="inline-session-configuration-docs-voice-agents-voice-agent-api-session-configuration-mdx"></a>

# Inline session configuration

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/session-configuration
Source: docs/voice-agents/voice-agent-api/session-configuration.mdx
Navigation: Voice Agent API > Reference
Description: Configure a voice agent at connect-time over the WebSocket with session.update, without storing an agent.

**Most agents should [create a stored agent](/voice-agents/voice-agent-api/create-agent) instead.** Storing your configuration server-side keeps secrets off the client, runs HTTP tools for you, and lets you deploy the same agent across the API, a browser, and Twilio by `agent_id`. This page covers the **inline** alternative: configuring an agent at connect-time without storing it, useful for fully dynamic or one-off agents. The field reference below applies to both: the same fields exist on a stored agent's `input`/`output` objects.

`session.update` is the configuration surface for an inline (non-stored) agent. Everything that shapes how the agent sounds and behaves lives here: its personality, its greeting, the tools it can call, how it interprets user speech, and what voice it speaks in.

Send a [`session.update`](/voice-agents/voice-agent-api/events-reference#session-update) as your first WebSocket message, and any time after, to update most fields. Some fields can only be set in the first update; see [Mutability after `session.ready`](#mutability-after-session-ready) below.

To use a stored agent instead, send `{ "agent_id": "<id>" }` as the only field in your first `session.update`; it's mutually exclusive with the inline fields below. See [Deploy your agent](/voice-agents/voice-agent-api/deploy).

Here's a full configuration showing every available field:

```json expandable
{
  "type": "session.update",
  "session": {
    "system_prompt": "You are a friendly support agent. Keep responses under 2 sentences.",
    "greeting": "Hi! How can I help you today?",
    "tools": [],
    "input": {
      "format": { "encoding": "audio/pcm" },
      "keyterms": ["AssemblyAI", "Universal"],
      "transcription_mode": "balanced",
      "transcription_prompt": "Expect product names and order IDs.",
      "language_codes": ["en"],
      "voice_focus": "near-field",
      "voice_focus_threshold": 0.5,
      "turn_detection": {
        "vad_threshold": 0.5,
        "min_silence": 1000,
        "max_silence": 3000,
        "interrupt_response": true,
        "interruption_delay": 100
      }
    },
    "output": {
      "voice": "alba",
      "format": { "encoding": "audio/pcm" },
      "volume": 100
    }
  }
}
```

Every field is optional. Include only what you want to set or change. Jump to any section below for details.

## Mutability after `session.ready`

The first `session.update` you send before `session.ready` initializes the session. After `session.ready`, only a subset of fields can be changed. Changing one of the immutable fields raises a [`session.error`](/voice-agents/voice-agent-api/events-reference#session-error) with code `immutable_field` and the rejected change is ignored.

| Field                          | Mutable after `session.ready`? |
| ------------------------------ | ------------------------------ |
| `session.system_prompt`        | Yes. Send a new prompt at any time to change the agent's behavior on the next turn. |
| `session.input.turn_detection` | Yes. Adjust VAD thresholds, silence windows, and barge-in on the fly.               |
| `session.input.keyterms`       | Yes. Replace the keyterms list at any time. The new list takes effect on the next user utterance. |
| `session.input.transcription_mode` | Yes. |
| `session.input.transcription_prompt` | Yes. |
| `session.input.language_codes` | Accepted mid-session, but applied on the next speech-to-text reconnect (not the current turn). |
| `session.input.voice_focus` / `voice_focus_threshold` | Set at connect. A mid-session change is accepted but only takes effect on the next speech-to-text reconnect. |
| `session.output.volume`        | Yes. Adjust playback volume on the fly. |
| `session.greeting`             | No. Raises `immutable_field`. The greeting is spoken once at session start.         |
| `session.output.voice`         | No. Raises `immutable_field`. The voice is bound to the TTS connection at session start. |
| `session.output.format`        | No. Raises `immutable_field`. The output audio encoding is fixed for the session. |

Other fields (`session.tools`, `session.input.format`) are also accepted in subsequent `session.update` messages and don't raise `immutable_field`.

## Fields

The `session` object carries the same fields as a stored agent. Each has a dedicated guide with details and examples; the cURL there maps one-to-one onto the `session.update` shape shown above.

- **`system_prompt`**: the agent's personality and behavior. Updatable mid-session. See the [Prompting guide](/voice-agents/voice-agent-api/prompting-guide).
- **`greeting`**: the exact words spoken on connect, sent straight to TTS. Immutable after `session.ready`. See [Greeting](/voice-agents/voice-agent-api/greeting).
- **`output.voice`**: the TTS voice, e.g. `"alba"`. Immutable after `session.ready`. See [Voices](/voice-agents/voice-agent-api/voices).
- **`input.format` / `output.format`**: audio encoding (default `audio/pcm` at 24 kHz). `output.format` is immutable after `session.ready`. See [Audio format](/voice-agents/voice-agent-api/audio-format#encodings).
- **`output.volume`**: playback volume, `0` to `100`. Updatable mid-session. See [Volume](/voice-agents/voice-agent-api/volume).
- **`input.keyterms`**: up to 100 transcription-bias terms. Updatable mid-session. See [Key terms](/voice-agents/voice-agent-api/transcription-prompt#key-terms).
- **`input.turn_detection`**: VAD threshold, silence windows (`min_silence`/`max_silence`, ms), barge-in (`interrupt_response`), and `interruption_delay` (ms, `0` to `1000`). Updatable mid-session. See [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions).
- **`input.transcription_mode`**: speech-to-text speed vs. accuracy: `balanced` (default), `min_latency`, or `max_accuracy`. Updatable mid-session. See [Turn-taking & interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions#transcription-mode).
- **`input.transcription_prompt`**: a prompt that biases transcription toward expected vocabulary (max 1750 characters). Updatable mid-session.
- **`input.language_codes`**: declare the spoken language(s); omit for automatic detection. Applied on the next speech-to-text connect.
- **`input.voice_focus` / `input.voice_focus_threshold`**: noise-suppression model (`near-field`, the default, or `far-field`) and strength (`0.0` to `1.0`, default `0.7`, requires `voice_focus`). Set at connect.
- **`tools`**: actions the agent can call, each optionally with `response_instructions` (`success`/`error` guidance the agent follows after the call). See [Add tools](/voice-agents/voice-agent-api/tools/overview).

---

<a id="events-reference-docs-voice-agents-voice-agent-api-events-reference-mdx"></a>

# Events reference

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/events-reference
Source: docs/voice-agents/voice-agent-api/events-reference.mdx
Navigation: Voice Agent API > Reference
Description: Every client-to-server and server-to-client event for the Voice Agent API.

Every message exchanged over the Voice Agent API WebSocket, grouped by direction. You'll send [`session.update`](#session-update) to configure, [`input.audio`](#input-audio) to stream mic audio, [`tool.result`](#tool-result) to respond to tool calls, and [`session.end`](#session-end) to cleanly end the call. The server streams everything else back, including [`session.ended`](#session-ended) on every clean teardown.

## Event flow

A typical voice agent session moves through the events in this order:

```
Client                              Server
  │                                   │
  │── WebSocket connect ─────────────►│
  │── session.update ────────────────►│  (system prompt + tools + greeting)
  │                                   │
  │◄─── session.ready ────────────────│  (save session_id)
  │                                   │
  │── input.audio (stream) ──────────►│  (only after session.ready)
  │── input.audio (stream) ──────────►│
  │                                   │
  │◄─── input.speech.started ─────────│
  │◄─── transcript.user.delta ────────│
  │◄─── input.speech.stopped ─────────│
  │◄─── transcript.user ──────────────│
  │                                   │
  │◄─── reply.started ────────────────│
  │◄─── reply.audio ──────────────────│
  │◄─── transcript.agent.delta ───────│
  │◄─── transcript.agent ─────────────│
  │◄─── reply.done ───────────────────│
  │                                   │
  │  [tool call flow]                 │
  │◄─── tool.call ────────────────────│  (arguments is a dict)
  │◄─── reply.done ───────────────────│  ← send tool.result here
  │── tool.result ───────────────────►│
  │◄─── reply.started ────────────────│
  │◄─── reply.audio ──────────────────│
  │◄─── reply.done ───────────────────│
  │                                   │
  │  [clean teardown]                 │
  │── session.end ───────────────────►│
  │◄─── session.ended ────────────────│
  │◄── WebSocket close ───────────────│
```

## Client → Server

### `input.audio`

Stream PCM16 audio to the agent.

```json
{
  "type": "input.audio",
  "audio": "<base64-encoded PCM16>"
}
```

| Field   | Type   | Description                            |
| ------- | ------ | -------------------------------------- |
| `audio` | string | Base64-encoded PCM16 mono 24kHz audio  |

See [Audio format](/voice-agents/voice-agent-api/audio-format) for the full format specification.

---

### `session.update`

Configure the session. Send immediately on WebSocket connect (before `session.ready`). Can also be sent mid-conversation to update most fields. See [Mutability after `session.ready`](/voice-agents/voice-agent-api/session-configuration#mutability-after-session-ready) for which fields can change once the session is established.

```json expandable
{
  "type": "session.update",
  "session": {
    "system_prompt": "You are a concise assistant.",
    "greeting": "Hi! How can I help?",
    "input": {
      "format": { "encoding": "audio/pcm" },
      "turn_detection": { "vad_threshold": 0.5 }
    },
    "output": {
      "voice": "alba",
      "format": { "encoding": "audio/pcm" },
      "volume": 100
    },
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
          "type": "object",
          "properties": { "city": { "type": "string" } },
          "required": ["city"]
        }
      }
    ]
  }
}
```

All fields are optional. Include only what you want to set or change. After `session.ready`, only a subset of fields can be changed; changing `greeting`, `session.output.voice`, or `session.output.format` raises `immutable_field`. `session.output.volume` is mutable mid-session. For the full field reference, defaults, and per-field mutability, see [Inline session configuration](/voice-agents/voice-agent-api/session-configuration).

| Field                                  | Type    | Description                                                      |
| -------------------------------------- | ------- | ---------------------------------------------------------------- |
| `session.agent_id`                     | string  | Bind a [stored agent](/voice-agents/voice-agent-api/deploy) by ID. First `session.update` only, and mutually exclusive with the inline fields below. |
| `session.system_prompt`                | string  | Sets the agent's personality and context. Mutable mid-session.   |
| `session.greeting`                     | string  | Spoken aloud at the start of the conversation. Immutable after the first update. |
| `session.input.format`                 | object  | Input audio format (`encoding`). See [Audio format](/voice-agents/voice-agent-api/audio-format) |
| `session.input.keyterms`               | array   | Up to 100 strings to boost in transcription. Mutable. See [Key terms](/voice-agents/voice-agent-api/transcription-prompt#key-terms) |
| `session.input.turn_detection`         | object  | VAD threshold, silence windows, barge-in, and interruption delay. Mutable. See [Turn detection](/voice-agents/voice-agent-api/turn-detection-and-interruptions) |
| `session.input.transcription_mode`     | string  | Speech-to-text speed vs. accuracy: `min_latency`, `balanced` (default), or `max_accuracy`. Mutable. |
| `session.input.transcription_prompt`   | string  | Prompt that biases transcription (max 1750 chars). Mutable. |
| `session.input.language_codes`         | array   | Declare the spoken language(s); omit for automatic detection. Applied on the next STT connect. |
| `session.input.voice_focus`            | string  | Noise-suppression model: `near-field` (default) or `far-field`. Set at connect. |
| `session.input.voice_focus_threshold`  | number  | Noise-suppression strength, `0.0` to `1.0` (default `0.7`); requires `voice_focus`. Set at connect. |
| `session.output.voice`                 | string  | The voice used for the agent's speech. Immutable after the first update. See [Voices](/voice-agents/voice-agent-api/voices) |
| `session.output.format`                | object  | Output audio format (`encoding`). Immutable after the first update. See [Audio format](/voice-agents/voice-agent-api/audio-format) |
| `session.output.volume`                | number  | Playback volume, `0` (silent) to `100` (loudest). Mutable mid-session. See [Output volume](/voice-agents/voice-agent-api/volume) |
| `session.tools`                        | array   | Tool definitions. See [Tool calling](/voice-agents/voice-agent-api/tools/overview) |

---

### `session.resume`

Reconnect to an existing session using the `session_id` from a previous `session.ready`. Preserves conversation context across dropped connections.

```json
{
  "type": "session.resume",
  "session_id": "sess_abc123"
}
```

Sessions are preserved for 30 seconds after every disconnection before expiring. If the session has expired, the server returns a `session.error` with code `session_not_found` or `session_forbidden`. Start a fresh connection without `session.resume`.

**Example.** Capture `session_id` from `session.ready` on the first connection, then send `session.resume` as the first message when reconnecting:

```python expandable
import json
import websockets

session_id: str | None = None

async def connect():
    global session_id
    async with websockets.connect(URL, additional_headers={"Authorization": f"Bearer {API_KEY}"}) as ws:
        # If we already have a session_id from a previous connection, resume it.
        if session_id:
            await ws.send(json.dumps({"type": "session.resume", "session_id": session_id}))
        else:
            await ws.send(json.dumps({"type": "session.update", "session": {...}}))

        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "session.ready":
                session_id = event["session_id"]  # save for next reconnect
            elif event["type"] == "session.error" and event["code"] in ("session_not_found", "session_forbidden"):
                session_id = None  # session expired - start fresh next time
            # ... handle other events

# On disconnect, call connect() again within 30 seconds to resume.
```

---

### `session.end`

End the session cleanly. Send this when the call is over and you don't intend to reconnect. The server emits a final [`session.ended`](#session-ended) and closes the WebSocket; the `session_id` is dead immediately and cannot be resumed.

```json
{ "type": "session.end" }
```

No other fields.

**What happens next:**

1. The server emits [`session.ended`](#session-ended).
2. The server closes the WebSocket.
3. The `session_id` is dead. Sending [`session.resume`](#session-resume) with it returns `session_not_found`.

**When to send it vs. just closing the socket.** If you just close the WebSocket, the server holds the session for 30 seconds so you can [`session.resume`](#session-resume) from a new connection, and that 30-second window is billable. `session.end` short-circuits the grace window and stops billing immediately.

| What you do                          | Server behavior                                   | Resumable?         | Billing                          |
| ------------------------------------ | ------------------------------------------------- | ------------------ | -------------------------------- |
| Send `session.end`, then close       | Closes immediately, emits `session.ended`        | No                 | Stops immediately                |
| Just close the WebSocket             | Holds the session for 30s for `session.resume`    | Yes, within 30s    | Continues during the 30s window  |

---

### `tool.result`

Send a tool result back to the agent. Send this when `reply.done` is the latest event you've received (and nothing has happened since). The simplest pattern is to accumulate on `tool.call` and drain inside the `reply.done` handler. See [Client-side tools](/voice-agents/voice-agent-api/tools/client-side-tools#returning-tool-results).

```json
{
  "type": "tool.result",
  "call_id": "call_abc123",
  "result": "{\"temp_c\": 22, \"description\": \"Sunny\"}",
  "is_error": false
}
```

| Field      | Type    | Description                                                              |
| ---------- | ------- | ------------------------------------------------------------------------ |
| `call_id`  | string  | The `call_id` from the `tool.call` event                                 |
| `result`   | string  | JSON string containing the tool result                                   |
| `is_error` | boolean | Optional, default `false`. Set `true` to signal the tool call failed so the agent can respond accordingly. |

---

### `reply.create`

Ask the agent to generate a reply right now, optionally with custom `instructions`. Useful for delivering status updates during long-running [`hold`-mode tool calls](/voice-agents/voice-agent-api/tools/overview#execution-modes), or any time you want the agent to speak without a user utterance triggering it.

```json
{
  "type": "reply.create",
  "instructions": "Let the customer know we're still processing the transfer."
}
```

| Field          | Type   | Description                                                                                       |
| -------------- | ------ | ------------------------------------------------------------------------------------------------- |
| `instructions` | string | Optional. One-shot instruction the agent uses to compose this reply. Does not modify `system_prompt`. |

The agent generates a normal reply (`reply.started` → `reply.audio` → `transcript.agent` → `reply.done`) using the provided instructions on top of the existing system prompt and conversation history.

---

### `conversation.message`

Inject a message into the conversation context without the user speaking it. Useful for seeding context or replaying prior history. This does not by itself make the agent reply; send [`reply.create`](#replycreate) if you want an immediate response.

```json
{
  "type": "conversation.message",
  "role": "user",
  "content": "I'd like to check my order status."
}
```

| Field     | Type   | Description                                  |
| --------- | ------ | -------------------------------------------- |
| `role`    | string | `"user"` or `"system"`.                      |
| `content` | string | The message text to add to the conversation. |

---

## Server → Client

### `session.ready`

Session is established and ready to receive audio. Save `session_id` for reconnection. Start sending `input.audio` only after this event. The event echoes the full resolved `config` (defaults filled in, and for a stored agent the expanded configuration).

```json
{
  "type": "session.ready",
  "session_id": "sess_abc123",
  "expires_at": 1717180000,
  "resume_token": "...",
  "config": { "system_prompt": "...", "output": { "voice": "alba" }, "...": "..." }
}
```

| Field          | Type    | Description                                                  |
| -------------- | ------- | ------------------------------------------------------------ |
| `session_id`   | string  | Always present. Save this value to reconnect with `session.resume`. |
| `config`       | object  | The full resolved session configuration the server applied. |
| `expires_at`   | integer | Unix epoch seconds when the session reaches its maximum duration. |
| `resume_token` | string  | Token used to resume the session after a drop. May be empty. |

---

### `session.updated`

Sent after `session.update` is applied successfully. Echoes the full resolved configuration after the update.

```json
{
  "type": "session.updated",
  "config": { "system_prompt": "...", "output": { "voice": "alba" }, "...": "..." }
}
```

| Field    | Type   | Description                                          |
| -------- | ------ | ---------------------------------------------------- |
| `config` | object | The full resolved session configuration after the update. |

---

### `session.ended`

Final event emitted on every clean teardown, right before the server closes the WebSocket. Always handle it.

```json
{
  "type": "session.ended",
  "session_duration_seconds": 42.7,
  "audio_duration_seconds": 38.2,
  "timestamp": 1717180000.123
}
```

| Field                       | Type            | Description                                                                |
| --------------------------- | --------------- | -------------------------------------------------------------------------- |
| `session_duration_seconds`  | number          | Total wall-clock duration of the session.                                  |
| `audio_duration_seconds`    | number \| null  | Total audio you streamed in. `null` if you streamed none.                  |
| `timestamp`                 | number          | Unix epoch seconds when the server emitted the event.                      |

**When you'll see it:**

- You sent [`session.end`](#session-end).
- The session hit `max_session_duration_seconds`.
- The server hit an unrecoverable error.
- You disconnected and the 30-second grace window expired.

**How to use it:**

- **Got `session.ended` before the WebSocket closed**: clean server-initiated close. Done. Don't try to resume.
- **WebSocket closed without `session.ended`**: probably a network drop. If you want to keep going, reconnect and call [`session.resume`](#session-resume) with your saved `session_id` within 30 seconds.

---

### `input.speech.started`

Turn detection determined the user has started speaking.

```json
{ "type": "input.speech.started" }
```

---

### `input.speech.stopped`

Turn detection determined the user has stopped speaking.

```json
{ "type": "input.speech.stopped" }
```

---

### `transcript.user.delta`

Partial transcript of what the user is saying, updating in real-time.

```json
{
  "type": "transcript.user.delta",
  "item_id": "item_abc123",
  "text": "What's the weather in"
}
```

| Field     | Type   | Description                                                     |
| --------- | ------ | --------------------------------------------------------------- |
| `item_id` | string | Conversation item this partial belongs to.                     |
| `text`    | string | The full transcript so far for this `item_id`.                 |

`text` is the **full transcript so far**, not an incremental chunk. Each delta supersedes the previous one for that `item_id`; render the latest and do not concatenate. Live user transcripts also pause while a [`hold`-mode tool](/voice-agents/voice-agent-api/tools/overview#execution-modes) is in flight and resume once the hold ends; anything said during the hold is preserved in the conversation context.

---

### `transcript.user`

Final transcript of the user's utterance.

```json
{
  "type": "transcript.user",
  "text": "What's the weather in Tokyo?",
  "item_id": "item_abc123"
}
```

---

### `reply.started`

Agent has begun generating a response.

```json
{
  "type": "reply.started",
  "reply_id": "reply_abc123",
  "item_id": "item_abc123"
}
```

| Field      | Type   | Description                                                        |
| ---------- | ------ | ----------------------------------------------------------------- |
| `reply_id` | string | ID of this reply. For a tool-call reply it is `fc-<call_id>`.      |
| `item_id`  | string | Conversation item ID for this reply.                              |

---

### `reply.audio`

A chunk of the agent's spoken response as base64 PCM16. Decode and play immediately.

```json
{
  "type": "reply.audio",
  "data": "<base64-encoded PCM16>"
}
```

See [Audio format](/voice-agents/voice-agent-api/audio-format#playing-output-audio) for playback guidance.

---

### `transcript.agent.delta`

Word-level streaming of the agent's response, aligned to the audio as it plays. Useful for live captioning in sync with playback. The final [`transcript.agent`](#transcriptagent) still follows once the whole reply has been delivered.

```json
{
  "type": "transcript.agent.delta",
  "reply_id": "reply_abc123",
  "item_id": "item_abc123",
  "delta": "sunny",
  "start_ms": 1200,
  "end_ms": 1560
}
```

| Field      | Type            | Description                                                         |
| ---------- | --------------- | ------------------------------------------------------------------ |
| `reply_id` | string          | ID of the reply this word belongs to.                              |
| `item_id`  | string          | Conversation item ID.                                              |
| `delta`    | string          | The next word (or token) of the agent's speech.                    |
| `start_ms` | integer \| null | Start time of the word within the reply audio, in ms. `null` if unavailable. |
| `end_ms`   | integer \| null | End time of the word within the reply audio, in ms. `null` if unavailable. |

---

### `transcript.agent`

Full text of the agent's response, sent after all audio for the response has been delivered. If the agent was interrupted, `interrupted` is `true` and `text` contains only what was actually spoken before the interruption.

```json
{
  "type": "transcript.agent",
  "text": "It's currently 22°C and sunny in Tokyo.",
  "reply_id": "reply_abc123",
  "item_id": "item_abc123",
  "interrupted": false
}
```

| Field         | Type    | Description                                                     |
| ------------- | ------- | --------------------------------------------------------------- |
| `text`        | string  | What the agent said (trimmed to interruption point if interrupted) |
| `reply_id`    | string  | ID of the reply                                                 |
| `item_id`     | string  | Conversation item ID                                            |
| `interrupted` | boolean | `true` if the user interrupted mid-response                     |

---

### `reply.done`

Agent has finished the reply. Always carries `reply_id` and a `status`.

```json
{
  "type": "reply.done",
  "reply_id": "reply_abc123",
  "status": "completed"
}
```

| Field      | Type   | Description                                                      |
| ---------- | ------ | ---------------------------------------------------------------- |
| `reply_id` | string | ID of the reply that finished. For a tool-call reply it is `fc-<call_id>`. |
| `status`   | string | `"completed"` for a normal finish, or `"interrupted"` if the user barged in. Always present. |

---

### `tool.call`

Agent wants to call a registered tool. `arguments` is a dict, ready to use directly as-is.

```json
{
  "type": "tool.call",
  "call_id": "call_abc123",
  "name": "get_weather",
  "arguments": { "location": "Tokyo" }
}
```

| Field     | Type   | Description                              |
| --------- | ------ | ---------------------------------------- |
| `call_id` | string | Include this in `tool.result`            |
| `name`    | string | Tool name to call                        |
| `arguments`    | object | Arguments as a dict (use directly)       |

See [Tool calling](/voice-agents/voice-agent-api/tools/overview) for the full pattern.

---

### `session.error`

Session or protocol error. The payload always includes `type`, `timestamp`, `code`, and `message`. Some errors (like `session.update` validation failures) also include a `param` field naming the offending field.

```json
{
  "type": "session.error",
  "code": "invalid_format",
  "message": "Invalid message format",
  "timestamp": "2025-01-01T00:00:00Z"
}
```

**Connection and handshake errors**

Sent before or instead of `session.ready`. The WebSocket closes after these with the indicated close code.

| Code               | Close code | Description                                                        |
| ------------------ | ---------- | ------------------------------------------------------------------ |
| `UNAUTHORIZED`     | 1008       | Missing or invalid `Authorization` token                           |
| `FORBIDDEN`        | 1008       | Valid token, but insufficient permissions                          |
| `server_error`     | 1008       | Service at capacity (try again later)                              |
| `INTERNAL_ERROR`   | 1011       | Unexpected exception during connection setup                       |

**Session resume errors**

Sent when `session.resume` fails. The WebSocket closes after these.

| Code               | Close code | Description                                                        |
| ------------------ | ---------- | ------------------------------------------------------------------ |
| `session_not_found`| 1008       | The `session_id` is unknown or the 30-second grace window expired  |
| `session_forbidden`| 1008       | The `session_id` belongs to a different account                    |
| `session_expired`  | 1008       | Session TTL elapsed during the grace window                        |

**Agent startup errors**

Sent after the WebSocket is accepted but before `session.ready`.

| Code                | Description                                                         |
| ------------------- | ------------------------------------------------------------------- |
| `agent_init_failed` | Voice agent worker reported initialization failure                  |
| `agent_timeout`     | Agent did not signal ready within 10 seconds                        |

**Client message errors**

Sent on the open socket when an inbound message is invalid. The session stays alive (except `session_expired`).

| Code               | Description                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------- |
| `invalid_format`   | Bad JSON, missing or unknown `type`, validation failure, or missing `audio` field on `input.audio` |
| `invalid_audio`    | `input.audio` payload failed base64 decode or PCM conversion                                  |
| `invalid_value`    | `session.update` with an invalid voice or field type                                          |
| `immutable_field`  | `session.update` tried to change `greeting`, `output.voice`, or `output.format` after the first update was applied. `output.volume` is mutable and does not raise this error. |
| `invalid_config`   | `session.update` raised a validation error                                                    |
| `server_error`     | Unexpected exception while applying `session.update`                                          |
| `agent_id_not_first` | `agent_id` was sent after the first `session.update`, or alongside inline fields in the same message |
| `agent_not_found`  | `agent_id` does not resolve to an agent on your account                                        |
| `audio_rate_violation` | `input.audio` streamed faster than real time                                              |

**Capacity errors (retryable)**

Reconnect with backoff; these are transient. The session-level codes `at_capacity`, `concurrency_exceeded`, and `internal_error` are the only retryable ones — every other code is fatal.

| Code                  | Description                                              |
| --------------------- | ------------------------------------------------------- |
| `at_capacity`         | Service is temporarily at capacity                      |
| `concurrency_exceeded`| Your account's concurrent-session limit was reached     |

**Live session errors**

| Code               | Close code | Description                                                                                              |
| ------------------ | ---------- | -------------------------------------------------------------------------------------------------------- |
| `session_expired`  | 1008       | Session duration TTL reached. There is no separate "closing soon" warning event before this, so run a client-side timer if you need to wrap up gracefully. |

If the server cancels the session due to an internal error, the WebSocket closes with code `1011` without any `session.error` payload. In browsers, pre-handshake failures (like `UNAUTHORIZED`) surface as a `close` event with code `1006`. You won't receive a `session.error`. Always fetch a fresh token immediately before each connection attempt.

---

## Interruptions

When the user speaks mid-response (barge-in), the server stops the agent and emits [`reply.done`](#reply-done) with `status: "interrupted"` and [`transcript.agent`](#transcript-agent) with `interrupted: true`. The decision is semantic. Back-channels like "uh-huh" don't trigger an interruption.

**On `reply.done` with `status: "interrupted"`:**

1. Flush your local audio playback buffer.
2. Discard any pending `tool.result` accumulators from the just-ended reply.
3. Restart the playback stream so it's ready for the next response.

See [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions) for how the model decides what counts as an interruption, and [Handling interruptions](/voice-agents/voice-agent-api/audio-format#handling-interruptions) for the platform-specific flush pattern.

---

<a id="troubleshooting-docs-voice-agents-voice-agent-api-troubleshooting-mdx"></a>

# Troubleshooting

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/troubleshooting
Source: docs/voice-agents/voice-agent-api/troubleshooting.mdx
Navigation: Voice Agent API > Reference
Description: Common issues and fixes when building with the Voice Agent API.

## Before you contact support

Persist the [`session_id`](/voice-agents/voice-agent-api/events-reference#session-ready) from `session.ready` for every session, not just when something goes wrong. If you contact [support@assemblyai.com](mailto:support@assemblyai.com) about a specific session (audio glitches, unexpected interruptions, tool-call issues, session-resume failures), this ID lets us locate it in our logs immediately.

Log at minimum:

- `session_id` from `session.ready`
- WebSocket close code and reason on disconnect
- Timestamp at session start
- Whether you connected to the US (`agents.assemblyai.com`) or EU endpoint

## Agent interrupts itself (echo / feedback loop)

**Symptom:** Every agent response ends with `(interrupted)` after about one second. The transcript shows the agent's own words echoed back as user speech.

**Cause:** The agent's TTS audio plays through speakers and loops back into the microphone. Terminal apps (Python with `sounddevice`) don't get OS-level acoustic echo cancellation (AEC).

**Fixes:**
- **Use headphones**: the simplest fix.
- **Switch to the browser**: browsers provide AEC automatically through `getUserMedia({ audio: { echoCancellation: true } })`. See [Browser integration](/voice-agents/voice-agent-api/browser-integration).

---

## Wrong sample rate

**Symptom:** Audio sounds garbled, pitched up/down, or plays at the wrong speed.

**Cause:** The Voice Agent API expects PCM16 mono at exactly **24,000 Hz**. If your mic captures at 48 kHz or your playback device runs at a different rate, the audio will be misinterpreted.

**Fixes:**
- **Python:** Set `samplerate=24000` on both `sd.InputStream` and `sd.OutputStream`.
- **Chrome / Edge / Firefox:** Create the AudioContext with `new AudioContext({ sampleRate: 24000 })`. This avoids manual resampling entirely. See the [Browser quickstart](/voice-agents/voice-agent-api/browser-integration#3-browser-client).
- **Safari (desktop and iOS):** Safari ignores the `sampleRate` constructor option and runs the `AudioContext` at the hardware rate (typically 48 kHz). The quickstart will silently produce garbled audio. Let Safari use its default rate and resample to/from 24 kHz inside the worklet. See [Browser compatibility › Safari](/voice-agents/voice-agent-api/browser-integration#safari-resample-inside-the-worklet) for a working pattern.
- If you can't control the device sample rate, resample to/from 24 kHz before encoding/decoding.

---

## Microphone permission denied

**Symptom:** `NotAllowedError` in the browser or `PortAudioError` in Python.

**Fixes:**
- **Browser:** The page must be served over HTTPS (or `localhost`). Check that the user granted microphone permission in the browser prompt.
- **macOS:** Go to System Settings → Privacy & Security → Microphone and enable access for your terminal app or browser.
- **Linux:** Check that your user has access to the audio device (`ls -la /dev/snd/`). You may need to add your user to the `audio` group.

---

## Firewall blocking WebSocket connection

**Symptom:** WebSocket connection hangs or fails with a timeout.

**Cause:** Corporate firewalls or proxies may block outbound WSS (WebSocket Secure) connections on port 443.

**Fixes:**
- Verify that `wss://agents.assemblyai.com` is reachable from your network.
- If behind a corporate proxy, configure your WebSocket client to use the proxy.
- Test from a different network to rule out firewall issues.

---

## Malformed base64 in `input.audio`

**Symptom:** `session.error` with code `invalid_audio`.

**Cause:** The `audio` field in `input.audio` failed base64 decode or PCM conversion. Common mistakes include sending raw binary instead of base64, or encoding audio in the wrong format (e.g., WAV headers included, float32 instead of int16).

**Fixes:**
- Verify you're encoding raw PCM16 bytes, not a WAV or other container format.
- Check that the data is base64-encoded: `base64.b64encode(pcm_bytes).decode()` in Python, or `btoa(String.fromCharCode(...new Uint8Array(buffer)))` in JavaScript.
- Confirm the audio is 16-bit signed integer (little-endian), mono, at 24 kHz.

If the message itself is malformed (bad JSON, missing `type`, or missing `audio` field), you'll get `invalid_format` instead. See the [error codes reference](/voice-agents/voice-agent-api/events-reference#session-error) for the full list.

---

## Token expired or invalid credentials

**Symptom:** WebSocket closes immediately with close code `1008` and an `UNAUTHORIZED` error, or with code `1006` in browsers (no body visible). No `session.ready` event is received.

**Cause:** The token or API key is missing, expired, or invalid. The server sends `UNAUTHORIZED` (close code 1008) before the session is established.

**Fixes:**
- Fetch a fresh token immediately before each connection attempt. Don't pre-fetch and store them.
- Keep `expires_in_seconds` at 60–300 seconds for a good balance between security and usability.
- If using [`session.resume`](/voice-agents/voice-agent-api/events-reference#session-resume), remember that each new WebSocket connection needs a new token.

See [Token expiry and failure modes](/voice-agents/voice-agent-api/browser-integration#token-expiry-and-failure-modes) for more detail.

---

## Unexpected billing after the call ended

**Symptom:** Sessions appear to be billed for ~30 seconds longer than the user was actually on the call.

**Cause:** When the client closes the WebSocket without sending [`session.end`](/voice-agents/voice-agent-api/events-reference#session-end), the server holds the session open for 30 seconds so you can reconnect with [`session.resume`](/voice-agents/voice-agent-api/events-reference#session-resume). That grace window is billable.

**Fix:** Send `session.end` before closing the socket on any intentional disconnect (user hung up, "End call" button, page unload). Skip it only when the disconnect is unintentional and you want the option to resume within 30 seconds.

```js
ws.send(JSON.stringify({ type: "session.end" }));
// Server emits session.ended and closes the WebSocket.
```

See [Ending the session cleanly](/voice-agents/voice-agent-api/browser-integration#6-ending-the-session-cleanly).

---

## Session resume fails

**Symptom:** `session.error` with code `session_not_found`, `session_forbidden`, or `session_expired` after sending `session.resume`.

**Causes:**
- `session_not_found`: the `session_id` is unknown or the 30-second grace window after disconnection has expired.
- `session_forbidden`: the `session_id` belongs to a different account.
- `session_expired`: the session's TTL elapsed during the grace window.

**Fix:** Catch these error codes and start a fresh session without `session.resume`. See the [session.resume example](/voice-agents/voice-agent-api/events-reference#session-resume).

---

<a id="message-sequence-docs-voice-agents-voice-agent-api-message-sequence-mdx"></a>

# Message Sequence

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/message-sequence
Source: docs/voice-agents/voice-agent-api/message-sequence.mdx
Navigation: Voice Agent API > API reference
Description: Canonical reference for the Voice Agent WebSocket protocol. Covers every client and server event and the order they appear in.

This page is the canonical reference for the Voice Agent WebSocket protocol: every event the client sends, every event the server emits, and the order they appear in. For the field-level schema of each event, see the [Voice Agent WebSocket API reference](/voice-agents/voice-agent-api/api-spec/voice-agent-websocket) and the [Events reference](/voice-agents/voice-agent-api/events-reference).

## Sequence diagram

```mermaid
sequenceDiagram
  participant C as Client
  participant S as Server
  C->>S: WebSocket connect (Authorization or ?token=)
  C->>S: session.update (agent_id OR inline system_prompt, tools, input, output)
  S->>C: session.ready (session_id)
  loop while conversing
    C->>S: input.audio (base64 PCM16, ~50 ms chunks)
    S->>C: input.speech.started
    S->>C: transcript.user.delta (partial) xN
    S->>C: input.speech.stopped
    S->>C: transcript.user (final)
    S->>C: reply.started
    S->>C: reply.audio (base64 PCM16 chunks) xN
    S->>C: transcript.agent
    opt tool call
      S->>C: tool.call (name, arguments)
      S->>C: reply.done
      C->>S: tool.result
      S->>C: reply.started
      S->>C: reply.audio xN
      S->>C: transcript.agent
    end
    S->>C: reply.done
  end
  C->>S: session.end
  S->>C: session.ended
  S->>C: close (1000)
```

## Events at a glance

**Client → server**

| Event | Purpose |
| --- | --- |
| `session.update` | Configure the agent. First message after connect; can also be sent mid-session to update mutable fields. |
| `session.resume` | Resume a previous session by `session_id`. |
| `input.audio` | Stream a chunk of PCM16 audio to the agent (base64-encoded). |
| `tool.result` | Return the result of a tool invocation the agent requested. |
| `reply.create` | Ask the agent to generate a reply now, optionally with one-shot instructions. |
| `session.end` | Cleanly end the session. |

**Server → client**

| Event | When | Purpose |
| --- | --- | --- |
| `session.ready` | Once, after `session.update` is applied | Session established. Includes the `session_id`. Start streaming audio. |
| `session.updated` | After a mid-session `session.update` | Acknowledges the update was applied. |
| `input.speech.started` | Per user turn | Turn detection determined the user started speaking. |
| `transcript.user.delta` | Repeatedly during a user turn | Partial transcript of the current user utterance. |
| `input.speech.stopped` | Per user turn | Turn detection determined the user stopped speaking. |
| `transcript.user` | Once per user turn | Final transcript of the user's utterance. |
| `reply.started` | Per agent reply | Agent has begun generating a reply. |
| `reply.audio` | Repeatedly during a reply | A chunk of the agent's spoken response as base64 PCM16. |
| `transcript.agent` | Per agent reply | Text of the agent's response, sent after all reply audio has been delivered. |
| `tool.call` | When the agent invokes a tool | The agent wants to run a registered tool. Respond with `tool.result`. |
| `reply.done` | End of each reply | Marks the end of a reply. If a `tool.call` preceded it, send `tool.result` next. |
| `session.ended` | Once, last message | Session totals and clean shutdown. Nothing follows it. |
| `session.error` | On failure | Session- or protocol-level error. |

## Session initialization

Immediately after the WebSocket opens, send a `session.update` to configure the agent. You have two ways to configure:

- **Stored agent.** Send `{ "agent_id": "<id>" }` to bind to an agent created via the [Agents REST API](/voice-agents/voice-agent-api/api-spec/create-agent). The stored `system_prompt`, `greeting`, `tools`, `input`, and `output` are applied server-side.
- **Inline configuration.** Omit `agent_id` and send `system_prompt`, `greeting`, `tools`, `input`, and `output` directly.

The two modes are mutually exclusive. See [Deploy your agent](/voice-agents/voice-agent-api/deploy) and [Session configuration](/voice-agents/voice-agent-api/session-configuration) for details.

```json
{
  "type": "session.update",
  "session": {
    "system_prompt": "You are a concise assistant.",
    "greeting": "Hi! How can I help?",
    "input":  { "format": { "encoding": "audio/pcm" } },
    "output": { "voice": "alba", "format": { "encoding": "audio/pcm" }, "volume": 100 }
  }
}
```

The server responds with `session.ready` once the session is established:

```json
{
  "type": "session.ready",
  "session_id": "3207b601-2054-48df-ba77-8784dfcf9fb8"
}
```

  Save `session_id` — you'll need it to reconnect with `session.resume` if the WebSocket drops mid-conversation.

## Sending audio

After `session.ready`, stream audio to the agent as `input.audio` events. Each event carries a base64-encoded chunk of PCM16 audio:

```json
{
  "type": "input.audio",
  "audio": "<base64-encoded PCM16>"
}
```

Send audio in chunks of roughly 50 ms. The default input format is mono 24 kHz PCM16; you can change encoding and sample rate through `session.input.format`. See [Audio format](/voice-agents/voice-agent-api/audio-format) for supported encodings and how to stream and play the bytes.

  Do not send `input.audio` before `session.ready`. Audio sent before the session is established is discarded.

## User turn

When turn detection decides the user has started speaking, the server emits `input.speech.started`. As speech continues, partial transcripts stream back as `transcript.user.delta`. When the user stops speaking, the server emits `input.speech.stopped` followed by a final `transcript.user`:

```json
{
  "type": "input.speech.started"
}
```

```json
{
  "type": "transcript.user.delta",
  "delta": "hey there"
}
```

```json
{
  "type": "input.speech.stopped"
}
```

```json
{
  "type": "transcript.user",
  "transcript": "Hey there, what's the weather in Paris?"
}
```

See [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions) for tuning turn detection.

## Agent reply

Once the user turn ends, the agent generates a reply. Every reply is bracketed by `reply.started` and `reply.done`. In between, audio streams back as `reply.audio` events (base64 PCM16), and the full text arrives as `transcript.agent` after all audio has been delivered:

```json
{ "type": "reply.started" }
```

```json
{ "type": "reply.audio", "audio": "<base64 PCM16 chunk>" }
```

```json
{
  "type": "transcript.agent",
  "transcript": "It's 18 degrees and sunny in Paris right now."
}
```

```json
{ "type": "reply.done" }
```

  `transcript.agent` is emitted **after** the final `reply.audio` chunk, so it can be used to display the agent's reply once playback completes.

## Tool calls

If the agent decides to invoke a registered tool, the server emits a `tool.call` event. The current reply ends with `reply.done`, and the client is expected to run the tool and return the result via `tool.result`. A fresh `reply.started` … `reply.done` cycle follows once the tool result is received:

```json
{
  "type": "tool.call",
  "tool_call_id": "call_01H...",
  "name": "get_weather",
  "arguments": { "city": "Paris" }
}
```

```json
{
  "type": "tool.result",
  "tool_call_id": "call_01H...",
  "result": { "temperature_c": 18, "condition": "sunny" }
}
```

See [Tool calling](/voice-agents/voice-agent-api/tools/overview) for the full tool-calling flow, including HTTP tools and client-side tools.

## Updating configuration mid-session

You can send another `session.update` at any time to change mutable fields such as `system_prompt`, `input.turn_detection`, or `output.volume`. The server acknowledges with `session.updated`. Changing immutable fields (for example `greeting`, `output.voice`, or `output.format`) raises `immutable_field`. See [Mutability after `session.ready`](/voice-agents/voice-agent-api/session-configuration#mutability-after-session-ready).

## Session termination

To end a session cleanly, send `session.end`. The server flushes any in-flight events, emits `session.ended`, then closes the WebSocket with code `1000`:

```json
{ "type": "session.end" }
```

```json
{
  "type": "session.ended",
  "session_duration_seconds": 123.4,
  "audio_duration_seconds": 45.6,
  "timestamp": 1752969600.123
}
```

  **Always send `session.end` before closing the WebSocket.** A bare `ws.close()` leaves the session in a 30-second [`session.resume`](/voice-agents/voice-agent-api/events-reference#session-resume) grace window that is billable. See [Ending the session cleanly](/voice-agents/voice-agent-api/browser-integration#6-ending-the-session-cleanly) for browser and server-client patterns.

## Errors

If the session fails at any point, the server sends a `session.error` event and closes the connection:

```json
{
  "type": "session.error",
  "error_code": "immutable_field",
  "message": "Field 'session.output.voice' cannot be changed after session.ready."
}
```

See [Troubleshooting](/voice-agents/voice-agent-api/troubleshooting) for the full list of error codes and recovery guidance.

---

<a id="voice-agent-websocket-api-docs-voice-agents-voice-agent-api-api-spec-voice-agent-websocket-mdx"></a>

# Voice Agent WebSocket API

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/voice-agent-websocket
Source: docs/voice-agents/voice-agent-api/api-spec/voice-agent-websocket.mdx
Navigation: Voice Agent API > API reference
Description: Full AsyncAPI reference for the Voice Agent WebSocket.

Metadata:
- asyncapi: /api-reference/specs/voice-agent-api.yaml voiceAgent

---

<a id="generate-voice-agent-token-docs-voice-agents-voice-agent-api-api-spec-generate-voice-agent-token-mdx"></a>

# Generate voice agent token

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/generate-voice-agent-token
Source: docs/voice-agents/voice-agent-api/api-spec/generate-voice-agent-token.mdx
Navigation: Voice Agent API > API reference
Description: Generate a temporary token for the Voice Agent API.

Metadata:
- openapi: /api-reference/specs/voice-agent-token.yaml GET /v1/token

---

<a id="speech-understanding-docs-speech-understanding-getting-started-mdx"></a>

# Speech Understanding

URL: https://www.assemblyai.com/docs/speech-understanding/getting-started
Source: docs/speech-understanding/getting-started.mdx
Navigation: Speech Understanding > Models & features
Description: Extract structured insights from audio with Speech Understanding models.

Speech Understanding models analyze your transcripts to extract meaningful information like speaker identities, sentiment, topics, and summaries.

  
    Generate action items for your transcripts.
  
  
    Automatically segment your transcript into chapters with summaries.
  
  
    Customize how your transcript text is formatted.
  
  
    Detect and classify entities like names, locations, and organizations in
    your transcripts.
  
  
    Extract the most important phrases and words from your transcript.
  
  
    Detect the sentiment of each sentence in your transcript.
  
  
    Identify and label speakers in your audio to attribute speech to the correct
    person.
  
  
    Generate summaries of your transcripts in different formats.
  
  
    Detect topics discussed in your audio using the IAB taxonomy.
  
  
    Translate transcripts into other languages.

---

<a id="action-items-docs-speech-understanding-action-items-mdx"></a>

# Action Items

URL: https://www.assemblyai.com/docs/speech-understanding/action-items
Source: docs/speech-understanding/action-items.mdx
Navigation: Speech Understanding > Models & features
Description: Generate action items of your audio transcripts

US & EU <br />

Generate action items of your audio transcripts w/ timestamps and a quote of where the action item is from.

Action items is in open beta. Provide us feedback if results are not as good as expected.

## Quickstart

Single call in async transcription:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": audio_url,
    "speech_understanding": {
        "request": {
            "action_items": {}
        }
    }
}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

print(transcription_result["speech_understanding"]["response"]["action_items"])

```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
  "content-type": "application/json",
};

// Step 1: Transcribe your audio file
const audioUrl = "https://assembly.ai/wildfires.mp3";

const data = {
  audio_url: audioUrl,
  speech_understanding: {
    request: {
      action_items: {}
    }
  }
};

const response = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers,
  body: JSON.stringify(data),
});

const { id: transcriptId } = await response.json();
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let transcriptionResult;
while (true) {
  const pollingResponse = await fetch(pollingEndpoint, { headers });
  transcriptionResult = await pollingResponse.json();

  if (transcriptionResult.status === "completed") {
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

console.log(transcriptionResult.speech_understanding.response.action_items)
```

### Example output

```plain
{
    "effort": "low",
    "items": [
      {
        "action_item": "Observe the expansion and contraction of the Arctic ice cap throughout the seasons.",
        "quote": "The Arctic ice cap is in a sense the beating heart of the global climate system. It expands in winter and contracts in summer.",
        "timestamp": 9520
      },
      {
        "action_item": "Analyze a fast-forwarded visualization of the Arctic ice cap's changes over the last 25 years.",
        "quote": "The next slide I show you will be a rapid fast forward of what's happened over the last 25 years.",
        "timestamp": 17680
      },
      {
        "action_item": "Examine the reduction of permanent Arctic ice (five years old or older).",
        "quote": "And the so called permanent ice, five years old or older, you can see it's almost like blood spilling out of the body here. In 25 years it's gone from this to this.",
        "timestamp": 32950
      },
      {
        "action_item": "Recognize the issue of warming heating frozen ground around the Arctic Ocean.",
        "quote": "This is a problem because the warming heats up the frozen ground around the Arctic Ocean",
        "timestamp": 46070
      },
      {
        "action_item": "Understand that thawing frozen carbon in the Arctic is converted to methane by microbes.",
        "quote": "where there is a massive amount of frozen carbon which when it thaws is turned into methane by microbes compared to the.",
        "timestamp": 52390
      }
    ],
    "status": "success"
}
```

## Customize your Action Items

You can control the action items output by adjusting the params.

### Include decisions

You can set the `include_decisions` param to `true` which will configure the call to include decisions made within the
text as action items.

## Effort

The `effort` param lets you determine whether to spend more processing power to get higher quality action items.

Currently there are two options: `low` which is the default, and `medium`.

Action items is not a high precision task, so generally the default `low` effort will be sufficient.

### Example

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file

upload_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker identification

data = {
    "audio_url": upload_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "action_items": {
                "effort": "medium"
            }
        }
    }
}

# Submit the transcription request

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
# ... fetch results in GET

```

### When to use medium

Sometimes you have a use case where you need to make sure specific details are captured, or you have more
challenging topics to extract action items form. In these cases `medium` `effort` will more intelligently parse out
important details.

Examples:
* important meetings where missed details are significant
* multi-lingual texts where it's important to capture all languages
* very long audio, generally 1.5h and longer

---

<a id="rate-limits-docs-speech-understanding-rate-limits-mdx"></a>

# Rate limits

URL: https://www.assemblyai.com/docs/speech-understanding/rate-limits
Source: docs/speech-understanding/rate-limits.mdx
Navigation: Speech Understanding > Models & features
Description: Rate limits documentation.

Requests to the `/understanding` endpoint — covering [Translation](/speech-understanding/translation), [Speaker Identification](/speech-understanding/speaker-identification), and [Custom Formatting](/speech-understanding/custom-formatting) — are rate limited per account, measured as requests within a 60-second window. A single request can include multiple features and counts once against this limit.

## Default limits

| Account type | Rate limit (requests/min) |
| ------------ | -------------------------- |
| Free         | 2                           |
| Paid         | 30                          |

  **Need a higher rate limit?**

  If you need a higher rate limit, [contact our support team](mailto:support@assemblyai.com).

---

<a id="auto-chapters-docs-speech-understanding-auto-chapters-mdx"></a>

# Auto Chapters

URL: https://www.assemblyai.com/docs/speech-understanding/auto-chapters
Source: docs/speech-understanding/auto-chapters.mdx
Navigation: Speech Understanding > Models & features
Description: Automatically segment your audio into chapters with summaries using LLM Gateway

US & EU <br />

Generate chapter summaries from your audio transcripts using [LLM Gateway](/llm-gateway/quickstart). This approach gives you full control over how chapters are created and summarized.

  The `auto_chapters` parameter on the transcription API is deprecated. Use LLM Gateway as shown below for more flexible and powerful chapter summaries.

## Quickstart

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

# Step 1: Transcribe your audio file
audio_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": audio_url,
    "language_detection": True
}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

# Step 2: Get paragraphs from the transcript
paragraphs = requests.get(polling_endpoint + '/paragraphs', headers=headers).json()['paragraphs']

# Step 3: Combine paragraphs into groups for chapter summaries
combined_paragraphs = []
step = 2  # Adjust to control chapter length

for i in range(0, len(paragraphs), step):
    paragraph_group = paragraphs[i : i + step]
    start = paragraph_group[0]['start']
    end = paragraph_group[-1]['end']
    text = " ".join(p['text'] for p in paragraph_group)
    combined_paragraphs.append({"text": text, "start": start, "end": end})

# Step 4: Generate chapter summaries with LLM Gateway
for chapter in combined_paragraphs:
    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": f"Provide a brief one-paragraph summary, a one-line gist, and a headline for this section of a transcript.\n\nText: {chapter['text']}"}
        ],
        "max_tokens": 500
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()["choices"][0]["message"]["content"]
    print(f"{chapter['start']}-{chapter['end']}: {result}\n")
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
  "content-type": "application/json",
};

// Step 1: Transcribe your audio file
const audioUrl = "https://assembly.ai/wildfires.mp3";

const data = {
  audio_url: audioUrl,
  language_detection: true,
};

const response = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers,
  body: JSON.stringify(data),
});

const { id: transcriptId } = await response.json();
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let transcriptionResult;
while (true) {
  const pollingResponse = await fetch(pollingEndpoint, { headers });
  transcriptionResult = await pollingResponse.json();

  if (transcriptionResult.status === "completed") {
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Step 2: Get paragraphs from the transcript
const paragraphsResponse = await fetch(`${pollingEndpoint}/paragraphs`, {
  headers,
});
const { paragraphs } = await paragraphsResponse.json();

// Step 3: Combine paragraphs into groups for chapter summaries
const step = 2; // Adjust to control chapter length
const combinedParagraphs = [];

for (let i = 0; i < paragraphs.length; i += step) {
  const group = paragraphs.slice(i, i + step);
  combinedParagraphs.push({
    text: group.map((p) => p.text).join(" "),
    start: group[0].start,
    end: group[group.length - 1].end,
  });
}

// Step 4: Generate chapter summaries with LLM Gateway
for (const chapter of combinedParagraphs) {
  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content: `Provide a brief one-paragraph summary, a one-line gist, and a headline for this section of a transcript.\n\nText: ${chapter.text}`,
      },
    ],
    max_tokens: 500,
  };

  const result = await fetch(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    {
      method: "POST",
      headers,
      body: JSON.stringify(llmGatewayData),
    }
  );

  const resultData = await result.json();
  console.log(
    `${chapter.start}-${chapter.end}: ${resultData.choices[0].message.content}\n`
  );
}
```

### Example output

```plain
240-60890: 
Headline: Canadian Wildfire Smoke Triggers Air Quality Alerts Across the US
Gist: Wildfire smoke affects US air quality
Summary: Smoke from hundreds of wildfires in Canada is causing hazy conditions and air quality alerts in multiple states. Peter DeCarlo, an environmental health expert from Johns Hopkins University, explains that dry conditions and specific weather patterns are channeling the smoke southward, affecting the mid-Atlantic and Northeast regions.

62270-113214:
Headline: Baltimore Air Quality Reaches Unhealthy Levels Due to Particulate Matter
Gist: Dangerous particulate matter levels in Baltimore
Summary: The air quality in Baltimore has reached unhealthy levels due to high concentrations of particulate matter. These microscopic particles can affect respiratory, cardiovascular, and neurological systems, measuring 150 micrograms per cubic meter—10 times higher than the annual average.
```

## Customize your chapters

You can adjust the chapter generation by modifying the `step` variable to control how many paragraphs are grouped into each chapter, and by customizing the prompt.

### Structured chapter output

For a more structured output, use [Structured Outputs](/llm-gateway/structured-outputs) or specify a JSON format in your prompt:

```python
prompt = """For this section of a transcript, provide the following in JSON format:
{
  "headline": "A single sentence headline",
  "gist": "A few words summarizing the section",
  "summary": "A one paragraph summary"
}

Text: """ + chapter['text']
```

## API reference

### Step 1: Transcribe audio

```bash
curl https://api.assemblyai.com/v2/transcript \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "audio_url": "YOUR_AUDIO_URL"
}'
```

### Step 2: Get paragraphs

Once the transcript is complete, fetch the paragraphs:

```bash
curl https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID/paragraphs \
--header "Authorization: <YOUR_API_KEY>"
```

### Step 3: Generate chapter summaries with LLM Gateway

```bash
curl https://llm-gateway.assemblyai.com/v1/chat/completions \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "model": "claude-sonnet-4-6",
  "messages": [
    {"role": "user", "content": "Provide a brief summary, gist, and headline for this section.\n\nText: YOUR_PARAGRAPH_TEXT"}
  ],
  "max_tokens": 500
}'
```

| Key           | Type   | Description                                                                                |
| ------------- | ------ | ------------------------------------------------------------------------------------------ |
| `model`       | string | The LLM model to use. See [available models](/llm-gateway/available-models). |
| `messages`    | array  | The messages to send to the model, including your prompt and paragraph text.                |
| `max_tokens`  | number | Maximum number of tokens in the response.                                                  |

## Next steps

- [LLM Gateway Overview](/llm-gateway/quickstart) - Learn more about available models and features
- [Apply LLM Gateway to pre-recorded audio](/llm-gateway/apply-llms-to-audio-files) - General guide for using LLM Gateway with transcripts
- [Structured Outputs](/llm-gateway/structured-outputs) - Constrain responses to a specific JSON schema

---

<a id="custom-formatting-docs-speech-understanding-custom-formatting-mdx"></a>

# Custom Formatting

URL: https://www.assemblyai.com/docs/speech-understanding/custom-formatting
Source: docs/speech-understanding/custom-formatting.mdx
Navigation: Speech Understanding > Models & features
Description: Automatically format dates, phone numbers, emails, and more in your transcripts

<br />

  
  <br />

  US & EU <br />

## Overview

The Custom Formatting feature automatically standardizes and formats specific types of information in your transcripts, ensuring consistency across dates, phone numbers, emails, and other data types. This eliminates the need for post-processing and provides clean, formatted output ready for your application.

**Key capabilities:**

- Format dates in your preferred style (US, European, ISO, etc.)
- Standardize phone number formats with custom patterns
- Control currency and decimal precision
- Convert spelled-out text into formatted patterns
- Format URLs as hyperlinks
- Apply multiple formatting rules simultaneously

**Common use cases:**

- Standardizing contact information in customer service transcripts
- Formatting financial data in earnings calls
- Preparing transcripts for CRM systems with specific format requirements
- Creating consistent documentation from meetings
- Processing legal or medical transcripts with strict formatting standards

## Quickstart

There are two ways to use Custom Formatting:

1. **Transcribe and format in one request** - Best when you're starting a new transcription and want to automatically format the transcript text as part of that process
2. **Transcribe and format in separate requests** - Best when you already have text that you would like to format or for more complicated workflows where you want to separate the transcription and formatting tasks

### Method 1: Transcribe and format in one request

This method is ideal when you're starting fresh and want both transcription and formatting in a single workflow.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
audio_url = "https://assembly.ai/phone-msg.m4a"

# Configure transcription with custom formatting
data = {
  "audio_url": audio_url,
  "language_detection": True,
  "speaker_labels": True,
  "speech_understanding": {
    "request": {
      "custom_formatting": {
        "date": "mm/dd/yyyy",
        "phone_number": "(xxx)xxx-xxxx",
        "email": "username@domain.com",
        "format_utterances": True
      }
    }
  }
}

# Submit transcription request
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription results
while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Access and display results
print("\n--- Formatting Details ---")
mapping = transcript['speech_understanding']['response']['custom_formatting']['mapping']
for original, formatted in mapping.items():
  print(f"Original: {original}")
  print(f"Formatted: {formatted}\n")
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
  "content-type": "application/json",
};

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const audioUrl = "https://assembly.ai/phone-msg.m4a";

// Configure transcription with custom formatting
const data = {
  audio_url: audioUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      custom_formatting: {
        date: "mm/dd/yyyy",
        phone_number: "(xxx)xxx-xxxx",
        email: "username@domain.com",
        format_utterances: true,
      },
    },
  },
};

// Submit transcription request
const response = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: headers,
  body: JSON.stringify(data),
});

const { id: transcriptId } = await response.json();
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

// Poll for transcription results
while (true) {
  const pollingResponse = await fetch(pollingEndpoint, { headers });
  const transcript = await pollingResponse.json();

  if (transcript.status === "completed") {
    // Access and display results
    console.log("\n--- Formatting Details ---");
    const mapping =
      transcript.speech_understanding.response.custom_formatting.mapping;
    for (const [original, formatted] of Object.entries(mapping)) {
      console.log(`Original: ${original}`);
      console.log(`Formatted: ${formatted}\n`);
    }
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

```python expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_url = "https://assembly.ai/phone-msg.m4a"

config = aai.TranscriptionConfig(
    language_detection=True,
    speaker_labels=True,
    speech_understanding=aai.SpeechUnderstandingRequest(
        request=aai.SpeechUnderstandingFeatureRequests(
            custom_formatting=aai.CustomFormattingRequest(
                phone_number="(XXX) XXX-XXXX",
            )
        )
    )
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_url, config)

# Print the formatted text
su = transcript.speech_understanding
if su and su.response and su.response.custom_formatting:
    fmt = su.response.custom_formatting
    print("Mapping:", fmt.mapping)
    print("\nFormatted text:", fmt.formatted_text)
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>"
});

const audioUrl = "https://assembly.ai/phone-msg.m4a";

const params = {
  audio: audioUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      custom_formatting: {
        phone_number: "(XXX) XXX-XXXX",
      }
    }
  }
};

const transcript = await client.transcripts.transcribe(params);

// Print the formatted text
const fmt = transcript.speech_understanding?.response?.custom_formatting;
if (fmt) {
  console.log("Mapping:", fmt.mapping);
  console.log("\nFormatted text:", fmt.formatted_text);
}
```

### Method 2: Transcribe and format in separate requests

This method is useful when you already have text that you would like to format or for more complicated workflows where you want to separate the transcription and formatting tasks.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
audio_url = "https://assembly.ai/phone-msg.m4a"

# Submit transcription request (without formatting)
data = {
  "audio_url": audio_url,
  "language_detection": True,
  "speaker_labels": True
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription completion
while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    print("Transcription completed!")
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Add custom formatting configuration to the completed transcript
understanding_body = {
  "transcript_id": transcript_id,
  "speech_understanding": {
    "request": {
      "custom_formatting": {
      "date": "mm/dd/yyyy",
      "phone_number": "(xxx)xxx-xxxx",
      "email": "username@domain.com",
      "format_utterances": True
      }
    }
  }
}

# Send to Speech Understanding API for formatting
result = requests.post(
  "https://llm-gateway.assemblyai.com/v1/understanding",
  headers=headers,
  json=understanding_body
).json()

print("Formatting completed!")

# Access and display results
print("\n--- Formatting Details ---")
mapping = result['speech_understanding']['response']['custom_formatting']['mapping']
for original, formatted in mapping.items():
  print(f"Original: {original}")
  print(f"Formatted: {formatted}\n")
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
  "content-type": "application/json",
};

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const audioUrl = "https://assembly.ai/phone-msg.m4a";

// Submit transcription request (without formatting)
const data = {
  audio_url: audioUrl,
  language_detection: true,
  speaker_labels: true,
};

async function transcribeAndFormat() {
  // Start transcription
  const response = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: headers,
    body: JSON.stringify(data),
  });

  const { id: transcriptId } = await response.json();
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for transcription completion
  while (true) {
    const pollingResponse = await fetch(pollingEndpoint, { headers });
    const transcript = await pollingResponse.json();

    if (transcript.status === "completed") {
      console.log("Transcription completed!");
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }

  // Add custom formatting configuration to the completed transcript
  const understandingBody = {
    transcript_id: transcriptId,
    speech_understanding: {
      request: {
        custom_formatting: {
          date: "mm/dd/yyyy",
          phone_number: "(xxx)xxx-xxxx",
          email: "username@domain.com",
          format_utterances: true,
        },
      },
    },
  };

  // Send to Speech Understanding API for formatting
  const understandingResponse = await fetch(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    {
      method: "POST",
      headers: headers,
      body: JSON.stringify(understandingBody),
    }
  );

  const result = await understandingResponse.json();

  // Access and display results
  console.log("\n--- Formatting Details ---");
  const mapping =
    result.speech_understanding.response.custom_formatting.mapping;
  for (const [original, formatted] of Object.entries(mapping)) {
    console.log(`Original: ${original}`);
    console.log(`Formatted: ${formatted}\n`);
  }
}

transcribeAndFormat();
```

**Expected output:**

```
--- Formatting Details ---
Original: Yes, I would appreciate it if you could call me back. My phone number is 555-679-3466. Also, my cell phone number is 555-679-8244. Once again, if you could call me back, I'd appreciate it. My phone number is 555-679-3466. Thanks.
Formatted: Yes, I would appreciate it if you could call me back. My phone number is (555)679-3466. Also, my cell phone number is (555)679-8244. Once again, if you could call me back, I'd appreciate it. My phone number is (555)679-3466. Thanks.
```

## Output format

Data from the Custom Formatting API will be returned in the `custom_formatted` object, which is contained in the `speech_understanding` object. The `formatted_text` key will included a formatted version of the transcript text.

If Speaker Diarization is used in the request a `formatted_utterances` key will be returned containing formatted utterances with preserved timestamps.

**Example response structure:**

```json expandable
{
  "id": "2accd7f2-445b-4d08-b10b-1bafdd5906ed",
  "status": "completed",
  "text": "Yes, I would appreciate it if you could call me back. My phone number is 555-679-3466...",
  "speech_understanding": {
    "request": {
      "custom_formatting": {
        "date": "mm/dd/yyyy",
        "phone_number": "(xxx)xxx-xxxx",
        "email": "username@domain.com",
        "format_utterances": true
      }
    },
    "response": {
      "custom_formatting": {
        "formatted_text": "Yes, I would appreciate it if you could call me back. My phone number is (555)679-3466...",
        "formatted_utterances": [
          {
            "confidence": 0.9920061471354167,
            "end": 26000,
            "speaker": "A",
            "start": 1920,
            "text": "Yes, I would appreciate it if you could call me back. My phone number is (555)679-3466...",
            "words": [
              {
                "speaker": "A",
                "start": 1920,
                "end": 2160,
                "text": "Yes,",
                "confidence": 0.808349609375
              }
              // ... more words
            ]
          }
        ],
        "mapping": {
          "555-679-3466": "(555)679-3466",
          "555-679-8244": "(555)679-8244"
        },
        "status": "success"
      }
    }
  }
}
```

**Key features of the output:**

- **Formatted Text**: Formatted text can be found in the `formatted_text` key
- **Formatted utterances**: When `format_utterances` is enabled, speaker-separated segments in the `formatted_utterances` key include formatted text
- **Preserved timestamps**: All word-level timestamps in `formatted_utterances` remain intact after formatting, allowing you to maintain temporal alignment with the audio
- **Mapping object**: Shows exactly what transformations were applied (original → formatted)

## Understanding the `custom_formatting` parameter

The `custom_formatting` parameter accepts an object with specific formatting rules for different data types in your transcript. Each property in the object defines how a particular type of information should be formatted.

### Available formatting options

| Parameter           | Type    | Description                                                                                                          | Example Values                                             |
| ------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `date`              | string  | Specifies the format pattern for dates in the transcript                                                             | `"mm/dd/yyyy"`, `"dd/mm/yyyy"`, `"yyyy-mm-dd"`             |
| `phone_number`      | string  | Specifies the format pattern for phone numbers                                                                       | `"(xxx)xxx-xxxx"`, `"xxx-xxx-xxxx"`, `"xxx.xxx.xxxx"`      |
| `email`             | string  | Specifies the format pattern for email addresses                                                                     | `"username@domain.com"`, `"firstname.lastname@domain.com"` |
| `format_utterances` | boolean | When true, applies formatting to utterances in addition to the main text field. Preserves all word-level timestamps. | `true`, `false` (default: `false`)                         |

**Example configuration:**

```json
{
  "custom_formatting": {
    "date": "mm/dd/yyyy",
    "phone_number": "(xxx)xxx-xxxx",
    "email": "username@domain.com",
    "format_utterances": true
  }
}
```

When you include this configuration in your transcription request, the API will automatically detect and format dates, phone numbers, and emails in your transcript according to the specified patterns. With `format_utterances` enabled, the formatting is applied to both the main transcript text and individual speaker utterances while preserving all timing information.

## Common formatting patterns

### Date formats

| Pattern      | Example Output | Description                      |
| ------------ | -------------- | -------------------------------- |
| `mm/dd/yyyy` | 09/19/1991     | US format (month/day/year)       |
| `dd/mm/yyyy` | 19/09/1991     | European format (day/month/year) |
| `yyyy-mm-dd` | 1991-09-19     | ISO 8601 format                  |
| `mm-dd-yyyy` | 09-19-1991     | US format with dashes            |
| `dd.mm.yyyy` | 19.09.1991     | European format with dots        |

### Phone number formats

| Pattern           | Example Output  | Description          |
| ----------------- | --------------- | -------------------- |
| `(xxx)xxx-xxxx`   | (555)679-3466   | Parentheses and dash |
| `xxx-xxx-xxxx`    | 555-679-3466    | Dashes only          |
| `xxx.xxx.xxxx`    | 555.679.3466    | Dots separator       |
| `+x(xxx)xxx-xxxx` | +1(555)679-3466 | International format |

### Email formats

| Pattern                         | Example Output       |
| ------------------------------- | -------------------- |
| `username@domain.com`           | john.doe@example.com |
| `firstname.lastname@domain.com` | john.doe@company.com |

## Best practices

1. **Choose appropriate formats**: Select formatting patterns that match your application's requirements and regional standards.

2. **Combine formatting rules**: You can apply multiple formatting rules simultaneously for comprehensive text standardization.

3. **Test with sample data**: Verify your formatting patterns work correctly with representative audio samples before processing large batches.

4. **Review the mapping**: Check the `mapping` object in the response to see exactly what was changed and verify the results.

5. **Consider regional differences**: Be mindful of date and phone number format differences when processing international content.

## API reference

### Request

#### Method 1: Transcribe and format in one request

When creating a new transcription, include the `speech_understanding` parameter directly in your transcription request:

```bash
curl -X POST \
  "https://api.assemblyai.com/v2/transcript" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://assembly.ai/phone-msg.m4a",
    "speaker_labels": true,
    "speech_understanding": {
      "request": {
        "custom_formatting": {
          "date": "mm/dd/yyyy",
          "phone_number": "(xxx)xxx-xxxx",
          "email": "username@domain.com",
          "format_utterances": true
        }
      }
    }
  }'
```

#### Method 2: Add formatting to existing transcripts

For existing transcripts, retrieve the completed transcript and send it to the Speech Understanding API:

```bash expandable
# Step 1: Get the completed transcript
transcript=$(curl -s -X GET \
  "https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID" \
  -H "Authorization: YOUR_API_KEY")

# Step 2: Add custom formatting and send to Speech Understanding API
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/understanding" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "transcript_id": "{transcript_id}",
    "speech_understanding": {
      "request": {
        "custom_formatting": {
          "date": "mm/dd/yyyy",
          "phone_number": "(xxx)xxx-xxxx",
          "email": "username@domain.com",
          "format_utterances": true
        }
      }
    }
  }'
```

| Key                                              | Type    | Required? | Description                                                                                                                            |
| ------------------------------------------------ | ------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `speech_understanding`                           | object  | Yes       | Container for speech understanding requests.                                                                                           |
| `speech_understanding.request`                   | object  | Yes       | The understanding request configuration.                                                                                               |
| `speech_understanding.request.custom_formatting` | object  | Yes       | Custom formatting configuration.                                                                                                       |
| `custom_formatting.date`                         | string  | No        | Date format pattern. Common patterns: `mm/dd/yyyy` (US), `dd/mm/yyyy` (European), `yyyy-mm-dd` (ISO).                                  |
| `custom_formatting.phone_number`                 | string  | No        | Phone number format pattern. Examples: `(xxx)xxx-xxxx`, `xxx-xxx-xxxx`, `xxx.xxx.xxxx`.                                                |
| `custom_formatting.email`                        | string  | No        | Email format pattern. Example: `username@domain.com`.                                                                                  |
| `custom_formatting.format_utterances`            | boolean | No        | When `true`, applies formatting to speaker utterances in addition to the main text. Preserves word-level timestamps. Default: `false`. |

### Response

The Custom Formatting API returns your original transcript response with formatting applied to the `text` field and additional formatting details in the `speech_understanding` object. When `format_utterances` is enabled, formatted utterances with preserved timestamps are also included.

```json expandable
{
  "id": "2accd7f2-445b-4d08-b10b-1bafdd5906ed",
  "status": "completed",
  "text": "Yes, I would appreciate it if you could call me back. My phone number is (555)679-3466...",
  "speech_understanding": {
    "request": {
      "custom_formatting": {
        "date": "mm/dd/yyyy",
        "phone_number": "(xxx)xxx-xxxx",
        "email": "username@domain.com",
        "format_utterances": true
      }
    },
    "response": {
      "custom_formatting": {
        "formatted_text": "Yes, I would appreciate it if you could call me back. My phone number is (555)679-3466...",
        "formatted_utterances": [
          {
            "confidence": 0.9920061471354167,
            "end": 26000,
            "speaker": "A",
            "start": 1920,
            "text": "Yes, I would appreciate it if you could call me back. My phone number is (555)679-3466...",
            "words": [
              {
                "speaker": "A",
                "start": 1920,
                "end": 2160,
                "text": "Yes,",
                "confidence": 0.808349609375
              }
              // ... more words
            ]
          }
        ],
        "mapping": {
          "555-679-3466": "(555)679-3466",
          "555-679-8244": "(555)679-8244"
        },
        "status": "success"
      }
    }
  }
}
```

| Key                                                                    | Type   | Description                                                                                                                                                                                                                                   |
| ---------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `text`                                                                 | string | The transcript text with custom formatting applied.                                                                                                                                                                                           |
| `speech_understanding`                                                 | object | Container for speech understanding request and response information.                                                                                                                                                                          |
| `speech_understanding.request`                                         | object | The original custom formatting request configuration that was submitted.                                                                                                                                                                      |
| `speech_understanding.request.custom_formatting`                       | object | The formatting parameters that were used.                                                                                                                                                                                                     |
| `speech_understanding.response`                                        | object | The response information from the formatting process.                                                                                                                                                                                         |
| `speech_understanding.response.custom_formatting`                      | object | Details about the formatting operation.                                                                                                                                                                                                       |
| `speech_understanding.response.custom_formatting.formatted_text`       | string | The complete transcript with custom formatting applied. Identical to the `text` field.                                                                                                                                                        |
| `speech_understanding.response.custom_formatting.formatted_utterances` | array  | Array of speaker utterances with formatting applied. Only present when `format_utterances` is `true`. Each utterance includes speaker label, timestamps, confidence scores, formatted text, and word-level details with preserved timestamps. |
| `speech_understanding.response.custom_formatting.mapping`              | object | An object showing the original text segments and their formatted versions. Keys are original text, values are formatted text.                                                                                                                 |
| `speech_understanding.response.custom_formatting.status`               | string | The status of the formatting operation. Will be `"success"` when formatting completes successfully.                                                                                                                                           |

#### Understanding formatted_utterances

When `format_utterances` is enabled, each object in the `formatted_utterances` array contains:

| Field        | Type    | Description                                                                                               |
| ------------ | ------- | --------------------------------------------------------------------------------------------------------- |
| `speaker`    | string  | Speaker identifier (e.g., "A", "B")                                                                       |
| `start`      | integer | Start time of the utterance in milliseconds                                                               |
| `end`        | integer | End time of the utterance in milliseconds                                                                 |
| `text`       | string  | The utterance text with custom formatting applied                                                         |
| `confidence` | number  | Confidence score for the utterance (0-1)                                                                  |
| `words`      | array   | Array of word objects with individual timestamps, text (formatted), confidence scores, and speaker labels |

**Important**: All timestamps in both utterances and words are preserved exactly as they appear in the original transcription, ensuring perfect temporal alignment with the audio even after formatting is applied.

#### Key differences from standard transcription

| Field                  | Standard Transcription                        | With Custom Formatting                                      | With format_utterances=true                            |
| ---------------------- | --------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------ |
| `text`                 | Transcribed text with default formatting      | Transcribed text with your custom formatting rules applied  | Same as custom formatting                              |
| `speech_understanding` | Not present                                   | Object containing formatting request, response, and mapping | Same, plus `formatted_text` and `formatted_utterances` |
| `utterances`           | Speaker-separated segments with original text | Unchanged                                                   | Unchanged (original utterances remain)                 |
| Word timestamps        | Original timestamps                           | Preserved exactly                                           | Preserved exactly in `formatted_utterances`            |

All other fields from the original transcript (`words`, `utterances`, `confidence`, etc.) remain unchanged. The `formatted_utterances` field provides an additional view of the data with formatting applied while maintaining complete timestamp fidelity.

---

<a id="entity-detection-docs-speech-understanding-entity-detection-mdx"></a>

# Entity Detection

URL: https://www.assemblyai.com/docs/speech-understanding/entity-detection
Source: docs/speech-understanding/entity-detection.mdx
Navigation: Speech Understanding > Models & features
Description: Extract named entities from your audio file

<br />

  
  <br />

  US & EU <br />

The Entity Detection model lets you automatically identify and categorize key information in transcribed audio content.

Here are a few examples of what you can detect:

- Names of people
- Organizations
- Addresses
- Phone numbers
- Medical data
- Social security numbers

For the full list of entities that you can detect, see [Supported entities](#supported-entities).

  **Supported Languages**

  Entity Detection is available in multiple languages. See [Supported
  languages](/pre-recorded-audio/supported-languages).

## Quickstart

  
  Enable Entity Detection by setting `entity_detection` to `True` in the JSON payload.

```python {19} expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./local_file.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "entity_detection": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

print(f"Transcript ID: {transcript_id}")

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
      for entity in transcription_result['entities']:
        print(entity['text'])
        print(entity['entity_type'])
        print(f"Timestamp: {entity['start']} - {entity['end']}\n")
      break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

  

  
  Enable Entity Detection by setting `entity_detection` to `True` in the transcription config.

```python {8}
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    entity_detection=True
)

transcript = aai.Transcriber().transcribe(audio_file, config)
print(f"Transcript ID: {transcript.id}")

for entity in transcript.entities:
    print(entity.text)
    print(entity.entity_type)
    print(f"Timestamp: {entity.start} - {entity.end}\n")
```

  

  
  Enable Entity Detection by setting `entity_detection` to `true` in the JSON payload.

```javascript {19} expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  entity_detection: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
console.log("Transcript ID: ", transcriptId);

const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    for (const entity of transcriptionResult.entities) {
      console.log(entity.text);
      console.log(entity.entity_type);
      console.log(`Timestamp: ${entity.start} - ${entity.end}\n`);
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

  
  Enable Entity Detection by setting `entity_detection` to `true` in the transcription config.

```javascript {12} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  entity_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  for (const entity of transcript.entities) {
    console.log(entity.text);
    console.log(entity.entity_type);
    console.log(`Timestamp: ${entity.start} - ${entity.end}\n`);
  }
};

run();
```

  

### Example output

```plain
Canada
location
Timestamp: 2548 - 3130

the US
location
Timestamp: 5498 - 6350

...
```

## API reference

### Request

```bash {6}
curl https://api.assemblyai.com/v2/transcript \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "audio_url": "YOUR_AUDIO_URL",
  "entity_detection": true
}'
```

| Key                | Type    | Description              |
| ------------------ | ------- | ------------------------ |
| `entity_detection` | boolean | Enable Entity Detection. |

### Response

| Key                       | Type   | Description                                                                                      |
| ------------------------- | ------ | ------------------------------------------------------------------------------------------------ |
| `entities`                | array  | An array of detected entities.                                                                   |
| `entities[i].entity_type` | string | The type of entity for the i-th detected entity.                                                 |
| `entities[i].text`        | string | The text for the i-th detected entity.                                                           |
| `entities[i].start`       | number | The starting time, in milliseconds, at which the i-th detected entity appears in the audio file. |
| `entities[i].end`         | number | The ending time, in milliseconds, for the i-th detected entity in the audio file.                |

The response also includes the request parameters used to generate the transcript.

## Supported entities

The model is designed to automatically detect and classify various types of entities within the transcription text. The detected entities and their corresponding types is listed individually in the entities key of the response object, ordered by when they first appear in the transcript.

| Entity name                 | Description                                                                                                    | Example                                                          |
| --------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `account_number`            | Customer account or membership identification number                                                           | Policy No. 10042992; Member ID: HZ-5235-001                      |
| `banking_information`       | Banking information, including account and routing numbers                                                     |                                                                  |
| `blood_type`                | Blood type                                                                                                     | O-, AB positive                                                  |
| `credit_card_cvv`           | Credit card verification code                                                                                  | CVV: 080                                                         |
| `credit_card_expiration`    | Expiration date of a credit card                                                                               |                                                                  |
| `credit_card_number`        | Credit card number                                                                                             |                                                                  |
| `date`                      | Specific calendar date                                                                                         | December 18                                                      |
| `date_interval`             | Broader time periods, including date ranges, months, seasons, years, and decades                               | 2020-2021; 5-9 May; January 1984                                 |
| `date_of_birth`             | Date of birth                                                                                                  | Date of Birth: March 7, 1961                                     |
| `drivers_license`           | Driver's license number                                                                                        | DL# 356933-540                                                   |
| `drug`                      | Medications, vitamins, or supplements                                                                          | Advil, Acetaminophen, Panadol                                    |
| `duration`                  | Periods of time, specified as a number and a unit of time                                                      | 8 months; 2 years                                                |
| `email_address`             | Email address                                                                                                  | support@assemblyai.com                                           |
| `event`                     | Name of an event or holiday                                                                                    | Olympics, Yom Kippur                                             |
| `filename`                  | Names of computer files, including the extension or filepath                                                   | Taxes/2012/brad-tax-returns.pdf                                  |
| `gender_sexuality`          | Terms indicating gender identity or sexual orientation, including slang terms                                  | female; bisexual; trans                                          |
| `healthcare_number`         | Healthcare numbers and health plan beneficiary numbers                                                         | Policy No.: 5584-486-674-YM                                      |
| `injury`                    | Injuries or health issues                                                                                      | I broke my arm, I have a sprained wrist                          |
| `ip_address`                | Internet IP address, including IPv4 and IPv6 formats                                                           | 192.168.0.1                                                      |
| `language`                  | Name of a natural language                                                                                     | Spanish, French                                                  |
| `location`                  | Any Location reference including mailing address, postal code, city, state, province, country, or coordinates. | Lake Victoria, 145 Windsor St., 90210                            |
| `marital_status`            | Terms indicating marital status                                                                                | Single; common-law; ex-wife; married                             |
| `medical_condition`         | Name of a medical condition, disease, syndrome, deficit, or disorder                                           | chronic fatigue syndrome, arrhythmia, depression                 |
| `medical_process`           | Medical process, including treatments, procedures, and tests                                                   | heart surgery, CT scan                                           |
| `money_amount`              | Name and/or amount of currency                                                                                 | 15 pesos, $94.50                                                 |
| `nationality`               | Terms indicating nationality, ethnicity, or race                                                               | American, Asian, Caucasian                                       |
| `number_sequence`           | Numerical PII (including alphanumeric strings) that doesn't fall under other categories                        |                                                                  |
| `occupation`                | Job title or profession                                                                                        | professor, actors, engineer, CPA                                 |
| `organization`              | Name of an organization                                                                                        | CNN, McDonalds, University of Alaska, Northwest General Hospital |
| `passport_number`           | Passport numbers, issued by any country                                                                        | PA4568332; NU3C6L86S12                                           |
| `password`                  | Account passwords, PINs, access keys, or verification answers                                                  | 27%alfalfa, temp1234, My mother's maiden name is Smith           |
| `person_age`                | Number associated with an age                                                                                  | 27, 75                                                           |
| `person_name`               | Name of a person                                                                                               | Bob, Doug Jones, Dr. Kay Martinez, MD                            |
| `phone_number`              | Telephone or fax number                                                                                        |                                                                  |
| `physical_attribute`        | Distinctive bodily attributes, including terms indicating race                                                 | I'm 190cm tall; He belongs to the Black students' association    |
| `political_affiliation`     | Terms referring to a political party, movement, or ideology                                                    | Republican, Liberal                                              |
| `religion`                  | Terms indicating religious affiliation                                                                         | Hindu, Catholic                                                  |
| `statistics`                | Medical statistics                                                                                             | 18%, 18 percent                                                  |
| `time`                      | Expressions indicating clock times                                                                             | 19:37:28; 10pm EST                                               |
| `url`                       | Internet addresses                                                                                             | https://www.assemblyai.com/                                      |
| `us_social_security_number` | Social Security Number or equivalent                                                                           |                                                                  |
| `username`                  | Usernames, login names, or handles                                                                             | @AssemblyAI                                                      |
| `vehicle_id`                | Vehicle identification numbers (VINs), vehicle serial numbers, and license plate numbers                       | 5FNRL38918B111818; BIF7547                                       |
| `zodiac_sign`               | Names of Zodiac signs                                                                                          | Aries; Taurus                                                    |

  Entity Detection returns every location reference under the single `location` entity type. It does **not** emit granular subtypes such as `location_address`, `location_city`, or `location_zip` — those are supported by [PII redaction](/guardrails/redact-pii-from-transcripts#granular-location-policies) only.

## Frequently asked questions

  
The model is capable of identifying entities with variations in spelling or formatting. However, the accuracy of the detection may depend on the severity of the variation or misspelling.

  
No, the Entity Detection model doesn't support the detection of custom entity types. However, the model is capable of detecting a wide range of predefined entity types, including people, organizations, locations, dates, times, addresses, phone numbers, medical data, and banking information, among others.

  
To improve the accuracy of the Entity Detection model, it's recommended to provide high-quality audio files with clear and distinct speech. In addition, it's important to ensure that the audio content is relevant to the use case and that the entities being detected are relevant to the intended analysis. Finally, it may be helpful to review and adjust the model's configuration parameters, such as the confidence threshold for entity detection, to optimize the results.

---

<a id="key-phrases-docs-speech-understanding-key-phrases-mdx"></a>

# Key Phrases

URL: https://www.assemblyai.com/docs/speech-understanding/key-phrases
Source: docs/speech-understanding/key-phrases.mdx
Navigation: Speech Understanding > Models & features
Description: Label key phrases that are spoken in your audio

<br />

  
  <br />

  US & EU <br />

The Key Phrases model identifies significant words and phrases in your transcript and lets you extract the most important concepts or highlights from your audio or video file.

  Key phrase results are not embedded inline in the transcript `text` field.
  They are returned in a separate `auto_highlights_result` object in the API
  response, which contains each phrase along with its relevancy rank, occurrence
  count, and timestamps. See the [Response](#response) section and code examples
  below for how to access this field.

## Quickstart

  
  Enable Key Phrases by setting `auto_highlights` to `True` in the JSON payload.

```python {19} expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./local_file.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "auto_highlights": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

print(f"Transcript ID: {transcript_id}")

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
        for result in transcription_result['auto_highlights_result']['results']:
            print(f"Highlight: {result['text']}, Count: {result['count']}, Rank: {result['rank']}, Timestamps: {result['timestamps']}")
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

  

  
  Enable Key Phrases by setting `auto_highlights` to `True` in the transcription config.

```python {8}
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    auto_highlights=True
)

transcript = aai.Transcriber().transcribe(audio_file, config)
print(f"Transcript ID: {transcript.id}")

for result in transcript.auto_highlights.results:
    print(f"Highlight: {result.text}, Count: {result.count}, Rank: {result.rank}, Timestamps: {result.timestamps}")
```

  

  
  Enable Key Phrases by setting `auto_highlights` to `true` in the JSON payload.

```javascript {19} expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  auto_highlights: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
console.log("Transcript ID: ", transcriptId);

const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    for (const result of transcriptionResult.auto_highlights_result.results) {
      const timestamps = result.timestamps
        .map(({ start, end }) => `[Timestamp(start=${start}, end=${end})]`)
        .join(", ");
      console.log(
        `Highlight: ${result.text}, Count: ${result.count}, Rank: ${result.rank}, Timestamps: ${timestamps}`
      );
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

  
  Enable Key Phrases by setting `auto_highlights` to `true` in the transcription config.

```javascript {12} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  auto_highlights: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  for (const result of transcript.auto_highlights_result.results) {
    const timestamps = result.timestamps
      .map(({ start, end }) => `[Timestamp(start=${start}, end=${end})]`)
      .join(", ");
    console.log(
      `Highlight: ${result.text}, Count: ${result.count}, Rank ${result.rank}, Timestamps: ${timestamps}`
    );
  }
};

run();
```

  

### Example output

```plain
Highlight: air quality alerts, Count: 1, Rank: 0.08, Timestamps: [Timestamp(start=3978, end=5114)]
Highlight: wide ranging air quality consequences, Count: 1, Rank: 0.08, Timestamps: [Timestamp(start=235388, end=238838)]
Highlight: more fires, Count: 1, Rank: 0.07, Timestamps: [Timestamp(start=184716, end=185186)]
...
```

## API reference

### Request

```bash {6}
curl https://api.assemblyai.com/v2/transcript \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "audio_url": "YOUR_AUDIO_URL",
  "auto_highlights": true
}'
```

| Key               | Type    | Description         |
| ----------------- | ------- | ------------------- |
| `auto_highlights` | boolean | Enable Key Phrases. |

### Response

| Key                                                     | Type   | Description                                                                                                                    |
| ------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------ |
| `auto_highlights_result`                                | object | The result of the Key Phrases model.                                                                                           |
| `auto_highlights_result.status`                         | string | Is either `success` or `unavailable` in the rare case that the Key Phrases model failed.                                       |
| `auto_highlights_result.results`                        | array  | A temporally-sequential array of key phrases.                                                                                  |
| `auto_highlights_result.results[i].count`               | number | The total number of times the i-th key phrase appears in the audio file.                                                       |
| `auto_highlights_result.results[i].rank`                | number | The total relevancy to the overall audio file of this key phrase. A greater number means that the key phrase is more relevant. |
| `auto_highlights_result.results[i].text`                | string | The text itself of the key phrase.                                                                                             |
| `auto_highlights_result.results[i].timestamps[j].start` | number | The starting time of the j-th appearance of the i-th key phrase.                                                               |
| `auto_highlights_result.results[i].timestamps[j].end`   | number | The ending time of the j-th appearance of the i-th key phrase.                                                                 |

The response also includes the request parameters used to generate the transcript.

## Frequently Asked Questions

  <p>
    The Key Phrases model uses natural language processing and machine learning
    algorithms to analyze the frequency and distribution of words and phrases in
    your transcription. The algorithm identifies key phrases based on their
    relevancy score, which takes into account factors such as the number of
    times a phrase occurs, the distance between occurrences, and the overall
    length of the transcription.
  </p>

  <p>
    The Key Phrases model is designed to identify important phrases and words in
    your transcription, whereas the Topic Detection model is designed to
    categorize your transcription into predefined topics. While both models use
    natural language processing and machine learning algorithms, they have
    different goals and approaches to analyzing your text.
  </p>

  <p>
    Yes, the Key Phrases model can handle misspelled or unrecognized words to
    some extent. However, the accuracy of the detection may depend on the
    severity of the misspelling or the obscurity of the word. It's recommended
    to provide high-quality, relevant audio files with accurate transcriptions
    for the best results.
  </p>

  <p>
    Some limitations of the Key Phrases model include its limited understanding
    of context, which may lead to inaccuracies in identifying the most important
    phrases in certain cases, such as text with heavy use of jargon or idioms.
    Additionally, the model assigns higher scores to words or phrases that occur
    more frequently in the text, which may lead to an over-representation of
    common words and phrases that may not be as important in the context of the
    text. Finally, the Key Phrases model is a general-purpose algorithm that
    can't be easily customized or fine-tuned for specific domains, meaning it
    may not perform as well for specialized texts where certain keywords or
    concepts may be more important than others.
  </p>

  <p>
    To optimize the performance of the Key Phrases model, it's recommended to
    provide high-quality, relevant audio files with accurate transcriptions, to
    review and adjust the model's configuration parameters, such as the
    confidence threshold for key phrase detection, and to refer to the list of
    identified key phrases to guide the analysis. It may also be helpful to
    consider adding additional training data to the model or consulting with
    AssemblyAI support for further assistance.
  </p>

---

<a id="sentiment-analysis-docs-speech-understanding-sentiment-analysis-mdx"></a>

# Sentiment Analysis

URL: https://www.assemblyai.com/docs/speech-understanding/sentiment-analysis
Source: docs/speech-understanding/sentiment-analysis.mdx
Navigation: Speech Understanding > Models & features
Description: Detect the sentiment of speech in your audio

<br />

  
  <br />

  US & EU <br />

The Sentiment Analysis model detects the sentiment of each spoken sentence in the transcript text. Use Sentiment Analysis to get a detailed analysis of the positive, negative, or neutral sentiment conveyed in the audio, along with a confidence score for each result.

## Quickstart

  
  Enable Sentiment Analysis by setting `sentiment_analysis` to `True` in the JSON payload.

```python {19} expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./local_file.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "sentiment_analysis": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

print(f"Transcript ID: {transcript_id}")

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
      for sentiment_result in transcription_result['sentiment_analysis_results']:
        print(sentiment_result['text'])
        print(sentiment_result['sentiment'])  # POSITIVE, NEUTRAL, or NEGATIVE
        print(sentiment_result['confidence'])
        print(f"Timestamp: {sentiment_result['start']} - {sentiment_result['end']}")
      break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

  

  
  Enable Sentiment Analysis by setting `sentiment_analysis` to `True` in the transcription config.

```python {8} expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    sentiment_analysis=True
)

transcript = aai.Transcriber().transcribe(audio_file, config)
print(f"Transcript ID: {transcript.id}")

for sentiment_result in transcript.sentiment_analysis:
    print(sentiment_result.text)
    print(sentiment_result.sentiment)  # POSITIVE, NEUTRAL, or NEGATIVE
    print(sentiment_result.confidence)
    print(f"Timestamp: {sentiment_result.start} - {sentiment_result.end}")
```

  

  
  Enable Sentiment Analysis by setting `sentiment_analysis` to `true` in the JSON payload.

```javascript {19} expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  sentiment_analysis: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
console.log("Transcript ID: ", transcriptId);

const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    for (const sentimentResult of transcriptionResult.sentiment_analysis_results) {
      console.log(sentimentResult.text);
      console.log(sentimentResult.sentiment); // POSITIVE, NEUTRAL, or NEGATIVE
      console.log(sentimentResult.confidence);
      console.log(
        `Timestamp: ${sentimentResult.start} - ${sentimentResult.end}`
      );
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

  
  Enable Sentiment Analysis by setting `sentiment_analysis` to `true` in the transcription config.

```javascript {12} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  sentiment_analysis: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log("Transcript ID: ", transcript.id);

  for (const result of transcript.sentiment_analysis_results) {
    console.log(result.text);
    console.log(result.sentiment); // POSITIVE, NEUTRAL, or NEGATIVE
    console.log(result.confidence);
    console.log(`Timestamp: ${result.start} - ${result.end}`);
  }
};
run();
```

  

### Example output

```plain
Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US.
NEGATIVE
0.8181032538414001
Timestamp: 250 - 6350
...
```

  **Sentiment Analysis Using LLM Gateway**

  Check out this cookbook [LLM Gateway for Customer Call Sentiment
  Analysis](/guides/call-sentiment-analysis) for an example of how to use
  LLM Gateway to analyze the sentiment of a customer call.

## Add speaker labels to sentiments

  
  To add speaker labels to each sentiment analysis result, using [Speaker Diarization](/pre-recorded-audio/label-speakers), enable `speaker_labels` in the JSON payload.

Each sentiment result will then have a `speaker` field that contains the speaker label.

```python {4}
data = {
    "audio_url": upload_url,
    "sentiment_analysis": True,
    "speaker_labels": True
}
# ...
      for sentiment_result in transcription_result['sentiment_analysis_results']:
        print(sentiment_result['speaker'])
      break
```

  

  
  To add speaker labels to each sentiment analysis result, using [Speaker Diarization](/pre-recorded-audio/label-speakers), enable `speaker_labels` in the transcription config.

Each sentiment result will then have a `speaker` field that contains the speaker label.

```python {3}
config = aai.TranscriptionConfig(
  sentiment_analysis=True,
  speaker_labels=True
)
# ...
for sentiment_result in transcript.sentiment_analysis:
  print(sentiment_result.speaker)
```

  

  
  To add speaker labels to each sentiment analysis result, using [Speaker Diarization](/pre-recorded-audio/label-speakers), enable `speaker_labels` in the JSON payload.

Each sentiment result will then have a `speaker` field that contains the speaker label.

```javascript {4}
const data = {
  audio_url: uploadUrl,
  sentiment_analysis: true,
  speaker_labels: true
}
// ...
    for (const sentimentResult of transcriptionResult.sentiment_analysis_results) {
      console.log(sentimentResult.speaker);
    }
    break;
```

  

  
  To add speaker labels to each sentiment analysis result, using [Speaker Diarization](/pre-recorded-audio/label-speakers), enable `speaker_labels` in the transcription config.

Each sentiment result will then have a `speaker` field that contains the speaker label.

```javascript {4}
const params = {
  audio: audioUrl,
  sentiment_analysis: true,
  speaker_labels: true,
};
// ...
for (const result of transcript.sentiment_analysis_results) {
  console.log(result.speaker);
}
```

  

## API reference

### Request

```bash {6}
curl https://api.assemblyai.com/v2/transcript \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "audio_url": "YOUR_AUDIO_URL",
  "sentiment_analysis": true
}'
```

| Key                  | Type    | Description                |
| -------------------- | ------- | -------------------------- |
| `sentiment_analysis` | boolean | Enable Sentiment Analysis. |

### Response

| Key                                        | Type           | Description                                                                                                                |
| ------------------------------------------ | -------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `sentiment_analysis_results`               | array          | A temporal sequence of Sentiment Analysis results for the audio file, one element for each sentence in the file.           |
| `sentiment_analysis_results[i].text`       | string         | The transcript of the i-th sentence.                                                                                       |
| `sentiment_analysis_results[i].start`      | number         | The starting time, in milliseconds, of the i-th sentence.                                                                  |
| `sentiment_analysis_results[i].end`        | number         | The ending time, in milliseconds, of the i-th sentence.                                                                    |
| `sentiment_analysis_results[i].sentiment`  | string         | The detected sentiment for the i-th sentence, one of `POSITIVE`, `NEUTRAL`, `NEGATIVE`.                                    |
| `sentiment_analysis_results[i].confidence` | number         | The confidence score for the detected sentiment of the i-th sentence, from 0 to 1.                                         |
| `sentiment_analysis_results[i].speaker`    | string or null | The speaker of the i-th sentence if [Speaker Diarization](/pre-recorded-audio/label-speakers) is enabled, else null. |

## Frequently asked questions

  
The Sentiment Analysis model is based on the interpretation of the transcript and may not always accurately capture the intended sentiment of the speaker. It's recommended to take into account the context of the transcript and to validate the sentiment analysis results with human judgment when possible.

  
The [Content Moderation model](/guardrails/detect-sensitive-content) can be used to identify and filter out sensitive or offensive content from the transcript.

  
It's important to ensure that the audio being analyzed is relevant to your use case. Additionally, it's recommended to take into account the context of the transcript and to evaluate the confidence score for each sentiment label.

  
The Sentiment Analysis model is designed to be fast and efficient, but processing times may vary depending on the size of the audio file and the complexity of the language used. If you experience longer processing times than expected, don't hesitate to contact our support team.

---

<a id="speaker-identification-docs-speech-understanding-speaker-identification-mdx"></a>

# Speaker Identification

URL: https://www.assemblyai.com/docs/speech-understanding/speaker-identification
Source: docs/speech-understanding/speaker-identification.mdx
Navigation: Speech Understanding > Models & features
Description: Identify speakers by name or role in your transcript

<br />

  
  <br />

  US & EU <br />

## Overview

Replace generic "Speaker A" and "Speaker B" labels with real names or roles, no voice enrollment needed. Speaker Identification uses conversation content to infer who's speaking and applies the identifiers you provide.

It can infer roles from context clues, and can associate names when the names are present within the transcript.

**Example transformation:**

**Before:**

```txt
Speaker A: Good morning, and welcome to the show.
Speaker B: Thanks for having me.
Speaker A: Let's dive into today's topic...
```

**After (by name):**

```txt
Michel Martin: Good morning, and welcome to the show.
Peter DeCarlo: Thanks for having me.
Michel Martin: Let's dive into today's topic...
```

**After (by role):**

```txt
Interviewer: Good morning, and welcome to the show.
Interviewee: Thanks for having me.
Interviewer: Let's dive into today's topic...
```

  Speaker Identification requires [Speaker Diarization](/pre-recorded-audio/label-speakers). You must set `speaker_labels: true` in your transcription request.

To reliably identify speakers, your audio should contain clear, distinguishable voices and sufficient spoken audio from each speaker. The accuracy of Speaker Diarization depends on the quality of the audio and the distinctiveness of each speaker's voice, which will have a downstream effect on the quality of Speaker Identification.

### Choosing how to identify speakers

You can identify speakers by name or by role:

- **Know the speakers' names?** Use `speaker_type: "name"` with the names in `speakers`. [Click here to learn more.](#identify-by-name)
- **Know their roles but not names?** Use `speaker_type: "role"` with roles like `"Interviewer"` or `"Agent"` in `speakers`. [Click here to learn more.](#identify-by-role)
- **Need better accuracy?** Use `speakers` with `description` fields that provide context about what each speaker typically discusses. [Click here to learn more.](#adding-speaker-metadata)

## How to use Speaker Identification

Include the `speech_understanding` parameter in your transcription request to identify speakers as part of transcription.

### Identify by name

To identify speakers by name, use `speaker_type: "name"` with a list of speaker names in `speakers`. This is the most common approach when you know who is speaking in the audio.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file

upload_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker identification

data = {
    "audio_url": upload_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  # Change these values to match the names of the speakers in your file
            }
        }
    }
}

# Submit the transcription request

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription results

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript["status"] == "completed":
        break

    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    else:
        time.sleep(3)

# Access the results and print utterances to the terminal

for utterance in transcript["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")

````

{/* 
```python expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
audio_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker identification
config = aai.TranscriptionConfig(
  speaker_labels=True,
  speech_understanding=aai.SpeechUnderstandingConfig(
    speaker_identification=aai.SpeakerIdentificationConfig(
      speaker_type="name",
      speakers=[{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  # Change these values to match the names of the speakers in your file
    )
  )
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_url, config)

# Access the results and print utterances to the terminal
for utterance in transcript.utterances:
  print(f"{utterance.speaker}: {utterance.text}")
````

 */}

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  "authorization": "<YOUR_API_KEY>",
  "content-type": "application/json"
};

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const uploadUrl = "https://assembly.ai/wildfires.mp3";

// Configure transcript with speaker identification
const data = {
  audio_url: uploadUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      speaker_identification: {
        speaker_type: "name",
        speakers: [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  // Change these values to match the names of the speakers in your file
      }
    }
  }
};

async function main() {
  // Submit the transcription request
  const response = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: headers,
    body: JSON.stringify(data)
  });

  const { id: transcriptId } = await response.json();
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for transcription results
  while (true) {
    const pollingResponse = await fetch(pollingEndpoint, { headers });
    const transcript = await pollingResponse.json();

    if (transcript.status === "completed") {
      // Access the results and print utterances to the console
      for (const utterance of transcript.utterances) {
        console.log(`${utterance.speaker}: ${utterance.text}`);
      }
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise(resolve => setTimeout(resolve, 3000));
    }
  }
}

main().catch(console.error);

````

```python expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
audio_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker identification
config = aai.TranscriptionConfig(
    language_detection=True,
    speaker_labels=True,
    speech_understanding=aai.SpeechUnderstandingRequest(
        request=aai.SpeechUnderstandingFeatureRequests(
            speaker_identification=aai.SpeakerIdentificationRequest(
                speaker_type="name",
                speakers=[{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  # Change these values to match the names of the speakers in your file
            )
        )
    )
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_url, config)

# Access the results and print utterances to the terminal
for utterance in transcript.utterances:
    print(f"{utterance.speaker}: {utterance.text}")
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>"
});

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const audioUrl = "https://assembly.ai/wildfires.mp3";

// Configure transcript with speaker identification
const params = {
  audio: audioUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      speaker_identification: {
        speaker_type: "name",
        speakers: [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  // Change these values to match the names of the speakers in your file
      }
    }
  }
};

const transcript = await client.transcripts.transcribe(params);

// Access the results and print utterances to the console
for (const utterance of transcript.utterances) {
  console.log(`${utterance.speaker}: ${utterance.text}`);
}
```

{/* 
```javascript expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>"
});

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const audioUrl = "https://assembly.ai/wildfires.mp3";

// Configure transcript with speaker identification
const config = {
  audio_url: audioUrl,
  speaker_labels: true,
  speech_understanding: {
    speaker_identification: {
      speaker_type: "name",
      speakers: [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  // Change these values to match the names of the speakers in your file
    }
  }
};

const transcript = await client.transcripts.transcribe(config);

// Access the results and print utterances to the console
for (const utterance of transcript.utterances) {
  console.log(`${utterance.speaker}: ${utterance.text}`);
}
````

 */}

### Identify by role

To identify speakers by role instead of name, use `speaker_type: "role"` with role labels in `speakers`. This is useful for customer service calls, interviews, or any scenario where you know the roles but not the names.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file

upload_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with role-based speaker identification

data = {
    "audio_url": upload_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "role",
                "speakers": [{"role": "Interviewer"}, {"role": "Interviewee"}]  # Change these values to match the roles of the speakers in your file
            }
        }
    }
}

# Submit the transcription request

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription results

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript["status"] == "completed":
        break

    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    else:
        time.sleep(3)

# Access the results and print utterances to the terminal

for utterance in transcript["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")

````

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  "authorization": "<YOUR_API_KEY>",
  "content-type": "application/json"
};

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const uploadUrl = "https://assembly.ai/wildfires.mp3";

// Configure transcript with role-based speaker identification
const data = {
  audio_url: uploadUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      speaker_identification: {
        speaker_type: "role",
        speakers: [{"role": "Interviewer"}, {"role": "Interviewee"}]  // Change these values to match the roles of the speakers in your file
      }
    }
  }
};

async function main() {
  // Submit the transcription request
  const response = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: headers,
    body: JSON.stringify(data)
  });

  const { id: transcriptId } = await response.json();
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for transcription results
  while (true) {
    const pollingResponse = await fetch(pollingEndpoint, { headers });
    const transcript = await pollingResponse.json();

    if (transcript.status === "completed") {
      // Access the results and print utterances to the console
      for (const utterance of transcript.utterances) {
        console.log(`${utterance.speaker}: ${utterance.text}`);
      }
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise(resolve => setTimeout(resolve, 3000));
    }
  }
}

main().catch(console.error);

````

```python expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_url = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    speaker_labels=True,
    speech_understanding=aai.SpeechUnderstandingRequest(
        request=aai.SpeechUnderstandingFeatureRequests(
            speaker_identification=aai.SpeakerIdentificationRequest(
                speaker_type="role",
                speakers=[{"role": "Interviewer"}, {"role": "Interviewee"}]  # Change these values to match the roles of the speakers in your file
            )
        )
    )
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_url, config)

for utterance in transcript.utterances:
    print(f"{utterance.speaker}: {utterance.text}")
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>"
});

const audioUrl = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      speaker_identification: {
        speaker_type: "role",
        speakers: [{"role": "Interviewer"}, {"role": "Interviewee"}]  // Change these values to match the roles of the speakers in your file
      }
    }
  }
};

const transcript = await client.transcripts.transcribe(params);

for (const utterance of transcript.utterances) {
  console.log(`${utterance.speaker}: ${utterance.text}`);
}
```

#### Common role combinations

- `[{"role": "Agent"}, {"role": "Customer"}]` - Customer service calls
- `[{"role": "AI Assistant"}, {"role": "User"}]` - AI chatbot interactions
- `[{"role": "Support"}, {"role": "Customer"}]` - Technical support calls
- `[{"role": "Interviewer"}, {"role": "Interviewee"}]` - Interview recordings
- `[{"role": "Host"}, {"role": "Guest"}]` - Podcast or show recordings
- `[{"role": "Moderator"}, {"role": "Panelist"}]` - Panel discussions

## Controlling Effort

Sometimes when you are using the speaker identification the results are not quite as good high quality as you want.
To assist with this there is a configuration to increase the quality of the results at a higher cost.

Speaker Identification has two effort modes currently: low and medium. The default is low.

### Example

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file

upload_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker identification

data = {
    "audio_url": upload_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  # Change these values to match the names of the speakers in your file
                "effort": "medium"
            }
        }
    }
}

# Submit the transcription request

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
# ... fetch results in GET

```

### When to use low/default effort

The low/default effort works best for lower complexity situations. Some examples:

* Transcripts less than 10m long
* Transcripts where there are clearly segmented speakers
  * doctor/patient meeting
  * podcasts
  * customer support calls
  * note: audio size is less restrictive here
* Names and/or roles are well demonstrated in the transcript
  * names clearly spoken
  * immediately clear who has which role
    * example: "Hi thank you for calling insert_company_name, how can I help you" clearly indicates a customer support role

### When to use medium effort

The medium effort works best for higher complexity transcripts, where information is less clear and/or speakers are
less defined. Some example:

* Meeting call in a conferance room
* Arguments and/or elevated conversations where individuals interrupt each other
* Transcripts where one or more names are not clearly stated
  * note: names cannot be synthesized from nothing, we cannot extract names if none are present in the transcript
* Transcripts where the roles are nuanced or not clear

## Apply to an existing transcript

If you already have a completed transcript, you can add Speaker Identification in a separate request to the Speech Understanding API. This is useful when you want to re-identify speakers with different parameters, or when your workflow separates transcription from post-processing.

First, transcribe your audio with `speaker_labels: true`. Once transcription is complete, send the `transcript_id` along with your speaker identification configuration to the Speech Understanding API.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file

upload_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": upload_url,
    "language_detection": True,
    "speaker_labels": True
}

# Transcribe file

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription results

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript["status"] == "completed":
        break

    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    else:
        time.sleep(3)

# Enable speaker identification

understanding_body = {
    "transcript_id": transcript_id,
    "speech_understanding": {
        "request": {
            "speaker_identification": {
                "speaker_type": "name",
                "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  # Change these values to match the names of the speakers in your file
            }
        }
    }
}

# Send the modified transcript to the Speech Understanding API

result = requests.post(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    headers=headers,
    json=understanding_body
).json()

# Access the results and print utterances to the terminal

for utterance in result["utterances"]:
    print(f"{utterance['speaker']}: {utterance['text']}")
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const apiKey = "<YOUR_API_KEY>";

const headers = {
  "authorization": apiKey,
  "content-type": "application/json"
};

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const uploadUrl = "https://assembly.ai/wildfires.mp3";

async function transcribeAndIdentifySpeakers() {
  // Transcribe file
  const transcriptResponse = await fetch(`${baseUrl}/v2/transcript`, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify({
      audio_url: uploadUrl,
      language_detection: true,
      speaker_labels: true
    })
  });

  const { id: transcriptId } = await transcriptResponse.json();
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for transcription results
  while (true) {
    const pollingResponse = await fetch(pollingEndpoint, { headers });
    const transcript = await pollingResponse.json();

    if (transcript.status === "completed") {
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise(resolve => setTimeout(resolve, 3000));
    }
  }

  // Enable speaker identification
  const understandingBody = {
    transcript_id: transcriptId,
    speech_understanding: {
      request: {
        speaker_identification: {
          speaker_type: "name",
          speakers: [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]  // Change these values to match the names of the speakers in your file
        }
      }
    }
  };

  // Send the modified transcript to the Speech Understanding API
  const understandingResponse = await fetch(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    {
      method: 'POST',
      headers: headers,
      body: JSON.stringify(understandingBody)
    }
  );

  const result = await understandingResponse.json();

  // Access the results and print utterances to the terminal
  for (const utterance of result.utterances) {
    console.log(`${utterance.speaker}: ${utterance.text}`);
  }
}

transcribeAndIdentifySpeakers();
```

The example above identifies speakers by name. To identify by role, keep the same two-step flow and set `speaker_type: "role"` with role labels in `speakers` (see [Identify by role](#identify-by-role)). The [`speakers` metadata](#adding-speaker-metadata) approach works with this flow too.

## Adding speaker metadata

The `speakers` parameter lets you provide additional metadata about each speaker to help the model identify speakers based on conversational context.

This is particularly useful when:

- Speakers have similar voices but distinct roles or topics
- You want to provide contextual clues about what each speaker typically discusses
- You need more precise identification in complex multi-speaker scenarios

Each speaker object must include either a `name` or `role` (depending on `speaker_type`). Beyond that, you can add any additional properties you want. The `name` and `role` fields are reserved as strings, but all other properties are flexible and can be any structure.

Examples in this section are shown in Python for brevity. The same `speaker_identification` configuration works in any language.

At its simplest, you can provide a `description` alongside each speaker's name or role:

```python expandable
data = {
  "audio_url": upload_url,
  "speaker_labels": True,
  "speech_understanding": {
    "request": {
      "speaker_identification": {
        "speaker_type": "role",
        "speakers": [
          {
            "role": "interviewer",
            "description": "Hosts the program and interviews the guests"
          },
          {
            "role": "guest",
            "description": "Answers questions from the interview"
          }
        ]
      }
    }
  }
}
```

For even more fine-tuned identification, you can include any additional custom properties on each speaker object, such as `company`, `title`, `department`, or any other fields that help describe the speaker:

```python expandable
data = {
  "audio_url": upload_url,
  "speaker_labels": True,
  "speech_understanding": {
    "request": {
      "speaker_identification": {
        "speaker_type": "name",
        "speakers": [
          {
            "name": "Michel Martin",
            "description": "Hosts the program and interviews the guests",
            "company": "NPR",
            "title": "Host Morning Edition"
          },
          {
            "name": "Peter DeCarlo",
            "description": "Answers questions from the interview",
            "company": "Johns Hopkins University",
            "title": "Professor and Vice Chair of Environmental Health and Engineering"
          }
        ]
      }
    }
  }
}
```

You can use the same custom properties with role-based identification by replacing `name` with `role` in each speaker object.

## API reference

### Request

#### In a transcription request

Include the `speech_understanding` parameter directly in your transcription request (shown here with name-based identification):

```bash
curl -X POST \
  "https://api.assemblyai.com/v2/transcript" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speaker_labels": true,
    "speech_understanding": {
      "request": {
        "speaker_identification": {
          "speaker_type": "name",
          "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]
        }
      }
    }
  }'
```

#### On an existing transcript

Transcribe with `speaker_labels: true`, then send the completed `transcript_id` to the Speech Understanding API:

```bash expandable
# Step 1: Submit transcription job
curl -X POST "https://api.assemblyai.com/v2/transcript" \
  -H "authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speaker_labels": true
  }'

# Save the transcript_id from the response above, then use it in the following commands

# Step 2: Poll for transcription status (repeat until status is "completed")
curl -X GET "https://api.assemblyai.com/v2/transcript/{transcript_id}" \
  -H "authorization: <YOUR_API_KEY>"

# Step 3: Once transcription is completed, enable speaker identification
curl -X POST "https://llm-gateway.assemblyai.com/v1/understanding" \
  -H "authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "transcript_id": "{transcript_id}",
    "speech_understanding": {
      "request": {
        "speaker_identification": {
          "speaker_type": "name",
          "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]
        }
      }
    }
  }'
```

#### Request parameters

The following parameters are nested under `speech_understanding.request.speaker_identification`:

| Key                      | Type   | Required?   | Description                                                                                                                                                                                                     |
| ------------------------ | ------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `speaker_type`           | string | Yes         | The type of speakers being identified, values accepted are "name" for actual names or "role" for roles/titles.                                                                                                  |
| `speakers`               | array  | Conditional | An array of speaker objects with metadata. You can include any additional custom properties beyond `name`/`role` and `description`. |
| `speakers[].role`        | string | Conditional | The role of the speaker. Required when `speaker_type` is "role".                                                                                                                                                |
| `speakers[].name`        | string | Conditional | The name of the speaker. Required when `speaker_type` is "name".                                                                                                                                                |
| `speakers[].description` | string | No          | A description of the speaker to help the model identify them based on conversational context.                                                                                                                   |
| `speakers[].<custom>`    | any    | No          | Any additional custom properties (e.g., `company`, `title`, `department`) to provide more context about the speaker. The `name` and `role` fields are reserved as strings, but all other properties are flexible. |

### Response

The Speaker Identification API returns a modified version of your transcript with updated speaker labels in the `utterances` key.

```json expandable
{
  "speech_understanding": {
    "request": {
      "speaker_identification": {
        "speaker_type": "name",
        "speakers": [{"name": "Michel Martin"}, {"name": "Peter DeCarlo"}]
      }
    },
    "response": {
      "speaker_identification": {
        "mapping": {
          "A": "Michel Martin",
          "B": "Peter DeCarlo"
        },
        "status": "success"
      }
    }
  },
  "utterances": [
    {
      "speaker": "Michel Martin",
      "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts...",
      "start": 240,
      "end": 26560,
      "confidence": 0.9815734,
      "words": [
        {
          "text": "Smoke",
          "start": 240,
          "end": 640,
          "confidence": 0.90152997,
          "speaker": "Michel Martin"
        }
        // ... more words
      ]
    }
    // ... more utterances
  ]
}
```

#### Response fields

| Key                                                            | Type   | Description                                                                                                                                                |
| -------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `speech_understanding.response.speaker_identification.mapping` | object | A mapping of the original generic speaker labels (e.g., "A", "B") to the identified speaker names or roles.                                                |
| `speech_understanding.response.speaker_identification.status`  | string | The status of the speaker identification request (e.g., "success").                                                                                        |
| `utterances`                                                   | array  | A turn-by-turn temporal sequence of the transcript, where the i-th element is an object containing information about the i-th utterance in the audio file. |
| `utterances[i].confidence`                                     | number | The confidence score for the transcript of this utterance.                                                                                                 |
| `utterances[i].end`                                            | number | The ending time, in milliseconds, of the utterance in the audio file.                                                                                      |
| `utterances[i].speaker`                                        | string | The identified speaker name or role for this utterance.                                                                                                    |
| `utterances[i].start`                                          | number | The starting time, in milliseconds, of the utterance in the audio file.                                                                                    |
| `utterances[i].text`                                           | string | The transcript for this utterance.                                                                                                                         |
| `utterances[i].words`                                          | array  | A sequential array for the words in the transcript, where the j-th element is an object containing information about the j-th word in the utterance.       |
| `utterances[i].words[j].text`                                  | string | The text of the j-th word in the i-th utterance.                                                                                                           |
| `utterances[i].words[j].start`                                 | number | The starting time for when the j-th word is spoken in the i-th utterance, in milliseconds.                                                                 |
| `utterances[i].words[j].end`                                   | number | The ending time for when the j-th word is spoken in the i-th utterance, in milliseconds.                                                                   |
| `utterances[i].words[j].confidence`                            | number | The confidence score for the transcript of the j-th word in the i-th utterance.                                                                            |
| `utterances[i].words[j].speaker`                               | string | The identified speaker name or role who uttered the j-th word in the i-th utterance.                                                                       |

With Speaker Identification, the `speaker` field in `utterances` and `words` contains the identified name or role (e.g., `"Michel Martin"` or `"Agent"`) instead of generic labels like `"A"`, `"B"`, `"C"`. All other fields (`text`, `start`, `end`, `confidence`, `words`) remain unchanged from the standard transcription response.

---

<a id="summarization-docs-speech-understanding-summarization-mdx"></a>

# Summarization

URL: https://www.assemblyai.com/docs/speech-understanding/summarization
Source: docs/speech-understanding/summarization.mdx
Navigation: Speech Understanding > Models & features
Description: Generate summaries of your audio transcripts

US & EU <br />

Generate summaries of your audio transcripts split into chapters with timestamps.

Summarization is in open beta. Provide us feedback if results are not as good as expected.

## Quickstart

Single call in async transcription:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": audio_url,
    "speech_understanding": {
        "request": {
            "summarization": {
                "summary_type": "paragraph"
            }
        }
    }
}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

print(transcription_result["speech_understanding"]["response"]["summarization"])

```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
  "content-type": "application/json",
};

// Step 1: Transcribe your audio file
const audioUrl = "https://assembly.ai/wildfires.mp3";

const data = {
  audio_url: audioUrl,
  speech_understanding: {
    request: {
      summarization: {
        summary_type: "paragraph"
      }
    }
  }
};

const response = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers,
  body: JSON.stringify(data),
});

const { id: transcriptId } = await response.json();
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let transcriptionResult;
while (true) {
  const pollingResponse = await fetch(pollingEndpoint, { headers });
  transcriptionResult = await pollingResponse.json();

  if (transcriptionResult.status === "completed") {
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

console.log(transcriptionResult.speech_understanding.response.summarization)
```

### Example output

```plain
{
    "status": "success",
    "summary": [
      {
        "start": 240,
        "end": 37100,
        "text": "Smoke from hundreds of Canadian wildfires is causing air quality alerts across the US, turning skylines gray and prompting warnings to stay inside.",
        "headline": "Smoke from Canadian Wildfires Affects US Air Quality"
      },
      {
        "start": 39100,
        "end": 60670,
        "text": "Professor Peter DeCarlo explains that dry conditions and specific weather systems are channeling smoke from Canadian wildfires into the Mid-Atlantic and Northeast regions of the US.",
        "headline": "Weather Systems Channeling Smoke"
      },
      {
        "start": 62350,
        "end": 92190,
        "text": "The harmful component of the haze is particulate matter, microscopic particles smaller than a hair's width that can penetrate the lungs and impact the respiratory, cardiovascular, and neurological systems.",
        "headline": "Particulate Matter Health Risks"
      },
      {
        "start": 93550,
        "end": 135990,
        "text": "Current particulate matter concentrations are extremely high, reaching 150 micrograms per meter cubed, which is over 10 times the annual average and four times the 24-hour limit, posing significant health risks.",
        "headline": "Extreme Concentration Levels"
      },
      {
        "start": 137610,
        "end": 170140,
        "text": "Vulnerable groups include children, the elderly, and individuals with pre-existing respiratory or heart conditions, as their bodies are less able to cope with high levels of air pollution.",
        "headline": "Vulnerable Populations"
      },
      {
        "start": 171020,
        "end": 193260,
        "text": "While some areas like New York currently face higher concentrations, the smoke is expected to shift as weather patterns change, moving the highest levels away from the current regions over the next few days.",
        "headline": "Forecast for Smoke Dispersion"
      },
      {
        "start": 198280,
        "end": 239850,
        "text": "The duration of the smoke impact depends on weather system changes; as the current systems shift, the smoke will be pushed elsewhere, ending the current impact on the region.",
        "headline": "Duration of Impact"
      },
      {
        "start": 241370,
        "end": 280530,
        "text": "Climate change is predicted to lead to longer and more frequent wildfire seasons, resulting in more widespread air quality consequences, particularly affecting the eastern US more often in the future.",
        "headline": "Future Implications of Climate Change"
      }
    ],
    "summary_type": "paragraph",
    "effort": "low"
}
```

## Customize your summary

You can control the summary output by adjusting the params.

### Bullets vs Paragrahs

You can pick between two different `summary_type` params of `bullets` and `paragraph`.

Bullets will get you short concise bullet point style chapter summaries. Meanwhile, paragraph will generally give you
longer more detailed summaries.

## Effort

The `effort` param lets you determine whether to spend more processing power to get higher quality summaries.

Currently there are two options: `low` which is the default, and `medium`.

Summaries are not a data intensive task, nor do they typically need utmost accuracy. For the majority of use
cases leaving `effort` at low/the default will be all you need.

### Example

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file

upload_url = "https://assembly.ai/wildfires.mp3"

# Configure transcript with speaker identification

data = {
    "audio_url": upload_url,
    "language_detection": True,
    "speaker_labels": True,
    "speech_understanding": {
        "request": {
            "summarization": {
                "summary_type": "bullets",
                "effort": "medium"
            }
        }
    }
}

# Submit the transcription request

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
# ... fetch results in GET

```

### When to use medium

Sometimes you have a use case where you need to make sure specific details are captured, or you have more
challenging topics to summarize. In these cases `medium` `effort` will more intelligently parse out important details.

Examples:
* important meetings where missed details are significant
* multi-lingual texts where it's important to capture all languages
* very long audio, generally 1.5h and longer

---

<a id="topic-detection-docs-speech-understanding-topic-detection-mdx"></a>

# Topic Detection

URL: https://www.assemblyai.com/docs/speech-understanding/topic-detection
Source: docs/speech-understanding/topic-detection.mdx
Navigation: Speech Understanding > Models & features
Description: Label topics that are mentioned in your audio file

<br />

  
  <br />

  US & EU <br />

The Topic Detection model lets you identify different topics in the transcript. The model uses the [IAB Content Taxonomy](https://airtable.com/shr7KNXOtvfhTTS4i/tblqVLDb7YSsCMXo4?backgroundColor=purple&viewControls=on), a standardized language for content description which consists of 698 comprehensive topics.

## Quickstart

  
  Enable Topic Detection by setting `iab_categories` to `true` in the JSON payload.

```python {19} expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./local_file.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "iab_categories": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

print(f"Transcript ID: {transcript_id}")

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
      # Get the parts of the transcript that were tagged with topics
      for result in transcription_result['iab_categories_result']['results']:
        print(result['text'])
        print(f"Timestamp: {result['timestamp']['start']} - {result['timestamp']['end']}")

        for label in result['labels']:
          print(f"{label['label']} ({label['relevance']})")

      # Get a summary of all topics in the transcript
      for topic, relevance in transcription_result['iab_categories_result']['summary'].items():
        print(f"Audio is {relevance * 100}% relevant to {topic}")
      break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

  

  
  Enable Topic Detection by setting `iab_categories` to `True` in the transcription parameters.

```python {8} expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    iab_categories=True
)

transcript = aai.Transcriber().transcribe(audio_file, config)
print(f"Transcript ID: {transcript.id}")

# Get the parts of the transcript that were tagged with topics
for result in transcript.iab_categories.results:
    print(result.text)
    print(f"Timestamp: {result.timestamp.start} - {result.timestamp.end}")
    for label in result.labels:
        print(f"{label.label} ({label.relevance})")

# Get a summary of all topics in the transcript
for topic, relevance in transcript.iab_categories.summary.items():
    print(f"Audio is {relevance * 100}% relevant to {topic}")
```

  

  
  Enable Topic Detection by setting `iab_categories` to `true` in the JSON payload.

```javascript {19} expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  iab_categories: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
console.log("Transcript ID: ", transcriptId);

const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    // Get the parts of the transcript that were tagged with topics
    for (const result of transcriptionResult.iab_categories_result.results) {
      console.log(result.text);
      console.log(
        `Timestamp: ${result.timestamp.start} - ${result.timestamp.end}`
      );

      for (const label of result.labels) {
        console.log(`${label.label} (${label.relevance})`);
      }
    }
    // Get a summary of all topics in the transcript
    for (const [topic, relevance] of Object.entries(
      transcriptionResult.iab_categories_result.summary
    )) {
      console.log(`Audio is ${relevance * 100} relevant to ${topic}`);
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

  
  Enable Topic Detection by setting `iab_categories` to `true` in the transcription config.

```javascript {12} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  iab_categories: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log("Transcript ID: ", transcript.id);

  // Get the parts of the transcript that were tagged with topics
  for (const result of transcript.iab_categories_result.results) {
    console.log(result.text);
    console.log(
      `Timestamp: ${result.timestamp?.start} - ${result.timestamp?.end}`
    );
    for (const label of result.labels) {
      console.log(`${label.label} (${label.relevance})`);
    }
  }

  // Get a summary of all topics in the transcript
  for (const [topic, relevance] of Object.entries(
    transcript.iab_categories_result.summary
  )) {
    console.log(`Audio is ${relevance * 100} relevant to ${topic}`);
  }
};

run();
```

  

### Example output

```plain
Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines...
Timestamp: 250 - 28920
Home&Garden>IndoorEnvironmentalQuality (0.9881)
NewsAndPolitics>Weather (0.5561)
MedicalHealth>DiseasesAndConditions>LungAndRespiratoryHealth (0.0042)
...
Audio is 100.0% relevant to NewsAndPolitics>Weather
Audio is 93.78% relevant to Home&Garden>IndoorEnvironmentalQuality
...
```

  **Topic Detection Using LLM Gateway**

  Check out this cookbook [Custom Topic Tags](/guides/custom-topic-tags)
  for an example of how to use LLM Gateway to create custom topic tags.

## API reference

### Request

```bash {6}
curl https://api.assemblyai.com/v2/transcript \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "audio_url": "YOUR_AUDIO_URL",
  "iab_categories": true
}'
```

| Key              | Type    | Description             |
| ---------------- | ------- | ----------------------- |
| `iab_categories` | boolean | Enable Topic Detection. |

### Response

| Key                                                    | Type   | Description                                                                                                                                |
| ------------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `iab_categories_result`                                | object | The result of the Topic Detection model.                                                                                                   |
| `iab_categories_result.status`                         | string | Is either `success`, or `unavailable` in the rare case that the Topic Detection model failed.                                           |
| `iab_categories_result.results`                        | array  | An array of the Topic Detection results.                                                                                                   |
| `iab_categories_result.results[i].text`                | string | The text in the transcript in which the i-th instance of a detected topic occurs.                                                          |
| `iab_categories_result.results[i].labels[j].relevance` | number | How relevant the j-th detected topic is in the i-th instance of a detected topic.                                                          |
| `iab_categories_result.results[i].labels[j].label`     | string | The IAB taxonomical label for the j-th label of the i-th instance of a detected topic, where `>` denotes supertopic/subtopic relationship. |
| `iab_categories_result.results[i].timestamp.start`     | number | The starting time in the audio file at which the i-th detected topic instance is discussed.                                                |
| `iab_categories_result.results[i].timestamp.end`       | number | The ending time in the audio file at which the i-th detected topic instance is discussed.                                                  |
| `iab_categories_result.summary`                        | object | Summary where each property is a detected topic.                                                                                           |
| `iab_categories_result.summary.topic`                  | number | The overall relevance of <i>topic</i> to the entire audio file.                                                                            |

The response also includes the request parameters used to generate the transcript.

## Frequently asked questions

{" "}

  <p>
    The Topic Detection model uses natural language processing and machine
    learning to identify related words and phrases even if they are misspelled
    or unrecognized. However, the accuracy of the detection may depend on the
    severity of the misspelling or the obscurity of the word.
  </p>

  <p>
    No, the Topic Detection model can only identify entities that are part of
    the IAB Taxonomy. The model is optimized for contextual targeting use cases,
    so using the predefined IAB categories ensures the most accurate results.
  </p>

  <p>
    There could be several reasons why you aren't getting any topic predictions
    for your audio file. One possible reason is that the audio file doesn't
    contain enough relevant content for the model to analyze. Additionally, the
    accuracy of the predictions may be affected by factors such as background
    noise, low-quality audio, or a low confidence threshold for topic detection.
    It's recommended to review and adjust the model's configuration parameters
    and to provide high-quality, relevant audio files for analysis.
  </p>

  <p>
    There could be several reasons why you're getting inaccurate or irrelevant
    topic predictions for your audio file. One possible reason is that the audio
    file contains background noise or other non-relevant content that's
    interfering with the model's analysis. Additionally, the accuracy of the
    predictions may be affected by factors such as low-quality audio, a low
    confidence threshold for topic detection, or insufficient training data.
    It's recommended to review and adjust the model's configuration parameters,
    to provide high-quality, relevant audio files for analysis, and to consider
    adding additional training data to the model.
  </p>

  <p>
    As of 2023, AssemblyAI is a partner with the Interactive Advertising Bureau
    (IAB), a certification and community for advertising across the internet.
    AssemblyAI built Topic Detection using the IAB Taxonomy, which is a
    blueprint of the approximately 700 topics used to categorize ads.
  </p>

---

<a id="translation-docs-speech-understanding-translation-mdx"></a>

# Translation

URL: https://www.assemblyai.com/docs/speech-understanding/translation
Source: docs/speech-understanding/translation.mdx
Navigation: Speech Understanding > Models & features
Description: Translate your transcripts from one language to another

<br />

  
  <br />

  US & EU <br />

## Overview

The Translation feature automatically converts your transcribed audio content from one language to another, enabling you to reach global audiences without manual translation work. You can translate transcripts into 86 languages with a single API request.

**Key capabilities:**

- Translate to multiple target languages simultaneously
- Choose between formal and informal translation styles
- Translate during transcription or add translations to existing transcripts
- Get full-text translations that preserve the original meaning and context
- Get per-speaker translated utterances when using Speaker Labels

**Common use cases:**

- Creating multilingual subtitles for video content
- Translating customer support calls for international teams
- Localizing podcast episodes for different markets
- Making educational content accessible in multiple languages
- Generating multilingual meeting summaries

## Quickstart

There are two ways to use Translation:

1. **Transcribe and translate in one request** - Best when you're starting a new transcription and want to automatically translate the transcript text as part of that process
2. **Transcribe and translate in separate requests** - Best when you already have text that you would like to translate or for more complicated workflows where you want to separate the transcription and translation tasks

### Method 1: Transcribe and translate in one request

This method is ideal when you're starting fresh and want both transcription and translation in a single workflow.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "YOUR_API_KEY"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
audio_url = "https://assembly.ai/wildfires.mp3"

# Configure transcription with translation
data = {
  "audio_url": audio_url,
  "language_detection": True,
  "speaker_labels": True,  # Enable speaker labels
  "speech_understanding": {
    "request": {
      "translation": {
        "target_languages": ["es", "de"],  # Translate to Spanish and German
        "formal": True  # Use formal language style
      }
    }
  }
}

# Submit transcription request
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll transcription results
while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Access and display results
print("\n--- Original Transcript ---")
print(transcript['text'][:200] + "...\n")

print("--- Translations ---")
for language_code, translated_text in transcript['translated_texts'].items():
  print(f"{language_code.upper()}:")
  print(translated_text[:200] + "...\n")
```

```javascript expandable
import fetch from "node-fetch";

const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "YOUR_API_KEY",
};

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const audioUrl = "https://assembly.ai/wildfires.mp3";

// Configure transcription with translation
const data = {
  audio_url: audioUrl,
  language_detection: true,
  speaker_labels: true, // Enable speaker labels
  speech_understanding: {
    request: {
      translation: {
        target_languages: ["es", "de"], // Translate to Spanish and German
        formal: true, // Use formal language style
      },
    },
  },
};

// Submit transcription request
async function transcribeAndTranslate() {
  const response = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: {
      ...headers,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(data),
  });

  const transcript = await response.json();
  const transcriptId = transcript.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll transcription results
  while (true) {
    const pollingResponse = await fetch(pollingEndpoint, { headers });
    const transcriptResult = await pollingResponse.json();

    if (transcriptResult.status === "completed") {
      // Access and display results
      console.log("\n--- Original Transcript ---");
      console.log(transcriptResult.text.substring(0, 200) + "...\n");

      console.log("--- Translations ---");
      for (const [languageCode, translatedText] of Object.entries(
        transcriptResult.translated_texts
      )) {
        console.log(`${languageCode.toUpperCase()}:`);
        console.log(translatedText.substring(0, 200) + "...\n");
      }
      break;
    } else if (transcriptResult.status === "error") {
      throw new Error(`Transcription failed: ${transcriptResult.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

transcribeAndTranslate();
```

```python expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_url = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    speaker_labels=True,
    speech_understanding=aai.SpeechUnderstandingRequest(
        request=aai.SpeechUnderstandingFeatureRequests(
            translation=aai.TranslationRequest(
                target_languages=["es", "fr"],
            )
        )
    )
)

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_url, config)

# Print the translated texts
for lang, text in transcript.json_response["translated_texts"].items():
    print(f"\n[{lang}]:\n{text}")
```

```javascript expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>"
});

const audioUrl = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioUrl,
  language_detection: true,
  speaker_labels: true,
  speech_understanding: {
    request: {
      translation: {
        target_languages: ["es", "fr"],
      }
    }
  }
};

const transcript = await client.transcripts.transcribe(params);

// Print the translated texts
for (const [lang, text] of Object.entries(transcript.translated_texts)) {
  console.log(`\n[${lang}]:\n${text}`);
}
```

### Method 2: Transcribe and translate in separate requests

This method is useful when you already have text that you would like to translate or for more complicated workflows where you want to separate the transcription and translation tasks.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
audio_url = "https://assembly.ai/wildfires.mp3"

# Submit transcription request (without translation)
data = {
  "audio_url": audio_url,
  "language_detection": True,
  "speaker_labels": True,
}

# Transcribe file
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for transcription completion
while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    print("Transcription completed!")
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Add translation configuration to the completed transcript
understanding_body = {
  "transcript_id": transcript_id,
  "speech_understanding": {
    "request": {
      "translation": {
      "target_languages": ["es", "de"],  # Translate to Spanish and German
      "formal": True  # Use formal language style
      }
    }
  }
}

# Send to Speech Understanding API for translation
result = requests.post(
  "https://llm-gateway.assemblyai.com/v1/understanding",
  headers=headers,
  json=understanding_body
).json()

# Access and display results
print("\n--- Original Transcript ---")
print(transcript['text'][:200] + "...\n")

print("--- Translations ---")
for language_code, translated_text in result['translated_texts'].items():
  print(f"{language_code.upper()}:")
  print(translated_text[:200] + "...\n")
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
  "content-type": "application/json",
};

// Need to transcribe a local file? Learn more here: https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
const audioUrl = "https://assembly.ai/wildfires.mp3";

// Helper function to sleep
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function main() {
  // Submit transcription request (without translation)
  const data = {
    audio_url: audioUrl,
    language_detection: true,
    speaker_labels: true,
  };

  // Transcribe file
  const response = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: headers,
    body: JSON.stringify(data),
  });

  const responseData = await response.json();
  const transcriptId = responseData.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for transcription completion
  let transcript;
  while (true) {
    const pollResponse = await fetch(pollingEndpoint, {
      method: "GET",
      headers: headers,
    });

    transcript = await pollResponse.json();

    if (transcript.status === "completed") {
      console.log("Transcription completed!");
      break;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await sleep(3000);
    }
  }

  // Add translation configuration to the completed transcript
  const understandingBody = {
    transcript_id: transcriptId,
    speech_understanding: {
      request: {
        translation: {
          target_languages: ["es", "de"], // Translate to Spanish and German
          formal: true, // Use formal language style
        },
      },
    },
  };

  // Send to Speech Understanding API for translation
  const resultResponse = await fetch(
    "https://llm-gateway.assemblyai.com/v1/understanding",
    {
      method: "POST",
      headers: headers,
      body: JSON.stringify(understandingBody),
    }
  );

  const result = await resultResponse.json();

  // Access and display results
  console.log("\n--- Original Transcript ---");
  console.log(transcript.text.substring(0, 200) + "...\n");

  console.log("--- Translations ---");
  for (const [languageCode, translatedText] of Object.entries(
    result.translated_texts
  )) {
    console.log(`${languageCode.toUpperCase()}:`);
    console.log(translatedText.substring(0, 200) + "...\n");
  }
}

// Run the main function
main().catch((error) => {
  console.error("Error:", error);
});
```

**Expected output:**

```
--- Original Transcript ---
Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US...

--- Translations ---

ES:
El humo de cientos de incendios forestales en Canadá está provocando alertas de calidad del aire...

DE:
Rauch von Hunderten von Waldbränden in Kanada löst in den gesamten USA Luftqualitätswarnungen aus...
```

## Output format

The Translation API returns translations in the `translated_texts` key of the response. This key contains an object where each property is a language code corresponding to one of your target languages, and the value is the full translated text.

**Example response structure:**

```json expandable
{
  "id": "735d90b6-2e8b-4748-b75d-d02b78eb7811",
  "status": "completed",
  "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts...",
  "translated_texts": {
    "es": "El humo de cientos de incendios forestales en Canadá está provocando alertas de calidad del aire...",
    "de": "Rauch von Hunderten von Waldbränden in Kanada löst in den gesamten USA Luftqualitätswarnungen aus..."
  },
  "speech_understanding": {
    "request": {
      "translation": {
        "formal": true,
        "target_languages": [
          "es",
          "de"
        ],
      }
    },
    "response": {
      "translation": {
        "status": "success"
      }
    }
  },
  "utterances": [
    {
      "speaker": "A",
      "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts...",
      "confidence": 0.9815734,
      "start": 240,
      "end": 26560,
      "words": [
        {
          "text": "Smoke",
          "start": 240,
          "end": 640,
          "confidence": 0.90152997,
          "speaker": "A"
        },
        // ... more words
      ],
      "translated_texts": {
        "es": "El humo de cientos de incendios forestales en Canadá está provocando alertas de calidad del aire...",
        "de": "Rauch von Hunderten von Waldbränden in Kanada löst in den gesamten USA Luftqualitätswarnungen aus..."
      }
    },
    // ... more utterances
  ],
  ...
}
```

## Translation with speaker labels

When you use Translation with [Speaker Labels](/pre-recorded-audio/label-speakers), you can get translated text for each individual utterance by setting `match_original_utterance` to `true`. This is useful for creating speaker-specific subtitles or analyzing conversations in multiple languages while preserving speaker attribution.

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "YOUR_API_KEY"
}

audio_url = "https://assembly.ai/wildfires.mp3"

# Configure transcription with translation and speaker labels
data = {
  "audio_url": audio_url,
  "language_detection": True,
  "speaker_labels": True,  # Enable speaker labels
  "speech_understanding": {
    "request": {
      "translation": {
        "target_languages": ["es"],
        "match_original_utterance": True,  # Get translated text per utterance
        "formal": True
      }
    }
  }
}

# Submit transcription request
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll transcription results
while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Access translated utterances
for utterance in transcript["utterances"]:
  print(f"Speaker {utterance['speaker']}:")
  print(f"  Original: {utterance['text'][:100]}...")
  print(f"  Spanish: {utterance['translated_texts']['es'][:100]}...")
  print()
```

```javascript expandable
import fetch from "node-fetch";

const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "YOUR_API_KEY",
};

const audioUrl = "https://assembly.ai/wildfires.mp3";

// Configure transcription with translation and speaker labels
const data = {
  audio_url: audioUrl,
  language_detection: true,
  speaker_labels: true, // Enable speaker labels
  speech_understanding: {
    request: {
      translation: {
        target_languages: ["es"],
        match_original_utterance: true, // Get translated text per utterance
        formal: true,
      },
    },
  },
};

async function transcribeWithTranslatedUtterances() {
  const response = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: {
      ...headers,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(data),
  });

  const transcript = await response.json();
  const transcriptId = transcript.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll transcription results
  while (true) {
    const pollingResponse = await fetch(pollingEndpoint, { headers });
    const transcriptResult = await pollingResponse.json();

    if (transcriptResult.status === "completed") {
      // Access translated utterances
      for (const utterance of transcriptResult.utterances) {
        console.log(`Speaker ${utterance.speaker}:`);
        console.log(`  Original: ${utterance.text.substring(0, 100)}...`);
        console.log(
          `  Spanish: ${utterance.translated_texts.es.substring(0, 100)}...`
        );
        console.log();
      }
      break;
    } else if (transcriptResult.status === "error") {
      throw new Error(`Transcription failed: ${transcriptResult.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

transcribeWithTranslatedUtterances();
```

**Example response:**

Each utterance in the `utterances` array includes a `translated_texts` object with the translation for that specific speaker's utterance:

```json expandable
{
  "utterances": [
    {
      "speaker": "A",
      "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts...",
      "confidence": 0.9815734,
      "start": 240,
      "end": 26560,
      "words": [...],
      "translated_texts": {
        "es": "El humo de cientos de incendios forestales en Canadá está activando alertas de calidad del aire..."
      }
    },
    {
      "speaker": "B",
      "text": "Good morning.",
      "confidence": 0.98217773,
      "start": 28060,
      "end": 28620,
      "words": [...],
      "translated_texts": {
        "es": "Buenos días."
      }
    }
  ]
}
```

## Force Translation

By default our translation system will skip translating if the target language and the detected language is the same.
For example if a transcript is english and you request a language to be translated to english then we will skip the
translation and prevent you from being charged for a useless action.

However, on occasion a transcript will have a brief intro in a language, but then the rest of the transcript is another.
Also some transcripts have multiple languages being spoken. If you know this is common and you see translations being
skipped, you can set `force_translation` to true to ensure the translation happens.

## API reference

### Request

#### Method 1: Transcribe and translate in one request

When creating a new transcription, include the `speech_understanding` parameter directly in your transcription request:

```bash
curl -X POST \
  "https://api.assemblyai.com/v2/transcript" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://assembly.ai/wildfires.mp3",
    "speaker_labels": true,
    "language_detection": true,
    "speech_understanding": {
      "request": {
        "translation": {
          "target_languages": ["es", "de"],
          "formal": true
        }
      }
    }
  }'
```

#### Method 2: Add translation to existing transcripts

For existing transcripts, retrieve the completed transcript and send it to the Speech Understanding API:

```bash {6}  expandable
# Step 1: Get the completed transcript
transcript=$(curl -s -X GET \
  "https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID" \
  -H "Authorization: YOUR_API_KEY")

# Step 2: Add translation and send to Speech Understanding API
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/understanding" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "transcript_id": "{transcript_id}",
    "speech_understanding": {
      "request": {
        "translation": {
          "target_languages": ["es", "de"],
          "formal": true
        }
      }
    }
  }'
```

| Key                                        | Type    | Required? | Description                                                                                                                                                                                                                               |
| ------------------------------------------ | ------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `speech_understanding`                     | object  | Yes       | Container for speech understanding requests.                                                                                                                                                                                              |
| `speech_understanding.request`             | object  | Yes       | The understanding request configuration.                                                                                                                                                                                                  |
| `speech_understanding.request.translation` | object  | Yes       | Translation configuration.                                                                                                                                                                                                                |
| `translation.target_languages`             | array   | Yes       | Array of language codes to translate the transcript into. See the supported languages table for available language codes.                                                                                                                 |
| `translation.formal`                       | boolean | No        | Whether to use formal language in translations. Defaults to `false`. When `true`, uses formal pronouns and grammatical forms.                                                                                                             |
| `translation.match_original_utterance`     | boolean | No        | Whether to include translated texts for each utterance. Defaults to `false`. When `true`, returns a `translated_texts` key within each utterance in the `utterances` array. Requires `speaker_labels` to be set to `true` in the request. |
| `translation.force_translation`     | boolean | No        | Force the system to translate even if it detects that the target language and detected language are the same.  |

### Response

The Translation API returns your original transcript response with an additional `translated_texts` key containing the translations. When `match_original_utterance` is enabled with `speaker_labels`, each utterance in the `utterances` array will also include its own `translated_texts` key.

```json expandable
{
  "id": "735d90b6-2e8b-4748-b75d-d02b78eb7811",
  "status": "completed",
  "text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US...",
  "translated_texts": {
    "es": "El humo de cientos de incendios forestales en Canadá está provocando alertas de calidad del aire en todo Estados Unidos...",
    "de": "Rauch von Hunderten von Waldbränden in Kanada löst in den gesamten USA Luftqualitätswarnungen aus..."
  },
  "speech_understanding": {
    "request": {
      "translation": {
        "formal": true,
        "target_languages": ["es", "de"]
      }
    },
    "response": {
      "translation": {
        "status": "success"
      }
    }
  }
}
```

| Key                                                | Type   | Description                                                                                                                                 |
| -------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `translated_texts`                                 | object | An object containing the translated texts, where each key is a language code and each value is the full translated transcript text.         |
| `utterances[].translated_texts`                    | object | (When `match_original_utterance` is `true`) An object containing the translations for this specific utterance, with language codes as keys. |
| `speech_understanding`                             | object | Container for speech understanding request and response information.                                                                        |
| `speech_understanding.request`                     | object | The original translation request configuration that was submitted.                                                                          |
| `speech_understanding.request.translation`         | object | The translation parameters that were used.                                                                                                  |
| `speech_understanding.response`                    | object | The response information from the translation process.                                                                                      |
| `speech_understanding.response.translation`        | object | Status information about the translation.                                                                                                   |
| `speech_understanding.response.translation.status` | string | The status of the translation. Will be `"success"` when translation completes successfully.                                                 |

#### Key differences from standard transcription

| Field                  | Standard Transcription | With Translation                                                  |
| ---------------------- | ---------------------- | ----------------------------------------------------------------- |
| `translated_texts`     | Not present            | Object with language codes as keys and translated texts as values |
| `speech_understanding` | Not present            | Object containing the translation request and response details    |

All other fields from the original transcript (`text`, `words`, `utterances`, `confidence`, etc.) remain unchanged.

---

<a id="migration-guide-auto-chapters-docs-speech-understanding-migration-guides-auto-chapters-mdx"></a>

# Migration guide: Auto Chapters

URL: https://www.assemblyai.com/docs/speech-understanding/migration-guides/auto-chapters
Source: docs/speech-understanding/migration-guides/auto-chapters.mdx
Navigation: Speech Understanding > Guides > Migration guides
Description: Migration guide: Auto Chapters documentation.

This guide walks through the process of upgrading from the deprecated `auto_chapters` parameter to [Summarization](/speech-understanding/summarization) under Speech Understanding.

  **This guide is for existing accounts using the deprecated `auto_chapters`
  parameter.** If you're adding chapters for the first time, go straight to the
  [Summarization documentation](/speech-understanding/summarization) — it
  returns topic summaries with headlines and timestamps in one call, which is
  what chapters are.

We replaced Auto Chapters with Summarization. Both return the transcript split into topic sections with a headline and timestamps per section, so the response shapes line up closely. The changes are to the request shape, the response location, and the fields on each section.

- **Topic sections replace chapters.** You still get the transcript broken into ordered sections with `start`, `end`, and a headline per section.
- **One `text` field per topic.** The deprecated parameter returned three overlapping text fields (`summary`, `gist`, `headline`) per chapter; Summarization returns `text` and `headline`.
- **Quality you control.** The new `effort` parameter lets you spend more processing on harder content — useful for meetings, multilingual audio, or files longer than about 1.5 hours.

Your existing API key and endpoint stay the same.

## Quick upgrade

Replace `auto_chapters: true` with a `speech_understanding.request.summarization` object:

```json
// Before (deprecated)
{
  "audio_url": "https://example.com/audio.mp3",
  "auto_chapters": true
}

// After
{
  "audio_url": "https://example.com/audio.mp3",
  "speech_understanding": {
    "request": {
      "summarization": { "summary_type": "paragraph" }
    }
  }
}
```

  **That's it for the request.** But the response moves too — read the sections
  from `speech_understanding.response.summarization.summary` instead of the
  top-level `chapters` field. Read on for the full migration details.

## What changes

This table covers the key parameter and response field differences. Use it as a migration checklist.

| What                 | Deprecated Auto Chapters                          | Speech Understanding Summarization                                | Action required                                                             |
| -------------------- | ------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **Where it's set**   | Top-level `auto_chapters: true`                    | `speech_understanding.request.summarization`                       | Replace `auto_chapters` with a `summarization` object under `speech_understanding.request` |
| **Request shape**    | Boolean flag                                       | Object holding the summary options                                 | Pass an object; `{}` uses the defaults                                       |
| **`summary_type`**   | Not a parameter                                    | `bullets` or `paragraph` (defaults to `paragraph`)                 | Optional — pick `paragraph` for chapter-style prose or `bullets` for short bullets |
| **`effort`**         | Not supported                                      | `low` (default) or `medium`                                        | New capability (optional) — set `medium` for harder content                  |
| **Response field**   | Top-level `chapters`                               | `speech_understanding.response.summarization.summary`              | Update your response parsing to read from the new path                       |
| **Per-section fields** | `headline`, `gist`, `summary`, `start`, `end`   | `headline`, `text`, `start`, `end`                                 | Read `text` in place of `summary`; drop `gist` (or fall back to `headline`)  |
| **Pricing**          | Bundled with the deprecated model                  | $0.03 per hour of audio                                            | Review your cost model                                                       |

### Mapping the per-chapter fields

| Deprecated field | Use instead | Notes                                                        |
| ---------------- | ----------- | ------------------------------------------------------------ |
| `headline`       | `headline`  | Same field, same purpose                                     |
| `summary`        | `text`      | The longer summary body for the section                      |
| `gist`           | `headline`  | No direct equivalent — use `headline` for the short overview |
| `start` / `end`  | `start` / `end` | Same field, milliseconds                                 |

## Side-by-side code comparison

Below is a side-by-side comparison of generating chapters with the deprecated parameter and with Speech Understanding Summarization:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": audio_url,
    "auto_chapters": True
}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

for chapter in transcription_result['chapters']:
    print(f"{chapter['start']} - {chapter['end']}: {chapter['headline']}")
    print(chapter['summary'])
```

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": audio_url,
    "speech_understanding": {
        "request": {
            "summarization": {
                "summary_type": "paragraph"
            }
        }
    }
}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

summarization = transcription_result["speech_understanding"]["response"]["summarization"]

for topic in summarization["summary"]:
    print(f"{topic['start']} - {topic['end']}: {topic['headline']}")
    print(topic["text"])
```

The same change in JavaScript:

```javascript
const data = {
  audio_url: audioUrl,
  auto_chapters: true,
};

// ... submit and poll until status is "completed"

for (const chapter of transcriptionResult.chapters) {
  console.log(`${chapter.start} - ${chapter.end}: ${chapter.headline}`);
  console.log(chapter.summary);
}
```

```javascript
const data = {
  audio_url: audioUrl,
  speech_understanding: {
    request: {
      summarization: {
        summary_type: "paragraph",
      },
    },
  },
};

// ... submit and poll until status is "completed"

const { summary } = transcriptionResult.speech_understanding.response.summarization;

for (const topic of summary) {
  console.log(`${topic.start} - ${topic.end}: ${topic.headline}`);
  console.log(topic.text);
}
```

## Reading the response

The deprecated parameter returned an array on the top-level `chapters` field. Speech Understanding returns an object on `speech_understanding.response.summarization`, with the sections themselves in the `summary` array:

```plain
{
    "status": "success",
    "summary": [
      {
        "start": 240,
        "end": 37100,
        "text": "Smoke from hundreds of Canadian wildfires is causing air quality alerts across the US, turning skylines gray and prompting warnings to stay inside.",
        "headline": "Smoke from Canadian Wildfires Affects US Air Quality"
      },
      {
        "start": 39100,
        "end": 60670,
        "text": "Professor Peter DeCarlo explains that dry conditions and specific weather systems are channeling smoke from Canadian wildfires into the Mid-Atlantic and Northeast regions of the US.",
        "headline": "Weather Systems Channeling Smoke"
      }
    ],
    "summary_type": "paragraph",
    "effort": "low"
}
```

Things to know when updating your response handling:

- `summary` is the array of sections — one per topic, in order. Treat each element as a chapter.
- `start` and `end` are timestamps in milliseconds, so you can link each section back to the audio.
- `text` replaces the deprecated `summary` field on each chapter. There is no separate `gist` — use `headline` for a short overview.
- Check `status` for `success` before reading the summaries.

## Choosing `summary_type`

Summarization takes a `summary_type` of `bullets` or `paragraph`:

- **`paragraph`** produces prose summaries per section. This is closest to what Auto Chapters returned in its `summary` field.
- **`bullets`** produces short bullet-style summaries per section. Reach for this if you're rendering chapter markers in a compact UI.

Either way, every section still carries its own `headline`, `start`, and `end`.

## Controlling quality with `effort`

The new `effort` parameter has no equivalent in the deprecated parameter. It controls how much processing power goes into the summary:

```json
{
  "audio_url": "https://example.com/audio.mp3",
  "speech_understanding": {
    "request": {
      "summarization": {
        "summary_type": "paragraph",
        "effort": "medium"
      }
    }
  }
}
```

`low` is the default and is the right choice for most use cases. Reach for `medium` when missed details matter — important meetings, multilingual audio, or long files (roughly 1.5 hours and up).

## Pricing

Summarization is **$0.03 per hour of audio**. See [Billing and pricing](/billing-and-pricing) for full rates.

## If you also need action items

Alongside Summarization we shipped [Action Items](/speech-understanding/action-items). Add an `action_items` object to the same `speech_understanding` request to get structured follow-ups from meetings and calls — pass `{}` to use the defaults:

```json
{
  "audio_url": "https://example.com/audio.mp3",
  "speech_understanding": {
    "request": {
      "summarization": { "summary_type": "paragraph" },
      "action_items": {}
    }
  }
}
```

Read the results from `speech_understanding.response.action_items`. Action Items takes the same `effort` parameter as Summarization, plus `include_decisions` to count decisions made in the audio as action items. Action Items is **$0.02 per hour of audio**.

## Next steps

- Read the full [Summarization documentation](/speech-understanding/summarization) for all parameters and options.
- Explore the other [Speech Understanding](/speech-understanding/getting-started) models you can request in the same call.
- Review the [AssemblyAI API Reference](/pre-recorded-audio/api-reference/transcripts/submit) for the complete request and response schema.

---

<a id="migration-guide-summarization-docs-speech-understanding-migration-guides-summarization-mdx"></a>

# Migration guide: Summarization

URL: https://www.assemblyai.com/docs/speech-understanding/migration-guides/summarization
Source: docs/speech-understanding/migration-guides/summarization.mdx
Navigation: Speech Understanding > Guides > Migration guides
Description: Migration guide: Summarization documentation.

This guide walks through the process of upgrading from the deprecated top-level `summarization` parameter to Summarization under Speech Understanding.

  **This guide is for existing accounts using the deprecated `summarization`
  parameter.** If you're adding summaries for the first time, go straight to the
  [Summarization documentation](/speech-understanding/summarization) — none of
  the legacy parameters below apply to you.

We upgraded Summarization rather than replacing it. It now runs on large language models through our [LLM Gateway](/llm-gateway/quickstart), which brings three improvements over the deprecated version:

- **Topic-based summaries instead of one block of text.** You get a set of summaries, each covering a distinct topic, rather than a single string for the whole file.
- **Headlines and timestamps on every summary.** Each topic carries its own headline and `start`/`end` times, so you can link summaries back to the audio — which also replaces the old `headline` and `gist` summary types.
- **Quality you control.** The new `effort` parameter lets you spend more processing on harder content, in place of the fixed `summary_model` choice.

Your existing API key and endpoint stay the same. The changes are to the request shape, the response location, and the response structure.

## Quick upgrade

Move `summarization` into `speech_understanding.request`, drop `summary_model`, and keep `summary_type`:

```json
// Before (deprecated)
{
  "audio_url": "https://example.com/audio.mp3",
  "summarization": true,
  "summary_model": "informative",
  "summary_type": "bullets"
}

// After
{
  "audio_url": "https://example.com/audio.mp3",
  "speech_understanding": {
    "request": {
      "summarization": { "summary_type": "bullets" }
    }
  }
}
```

  **That's it for the request.** But the response moves too — read the summary
  from `speech_understanding.response.summarization` instead of the top-level
  `summary` field, and expect an array of topic summaries rather than a single
  string. Read on for the full migration details.

## What changes

This table covers the key parameter and response field differences. Use it as a migration checklist.

| What                 | Deprecated Summarization                               | Speech Understanding Summarization                                        | Action required                                                             |
| -------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **Where it's set**   | Top-level `summarization: true`                         | `speech_understanding.request.summarization`                               | Nest the parameter under `speech_understanding.request`                      |
| **`summarization`**  | Boolean flag                                            | Object holding the summary options                                         | Replace `true` with an object (`{}` for defaults)                            |
| **`summary_model`**  | `informative`, `conversational`, `catchy`               | Not a parameter — summaries run on LLMs through the LLM Gateway            | Remove `summary_model` from your request                                     |
| **`summary_type`**   | `bullets`, `bullets_verbose`, `gist`, `headline`, `paragraph` | `bullets` or `paragraph`                                              | Map your existing value to `bullets` or `paragraph` (see the table below)    |
| **`effort`**         | Not supported                                           | `low` (default) or `medium`                                                | New capability (optional) — set `medium` for harder content                  |
| **Response field**   | Top-level `summary`                                     | `speech_understanding.response.summarization`                              | Update your response parsing to read from the new path                       |
| **Response shape**   | A single string                                         | An object with `status`, `summary`, `summary_type`, and `effort`           | Iterate over `summary`, an array of topic summaries                          |
| **Per-summary data** | None — one summary for the whole file                   | `headline`, `text`, `start`, and `end` per topic                           | New capability — use headlines and timestamps to link summaries back to audio |
| **Pricing**          | Bundled with the deprecated model                       | $0.03 per hour of audio                                                    | Review your cost model                                                       |

### Mapping `summary_type`

| Deprecated value  | Use instead | Notes                                                        |
| ----------------- | ----------- | ------------------------------------------------------------ |
| `bullets`         | `bullets`   | Short, concise bullet-point summaries per topic               |
| `bullets_verbose` | `paragraph` | `paragraph` gives the longer, more detailed output            |
| `paragraph`       | `paragraph` | Now returned per topic rather than as one block               |
| `gist`            | `bullets`   | Use the per-topic `headline` fields for the shortest overview |
| `headline`        | `bullets`   | Every topic summary now includes its own `headline`           |

## Side-by-side code comparison

Below is a side-by-side comparison of summarizing a pre-recorded audio file with the deprecated parameter and with Speech Understanding:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": audio_url,
    "summarization": True,
    "summary_model": "informative",
    "summary_type": "bullets"
}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

print(transcription_result["summary"])
```

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/wildfires.mp3"

data = {
    "audio_url": audio_url,
    "speech_understanding": {
        "request": {
            "summarization": {
                "summary_type": "bullets"
            }
        }
    }
}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

summarization = transcription_result["speech_understanding"]["response"]["summarization"]

for topic in summarization["summary"]:
    print(f"{topic['start']} - {topic['end']}: {topic['headline']}")
    print(topic["text"])
```

The same change in JavaScript:

```javascript
const data = {
  audio_url: audioUrl,
  summarization: true,
  summary_model: "informative",
  summary_type: "bullets",
};

// ... submit and poll until status is "completed"

console.log(transcriptionResult.summary);
```

```javascript
const data = {
  audio_url: audioUrl,
  speech_understanding: {
    request: {
      summarization: {
        summary_type: "bullets",
      },
    },
  },
};

// ... submit and poll until status is "completed"

const { summary } = transcriptionResult.speech_understanding.response.summarization;

for (const topic of summary) {
  console.log(`${topic.start} - ${topic.end}: ${topic.headline}`);
  console.log(topic.text);
}
```

## Reading the response

The deprecated parameter returned one string on the top-level `summary` field. Speech Understanding returns an object on `speech_understanding.response.summarization`, with the summaries themselves in the `summary` array:

```plain
{
    "status": "success",
    "summary": [
      {
        "start": 240,
        "end": 37100,
        "text": "Smoke from hundreds of Canadian wildfires is causing air quality alerts across the US, turning skylines gray and prompting warnings to stay inside.",
        "headline": "Smoke from Canadian Wildfires Affects US Air Quality"
      },
      {
        "start": 39100,
        "end": 60670,
        "text": "Professor Peter DeCarlo explains that dry conditions and specific weather systems are channeling smoke from Canadian wildfires into the Mid-Atlantic and Northeast regions of the US.",
        "headline": "Weather Systems Channeling Smoke"
      }
    ],
    "summary_type": "bullets",
    "effort": "low"
}
```

Things to know when updating your response handling:

- `summary` is an array of topic-based summaries, not a single string. If you need one block of text, join the `text` fields.
- `start` and `end` are timestamps in milliseconds, so you can link each summary back to the audio.
- Each topic carries its own `headline`, which replaces the deprecated `headline` and `gist` summary types.
- Check `status` for `success` before reading the summaries.

## Controlling quality with `effort`

The new `effort` parameter has no equivalent in the deprecated parameter. It controls how much processing power goes into the summary:

```json
{
  "audio_url": "https://example.com/audio.mp3",
  "speech_understanding": {
    "request": {
      "summarization": {
        "summary_type": "bullets",
        "effort": "medium"
      }
    }
  }
}
```

`low` is the default and is the right choice for most use cases. Reach for `medium` when missed details matter — important meetings, multilingual audio, or long files (roughly 1.5 hours and up).

## Pricing

The upgraded Summarization is **$0.03 per hour of audio**. See [Billing and pricing](/billing-and-pricing) for full rates.

## If you also need action items

Alongside the upgraded Summarization we shipped [Action Items](/speech-understanding/action-items). Add an `action_items` object to the same `speech_understanding` request to get structured follow-ups from meetings and calls — pass `{}` to use the defaults:

```json
{
  "audio_url": "https://example.com/audio.mp3",
  "speech_understanding": {
    "request": {
      "summarization": { "summary_type": "bullets" },
      "action_items": {}
    }
  }
}
```

Read the results from `speech_understanding.response.action_items`. Action Items takes the same `effort` parameter as Summarization, plus `include_decisions` to count decisions made in the audio as action items. Action Items is **$0.02 per hour of audio**.

## Next steps

- Read the full [Summarization documentation](/speech-understanding/summarization) for all parameters and options.
- Explore the other [Speech Understanding](/speech-understanding/getting-started) models you can request in the same call.
- Review the [AssemblyAI API Reference](/pre-recorded-audio/api-reference/transcripts/submit) for the complete request and response schema.

---

<a id="guardrails-docs-guardrails-getting-started-mdx"></a>

# Guardrails

URL: https://www.assemblyai.com/docs/guardrails/getting-started
Source: docs/guardrails/getting-started.mdx
Navigation: Guardrails > Models & features
Description: Protect sensitive content in your transcripts with guardrail features.

Guardrails help you filter, redact, and moderate transcript content to meet compliance and safety requirements.

  
    Automatically detect and redact personally identifiable information from
    transcripts and audio.
  
  
    Detect sensitive content like hate speech, violence, and other topics in
    your audio.
  
  
    Filter profanity from your transcripts by replacing profane words with
    asterisks.
  
  
    Prevent transcription of files with insufficient speech, reducing
    unnecessary processing.

---

<a id="pii-redaction-docs-guardrails-redact-pii-from-transcripts-mdx"></a>

# PII Redaction

URL: https://www.assemblyai.com/docs/guardrails/redact-pii-from-transcripts
Source: docs/guardrails/redact-pii-from-transcripts.mdx
Navigation: Guardrails > Models & features
Description: Redact PII that is spoken in your audio

<br />

  
  <br />

US & EU

The PII Redaction model lets you minimize sensitive information about individuals by automatically identifying and removing it from your transcript.

Personal Identifiable Information (PII) is any information that can be used to identify a person, such as a name, email address, or phone number. To redact PII from text rather than audio, see [Redact PII from Text Using LLM Gateway](/guides/llm-gateway-pii-redaction).

When you enable the PII Redaction model, your transcript will look like this:

- With `hash` substitution: `Hi, my name is ####!`
- With `entity_name` substitution: `Hi, my name is [PERSON_NAME]!`

You can also [Create redacted audio files](#create-redacted-audio-files) to replace sensitive information in the audio with a beeping sound or silence.

  **Redacted Properties**

  PII only redacts words in the `text` property. Properties from other features
  may still include PII, such as `entities` from [Entity
  Detection](/speech-understanding/entity-detection) or `summary` from
  [Summarization](/speech-understanding/summarization).

## Quickstart

Enable Topic Detection by setting `redact_pii` to `True` in the JSON payload.

Set `redact_pii_policies` to specify the information you want to redact. For the full list of policies, see [PII policies](#pii-policies).

Set `redact_pii_sub` to specify the replacement for redacted information.

```python {19-21} expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./local_file.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "organization", "occupation"],
    "redact_pii_sub": "hash"
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

print(f"Transcript ID: {transcript_id}")

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
        print(transcription_result['text'])
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

  

  Enable PII Redaction on the `TranscriptionConfig` using the `set_redact_pii()`
method.

Set `policies` to specify the information you want to redact. For the full list of policies, see [PII policies](#pii-policies).

```python {8-15} expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
).set_redact_pii(
    policies=[
        aai.PIIRedactionPolicy.person_name,
        aai.PIIRedactionPolicy.organization,
        aai.PIIRedactionPolicy.occupation,
    ],
    substitution=aai.PIISubstitutionPolicy.hash,
)

transcript = aai.Transcriber().transcribe(audio_file, config)
print(f"Transcript ID: {transcript.id}")

print(transcript.text)
```

  

    Enable Topic Detection by setting `redact_pii` to `true` in the JSON payload.

Set `redact_pii_policies` to specify the information you want to redact. For the full list of policies, see [PII policies](#pii-policies).

Set `redact_pii_sub` to specify the replacement for redacted information.

```javascript {19-21} expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  redact_pii: true,
  redact_pii_policies: ["person_name", "organization", "occupation"],
  redact_pii_sub: "hash",
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
console.log("Transcript ID: ", transcriptId);

const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

Enable PII Redaction by setting `redact_pii` to `true` in the transcription
config.

Use `redact_pii_policies` to specify the information you want to
redact. For the full list of policies, see [PII policies](#pii-policies).

```javascript {12-14} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  redact_pii: true,
  redact_pii_policies: ["person_name", "organization", "occupation"],
  redact_pii_sub: "hash",
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

  

### Example output

```plain
Smoke from hundreds of wildfires in Canada is triggering air quality alerts
throughout the US. Skylines from Maine to Maryland to Minnesota are gray and
smoggy. And in some places, the air quality warnings include the warning to stay
inside. We wanted to better understand what's happening here and why, so we
called ##### #######, an ######### ######### in the ########## ## #############
###### ### ########### at ##### ####### ##########. Good morning, #########.
Good morning. So what is it about the conditions right now that have caused this
round of wildfires to affect so many people so far away? Well, there's a couple
of things. The season has been pretty dry already, and then the fact that we're
getting hit in the US. Is because there's a couple of weather systems that ...
```

  **PII Redaction Using LLM Gateway**

  If you would like the option to use LLM Gateway for custom PII redaction,
  check out this guide [Redact PII from Text Using LLM
  Gateway](/guides/llm-gateway-pii-redaction).

## Create redacted audio files

In addition to redacting sensitive information from the transcription text, you can also generate a version of the original audio file with the PII "beeped" out. You can optionally use silence instead of a beep by setting `override_audio_redaction_method` to `"silence"` within `redact_pii_audio_options`.

  **Redacted Audio Endpoint**

Retrieve the redacted audio file using the `transcript_id` for a transcript where `redact_pii_audio` was enabled during submission:

View the endpoint reference.

Redacted audio files are only available for 24 hours. Make sure to download the file within this time frame.

    To create a redacted version of the audio file, set `redact_pii_audio` to `True` on the JSON payload.
    Set `redact_pii_audio_quality` to specify the quality of the redacted audio file.

Use the transcript ID to poll the [GET redacted audio endpoint](/api-reference/transcripts/get-redacted-audio) every few seconds to check the status of the redacted audio. Once the status is `redacted_audio_ready`, you can retrieve the audio URL from the API response.

```python {6-10,14-25} expandable
data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "organization", "occupation"],
    "redact_pii_sub": "hash",
    "redact_pii_audio": True,
    "redact_pii_audio_quality": "wav", # Optional. Defaults to "mp3"
    "redact_pii_audio_options": {
        "override_audio_redaction_method": "silence" # Optional. Omit for default (beep)
    }
}

# ...
redacted_audio_polling_endpoint = base_url + "/v2/transcript/" + transcript_id + "/redacted-audio"

while True:
    redacted_audio_result = requests.get(redacted_audio_polling_endpoint, headers=headers).json()

    if redacted_audio_result['status'] == 'redacted_audio_ready':
        print(redacted_audio_result['redacted_audio_url'])
        break
    elif redacted_audio_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {redacted_audio_result['error']}")
    else:
        time.sleep(3)
```

  

    To create a redacted version of the audio file, use the `set_redact_pii()` method on the `TranscriptionConfig` with `redact_audio` to `True`.

Use `get_redacted_audio_url()` on the transcript to get the URL to the redacted audio file.

```python {7-8,13}
config = aai.TranscriptionConfig().set_redact_pii(
    policies=[
        aai.PIIRedactionPolicy.person_name,
        aai.PIIRedactionPolicy.organization,
        aai.PIIRedactionPolicy.occupation,
    ],
    redact_audio=True,
    redact_audio_options={"override_audio_redaction_method": "silence"}  # Optional. Omit for default (beep)
)

transcript = aai.Transcriber().transcribe(audio_url, config)

print(transcript.get_redacted_audio_url())
```

  

    To create a redacted version of the audio file, set `redact_pii_audio` to `true` on the JSON payload.
    Set `redact_pii_audio_quality` to specify the quality of the redacted audio file.

Use the transcript ID to poll the [GET redacted audio endpoint](/api-reference/transcripts/get-redacted-audio) every few seconds to check the status of the redacted audio. Once the status is `redacted_audio_ready`, you can retrieve the audio URL from the API response.

```javascript {6-10,14,17-29} expandable
const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  redact_pii: true,
  redact_pii_policies: ["person_name", "organization", "occupation"],
  redact_pii_sub: "hash",
  redact_pii_audio: true,
  redact_pii_audio_quality: "wav", // Optional. Defaults to "mp3"
  redact_pii_audio_options: {
    override_audio_redaction_method: "silence", // Optional. Omit for default (beep)
  },
};

// ...
const redactedAudioPollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}/redacted-audio`;

while (true) {
  let res = await fetch(redactedAudioPollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const redactedAudioResult = await res.json();

  if (redactedAudioResult.status === "redacted_audio_ready") {
    console.log(redactedAudioResult.redacted_audio_url);
    break;
  } else if (redactedAudioResult.status === "error") {
    throw new Error(`Transcription failed: ${redactedAudioResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

To create a redacted version of the audio file, set `redact_pii_audio` to
`true` in the transcription config. Use `redact_pii_audio_quality` to specify
the quality of the redacted audio file.

Use redactedAudio() on the transcript to get the URL to the redacted audio file.

```javascript {5-9, 15-17} expandable
const params = {
  audio: audioFile,
  redact_pii: true,
  redact_pii_policies: ["person_name", "organization", "occupation"],
  redact_pii_audio: true,
  redact_pii_audio_quality: "wav", // Optional. Defaults to "mp3"
  redact_pii_audio_options: {
    override_audio_redaction_method: "silence", // Optional. Omit for default (beep)
  },
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  const { status, redacted_audio_url } = await client.transcripts.redactedAudio(
    transcript.id
  );

  console.log(`Status: ${status}, Redacted audio URL: ${redacted_audio_url}`);
};

run();
```

You can also retrieve the redacted audio file itself using `redactedAudioFile()`.
The following code writes the redacted audio file to a local file, using `writeFile()` from Node.js.

```javascript
import fs from "fs/promises";

...

const audioFile = await client.transcripts.redactedAudioFile(transcript.id);
await fs.writeFile('./redacted-audio.wav', audioFile.body, 'binary');
```

  

  **Maximum Audio File Size**

  You can only create redacted versions of audio files if the original file is
  smaller than 1 GB. For files over 1 GB, see the [Downsampling guide](/pre-recorded-audio/guides/downsampling) to reduce file size before redaction.

  **Redacted Audio for Silent Files**

By default, audio redaction provides redacted audio URLs only when speech is detected. However, if your use-case specifically requires redacted audio files even for silent audio files without any dialogue, you can now opt to receive these URLs. Enable this by setting the optional parameter `"return_redacted_no_speech_audio": true` within `redact_pii_audio_options` in your `POST` request body.

```json
{
    "audio_url": "YOUR_AUDIO_URL",
    "redact_pii": true,
    "redact_pii_audio": true,
    "redact_pii_audio_options": {
          "return_redacted_no_speech_audio": true
     },
     "redact_pii_policies": ["credit_card_number"]
}
```

### Example output

```plain
https://s3.us-west-2.amazonaws.com/api.assembly.ai.usw2/redacted-audio/ac06721c-d1ea-41a7-95f7-a9463421e6b1.mp3?AWSAccessKeyId=...
```

## Return the unredacted transcript

If your workflow needs both the redacted and unredacted transcripts, you can request both in a single transcription call by setting `redact_pii_return_unredacted` to `true`. This avoids the need to send a second API request without redaction.

  **This is an opt-in convenience feature**

  Enabling PII Redaction by itself does **not** return the unredacted transcript — the default behavior is to redact `text`, `words`, and `utterances` and return only the redacted versions. The unredacted transcript is only returned when you explicitly set `redact_pii_return_unredacted` to `true`, and `redact_pii` must also be `true` (otherwise the request is rejected with a 400).

  Only use this feature if your workflow specifically requires access to both the redacted and unredacted transcripts from the same request. If you only need the redacted transcript, leave this parameter off. When `redact_pii_return_unredacted` is `false` or omitted, the three `unredacted_*` fields are not included in the response at all.

When `redact_pii_return_unredacted` is `true`, the response includes three additional fields alongside their redacted counterparts:

| Field                   | Type                | Description                                                                       |
| ----------------------- | ------------------- | --------------------------------------------------------------------------------- |
| `unredacted_text`       | string              | The original transcript text before PII redaction was applied.                    |
| `unredacted_words`      | array of Word       | The original word objects before redaction. Same shape as `words`.                |
| `unredacted_utterances` | array of Utterance  | The original utterance objects before redaction. Same shape as `utterances`.      |

Set `redact_pii_return_unredacted` to `True` alongside the existing PII parameters. The completed transcript will include `unredacted_text`, `unredacted_words`, and `unredacted_utterances` in addition to the redacted versions.

```python {6}
data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "phone_number", "email_address"],
    "redact_pii_sub": "entity_name",
    "redact_pii_return_unredacted": True
}

# ...
# After polling completes:
print("Redacted:  ", transcription_result["text"])
print("Original:  ", transcription_result["unredacted_text"])
```

Pass `return_unredacted=True` to `set_redact_pii()`. The transcript object will include `unredacted_text`, `unredacted_words`, and `unredacted_utterances`.

```python {11}
config = aai.TranscriptionConfig(
    language_detection=True,
).set_redact_pii(
    policies=[
        aai.PIIRedactionPolicy.person_name,
        aai.PIIRedactionPolicy.phone_number,
        aai.PIIRedactionPolicy.email_address,
    ],
    substitution=aai.PIISubstitutionPolicy.entity_name,
    return_unredacted=True,
)

transcript = aai.Transcriber().transcribe(audio_file, config)

print("Redacted:  ", transcript.text)
print("Original:  ", transcript.unredacted_text)
```

Set `redact_pii_return_unredacted` to `true` alongside the existing PII parameters. The completed transcript will include `unredacted_text`, `unredacted_words`, and `unredacted_utterances` in addition to the redacted versions.

```javascript {6}
const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  redact_pii: true,
  redact_pii_policies: ["person_name", "phone_number", "email_address"],
  redact_pii_sub: "entity_name",
  redact_pii_return_unredacted: true,
};

// ...
// After polling completes:
console.log("Redacted:  ", transcriptionResult.text);
console.log("Original:  ", transcriptionResult.unredacted_text);
```

Set `redact_pii_return_unredacted` to `true` in the transcription config. The transcript object will include `unredacted_text`, `unredacted_words`, and `unredacted_utterances`.

```javascript {8}
const params = {
  audio: audioFile,
  language_detection: true,
  redact_pii: true,
  redact_pii_policies: ["person_name", "phone_number", "email_address"],
  redact_pii_sub: "entity_name",
  redact_pii_return_unredacted: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log("Redacted:  ", transcript.text);
  console.log("Original:  ", transcript.unredacted_text);
};

run();
```

### Example response

```json
{
  "text": "[PERSON_NAME] called from [PHONE_NUMBER]...",
  "words": [{ "text": "[PERSON_NAME]", "start": 250, "end": 650, "confidence": 0.98 }],
  "utterances": [{ "text": "[PERSON_NAME] called from [PHONE_NUMBER]...", "start": 250, "end": 4820, "speaker": "A" }],
  "redact_pii_return_unredacted": true,
  "unredacted_text": "Mary called from 555-0123...",
  "unredacted_words": [{ "text": "Mary", "start": 250, "end": 650, "confidence": 0.98 }],
  "unredacted_utterances": [{ "text": "Mary called from 555-0123...", "start": 250, "end": 4820, "speaker": "A" }]
}
```

## Static Entity Redaction

PII Redaction detects broad categories of sensitive information (names, phone numbers, and the other [PII policies](#pii-policies)) using an AI model. **Static Entity Redaction** complements this for cases where you need to redact specific, predefined terms that may not fit into general PII categories — for example a proprietary product name, an internal project codename, or an org-specific identifier.

You supply a map of `redact_static_entities`, where each key is a label and each value is a list of **exact terms** to redact. Any matching term in the transcript is replaced with that label, on top of the standard PII redaction.

  Static Entity Redaction matches the exact terms you provide (tolerant of casing, surrounding punctuation, and minor spacing/hyphenation variation from the transcription), rather than inferring a category. For example, listing `["Bearclaw"]` redacts `Bearclaw`, `bearclaw`, `Bearclaw's`, `Bear-claw`, and `Bear claw` — but a different internal tool name you didn't list is not redacted. You define and manage the term list.

  `redact_static_entities` requires `redact_pii` to be `true`. It runs as the final step of redaction, after the standard PII pass.

The substitution follows your existing `redact_pii_sub` setting:

- `entity_name` — the matched term is replaced with `[LABEL]` (the uppercased map key).
- `hash` — the matched term's characters are replaced with `#`.

When [redacted audio](#create-redacted-audio-files) is enabled (`redact_pii_audio: true`), the matched terms are also bleeped from the audio output alongside standard PII.

Pass `redact_static_entities` alongside your existing PII parameters. Each key is the redaction label; each value is the list of terms to match.

```python {6-9}
data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "phone_number"],
    "redact_pii_sub": "entity_name",
    "redact_static_entities": {
        "INTERNAL_TOOL": ["Bearclaw", "Cubclaw"],
        "PROJECT": ["Universal-3.5 Pro"]
    }
}
```

Pass `redact_static_entities` alongside your existing PII parameters. Each key is the redaction label; each value is the list of terms to match.

```javascript {6-9}
const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  redact_pii: true,
  redact_pii_policies: ["person_name", "phone_number"],
  redact_pii_sub: "entity_name",
  redact_static_entities: {
    INTERNAL_TOOL: ["Bearclaw", "Cubclaw"],
    PROJECT: ["Universal-3.5 Pro"],
  },
};
```

### Example response

Matched terms appear redacted inline in `text`, `words`, and `utterances`, exactly like standard PII. There are no additional response fields. For the transcript _"Our internal tool Bearclaw is powered by Universal-3.5 Pro."_ with the request above:

```json
{
  "text": "Our internal tool [INTERNAL_TOOL] is powered by [PROJECT].",
  "words": [{ "text": "[INTERNAL_TOOL]", "start": 820, "end": 1180, "confidence": 0.98 }],
  "utterances": [{ "text": "Our internal tool [INTERNAL_TOOL] is powered by [PROJECT].", "start": 250, "end": 3120, "speaker": "A" }]
}
```

## API reference

### Request

```bash {6-13}
curl https://api.assemblyai.com/v2/transcript \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "audio_url": "YOUR_AUDIO_URL",
  "redact_pii": true,
  "redact_pii_policies": ["us_social_security_number", "credit_card_number"],
  "redact_pii_sub": "hash",
  "redact_pii_audio": true,
  "redact_pii_audio_quality": "mp3",
  "redact_pii_audio_options": {
    "override_audio_redaction_method": "silence"
  }
}'
```

| Key                        | Type    | Description                                                                      |
| -------------------------- | ------- | -------------------------------------------------------------------------------- |
| `redact_pii`               | boolean | Enable PII Redaction.                                                            |
| `redact_pii_policies`      | array   | [PII policies](#pii-policies) for what information to redact.                    |
| `redact_pii_sub`           | string  | Method used to substitute PII in the transcript. Can be `entity_name` or `hash`. |
| `redact_pii_audio`         | boolean | Create a [redacted version of the audio file](#create-redacted-audio-files).     |
| `redact_pii_audio_quality` | string  | Quality of the redacted PII audio file. Can be `mp3` or `wav`.                   |
| `redact_pii_audio_options` | object  | Options for PII-redacted audio. See [Create redacted audio files](#create-redacted-audio-files). |
| `redact_pii_audio_options.override_audio_redaction_method` | string | The method used to redact audio. Set to `silence` to replace PII with silence instead of the default beep. |
| `redact_pii_return_unredacted` | boolean | Opt-in. When `true`, [returns the unredacted transcript](#return-the-unredacted-transcript) alongside the redacted one. Requires `redact_pii: true`. Defaults to `false`, in which case only the redacted transcript is returned. |
| `redact_static_entities` | object | A map of `{ "LABEL": ["term", ...] }` for [Static Entity Redaction](#static-entity-redaction). Each listed term is redacted (using `redact_pii_sub`) wherever it appears, on top of standard PII. Requires `redact_pii: true`. |

### Response

| Key    | Type   | Description                   |
| ------ | ------ | ----------------------------- |
| `text` | string | Transcript with redacted PII. |

The response also includes the request parameters used to generate the transcript.

### Request for Redacted Audio

In the request URL, replace transcript_id with the ID of the transcript where `redact_pii_audio` is set to `true`.

```bash {6-10}
curl https://api.assemblyai.com/v2/transcript/transcript_id/redacted-audio \
--header "Authorization: <YOUR_API_KEY>"
```

### Response for Redacted Audio

| Key                  | Type   | Description                         |
| -------------------- | ------ | ----------------------------------- |
| `status`             | string | The status of the redacted audio.   |
| `redacted_audio_url` | string | The URL of the redacted audio file. |

### PII policies

| Policy name                 | Description                                                                                                    | Example                                                          |
| --------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `account_number`            | Customer account or membership identification number                                                           | `Policy No. 10042992; Member ID: HZ-5235-001`                    |
| `banking_information`       | Banking information, including account and routing numbers                                                     |                                                                  |
| `blood_type`                | Blood type                                                                                                     | O-, AB positive                                                  |
| `credit_card_cvv`           | Credit card verification code                                                                                  | CVV: 080                                                         |
| `credit_card_expiration`    | Expiration date of a credit card                                                                               |                                                                  |
| `credit_card_number`        | Credit card number                                                                                             |                                                                  |
| `date`                      | Specific calendar date                                                                                         | December 18                                                      |
| `date_interval`             | Broader time periods, including date ranges, months, seasons, years, and decades                               | 2020-2021, 5-9 May, January 1984                                 |
| `date_of_birth`             | Date of birth                                                                                                  | Date of Birth: March 7, 1961                                     |
| `drivers_license`           | Driver's license number                                                                                        | DL# 356933-540                                                   |
| `drug`                      | Medications, vitamins, or supplements                                                                          | Advil, Acetaminophen, Panadol                                    |
| `duration`                  | Measurements of time expressed as a numerical value plus a unit                                                | 8 months, 2 years                                                |
| `email_address`             | Email address                                                                                                  | support@assemblyai.com                                           |
| `event`                     | Name of an event or holiday                                                                                    | Olympics, Yom Kippur                                             |
| `filename`                  | Names of computer files, including the extension or filepath                                                   | Taxes/2012/brad-tax-returns.pdf                                  |
| `gender_sexuality`          | Terms indicating gender identity or sexual orientation, including slang terms                                  | female, bisexual, trans                                          |
| `healthcare_number`         | Healthcare numbers and health plan beneficiary numbers                                                         | Policy No.: 5584-486-674-YM                                      |
| `injury`                    | Bodily injury                                                                                                  | I broke my arm, I have a sprained wrist                          |
| `ip_address`                | Internet IP address, including IPv4 and IPv6 formats                                                           | 192.168.0.1                                                      |
| `language`                  | Name of a natural language                                                                                     | Spanish, French                                                  |
| `location`                  | Any Location reference including mailing address, postal code, city, state, province, country, or coordinates. Also required to redact combined location phrases like `Toronto, Canada` — see [Granular location policies](#granular-location-policies) below. | Lake Victoria, 145 Windsor St., 90210                            |
| `location_address`          | A complete mailing address                                                                                     | 145 Windsor St., Toronto, ON M5A 2P5                             |
| `location_address_street`   | Street-level component of an address                                                                           | 145 Windsor St.                                                  |
| `location_city`             | City name                                                                                                      | Toronto, San Francisco                                           |
| `location_coordinate`       | Geographic coordinates                                                                                         | 40.7128° N, 74.0060° W                                           |
| `location_country`          | Country name                                                                                                   | Canada, United States                                            |
| `location_state`            | State, province, or region name                                                                                | Ontario, California                                              |
| `location_zip`              | Postal or ZIP code                                                                                             | M5A 2P5, 90210                                                   |
| `marital_status`            | Terms indicating marital status                                                                                | Single, common-law, ex-wife, married                             |
| `medical_condition`         | Name of a medical condition, disease, syndrome, deficit, or disorder                                           | chronic fatigue syndrome, arrhythmia, depression                 |
| `medical_process`           | Medical process, including treatments, procedures, and tests                                                   | heart surgery, CT scan                                           |
| `money_amount`              | Name and/or amount of currency                                                                                 | 15 pesos, $94.50                                                 |
| `nationality`               | Terms indicating nationality, ethnicity, or race                                                               | American, Asian, Caucasian                                       |
| `number_sequence`           | Numerical PII (including alphanumeric strings) that doesn't fall under other categories                        |                                                                  |
| `occupation`                | Job title or profession                                                                                        | professor, actors, engineer, CPA                                 |
| `organization`              | Name of an organization                                                                                        | CNN, McDonalds, University of Alaska, Northwest General Hospital |
| `passport_number`           | Passport numbers, issued by any country                                                                        | PA4568332, NU3C6L86S12                                           |
| `password`                  | Account passwords, PINs, access keys, or verification answers                                                  | 27%alfalfa, temp1234, My mother's maiden name is Smith           |
| `person_age`                | Number associated with an age                                                                                  | 27, 75                                                           |
| `person_name`               | Name of a person                                                                                               | Bob, Doug Jones, Dr. Kay Martinez, MD                            |
| `phone_number`              | Telephone or fax number                                                                                        |                                                                  |
| `physical_attribute`        | Distinctive bodily attributes, including race                                                                  | I'm 190cm tall                                                   |
| `political_affiliation`     | Terms referring to a political party, movement, or ideology                                                    | Republican, Liberal                                              |
| `religion`                  | Terms indicating religious affiliation                                                                         | Hindu, Catholic                                                  |
| `statistics`                | Medical statistics                                                                                             | 18%, 18 percent                                                  |
| `time`                      | Expressions indicating clock times                                                                             | 19:37:28, 10pm EST                                               |
| `url`                       | Internet addresses                                                                                             | https://www.assemblyai.com/                                      |
| `us_social_security_number` | Social Security Number or equivalent                                                                           |                                                                  |
| `username`                  | Usernames, login names, or handles                                                                             | @AssemblyAI                                                      |
| `vehicle_id`                | Vehicle identification numbers (VINs), vehicle serial numbers, and license plate numbers                       | 5FNRL38918B111818, BIF7547                                       |
| `zodiac_sign`               | Names of Zodiac signs                                                                                          | Aries, Taurus                                                    |

#### Granular location policies

The `location_*` policies let you redact specific parts of a location without redacting the rest — for example, redacting street addresses while keeping city and state visible.

Which policy matches a given span is **shape-dependent**:

* A **full contiguous mailing address** (street, city, state, and ZIP together, e.g. `145 Windsor St., Toronto, ON M5A 2P5`) is treated as a single `location_address` span. To redact it, include `location_address` in `redact_pii_policies`.
* A **standalone location fragment** is tagged as its specific subtype. A lone street redacts as `location_address_street`, a lone city as `location_city`, a lone state as `location_state`, a lone country as `location_country`, a lone ZIP as `location_zip`, and coordinates as `location_coordinate`.
* A **combined phrase** that mixes location parts without forming a full address — for example `Toronto, Canada` — is treated as a single generic `location` span. To redact it, include the broad `location` policy in `redact_pii_policies`. The granular subtypes alone won't match it.

  These granular subtypes apply to **PII redaction only**. [Entity Detection](/speech-understanding/entity-detection) returns all location references under the single `location` entity type, not as `location_address`, `location_city`, and so on.

## Troubleshooting

  Make sure that at least one [PII policy](#pii-policies) has been specified in
  your request, using the `redact_pii_policies` parameter. If you're still
  experiencing issues, please reach out to our support team for assistance.

  There could be several reasons why your webhook isn't being sent, such as a
  misconfigured URL, an unreachable endpoint, or an issue with the
  authentication headers. Double-check your request and ensure that the{" "}
  `webhook_url` parameter is included with a valid URL that can be reached by
  AssemblyAI's API. If you're using custom authentication headers, ensure that
  the `webhook_auth_header_name` and `webhook_auth_header_value` parameters are
  included and are correct. If you're still having issues, please contact our
  support team for assistance.

  By default, the API returns redacted audio files in MP3 format, a lossy
  format. Lossy formats remove audio information to reduce file size, which may
  cause a reduction in quality. The difference may be particularly noticeable if
  the submitted audio is in a lossless file format. To retain as much quality as
  possible, you can instead return your redacted audio files in a lossless
  format, by setting `redact_pii_audio_quality` to `wav`.

---

<a id="content-moderation-docs-guardrails-detect-sensitive-content-mdx"></a>

# Content Moderation

URL: https://www.assemblyai.com/docs/guardrails/detect-sensitive-content
Source: docs/guardrails/detect-sensitive-content.mdx
Navigation: Guardrails > Models & features
Description: Detect sensitive content in your audio files

<br />

  
  <br />

  US & EU <br />

The Content Moderation model lets you detect inappropriate content in audio files to ensure that your content is safe for all audiences.

The model pinpoints sensitive discussions in spoken data and their severity.

## Quickstart

  
  Enable Content Moderation by setting `content_safety` to `True` in the JSON payload.

```python {19} expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./local_file.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "content_safety": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

print(f"Transcript ID: {transcript_id}")

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':

        for result in transcription_result['content_safety_labels']['results']:
            print(result['text'])
            print(f"Timestamp: {result['timestamp']['start']} - {result['timestamp']['end']}")

            # Get category, confidence, and severity.
            for label in result['labels']:
                print(f"{label['label']} - {label['confidence']} - {label['severity']}")  # content safety category

        # Get the confidence of the most common labels in relation to the entire audio file.
        for label, confidence in transcription_result['content_safety_labels']['summary'].items():
            print(f"{confidence * 100}% confident that the audio contains {label}")

        # Get the overall severity of the most common labels in relation to the entire audio file.
        for label, severity_confidence in transcription_result['content_safety_labels']['severity_score_summary'].items():
            print(f"{severity_confidence['low'] * 100}% confident that the audio contains low-severity {label}")
            print(f"{severity_confidence['medium'] * 100}% confident that the audio contains medium-severity {label}")
            print(f"{severity_confidence['high'] * 100}% confident that the audio contains high-severity {label}")
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

  

  
  Enable Content Moderation by setting `content_safety` to `True` in the transcription config.

```python {8} expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    content_safety=True
)

transcript = aai.Transcriber().transcribe(audio_file, config)

print(f"Transcript ID: {transcript.id}")

for result in transcript.content_safety.results:
    print(result.text)
    print(f"Timestamp: {result.timestamp.start} - {result.timestamp.end}")

    # Get category, confidence, and severity.
    for label in result.labels:
        print(f"{label.label} - {label.confidence} - {label.severity}")  # content safety category

# Get the confidence of the most common labels in relation to the entire audio file.
for label, confidence in transcript.content_safety.summary.items():
    print(f"{confidence * 100}% confident that the audio contains {label}")

# Get the overall severity of the most common labels in relation to the entire audio file.
for label, severity_confidence in transcript.content_safety.severity_score_summary.items():
    print(f"{severity_confidence.low * 100}% confident that the audio contains low-severity {label}")
    print(f"{severity_confidence.medium * 100}% confident that the audio contains medium-severity {label}")
    print(f"{severity_confidence.high * 100}% confident that the audio contains high-severity {label}")
```

  

  
  Enable Content Moderation by setting `content_safety` to `true` in the JSON payload.

```javascript {19} expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  content_safety: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
console.log("Transcript ID: ", transcriptId);

const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    const contentSafetyLabels = transcriptionResult.content_safety_labels;
    for (const result of contentSafetyLabels.results) {
      console.log(result.text);
      console.log(
        `Timestamp: ${result.timestamp.start} - ${result.timestamp.end}`
      );

      // Get category, confidence, and severity.
      for (const label of result.labels) {
        console.log(`${label.label} - ${label.confidence} - ${label.severity}`); // content safety category
      }
    }
    // Get the confidence of the most common labels in relation to the entire audio file
    for (const [label, confidence] of Object.entries(
      contentSafetyLabels.summary
    )) {
      console.log(
        `${confidence * 100}% confident that the audio contains ${label}`
      );
    }
    // Get the confidence of the most common labels in relation to the entire audio file.
    for (const [label, severity_confidence] of Object.entries(
      contentSafetyLabels.severity_score_summary
    )) {
      console.log(
        `${severity_confidence.low * 100}% confident that the audio contains low-severity ${label}`
      );
      console.log(
        `${severity_confidence.medium * 100}% confident that the audio contains medium-severity ${label}`
      );
      console.log(
        `${severity_confidence.high * 100}% confident that the audio contains high-severity ${label}`
      );
    }

    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

  
  Enable Content Moderation by setting `content_safety` to `true` in the transcription config.

```javascript {12} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  content_safety: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log(`Transcript ID: ${transcript.id}`);
  const contentSafetyLabels = transcript.content_safety_labels;

  // Get the parts of the transcript which were flagged as sensitive
  for (const result of contentSafetyLabels.results) {
    console.log(result.text);
    console.log(
      `Timestamp: ${result.timestamp.start} - ${result.timestamp.end}`
    );

    // Get category, confidence, and severity
    for (const label of result.labels) {
      console.log(`${label.label} - ${label.confidence} - ${label.severity}`);
    }
  }

  // Get the confidence of the most common labels in relation to the entire audio file
  for (const [label, confidence] of Object.entries(
    contentSafetyLabels.summary
  )) {
    console.log(
      `${confidence * 100}% confident that the audio contains ${label}`
    );
  }

  // Get the overall severity of the most common labels in relation to the entire audio file
  for (const [label, severity_confidence] of Object.entries(
    contentSafetyLabels.severity_score_summary
  )) {
    console.log(
      `${severity_confidence.low * 100}% confident that the audio contains low-severity ${label}`
    );
    console.log(
      `${severity_confidence.medium * 100}% confident that the audio contains medium-severity ${label}`
    );
    console.log(
      `${severity_confidence.high * 100}% confident that the audio contains high-severity ${label}`
    );
  }
};
run();
```

  

### Example output

```plain expandable
Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines...
Timestamp: 250 - 28920
disasters - 0.8141 - 0.4014

So what is it about the conditions right now that have caused this round of wildfires to...
Timestamp: 29290 - 56190
disasters - 0.9217 - 0.5665

So what is it in this haze that makes it harmful? And I'm assuming it is...
Timestamp: 56340 - 88034
health_issues - 0.9358 - 0.8906

...

99.42% confident that the audio contains disasters
92.70% confident that the audio contains health_issues

57.43% confident that the audio contains low-severity disasters
42.56% confident that the audio contains mid-severity disasters
0.0% confident that the audio contains high-severity disasters
23.57% confident that the audio contains low-severity health_issues
30.22% confident that the audio contains mid-severity health_issues
46.19% confident that the audio contains high-severity health_issues
```

## Adjust the confidence threshold

The confidence threshold determines how likely something is to be flagged as inappropriate content. A threshold of 50% (which is the default) means any label with a confidence score of 50% or greater is flagged.

  
  To adjust the confidence threshold for your transcription, include `content_safety_confidence` in the JSON payload.

```python {5}
# Setting the content safety confidence threshold to 60%.
data = {
    "audio_url": upload_url,
    "content_safety": True,
    "content_safety_confidence": 60
}
```

  

  
  To adjust the confidence threshold for your transcription, include `content_safety_confidence` in the transcription config.

```python {4}
# Setting the content safety confidence threshold to 60%.
config = aai.TranscriptionConfig(
  content_safety=True,
  content_safety_confidence=60
)
```

  

  
  To adjust the confidence threshold for your transcription, include `content_safety_confidence` in the JSON payload.

```javascript {5}
// Setting the content safety confidence threshold to 60%.
const data = {
  audio_url: uploadUrl,
  content_safety: true,
  content_safety_confidence: 60,
};
```

  

  
  To adjust the confidence threshold for your transcription, include `content_safety_confidence` in the transcription config.

```javascript {5}
// Setting the content safety confidence threshold to 60%.
const params = {
  audio: audioFile,
  content_safety: true,
  content_safety_confidence: 60,
};
```

  

## API reference

### Request

```bash {6-7}
curl https://api.assemblyai.com/v2/transcript \
--header "Authorization: <YOUR_API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "audio_url": "YOUR_AUDIO_URL",
  "content_safety": true,
  "content_safety_confidence": 60
}'
```

| Key                         | Type    | Description                                                                         |
| --------------------------- | ------- | ----------------------------------------------------------------------------------- |
| `content_safety`            | boolean | Enable Content Moderation.                                                          |
| `content_safety_confidence` | integer | The confidence threshold for content moderation. Values must be between 25 and 100. |

### Response

| Key                                                                      | Type   | Description                                                                                                                 |
| ------------------------------------------------------------------------ | ------ | --------------------------------------------------------------------------------------------------------------------------- |
| `content_safety_labels`                                                  | object | An object containing all results of the Content Moderation model.                                                           |
| `content_safety_labels.status`                                           | string | Is either `success`, or `unavailable` in the rare case that the Content Moderation model failed.                            |
| `content_safety_labels.results`                                          | array  | An array of objects, one for each section in the audio file, that the Content Moderation file flagged.                      |
| `content_safety_labels.results[i].text`                                  | string | The transcript of the i-th section flagged by the Content Moderation model.                                                 |
| `content_safety_labels.results[i].labels`                                | array  | An array of objects, one per sensitive topic, that was detected in the i-th section.                                        |
| `content_safety_labels.results[i].labels[j].label`                       | string | The label of the sensitive topic.                                                                                           |
| `content_safety_labels.results[i].labels[j].confidence`                  | number | The confidence score for the j-th topic being discussed in the i-th section, from 0 to 1.                                   |
| `content_safety_labels.results[i].labels[j].severity`                    | number | How severely the j-th topic is discussed in the i-th section, from 0 to 1.                                                  |
| `content_safety_labels.results[i].sentences_idx_start`                   | number | The sentence index at which the i-th section begins.                                                                        |
| `content_safety_labels.results[i].sentences_idx_end`                     | number | The sentence index at which the i-th section ends.                                                                          |
| `content_safety_labels.results[i].timestamp`                             | object | Timestamp information for the i-th section.                                                                                 |
| `content_safety_labels.results[i].timestamp.start`                       | number | The time, in milliseconds, at which the i-th section begins.                                                                |
| `content_safety_labels.results[i].timestamp.end`                         | number | The time, in milliseconds, at which the i-th section ends.                                                                  |
| `content_safety_labels.summary`                                          | object | A summary of the Content Moderation confidence results for the entire audio file.                                           |
| `content_safety_labels.summary.topic`                                    | number | A confidence score for the presence of the sensitive topic "topic" across the entire audio file.                            |
| `content_safety_labels.severity_score_summary`                           | object | A summary of the Content Moderation severity results for the entire audio file.                                             |
| `content_safety_labels.severity_score_summary.topic.[low, medium, high]` | number | A distribution across the values "low", "medium", and "high" for the severity of the presence of "topic" in the audio file. |

The response also includes the request parameters used to generate the transcript.

## Supported labels

| Label                   | Description                                                                                                                                                                                                      | Model output              | Severity |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | -------- |
| Accidents               | Any man-made incident that happens unexpectedly and results in damage, injury, or death.                                                                                                                         | `accidents`               | Yes      |
| Alcohol                 | Content that discusses any alcoholic beverage or its consumption.                                                                                                                                                | `alcohol`                 | Yes      |
| Company Financials      | Content that discusses any sensitive company financial information.                                                                                                                                              | `financials`              | No       |
| Crime Violence          | Content that discusses any type of criminal activity or extreme violence that is criminal in nature.                                                                                                             | `crime_violence`          | Yes      |
| Drugs                   | Content that discusses illegal drugs or their usage.                                                                                                                                                             | `drugs`                   | Yes      |
| Gambling                | Includes gambling on casino-based games such as poker, slots, etc. as well as sports betting.                                                                                                                    | `gambling`                | Yes      |
| Hate Speech             | Content that's a direct attack against people or groups based on their sexual orientation, gender identity, race, religion, ethnicity, national origin, disability, etc.                                         | `hate_speech`             | Yes      |
| Health Issues           | Content that discusses any medical or health-related problems.                                                                                                                                                   | `health_issues`           | Yes      |
| Manga                   | Mangas are comics or graphic novels originating from Japan with some of the more popular series being "Pokemon", "Naruto", "Dragon Ball Z", "One Punch Man", and "Sailor Moon".                                  | `manga`                   | No       |
| Marijuana               | This category includes content that discusses marijuana or its usage.                                                                                                                                            | `marijuana`               | Yes      |
| Natural Disasters       | Phenomena that happens infrequently and results in damage, injury, or death. Such as hurricanes, tornadoes, earthquakes, volcano eruptions, and firestorms.                                                      | `disasters`               | Yes      |
| Negative News           | News content with a negative sentiment which typically occur in the third person as an unbiased recapping of events.                                                                                             | `negative_news`           | No       |
| NSFW (Adult Content)    | Content considered "Not Safe for Work" and consists of content that a viewer would not want to be heard/seen in a public environment.                                                                            | `nsfw`                    | No       |
| Pornography             | Content that discusses any sexual content or material.                                                                                                                                                           | `pornography`             | Yes      |
| Profanity               | Any profanity or cursing.                                                                                                                                                                                        | `profanity`               | Yes      |
| Sensitive Social Issues | This category includes content that may be considered insensitive, irresponsible, or harmful to certain groups based on their beliefs, political affiliation, sexual orientation, or gender identity.            | `sensitive_social_issues` | No       |
| Terrorism               | Includes terrorist acts as well as terrorist groups. Examples include bombings, mass shootings, and ISIS. Note that many texts corresponding to this topic may also be classified into the crime violence topic. | `terrorism`               | Yes      |
| Tobacco                 | Text that discusses tobacco and tobacco usage, including e-cigarettes, nicotine, vaping, and general discussions about smoking.                                                                                  | `tobacco`                 | Yes      |
| Weapons                 | Text that discusses any type of weapon including guns, ammunition, shooting, knives, missiles, torpedoes, etc.                                                                                                   | `weapons`                 | Yes      |

## Frequently asked questions

  
There could be a few reasons for this. First, make sure that the audio file contains speech, and not just background noise or music. Additionally, the model may not have been trained on the specific type of sensitive content you're looking for. If you believe the model should be able to detect the content but it's not, you can reach out to AssemblyAI's support team for assistance.

  
The model may occasionally flag content as sensitive that isn't actually problematic. This can happen if the model isn't trained on the specific context or nuances of the language being used. In these cases, you can manually review the flagged content and determine if it's actually sensitive or not. If you believe the model is consistently flagging content incorrectly, you can contact AssemblyAI's support team to report the issue.

  
The Content Moderation model provides segment-level results that pinpoint where in the audio the sensitive content was discussed, as well as the degree to which it was discussed. You can access this information in the results key of the API response. Each result in the list contains a text key that shows the sensitive content, and a labels key that shows the detected sensitive topics along with their confidence and severity scores.

  
The model is designed to process batches of segments in significantly less than 1 second, making it suitable for real-time applications. However, keep in mind that the actual processing time depends on the length of the audio file and the number of segments it's divided into. Additionally, the model may occasionally require additional time to process particularly complex or long segments.

  
If you receive an error message, it may be due to an issue with your request format or parameters. Double-check that your request includes the correct `audio_url` parameter. If you continue to experience issues, you can reach out to AssemblyAI's support team for assistance.

---

<a id="profanity-filtering-docs-guardrails-filter-profanity-from-transcripts-mdx"></a>

# Profanity Filtering

URL: https://www.assemblyai.com/docs/guardrails/filter-profanity-from-transcripts
Source: docs/guardrails/filter-profanity-from-transcripts.mdx
Navigation: Guardrails > Models & features
Description: Profanity Filtering documentation.

**Streaming profanity filtering**

  For real-time profanity filtering on streaming transcripts, see [Filter profanity](/streaming/filter-profanity-from-transcripts).

  
  <br />

  
  <br />

  US & EU <br />

You can automatically filter out profanity from the transcripts by setting `filter_profanity` to `true` in your transcription config.

Any profanity in the returned `text` will be replaced with asterisks.

```python title="Python SDK" highlight={8} 
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    filter_profanity=True
)

transcript = aai.Transcriber(config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" highlight={19}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "filter_profanity": True
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" highlight={12}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  filter_profanity: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={19}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  filter_profanity: true,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  Profanity filter isn't perfect. Certain words may still be missed or
  improperly filtered.

---

<a id="speech-threshold-docs-guardrails-set-minimum-speech-threshold-mdx"></a>

# Speech Threshold

URL: https://www.assemblyai.com/docs/guardrails/set-minimum-speech-threshold
Source: docs/guardrails/set-minimum-speech-threshold.mdx
Navigation: Guardrails > Models & features
Description: Speech Threshold documentation.

**Supported Languages, Regions, and Models**

  Speech threshold is supported for all languages, regions, and models.

To only transcribe files that contain at least a specified percentage of spoken audio, you can set the `speech_threshold` parameter. You can pass any value between 0 and 1.

If the percentage of speech in the audio file is below the provided threshold, the value of `text` is `None` and the response contains an `error` message:

```plain 
Audio speech threshold 0.9461 is below the requested speech threshold value 1.0
```

  A file must contain at least 30 seconds of audio for the amount of spoken
  audio to reliably be determined.

```python title="Python SDK" highlight={8} 
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# audio_file = "./local_file.mp3"
audio_file = "https://assembly.ai/wildfires.mp3"

config = aai.TranscriptionConfig(
    language_detection=True,
    speech_threshold=0.5
)

transcript = aai.Transcriber(config=config).transcribe(audio_file)

if transcript.status == "error":
  raise RuntimeError(f"Transcription failed: {transcript.error}")

print(transcript.text)
```

```python title="Python" highlight={19}  expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "language_detection": True,
    "speech_threshold": 0.5
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    print(f"Transcript ID: {transcript_id}")
    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)

```

```javascript title="JavaScript SDK" highlight={12}  expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// const audioFile = './local_file.mp3'
const audioFile = "https://assembly.ai/wildfires.mp3";

const params = {
  audio: audioFile,
  language_detection: true,
  speech_threshold: 0.5,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);

  if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }

  console.log(transcript.text);
};

run();
```

```javascript title="JavaScript" highlight={19}  expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  language_detection: true,
  speech_threshold: 0.5,
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

---

<a id="quickstart-docs-llm-gateway-quickstart-mdx"></a>

# Quickstart

URL: https://www.assemblyai.com/docs/llm-gateway/quickstart
Source: docs/llm-gateway/quickstart.mdx
Navigation: LLM Gateway > Getting started
Description: AssemblyAI's LLM Gateway is a unified API providing access to 25+ models from Claude, GPT, Gemini, and more through a single interface.

## Overview

**AssemblyAI's LLM Gateway** is a unified, Open AI sdk compatible, interface that allows you to connect across multiple LLMs through a single API with access to automatic fallbacks.
Use as a standalone service or make use of our easy integration with AssemblyAI speech-to-text products.

We focus on text to text LLM providers with regional compliance, security, and ZDR. Additionally, we have optional
but tight integration with other AssemblyAI products for a more seamless experience.

## Quickstart

```python {11}
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers = headers,
    json = {
        "model": "qwen3.5-4b-32k-fast",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript {10}
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "qwen3.5-4b-32k-fast",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

```python {11}
from openai import OpenAI
import openai.types.chat.chat_completion as types

client = OpenAI(
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key="<YOUR_API_KEY>",
)

import json

messages = [{"role": "user", "content": "What is the capital of france"}]
response = client.chat.completions.create(model="qwen3.5-4b-32k-fast", messages=messages)
print(response.choices[0].message.content)
```

Simply change the `model` parameter to use any of the available models listed in the [Available models](/llm-gateway/available-models) page.

  Want to compare models side-by-side? Try the [Model Comparison
  Tool](https://assemblyai-model-comparison-llmgw.lovable.app/), a Lovable
  application, to test different LLM models and see how they perform.

## Non-standard LLM Enhancements

Here are some bonus features LLM Gateway provides over other routing options

* Automatic retries
  * 5xx responses will automatically be retried on our side by default
  * configure with [fallback_config](/api-reference/llm-gateway/create-chat-completion#body-fallback-config)
* Model Fallbacks
  * Configure alternative models/providers in case a model isn't working
  * Even change the prompt and/or params to better suit a different model
  * See [docs](/llm-gateway/fallback)
* Automatically apply LLM Gateway requests on each real-time STT turn for reduced E2E latency
  * see [docs](/guides/real_time_llm_gateway)
* Post Processing
  * Automatically apply post processing to the LLM response
  * Repair broken JSON in tool calls and structured output with `json-repair`
  * see [docs](/llm-gateway/structured-outputs#post-processing)
* Inject Transcript Text
  * Call LLM Gateway with a `transcript_id` and we will handle fetching and adding the transcript text to the LLM message.
  * Useful if you don't want to store the transcript.
  * see [docs](/llm-gateway/chat-completions#inject-a-transcript-by-id)

## Regional compliance

| Endpoint     | Base URL                                                    |
| ------------ | ----------------------------------------------------------- |
| US (default) | `https://llm-gateway.assemblyai.com/v1/chat/completions`   |
| EU           | `https://llm-gateway.eu.assemblyai.com/v1/chat/completions`|

  The LLM Gateway is available in both US and EU regions. Use the EU endpoint to
  ensure your data stays within the European Union. Currently, Anthropic Claude
  and most Google Gemini models are supported in the EU (except where otherwise noted). OpenAI models are only
  available in the US region. See [Cloud Endpoints and Data
  Residency](/llm-gateway/cloud-endpoints-and-data-residency) for more details.

The LLM Gateway provides access to 25+ models across major AI providers with support for:

- **Basic Chat Completions** - Simple request/response interactions
- **Streamed Responses** - Stream output as it's generated (OpenAI models)
- **Multi-turn Conversations** - Maintain context across multiple exchanges
- **Structured Outputs** - Constrain responses to a specific JSON schema
- **Tool/Function Calling** - Enable models to execute custom functions
- **Agentic Workflows** - Multi-step reasoning with automatic tool chaining
- **Unified Interface** - One API for Claude, GPT, Gemini, Qwen, Kimi, and more
- **Post-processing** - Automatically repair malformed JSON responses with built-in JSON repair

### Global model routing

If your request/needs do not require region locked routing for either latency or compliance, you can opt to set the
`model_region` to `global` in the api request body for a discount.

See model [pricing](https://www.assemblyai.com/pricing) for which models have global discounts.

## Rate limits

LLM Gateway is rate limited per model, measured as requests within a 60-second window. For account-specific limits and how to request a higher limit, see [Rate limits](/llm-gateway/rate-limits).

## Logging and troubleshooting

Every LLM Gateway response includes a `request_id` field — a unique identifier for that request. Persist it (along with the model, the API region, and a timestamp) for every call you make, not just when something goes wrong. If you contact [support@assemblyai.com](mailto:support@assemblyai.com) about a specific request (latency spikes, unexpected output, rate-limit errors, content moderation surprises), this ID lets us locate the exact request in our logs immediately.

We recommend logging, at minimum:

- `request_id` from the response body
- The `model` parameter used
- The API region (US: `llm-gateway.assemblyai.com`, EU: `llm-gateway.eu.assemblyai.com`)
- A timestamp for when the request was sent
- The full error response body when a non-2xx status code is returned

For details on debugging specific status codes (400/401/403/429/5xx) and what information to include when filing a support request, see the [Troubleshooting](/llm-gateway/troubleshooting) page.

## Next steps

- [Basic Chat Completions](/llm-gateway/chat-completions) - Learn how to send simple messages and receive responses
- [Multi-turn Conversations](/llm-gateway/conversations) - Maintain context across multiple exchanges
- [Structured Outputs](/llm-gateway/structured-outputs) - Constrain model responses to follow a specific JSON schema
- [Tool Calling](/llm-gateway/tool-calling) - Enable models to execute custom functions
- [Agentic Workflows](/llm-gateway/agentic-workflows) - Build multi-step reasoning applications
- [Post-processing](/llm-gateway/structured-outputs#post-processing) - Automatically repair malformed JSON in model responses
- [Prompt Caching](/llm-gateway/prompt-caching) - Save money on similar calls with prompt caching

  The LLM Gateway API is separate from the Speech-to-Text and Speech
  Understanding APIs. It provides a unified interface to work with large
  language models across multiple providers.

---

<a id="models-docs-llm-gateway-available-models-mdx"></a>

# Models

URL: https://www.assemblyai.com/docs/llm-gateway/available-models
Source: docs/llm-gateway/available-models.mdx
Navigation: LLM Gateway > Getting started
Description: Available models on LLM Gateway

## Available models

| Model | ID | Supported parameters | Max Context | Retirement Date |
| --- | --- | --- | --- | --- |
| **Haiku 4.5** | `claude-haiku-4-5-20251001` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream` | `200000` |  |
| **Opus 4.5** | `claude-opus-4-5-20251101` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream` | `200000` |  |
| **Opus 4.6** | `claude-opus-4-6` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream` | `200000` |  |
| **Opus 4.7** | `claude-opus-4-7` | `max_tokens`, `tools`, `tool_choice`, `stream` | `1000000` |  |
| **Opus 4.8** | `claude-opus-4-8` | `max_tokens`, `tools`, `tool_choice`, `stream` | `1000000` |  |
| **Sonnet 4.5** | `claude-sonnet-4-5-20250929` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream` | `200000` |  |
| **Sonnet 4.6** | `claude-sonnet-4-6` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream` | `200000` |  |
| **Sonnet 5** | `claude-sonnet-5` | `max_tokens`, `tools`, `tool_choice`, `stream` | `200000` |  |
| **Gemini 2.5 Flash** | `gemini-2.5-flash` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `1048576` |  |
| **Gemini 2.5 Flash Lite** | `gemini-2.5-flash-lite` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `1048576` |  |
| **Gemini 2.5 Pro** | `gemini-2.5-pro` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `200000` |  |
| **Gemini 3.1 Flash Lite** | `gemini-3.1-flash-lite` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `1048575` | 2027-05-07 |
| **Gemini 3.5 Flash** | `gemini-3.5-flash` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `1048575` |  |
| **Gemini 3.5 Flash Lite** | `gemini-3.5-flash-lite` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `1048575` |  |
| **Gemini 3.6 Flash** | `gemini-3.6-flash` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `1048575` |  |
| **Gemini 3.7 Flash** | `gemini-3.7-flash` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `1048575` |  |
| **GPT OSS 120B** | `gpt-oss-120b` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice` | `131072` |  |
| **GPT OSS 20B** | `gpt-oss-20b` | `max_tokens`, `temperature`, `tools`, `tool_choice` | `131072` |  |
| **GPT-4.1** | `gpt-4.1` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `stream` | `1047576` |  |
| **GPT-5** | `gpt-5` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `stream`, `response_format` | `400000` |  |
| **GPT-5 Nano** | `gpt-5-nano` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `400000` |  |
| **GPT-5 mini** | `gpt-5-mini` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `400000` |  |
| **GPT-5.1** | `gpt-5.1` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `400000` |  |
| **GPT-5.2** | `gpt-5.2` | `max_tokens`, `response_format`, `temperature`, `tools`, `tool_choice`, `stream` | `400000` |  |
| **GPT-5.5** | `gpt-5.5` | `max_tokens`, `response_format`, `tools`, `tool_choice`, `stream` | `272000` |  |
| **GPT-5.6 Luna** | `gpt-5.6-luna` | `max_tokens`, `response_format`, `tools`, `tool_choice`, `stream` | `270000` |  |
| **GPT-5.6 Terra** | `gpt-5.6-terra` | `max_tokens`, `response_format`, `tools`, `tool_choice`, `stream` | `270000` |  |
| **Qwen3 32B** | `qwen3-32B` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream` | `200000` |  |
| **Qwen3 Next 80B A3B** | `qwen3-next-80b-a3b` | `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream` | `200000` |  |
| **Qwen3.5 4B Fast** | `qwen3.5-4b-32k-fast` | `max_tokens`, `temperature`, `stream` | `32768` |  |

### Pricing details

| Model | Parameter | Prompt (per 1M) | Completion (per 1M) | Cache read (per 1M) | Cache write (per 1M) | Cache write 1h (per 1M) | Regional surcharge |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Haiku 4.5** | `claude-haiku-4-5-20251001` | $1 | $5 | $0.1 | $1.25 | $2 | +10% |
| **Opus 4.5** | `claude-opus-4-5-20251101` | $5 | $25 | $0.5 | $6.25 | $10 | +10% |
| **Opus 4.6** | `claude-opus-4-6` | $5 | $25 | $0.5 | $6.25 | $10 | +10% |
| **Opus 4.7** | `claude-opus-4-7` | $5 | $25 | $0.5 | $6.25 | $10 | +10% |
| **Opus 4.8** | `claude-opus-4-8` | $5 | $25 | $0.5 | $6.25 | $10 | +10% |
| **Sonnet 4.5** | `claude-sonnet-4-5-20250929` | $3 | $15 | $0.3 | $3.75 | $6 | +10% |
| **Sonnet 4.6** | `claude-sonnet-4-6` | $3 | $15 | $0.3 | $3.75 | $6 | +10% |
| **Sonnet 5** | `claude-sonnet-5` | $3 | $15 | $0.3 | $3.75 | $6 | +10% |
| **Gemini 2.5 Flash** | `gemini-2.5-flash` | $0.3 | $2.5 | $0.03 |  |  | +10% |
| **Gemini 2.5 Flash Lite** | `gemini-2.5-flash-lite` | $0.1 | $0.4 | $0.01 |  |  | +10% |
| **Gemini 2.5 Pro** | `gemini-2.5-pro` | $1.25 | $10 | $0.125 |  |  | +10% |
| **Gemini 3.1 Flash Lite** | `gemini-3.1-flash-lite` | $0.25 | $1.5 | $0.025 |  |  | +10% |
| **Gemini 3.5 Flash** | `gemini-3.5-flash` | $1.25 | $9 | $0.125 |  |  | +10% |
| **Gemini 3.5 Flash Lite** | `gemini-3.5-flash-lite` | $0.3 | $2.5 | $0.03 |  |  | +10% |
| **Gemini 3.6 Flash** | `gemini-3.6-flash` | $1.5 | $7.5 | $0.15 |  |  | +10% |
| **Gemini 3.7 Flash** | `gemini-3.7-flash` | $0.75 | $3.75 | $0.075 |  |  | +10% |
| **GPT OSS 120B** | `gpt-oss-120b` | $0.15 | $0.6 |  |  |  |  |
| **GPT OSS 20B** | `gpt-oss-20b` | $0.07 | $0.3 |  |  |  |  |
| **GPT-4.1** | `gpt-4.1` | $2 | $8 | $0.5 |  |  | +10% |
| **GPT-5** | `gpt-5` | $1.25 | $10 | $0.125 |  |  | +10% |
| **GPT-5 Nano** | `gpt-5-nano` | $0.05 | $0.4 | $0.005 |  |  | +10% |
| **GPT-5 mini** | `gpt-5-mini` | $0.25 | $2 | $0.025 |  |  | +10% |
| **GPT-5.1** | `gpt-5.1` | $1.25 | $10 | $0.125 |  |  | +10% |
| **GPT-5.2** | `gpt-5.2` | $1.75 | $14 | $0.175 |  |  | +10% |
| **GPT-5.5** | `gpt-5.5` | $5 | $30 | $0.5 |  |  | +10% |
| **GPT-5.6 Luna** | `gpt-5.6-luna` | $1 | $6 | $0.1 | $1.25 |  | +10% |
| **GPT-5.6 Terra** | `gpt-5.6-terra` | $2.5 | $15 | $0.25 | $3.125 |  | +10% |
| **Qwen3 32B** | `qwen3-32B` | $0.15 | $0.6 |  |  |  |  |
| **Qwen3 Next 80B A3B** | `qwen3-next-80b-a3b` | $0.15 | $1.2 |  |  |  |  |
| **Qwen3.5 4B Fast** | `qwen3.5-4b-32k-fast` | $0.1 | $0.5 |  |  |  |  |

  For information on data retention and model training policies for each
  provider, see [Data Retention and Model Training](/data-retention-and-model-training#llm-gateway-production-environment).

  Head to [our Playground](https://www.assemblyai.com/dashboard/home) to
  test out LLM Gateway without having to write any code!

---

<a id="providers-docs-llm-gateway-providers-mdx"></a>

# Providers

URL: https://www.assemblyai.com/docs/llm-gateway/providers
Source: docs/llm-gateway/providers.mdx
Navigation: LLM Gateway > Getting started
Description: Model providers available through LLM Gateway, including data retention policies and regional availability

## Overview

LLM Gateway routes requests to models hosted by several third-party providers. This page summarizes each provider's data retention posture, whether they require opt-in, and where to find their public terms.

For a full list of models and IDs, see [Models](/llm-gateway/available-models). For which models are available in which regions, see [Cloud endpoints & data residency](/llm-gateway/cloud-endpoints-and-data-residency#model-availability-by-region). For AssemblyAI's overall data retention policies, see [Data retention and model training](/data-retention-and-model-training#llm-gateway-production-environment).

## Provider summary

| Provider     | Retention                                                                                       | Availability  | Terms                                                                  |
| ------------ | ------------------------------------------------------------------------------------------------ | ------------- | ------------------------------------------------------------------------ |
| AssemblyAI   | [ZDR](/data-retention-and-model-training)                                                       | On by default | [Terms](https://www.assemblyai.com/legal/terms-of-service)             |
| AWS Bedrock  | [ZDR](https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html)                | On by default | [Terms](https://aws.amazon.com/service-terms/)                         |
| Google       | [ZDR](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/vertex-ai-zero-data-retention) | On by default | [Terms](https://cloud.google.com/terms)                                |
| OpenAI       | [ZDR](https://developers.openai.com/api/docs/guides/your-data#zero-data-retention)              | On by default | [Terms](https://openai.com/policies/business-terms/)                   |
| Together AI  | [ZDR](https://www.together.ai/privacy)                                                          | Opt-in        | [Terms](https://www.together.ai/terms-of-service)                      |
| Fireworks AI | [ZDR](https://docs.fireworks.ai/guides/security_compliance/data_handling#zero-data-retention)   | Opt-in        | [Terms](https://fireworks.ai/terms-of-service)                         |
| DigitalOcean | [See provider](https://docs.digitalocean.com/products/inference/details/data-privacy/)          | Opt-in        | [Terms](https://www.digitalocean.com/legal/terms-of-service-agreement) |

Providers marked **Opt-in** are disabled by default. To use them, you need to manually turn on their toggles on the [Data Controls](https://www.assemblyai.com/dashboard/settings/data-controls) page in your dashboard.

  No LLM Gateway providers train on your data. AssemblyAI has opted out of model training with all LLM Gateway providers. This is separate from whether AssemblyAI trains our own models with your data. See [Data retention and model training](/data-retention-and-model-training) for details.

---

<a id="model-fallbacks-docs-llm-gateway-fallback-mdx"></a>

# Model Fallbacks

URL: https://www.assemblyai.com/docs/llm-gateway/fallback
Source: docs/llm-gateway/fallback.mdx
Navigation: LLM Gateway > Features
Description: Automatically fall back to an alternative LLM model when your primary model fails

## Overview

The Fallback feature lets you specify one or more backup models that the LLM Gateway will automatically try if your primary model fails. This ensures your application stays resilient without requiring complex retry logic on your end.

  The LLM Gateway is available in both US and EU regions. Fallback behavior works the same way on both endpoints. See [Cloud endpoints and data residency](/llm-gateway/cloud-endpoints-and-data-residency) for more details.

## Basic usage

To add a fallback, include a `fallbacks` array in your request. Each entry specifies an alternative `model` to use if the primary model is unavailable:

```python expandable
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gemini-2.5-flash-lite",
        "messages": [
            {"role": "user", "content": "Tell me a short fairy tale."}
        ],
        "max_tokens": 400,
        "fallbacks": [
            {"model": "claude-sonnet-4-6"}
        ]
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gemini-2.5-flash-lite",
      messages: [{ role: "user", content: "Tell me a short fairy tale." }],
      max_tokens: 400,
      fallbacks: [
        { model: "claude-sonnet-4-6" }
      ],
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

If `gemini-2.5-flash-lite` fails, the LLM Gateway automatically retries the request using `claude-sonnet-4-6`.

  You can chain up to two fallback models by setting `fallback_config.depth` to `2`. The LLM Gateway tries each fallback in order until one succeeds.

## Override fields per fallback

In the advanced case, you can override specific request fields for each fallback model. For example, you can change the `messages` or `temperature` for the fallback:

```python expandable
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gemini-2.5-flash-lite",
        "messages": [
            {"role": "user", "content": "Tell me a short fairy tale."}
        ],
        "temperature": 0.2,
        "max_tokens": 400,
        "fallbacks": [
            {
                "model": "claude-sonnet-4-6",
                "messages": [
                    {"role": "user", "content": "Tell me a fairy tale, but be very concise."}
                ],
                "temperature": 0.4
            }
        ]
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript expandable
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gemini-2.5-flash-lite",
      messages: [{ role: "user", content: "Tell me a short fairy tale." }],
      temperature: 0.2,
      max_tokens: 400,
      fallbacks: [
        {
          model: "claude-sonnet-4-6",
          messages: [
            { role: "user", content: "Tell me a fairy tale, but be very concise." },
          ],
          temperature: 0.4,
        },
      ],
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

Any field you don't override in the fallback inherits the value from the original request. Only the fields you explicitly set in the fallback object are changed.

## Retry behavior

If no fallbacks are set, the API automatically retries the LLM request once after 500ms. This is because `fallback_config.retry` defaults to `true`, providing a zero-config way to handle transient failures.

For more control over retries, set `retry` to `false` and implement your own exponential backoff:

```json
{
  "model": "gemini-2.5-flash-lite",
  "messages": [{"role": "user", "content": "Tell me a short fairy tale."}],
  "max_tokens": 400,
  "fallback_config": {
    "retry": false
  }
}
```

## Response behavior

When a fallback is used, the response looks exactly as if you had made the original request with the fallback model. The `model` field in the response reflects the fallback model that was used, and billing is charged only for that model.

## API reference

### Request parameters

| Key                      | Type    | Required? | Description                                                                                                                           |
| ------------------------ | ------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `model`                  | string  | Yes       | The primary model to use for completion. See [Available models](/llm-gateway/available-models) for supported values.     |
| `messages`               | array   | Yes       | An array of message objects representing the conversation history.                                                                    |
| `fallbacks`              | array   | No        | An array of fallback objects. Each object must include a `model` and can override any field available in the original request.         |
| `fallback_config`        | object  | No        | Configuration for fallback behavior.                                                                                                  |
| `fallback_config.retry`  | boolean | No        | Whether to automatically retry the request once after 500ms on failure. Defaults to `true`.                                           |
| `fallback_config.depth`  | number  | No        | Max fallbacks to traverse. Default 1, max 2.                                                                                          |

### Fallback object

Each object in the `fallbacks` array must include a `model` and can override any field available in the original request. For example:

| Key           | Type   | Required? | Description                                                                                                                       |
| ------------- | ------ | --------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `model`       | string | Yes       | The fallback model to use. See [Available models](/llm-gateway/available-models) for supported values.               |
| `messages`    | array  | No        | Override the messages for the fallback request.                                                                                   |
| `temperature` | number | No        | Override the temperature for the fallback request.                                                                                |
| `max_tokens`  | number | No        | Override the max tokens for the fallback request.                                                                                 |

Any field from the original request can be included in a fallback object. Fields not specified in the fallback inherit the values from the original request.

## Next steps

- [Basic chat completions](/llm-gateway/chat-completions) - Send simple messages and receive responses
- [Multi-turn conversations](/llm-gateway/conversations) - Maintain context across multiple exchanges
- [Tool calling](/llm-gateway/tool-calling) - Enable models to execute custom functions
- [Cloud endpoints and data residency](/llm-gateway/cloud-endpoints-and-data-residency) - Learn about regional endpoint options

---

<a id="prompt-caching-docs-llm-gateway-prompt-caching-mdx"></a>

# Prompt Caching

URL: https://www.assemblyai.com/docs/llm-gateway/prompt-caching
Source: docs/llm-gateway/prompt-caching.mdx
Navigation: LLM Gateway > Features
Description: Reduce latency and costs by caching repeated prompt content across LLM Gateway requests

**Public Beta**

  Prompt caching is available in Public Beta.

## Overview

Prompt caching lets you avoid reprocessing the same prompt content on every request. When you send a long system prompt, tool definitions, or conversation history repeatedly, the LLM provider can cache that content and reuse it on subsequent requests — reducing both latency and cost.

The LLM Gateway supports prompt caching across all major providers, with each provider using its own caching mechanism:

| Provider  | Caching behavior | Configuration required? |
| --------- | ---------------- | ----------------------- |
| Claude    | Explicit opt-in  | Yes — add `cache_control` to messages |
| OpenAI    | Automatic        | No — caching happens implicitly |
| Gemini    | Automatic        | No — caching happens implicitly |

  Cached input tokens are billed at a discounted rate compared to regular input tokens. The exact discount depends on the model and provider.

  Prompt caching only activates when the cacheable portion of your prompt meets a minimum token threshold. Claude's minimum varies by model; see [Minimum cacheable prompt length](#minimum-cacheable-prompt-length) for links to each provider's documentation. Shorter prompts won't benefit from caching.

## Claude models

Claude models require you to explicitly mark which content blocks to cache using the `cache_control` field. Add `cache_control` with `type` set to `"ephemeral"` on any message you want cached.

```python expandable
import os
import requests

headers = {
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
}

system_prompt = (
    "You are a customer support agent for Acme Corp. "
    "You have access to our full product catalog, pricing, "
    "and policy documentation. Always be helpful and concise."
)

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "system",
                "content": system_prompt,
                "cache_control": {"type": "ephemeral"}
            },
            {
                "role": "user",
                "content": "What is your return policy?"
            }
        ],
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])

# Check cache usage in the response
usage = result["usage"]
cache_details = usage.get("prompt_tokens_details", {})
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")
```

```javascript expandable
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: process.env.ASSEMBLYAI_API_KEY,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [
        {
          role: "system",
          content:
            "You are a customer support agent for Acme Corp. " +
            "You have access to our full product catalog, pricing, " +
            "and policy documentation. Always be helpful and concise.",
          cache_control: { type: "ephemeral" },
        },
        {
          role: "user",
          content: "What is your return policy?",
        },
      ],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);

// Check cache usage in the response
const cacheDetails = result.usage?.prompt_tokens_details;
console.log(`Cached tokens: ${cacheDetails?.cached_tokens ?? 0}`);
```

You can also set `cache_control` on tool result messages to cache tool interaction history in multi-turn agentic conversations.

### Minimum cacheable prompt length

Claude only caches prompts that meet a minimum token threshold, and the threshold depends on the model. If the cacheable portion of your prompt falls below this threshold, the request is processed without caching and no error is returned.

For per-model thresholds, see Anthropic's [prompt caching cache limitations](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#cache-limitations). OpenAI's minimum is documented in [OpenAI's prompt caching guide](https://developers.openai.com/docs/guides/prompt-caching), and Gemini's in [Google's context caching guide](https://ai.google.dev/gemini-api/docs/caching).

### Cache control with TTL

The LLM Gateway extends Anthropic's native `cache_control` with an optional `ttl` field for specifying cache duration. This is a Gateway-specific parameter — Anthropic's native API does not support it.

```json
{
  "cache_control": {
    "type": "ephemeral",
    "ttl": "5m"
  }
}
```

  The `ttl` field is a Gateway extension, not part of Anthropic's native API. If omitted, Anthropic's default cache duration applies.

### Where to place cache_control

The `cache_control` field can be placed on:

- **System messages** — Cache long system prompts that don't change between requests
- **User and assistant messages** — Cache conversation history in multi-turn flows
- **Tool result messages** — Cache tool call outputs in agentic workflows

  For best results with Claude, place `cache_control` on the content that stays the same across requests — typically the system prompt and any static context. Content after the last cache breakpoint is not cached.

## OpenAI models

OpenAI models cache prompts automatically. No configuration is needed — the gateway passes your requests through and caching happens on OpenAI's infrastructure.

```python expandable
import os
import requests

headers = {
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
}

# OpenAI models cache automatically — no cache_control needed
response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gpt-4.1",
        "messages": [
            {
                "role": "system",
                "content": "You are a customer support agent..."
            },
            {
                "role": "user",
                "content": "What is your return policy?"
            }
        ],
        "max_tokens": 1000
    }
)

result = response.json()

# Cached tokens still appear in usage
cache_details = result["usage"].get("prompt_tokens_details", {})
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")
```

```javascript expandable
// OpenAI models cache automatically — no cache_control needed
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: process.env.ASSEMBLYAI_API_KEY,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gpt-4.1",
      messages: [
        {
          role: "system",
          content: "You are a customer support agent...",
        },
        {
          role: "user",
          content: "What is your return policy?",
        },
      ],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
const cacheDetails = result.usage?.prompt_tokens_details;
console.log(`Cached tokens: ${cacheDetails?.cached_tokens ?? 0}`);
```

You can optionally configure cache behavior with two additional request-level fields:

| Field                    | Type   | Description |
| ------------------------ | ------ | ----------- |
| `prompt_cache_retention` | string | Controls how long cached content is retained on OpenAI's infrastructure. These values are passed through to OpenAI's API — refer to [OpenAI's documentation](https://developers.openai.com/docs/guides/prompt-caching) for current allowed values. |
| `prompt_cache_key`       | string | A custom key to group related requests for caching. Requests with the same key are more likely to share cached content. |

## Gemini models

Gemini models also cache automatically — no configuration needed.

```python expandable
import os
import requests

headers = {
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
}

# Gemini models cache automatically
response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gemini-2.5-flash",
        "messages": [
            {
                "role": "system",
                "content": "You are a customer support agent..."
            },
            {
                "role": "user",
                "content": "What is your return policy?"
            }
        ],
        "max_tokens": 1000
    }
)

result = response.json()

# Cached tokens appear in usage
cache_details = result["usage"].get("prompt_tokens_details", {})
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")
```

```javascript expandable
// Gemini models cache automatically
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: process.env.ASSEMBLYAI_API_KEY,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gemini-2.5-flash",
      messages: [
        {
          role: "system",
          content: "You are a customer support agent...",
        },
        {
          role: "user",
          content: "What is your return policy?",
        },
      ],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
const cacheDetails = result.usage?.prompt_tokens_details;
console.log(`Cached tokens: ${cacheDetails?.cached_tokens ?? 0}`);
```

## Reading cache metrics from the response

All providers return cache usage data in the `usage.prompt_tokens_details` field of the response:

```json
{
  "usage": {
    "input_tokens": 500,
    "output_tokens": 150,
    "total_tokens": 650,
    "prompt_tokens_details": {
      "cached_tokens": 450,
      "cache_creation": {
        "ephemeral_5m_input_tokens": 0,
        "ephemeral_1h_input_tokens": 0
      }
    }
  }
}
```

| Field | Description |
| ----- | ----------- |
| `cached_tokens` | Number of input tokens read from cache (cost savings). |
| `cache_creation.ephemeral_5m_input_tokens` | Tokens written to a 5-minute ephemeral cache (Claude only). |
| `cache_creation.ephemeral_1h_input_tokens` | Tokens written to a 1-hour ephemeral cache (Claude only). |

When `cached_tokens` is greater than zero, those tokens were served from cache and billed at the discounted cached input rate rather than the standard input rate.

## Best practices

- **Cache your system prompt** — System prompts are the best candidates for caching since they stay the same across requests. Place `cache_control` on the system message for Claude, or rely on automatic caching for OpenAI and Gemini.
- **Cache tool definitions** — If you use the same tools across multiple requests, the tool definitions in your prompt are automatically eligible for caching.
- **Order messages for maximum cache hits** — Put static content (system prompt, tool definitions) at the beginning of the message array. Content before the cache breakpoint is more likely to match across requests.
- **Monitor cache metrics** — Check `prompt_tokens_details.cached_tokens` in responses to verify caching is working and estimate your cost savings.

## API reference

### Request

#### Top-level request parameters

These fields are set at the top level of the request body:

| Key | Type | Required? | Description |
| --- | ---- | --------- | ----------- |
| `cache_control` | object | No | Default cache control applied to the entire request (Claude models). When set at the request level, it acts as a default for all messages. Contains `type` and optional `ttl`. |
| `prompt_cache_retention` | string | No | Controls cache retention duration (OpenAI models). Passed through to OpenAI's API. |
| `prompt_cache_key` | string | No | Custom cache key for grouping requests (OpenAI models). |

#### Message-level cache_control

The `cache_control` field can also be set on individual messages. Message-level `cache_control` lets you mark specific cache breakpoints — the provider caches all content up to and including the marked message. This is the recommended approach for Claude models.

| Key | Type | Required? | Description |
| --- | ---- | --------- | ----------- |
| `cache_control` | object | No | Cache control for this specific message. Marks a cache breakpoint at this position in the conversation. |

#### Cache control object

The `cache_control` object has the same structure whether used at the request level or message level:

| Key | Type | Required? | Description |
| --- | ---- | --------- | ----------- |
| `type` | string | Yes | The cache type. Use `"ephemeral"` for standard caching. |
| `ttl` | string | No | Time-to-live for the cached content (Gateway extension — not part of Anthropic's native API). |

### Response

#### Usage fields for cache metrics

| Key | Type | Description |
| --- | ---- | ----------- |
| `usage.prompt_tokens_details.cached_tokens` | number | Input tokens served from cache. |
| `usage.prompt_tokens_details.cache_creation.ephemeral_5m_input_tokens` | number | Tokens written to 5-minute cache. |
| `usage.prompt_tokens_details.cache_creation.ephemeral_1h_input_tokens` | number | Tokens written to 1-hour cache. |

---

<a id="tool-calling-docs-llm-gateway-tool-calling-mdx"></a>

# Tool Calling

URL: https://www.assemblyai.com/docs/llm-gateway/tool-calling
Source: docs/llm-gateway/tool-calling.mdx
Navigation: LLM Gateway > Features
Description: Enable models to execute custom functions to access external data or functionality

## Overview

Tool calling allows you to define functions that the model can call to access external data or perform actions. This enables the model to interact with databases, APIs, and other external systems.

Tool call objects follow an OpenAI-compatible structure nested under `choices[i].message.tool_calls`, so the same parser works across Claude, OpenAI, and Gemini. `finish_reason` is passed through provider-native (`tool_use` / `end_turn` for Claude, `tool_calls` / `stop` for OpenAI), so handle both families when branching on it.

## Defining tools

Define functions the model can call to access external data or functionality:

```python expandable
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_employees",
            "description": "Search employee database by name, department, or other criteria",
            "parameters": {
                "type": "object",
                "properties": {
                    "first_name": {
                        "type": "string",
                        "description": "Employee's first name"
                    },
                    "department": {
                        "type": "string",
                        "description": "Department name"
                    }
                },
                "required": []
            }
        }
    }
]

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers = headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": "Find employees in Engineering"}],
        "tools": tools
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

## Handling tool calls

Execute tools and send results back to continue the conversation:

```python expandable
import requests
import json

headers = {
  "authorization": "<YOUR_API_KEY>"
}

choice = response.json()["choices"][0]

if choice.get("message", {}).get("tool_calls"):
    tool_call = choice["message"]["tool_calls"][0]

    # Add assistant message with tool calls to history
    conversation_history.append({
        "role": "assistant",
        "content": None,
        "tool_calls": [tool_call]
    })

    # Execute your function
    result = execute_function(
        tool_call["function"]["name"],
        json.loads(tool_call["function"]["arguments"])
    )

    # Add tool result to history
    conversation_history.append({
        "role": "tool",
        "tool_call_id": tool_call["id"],
        "content": json.dumps({"result": result})
    })

    # Continue conversation with the result
    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers = headers,
        json = {
            "model": "claude-sonnet-4-6",
            "messages": conversation_history,
            "tools": tools
        }
    )

result = response.json()
print(result["choices"][0]["message"]["content"])
```

## Tool call flow

The complete tool calling process:

1. Send user message with `tools` array defining available functions
2. Model responds with `tool_calls` if it needs to use a tool
3. Execute the tool in your application code
4. Add the assistant message (with `tool_calls`) and a `tool` message (with the result) to conversation history
5. Send updated history back to the API
6. Model incorporates results and either calls more tools or provides final answer

## Message types

When using tools, structure your conversation history with these message types to track the complete interaction flow:

- **user** - Messages from the user
- **assistant** - Messages from the AI model
- **system** - System instructions or context
- **tool** - Results from executing a tool call

## API reference

### Request

The LLM Gateway accepts POST requests to `https://llm-gateway.assemblyai.com/v1/chat/completions` with the following parameters:

```bash expandable
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "Find employees in Engineering"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "search_employees",
          "description": "Search employee database by name, department, or other criteria",
          "parameters": {
            "type": "object",
            "properties": {
              "department": {
                "type": "string",
                "description": "Department name"
              }
            }
          }
        }
      }
    ],
    "max_tokens": 1000
  }'
```

#### Request parameters

| Key           | Type             | Required? | Description                                                                                                               |
| ------------- | ---------------- | --------- | ------------------------------------------------------------------------------------------------------------------------- |
| `model`       | string           | Yes       | The model to use for completion. See [Available models](/llm-gateway/available-models) for supported values. |
| `messages`    | array            | Yes       | An array of message objects representing the conversation history.                                                        |
| `tools`       | array            | Yes       | An array of tool definitions that the model can call.                                                                     |
| `tool_choice` | string or object | No        | Controls which tools the model can call. Options: `"none"`, `"auto"`, or an object specifying a specific function.        |
| `max_tokens`  | number           | No        | The maximum number of tokens to generate. Default: 1000. Range: [1, context_length).                                      |
| `temperature` | number           | No        | Controls randomness in the output. Higher values make output more random. Range: [0, 2].                                  |

#### Message object

| Key            | Type            | Required? | Description                                                                                                    |
| -------------- | --------------- | --------- | -------------------------------------------------------------------------------------------------------------- |
| `role`         | string          | Yes       | The role of the message sender. Valid values: `"user"`, `"assistant"`, `"system"`, or `"tool"`.                |
| `content`      | string or array | Yes       | The message content. Can be a string or an array of content parts for the `"user"` role.                       |
| `name`         | string          | No        | An optional name for the message sender. For non-OpenAI models, this will be prepended as `{name}: {content}`. |
| `tool_call_id` | string          | No\*      | Required when `role` is `"tool"`. The ID of the tool call this message is responding to.                       |

#### Tool object

| Key                    | Type   | Required? | Description                                                 |
| ---------------------- | ------ | --------- | ----------------------------------------------------------- |
| `type`                 | string | Yes       | The type of tool. Currently only `"function"` is supported. |
| `function`             | object | Yes       | The function definition.                                    |
| `function.name`        | string | Yes       | The name of the function.                                   |
| `function.description` | string | No        | A description of what the function does.                    |
| `function.parameters`  | object | Yes       | A JSON Schema object describing the function parameters.    |

#### Tool choice object

The `tool_choice` parameter can be:

- `"none"` - The model will not call any tools
- `"auto"` - The model can choose whether to call tools
- An object with `type: "function"` and a `function.name` to force calling a specific function

### Response

The API returns a JSON response. When the model wants to call a tool:

```json expandable
{
  "request_id": "abc123",
  "choices": [
    {
      "index": 0,
      "finish_reason": "tool_use",
      "message": {
        "role": "assistant",
        "content": "I'll search the engineering department."
      }
    },
    {
      "index": 1,
      "finish_reason": "tool_use",
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_123",
            "type": "function",
            "function": {
              "name": "search_employees",
              "arguments": "{\"department\": \"Engineering\"}"
            }
          }
        ]
      }
    }
  ],
  "request": { "model": "claude-sonnet-4-6", "max_tokens": 1000 },
  "usage": { "input_tokens": 120, "output_tokens": 25, "total_tokens": 145 }
}
```

#### Response fields

| Key                          | Type   | Description                                                                                   |
| ---------------------------- | ------ | --------------------------------------------------------------------------------------------- |
| `request_id`                 | string | A unique identifier for the request.                                                          |
| `choices`                    | array  | An array of completion choices. Typically contains one choice.                                |
| `choices[i].message`         | object | The message object containing the model's response.                                           |
| `choices[i].message.role`    | string | The role of the message, typically `"assistant"`.                                             |
| `choices[i].message.content` | string | The text content of the model's response. May be null when calling tools.                     |
| `choices[i].finish_reason`   | string | The reason the model stopped generating. Common values: `"stop"`, `"length"`, `"tool_calls"` (OpenAI models), `"tool_use"`, `"end_turn"` (Claude models). LLM Gateway passes the provider-native value through; client code should accept either family. |
| `choices[i].message.tool_calls` | array  | Present on assistant message choices when the model calls a tool. Each tool call appears in its own choice with the calling content nested under `message`. |
| `request`                    | object | Echo of the request parameters (excluding `messages`).                                        |
| `usage`                      | object | Token usage statistics for the request.                                                       |
| `usage.input_tokens`         | number | Number of tokens in the prompt.                                                               |
| `usage.output_tokens`        | number | Number of tokens in the completion.                                                           |
| `usage.total_tokens`         | number | Total tokens used (prompt + completion).                                                      |

#### Tool call object

When the model wants to call a tool, the response includes a `tool_calls` array:

| Key                                | Type   | Description                                      |
| ---------------------------------- | ------ | ------------------------------------------------ |
| `tool_calls[i].id`                 | string | A unique identifier for the tool call.           |
| `tool_calls[i].type`               | string | The type of tool call, always `"function"`.      |
| `tool_calls[i].function`           | object | The function call details.                       |
| `tool_calls[i].function.name`      | string | The name of the function to call.                |
| `tool_calls[i].function.arguments` | string | A JSON string containing the function arguments. |

  Models occasionally return malformed JSON in `tool_calls[i].function.arguments`. Add `post_processing_steps: [{"type": "json-repair"}]` to your request to automatically repair common JSON errors. See [Post-processing](/llm-gateway/structured-outputs#post-processing).

### Error response

If an error occurs, the API returns an error response:

```json
{
  "code": 400,
  "message": "invalid request body",
  "request_id": "2a9adf03-c73e-4333-a42d-54b515e6afbd",
  "metadata": {
    "errors": [
      "one of messages or prompt required"
    ]
  }
}
```

| Key               | Type   | Description                                                              |
| ----------------- | ------ | ------------------------------------------------------------------------ |
| `code`            | number | HTTP status code for the error.                                          |
| `message`         | string | A human-readable description of the error.                               |
| `request_id`      | string | Unique identifier for the request. Include this when contacting support. |
| `metadata`        | object | Optional. Present on 400 responses with per-field validation details.    |
| `metadata.errors` | array  | List of specific validation failure messages.                            |

#### Common error codes

| Code | Description                               |
| ---- | ----------------------------------------- |
| 400  | Bad Request - Invalid request parameters  |
| 401  | Unauthorized - Invalid or missing API key |
| 403  | Forbidden - Insufficient permissions      |
| 404  | Not Found - Invalid endpoint or model     |
| 429  | Too Many Requests - Rate limit exceeded   |
| 500  | Internal Server Error - Server-side error |

## Handling malformed tool call arguments

LLMs occasionally return malformed JSON in tool call arguments. Add `post_processing_steps` to your request to automatically repair these before they reach your application:

```json
{
  "post_processing_steps": [{"type": "json-repair"}]
}
```

See [Post-processing](/llm-gateway/structured-outputs#post-processing) for details.

---

<a id="structured-outputs-docs-llm-gateway-structured-outputs-mdx"></a>

# Structured Outputs

URL: https://www.assemblyai.com/docs/llm-gateway/structured-outputs
Source: docs/llm-gateway/structured-outputs.mdx
Navigation: LLM Gateway > Features
Description: Constrain model responses to follow a specific JSON schema for reliable, parseable outputs

## Overview

Structured outputs allow you to constrain the model's response to follow a specific JSON schema. This ensures the model returns data in a predictable format that can be reliably parsed and processed by your application.

  To avoid JSON parse errors, add `post_processing_steps: [{"type": "json-repair"}]` to your request. The LLM Gateway will automatically repair common JSON errors before returning the response. See [Post-processing](#post-processing).

## Getting started

To use structured outputs, include the `response_format` parameter in your request with a `json_schema` type:

```python expandable
import requests

headers = {
    "authorization": "<YOUR_API_KEY>",
    "content-type": "application/json"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gemini-2.5-flash-lite",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful math tutor. Guide the user through the solution step by step."
            },
            {
                "role": "user",
                "content": "how can I solve 8x + 7 = -23"
            }
        ],
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "math_reasoning",
                "schema": {
                    "type": "object",
                    "properties": {
                        "steps": {
                            "type": "array",
                            "items": {
                                "type": "object",
                                "properties": {
                                    "explanation": {"type": "string"},
                                    "output": {"type": "string"}
                                },
                                "required": ["explanation", "output"],
                                "additionalProperties": False
                            }
                        },
                        "final_answer": {"type": "string"}
                    },
                    "required": ["steps", "final_answer"],
                    "additionalProperties": False
                },
                "strict": True
            }
        }
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript expandable
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gemini-2.5-flash-lite",
      messages: [
        {
          role: "system",
          content:
            "You are a helpful math tutor. Guide the user through the solution step by step.",
        },
        {
          role: "user",
          content: "how can I solve 8x + 7 = -23",
        },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "math_reasoning",
          schema: {
            type: "object",
            properties: {
              steps: {
                type: "array",
                items: {
                  type: "object",
                  properties: {
                    explanation: { type: "string" },
                    output: { type: "string" },
                  },
                  required: ["explanation", "output"],
                  additionalProperties: false,
                },
              },
              final_answer: { type: "string" },
            },
            required: ["steps", "final_answer"],
            additionalProperties: false,
          },
          strict: true,
        },
      },
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

```bash expandable
curl -X POST "https://llm-gateway.assemblyai.com/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: <YOUR_API_KEY>" \
    -d '{
          "model": "gemini-2.5-flash-lite",
          "messages": [
            {
              "role": "system",
              "content": "You are a helpful math tutor. Guide the user through the solution step by step."
            },
            {
              "role": "user",
              "content": "how can I solve 8x + 7 = -23"
            }
          ],
          "response_format": {
            "type": "json_schema",
            "json_schema": {
              "name": "math_reasoning",
              "schema": {
                "type": "object",
                "properties": {
                  "steps": {
                    "type": "array",
                    "items": {
                      "type": "object",
                      "properties": {
                        "explanation": { "type": "string" },
                        "output": { "type": "string" }
                      },
                      "required": ["explanation", "output"],
                      "additionalProperties": false
                    }
                  },
                  "final_answer": { "type": "string" }
                },
                "required": ["steps", "final_answer"],
                "additionalProperties": false
              },
              "strict": true
            }
          }
        }'
```

## Example response

When using structured outputs, the model's response will be a JSON string that conforms to your schema:

```json
{
  "request_id": "abc123",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "{\"steps\":[{\"explanation\":\"Start with the equation 8x + 7 = -23\",\"output\":\"8x + 7 = -23\"},{\"explanation\":\"Subtract 7 from both sides to isolate the term with x\",\"output\":\"8x = -30\"},{\"explanation\":\"Divide both sides by 8 to solve for x\",\"output\":\"x = -30/8 = -15/4 = -3.75\"}],\"final_answer\":\"x = -3.75\"}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "input_tokens": 85,
    "output_tokens": 120,
    "total_tokens": 205
  }
}
```

You can parse the content as JSON in your application:

```python
import json

content = result["choices"][0]["message"]["content"]
parsed = json.loads(content)

for step in parsed["steps"]:
    print(f"{step['explanation']}: {step['output']}")

print(f"Final answer: {parsed['final_answer']}")
```

## Supported models

Structured outputs are supported by the following model families:

| Provider                   | Supported |
| -------------------------- | --------- |
| OpenAI (GPT-4.1, GPT-5.x) | Yes       |
| Gemini                     | Yes       |
| Claude (4.5+)              | Yes       |
| Alibaba Cloud Qwen         | Yes       |
| Moonshot AI Kimi           | Yes       |
| gpt-oss                    | No        |

## Post-processing

Post-processing steps let you apply automatic fixes to model responses after generation. You can specify an ordered list of steps in the `post_processing_steps` parameter on any chat completions request. Steps run server-side on all LLM Gateway models in both US and EU regions.

Currently, JSON repair (`json-repair`) is the only supported step type.

### JSON repair

JSON repair corrects common JSON errors — such as trailing commas, unescaped characters, and missing quotes — that LLMs occasionally produce. This is especially useful when using structured outputs or [tool calling](/llm-gateway/tool-calling), where invalid JSON would otherwise require client-side retry logic.

#### Getting started

Add `post_processing_steps` to any chat completions request:

```python expandable
import requests
import json

headers = {
    "authorization": "<YOUR_API_KEY>",
    "content-type": "application/json"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gemini-2.5-flash-lite",
        "messages": [
            {
                "role": "user",
                "content": "Extract the user name and age from: John is 30 years old. Return as JSON."
            }
        ],
        "post_processing_steps": [{"type": "json-repair"}]
    }
)

result = response.json()
parsed = json.loads(result["choices"][0]["message"]["content"])
print(parsed)
```

```javascript expandable
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gemini-2.5-flash-lite",
      messages: [
        {
          role: "user",
          content: "Extract the user name and age from: John is 30 years old. Return as JSON.",
        },
      ],
      post_processing_steps: [{ type: "json-repair" }],
    }),
  }
);

const result = await response.json();
const parsed = JSON.parse(result.choices[0].message.content);
console.log(parsed);
```

#### What JSON repair fixes

The JSON repair step corrects the most common JSON errors produced by LLMs:

| Error type | Example (broken) | After repair |
|---|---|---|
| Trailing comma | `{"name": "John",}` | `{"name": "John"}` |
| Unescaped characters | `{"note": "say "hi""}` | `{"note": "say \"hi\""}` |
| Missing closing bracket | `{"name": "John"` | `{"name": "John"}` |
| Single-quoted strings | `{'name': 'John'}` | `{"name": "John"}` |

The step applies to both message content and tool call arguments in the response. `post_processing_steps` runs independently of `response_format`, so you can combine JSON repair with a `json_schema` for maximum reliability.

  If the JSON cannot be repaired, the request returns an HTTP 500 error. The raw malformed response is never passed through.

## API reference

### Request parameters

The `response_format` parameter controls how the model formats its response:

| Key                           | Type   | Required? | Description                                                              |
| ----------------------------- | ------ | --------- | ------------------------------------------------------------------------ |
| `response_format`             | object | No        | Specifies the format of the model's response.                            |
| `response_format.type`        | string | Yes       | The type of response format. Use `"json_schema"` for structured outputs. |
| `response_format.json_schema` | object | Yes       | The JSON schema configuration object.                                    |

### JSON schema object

| Key                  | Type    | Required? | Description                                                                                  |
| -------------------- | ------- | --------- | -------------------------------------------------------------------------------------------- |
| `json_schema.name`   | string  | Yes       | A name for the schema. Used for identification purposes.                                     |
| `json_schema.schema` | object  | Yes       | A valid JSON Schema object that defines the structure of the expected response.              |
| `json_schema.strict` | boolean | No        | When `true`, the model will strictly adhere to the schema. Recommended for reliable parsing. |

### Schema definition

The `schema` object follows the [JSON Schema](https://json-schema.org/) specification. Common properties include:

| Property               | Type    | Description                                                                |
| ---------------------- | ------- | -------------------------------------------------------------------------- |
| `type`                 | string  | The data type: `"object"`, `"array"`, `"string"`, `"number"`, `"boolean"`. |
| `properties`           | object  | For objects, defines the properties and their schemas.                     |
| `items`                | object  | For arrays, defines the schema for array items.                            |
| `required`             | array   | List of required property names.                                           |
| `additionalProperties` | boolean | When `false`, prevents additional properties not defined in the schema.    |

### Post-processing parameters

| Key | Type | Required? | Description |
|-----|------|-----------|-------------|
| `post_processing_steps` | array | No | An ordered list of post-processing steps to apply to the response. Omit to disable post-processing. |
| `post_processing_steps[i].type` | string | Yes | The step type. Currently `"json-repair"` is supported. |

Supported step types:

| Type          | Applies to                              | Models                 | Regions   |
|---------------|-----------------------------------------|------------------------|-----------|
| `json-repair` | Message content and tool call arguments | All LLM Gateway models | US and EU |

## Best practices

When using structured outputs, keep these recommendations in mind:

Set `strict: true` to ensure the model's response strictly adheres to your schema. This is especially important when your application depends on specific fields being present.

Use `additionalProperties: false` at each level of your schema to prevent the model from adding unexpected fields to the response.

Keep your schemas focused and specific. Complex schemas with many nested levels may increase latency and token usage.

Include clear descriptions in your system or user messages to help the model understand what data to extract or generate for each field.

## Error handling

If the model cannot generate a valid response that matches your schema, you may receive an error or a response that doesn't fully conform to the schema. Always validate the parsed JSON against your expected structure:

```python
import json

try:
    content = result["choices"][0]["message"]["content"]
    parsed = json.loads(content)

    # Validate required fields exist
    if "steps" not in parsed or "final_answer" not in parsed:
        raise ValueError("Missing required fields in response")

except json.JSONDecodeError as e:
    print(f"Failed to parse response as JSON: {e}")
except KeyError as e:
    print(f"Unexpected response structure: {e}")
```

---

<a id="cloud-endpoints-and-data-residency-docs-llm-gateway-cloud-endpoints-and-data-residency-mdx"></a>

# Cloud Endpoints and Data Residency

URL: https://www.assemblyai.com/docs/llm-gateway/cloud-endpoints-and-data-residency
Source: docs/llm-gateway/cloud-endpoints-and-data-residency.mdx
Navigation: LLM Gateway > Features
Description: Choose LLM Gateway endpoints for data residency requirements.

Choose the endpoint that best fits your application's requirements—whether that's using the default US region or ensuring your data stays within the European Union.

## Default Endpoint

The default endpoint (`llm-gateway.assemblyai.com`) processes your LLM Gateway requests in the US region. If you don't specify a base URL, this is the endpoint used by default.

## EU Data Residency

The EU endpoint (`llm-gateway.eu.assemblyai.com`) guarantees your data never leaves the European Union. This is designed for organizations with strict data residency and governance requirements—your request and response data will remain entirely within the EU.

## Endpoints

| Endpoint     | Base URL                                                    | Description          |
| ------------ | ----------------------------------------------------------- | -------------------- |
| US (default) | `https://llm-gateway.assemblyai.com/v1/chat/completions`    | Data stays in the US |
| EU           | `https://llm-gateway.eu.assemblyai.com/v1/chat/completions` | Data stays in the EU |

  Effective July 1, 2026, in-region LLM Gateway requests will increase by 10% as a direct pass-through of provider price increases, with no AssemblyAI upcharge. Opt into [global routing](#global-routing) to keep current pricing.

## Model availability by region

The EU endpoint currently supports Anthropic Claude and Google Gemini models. All other models are available through the US endpoint or [global routing](#global-routing) only.

| Provider                     | US  | EU  | Global      |
| ---------------------------- | --- | --- | ----------- |
| Anthropic Claude             | Yes | Yes | Yes         |
| Google Gemini                | Yes | Yes | Yes         |
| OpenAI GPT                   | Yes | No  | No          |
| Together AI (opt-in)         | No  | No  | Yes         |
| Fireworks AI (opt-in)        | Yes | No  | No          |
| DigitalOcean (opt-in)        | No  | No  | Yes         |

Providers marked **opt-in** are disabled by default. To use them, manually turn on their toggles on the [Data Controls](https://www.assemblyai.com/dashboard/settings/data-controls) page in your dashboard.

For a full list of available models, see the [LLM Gateway Models](/llm-gateway/available-models). For each provider's data retention posture and terms, see [Providers](/llm-gateway/providers).

## Which endpoint should I use?

- **No data residency requirements?** Use the **default endpoint**. No configuration change is needed.
- **Need EU data residency?** Use the **EU endpoint** to ensure your data stays within the European Union. Note that only Claude and Gemini models are available in the EU region.
- **No data residency, compliance, or latency needs, and want cheaper calls?** Opt into [**global routing**](#global-routing) on top of the default endpoint to route requests to the provider's global endpoints at lower cost.

## How to use it

Update your request URL to your preferred endpoint. Select an endpoint tab below to see examples for each.

The US endpoint is the default. No configuration change is required.

```python title="Python"  expandable
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# No URL change needed — US is the default
response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript title="JavaScript" 
// No URL change needed — US is the default
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

```bash title="cURL" 
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "max_tokens": 1000
  }'
```

Use the EU endpoint to keep your data within the European Union.

```python title="Python" highlight={9} 
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.eu.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript title="JavaScript" highlight={3} 
const response = await fetch(
  "https://llm-gateway.eu.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

```bash title="cURL" highlight={2} 
curl -X POST \
  "https://llm-gateway.eu.assemblyai.com/v1/chat/completions" \
  -H "Authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "max_tokens": 1000
  }'
```

## Global routing

Global routing is an opt-in option that routes your request to the provider's global (non-region) endpoints for lower-cost processing. Set `model_region` to `"global"` in your request body to enable it. Omit the parameter for default in-region processing.

`"global"` is the only valid value for `model_region`.

### Who should use global routing?

Global routing is designed for customers who:

- Do **not** have data residency or compliance requirements that tie processing to a specific region.
- Do **not** have strict latency requirements that depend on regional proximity.
- Want lower-cost calls by using the provider's global endpoints.

Global routing is always opt-in. If you don't set `model_region`, requests continue to be processed in-region (US or EU), so data residency and compliance remain the default behavior. If you have data residency, compliance, or latency needs, keep using the default in-region processing on the [US or EU endpoint](#endpoints).

### Availability

Global routing is live for Anthropic Claude models. Support for Google Gemini 3 series models is coming soon.

### Usage tracking

Global-routed usage appears as a new **Global** region in the spend and usage dashboard, separate from US and EU usage.

### How to use it

Include `model_region: "global"` in your request body alongside `model` and `messages`.

```python title="Python" highlight={15}
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "model_region": "global",
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript title="JavaScript" highlight={12}
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      model_region: "global",
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

```bash title="cURL" highlight={13}
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "model_region": "global",
    "max_tokens": 1000
  }'
```

---

<a id="create-a-chat-completion-docs-llm-gateway-api-reference-create-chat-completion-mdx"></a>

# Create a chat completion

URL: https://www.assemblyai.com/docs/llm-gateway/api-reference/create-chat-completion
Source: docs/llm-gateway/api-reference/create-chat-completion.mdx
Navigation: LLM Gateway > API reference
Description: Create a chat completion with LLM Gateway.

Metadata:
- openapi: ../../api-reference/specs/llm-gateway.yaml POST /chat/completions

---

<a id="create-speech-understanding-docs-llm-gateway-api-reference-create-speech-understanding-mdx"></a>

# Create speech understanding

URL: https://www.assemblyai.com/docs/llm-gateway/api-reference/create-speech-understanding
Source: docs/llm-gateway/api-reference/create-speech-understanding.mdx
Navigation: LLM Gateway > API reference
Description: Apply LLM Gateway to audio transcripts.

Metadata:
- openapi: ../../api-reference/specs/llm-gateway.yaml POST /understanding

---

<a id="list-available-models-docs-llm-gateway-api-reference-list-available-models-mdx"></a>

# List available models

URL: https://www.assemblyai.com/docs/llm-gateway/api-reference/list-available-models
Source: docs/llm-gateway/api-reference/list-available-models.mdx
Navigation: LLM Gateway > API reference
Description: List the models available through LLM Gateway.

Metadata:
- openapi: ../../api-reference/specs/llm-gateway.yaml GET /models

---

<a id="improve-latency-docs-llm-gateway-improve-latency-mdx"></a>

# Improve Latency

URL: https://www.assemblyai.com/docs/llm-gateway/improve-latency
Source: docs/llm-gateway/improve-latency.mdx
Navigation: LLM Gateway > Core concepts
Description: Use keep-alive connections to minimize latency

# Summary

A common enhancement to minimize latency for http calls is to make use of a keep-alive connection. This prevents a
lot of the back and forth chatter between the server and your client and reduces latency on subsequent calls.
The further your distance to the server, the more significant the improvement.

Keep-alive is a fundamental HTTP optimization, and it matters because of how connections are established. Every new
HTTPS connection has to complete a TCP handshake and then a TLS handshake before your request can even be sent —
several round trips to the server in total. That setup cost is pure added latency on every call that opens a fresh
connection. With keep-alive, your HTTP client keeps the connection open after a response and reuses it for the next
request, skipping the connection setup entirely. (Servers still close connections that sit idle too long, but for
back-to-back requests you get the reuse for free.)

## How to do this

The mechanism is similar across languages and libraries: create **one** HTTP client (or session) and reuse it for every request, so the underlying TCP/TLS connection is pooled and reused instead of being re-established on every call.
Some libraries may require you to explicitly set a keep-alive connection. In most languages/libraries, creating a new
client per request defeats keep-alive even if the client's default settings otherwise support it.

Some notes on common defaults:

* **Python `requests`** — doesn't pool connections unless you create a `requests.Session()` and reuse the session object across calls.
* **Python `httpx`** — pools connections by default, as long as you reuse the same `httpx.Client()`.
* **OpenAI SDK** — built on `httpx`, so it pools connections by default when you reuse one client instance.
* **Node.js `fetch`** (undici) — pools connections by default; create a custom `Agent` if you want to tune pool size or keep-alive timeouts.
* **Go `net/http`** — keep-alives are on by default, but only help if you reuse the same `http.Client`/`Transport` — a fresh `http.Client{}` per request still pays the handshake every time.
* **Java** — both `java.net.http.HttpClient` and OkHttp pool connections by default when the same client instance is reused across calls.

When architecting your code, create the client once at startup and share that single instance everywhere — either as
a singleton (one global instance the whole app uses) or via dependency injection (the shared client is passed in to
the code that needs it). Both patterns guarantee you never accidentally create a new client per request. If you are
using LLM agents to set up your code for LLM Gateway, you can use the following instructions:

```
Write code that calls the AssemblyAI LLM Gateway using keep-alive connections.

Requirements:
- Create exactly one shared HTTP client (a singleton) and reuse it for every request. Never create a new client per request.
- Base URL: https://llm-gateway.assemblyai.com/v1 — send chat requests to POST /chat/completions.
- Set the "authorization" header to my AssemblyAI API key, read from the ASSEMBLYAI_API_KEY environment variable.
- Include a small example function that sends a chat message and returns the response text.

Docs for reference: https://www.assemblyai.com/docs/llm-gateway/quickstart

Write the code in Python using httpx.
```

You can replace the last line's `Python` and `httpx` with your applicable language/library.

### Simple examples

Below are concrete, copy-pasteable examples for common languages and libraries. Each one builds a single client and sends two requests through it — subsequent requests reuse pooled connections instead of opening new ones.

```python title="requests"
import requests

# Create the Session once and reuse it for every request — this is
# what pools and reuses the underlying TCP/TLS connection.
session = requests.Session()
session.headers.update({"authorization": "<YOUR_API_KEY>"})

def ask(question):
    response = session.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        json={
            "model": "claude-sonnet-4-6",
            "messages": [{"role": "user", "content": question}],
            "max_tokens": 1000,
        },
    )
    return response.json()["choices"][0]["message"]["content"]

# Both calls reuse `session` -> the second call reuses the pooled connection.
print(ask("What is the capital of France?"))
print(ask("What is the capital of Germany?"))
```

```python title="httpx"
import httpx

# httpx.Client pools connections by default. Tune `limits` if you're
# making many concurrent or sequential requests.
client = httpx.Client(
    headers={"authorization": "<YOUR_API_KEY>"},
    limits=httpx.Limits(max_keepalive_connections=20, max_connections=100),
)

def ask(question):
    response = client.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        json={
            "model": "claude-sonnet-4-6",
            "messages": [{"role": "user", "content": question}],
            "max_tokens": 1000,
        },
    )
    return response.json()["choices"][0]["message"]["content"]

print(ask("What is the capital of France?"))
print(ask("What is the capital of Germany?"))
```

```javascript title="fetch (undici)"
import { Agent, fetch } from "undici";

// A single Agent pools and reuses connections across every fetch call
// that uses it. Node's global fetch already pools connections by
// default -- create a custom Agent when you want to tune pool size or
// keep-alive timeouts.
const agent = new Agent({ keepAliveTimeout: 30_000, connections: 10 });

async function ask(question) {
  const response = await fetch(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    {
      method: "POST",
      dispatcher: agent,
      headers: {
        authorization: "<YOUR_API_KEY>",
        "content-type": "application/json",
      },
      body: JSON.stringify({
        model: "claude-sonnet-4-6",
        messages: [{ role: "user", content: question }],
        max_tokens: 1000,
      }),
    }
  );
  const result = await response.json();
  return result.choices[0].message.content;
}

// Both calls share `agent`, so sockets are pooled and reused across subsequent calls.
console.log(await ask("What is the capital of France?"));
console.log(await ask("What is the capital of Germany?"));
```

```javascript title="axios"
import axios from "axios";
import { Agent as HttpsAgent } from "node:https";

// Reusing one Axios instance (backed by one Agent) is what lets later
// requests reuse the pooled connection instead of opening a new
// socket per call.
const client = axios.create({
  baseURL: "https://llm-gateway.assemblyai.com/v1",
  httpsAgent: new HttpsAgent({ keepAlive: true, maxSockets: 10 }),
  headers: { authorization: "<YOUR_API_KEY>" },
});

async function ask(question) {
  const { data } = await client.post("/chat/completions", {
    model: "claude-sonnet-4-6",
    messages: [{ role: "user", content: question }],
    max_tokens: 1000,
  });
  return data.choices[0].message.content;
}

console.log(await ask("What is the capital of France?"));
console.log(await ask("What is the capital of Germany?"));
```

  Both examples work as-is in TypeScript — add types for the response shape if you want them.

```go title="Go"
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"time"
)

// Build one http.Client (and Transport) and share it across every
// request -- that's what lets connections be pooled and reused.
// Keep-alives are on by default; MaxIdleConnsPerHost controls how
// many idle connections per host stay open for reuse.
var client = &http.Client{
	Timeout: 30 * time.Second,
	Transport: &http.Transport{
		MaxIdleConnsPerHost: 10,
		IdleConnTimeout:     90 * time.Second,
	},
}

func ask(question string) (string, error) {
	payload, _ := json.Marshal(map[string]any{
		"model":      "claude-sonnet-4-6",
		"messages":   []map[string]string{{"role": "user", "content": question}},
		"max_tokens": 1000,
	})

	req, err := http.NewRequest(
		"POST",
		"https://llm-gateway.assemblyai.com/v1/chat/completions",
		bytes.NewReader(payload),
	)
	if err != nil {
		return "", err
	}
	req.Header.Set("authorization", "<YOUR_API_KEY>")
	req.Header.Set("content-type", "application/json")

	resp, err := client.Do(req) // reuses the pooled connection
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()

	var result struct {
		Choices []struct {
			Message struct {
				Content string `json:"content"`
			} `json:"message"`
		} `json:"choices"`
	}
	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		return "", err
	}
	return result.Choices[0].Message.Content, nil
}

func main() {
	// Both calls reuse `client` -> the second call reuses the pooled connection.
	first, _ := ask("What is the capital of France?")
	second, _ := ask("What is the capital of Germany?")
	fmt.Println(first)
	fmt.Println(second)
}
```

```java title="java.net.http"
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class LlmGatewayClient {

    // Build one HttpClient and share it -- it owns the connection
    // pool, and reusing it (instead of calling HttpClient.newHttpClient()
    // per request) is what keeps connections alive between requests.
    private static final HttpClient CLIENT = HttpClient.newHttpClient();

    static String ask(String question) throws Exception {
        String body = """
            {"model":"claude-sonnet-4-6","messages":[{"role":"user","content":"%s"}],"max_tokens":1000}
            """.formatted(question);

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://llm-gateway.assemblyai.com/v1/chat/completions"))
                .header("authorization", "<YOUR_API_KEY>")
                .header("content-type", "application/json")
                .POST(HttpRequest.BodyPublishers.ofString(body))
                .build();

        // Reusing CLIENT keeps the underlying connection alive between requests.
        HttpResponse<String> response = CLIENT.send(request, HttpResponse.BodyHandlers.ofString());
        return response.body();
    }

    public static void main(String[] args) throws Exception {
        System.out.println(ask("What is the capital of France?"));
        System.out.println(ask("What is the capital of Germany?"));
    }
}
```

```java title="OkHttp"
import okhttp3.*;

import java.util.concurrent.TimeUnit;

public class LlmGatewayClient {

    private static final MediaType JSON = MediaType.get("application/json");

    // OkHttpClient owns a ConnectionPool. Build one instance (e.g. as
    // a static field) and share it across every call -- creating a
    // new OkHttpClient per request opens a new pool and a new connection.
    private static final OkHttpClient CLIENT = new OkHttpClient.Builder()
            .connectionPool(new ConnectionPool(10, 5, TimeUnit.MINUTES))
            .build();

    static String ask(String question) throws Exception {
        String json = "{\"model\":\"claude-sonnet-4-6\",\"messages\":[{\"role\":\"user\",\"content\":\""
                + question + "\"}],\"max_tokens\":1000}";

        Request request = new Request.Builder()
                .url("https://llm-gateway.assemblyai.com/v1/chat/completions")
                .addHeader("authorization", "<YOUR_API_KEY>")
                .post(RequestBody.create(json, JSON))
                .build();

        // Reusing CLIENT is what lets OkHttp pull a pooled connection
        // instead of opening a new one.
        try (Response response = CLIENT.newCall(request).execute()) {
            return response.body().string();
        }
    }

    public static void main(String[] args) throws Exception {
        System.out.println(ask("What is the capital of France?"));
        System.out.println(ask("What is the capital of Germany?"));
    }
}
```

---

<a id="ask-questions-about-your-audio-transcripts-docs-llm-gateway-ask-questions-mdx"></a>

# Ask Questions About Your Audio Transcripts

URL: https://www.assemblyai.com/docs/llm-gateway/ask-questions
Source: docs/llm-gateway/ask-questions.mdx
Navigation: LLM Gateway > Core concepts
Description: Ask questions about your audio transcripts using LLM Gateway.

In this guide, you'll learn how to use LLM Gateway to ask questions and get answers about your audio transcripts.

If you want a Quickstart, see [Apply LLM Gateway to Audio Transcripts](/llm-gateway/apply-llms-to-audio-files).

## Basic Q&A example

To ask a question about your audio transcript, define a prompt with your questions, reference the transcript text with a `{{ transcript }}` tag, and pass the transcript ID alongside the prompt to the chat completions API.

  Only the first occurrence of `{{ transcript }}` in the first message that contains it is substituted — additional tags or tags in later messages are left as-is. The tag must be exactly `{{ transcript }}` (with the spaces); variants like `{{transcript}}` or `{{ TRANSCRIPT }}` are not substituted. The endpoint returns 404 if the transcript ID does not exist or belongs to a different account.

```python {42-43,45-53} expandable
import requests
import time

# Step 1: Transcribe an audio file.
base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# You can use a local filepath:
# with open("./my-audio.mp3", "rb") as f:
# response = requests.post(base_url + "/v2/upload",
#                         headers=headers,
#                         data=f)
# upload_url = response.json()["upload_url"]

# Or use a publicly-accessible URL:
upload_url = "https://assembly.ai/sports_injuries.mp3"

data = {
  "audio_url": upload_url
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Step 2: Define a prompt with your question(s).
prompt = "What is a runner's knee?"

# Step 3: Send to LLM Gateway.
llm_gateway_data = {
  "model": "claude-sonnet-4-6",
  "messages": [
    {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
  ],
  "transcript_id": transcript_id,
  "max_tokens": 1000
}

result = requests.post(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  headers=headers,
  json=llm_gateway_data
)
print(result.json()["choices"][0]["message"]["content"])
```

  
```javascript {44-45,47-52} expandable
// Step 1: Transcribe an audio file.
const base_url = "https://api.assemblyai.com";

const headers = {
authorization: "<YOUR_API_KEY>",
};

// import fs from 'fs-extra';
// const path = './my-audio.mp3';
// const audioData = await fs.readFile(path)
// const uploadRes = await fetch(`${base_url}/v2/upload`, {
//   method: "POST",
//   headers,
//   body: audioData,
// });
// if (!uploadRes.ok) throw new Error(`Error: ${uploadRes.status}`);
// const uploadResponse = await uploadRes.json();
// const uploadUrl = uploadResponse.upload_url

// Or use a publicly-accessibly URL:
const uploadUrl = "https://assembly.ai/sports_injuries.mp3";

const data = {
audio_url: uploadUrl,
};

let res = await fetch(base_url + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcript_id = response.id;
const polling_endpoint = base_url + `/v2/transcript/${transcript_id}`;

let transcript;
while (true) {
res = await fetch(polling_endpoint, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
transcript = await res.json();

if (transcript.status === "completed") {
break;
} else if (transcript.status === "error") {
throw new Error(`Transcription failed: ${transcript.error}`);
} else {
await new Promise((resolve) => setTimeout(resolve, 3000));
}
}

// Step 2: Define a prompt with your question(s).
const prompt = "What is a runner's knee?";

// Step 3: Send to LLM Gateway.
const llm_gateway_data = {
model: "claude-sonnet-4-6",
messages: [
{ role: "user", content: `${prompt}\n\n{{ transcript }}` }
],
transcript_id: transcript_id,
max_tokens: 1000
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llm_gateway_data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();
console.log(result.choices[0].message.content);

````

### Example output

```plain
Based on the transcript, runner's knee is a condition characterized
by pain behind or around the kneecap. It is caused by overuse,
muscle imbalance and inadequate stretching. Symptoms include pain
under or around the kneecap and pain when walking.
````

## Structured Q&A with LLM Gateway

You can achieve structured question-and-answer outputs by crafting specific prompts that guide the LLM to format responses in a consistent way. Here's how to create structured Q&A responses using LLM Gateway:

```python expandable
import requests
import time

# Step 1: Transcribe the audio
base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/meeting.mp4"
data = {"audio_url": audio_url}
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

# Step 2: Create structured Q&A prompt
questions = [
    "What are the top level KPIs for engineering? (KPI stands for key performance indicator)",
    "How many days has it been since the data team has gotten updated metrics? (Choose from: 1, 2, 3, 4, 5, 6, 7, more than 7)"
]

prompt = f"""Answer the following questions based on the meeting transcript. Format your response as:
Q1: [question]
A1: [answer]
Q2: [question]
A2: [answer]

Questions:
{". ".join([f"{i+1}. {q}" for i, q in enumerate(questions)])}

Context: This is a GitLab meeting to discuss logistics."""

# Step 3: Send to LLM Gateway
llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
    ],
    "transcript_id": transcript_id,
    "max_tokens": 1000
}

result = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)
print(result.json()["choices"][0]["message"]["content"])
```

```javascript expandable
// Step 1: Transcribe the audio
const base_url = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

const audio_url = "https://assembly.ai/meeting.mp4";
const data = { audio_url };

let res = await fetch(base_url + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();
const transcript_id = response.id;
const polling_endpoint = base_url + `/v2/transcript/${transcript_id}`;

let transcript;
while (true) {
  res = await fetch(polling_endpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();
  if (transcript.status === "completed") break;
  if (transcript.status === "error")
    throw new Error(`Transcription failed: ${transcript.error}`);
  await new Promise((resolve) => setTimeout(resolve, 3000));
}

// Step 2: Create structured Q&A prompt
const questions = [
  "What are the top level KPIs for engineering? (KPI stands for key performance indicator)",
  "How many days has it been since the data team has gotten updated metrics? (Choose from: 1, 2, 3, 4, 5, 6, 7, more than 7)",
];

const prompt = `Answer the following questions based on the meeting transcript. Format your response as:
Q1: [question]
A1: [answer]
Q2: [question]
A2: [answer]

Questions:
${questions.map((q, i) => `${i + 1}. ${q}`).join("\n")}

Context: This is a GitLab meeting to discuss logistics.`;

// Step 3: Send to LLM Gateway
const llm_gateway_data = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\n{{ transcript }}` },
  ],
  transcript_id: transcript_id,
  max_tokens: 1000,
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llm_gateway_data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();
console.log(result.choices[0].message.content);
```

## Advanced Q&A with LLM Gateway

For more sophisticated question-answering scenarios, you can create detailed prompts that guide the LLM to produce highly structured and context-aware responses using LLM Gateway's flexible chat completions API.

## More Q&A prompt examples

Try any of these prompts to get started:

| Use case               | Example prompt                                                            |
| ---------------------- | ------------------------------------------------------------------------- |
| Question and answer    | <i>"Identify any patterns or trends based on the transcript"</i>          |
| Closed-ended questions | <i>"Did the customer express a positive sentiment in the phone call?"</i> |
| Sentiment analysis     | <i>"What was the emotional sentiment of the phone call?"</i>              |

For more examples and techniques, experiment with different prompt structures and try various LLM models available through LLM Gateway.

## API reference

- [LLM Gateway Overview](/llm-gateway/quickstart)
- [Basic Chat Completions](/llm-gateway/chat-completions)

## Improve the results

To improve your Q&A results with LLM Gateway:

- **Experiment with different models**: Try Claude, GPT, or Gemini models to find the best fit for your use case
- **Refine your prompts**: Use clear, specific instructions and examples to guide the model's responses
- **Add context**: Provide relevant background information to help the model understand the domain
- **Structure your questions**: Use consistent formatting to get more predictable outputs

---

<a id="agentic-workflows-docs-llm-gateway-agentic-workflows-mdx"></a>

# Agentic Workflows

URL: https://www.assemblyai.com/docs/llm-gateway/agentic-workflows
Source: docs/llm-gateway/agentic-workflows.mdx
Navigation: LLM Gateway > Core concepts
Description: Build multi-step reasoning applications with automatic tool chaining

## Overview

Agentic workflows enable the model to make multiple sequential tool calls for complex multi-step reasoning. The model autonomously decides which tools to use, chains them together, and iterates until it has sufficient information to answer the question.

  Because every iteration depends on the model emitting valid JSON in `tool_calls[i].function.arguments`, a single malformed argument can break the entire chain. Add `post_processing_steps: [{"type": "json-repair"}]` to your request to automatically repair common JSON errors before they reach your loop. See [Post-processing](/llm-gateway/structured-outputs#post-processing).

## Getting started

Enable the model to make multiple sequential tool calls:

```python expandable
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

conversation_history = [
    {"role": "user", "content": "Where does Sarah work and what city is that in?"}
]

max_iterations = 10
iteration = 0

while iteration < max_iterations:
    iteration += 1
    response = chat(conversation_history, model)
    choice = response["choices"][0]

    if choice.get("message", {}).get("tool_calls"):
        # Execute tools and add results to history
        handle_tool_calls(choice["message"]["tool_calls"], conversation_history)
        # Continue loop - model can make more tool calls
        continue
    else:
        # Model has final answer
        print(choice["message"]["content"])
        break
```

## Example interaction

```
You: Where does Andrew live?
🔧 Calling: search_employees(first_name="Andrew")
✅ Result: Found 1 result
🔧 Calling: search_tavily(query="33186 zip code location")
✅ Result: The zip code 33186 is located in Miami, Florida...
Assistant: Andrew lives in Miami, Florida (zip code 33186).
```

## How agentic behavior works

The model autonomously:

- Decides which tools to use based on the question
- Chains multiple tools together (e.g., get employee info → lookup location details)
- Iterates until it has sufficient information to answer
- Handles complex multi-step reasoning automatically

This is achieved by running a loop that continues making requests as long as the model is calling tools.

## API reference

### Request

The LLM Gateway accepts POST requests to `https://llm-gateway.assemblyai.com/v1/chat/completions` with the following parameters:

```bash expandable
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "Where does Sarah work and what city is that in?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "search_employees",
          "description": "Search employee database",
          "parameters": {
            "type": "object",
            "properties": {
              "first_name": {
                "type": "string"
              }
            }
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "search_web",
          "description": "Search the web for information",
          "parameters": {
            "type": "object",
            "properties": {
              "query": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "max_tokens": 1000
  }'
```

#### Request parameters

| Key           | Type             | Required? | Description                                                                                                               |
| ------------- | ---------------- | --------- | ------------------------------------------------------------------------------------------------------------------------- |
| `model`       | string           | Yes       | The model to use for completion. See [Available models](/llm-gateway/available-models) for supported values. |
| `messages`    | array            | Yes       | An array of message objects representing the conversation history.                                                        |
| `tools`       | array            | Yes       | An array of tool definitions that the model can call.                                                                     |
| `tool_choice` | string or object | No        | Controls which tools the model can call. Options: `"none"`, `"auto"`, or an object specifying a specific function.        |
| `max_tokens`  | number           | No        | The maximum number of tokens to generate. Default: 1000. Range: [1, context_length).                                      |
| `temperature` | number           | No        | Controls randomness in the output. Higher values make output more random. Range: [0, 2].                                  |

#### Message object

| Key            | Type            | Required? | Description                                                                                                    |
| -------------- | --------------- | --------- | -------------------------------------------------------------------------------------------------------------- |
| `role`         | string          | Yes       | The role of the message sender. Valid values: `"user"`, `"assistant"`, `"system"`, or `"tool"`.                |
| `content`      | string or array | Yes       | The message content. Can be a string or an array of content parts for the `"user"` role.                       |
| `name`         | string          | No        | An optional name for the message sender. For non-OpenAI models, this will be prepended as `{name}: {content}`. |
| `tool_call_id` | string          | No\*      | Required when `role` is `"tool"`. The ID of the tool call this message is responding to.                       |

#### Tool object

| Key                    | Type   | Required? | Description                                                 |
| ---------------------- | ------ | --------- | ----------------------------------------------------------- |
| `type`                 | string | Yes       | The type of tool. Currently only `"function"` is supported. |
| `function`             | object | Yes       | The function definition.                                    |
| `function.name`        | string | Yes       | The name of the function.                                   |
| `function.description` | string | No        | A description of what the function does.                    |
| `function.parameters`  | object | Yes       | A JSON Schema object describing the function parameters.    |

#### Tool choice object

The `tool_choice` parameter can be:

- `"none"` - The model will not call any tools
- `"auto"` - The model can choose whether to call tools
- An object with `type: "function"` and a `function.name` to force calling a specific function

### Response

The API returns a JSON response. In agentic workflows, the model may make multiple tool calls before providing a final answer.

#### First response (tool call)

```json expandable
{
  "request_id": "abc123",
  "choices": [
    {
      "index": 0,
      "finish_reason": "tool_use",
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_123",
            "type": "function",
            "function": {
              "name": "search_employees",
              "arguments": "{\"first_name\": \"Sarah\"}"
            }
          }
        ]
      }
    }
  ],
  "request": { "model": "claude-sonnet-4-6", "max_tokens": 1000 },
  "usage": { "input_tokens": 150, "output_tokens": 30, "total_tokens": 180 }
}
```

#### Subsequent response (another tool call)

After adding the function result to conversation history:

```json expandable
{
  "request_id": "def456",
  "choices": [
    {
      "index": 0,
      "finish_reason": "tool_use",
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_456",
            "type": "function",
            "function": {
              "name": "search_web",
              "arguments": "{\"query\": \"Acme Corp headquarters location\"}"
            }
          }
        ]
      }
    }
  ],
  "request": { "model": "claude-sonnet-4-6", "max_tokens": 1000 },
  "usage": { "input_tokens": 220, "output_tokens": 35, "total_tokens": 255 }
}
```

#### Final response (answer)

After all tool calls are complete:

```json expandable
{
  "request_id": "ghi789",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Sarah works at Acme Corp, which is headquartered in San Francisco, California."
      },
      "finish_reason": "stop"
    }
  ],
  "request": {
    "model": "claude-sonnet-4-6",
    "max_tokens": 1000
  },
  "usage": {
    "input_tokens": 280,
    "output_tokens": 20,
    "total_tokens": 300
  }
}
```

#### Response fields

| Key                          | Type   | Description                                                                                   |
| ---------------------------- | ------ | --------------------------------------------------------------------------------------------- |
| `request_id`                 | string | A unique identifier for the request.                                                          |
| `choices`                    | array  | An array of completion choices. Typically contains one choice.                                |
| `choices[i].message`         | object | The message object containing the model's response.                                           |
| `choices[i].message.role`    | string | The role of the message, typically `"assistant"`.                                             |
| `choices[i].message.content` | string | The text content of the model's response. May be null when calling tools.                     |
| `choices[i].finish_reason`   | string | The reason the model stopped generating. Common values: `"stop"`, `"length"`, `"tool_calls"` (OpenAI models), `"tool_use"`, `"end_turn"` (Claude models). LLM Gateway passes the provider-native value through; client code should accept either family. |
| `choices[i].message.tool_calls` | array  | Present on assistant message choices when the model calls a tool. Each tool call appears in its own choice with the calling content nested under `message`. |
| `request`                    | object | Echo of the request parameters (excluding `messages`).                                        |
| `usage`                      | object | Token usage statistics for the request.                                                       |
| `usage.input_tokens`         | number | Number of tokens in the prompt.                                                               |
| `usage.output_tokens`        | number | Number of tokens in the completion.                                                           |
| `usage.total_tokens`         | number | Total tokens used (prompt + completion).                                                      |

#### Tool call object

When the model wants to call a tool, the response includes a `tool_calls` array:

| Key                                | Type   | Description                                      |
| ---------------------------------- | ------ | ------------------------------------------------ |
| `tool_calls[i].id`                 | string | A unique identifier for the tool call.           |
| `tool_calls[i].type`               | string | The type of tool call, always `"function"`.      |
| `tool_calls[i].function`           | object | The function call details.                       |
| `tool_calls[i].function.name`      | string | The name of the function to call.                |
| `tool_calls[i].function.arguments` | string | A JSON string containing the function arguments. |

### Error response

If an error occurs, the API returns an error response:

```json
{
  "code": 400,
  "message": "invalid request body",
  "request_id": "2a9adf03-c73e-4333-a42d-54b515e6afbd",
  "metadata": {
    "errors": [
      "one of messages or prompt required"
    ]
  }
}
```

| Key               | Type   | Description                                                              |
| ----------------- | ------ | ------------------------------------------------------------------------ |
| `code`            | number | HTTP status code for the error.                                          |
| `message`         | string | A human-readable description of the error.                               |
| `request_id`      | string | Unique identifier for the request. Include this when contacting support. |
| `metadata`        | object | Optional. Present on 400 responses with per-field validation details.    |
| `metadata.errors` | array  | List of specific validation failure messages.                            |

#### Common error codes

| Code | Description                               |
| ---- | ----------------------------------------- |
| 400  | Bad Request - Invalid request parameters  |
| 401  | Unauthorized - Invalid or missing API key |
| 403  | Forbidden - Insufficient permissions      |
| 404  | Not Found - Invalid endpoint or model     |
| 429  | Too Many Requests - Rate limit exceeded   |
| 500  | Internal Server Error - Server-side error |

---

<a id="basic-chat-completions-docs-llm-gateway-chat-completions-mdx"></a>

# Basic Chat Completions

URL: https://www.assemblyai.com/docs/llm-gateway/chat-completions
Source: docs/llm-gateway/chat-completions.mdx
Navigation: LLM Gateway > Core concepts
Description: Send simple messages and receive responses from LLM models

## Overview

Basic chat completions allow you to send a message and receive a response from the model. This is the simplest way to interact with the LLM Gateway.

## Getting started

Send a message and receive a response:

```python
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers = headers,
    json = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

## Streamed responses

You can stream responses from OpenAI models by setting `stream` to `true`. This returns partial responses as server-sent events (SSE), allowing you to display output as it's generated.

  Streamed responses are currently supported on OpenAI models only.

```python expandable
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gpt-4.1",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "stream": True,
        "max_tokens": 1000
    },
    stream=True
)

for line in response.iter_lines():
    if line:
        print(line.decode("utf-8"))
```

```javascript expandable
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gpt-4.1",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      stream: true,
      max_tokens: 1000,
    }),
  }
);

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  console.log(decoder.decode(value));
}
```

## API reference

### Request

The LLM Gateway accepts POST requests to `https://llm-gateway.assemblyai.com/v1/chat/completions` with the following parameters:

```bash
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "max_tokens": 1000
  }'
```

#### Request parameters

| Key           | Type    | Required? | Description                                                                                                               |
| ------------- | ------- | --------- | ------------------------------------------------------------------------------------------------------------------------- |
| `model`       | string  | Yes       | The model to use for completion. See [Available models](/llm-gateway/available-models) for supported values. |
| `messages`    | array   | Yes\*     | An array of message objects representing the conversation history. Either `messages` or `prompt` is required.             |
| `prompt`      | string  | Yes\*     | A simple string prompt for single request/response interactions. Either `messages` or `prompt` is required.               |
| `stream`      | boolean | No        | When `true`, responses are streamed as server-sent events (SSE). Supported on OpenAI models only.                        |
| `max_tokens`  | number  | No        | The maximum number of tokens to generate. Default: 1000. Range: [1, context_length).                                      |
| `temperature` | number  | No        | Controls randomness in the output. Higher values make output more random. Range: [0, 2].                                  |
| `post_processing_steps` | array | No | An ordered list of post-processing steps to apply to the response. See [Post-processing](/llm-gateway/structured-outputs#post-processing). |
| `transcript_id` | string | No | Inject an AssemblyAI transcript's text into the prompt. The first `{{ transcript }}` tag in the first message that contains it is replaced with the transcript text. See [Inject a transcript by ID](#inject-a-transcript-by-id). |
| `model_region` | string | No | Opt into global routing for lower-cost processing. The only accepted value is `"global"`; omit for default in-region processing. See [Global routing](/llm-gateway/cloud-endpoints-and-data-residency#global-routing). |

#### Message object

| Key       | Type            | Required? | Description                                                                                                    |
| --------- | --------------- | --------- | -------------------------------------------------------------------------------------------------------------- |
| `role`    | string          | Yes       | The role of the message sender. Valid values: `"user"`, `"assistant"`, `"system"`, or `"tool"`.                |
| `content` | string or array | Yes       | The message content. Can be a string or an array of content parts for the `"user"` role.                       |
| `name`    | string          | No        | An optional name for the message sender. For non-OpenAI models, this will be prepended as `{name}: {content}`. |

#### Content part object

| Key    | Type   | Required? | Description                                                |
| ------ | ------ | --------- | ---------------------------------------------------------- |
| `type` | string | Yes       | The type of content. Currently only `"text"` is supported. |
| `text` | string | Yes       | The text content.                                          |

### Response

The API returns a JSON response with the model's completion:

```json expandable
{
  "request_id": "abc123",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "request": {
    "model": "claude-sonnet-4-6",
    "max_tokens": 1000
  },
  "usage": {
    "input_tokens": 15,
    "output_tokens": 8,
    "total_tokens": 23
  }
}
```

#### Response fields

| Key                          | Type   | Description                                                                   |
| ---------------------------- | ------ | ----------------------------------------------------------------------------- |
| `request_id`                 | string | A unique identifier for the request.                                          |
| `choices`                    | array  | An array of completion choices. Typically contains one choice.                |
| `choices[i].message`         | object | The message object containing the model's response.                           |
| `choices[i].message.role`    | string | The role of the message, typically `"assistant"`.                             |
| `choices[i].message.content` | string | The text content of the model's response.                                     |
| `choices[i].finish_reason`   | string | The reason the model stopped generating. Common values: `"stop"`, `"length"`. |
| `request`                    | object | Echo of the request parameters (excluding `prompt` and `messages`).           |
| `usage`                      | object | Token usage statistics for the request.                                       |
| `usage.input_tokens`         | number | Number of tokens in the prompt.                                               |
| `usage.output_tokens`        | number | Number of tokens in the completion.                                           |
| `usage.total_tokens`         | number | Total tokens used (prompt + completion).                                      |

### Inject a transcript by ID

Pass `transcript_id` at the top level of the request to inject a transcript's text into the prompt. The API replaces the first occurrence of the literal tag `{{ transcript }}` in the first message containing it with the transcript's `text` field, then runs the completion.

  Only the first occurrence of `{{ transcript }}` in the first message that contains it is substituted — additional tags or tags in later messages are left as-is. The tag must be exactly `{{ transcript }}` (with the spaces); variants like `{{transcript}}` or `{{ TRANSCRIPT }}` are not substituted. The endpoint returns 404 if the transcript ID does not exist or belongs to a different account.

```bash
curl -X POST "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-2.5-flash-lite",
    "messages": [
      {"role": "user", "content": "hi there"},
      {"role": "assistant", "content": "Hi! How can I help?"},
      {"role": "user", "content": "Here is a transcript: {{ transcript }}. Return the text verbatim."}
    ],
    "transcript_id": "065a71ac-dc3e-4e38-9374-e54c0bea564f"
  }'
```

### Error response

If an error occurs, the API returns an error response:

```json
{
  "code": 400,
  "message": "invalid request body",
  "request_id": "2a9adf03-c73e-4333-a42d-54b515e6afbd",
  "metadata": {
    "errors": [
      "one of messages or prompt required"
    ]
  }
}
```

| Key               | Type   | Description                                                              |
| ----------------- | ------ | ------------------------------------------------------------------------ |
| `code`            | number | HTTP status code for the error.                                          |
| `message`         | string | A human-readable description of the error.                               |
| `request_id`      | string | Unique identifier for the request. Include this when contacting support. |
| `metadata`        | object | Optional. Present on 400 responses with per-field validation details.    |
| `metadata.errors` | array  | List of specific validation failure messages.                            |

#### Common error codes

| Code | Description                               |
| ---- | ----------------------------------------- |
| 400  | Bad Request - Invalid request parameters  |
| 401  | Unauthorized - Invalid or missing API key |
| 403  | Forbidden - Insufficient permissions      |
| 404  | Not Found - Invalid endpoint or model     |
| 429  | Too Many Requests - Rate limit exceeded   |
| 500  | Internal Server Error - Server-side error |

---

<a id="multi-turn-conversations-docs-llm-gateway-conversations-mdx"></a>

# Multi-turn Conversations

URL: https://www.assemblyai.com/docs/llm-gateway/conversations
Source: docs/llm-gateway/conversations.mdx
Navigation: LLM Gateway > Core concepts
Description: Maintain context across multiple exchanges with conversation history

## Overview

Multi-turn conversations allow you to maintain context across multiple exchanges by including conversation history in your API requests. This enables the model to understand and reference previous messages, creating natural, coherent dialogues.

### Why use multi-turn conversations?

With conversation history, you can:

- **Ask follow-up questions** - Ask "What's the population?" and the model knows you're referring to Paris from the previous message
- **Build on previous responses** - Request clarifications, expansions, or corrections without repeating context
- **Create interactive experiences** - Build chatbots, assistants, and conversational interfaces that feel natural

### How it works

Each API request includes an array of previous messages. The model uses this history to understand context and maintain coherence across the conversation:

```python
# First exchange
messages = [
    {"role": "user", "content": "What is the capital of France?"}
]
# Response: "The capital of France is Paris."

# Second exchange - model remembers Paris
messages = [
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
    {"role": "user", "content": "What's the population?"}
]
# Response: "As of the latest estimates, the population of Paris is approximately 2.2 million..."
```

**Note:** You're responsible for managing conversation history. Each request must include all relevant previous messages - the API doesn't store history between requests.

## Getting started

Maintain context by including conversation history:

```python expandable
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

conversation_history = [
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
    {"role": "user", "content": "What's the population?"}
]

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers = headers,
    json = {
        "model": "claude-sonnet-4-6",
        "messages": conversation_history,
        "max_tokens": 1000
    }
)

result = response.json()
agent_response = result["choices"][0]["message"]["content"]
print(agent_response)

# Append the assistant's response to conversation history
conversation_history.append({"role": "assistant", "content": agent_response})
```

```javascript expandable
const headers = {
  authorization: "<YOUR_API_KEY>",
  "Content-Type": "application/json",
};

const conversation_history = [
  { role: "user", content: "What is the capital of France?" },
  { role: "assistant", content: "The capital of France is Paris." },
  { role: "user", content: "What's the population?" },
];

fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: headers,
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: conversation_history,
    max_tokens: 1000,
  }),
})
  .then((response) => response.json())
  .then((result) => {
    const agent_response = result.choices[0].message.content;
    console.log(agent_response);

    // Append the assistant's response to conversation history
    conversation_history.push({ role: "assistant", content: agent_response });
  })
  .catch((error) => {
    console.error("Error:", error);
  });
```

## Message types

When building conversation history, you can use the following message types:

- **user** - Messages from the user
- **assistant** - Messages from the AI model
- **system** - System instructions or context

Structure your conversation history with these message types to track the complete interaction flow between the user and model.

## API reference

### Request

The LLM Gateway accepts POST requests to `https://llm-gateway.assemblyai.com/v1/chat/completions` with the following parameters:

```bash expandable
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      },
      {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      {
        "role": "user",
        "content": "What'\''s the population?"
      }
    ],
    "max_tokens": 1000
  }'
```

#### Request parameters

| Key           | Type   | Required? | Description                                                                                                               |
| ------------- | ------ | --------- | ------------------------------------------------------------------------------------------------------------------------- |
| `model`       | string | Yes       | The model to use for completion. See [Available models](/llm-gateway/available-models) for supported values. |
| `messages`    | array  | Yes       | An array of message objects representing the conversation history.                                                        |
| `max_tokens`  | number | No        | The maximum number of tokens to generate. Default: 1000. Range: [1, context_length).                                      |
| `temperature` | number | No        | Controls randomness in the output. Higher values make output more random. Range: [0, 2].                                  |

#### Message object

| Key       | Type            | Required? | Description                                                                                                    |
| --------- | --------------- | --------- | -------------------------------------------------------------------------------------------------------------- |
| `role`    | string          | Yes       | The role of the message sender. Valid values: `"user"`, `"assistant"`, `"system"`, or `"tool"`.                |
| `content` | string or array | Yes       | The message content. Can be a string or an array of content parts for the `"user"` role.                       |
| `name`    | string          | No        | An optional name for the message sender. For non-OpenAI models, this will be prepended as `{name}: {content}`. |

#### Content part object

| Key    | Type   | Required? | Description                                                |
| ------ | ------ | --------- | ---------------------------------------------------------- |
| `type` | string | Yes       | The type of content. Currently only `"text"` is supported. |
| `text` | string | Yes       | The text content.                                          |

### Response

The API returns a JSON response with the model's completion:

```json expandable
{
  "request_id": "abc123",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "As of the latest estimates, the population of Paris is approximately 2.2 million people within the city proper, and around 12 million in the greater metropolitan area."
      },
      "finish_reason": "stop"
    }
  ],
  "request": {
    "model": "claude-sonnet-4-6",
    "max_tokens": 1000
  },
  "usage": {
    "input_tokens": 45,
    "output_tokens": 35,
    "total_tokens": 80
  }
}
```

#### Response fields

| Key                          | Type   | Description                                                                   |
| ---------------------------- | ------ | ----------------------------------------------------------------------------- |
| `request_id`                 | string | A unique identifier for the request.                                          |
| `choices`                    | array  | An array of completion choices. Typically contains one choice.                |
| `choices[i].message`         | object | The message object containing the model's response.                           |
| `choices[i].message.role`    | string | The role of the message, typically `"assistant"`.                             |
| `choices[i].message.content` | string | The text content of the model's response.                                     |
| `choices[i].finish_reason`   | string | The reason the model stopped generating. Common values: `"stop"`, `"length"`. |
| `request`                    | object | Echo of the request parameters (excluding `messages`).                        |
| `usage`                      | object | Token usage statistics for the request.                                       |
| `usage.input_tokens`         | number | Number of tokens in the prompt.                                               |
| `usage.output_tokens`        | number | Number of tokens in the completion.                                           |
| `usage.total_tokens`         | number | Total tokens used (prompt + completion).                                      |

### Error response

If an error occurs, the API returns an error response:

```json
{
  "code": 400,
  "message": "invalid request body",
  "request_id": "2a9adf03-c73e-4333-a42d-54b515e6afbd",
  "metadata": {
    "errors": [
      "one of messages or prompt required"
    ]
  }
}
```

| Key               | Type   | Description                                                              |
| ----------------- | ------ | ------------------------------------------------------------------------ |
| `code`            | number | HTTP status code for the error.                                          |
| `message`         | string | A human-readable description of the error.                               |
| `request_id`      | string | Unique identifier for the request. Include this when contacting support. |
| `metadata`        | object | Optional. Present on 400 responses with per-field validation details.    |
| `metadata.errors` | array  | List of specific validation failure messages.                            |

#### Common error codes

| Code | Description                               |
| ---- | ----------------------------------------- |
| 400  | Bad Request - Invalid request parameters  |
| 401  | Unauthorized - Invalid or missing API key |
| 403  | Forbidden - Insufficient permissions      |
| 404  | Not Found - Invalid endpoint or model     |
| 429  | Too Many Requests - Rate limit exceeded   |
| 500  | Internal Server Error - Server-side error |

---

<a id="prompt-engineering-for-llm-gateway-docs-llm-gateway-prompt-engineering-mdx"></a>

# Prompt Engineering for LLM Gateway

URL: https://www.assemblyai.com/docs/llm-gateway/prompt-engineering
Source: docs/llm-gateway/prompt-engineering.mdx
Navigation: LLM Gateway > Core concepts
Description: Learn how to improve your prompts for LLM Gateway.

Learn how to get better results from LLM Gateway by using prompting techniques to optimize your prompts. This page walks you through the parts of a prompt and gives you tips on how to use them to improve your interactions with various LLM models.

  **Large language models**

  If you want to learn about LLMs first, see our blog post [Introduction to
  Large Language
  Models](https://www.assemblyai.com/blog/introduction-large-language-models-generative-ai/).

  **Prompt engineering**

  Writing good prompts is both an art and a science that has given rise to an entire field called _prompt engineering_.

  If you want to learn more, see our crash course on prompt engineering:

  <iframe
    width="560"
    height="315"
    src="https://www.youtube.com/embed/aOm75o2Z5-o?si=bDWFpSPQ0PLpW6kK"
    title="Prompt engineering crash course"
    frameborder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowfullscreen
  ></iframe>

## Anatomy of a prompt

Through a learning process called instruction fine-tuning, modern LLMs are now capable of performing specific tasks based on explicit instructions, also called _prompts_.

A prompt is a piece of text that guides the LLM to generate a response.

It can be as simple as a single sentence:

```
Provide a summary of the meeting.
```

However, if you want to improve your response, you can include additional information to help the LLM generate more desirable results.

A prompt can be broken down into three parts:

- **Instruction:** The instruction, or question, that you want the LLM to help you with.
- **Context:** Additional information that help the LLM understand the instruction.
- **Format:** The format of the response that you want the LLM to generate.

At a minimum, a prompt should include an instruction. However, by specifying context and the desired format, you can greatly improve the quality of the response.

If you include at least one of context and format, use the following format:

```plain
<YOUR_INSTRUCTION>

Context:
<YOUR_CONTEXT>

Answer Format:
<YOUR_FORMAT>
```

Replace `<YOUR_INSTRUCTION>`, `<YOUR_CONTEXT>`, and `<YOUR_FORMAT>` with your own text.

In the following sections, you'll learn about each part in more detail.

## Instruction

A good instruction is clear and concise. It should tell the LLM exactly what you want it to generate. It should also be short enough that the LLM can understand it.

### Prefer instructions over questions

Questions can often be ambiguous and may lead to unexpected results. Instead, use instructions starting with an imperative verb to tell the LLM what you want.

- **Recommended:** `List the key features of the product.`
- **Not recommended:** `What are the features of the product?`

### Avoid compound instructions

Focus on one task at a time. Including multiple instructions in a single prompt may confuse the LLM and lead to less accurate results.

- **Recommended:** `Identify action items from the meeting.`
- **Not recommended:** `Identify action items from the meeting and list highlights from each speaker.`

## Context

Add a `Context:` section to your prompt to provide additional information to the LLM. This can help the LLM generate more accurate results.

  **Transcription text**

  With LLM Gateway, you'll include the transcription text directly in your chat
  message. This gives you full control over how the transcript context is
  provided to the model.

### Provide definitions

You can use context to define any terms that the LLM may not understand.

```plain
Identify action items from the meeting.

Context:
Action items are tasks for the participants to complete after the meeting.
```

### Provide examples

Rather than telling the LLM what to generate, you can show it examples of the output you want to generate.

```plain
Identify action items from the meeting.

Context:
Action item examples from other meetings:
- Schedule a follow-up meeting with the client to address their concerns.
- Review the proposal and provide your feedback by the end of the week.
- Complete the data analysis and share the results with the team.
```

## Format

You can tell the LLM how the format should look like by adding an `Answer Format:` section to your prompt.

```plain
Provide a summary of the podcast.

Context:
This is an episode of the Lex Fridman podcast with guest Sam Altman, the CEO of OpenAI.

Answer Format:
catchy title, no longer than 10 words
```

You can also use the answer format to generate structured data, such as JSON or Markdown.

**JSON:**

```plain
Identify action items from the meeting.

Answer Format:
{
  "assignee": <assignee>,
  "action_item": <action item>,
  "due_date": <due_date>
}
```

**Markdown:**

```plain
Provide a summary of the meeting.

Context:
The AssemblyAI marketing team is meeting to discuss a new marketing campaign.

Answer Format:
**<topic header>**
<topic summary>
```

### Use tags to define placeholders

Use tags to insert dynamic content into your prompt. A tag is a description of a piece of information, surrounded by angle brackets, `<...>`.

In the format section of your prompt, place the tag where you want to insert the value.

```plain
Answer Format:
<OVERALL SUMMARY>

Number of participants: <NUMBER OF PARTICIPANTS>
```

### Remove the preamble

Sometimes the LLM generates a response that includes an introduction to the result, also known as a _preamble_. For example:

```
Here's a summary of the meeting:

...
```

To remove the preamble, you can ask the LLM to leave it out:

```
Provide a summary of the meeting. Do not provide a preamble.
```

## Learn more

- [LLM Gateway Overview](/llm-gateway/quickstart) - Learn about available models and capabilities
- [Apply LLM Gateway to Audio Transcripts](/llm-gateway/apply-llms-to-audio-files) - Complete guide to using LLM Gateway

For more advanced prompt engineering techniques, see [Awesome Prompt Engineering](https://github.com/promptslab/Awesome-Prompt-Engineering) on GitHub.

---

<a id="rate-limits-docs-llm-gateway-rate-limits-mdx"></a>

# Rate limits

URL: https://www.assemblyai.com/docs/llm-gateway/rate-limits
Source: docs/llm-gateway/rate-limits.mdx
Navigation: LLM Gateway > Core concepts
Description: Rate limits documentation.

LLM Gateway is rate limited per model, measured as requests within a 60-second window.

| Account type | Rate limit (requests/min, per model) |
| ------------ | ------------------------------------- |
| Free         | Not available                          |
| Paid         | 30                                      |

  **Need a higher rate limit?**

  If you need a higher rate limit, [contact our support team](mailto:support@assemblyai.com).

## Response headers

If you exceed the limit, the API responds with a `429` status code. To see your remaining quota, check the following response headers:

| Header                  | Description                                                                                |
| ----------------------- | ------------------------------------------------------------------------------------------ |
| `X-RateLimit-Limit`     | Maximum number of allowed requests in a 60 second window.                                  |
| `X-RateLimit-Remaining` | Number of remaining requests in the current time window.                                   |
| `X-RateLimit-Reset`     | Number of seconds until the remaining requests resets to the value of `X-RateLimit-Limit`. |
| `X-RateLimit-Model`     | The model the rate limit applies to. Same as "model" param in the request.                 |
| `X-RateLimit-Service`   | Denotes the service used, LLM Gateway or Speech Understanding.                             |

If the response doesn't include `X-RateLimit` headers, the endpoint doesn't have rate limits.

---

<a id="troubleshooting-docs-llm-gateway-troubleshooting-mdx"></a>

# Troubleshooting

URL: https://www.assemblyai.com/docs/llm-gateway/troubleshooting
Source: docs/llm-gateway/troubleshooting.mdx
Navigation: LLM Gateway > Core concepts
Description: Common issues and fixes when using the LLM Gateway.

## What to log for support

Every LLM Gateway response includes a `request_id` — a unique identifier for that specific request. **Log this ID for every call**, not just when something goes wrong. When you reach out to [support@assemblyai.com](mailto:support@assemblyai.com), including the `request_id` lets us find the exact request in our logs in seconds.

At minimum, capture the following for every request:

- `request_id` from the response body
- The `model` parameter used
- The API region (US: `llm-gateway.assemblyai.com`, EU: `llm-gateway.eu.assemblyai.com`)
- A timestamp for when the request was sent
- The full HTTP status code and response body when a non-2xx response is returned

A minimal logging example:

```python
import requests
import time

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers={"authorization": "<YOUR_API_KEY>"},
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": "What is the capital of France?"}],
        "max_tokens": 1000,
    },
)

result = response.json()
log_entry = {
    "timestamp": time.time(),
    "region": "us",
    "model": "claude-sonnet-4-6",
    "status_code": response.status_code,
    "request_id": result.get("request_id"),
    "error": result.get("error"),
}
print(log_entry)
```

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log({
  timestamp: Date.now(),
  region: "us",
  model: "claude-sonnet-4-6",
  status_code: response.status,
  request_id: result.request_id,
  error: result.error,
});
```

---

## Authentication errors (401 / 403)

**Symptom:** The API responds with `401 Unauthorized` or `403 Forbidden`.

```json
{
  "error": "Authentication error, API token missing/invalid",
  "status": "error",
  "request_id": "6e6f340d-6580-4cd4-a3f9-d6cb7323a9bd"
}
```

**Causes:**

- API key is missing, malformed, or expired.
- API key is from a different account or region.
- The `Authorization` header is misspelled (e.g. `Authorisation` or missing the header entirely).

**Fixes:**

- Confirm your API key on the [API Keys page](https://www.assemblyai.com/dashboard/home).
- Pass the key in the `Authorization` header — not as a query parameter and not prefixed with `Bearer`.
- If you're using EU data residency, make sure the key was generated for the EU region. See [Cloud endpoints and data residency](/llm-gateway/cloud-endpoints-and-data-residency).

---

## Bad request (400)

**Symptom:** The API responds with `400 Bad Request`.

```json
{
  "code": 400,
  "message": "invalid request body",
  "request_id": "2a9adf03-c73e-4333-a42d-54b515e6afbd",
  "metadata": {
    "errors": [
      "one of messages or prompt required"
    ]
  }
}
```

**Causes:**

- A required field is missing (`model`, plus either `messages` or `prompt`).
- The `model` value is not a recognized model ID — see [Available models](/llm-gateway/available-models).
- `max_tokens` is outside the valid range or exceeds the model's context window.
- A field is the wrong type (e.g. `messages` sent as a string instead of an array).

**Fixes:**

- Validate your request payload against the [Basic chat completions reference](/llm-gateway/chat-completions#api-reference).
- Check the `metadata.errors` array in the response — it lists every field that failed validation.

**Validation errors in detail**

The `metadata.errors` array lists every field that failed validation:

| String | Meaning |
|--------|---------|
| `"one of messages or prompt required"` | Neither `messages` nor `prompt` was provided |
| `"model {model} is not supported"` | The `model` value is not a recognized model ID |
| `"model context limit exceeded"` | The input exceeds the model's context window |
| `"model_region can only be set to global"` | `model_region` was set to a value other than `"global"` |
| `"fallback_config depth cannot be greater than 2"` | `fallback_config.depth` exceeds the maximum of 2 |
| `"response_format is invalid: {detail}"` | The `response_format` object failed schema validation |

---

## Rate limit exceeded (429)

**Symptom:** The API responds with `429 Too Many Requests`.

**Cause:** You exceeded the per-model rate limit within a 60-second window. Each model has its own limit.

**Fixes:**

- Read the rate limit headers on every response (`X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`) to back off gracefully. See [Rate limits](/llm-gateway/rate-limits#response-headers) for the full header reference.
- Implement exponential backoff with jitter when you receive a 429.
- Consider [specifying fallback models](/llm-gateway/fallback) so traffic spills over to a different model when the primary is rate-limited.
- If you need a higher rate limit, [contact support](mailto:support@assemblyai.com).

---

## Transcript not found (404)

**Symptom:** The API responds with `404 Not Found` when you pass a `transcript_id` that can't be found.

```json
{
  "code": 404,
  "message": "transcript not found",
  "request_id": "bf08febb-ee48-4ce1-b473-7c7b15561033"
}
```

**Causes:**

- The `transcript_id` belongs to a different account or was created under a different API key.
- The transcript was deleted (by you or via a data retention policy). In that case the message will be `"transcript deleted"`.
- The transcript was created in a different region — US transcripts are not accessible from the EU endpoint and vice versa.

**Fixes:**

- Confirm the transcript ID is correct and belongs to the account associated with the API key you're using.
- If the transcript was deleted, re-transcribe the audio and use the new transcript ID.
- Make sure the LLM Gateway region matches the region where the transcript was created.

---

## Server errors (5xx)

**Symptom:** The API responds with `500`, `502`, `503`, or `504`.

**Causes:**

- Transient issues on AssemblyAI's side or with the upstream model provider.
- The upstream provider returned a timeout or unavailable response.

**Fixes:**

- Retry with exponential backoff and jitter. Most 5xx errors are transient.
- Check the [AssemblyAI Status page](https://status.assemblyai.com) for ongoing incidents.
- If the error persists, [contact support](mailto:support@assemblyai.com) with the `request_id`, the model used, the timestamp, and the full error response body.

---

## Streamed responses don't appear

**Symptom:** You set `stream: true` but receive a single non-streamed response — or no response at all.

**Causes:**

- Streaming is currently supported on OpenAI models only. Other providers ignore the `stream` flag and return a regular response.
- The HTTP client isn't reading the response body as a stream of server-sent events (SSE).

**Fixes:**

- Confirm the model is from OpenAI. See [Available models](/llm-gateway/available-models).
- Use a client that reads SSE chunks (e.g. `response.iter_lines()` in Python `requests`, or the streaming `fetch` body reader in JavaScript). See [Basic chat completions — Streamed responses](/llm-gateway/chat-completions#streamed-responses).

---

## Unexpected output or quality issues

**Symptom:** The model returns content you didn't expect — wrong format, wrong language, hallucinations, or refusals.

**Fixes:**

- Capture the full request payload (model, messages, parameters), the full response, and the `request_id`. Send all three to [support@assemblyai.com](mailto:support@assemblyai.com) — quality issues are difficult to diagnose without the exact prompt.
- For structured output, use [Structured outputs](/llm-gateway/structured-outputs) with a JSON schema rather than prompting for JSON in free text.
- For malformed JSON, enable [Post-processing](/llm-gateway/structured-outputs#post-processing) to automatically repair responses.
- Try a different model. Quality varies across models; see [Available models](/llm-gateway/available-models) for the full list.

---

## Contacting support

If you've worked through the steps above and still need help, email [support@assemblyai.com](mailto:support@assemblyai.com) with:

- The `request_id` from the failing response (or several, for intermittent issues)
- The `model` parameter used
- The API region (US or EU)
- A timestamp for when the request was sent
- The HTTP status code and full error response body
- A minimal reproducible example of the request payload (with your API key redacted)

---

<a id="apply-llm-gateway-to-audio-transcripts-docs-llm-gateway-apply-llms-to-audio-files-mdx"></a>

# Apply LLM Gateway to Audio Transcripts

URL: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files
Source: docs/llm-gateway/apply-llms-to-audio-files.mdx
Navigation: LLM Gateway > Voice AI Guides
Description: Learn how to analyze audio transcripts using LLM Gateway.

## Overview

A Large Language Model (LLM) is a machine learning model that uses natural language processing (NLP) to generate text. [LLM Gateway](/llm-gateway/quickstart) is a unified API that provides access to 25+ models from Claude, GPT, Gemini, and more through a single interface. You can use LLM Gateway to analyze audio transcripts, for example to ask questions about a call, or to summarize a meeting.

By the end of this tutorial, you'll be able to use LLM Gateway to summarize an audio file.

Here's the full sample code for what you'll build in this tutorial:

```python expandable
import requests
import time

# Step 1: Transcribe the audio
base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# You can use a local filepath:
# with open("./my-audio.mp3", "rb") as f:
# response = requests.post(base_url + "/v2/upload",
# headers=headers,
# data=f)
# upload_url = response.json()["upload_url"]
# Or use a publicly-accessible URL:

upload_url = "https://assembly.ai/sports_injuries.mp3"

data = {
  "audio_url": upload_url
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Step 2: Send transcript to LLM Gateway
prompt = "Provide a brief summary of the transcript."

llm_gateway_data = {
  "model": "claude-sonnet-4-6",
  "messages": [
    {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
  ],
  "transcript_id": transcript_id,
  "max_tokens": 1000
}

response = requests.post(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  headers=headers,
  json=llm_gateway_data
)
print(response.json()["choices"][0]["message"]["content"])
```

```javascript expandable
import fs from "fs-extra";

// Step 1: Transcribe the audio
const base_url = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${base_url}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();

const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL of an audio or video file on the web
};

res = await fetch(base_url + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcript_id = response.id;
const polling_endpoint = base_url + `/v2/transcript/${transcript_id}`;

let transcript;
while (true) {
  res = await fetch(polling_endpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();

  if (transcript.status === "completed") {
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Step 2: Send transcript to LLM Gateway
const prompt = "Provide a brief summary of the transcript.";

const llm_gateway_data = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\n{{ transcript }}` },
  ],
  transcript_id: transcript_id,
  max_tokens: 1000,
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llm_gateway_data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();
console.log(result.choices[0].message.content);
```

If you run the code above, you'll see the following output:

```plain
The transcript describes several common sports injuries - runner's knee,
sprained ankle, meniscus tear, rotator cuff tear, and ACL tear. It provides
definitions, causes, and symptoms for each injury. The transcript seems to be
narrating sports footage and describing injuries as they occur to the athletes.
Overall, it provides an overview of these common sports injuries that can result
from overuse or sudden trauma during athletic activities
```

## Before you begin

To complete this tutorial, you need:

- [Python](https://www.python.org/) or [Node](https://nodejs.org/en) installed.
- An <a href="https://www.assemblyai.com/dashboard/signup" target="_blank">AssemblyAI account with a credit card set up</a>.
- Basic understanding of how to [Transcribe an audio file](/pre-recorded-audio/getting-started/transcribe-an-audio-file).

## Step 1: Install prerequisites

Install the package via pip:

```bash
pip install requests
```

No additional packages are required. This guide uses the built-in [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API).

## Step 2: Transcribe an audio file

When you pass `transcript_id` to LLM Gateway and include the `{{ transcript }}` tag in a message, the API substitutes the tag with the transcript's `text` field before running the completion. It does not include other fields such as utterances or speaker labels. If you need speaker-separated context, format the utterances yourself and include them in your prompt.

In this step, you'll transcribe an audio file that you can later use with LLM Gateway.

For more information about transcribing audio, see [Transcribe an audio file](/pre-recorded-audio/getting-started/transcribe-an-audio-file).

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {"authorization": "<YOUR_API_KEY>"}

# You can use a local filepath:
# with open("./my-audio.mp3", "rb") as f:
#     response = requests.post(base_url + "/v2/upload", headers=headers, data=f)
#     upload_url = response.json()["upload_url"]

# Or use a publicly-accessible URL:
upload_url = "https://assembly.ai/sports_injuries.mp3"

data = {"audio_url": upload_url}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript["status"] == "completed":
        break

    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    else:
        time.sleep(3)
```

```javascript expandable
import fs from "fs-extra";

const base_url = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${base_url}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL of an audio or video file on the web
};

res = await fetch(base_url + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcript_id = response.id;
const polling_endpoint = base_url + `/v2/transcript/${transcript_id}`;

while (true) {
  res = await fetch(polling_endpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcript = await res.json();

  if (transcript.status === "completed") {
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  **Use existing transcript**

If you've already transcribed an audio file you want to use, you can get an existing transcript using its ID. You can find the ID for previously transcribed audio files in the <a href="https://www.assemblyai.com/dashboard/home" target="_blank">Processing queue</a>.

```python
transcript = requests.get("https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID", headers=headers).json()
```

```javascript
const res = await fetch("https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID", { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcript = await res.json();
```

## Step 3: Send transcript to LLM Gateway

In this step, you'll send the transcript ID to LLM Gateway along with a prompt to generate text output.

The prompt is a text string that provides the LLM with instructions on how to generate the text output. You'll write a prompt that references the transcript with a `{{ transcript }}` tag and send it to LLM Gateway using the chat completions API. Pass the transcript ID as the top-level `transcript_id` field — the API substitutes the tag with the transcript's text before running the completion.

  Only the first occurrence of `{{ transcript }}` in the first message that contains it is substituted — additional tags or tags in later messages are left as-is. The tag must be exactly `{{ transcript }}` (with the spaces); variants like `{{transcript}}` or `{{ TRANSCRIPT }}` are not substituted. The endpoint returns 404 if the transcript ID does not exist or belongs to a different account.

Write a prompt with instructions on how the LLM should generate the text output.

```python
prompt = "Provide a brief summary of the transcript."
```

Write a prompt with instructions on how the LLM should generate the text output.

```javascript
const prompt = "Provide a brief summary of the transcript.";
```

Send the transcript ID and prompt to LLM Gateway. The model parameter defines which LLM to use. For available models, see [LLM Gateway Overview](/llm-gateway/quickstart).

```python
llm_gateway_data = {
  "model": "claude-sonnet-4-6",
  "messages": [
    {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
  ],
  "transcript_id": transcript_id,
  "max_tokens": 1000
}

result = requests.post(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  headers=headers,
  json=llm_gateway_data
)
```

Send the transcript ID and prompt to LLM Gateway. The model parameter defines which LLM to use. For available models, see [LLM Gateway Overview](/llm-gateway/quickstart).

```javascript
const llm_gateway_data = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\n{{ transcript }}` },
  ],
  transcript_id: transcript_id,
  max_tokens: 1000,
};

let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llm_gateway_data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();
```

Print the result.

```python
print(result.json()["choices"][0]["message"]["content"])
```

```javascript
console.log(result.choices[0].message.content);
```

The output will look something like this:

```
 The transcript describes several common sports injuries - runner's knee,
 sprained ankle, meniscus tear, rotator cuff tear, and ACL tear. It provides
 definitions, causes, and symptoms for each injury. The transcript seems to be
 narrating sports footage and describing injuries as they occur to the athletes.
 Overall, it provides an overview of these common sports injuries that can
 result from overuse or sudden trauma during athletic activities
```

  Want to make your LLM requests more resilient? Use [fallback models](/llm-gateway/fallback) to automatically switch to a backup model if your primary model is unavailable.

## Next steps

In this tutorial, you've learned how to generate LLM output based on your audio transcripts using LLM Gateway. The type of output depends on your prompt, so try exploring different prompts to see how they affect the output. Here's a few more prompts to try.

- "Provide an analysis of the transcript and offer areas to improve with exact quotes."
- "What's the main take-away from the transcript?"
- "Generate a set of action items from this transcript."

To learn more about LLM Gateway and working with different models, see the following resources:

- [LLM Gateway Overview](/llm-gateway/quickstart)
- [Basic Chat Completions](/llm-gateway/chat-completions)
- [Multi-turn Conversations](/llm-gateway/conversations)

## Need some help?

If you get stuck, or have any other questions, we'd love to help you out. Contact our support team at support@assemblyai.com or create a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="setup-an-ai-coach-with-llm-gateway-docs-guides-task-endpoint-ai-coach-mdx"></a>

# Setup An AI Coach With LLM Gateway

URL: https://www.assemblyai.com/docs/guides/task-endpoint-ai-coach
Source: docs/guides/task-endpoint-ai-coach.mdx
Navigation: LLM Gateway > Voice AI Guides > Basic LLM Gateway workflows
Description: Setup An AI Coach With LLM Gateway documentation.

This tutorial will demonstrate how to use AssemblyAI's [LLM Gateway](/llm-gateway) framework to receive AI coaching. LLM Gateway provides access to multiple LLM providers through a unified API.

## Quickstart

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

# Use a publicly-accessible URL:
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"

# with open("/your_audio_file.mp3", "rb") as f:
#     response = requests.post(base_url + "/v2/upload", headers=headers, data=f)
#     if response.status_code != 200:
#         print(f"Error: {response.status_code}, Response: {response.text}")
#         response.raise_for_status()
#     upload_json = response.json()
#     audio_url = upload_json["upload_url"]

data = {
    "audio_url": audio_url
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")

transcript_json = response.json()
transcript_id = transcript_json["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript['id'])
        print(f" \nFull Transcript: \n\n{transcript['text']}\n")

        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

prompt = f"""
        - You are an expert at providing valuable feedback to individuals.
        - You possess exceptionally high emotional intelligence.
        - You excel at analyzing the behavior of individuals in the given transcript and providing insights on how they could improve.
        - You emphasize constructive criticism in your feedback.
        - The feedback focuses on how people can better achieve their objectives.
        - You avoid providing unjustified or unfounded feedback.
        - Your communication is clear, accurate and concise, and you write with perfect English.
        - Directly start with the feedback without any preamble or introduction.
        """

llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": f"{prompt} Please provide feedback for this transcript: \n\n{{{{ transcript }}}}"
      }
    ],
    "transcript_id": transcript_id,
    "max_tokens": 1500
  }

response = requests.post(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  headers=headers,
  json=llm_gateway_data
)

result = response.json()

if "error" in result:
    print(f"\nError from LLM Gateway: {result['error']}")
else:
    response_text = result['choices'][0]['message']['content']
    print(f"\nResponse ID: {result['request_id']}\n")
    print(response_text)
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

// Use a publicly-accessible URL:
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4";

// Or upload a local file:
// import fs from "fs-extra";
// const audioData = await fs.readFile("./your_audio_file.mp3");
// const uploadRes = await fetch(`${baseUrl}/v2/upload`, {
//   method: "POST",
//   headers,
//   body: audioData,
// });
// if (!uploadRes.ok) throw new Error(`Error: ${uploadRes.status}`);
// const uploadResponse = await uploadRes.json();
// const audioUrl = uploadResponse.upload_url;

const data = {
  audio_url: audioUrl
};

let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let transcript;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();
  if (transcript.status === "completed") {
    console.log(transcript.id);
    console.log(`\nFull Transcript:\n\n${transcript.text}\n`);
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

const prompt = `
        - You are an expert at providing valuable feedback to individuals.
        - You possess exceptionally high emotional intelligence.
        - You excel at analyzing the behavior of individuals in the given transcript and providing insights on how they could improve.
        - You emphasize constructive criticism in your feedback.
        - The feedback focuses on how people can better achieve their objectives.
        - You avoid providing unjustified or unfounded feedback.
        - Your communication is clear, accurate and concise, and you write with perfect English.
        - Directly start with the feedback without any preamble or introduction.
        `;

const llmGatewayData = {
  model: "claude-sonnet-4-6",
  messages: [
    {
      role: "user",
      content: `${prompt} Please provide feedback for this transcript: \n\n{{ transcript }}`,
    },
  ],
  transcript_id: transcriptId,
  max_tokens: 1500,
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llmGatewayData),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();

if (result.error) {
  console.log(`\nError from LLM Gateway: ${result.error}`);
} else {
  const responseText = result.choices[0].message.content;
  console.log(`\nResponse ID: ${result.request_id}\n`);
  console.log(responseText);
}
```

## Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

Find more details on the current LLM Gateway pricing on the AssemblyAI [pricing page](https://www.assemblyai.com/pricing).

## Step-by-Step Instructions

In this guide, we'll prompt LLM Gateway to perform some AI coaching.

Install the required packages:

```bash
pip install requests
```

Import the required packages and set the base URL and headers.

```python
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}
```

```javascript
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };
```

Use AssemblyAI to transcribe a file and save the transcript.

```python expandable
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"

# with open("/your_audio_file.mp3", "rb") as f:
#     response = requests.post(base_url + "/v2/upload", headers=headers, data=f)
#     if response.status_code != 200:
#         print(f"Error: {response.status_code}, Response: {response.text}")
#         response.raise_for_status()
#     upload_json = response.json()
#     audio_url = upload_json["upload_url"]

data = {
    "audio_url": audio_url
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")

transcript_json = response.json()
transcript_id = transcript_json["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript['id'])
        print(f" \nFull Transcript: \n\n{transcript['text']}\n")

        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```javascript expandable
// Use a publicly-accessible URL:
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4";

// Or upload a local file:
// import fs from "fs-extra";
// const audioData = await fs.readFile("./your_audio_file.mp3");
// const uploadRes = await fetch(`${baseUrl}/v2/upload`, {
//   method: "POST",
//   headers,
//   body: audioData,
// });
// if (!uploadRes.ok) throw new Error(`Error: ${uploadRes.status}`);
// const uploadResponse = await uploadRes.json();
// const audioUrl = uploadResponse.upload_url;

const data = {
  audio_url: audioUrl
};

let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let transcript;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();
  if (transcript.status === "completed") {
    console.log(transcript.id);
    console.log(`\nFull Transcript:\n\n${transcript.text}\n`);
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

Define your detailed prompt instructions for generating feedback on the transcript text of your sales call, meeting, or other content. This is an example prompt, which you can modify to suit your specific requirements.

```python
prompt = f"""
        - You are an expert at providing valuable feedback to individuals.
        - You possess exceptionally high emotional intelligence.
        - You excel at analyzing the behavior of individuals in the given transcript and providing insights on how they could improve.
        - You emphasize constructive criticism in your feedback.
        - The feedback focuses on how people can better achieve their objectives.
        - You avoid providing unjustified or unfounded feedback.
        - Your communication is clear, accurate and concise, and you write with perfect English.
        - Directly start with the feedback without any preamble or introduction.
        """
```

```javascript
const prompt = `
        - You are an expert at providing valuable feedback to individuals.
        - You possess exceptionally high emotional intelligence.
        - You excel at analyzing the behavior of individuals in the given transcript and providing insights on how they could improve.
        - You emphasize constructive criticism in your feedback.
        - The feedback focuses on how people can better achieve their objectives.
        - You avoid providing unjustified or unfounded feedback.
        - Your communication is clear, accurate and concise, and you write with perfect English.
        - Directly start with the feedback without any preamble or introduction.
        `;
```

Generate the custom feedback using LLM Gateway.

```python
llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": f"{prompt} Please provide feedback for this transcript: \n\n{{{{ transcript }}}}"
      }
    ],
    "transcript_id": transcript_id,
    "max_tokens": 1500,
    "temperature": 0
  }

response = requests.post(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  headers=headers,
  json=llm_gateway_data
)
```

```javascript
const llmGatewayData = {
  model: "claude-sonnet-4-6",
  messages: [
    {
      role: "user",
      content: `${prompt} Please provide feedback for this transcript: \n\n{{ transcript }}`,
    },
  ],
  transcript_id: transcriptId,
  max_tokens: 1500,
  temperature: 0,
};

const res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llmGatewayData),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();
```

Finally, save and return the LLM response.

```python
result = response.json()

if "error" in result:
    print(f"\nError from LLM Gateway: {result['error']}")
else:
    response_text = result['choices'][0]['message']['content']
    print(f"\nResponse ID: {result['request_id']}\n")
    print(response_text)
```

```javascript
if (response.error) {
  console.log(`\nError from LLM Gateway: ${response.error}`);
} else {
  const responseText = response.choices[0].message.content;
  console.log(`\nResponse ID: ${response.request_id}\n`);
  console.log(responseText);
}
```

---

<a id="generate-action-items-with-llm-gateway-docs-guides-task-endpoint-action-items-mdx"></a>

# Generate Action Items with LLM Gateway

URL: https://www.assemblyai.com/docs/guides/task-endpoint-action-items
Source: docs/guides/task-endpoint-action-items.mdx
Navigation: LLM Gateway > Voice AI Guides > Basic LLM Gateway workflows
Description: Generate Action Items with LLM Gateway documentation.

This tutorial will demonstrate how to use AssemblyAI's [LLM Gateway](/llm-gateway) framework to create action items from a transcript.

## Quickstart

```python expandable
import requests
import time

API_KEY = "YOUR_API_KEY"
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"

# Step 1: Upload or provide audio URL and start transcription
transcript_request = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": API_KEY, "content-type": "application/json"},
    json={"audio_url": audio_url},
)

transcript_id = transcript_request.json()["id"]

# Step 2: Poll until transcription completes
while True:
    polling_response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": API_KEY},
    )
    status = polling_response.json()["status"]

    if status == "completed":
        break
    elif status == "error":
        raise RuntimeError(f"Transcription failed: {polling_response.json()['error']}")
    else:
        print(f"Transcription status: {status}")
        time.sleep(3)

# Step 3: Build the prompt
prompt = """
Here are guidelines to follow:
- You are an expert at understanding transcripts of conversations, calls and meetings.
- You are an expert at coming up with ideal action items based on the contents of the transcripts.
- Action items are things that the transcript implies should get done.
- Your action item ideas do not make stuff up that isn't relevant to the transcript.
- You do not needlessly make up action items - you stick to important tasks.
- You are useful, true and concise, and write in perfect English.
- Your action items can be tied back to direct quotes in the transcript.
- You do not cite the quotes the action items relate to.
- The action items are written succinctly.
- Please give useful action items based on the transcript.
"""

answer_format = "Bullet Points"
if answer_format:
    prompt += f"\nYour response should have the following format: {answer_format}"

# Step 4: Send transcript ID to LLM Gateway
headers = {"authorization": API_KEY}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": f"{prompt}\n\n{{{{ transcript }}}}",
            }
        ],
        "transcript_id": transcript_id,
        "max_tokens": 1000,
    },
)

# Step 5: Print the LLM-generated action items
response_json = response.json()
print(response_json["choices"][0]["message"]["content"])
```

```javascript expandable
const API_KEY = "YOUR_API_KEY";
const headers = { authorization: API_KEY };
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4";

// Step 1: Upload or provide audio URL and start transcription
let res = await fetch("https://api.assemblyai.com/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({ audio_url: audioUrl }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptRequest = await res.json();

const transcriptId = transcriptRequest.id;

// Step 2: Poll until transcription completes
while (true) {
  res = await fetch(`https://api.assemblyai.com/v2/transcript/${transcriptId}`, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const pollingResponse = await res.json();
  const status = pollingResponse.status;

  if (status === "completed") {
    break;
  } else if (status === "error") {
    throw new Error(`Transcription failed: ${pollingResponse.error}`);
  } else {
    console.log(`Transcription status: ${status}`);
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Step 3: Build the prompt
let prompt = `
Here are guidelines to follow:
- You are an expert at understanding transcripts of conversations, calls and meetings.
- You are an expert at coming up with ideal action items based on the contents of the transcripts.
- Action items are things that the transcript implies should get done.
- Your action item ideas do not make stuff up that isn't relevant to the transcript.
- You do not needlessly make up action items - you stick to important tasks.
- You are useful, true and concise, and write in perfect English.
- Your action items can be tied back to direct quotes in the transcript.
- You do not cite the quotes the action items relate to.
- The action items are written succinctly.
- Please give useful action items based on the transcript.
`;

const answerFormat = "Bullet Points";
if (answerFormat) {
  prompt += `\nYour response should have the following format: ${answerFormat}`;
}

// Step 4: Send transcript ID to LLM Gateway
res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content: `${prompt}\n\n{{ transcript }}`,
      },
    ],
    transcript_id: transcriptId,
    max_tokens: 1000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

// Step 5: Print the LLM-generated action items
console.log(response.choices[0].message.content);
```

## Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

Find more details on the current LLM Gateway pricing in the AssemblyAI [pricing page](https://www.assemblyai.com/pricing).

## Step-by-Step Instructions

In this guide, we will prompt LLM Gateway to generate action items from our transcript.

Install the required packages:

```bash
pip install requests
```

Import the necessary libraries and set AssemblyAI API key.

```python
import requests
import time

API_KEY = "YOUR_API_KEY"
```

```javascript
const API_KEY = "YOUR_API_KEY";
const headers = { authorization: API_KEY };
```

Use AssemblyAI to transcribe a file and save the transcript.

```python expandable
# Step 1: Upload or provide audio URL and start transcription
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"

transcript_request = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": API_KEY, "content-type": "application/json"},
    json={"audio_url": audio_url},
)

transcript_id = transcript_request.json()["id"]

# Step 2: Poll until transcription completes
while True:
    polling_response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": API_KEY},
    )
    status = polling_response.json()["status"]

    if status == "completed":
        break
    elif status == "error":
        raise RuntimeError(f"Transcription failed: {polling_response.json()['error']}")
    else:
        print(f"Transcription status: {status}")
        time.sleep(3)
```

```javascript expandable
// Step 1: Upload or provide audio URL and start transcription
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4";

let res = await fetch("https://api.assemblyai.com/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({ audio_url: audioUrl }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptRequest = await res.json();

const transcriptId = transcriptRequest.id;

// Step 2: Poll until transcription completes
while (true) {
  res = await fetch(`https://api.assemblyai.com/v2/transcript/${transcriptId}`, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const pollingResponse = await res.json();
  const status = pollingResponse.status;

  if (status === "completed") {
    break;
  } else if (status === "error") {
    throw new Error(`Transcription failed: ${pollingResponse.error}`);
  } else {
    console.log(`Transcription status: ${status}`);
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

Provide detailed instructions to prompt LLM Gateway to create action items from the transcript.

```python
prompt = f"""
        Here are guidelines to follow:
        - You are an expert at understanding transcripts of conversations, calls and meetings.
        - You are an expert at coming up with ideal action items based on the contents of the transcripts.
        - Action items are things that the transcript implies should get done.
        - Your action item ideas do not make stuff up that isn't relevant to the transcript.
        - You do not needlessly make up action items - you stick to important tasks.
        - You are useful, true and concise, and write in perfect English.
        - Your action items can be tied back to direct quotes in the transcript.
        - You do not cite the quotes the action items relate to.
        - The action items are written succinctly.
        - Please give useful action items based on the transcript.
        - Your response should be formatted in bullet points.
        """
```

```javascript
let prompt = `
        Here are guidelines to follow:
        - You are an expert at understanding transcripts of conversations, calls and meetings.
        - You are an expert at coming up with ideal action items based on the contents of the transcripts.
        - Action items are things that the transcript implies should get done.
        - Your action item ideas do not make stuff up that isn't relevant to the transcript.
        - You do not needlessly make up action items - you stick to important tasks.
        - You are useful, true and concise, and write in perfect English.
        - Your action items can be tied back to direct quotes in the transcript.
        - You do not cite the quotes the action items relate to.
        - The action items are written succinctly.
        - Please give useful action items based on the transcript.
        - Your response should be formatted in bullet points.
        `;
```

Generate the custom action items using LLM Gateway.

```python
answer_format = "Bullet Points"
if answer_format:
    prompt += f"\nYour response should have the following format: {answer_format}"
```

```javascript
const answerFormat = "Bullet Points";
if (answerFormat) {
  prompt += `\nYour response should have the following format: ${answerFormat}`;
}
```

Prompt LLM Gateway using the transcript results and return the response.

```python expandable
# Step 4: Send transcript ID to LLM Gateway
headers = {"authorization": API_KEY}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": f"{prompt}\n\n{{{{ transcript }}}}",
            }
        ],
        "transcript_id": transcript_id,
        "max_tokens": 1000,
    },
)

# Step 5: Print the LLM-generated action items
response_json = response.json()
print(response_json["choices"][0]["message"]["content"])

```

```javascript expandable
// Step 4: Send transcript ID to LLM Gateway
let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content: `${prompt}\n\n{{ transcript }}`,
      },
    ],
    transcript_id: transcriptId,
    max_tokens: 1000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

// Step 5: Print the LLM-generated action items
console.log(response.choices[0].message.content);
```

---

<a id="prompt-a-structured-q-a-response-using-llm-gateway-docs-guides-task-endpoint-structured-qa-mdx"></a>

# Prompt A Structured Q&A Response Using LLM Gateway

URL: https://www.assemblyai.com/docs/guides/task-endpoint-structured-QA
Source: docs/guides/task-endpoint-structured-QA.mdx
Navigation: LLM Gateway > Voice AI Guides > Basic LLM Gateway workflows
Description: Prompt A Structured Q&A Response Using LLM Gateway documentation.

This cookbook will demonstrate how to use AssemblyAI's [LLM Gateway](/llm-gateway) framework to prompt a structured question and answer response.

## Quickstart

```python expandable
import requests
import time
import xml.etree.ElementTree as ET

API_KEY = "YOUR_API_KEY"
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"

# -------------------------------
# Step 1: Transcribe the audio
# -------------------------------
transcript_request = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": API_KEY, "content-type": "application/json"},
    json={"audio_url": audio_url},
)

transcript_id = transcript_request.json()["id"]

# Poll for completion
while True:
    polling_response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": API_KEY},
    )
    status = polling_response.json()["status"]

    if status == "completed":
        break
    elif status == "error":
        raise RuntimeError(f"Transcription failed: {polling_response.json()['error']}")
    else:
        print(f"Transcription status: {status}")
        time.sleep(3)

# -------------------------------
# Step 2: Build question helper functions
# -------------------------------
def construct_question(question):
    question_str = f"Question: {question['question']}"

    if question.get("context"):
        question_str += f"\nContext: {question['context']}"

    # Default answer_format
    if not question.get("answer_format"):
        question["answer_format"] = "short sentence"

    question_str += f"\nAnswer Format: {question['answer_format']}"

    if question.get("answer_options"):
        options_str = ", ".join(question["answer_options"])
        question_str += f"\nOptions: {options_str}"

    return question_str + "\n"

def escape_xml_characters(xml_string):
    return xml_string.replace("&", "&amp;")

# -------------------------------
# Step 3: Define questions
# -------------------------------
questions = [
    {
        "question": "What are the top level KPIs for engineering?",
        "context": "KPI stands for key performance indicator",
        "answer_format": "short sentence",
    },
    {
        "question": "How many days has it been since the data team has gotten updated metrics?",
        "answer_options": ["1", "2", "3", "4", "5", "6", "7", "more than 7"],
    },
    {"question": "What are the future plans for the project?"},
]

question_str = "\n".join(construct_question(q) for q in questions)

# -------------------------------
# Step 4: Build the LLM prompt
# -------------------------------
prompt = f"""You are an expert at giving accurate answers to questions about texts.
No preamble.
Given the series of questions, answer the questions.
Each question may follow up with answer format, answer options, and context for each question.
It is critical that you follow the answer format and answer options for each question.
When context is provided with a question, refer to it when answering the question.
You are useful, true and concise, and write in perfect English.
Only the question is allowed between the <question> tag. Do not include the answer format, options, or question context in your response.
Only text is allowed between the <question> and <answer> tags.
XML tags are not allowed between the <question> and <answer> tags.
End your response with a closing </responses> tag.
For each question-answer pair, format your response according to the template provided below:

Template for response:
<responses>
  <response>
    <question>The question</question>
    <answer>Your answer</answer>
  </response>
  <response>
    ...
  </response>
  ...
</responses>

These are the questions:
{question_str}

Transcript:
{{{{ transcript }}}}
"""

# -------------------------------
# Step 5: Query LLM Gateway
# -------------------------------
headers = {"authorization": API_KEY}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": prompt}],
        "transcript_id": transcript_id,
        "max_tokens": 2000,
    },
)

response_json = response.json()
llm_output = response_json["choices"][0]["message"]["content"]

# -------------------------------
# Step 6: Parse and print XML response
# -------------------------------
clean_response = escape_xml_characters(llm_output).strip()

try:
    root = ET.fromstring(clean_response)
    for resp in root.findall("response"):
        question = resp.find("question").text
        answer = resp.find("answer").text
        print(f"Question: {question}")
        print(f"Answer: {answer}\n")
except ET.ParseError as e:
    print("Could not parse XML response.")
    print("Raw model output:\n", llm_output)
```

```javascript expandable
import { DOMParser } from "@xmldom/xmldom";

const API_KEY = "YOUR_API_KEY";
const headers = { authorization: API_KEY };
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4";

// -------------------------------
// Step 1: Transcribe the audio
// -------------------------------
let res = await fetch("https://api.assemblyai.com/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({ audio_url: audioUrl }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptRequest = await res.json();

const transcriptId = transcriptRequest.id;

// Poll for completion
while (true) {
  const pollingRes = await fetch(`https://api.assemblyai.com/v2/transcript/${transcriptId}`, { headers });
  if (!pollingRes.ok) throw new Error(`Error: ${pollingRes.status}`);
  const pollingResponse = await pollingRes.json();
  const status = pollingResponse.status;

  if (status === "completed") {
    break;
  } else if (status === "error") {
    throw new Error(`Transcription failed: ${pollingResponse.error}`);
  } else {
    console.log(`Transcription status: ${status}`);
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// -------------------------------
// Step 2: Build question helper functions
// -------------------------------
function constructQuestion(question) {
  let questionStr = `Question: ${question.question}`;

  if (question.context) {
    questionStr += `\nContext: ${question.context}`;
  }

  // Default answer_format
  if (!question.answer_format) {
    question.answer_format = "short sentence";
  }

  questionStr += `\nAnswer Format: ${question.answer_format}`;

  if (question.answer_options) {
    const optionsStr = question.answer_options.join(", ");
    questionStr += `\nOptions: ${optionsStr}`;
  }

  return questionStr + "\n";
}

function escapeXmlCharacters(xmlString) {
  return xmlString.replace(/&/g, "&amp;");
}

// -------------------------------
// Step 3: Define questions
// -------------------------------
const questions = [
  {
    question: "What are the top level KPIs for engineering?",
    context: "KPI stands for key performance indicator",
    answer_format: "short sentence",
  },
  {
    question: "How many days has it been since the data team has gotten updated metrics?",
    answer_options: ["1", "2", "3", "4", "5", "6", "7", "more than 7"],
  },
  { question: "What are the future plans for the project?" },
];

const questionStr = questions.map((q) => constructQuestion(q)).join("\n");

// -------------------------------
// Step 4: Build the LLM prompt
// -------------------------------
const prompt = `You are an expert at giving accurate answers to questions about texts.
No preamble.
Given the series of questions, answer the questions.
Each question may follow up with answer format, answer options, and context for each question.
It is critical that you follow the answer format and answer options for each question.
When context is provided with a question, refer to it when answering the question.
You are useful, true and concise, and write in perfect English.
Only the question is allowed between the <question> tag. Do not include the answer format, options, or question context in your response.
Only text is allowed between the <question> and <answer> tags.
XML tags are not allowed between the <question> and <answer> tags.
End your response with a closing </responses> tag.
For each question-answer pair, format your response according to the template provided below:

Template for response:
<responses>
  <response>
    <question>The question</question>
    <answer>Your answer</answer>
  </response>
  <response>
    ...
  </response>
  ...
</responses>

These are the questions:
${questionStr}

Transcript:
{{ transcript }}
`;

// -------------------------------
// Step 5: Query LLM Gateway
// -------------------------------
res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [{ role: "user", content: prompt }],
    transcript_id: transcriptId,
    max_tokens: 2000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const llmOutput = response.choices[0].message.content;

// -------------------------------
// Step 6: Parse and print XML response
// -------------------------------
const cleanResponse = escapeXmlCharacters(llmOutput).trim();

try {
  const root = new DOMParser().parseFromString(cleanResponse, "text/xml");
  const responses = Array.from(root.getElementsByTagName("response"));
  for (const resp of responses) {
    const question = resp.getElementsByTagName("question")[0].textContent;
    const answer = resp.getElementsByTagName("answer")[0].textContent;
    console.log(`Question: ${question}`);
    console.log(`Answer: ${answer}\n`);
  }
} catch (e) {
  console.log("Could not parse XML response.");
  console.log("Raw model output:\n", llmOutput);
}
```

## Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

Find more details on the current LLM Gateway pricing in the AssemblyAI [pricing page](https://www.assemblyai.com/pricing).

## Step-by-Step Instructions

In this guide, we will prompt LLM Gateway with a structured Q&A format and generate an XML response.

Install the required packages:

```bash
pip install requests
```

```bash
npm install @xmldom/xmldom
```

Import the necessary libraries and set API key.

```python
import requests
import time
import xml.etree.ElementTree as ET

API_KEY = "YOUR_API_KEY"
```

```javascript
import { DOMParser } from "@xmldom/xmldom";

const API_KEY = "YOUR_API_KEY";
const headers = { authorization: API_KEY };
```

Next, we'll use AssemblyAI to transcribe a file and save our transcript.

```python expandable
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"

# -------------------------------
# Step 1: Transcribe the audio
# -------------------------------
transcript_request = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": API_KEY, "content-type": "application/json"},
    json={"audio_url": audio_url},
)

transcript_id = transcript_request.json()["id"]

# Poll for completion
while True:
    polling_response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": API_KEY},
    )
    status = polling_response.json()["status"]

    if status == "completed":
        break
    elif status == "error":
        raise RuntimeError(f"Transcription failed: {polling_response.json()['error']}")
    else:
        print(f"Transcription status: {status}")
        time.sleep(3)
```

```javascript expandable
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4";

// -------------------------------
// Step 1: Transcribe the audio
// -------------------------------
let res = await fetch("https://api.assemblyai.com/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({ audio_url: audioUrl }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptRequest = await res.json();

const transcriptId = transcriptRequest.id;

// Poll for completion
while (true) {
  const pollingRes = await fetch(`https://api.assemblyai.com/v2/transcript/${transcriptId}`, { headers });
  if (!pollingRes.ok) throw new Error(`Error: ${pollingRes.status}`);
  const pollingResponse = await pollingRes.json();
  const status = pollingResponse.status;

  if (status === "completed") {
    break;
  } else if (status === "error") {
    throw new Error(`Transcription failed: ${pollingResponse.error}`);
  } else {
    console.log(`Transcription status: ${status}`);
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

Construct a formatted string to structure the questions. This includes the question text, context, an answer format, any answer options, then returns the formatted string.

```python expandable
# -------------------------------
# Step 2: Build question helper functions
# -------------------------------
def construct_question(question):
    question_str = f"Question: {question['question']}"

    if question.get("context"):
        question_str += f"\nContext: {question['context']}"

    # Default answer_format
    if not question.get("answer_format"):
        question["answer_format"] = "short sentence"

    question_str += f"\nAnswer Format: {question['answer_format']}"

    if question.get("answer_options"):
        options_str = ", ".join(question["answer_options"])
        question_str += f"\nOptions: {options_str}"

    return question_str + "\n"

def escape_xml_characters(xml_string):
    return xml_string.replace("&", "&amp;")
```

```javascript expandable
// -------------------------------
// Step 2: Build question helper functions
// -------------------------------
function constructQuestion(question) {
  let questionStr = `Question: ${question.question}`;

  if (question.context) {
    questionStr += `\nContext: ${question.context}`;
  }

  // Default answer_format
  if (!question.answer_format) {
    question.answer_format = "short sentence";
  }

  questionStr += `\nAnswer Format: ${question.answer_format}`;

  if (question.answer_options) {
    const optionsStr = question.answer_options.join(", ");
    questionStr += `\nOptions: ${optionsStr}`;
  }

  return questionStr + "\n";
}

function escapeXmlCharacters(xmlString) {
  return xmlString.replace(/&/g, "&amp;");
}
```

Define a list of questions. For each question, you can define additional `context` and specify either an `answer_format` or a list of `answer_options`.

```python
# -------------------------------
# Step 3: Define questions
# -------------------------------
questions = [
    {
        "question": "What are the top level KPIs for engineering?",
        "context": "KPI stands for key performance indicator",
        "answer_format": "short sentence",
    },
    {
        "question": "How many days has it been since the data team has gotten updated metrics?",
        "answer_options": ["1", "2", "3", "4", "5", "6", "7", "more than 7"],
    },
    {"question": "What are the future plans for the project?"},
]
```

```javascript
// -------------------------------
// Step 3: Define questions
// -------------------------------
const questions = [
  {
    question: "What are the top level KPIs for engineering?",
    context: "KPI stands for key performance indicator",
    answer_format: "short sentence",
  },
  {
    question: "How many days has it been since the data team has gotten updated metrics?",
    answer_options: ["1", "2", "3", "4", "5", "6", "7", "more than 7"],
  },
  { question: "What are the future plans for the project?" },
];
```

Construct the formatted question string for all the questions and build the LLM prompt.

```python
question_str = '\n'.join(construct_question(q) for q in questions)
```

```javascript
const questionStr = questions.map((q) => constructQuestion(q)).join("\n");
```

Provide detailed instructions to prompt LLM Gateway to answer a series of questions. This also defines a structured XML template for the responses.

```python expandable
# -------------------------------
# Step 4: Build the LLM prompt
# -------------------------------
prompt = f"""You are an expert at giving accurate answers to questions about texts.
No preamble.
Given the series of questions, answer the questions.
Each question may follow up with answer format, answer options, and context for each question.
It is critical that you follow the answer format and answer options for each question.
When context is provided with a question, refer to it when answering the question.
You are useful, true and concise, and write in perfect English.
Only the question is allowed between the <question> tag. Do not include the answer format, options, or question context in your response.
Only text is allowed between the <question> and <answer> tags.
XML tags are not allowed between the <question> and <answer> tags.
End your response with a closing </responses> tag.
For each question-answer pair, format your response according to the template provided below:

Template for response:
<responses>
  <response>
    <question>The question</question>
    <answer>Your answer</answer>
  </response>
  <response>
    ...
  </response>
  ...
</responses>

These are the questions:
{question_str}

Transcript:
{{{{ transcript }}}}
"""

# -------------------------------
# Step 5: Query LLM Gateway
# -------------------------------
headers = {"authorization": API_KEY}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": prompt}],
        "transcript_id": transcript_id,
        "max_tokens": 2000,
    },
)

response_json = response.json()
llm_output = response_json["choices"][0]["message"]["content"]

# -------------------------------
# Step 6: Parse and print XML response
# -------------------------------
clean_response = escape_xml_characters(llm_output).strip()

try:
    root = ET.fromstring(clean_response)
    for resp in root.findall("response"):
        question = resp.find("question").text
        answer = resp.find("answer").text
        print(f"Question: {question}")
        print(f"Answer: {answer}\n")
except ET.ParseError as e:
    print("Could not parse XML response.")
    print("Raw model output:\n", llm_output)
```

```javascript expandable
// -------------------------------
// Step 4: Build the LLM prompt
// -------------------------------
const prompt = `You are an expert at giving accurate answers to questions about texts.
No preamble.
Given the series of questions, answer the questions.
Each question may follow up with answer format, answer options, and context for each question.
It is critical that you follow the answer format and answer options for each question.
When context is provided with a question, refer to it when answering the question.
You are useful, true and concise, and write in perfect English.
Only the question is allowed between the <question> tag. Do not include the answer format, options, or question context in your response.
Only text is allowed between the <question> and <answer> tags.
XML tags are not allowed between the <question> and <answer> tags.
End your response with a closing </responses> tag.
For each question-answer pair, format your response according to the template provided below:

Template for response:
<responses>
  <response>
    <question>The question</question>
    <answer>Your answer</answer>
  </response>
  <response>
    ...
  </response>
  ...
</responses>

These are the questions:
${questionStr}

Transcript:
{{ transcript }}
`;

// -------------------------------
// Step 5: Query LLM Gateway
// -------------------------------
const res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [{ role: "user", content: prompt }],
    transcript_id: transcriptId,
    max_tokens: 2000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const llmOutput = response.choices[0].message.content;

// -------------------------------
// Step 6: Parse and print XML response
// -------------------------------
const cleanResponse = escapeXmlCharacters(llmOutput).trim();

try {
  const root = new DOMParser().parseFromString(cleanResponse, "text/xml");
  const responses = Array.from(root.getElementsByTagName("response"));
  for (const resp of responses) {
    const question = resp.getElementsByTagName("question")[0].textContent;
    const answer = resp.getElementsByTagName("answer")[0].textContent;
    console.log(`Question: ${question}`);
    console.log(`Answer: ${answer}\n`);
  }
} catch (e) {
  console.log("Could not parse XML response.");
  console.log("Raw model output:\n", llmOutput);
}
```

---

<a id="estimate-input-token-costs-for-llm-gateway-docs-guides-counting-tokens-mdx"></a>

# Estimate Input Token Costs for LLM Gateway

URL: https://www.assemblyai.com/docs/guides/counting-tokens
Source: docs/guides/counting-tokens.mdx
Navigation: LLM Gateway > Voice AI Guides > Basic LLM Gateway workflows
Description: Estimate Input Token Costs for LLM Gateway documentation.

AssemblyAI's [LLM Gateway](/llm-gateway/quickstart) is a unified API providing access to 25+ models from Claude, GPT, Gemini, and more through a single interface.
It's a powerful way to extract insights from transcripts generated from audio and video files. Given how varied the type of input and output could be for these use cases, the [pricing](https://www.assemblyai.com/pricing) for LLM Gateway is based on **both** input and output tokens.

Output tokens will vary depending on the model and the complexity of your request, but how do you determine the amount of input tokens you'll be sending to LLM Gateway?
How many tokens does an audio file and your prompt contain? This guide will show you how to roughly calculate that information to help predict LLM Gateway's input token cost ahead of time.

This guide calculates **input token costs only**.
Output token costs will vary based on the model used and the length of the generated response.

To see the specific cost of each model (per 1M input and output tokens) applicable to your AssemblyAI account, refer to the Rates table on the [Billing page](https://www.assemblyai.com/dashboard/home) of the dashboard.

## Quickstart

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}

# Transcribe audio file
audio_url = "https://assembly.ai/wildfires.mp3"
data = {"audio_url": audio_url}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for completion
print("Waiting for transcription to complete...")

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    time.sleep(3)

# Define your prompt
prompt = "Provide a brief summary of the transcript."

# Calculate character count (transcript + prompt)
transcript_chars = len(transcript["text"])
prompt_chars = len(prompt)
total_chars = transcript_chars + prompt_chars
print(f"\nTotal characters: {total_chars}")

# Estimate tokens (roughly 4 characters = 1 token)
estimated_tokens = total_chars / 4
tokens_in_millions = estimated_tokens / 1_000_000

# Calculate input costs for different models
gpt5_cost = 1.25 * tokens_in_millions
claude_sonnet_cost = 3.00 * tokens_in_millions
gemini_pro_cost = 1.25 * tokens_in_millions

print(f"Estimated input tokens: {estimated_tokens:,.0f}")
print(f"\nEstimated input costs:")
print(f"GPT-5: ${gpt5_cost:.4f}")
print(f"Claude Sonnet 4.6: ${claude_sonnet_cost:.4f}")
print(f"Gemini 2.5 Pro: ${gemini_pro_cost:.4f}")
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "YOUR_API_KEY" };

// Transcribe audio file
const audioUrl = "https://assembly.ai/wildfires.mp3";
const data = { audio_url: audioUrl };

let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

// Poll for completion
console.log("Waiting for transcription to complete...");

let transcript;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();
  if (transcript.status === "completed") {
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }
  await new Promise((resolve) => setTimeout(resolve, 3000));
}

// Define your prompt
const prompt = "Provide a brief summary of the transcript.";

// Calculate character count (transcript + prompt)
const transcriptChars = transcript.text.length;
const promptChars = prompt.length;
const totalChars = transcriptChars + promptChars;
console.log(`\nTotal characters: ${totalChars}`);

// Estimate tokens (roughly 4 characters = 1 token)
const estimatedTokens = totalChars / 4;
const tokensInMillions = estimatedTokens / 1_000_000;

// Calculate input costs for different models
const gpt5Cost = 1.25 * tokensInMillions;
const claudeSonnetCost = 3.00 * tokensInMillions;
const geminiProCost = 1.25 * tokensInMillions;

console.log(`Estimated input tokens: ${estimatedTokens.toLocaleString("en-US", { maximumFractionDigits: 0 })}`);
console.log(`\nEstimated input costs:`);
console.log(`GPT-5: $${gpt5Cost.toFixed(4)}`);
console.log(`Claude Sonnet 4.6: $${claudeSonnetCost.toFixed(4)}`);
console.log(`Gemini 2.5 Pro: $${geminiProCost.toFixed(4)}`);
```

## Step-by-Step Guide

### Install dependencies

Install the required library:

```bash
pip install requests
```

### Set up your API key

Import the necessary libraries and set your AssemblyAI API key, which can be found on your account [dashboard](https://www.assemblyai.com/dashboard/home):

```python
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "YOUR_API_KEY"}
```

```javascript
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "YOUR_API_KEY" };
```

### Transcribe your audio file

Transcribe your audio file using AssemblyAI:

```python
audio_url = "https://assembly.ai/wildfires.mp3"
data = {"audio_url": audio_url}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

# Poll for completion
print("Waiting for transcription to complete...")

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    time.sleep(3)
```

```javascript expandable
const audioUrl = "https://assembly.ai/wildfires.mp3";
const data = { audio_url: audioUrl };

let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

// Poll for completion
console.log("Waiting for transcription to complete...");

let transcript;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();
  if (transcript.status === "completed") {
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }
  await new Promise((resolve) => setTimeout(resolve, 3000));
}
```

### Calculate character count

We'll count the characters in both the transcript and your prompt:

```python
# Define your prompt
prompt = "Provide a brief summary of the transcript."

# Calculate character count (transcript + prompt)
transcript_chars = len(transcript["text"])
prompt_chars = len(prompt)
total_chars = transcript_chars + prompt_chars
print(f"\nTotal characters: {total_chars}")
```

```javascript
// Define your prompt
const prompt = "Provide a brief summary of the transcript.";

// Calculate character count (transcript + prompt)
const transcriptChars = transcript.text.length;
const promptChars = prompt.length;
const totalChars = transcriptChars + promptChars;
console.log(`\nTotal characters: ${totalChars}`);
```

For this specific file with the example prompt, the transcript contains approximately 4,880 characters and the prompt contains 42 characters, for a total of 4,922 characters.

### Estimate tokens

Different LLM providers use different tokenization methods, but a rough estimate is that **4 characters equals approximately 1 token**. This is based on guidance from:

- [Claude tokenization documentation](https://docs.claude.com/en/docs/about-claude/pricing#frequently-asked-questions)
- [OpenAI token counting guide](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)
- [Gemini tokens documentation](https://ai.google.dev/gemini-api/docs/tokens?lang=python)

```python
# Estimate tokens (roughly 4 characters = 1 token)
estimated_tokens = total_chars / 4
tokens_in_millions = estimated_tokens / 1_000_000

print(f"Estimated input tokens: {estimated_tokens:,.0f}")
```

```javascript
// Estimate tokens (roughly 4 characters = 1 token)
const estimatedTokens = totalChars / 4;
const tokensInMillions = estimatedTokens / 1_000_000;

console.log(`Estimated input tokens: ${estimatedTokens.toLocaleString("en-US", { maximumFractionDigits: 0 })}`);
```

  **Language considerations**

  Token counts can differ significantly across languages. Non-English languages
  typically require more tokens per character than English. For instance, text
  in languages like Spanish, Chinese, or Arabic may use 2-3 characters per token
  instead of 4, resulting in higher token costs for the same amount of content.

### Calculate input token costs

LLM Gateway's pricing is calculated per 1M input tokens. Here are the [current rates](https://www.assemblyai.com/pricing) for popular models:

```python
# Calculate input costs for different models (rates per 1M tokens)
gpt5_cost = 1.25 * tokens_in_millions
claude_sonnet_cost = 3.00 * tokens_in_millions
gemini_pro_cost = 1.25 * tokens_in_millions

print(f"\nEstimated input costs:")
print(f"GPT-5: ${gpt5_cost:.4f}")
print(f"Claude Sonnet 4.6: ${claude_sonnet_cost:.4f}")
print(f"Gemini 2.5 Pro: ${gemini_pro_cost:.4f}")
```

```javascript
// Calculate input costs for different models (rates per 1M tokens)
const gpt5Cost = 1.25 * tokensInMillions;
const claudeSonnetCost = 3.00 * tokensInMillions;
const geminiProCost = 1.25 * tokensInMillions;

console.log(`\nEstimated input costs:`);
console.log(`GPT-5: $${gpt5Cost.toFixed(4)}`);
console.log(`Claude Sonnet 4.6: $${claudeSonnetCost.toFixed(4)}`);
console.log(`Gemini 2.5 Pro: $${geminiProCost.toFixed(4)}`);
```

For our example file with approximately 1,230 input tokens:

- **GPT-5** (`gpt-5`): ~$0.0015
- **Claude Sonnet 4.6** (`claude-sonnet-4-6`): ~$0.0037
- **Gemini 2.5 Pro** (`gemini-2.5-pro`): ~$0.0015

These calculations estimate **input token costs only**. Output tokens are not included and will vary based on:
- The model you choose
- The complexity of your request
- The length of the generated response

To see the complete pricing for both input and output tokens for all available models, visit the Rates table on the [Billing page](https://www.assemblyai.com/dashboard/home) of your dashboard.

## Next steps

- [LLM Gateway Overview](/llm-gateway/quickstart) - Learn about all available models and capabilities
- [Apply LLM Gateway to Audio Transcripts](/llm-gateway/apply-llms-to-audio-files) - Complete guide to using LLM Gateway with transcripts
- [Billing page](https://www.assemblyai.com/dashboard/home) - View LLM pricing for your account

---

<a id="extract-dialogue-data-with-llm-gateway-and-json-docs-guides-dialogue-data-mdx"></a>

# Extract Dialogue Data with LLM Gateway and JSON

URL: https://www.assemblyai.com/docs/guides/dialogue-data
Source: docs/guides/dialogue-data.mdx
Navigation: LLM Gateway > Voice AI Guides > Get quotes and citations with LLM Gateway
Description: Extract Dialogue Data with LLM Gateway and JSON documentation.

In this guide, we'll show you how to use AssemblyAI's [LLM Gateway](/llm-gateway) framework to process several audio files, and then format your results in JSON (JavaScript Object Notation) format.

JSON allows you to programmatically format, parse, and transfer responses from LLM Gateway, which is useful for implementing LLM Gateway with a wide range of other applications.

In this example, we will leverage the JSON formatting to create a .csv file from a directory of files that must be transcribed and submitted to LLM Gateway. However, you can use the same concepts in this guide to generate a JSON-formatted response, which you can then use to update a database table or interact with other APIs.

## Quickstart

````python expandable
import requests
import json
import os
import csv
import time
import re

# Configuration
api_key = "<YOUR_API_KEY>"
base_url = "https://api.assemblyai.com"
headers = {"authorization": api_key}
output_filename = "profiles.csv"

def extract_json(text):
    """Extract JSON from text that might contain markdown or extra text"""
    # First, try to remove markdown code blocks
    text = text.strip()

    # Remove ```json and ``` markers
    if text.startswith("```"):
        text = re.sub(r'^```(?:json)?\s*', '', text)
        text = re.sub(r'\s*```$', '', text)

    # Find the first { and last } to extract just the JSON object
    first_brace = text.find('{')
    last_brace = text.rfind('}')

    if first_brace != -1 and last_brace != -1:
        json_str = text[first_brace:last_brace + 1]
        return json.loads(json_str)

    # If that didn't work, try parsing the whole thing
    return json.loads(text)

def upload_file(file_path):
    """Upload a local audio file to AssemblyAI"""
    with open(file_path, "rb") as f:
        response = requests.post(f"{base_url}/v2/upload", headers=headers, data=f)
        if response.status_code != 200:
            print(f"Error uploading {file_path}: {response.status_code}, {response.text}")
            response.raise_for_status()
        return response.json()["upload_url"]

def transcribe_audio(audio_url):
    """Submit audio for transcription and poll until complete"""
    # Submit transcription request
    data = {"audio_url": audio_url}
    response = requests.post(f"{base_url}/v2/transcript", headers=headers, json=data)

    if response.status_code != 200:
        print(f"Error submitting transcription: {response.status_code}, {response.text}")
        response.raise_for_status()

    transcript_id = response.json()["id"]
    polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

    # Poll for completion
    while True:
        transcript = requests.get(polling_endpoint, headers=headers).json()
        if transcript["status"] == "completed":
            return transcript_id
        elif transcript["status"] == "error":
            raise RuntimeError(f"Transcription failed: {transcript['error']}")
        else:
            time.sleep(3)

def process_with_llm_gateway(transcript_id, prompt):
    """Send transcript to LLM Gateway for processing"""
    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": f"{prompt}\n\n{{{{ transcript }}}}"
            }
        ],
        "transcript_id": transcript_id,
        "max_tokens": 1500
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()

    if "error" in result:
        raise RuntimeError(f"LLM Gateway error: {result['error']}")

    return result['choices'][0]['message']['content']

# Main execution
prompt = """
        You are an HR executive scanning through an interview transcript to extract information about a candidate.
        You are required to create a JSON response with key information about the candidate.
        You will use this template for your answer:
        {
            "Name": "<candidate-name>",
            "Position": "<job position that candidate is applying for>",
            "Past experience": "<A short phrase describing the candidate's relevant past experience for the role>"
        }
        Do not include any other text in your response. Only respond in JSON format that is not surrounded by markdown code, as your response will be parsed programmatically as JSON.
        """

# Get all files from interviews directory
interview_files = [os.path.join("interviews", file) for file in os.listdir("interviews")]

with open(output_filename, "w", newline="") as file:
    writer = csv.writer(file)
    header = ["Name", "Position", "Past Experience"]
    writer.writerow(header)

    print(f"Processing {len(interview_files)} interview files...")

    for interview_file in interview_files:
        print(f"\nProcessing: {interview_file}")

        # Upload file and get URL
        print("  Uploading file...")
        audio_url = upload_file(interview_file)

        # Transcribe audio
        print("  Transcribing...")
        transcript_id = transcribe_audio(audio_url)

        # Process with LLM Gateway
        print("  Analyzing with LLM Gateway...")
        llm_response = process_with_llm_gateway(transcript_id, prompt)

        # Parse JSON response
        interviewee_data = extract_json(llm_response)
        writer.writerow(interviewee_data.values())
        print(f"  Completed: {interviewee_data['Name']}")

print(f"\nCreated .csv file {output_filename}")
````

```javascript expandable
import { readdirSync } from "fs";
import fs from "fs";
import path from "path";

// Configuration
const apiKey = "<YOUR_API_KEY>";
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: apiKey };
const outputFilename = "profiles.csv";

function extractJson(text) {
  // Extract JSON from text that might contain markdown or extra text
  text = text.trim();

  // Remove ```json and ``` markers
  if (text.startsWith("```")) {
    text = text.replace(/^```(?:json)?\s*/g, "");
    text = text.replace(/\s*```$/g, "");
  }

  // Find the first { and last } to extract just the JSON object
  const firstBrace = text.indexOf("{");
  const lastBrace = text.lastIndexOf("}");

  if (firstBrace !== -1 && lastBrace !== -1) {
    const jsonStr = text.slice(firstBrace, lastBrace + 1);
    return JSON.parse(jsonStr);
  }

  // If that didn't work, try parsing the whole thing
  return JSON.parse(text);
}

async function uploadFile(filePath) {
  // Upload a local audio file to AssemblyAI
  const fileData = await fs.promises.readFile(filePath);
  const res = await fetch(`${baseUrl}/v2/upload`, {
    method: "POST",
    headers,
    body: fileData,
  });
  if (!res.ok) {
    throw new Error(`Error uploading ${filePath}: ${res.status}`);
  }
  const response = await res.json();
  return response.upload_url;
}

async function transcribeAudio(audioUrl) {
  // Submit audio for transcription and poll until complete
  const data = { audio_url: audioUrl };
  const res = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(data),
  });

  if (!res.ok) {
    throw new Error(`Error submitting transcription: ${res.status}`);
  }

  const response = await res.json();
  const transcriptId = response.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for completion
  while (true) {
    const res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    const transcript = await res.json();
    if (transcript.status === "completed") {
      return transcriptId;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

async function processWithLlmGateway(transcriptId, prompt) {
  // Send transcript to LLM Gateway for processing
  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content: `${prompt}\n\n{{ transcript }}`,
      },
    ],
    transcript_id: transcriptId,
    max_tokens: 1500,
  };

  const res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  if (response.error) {
    throw new Error(`LLM Gateway error: ${response.error}`);
  }

  return response.choices[0].message.content;
}

// Main execution
const prompt = `
        You are an HR executive scanning through an interview transcript to extract information about a candidate.
        You are required to create a JSON response with key information about the candidate.
        You will use this template for your answer:
        {
            "Name": "<candidate-name>",
            "Position": "<job position that candidate is applying for>",
            "Past experience": "<A short phrase describing the candidate's relevant past experience for the role>"
        }
        Do not include any other text in your response. Only respond in JSON format that is not surrounded by markdown code, as your response will be parsed programmatically as JSON.
        `;

// Get all files from interviews directory
const interviewFiles = readdirSync("interviews").map((file) =>
  path.join("interviews", file)
);

const header = ["Name", "Position", "Past Experience"];
const csvRows = [header.join(",")];

console.log(`Processing ${interviewFiles.length} interview files...`);

for (const interviewFile of interviewFiles) {
  console.log(`\nProcessing: ${interviewFile}`);

  // Upload file and get URL
  console.log("  Uploading file...");
  const audioUrl = await uploadFile(interviewFile);

  // Transcribe audio
  console.log("  Transcribing...");
  const transcriptId = await transcribeAudio(audioUrl);

  // Process with LLM Gateway
  console.log("  Analyzing with LLM Gateway...");
  const llmResponse = await processWithLlmGateway(transcriptId, prompt);

  // Parse JSON response
  const intervieweeData = extractJson(llmResponse);
  csvRows.push(Object.values(intervieweeData).join(","));
  console.log(`  Completed: ${intervieweeData["Name"]}`);
}

fs.writeFileSync(outputFilename, csvRows.join("\n"));
console.log(`\nCreated .csv file ${outputFilename}`);
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-Step Instructions

In this guide, we will ask the same questions to LLM Gateway about multiple files. Then, we will collate the answers in a .csv file.

Install the required packages:

```bash
pip install requests
```

Import the necessary libraries for making an HTTP request and set your API key, headers, and base URL.

```python
import requests
import json
import os
import csv
import time
import re

# Configuration
api_key = "<YOUR_API_KEY>"
base_url = "https://api.assemblyai.com"
headers = {"authorization": api_key}
output_filename = "profiles.csv"
```

```javascript
import { readdirSync } from "fs";
import fs from "fs";
import path from "path";

// Configuration
const apiKey = "<YOUR_API_KEY>";
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: apiKey };
const outputFilename = "profiles.csv";
```

Define a function to extract the JSON text from the response from LLM Gateway.

````python
def extract_json(text):
    """Extract JSON from text that might contain markdown or extra text"""
    # First, try to remove markdown code blocks
    text = text.strip()

    # Remove ```json and ``` markers
    if text.startswith("```"):
        text = re.sub(r'^```(?:json)?\s*', '', text)
        text = re.sub(r'\s*```$', '', text)

    # Find the first { and last } to extract just the JSON object
    first_brace = text.find('{')
    last_brace = text.rfind('}')

    if first_brace != -1 and last_brace != -1:
        json_str = text[first_brace:last_brace + 1]
        return json.loads(json_str)

    # If that didn't work, try parsing the whole thing
    return json.loads(text)
````

```javascript expandable
function extractJson(text) {
  // Extract JSON from text that might contain markdown or extra text
  text = text.trim();

  // Remove ```json and ``` markers
  if (text.startsWith("```")) {
    text = text.replace(/^```(?:json)?\s*/g, "");
    text = text.replace(/\s*```$/g, "");
  }

  // Find the first { and last } to extract just the JSON object
  const firstBrace = text.indexOf("{");
  const lastBrace = text.lastIndexOf("}");

  if (firstBrace !== -1 && lastBrace !== -1) {
    const jsonStr = text.slice(firstBrace, lastBrace + 1);
    return JSON.parse(jsonStr);
  }

  // If that didn't work, try parsing the whole thing
  return JSON.parse(text);
}
```

Define functions to upload and transcribe each file using AssemblyAI's Async API.

```python expandable
def upload_file(file_path):
    """Upload a local audio file to AssemblyAI"""
    with open(file_path, "rb") as f:
        response = requests.post(f"{base_url}/v2/upload", headers=headers, data=f)
        if response.status_code != 200:
            print(f"Error uploading {file_path}: {response.status_code}, {response.text}")
            response.raise_for_status()
        return response.json()["upload_url"]

def transcribe_audio(audio_url):
    """Submit audio for transcription and poll until complete"""
    # Submit transcription request
    data = {"audio_url": audio_url}
    response = requests.post(f"{base_url}/v2/transcript", headers=headers, json=data)

    if response.status_code != 200:
        print(f"Error submitting transcription: {response.status_code}, {response.text}")
        response.raise_for_status()

    transcript_id = response.json()["id"]
    polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

    # Poll for completion
    while True:
        transcript = requests.get(polling_endpoint, headers=headers).json()
        if transcript["status"] == "completed":
            return transcript_id
        elif transcript["status"] == "error":
            raise RuntimeError(f"Transcription failed: {transcript['error']}")
        else:
            time.sleep(3)
```

```javascript expandable
async function uploadFile(filePath) {
  // Upload a local audio file to AssemblyAI
  const fileData = await fs.promises.readFile(filePath);
  const res = await fetch(`${baseUrl}/v2/upload`, {
    method: "POST",
    headers,
    body: fileData,
  });
  if (!res.ok) {
    throw new Error(`Error uploading ${filePath}: ${res.status}`);
  }
  const response = await res.json();
  return response.upload_url;
}

async function transcribeAudio(audioUrl) {
  // Submit audio for transcription and poll until complete
  const data = { audio_url: audioUrl };
  const res = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(data),
  });

  if (!res.ok) {
    throw new Error(`Error submitting transcription: ${res.status}`);
  }

  const response = await res.json();
  const transcriptId = response.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  // Poll for completion
  while (true) {
    const res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    const transcript = await res.json();
    if (transcript.status === "completed") {
      return transcriptId;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}
```

Define a function to process each transcript text with LLM Gateway.

```python expandable
def process_with_llm_gateway(transcript_id, prompt):
    """Send transcript to LLM Gateway for processing"""
    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": f"{prompt}\n\n{{{{ transcript }}}}"
            }
        ],
        "transcript_id": transcript_id,
        "max_tokens": 1500
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()

    if "error" in result:
        raise RuntimeError(f"LLM Gateway error: {result['error']}")

    return result['choices'][0]['message']['content']
```

```javascript expandable
async function processWithLlmGateway(transcriptId, prompt) {
  // Send transcript to LLM Gateway for processing
  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content: `${prompt}\n\n{{ transcript }}`,
      },
    ],
    transcript_id: transcriptId,
    max_tokens: 1500,
  };

  const res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  if (response.error) {
    throw new Error(`LLM Gateway error: ${response.error}`);
  }

  return response.choices[0].message.content;
}
```

Define your LLM Gateway request prompt.

```python
prompt = """
        You are an HR executive scanning through an interview transcript to extract information about a candidate.
        You are required to create a JSON response with key information about the candidate.
        You will use this template for your answer:
        {
            "Name": "<candidate-name>",
            "Position": "<job position that candidate is applying for>",
            "Past experience": "<A short phrase describing the candidate's relevant past experience for the role>"
        }
        Do not include any other text in your response. Only respond in JSON format that is not surrounded by markdown code, as your response will be parsed programmatically as JSON.
        """
```

```javascript
const prompt = `
        You are an HR executive scanning through an interview transcript to extract information about a candidate.
        You are required to create a JSON response with key information about the candidate.
        You will use this template for your answer:
        {
            "Name": "<candidate-name>",
            "Position": "<job position that candidate is applying for>",
            "Past experience": "<A short phrase describing the candidate's relevant past experience for the role>"
        }
        Do not include any other text in your response. Only respond in JSON format that is not surrounded by markdown code, as your response will be parsed programmatically as JSON.
        `;
```

Retrieve and process each file in the `interviews` folder and create a .csv file with the results.

```python expandable
interview_files = [os.path.join("interviews", file) for file in os.listdir("interviews")]

with open(output_filename, "w", newline="") as file:
    writer = csv.writer(file)
    header = ["Name", "Position", "Past Experience"]
    writer.writerow(header)

    print(f"Processing {len(interview_files)} interview files...")

    for interview_file in interview_files:
        print(f"\nProcessing: {interview_file}")

        # Upload file and get URL
        print("  Uploading file...")
        audio_url = upload_file(interview_file)

        # Transcribe audio
        print("  Transcribing...")
        transcript_id = transcribe_audio(audio_url)

        # Process with LLM Gateway
        print("  Analyzing with LLM Gateway...")
        llm_response = process_with_llm_gateway(transcript_id, prompt)

        # Parse JSON response
        interviewee_data = extract_json(llm_response)
        writer.writerow(interviewee_data.values())
        print(f"  Completed: {interviewee_data['Name']}")

print(f"\nCreated .csv file {output_filename}")
```

```javascript expandable
// Get all files from interviews directory
const interviewFiles = readdirSync("interviews").map((file) =>
  path.join("interviews", file)
);

const header = ["Name", "Position", "Past Experience"];
const csvRows = [header.join(",")];

console.log(`Processing ${interviewFiles.length} interview files...`);

for (const interviewFile of interviewFiles) {
  console.log(`\nProcessing: ${interviewFile}`);

  // Upload file and get URL
  console.log("  Uploading file...");
  const audioUrl = await uploadFile(interviewFile);

  // Transcribe audio
  console.log("  Transcribing...");
  const transcriptId = await transcribeAudio(audioUrl);

  // Process with LLM Gateway
  console.log("  Analyzing with LLM Gateway...");
  const llmResponse = await processWithLlmGateway(transcriptId, prompt);

  // Parse JSON response
  const intervieweeData = extractJson(llmResponse);
  csvRows.push(Object.values(intervieweeData).join(","));
  console.log(`  Completed: ${intervieweeData["Name"]}`);
}

fs.writeFileSync(outputFilename, csvRows.join("\n"));
console.log(`\nCreated .csv file ${outputFilename}`);
```

For context, this is the response from LLM Gateway with our prompt.

```json
{
  "Name": "John Smith",
  "Position": "software engineer",
  "Past experience": "three years of experience at Google"
}
```

You can now run your Python script and you should see that a `profiles.csv` file is generated.

---

<a id="extract-quotes-with-timestamps-using-llm-gateway-semantic-search-docs-guides-transcript-citations-mdx"></a>

# Extract Quotes with Timestamps Using LLM Gateway + Semantic Search

URL: https://www.assemblyai.com/docs/guides/transcript-citations
Source: docs/guides/transcript-citations.mdx
Navigation: LLM Gateway > Voice AI Guides > Get quotes and citations with LLM Gateway
Description: Extract Quotes with Timestamps Using LLM Gateway + Semantic Search documentation.

This guide will demonstrate how to use AssemblyAI's [LLM Gateway](/llm-gateway) framework to process an audio file and find the best quotes included in it through [Semantic Search](https://www.elastic.co/what-is/semantic-search).

## Quickstart

```python expandable
import datetime
import numpy as np
import requests
import time
from sklearn.neighbors import NearestNeighbors
from sentence_transformers import SentenceTransformer

# Configuration
api_key = "<YOUR_API_KEY>"
base_url = "https://api.assemblyai.com"
headers = {"authorization": api_key}

def upload_file(file_path):
    """Upload a local audio file to AssemblyAI"""
    with open(file_path, "rb") as f:
        response = requests.post(f"{base_url}/v2/upload", headers=headers, data=f)
        if response.status_code != 200:
            print(f"Error uploading: {response.status_code}, {response.text}")
            response.raise_for_status()
        return response.json()["upload_url"]

def transcribe_audio(audio_url):
    """Submit audio for transcription with sentences enabled and poll until complete"""
    data = {
        "audio_url": audio_url,
        "auto_highlights": False,
        "sentiment_analysis": False,
        "entity_detection": False
    }

    response = requests.post(f"{base_url}/v2/transcript", headers=headers, json=data)

    if response.status_code != 200:
        print(f"Error submitting transcription: {response.status_code}, {response.text}")
        response.raise_for_status()

    transcript_id = response.json()["id"]
    polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

    print("Transcribing...")
    while True:
        transcript = requests.get(polling_endpoint, headers=headers).json()
        if transcript["status"] == "completed":
            print("Transcription completed!")
            return transcript
        elif transcript["status"] == "error":
            raise RuntimeError(f"Transcription failed: {transcript['error']}")
        else:
            time.sleep(3)

def get_sentences(transcript_id):
    """Get sentences from a completed transcript"""
    sentences_endpoint = f"{base_url}/v2/transcript/{transcript_id}/sentences"
    response = requests.get(sentences_endpoint, headers=headers)

    if response.status_code != 200:
        print(f"Error getting sentences: {response.status_code}, {response.text}")
        response.raise_for_status()

    return response.json()["sentences"]

def process_with_llm_gateway(transcript_text, question, context=""):
    """Send transcript to LLM Gateway for question answering"""
    prompt = f"""Based on the following transcript, please answer this question:
            Question: {question}
            Context: {context}
            Transcript: {transcript_text}
            Please provide a clear and specific answer."""

    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": prompt
            }
        ],
        "max_tokens": 2000
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()

    if "error" in result:
        raise RuntimeError(f"LLM Gateway error: {result['error']}")

    return result['choices'][0]['message']['content']

def sliding_window(elements, distance, stride):
    """Create sliding windows of elements"""
    idx = 0
    results = []
    while idx + distance < len(elements):
        results.append(elements[idx:idx + distance])
        idx += (distance - stride)
    return results

# Main execution
# If using a local file:
audio_url = upload_file("<YOUR_AUDIO FILE>")

# If using a public URL:
# audio_url = "<YOUR_AUDIO_URL>"

# Transcribe audio
transcript = transcribe_audio(audio_url)
transcript_text = transcript["text"]
transcript_id = transcript["id"]

# Get sentences
print("Getting sentences...")
sentences = get_sentences(transcript_id)

# Initialize embedder
embedder = SentenceTransformer("multi-qa-mpnet-base-dot-v1")
embeddings = {}

# Create sliding window of sentences and generate embeddings
print("Creating embeddings...")
sentence_groups = sliding_window(sentences, 5, 2)

for sentence_group in sentence_groups:
    combined_text = " ".join([sentence["text"] for sentence in sentence_group])
    start = sentence_group[0]["start"]
    end = sentence_group[-1]["end"]

    embeddings[(start, end, transcript_id, combined_text)] = embedder.encode(combined_text)

# Use LLM Gateway to find the best quotes
print("Asking LLM Gateway for best quotes...")
question = "What are the 3 best quotes from this video?"
context = "Please provide exactly 3 quotes."

llm_answer = process_with_llm_gateway(transcript_text, question, context)
print(f"\nLLM Gateway Response:\n{llm_answer}\n")

# Embed the LLM output
llm_gateway_embedding = embedder.encode(llm_answer)

# Vectorize transcript embeddings
np_embeddings = np.array(list(embeddings.values()))
metadata = list(embeddings.keys())

# Find the top 3 most similar quotes
print("Finding matching quotes in transcript...")
knn = NearestNeighbors(n_neighbors=3, metric="cosine")
knn.fit(np_embeddings)
distances, indices = knn.kneighbors([llm_gateway_embedding])

matches = []
for distance, index in zip(distances[0], indices[0]):
    result_metadata = metadata[index]
    matches.append(
        {
            "start_timestamp": result_metadata[0],
            "end_timestamp": result_metadata[1],
            "transcript_id": result_metadata[2],
            "text": result_metadata[3],
            "confidence": 1 - distance,
        }
    )

# Display results
print("\n" + "="*80)
print("BEST MATCHING QUOTES FROM TRANSCRIPT:")
print("="*80 + "\n")

for index, m in enumerate(matches):
    print('QUOTE #{}: "{}"'.format(index + 1, m['text']))
    print('START TIMESTAMP:', str(datetime.timedelta(seconds=m['start_timestamp']/1000)))
    print('END TIMESTAMP:', str(datetime.timedelta(seconds=m['end_timestamp']/1000)))
    print('CONFIDENCE:', m['confidence'])
    print()
```

```javascript expandable
import { pipeline } from "@xenova/transformers";

// Configuration
const apiKey = "<YOUR_API_KEY>";
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: apiKey };

function msToTime(ms) {
  const totalSeconds = Math.floor(ms / 1000);
  const h = Math.floor(totalSeconds / 3600);
  const m = Math.floor((totalSeconds % 3600) / 60);
  const s = totalSeconds % 60;
  return `${h}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}`;
}

function cosineSimilarity(a, b) {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}

async function uploadFile(filePath) {
  // Upload a local audio file to AssemblyAI
  const { default: fs } = await import("fs");
  const fileData = await fs.promises.readFile(filePath);
  const res = await fetch(`${baseUrl}/v2/upload`, {
    method: "POST",
    headers,
    body: fileData,
  });
  if (!res.ok) {
    throw new Error(`Error uploading: ${res.status}`);
  }
  const response = await res.json();
  return response.upload_url;
}

async function transcribeAudio(audioUrl) {
  // Submit audio for transcription with sentences enabled and poll until complete
  const data = {
    audio_url: audioUrl,
    auto_highlights: false,
    sentiment_analysis: false,
    entity_detection: false,
  };

  const res = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(data),
  });

  if (!res.ok) {
    throw new Error(`Error submitting transcription: ${res.status}`);
  }

  const response = await res.json();
  const transcriptId = response.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  console.log("Transcribing...");
  while (true) {
    const res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    const transcript = await res.json();
    if (transcript.status === "completed") {
      console.log("Transcription completed!");
      return transcript;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

async function getSentences(transcriptId) {
  // Get sentences from a completed transcript
  const sentencesEndpoint = `${baseUrl}/v2/transcript/${transcriptId}/sentences`;
  const res = await fetch(sentencesEndpoint, { headers });

  if (!res.ok) {
    throw new Error(`Error getting sentences: ${res.status}`);
  }

  const response = await res.json();
  return response.sentences;
}

async function processWithLlmGateway(transcriptText, question, context = "") {
  // Send transcript to LLM Gateway for question answering
  const prompt = `Based on the following transcript, please answer this question:
            Question: ${question}
            Context: ${context}
            Transcript: ${transcriptText}
            Please provide a clear and specific answer.`;

  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content: prompt,
      },
    ],
    max_tokens: 2000,
  };

  const res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  if (response.error) {
    throw new Error(`LLM Gateway error: ${response.error}`);
  }

  return response.choices[0].message.content;
}

function slidingWindow(elements, distance, stride) {
  // Create sliding windows of elements
  let idx = 0;
  const results = [];
  while (idx + distance < elements.length) {
    results.push(elements.slice(idx, idx + distance));
    idx += distance - stride;
  }
  return results;
}

// Main execution
// If using a local file:
const audioUrl = await uploadFile("<YOUR_AUDIO FILE>");

// If using a public URL:
// const audioUrl = "<YOUR_AUDIO_URL>";

// Transcribe audio
const transcript = await transcribeAudio(audioUrl);
const transcriptText = transcript.text;
const transcriptId = transcript.id;

// Get sentences
console.log("Getting sentences...");
const sentences = await getSentences(transcriptId);

// Initialize embedder
const embedder = await pipeline(
  "feature-extraction",
  "Xenova/multi-qa-mpnet-base-dot-v1"
);
const embeddingKeys = [];
const embeddingValues = [];

// Create sliding window of sentences and generate embeddings
console.log("Creating embeddings...");
const sentenceGroups = slidingWindow(sentences, 5, 2);

for (const sentenceGroup of sentenceGroups) {
  const combinedText = sentenceGroup.map((s) => s.text).join(" ");
  const start = sentenceGroup[0].start;
  const end = sentenceGroup[sentenceGroup.length - 1].end;

  const output = await embedder(combinedText, {
    pooling: "mean",
    normalize: true,
  });
  const embedding = Array.from(output.data);

  embeddingKeys.push({ start, end, transcriptId, combinedText });
  embeddingValues.push(embedding);
}

// Use LLM Gateway to find the best quotes
console.log("Asking LLM Gateway for best quotes...");
const question = "What are the 3 best quotes from this video?";
const context = "Please provide exactly 3 quotes.";

const llmAnswer = await processWithLlmGateway(
  transcriptText,
  question,
  context
);
console.log(`\nLLM Gateway Response:\n${llmAnswer}\n`);

// Embed the LLM output
const llmOutput = await embedder(llmAnswer, { pooling: "mean", normalize: true });
const llmGatewayEmbedding = Array.from(llmOutput.data);

// Find the top 3 most similar quotes
console.log("Finding matching quotes in transcript...");
const similarities = embeddingKeys.map((key, idx) => ({
  key,
  similarity: cosineSimilarity(llmGatewayEmbedding, embeddingValues[idx]),
}));
similarities.sort((a, b) => b.similarity - a.similarity);
const matches = similarities.slice(0, 3).map(({ key, similarity }) => ({
  startTimestamp: key.start,
  endTimestamp: key.end,
  transcriptId: key.transcriptId,
  text: key.combinedText,
  confidence: similarity,
}));

// Display results
console.log("\n" + "=".repeat(80));
console.log("BEST MATCHING QUOTES FROM TRANSCRIPT:");
console.log("=".repeat(80) + "\n");

matches.forEach((m, index) => {
  console.log(`QUOTE #${index + 1}: "${m.text}"`);
  console.log("START TIMESTAMP:", msToTime(m.startTimestamp));
  console.log("END TIMESTAMP:", msToTime(m.endTimestamp));
  console.log("CONFIDENCE:", m.confidence);
  console.log();
});
```

## Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://www.assemblyai.com/dashboard/signup) for an AssemblyAI account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home).

## Step-by-Step Instructions

Install the required packages:

You'll need to install a few libraries that this code depends on:

- [Numpy](https://numpy.org/), a scientific computing library.
- [Sciki-Learn](https://scikit-learn.org/stable/), a library for predictive data analysis.
- [Sentence-Transformers](https://www.sbert.net/index.html), a framework for state-of-the-art sentence and text embedding.

```bash
pip install -U numpy scikit-learn sentence-transformers
```

```bash
npm install @xenova/transformers
```

Then import all of these libraries and set our AssemblyAI API key, headers, and base URL.

```python
import datetime
import numpy as np
import requests
import time
from sklearn.neighbors import NearestNeighbors
from sentence_transformers import SentenceTransformer

# Configuration
api_key = "<YOUR_API_KEY>"
base_url = "https://api.assemblyai.com"
headers = {"authorization": api_key}
```

```javascript
import { pipeline } from "@xenova/transformers";

// Configuration
const apiKey = "<YOUR_API_KEY>";
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: apiKey };
```

Next, define functions to upload and transcribe files using AssemblyAI's Async API, as well as request sentences.

```python expandable
def upload_file(file_path):
    """Upload a local audio file to AssemblyAI"""
    with open(file_path, "rb") as f:
        response = requests.post(f"{base_url}/v2/upload", headers=headers, data=f)
        if response.status_code != 200:
            print(f"Error uploading: {response.status_code}, {response.text}")
            response.raise_for_status()
        return response.json()["upload_url"]

def transcribe_audio(audio_url):
    """Submit audio for transcription with sentences enabled and poll until complete"""
    data = {
        "audio_url": audio_url,
        "auto_highlights": False,
        "sentiment_analysis": False,
        "entity_detection": False
    }

    response = requests.post(f"{base_url}/v2/transcript", headers=headers, json=data)

    if response.status_code != 200:
        print(f"Error submitting transcription: {response.status_code}, {response.text}")
        response.raise_for_status()

    transcript_id = response.json()["id"]
    polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

    print("Transcribing...")
    while True:
        transcript = requests.get(polling_endpoint, headers=headers).json()
        if transcript["status"] == "completed":
            print("Transcription completed!")
            return transcript
        elif transcript["status"] == "error":
            raise RuntimeError(f"Transcription failed: {transcript['error']}")
        else:
            time.sleep(3)

def get_sentences(transcript_id):
    """Get sentences from a completed transcript"""
    sentences_endpoint = f"{base_url}/v2/transcript/{transcript_id}/sentences"
    response = requests.get(sentences_endpoint, headers=headers)

    if response.status_code != 200:
        print(f"Error getting sentences: {response.status_code}, {response.text}")
        response.raise_for_status()

    return response.json()["sentences"]
```

```javascript expandable
async function uploadFile(filePath) {
  // Upload a local audio file to AssemblyAI
  const { default: fs } = await import("fs");
  const fileData = await fs.promises.readFile(filePath);
  const res = await fetch(`${baseUrl}/v2/upload`, {
    method: "POST",
    headers,
    body: fileData,
  });
  if (!res.ok) {
    throw new Error(`Error uploading: ${res.status}`);
  }
  const response = await res.json();
  return response.upload_url;
}

async function transcribeAudio(audioUrl) {
  // Submit audio for transcription with sentences enabled and poll until complete
  const data = {
    audio_url: audioUrl,
    auto_highlights: false,
    sentiment_analysis: false,
    entity_detection: false,
  };

  const res = await fetch(`${baseUrl}/v2/transcript`, {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(data),
  });

  if (!res.ok) {
    throw new Error(`Error submitting transcription: ${res.status}`);
  }

  const response = await res.json();
  const transcriptId = response.id;
  const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

  console.log("Transcribing...");
  while (true) {
    const res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    const transcript = await res.json();
    if (transcript.status === "completed") {
      console.log("Transcription completed!");
      return transcript;
    } else if (transcript.status === "error") {
      throw new Error(`Transcription failed: ${transcript.error}`);
    } else {
      await new Promise((resolve) => setTimeout(resolve, 3000));
    }
  }
}

async function getSentences(transcriptId) {
  // Get sentences from a completed transcript
  const sentencesEndpoint = `${baseUrl}/v2/transcript/${transcriptId}/sentences`;
  const res = await fetch(sentencesEndpoint, { headers });

  if (!res.ok) {
    throw new Error(`Error getting sentences: ${res.status}`);
  }

  const response = await res.json();
  return response.sentences;
}
```

Then define a function to process each transcript text with LLM Gateway.

```python expandable
def process_with_llm_gateway(transcript_text, question, context=""):
    """Send transcript to LLM Gateway for question answering"""
    prompt = f"""Based on the following transcript, please answer this question:
            Question: {question}
            Context: {context}
            Transcript: {transcript_text}
            Please provide a clear and specific answer."""

    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {
                "role": "user",
                "content": prompt
            }
        ],
        "max_tokens": 2000
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()

    if "error" in result:
        raise RuntimeError(f"LLM Gateway error: {result['error']}")

    return result['choices'][0]['message']['content']
```

```javascript expandable
async function processWithLlmGateway(transcriptText, question, context = "") {
  // Send transcript to LLM Gateway for question answering
  const prompt = `Based on the following transcript, please answer this question:
            Question: ${question}
            Context: ${context}
            Transcript: ${transcriptText}
            Please provide a clear and specific answer.`;

  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      {
        role: "user",
        content: prompt,
      },
    ],
    max_tokens: 2000,
  };

  const res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  if (response.error) {
    throw new Error(`LLM Gateway error: ${response.error}`);
  }

  return response.choices[0].message.content;
}
```

Define a function to implement a [sliding window](https://blandthony.medium.com/methods-for-semantic-text-segmentation-prior-to-generating-text-embeddings-vectorization-6442afdb086), which allows us to group sentences together in different combinations to retain their semantic meaning and context while also enabling us to customize the length (and thus duration) of the quotes.

```python
def sliding_window(elements, distance, stride):
    """Create sliding windows of elements"""
    idx = 0
    results = []
    while idx + distance < len(elements):
        results.append(elements[idx:idx + distance])
        idx += (distance - stride)
    return results
```

```javascript
function slidingWindow(elements, distance, stride) {
  // Create sliding windows of elements
  let idx = 0;
  const results = [];
  while (idx + distance < elements.length) {
    results.push(elements.slice(idx, idx + distance));
    idx += distance - stride;
  }
  return results;
}
```

Execute all upload and transcription functions.

```python
# Main execution
# If using a local file:
audio_url = upload_file("<YOUR_AUDIO FILE>")

# If using a public URL:
# audio_url = "<YOUR_AUDIO_URL>"

# Transcribe audio
transcript = transcribe_audio(audio_url)
transcript_text = transcript["text"]
transcript_id = transcript["id"]

# Get sentences
print("Getting sentences...")
sentences = get_sentences(transcript_id)
```

```javascript
// Main execution
// If using a local file:
const audioUrl = await uploadFile("<YOUR_AUDIO FILE>");

// If using a public URL:
// const audioUrl = "<YOUR_AUDIO_URL>";

// Transcribe audio
const transcript = await transcribeAudio(audioUrl);
const transcriptText = transcript.text;
const transcriptId = transcript.id;

// Get sentences
console.log("Getting sentences...");
const sentences = await getSentences(transcriptId);
```

Now we can iterate over all of the sentences in our transcript and create [embeddings](https://github.com/microsoft/semantic-kernel/blob/main/docs/EMBEDDINGS.md) for them to use as part of our Semantic Search later.

We'll be relying on SentenceTransformer's `multi-qa-mpnet-base-dot-v1` model, which has been fine-tuned specifically for Semantic Search, and is their [highest-performing model](https://www.sbert.net/docs/pretrained_models.html) for this task.

By default, we'll group 5 sentences together while having 2 of them overlap when the window moves. This should give us quotes around 30 seconds in length at most.

```python
# Initialize embedder
embedder = SentenceTransformer("multi-qa-mpnet-base-dot-v1")
embeddings = {}

# Create sliding window of sentences and generate embeddings
print("Creating embeddings...")
sentence_groups = sliding_window(sentences, 5, 2)

for sentence_group in sentence_groups:
    combined_text = " ".join([sentence["text"] for sentence in sentence_group])
    start = sentence_group[0]["start"]
    end = sentence_group[-1]["end"]

    embeddings[(start, end, transcript_id, combined_text)] = embedder.encode(combined_text)
```

```javascript expandable
// Initialize embedder
const embedder = await pipeline(
  "feature-extraction",
  "Xenova/multi-qa-mpnet-base-dot-v1"
);
const embeddingKeys = [];
const embeddingValues = [];

// Create sliding window of sentences and generate embeddings
console.log("Creating embeddings...");
const sentenceGroups = slidingWindow(sentences, 5, 2);

for (const sentenceGroup of sentenceGroups) {
  const combinedText = sentenceGroup.map((s) => s.text).join(" ");
  const start = sentenceGroup[0].start;
  const end = sentenceGroup[sentenceGroup.length - 1].end;

  const output = await embedder(combinedText, {
    pooling: "mean",
    normalize: true,
  });
  const embedding = Array.from(output.data);

  embeddingKeys.push({ start, end, transcriptId, combinedText });
  embeddingValues.push(embedding);
}
```

Now we can query LLM Gateway to provide the type of quotes we want. In this case, let's prompt LLM Gateway to find the best 3 quotes out of a video that we transcribed.

```python
print("Asking LLM Gateway for best quotes...")
question = "What are the 3 best quotes from this video?"
context = "Please provide exactly 3 quotes."

llm_answer = process_with_llm_gateway(transcript_text, question, context)
print(f"\nLLM Gateway Response:\n{llm_answer}\n")
```

```javascript
console.log("Asking LLM Gateway for best quotes...");
const question = "What are the 3 best quotes from this video?";
const context = "Please provide exactly 3 quotes.";

const llmAnswer = await processWithLlmGateway(
  transcriptText,
  question,
  context
);
console.log(`\nLLM Gateway Response:\n${llmAnswer}\n`);
```

Now we can take the embeddings from the transcript text, as well as the embeddings from LLM Gateway's output, and use them in our [k-nearest neighbors](https://www.ibm.com/topics/knn) algorithm to determine their similarity. The most similar quotes to what LLM Gateway identified will be surfaced as our 3 best quotes, along with their timestamps and confidence scores.

We'll be relying on [cosine similarity](https://github.com/microsoft/semantic-kernel/blob/main/docs/COSINE_SIMILARITY.md) rather than the default Euclidean distance metric since it takes into account both the magnitude and direction of our vectors.

```python expandable
# Embed the LLM output
llm_gateway_embedding = embedder.encode(llm_answer)

# Vectorize transcript embeddings
np_embeddings = np.array(list(embeddings.values()))
metadata = list(embeddings.keys())

# Find the top 3 most similar quotes
print("Finding matching quotes in transcript...")
knn = NearestNeighbors(n_neighbors=3, metric="cosine")
knn.fit(np_embeddings)
distances, indices = knn.kneighbors([llm_gateway_embedding])

matches = []
for distance, index in zip(distances[0], indices[0]):
    result_metadata = metadata[index]
    matches.append(
        {
            "start_timestamp": result_metadata[0],
            "end_timestamp": result_metadata[1],
            "transcript_id": result_metadata[2],
            "text": result_metadata[3],
            "confidence": 1 - distance,
        }
    )

# Display results
print("\n" + "="*80)
print("BEST MATCHING QUOTES FROM TRANSCRIPT:")
print("="*80 + "\n")

for index, m in enumerate(matches):
    print('QUOTE #{}: "{}"'.format(index + 1, m['text']))
    print('START TIMESTAMP:', str(datetime.timedelta(seconds=m['start_timestamp']/1000)))
    print('END TIMESTAMP:', str(datetime.timedelta(seconds=m['end_timestamp']/1000)))
    print('CONFIDENCE:', m['confidence'])
    print()
```

```javascript expandable
// Embed the LLM output
const llmOutput = await embedder(llmAnswer, {
  pooling: "mean",
  normalize: true,
});
const llmGatewayEmbedding = Array.from(llmOutput.data);

// Find the top 3 most similar quotes
console.log("Finding matching quotes in transcript...");

function cosineSimilarity(a, b) {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}

function msToTime(ms) {
  const totalSeconds = Math.floor(ms / 1000);
  const h = Math.floor(totalSeconds / 3600);
  const m = Math.floor((totalSeconds % 3600) / 60);
  const s = totalSeconds % 60;
  return `${h}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}`;
}

const similarities = embeddingKeys.map((key, idx) => ({
  key,
  similarity: cosineSimilarity(llmGatewayEmbedding, embeddingValues[idx]),
}));
similarities.sort((a, b) => b.similarity - a.similarity);
const matches = similarities.slice(0, 3).map(({ key, similarity }) => ({
  startTimestamp: key.start,
  endTimestamp: key.end,
  transcriptId: key.transcriptId,
  text: key.combinedText,
  confidence: similarity,
}));

// Display results
console.log("\n" + "=".repeat(80));
console.log("BEST MATCHING QUOTES FROM TRANSCRIPT:");
console.log("=".repeat(80) + "\n");

matches.forEach((m, index) => {
  console.log(`QUOTE #${index + 1}: "${m.text}"`);
  console.log("START TIMESTAMP:", msToTime(m.startTimestamp));
  console.log("END TIMESTAMP:", msToTime(m.endTimestamp));
  console.log("CONFIDENCE:", m.confidence);
  console.log();
});
```

---

<a id="extract-transcript-quotes-with-llm-gateway-docs-guides-timestamped-transcripts-mdx"></a>

# Extract Transcript Quotes with LLM Gateway

URL: https://www.assemblyai.com/docs/guides/timestamped-transcripts
Source: docs/guides/timestamped-transcripts.mdx
Navigation: LLM Gateway > Voice AI Guides > Get quotes and citations with LLM Gateway
Description: Extract Transcript Quotes with LLM Gateway documentation.

This guide will demonstrate how to use AssemblyAI's [LLM Gateway](/llm-gateway/quickstart) to process an audio file and find the best quotes via the [chat completions endpoint](/api-reference/llm-gateway/create-chat-completion).

## Quickstart

```python expandable
import requests
import json
import time

API_KEY = "YOUR_API_KEY"
headers = {"authorization": API_KEY}

# Transcribe the audio file

print("Submitting audio for transcription...")
transcript_response = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers=headers,
    json={
        "audio_url": "https://assembly.ai/wildfires.mp3",
        "speaker_labels": True,
        "language_detection": True
    }
)

transcript_id = transcript_response.json()["id"]

# Poll for transcription completion

while True:
    transcript_result = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers=headers
    ).json()

    if transcript_result["status"] == "completed":
        break
    elif transcript_result["status"] == "error":
        raise Exception(f"Transcription failed: {transcript_result['error']}")

    time.sleep(3)

# Extract utterances with timestamps

utterances_data = [
    {"text": u["text"], "start": u["start"], "end": u["end"], "speaker": u["speaker"]}
    for u in transcript_result["utterances"]
]

# Create prompt with timestamped utterances

prompt = f"""You are analyzing a transcript with timestamped utterances. Each utterance includes the text content, speaker label, and start/end timestamps in milliseconds.

Here is the transcript data:
{json.dumps(utterances_data, indent=2)}

Task: Identify the 3-5 most engaging, impactful, or quotable utterances from this transcript.

Return your response as a JSON array with the following structure:
{{
  "quotes": [
    {{
      "text": "exact quote text",
      "start": start_timestamp_in_milliseconds,
      "end": end_timestamp_in_milliseconds,
      "speaker": "speaker_label",
      "reason": "brief explanation of why this quote is engaging"
    }}
]
}}

Return ONLY valid JSON, no additional text."""

# Use LLM Gateway to extract quotes

print("Submitting transcript to LLM Gateway for quote extraction...")
gateway_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": prompt}
        ]
    }
)

result = gateway_response.json()
content = result["choices"][0]["message"]["content"].strip().strip("```").removeprefix("json").strip()
quotes_json = json.loads(content)
print(json.dumps(quotes_json, indent=2))

````

```javascript expandable
const API_KEY = "YOUR_API_KEY";

const headers = {
  "authorization": API_KEY,
  "Content-Type": "application/json"
};

// Transcribe the audio file
console.log("Submitting audio for transcription...");
let res = await fetch(
  "https://api.assemblyai.com/v2/transcript",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      audio_url: "https://assembly.ai/wildfires.mp3",
      speaker_labels: true,
      language_detection: true
    })
  }
);
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;

// Poll for transcription completion
let transcriptResult;
while (true) {
  res = await fetch(
    `https://api.assemblyai.com/v2/transcript/${transcriptId}`,
    { headers }
  );
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcriptResult = await res.json();

  if (transcriptResult.status === "completed") {
    break;
  } else if (transcriptResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptResult.error}`);
  }

  await new Promise(resolve => setTimeout(resolve, 3000));
}

// Extract utterances with timestamps
const utterancesData = transcriptResult.utterances.map(u => ({
  text: u.text,
  start: u.start,
  end: u.end,
  speaker: u.speaker
}));

// Create prompt with timestamped utterances
const prompt = `You are analyzing a transcript with timestamped utterances. Each utterance includes the text content, speaker label, and start/end timestamps in milliseconds.

Here is the transcript data:
${JSON.stringify(utterancesData, null, 2)}

Task: Identify the 3-5 most engaging, impactful, or quotable utterances from this transcript.

Return your response as a JSON array with the following structure:
{
  "quotes": [
    {
      "text": "exact quote text",
      "start": start_timestamp_in_milliseconds,
      "end": end_timestamp_in_milliseconds,
      "speaker": "speaker_label",
      "reason": "brief explanation of why this quote is engaging"
    }
  ]
}

Return ONLY valid JSON, no additional text.`;

// Use LLM Gateway to extract quotes
console.log("Submitting transcript to LLM Gateway for quote extraction...");
res = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [
        { role: "user", content: prompt }
      ]
    })
  }
);
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();

const quotesJson = JSON.parse(result.choices[0].message.content.replace(/^```json\n?/, '').replace(/```$/, '').trim());
console.log(JSON.stringify(quotesJson, null, 2));
````

## Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://www.assemblyai.com/dashboard/signup/) for an AssemblyAI account and get your API key from your [dashboard](https://www.assemblyai.com/dashboard/home).

## Step-by-Step Instructions

### Step 1: Install dependencies

Install the required library:

```bash
pip install requests
```

### Step 2: Set up your API key and headers

```python
import requests
import json
import time

API_KEY = "YOUR_API_KEY"
headers = {"authorization": API_KEY}

````

```javascript
const API_KEY = "YOUR_API_KEY";

const headers = {
  "authorization": API_KEY,
  "Content-Type": "application/json"
};

````

### Step 3: Transcribe the audio file

Next, we'll use AssemblyAI to transcribe a file and save our transcript for later use. We'll enable `speaker_labels` to get utterances grouped by speaker.

```python expandable
# Transcribe the audio file
print("Submitting audio for transcription...")
transcript_response = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers=headers,
    json={
        "audio_url": "https://assembly.ai/wildfires.mp3",
        "speaker_labels": True,
        "language_detection": True
    }
)

transcript_id = transcript_response.json()["id"]

# Poll for transcription completion

while True:
    transcript_result = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers=headers
    ).json()

    if transcript_result["status"] == "completed":
        break
    elif transcript_result["status"] == "error":
        raise Exception(f"Transcription failed: {transcript_result['error']}")

    time.sleep(3)

````

```javascript expandable
// Transcribe the audio file
console.log("Submitting audio for transcription...");
let res = await fetch(
  "https://api.assemblyai.com/v2/transcript",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      audio_url: "https://assembly.ai/wildfires.mp3",
      speaker_labels: true,
      language_detection: true
    })
  }
);
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;

// Poll for transcription completion
let transcriptResult;
while (true) {
  res = await fetch(
    `https://api.assemblyai.com/v2/transcript/${transcriptId}`,
    { headers }
  );
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcriptResult = await res.json();

  if (transcriptResult.status === "completed") {
    break;
  } else if (transcriptResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptResult.error}`);
  }

  await new Promise(resolve => setTimeout(resolve, 3000));
}
````

### Step 4: Extract utterances with timestamps

Then we'll take the timestamped `utterances` array from our transcript and format it as structured data. Utterances are grouped by speaker and include continuous speech segments.

```python
utterances_data = [
    {"text": u["text"], "start": u["start"], "end": u["end"], "speaker": u["speaker"]} 
    for u in transcript_result["utterances"]
]
```

```javascript
const utterancesData = transcriptResult.utterances.map(u => ({
  text: u.text,
  start: u.start,
  end: u.end,
  speaker: u.speaker
}));
```

### Step 5: Use LLM Gateway to extract engaging quotes

Finally, we'll provide the timestamped utterances to the LLM Gateway chat completions endpoint to extract the most engaging quotes from this transcript with their associated timestamps in a structured JSON format.

```python expandable
# Create prompt with timestamped utterances
prompt = f"""You are analyzing a transcript with timestamped utterances. Each utterance includes the text content, speaker label, and start/end timestamps in milliseconds.

Here is the transcript data:
{json.dumps(utterances_data, indent=2)}

Task: Identify the 3-5 most engaging, impactful, or quotable utterances from this transcript.

Return your response as a JSON array with the following structure:
{{
  "quotes": [
    {{
      "text": "exact quote text",
      "start": start_timestamp_in_milliseconds,
      "end": end_timestamp_in_milliseconds,
      "speaker": "speaker_label",
      "reason": "brief explanation of why this quote is engaging"
    }}
]
}}

Return ONLY valid JSON, no additional text."""

# Use LLM Gateway to extract quotes

print("Submitting transcript to LLM Gateway for quote extraction...")
gateway_response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": prompt}
        ]
    }
)

result = gateway_response.json()
content = result["choices"][0]["message"]["content"].strip().strip("```").removeprefix("json").strip()
quotes_json = json.loads(content)
print(json.dumps(quotes_json, indent=2))

````

```javascript expandable
// Create prompt with timestamped utterances
const prompt = `You are analyzing a transcript with timestamped utterances. Each utterance includes the text content, speaker label, and start/end timestamps in milliseconds.

Here is the transcript data:
${JSON.stringify(utterancesData, null, 2)}

Task: Identify the 3-5 most engaging, impactful, or quotable utterances from this transcript.

Return your response as a JSON array with the following structure:
{
  "quotes": [
    {
      "text": "exact quote text",
      "start": start_timestamp_in_milliseconds,
      "end": end_timestamp_in_milliseconds,
      "speaker": "speaker_label",
      "reason": "brief explanation of why this quote is engaging"
    }
  ]
}

Return ONLY valid JSON, no additional text.`;

// Use LLM Gateway to extract quotes
console.log("Submitting transcript to LLM Gateway for quote extraction...");
let res = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [
        { role: "user", content: prompt }
      ]
    })
  }
);
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();

const quotesJson = JSON.parse(result.choices[0].message.content.replace(/^```json\n?/, '').replace(/```$/, '').trim());
console.log(JSON.stringify(quotesJson, null, 2));
````

## Example Response

```json expandable
{
  "quotes": [
    {
      "text": "It is, it is. The levels outside right now in Baltimore are considered unhealthy. And most of that is due to what's called particulate matter, which are tiny particles, microscopic, smaller than the width of your hair, that can get into your lungs and impact your respiratory system, your cardiovascular system, and even your neurological, your brain.",
      "start": 62350,
      "end": 82590,
      "speaker": "B",
      "reason": "Defines particulate matter and explains how it harms health."
    },
    {
      "text": "Yeah. So the concentration of particulate matter, I was looking at some of the monitors that we have was reaching levels of what are, in science speak, 150 micrograms per meter cubed, which is more than 10 times what the annual average should be in about four times higher than what you're supposed to have on a 24 hour average. And so the concentrations of these particles in the air are just much, much, much higher than we typically see. And exposure to those high levels can lead to a host of health problems.",
      "start": 93550,
      "end": 123350,
      "speaker": "B",
      "reason": "Gives specific concentration figures and links to health risks."
    },
    {
      "text": "It's the youngest. So children, obviously, whose bodies are still developing, the elderly who are, you know, their bodies are more in decline and they're more susceptible to the health impacts of breathing, the poor air quality. And then people who have pre existing health conditions, people with respiratory conditions or heart conditions, can be triggered by high levels of air pollution.",
      "start": 137610,
      "end": 156650,
      "speaker": "B",
      "reason": "Highlights the most vulnerable groups affected by poor air quality."
    },
    {
      "text": "Well, I think the fires are going to burn for a little bit longer. But the key for us in the US Is the weather system changing. Right now it's the weather systems that are pulling that air into our Mid Atlantic and Northeast region. As those weather systems change and shift, we'll see that smoke going elsewhere and not impact us in this region as much. I think that's going to be the defining factor. I think the next couple days we're going to see a shift in that weather pattern and start to push the smoke away from where we are.",
      "start": 198280,
      "end": 227480,
      "speaker": "B",
      "reason": "Offers an outlook on how weather patterns may reduce exposure."
    },
    {
      "text": "I mean, that is one of the predictions for climate change. Looking into the future, the fire season is starting earlier and lasting longer and we're seeing more frequent fires. So yeah, this is probably something that we'll be seeing more, more frequently. This tends to be much more of an issue in the western U.S. so the eastern U.S. getting hit right now is a little bit new. But yeah, I think with climate change moving forward, this is something that is going to happen more frequently.",
      "start": 241370,
      "end": 267570,
      "speaker": "B",
      "reason": "Connects current event to longer-term climate change trends and future frequency."
    }
  ]
}
```

---

<a id="analyze-the-sentiment-of-a-customer-call-using-llm-gateway-docs-guides-call-sentiment-analysis-mdx"></a>

# Analyze The Sentiment Of A Customer Call using LLM Gateway

URL: https://www.assemblyai.com/docs/guides/call-sentiment-analysis
Source: docs/guides/call-sentiment-analysis.mdx
Navigation: LLM Gateway > Voice AI Guides > Substitute Speech Understanding with LLM Gateway
Description: Analyze The Sentiment Of A Customer Call using LLM Gateway documentation.

In this guide, we'll show you how to use AssemblyAI's LLM Gateway framework to process an audio file and then use [LLM Gateway](/llm-gateway) to automatically detect sentiment analysis from customer calls as "positive", "negative", or "neutral". In addition, we will glean additional insights beyond these three sentiments and learn the reasoning behind these detected sentiments.

## Quickstart

```python expandable
import requests
import time

API_KEY = "YOUR_API_KEY"
audio_file_path = "./meeting.mp3"

# ------------------------------------------
# Step 1: Upload the audio file
# ------------------------------------------
def upload_file(filename):
    with open(filename, "rb") as f:
        upload_url = "https://api.assemblyai.com/v2/upload"
        headers = {"authorization": API_KEY}
        response = requests.post(upload_url, headers=headers, data=f)
        response.raise_for_status()
        return response.json()["upload_url"]

audio_url = upload_file(audio_file_path)
print(f"Uploaded audio file. URL: {audio_url}")

# ------------------------------------------
# Step 2: Request transcription
# ------------------------------------------
transcript_request = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": API_KEY, "content-type": "application/json"},
    json={"audio_url": audio_url},
)

transcript_id = transcript_request.json()["id"]

# Poll until completed
while True:
    polling_response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": API_KEY},
    )
    status = polling_response.json()["status"]

    if status == "completed":
        break
    elif status == "error":
        raise RuntimeError(f"Transcription failed: {polling_response.json()['error']}")
    else:
        print(f"Transcription status: {status}")
        time.sleep(3)

print("\nTranscription complete.\n")

# ------------------------------------------
# Step 3: Define questions
# ------------------------------------------
agent_context = "The agent is trying to get the customer to go through with the update to their car."
customer_context = "The customer is calling to check how much it would cost to update the map in his car."

answer_format = "<answer in one sentence> <reason in one sentence>"

questions = [
    {
        "question": "What was the overall sentiment of the call?",
        "context": customer_context,
        "answer_format": answer_format,
    },
    {
        "question": "What was the sentiment of the agent in this call?",
        "context": agent_context,
        "answer_format": answer_format,
    },
    {
        "question": "What was the sentiment of the customer in this call?",
        "context": customer_context,
        "answer_format": answer_format,
    },
    {
        "question": "What quote best demonstrates the customer's level of interest?",
        "context": customer_context,
        "answer_format": answer_format,
    },
    {
        "question": "Provide a quote from the agent that demonstrates their level of enthusiasm.",
        "context": agent_context,
        "answer_format": answer_format,
    },
]

# ------------------------------------------
# Step 4: Build prompt for the LLM
# ------------------------------------------
question_strs = []
for q in questions:
    q_str = f"Question: {q['question']}"
    if q.get("context"):
        q_str += f"\nContext: {q['context']}"
    if q.get("answer_format"):
        q_str += f"\nAnswer Format: {q['answer_format']}"
    question_strs.append(q_str)

questions_prompt = "\n\n".join(question_strs)

prompt = f"""
You are an expert at analyzing call transcripts.
Given the series of questions below, answer them accurately and concisely.
When context or answer format is provided, use it to guide your answers.

Transcript:
{{{{ transcript }}}}

Questions:
{questions_prompt}
"""

# ------------------------------------------
# Step 5: Query the LLM Gateway
# ------------------------------------------
headers = {"authorization": API_KEY}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": prompt}],
        "transcript_id": transcript_id,
        "max_tokens": 2000,
    },
)

response_json = response.json()
llm_output = response_json["choices"][0]["message"]["content"]

# ------------------------------------------
# Step 6: Parse and display the results
# ------------------------------------------
print("\n--- LLM Responses ---\n")
print(llm_output)
```

```javascript expandable
import fs from "fs-extra";

const API_KEY = "YOUR_API_KEY";
const audioFilePath = "./meeting.mp3";
const headers = { authorization: API_KEY };

// ------------------------------------------
// Step 1: Upload the audio file
// ------------------------------------------
const audioData = await fs.readFile(audioFilePath);
let res = await fetch("https://api.assemblyai.com/v2/upload", {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const audioUrl = uploadResponse.upload_url;
console.log(`Uploaded audio file. URL: ${audioUrl}`);

// ------------------------------------------
// Step 2: Request transcription
// ------------------------------------------
res = await fetch("https://api.assemblyai.com/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({ audio_url: audioUrl }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptRequest = await res.json();

const transcriptId = transcriptRequest.id;

// Poll until completed
while (true) {
  res = await fetch(`https://api.assemblyai.com/v2/transcript/${transcriptId}`, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const pollingResponse = await res.json();
  const status = pollingResponse.status;

  if (status === "completed") {
    break;
  } else if (status === "error") {
    throw new Error(`Transcription failed: ${pollingResponse.error}`);
  } else {
    console.log(`Transcription status: ${status}`);
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

console.log("\nTranscription complete.\n");

// ------------------------------------------
// Step 3: Define questions
// ------------------------------------------
const agentContext = "The agent is trying to get the customer to go through with the update to their car.";
const customerContext = "The customer is calling to check how much it would cost to update the map in his car.";

const answerFormat = "<answer in one sentence> <reason in one sentence>";

const questions = [
  {
    question: "What was the overall sentiment of the call?",
    context: customerContext,
    answer_format: answerFormat,
  },
  {
    question: "What was the sentiment of the agent in this call?",
    context: agentContext,
    answer_format: answerFormat,
  },
  {
    question: "What was the sentiment of the customer in this call?",
    context: customerContext,
    answer_format: answerFormat,
  },
  {
    question: "What quote best demonstrates the customer's level of interest?",
    context: customerContext,
    answer_format: answerFormat,
  },
  {
    question: "Provide a quote from the agent that demonstrates their level of enthusiasm.",
    context: agentContext,
    answer_format: answerFormat,
  },
];

// ------------------------------------------
// Step 4: Build prompt for the LLM
// ------------------------------------------
const questionStrs = [];
for (const q of questions) {
  let qStr = `Question: ${q.question}`;
  if (q.context) {
    qStr += `\nContext: ${q.context}`;
  }
  if (q.answer_format) {
    qStr += `\nAnswer Format: ${q.answer_format}`;
  }
  questionStrs.push(qStr);
}

const questionsPrompt = questionStrs.join("\n\n");

const prompt = `
You are an expert at analyzing call transcripts.
Given the series of questions below, answer them accurately and concisely.
When context or answer format is provided, use it to guide your answers.

Transcript:
{{ transcript }}

Questions:
${questionsPrompt}
`;

// ------------------------------------------
// Step 5: Query the LLM Gateway
// ------------------------------------------
res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [{ role: "user", content: prompt }],
    transcript_id: transcriptId,
    max_tokens: 2000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const llmOutput = response.choices[0].message.content;

// ------------------------------------------
// Step 6: Parse and display the results
// ------------------------------------------
console.log("\n--- LLM Responses ---\n");
console.log(llmOutput);
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

See our [pricing page](https://www.assemblyai.com/pricing) for LLM Gateway pricing rates.

## Step-by-Step Instructions

In this guide, we will ask five questions to learn about the sentiment of the customer and agent. You can adjust the questions to suit your project's needs.

### Install dependencies

Install the required packages.

```bash
pip install requests
```

```bash
npm install fs-extra
```

Import the required libraries and set your AssemblyAI API key.

```python
import requests
import time

API_KEY = "YOUR_API_KEY"
```

```javascript
import fs from "fs-extra";

const API_KEY = "YOUR_API_KEY";
```

Next, you'll upload your audio file to AssemblyAI's servers.
Once the upload is complete, the API will return a temporary URL that can be used to start the transcription.

After submitting the transcription request, your script will poll the API until the transcription is finished.

```python expandable
audio_file_path = "./meeting.mp3"
# ------------------------------------------
# Step 1: Upload the audio file
# ------------------------------------------
def upload_file(filename):
    with open(filename, "rb") as f:
        upload_url = "https://api.assemblyai.com/v2/upload"
        headers = {"authorization": API_KEY}
        response = requests.post(upload_url, headers=headers, data=f)
        response.raise_for_status()
        return response.json()["upload_url"]
audio_url = upload_file(audio_file_path)
print(f"Uploaded audio file. URL: {audio_url}")
# ------------------------------------------
# Step 2: Request transcription
# ------------------------------------------
transcript_request = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers={"authorization": API_KEY, "content-type": "application/json"},
    json={"audio_url": audio_url},
)
transcript_id = transcript_request.json()["id"]
# Poll until completed
while True:
    polling_response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": API_KEY},
    )
    status = polling_response.json()["status"]
    if status == "completed":
        break
    elif status == "error":
        raise RuntimeError(f"Transcription failed: {polling_response.json()['error']}")
    else:
        print(f"Transcription status: {status}")
        time.sleep(3)
print("\nTranscription complete.\n")
```

```javascript expandable
const audioFilePath = "./meeting.mp3";
const headers = { authorization: API_KEY };
// ------------------------------------------
// Step 1: Upload the audio file
// ------------------------------------------
const audioData = await fs.readFile(audioFilePath);
let res = await fetch("https://api.assemblyai.com/v2/upload", {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const audioUrl = uploadResponse.upload_url;
console.log(`Uploaded audio file. URL: ${audioUrl}`);
// ------------------------------------------
// Step 2: Request transcription
// ------------------------------------------
res = await fetch("https://api.assemblyai.com/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({ audio_url: audioUrl }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptRequest = await res.json();
const transcriptId = transcriptRequest.id;
// Poll until completed
while (true) {
  res = await fetch(`https://api.assemblyai.com/v2/transcript/${transcriptId}`, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const pollingResponse = await res.json();
  const status = pollingResponse.status;
  if (status === "completed") {
    break;
  } else if (status === "error") {
    throw new Error(`Transcription failed: ${pollingResponse.error}`);
  } else {
    console.log(`Transcription status: ${status}`);
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
console.log("\nTranscription complete.\n");
```

Once you have the transcript, you'll define short context strings for both the agent and the customer.
These will help the model better understand the roles and perspectives in the conversation.

```python
agent_context = "The agent is trying to get the customer to go through with the update to their car."
customer_context = "The customer is calling to check how much it would cost to update the map in his car."
```

```javascript
const agentContext = "The agent is trying to get the customer to go through with the update to their car.";
const customerContext = "The customer is calling to check how much it would cost to update the map in his car.";
```

You can now specify the exact questions you want the LLM Gateway to answer.
Each question can include optional context and an answer format that tells the model how to structure its response.

```python expandable
answer_format = "<answer in one sentence> <reason in one sentence>"
questions = [
    {
        "question": "What was the overall sentiment of the call?",
        "context": customer_context,
        "answer_format": answer_format,
    },
    {
        "question": "What was the sentiment of the agent in this call?",
        "context": agent_context,
        "answer_format": answer_format,
    },
    {
        "question": "What was the sentiment of the customer in this call?",
        "context": customer_context,
        "answer_format": answer_format,
    },
    {
        "question": "What quote best demonstrates the customer's level of interest?",
        "context": customer_context,
        "answer_format": answer_format,
    },
    {
        "question": "Provide a quote from the agent that demonstrates their level of enthusiasm.",
        "context": agent_context,
        "answer_format": answer_format,
    },
]
```

```javascript expandable
const answerFormat = "<answer in one sentence> <reason in one sentence>";
const questions = [
  {
    question: "What was the overall sentiment of the call?",
    context: customerContext,
    answer_format: answerFormat,
  },
  {
    question: "What was the sentiment of the agent in this call?",
    context: agentContext,
    answer_format: answerFormat,
  },
  {
    question: "What was the sentiment of the customer in this call?",
    context: customerContext,
    answer_format: answerFormat,
  },
  {
    question: "What quote best demonstrates the customer's level of interest?",
    context: customerContext,
    answer_format: answerFormat,
  },
  {
    question: "Provide a quote from the agent that demonstrates their level of enthusiasm.",
    context: agentContext,
    answer_format: answerFormat,
  },
];
```

Now that the questions are defined, combine them into a single formatted prompt.
This prompt includes both the call transcript and the questions you want the model to address.
The model will use these details to generate accurate and concise responses.

```python expandable
# ------------------------------------------
# Step 4: Build prompt for the LLM
# ------------------------------------------
question_strs = []
for q in questions:
    q_str = f"Question: {q['question']}"
    if q.get("context"):
        q_str += f"\nContext: {q['context']}"
    if q.get("answer_format"):
        q_str += f"\nAnswer Format: {q['answer_format']}"
    question_strs.append(q_str)
questions_prompt = "\n\n".join(question_strs)
prompt = f"""
You are an expert at analyzing call transcripts.
Given the series of questions below, answer them accurately and concisely.
When context or answer format is provided, use it to guide your answers.
Transcript:
{{{{ transcript }}}}
Questions:
{questions_prompt}
"""
```

```javascript expandable
// ------------------------------------------
// Step 4: Build prompt for the LLM
// ------------------------------------------
const questionStrs = [];
for (const q of questions) {
  let qStr = `Question: ${q.question}`;
  if (q.context) {
    qStr += `\nContext: ${q.context}`;
  }
  if (q.answer_format) {
    qStr += `\nAnswer Format: ${q.answer_format}`;
  }
  questionStrs.push(qStr);
}
const questionsPrompt = questionStrs.join("\n\n");
const prompt = `
You are an expert at analyzing call transcripts.
Given the series of questions below, answer them accurately and concisely.
When context or answer format is provided, use it to guide your answers.
Transcript:
{{ transcript }}
Questions:
${questionsPrompt}
`;
```

With the prompt prepared, query LLM Gateway then extract and print the answers returned by the LLM Gateway.
This step displays the model's assessment of each question, including the identified sentiments and their reasoning.

```python expandable
# ------------------------------------------
# Step 5: Query the LLM Gateway
# ------------------------------------------
headers = {"authorization": API_KEY}
response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": prompt}],
        "transcript_id": transcript_id,
        "max_tokens": 2000,
    },
)
response_json = response.json()
llm_output = response_json["choices"][0]["message"]["content"]
# ------------------------------------------
# Step 6: Parse and display the results
# ------------------------------------------
print("\n--- LLM Responses ---\n")
print(llm_output)
```

```javascript expandable
// ------------------------------------------
// Step 5: Query the LLM Gateway
// ------------------------------------------
let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    messages: [{ role: "user", content: prompt }],
    transcript_id: transcriptId,
    max_tokens: 2000,
  }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();
const llmOutput = response.choices[0].message.content;
// ------------------------------------------
// Step 6: Parse and display the results
// ------------------------------------------
console.log("\n--- LLM Responses ---\n");
console.log(llmOutput);
```

---

<a id="custom-topic-tags-using-llm-gateway-docs-guides-custom-topic-tags-mdx"></a>

# Custom Topic Tags Using LLM Gateway

URL: https://www.assemblyai.com/docs/guides/custom-topic-tags
Source: docs/guides/custom-topic-tags.mdx
Navigation: LLM Gateway > Voice AI Guides > Substitute Speech Understanding with LLM Gateway
Description: Custom Topic Tags Using LLM Gateway documentation.

In this guide we will show you how to label content with custom topic tags using AssemblyAI's LLM Gateway.

## Quickstart

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

# Step 1: Transcribe the audio
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"
data = {"audio_url": audio_url}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

# Step 2: Generate topic tags with LLM Gateway
tag_list = {
    'Sports': 'News and updates on various athletic events, teams, and sports personalities.',
    'Politics': 'Coverage and discussion of government activities, policies, and political events.',
    'Entertainment': 'Information on movies, music, television, celebrities, and arts.',
    'Technology': 'News and reviews on gadgets, software, tech advancements, and trends.',
    'Health': 'Articles focusing on medical news, wellness, and health-related topics.',
    'Business': 'Updates on markets, industries, companies, and economic trends.',
    'Science': 'News and insights into scientific discoveries, research, and innovations.',
    'Education': 'Coverage of topics related to schools, educational policies, and learning.',
    'Travel': 'Information on destinations, travel tips, and tourism news.',
    'Lifestyle': 'Articles on fashion, hobbies, personal interests, and daily life.',
    'Environment': 'News and discussion about environmental issues and sustainability.',
    'Finance': 'Information on personal finance, investments, banking, and economic news.',
    'World News': 'International news covering global events and issues.',
    'Crime': 'Reports and updates on criminal activities, law enforcement, and legal cases.',
    'Culture': 'Coverage of cultural events, traditions, and societal norms.'
}

prompt = f"""
You are a helpful assistant designed to label video content with topic tags.

I will give you a list of topics and definitions. Select the most relevant topic from the list. Return your selection and nothing else.

<topics_list>
{tag_list}
</topics_list>
"""

llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
    ],
    "transcript_id": transcript_id,
    "max_tokens": 500
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)

result = response.json()["choices"][0]["message"]["content"]
print(result.strip())
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

// Step 1: Transcribe the audio
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4";
const data = { audio_url: audioUrl };

let res = await fetch(baseUrl + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = baseUrl + "/v2/transcript/" + transcriptId;

let transcriptionResult;
while (true) {
    res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    transcriptionResult = await res.json();
    if (transcriptionResult.status === "completed") {
        break;
    } else if (transcriptionResult.status === "error") {
        throw new Error(`Transcription failed: ${transcriptionResult.error}`);
    } else {
        await new Promise((resolve) => setTimeout(resolve, 3000));
    }
}

// Step 2: Generate topic tags with LLM Gateway
const tagList = {
    Sports: "News and updates on various athletic events, teams, and sports personalities.",
    Politics: "Coverage and discussion of government activities, policies, and political events.",
    Entertainment: "Information on movies, music, television, celebrities, and arts.",
    Technology: "News and reviews on gadgets, software, tech advancements, and trends.",
    Health: "Articles focusing on medical news, wellness, and health-related topics.",
    Business: "Updates on markets, industries, companies, and economic trends.",
    Science: "News and insights into scientific discoveries, research, and innovations.",
    Education: "Coverage of topics related to schools, educational policies, and learning.",
    Travel: "Information on destinations, travel tips, and tourism news.",
    Lifestyle: "Articles on fashion, hobbies, personal interests, and daily life.",
    Environment: "News and discussion about environmental issues and sustainability.",
    Finance: "Information on personal finance, investments, banking, and economic news.",
    "World News": "International news covering global events and issues.",
    Crime: "Reports and updates on criminal activities, law enforcement, and legal cases.",
    Culture: "Coverage of cultural events, traditions, and societal norms.",
};

const prompt = `
You are a helpful assistant designed to label video content with topic tags.

I will give you a list of topics and definitions. Select the most relevant topic from the list. Return your selection and nothing else.

<topics_list>
${JSON.stringify(tagList, null, 2)}
</topics_list>
`;

const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
        { role: "user", content: `${prompt}\n\n{{ transcript }}` },
    ],
    transcript_id: transcriptId,
    max_tokens: 500,
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llmGatewayData),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const result = response.choices[0].message.content;
console.log(result.trim());
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-Step Instructions

### Install dependencies

Install the required packages:

```bash
pip install requests
```

Set up your API client and transcribe the audio file:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

# Transcribe the audio
audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"
data = {"audio_url": audio_url}  # You can also use a URL to an audio or video file on the web

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

// Transcribe the audio
const audioUrl = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"; // You can also use a URL to an audio or video file on the web
const data = { audio_url: audioUrl };

let res = await fetch(baseUrl + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = baseUrl + "/v2/transcript/" + transcriptId;

let transcriptionResult;
while (true) {
    res = await fetch(pollingEndpoint, { headers });
    if (!res.ok) throw new Error(`Error: ${res.status}`);
    transcriptionResult = await res.json();
    if (transcriptionResult.status === "completed") {
        break;
    } else if (transcriptionResult.status === "error") {
        throw new Error(`Transcription failed: ${transcriptionResult.error}`);
    } else {
        await new Promise((resolve) => setTimeout(resolve, 3000));
    }
}
```

Create a `tag_list` of custom topics, which consists of a key that is the topic and a value that is a short description of what qualifies a file to be labeled with that topic.

Here is an example of a `tag_list` that can be used for videos or podcasts:

```python
tag_list = {
    'Sports': 'News and updates on various athletic events, teams, and sports personalities.',
    'Politics': 'Coverage and discussion of government activities, policies, and political events.',
    'Entertainment': 'Information on movies, music, television, celebrities, and arts.',
    'Technology': 'News and reviews on gadgets, software, tech advancements, and trends.',
    'Health': 'Articles focusing on medical news, wellness, and health-related topics.',
    'Business': 'Updates on markets, industries, companies, and economic trends.',
    'Science': 'News and insights into scientific discoveries, research, and innovations.',
    'Education': 'Coverage of topics related to schools, educational policies, and learning.',
    'Travel': 'Information on destinations, travel tips, and tourism news.',
    'Lifestyle': 'Articles on fashion, hobbies, personal interests, and daily life.',
    'Environment': 'News and discussion about environmental issues and sustainability.',
    'Finance': 'Information on personal finance, investments, banking, and economic news.',
    'World News': 'International news covering global events and issues.',
    'Crime': 'Reports and updates on criminal activities, law enforcement, and legal cases.',
    'Culture': 'Coverage of cultural events, traditions, and societal norms.'
}
```

Create a `tagList` of custom topics, which consists of a key that is the topic and a value that is a short description of what qualifies a file to be labeled with that topic.

Here is an example of a `tagList` that can be used for videos or podcasts:

```javascript
const tagList = {
    Sports: "News and updates on various athletic events, teams, and sports personalities.",
    Politics: "Coverage and discussion of government activities, policies, and political events.",
    Entertainment: "Information on movies, music, television, celebrities, and arts.",
    Technology: "News and reviews on gadgets, software, tech advancements, and trends.",
    Health: "Articles focusing on medical news, wellness, and health-related topics.",
    Business: "Updates on markets, industries, companies, and economic trends.",
    Science: "News and insights into scientific discoveries, research, and innovations.",
    Education: "Coverage of topics related to schools, educational policies, and learning.",
    Travel: "Information on destinations, travel tips, and tourism news.",
    Lifestyle: "Articles on fashion, hobbies, personal interests, and daily life.",
    Environment: "News and discussion about environmental issues and sustainability.",
    Finance: "Information on personal finance, investments, banking, and economic news.",
    "World News": "International news covering global events and issues.",
    Crime: "Reports and updates on criminal activities, law enforcement, and legal cases.",
    Culture: "Coverage of cultural events, traditions, and societal norms.",
};
```

Here is another example of a `tag_list` that can be used for support calls:

```python
tag_list = {
    'Account Issues': 'Problems related to user accounts, such as login difficulties or account access.',
    'Technical Support': 'Inquiries regarding software or hardware functionality and troubleshooting.',
    'Billing and Payments': 'Questions or problems about invoices, payments, or subscription plans.',
    'Product Inquiry': 'Requests for information about product features, capabilities, or availability.',
    'Service Disruption': 'Reports of outages or interruptions in service performance or availability.'
}
```

Here is another example of a `tagList` that can be used for support calls:

```javascript
const tagList = {
    "Account Issues": "Problems related to user accounts, such as login difficulties or account access.",
    "Technical Support": "Inquiries regarding software or hardware functionality and troubleshooting.",
    "Billing and Payments": "Questions or problems about invoices, payments, or subscription plans.",
    "Product Inquiry": "Requests for information about product features, capabilities, or availability.",
    "Service Disruption": "Reports of outages or interruptions in service performance or availability.",
};
```

Use LLM Gateway to analyze the transcript and select the most relevant topic tag. This is an example prompt, which you can modify to suit your specific requirements.

```python expandable
prompt = f"""
You are a helpful assistant designed to label video content with topic tags.

I will give you a list of topics and definitions. Select the most relevant topic from the list. Return your selection and nothing else.

<topics_list>
{tag_list}
</topics_list>
"""

llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
    ],
    "transcript_id": transcript_id,
    "max_tokens": 500
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)

result = response.json()["choices"][0]["message"]["content"]
print(result.strip())
```

```javascript expandable
const prompt = `
You are a helpful assistant designed to label video content with topic tags.

I will give you a list of topics and definitions. Select the most relevant topic from the list. Return your selection and nothing else.

<topics_list>
${JSON.stringify(tagList, null, 2)}
</topics_list>
`;

const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
        { role: "user", content: `${prompt}\n\n{{ transcript }}` },
    ],
    transcript_id: transcriptId,
    max_tokens: 500,
};

let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llmGatewayData),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const result = response.choices[0].message.content;
console.log(result.trim());
```

---

<a id="redact-pii-from-text-using-llm-gateway-docs-guides-llm-gateway-pii-redaction-mdx"></a>

# Redact PII from Text Using LLM Gateway

URL: https://www.assemblyai.com/docs/guides/llm-gateway-pii-redaction
Source: docs/guides/llm-gateway-pii-redaction.mdx
Navigation: LLM Gateway > Voice AI Guides > Substitute Speech Understanding with LLM Gateway
Description: Redact PII from Text Using LLM Gateway documentation.

This guide will show you how to use AssemblyAI's LLM Gateway to redact personally identifiable information (PII) from text.

## Quickstart

```python expandable
import requests
import time
import json
import re

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

def generate_ner(transcript_text):
    prompt = '''
    You will be given a transcript of a conversation or text. Your task is to generate named entities from the given transcript text.

    Please identify and extract the following named entities from the transcript:

    1. Person names
    2. Organization names
    3. Email addresses
    4. Phone numbers
    5. Full addresses

    When extracting these entities, make sure to return the exact spelling and formatting as they appear in the transcript. Do not modify or standardize the entities in any way.

    Present your results in a JSON format with a single field named "named_entities". This field should contain an array of strings, where each string is a named entity you've identified. For example:
    {
      "named_entities": ["John Doe", "Acme Corp", "john.doe@example.com", "123-456-7890", "123 Main St, Anytown, USA 12345"]
    }

    Important: Do not include any other information, explanations, or text in your response. Your output should consist solely of the JSON object containing the named entities.

    If you do not find any named entities of a particular type, simply return an empty array for the "named_entities" field.
    '''

    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": f"{prompt}\n\nTranscript: {transcript_text}"}
        ],
        "max_tokens": 1000,
        "temperature": 0.0
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()["choices"][0]["message"]["content"]

    try:
        res_json = json.loads(result)
    except:
        res_json = {'named_entities': []}

    named_entities = res_json.get('named_entities', [])
    return named_entities

# Step 1: Transcribe audio
with open("./my-audio.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload", headers=headers, data=f)

upload_url = response.json()["upload_url"]
data = {"audio_url": upload_url}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

# Step 2: Split transcript into sentences and redact PII
transcript_text = transcription_result['text']
sentences = re.split(r'[.!?]+', transcript_text)
redacted_transcript = ''

for sentence in sentences:
    sentence = sentence.strip()
    if not sentence:
        continue

    generated_entities = generate_ner(sentence)
    redacted_sentence = sentence

    for entity in generated_entities:
        redacted_sentence = redacted_sentence.replace(entity, '#' * len(entity))

    redacted_transcript += redacted_sentence + '. '
    print(redacted_sentence)

print('\nFull redacted transcript:')
print(redacted_transcript)
```

```javascript expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

async function generateNer(transcriptText) {
  const prompt = `
    You will be given a transcript of a conversation or text. Your task is to generate named entities from the given transcript text.

    Please identify and extract the following named entities from the transcript:

    1. Person names
    2. Organization names
    3. Email addresses
    4. Phone numbers
    5. Full addresses

    When extracting these entities, make sure to return the exact spelling and formatting as they appear in the transcript. Do not modify or standardize the entities in any way.

    Present your results in a JSON format with a single field named "named_entities". This field should contain an array of strings, where each string is a named entity you've identified. For example:
    {
      "named_entities": ["John Doe", "Acme Corp", "john.doe@example.com", "123-456-7890", "123 Main St, Anytown, USA 12345"]
    }

    Important: Do not include any other information, explanations, or text in your response. Your output should consist solely of the JSON object containing the named entities.

    If you do not find any named entities of a particular type, simply return an empty array for the "named_entities" field.
    `;

  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      { role: "user", content: `${prompt}\n\nTranscript: ${transcriptText}` },
    ],
    max_tokens: 1000,
    temperature: 0.0,
  };

  let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  const result = response.choices[0].message.content;

  let resJson;
  try {
    resJson = JSON.parse(result);
  } catch (e) {
    resJson = { named_entities: [] };
  }

  const namedEntities = resJson.named_entities || [];
  return namedEntities;
}

// Step 1: Transcribe audio
const fileData = await fs.readFile("./my-audio.mp3");
let res = await fetch(baseUrl + "/v2/upload", {
  method: "POST",
  headers,
  body: fileData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();

const uploadUrl = uploadResponse.upload_url;
const data = { audio_url: uploadUrl };

res = await fetch(baseUrl + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = baseUrl + "/v2/transcript/" + transcriptId;

let transcriptionResult;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcriptionResult = await res.json();
  if (transcriptionResult.status === "completed") {
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Step 2: Split transcript into sentences and redact PII
const transcriptText = transcriptionResult.text;
const sentences = transcriptText.split(/[.!?]+/);
let redactedTranscript = "";

for (let sentence of sentences) {
  sentence = sentence.trim();
  if (!sentence) continue;

  const generatedEntities = await generateNer(sentence);
  let redactedSentence = sentence;

  for (const entity of generatedEntities) {
    redactedSentence = redactedSentence.replaceAll(entity, "#".repeat(entity.length));
  }

  redactedTranscript += redactedSentence + ". ";
  console.log(redactedSentence);
}

console.log("\nFull redacted transcript:");
console.log(redactedTranscript);
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for an account and get your API key from your dashboard.

## Step-by-Step Instructions

### Install dependencies

Install the required packages:

```bash
pip install requests
```

```bash
npm install fs-extra
```

Import the required packages and set up your API client:

```python
import requests
import time
import json
import re

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}
```

```javascript
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };
```

Define a function `generate_ner` that uses LLM Gateway to identify named entities (person names, organizations, emails, phone numbers, addresses) in a given text.

```python expandable
def generate_ner(transcript_text):
    prompt = '''
    You will be given a transcript of a conversation or text. Your task is to generate named entities from the given transcript text.

    Please identify and extract the following named entities from the transcript:

    1. Person names
    2. Organization names
    3. Email addresses
    4. Phone numbers
    5. Full addresses

    When extracting these entities, make sure to return the exact spelling and formatting as they appear in the transcript. Do not modify or standardize the entities in any way.

    Present your results in a JSON format with a single field named "named_entities". This field should contain an array of strings, where each string is a named entity you've identified. For example:
    {
      "named_entities": ["John Doe", "Acme Corp", "john.doe@example.com", "123-456-7890", "123 Main St, Anytown, USA 12345"]
    }

    Important: Do not include any other information, explanations, or text in your response. Your output should consist solely of the JSON object containing the named entities.

    If you do not find any named entities of a particular type, simply return an empty array for the "named_entities" field.
    '''

    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": f"{prompt}\n\nTranscript: {transcript_text}"}
        ],
        "max_tokens": 1000,
        "temperature": 0.0
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()["choices"][0]["message"]["content"]

    try:
        res_json = json.loads(result)
    except:
        res_json = {'named_entities': []}

    named_entities = res_json.get('named_entities', [])
    return named_entities
```

Define a function `generateNer` that uses LLM Gateway to identify named entities (person names, organizations, emails, phone numbers, addresses) in a given text.

```javascript expandable
async function generateNer(transcriptText) {
  const prompt = `
    You will be given a transcript of a conversation or text. Your task is to generate named entities from the given transcript text.

    Please identify and extract the following named entities from the transcript:

    1. Person names
    2. Organization names
    3. Email addresses
    4. Phone numbers
    5. Full addresses

    When extracting these entities, make sure to return the exact spelling and formatting as they appear in the transcript. Do not modify or standardize the entities in any way.

    Present your results in a JSON format with a single field named "named_entities". This field should contain an array of strings, where each string is a named entity you've identified. For example:
    {
      "named_entities": ["John Doe", "Acme Corp", "john.doe@example.com", "123-456-7890", "123 Main St, Anytown, USA 12345"]
    }

    Important: Do not include any other information, explanations, or text in your response. Your output should consist solely of the JSON object containing the named entities.

    If you do not find any named entities of a particular type, simply return an empty array for the "named_entities" field.
    `;

  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      { role: "user", content: `${prompt}\n\nTranscript: ${transcriptText}` },
    ],
    max_tokens: 1000,
    temperature: 0.0,
  };

  let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  const result = response.choices[0].message.content;

  let resJson;
  try {
    resJson = JSON.parse(result);
  } catch (e) {
    resJson = { named_entities: [] };
  }

  const namedEntities = resJson.named_entities || [];
  return namedEntities;
}
```

Transcribe an audio file using the AssemblyAI API:

```python
with open("./my-audio.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload", headers=headers, data=f)

upload_url = response.json()["upload_url"]
data = {"audio_url": upload_url}  # You can also use a URL to an audio or video file on the web

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

```javascript expandable
const fileData = await fs.readFile("./my-audio.mp3");
let res = await fetch(baseUrl + "/v2/upload", {
  method: "POST",
  headers,
  body: fileData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();

const uploadUrl = uploadResponse.upload_url;
const data = { audio_url: uploadUrl }; // You can also use a URL to an audio or video file on the web

res = await fetch(baseUrl + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = baseUrl + "/v2/transcript/" + transcriptId;

let transcriptionResult;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcriptionResult = await res.json();
  if (transcriptionResult.status === "completed") {
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

Split the transcript into sentences, identify named entities using `generate_ner`, and replace them with # characters:

```python
transcript_text = transcription_result['text']
sentences = re.split(r'[.!?]+', transcript_text)
redacted_transcript = ''

for sentence in sentences:
    sentence = sentence.strip()
    if not sentence:
        continue

    generated_entities = generate_ner(sentence)
    redacted_sentence = sentence

    for entity in generated_entities:
        redacted_sentence = redacted_sentence.replace(entity, '#' * len(entity))

    redacted_transcript += redacted_sentence + '. '
    print(redacted_sentence)
```

Split the transcript into sentences, identify named entities using `generateNer`, and replace them with # characters:

```javascript
const transcriptText = transcriptionResult.text;
const sentences = transcriptText.split(/[.!?]+/);
let redactedTranscript = "";

for (let sentence of sentences) {
  sentence = sentence.trim();
  if (!sentence) continue;

  const generatedEntities = await generateNer(sentence);
  let redactedSentence = sentence;

  for (const entity of generatedEntities) {
    redactedSentence = redactedSentence.replaceAll(entity, "#".repeat(entity.length));
  }

  redactedTranscript += redactedSentence + ". ";
  console.log(redactedSentence);
}
```

Print the fully redacted transcript:

```python
print('\nFull redacted transcript:')
print(redacted_transcript)
```

```javascript
console.log("\nFull redacted transcript:");
console.log(redactedTranscript);
```

---

<a id="apply-llm-gateway-to-streaming-docs-guides-real-time-llm-gateway-mdx"></a>

# Apply LLM Gateway to Streaming

URL: https://www.assemblyai.com/docs/guides/real_time_llm_gateway
Source: docs/guides/real_time_llm_gateway.mdx
Navigation: LLM Gateway > Voice AI Guides > Streaming with LLM Gateway
Description: Learn how to analyze streaming audio transcripts using LLM Gateway.

## Overview

A Large Language Model (LLM) is a machine learning model that uses natural language processing (NLP) to generate text. [LLM Gateway](/llm-gateway/quickstart) is a unified API that provides access to 25+ models from Claude, GPT, Gemini, and more through a single interface. You can use LLM Gateway to analyze streaming audio transcripts in real time, for example to summarize a live conversation or extract action items as they happen.

By the end of this tutorial, you'll be able to use LLM Gateway to analyze a streaming audio transcript from your microphone.

Here's the full sample code for what you'll build in this tutorial:

```python expandable
import pyaudio
import websocket
import json
import threading
import time
import wave
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "<YOUR_API_KEY>"  # Replace with your actual API key

# LLM Gateway Configuration
PROMPT = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"
LLM_GATEWAY_CONFIG = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": PROMPT}
    ],
    "max_tokens": 4000
}

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,  # Request formatted final transcripts
    "speech_model": "universal-3-5-pro",
    "llm_gateway": json.dumps(LLM_GATEWAY_CONFIG)  # LLM Gateway configuration
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# WAV recording variables
recorded_frames = []  # Store audio frames for WAV file
recording_lock = threading.Lock()  # Thread-safe access to recorded_frames

def save_wav_file():
    """Save recorded audio frames to a WAV file."""
    if not recorded_frames:
        print("No audio data recorded.")
        return

    # Generate filename with timestamp
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"recorded_audio_{timestamp}.wav"

    try:
        with wave.open(filename, 'wb') as wf:
            wf.setnchannels(CHANNELS)
            wf.setsampwidth(2)  # 16-bit = 2 bytes
            wf.setframerate(SAMPLE_RATE)

            # Write all recorded frames
            with recording_lock:
                wf.writeframes(b''.join(recorded_frames))

        print(f"Audio saved to: {filename}")
        print(f"Duration: {len(recorded_frames) * FRAMES_PER_BUFFER / SAMPLE_RATE:.2f} seconds")

    except Exception as e:
        print(f"Error saving WAV file: {e}")

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                # Store audio data for WAV recording
                with recording_lock:
                    recorded_frames.append(audio_data)

                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"Session started: {session_id}")
        elif msg_type == "Turn":
            end_of_turn = data.get('end_of_turn', False)

            if end_of_turn:
                transcript = data.get('transcript', '')
                print(f"\nTranscript:\n{transcript}\n")
        elif msg_type == "LLMGatewayResponse":
            # Extract the LLM response content
            llm_data = data.get('data', {})
            llm_content = llm_data.get("choices", [{}])[0].get("message", {}).get("content", "")
            print(f"LLM Response:\n{llm_content}\n")
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"Session terminated: {audio_duration} seconds of audio processed")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    # Save recorded audio to WAV file
    save_wav_file()

    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)

# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
        print("Audio will be saved to a WAV file when the session ends.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

```python expandable
import logging
from typing import Type

import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    LLMGatewayResponseEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TurnEvent,
    TerminationEvent,
)
from assemblyai.streaming.v3.models import LLMGatewayConfig, LLMGatewayMessage

api_key = "<YOUR_API_KEY>"
prompt = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()

def on_begin(self: Type[RealTimeTranscriber], event: BeginEvent):
    print(f"Session started: {event.id}")

def on_turn(self: Type[RealTimeTranscriber], event: TurnEvent):
    if event.end_of_turn:
        print(f"\nTranscript:\n{event.transcript}\n")

def on_llm_response(self: Type[RealTimeTranscriber], event: LLMGatewayResponseEvent):
    # Extract the actual LLM response content from the data
    llm_content = event.data.get("choices", [{}])[0].get("message", {}).get("content", "")
    print(f"LLM Response:\n{llm_content}\n")

def on_terminated(self: Type[RealTimeTranscriber], event: TerminationEvent):
    print(
        f"Session terminated: {event.audio_duration_seconds} seconds of audio processed"
    )

def on_error(self: Type[RealTimeTranscriber], error: RealTimeError):
    print(f"Error occurred: {error}")

def main():
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(
            api_host="streaming.assemblyai.com",
        ),
        api_key=api_key,
    )

    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.LLMGatewayResponse, on_llm_response)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    client.connect(
        RealTimeParameters(
            sample_rate=16000,
            speech_model="universal-3-5-pro",
            format_turns=True,
            llm_gateway=LLMGatewayConfig(
                model="claude-sonnet-4-6",
                messages=[
                    LLMGatewayMessage(role="user", content=prompt)
                ],
                max_tokens=4000
            )
        )
    )

    try:
        client.stream(microphone_stream())
    finally:
        client.disconnect(terminate=True)

if __name__ == "__main__":
    main()
```

```javascript expandable
const WebSocket = require("ws");
const mic = require("mic");
const querystring = require("querystring");
const fs = require("fs");

// --- Configuration ---
const YOUR_API_KEY = "<YOUR_API_KEY>"; // Replace with your actual API key

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};

const CONNECTION_PARAMS = {
  sample_rate: 16000,
  format_turns: true, // Request formatted final transcripts
  speech_model: "universal-3-5-pro",
  llm_gateway: JSON.stringify(LLM_GATEWAY_CONFIG), // LLM Gateway configuration
};
const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

// Audio Configuration
const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;
const CHANNELS = 1;

// Global variables
let micInstance = null;
let micInputStream = null;
let ws = null;
let stopRequested = false;

// WAV recording variables
let recordedFrames = []; // Store audio frames for WAV file

// --- Helper functions ---
function formatTimestamp(timestamp) {
  return new Date(timestamp * 1000).toISOString();
}

function createWavHeader(sampleRate, channels, dataLength) {
  const buffer = Buffer.alloc(44);

  // RIFF header
  buffer.write("RIFF", 0);
  buffer.writeUInt32LE(36 + dataLength, 4);
  buffer.write("WAVE", 8);

  // fmt chunk
  buffer.write("fmt ", 12);
  buffer.writeUInt32LE(16, 16); // fmt chunk size
  buffer.writeUInt16LE(1, 20); // PCM format
  buffer.writeUInt16LE(channels, 22);
  buffer.writeUInt32LE(sampleRate, 24);
  buffer.writeUInt32LE(sampleRate * channels * 2, 28); // byte rate
  buffer.writeUInt16LE(channels * 2, 32); // block align
  buffer.writeUInt16LE(16, 34); // bits per sample

  // data chunk
  buffer.write("data", 36);
  buffer.writeUInt32LE(dataLength, 40);

  return buffer;
}

function saveWavFile() {
  if (recordedFrames.length === 0) {
    console.log("No audio data recorded.");
    return;
  }

  // Generate filename with timestamp
  const timestamp = new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19);
  const filename = `recorded_audio_${timestamp}.wav`;

  try {
    // Combine all recorded frames
    const audioData = Buffer.concat(recordedFrames);
    const dataLength = audioData.length;

    // Create WAV header
    const wavHeader = createWavHeader(SAMPLE_RATE, CHANNELS, dataLength);

    // Write WAV file
    const wavFile = Buffer.concat([wavHeader, audioData]);
    fs.writeFileSync(filename, wavFile);

    console.log(`Audio saved to: ${filename}`);
    console.log(
      `Duration: ${(dataLength / (SAMPLE_RATE * CHANNELS * 2)).toFixed(2)} seconds`
    );
  } catch (error) {
    console.error(`Error saving WAV file: ${error}`);
  }
}

// --- Main function ---
async function run() {
  console.log(
    "Starting AssemblyAI streaming transcription with LLM Gateway..."
  );
  console.log("Audio will be saved to a WAV file when the session ends.");

  // Initialize WebSocket connection
  ws = new WebSocket(API_ENDPOINT, {
    headers: {
      Authorization: YOUR_API_KEY,
    },
  });

  // Setup WebSocket event handlers
  ws.on("open", () => {
    console.log("WebSocket connection opened.");
    console.log(`Connected to: ${API_ENDPOINT_BASE_URL}`);
    // Start the microphone
    startMicrophone();
  });

  ws.on("message", (message) => {
    try {
      const data = JSON.parse(message);
      const msgType = data.type;

      if (msgType === "Begin") {
        const sessionId = data.id;
        console.log(`Session started: ${sessionId}`);
      } else if (msgType === "Turn") {
        const endOfTurn = data.end_of_turn;

        if (endOfTurn) {
          const transcript = data.transcript || "";
          console.log(`\nTranscript:\n${transcript}\n`);
        }
      } else if (msgType === "LLMGatewayResponse") {
        // Extract the LLM response content
        const llmData = data.data || {};
        const llmContent = llmData.choices?.[0]?.message?.content || "";
        console.log(`LLM Response:\n${llmContent}\n`);
      } else if (msgType === "Termination") {
        const audioDuration = data.audio_duration_seconds;
        console.log(
          `Session terminated: ${audioDuration} seconds of audio processed`
        );
      }
    } catch (error) {
      console.error(`\nError handling message: ${error}`);
      console.error(`Message data: ${message}`);
    }
  });

  ws.on("error", (error) => {
    console.error(`\nWebSocket Error: ${error}`);
    cleanup();
  });

  ws.on("close", (code, reason) => {
    console.log(`\nWebSocket Disconnected: Status=${code}, Msg=${reason}`);
    cleanup();
  });

  // Handle process termination
  setupTerminationHandlers();
}

function startMicrophone() {
  try {
    micInstance = mic({
      rate: SAMPLE_RATE.toString(),
      channels: CHANNELS.toString(),
      debug: false,
      exitOnSilence: 6, // This won't actually exit, just a parameter for mic
    });

    micInputStream = micInstance.getAudioStream();

    micInputStream.on("data", (data) => {
      if (ws && ws.readyState === WebSocket.OPEN && !stopRequested) {
        // Store audio data for WAV recording
        recordedFrames.push(Buffer.from(data));

        // Send audio data to WebSocket
        ws.send(data);
      }
    });

    micInputStream.on("error", (err) => {
      console.error(`Microphone Error: ${err}`);
      cleanup();
    });

    micInstance.start();
    console.log("Microphone stream opened successfully.");
    console.log("Speak into your microphone. Press Ctrl+C to stop.");
  } catch (error) {
    console.error(`Error opening microphone stream: ${error}`);
    cleanup();
  }
}

function cleanup() {
  stopRequested = true;

  // Save recorded audio to WAV file
  saveWavFile();

  // Stop microphone if it's running
  if (micInstance) {
    try {
      micInstance.stop();
    } catch (error) {
      console.error(`Error stopping microphone: ${error}`);
    }
    micInstance = null;
  }

  // Close WebSocket connection if it's open
  if (ws && [WebSocket.OPEN, WebSocket.CONNECTING].includes(ws.readyState)) {
    try {
      // Send termination message if possible
      if (ws.readyState === WebSocket.OPEN) {
        const terminateMessage = { type: "Terminate" };
        console.log(
          `Sending termination message: ${JSON.stringify(terminateMessage)}`
        );
        ws.send(JSON.stringify(terminateMessage));
      }
      ws.close();
    } catch (error) {
      console.error(`Error closing WebSocket: ${error}`);
    }
    ws = null;
  }

  console.log("Cleanup complete.");
}

function setupTerminationHandlers() {
  // Handle Ctrl+C and other termination signals
  process.on("SIGINT", () => {
    console.log("\nCtrl+C received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  process.on("SIGTERM", () => {
    console.log("\nTermination signal received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  // Handle uncaught exceptions
  process.on("uncaughtException", (error) => {
    console.error(`\nUncaught exception: ${error}`);
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(1), 1000);
  });
}

// Start the application
run();
```

```javascript expandable
import { AssemblyAI } from "assemblyai";
import recorder from "node-record-lpcm16";
import { Readable } from "stream";

// --- Configuration ---
const API_KEY = "<YOUR_API_KEY>";

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};

const run = async () => {
  const client = new AssemblyAI({
    apiKey: API_KEY,
  });

  const transcriber = client.streaming.transcriber({
    sampleRate: 16000,
    formatTurns: true,
    speechModel: "universal-3-5-pro",
    llmGateway: LLM_GATEWAY_CONFIG,
  });

  // Session started event
  transcriber.on("open", ({ id }) => {
    console.log(`Session started: ${id}`);
  });

  // Error event
  transcriber.on("error", (error) => {
    console.error(`\nError: ${error}`);
  });

  // Close event
  transcriber.on("close", (code, reason) => {
    console.log("Session closed:", code, reason || "");
  });

  // Turn event - displays transcript when turn ends
  transcriber.on("turn", (turn) => {
    if (turn.end_of_turn) {
      console.log(`\nTranscript:\n${turn.transcript}\n`);
    }
  });

  // LLM Gateway Response event - displays LLM summary
  transcriber.on("llmGatewayResponse", (response) => {
    const llmContent = response.data?.choices?.[0]?.message?.content || "";
    console.log(`\nLLM Response:\n${llmContent}\n`);
  });

  // Termination event
  transcriber.on("termination", (event) => {
    console.log(
      `Session terminated: ${event.audio_duration_seconds} seconds of audio processed`
    );
  });

  try {
    await transcriber.connect();

    console.log("Connecting to streaming transcript service");
    console.log("Starting recording");

    const recording = recorder.record({
      channels: 1,
      sampleRate: 16000,
      audioType: "wav", // Linear PCM
    });

    Readable.toWeb(recording.stream()).pipeTo(transcriber.stream());

    // Stop recording and close connection using Ctrl-C
    process.on("SIGINT", async function () {
      console.log();
      console.log("Stopping recording");
      recording.stop();

      console.log("Closing streaming transcript connection");
      await transcriber.close();

      process.exit();
    });
  } catch (error) {
    console.error(`Error: ${error}`);
  }
};

run();
```

## Before you begin

To complete this tutorial, you need:

- [Python](https://www.python.org/) or [Node](https://nodejs.org/en) installed.
- An <a href="https://www.assemblyai.com/dashboard/signup" target="_blank">AssemblyAI account with a credit card set up</a>.
- A microphone connected to your computer.
- Basic understanding of how to [Transcribe streaming audio](/streaming/getting-started/transcribe-streaming-audio).

## Step 1: Install prerequisites

Install the required packages via pip:

```bash
pip install pyaudio websocket-client
```

Install the AssemblyAI Python SDK and a recording package via pip:

```bash
pip install assemblyai pyaudio
```

Install the required packages via NPM:

```bash
npm install ws mic
```

Install the AssemblyAI JavaScript SDK and a recording package via NPM:

```bash
npm install assemblyai node-record-lpcm16
```

## Step 2: Connect to Universal Streaming

In this step, you'll set up a connection to the Universal Streaming API with the `llm_gateway` parameter. This parameter configures LLM Gateway to process your streaming transcripts.

For more information about streaming transcription, see [Transcribe streaming audio](/streaming/getting-started/transcribe-streaming-audio).

```python expandable
import pyaudio
import websocket
import json
import threading
import time
import wave
from urllib.parse import urlencode
from datetime import datetime

# --- Configuration ---
YOUR_API_KEY = "<YOUR_API_KEY>"  # Replace with your actual API key

# LLM Gateway Configuration
PROMPT = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"
LLM_GATEWAY_CONFIG = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": PROMPT}
    ],
    "max_tokens": 4000
}

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,  # Request formatted final transcripts
    "speech_model": "universal-3-5-pro",
    "llm_gateway": json.dumps(LLM_GATEWAY_CONFIG)  # LLM Gateway configuration
}
API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

# Audio Configuration
FRAMES_PER_BUFFER = 800  # 50ms of audio (0.05s * 16000Hz)
SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
CHANNELS = 1
FORMAT = pyaudio.paInt16

# Global variables for audio stream and websocket
audio = None
stream = None
ws_app = None
audio_thread = None
stop_event = threading.Event()  # To signal the audio thread to stop

# WAV recording variables
recorded_frames = []  # Store audio frames for WAV file
recording_lock = threading.Lock()  # Thread-safe access to recorded_frames
```

```python expandable
import logging
from typing import Type

import pyaudio
from assemblyai.streaming.v3 import (
    BeginEvent,
    LLMGatewayResponseEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TurnEvent,
    TerminationEvent,
)
from assemblyai.streaming.v3.models import LLMGatewayConfig, LLMGatewayMessage

api_key = "<YOUR_API_KEY>"
prompt = "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}"

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# The SDK doesn't capture audio — supply 16-bit PCM chunks yourself.
def microphone_stream(sample_rate=16000, frames_per_buffer=800):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        input=True,
        format=pyaudio.paInt16,
        channels=1,
        rate=sample_rate,
        frames_per_buffer=frames_per_buffer,
    )
    try:
        while True:
            yield stream.read(frames_per_buffer, exception_on_overflow=False)
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()
```

```javascript expandable
const WebSocket = require("ws");
const mic = require("mic");
const querystring = require("querystring");
const fs = require("fs");

// --- Configuration ---
const YOUR_API_KEY = "<YOUR_API_KEY>"; // Replace with your actual API key

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};

const CONNECTION_PARAMS = {
  sample_rate: 16000,
  format_turns: true, // Request formatted final transcripts
  speech_model: "universal-3-5-pro",
  llm_gateway: JSON.stringify(LLM_GATEWAY_CONFIG), // LLM Gateway configuration
};
const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${querystring.stringify(CONNECTION_PARAMS)}`;

// Audio Configuration
const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;
const CHANNELS = 1;

// Global variables
let micInstance = null;
let micInputStream = null;
let ws = null;
let stopRequested = false;

// WAV recording variables
let recordedFrames = []; // Store audio frames for WAV file
```

```javascript
import { AssemblyAI } from "assemblyai";
import recorder from "node-record-lpcm16";
import { Readable } from "stream";

// --- Configuration ---
const API_KEY = "<YOUR_API_KEY>";

// LLM Gateway Configuration
const PROMPT =
  "Provide a brief summary of the transcript.\n\nTranscript: {{turn}}";
const LLM_GATEWAY_CONFIG = {
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: PROMPT }],
  max_tokens: 4000,
};
```

The `llm_gateway` parameterisa JSON-stringified object that follows the same interface as the [LLM Gateway chat completions API](/llm-gateway/chat-completions). It accepts the following fields:

| Key          | Type   | Description                                                                            |
| ------------ | ------ | -------------------------------------------------------------------------------------- |
| `model`      | string | The model to use. See [Available models](/llm-gateway/available-models). |
| `messages`   | array  | An array of message objects. The `content` field contains your prompt.                 |
| `max_tokens` | number | The maximum number of tokens to generate.                                              |

## Step 3: Stream audio and analyze with LLM Gateway

In this step, you'll stream audio from your microphone, collect the transcribed text from completed turns, and then send the accumulated transcript to LLM Gateway for analysis when the session ends.

Set up the event handlers to stream audio and collect transcripts from completed turns.

```python expandable
def save_wav_file():
    """Save recorded audio frames to a WAV file."""
    if not recorded_frames:
        print("No audio data recorded.")
        return

    # Generate filename with timestamp
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"recorded_audio_{timestamp}.wav"

    try:
        with wave.open(filename, 'wb') as wf:
            wf.setnchannels(CHANNELS)
            wf.setsampwidth(2)  # 16-bit = 2 bytes
            wf.setframerate(SAMPLE_RATE)

            # Write all recorded frames
            with recording_lock:
                wf.writeframes(b''.join(recorded_frames))

        print(f"Audio saved to: {filename}")
        print(f"Duration: {len(recorded_frames) * FRAMES_PER_BUFFER / SAMPLE_RATE:.2f} seconds")

    except Exception as e:
        print(f"Error saving WAV file: {e}")

# --- WebSocket Event Handlers ---

def on_open(ws):
    """Called when the WebSocket connection is established."""
    print("WebSocket connection opened.")
    print(f"Connected to: {API_ENDPOINT_BASE_URL}")

    # Start sending audio data in a separate thread
    def stream_audio():
        global stream
        print("Starting audio streaming...")
        while not stop_event.is_set():
            try:
                audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)

                # Store audio data for WAV recording
                with recording_lock:
                    recorded_frames.append(audio_data)

                # Send audio data as binary message
                ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
            except Exception as e:
                print(f"Error streaming audio: {e}")
                # If stream read fails, likely means it's closed, stop the loop
                break
        print("Audio streaming stopped.")

    global audio_thread
    audio_thread = threading.Thread(target=stream_audio)
    audio_thread.daemon = (
        True  # Allow main thread to exit even if this thread is running
    )
    audio_thread.start()

def on_message(ws, message):
    try:
        data = json.loads(message)
        msg_type = data.get('type')

        if msg_type == "Begin":
            session_id = data.get('id')
            expires_at = data.get('expires_at')
            print(f"Session started: {session_id}")
        elif msg_type == "Turn":
            end_of_turn = data.get('end_of_turn', False)

            if end_of_turn:
                transcript = data.get('transcript', '')
                print(f"\nTranscript:\n{transcript}\n")
        elif msg_type == "LLMGatewayResponse":
            # Extract the LLM response content
            llm_data = data.get('data', {})
            llm_content = llm_data.get("choices", [{}])[0].get("message", {}).get("content", "")
            print(f"LLM Response:\n{llm_content}\n")
        elif msg_type == "Termination":
            audio_duration = data.get('audio_duration_seconds', 0)
            session_duration = data.get('session_duration_seconds', 0)
            print(f"Session terminated: {audio_duration} seconds of audio processed")
    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")

def on_error(ws, error):
    """Called when a WebSocket error occurs."""
    print(f"\nWebSocket Error: {error}")
    # Attempt to signal stop on error
    stop_event.set()

def on_close(ws, close_status_code, close_msg):
    """Called when the WebSocket connection is closed."""
    print(f"\nWebSocket Disconnected: Status={close_status_code}, Msg={close_msg}")

    # Save recorded audio to WAV file
    save_wav_file()

    # Ensure audio resources are released
    global stream, audio
    stop_event.set()  # Signal audio thread just in case it's still running

    if stream:
        if stream.is_active():
            stream.stop_stream()
        stream.close()
        stream = None
    if audio:
        audio.terminate()
        audio = None
    # Try to join the audio thread to ensure clean exit
    if audio_thread and audio_thread.is_alive():
        audio_thread.join(timeout=1.0)
```

```python
def on_begin(self: Type[RealTimeTranscriber], event: BeginEvent):
    print(f"Session started: {event.id}")

def on_turn(self: Type[RealTimeTranscriber], event: TurnEvent):
    if event.end_of_turn:
        print(f"\nTranscript:\n{event.transcript}\n")

def on_llm_response(self: Type[RealTimeTranscriber], event: LLMGatewayResponseEvent):
    # Extract the actual LLM response content from the data
    llm_content = event.data.get("choices", [{}])[0].get("message", {}).get("content", "")
    print(f"LLM Response:\n{llm_content}\n")

def on_terminated(self: Type[RealTimeTranscriber], event: TerminationEvent):
    print(
        f"Session terminated: {event.audio_duration_seconds} seconds of audio processed"
    )

def on_error(self: Type[RealTimeTranscriber], error: RealTimeError):
    print(f"Error occurred: {error}")
```

```javascript expandable
// --- Helper functions ---
function formatTimestamp(timestamp) {
  return new Date(timestamp * 1000).toISOString();
}

function createWavHeader(sampleRate, channels, dataLength) {
  const buffer = Buffer.alloc(44);

  // RIFF header
  buffer.write("RIFF", 0);
  buffer.writeUInt32LE(36 + dataLength, 4);
  buffer.write("WAVE", 8);

  // fmt chunk
  buffer.write("fmt ", 12);
  buffer.writeUInt32LE(16, 16); // fmt chunk size
  buffer.writeUInt16LE(1, 20); // PCM format
  buffer.writeUInt16LE(channels, 22);
  buffer.writeUInt32LE(sampleRate, 24);
  buffer.writeUInt32LE(sampleRate * channels * 2, 28); // byte rate
  buffer.writeUInt16LE(channels * 2, 32); // block align
  buffer.writeUInt16LE(16, 34); // bits per sample

  // data chunk
  buffer.write("data", 36);
  buffer.writeUInt32LE(dataLength, 40);

  return buffer;
}

function saveWavFile() {
  if (recordedFrames.length === 0) {
    console.log("No audio data recorded.");
    return;
  }

  // Generate filename with timestamp
  const timestamp = new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19);
  const filename = `recorded_audio_${timestamp}.wav`;

  try {
    // Combine all recorded frames
    const audioData = Buffer.concat(recordedFrames);
    const dataLength = audioData.length;

    // Create WAV header
    const wavHeader = createWavHeader(SAMPLE_RATE, CHANNELS, dataLength);

    // Write WAV file
    const wavFile = Buffer.concat([wavHeader, audioData]);
    fs.writeFileSync(filename, wavFile);

    console.log(`Audio saved to: ${filename}`);
    console.log(
      `Duration: ${(dataLength / (SAMPLE_RATE * CHANNELS * 2)).toFixed(2)} seconds`
    );
  } catch (error) {
    console.error(`Error saving WAV file: ${error}`);
  }
}

function startMicrophone() {
  try {
    micInstance = mic({
      rate: SAMPLE_RATE.toString(),
      channels: CHANNELS.toString(),
      debug: false,
      exitOnSilence: 6, // This won't actually exit, just a parameter for mic
    });

    micInputStream = micInstance.getAudioStream();

    micInputStream.on("data", (data) => {
      if (ws && ws.readyState === WebSocket.OPEN && !stopRequested) {
        // Store audio data for WAV recording
        recordedFrames.push(Buffer.from(data));

        // Send audio data to WebSocket
        ws.send(data);
      }
    });

    micInputStream.on("error", (err) => {
      console.error(`Microphone Error: ${err}`);
      cleanup();
    });

    micInstance.start();
    console.log("Microphone stream opened successfully.");
    console.log("Speak into your microphone. Press Ctrl+C to stop.");
  } catch (error) {
    console.error(`Error opening microphone stream: ${error}`);
    cleanup();
  }
}
```

```javascript expandable
// Session started event
transcriber.on("open", ({ id }) => {
  console.log(`Session started: ${id}`);
});

// Error event
transcriber.on("error", (error) => {
  console.error(`\nError: ${error}`);
});

// Close event
transcriber.on("close", (code, reason) => {
  console.log("Session closed:", code, reason || "");
});

// Turn event - displays transcript when turn ends
transcriber.on("turn", (turn) => {
  if (turn.end_of_turn) {
    console.log(`\nTranscript:\n${turn.transcript}\n`);
  }
});

// LLM Gateway Response event - displays LLM summary
transcriber.on("llmGatewayResponse", (response) => {
  const llmContent = response.data?.choices?.[0]?.message?.content || "";
  console.log(`\nLLM Response:\n${llmContent}\n`);
});

// Termination event
transcriber.on("termination", (event) => {
  console.log(
    `Session terminated: ${event.audio_duration_seconds} seconds of audio processed`
  );
});
```

Define a function to send the accumulated transcript to LLM Gateway for analysis. This function uses the [LLM Gateway chat completions API](/llm-gateway/chat-completions) to process the transcript with your prompt.

When using the raw WebSocket approach with `llm_gateway` in the connection parameters, LLM Gateway responses are received as `LLMGatewayResponse` messages through the WebSocket, handled by the `on_message` callback registered in the previous step. No separate API call is needed.

When using the Python SDK with `LLMGatewayConfig`, analysis responses are received automatically through the `LLMGatewayResponseEvent` event handler registered in the previous step. No separate API call is needed.

When using the raw WebSocket approach with `llm_gateway` in the connection parameters, LLM Gateway responses are received as `LLMGatewayResponse` messages through the WebSocket, handled by the `on_message` callback in the `run()` function. No separate API call is needed.

When using the JavaScript SDK with `llmGateway` in the transcriber options, LLM Gateway responses are received automatically through the `llmGatewayResponse` event handler registered in the previous step. No separate API call is needed.

Run the streaming session and analyze the transcript with LLM Gateway when the session ends.

```python expandable
# --- Main Execution ---
def run():
    global audio, stream, ws_app

    # Initialize PyAudio
    audio = pyaudio.PyAudio()

    # Open microphone stream
    try:
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Microphone stream opened successfully.")
        print("Speak into your microphone. Press Ctrl+C to stop.")
        print("Audio will be saved to a WAV file when the session ends.")
    except Exception as e:
        print(f"Error opening microphone stream: {e}")
        if audio:
            audio.terminate()
        return  # Exit if microphone cannot be opened

    # Create WebSocketApp
    ws_app = websocket.WebSocketApp(
        API_ENDPOINT,
        header={"Authorization": YOUR_API_KEY},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )

    # Run WebSocketApp in a separate thread to allow main thread to catch KeyboardInterrupt
    ws_thread = threading.Thread(target=ws_app.run_forever)
    ws_thread.daemon = True
    ws_thread.start()

    try:
        # Keep main thread alive until interrupted
        while ws_thread.is_alive():
            time.sleep(0.1)
    except KeyboardInterrupt:
        print("\nCtrl+C received. Stopping...")
        stop_event.set()  # Signal audio thread to stop

        # Send termination message to the server
        if ws_app and ws_app.sock and ws_app.sock.connected:
            try:
                terminate_message = {"type": "Terminate"}
                print(f"Sending termination message: {json.dumps(terminate_message)}")
                ws_app.send(json.dumps(terminate_message))
                # Give a moment for messages to process before forceful close
                time.sleep(5)
            except Exception as e:
                print(f"Error sending termination message: {e}")

        # Close the WebSocket connection (will trigger on_close)
        if ws_app:
            ws_app.close()

        # Wait for WebSocket thread to finish
        ws_thread.join(timeout=2.0)

    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")
        stop_event.set()
        if ws_app:
            ws_app.close()
        ws_thread.join(timeout=2.0)

    finally:
        # Final cleanup (already handled in on_close, but good as a fallback)
        if stream and stream.is_active():
            stream.stop_stream()
        if stream:
            stream.close()
        if audio:
            audio.terminate()
        print("Cleanup complete. Exiting.")

if __name__ == "__main__":
    run()
```

```python expandable
def main():
    client = RealTimeTranscriber(
        RealTimeTranscriberOptions(
            api_host="streaming.assemblyai.com",
        ),
        api_key=api_key,
    )

    client.on(RealTimeEvents.Begin, on_begin)
    client.on(RealTimeEvents.Turn, on_turn)
    client.on(RealTimeEvents.LLMGatewayResponse, on_llm_response)
    client.on(RealTimeEvents.Termination, on_terminated)
    client.on(RealTimeEvents.Error, on_error)

    client.connect(
        RealTimeParameters(
            sample_rate=16000,
            speech_model="universal-3-5-pro",
            format_turns=True,
            llm_gateway=LLMGatewayConfig(
                model="claude-sonnet-4-6",
                messages=[
                    LLMGatewayMessage(role="user", content=prompt)
                ],
                max_tokens=4000
            )
        )
    )

    try:
        client.stream(microphone_stream())
    finally:
        client.disconnect(terminate=True)

if __name__ == "__main__":
    main()
```

```javascript expandable
// --- Main function ---
async function run() {
  console.log(
    "Starting AssemblyAI streaming transcription with LLM Gateway..."
  );
  console.log("Audio will be saved to a WAV file when the session ends.");

  // Initialize WebSocket connection
  ws = new WebSocket(API_ENDPOINT, {
    headers: {
      Authorization: YOUR_API_KEY,
    },
  });

  // Setup WebSocket event handlers
  ws.on("open", () => {
    console.log("WebSocket connection opened.");
    console.log(`Connected to: ${API_ENDPOINT_BASE_URL}`);
    // Start the microphone
    startMicrophone();
  });

  ws.on("message", (message) => {
    try {
      const data = JSON.parse(message);
      const msgType = data.type;

      if (msgType === "Begin") {
        const sessionId = data.id;
        console.log(`Session started: ${sessionId}`);
      } else if (msgType === "Turn") {
        const endOfTurn = data.end_of_turn;

        if (endOfTurn) {
          const transcript = data.transcript || "";
          console.log(`\nTranscript:\n${transcript}\n`);
        }
      } else if (msgType === "LLMGatewayResponse") {
        // Extract the LLM response content
        const llmData = data.data || {};
        const llmContent = llmData.choices?.[0]?.message?.content || "";
        console.log(`LLM Response:\n${llmContent}\n`);
      } else if (msgType === "Termination") {
        const audioDuration = data.audio_duration_seconds;
        console.log(
          `Session terminated: ${audioDuration} seconds of audio processed`
        );
      }
    } catch (error) {
      console.error(`\nError handling message: ${error}`);
      console.error(`Message data: ${message}`);
    }
  });

  ws.on("error", (error) => {
    console.error(`\nWebSocket Error: ${error}`);
    cleanup();
  });

  ws.on("close", (code, reason) => {
    console.log(`\nWebSocket Disconnected: Status=${code}, Msg=${reason}`);
    cleanup();
  });

  // Handle process termination
  setupTerminationHandlers();
}

function cleanup() {
  stopRequested = true;

  // Save recorded audio to WAV file
  saveWavFile();

  // Stop microphone if it's running
  if (micInstance) {
    try {
      micInstance.stop();
    } catch (error) {
      console.error(`Error stopping microphone: ${error}`);
    }
    micInstance = null;
  }

  // Close WebSocket connection if it's open
  if (ws && [WebSocket.OPEN, WebSocket.CONNECTING].includes(ws.readyState)) {
    try {
      // Send termination message if possible
      if (ws.readyState === WebSocket.OPEN) {
        const terminateMessage = { type: "Terminate" };
        console.log(
          `Sending termination message: ${JSON.stringify(terminateMessage)}`
        );
        ws.send(JSON.stringify(terminateMessage));
      }
      ws.close();
    } catch (error) {
      console.error(`Error closing WebSocket: ${error}`);
    }
    ws = null;
  }

  console.log("Cleanup complete.");
}

function setupTerminationHandlers() {
  // Handle Ctrl+C and other termination signals
  process.on("SIGINT", () => {
    console.log("\nCtrl+C received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  process.on("SIGTERM", () => {
    console.log("\nTermination signal received. Stopping...");
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(0), 1000);
  });

  // Handle uncaught exceptions
  process.on("uncaughtException", (error) => {
    console.error(`\nUncaught exception: ${error}`);
    cleanup();
    // Give time for cleanup before exiting
    setTimeout(() => process.exit(1), 1000);
  });
}

// Start the application
run();
```

```javascript expandable
const run = async () => {
  const client = new AssemblyAI({
    apiKey: API_KEY,
  });

  const transcriber = client.streaming.transcriber({
    sampleRate: 16000,
    formatTurns: true,
    speechModel: "universal-3-5-pro",
    llmGateway: LLM_GATEWAY_CONFIG,
  });

  // ... (event handlers from Step 1)

  try {
    await transcriber.connect();

    console.log("Connecting to streaming transcript service");
    console.log("Starting recording");

    const recording = recorder.record({
      channels: 1,
      sampleRate: 16000,
      audioType: "wav", // Linear PCM
    });

    Readable.toWeb(recording.stream()).pipeTo(transcriber.stream());

    // Stop recording and close connection using Ctrl-C
    process.on("SIGINT", async function () {
      console.log();
      console.log("Stopping recording");
      recording.stop();

      console.log("Closing streaming transcript connection");
      await transcriber.close();

      process.exit();
    });
  } catch (error) {
    console.error(`Error: ${error}`);
  }
};

run();
```

The output will look something like this:

```
Session started: de5d9927-73a6-4be8-b52d-b4c07be37e6b

Transcript: Hi, my name is Sonny.

Transcript: I am a voice agent.

Stopping...

Session terminated: 12s of audio processed

Analyzing conversation with LLM Gateway...
The speaker introduces themselves as Sonny and identifies as a voice agent.
```

  Want to make your LLM requests more resilient? Use [fallback models](/llm-gateway/fallback) to automatically switch to a backup model if your primary model is unavailable.

## Next steps

In this tutorial, you've learned how to analyze streaming audio transcripts using LLM Gateway. The type of output depends on your prompt, so try exploring different prompts to see how they affect the output. Here are a few more prompts to try:

- "Provide an analysis of the transcript and offer areas to improve with exact quotes."
- "What's the main take-away from the transcript?"
- "Generate a set of action items from this transcript."

To learn more about LLM Gateway and streaming, see the following resources:

- [LLM Gateway Overview](/llm-gateway/quickstart)
- [Basic Chat Completions](/llm-gateway/chat-completions)
- [Model Selection](/streaming)
- [Transcribe streaming audio](/streaming/getting-started/transcribe-streaming-audio)

## Need some help?

If you get stuck, or have any other questions, we'd love to help you out. Contact our support team at support@assemblyai.com or create a [support ticket](https://www.assemblyai.com/contact/support).

---

<a id="translate-real-time-stt-transcripts-with-llm-gateway-docs-guides-real-time-translation-mdx"></a>

# Translate Real-time STT Transcripts with LLM Gateway

URL: https://www.assemblyai.com/docs/guides/real_time_translation
Source: docs/guides/real_time_translation.mdx
Navigation: LLM Gateway > Voice AI Guides > Streaming with LLM Gateway
Description: Translate Real-time STT Transcripts with LLM Gateway documentation.

In this guide, you'll learn how to implement real-time translation of final transcripts using AssemblyAI's Streaming API and LLM Gateway.

## Quickstart

    

    ```python expandable
    import pyaudio
    import websocket
    import json
    import threading
    import time
    import requests
    from urllib.parse import urlencode

    YOUR_API_KEY = "YOUR_API_KEY"  # Replace with your actual API key

    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "speech_model": "universal-3-5-pro",
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    FRAMES_PER_BUFFER = 800
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()

    def translate_text(text):
        """Called when translating final transcripts."""
        headers = {
            "authorization": YOUR_API_KEY
        }

        llm_gateway_data = {
            "model": "gemini-2.5-flash-lite",
            "messages": [
                {"role": "user", "content": f"Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: {text}"}
            ],
            "max_tokens": 1000
        }

        result = requests.post(
            "https://llm-gateway.assemblyai.com/v1/chat/completions",
            headers=headers,
            json=llm_gateway_data
        )
        return result.json()["choices"][0]["message"]["content"]

    def on_open(ws):
        print("WebSocket connection opened.")
        def stream_audio():
            global stream
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    break

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = True
        audio_thread.start()

    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get("type")

            if msg_type == "Begin":
                print(f"Session began: ID={data.get('id')}")
            elif msg_type == "Turn":
                transcript = data.get("transcript", "")
                if data.get("end_of_turn"):
                    print(f"\r{' ' * 80}\r", end="")
                    print(translate_text(transcript))
                else:
                    print(f"\r{transcript}", end="")
            elif msg_type == "Termination":
                print(f"\nSession terminated: {data.get('audio_duration_seconds', 0)}s of audio")
        except Exception as e:
            print(f"Error handling message: {e}")

    def on_error(ws, error):
        print(f"\nWebSocket Error: {error}")
        stop_event.set()

    def on_close(ws, close_status_code, close_msg):
        print(f"\nWebSocket Disconnected: Status={close_status_code}")
        global stream, audio
        stop_event.set()
        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
        if audio:
            audio.terminate()

    def run():
        global audio, stream, ws_app

        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Speak into your microphone. Press Ctrl+C to stop.")

        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nStopping...")
            stop_event.set()
            if ws_app and ws_app.sock and ws_app.sock.connected:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

    if __name__ == "__main__":
        run()
    ```

    
    

    ```javascript expandable
    import WebSocket from "ws";
    import mic from "mic";

    const YOUR_API_KEY = "YOUR_API_KEY";
    const CONNECTION_PARAMS = {
      sample_rate: 16000,
      speech_model: "universal-3-5-pro",
    };
    const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
    const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${new URLSearchParams(CONNECTION_PARAMS).toString()}`;
    const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;

    let micInstance = null;
    let ws = null;

    async function translateText(text) {
      const response = await fetch(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        {
          method: "POST",
          headers: {
            Authorization: YOUR_API_KEY,
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
            model: "gemini-2.5-flash-lite",
            messages: [
              { role: "user", content: `Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: ${text}` },
            ],
            max_tokens: 1000,
          }),
        }
      );
      const data = await response.json();
      return data.choices[0].message.content;
    }

    function run() {
      ws = new WebSocket(API_ENDPOINT, {
        headers: { Authorization: YOUR_API_KEY },
      });

      ws.on("open", () => {
        console.log("WebSocket connection opened.");

        micInstance = mic({
          rate: String(SAMPLE_RATE),
          channels: "1",
          bitwidth: "16",
          encoding: "signed-integer",
          endian: "little",
        });

        const micInputStream = micInstance.getAudioStream();
        micInputStream.on("data", (data) => {
          if (ws.readyState === WebSocket.OPEN) {
            ws.send(data);
          }
        });

        micInstance.start();
        console.log("Speak into your microphone. Press Ctrl+C to stop.");
      });

      ws.on("message", async (data) => {
        try {
          const msg = JSON.parse(data);
          if (msg.type === "Begin") {
            console.log(`Session began: ID=${msg.id}`);
          } else if (msg.type === "Turn") {
            const transcript = msg.transcript || "";
            if (msg.end_of_turn) {
              process.stdout.write("\r" + " ".repeat(80) + "\r");
              const translated = await translateText(transcript);
              console.log(translated);
            } else {
              process.stdout.write(`\r${transcript}`);
            }
          } else if (msg.type === "Termination") {
            console.log(
              `\nSession terminated: ${msg.audio_duration_seconds}s of audio`
            );
          }
        } catch (e) {
          console.error("Error handling message:", e);
        }
      });

      ws.on("error", (error) => {
        console.error("WebSocket error:", error);
      });

      ws.on("close", (code) => {
        console.log(`WebSocket closed: ${code}`);
        if (micInstance) micInstance.stop();
      });

      process.on("SIGINT", () => {
        console.log("\nStopping...");
        if (micInstance) micInstance.stop();
        if (ws && ws.readyState === WebSocket.OPEN) {
          ws.send(JSON.stringify({ type: "Terminate" }));
          setTimeout(() => {
            ws.close();
            process.exit(0);
          }, 2000);
        } else {
          process.exit(0);
        }
      });
    }

    run();
    ```

    

## Step-by-Step Instructions

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) and get your API key from your dashboard.

### Install Dependencies

    

    ```bash
    pip install websocket-client pyaudio requests
    ```

    
    

    ```bash
    npm install ws mic
    ```

    

### Import Packages & Set API Key

    

    ```python
    import pyaudio
    import websocket
    import json
    import threading
    import time
    import requests
    from urllib.parse import urlencode

    YOUR_API_KEY = "YOUR_API_KEY"  # Replace with your actual API key
    ```

    
    

    ```javascript
    import WebSocket from "ws";
    import mic from "mic";

    const YOUR_API_KEY = "YOUR_API_KEY";
    ```

    

### Audio Configuration & Global Variables

Set all of your audio configurations and global variables.

    

    ```python
    CONNECTION_PARAMS = {
        "sample_rate": 16000,
        "speech_model": "universal-3-5-pro",
    }
    API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws"
    API_ENDPOINT = f"{API_ENDPOINT_BASE_URL}?{urlencode(CONNECTION_PARAMS)}"

    FRAMES_PER_BUFFER = 800
    SAMPLE_RATE = CONNECTION_PARAMS["sample_rate"]
    CHANNELS = 1
    FORMAT = pyaudio.paInt16

    audio = None
    stream = None
    ws_app = None
    audio_thread = None
    stop_event = threading.Event()
    ```

    
    

    ```javascript
    const CONNECTION_PARAMS = {
      sample_rate: 16000,
      speech_model: "universal-3-5-pro",
    };
    const API_ENDPOINT_BASE_URL = "wss://streaming.assemblyai.com/v3/ws";
    const API_ENDPOINT = `${API_ENDPOINT_BASE_URL}?${new URLSearchParams(CONNECTION_PARAMS).toString()}`;
    const SAMPLE_RATE = CONNECTION_PARAMS.sample_rate;

    let micInstance = null;
    let ws = null;
    ```

    

### Define Translate Text Function

Define a function called `translate_text` (Python) or `translateText` (JavaScript), which uses LLM Gateway to translate the English final transcripts into another language. This example is translating the text into Spanish. To set this to a different language, just replace "Spanish" in the prompt with your language of choice.

    

    ```python
    def translate_text(text):
        """Called when translating final transcripts."""
        headers = {
            "authorization": YOUR_API_KEY
        }

        llm_gateway_data = {
            "model": "gemini-2.5-flash-lite",
            "messages": [
                {"role": "user", "content": f"Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: {text}"}
            ],
            "max_tokens": 1000
        }

        result = requests.post(
            "https://llm-gateway.assemblyai.com/v1/chat/completions",
            headers=headers,
            json=llm_gateway_data
        )
        return result.json()["choices"][0]["message"]["content"]
    ```

    
    

    ```javascript expandable
    async function translateText(text) {
      const response = await fetch(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        {
          method: "POST",
          headers: {
            Authorization: YOUR_API_KEY,
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
            model: "gemini-2.5-flash-lite",
            messages: [
              { role: "user", content: `Translate the following text into Spanish. Do not write a preamble. Just return the translated text.\n\nText: ${text}` },
            ],
            max_tokens: 1000,
          }),
        }
      );
      const data = await response.json();
      return data.choices[0].message.content;
    }
    ```

    

### Websocket Event Handlers

#### Open Websocket

    

    ```python
    def on_open(ws):
        print("WebSocket connection opened.")
        def stream_audio():
            global stream
            while not stop_event.is_set():
                try:
                    audio_data = stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
                    ws.send(audio_data, websocket.ABNF.OPCODE_BINARY)
                except Exception as e:
                    print(f"Error streaming audio: {e}")
                    break

        global audio_thread
        audio_thread = threading.Thread(target=stream_audio)
        audio_thread.daemon = True
        audio_thread.start()
    ```

    
    

    ```javascript expandable
    ws.on("open", () => {
      console.log("WebSocket connection opened.");

      micInstance = mic({
        rate: String(SAMPLE_RATE),
        channels: "1",
        bitwidth: "16",
        encoding: "signed-integer",
        endian: "little",
      });

      const micInputStream = micInstance.getAudioStream();
      micInputStream.on("data", (data) => {
        if (ws.readyState === WebSocket.OPEN) {
          ws.send(data);
        }
      });

      micInstance.start();
      console.log("Speak into your microphone. Press Ctrl+C to stop.");
    });
    ```

    

#### Handle Websocket Messages

In this function, use the previously defined `translate_text` / `translateText` to translate all final transcripts.

    

    ```python {10-12}
    def on_message(ws, message):
        try:
            data = json.loads(message)
            msg_type = data.get("type")

            if msg_type == "Begin":
                print(f"Session began: ID={data.get('id')}")
            elif msg_type == "Turn":
                transcript = data.get("transcript", "")
                if data.get("end_of_turn"):
                    print(f"\r{' ' * 80}\r", end="")
                    print(translate_text(transcript))
                else:
                    print(f"\r{transcript}", end="")
            elif msg_type == "Termination":
                print(f"\nSession terminated: {data.get('audio_duration_seconds', 0)}s of audio")
        except Exception as e:
            print(f"Error handling message: {e}")
    ```

    
    

    ```javascript {8-11} expandable
    ws.on("message", async (data) => {
      try {
        const msg = JSON.parse(data);
        if (msg.type === "Begin") {
          console.log(`Session began: ID=${msg.id}`);
        } else if (msg.type === "Turn") {
          const transcript = msg.transcript || "";
          if (msg.end_of_turn) {
            process.stdout.write("\r" + " ".repeat(80) + "\r");
            const translated = await translateText(transcript);
            console.log(translated);
          } else {
            process.stdout.write(`\r${transcript}`);
          }
        } else if (msg.type === "Termination") {
          console.log(
            `\nSession terminated: ${msg.audio_duration_seconds}s of audio`
          );
        }
      } catch (e) {
        console.error("Error handling message:", e);
      }
    });
    ```

    

#### Close Websocket

    

    ```python
    def on_close(ws, close_status_code, close_msg):
        print(f"\nWebSocket Disconnected: Status={close_status_code}")
        global stream, audio
        stop_event.set()
        if stream:
            if stream.is_active():
                stream.stop_stream()
            stream.close()
        if audio:
            audio.terminate()
    ```

    
    

    ```javascript
    ws.on("close", (code) => {
      console.log(`WebSocket closed: ${code}`);
      if (micInstance) micInstance.stop();
    });
    ```

    

#### Websocket Error Handling

    

    ```python
    def on_error(ws, error):
        print(f"\nWebSocket Error: {error}")
        stop_event.set()
    ```

    
    

    ```javascript
    ws.on("error", (error) => {
      console.error("WebSocket error:", error);
    });
    ```

    

### Begin Real-time STT Transcription

    

    ```python expandable
    def run():
        global audio, stream, ws_app

        audio = pyaudio.PyAudio()
        stream = audio.open(
            input=True,
            frames_per_buffer=FRAMES_PER_BUFFER,
            channels=CHANNELS,
            format=FORMAT,
            rate=SAMPLE_RATE,
        )
        print("Speak into your microphone. Press Ctrl+C to stop.")

        ws_app = websocket.WebSocketApp(
            API_ENDPOINT,
            header={"Authorization": YOUR_API_KEY},
            on_open=on_open,
            on_message=on_message,
            on_error=on_error,
            on_close=on_close,
        )

        ws_thread = threading.Thread(target=ws_app.run_forever)
        ws_thread.daemon = True
        ws_thread.start()

        try:
            while ws_thread.is_alive():
                time.sleep(0.1)
        except KeyboardInterrupt:
            print("\nStopping...")
            stop_event.set()
            if ws_app and ws_app.sock and ws_app.sock.connected:
                ws_app.send(json.dumps({"type": "Terminate"}))
                time.sleep(2)
            if ws_app:
                ws_app.close()
            ws_thread.join(timeout=2.0)

    if __name__ == "__main__":
        run()
    ```

    
    

    ```javascript expandable
    function run() {
      ws = new WebSocket(API_ENDPOINT, {
        headers: { Authorization: YOUR_API_KEY },
      });

      // ... event handlers defined above ...

      process.on("SIGINT", () => {
        console.log("\nStopping...");
        if (micInstance) micInstance.stop();
        if (ws && ws.readyState === WebSocket.OPEN) {
          ws.send(JSON.stringify({ type: "Terminate" }));
          setTimeout(() => {
            ws.close();
            process.exit(0);
          }, 2000);
        } else {
          process.exit(0);
        }
      });
    }

    run();
    ```

---

<a id="implement-a-sales-playbook-using-llm-gateway-docs-guides-sales-playbook-mdx"></a>

# Implement a Sales Playbook Using LLM Gateway

URL: https://www.assemblyai.com/docs/guides/sales-playbook
Source: docs/guides/sales-playbook.mdx
Navigation: LLM Gateway > Voice AI Guides > Use-case specific LLM Gateway workflows
Description: Implement a Sales Playbook Using LLM Gateway documentation.

This guide will show you how to use AssemblyAI's LLM Gateway to implement a sales playbook with a call from a sales representative to a client.

This guide aims to show different ways of using structured prompts with a hypothetical sales use case to produce personalized, precise responses. Using LLM Gateway, a user can immediately evaluate large numbers of sales calls and ensure that prospecting steps are followed, including quotes in the response, which can inform future sales by identifying trends and quantitative performance tracking.

In this example, we will demonstrate how to use structured prompts with context, answer formats, and answer options to create effective sales call evaluations with [LLM Gateway](/llm-gateway/quickstart). You can use the concepts in this guide to create custom specifications to evaluate your sales representatives.

## Quickstart

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

# Step 1: Transcribe the sales call
with open("./sales-call.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload", headers=headers, data=f)

upload_url = response.json()["upload_url"]
data = {"audio_url": upload_url}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

# Step 2: Evaluate with LLM Gateway
context = "There are sales interactions between a salesperson who is selling an internet plan to customers who are warm leads."
answer_format = """
Answer with JSON in the following format:
{
    "Answer": "<answer_options>",
    "Reason": "<justification for the answer in one sentence including quotes>"
}
"""

questions = [
    {
        "question": "Did the salesperson start the conversation with a professional greeting?",
        "answer_options": ["Poor", "Satisfactory", "Excellent"]
    },
    {
        "question": "How well did the salesperson answer questions during the call?",
        "answer_options": ["Poor", "Good", "Excellent"]
    },
    {
        "question": "Did the salesperson discuss next steps clearly?",
        "answer_options": ["Yes", "No"]
    }
]

for q in questions:
    prompt = f"""
{q['question']}

Context: {context}

Answer Options: {', '.join(q['answer_options'])}

{answer_format}
"""

    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
        ],
        "transcript_id": transcript_id,
        "max_tokens": 500
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()["choices"][0]["message"]["content"]
    print(f"Question: {q['question']}")
    print(f"Answer: {result}")
    print()
```

```javascript expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

// Step 1: Transcribe the sales call
const audioData = await fs.readFile("./sales-call.mp3");
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = { audio_url: uploadUrl };

res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let transcriptionResult;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcriptionResult = await res.json();
  if (transcriptionResult.status === "completed") {
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Step 2: Evaluate with LLM Gateway
const context = "There are sales interactions between a salesperson who is selling an internet plan to customers who are warm leads.";
const answerFormat = `
Answer with JSON in the following format:
{
    "Answer": "<answer_options>",
    "Reason": "<justification for the answer in one sentence including quotes>"
}
`;

const questions = [
  {
    question: "Did the salesperson start the conversation with a professional greeting?",
    answerOptions: ["Poor", "Satisfactory", "Excellent"]
  },
  {
    question: "How well did the salesperson answer questions during the call?",
    answerOptions: ["Poor", "Good", "Excellent"]
  },
  {
    question: "Did the salesperson discuss next steps clearly?",
    answerOptions: ["Yes", "No"]
  }
];

for (const q of questions) {
  const prompt = `
${q.question}

Context: ${context}

Answer Options: ${q.answerOptions.join(", ")}

${answerFormat}
`;

  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      { role: "user", content: `${prompt}\n\n{{ transcript }}` }
    ],
    transcript_id: transcriptId,
    max_tokens: 500
  };

  res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  const result = response.choices[0].message.content;
  console.log(`Question: ${q.question}`);
  console.log(`Answer: ${result}`);
  console.log();
}
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-Step Instructions

In this guide, we will ask three questions evaluating the prospecting performance of the sales representative. Each question has slightly different parameters based on the use case but largely has a fixed `context` that we will apply to each question.

Install the required packages:

```bash
pip install requests
```

```bash
npm install fs-extra
```

Set up the API client:

```python
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}
```

```javascript
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };
```

Transcribe the sales call audio file:

```python
with open("./sales-call.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload", headers=headers, data=f)

upload_url = response.json()["upload_url"]
data = {"audio_url": upload_url}  # You can also use a URL to an audio or video file on the web

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
        print(f"Transcription completed: {transcript_id}")
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

```javascript expandable
const audioData = await fs.readFile("./sales-call.mp3");
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = { audio_url: uploadUrl }; // You can also use a URL to an audio or video file on the web

res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

let transcriptionResult;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(`Transcription completed: ${transcriptId}`);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

Define your evaluation context and answer format for structured responses:

```python
context = "There are sales interactions between a salesperson who is selling an internet plan to customers who are warm leads."
answer_format = """
Answer with JSON in the following format:
{
    "Answer": "<answer_options>",
    "Reason": "<justification for the answer in one sentence including quotes>"
}
"""
```

```javascript
const context = "There are sales interactions between a salesperson who is selling an internet plan to customers who are warm leads.";
const answerFormat = `
Answer with JSON in the following format:
{
    "Answer": "<answer_options>",
    "Reason": "<justification for the answer in one sentence including quotes>"
}
`;
```

Next, define your evaluation questions for your sales playbook processes. Note: You can edit the questions and answer options to provide custom evaluations for each aspect of the sales call.

```python
questions = [
    {
        "question": "Did the salesperson start the conversation with a professional greeting?",
        "answer_options": ["Poor", "Satisfactory", "Excellent"]
    },
    {
        "question": "How well did the salesperson answer questions during the call?",
        "answer_options": ["Poor", "Good", "Excellent"]
    },
    {
        "question": "Did the salesperson discuss next steps clearly?",
        "answer_options": ["Yes", "No"]
    }
]
```

```javascript
const questions = [
  {
    question: "Did the salesperson start the conversation with a professional greeting?",
    answerOptions: ["Poor", "Satisfactory", "Excellent"]
  },
  {
    question: "How well did the salesperson answer questions during the call?",
    answerOptions: ["Poor", "Good", "Excellent"]
  },
  {
    question: "Did the salesperson discuss next steps clearly?",
    answerOptions: ["Yes", "No"]
  }
];
```

Evaluate each question using LLM Gateway and print the results:

```python expandable
for q in questions:
    prompt = f"""
{q['question']}

Context: {context}

Answer Options: {', '.join(q['answer_options'])}

{answer_format}
"""

    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
        ],
        "transcript_id": transcript_id,
        "max_tokens": 500
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()["choices"][0]["message"]["content"]
    print(f"Question: {q['question']}")
    print(f"Answer: {result}")
    print()
```

```javascript expandable
for (const q of questions) {
  const prompt = `
${q.question}

Context: ${context}

Answer Options: ${q.answerOptions.join(", ")}

${answerFormat}
`;

  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      { role: "user", content: `${prompt}\n\n{{ transcript }}` }
    ],
    transcript_id: transcriptId,
    max_tokens: 500
  };

  let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  const result = response.choices[0].message.content;
  console.log(`Question: ${q.question}`);
  console.log(`Answer: ${result}`);
  console.log();
}
```

---

<a id="segment-a-phone-call-using-llm-gateway-docs-guides-phone-call-segmentation-mdx"></a>

# Segment A Phone Call using LLM Gateway

URL: https://www.assemblyai.com/docs/guides/phone-call-segmentation
Source: docs/guides/phone-call-segmentation.mdx
Navigation: LLM Gateway > Voice AI Guides > Use-case specific LLM Gateway workflows
Description: Segment A Phone Call using LLM Gateway documentation.

In this guide we will show you how to use AssemblyAI's LLM Gateway to segment a phone call. LLM Gateway provides access to multiple LLM providers through a unified API.

## Quickstart

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

# Step 1: Transcribe and get VTT subtitles
with open("./my-audio.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload", headers=headers, data=f)

upload_url = response.json()["upload_url"]
data = {"audio_url": upload_url}

response = requests.post(base_url + "/v2/transcript", json=data, headers=headers)
transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()
    if transcription_result['status'] == 'completed':
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)

# Get VTT subtitles
vtt_response = requests.get(f"{polling_endpoint}/vtt", headers=headers)
vtt_content = vtt_response.text

# Step 2: Define phases and analyze with LLM Gateway
phases = ["Introduction", "Complaint", "Resolution", "Goodbye"]

prompt = f'''
Analyze the following transcript of a phone call conversation and divide it into the following phases:
{', '.join(phases)}

You will be given the transcript in the format of VTT captions.

For each phase:
1. Identify the start and end timestamps (in seconds)
2. Provide a brief summary of what happened in that phase

Format your response as a JSON object with the following structure:
{{
    "phases": [
        {{
            "name": "Phase Name",
            "start_time": start_time_in_seconds,
            "end_time": end_time_in_seconds,
            "summary": "Brief summary of the phase"
        }},
        ...
    ]
}}

Ensure that all parts of the conversation are covered by a phase, using "Other" for any parts that don't fit into the specified phases.
'''

llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\nVTT Transcript:\n{vtt_content}"}
    ],
    "max_tokens": 2000
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)

result = response.json()["choices"][0]["message"]["content"]
print(result)
```

```javascript expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

// Step 1: Transcribe and get VTT subtitles
const audioData = await fs.readFile("./my-audio.mp3");
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = { audio_url: uploadUrl };

res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();
const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();
  if (transcriptionResult.status === "completed") {
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Get VTT subtitles
res = await fetch(`${pollingEndpoint}/vtt`, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const vttContent = await res.text();

// Step 2: Define phases and analyze with LLM Gateway
const phases = ["Introduction", "Complaint", "Resolution", "Goodbye"];

const prompt = `
Analyze the following transcript of a phone call conversation and divide it into the following phases:
${phases.join(", ")}

You will be given the transcript in the format of VTT captions.

For each phase:
1. Identify the start and end timestamps (in seconds)
2. Provide a brief summary of what happened in that phase

Format your response as a JSON object with the following structure:
{
    "phases": [
        {
            "name": "Phase Name",
            "start_time": start_time_in_seconds,
            "end_time": end_time_in_seconds,
            "summary": "Brief summary of the phase"
        },
        ...
    ]
}

Ensure that all parts of the conversation are covered by a phase, using "Other" for any parts that don't fit into the specified phases.
`;

const llmGatewayData = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\nVTT Transcript:\n${vttContent}` },
  ],
  max_tokens: 2000,
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llmGatewayData),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const result = response.choices[0].message.content;
console.log(result);
```

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard.

## Step-by-Step Instructions

Install the required packages:

```bash
pip install requests
```

```bash
npm install fs-extra
```

Set up your API client and transcribe the audio file:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./my-audio.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url # You can also use a URL to an audio or video file on the web
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
        print(f"Transcript ID: {transcript_id}")
        break

    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

    else:
        time.sleep(3)

# Get VTT subtitles for timestamp information
vtt_response = requests.get(f"{polling_endpoint}/vtt", headers=headers)
vtt_content = vtt_response.text

with open(f"transcript_{transcript_id}.vtt", "w") as vtt_file:
    vtt_file.write(vtt_content)
```

```javascript expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const audioData = await fs.readFile("./my-audio.mp3");
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl // You can also use a URL to an audio or video file on the web
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();

const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log("Transcript ID:", transcriptId);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Get VTT subtitles for timestamp information
res = await fetch(`${pollingEndpoint}/vtt`, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const vttContent = await res.text();

fs.writeFileSync(`transcript_${transcriptId}.vtt`, vttContent);
```

Define the phases you want to identify.

Here is an example of `phases` that can be used for customer support calls:

```python
phases = [
    "Introduction",
    "Complaint",
    "Resolution",
    "Goodbye"
]
```

```javascript
const phases = [
  "Introduction",
  "Complaint",
  "Resolution",
  "Goodbye",
];
```

Use LLM Gateway to analyze the transcript and divide it into phases. This is an example prompt, which you can modify to suit your specific requirements.

```python expandable
prompt = f'''
Analyze the following transcript of a phone call conversation and divide it into the following phases:
{', '.join(phases)}

You will be given the transcript in the format of VTT captions.

For each phase:
1. Identify the start and end timestamps (in seconds)
2. Provide a brief summary of what happened in that phase

Format your response as a JSON object with the following structure:
{{
    "phases": [
        {{
            "name": "Phase Name",
            "start_time": start_time_in_seconds,
            "end_time": end_time_in_seconds,
            "summary": "Brief summary of the phase"
        }},
        ...
    ]
}}

Ensure that all parts of the conversation are covered by a phase, using "Other" for any parts that don't fit into the specified phases.
'''

llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\nVTT Transcript:\n{vtt_content}"}
    ],
    "max_tokens": 2000
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)

result = response.json()["choices"][0]["message"]["content"]
print(result)
```

```javascript expandable
const prompt = `
Analyze the following transcript of a phone call conversation and divide it into the following phases:
${phases.join(", ")}

You will be given the transcript in the format of VTT captions.

For each phase:
1. Identify the start and end timestamps (in seconds)
2. Provide a brief summary of what happened in that phase

Format your response as a JSON object with the following structure:
{
    "phases": [
        {
            "name": "Phase Name",
            "start_time": start_time_in_seconds,
            "end_time": end_time_in_seconds,
            "summary": "Brief summary of the phase"
        },
        ...
    ]
}

Ensure that all parts of the conversation are covered by a phase, using "Other" for any parts that don't fit into the specified phases.
`;

const llmGatewayData = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\nVTT Transcript:\n${vttContent}` },
  ],
  max_tokens: 2000,
};

let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llmGatewayData),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const result = response.choices[0].message.content;
console.log(result);
```

Example output of the analysis of a transcript divided into phases and formatted as a JSON object:

``` expandable

{
    "phases": [
        {
            "name": "Introduction",
            "start_time": 1.52,
            "end_time": 15.57,
            "summary": "The customer service representative greets the caller and asks how they can help. The caller states they want to know the status of their order refund."
        },
        {
            "name": "Complaint",
            "start_time": 15.57,
            "end_time": 59.41,
            "summary": "The representative asks for the order ID, which the caller provides. The representative confirms the order details and that it was cancelled. The caller mentions they couldn't complete their test."
        },
        {
            "name": "Resolution",
            "start_time": 59.41,
            "end_time": 210.01,
            "summary": "The representative informs the caller that the refund was initiated on April 8th and will be credited by April 21st. They explain the refund timeline and bank processing days. The caller expresses some confusion about the timeline, and the representative clarifies the process."
        },
        {
            "name": "Goodbye",
            "start_time": 210.01,
            "end_time": 235.8,
            "summary": "The caller accepts the explanation. The representative asks if there's anything else they can help with, requests feedback, and concludes the call with a farewell."
        }
    ]
}
```

---

<a id="generate-soap-notes-using-llm-gateway-docs-guides-soap-note-generation-mdx"></a>

# Generate SOAP Notes using LLM Gateway

URL: https://www.assemblyai.com/docs/guides/soap-note-generation
Source: docs/guides/soap-note-generation.mdx
Navigation: LLM Gateway > Voice AI Guides > Use-case specific LLM Gateway workflows
Description: Generate SOAP Notes using LLM Gateway documentation.

The acronym SOAP stands for Subjective, Objective, Assessment, and Plan. This standardized method of documenting patient encounters allows providers to concisely record patient information. This guide walks through how to generate SOAP notes with AssemblyAI's LLM Gateway, which provides access to multiple LLM providers through a unified API.

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up for an AssemblyAI account](https://www.assemblyai.com/dashboard/home) and get your API key from your dashboard. You will need to upgrade your account by adding a credit card to have access to LLM Gateway.

Find more details on the current pricing in the AssemblyAI [pricing page](https://www.assemblyai.com/pricing).

## Step-by-Step Instructions

Install the required packages:

```bash
pip install requests
```

```bash
npm install fs-extra
```

Set up your transcription with PII redaction enabled to protect patient information:

```python expandable
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

with open("./local_file.mp3", "rb") as f:
    response = requests.post(base_url + "/v2/upload",
                            headers=headers,
                            data=f)

upload_url = response.json()["upload_url"]

data = {
    "audio_url": upload_url, # You can also use a URL to an audio or video file on the web
    "redact_pii": True,
    "redact_pii_policies": ["person_name", "organization", "occupation"],
    "redact_pii_sub": "hash"
}

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

transcript_id = response.json()['id']
polling_endpoint = base_url + "/v2/transcript/" + transcript_id

print(f"Transcript ID: {transcript_id}")

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
        print(transcription_result['text'])
        break
    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")
    else:
        time.sleep(3)
```

```javascript expandable
import fs from "fs-extra";

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const audioData = await fs.readFile("./local_file.mp3");
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL to an audio or video file on the web
  redact_pii: true,
  redact_pii_policies: ["person_name", "organization", "occupation"],
  redact_pii_sub: "hash",
};

const url = `${baseUrl}/v2/transcript`;
res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcriptResponse = await res.json();

const transcriptId = transcriptResponse.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

console.log("Transcript ID:", transcriptId);

let transcriptionResult;
while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

## Two different methods of generating SOAP notes

### Method 1: Complete SOAP Note Generation

This method uses LLM Gateway to generate the entire SOAP note in one request. The prompt includes context about the audio file, a specific format for the SOAP note, and detailed instructions for each section.

```python expandable
prompt = """
Generate a SOAP note summary based on the following doctor-patient appointment transcript.

Context: This is a doctor appointment between patient and provider. The personal identification information has been redacted.

Format the response as follows:

Subjective
This is typically the shortest section (only 2-3 sentences) and it describes the patient's affect, as the professional sees it. This information is all subjective (it isn't measureable).
- Include information that may have affected the patient's performance, such as if they were sick, tired, attentive, distractible, etc.
- Was the patient on time or did they come late?
- May include a quote of something the patient said, or how they reported feeling

Objective
This section includes factual, measurable, and objective information. This may include:
- Direct patient quotes
- Measurements
- Data on patient performance

Assessment
This section should be the meat of the SOAP note. It contains a narrative of what actually happened during the session. There may be information regarding:
- Whether improvements have been made since the last session
- Any potential barriers to success
- Clinician's interpretation of the results of the session

Plan
This is another short section that states the plan for future sessions. In most settings, this section may be bulleted
"""

# Send to LLM Gateway
llm_gateway_data = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
    ],
    "transcript_id": transcript_id,
    "max_tokens": 2000
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)

result = response.json()["choices"][0]["message"]["content"]
print(result.strip())
```

```javascript expandable
const prompt = `
Generate a SOAP note summary based on the following doctor-patient appointment transcript.

Context: This is a doctor appointment between patient and provider. The personal identification information has been redacted.

Format the response as follows:

Subjective
This is typically the shortest section (only 2-3 sentences) and it describes the patient's affect, as the professional sees it. This information is all subjective (it isn't measureable).
- Include information that may have affected the patient's performance, such as if they were sick, tired, attentive, distractible, etc.
- Was the patient on time or did they come late?
- May include a quote of something the patient said, or how they reported feeling

Objective
This section includes factual, measurable, and objective information. This may include:
- Direct patient quotes
- Measurements
- Data on patient performance

Assessment
This section should be the meat of the SOAP note. It contains a narrative of what actually happened during the session. There may be information regarding:
- Whether improvements have been made since the last session
- Any potential barriers to success
- Clinician's interpretation of the results of the session

Plan
This is another short section that states the plan for future sessions. In most settings, this section may be bulleted
`;

// Send to LLM Gateway
const llmGatewayData = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\n{{ transcript }}` },
  ],
  transcript_id: transcriptId,
  max_tokens: 2000,
};

let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llmGatewayData),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const result = response.choices[0].message.content;
console.log(result.trim());
```

### Method 2: Section-by-Section Generation

This method generates each section of the SOAP note separately using LLM Gateway. This makes it easy to regenerate one or more sections individually if needed. Each section is generated with a specific prompt tailored to that part of the SOAP note.

```python expandable
# Define questions for each SOAP section
questions = [
    {
        "section": "Subjective",
        "question": "What are the patient's current symptoms or concerns?",
        "context": "Gather information about the patient's subjective experience. Example: The patient reports experiencing persistent headaches and dizziness.",
        "format": "<patient's symptoms or concerns>, [exact quote from patient]"
    },
    {
        "section": "Objective",
        "question": "What are the measurable and observable findings from the examination?",
        "context": "Collect data on the patient's objective signs and measurements. Example: The examination reveals an elevated body temperature and increased heart rate.",
        "format": "<measurable and observable findings>"
    },
    {
        "section": "Assessment",
        "question": "Based on the patient's history and examination, what is your assessment or diagnosis?",
        "context": "Formulate a professional assessment based on the gathered information. Example: Based on the patient's symptoms, examination, and medical history, the preliminary diagnosis is migraine.",
        "format": "<assessment or diagnosis>"
    },
    {
        "section": "Plan",
        "question": "What is the plan of action or treatment for the patient?",
        "context": "Outline the intended course of action or treatment. Example: The treatment plan includes prescribing medication, recommending rest, and scheduling a follow-up appointment in two weeks.",
        "format": "<plan of action or treatment>, [exact quote from provider]"
    }
]

# Process each section
for q in questions:
    prompt = f"""
{q['question']}

Context: This is a doctor appointment between patient and provider. The personal identification information has been redacted. {q['context']}

Answer Format: {q['format']}
"""

    llm_gateway_data = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
        ],
        "transcript_id": transcript_id,
        "max_tokens": 1000
    }

    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers=headers,
        json=llm_gateway_data
    )

    result = response.json()["choices"][0]["message"]["content"]
    print(f"{q['section']}: {q['question']}")
    print(result.strip())
    print()
```

```javascript expandable
// Define questions for each SOAP section
const questions = [
  {
    section: "Subjective",
    question: "What are the patient's current symptoms or concerns?",
    context: "Gather information about the patient's subjective experience. Example: The patient reports experiencing persistent headaches and dizziness.",
    format: "<patient's symptoms or concerns>, [exact quote from patient]",
  },
  {
    section: "Objective",
    question: "What are the measurable and observable findings from the examination?",
    context: "Collect data on the patient's objective signs and measurements. Example: The examination reveals an elevated body temperature and increased heart rate.",
    format: "<measurable and observable findings>",
  },
  {
    section: "Assessment",
    question: "Based on the patient's history and examination, what is your assessment or diagnosis?",
    context: "Formulate a professional assessment based on the gathered information. Example: Based on the patient's symptoms, examination, and medical history, the preliminary diagnosis is migraine.",
    format: "<assessment or diagnosis>",
  },
  {
    section: "Plan",
    question: "What is the plan of action or treatment for the patient?",
    context: "Outline the intended course of action or treatment. Example: The treatment plan includes prescribing medication, recommending rest, and scheduling a follow-up appointment in two weeks.",
    format: "<plan of action or treatment>, [exact quote from provider]",
  },
];

// Process each section
for (const q of questions) {
  const prompt = `
${q.question}

Context: This is a doctor appointment between patient and provider. The personal identification information has been redacted. ${q.context}

Answer Format: ${q.format}
`;

  const llmGatewayData = {
    model: "claude-sonnet-4-6",
    messages: [
      { role: "user", content: `${prompt}\n\n{{ transcript }}` },
    ],
    transcript_id: transcriptId,
    max_tokens: 1000,
  };

  let res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
    method: "POST",
    headers: { ...headers, "Content-Type": "application/json" },
    body: JSON.stringify(llmGatewayData),
  });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const response = await res.json();

  const result = response.choices[0].message.content;
  console.log(`${q.section}: ${q.question}`);
  console.log(result.trim());
  console.log();
}
```

---

<a id="frequently-asked-questions-docs-faq-mdx"></a>

# Frequently Asked Questions

URL: https://www.assemblyai.com/docs/faq
Source: docs/faq.mdx
Navigation: FAQ > Overview
Description: Find answers to common questions about AssemblyAI's services, features, and pricing.

<style>
  {`
  /* Styles for custom list items */
  .no-bullets {
    list-style-type: none;
    padding-left: 0;
    margin-left: 0;
  }

  .no-bullets li {
    margin: 0;
    padding-left: 0;

    &:before {
      display: none !important;
    }
  }

  .link-cta {
    text-decoration: none;
    padding: 8px 12px;
    margin-left: -12px;
    border-radius: 0.5rem;

    p {
      margin: 0 !important;
    }
  }

  .link-cta:hover {
    background-color: var(--accent-3);
  }

  .link-cta svg {
    transition: transform 0.2s ease;
  }

  .link-cta:hover svg {
    transform: translateX(4px);
  }

  .card-stack {
    position: relative;
    width: 100%;
    margin-bottom: 20px;
  }

  body#fern-docs .card-stack > div {
    height: 100%;
  }

  .card-stack .fern-card {
    background: var(--grayscale-1);
    height: 100%;
    overflow: hidden;
  }

  .card-stack .fern-card > div {
    height: 100%;
  }

   .card-stack .fern-card > div > div {
    height: 100%;
  }

  .card-stack .fern-card > div > div > div:nth-child(2) {
    height: 100%;
  }

  /* Animation to change the color of the CTA text when hovering over the featured FAQ */
  .card-stack .fern-card:hover .cta {
    color: rgba(var(--accent-aaa),var(--tw-text-opacity,1));
    transition: color 0.2s ease;
  }

  .card-stack .fern-card .cta svg {
    transform: translateX(0);
    transition: color 0.2s ease, transform 0.2s ease;
  }

  .card-stack .fern-card:hover .cta svg {
    background-color: rgba(var(--accent-aaa),var(--tw-text-opacity,1));
    transform: translateX(4px);
  }

  /* Styling for the card stack effect behind each featured FAQ */
  .card-stack::before,
  .card-stack::after {
    content: '';
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    border-radius: 1rem;
    z-index: -1;
    border: 1px solid var(--border);
    background-color: var(--grayscale-1);
  }

  .card-stack::before {
    transform: translateY(8px) translateX(-8px);
    border-radius: 1.25rem;
  }

  .card-stack::after {
    transform: translateY(4px) translateX(-4px);
  }

  /* Animation to collapse card stack when hovering over the featured FAQ */
  .card-stack:hover::before,
  .card-stack:hover::after {
    transform: translateY(0) translateX(0);
    transition: transform 0.2s ease;
  }

  .bg-noise {
    background-position: 0 0;
    background-size: 5rem 10rem;
    background-image: url(https://fern-image-hosting.s3.us-east-1.amazonaws.com/6789751936e0c7bd371e8cf2_noise-full.webp);
    z-index: -1;
    opacity: 0.7;
    justify-content: center;
    align-items: center;
    width: calc(100% + 5rem);
    height: calc(100% + 5rem);
    display: flex;
    position: absolute;
    top: 0;
    left: 0;
  }
`}
</style>

Get quick answers to the most common questions about AssemblyAI's features, pricing, and services.

  Use the Search bar above to find specific information, or ask our AI assistant
  any question about AssemblyAI by typing in the search box!

## **Popular**

  <div className="card-stack">
    
      <div className="bg-noise"></div>
      <div className="flex flex-col h-full justify-between">
        <div className="flex flex-col gap-4 mb-auto">
          <div className="t-default text-lg font-semibold">
            How to Opt Out of Data Sharing for our Model Improvement Program
          </div>
          <div className="flex flex-row gap-2">
            <p className="t-muted text-sm">Privacy & Security</p>
          </div>
        </div>
      </div>
    
  </div>

  <div className="card-stack">
    
      <div className="bg-noise"></div>
      <div className="flex flex-col h-full justify-between">
        <div className="flex flex-col gap-4 mb-auto">
          <div className="t-default text-lg font-semibold">
            Are there any limits on file size or file duration for files submitted to the API?
          </div>
          <div className="flex flex-row gap-2">
            <p className="t-muted text-sm">Pre-recorded audio</p>
          </div>
        </div>
      </div>
    
  </div>

  <div className="card-stack">
    
      <div className="bg-noise"></div>
      <div className="flex flex-col h-full justify-between">
        <div className="flex flex-col gap-4 mb-auto">
          <div className="t-default text-lg font-semibold">
            What audio and video file types are supported by your API?
          </div>
          <div className="flex flex-row gap-2">
            <p className="t-muted text-sm">Pre-recorded audio</p>
          </div>
        </div>
      </div>
    
  </div>

  <div className="card-stack">
    
      <div className="bg-noise"></div>
      <div className="flex flex-col h-full justify-between">
        <div className="flex flex-col gap-4 mb-auto">
          <div className="t-default text-lg font-semibold">
            Will AssemblyAI sign a Business Associate Addendum (BAA)?
          </div>
          <div className="flex flex-row gap-2">
            <p className="t-muted text-sm">Privacy & Security</p>
          </div>
        </div>
      </div>
    
  </div>

  <div className="card-stack">
    
      <div className="bg-noise"></div>
      <div className="flex flex-col h-full justify-between">
        <div className="flex flex-col gap-4 mb-auto">
          <div className="t-default text-lg font-semibold">
            How does the automatically scaling streaming rate limit for Real-time STT work?
          </div>
          <div className="flex flex-row gap-2">
            <p className="t-muted text-sm">Streaming audio</p>
          </div>
        </div>
      </div>
    
  </div>

  <div className="card-stack">
    
      <div className="bg-noise"></div>
      <div className="flex flex-col h-full justify-between">
        <div className="flex flex-col gap-4 mb-auto">
          <div className="t-default text-lg font-semibold">
            Can I sign up for free?
          </div>
          <div className="flex flex-row gap-2">
            <p className="t-muted text-sm">Account, Billing & Payments</p>
          </div>
        </div>
      </div>
    
  </div>

---

<a id="has-assemblyai-certified-to-the-eu-u-s-data-privacy-framework-docs-faq-has-assemblyai-certified-to-the-eu-us-data-privacy-framework-mdx"></a>

# Has AssemblyAI certified to the EU-U.S. Data Privacy Framework?

URL: https://www.assemblyai.com/docs/faq/has-assemblyai-certified-to-the-eu-us-data-privacy-framework
Source: docs/faq/has-assemblyai-certified-to-the-eu-us-data-privacy-framework.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Has AssemblyAI certified to the EU-U.S. Data Privacy Framework? documentation.

Yes, AssemblyAI has certified to the EU-U.S. Data Privacy Framework (DPF) and adheres to the [DPF Principles](https://www.dataprivacyframework.gov/s/framework-text). You can view the AssemblyAI DPF certification [here](https://www.dataprivacyframework.gov/list). Please note that to locate the certification, search for "AssemblyAI" in the search bar.

---

<a id="can-i-sign-a-dpa-agreement-with-assemblyai-docs-faq-can-i-sign-a-dpa-agreement-with-assemblyai-mdx"></a>

# Can I sign a DPA agreement with AssemblyAI?

URL: https://www.assemblyai.com/docs/faq/can-i-sign-a-dpa-agreement-with-assemblyai
Source: docs/faq/can-i-sign-a-dpa-agreement-with-assemblyai.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Can I sign a DPA agreement with AssemblyAI? documentation.

Yes. As an AssemblyAI customer, you automatically agree to a Data Processing Addendum (DPA) as part of our terms of service. You do not need to sign a separate agreement.

You can review the full DPA here: [AssemblyAI Data Processing Addendum](https://www.assemblyai.com/legal/data-processing-addendum).

For additional details about how AssemblyAI handles your data, see our [Data retention and model training](/data-retention-and-model-training) documentation.

---

<a id="can-you-provide-a-copy-of-your-most-recent-penetration-test-executive-summary-docs-faq-can-you-provide-a-copy-of-your-most-recent-penetration-test-executive-summary-mdx"></a>

# Can you provide a copy of your most recent penetration test executive summary?

URL: https://www.assemblyai.com/docs/faq/can-you-provide-a-copy-of-your-most-recent-penetration-test-executive-summary
Source: docs/faq/can-you-provide-a-copy-of-your-most-recent-penetration-test-executive-summary.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Can you provide a copy of your most recent penetration test executive summary? documentation.

A copy of our most recent external penetration test can be found on our [Trust Center](https://app.vanta.com/assemblyai/trust/7n80syl8zln1bn1qm3x8eg).

We also conduct credentialed penetration tests however we do not provide those to customers because they provide very detailed information about our production systems.

---

<a id="can-you-provide-a-recent-vulnerability-scan-docs-faq-can-you-provide-a-recent-vulnerability-scan-mdx"></a>

# Can you provide a recent vulnerability scan?

URL: https://www.assemblyai.com/docs/faq/can-you-provide-a-recent-vulnerability-scan
Source: docs/faq/can-you-provide-a-recent-vulnerability-scan.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Can you provide a recent vulnerability scan? documentation.

We can provide ASV scans to customers with a signed NDA. ASV scans are uncredentialed vulnerability scans directed to production endpoints that report on exposed vulnerabilities. We produce these quarterly at a minimum to maintain our PCI certification.

---

<a id="will-assemblyai-sign-a-business-associate-addendum-baa-as-described-in-the-hipaa-rules-and-regulations-docs-faq-can-you-sign-a-baa-mdx"></a>

# Will AssemblyAI sign a Business Associate Addendum (BAA) as described in the HIPAA rules and regulations?

URL: https://www.assemblyai.com/docs/faq/can-you-sign-a-baa
Source: docs/faq/can-you-sign-a-baa.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Will AssemblyAI sign a Business Associate Addendum (BAA) as described in the HIPAA rules and regulations? documentation.

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AAI appropriately safeguards PHI.

You can review our [standard Business Associate Addendum (BAA)](https://www.assemblyai.com/legal/business-associate-agreement).

Paid customers can review and sign our standard online BAA terms self-serve, at no additional cost, from the [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the AssemblyAI dashboard. Only members with the **Owner** or **Admin** role can sign. For more detail, see [Data Controls](/data-controls#business-associate-agreement-baa).

  You must have an upgraded (paid) account to see the option to initiate a BAA in the dashboard. Free accounts do not see the BAA controls — [upgrade to a paid plan](https://www.assemblyai.com/dashboard/pricing) first.

  Create your AssemblyAI account before completing the BAA agreement.

  Once you have signed a BAA, you are automatically opted out of the model improvement program upon signature, and files submitted under the BAA are not used for model training.

---

<a id="do-you-have-a-formal-risk-assessment-policy-or-process-docs-faq-do-you-have-a-formal-risk-assessment-policy-or-process-mdx"></a>

# Do you have a formal risk assessment policy or process?

URL: https://www.assemblyai.com/docs/faq/do-you-have-a-formal-risk-assessment-policy-or-process
Source: docs/faq/do-you-have-a-formal-risk-assessment-policy-or-process.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Do you have a formal risk assessment policy or process? documentation.

Yes, we have a formal risk assessment policy. We conduct quarterly risk reviews including conducting business impact analysis on high-risk events, and keeping track of risks within Jira, our ticketing system.

Samples of old risk register exports can be provided to select customers with signed NDAs, but not current versions as some risks may still be open.

---

<a id="do-you-have-documented-information-security-policies-if-so-how-frequently-are-they-updated-docs-faq-do-you-have-documented-information-security-policies-if-so-how-frequently-are-they-updated-mdx"></a>

# Do you have documented information security policies? If so, how frequently are they updated?

URL: https://www.assemblyai.com/docs/faq/do-you-have-documented-information-security-policies-if-so-how-frequently-are-they-updated
Source: docs/faq/do-you-have-documented-information-security-policies-if-so-how-frequently-are-they-updated.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Do you have documented information security policies? If so, how frequently are they updated? documentation.

Yes, AssemblyAI maintains documented information security policies. These policies are available on our [Trust Center](https://app.vanta.com/assemblyai/trust/7n80syl8zln1bn1qm3x8eg) and are reviewed and updated annually.

AssemblyAI is SOC 2 Type II certified and HIPAA compliant. You can access our security reports and certifications through the [Trust Center](https://app.vanta.com/assemblyai/trust/7n80syl8zln1bn1qm3x8eg). For more information about how we handle your data, see [Data retention and model training](/data-retention-and-model-training).

---

<a id="do-you-offer-eu-data-residency-docs-faq-do-you-offer-eu-data-residency-mdx"></a>

# Do you offer EU Data Residency?

URL: https://www.assemblyai.com/docs/faq/do-you-offer-eu-data-residency
Source: docs/faq/do-you-offer-eu-data-residency.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Do you offer EU Data Residency? documentation.

For users who need to process data within European Union data centers, we offer access to European servers via our EU endpoint, which runs on AWS servers located in the EU West 1 (Dublin) region.

Here are some benefits of EU Data Residency:

- Instant EU processing via API parameters and dedicated endpoint
- Continued data sovereignty for EU operations
- Regional usage tracking and cost management in your dashboard

See [our documentation](/pre-recorded-audio/select-the-region) to learn more about using our EU endpoint.

---

<a id="do-you-offer-self-hosted-solutions-docs-faq-do-you-offer-self-hosted-solutions-mdx"></a>

# Do you offer self-hosted solutions?

URL: https://www.assemblyai.com/docs/faq/do-you-offer-self-hosted-solutions
Source: docs/faq/do-you-offer-self-hosted-solutions.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Do you offer self-hosted solutions? documentation.

Yes, we offer self-hosted solutions for organizations that require greater control over their data and infrastructure. Our self-hosted options allow you to deploy our services within your own environment, ensuring compliance with your organization's security policies and regulatory requirements.

For more information on self-hosted streaming speech-to-text, see [Self-Hosted Streaming](/streaming/self-hosted-streaming).

[Contact our sales team](https://www.assemblyai.com/contact/sales) to discuss your specific needs and to learn more about our self-hosted offerings.

---

<a id="do-you-offer-servers-in-the-eu-docs-faq-do-you-offer-servers-in-the-eu-mdx"></a>

# Do you offer servers in the EU?

URL: https://www.assemblyai.com/docs/faq/do-you-offer-servers-in-the-eu
Source: docs/faq/do-you-offer-servers-in-the-eu.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Do you offer servers in the EU? documentation.

Yes, we have EU endpoints for Async transcription, Streaming Speech-to-Text, LLM Gateway, and the Voice Agent API if your use case requires EU data residency.

- **Async transcription** (AWS eu-west-1, Dublin): use the base URL `api.eu.assemblyai.com`.
- **Streaming Speech-to-Text** (AWS eu-north-1, Stockholm): use `streaming.eu.assemblyai.com`.
- **LLM Gateway** (AWS eu-west-1, Dublin): use `llm-gateway.eu.assemblyai.com/v1/chat/completions`. The EU endpoint supports Anthropic Claude and Google Gemini models.
- **Voice Agent API** (AWS eu-west-1, Dublin): use `agents.eu.assemblyai.com`.

See [Cloud endpoints and data residency](/pre-recorded-audio/select-the-region) for code examples showing how to change the base URL with our SDKs.

---

<a id="do-you-support-saml-in-your-product-docs-faq-do-you-support-saml-in-your-product-mdx"></a>

# Do you support SAML in your product?

URL: https://www.assemblyai.com/docs/faq/do-you-support-saml-in-your-product
Source: docs/faq/do-you-support-saml-in-your-product.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Do you support SAML in your product? documentation.

We do not currently support SAML-based SSO, but it is on our roadmap and expected to be available soon.

  Interested in SAML SSO? Reach out to our team at [support@assemblyai.com](mailto:support@assemblyai.com) for more information.

---

<a id="how-long-are-outputs-maintained-docs-faq-how-long-are-outputs-retained-mdx"></a>

# How long are outputs maintained?

URL: https://www.assemblyai.com/docs/faq/how-long-are-outputs-retained
Source: docs/faq/how-long-are-outputs-retained.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: How long are outputs maintained? documentation.

For a full breakdown of how AssemblyAI maintains audio and transcript data, see [Data retention and model training](/data-retention-and-model-training).

For Async STT, Final Transcription Artifacts (transcripts) are stored for **30 days by default and are automatically deleted after that**. You can shorten this window by setting a time-to-live (TTL) on the [Data Controls page](https://www.assemblyai.com/dashboard/settings/data-controls) in the dashboard — presets of 1 day, 3 days, 7 days, and 30 days are available, or you can set a **Custom** value. See [Set a time-to-live for audio and transcripts](/data-controls#set-a-time-to-live-for-audio-and-transcripts).

For customers with a BAA, all transcripts generated via the async transcription endpoint are deleted after the TTL period (currently set to 3 days by default).

You can also delete individual transcripts on demand via the [DELETE endpoint](/pre-recorded-audio/delete-transcripts).

---

<a id="does-assemblyai-utilize-an-anti-virus-anti-malware-solution-across-all-relevant-infrastructure-workstations-and-servers-and-are-appropriate-response-capabilities-deployed-to-respond-to-alerts-docs-faq-does-assemblyai-utilize-an-anti-virus-anti-malware-solution-across-all-relevant-infrastructure-workstations-and-servers-and-are-appropriate-response-capabilities-deployed-to-respond-to-ale-mdx"></a>

# Does AssemblyAI utilize an anti-virus/anti-malware solution across all relevant infrastructure (workstations and servers), and are appropriate response capabilities deployed to respond to alerts?

URL: https://www.assemblyai.com/docs/faq/does-assemblyai-utilize-an-anti-virus-anti-malware-solution-across-all-relevant-infrastructure-workstations-and-servers-and-are-appropriate-response-capabilities-deployed-to-respond-to-ale
Source: docs/faq/does-assemblyai-utilize-an-anti-virus-anti-malware-solution-across-all-relevant-infrastructure-workstations-and-servers-and-are-appropriate-response-capabilities-deployed-to-respond-to-ale.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Does AssemblyAI utilize an anti-virus/anti-malware solution across all relevant infrastructure (workstations and servers), and are appropriate response capabilities deployed to respond to alerts? documentation.

Yes. We have antimalware installed on corporate endpoints and ingress points within our production environment. We monitor for suspicious activity and have data loss prevention (DLP) alerts set up so that we can respond as quickly as possible.

---

<a id="how-are-incidents-escalated-within-your-organization-docs-faq-how-are-incidents-escalated-within-your-organization-mdx"></a>

# How are incidents escalated within your organization?

URL: https://www.assemblyai.com/docs/faq/how-are-incidents-escalated-within-your-organization
Source: docs/faq/how-are-incidents-escalated-within-your-organization.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: How are incidents escalated within your organization? documentation.

Incidents are reported via email and escalated based on severity, using Firehydrant, our incident response tool. Firehydrant integrates multiple tools including ticketing and drives while logging actions, checkpoints, and other activities handled during an incident.

---

<a id="how-do-we-securely-use-your-service-docs-faq-how-do-we-securely-use-your-service-mdx"></a>

# How do we securely use your service?

URL: https://www.assemblyai.com/docs/faq/how-do-we-securely-use-your-service
Source: docs/faq/how-do-we-securely-use-your-service.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: How do we securely use your service? documentation.

Our service is reached over TLS by connecting to our API hostnames, such as `api.assemblyai.com` for REST, `streaming.assemblyai.com` for streaming, and `llm-gateway.assemblyai.com` for LLM Gateway (with `*.eu.assemblyai.com` equivalents for the EU region). All connections use HTTPS or WSS (WebSocket over TLS) so that traffic is encrypted in transit.

If your network requires allowlisting fixed IP addresses for inbound webhook deliveries or audio retrieval requests, see [What IP Address Should I Whitelist for AssemblyAI?](/faq/what-ip-address-should-i-whitelist-for-assemblyai).

Outputs are text payloads, so conducting file scanning on responses at the corporate firewall can be useful in maintaining a strong security posture while using our services.

---

<a id="how-do-you-protect-production-code-docs-faq-how-do-you-protect-production-code-mdx"></a>

# How do you protect production code?

URL: https://www.assemblyai.com/docs/faq/how-do-you-protect-production-code
Source: docs/faq/how-do-you-protect-production-code.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: How do you protect production code? documentation.

We have a peer review (PR) process in place. The PR process requires code to be reviewed by a senior reviewer or code owner before being merged into production. We also use GitHub Dependabot to ensure we are not errantly introducing insecure libraries to our development repositories.

---

<a id="how-to-access-assemblyai-s-security-reports-docs-faq-how-to-access-assemblyai-s-security-reports-mdx"></a>

# How to Access AssemblyAI's Security Reports

URL: https://www.assemblyai.com/docs/faq/how-to-access-assemblyai-s-security-reports
Source: docs/faq/how-to-access-assemblyai-s-security-reports.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: How to Access AssemblyAI's Security Reports documentation.

AssemblyAI provides easy access to our security reports, including our SOC 2 report, through our dedicated security center. This article will guide you on how to obtain these important documents.

## Accessing the Security Center

To access AssemblyAI's security reports:

1. Visit our [AssemblyAI Trust Center](https://app.vanta.com/assemblyai/trust/7n80syl8zln1bn1qm3x8eg)

2. Navigate through the available reports, which may include:

   - SOC 2 Report
   - Other relevant security and compliance documents

3. Download the reports you need directly from this portal

## Unable to Access the Security Center?

If you encounter any issues accessing the security center or downloading the reports, please contact our support team. They will be happy to assist you in obtaining the necessary security documentation.

  Our security reports are regularly updated. Always ensure you're referencing
  the most recent version available in our Trust Portal.

---

<a id="how-to-opt-out-of-data-sharing-for-our-model-improvement-program-docs-faq-how-to-opt-out-of-data-sharing-for-our-model-improvement-program-mdx"></a>

# How to Opt Out of Data Sharing for our Model Improvement Program

URL: https://www.assemblyai.com/docs/faq/how-to-opt-out-of-data-sharing-for-our-model-improvement-program
Source: docs/faq/how-to-opt-out-of-data-sharing-for-our-model-improvement-program.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: How to Opt Out of Data Sharing for our Model Improvement Program documentation.

AssemblyAI provides an option for customers to opt out of AssemblyAI using your data to train our artificial intelligence and machine learning models. If you opt out of data sharing for our model improvement program (to the extent applicable to your pricing plan), at this time:

- AssemblyAI will not use your Customer Data to train its artificial intelligence and machine learning models, as set forth in Section 4.3(b)(iii) of the [Terms of Service](https://www.assemblyai.com/legal/terms-of-service).
- AssemblyAI will not use your Customer Data to perform benchmarking, as set forth in Section 4.3(b)(ii) of the [Terms of Service](https://www.assemblyai.com/legal/terms-of-service).
- AssemblyAI will not use your Deidentified Data to train its artificial intelligence and machine learning models, as set forth in Section 4.6 of the [Terms of Service](https://www.assemblyai.com/legal/terms-of-service).

If you wish to opt out of data sharing for our model improvement program, you can do so self-serve from the AssemblyAI dashboard.

## Opting out from the dashboard

Paid customers can opt out self-serve, at no additional cost, from the [**Data Controls** page](https://www.assemblyai.com/dashboard/settings/data-controls) in the AssemblyAI dashboard. Toggle **Opt Out of Data Sharing for Model Improvement Program** to update your status. Only members with the **Owner** or **Admin** role can change this setting. For more detail, see [Data Controls](/data-controls#opt-out-of-the-model-improvement-program).

  If you have signed a BAA, you are automatically opted out upon signature and the opt-out toggle can no longer be changed.

  Free users do not have the ability to opt out of the model improvement program. If
  you would like to move to a paid plan to access opt-out functionality, please
  upgrade [here](https://www.assemblyai.com/dashboard/pricing).

## Important Notes

For further information on how we train our models, please see [our FAQ on model training](/data-retention-and-model-training).

Opt-out changes are forward-looking only and apply to subsequent new requests. At this time, we do not have the ability to retroactively apply opt-out requests.

If you have any questions about this process, you can contact our support team for assistance via live chat or email at [https://www.assemblyai.com/contact/support](https://www.assemblyai.com/contact/support).

---

<a id="is-multi-factor-authentication-enforced-for-all-access-to-scoped-systems-and-data-docs-faq-is-multi-factor-authentication-enforced-for-all-access-to-scoped-systems-and-data-mdx"></a>

# Is multi-factor authentication enforced for all access to scoped systems and data?

URL: https://www.assemblyai.com/docs/faq/is-multi-factor-authentication-enforced-for-all-access-to-scoped-systems-and-data
Source: docs/faq/is-multi-factor-authentication-enforced-for-all-access-to-scoped-systems-and-data.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Is multi-factor authentication enforced for all access to scoped systems and data? documentation.

Multifactor authentication is enforced on internal systems that interface with customer systems and data.

MFA is available for all customers using our dashboard.

---

<a id="does-assemblyai-have-a-documented-process-for-reviewing-and-approving-third-party-service-providers-docs-faq-is-there-a-documented-process-for-reviewing-and-approving-third-party-service-providers-mdx"></a>

# Does AssemblyAI have a documented process for reviewing and approving third-party service providers?

URL: https://www.assemblyai.com/docs/faq/is-there-a-documented-process-for-reviewing-and-approving-third-party-service-providers
Source: docs/faq/is-there-a-documented-process-for-reviewing-and-approving-third-party-service-providers.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Does AssemblyAI have a documented process for reviewing and approving third-party service providers? documentation.

Yes. We have a documented Access Control Policy. We conduct quarterly access reviews and security reviews on our vendors ensuring that they maintain their security posture. We do this by having cadences with our vendor contacts and documenting risk scores based on what they process, and controls they have in place. Considerations are their SOC, ISO or other attestation reports are considered and attached to the vendor profile.

---

<a id="does-assemblyai-have-an-incident-response-plan-docs-faq-please-describe-the-incident-response-plan-mdx"></a>

# Does AssemblyAI have an incident response plan?

URL: https://www.assemblyai.com/docs/faq/please-describe-the-incident-response-plan
Source: docs/faq/please-describe-the-incident-response-plan.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Does AssemblyAI have an incident response plan? documentation.

AssemblyAI has an Incident Response Plan and runbooks for managing incidents effectively. Runbooks exist for incidents that have been practiced, or have occurred before. If required, please contact [support@assemblyai.com](mailto:support@assemblyai.com) for generic samples.

---

<a id="what-are-your-recovery-time-and-recovery-point-objectives-docs-faq-what-are-your-recovery-time-and-recovery-point-objectives-mdx"></a>

# What are your recovery time and recovery point objectives?

URL: https://www.assemblyai.com/docs/faq/what-are-your-recovery-time-and-recovery-point-objectives
Source: docs/faq/what-are-your-recovery-time-and-recovery-point-objectives.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: What are your recovery time and recovery point objectives? documentation.

After conducting a business impact analysis, we have determined our RTO is ≤4.5 days and our RPO ≤7 days. We aim to reassess these targets annually.

---

<a id="what-is-your-sla-for-repairing-critical-high-medium-vulnerabilities-docs-faq-what-is-your-sla-for-repairing-critical-high-medium-vulnerabilities-mdx"></a>

# What is your SLA for repairing Critical/High/Medium vulnerabilities?

URL: https://www.assemblyai.com/docs/faq/what-is-your-sla-for-repairing-critical-high-medium-vulnerabilities
Source: docs/faq/what-is-your-sla-for-repairing-critical-high-medium-vulnerabilities.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: What is your SLA for repairing Critical/High/Medium vulnerabilities? documentation.

Critical and High vulnerabilities are remediated within 30 days, Medium within 60 days.

We are exploring CNAPP and base image hardening tools to speed up our time to remediation on all vulnerabilities with the goal of addressing critical vulnerabilities in 10 days.

---

<a id="what-logs-are-available-to-customers-docs-faq-what-logs-are-available-to-customers-mdx"></a>

# What logs are available to customers?

URL: https://www.assemblyai.com/docs/faq/what-logs-are-available-to-customers
Source: docs/faq/what-logs-are-available-to-customers.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: What logs are available to customers? documentation.

Available logs include system, application, network, and audit logs for access, download, and deletion.

The following logs can be provided to Enterprise customers upon request:

- Header Keys
- Customer header keys included on API request
- Customer request headers included on API request
- Host information
- Customer specified API endpoint (based on model and region)
- Customer specified API endpoint method
- Customer specific API request full path

---

<a id="what-standards-do-your-internal-password-policies-follow-docs-faq-what-standards-do-your-internal-password-policies-follow-mdx"></a>

# What standards do your internal password policies follow?

URL: https://www.assemblyai.com/docs/faq/what-standards-do-your-internal-password-policies-follow
Source: docs/faq/what-standards-do-your-internal-password-policies-follow.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: What standards do your internal password policies follow? documentation.

We use a password manager that defaults passwords to 14 characters with uppercase, lowercase, numbers and special characters.

To access our dashboard and sign into accounts, users of our services must have 8 characters, uppercase, lowercase, numbers and special characters matching NIST's requirements.

---

<a id="where-are-your-servers-located-docs-faq-where-are-your-servers-located-mdx"></a>

# Where are your servers located?

URL: https://www.assemblyai.com/docs/faq/where-are-your-servers-located
Source: docs/faq/where-are-your-servers-located.mdx
Navigation: FAQ > Overview > Privacy & Security
Description: Where are your servers located? documentation.

Our production server resides in AWS US West 2, AWS EU West 1. These are AWS Oregon and Dublin regions respectively.

---

<a id="am-i-charged-for-transcribing-silent-audio-docs-faq-am-i-charged-for-transcribing-silent-audio-mdx"></a>

# Am I charged for transcribing silent audio?

URL: https://www.assemblyai.com/docs/faq/am-i-charged-for-transcribing-silent-audio
Source: docs/faq/am-i-charged-for-transcribing-silent-audio.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Am I charged for transcribing silent audio? documentation.

Yes, our billing is based on audio duration, not the amount of speech content. However, you can use the `speech_threshold` parameter to avoid processing files with minimal speech content. This parameter lets you specify a minimum percentage of speech required for transcription.

[Learn more about speech threshold](/guardrails/set-minimum-speech-threshold)

---

<a id="are-custom-models-more-accurate-than-general-models-docs-faq-are-custom-models-more-accurate-than-general-models-mdx"></a>

# Are Custom Models More Accurate than General Models?

URL: https://www.assemblyai.com/docs/faq/are-custom-models-more-accurate-than-general-models
Source: docs/faq/are-custom-models-more-accurate-than-general-models.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Are Custom Models More Accurate than General Models? documentation.

In the field of ASR, custom models are rarely more accurate than the best general models (learn more about one measure of accuracy, [Word Error Rate or WER](https://www.assemblyai.com/blog/word-error-rate/?utm_source=google&utm_medium=cpc&utm_campaign=brand), here). This is because general models are trained on huge datasets, and are constantly maintained and updated using the latest deep learning research.

For example, at AssemblyAI, we train large deep neural networks on over 12.5 million hours of speech data. This training data is a mix of many different types of audio (broadcast TV recordings, phone calls, Zoom meetings, videos, etc), accents, and speakers. This massive amount of diverse training data helps our ASR models to generalize extremely well across all types of audio/data, speakers, recording quality, and accents when converting Speech-to-Text in the real world.

Custom models usually come into the mix when dealing with audio data that have unique characteristics unseen by a general model. However, because large, accurate general models see most types of audio data during training, there are not many "unique characteristics" that would trip up a general model - or that a custom model would even be able to learn.

To learn more about this topic, see [this blog post](https://www.assemblyai.com/blog/do-i-need-a-custom-speech-recognition-model/).

---

<a id="do-i-get-charged-for-failed-api-calls-docs-faq-are-customers-charged-for-api-calls-that-result-in-errors-mdx"></a>

# Do I Get Charged for Failed API Calls?

URL: https://www.assemblyai.com/docs/faq/are-customers-charged-for-api-calls-that-result-in-errors
Source: docs/faq/are-customers-charged-for-api-calls-that-result-in-errors.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Do I Get Charged for Failed API Calls? documentation.

You won't be charged for failed API calls. If your transcript response returns an "error" status, the request won't count against your credits.

You can review any failed transcripts in your [Async Transcription Processing Queue](https://www.assemblyai.com/dashboard/home).

  <img src="/assets/img/faq/error2.jpg" alt="Failed transcripts shown in the dashboard Async Transcription Processing Queue" />

---

<a id="are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api-docs-faq-are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api-mdx"></a>

# Are there any limits on file size or file duration for files submitted to the API?

URL: https://www.assemblyai.com/docs/faq/are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api
Source: docs/faq/are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Are there any limits on file size or file duration for files submitted to the API? documentation.

There is both a maximum file size and a maximum audio duration for files that can be submitted to our API for transcription. The maximum file size that can be submitted to the `/v2/transcript` endpoint for transcription is 5GB, and the maximum duration is 10 hours.

There is also a maximum file size for a local file uploaded to the API via the `/v2/upload` endpoint, which is 2.2GB.

---

<a id="can-i-customize-how-words-are-spelled-by-the-model-docs-faq-can-i-customize-how-words-are-spelled-by-the-model-mdx"></a>

# Can I customize how words are spelled by the model?

URL: https://www.assemblyai.com/docs/faq/can-i-customize-how-words-are-spelled-by-the-model
Source: docs/faq/can-i-customize-how-words-are-spelled-by-the-model.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Can I customize how words are spelled by the model? documentation.

Yes. The Custom Spelling feature gives you the ability to specify how words are spelled or formatted in the transcript text. For example, Custom Spelling could be used to change the spelling of all instances of the word "Ariana" to "Arianna". It could also be used to change the formatting of "CS 50" to "CS50".

For more information, check our docs on [Custom Spelling](/pre-recorded-audio/correct-spelling-of-terms).

---

<a id="can-i-delete-the-transcripts-i-have-created-using-the-api-docs-faq-can-i-delete-the-transcripts-i-have-created-using-the-api-mdx"></a>

# Can I delete the transcripts I have created using the API?

URL: https://www.assemblyai.com/docs/faq/can-i-delete-the-transcripts-i-have-created-using-the-api
Source: docs/faq/can-i-delete-the-transcripts-i-have-created-using-the-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Can I delete the transcripts I have created using the API? documentation.

You can! Completed transcripts are stored for **30 days by default** and are automatically deleted after that. You can also delete them earlier at any time using the `DELETE` endpoint covered on [this page](/pre-recorded-audio/delete-transcripts) of our documentation, or shorten the retention window by setting a [time-to-live (TTL)](/data-controls#set-a-time-to-live-for-audio-and-transcripts) on the Data Controls page in the dashboard.

If you are looking to delete transcriptions created via our Playground please reach out to our Support team at [support@assemblyai.com](mailto:support@assemblyai.com).

---

<a id="can-i-get-a-list-of-all-transcripts-i-have-created-docs-faq-can-i-get-a-list-of-all-transcripts-i-have-created-mdx"></a>

# Can I get a list of all transcripts I have created?

URL: https://www.assemblyai.com/docs/faq/can-i-get-a-list-of-all-transcripts-i-have-created
Source: docs/faq/can-i-get-a-list-of-all-transcripts-i-have-created.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Can I get a list of all transcripts I have created? documentation.

You can retrieve a list of all transcripts that you have created by [making a `GET` request to the API](/api-reference/transcripts/list).

Note historical transcripts can only be retrieved for the last 90 days via this endpoint. If you need transcript information from farther back please reach out to our Support team at support@assemblyai.com.

---

<a id="can-i-send-audio-to-assemblyai-in-segments-and-still-get-speaker-labels-for-the-whole-recording-docs-faq-can-i-send-audio-to-assemblyai-in-segments-and-still-get-speaker-labels-for-the-whole-recording-mdx"></a>

# Can I send audio to AssemblyAI in segments and still get speaker labels for the whole recording?

URL: https://www.assemblyai.com/docs/faq/can-i-send-audio-to-assemblyai-in-segments-and-still-get-speaker-labels-for-the-whole-recording
Source: docs/faq/can-i-send-audio-to-assemblyai-in-segments-and-still-get-speaker-labels-for-the-whole-recording.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Can I send audio to AssemblyAI in segments and still get speaker labels for the whole recording? documentation.

While AssemblyAI doesn't provide a built-in way to send audio in segments and receive speaker labels for the entire recording, you can achieve this functionality using TitaNet. We have a detailed cookbook that demonstrates this workflow.

## Using TitaNet for Segmented Audio Processing

To process audio in segments while maintaining speaker labels across the entire recording:

1. Use TitaNet to segment your audio
2. Send these segments to AssemblyAI for transcription
3. Combine the results to get speaker labels for the whole recording

## Step-by-Step Guide

For a detailed walkthrough of this process, including code examples, please refer to our cookbook:

[Speaker Diarization with Async Chunking](/pre-recorded-audio/guides/speaker-diarization-with-async-chunking)

This cookbook provides a comprehensive guide on how to implement speaker diarization with asynchronous chunking, allowing you to process large audio files in segments while maintaining accurate speaker labels across the entire recording.

---

<a id="can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive-docs-faq-can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive-mdx"></a>

# Can I submit files to the API that are stored in a Google Drive?

URL: https://www.assemblyai.com/docs/faq/can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive
Source: docs/faq/can-i-submit-files-to-the-api-that-are-stored-in-a-google-drive.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Can I submit files to the API that are stored in a Google Drive? documentation.

Yes, you can but the file needs to be shared publicly, smaller than 100MB, and the URL needs to be a downloadable link, which should look something like this:

```
https://drive.google.com/u/0/uc?id=1OMjFJ1VUX0ES5tMSdslybvPLxYTjdqLZ&export=download
```

See [this guide](/pre-recorded-audio/guides/transcribing-google-drive-file) from our AssemblyAI Cookbooks for a detailed walkthrough of how to submit files from Google Drive.

---

<a id="can-i-use-the-api-without-internet-access-docs-faq-can-i-use-the-api-without-internet-access-mdx"></a>

# Can I use the API without internet access?

URL: https://www.assemblyai.com/docs/faq/can-i-use-the-api-without-internet-access
Source: docs/faq/can-i-use-the-api-without-internet-access.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Can I use the API without internet access? documentation.

No - you won't be able to reach AssemblyAI's API without an internet connection. A fast and reliable internet connection is especially recommended when using our upload endpoint or Streaming services.

---

<a id="do-we-have-resources-for-building-with-make-docs-faq-do-we-have-resources-for-building-with-make-mdx"></a>

# Do we have resources for building with Make?

URL: https://www.assemblyai.com/docs/faq/do-we-have-resources-for-building-with-make-
Source: docs/faq/do-we-have-resources-for-building-with-make-.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Do we have resources for building with Make? documentation.

Yes, we have a Make integration which you can find on the [AssemblyAI page in the Make integrations directory](https://www.make.com/en/integrations/assembly-ai).

Make is a no-code platform that enables users to build their own workflows through their drag-and-drop features. Users can automate tasks involving voice and text data processing using AssemblyAI on the Make platform without needing to invest in custom software development.

You can also build an application that connects to various other platforms such as Gmail, Slack, Stripe, etc. For example, an audio file is transcribed by AssemblyAI, and the transcription results are then uploaded to a Google Drive.

The following modules are supported:

- Upload a File
- Transcripts
- Transcribe an Audio File (submit or poll)
- Wait until Transcript is Ready (explicitly poll after submit)
- Watch for Transcript Ready Notification (this is the transcript webhook)
- Get a Transcript
- Get Paragraphs of a Transcript
- Get Sentences of a Transcript
- Get Subtitles for a Transcript
- Get Redacted Audio of a Transcript (including file download)
- Search for Words in a Transcript
- List Transcripts
- Delete a Transcript
- Other modules
- Make an API Call

---

<a id="do-you-have-any-examples-for-how-to-use-your-api-docs-faq-do-you-have-any-examples-for-how-to-use-your-api-mdx"></a>

# Do you have any examples for how to use your API?

URL: https://www.assemblyai.com/docs/faq/do-you-have-any-examples-for-how-to-use-your-api-
Source: docs/faq/do-you-have-any-examples-for-how-to-use-your-api-.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Do you have any examples for how to use your API? documentation.

We have a plethora of resources to help users implement and try out our API! Here are a few resources that could help you.

## Documentation

Our documentation provides info on our various features and models, as well as guides on how to use our API that include our code examples:

[https://www.assemblyai.com/docs/](/)

## Cookbook

We have use-case-specific cookbooks that may apply to what you're looking to build:

[https://www.assemblyai.com/docs/getting-started/end-to-end-examples](/getting-started/end-to-end-examples)

## YouTube

We have an active YouTube channel with more tutorials and AI Content. If you prefer video guides, this is the place for you! You can follow our YouTube channel here:

[AssemblyAI on YouTube](https://www.youtube.com/@AssemblyAI)

## AssemblyAI Blog

The AssemblyAI Blog covers product news, company news, the latest AI research, and AssemblyAI tutorials. The tutorials often cover specific integrations and ideas to implement with our API.

[AssemblyAI Blog](https://www.assemblyai.com/blog/)

For any questions about any of these resources, please reach out to us via email [support@assemblyai.com](mailto:support@assemblyai.com) with a detailed summary of how we can help!

---

<a id="do-you-have-example-use-cases-for-using-assemblyai-docs-faq-do-you-have-example-use-cases-for-using-assemblyai-mdx"></a>

# Do you have example use cases for using AssemblyAI?

URL: https://www.assemblyai.com/docs/faq/do-you-have-example-use-cases-for-using-assemblyai-
Source: docs/faq/do-you-have-example-use-cases-for-using-assemblyai-.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Do you have example use cases for using AssemblyAI? documentation.

Yes! You can find the AssemblyAI Cookbooks here:

[https://www.assemblyai.com/docs/getting-started/end-to-end-examples](/getting-started/end-to-end-examples)

AssemblyAI Cookbooks are a resource of code examples, guides, and tutorials for using the AssemblyAI API. They are intended to demonstrate and provide an accompanying code example for use cases using our API. Our team is constantly updating this resource based on our user feedback and feature releases.

Please reach out if you have any ideas or would like to contribute to the code examples!

---

<a id="do-you-offer-cross-file-speaker-identification-docs-faq-do-you-offer-cross-file-speaker-identification-mdx"></a>

# Do you offer cross-file Speaker Identification?

URL: https://www.assemblyai.com/docs/faq/do-you-offer-cross-file-speaker-identification
Source: docs/faq/do-you-offer-cross-file-speaker-identification.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Do you offer cross-file Speaker Identification? documentation.

Yes! Speaker Identification allows you to identify speakers by their actual names or roles, transforming generic labels like `"A", "B", "C", ...` into meaningful identifiers that you provide. Speaker identities are inferred based on the conversation content. For more information, see [Speaker Identification](/speech-understanding/speaker-identification).

For speaker identification across multiple recordings, you can implement speaker identification using audio embeddings. For this you would first submit your audio file to AssemblyAI for diarization with speaker labels, and then use a model like [Nvidia Titanet](https://huggingface.co/nvidia/speakerverification_en_titanet_large) to generate speaker embeddings from the audio. Then, you would match these embeddings against a vector database of known speakers before replacing our generic labels ("Speaker A/B") with actual names. Refer to our [speaker identification cookbook](/pre-recorded-audio/guides/titanet-speaker-identification) for more details.

---

<a id="do-you-offer-translation-docs-faq-do-you-offer-translation-mdx"></a>

# Do you offer translation?

URL: https://www.assemblyai.com/docs/faq/do-you-offer-translation
Source: docs/faq/do-you-offer-translation.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Do you offer translation? documentation.

We do! Translation is one of our Speech Understanding features, with support for 99 languages. See our [translation guide](/pre-recorded-audio/guides/translate_transcripts) for more information.

---

<a id="do-you-offer-voice-to-voice-or-text-to-speech-tts-docs-faq-do-you-offer-voice-to-voice-or-text-to-speech-tts-mdx"></a>

# Do you offer voice-to-voice or text-to-speech (TTS)?

URL: https://www.assemblyai.com/docs/faq/do-you-offer-voice-to-voice-or-text-to-speech-tts
Source: docs/faq/do-you-offer-voice-to-voice-or-text-to-speech-tts.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Do you offer voice-to-voice or text-to-speech (TTS)? documentation.

Yes! The [Voice Agent API](/voice-agents/voice-agent-api) provides a complete voice-to-voice pipeline through a single WebSocket connection. It combines AssemblyAI's speech-to-text, LLM reasoning, and text-to-speech into one integrated service — you stream audio in and receive spoken audio back in real time.

The Voice Agent API is billed at a single all-in rate of $4.50/hr covering STT, LLM reasoning, and TTS. See the [Voice Agent API documentation](/voice-agents/voice-agent-api) to get started.

  AssemblyAI does not offer standalone text-to-speech as a separate service. TTS is available as part of the Voice Agent API pipeline.

---

<a id="does-it-cost-extra-to-export-srt-or-vtt-captions-docs-faq-does-it-cost-extra-to-export-srt-or-vtt-captions-mdx"></a>

# Does it cost extra to export SRT or VTT captions?

URL: https://www.assemblyai.com/docs/faq/does-it-cost-extra-to-export-srt-or-vtt-captions
Source: docs/faq/does-it-cost-extra-to-export-srt-or-vtt-captions.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Does it cost extra to export SRT or VTT captions? documentation.

It does not!

Customers are not charged an additional amount for exporting `SRT` or `VTT` files. Once a transcription job for a specific transcript ID is completed, the customer will be charged based on the audio duration of the file and the features/models included in that `POST` request. Customers are not charged for `GET` requests including those to export `SRT` or `VTT` captions.

---

<a id="is-there-a-way-to-generate-srt-or-vtt-captions-with-speaker-labels-docs-faq-is-there-a-way-to-generate-srt-or-vtt-captions-with-speaker-labels-mdx"></a>

# Is there a way to generate SRT or VTT captions with speaker labels?

URL: https://www.assemblyai.com/docs/faq/is-there-a-way-to-generate-srt-or-vtt-captions-with-speaker-labels
Source: docs/faq/is-there-a-way-to-generate-srt-or-vtt-captions-with-speaker-labels.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Is there a way to generate SRT or VTT captions with speaker labels? documentation.

The [export captions endpoint](/api-reference/transcripts/get-subtitles) does not include speaker label information by default, but you can generate captions with speaker labels using the information from the JSON response for the completed transcript.

When you enable [Speaker Diarization](/pre-recorded-audio/label-speakers) in your transcription request, the response includes speaker information for each word and utterance. You can use this data along with the word-level timestamps to build custom SRT or VTT files that include speaker identification.

See the [Create Subtitles with Speaker Labels](/pre-recorded-audio/guides/speaker_labelled_subtitles) cookbook for a step-by-step guide on how to accomplish this.

---

<a id="does-it-cost-more-to-transcribe-an-audio-or-video-docs-faq-does-it-cost-more-to-transcribe-an-audio-or-video-mdx"></a>

# Does it cost more to transcribe an audio or video?

URL: https://www.assemblyai.com/docs/faq/does-it-cost-more-to-transcribe-an-audio-or-video
Source: docs/faq/does-it-cost-more-to-transcribe-an-audio-or-video.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Does it cost more to transcribe an audio or video? documentation.

Our pricing is based solely on duration. Audio and video files are charged at the same rate - you only pay for the length of the content, not the file type.

---

<a id="does-your-api-return-timestamps-for-individual-words-docs-faq-does-your-api-return-timestamps-for-individual-words-mdx"></a>

# Does your API return timestamps for individual words?

URL: https://www.assemblyai.com/docs/faq/does-your-api-return-timestamps-for-individual-words
Source: docs/faq/does-your-api-return-timestamps-for-individual-words.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Does your API return timestamps for individual words? documentation.

Yes! The response for a completed request includes `start` and `end` keys. These keys are timestamp values for when a given word, phrase, or sentence starts and ends. These values are in milliseconds and are accurate to within about 400 milliseconds.

To convert these timestamps from milliseconds to seconds, divide the timestamp value by 1000.

See [this section](/api-reference/transcripts/get) of our API reference for an example of the JSON response for a completed transcript, which includes these `start` and `end` keys.

---

<a id="how-are-individual-speakers-identified-and-how-does-the-speaker-label-feature-work-docs-faq-how-are-individual-speakers-identified-and-how-does-the-speaker-label-feature-work-mdx"></a>

# How are individual speakers identified and how does the Speaker Label feature work?

URL: https://www.assemblyai.com/docs/faq/how-are-individual-speakers-identified-and-how-does-the-speaker-label-feature-work
Source: docs/faq/how-are-individual-speakers-identified-and-how-does-the-speaker-label-feature-work.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How are individual speakers identified and how does the Speaker Label feature work? documentation.

When an audio file is submitted with `speaker_labels` set to `true` word timings are used to cut the audio into separate chunks of words. Those chunks are fed into a model to build a "speaker embedding", which is a representation of a speaker. An algorithm is then used to cluster speaker embeddings that are similar to each other. If there are two speakers, there will be two distinct clusters of speaker embeddings. Those clusters are then used to assign speakers.

When the model is determining speakers, the entire audio is taken into consideration. It takes a certain amount of audio for a person to be identified as a unique speaker (we typically say 30 seconds). If a person doesn't speak much over the course of a file or they reply in short phrases like "okay" and "right" they may not get identified as a unique speaker. If they aren't identified as a unique speaker, their words will be attributed to the speaker embedding the model feels is most similar.

For more information on Speaker Labels, see [this blog post](https://www.assemblyai.com/blog/top-speaker-diarization-libraries-and-apis).

---

<a id="how-are-paragraphs-created-for-the-paragraphs-endpoint-docs-faq-how-are-paragraphs-created-for-the-paragraphs-endpoint-mdx"></a>

# How are paragraphs created for the /paragraphs endpoint?

URL: https://www.assemblyai.com/docs/faq/how-are-paragraphs-created-for-the-paragraphs-endpoint
Source: docs/faq/how-are-paragraphs-created-for-the-paragraphs-endpoint.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How are paragraphs created for the /paragraphs endpoint? documentation.

With our `/v2/transcript/{TRANSCRIPT-ID}/paragraphs` endpoint you can export a completed transcript broken into paragraphs. These paragraphs are created using an algorithm that looks at certain heuristics like speaking time, text length, pauses, etc.

See [this section](/pre-recorded-audio/export-transcripts-as-srt-vtt-or-text) of our documentation to learn more about exporting paragraphs.

---

<a id="how-are-word-transcript-level-confidence-scores-calculated-docs-faq-how-are-word-transcript-level-confidence-scores-calculated-mdx"></a>

# How are word/transcript level confidence scores calculated?

URL: https://www.assemblyai.com/docs/faq/how-are-word-transcript-level-confidence-scores-calculated
Source: docs/faq/how-are-word-transcript-level-confidence-scores-calculated.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How are word/transcript level confidence scores calculated? documentation.

The confidence scores are derived from the combination of the [softmax layer](https://en.wikipedia.org/wiki/Softmax_function) from the acoustic model and language model probabilities. The models produce probabilities for each predicted word, and we use those probabilities as confidence scores. The transcript level confidence score is calculated by taking the average of all word-level confidence scores.

Confidence scores fall between `0.0` (low confidence) and `1.0` (high confidence).

---

<a id="how-can-i-integrate-assemblyai-with-other-services-docs-faq-how-can-i-integrate-assemblyai-with-other-services-mdx"></a>

# How can I integrate AssemblyAI with other services?

URL: https://www.assemblyai.com/docs/faq/how-can-i-integrate-assemblyai-with-other-services-
Source: docs/faq/how-can-i-integrate-assemblyai-with-other-services-.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How can I integrate AssemblyAI with other services? documentation.

We have existing integrations with services such as Zapier, Make, AWS, and more! You can find our existing integrations with other tools and services in our [integrations documentation](/integrations).

Using AssemblyAI API involves making GET and POST requests so as long as you can make those you can use our API with any language, framework, or library.

---

<a id="how-can-i-make-certain-words-more-likely-to-be-transcribed-docs-faq-how-can-i-make-certain-words-more-likely-to-be-transcribed-mdx"></a>

# How can I make certain words more likely to be transcribed?

URL: https://www.assemblyai.com/docs/faq/how-can-i-make-certain-words-more-likely-to-be-transcribed
Source: docs/faq/how-can-i-make-certain-words-more-likely-to-be-transcribed.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How can I make certain words more likely to be transcribed? documentation.

Our speech recognition models support a feature called **keyterms prompting**, which lets you provide a list of words or phrases that you want the model to be more likely to transcribe accurately. How this feature works varies slightly by model, so see these specific sections of our documentation for more details:

- [Pre-recorded (Universal-3.5 Pro and Universal-2)](/pre-recorded-audio/universal-3-5-pro/prompting#keyterms-prompting)
- [Streaming (Universal-3.5 Pro Streaming and Universal-Streaming)](/streaming/prompting-and-keyterms)

---

<a id="how-can-i-test-assemblyai-without-writing-code-docs-faq-how-can-i-test-assemblyai-without-writing-code-mdx"></a>

# How can I test AssemblyAI without writing code?

URL: https://www.assemblyai.com/docs/faq/how-can-i-test-assemblyai-without-writing-code-
Source: docs/faq/how-can-i-test-assemblyai-without-writing-code-.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How can I test AssemblyAI without writing code? documentation.

Start exploring our AI capabilities instantly using the [AssemblyAI Playground](https://www.assemblyai.com/playground). Our Playground allows you to transcribe pre-recorded and streaming audio, extract insights from your audio using our various Speech Understanding features, and connect to multiple LLM providers using our LLM Gateway feature.

---

<a id="how-can-i-transcribe-youtube-videos-docs-faq-how-can-i-transcribe-youtube-videos-mdx"></a>

# How can I transcribe YouTube videos?

URL: https://www.assemblyai.com/docs/faq/how-can-i-transcribe-youtube-videos
Source: docs/faq/how-can-i-transcribe-youtube-videos.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How can I transcribe YouTube videos? documentation.

YouTube URLs are not supported in the `audio_url` parameter since it requires a direct link to a downloadable audio file. To transcribe a YouTube video:

1. Download the video's audio
2. Upload the audio file to our API

See [How to transcribe YouTube videos cookbook](/pre-recorded-audio/guides/transcribe_youtube_videos) (Python) for more information.

---

<a id="how-do-i-generate-subtitles-docs-faq-how-do-i-generate-subtitles-mdx"></a>

# How do I generate subtitles?

URL: https://www.assemblyai.com/docs/faq/how-do-i-generate-subtitles
Source: docs/faq/how-do-i-generate-subtitles.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How do I generate subtitles? documentation.

AssemblyAI's API supports exporting transcripts as SRT or VTT files, which are standard subtitle formats used for video captioning. You can learn how to generate subtitles using:

- [Generate Subtitles for Videos](/pre-recorded-audio/guides/subtitles)

---

<a id="how-does-assemblyai-compare-to-other-asr-providers-docs-faq-how-does-assemblyai-compare-to-other-asr-providers-mdx"></a>

# How does AssemblyAI compare to other ASR providers?

URL: https://www.assemblyai.com/docs/faq/how-does-assemblyai-compare-to-other-asr-providers
Source: docs/faq/how-does-assemblyai-compare-to-other-asr-providers.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How does AssemblyAI compare to other ASR providers? documentation.

Our state-of-the-art speech recognition models are trained on extremely large amounts of data to achieve near human-level performance and robustness across a variety of data. As a result, our models typically perform as good or better than the competition. See [our latest benchmark data](https://www.assemblyai.com/benchmarks) to see how we compare to other ASR providers (including Amazon, Microsoft, OpenAI, and more) on a wide range of file types.

---

<a id="how-does-automatic-language-detection-work-docs-faq-how-does-language-detection-work-for-transcriptions-mdx"></a>

# How does Automatic Language Detection work?

URL: https://www.assemblyai.com/docs/faq/how-does-language-detection-work-for-transcriptions
Source: docs/faq/how-does-language-detection-work-for-transcriptions.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How does Automatic Language Detection work? documentation.

Our Automatic Language Detection (ALD) model analyzes samples of the audio to determine the language spoken. It randomly selects up to 3 clips of 30 seconds each from the middle 50% of the audio duration (between 25% and 75% of the total length).

These 3 clips are passed through our ALD model, which predicts the language probabilities for each clip. The probabilities are then averaged across the clips, and the languages are sorted by their average probability scores.

This approach helps ensure that the language detection is based on a representative sample of the audio, rather than just the beginning or end portions which may contain greetings, silence, or other non-representative speech.

If you are seeing low confidence scores for transcriptions in a particular language, it may be due to factors like background noise, accents, or audio quality issues.

---

<a id="how-does-the-api-handle-files-that-contain-spoken-audio-in-multiple-languages-docs-faq-how-does-the-api-handle-files-that-contain-spoken-audio-in-multiple-languages-mdx"></a>

# How does the API handle files that contain spoken audio in multiple languages?

URL: https://www.assemblyai.com/docs/faq/how-does-the-api-handle-files-that-contain-spoken-audio-in-multiple-languages
Source: docs/faq/how-does-the-api-handle-files-that-contain-spoken-audio-in-multiple-languages.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How does the API handle files that contain spoken audio in multiple languages? documentation.

Our models are capable of transcribing audio files that contain spoken audio in more than one language. This is referred to as code switching. Code switching performance varies by language pair. See [this section](/pre-recorded-audio/code-switching) of our documentation to learn more.

---

<a id="how-long-does-it-take-to-transcribe-a-file-docs-faq-how-long-does-it-take-to-transcribe-a-file-mdx"></a>

# How long does it take to transcribe a file?

URL: https://www.assemblyai.com/docs/faq/how-long-does-it-take-to-transcribe-a-file
Source: docs/faq/how-long-does-it-take-to-transcribe-a-file.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: How long does it take to transcribe a file? documentation.

Processing times for our asynchronous transcription API are based on the duration of the submitted audio and models enabled in the request but the vast majority of files sent to our API will complete in **under 45 seconds**, and with a Real-Time-Factor (RTF) as low as .008x.

To put an RTF of .008x into perspective, this means you can convert a:

- 1h3min (75MB) meeting in 35 seconds
- 3h15min (191MB) podcast in 133 seconds
- 8h21min (464MB) video course in 300 seconds

Files submitted for Streaming Speech-to-Text receive a response within a few hundred milliseconds.

---

<a id="what-should-i-do-if-i-m-getting-an-error-docs-faq-i-am-getting-an-error-what-should-i-do-mdx"></a>

# What should I do if I'm getting an error?

URL: https://www.assemblyai.com/docs/faq/i-am-getting-an-error-what-should-i-do
Source: docs/faq/i-am-getting-an-error-what-should-i-do.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What should I do if I'm getting an error? documentation.

Any time you make a request to the API, you should receive a JSON response. If you don't receive the expected output, the JSON contains an `error` key with a message value describing the error.

To retrieve the error from our API:

1. Make a GET request to fetch the transcript details:

```
curl https://api.assemblyai.com/v2/transcript/<transcript_id> \
     -H "Authorization: <apiKey>"
```

The response will include an `error` key with details about what went wrong:

```
{
    "status": "error",
    "error": "Download error, unable to access file at https://example.com/audio.mp3",
    ...
}
```

2. Check our [Common Errors Guide](/pre-recorded-audio/guides/common_errors_and_solutions) for solutions to frequently encountered errors and how to resolve them.

3. Still stuck? Contact our support team at [support@assemblyai.com](mailto:support@assemblyai.com). Please include your transcript ID, the audio file/URL you're trying to transcribe, and the JSON response from our server so we can help you faster.

---

<a id="is-there-a-postman-collection-for-using-the-api-docs-faq-is-there-a-postman-collection-for-using-the-api-mdx"></a>

# Is there a Postman collection for using the API?

URL: https://www.assemblyai.com/docs/faq/is-there-a-postman-collection-for-using-the-api
Source: docs/faq/is-there-a-postman-collection-for-using-the-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Is there a Postman collection for using the API? documentation.

There is! You can find more information on our Postman collection in [this section](/integrations/postman) of our official documentation. See [this blog post](https://www.assemblyai.com/blog/postman-collection) to learn more.

---

<a id="is-there-a-way-for-us-to-send-the-start-time-end-time-for-transcription-instead-of-transcribing-the-whole-length-of-a-call-recording-docs-faq-is-there-a-way-for-us-to-send-the-start-time-end-time-for-transcription-instead-of-transcribing-the-whole-length-of-a-call-recording-mdx"></a>

# Is there a way for us to send the start time / end time for transcription instead of transcribing the whole length of a call recording?

URL: https://www.assemblyai.com/docs/faq/is-there-a-way-for-us-to-send-the-start-time-end-time-for-transcription-instead-of-transcribing-the-whole-length-of-a-call-recording
Source: docs/faq/is-there-a-way-for-us-to-send-the-start-time-end-time-for-transcription-instead-of-transcribing-the-whole-length-of-a-call-recording.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Is there a way for us to send the start time / end time for transcription instead of transcribing the whole length of a call recording? documentation.

Yes, you can!

There are parameters that you can add to your `POST` request that allow you to set the start time and end time of where you would like to start/stop transcribing your audio file.

More specifically you can use the `audio_start_from` parameter and enter the time in milliseconds of where you would like to start transcribing your media file and the `audio_end_at` parameter and enter the time in milliseconds of where you'd like to stop transcribing your media file.

The parameters are referenced in the [Set the start and end of the transcript](/pre-recorded-audio/set-the-start-and-end-of-the-transcript) section of our documentation.

---

<a id="is-there-an-openapi-spec-schema-for-the-api-docs-faq-is-there-an-openapi-spec-schema-for-the-api-mdx"></a>

# Is there an OpenAPI spec/schema for the API?

URL: https://www.assemblyai.com/docs/faq/is-there-an-openapi-spec-schema-for-the-api
Source: docs/faq/is-there-an-openapi-spec-schema-for-the-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Is there an OpenAPI spec/schema for the API? documentation.

Yes, AssemblyAI publishes an OpenAPI 3.1.0 specification for the REST API:

- [openapi.yaml](/openapi.yaml) — OpenAPI 3.1.0 spec (YAML)
- [openapi.json](https://www.assemblyai.com/docs/openapi.json) — OpenAPI 3.1.0 spec (JSON)

The spec covers all REST API endpoints including file upload, transcript submission and retrieval, and the LLM Gateway. You can use the spec to generate client libraries, validate requests, or import into tools like [Postman](/integrations/postman).

---

<a id="what-causes-a-read-operation-timed-out-error-docs-faq-read-operation-timed-out-error-mdx"></a>

# What causes a "read operation timed out" error?

URL: https://www.assemblyai.com/docs/faq/read-operation-timed-out-error
Source: docs/faq/read-operation-timed-out-error.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What causes a "read operation timed out" error? documentation.

This error is a byproduct of a request taking longer than the default timeout value built into the Python SDK, which is 15 seconds.

This shouldn't be a common occurrence but to allow for the occasional slowdown this setting can be changed like this:

```python
import assemblyai as aai

aai.settings.api_key = f"{ASSEMBLYAI_API_KEY}"
aai.settings.http_timeout = 30.0
```

---

<a id="should-i-use-speaker-labels-or-multi-channel-docs-faq-should-i-use-speaker-labels-or-multi-channel-mdx"></a>

# Should I use Speaker Labels or Multi-channel?

URL: https://www.assemblyai.com/docs/faq/should-i-use-speaker-labels-or-multi-channel
Source: docs/faq/should-i-use-speaker-labels-or-multi-channel.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Should I use Speaker Labels or Multi-channel? documentation.

The choice depends on your audio source:

## Use Multi-channel when:

You have audio where each speaker is recorded on a separate channel, such as:

- Twilio calls with configured channel separation
- Call center software that exports multiple audio channels
- Any recording where speakers are pre-separated into channels

Multichannel is more accurate since each speaker's audio is processed independently.

  Multichannel audio will not automatically separate multiple speakers on the
  same channel. If you have multiple speakers on a single channel, speaker
  labels are required for speaker separation.

## Use Speaker Labels when:

You have single-channel audio with multiple speakers, such as:

- Standard audio recordings with multiple participants
- Calls recorded as a single track
- Any audio where speakers share the same channel

For speaker labels, you can improve accuracy by specifying the expected number of speakers in advance [using the `speakers_expected` parameter](/api-reference/transcripts/submit#request.body.speakers_expected), or set a range of possible speakers [using the `speaker_options` parameter](/pre-recorded-audio/label-speakers#configuration).

---

<a id="should-i-use-the-na-or-eu-endpoint-for-my-speech-to-text-requests-docs-faq-should-i-use-the-na-or-eu-endpoint-mdx"></a>

# Should I use the NA or EU endpoint for my Speech-to-Text requests?

URL: https://www.assemblyai.com/docs/faq/should-i-use-the-na-or-eu-endpoint
Source: docs/faq/should-i-use-the-na-or-eu-endpoint.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Should I use the NA or EU endpoint for my Speech-to-Text requests? documentation.

By default, Speech-to-Text requests using pre-recorded audio are sent to our API via the `https://api.assemblyai.com/v2/transcript` endpoint. We also have an option EU endpoint, `https://api.eu.assemblyai.com/v2/transcript` endpoint that can be used for Speech-to-Text requests using pre-recorded audio.

Users should only use this EU endpoint when they have data residency needs that require that data be stored in the EU.

Otherwise, users should use the default NA endpoint as it will generally result in the fastest processing time regardless of what region the user is in.

---

<a id="what-are-the-recommended-options-for-audio-noise-reduction-docs-faq-what-are-the-recommended-options-for-audio-noise-reduction-mdx"></a>

# What are the recommended options for audio noise reduction?

URL: https://www.assemblyai.com/docs/faq/what-are-the-recommended-options-for-audio-noise-reduction
Source: docs/faq/what-are-the-recommended-options-for-audio-noise-reduction.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What are the recommended options for audio noise reduction? documentation.

When working with audio files, you may want to reduce background noise to improve the listening experience for your users. There are different solutions available depending on your specific needs and resources.

There are two main approaches you can take for audio noise reduction:

## 1. Commercial Solution

[Krisp](https://krisp.ai/developers/#technologies) is a recommended commercial solution for background noise reduction. They offer an SDK that can be integrated into your application.

## 2. Open Source Solution

If you prefer more customization and control over processing costs, you can use open-source solutions such as the [noisereduce](https://github.com/timsainb/noisereduce) library.

For implementation examples with real-time audio using the open-source approach, you can refer to our [streaming API cookbook](/streaming/guides/noise_reduction_streaming).

  If you're using our transcription service, we recommend applying noise
  reduction for playback purposes only, not before transcription.

---

<a id="what-audio-and-video-file-types-are-supported-by-your-api-docs-faq-what-audio-and-video-file-types-are-supported-by-your-api-mdx"></a>

# What audio and video file types are supported by your API?

URL: https://www.assemblyai.com/docs/faq/what-audio-and-video-file-types-are-supported-by-your-api
Source: docs/faq/what-audio-and-video-file-types-are-supported-by-your-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What audio and video file types are supported by your API? documentation.

The AssemblyAI API supports most common audio and video file formats. We recommend that you submit your audio in its native format without additional transcoding or file conversion. Transcoding or converting it to another format can sometimes result in a loss of quality, especially if you're converting compressed formats like `.mp3`. The AssemblyAI API converts all files to 16khz uncompressed audio as part of our transcription pipeline.

Note that when you upload a video to our API, the audio will be extracted from it and processed independently, so the list of supported video formats isn't exhaustive.

| Supported audio file types | Supported video file types  |
| -------------------------- | --------------------------- |
| .3ga                       | .webm                       |
| .8svx                      | .mts, .m2ts, .ts            |
| .aac                       | .mov                        |
| .ac3                       | .mp2                        |
| .aif                       | .mp4, .m4p (with DRM), .m4v |
| .aiff                      | .mxf                        |
| .alac                      |                             |
| .amr                       |                             |
| .ape                       |                             |
| .au                        |                             |
| .dss                       |                             |
| .flac                      |                             |
| .flv                       |                             |
| .m4a                       |                             |
| .m4b                       |                             |
| .m4p                       |                             |
| .m4r                       |                             |
| .mp3                       |                             |
| .mpga                      |                             |
| .ogg, .oga, .mogg          |                             |
| .opus                      |                             |
| .qcp                       |                             |
| .tta                       |                             |
| .voc                       |                             |
| .wav                       |                             |
| .wma                       |                             |
| .wv                        |                             |

---

<a id="what-ip-address-should-i-whitelist-for-assemblyai-docs-faq-what-ip-address-should-i-whitelist-for-assemblyai-mdx"></a>

# What IP Address Should I Whitelist for AssemblyAI?

URL: https://www.assemblyai.com/docs/faq/what-ip-address-should-i-whitelist-for-assemblyai
Source: docs/faq/what-ip-address-should-i-whitelist-for-assemblyai.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What IP Address Should I Whitelist for AssemblyAI? documentation.

AssemblyAI sends all webhook deliveries from fixed IP addresses:

| Region | IP Address   |
| ------ | ------------ |
| US     | 44.238.19.20 |
| EU     | 54.220.25.36 |

When retrieving audio files submitted by your account, our systems use dynamic IP addresses by default. If your network security requirements require a static IP address for these retrieval requests, please [contact us](https://assembly.ai/support) to enable this feature for your account. Once enabled, audio retrieval requests will be routed through the same fixed IP addresses listed above.

---

<a id="what-is-the-minimum-audio-duration-that-the-api-can-transcribe-docs-faq-what-is-the-minimum-audio-duration-that-the-api-can-transcribe-mdx"></a>

# What is the minimum audio duration that the API can transcribe?

URL: https://www.assemblyai.com/docs/faq/what-is-the-minimum-audio-duration-that-the-api-can-transcribe
Source: docs/faq/what-is-the-minimum-audio-duration-that-the-api-can-transcribe.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What is the minimum audio duration that the API can transcribe? documentation.

The minimum audio duration for a file submitted to our API is 160ms.

Files submitted to our API that are shorter than 160ms will return an error:

```json
{
  ...
  "status": "error",
  "audio_url": "https://foo.bar",
  "error": "Audio duration is too short.",
  ...
}
```

**To fix this issue**:

- **Add error handling for this error message**: When this error occurs, handle it safely by checking the error string and returning the error.
- **Add pre-submit checks for the duration of the audio file**: Prior to submitting a file for transcription, check the duration using a tool like soxi (part of the SoX package): `soxi -D audio.mp3`

---

<a id="what-is-the-recommended-file-type-for-using-your-api-docs-faq-what-is-the-recommended-file-type-for-using-your-api-mdx"></a>

# What is the recommended file type for using your API?

URL: https://www.assemblyai.com/docs/faq/what-is-the-recommended-file-type-for-using-your-api
Source: docs/faq/what-is-the-recommended-file-type-for-using-your-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What is the recommended file type for using your API? documentation.

Our API supports a wide range of file types so we typically recommend customers submit their audio in whatever its raw format is without any type of transcoding or file conversion. The reason for this is that transcoding or converting it to another format can sometimes result in a loss of quality.

Some formats like `.wav` are lossless while others like `.mp3` and `.acc` are lossy. See [this article](https://www.geeksforgeeks.org/computer-networks/difference-between-lossy-compression-and-lossless-compression/) to learn more. This difference between lossless and lossy is part of the reason we suggest people submit their audio as is without any modification.

We have transcoding on our end that will take the incoming files and convert them to 16khz `.wav` files so regardless of the file type submitted they will be able to work with our transcription pipeline.

---

<a id="what-types-of-audio-urls-can-i-use-with-the-api-docs-faq-what-types-of-audio-urls-can-i-use-with-the-api-mdx"></a>

# What types of audio URLs can I use with the API?

URL: https://www.assemblyai.com/docs/faq/what-types-of-audio-urls-can-i-use-with-the-api
Source: docs/faq/what-types-of-audio-urls-can-i-use-with-the-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: What types of audio URLs can I use with the API? documentation.

Your audio URL must:

- Be accessible by our servers
- Point directly to an audio/video file

You can upload your file directly to AssemblyAI's servers using our upload endpoint - [check out our transcription guide to get started](/pre-recorded-audio/getting-started/transcribe-an-audio-file).

Alternatively, you can host your files elsewhere. Common approaches are to:

- AWS presigned URLs ([cookbook](/pre-recorded-audio/guides/transcribe_from_s3))
- Google Drive ([cookbook](/pre-recorded-audio/guides/transcribing-google-drive-file)) - for files smaller than 100MB
- GitHub ([cookbook](/pre-recorded-audio/guides/transcribing-github-files)) - for files smaller than 100MB
- Cloud storage services (GCP/Azure/AWS)
- Any direct download URL that serves audio/video content

**Time-limited URLs must be fresh at submission time.** If you use presigned URLs (AWS S3, GCP, Azure SAS tokens) or CDN URLs from platforms like TikTok or Instagram, generate the URL immediately before submitting your request to `/v2/transcript`. If the URL expires before AssemblyAI downloads the file, the transcription will fail with a download error. For more details, see the [Troubleshoot Common Errors](/pre-recorded-audio/guides/common_errors_and_solutions) guide.

API Reference: [Upload endpoint](/api-reference/files/upload)

---

<a id="where-can-i-find-a-list-of-recent-changes-to-the-api-docs-faq-where-can-i-find-a-list-of-recent-changes-to-the-api-mdx"></a>

# Where can I find a list of recent changes to the API?

URL: https://www.assemblyai.com/docs/faq/where-can-i-find-a-list-of-recent-changes-to-the-api
Source: docs/faq/where-can-i-find-a-list-of-recent-changes-to-the-api.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Where can I find a list of recent changes to the API? documentation.

You can find a list of recent updates, improvements, and new features for our API in our [official Changelog](https://www.assemblyai.com/changelog). Be sure to subscribe to that page to receive notifications on future updates!

---

<a id="where-can-i-find-curl-code-examples-docs-faq-where-can-i-find-curl-code-examples-mdx"></a>

# Where can I find cURL code examples?

URL: https://www.assemblyai.com/docs/faq/where-can-i-find-curl-code-examples
Source: docs/faq/where-can-i-find-curl-code-examples.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Where can I find cURL code examples? documentation.

You can find cURL examples throughout our API reference documentation. Each endpoint includes sample cURL requests that you can copy and modify for your use case.

## Where to find cURL examples

Visit the [API reference section](/api-reference/overview) of our documentation. Each API endpoint page includes cURL code examples.

For example, the [Submit a file for transcription page](/api-reference/transcripts/submit) shows the cURL request format for submitting audio files.

## Converting other code examples to cURL

If you see code examples in other programming languages in our documentation, you can use AI tools like ChatGPT or Claude to convert those examples into cURL format. Simply provide the code example and ask the AI tool to recreate it as a cURL request.

---

<a id="why-can-t-i-access-recording-urls-from-the-upload-endpoint-directly-docs-faq-why-cant-i-access-urls-from-the-upload-endpoint-directly-mdx"></a>

# Why can't I access recording URLs from the /upload endpoint directly?

URL: https://www.assemblyai.com/docs/faq/why-cant-i-access-urls-from-the-upload-endpoint-directly
Source: docs/faq/why-cant-i-access-urls-from-the-upload-endpoint-directly.mdx
Navigation: FAQ > Overview > Pre-recorded audio
Description: Why can't I access recording URLs from the /upload endpoint directly? documentation.

The URLs generated by AssemblyAI's `/upload` endpoint are designed exclusively for transcript processing and are only accessible by our API. These URLs cannot be accessed directly, which is why you receive an "AccessDenied" error when you try to navigate to the URLs or use them in other applications.

## Recommended Solution

If you need a publicly accessible URL for your audio files to use in automation workflows or other applications, we recommend hosting your files in your own cloud storage solution, such as:

- Amazon S3 bucket
- Google Cloud Storage
- Azure Blob Storage
- Other cloud storage providers

## Best Practice Workflow

For automation workflows that require access to the original audio file:

1. Upload your audio file to your own cloud storage
2. Generate a public or pre-signed URL from your storage provider
3. Use that URL in your automation tools (like Make)
4. Submit the same URL to AssemblyAI for transcription processing

This approach ensures you maintain control over file access while still being able to process transcripts through AssemblyAI's API.

---

<a id="can-i-use-speaker-diarization-with-streaming-speech-to-text-docs-faq-can-i-use-speaker-diarization-with-live-audio-transcription-mdx"></a>

# Can I use speaker diarization with Streaming Speech-to-Text?

URL: https://www.assemblyai.com/docs/faq/can-i-use-speaker-diarization-with-live-audio-transcription
Source: docs/faq/can-i-use-speaker-diarization-with-live-audio-transcription.mdx
Navigation: FAQ > Overview > Streaming audio
Description: Can I use speaker diarization with Streaming Speech-to-Text? documentation.

Yes. Streaming Diarization is supported on all streaming models, including `universal-3-5-pro`, `universal-streaming-english`, and `universal-streaming-multilingual`. Enable it by adding `speaker_labels: true` to your connection parameters. See our [Streaming Diarization documentation](/streaming/label-speakers-and-separate-channels) to learn more.

Alternatively, you can use multichannel audio (i.e., an audio stream for each speaker) to separate speakers by creating a separate session for each channel. See the [Multichannel streaming audio](/streaming/label-speakers-and-separate-channels#multichannel-streaming-audio) section for details on this approach.

---

<a id="how-accurate-is-your-streaming-transcription-compared-to-async-transcription-docs-faq-how-accurate-is-your-real-time-transcription-compared-to-async-transcription-mdx"></a>

# How accurate is your Streaming transcription compared to Async transcription?

URL: https://www.assemblyai.com/docs/faq/how-accurate-is-your-real-time-transcription-compared-to-async-transcription
Source: docs/faq/how-accurate-is-your-real-time-transcription-compared-to-async-transcription.mdx
Navigation: FAQ > Overview > Streaming audio
Description: How accurate is your Streaming transcription compared to Async transcription? documentation.

Since Streaming Speech-to-Text models are built for low latency, they are only able to use limited historical context when making predictions. This is compared to our Async models, which can look at complete historical and future context to make predictions (since they have access to the entire audio file). As a result, Streaming models are usually a few percentage points less accurate than Async models (~2-3% absolute). Overall, the results are still quite good!

---

<a id="how-does-universal-streaming-session-based-pricing-work-docs-faq-how-does-universal-streaming-session-based-pricing-work-mdx"></a>

# How does Universal Streaming session-based pricing work?

URL: https://www.assemblyai.com/docs/faq/how-does-universal-streaming-session-based-pricing-work
Source: docs/faq/how-does-universal-streaming-session-based-pricing-work.mdx
Navigation: FAQ > Overview > Streaming audio
Description: How does Universal Streaming session-based pricing work? documentation.

Universal Streaming is billed on the total duration of each streaming session — the entire time the WebSocket connection stays open — not on the amount of audio sent through it. You're charged for idle time on an open session just the same as you are for time when audio is actively flowing.

Each open session is billed independently, so concurrent sessions accumulate billed time in parallel. For example, a single call that is dual-streamed under two separate session IDs for 5 minutes bills as 10 minutes of session time. See the [Models pricing section](/getting-started/models#pricing) for current rates.

This gives you full control over cost and latency. You can keep a stream open continuously for instant response, or open streams strategically on demand to minimize spend.

For the current per-hour streaming rates, see the [pricing page](https://www.assemblyai.com/pricing) or the [Models pricing section](/getting-started/models#pricing).

## Best practices to avoid unexpected charges

- **Always send a session termination message** when your application is finished with a stream. This is what stops billing for the session. See [Session termination](/streaming/getting-started/transcribe-streaming-audio) for details.
- **Sessions that are not terminated auto-close after 3 hours**, and you'll be billed for the full 3-hour session duration regardless of how much audio was actually streamed.
- Improperly closed streaming sessions are the most common cause of unexpected charges that lead to negative account balances.

## Additional resources

- [Billing and pricing](/billing-and-pricing) in Account Management
- [How does the automatically scaling streaming rate limit for Real-time STT work?](/streaming/rate-limits)

---

<a id="what-languages-are-supported-for-streaming-speech-to-text-docs-faq-language-support-for-real-time-transcription-mdx"></a>

# What languages are supported for Streaming Speech-to-text?

URL: https://www.assemblyai.com/docs/faq/language-support-for-real-time-transcription
Source: docs/faq/language-support-for-real-time-transcription.mdx
Navigation: FAQ > Overview > Streaming audio
Description: What languages are supported for Streaming Speech-to-text? documentation.

Language support for Real-time STT depends on the model you select. AssemblyAI offers several streaming models with different language capabilities:

## Available Models and Language Support

| Model | Languages Supported |
| --- | --- |
| **Universal-3.5 Pro Streaming** | English, Spanish, German, French, Portuguese, Italian, Turkish, Dutch, Swedish, Norwegian, Danish, Finnish, Hindi, Vietnamese, Arabic, Hebrew, Japanese, Mandarin (native code switching across 18 languages) |
| **Universal-Streaming English** | English only |
| **Universal-Streaming Multilingual** | English, Spanish, German, French, Portuguese, Italian (per turn) |

## Choosing the Right Model

- If you need the highest accuracy with multilingual code switching across up to 18 languages, use [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio) (`universal-3-5-pro`).
- If you only need English transcription at the lowest cost, use Universal-Streaming English (`universal-streaming-english`).
- If you need multilingual support at a lower cost, use Universal-Streaming Multilingual (`universal-streaming-multilingual`).

For more details on model selection, see the [Model selection](/streaming/select-the-speech-model) page.

## Difference From Pre-recorded STT

It's important to note that the `language_code` parameter mentioned in some AssemblyAI documentation applies to the Pre-recorded STT feature, not the Streaming Transcription feature. For real-time STT, you specify the model using the `speech_model` parameter.

  To stay informed about new features and improvements, including language
  support updates, you can follow our
  [Changelog](https://www.assemblyai.com/changelog).

---

<a id="resolving-ssl-certificate-verification-error-when-trying-to-use-real-time-stt-docs-faq-resolving-ssl-certificate-verification-error-in-assemblyai-real-time-transcriber-mdx"></a>

# Resolving SSL Certificate Verification Error When Trying to Use Real-time STT

URL: https://www.assemblyai.com/docs/faq/resolving-ssl-certificate-verification-error-in-assemblyai-real-time-transcriber
Source: docs/faq/resolving-ssl-certificate-verification-error-in-assemblyai-real-time-transcriber.mdx
Navigation: FAQ > Overview > Streaming audio
Description: Resolving SSL Certificate Verification Error When Trying to Use Streaming STT documentation.

When using AssemblyAI's Streaming Speech-to-Text feature (for example, the `StreamingClient` SDK connecting to `wss://streaming.assemblyai.com/v3/ws`), you may encounter the following error:

```
An error occurred: Could not connect to the streaming service: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)
```

This error is typically caused by outdated or incomplete SSL certificates on your system, particularly Python certificates.

## How to Resolve the Issue

1. Update your system's SSL certificates, focusing on Python certificates.

2. If you're using Python, you may need to install or update the `certifi` package, which provides Mozilla's root certificate bundle.

3. On some systems, you might need to run a command to install certificates for Python. The exact command can vary depending on your operating system and Python installation.

## For macOS Users

If you're using macOS, you can try running the following command in your terminal:

```
/Applications/Python 3.x/Install Certificates.command
```

Replace "3.x" with your Python version number.

## For Windows Users

On Windows, you might need to download and install the certificates manually. You can find instructions for this process online, specific to your version of Windows and Python.

## Additional Resources

For more detailed information on resolving SSL certificate issues in Python, you can refer to this [Stack Overflow article](https://stackoverflow.com/questions/50236117/scraping-ssl-certificate-verify-failed-error-for-http-en-wikipedia-org).

If you continue to experience issues after updating your certificates, please contact AssemblyAI support for further assistance.

---

<a id="i-am-getting-a-model-deprecated-see-docs-for-new-model-information-error-message-what-does-it-mean-docs-faq-upgrading-to-the-universal-streaming-model-mdx"></a>

# I am getting a "Model deprecated. See docs for new model information" error message. What does it mean?

URL: https://www.assemblyai.com/docs/faq/upgrading-to-the-universal-streaming-model
Source: docs/faq/upgrading-to-the-universal-streaming-model.mdx
Navigation: FAQ > Overview > Streaming audio
Description: I am getting a "Model deprecated. See docs for new model information" error message. What does it mean? documentation.

If you receive an error message indicating `Model deprecated, see our docs for new model information: https://www.assemblyai.com/docs/streaming`, it means you are using an older version of our Real-time STT feature that is no longer supported.

## How to upgrade to a supported streaming model

You'll need to update your implementation to use one of our current streaming models. We recommend **Universal-3.5 Pro Streaming** (`universal-3-5-pro`) as the primary streaming model. It delivers the highest accuracy with sub-300ms latency and native code switching across English, Spanish, German, French, Portuguese, and Italian.

1. Review the [Streaming documentation](/streaming) for model selection and implementation guidance.

2. For a quick start guide, see our [Getting Started with Streaming Audio](/streaming/getting-started/transcribe-streaming-audio) documentation.

  Universal-3.5 Pro Streaming and the Universal-Streaming family of models offer
  improved accuracy and reliability compared to our legacy streaming
  implementation.

---

<a id="how-do-content-moderation-severity-scores-work-docs-faq-how-do-content-moderation-severity-scores-work-mdx"></a>

# How do Content Moderation severity scores work?

URL: https://www.assemblyai.com/docs/faq/how-do-content-moderation-severity-scores-work
Source: docs/faq/how-do-content-moderation-severity-scores-work.mdx
Navigation: FAQ > Overview > Speech understanding
Description: How do Content Moderation severity scores work? documentation.

AssemblyAI's content moderation API implements a context-aware severity scoring system that helps developers build more intelligent content filtering solutions. This guide examines how the severity parameter works through practical examples.

## Severity Score Structure

The API returns severity scores across three levels for each detected label:

```plaintext
"severity_score_summary": {
    "label_name": {
        "low": float,    // 0-1
        "medium": float, // 0-1
        "high": float    // 0-1
    }
}
```

Please refer to our API Reference for Content Moderation for the full response.

The scores represent probability distributions across severity levels, always summing to 1.0.

### Case Study: Contextual Analysis of Alcohol References:

To demonstrate how the system evaluates context, let's analyze three different transcripts discussing alcohol:

**Educational Context**

**Source**: What Are Alcohols? | Organic Chemistry ([full transcript + audio](https://www.assemblyai.com/playground/transcript/fdf06be8-b480-4686-9d92-6fe2d7ce2753))

```
"severity_score_summary": {
    "alcohol": {
        "low": 1.0,
        "medium": 0,
        "high": 0
    }
}
```

The system identifies the educational context, assigning maximum low severity despite explicit alcohol-related terminology.

**Social Context**

**Source**: How I Learned To Go To A Bar Alone And Meet Women ([full transcript + audio](https://www.assemblyai.com/playground/transcript/da4e4612-240e-499b-a8c9-4f14ef4064b2))

```
"severity_score_summary": {
    "alcohol": {
        "low": 0.55243415,
        "medium": 0.44756585,
        "high": 0
    }
}
```

References to bars and casual drinking split the probability between low and medium severity levels.

**Substance Abuse Context**

**Source**: Steve-O: I'm grateful my alcoholism was severe ([full transcript + audio](https://www.assemblyai.com/playground/transcript/aedd18f2-515a-4882-bf8f-d29e78d98b65))

```
"severity_score_summary": {
    "alcohol": {
        "medium": 0.22141029,
        "high": 0.7785897,
        "low": 0
    }
}
```

Discussion of addiction and abuse triggers high severity scores, reflecting potentially sensitive content.

### Conclusion

The severity scoring system provides a sophisticated way to understand not just what topics are mentioned, but how they're discussed. Each severity level (low, medium, high) represents a probability score between 0 and 1, allowing developers to implement precise content filtering based on their specific needs. For example, developers can set custom thresholds for content classification (e.g., flagging content with high severity > 0.7) or create tiered content ratings based on severity distributions. This granular approach to content moderation enables more accurate and context-aware content filtering systems.

---

<a id="how-can-i-summarize-my-audio-file-docs-faq-how-do-your-summarization-models-work-mdx"></a>

# How can I summarize my audio file?

URL: https://www.assemblyai.com/docs/faq/how-do-your-summarization-models-work
Source: docs/faq/how-do-your-summarization-models-work.mdx
Navigation: FAQ > Overview > Speech understanding
Description: How can I summarize my audio file? documentation.

## [Summarization](/speech-understanding/summarization)

Our standard summarization feature generates a single abstractive summary of your entire audio file.

---

<a id="is-mistral-still-supported-docs-faq-is-mistral-still-supported-mdx"></a>

# Is Mistral still supported?

URL: https://www.assemblyai.com/docs/faq/is-mistral-still-supported
Source: docs/faq/is-mistral-still-supported.mdx
Navigation: FAQ > Overview > Speech understanding
Description: Is Mistral still supported? documentation.

Mistral has been sunsetted and is no longer available. See [this section](/llm-gateway/quickstart) of our documentation for more information on which models are available through LLM Gateway.

---

<a id="is-pricing-for-speech-understanding-per-feature-or-all-inclusive-docs-faq-is-pricing-for-audio-intelligence-per-feature-or-all-inclusive-mdx"></a>

# Is pricing for Speech Understanding per feature or all-inclusive?

URL: https://www.assemblyai.com/docs/faq/is-pricing-for-audio-intelligence-per-feature-or-all-inclusive
Source: docs/faq/is-pricing-for-audio-intelligence-per-feature-or-all-inclusive.mdx
Navigation: FAQ > Overview > Speech understanding
Description: Is pricing for Speech Understanding per feature or all-inclusive? documentation.

Pricing for our [Speech Understanding features](https://www.assemblyai.com/products/speech-understanding) is based on which models are used in the request and the duration of the audio file submitted. See [our Pricing page](https://www.assemblyai.com/pricing) for more information on the pricing for each model.

---

<a id="understanding-input-and-output-tokens-for-llm-gateway-docs-faq-understanding-input-and-output-tokens-for-llm-gateway-mdx"></a>

# Understanding Input and Output Tokens for LLM Gateway

URL: https://www.assemblyai.com/docs/faq/understanding-input-and-output-tokens-for-llm-gateway
Source: docs/faq/understanding-input-and-output-tokens-for-llm-gateway.mdx
Navigation: FAQ > Overview > LLM Gateway
Description: Understanding Input and Output Tokens for LLM Gateway documentation.

## What are Input Tokens?

Input tokens represent the number of tokens provided to the AI model in your initial request. These include:

- The prompt or question
- Transcript text
- Context information
- Any other data included in your request

## What are Output Tokens?

Output tokens represent the number of tokens generated by the AI model in response to your input. This typically includes:

- The text response from the model

## Key Differences

There are two main differences between input and output tokens:

1. **Quantity:** Usually, the number of input tokens is higher than the number of output tokens. This is because your initial request often contains more information than the model's response.

2. **Cost:** Output tokens generally cost more than input tokens. This is due to the computational resources required for the model to generate a response.

## Why This Matters

Understanding the difference between input and output tokens can help you:

- Optimize your requests to the AI model
- Manage costs associated with using the AI service
- Improve the efficiency of your interactions with the AI model

If you have any further questions about input and output tokens or how they affect your usage of our AI service, please don't hesitate to contact our support team.

---

<a id="can-you-use-the-playground-with-files-in-languages-other-than-english-docs-faq-can-you-use-the-playground-with-files-in-languages-other-than-english-mdx"></a>

# Can you use the Playground with files in languages other than English?

URL: https://www.assemblyai.com/docs/faq/can-you-use-the-playground-with-files-in-languages-other-than-english
Source: docs/faq/can-you-use-the-playground-with-files-in-languages-other-than-english.mdx
Navigation: FAQ > Overview > Playground
Description: Can you use the Playground with files in languages other than English? documentation.

You can! Automatic Language Detection is selected by default in the Playground. If you know the language of your file you can specify the language by opening up the dropdown menu below where you upload your file.

  <img src="/assets/img/faq/playground2.png" alt="Playground language selection dropdown menu below the file upload area" />

---

<a id="how-do-i-delete-a-transcript-i-created-using-the-playground-docs-faq-how-do-i-delete-a-transcript-i-created-using-the-playground-mdx"></a>

# How do I delete a transcript I created using the Playground?

URL: https://www.assemblyai.com/docs/faq/how-do-i-delete-a-transcript-i-created-using-the-playground
Source: docs/faq/how-do-i-delete-a-transcript-i-created-using-the-playground.mdx
Navigation: FAQ > Overview > Playground
Description: How do I delete a transcript I created using the Playground? documentation.

On the right side of the Pre-recorded Audio section of the Playground, you will see a "History" sidebar. To the left of each transcript id, you will see three dots. Click those dots and select "Delete" to delete that transcript.

---

<a id="why-is-the-transcription-i-am-receiving-using-the-playground-in-a-different-language-docs-faq-why-is-the-transcription-i-am-receiving-using-the-playground-in-a-different-language-mdx"></a>

# Why is the transcription I am receiving using the Playground in a different language?

URL: https://www.assemblyai.com/docs/faq/why-is-the-transcription-i-am-receiving-using-the-playground-in-a-different-language
Source: docs/faq/why-is-the-transcription-i-am-receiving-using-the-playground-in-a-different-language.mdx
Navigation: FAQ > Overview > Playground
Description: Why is the transcription I am receiving using the Playground in a different language? documentation.

Your file was likely submitted to the wrong language model. The Playground has Automatic Language Detection (ALD) enabled by default so it will detect the dominant language of the file and route it to the appropriate model for transcription.

If you know the dominant language spoken in your audio file you should select that when submitting your file rather than using ALD.

---

<a id="do-you-have-an-affiliate-marketing-program-docs-faq-do-you-have-an-affiliate-marketing-program-mdx"></a>

# Do you have an affiliate marketing program?

URL: https://www.assemblyai.com/docs/faq/do-you-have-an-affiliate-marketing-program
Source: docs/faq/do-you-have-an-affiliate-marketing-program.mdx
Navigation: FAQ > Overview > Company
Description: Do you have an affiliate marketing program? documentation.

No - we do not currently offer an affiliate marketing program.

---

<a id="do-you-have-any-job-openings-or-internship-opportunities-docs-faq-do-you-have-any-job-openings-or-internship-opportunities-mdx"></a>

# Do you have any job openings or internship opportunities?

URL: https://www.assemblyai.com/docs/faq/do-you-have-any-job-openings-or-internship-opportunities
Source: docs/faq/do-you-have-any-job-openings-or-internship-opportunities.mdx
Navigation: FAQ > Overview > Company
Description: Do you have any job openings or internship opportunities? documentation.

As a fast-growing start-up, we are regularly growing our team! Check out our [Careers page](https://www.assemblyai.com/careers) to learn more about our current open positions. We are regularly posting new openings so if you don't see anything that is a fit for you now continue to check back in the future.

---

<a id="how-do-i-contact-support-docs-faq-how-do-i-contact-support-mdx"></a>

# How do I contact support?

URL: https://www.assemblyai.com/docs/faq/how-do-i-contact-support
Source: docs/faq/how-do-i-contact-support.mdx
Navigation: FAQ > Overview > Company
Description: How do I contact support? documentation.

When logged in to your account dashboard, you can chat with us via our chat widget.

  <img src="/assets/img/faq/support2.png" alt="Support chat widget open in the AssemblyAI account dashboard" />

You can also reach out to our support team at any time by sending an email to [support@assemblyai.com](mailto:support@assemblyai.com). When reaching out, please include a detailed description of any issues you're experiencing as well as transcript IDs for affected requests, if possible.

---

<a id="how-do-i-get-in-touch-with-your-sales-team-docs-faq-how-do-i-get-in-touch-with-your-sales-team-mdx"></a>

# How do I get in touch with your Sales team?

URL: https://www.assemblyai.com/docs/faq/how-do-i-get-in-touch-with-your-sales-team
Source: docs/faq/how-do-i-get-in-touch-with-your-sales-team.mdx
Navigation: FAQ > Overview > Company
Description: How do I get in touch with your Sales team? documentation.

You can reach out to our Sales team by filling out [this contact form](https://www.assemblyai.com/contact/sales). Some of the things they can help with include discussing pricing, plans, and contracts, or scheduling a demo of our product.

  If you have a technical issue with our API, please reach out to our Support
  team by either filling out [this
  form](https://www.assemblyai.com/contact/support) or sending an email to
  support@assemblyai.com.

---

<a id="i-ve-spotted-an-issue-with-the-website-what-should-i-do-docs-faq-ive-spotted-an-issue-with-the-website-what-should-i-do-mdx"></a>

# I’ve spotted an issue with the website, what should I do?

URL: https://www.assemblyai.com/docs/faq/ive-spotted-an-issue-with-the-website-what-should-i-do
Source: docs/faq/ive-spotted-an-issue-with-the-website-what-should-i-do.mdx
Navigation: FAQ > Overview > Company
Description: I’ve spotted an issue with the website, what should I do? documentation.

If you encounter an issue or problem with our website please reach out to us at [support@assemblyai.com](mailto:support@assemblyai.com) and we would be happy to look into it for you.

---

<a id="what-are-your-support-hours-and-response-time-slas-docs-faq-what-are-your-support-hours-and-response-time-slas-mdx"></a>

# What are your support hours and response time SLAs?

URL: https://www.assemblyai.com/docs/faq/what-are-your-support-hours-and-response-time-slas
Source: docs/faq/what-are-your-support-hours-and-response-time-slas.mdx
Navigation: FAQ > Overview > Company
Description: What are your support hours and response time SLAs? documentation.

AssemblyAI provides technical support to Customer via telephone, live chat, and email on weekdays during the hours of 6:00 pm PT Sunday through 6:00 pm PT Friday, with the exclusion of federal holidays ("Support Hours").

Customers may initiate a helpdesk ticket by emailing [support@assemblyai.com](mailto:support@assemblyai.com), or the support widget on AssemblyAI's website [www.AssemblyAI.com](https://www.assemblyai.com/). Some select customers may also have a Slack Connect channel which is configured and monitored by the Support Team. All message services may be accessed 24/7.

When reporting problems experienced with the Services, Customer shall provide AssemblyAI with the following information, as applicable, in the call or email:

1. Company name
2. Contact name and callback number
3. Problem description
4. Date and time problem was detected
5. Error messages

## Ticket Severity and Response SLAs

AssemblyAI shall assign a trouble ticket to Customer, and shall use commercially reasonable efforts to resolve each trouble ticket according to the Severity Levels set forth below.

| Severity Level | Definition                                                                                                                                                                         | Initial Response | Rate of Work                                                                                                          | Status Updates |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------- | -------------- |
| HIGH           | Service is unavailable or inoperable (including Downtime).                                                                                                                         | 60 min           | 24 x 7 until a resolution is achieved                                                                                 | Hourly         |
| MEDIUM         | Service is available and usable, but certain non-critical functions not working properly.                                                                                          | 4 hours          | Work shall be performed during AssemblyAI's normal business hours until a resolution is achieved                      | Weekly         |
| LOW            | Service is available and all functions are working as advertised. This category would include feature enhancement requests, or service questions requiring research by AssemblyAI. | 24 hours         | Work shall be performed during AssemblyAI's normal business hours, based on nature, scope, and complexity of request. | Monthly        |

---

<a id="what-is-your-api-uptime-sla-docs-faq-what-is-your-api-uptime-sla-mdx"></a>

# What is your API Uptime SLA?

URL: https://www.assemblyai.com/docs/faq/what-is-your-api-uptime-sla
Source: docs/faq/what-is-your-api-uptime-sla.mdx
Navigation: FAQ > Overview > Company
Description: What is your API Uptime SLA? documentation.

Our SLA guarantees an  uptime of 99.9%  for our API service. This commitment means that, within a given year our service will not be unavailable for more than 8 hours and 46 minutes (526 mins).

Our service's uptime is measured on an annual basis. We continuously monitor the availability of our API endpoints. A period is considered "downtime" when a significant number of requests to our API fail. Scheduled maintenance periods, which we announce in advance, are not included in this calculation.

    We use Amazon Web Services (AWS) as our cloud provider. Our infrastructure is designed to be highly available, utilizing multiple AWS Availability Zones (AZs) to protect against single points of failure. Even if an entire AWS AZ were to go down, our service would continue operating from other healthy zones, however we must account for AWS' SLAs for uptime in our calculations.

Please refer to our [Terms of Service](https://www.assemblyai.com/legal/terms-of-service) regarding credits you may be entitled to.

---

<a id="where-can-i-find-assemblyai-s-product-roadmap-docs-faq-where-can-i-find-assemblyais-product-roadmap-mdx"></a>

# Where can I find AssemblyAI's product roadmap?

URL: https://www.assemblyai.com/docs/faq/where-can-i-find-assemblyais-product-roadmap
Source: docs/faq/where-can-i-find-assemblyais-product-roadmap.mdx
Navigation: FAQ > Overview > Company
Description: Where can I find AssemblyAI's product roadmap? documentation.

You can view our [public product roadmap](https://assemblyai.com/roadmap).

---

<a id="assemblyai-integration-prompt-for-ai-coding-agents-docs-agent-instructions-mdx"></a>

# AssemblyAI integration prompt for AI coding agents

URL: https://www.assemblyai.com/docs/agent-instructions
Source: docs/agent-instructions.mdx
Navigation: Additional pages
Description: The full integration prompt AI coding agents fetch when building with AssemblyAI. Pin the URL https://www.assemblyai.com/docs/agent-instructions.md in your agent's project instructions, or copy the prompt below.

# AssemblyAI Integration — Coding Agent Instructions

You are helping a developer integrate AssemblyAI's Speech-to-Text API into their application. Your job is to understand their context through discovery, produce a concrete implementation plan, get their approval, and then write correct, production-ready code.

This is a public API. The developer creates their own key at [assemblyai.com/dashboard/api-keys](https://www.assemblyai.com/dashboard/home).

**Official documentation.** Two ways to wire your coding agent up to live docs (both recommended — they layer):

1. **Project instructions** (every prompt): add to `CLAUDE.md`, `.cursorrules`, `AGENTS.md`, or equivalent:

   ```
   Always fetch https://www.assemblyai.com/docs/llms.txt before writing AssemblyAI code.
   The API has changed — do not rely on memorized parameter names.
   ```

   `llms.txt` is the structured index. For full content use `llms-full.txt`; narrow with `?lang=python` or `?lang=typescript`, or add `?excludeSpec=true` to skip the API spec.

2. **Docs MCP server** (on-demand lookups): `https://assemblyai.com/docs/mcp` — Streamable HTTP transport. Lets the agent search and fetch AssemblyAI documentation pages on demand.

   ```bash
   # Claude Code
   claude mcp add assemblyai-docs --transport http https://assemblyai.com/docs/mcp
   ```

   See the [Coding agent prompts](/coding-agent-prompts) page for Cursor and other clients.

---

## 0. Operating rules

1. **Discovery first, code later.** Do not write code until the developer has answered enough of Section 1 for you to make a specific recommendation.
2. **One question per message.** Never batch discovery questions. Wait for an answer before asking the next one.
3. **Plan before you build.** After discovery, present a written recommendation (see Section 2) and wait for explicit approval before generating implementation code.
4. **Prefer the official SDKs.** Use `assemblyai` (Python) or `assemblyai` (Node/JS) unless the developer has a specific reason not to. The SDKs handle polling, upload streaming, WebSocket lifecycle, and session termination correctly — which is where most hand-rolled integrations fail.
5. **Never expose the API key in client-side code.** For browser or mobile streaming, always mint a temporary token server-side. For pre-recorded, proxy uploads and submissions through your server.
6. **Authorization header is the raw key — no `Bearer` prefix.** This trips up everyone. **One exception:** the Voice Agent API (Section 10) requires `Authorization: Bearer YOUR_API_KEY`. Don't generalize either rule across products.
7. **`speech_models` is optional on pre-recorded requests.** If omitted, the request defaults to `["universal-3-5-pro", "universal-2"]`. You can still set it explicitly — see Section 5 for semantics.
8. **Always terminate streaming sessions explicitly.** An abandoned WebSocket keeps accruing charges until the 3-hour cap.
9. **Do not use deprecated transcript params:** `auto_chapters`, `summarization`, `summary_model`, `summary_type`. Use LLM Gateway instead (Section 8).
10. **If the developer's answers are inconsistent, stop and surface the conflict.** Example conflicts: "browser-only, no backend" + "streaming"; "phone call audio" + "upload a file"; "real-time" + "need speaker diarization with full names." Don't paper over these — ask.
11. **Be flexible.** If something the developer says doesn't match the shape of the API (e.g., they describe a use case that isn't supported — see Section 13), say so directly and propose the closest supported alternative.
12. **Verify parameters against live docs before recommending.** This file is a snapshot — features move between beta and GA, model-specific behaviors change, and new knobs ship regularly. Before posting the Section 2 recommendation, confirm each parameter you plan to use is supported for the chosen **mode** (pre-recorded vs streaming) *and* **model** (U3 Pro, U2, U3 Pro Streaming, Universal-Streaming). Do not assume a pre-recorded flag works on streaming, or that a parameter supported on U2 still behaves the same on U3 Pro. Pull the current reference rather than memorizing. Primary sources, in order of preference:
    - `https://www.assemblyai.com/docs/llms-full.txt` — the canonical machine-readable reference
    - Per-mode docs: `/docs/pre-recorded-audio/*` (pre-recorded) and `/docs/streaming/*` (streaming), including the model-specific overview page (e.g., `/docs/streaming/getting-started/transcribe-streaming-audio` and `/docs/streaming/select-the-speech-model`) which lists *exactly* which parameters are honored/ignored by that model
    - The OpenAPI-backed API reference at `/docs/api-reference/*` for request/response schemas
    - For LLM Gateway: `/docs/llm-gateway/quickstart` lists the current valid `model` strings — don't guess short names like `claude-sonnet-4`

  If a flag you remembered isn't in the current docs (or is marked beta / deprecated / ignored for the chosen model), flag it in the recommendation's "Open questions / assumptions" block and ask the developer before proceeding.

---

## 1. Discovery questions

Ask these **one at a time**, in order. Skip any question already answered in the conversation. Adapt wording to sound natural, but cover the substance of each.

1. **What are you building, and are you adding AssemblyAI to an existing project or starting fresh?** (A short description of the product is usually enough.)
2. **What do you need: pre-recorded transcription, real-time real-time STT, or a managed voice agent?**
   - Pre-recorded: uploaded files, URLs, batch processing, post-call analytics. → Section 6.
   - Real-time STT: live transcripts only (you bring your own LLM/TTS). Live captioning, voice-agent STT, meeting notetaking, dictation. → Section 9.
   - Voice Agent API (managed): full-duplex speech-in/speech-out — STT + LLM + TTS + turn detection + tool calling, all in one WebSocket. Right answer when "I want to talk to an AI" is the whole product. → Section 10.
3. **Where is your audio coming from?** (e.g., uploaded files, public URLs, browser microphone, mobile app, Twilio/Telnyx phone numbers, SIP trunks.)
4. **What language and framework are you using?** (e.g., Python + FastAPI, Node + Next.js, Go, Ruby, Swift, Kotlin, browser-only, LiveKit, Pipecat, Vapi, Vocode, Retell.)
5. **Do you already have an AssemblyAI API key, or do you need to create one?** (If needed: [assemblyai.com/dashboard/api-keys](https://www.assemblyai.com/dashboard/home).)
6. **Do you have a data residency requirement?** (US vs EU — this changes the base URL.)
7. **Anything beyond a plain transcript?** Don't read off a checklist. Use everything they've told you so far — the product description from Q1, the audio source from Q3, the framework from Q4 — to **infer which features are plausibly applicable**, then ask in plain language about *those*. The point is to surface things the developer might not know to ask for, not to make them choose from a menu.

   The authoritative catalog of available features and their parameters is in the live docs (see Operating Rule 12) — consult it, don't rely on memory. Section 3 of this file is a starting reference, not the final word.

   Calibrate to mode and use case. Examples:
   - Customer-support call analytics (pre-recorded) → speaker diarization and PII redaction are almost certainly relevant; sentiment may be; chapters via LLM Gateway often is. Ask about those, not about live-streaming features.
   - Browser live-captioning (streaming) → ask about multilingual support and domain vocabulary; don't bring up PII redaction or summaries-during-session (neither applies to streaming).
   - Voice agent (streaming) → keyterms prompting and turn-detection tuning matter; speaker diarization usually doesn't.
   - Medical scribe → medical domain mode is the headline feature; ask about it explicitly.

   Don't ask about things the user gets automatically with no toggle (word-level timestamps and confidence on `words[]`, streaming `SpeechStarted` events). Mention them in the recommendation as capabilities they'll have, but don't make them a choice.

   If you're confident from context that a feature is needed (e.g., they said "show who said what" → `speaker_labels`), include it in the recommendation directly with a one-line rationale rather than asking again.

---

## 2. Recommendation template (after discovery)

Before writing code, post a plan with all of the following. Get explicit approval.

````
## Recommendation

**Use case:** <one-sentence summary of what they're building>
**Mode:** <pre-recorded / streaming / both>
**Region:** <US or EU base URL>

**Model:**
- <model name> — <one-line rationale>
- <fallback model, if applicable>

**Endpoints:**
- <endpoint 1>
- <endpoint 2>

**Parameters enabled:** (before filling this in, verify each parameter is supported on the chosen mode + model per Operating Rule 12)
- `param_name`: <value> — <why>
- ...

**Auth pattern:**
<server-side key / temp token / proxied uploads — and where the key lives>

**Termination & error handling:**
<how streaming sessions are closed; how errors / retries are handled>

**Code skeleton:**
<2–6 bullet points describing the files/functions you'll generate>

**Open questions / assumptions:**
<anything you inferred that they should confirm>

Ready to proceed?
````

If they say yes, write the code. If they push back on any piece, revise the plan — don't just start coding around objections.

---

## 3. Feature selection guide (agent reference)

Use this to build the recommendation. Do not dump it on the user.

| Developer need | Parameter / approach |
|---|---|
| Speaker diarization | `speaker_labels: true` (pre-recorded, and streaming — streaming adds a `speaker_label` to each Turn event) |
| Automatic language detection | `language_detection: true` (pre-recorded; on streaming, only available on Universal-Streaming Multilingual — adds `language_code` + `language_confidence` to Turn events. **Not** supported on U3 Pro Streaming.) |
| Specific language | `language_code: "es"` etc. (pre-recorded and U3 Pro Streaming — biases the multilingual model toward one language) |
| Multilingual / code-switching | `speech_models: ["universal-3-5-pro"]` + `prompt` parameter — see [U3 Pro prompting guide](/pre-recorded-audio/universal-3-5-pro/prompting) |
| Domain-specific vocabulary | `keyterms_prompt: [...]` (pre-recorded: up to 1,000 terms with U3 Pro / 200 with U2; streaming: up to 100 terms, each ≤50 chars) |
| Medical domain | `domain: "medical-v1"` (pre-recorded *and* streaming; supported languages: en, es, de, fr) |
| PII redaction in text | `redact_pii: true` + `redact_pii_policies: [...]` + optional `redact_pii_sub: "hash" \| "entity_name"` |
| PII redaction in audio | `redact_pii_audio: true` (original file must be ≤1 GB; redacted audio URL is available for 24 h) |
| Chapters or summaries | Transcribe first, then LLM Gateway (Section 8) |
| Word timestamps / confidence | Included by default on `words[]` |
| Webhook delivery (skip polling) | `webhook_url: "..."` (Section 7) |
| Managed voice agent (speech-in / speech-out) | Voice Agent API (Section 10) — one WebSocket, no separate STT/LLM/TTS |
| Custom voice agent (your LLM + TTS) | Real-time STT + framework integration (Section 11) |
| Multilingual streaming | Universal-3 Pro Streaming (native code-switching; `language_code` to bias toward one language) |

---

## 4. API overview

- **REST base URL (US):** `https://api.assemblyai.com`
- **REST base URL (EU):** `https://api.eu.assemblyai.com`
- **Streaming WebSocket (Edge, default):** `wss://streaming.assemblyai.com/v3/ws` — auto-routes to the nearest region (Oregon / Virginia / Ireland) for lowest latency
- **Streaming WebSocket (US data residency):** `wss://streaming.us.assemblyai.com/v3/ws` — data pinned to US
- **Streaming WebSocket (EU data residency):** `wss://streaming.eu.assemblyai.com/v3/ws` — data pinned to EU
- **LLM Gateway (US):** `https://llm-gateway.assemblyai.com/v1/chat/completions`
- **LLM Gateway (EU):** `https://llm-gateway.eu.assemblyai.com/v1/chat/completions` — Claude and Gemini only; OpenAI and Qwen are US-only
- **Auth header:** `Authorization: YOUR_API_KEY` (no `Bearer`). Same header is used for REST, streaming WS upgrade, temp-token minting, and LLM Gateway
- **Content type:** `application/json` for submit/poll and LLM Gateway; `application/octet-stream` (raw binary) for `/v2/upload`

Core REST endpoints:
- `POST /v2/upload` — upload a local file (raw binary body, **not multipart**). Returns `{ "upload_url": "..." }`. Max 2.2 GB.
- `POST /v2/transcript` — submit a job. Returns transcript object with `id` and `status: "queued"`. Max 5 GB / 10 hours.
- `GET /v2/transcript/{id}` — poll. Statuses: `queued`, `processing`, `completed`, `error`.

Streaming:
- `wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro`
- `GET https://streaming.assemblyai.com/v3/token?expires_in_seconds=60` — mint a single-use temp token for browser/mobile clients. Optional `max_session_duration_seconds` (60–10800, defaults to 3 h) caps the downstream session length.

---

## 5. `speech_models` semantics

`speech_models` on pre-recorded requests is an **ordered fallback list**, not parallel execution. The first model in the array is tried; if it's unavailable (e.g., not yet rolled out to the account, or temporarily unhealthy), the next is used. A single transcript is produced by exactly one model.

Recommended default: `["universal-3-5-pro", "universal-2"]` — tries the latest model first, falls back to the stable predecessor.

On streaming, the parameter is **singular** (`speech_model=universal-3-5-pro`) — there is no fallback list. Easy to mix up.

---

## 6. Pre-recorded quick start

### SDK (recommended)

**Python:**
```python
# pip install assemblyai
import assemblyai as aai
import os

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],  # fallback handled by SDK
    speaker_labels=True,
)

transcript = aai.Transcriber(config=config).transcribe("https://assembly.ai/wildfires.mp3")
# Or a local path: .transcribe("./recording.wav")

if transcript.status == aai.TranscriptStatus.error:
    raise RuntimeError(transcript.error)
print(transcript.text)
```

**Node/JS:**
```javascript
// npm install assemblyai
import { AssemblyAI } from 'assemblyai';

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const transcript = await client.transcripts.transcribe({
  audio: 'https://assembly.ai/wildfires.mp3', // or a local file path / Buffer / stream
  speech_models: ["universal-3-5-pro", "universal-2"],
  speaker_labels: true,
});

if (transcript.status === 'error') throw new Error(transcript.error);
console.log(transcript.text);
```

The SDK handles upload, submit, and polling. You don't need to write the polling loop yourself.

### Raw HTTP (fallback — use only if SDK isn't an option)

**Upload a local file** (raw bytes, not multipart):
```bash
curl -X POST https://api.assemblyai.com/v2/upload \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  --data-binary @recording.wav
# -> { "upload_url": "https://cdn.assemblyai.com/upload/..." }
```

**Submit and poll (Python):**
```python
import os, time, requests

headers = {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}

submit = requests.post(
    "https://api.assemblyai.com/v2/transcript",
    headers=headers,
    json={
        "audio_url": "https://assembly.ai/wildfires.mp3",
        "speech_models": ["universal-3-5-pro", "universal-2"],
        "speaker_labels": True,
    },
)
transcript_id = submit.json()["id"]

while True:
    res = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers=headers,
    ).json()
    if res["status"] == "completed":
        print(res["text"]); break
    if res["status"] == "error":
        raise RuntimeError(res["error"])
    time.sleep(3)
```

Common optional params: `speaker_labels`, `language_detection`, `language_code`, `punctuate`, `format_text`, `redact_pii`, `redact_pii_audio`, `keyterms_prompt`, `webhook_url`, `prompt`.

---

## 7. Webhooks (skip polling)

Provide `webhook_url` on submit; AssemblyAI POSTs when the job finishes:

```json
{ "transcript_id": "5552493-16d8-42d8-8feb-c2a16b56f6e8", "status": "completed" }
```

Handler requirements:
- Return 2xx within **10 seconds**. Otherwise retried up to 10 times, 10s apart. 4xx is not retried.
- On receipt, call `GET /v2/transcript/{id}` to fetch the full result — the webhook payload doesn't include it.

Optional custom auth on your webhook: set `webhook_auth_header_name` and `webhook_auth_header_value` when submitting.

**Source IPs** (for allowlists): US `44.238.19.20`, EU `54.220.25.36`.

**Local dev note:** Webhook URLs must be publicly reachable. Use ngrok, Cloudflare Tunnel, or similar during development.

---

## 8. LLM Gateway (chapters, summaries, custom analysis)

LLM Gateway replaces both the deprecated transcript params (`auto_chapters`, `summarization`, `summary_model`, `summary_type`) and the legacy **LeMUR** API, which sunset on 2026-03-31. If a developer mentions LeMUR or `transcript_ids`, point them at LLM Gateway and the [migration guide](/llm-gateway/quickstart). Workflow:

1. Transcribe normally with `POST /v2/transcript`.
2. Once `status == "completed"`, POST to LLM Gateway with the transcript text (or paragraphs from `GET /v2/transcript/{id}/paragraphs` for chapter-style output):

```http
POST https://llm-gateway.assemblyai.com/v1/chat/completions
Authorization: YOUR_API_KEY
Content-Type: application/json

{
  "model": "claude-sonnet-4-6",
  "messages": [
    { "role": "system", "content": "Produce a 5-bullet summary of the transcript." },
    { "role": "user", "content": "<transcript.text here>" }
  ],
  "max_tokens": 1000
}
```

Model IDs are exact strings — see the [LLM Gateway Overview](/llm-gateway/quickstart) for the current list. Examples: `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`, `gpt-5.2`, `gpt-5.1`, `gpt-4.1`, `gemini-3.5-flash`, `gemini-2.5-pro`, `gemini-2.5-flash`, `qwen3-next-80b-a3b`. `claude-sonnet-4` by itself is **not** valid — always include the version suffix. EU region (`llm-gateway.eu.assemblyai.com`) supports Anthropic and Google only.

Do not submit with `auto_chapters` and `summarization` both enabled — the API rejects it (`Only one of the following models can be enabled at a time: auto_chapters, summarization.`). But the broader rule is simpler: **don't use either.**

---

## 9. Streaming — Universal-3.5 Pro

**WebSocket (default, Edge Routing):** `wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro`

For data residency, swap the host: `streaming.us.assemblyai.com` (US-pinned) or `streaming.eu.assemblyai.com` (EU-pinned). The default host auto-routes to the nearest region.

**Audio format:** PCM16 signed little-endian, mono, 16 kHz. Binary WebSocket frames, **50–1000 ms per chunk**, no faster than real-time. Phone audio (`encoding=pcm_mulaw`, `sample_rate=8000`) is sent as-is — don't upsample.

**Auth:**
- Server-side: `Authorization` header on the WS upgrade.
- Browser/mobile: mint a short-lived token server-side and pass it as `?token=<token>` (no Authorization header).

Mint a token:
```bash
curl -s "https://streaming.assemblyai.com/v3/token?expires_in_seconds=60" \
  -H "Authorization: $ASSEMBLYAI_API_KEY"
# { "token": "..." }
```
`expires_in_seconds` must be 1–600. Tokens are single-use per session.

### Server messages (JSON)

- `Begin` — `{ type, id, expires_at }`
- `SpeechStarted` — `{ type, timestamp, confidence }`
- `Turn` — `{ type, turn_order, end_of_turn, transcript, end_of_turn_confidence, words:[...], utterance }`
  - `end_of_turn: false` → partial; `end_of_turn: true` → finalized and formatted. Always read `transcript` for current text.
- `Termination` — `{ type, audio_duration_seconds, session_duration_seconds }`

### Client messages

- Binary PCM16 frames — audio.
- `{ "type": "Terminate" }` — graceful end. **Always send this when done.**
- `{ "type": "ForceEndpoint" }` — force current turn to end.
- `{ "type": "KeepAlive" }` — only needed if `inactivity_timeout` is set.
- `{ "type": "UpdateConfiguration", "keyterms_prompt": [...], "min_turn_silence": 100, "max_turn_silence": 1000 }` — adjust mid-session.

### SDK (recommended)

**Python:**
```python
# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    StreamingClient,
    StreamingClientOptions,
    StreamingEvents,
    StreamingParameters,
    TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = StreamingClient(
    StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"])
)
client.on(StreamingEvents.Turn, on_turn)
client.connect(StreamingParameters(sample_rate=16000, speech_model="universal-3-5-pro"))

# Feed 16 kHz mono PCM16 chunks (50–1000ms each) via client.stream(chunk)
# When finished:
client.disconnect(terminate=True)  # sends Terminate and closes cleanly
```

**Node/JS:**
```javascript
// npm install assemblyai
import { AssemblyAI } from 'assemblyai';

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const rt = client.streaming.transcriber({
  sampleRate: 16000,
  speechModel: 'universal-3-5-pro',
});

rt.on('turn', (turn) => {
  const tag = turn.end_of_turn ? 'FINAL' : 'partial';
  console.log(`${tag}: ${turn.transcript}`);
});
rt.on('error', (err) => console.error(err));

await rt.connect();
// rt.sendAudio(pcm16Buffer) for each 50–1000ms chunk
// When done:
await rt.close(); // sends Terminate and closes
```

### Raw WebSocket (fallback)

**Node.js (`ws`):**
```javascript
import WebSocket from 'ws';

const ws = new WebSocket(
  'wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro',
  { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
);

ws.on('open', () => {
  // Feed PCM16 16kHz mono chunks here, 50–1000ms each.
  // Example: audioStream.on('data', (chunk) => ws.send(chunk));
});

ws.on('message', (raw) => {
  const msg = JSON.parse(raw.toString());
  if (msg.type === 'Turn') {
    console.log(msg.end_of_turn ? `FINAL: ${msg.transcript}` : `partial: ${msg.transcript}`);
  }
});

function stop() {
  ws.send(JSON.stringify({ type: 'Terminate' })); // required!
}
```

**Python (`websockets`):**
```python
import asyncio, json, os, websockets

URL = "wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro"

async def run(audio_source):
    """audio_source: async iterator yielding 50–1000ms PCM16 chunks at 16kHz mono."""
    async with websockets.connect(
        URL,
        additional_headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    ) as ws:
        async def send_audio():
            async for chunk in audio_source:
                await ws.send(chunk)
            await ws.send(json.dumps({"type": "Terminate"}))

        async def recv_loop():
            async for raw in ws:
                msg = json.loads(raw)
                if msg["type"] == "Turn":
                    tag = "FINAL" if msg["end_of_turn"] else "partial"
                    print(f"{tag}: {msg['transcript']}")
                elif msg["type"] == "Termination":
                    return

        await asyncio.gather(send_audio(), recv_loop())

# asyncio.run(run(my_audio_iterator()))
```

---

## 10. Voice Agent API (managed speech-in / speech-out)

Use this when the developer wants a complete spoken AI agent — not just transcription. Single WebSocket, audio in and audio out, with STT + LLM + TTS + turn detection + tool calling all managed by AssemblyAI.

**Endpoint:** `wss://agents.assemblyai.com/v1/ws`

**Auth:** `Authorization: Bearer YOUR_API_KEY` — the Bearer prefix is **required** on this product (different from STT and LLM Gateway, which take the raw key). For browsers/mobile, mint a temp token instead and pass it as `?token=<token>`.

**Token endpoint (for browser/mobile clients):**
```bash
curl -s "https://agents.assemblyai.com/v1/token?expires_in_seconds=300&max_session_duration_seconds=8640" \
  -H "Authorization: Bearer $ASSEMBLYAI_API_KEY"
# { "token": "..." }
```
- `expires_in_seconds`: 1–600 (controls how long the token can be redeemed for)
- `max_session_duration_seconds`: 60–10800 (caps the resulting session; defaults to the 3-hour max)
- Tokens are **single-use** per session — get a fresh one for every reconnect (including `session.resume`).

**Audio format:** PCM16 mono **24 kHz**, **base64-encoded inside JSON events** (not raw binary frames — this is different from real-time STT). ~50 ms chunks (2,400 bytes) is fine; the server buffers continuously, exact chunk size doesn't matter.

### Lifecycle (the events that matter)

1. Client connects, sends `session.update` immediately (don't wait for `session.ready`):
   ```json
   {
     "type": "session.update",
     "session": {
       "system_prompt": "You are a helpful assistant.",
       "greeting": "Hi there! How can I help?",
       "input": {
         "format": { "encoding": "audio/pcm" },
         "keyterms": ["AssemblyAI", "Universal-3"],
         "turn_detection": {
           "vad_threshold": 0.5,
           "min_silence": 200,
           "max_silence": 1000,
           "interrupt_response": true
         }
       },
       "output": {
         "voice": "anna",
         "format": { "encoding": "audio/pcm" }
       },
       "tools": [ /* flat-schema tool defs, see step 5 */ ]
     }
   }
   ```
   Output `encoding` accepts `audio/pcm` (24 kHz, default), `audio/pcmu` (G.711 μ-law, 8 kHz), or `audio/pcma` (G.711 A-law, 8 kHz) — use the G.711 variants for telephony bridges (Twilio, etc.) so you don't have to resample.
2. Server replies with `session.ready` (capture `session_id` for `session.resume` if you reconnect within 30 s of a disconnect).
3. **Only after `session.ready`**, start streaming mic audio:
   ```json
   { "type": "input.audio", "audio": "<base64 PCM16 24kHz>" }
   ```
4. Server emits, in roughly this order, per turn:
   - `input.speech.started` / `input.speech.stopped` (VAD)
   - `transcript.user.delta` (partials) and `transcript.user` (final)
   - `reply.started`, `reply.audio` (multiple base64 PCM16 chunks — write directly into an output buffer at 24 kHz), `transcript.agent`, `reply.done`
   - **Field-name asymmetry:** `input.audio` carries audio in the `audio` field; `reply.audio` carries it in the `data` field. Easy to miss — copying `event["audio"]` from input handling will silently return nothing on output.
5. **Tool calls:** tool definitions in `session.tools` use a **flat** schema — *not* OpenAI's nested `{type: "function", function: {...}}` form:
   ```json
   {
     "type": "function",
     "name": "get_weather",
     "description": "Get the current weather for a city.",
     "parameters": {
       "type": "object",
       "properties": { "location": { "type": "string" } },
       "required": ["location"]
     }
   }
   ```
   Server sends `tool.call` with `{call_id, name, arguments}`. Accumulate the result locally, then send `tool.result` with the matching `call_id` *after* `reply.done` fires. If `reply.done.status == "interrupted"` (user barge-in), discard pending tool results.
6. **Resume after disconnect:** within 30 s, reconnect with a *new* token and send `session.resume` carrying the previous `session_id` to keep conversation context. After 30 s, start a new session.

### Voices

Voice IDs are **exact strings** — invented or remembered values silently fail. Pick from the catalog below or call `GET https://agents.assemblyai.com/v1/voices` for the live list.

**English** — `alba`, `eve`, `george`, `jane`, `jean`, `mary`, `michael` (🇺🇸 American), `anna`, `charles`, `paul`, `vera` (🇬🇧 British).

**Language-specific** (each speaks the named language plus English): `giovanni` (🇮🇹 Italian), `lola` (🇪🇸 Spanish), `juergen` (🇩🇪 German), `rafael` (🇵🇹 Portuguese), `estelle` (🇫🇷 French).

If the developer needs a voice not in this list, *don't* substitute a similar-sounding name — say so and ask. Pre-Voice-Agent-API names like `claire`, `dawn`, `josh`, `grace`, `pete` are **no longer valid** and will be rejected at `session.update`. The legacy voices `ivy`, `james`, `tyler`, `winter`, `bella`, `david`, `kyle`, `helen`, `martha`, `river`, `emma`, `victor`, `eleanor`, `arjun`, `dmitri`, `pierre`, `giulia`, `luca`, `lucia`, `mateo`, and `diego` have been removed — pick a current voice from the catalog above.

### Playback gotcha

Don't sleep-schedule audio chunks. Write each `reply.audio` PCM directly to an OS audio buffer (e.g., `sounddevice.OutputStream.write()`) — the OS drains at exactly 24 kHz and absorbs network jitter. Sleep-based timing drifts and produces pops/gaps.

On `reply.done.status == "interrupted"`, flush the output buffer (e.g., `speaker.abort(); speaker.start()`) so the user doesn't hear stale agent speech.

### Quickstart pattern (Python sketch)

```python
# pip install websockets sounddevice numpy
import asyncio, base64, json, os
import sounddevice as sd
import websockets

URL = "wss://agents.assemblyai.com/v1/ws"
SAMPLE_RATE = 24_000

async def main():
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a helpful assistant.",
                "greeting": "Hi! How can I help?",
                "output": {"voice": "anna"},
            },
        }))

        ready = asyncio.Event()
        loop = asyncio.get_running_loop()
        mic_q: asyncio.Queue = asyncio.Queue()

        def on_mic(indata, *_):
            if ready.is_set():
                loop.call_soon_threadsafe(mic_q.put_nowait, bytes(indata))

        async def pump_mic():
            while True:
                chunk = await mic_q.get()
                await ws.send(json.dumps({
                    "type": "input.audio",
                    "audio": base64.b64encode(chunk).decode(),
                }))

        with sd.InputStream(samplerate=SAMPLE_RATE, channels=1,
                            dtype="int16", callback=on_mic), \
             sd.OutputStream(samplerate=SAMPLE_RATE, channels=1,
                             dtype="int16") as speaker:
            asyncio.create_task(pump_mic())
            async for raw in ws:
                ev = json.loads(raw)
                if ev["type"] == "session.ready":
                    ready.set()
                elif ev["type"] == "reply.audio":
                    import numpy as np
                    speaker.write(np.frombuffer(base64.b64decode(ev["data"]), dtype=np.int16))
                elif ev["type"] == "reply.done" and ev.get("status") == "interrupted":
                    speaker.abort(); speaker.start()

asyncio.run(main())
```

For a complete worked example (MCP-tooled agent that talks back), see the [Voice Agent API quickstart](/voice-agents/voice-agent-api). For browser integration, see the [browser integration guide](/voice-agents/voice-agent-api/browser-integration).

### When to choose Voice Agent API vs Real-time STT + your own LLM/TTS

- **Voice Agent API (Section 10):** end-to-end conversational agents, fastest to ship, AssemblyAI manages the pipeline. Use when "speech in, speech out" is the whole product.
- **Real-time STT + framework (Section 11):** you need a specific LLM, a specific TTS provider, custom turn-detection logic, complex orchestration (LiveKit/Pipecat/Vapi/Vocode/Retell), or features the managed pipeline doesn't expose yet.

If they're not sure, ask: *do you want to choose your own LLM and TTS, or is a managed pipeline fine?* That single answer routes them.

---

## 11. Voice Agent framework configs (real-time STT + your own pipeline)

This section is for developers who are NOT using the Voice Agent API (Section 10) — they're wiring AssemblyAI Real-time STT into LiveKit, Pipecat, Vapi, Vocode, Retell, or similar, and bringing their own LLM and TTS.

The defaults will not be good enough. Common tuning:

- **`keyterms_prompt`** — pass proper nouns, product names, and domain terms. For dynamic values (usernames, order IDs), update mid-session via `UpdateConfiguration`.
- **Turn silence bounds** — `min_turn_silence` and `max_turn_silence` (ms). Lower values fire end-of-turn faster but risk cutting speakers off. Higher values reduce false finalizations. Form-filling and dictation use cases often want wider windows.
- **Multilingual** — Universal-3 Pro Streaming code-switches natively and ignores `end_of_turn_confidence_threshold`. Pass `language_code` to bias the model toward a single language.
- **Barge-in / false SpeechStarted** — ambient noise, TTS bleed-through, and PSTN echo can cause spurious `SpeechStarted` events. If the agent is interrupting itself, look here first. Framework-level knobs (e.g., LiveKit's `min_interruption_duration`) often complement, not replace, server-side tuning.
- **Phone audio** — 8 kHz mu-law (`pcm_mulaw` at 8000 Hz) should be sent as-is, not upsampled to 16 kHz. Upsampling degrades accuracy.

When the developer names one of these frameworks, ask about their specific turn-taking and interruption requirements before defaulting.

---

## 12. Browser patterns

**Never put the API key in client code.**

### Pre-recorded — proxy upload + submit through your server

```javascript
// Next.js route handler (server)
export async function POST(request) {
  const incoming = await request.formData();
  const file = incoming.get('file'); // Blob

  const upload = await fetch('https://api.assemblyai.com/v2/upload', {
    method: 'POST',
    headers: { authorization: process.env.ASSEMBLYAI_API_KEY },
    body: file.stream(),
    duplex: 'half',
  });
  const { upload_url } = await upload.json();

  const submit = await fetch('https://api.assemblyai.com/v2/transcript', {
    method: 'POST',
    headers: {
      authorization: process.env.ASSEMBLYAI_API_KEY,
      'content-type': 'application/json',
    },
    body: JSON.stringify({
      audio_url: upload_url,
      speech_models: ['universal-3-5-pro', 'universal-2'],
    }),
  });
  return Response.json(await submit.json());
}
```

### Streaming — server mints a temp token, client connects directly

```javascript
// Server
export async function GET() {
  const res = await fetch(
    'https://streaming.assemblyai.com/v3/token?expires_in_seconds=60',
    { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
  );
  return Response.json(await res.json()); // { token }
}
```

```javascript
// Client
const { token } = await fetch('/api/aai-token').then((r) => r.json());
const ws = new WebSocket(
  `wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro&token=${token}`,
);
ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'Turn') console.log(msg.transcript, msg.end_of_turn);
};
```

### Capturing mic audio in the browser

`MediaRecorder` does not emit PCM16. You need an `AudioWorklet` (preferred) or `ScriptProcessorNode` to:
1. Capture raw Float32 samples.
2. Downsample to 16 kHz.
3. Convert Float32 → Int16.
4. Send each ~50 ms chunk as a binary WS frame.

Reference: [AssemblyAI realtime-transcription-browser-js-example](https://github.com/AssemblyAI/realtime-transcription-browser-js-example).

---

## 13. Not supported / out of scope

If a developer asks for any of these, say so directly and propose the closest supported alternative. Do not improvise.

- **Real-time translation.** AssemblyAI transcribes, it doesn't translate. Suggest: transcribe with U3 Pro, then translate via LLM Gateway.
- **On-device / offline STT.** Cloud API only.
- **Speaker identification (matching voices to known people).** `speaker_labels` does diarization (Speaker A, B, C) but does not recognize specific individuals.
- **Standalone TTS.** Not an AssemblyAI product *as a separate API*. TTS is bundled into the Voice Agent API (Section 10) — if they need just-TTS, point them to a dedicated provider.
- **Voice activity detection as a standalone product.** VAD is internal to the streaming pipeline and surfaced via `SpeechStarted` / turn events, not exposed separately.

---

## 14. Error handling

### REST (pre-recorded)

- **401** — Missing/invalid Authorization, disabled account, or insufficient balance. Double-check there's no `Bearer` prefix.
- **Transcript `status: "error"`** — Read the `error` field on `GET /v2/transcript/{id}`.
- **Retries** — Exponential backoff on 5xx. For 429, respect the `Retry-After` header.
- **Limits** — `/v2/upload` max 2.2 GB; `/v2/transcript` max 5 GB / 10 hr per file.
- **Scoping** — An API key can only transcribe files uploaded under the same project.

### Streaming — handshake

- **HTTP 410** — The old `v2` streaming endpoint is deprecated. Upgrade to `/v3/ws`. This is an HTTP status on the upgrade request, not a WebSocket close code.

### Streaming — WebSocket close codes

| Code | Meaning |
|------|---------|
| `1008` | Unauthorized: missing/invalid Authorization or token |
| `3005` | Session cancelled (server-side error) |
| `3006` | Invalid message type / invalid JSON |
| `3007` | Audio chunk outside 50–1000 ms, or sent faster than real-time |
| `3008` | Session expired (3-hour cap) |
| `3009` | Too many concurrent sessions |

### Streaming gotchas

- `speech_model` (streaming, singular) vs `speech_models` (pre-recorded, plural). Don't mix up.
- On U3 Pro Streaming, `end_of_turn_confidence_threshold` is silently ignored; `language_code` biases the multilingual model toward a single language.
- Always send `{ "type": "Terminate" }` when finished. An abandoned session stays billable until the 3-hour cap (`3008`).
- Chunk size matters: frames outside 50–1000 ms will close the socket with `3007`.

---

## 15. Quick-reference gotchas

- No `Bearer` prefix on the Authorization header — *except* for the Voice Agent API (Section 10), which requires `Authorization: Bearer ...`.
- `speech_models` is **optional** on pre-recorded submits — it defaults to `["universal-3-5-pro", "universal-2"]` and is an **ordered fallback list** when provided.
- `/v2/upload` takes **raw binary**, not multipart.
- Webhook handlers must return 2xx in ≤10 seconds.
- Local webhook development needs a public tunnel (ngrok, Cloudflare Tunnel).
- Browser code never holds the API key. Proxy uploads, or mint temp tokens for streaming.
- Always `Terminate` streaming sessions.
- Don't use `auto_chapters`, `summarization`, `summary_model`, `summary_type`. Use LLM Gateway.
- Medical mode is `domain: "medical-v1"` (pre-recorded body param / streaming query param). The legacy `medical_mode` flag is **not** the right name.
- LLM Gateway model IDs are exact and versioned (e.g., `claude-sonnet-4-6`, `gpt-5.2`, `gemini-2.5-pro`). Shorthand like `claude-sonnet-4` is invalid.
- Phone audio stays at native 8 kHz mu-law (`encoding=pcm_mulaw`) — don't upsample.
- EU customers use `api.eu.assemblyai.com`, `streaming.eu.assemblyai.com`, and `llm-gateway.eu.assemblyai.com`. The default streaming host (`streaming.assemblyai.com`) is **Edge Routing**, not US-pinned — use `streaming.us.assemblyai.com` if you need data residency guarantees on the US side.
- Speech-model values are **raw strings** in the SDKs (`"universal-3-5-pro"`, `"universal-2"`). Enum aliases like `aai.SpeechModel.universal_3_pro` do **not** exist — agents that hallucinate them produce code that imports cleanly and fails at runtime.
- LeMUR has fully sunset (2026-03-31). Don't generate code that calls LeMUR endpoints or passes `transcript_ids` to a chat-completions API — use LLM Gateway with the transcript text in `messages` instead.

---

<a id="deployment-docs-deployment-mdx"></a>

# Deployment

URL: https://www.assemblyai.com/docs/deployment
Source: docs/deployment.mdx
Navigation: Additional pages
Description: Take AssemblyAI to production with these best practices.

Receive transcription results in real-time instead of polling.
  
  
    Manage Usage limits and API keys for your account.
  
  
    Meet data residency requirements for your region.
  
  
    Access our Trust Center and security documentation. 
  
  
    Learn about common errors and handle them gracefully.
  
  
    Stuck with your implementation? Get in touch with our team.

---

<a id="select-the-eu-region-for-eu-data-residency-docs-pre-recorded-audio-guides-how-to-use-the-eu-endpoint-mdx"></a>

# Select The EU Region for EU Data Residency

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/how_to_use_the_eu_endpoint
Source: docs/pre-recorded-audio/guides/how_to_use_the_eu_endpoint.mdx
Navigation: Additional pages
Description: Select The EU Region for EU Data Residency documentation.

In this guide, we'll show you how to configure your code to utilise the EU endpoint when interacting directly with the API, as well as across our SDKs.

  Streaming is now available in EU-West via `streaming.eu.assemblyai.com`

## Get Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

## Python (Requests)

The only difference here is adding the `eu.` subdomain to the base URL. Everything else in your request code stays the same:

```python
# US endpoint (default)
url = "https://api.assemblyai.com"

# EU endpoint - just change the base URL
url = "https://api.eu.assemblyai.com"
```

## Python SDK

To set the EU endpoint as the base URL when using the Python SDK, add the following line of code before setting the `Transcriber` object:

```python
import assemblyai as aai

aai.settings.base_url = "https://api.eu.assemblyai.com"  # Set the base URL to the EU endpoint
```

## JavaScript (Axios)

Similar to the requests library for Python, the only difference here is adding the `eu.` subdomain to the base URL as follows:

```javascript
// US endpoint (default)
const url = "https://api.assemblyai.com";

// EU endpoint - just change the base URL
const url = "https://api.eu.assemblyai.com";
```

## JavaScript SDK

To set the EU endpoint as the base URL when using the JavaScript SDK, set the `baseUrl` parameter in your AssemblyAI client as follows:

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: process.env.ASSEMBLYAI_API_KEY,
  baseUrl: "https://api.eu.assemblyai.com", // Set the baseUrl to the EU endpoint
});
```

## Streaming Speech-to-Text

For streaming transcription, use the EU streaming endpoint `streaming.eu.assemblyai.com`.

### Python SDK (Streaming)

To use the EU streaming endpoint with the Python SDK:

```python
import assemblyai as aai
from assemblyai.streaming.v3 import StreamingClient, StreamingClientOptions

client = StreamingClient(
    StreamingClientOptions(
        api_key="YOUR_API_KEY",
        api_host="streaming.eu.assemblyai.com"  # EU streaming endpoint
    )
)
```

### JavaScript SDK (Streaming)

To use the EU streaming endpoint with the JavaScript SDK:

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "YOUR_API_KEY",
});

const transcriber = client.streaming.transcriber({
  sampleRate: 16_000,
  apiHost: "streaming.eu.assemblyai.com", // EU streaming endpoint
});
```

### WebSocket (Direct)

When connecting directly via WebSocket, use the EU streaming base URL:

```text
# US endpoint (default)
wss://streaming.assemblyai.com/v3/ws

# EU endpoint
wss://streaming.eu.assemblyai.com/v3/ws
```

If you encounter any issues or have any questions, reach out to our [Support team.](https://www.assemblyai.com/contact/support)

---

<a id="identifying-speakers-in-audio-recordings-docs-pre-recorded-audio-guides-identifying-speakers-in-audio-recordings-mdx"></a>

# Identifying speakers in audio recordings

URL: https://www.assemblyai.com/docs/pre-recorded-audio/guides/identifying-speakers-in-audio-recordings
Source: docs/pre-recorded-audio/guides/identifying-speakers-in-audio-recordings.mdx
Navigation: Additional pages
Description: Add speaker labels to your transcript

When applying the [Speaker Diarization model](/pre-recorded-audio/label-speakers), the transcription not only contains the text but also includes speaker labels, enhancing the overall structure and organization of the output.

In this step-by-step guide, you'll learn how to apply the model. In short, you have to send the `speaker_labels` parameter in your request, and then find the results inside a field called `utterances`.

## Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can [sign up](https://assemblyai.com/dashboard/signup) for a free account and get your API key from your dashboard.

The complete source code for this guide can be viewed [here](https://github.com/AssemblyAI-Community/docs-snippets/tree/main/speakers).

Here is an audio example for this guide:

```bash
https://assembly.ai/wildfires.mp3
```

## Step-by-step instructions

Install the SDK.

  

Create a new file and
request.

  

```bash
pip install requests
```

  

```bash
pip install -U assemblyai
```

  

No additional packages are required. This guide uses the built-in [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API).

  

Import the `assemblyai` package and set the API key.

  

Set up the API endpoint and headers. The headers should include your API
key.

  

```python
base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}
```

  

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"
```

  

```javascript
const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};
```

  

Create a `TranscriptionConfig` with `speaker_labels` set to `True`.

  

Upload your local file to the AssemblyAI API.

  

```python
with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]
```

  

```python
# highlight-next-line
config = aai.TranscriptionConfig(speaker_labels=True)
```

  

```javascript
const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;
```

  

Create a `Transcriber` object and pass in the configuration.

  

Use the `upload_url` returned by the AssemblyAI API to create a JSON payload
containing the `audio_url` parameter and the `speaker_labels` paramter set to
`True`.

  

```python
data = {
    "audio_url": upload_url,
    "speaker_labels": True
}
```

  

```python
transcriber = aai.Transcriber(config=config)
```

  

```javascript
const data = {
  audio_url: uploadUrl,
  speaker_labels: true,
};
```

  

Use the `Transcriber` object's transcribe method and pass in the audio file's
path as a parameter. The `transcribe` method saves the results of the transcription to the `Transcriber` object's `transcript` attribute.

  

Make a `POST` request to the AssemblyAI API endpoint with the payload and
headers.

  

```python
url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)
```

  

```python
FILE_URL = "https://assembly.ai/wildfires.mp3"

transcript = transcriber.transcribe(FILE_URL)
```

  

```javascript
const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();
```

  

You can access the speaker label results through the transcription object's `utterances` attribute.

  

After making the request, you'll receive an ID for the transcription. Use it
to poll the API every few seconds to check the status of the transcript job.
Once the status is `completed`, you can retrieve the transcript from the API
response, using the `utterances` key to access the results.

  

```python expandable
transcript_id = response.json()['id']
polling_endpoint = f"https://api.assemblyai.com/v2/transcript/{transcript_id}"

while True:
  transcription_result = requests.get(polling_endpoint, headers=headers).json()

  if transcription_result['status'] == 'completed':
    # when the transcript is complete, extract all utterances from the response
    transcript_text = transcription_result['text']
    utterances = transcription_result['utterances']

    # For each utterance, print its speaker and what was said
    for utterance in utterances:
        speaker = utterance['speaker']
        text = utterance['text']
        print(f"Speaker {speaker}: {text}")

    break

  elif transcription_result['status'] == 'error':
    raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

  else:
    time.sleep(3)
```

  

```python
# extract all utterances from the response
utterances = transcript.utterances

# For each utterance, print its speaker and what was said
for utterance in utterances:
  speaker = utterance.speaker
  text = utterance.text
  print(f"Speaker {speaker}: {text}")
```

  

```javascript expandable
const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  let res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    const utterances = transcriptionResult.utterances;

    // Iterate through each utterance and print the speaker and the text they spoke
    for (const utterance of utterances) {
      const speaker = utterance.speaker;
      const text = utterance.text;
      console.log(`Speaker ${speaker}: ${text}`);
    }

    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

  

## Understanding the response

The speaker label information is included in the `utterances` key of the response. Each utterance object in the list includes a `speaker` field, which contains a string identifier for the speaker (e.g., "A", "B", etc.). The utterances list also contains a `text` field for each utterance containing the spoken text, and `confidence` scores both for utterances and their individual words.

```js expandable
{
      utterances: [
        {
          confidence: 0.7246133333333334,
          end: 3738,
          speaker: "A",
          start: 570,
          text: "Um hey, Erica.",
          words: [
            {
              text: "Um",
              start: 570,
              end: 1120,
              confidence: 0.42915,
              speaker: "A",
            },
            {
              text: "hey,",
              start: 2690,
              end: 3054,
              confidence: 0.98465,
              speaker: "A",
            },
            {
              text: "Erica.",
              start: 3092,
              end: 3738,
              confidence: 0.76004,
              speaker: "A",
            },
          ],
        },
        {
          confidence: 0.6015349999999999,
          end: 4430,
          speaker: "B",
          start: 3834,
          text: "One in.",
          words: [
            {
              text: "One",
              start: 3834,
              end: 4094,
              confidence: 0.25,
              speaker: "B",
            },
            {
              text: "in.",
              start: 4132,
              end: 4430,
              confidence: 0.95307,
              speaker: "B",
            },
          ],
        },
      ],
    
}
```

For more information, see the [Speaker Diarization model documentation](/pre-recorded-audio/label-speakers#specifying-the-number-of-speakers) or see the [API reference](/pre-recorded-audio/api-reference/transcripts/submit).

## Specifying the number of speakers

You can provide the optional parameter `speakers_expected`, that can be used to specify the expected number of speakers in an audio file.

[API/Model Reference](/pre-recorded-audio/label-speakers#specifying-the-number-of-speakers)

## Conclusion

Automatically identifying different speakers from an audio recording, also called **speaker diarization**, is a multi-step process. It can unlock additional value from many genres of recording, including conference call transcripts, broadcast media, podcasts, and more. You can learn more about use cases for speaker diarization and the underlying research from the [AssemblyAI blog](https://www.assemblyai.com/blog/speaker-diarization-speaker-labels-for-mono-channel-files).

---

<a id="streaming-v2-legacy-docs-pre-recorded-audio-legacy-streaming-mdx"></a>

# Streaming v2 (Legacy)

URL: https://www.assemblyai.com/docs/pre-recorded-audio/legacy-streaming
Source: docs/pre-recorded-audio/legacy-streaming.mdx
Navigation: Additional pages
Description: Transcribe live audio with Streaming Speech-to-Text

This documentation is for a previous, legacy version, of our Real-time STT
  feature. If you are looking for Real-time STT please use our latest Universal
  Streaming model, which you can learn more about in the [Universal Streaming documentation](/streaming)

  Need to migrate to Universal-Streaming from this old model? See [this
  migration guide](/streaming/guides/v2_to_v3_migration)!

AssemblyAI's Streaming Speech-to-Text (STT) allows you to transcribe live audio streams with high accuracy and low latency. By streaming your audio data to our secure WebSocket API, you can receive transcripts back within a few hundred milliseconds.

  By default, Universal-Streaming is set to transcribe English audio. If you'd
  like to enable multilingual streaming (support for English, Spanish, French,
  German, Italian, and Portuguese), enable [multilingual
  transcription](/streaming/getting-started/transcribe-streaming-audio) instead.

## Getting started

Check out our getting started guides:

  **Getting Started Guides**

- [Python](/streaming/getting-started/transcribe-streaming-audio)
- [JavaScript](/streaming/getting-started/transcribe-streaming-audio)

If your programming language isn't supported yet, see the WebSocket API:

- [Streaming API reference](/streaming/api-spec/streaming-websocket)

## Audio requirements

The audio format must conform to the following requirements:

- PCM16 or Mu-law encoding (See [Specify the encoding](#specify-the-encoding))
- A sample rate that matches the value of the supplied `sample_rate` parameter
- Single-channel
- 100 to 2000 milliseconds of audio per message

  Audio segments with a duration between 100 ms and 450 ms produce the best
  results in transcription accuracy.

## Specify the encoding

By default, transcriptions expect [PCM16 encoding](http://trac.ffmpeg.org/wiki/audio%20types). If you want to use [Mu-law encoding](http://trac.ffmpeg.org/wiki/audio%20types), you must set the `encoding` parameter to `pcm_mulaw`:

```python {2}
ws = websocket.WebSocketApp(
    f'wss://api.assemblyai.com/v2/realtime/ws?sample_rate={SAMPLE_RATE}&encoding=pcm_mulaw',
    ...,
)
```

By default, transcriptions expect [PCM16 encoding](http://trac.ffmpeg.org/wiki/audio%20types). If you want to use [Mu-law encoding](http://trac.ffmpeg.org/wiki/audio%20types), you must set the `encoding` parameter to `aai.AudioEncoding.pcm_mulaw`:

```python {3}
transcriber = aai.RealtimeTranscriber(
    ...,
    encoding=aai.AudioEncoding.pcm_mulaw
)
```

By default, transcriptions expect [PCM16 encoding](http://trac.ffmpeg.org/wiki/audio%20types). If you want to use [Mu-law encoding](http://trac.ffmpeg.org/wiki/audio%20types), you must set the `encoding` parameter to `pcm_mulaw`:

```javascript {2}
const ws = new WebSocket(
    `wss://api.assemblyai.com/v2/realtime/ws?sample_rate=${SAMPLE_RATE}&encoding=pcm_mulaw`,
    ...,
  );
```

By default, transcriptions expect [PCM16 encoding](http://trac.ffmpeg.org/wiki/audio%20types). If you want to use [Mu-law encoding](http://trac.ffmpeg.org/wiki/audio%20types), you must set the `encoding` parameter to `'pcm_mulaw'`:

```javascript {3}
const rt = client.realtime.transcriber({
  ...,
  encoding: 'pcm_mulaw'
})
```

| Encoding            | SDK Parameter | Description                      |
| ------------------- | ------------- | -------------------------------- |
| **PCM16** (default) | `'pcm_s16le'` | PCM signed 16-bit little-endian. |
| **Mu-law**          | `'pcm_mulaw'` | PCM Mu-law.                      |

| Encoding            | SDK Parameter                 | Description                      |
| ------------------- | ----------------------------- | -------------------------------- |
| **PCM16** (default) | `aai.AudioEncoding.pcm_s16le` | PCM signed 16-bit little-endian. |
| **Mu-law**          | `aai.AudioEncoding.pcm_mulaw` | PCM Mu-law.                      |

| Encoding            | SDK Parameter | Description                      |
| ------------------- | ------------- | -------------------------------- |
| **PCM16** (default) | `'pcm_s16le'` | PCM signed 16-bit little-endian. |
| **Mu-law**          | `'pcm_mulaw'` | PCM Mu-law.                      |

| Encoding            | SDK Parameter | Description                      |
| ------------------- | ------------- | -------------------------------- |
| **PCM16** (default) | `'pcm_s16le'` | PCM signed 16-bit little-endian. |
| **Mu-law**          | `'pcm_mulaw'` | PCM Mu-law.                      |

## Add custom vocabulary

You can add up to 2500 characters of custom vocabulary to boost their transcription probability.

For this, create a list of strings, URL encode them and add them to the URL:

```python {2,4}
list = ["aws", "azure", "google cloud"]
word_boost = urllib.parse.quote(json.dumps(list))
ws = websocket.WebSocketApp(
    f'wss://api.assemblyai.com/v2/realtime/ws?sample_rate={SAMPLE_RATE}&encoding=pcm_mulaw&word_boost={word_boost}',
    ...,
)
```

For this, create a list of strings and set the `word_boost` parameter:

```python {3}
transcriber = aai.RealtimeTranscriber(
    ...,
    word_boost=["aws", "azure", "google cloud"]
)
```

For this, create a list of strings, URL encode them and add them to the URL:

```javascript {2,4}
const list = ["aws", "azure", "google cloud"];
const wordBoost = encodeURIComponent(JSON.stringify(list));
const ws = new WebSocket(
    `wss://api.assemblyai.com/v2/realtime/ws?sample_rate=${SAMPLE_RATE}&word_boost=${wordBoost}`,
    ...,
  );
```

For this, create a list of strings and set the `wordBoost` parameter:

```javascript {3}
const rt = client.realtime.transcriber({
  ...,
  wordBoost:['aws', 'azure', 'google cloud']
})
```

  If you're not using one of the SDKs, you must ensure that the `word_boost`
  parameter is a JSON array that is URL encoded. See this [code
  example](/streaming/getting-started/transcribe-streaming-audio#adding-custom-vocabulary).

## Authenticate with a temporary token

If you need to authenticate on the client, you can avoid exposing your API key by using temporary authentication tokens.
You should generate this token on your server and pass it to the client.

To generate a temporary token, make a POST request to the temporary token endpoint.

Use the `expires_in` parameter to specify the session duration in seconds for which the token will remain valid.

```python
token = requests.post(
    'https://api.assemblyai.com/v2/realtime/token',
    headers={
        'Authorization': '<apiKey>',
        'Content-Type': 'application/json'
    },
    json={'expires_in': 60}
).json()['token']
```

To generate a temporary token, call `aai.RealtimeTranscriber.create_temporary_token()`.

Use the `expires_in` parameter to specify the session duration in seconds for which the token will remain valid.

```python
token = aai.RealtimeTranscriber.create_temporary_token(
    expires_in=60
)
```

To generate a temporary token, make a POST request to the temporary token endpoint.

Use the `expires_in` parameter to specify the session duration in seconds for which the token will remain valid.

```javascript
const res = await fetch("https://api.assemblyai.com/v2/realtime/token", {
  method: "POST",
  headers: {
    Authorization: "<apiKey>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ expires_in: 60 }),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const data = await res.json();
const token = data.token;
```

To generate a temporary token, call `client.realtime.createTemporaryToken()`.

Use the `expires_in` parameter to specify the session duration in seconds for which the token will remain valid.

```javascript
const token = await client.realtime.createTemporaryToken({ expires_in: 60 });
```

To generate a temporary token, make a POST request to the temporary token endpoint.

Use the `expires_in` parameter to specify the session duration in seconds for which the token will remain valid.

```
curl -X POST https://api.assemblyai.com/v2/realtime/token \
     -H "Authorization: <apiKey>" \
     -H "Content-Type: application/json" \
     -d '{
  "expires_in": 60
}'
```

The expiration time must be a value between 60 and 360000 seconds.

The client should retrieve the token from the server and use the token to authenticate the transcriber.

  Each token has a one-time use restriction and can only be used for a single
  session. Any usage associated with a temporary token will be attributed to the
  API key that generated it.

  
To use it, specify the `token` parameter as a query parameter in the WebSocket URL.

```python {2}
ws = websocket.WebSocketApp(
    f'wss://api.assemblyai.com/v2/realtime/ws?sample_rate={SAMPLE_RATE}&token={token}',
    ...,
)
```

  
To use it, specify the `token` parameter when initializing the streaming transcriber.

```python {3}
transcriber = aai.RealtimeTranscriber(
    ...,
    token=token
)
```

  
To use it, specify the `token` parameter as a query parameter in the WebSocket URL.

```javascript {2}
const ws = new WebSocket(
    `wss://api.assemblyai.com/v2/realtime/ws?sample_rate=${SAMPLE_RATE}&token=${token}`,
    ...,
  );
```

  
To use it, specify the `token` parameter when initializing the streaming transcriber.

```javascript {3}
const rt = new RealtimeTranscriber({
  ...,
  token
})
```

## Manually end current utterance

To manually end an utterance, send the following message:

```python
message = {"force_end_utterance": True}
ws.send(json.dumps(message))
```

To manually end an utterance, call `force_end_utterance()`:

```python
transcriber.force_end_utterance()
```

To manually end an utterance, send the following message:

```javascript
const message = JSON.stringify({ force_end_utterance: true });
ws.send(message);
```

To manually end an utterance, call `forceEndUtterance()`:

```javascript
rt.forceEndUtterance();
```

Manually ending an utterance immediately produces a final transcript.

## Configure the threshold for automatic utterance detection

You can configure the threshold for how long to wait before ending an utterance.

After the session has started, you can change the threshold by sending the following message:

```python {2}
message = {"end_utterance_silence_threshold": 300}
ws.send(json.dumps(message))
```

To change the threshold, you can specify the `end_utterance_silence_threshold` parameter when initializing the streaming transcriber.

After the session has started, you can change the threshold by calling `configure_end_utterance_silence_threshold()`.

```python {3,7}
transcriber = aai.RealtimeTranscriber(
    ...,
    end_utterance_silence_threshold=500
)

# after connecting
transcriber.configure_end_utterance_silence_threshold(300)
```

After the session has started, you can change the threshold by sending the following message:

```javascript {2}
const message = JSON.stringify({ end_utterance_silence_threshold: 300 });
ws.send(message);
```

To change the threshold, you can specify the `endUtteranceSilenceThreshold` parameter when initializing the streaming transcriber.

After the session has started, you can change the threshold by calling `configureEndUtteranceSilenceThreshold()`.

```javascript {3,8}
const rt = client.realtime.transcriber({
  ...,
  endUtteranceSilenceThreshold: 500
})

// after connecting
rt.configureEndUtteranceSilenceThreshold(300)
```

  By default, Streaming Speech-to-Text ends an utterance after 700 milliseconds
  of silence. You can configure the duration threshold any number of times
  during a session after the session has started. The valid range is between 0
  and 20000.

## Disable partial transcripts

If you're only using the final transcript, you can disable partial transcripts to reduce network traffic.

To disable partial transcripts, set the `disable_partial_transcripts` parameter to `true`.

```python {2}
ws = websocket.WebSocketApp(
    f'wss://api.assemblyai.com/v2/realtime/ws?sample_rate={SAMPLE_RATE}&disable_partial_transcripts=true',
    ...,
)
```

To disable partial transcripts, set the `disable_partial_transcripts` parameter to `True`.

```python {3}
transcriber = aai.RealtimeTranscriber(
    ...,
    disable_partial_transcripts=True
)
```

To disable partial transcripts, set the `disable_partial_transcripts` parameter to `true`.

```javascript {2}
const ws = new WebSocket(
    `wss://api.assemblyai.com/v2/realtime/ws?sample_rate=${SAMPLE_RATE}&disable_partial_transcripts=true`,
    ...,
  );
```

To disable partial transcripts, set the `disablePartialTranscripts` parameter to `true`.

```javascript {3}
const rt = client.realtime.transcriber({
  ...,
  disablePartialTranscripts: true
})
```

## Enable extra session information

If you enable extra session information, the client receives a `SessionInformation` message right before receiving the session termination message.

To enable it, add the message handler and configure the `on_extra_session_information` parameter.

```python {2,8}
# Define a callback to handle the session information message
if msg_type == 'SessionInformation':
    audio_duration_seconds = msg.get('audio_duration_seconds')
    print(f"Audio duration: {audio_duration_seconds}")

# Configure the on_extra_session_information parameter
ws = websocket.WebSocketApp(
    f'wss://api.assemblyai.com/v2/realtime/ws?sample_rate={SAMPLE_RATE}&enable_extra_session_information=true',
    ...,
)
```

If you enable extra session information, the client receives a `RealtimeSessionInformation` message right before receiving the session termination message.

To enable it, define a callback function to handle the event and configure the `on_extra_session_information` parameter.

```python {2,8}
# Define a callback to handle the session information message
def on_extra_session_information(data: aai.RealtimeSessionInformation):
    print(data.audio_duration_seconds)

# Configure the RealtimeTranscriber
transcriber = aai.RealtimeTranscriber(
    ...,
    on_extra_session_information=on_extra_session_information,
)
```

If you enable extra session information, the client receives a `SessionInformation` message right before receiving the session termination message.

To enable it, add the message handler and configure the `on_extra_session_information` parameter.

```javascript {2,9}
// Define a callback to handle the session information message
if (msgType === 'SessionInformation') {
      const audioDurationSeconds = msg.audio_duration_seconds;
      console.log(`Audio duration: ${audioDurationSeconds}`);
    }

// Configure the on_extra_session_information parameter
const ws = new WebSocket(
    `wss://api.assemblyai.com/v2/realtime/ws?sample_rate=${SAMPLE_RATE}&enable_extra_session_information=true`,
    ...,
  );
```

The client receives a `SessionInformation` message right before receiving the session termination message.
Handle the `session_information` event to receive the message.

```javascript {5}
const rt = client.realtime.transcriber({
  ...
})

rt.on('session_information', (info: SessionInformation) => console.log(info));
```

## Best practices

Here are some best practices to get the best results from Streaming Speech-to-Text:

### Audio quality

- Use a sample rate of at least 16000 Hz for better transcription accuracy. Higher quality audio input generally leads to better results.
- For noisy environments, see our [Cookbook example](/streaming/guides/noise_reduction_streaming) for implementing noise reduction in your streaming pipeline.

### Connection management

- Keep the WebSocket connection open for the entire duration of a session instead of frequently reconnecting. Reconnecting adds latency overhead and can impact real-time performance.
- For client-side implementations of our Streaming API, we recommend first creating a temporary authentication token using a [temporary token](/streaming/getting-started/transcribe-streaming-audio#authenticate-with-a-temporary-token) to avoid exposing your API key.

### Optimizing for latency

You can optimize for lower latency in several ways:

- Use partial transcripts instead of waiting for final transcripts. Partial transcripts provide faster feedback but may be less accurate. See our [Cookbook example](/streaming/getting-started/transcribe-streaming-audio) for more information.
- [Configure the threshold for silence between speaker turns](/streaming/getting-started/transcribe-streaming-audio#configure-the-threshold-for-automatic-utterance-detection) using `end_utterance_silence_threshold` to control when utterances are finalized.
- If you don't need real-time feedback, you can disable partial transcripts to reduce network traffic.

## Learn more

To learn about using Streaming Speech-to-Text, see the following resources:

- [Blog post: Automatically Transcribe Zoom Calls in Real Time](https://www.assemblyai.com/blog/how-to-automatically-transcribe-zoom-calls/)
- [Blog post: Transcribe Twilio Phone Calls](https://www.assemblyai.com/blog/transcribe-twilio-phone-calls-in-real-time-with-assemblyai/)
- [Blog post: Real Time Speech Recognition with Python and PyAudio](https://www.assemblyai.com/blog/real-time-speech-recognition-with-python/)
- [GitHub: End-to-end examples](https://github.com/AssemblyAI-Community/docs-snippets/tree/main/real-time)
- [GitHub: Cookbook examples](https://github.com/AssemblyAI/cookbook/tree/master/streaming-stt)
- [GitHub: Use Express.js for Streaming Speech-to-Text](https://github.com/AssemblyAI/realtime-transcription-browser-js-example)
- [Streaming API reference](/streaming/api-spec/streaming-websocket)

---

<a id="universal-3-5-pro-docs-pre-recorded-audio-universal-3-5-pro-mdx"></a>

# Universal-3.5 Pro

URL: https://www.assemblyai.com/docs/pre-recorded-audio/universal-3-5-pro
Source: docs/pre-recorded-audio/universal-3-5-pro.mdx
Navigation: Additional pages
Description: Set up and configure Universal-3.5 Pro for pre-recorded audio transcription.

Universal-3.5 Pro is our latest Voice AI model. It delivers:

- State-of-the-art transcription accuracy across [18 languages](/pre-recorded-audio/universal-3-5-pro#support-for-99-languages)
- Strong performance on accented English and seamless [code-switching](/pre-recorded-audio/universal-3-5-pro#code-switching) between supported languages
- A more intelligent [contextual prompting](/pre-recorded-audio/universal-3-5-pro#contextual-prompting) that reliably follows the context you provide — from a single topic label to a full description of your audio

## Quickstart

Get started with Universal-3.5 Pro using the code below. This example transcribes a pre-recorded audio file and prints the transcript text to your terminal.

    
    Install the required library

    ```bash
    pip install requests
    ```

    
    
    Create a new file `main.py` and paste the code below. Replace `<YOUR_API_KEY>` with your API key.
    
    
    Run with `python main.py`.
    

```python  expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assembly.ai/new-approved-medication",
    "language_detection": True,
    "speech_models": ["universal-3-5-pro"]
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_response = response.json()
transcript_id = transcript_response["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript["text"])
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

    
    Install the required library

    ```bash
    pip install "assemblyai>=1.0.0"
    ```

    
    
    Create a new file `main.py` and paste the code below. Replace `<YOUR_API_KEY>` with your API key.
    
    
    Run with `python main.py`.
    

```python 
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_file = "https://assembly.ai/new-approved-medication"

config = aai.TranscriptionConfig(
  speech_models=["universal-3-5-pro"],
  language_detection=True,
)

transcript = aai.Transcriber().transcribe(audio_file, config)

print(transcript.text)
```

    
    Create a new file `index.mjs` and paste the code below. Replace `<YOUR_API_KEY>` with your API key.
    
    
    Run with `node index.mjs`.
    

```javascript  expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assembly.ai/new-approved-medication",
  language_detection: true,
  speech_models: ["universal-3-5-pro"],
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

    
    Install the required library

    ```bash
    npm install assemblyai
    ```

    
    
    Create a new file `index.mjs` and paste the code below. Replace `<YOUR_API_KEY>` with your API key.
    
    
    Run with `node index.mjs`.
    

```javascript 
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assembly.ai/new-approved-medication";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log(transcript.text);
};

run();
```

## Code switching

Universal-3.5 Pro natively handles code switching across 18 languages — _no prompt configuration needed_. When speakers shift mid-sentence between languages like English and Spanish, French, Hindi or Mandarin, the model follows seamlessly, preserving exactly what was said without translating everything into a single language.

See Universal-3.5 Pro code switching in action.

### **English \<\> French**

```txt
I said something like, j'ai dit à mes étudiants que it's time to really pay attention to what the idea of code-switching is.
```

### **English \<\> Hindi**

```txt
मेरा रुकने का तो बहुत मन है, but I have an exam to give tomorrow.
```

### **English \<\> Mandarin**

```txt
But this sentence, 我父母不工作了, you can see the 了 at the end of the sentence indicates that the situation now, my parents don't work, is different from what it was before.
```

## Contextual prompting

Universal-3.5 Pro is highly accurate out of the box, but for challenging audio like short clips with limited context, noisy environments, or audio with very niche references, providing a brief description in the `prompt` parameter can meaningfully improve accuracy.

For example, this is a 2-second clip from a League of Legends pro interview:

Without prompt:

```txt
And so look who I've been a dear.
```

With prompt:

```txt
In solo queue, I ban Azir.
```

```python {11} expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assembly.ai/prompt-8",
    "language_detection": True,
    "speech_models": ["universal-3-5-pro"],
    "prompt": "League of Legend roles"
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_response = response.json()
transcript_id = transcript_response["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript["text"])
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```python {10} expandable
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_file = "https://assembly.ai/prompt-8"

config = aai.TranscriptionConfig(
  speech_models=["universal-3-5-pro"],
  language_detection=True,
  prompt="League of Legend roles",
)

transcript = aai.Transcriber().transcribe(audio_file, config)

print(transcript.text)
```

```javascript {10} expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assembly.ai/prompt-8",
  language_detection: true,
  speech_models: ["universal-3-5-pro"],
  prompt: "League of Legend roles",
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

```javascript {13} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assembly.ai/prompt-8";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
  prompt: "League of Legend roles",
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log(transcript.text);
};

run();
```

### Prompting guide

Contextual prompts work at three levels of specificity. Use the least specific level that covers your use case, and add detail when your audio contains uncommon names or terms the model can't otherwise know.

| Level | Length | What it contains | Example |
| --- | --- | --- | --- |
| **Domain** | 2–5 words | The domain only | `Medical consultation call.` |
| **Scenario** | 5–15 words | What the conversation is about | `Cardiology consultation about chest pain symptoms.` |
| **Detailed** | 20–50 words | Full description, including names, products, or identifiers | `Cardiology consultation between Dr. Smith and an elderly patient regarding recurring chest pain, ECG results, and medication adjustment for hypertension.` |

Guidelines for writing contextual prompts:

- Write plain, complete sentences that describe the audio
- Keep it to one short block of text. Don't pack lists of keywords into the contextual prompt

## Keyterms prompting

Universal-3.5 Pro has strong out-of-the-box vocabulary coverage, so in many cases you won't need keyterms prompting at all. But when you're dealing with highly specific names, brands, acronyms, or niche industry jargon, the `keyterms_prompt` parameter gives you an extra layer of control. You can pass up to 1,000 words or phrases (max 6 words per phrase) to boost recognition for those terms and contextually similar variations.

See keyterms prompt in action:

Without keyterms prompting:

```txt
Hi, this is Kelly Byrne Donahue
```

With keyterms prompting:

```txt
Hi, this is Kelly Byrne-Donoghue
```

```python {11} expandable
import requests
import time

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

data = {
    "audio_url": "https://assemblyaiassets.com/audios/keyterms_prompting.wav",
    "language_detection": True,
    "speech_models": ["universal-3-5-pro"],
    "keyterms_prompt": ["Kelly Byrne-Donoghue"]
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

if response.status_code != 200:
    print(f"Error: {response.status_code}, Response: {response.text}")
    response.raise_for_status()

transcript_response = response.json()
transcript_id = transcript_response["id"]
polling_endpoint = f"{base_url}/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        print(transcript["text"])
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)
```

```python {10}
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_file = "https://assemblyaiassets.com/audios/keyterms_prompting.wav"

config = aai.TranscriptionConfig(
  speech_models=["universal-3-5-pro"],
  language_detection=True,
  keyterms_prompt=["Kelly Byrne-Donoghue"],
)

transcript = aai.Transcriber().transcribe(audio_file, config)

print(transcript.text)
```

```javascript {12} expandable
const baseUrl = "https://api.assemblyai.com";
const headers = {
  authorization: "<YOUR_API_KEY>",
};

const data = {
  audio_url: "https://assemblyaiassets.com/audios/keyterms_prompting.wav",
  language_detection: true,
  speech_models: ["universal-3-5-pro"],
  keyterms_prompt: ["Kelly Byrne-Donoghue"],
};

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    console.log(transcriptionResult.text);
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

```javascript {13} expandable
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

const audioFile = "https://assemblyaiassets.com/audios/keyterms_prompting.wav";

const params = {
  audio: audioFile,
  speech_models: ["universal-3-5-pro"],
  language_detection: true,
  keyterms_prompt: ["Kelly Byrne-Donoghue"],
};

const run = async () => {
  const transcript = await client.transcripts.transcribe(params);
  console.log(transcript.text);
};

run();
```

## Support for 99 languages

Universal-3.5 Pro supports 18 languages, and for anything outside that set, the system automatically falls back to Universal-2, giving you coverage across 99 languages total without any extra configuration.

| Model | Supported languages |
| --- | --- |
| `universal-3-5-pro` | Global English, Australian English, British English, US English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Finnish, Hebrew, Hindi, Japanese, Mandarin, Norwegian, Swedish, Turkish, Vietnamese |
| `universal-2` | Global English, Australian English, British English, US English, Spanish, French, German, Italian, Portuguese, Dutch, Hindi, Japanese, Chinese, Finnish, Korean, Polish, Russian, Turkish, Ukrainian, Vietnamese, Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani, Bashkir, Basque, Belarusian, Bengali, Bosnian, Breton, Bulgarian, Burmese, Catalan, Croatian, Czech, Danish, Estonian, Faroese, Galician, Georgian, Greek, Gujarati, Haitian, Hausa, Hawaiian, Hebrew, Hungarian, Icelandic, Indonesian, Javanese, Kannada, Kazakh, Khmer, Lao, Latin, Latvian, Lingala, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian, Norwegian Nynorsk, Occitan, Panjabi, Pashto, Persian, Romanian, Sanskrit, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Sundanese, Swahili, Swedish, Swiss German, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Tibetan, Turkmen, Urdu, Uzbek, Welsh |

---

<a id="tracking-customer-transcription-usage-docs-tracking-your-customers-usage-mdx"></a>

# Tracking customer transcription usage

URL: https://www.assemblyai.com/docs/tracking-your-customers-usage
Source: docs/tracking-your-customers-usage.mdx
Navigation: Additional pages
Description: Learn how to track individual customer usage within your product using AssemblyAI's webhook system and metadata.

This guide explains how to track individual customer usage within your product for billing purposes.

  This is the recommended approach for tracking customer usage. Creating
  separate API keys for each of your customers is not an optimal strategy for
  usage tracking, as it adds unnecessary complexity and makes it harder to
  manage your account.

There are two separate methods depending on which transcription approach you use:

1. **Async transcription**: Use webhooks with custom query parameters to associate transcriptions with customers, then retrieve the `audio_duration` from the transcript response.

2. **Streaming transcription**: Manage customer IDs in your application state and capture the `session_duration_seconds` from the WebSocket Termination event.

This guide covers both methods in detail.

## Async transcription usage tracking

By combining webhooks with custom metadata, you can track audio duration per customer and monitor their usage of your transcription service.

### Step 1: Set up webhooks with customer metadata

When submitting a transcription request, include your webhook URL with the customer ID as a query parameter. This allows you to associate each transcription with a specific customer.

```python
import requests

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

# Add customer_id as a query parameter to your webhook URL
webhook_url = "https://your-domain.com/webhook?customer_id=customer_123"

data = {
    "audio_url": "https://example.com/audio.mp3",
    "speech_models": ["universal-3-5-pro", "universal-2"],
    "language_detection": True,
    "webhook_url": webhook_url
}

# Submit without waiting for completion
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
```

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# Add customer_id as a query parameter to your webhook URL
webhook_url = "https://your-domain.com/webhook?customer_id=customer_123"

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
).set_webhook(webhook_url)

# Submit without waiting for completion
aai.Transcriber().submit("https://example.com/audio.mp3", config)
```

```javascript expandable
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

// Add customer_id as a query parameter to your webhook URL
const webhookUrl = "https://your-domain.com/webhook?customer_id=customer_123";

const data = {
  audio_url: "https://example.com/audio.mp3",
  speech_models: ["universal-3-5-pro", "universal-2"],
  language_detection: true,
  webhook_url: webhookUrl,
};

// Submit without waiting for completion
let res = await fetch(`${baseUrl}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();
const transcriptId = response.id;
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// Add customer_id as a query parameter to your webhook URL
const webhookUrl = "https://your-domain.com/webhook?customer_id=customer_123";

const transcript = await client.transcripts.submit({
  audio: "https://example.com/audio.mp3",
  speech_models: ["universal-3-5-pro", "universal-2"],
  language_detection: true,
  webhook_url: webhookUrl,
});
```

You can add multiple query parameters to track additional information:

```
https://your-domain.com/webhook?customer_id=123&project_id=456&order_id=789
```

This allows you to track usage across multiple dimensions (customer, project, order, etc.).

### Step 2: Handle the webhook delivery

When the transcription completes, AssemblyAI sends a POST request to your webhook URL with the following payload:

```json
{
  "transcript_id": "5552493-16d8-42d8-8feb-c2a16b56f6e8",
  "status": "completed"
}
```

Extract both the `transcript_id` from the payload and the `customer_id` from your URL query parameters.

### Step 3: Retrieve the transcript with audio duration

Use the transcript ID to fetch the complete transcript details, which includes the `audio_duration` field (in seconds).

```python
import requests

base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

# Get transcript using the ID from webhook
response = requests.get(base_url + "/v2/transcript/<TRANSCRIPT_ID>", headers=headers)
transcript = response.json()

if transcript["status"] == "completed":
    audio_duration = transcript["audio_duration"]  # Duration in seconds
    # Use audio_duration for billing/tracking
```

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# Get transcript using the ID from webhook
transcript = aai.Transcript.get_by_id("<TRANSCRIPT_ID>")

if transcript.status == aai.TranscriptStatus.completed:
    audio_duration = transcript.audio_duration  # Duration in seconds
    # Use audio_duration for billing/tracking
```

```javascript
const baseUrl = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

// Get transcript using the ID from webhook
let res = await fetch(`${baseUrl}/v2/transcript/<TRANSCRIPT_ID>`, { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcript = await res.json();

if (transcript.status === "completed") {
  const audioDuration = transcript.audio_duration; // Duration in seconds
  // Use audioDuration for billing/tracking
}
```

```javascript
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: "<YOUR_API_KEY>",
});

// Get transcript using the ID from webhook
const transcript = await client.transcripts.get("<TRANSCRIPT_ID>");

if (transcript.status === "completed") {
  const audioDuration = transcript.audio_duration; // Duration in seconds
  // Use audioDuration for billing/tracking
}
```

### Step 4: Track usage per customer

In your webhook handler, combine the customer ID from your webhook URL query parameters with the audio duration from the transcript to record usage:

1. Extract the `customer_id` from the webhook URL query parameters
2. Extract the `transcript_id` from the webhook payload
3. If the status is `completed`, fetch the transcript using the SDK to get the `audio_duration`
4. Store the usage record in your database with the customer ID, transcript ID, audio duration, and timestamp

This allows you to aggregate usage per customer for billing purposes.

## Streaming transcription usage tracking

Unlike async transcription which uses webhooks, streaming transcription requires a different approach. You'll track usage by managing customer IDs in your own application state/session management, capturing the `session_duration_seconds` from the Termination event, and associating the duration with the customer ID for billing/tracking. AssemblyAI bills streaming based on session duration, so this is the metric you should track.

### Step 1: Set up your WebSocket connection

Connect to AssemblyAI's streaming service. The customer ID is managed entirely in your application and is never sent to AssemblyAI.

```python
import websocket
import json
from urllib.parse import urlencode
from datetime import datetime

# Configuration
YOUR_API_KEY = "<YOUR_API_KEY>"

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "format_turns": True,
}

API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"
```

### Step 2: Capture audio duration from the Termination event

The key to tracking usage is capturing the `audio_duration_seconds` field from the Termination message. This is sent when the streaming session ends.

```python expandable
def on_message(ws, message):
    """Handle WebSocket messages"""
    try:
        data = json.loads(message)
        msg_type = data.get("type")

        if msg_type == "Begin":
            session_id = data.get("id")
            print(f"Session started: {session_id}")

        elif msg_type == "Turn":
            transcript = data.get("transcript", "")
            if data.get("end_of_turn"):
                print(f"Transcript: {transcript}")

        elif msg_type == "Termination":
            # Extract audio duration - this is what you need for billing
            audio_duration_seconds = data.get("audio_duration_seconds", 0)
            session_duration_seconds = data.get("session_duration_seconds", 0)

            print(f"\nSession terminated:")
            print(f"  Audio Duration: {audio_duration_seconds} seconds")
            print(f"  Session Duration: {session_duration_seconds} seconds")

            # Here you would associate audio_duration_seconds with your customer
            # using whatever session management system you have in place
            customer_id = get_customer_id_from_session()  # Your implementation
            log_customer_usage(customer_id, session_duration_seconds)

    except json.JSONDecodeError as e:
        print(f"Error decoding message: {e}")
    except Exception as e:
        print(f"Error handling message: {e}")
```

### Step 3: Log customer usage

When you receive the Termination event, store the session duration for billing/tracking:

1. Retrieve the customer ID from your session management system (authentication tokens, session cookies, etc.)
2. Extract the `session_duration_seconds` from the Termination event
3. Store the usage record in your database with the customer ID, session duration, and timestamp

Since AssemblyAI bills streaming based on `session_duration_seconds`, this is the metric you should track for accurate billing.

### Session duration vs audio duration

From the Termination event, you receive two fields:

| Field                      | Description                               |
| -------------------------- | ----------------------------------------- |
| `session_duration_seconds` | Total time the session was open           |
| `audio_duration_seconds`   | Total seconds of audio actually processed |

  Streaming transcription is billed based on `session_duration_seconds`, not
  `audio_duration_seconds`. Make sure you track the correct metric for accurate
  billing.

### Session management

You need to implement your own session management to associate WebSocket connections with customer IDs. This could be through user authentication tokens, session cookies, database lookups, or in-memory session stores. Track the customer ID throughout the WebSocket lifecycle so you can associate it with the session duration when the Termination event arrives.

### Proper session termination

Always close sessions properly to ensure you receive the Termination event and avoid unexpected costs:

```python
# Send termination message when done
terminate_message = {"type": "Terminate"}
ws.send(json.dumps(terminate_message))
```

## Best practices

When implementing billing tracking, consider the following best practices:

1. **Store the transcript/session ID**: Always store the identifier alongside usage records. This allows you to audit and verify billing data.

2. **Handle errors gracefully**: If a transcription fails (`status: "error"`), don't bill the customer for that request. You may want to log failed transcriptions for debugging.

3. **Secure your webhooks**: Use the `webhook_auth_header_name` and `webhook_auth_header_value` parameters to verify that webhook requests are from AssemblyAI.

4. **Consider time zones**: Store timestamps in UTC to avoid confusion when generating billing reports.

## Next steps

- Learn more about [webhooks](/pre-recorded-audio/webhooks) and their configuration options
- Explore the [Submit Transcript API](/api-reference/transcripts/submit) for async transcription
- Explore the [Get Transcript API](/api-reference/transcripts/get) for retrieving transcript details
- Review the [Streaming API](/streaming/api-spec/streaming-websocket) for real-time transcription

---

<a id="use-case-guides-docs-use-cases-mdx"></a>

# Use case guides

URL: https://www.assemblyai.com/docs/use-cases
Source: docs/use-cases.mdx
Navigation: Additional pages
Description: Build real-world applications with AssemblyAI.

Whether you're building a meeting notetaker, a medical scribe, a voice agent, or a contact center platform, these guides walk you through architecture decisions, best practices, and implementation details for each use case.

## Core use cases

  
    Build an automated meeting notetaker that captures, transcribes, and
    summarizes conversations in real time or from recordings.
  
  
    Build an AI-powered medical scribe that transcribes patient encounters and
    generates clinical documentation.
  
  
    Build a conversational voice agent using streaming speech-to-text with
    low-latency turn detection.
  
  
    Build a contact center platform with call transcription, agent assist, and
    post-call analytics.
  

## More guides

Explore additional guides for async and streaming transcription workflows, including batch processing, subtitle generation, speaker identification, and more.

  
    Cookbooks and tutorials for pre-recorded audio transcription.
  
  
    Cookbooks and tutorials for real-time streaming transcription.

---

<a id="universal-3-5-pro-streaming-api-docs-voice-agents-universal-3-5-pro-streaming-api-mdx"></a>

# Universal-3.5 Pro Streaming API

URL: https://www.assemblyai.com/docs/voice-agents/universal-3-5-pro-streaming-api
Source: docs/voice-agents/universal-3-5-pro-streaming-api.mdx
Navigation: Additional pages
Description: Build a custom voice agent on AssemblyAI's streaming WebSocket with turn detection, barge-in, and interruption handling without a third-party orchestrator.

## Overview

This guide is for voice agents that connect AssemblyAI's [Universal-3.5 Pro Streaming](/streaming/getting-started/transcribe-streaming-audio) WebSocket directly to a custom LLM and TTS, with no LiveKit, Pipecat, or other orchestrator in the loop.

Universal-3.5 Pro Streaming is optimized for real-time audio under 10 seconds with low-latency turn detection, native multilingual code switching, and prompting support. The protocol is documented in detail on the [Universal-3.5 Pro overview](/streaming/getting-started/transcribe-streaming-audio) and [message sequence](/streaming/getting-started/transcribe-streaming-audio) pages. This guide focuses on the voice-agent loop and how to handle barge-in and interruptions correctly.

  If you're building on AssemblyAI's [Voice Agent API](/voice-agents/voice-agent-api) (a managed endpoint with built-in LLM and turn detection), see [Turn detection and interruptions](/voice-agents/voice-agent-api/turn-detection-and-interruptions) instead. Semantic interruption handling is built in there.

## Quickstart

A minimal Python consumer that connects to the streaming WebSocket and reacts to `Begin`, `Turn`, `SpeechStarted`, and `Termination` events:

```python expandable
import json
import websocket
from urllib.parse import urlencode

API_KEY = "YOUR_API_KEY"
SAMPLE_RATE = 16000

CONNECTION_PARAMS = {
    "sample_rate": SAMPLE_RATE,
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 1000,
}

API_ENDPOINT = (
    "wss://streaming.assemblyai.com/v3/ws?" + urlencode(CONNECTION_PARAMS)
)

def on_message(ws, message):
    data = json.loads(message)
    msg_type = data.get("type")

    if msg_type == "Begin":
        print(f"Session started: {data.get('id')}")

    elif msg_type == "Turn":
        transcript = data.get("transcript", "")
        end_of_turn = data.get("end_of_turn", False)
        if end_of_turn:
            # Final transcript - send to your LLM
            print(f"Final: {transcript}")
        else:
            # Partial - optionally start pre-emptive LLM generation
            print(f"Partial: {transcript}")

    elif msg_type == "SpeechStarted":
        # User started speaking - interrupt the agent's TTS if it's playing
        print("Speech detected, interrupt agent if speaking")

    elif msg_type == "Termination":
        print("Session ended")

ws = websocket.WebSocketApp(
    API_ENDPOINT,
    header={"Authorization": API_KEY},
    on_message=on_message,
)
ws.run_forever()
```

For the full message protocol, including all event fields, audio framing, and termination, see the [Universal-3.5 Pro message sequence](/streaming/getting-started/transcribe-streaming-audio) reference.

## Turn detection

Universal-3.5 Pro Streaming uses punctuation-based turn detection controlled by two parameters:

| Parameter          | Default   | Description                                                              |
| ------------------ | --------- | ------------------------------------------------------------------------ |
| `min_turn_silence` | `100` ms  | Silence before a speculative end-of-turn check fires.                    |
| `max_turn_silence` | `1000` ms | Maximum silence before forcing the turn to end.                          |

Lower values produce faster transcripts at the cost of occasional entity splits across turns. See the [Universal-3.5 Pro overview](/streaming/getting-started/transcribe-streaming-audio) for tuning guidance and the [message sequence reference](/streaming/getting-started/transcribe-streaming-audio) for the full event protocol.

## Interruption handling

While the agent is speaking, users often produce backchannel utterances ("mhm", "yeah", "um", "okay") that you don't want to treat as interruptions. A barge-in trigger that fires on every `SpeechStarted` (or every short `Turn`) will cause the agent to stop mid-sentence even though the user didn't intend to interrupt.

The recommended fix is a single combined filter applied to each `Turn` event during agent speech: skip the barge-in if the transcript is short **or** if every token is a known backchannel. Reset the filter once the agent has finished speaking.

```python expandable
import json
import string
import time
import websocket
from urllib.parse import urlencode

# "yes" / "no" deliberately omitted - in a booking flow a bare "yes"
# is a real confirmation. Edit for your domain.
BACKCHANNELS = frozenset({
    "mhm", "mm", "mmhm", "mmhmm",
    "uh", "uhhuh", "huh",
    "um", "umm", "uhm",
    "er", "erm",
    "hmm", "hm",
    "ah", "oh",
    "yeah", "yep", "yup",
    "okay", "ok",
    "right", "alright", "gotcha",
})

_PUNCT_STRIP = str.maketrans("", "", string.punctuation)
MIN_WORDS = 2          # Utterances below this are treated as filler
FILTER_GRACE_S = 1.0   # Keep filtering for 1s after agent stops speaking

# These flags are owned by your TTS layer.
agent_speaking = False
last_speaking_at = 0.0

def _is_all_backchannel(text: str) -> bool:
    tokens = text.lower().translate(_PUNCT_STRIP).split()
    return bool(tokens) and all(tok in BACKCHANNELS for tok in tokens)

def _should_suppress_interrupt(text: str) -> bool:
    now = time.monotonic()
    if agent_speaking:
        globals()["last_speaking_at"] = now
    elif now - last_speaking_at > FILTER_GRACE_S:
        return False

    word_count = len(text.split())
    return word_count < MIN_WORDS or _is_all_backchannel(text)

def on_message(ws, message):
    data = json.loads(message)
    msg_type = data.get("type")

    if msg_type == "Turn":
        transcript = data.get("transcript", "")
        end_of_turn = data.get("end_of_turn", False)

        if _should_suppress_interrupt(transcript):
            # Backchannel during agent speech - drop it.
            return

        if end_of_turn:
            handle_user_turn(transcript)  # send to LLM
        else:
            handle_partial(transcript)

    elif msg_type == "SpeechStarted":
        if agent_speaking:
            # Don't interrupt yet - wait for the Turn event,
            # which is gated by _should_suppress_interrupt above.
            return
        # Otherwise: normal barge-in path.
```

**How it works:**

1. While the agent is speaking (plus a 1-second grace window after speech ends), each `Turn` event is checked.
2. `_should_suppress_interrupt` returns `True` when the transcript has fewer than `MIN_WORDS` tokens **or** when every token is a known backchannel. Either condition drops the event.
3. Utterances with any non-filler content past the threshold (e.g., "yeah I'd like the suite") always pass through.
4. `SpeechStarted` is gated through the same `Turn`-level check rather than firing barge-in directly. This prevents a race where a backchannel triggers `SpeechStarted` before the gating logic sees the transcript.

  The `BACKCHANNELS` set is domain-customizable. "yes" and "no" are deliberately excluded because they often represent genuine confirmations in booking or IVR scenarios. Edit the set for your use case. `MIN_WORDS = 2` is a reasonable default. Raise it if you see frequent two-word filler ("uh okay", "yeah right") slipping through.

  If you're building on LiveKit, prefer LiveKit's adaptive interruption handling. See [LiveKit > Interruption handling](/voice-agents/livekit-universal-3-5-pro#interruption-handling). The strategy on this page is the equivalent for direct-WebSocket integrations.

## Recommended configuration

Three presets covering most voice-agent use cases:

```python
# Fast - quick confirmations, IVR, yes/no questions
fast_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 800,
}

# Balanced - most voice agent conversations (recommended)
balanced_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 100,
    "max_turn_silence": 1000,
}

# Patient - entity dictation, complex instructions
patient_params = {
    "speech_model": "universal-3-5-pro",
    "min_turn_silence": 200,
    "max_turn_silence": 2000,
}
```

For cross-cutting topics like dynamic configuration updates, scaling, latency budgeting, and evals, see the [voice agent best practices](/voice-agents/best-practices) guide.

## Related

- [Universal-3.5 Pro Streaming overview](/streaming/getting-started/transcribe-streaming-audio)
- [Universal-3.5 Pro message sequence](/streaming/getting-started/transcribe-streaming-audio)
- [LiveKit integration](/voice-agents/livekit-universal-3-5-pro)
- [Pipecat integration](/voice-agents/pipecat-universal-3-5-pro)
- [Voice agent best practices](/voice-agents/best-practices)

---

<a id="create-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-create-webhook-subscription-mdx"></a>

# Create a webhook subscription

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/create-webhook-subscription
Source: docs/voice-agents/voice-agent-api/api-spec/create-webhook-subscription.mdx
Navigation: Additional pages
Description: Create a webhook subscription for Voice Agent API events.

Metadata:
- openapi: /api-reference/specs/webhooks.yaml POST /v1/webhook-subscriptions

---

<a id="delete-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-delete-webhook-subscription-mdx"></a>

# Delete a webhook subscription

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/delete-webhook-subscription
Source: docs/voice-agents/voice-agent-api/api-spec/delete-webhook-subscription.mdx
Navigation: Additional pages
Description: Delete a webhook subscription.

Metadata:
- openapi: /api-reference/specs/webhooks.yaml DELETE /v1/webhook-subscriptions/{subscription_id}

---

<a id="retrieve-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-get-webhook-subscription-mdx"></a>

# Retrieve a webhook subscription

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/get-webhook-subscription
Source: docs/voice-agents/voice-agent-api/api-spec/get-webhook-subscription.mdx
Navigation: Additional pages
Description: Retrieve a webhook subscription by ID.

Metadata:
- openapi: /api-reference/specs/webhooks.yaml GET /v1/webhook-subscriptions/{subscription_id}

---

<a id="list-webhook-subscriptions-docs-voice-agents-voice-agent-api-api-spec-list-webhook-subscriptions-mdx"></a>

# List webhook subscriptions

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/list-webhook-subscriptions
Source: docs/voice-agents/voice-agent-api/api-spec/list-webhook-subscriptions.mdx
Navigation: Additional pages
Description: List all webhook subscriptions in your account.

Metadata:
- openapi: /api-reference/specs/webhooks.yaml GET /v1/webhook-subscriptions

---

<a id="update-a-webhook-subscription-docs-voice-agents-voice-agent-api-api-spec-update-webhook-subscription-mdx"></a>

# Update a webhook subscription

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/api-spec/update-webhook-subscription
Source: docs/voice-agents/voice-agent-api/api-spec/update-webhook-subscription.mdx
Navigation: Additional pages
Description: Update a webhook subscription.

Metadata:
- openapi: /api-reference/specs/webhooks.yaml PATCH /v1/webhook-subscriptions/{subscription_id}

---

<a id="webhooks-docs-voice-agents-voice-agent-api-webhooks-mdx"></a>

# Webhooks

URL: https://www.assemblyai.com/docs/voice-agents/voice-agent-api/webhooks
Source: docs/voice-agents/voice-agent-api/webhooks.mdx
Navigation: Additional pages
Description: Subscribe to session and call lifecycle events for your agents. AssemblyAI POSTs a signed JSON payload to your URL when each event fires.

Webhooks let your backend react to what your agents do without polling. Register a URL and the events you care about, and AssemblyAI sends a signed `POST` to that URL whenever one fires (when a call connects, a session ends, and so on). This is how you log calls, kick off post-call processing, or update your own systems for agents you created over [REST](/voice-agents/voice-agent-api/manage-agents).

**Base URL:** `https://agents.assemblyai.com`. Manage subscriptions with your API key in the `Authorization` header, same as the rest of the API.

## Events

| Event              | Fires when |
| ------------------ | ---------- |
| `session.started`  | A voice agent session begins. |
| `session.completed`| A session ends (delivered with final duration and close reason). |
| `call.connected`   | An inbound phone call connects to an agent. |
| `call.ended`       | A call ends. |
| `call.failed`      | A call fails to connect or errors out. |

## Create a subscription

`POST /v1/webhook-subscriptions`. Subscribe to one or more events; scope to a single agent with `agent_id`, or omit it to receive events for **all** your agents.

```bash cURL
curl -X POST https://agents.assemblyai.com/v1/webhook-subscriptions \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-app.com/webhooks/assemblyai",
    "events": ["session.completed", "call.ended"],
    "secret": "a-long-random-signing-secret-at-least-32-chars",
    "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9"
  }'
```

```python Python
# pip install requests
import os
import requests

resp = requests.post(
    "https://agents.assemblyai.com/v1/webhook-subscriptions",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
    json={
        "url": "https://your-app.com/webhooks/assemblyai",
        "events": ["session.completed", "call.ended"],
        "secret": "a-long-random-signing-secret-at-least-32-chars",
        "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9",
    },
)
resp.raise_for_status()
print(resp.json()["id"])
```

```javascript Node.js
// Node 18+ has fetch built in
const res = await fetch("https://agents.assemblyai.com/v1/webhook-subscriptions", {
  method: "POST",
  headers: {
    Authorization: process.env.ASSEMBLYAI_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://your-app.com/webhooks/assemblyai",
    events: ["session.completed", "call.ended"],
    secret: "a-long-random-signing-secret-at-least-32-chars",
    agent_id: "7ad24396-b822-4dca-871a-be9cc4781cf9",
  }),
});
const data = await res.json();
console.log(data);
```

```json
{
  "id": "0d75ebd7-937c-495c-b638-ba582ec05b39",
  "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9",
  "url": "https://your-app.com/webhooks/assemblyai",
  "events": ["session.completed", "call.ended"],
  "enabled": true,
  "secret_version": 1,
  "created_at": "2026-06-10T14:16:59Z",
  "updated_at": "2026-06-10T14:16:59Z"
}
```

| Field      | Required | Notes |
| ---------- | -------- | ----- |
| `url`      | Yes      | Where deliveries are POSTed. Must be `https` and a public host. |
| `events`   | Yes      | One or more of the [events](#events) above. No duplicates. |
| `secret`   | Yes      | Your signing secret, 32-256 printable ASCII chars (no whitespace). Used to sign every delivery; store it, since it's **never returned**. |
| `agent_id` | No       | Scope to one agent. Omit for account-wide events. |
| `enabled`  | No       | Defaults to `true`. Set `false` to pause deliveries. |

The response returns a `secret_version` (an integer that increments when you rotate the secret), never the secret itself. Keep your own copy of the secret to verify signatures.

## Receiving a delivery

When an event fires, AssemblyAI sends a `POST` to your `url` with these headers:

| Header             | Value |
| ------------------ | ----- |
| `X-AAI-Event`      | The event type, e.g. `session.completed`. |
| `X-AAI-Signature`  | `sha256=<hex HMAC-SHA256 of the raw body, keyed by your secret>`. |
| `X-AAI-Delivery-Id`| Unique per delivery attempt; use it to dedupe. |

**Session events** carry a `session` object:

```json
{
  "event_id": "f1d2...",
  "event": "session.completed",
  "timestamp": "2026-06-10T14:20:31Z",
  "session": {
    "session_id": "sess_abc123",
    "account_id": 325374,
    "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9",
    "status": "completed",
    "duration_seconds": 142.6,
    "public_close_reason": "client_ended",
    "s3_prefix": "…",
    "created_at": "2026-06-10T14:18:08Z",
    "ended_at": "2026-06-10T14:20:31Z"
  }
}
```

**Call events** carry a `call` object (with recording/transcript links where available):

```json
{
  "event_id": "a93c...",
  "event": "call.ended",
  "timestamp": "2026-06-10T14:20:32Z",
  "call": {
    "call_id": "call_8b2b1e8c...",
    "account_id": 325374,
    "agent_id": "7ad24396-b822-4dca-871a-be9cc4781cf9",
    "status": "completed",
    "direction": "inbound",
    "from_number": "+14155550123",
    "to_number": "+19016659697",
    "session_id": "sess_abc123",
    "recording_url": "https://…",
    "transcript_url": "https://…",
    "created_at": "2026-06-10T14:18:05Z",
    "ended_at": "2026-06-10T14:20:32Z"
  }
}
```

Respond with a `2xx` status promptly. Failed deliveries are retried with backoff, so make your handler **idempotent**: dedupe on `event_id` (or `X-AAI-Delivery-Id`).

## Verify the signature

Always verify `X-AAI-Signature` before trusting a delivery. Compute `HMAC-SHA256` over the **exact raw request body** (don't re-serialize the JSON) using your subscription secret, and compare in constant time:

```python Python
import hashlib, hmac

def verify(raw_body: bytes, signature_header: str, secret: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header or "")

# Flask example
@app.post("/webhooks/assemblyai")
def handle():
    if not verify(request.get_data(), request.headers.get("X-AAI-Signature"), SECRET):
        return "", 401
    event = request.get_json()
    # ... handle event["event"], event["session"] / event["call"]
    return "", 200
```

```javascript Node
import crypto from "node:crypto";

function verify(rawBody, signatureHeader, secret) {
  const expected = "sha256=" + crypto.createHmac("sha256", secret).update(rawBody).digest("hex");
  const a = Buffer.from(expected), b = Buffer.from(signatureHeader || "");
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

// Express: capture the raw body for verification
app.post("/webhooks/assemblyai",
  express.raw({ type: "application/json" }),
  (req, res) => {
    if (!verify(req.body, req.get("X-AAI-Signature"), SECRET)) return res.sendStatus(401);
    const event = JSON.parse(req.body.toString());
    // ... handle event
    res.sendStatus(200);
  });
```

## Manage subscriptions

```bash cURL expandable
# List (paginated, newest first)
curl https://agents.assemblyai.com/v1/webhook-subscriptions -H "Authorization: $ASSEMBLYAI_API_KEY"

# Retrieve one
curl https://agents.assemblyai.com/v1/webhook-subscriptions/{id} -H "Authorization: $ASSEMBLYAI_API_KEY"

# Update: change the URL/events, pause, or rotate the secret (bumps secret_version)
curl -X PATCH https://agents.assemblyai.com/v1/webhook-subscriptions/{id} \
  -H "Authorization: $ASSEMBLYAI_API_KEY" -H "Content-Type: application/json" \
  -d '{ "enabled": false }'

# Delete
curl -X DELETE https://agents.assemblyai.com/v1/webhook-subscriptions/{id} -H "Authorization: $ASSEMBLYAI_API_KEY"
```

```python Python expandable
# pip install requests
import os
import requests

headers = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

# List (paginated, newest first)
resp = requests.get("https://agents.assemblyai.com/v1/webhook-subscriptions", headers=headers)
resp.raise_for_status()
print(resp.json())

# Retrieve one
resp = requests.get("https://agents.assemblyai.com/v1/webhook-subscriptions/{id}", headers=headers)
resp.raise_for_status()
print(resp.json())

# Update: change the URL/events, pause, or rotate the secret (bumps secret_version)
resp = requests.patch(
    "https://agents.assemblyai.com/v1/webhook-subscriptions/{id}",
    headers=headers,
    json={"enabled": False},
)
resp.raise_for_status()
print(resp.json())

# Delete
resp = requests.delete("https://agents.assemblyai.com/v1/webhook-subscriptions/{id}", headers=headers)
resp.raise_for_status()
```

```javascript Node.js expandable
// Node 18+ has fetch built in
const auth = { Authorization: process.env.ASSEMBLYAI_API_KEY };

// List (paginated, newest first)
let res = await fetch("https://agents.assemblyai.com/v1/webhook-subscriptions", {
  method: "GET",
  headers: auth,
});
console.log(await res.json());

// Retrieve one
res = await fetch("https://agents.assemblyai.com/v1/webhook-subscriptions/{id}", {
  method: "GET",
  headers: auth,
});
console.log(await res.json());

// Update: change the URL/events, pause, or rotate the secret (bumps secret_version)
res = await fetch("https://agents.assemblyai.com/v1/webhook-subscriptions/{id}", {
  method: "PATCH",
  headers: { ...auth, "Content-Type": "application/json" },
  body: JSON.stringify({ enabled: false }),
});
console.log(await res.json());

// Delete
res = await fetch("https://agents.assemblyai.com/v1/webhook-subscriptions/{id}", {
  method: "DELETE",
  headers: auth,
});
```

`PATCH` accepts any of `url`, `events`, `secret`, `enabled`. Sending a new `secret` rotates it and increments `secret_version`.

## Endpoint summary

| Method   | Path                                  | Description |
| -------- | ------------------------------------- | ----------- |
| `POST`   | `/v1/webhook-subscriptions`           | Create a subscription. |
| `GET`    | `/v1/webhook-subscriptions`           | List your subscriptions. |
| `GET`    | `/v1/webhook-subscriptions/{id}`      | Retrieve one. |
| `PATCH`  | `/v1/webhook-subscriptions/{id}`      | Update (url, events, secret, enabled). |
| `DELETE` | `/v1/webhook-subscriptions/{id}`      | Delete. |

---

<a id="api-specs"></a>

# API specs

## docs/openapi.yaml

URL: https://www.assemblyai.com/docs/openapi.yaml

```yaml
openapi: 3.1.0

info:
  title: AssemblyAI API
  description: AssemblyAI API
  version: 1.3.4
  termsOfService: https://www.assemblyai.com/legal/terms-of-service
  contact:
    name: API Support
    email: support@assemblyai.com
    url: https://www.assemblyai.com/docs/

servers:
  - url: https://api.assemblyai.com
    description: AssemblyAI API
    x-fern-server-name: Default

tags:
  - name: transcript
    description: Transcript related operations
    externalDocs:
      url: https://www.assemblyai.com/docs/guides/transcribing-an-audio-file
  - name: streaming
    description: Streaming Speech-to-Text
    externalDocs:
      url: https://www.assemblyai.com/docs/streaming/universal-streaming

security:
  - ApiKey: []

paths:
  /v2/upload:
    post:
      tags:
        - transcript
      summary: Upload a media file
      description: |
        Upload a media file to AssemblyAI's servers.

        <Note>To upload a media file to our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        <Warning>Requests to transcribe uploaded files must use an API key from the same project as the key that was used to upload the file. If you use an API key from a different project you will get a `403` error and "Cannot access uploaded file" message.</Warning>
      operationId: uploadFile
      x-fern-sdk-group-name: files
      x-fern-sdk-method-name: upload
      requestBody:
        content:
          application/octet-stream:
            schema:
              type: string
              format: binary
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            from assemblyai.prerecorded.v2 import Transcriber

            transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])

            upload_url = transcriber.upload_file("./audio.mp3")
            print(upload_url)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const uploadUrl = await client.files.upload("./audio.mp3");
            console.log(uploadUrl);
        - lang: bash
          label: cURL
          source: |
            curl -s -X POST https://api.assemblyai.com/v2/upload \
              -H "authorization: $ASSEMBLYAI_API_KEY" \
              --data-binary @./audio.mp3 | jq -r .upload_url
        - lang: python
          label: Python
          source: |
            import os

            import requests

            headers = {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}

            with open("./audio.mp3", "rb") as f:
                response = requests.post(
                    "https://api.assemblyai.com/v2/upload", headers=headers, data=f
                )

            response.raise_for_status()
            print(response.json()["upload_url"])
        - lang: javascript
          label: JavaScript
          source: |
            import { readFileSync } from "fs";

            const audio = readFileSync("./audio.mp3");

            const response = await fetch("https://api.assemblyai.com/v2/upload", {
              method: "POST",
              headers: { authorization: process.env.ASSEMBLYAI_API_KEY },
              body: audio,
            });

            const { upload_url } = await response.json();
            console.log(upload_url);
      responses:
        "200":
          x-label: Media file uploaded
          description: Media file uploaded successfully
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/UploadedFile"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "403":
          $ref: "#/components/responses/CannotAccessUploadedFile"
        "404":
          $ref: "#/components/responses/NotFound"
        "422":
          x-label: Upload failed
          description: |
            Upload failed (for example, when the request body is empty). The response body is plain text, not JSON.
          content:
            text/plain:
              schema:
                type: string
              example: "Upload failed, please try again. If you continue to have issues please reach out to support@assemblyai.com"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

  /v2/transcript:
    post:
      tags:
        - transcript
      summary: Transcribe audio
      description: |
        <Note>To use our EU server for transcription, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        <Tip>Building a load test or submitting a large batch? [See our guide →](/pre-recorded-audio/guides/bulk-transcription-and-load-tests-at-scale)</Tip>
        Create a transcript from a media file that is accessible via a URL.
      operationId: createTranscript
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: submit
      x-fern-request-name: TranscriptParams
      requestBody:
        description: Params to create a transcript
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/TranscriptParams"
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            from assemblyai.prerecorded.v2 import Transcriber

            transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])

            # `submit` queues the job and returns right away; `transcribe` submits and polls.
            transcript = transcriber.submit("https://assembly.ai/wildfires.mp3")
            print(transcript.id, transcript.status)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            // `submit` queues the job and returns right away; `transcribe` submits and polls.
            const transcript = await client.transcripts.submit({
              audio_url: "https://assembly.ai/wildfires.mp3",
            });
            console.log(transcript.id, transcript.status);
        - lang: bash
          label: cURL
          source: |
            curl -s -X POST https://api.assemblyai.com/v2/transcript \
              -H "authorization: $ASSEMBLYAI_API_KEY" \
              -H "content-type: application/json" \
              -d '{"audio_url": "https://assembly.ai/wildfires.mp3"}' | jq -r .id
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.post(
                "https://api.assemblyai.com/v2/transcript",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
                json={"audio_url": "https://assembly.ai/wildfires.mp3"},
            )

            response.raise_for_status()
            print(response.json()["id"])
        - lang: javascript
          label: JavaScript
          source: |
            const response = await fetch("https://api.assemblyai.com/v2/transcript", {
              method: "POST",
              headers: {
                authorization: process.env.ASSEMBLYAI_API_KEY,
                "content-type": "application/json",
              },
              body: JSON.stringify({ audio_url: "https://assembly.ai/wildfires.mp3" }),
            });

            const transcript = await response.json();
            console.log(transcript.id, transcript.status);
      responses:
        "200":
          x-label: Transcript created
          description: Transcript created and queued for processing
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Transcript"
              example:
                id: 0072a82b-aa22-4962-add2-6121c36c17c6
                language_model: assemblyai_default
                acoustic_model: assemblyai_default
                language_code: null
                speech_understanding: null
                translated_texts: null
                status: processing
                audio_url: "https://assembly.ai/wildfires.mp3"
                text: null
                words: null
                utterances: null
                confidence: null
                audio_duration: null
                punctuate: true
                format_text: true
                dual_channel: false
                webhook_url: null
                webhook_status_code: null
                webhook_auth: false
                webhook_auth_header_name: null
                speed_boost: false
                auto_highlights_result: null
                auto_highlights: true
                audio_start_from: null
                audio_end_at: null
                word_boost: []
                boost_param: null
                prompt: null
                keyterms_prompt: null
                filter_profanity: true
                redact_pii: false
                redact_pii_audio: false
                redact_pii_audio_quality: null
                redact_pii_audio_options: null
                redact_pii_policies: null
                redact_pii_sub: null
                redact_static_entities: null
                speaker_labels: true
                speaker_options:
                  min_speakers_expected: 2
                  max_speakers_expected: 2
                  advanced_speaker_segmentation: true
                content_safety: true
                iab_categories: true
                content_safety_labels: {}
                iab_categories_result: {}
                language_detection: true
                language_detection_options: null
                language_detection_results: null
                language_confidence_threshold: 0.7
                language_confidence: null
                custom_spelling: null
                throttled: false
                auto_chapters: false
                summarization: false
                summary_type: null
                summary_model: null
                custom_topics: false
                topics: []
                speech_threshold: null
                speech_model: null
                speech_models:
                  - universal-3-pro
                  - universal-2
                speech_model_used: null
                temperature: null
                remove_audio_tags: all
                chapters: null
                disfluencies: false
                entity_detection: true
                sentiment_analysis: true
                sentiment_analysis_results: null
                entities: null
                speakers_expected: 2
                summary: null
                custom_topics_results: null
                is_deleted: null
                multichannel: null
                project_id: 14296
                token_id: 14296
          links:
            GetTranscriptById:
              $ref: "#/components/links/GetTranscriptById"
            GetTranscriptSentencesById:
              $ref: "#/components/links/GetTranscriptSentencesById"
            GetTranscriptParagraphsById:
              $ref: "#/components/links/GetTranscriptParagraphsById"
            GetTranscriptSubtitlesById:
              $ref: "#/components/links/GetTranscriptSubtitlesById"
            GetTranscriptRedactedAudioById:
              $ref: "#/components/links/GetTranscriptRedactedAudioById"
            WordSearchByTranscriptId:
              $ref: "#/components/links/WordSearchByTranscriptId"
            DeleteTranscriptById:
              $ref: "#/components/links/DeleteTranscriptById"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"
      callbacks:
        transcriptReadyWebhook:
          "{$request.body#/webhook_url}":
            post:
              summary: Transcript ready webhook
              requestBody:
                required: true
                content:
                  application/json:
                    schema:
                      $ref: "#/components/schemas/TranscriptReadyNotification"
              responses:
                "2XX":
                  description: Successfully received the notification
                "4XX":
                  description: Invalid request
                "5XX":
                  description: Unexpected error
        redactedAudioWebhook:
          "{$request.body#/webhook_url}":
            post:
              summary: Redacted audio ready webhook
              requestBody:
                required: true
                content:
                  application/json:
                    schema:
                      $ref: "#/components/schemas/RedactedAudioNotification"
              responses:
                "2XX":
                  description: Successfully received the notification
                "4XX":
                  description: Invalid request
                "5XX":
                  description: Unexpected error
    get:
      tags:
        - transcript
      summary: List transcripts
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: list
      x-fern-request-name: ListTranscriptParams
      operationId: listTranscripts
      description: |
        <Note>To retrieve your transcriptions on our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        Retrieve a list of transcripts you created.
        Transcripts are sorted from newest to oldest and can be retrieved for the last 90 days of usage. The previous URL always points to a page with older transcripts.

        If you need to retrieve transcripts from more than 90 days ago please reach out to our Support team at support@assemblyai.com.

        **Pagination**

        This endpoint returns paginated results. The response includes a `page_details` object with the following properties:
        - `page_details.limit` - Maximum number of transcripts per page.
        - `page_details.result_count` - Total number of transcripts returned on the current page.
        - `page_details.current_url` - URL to the current page.
        - `page_details.prev_url` - URL to the previous page of older transcripts.
        - `page_details.next_url` - URL to the next page of newer transcripts.
      parameters:
        - name: limit
          x-label: Limit
          in: query
          description: Maximum amount of transcripts to retrieve
          schema:
            type: integer
            minimum: 1
            maximum: 200
            default: 10
        - name: status
          x-label: Status
          in: query
          description: Filter by transcript status
          schema:
            $ref: "#/components/schemas/TranscriptStatus"

        - name: created_on
          x-label: Created on
          in: query
          description: Only get transcripts created on this date
          schema:
            type: string
            format: date

        - name: before_id
          x-label: Before ID
          in: query
          description: Get transcripts that were created before this transcript ID
          schema:
            type: string
            format: uuid

        - name: after_id
          x-label: After ID
          in: query
          description: Get transcripts that were created after this transcript ID
          schema:
            type: string
            format: uuid

        - name: throttled_only
          x-label: Throttled only
          in: query
          description: Only get throttled transcripts, overrides the status filter
          schema:
            type: boolean
            default: false
          deprecated: true
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            from assemblyai import ListTranscriptParameters
            from assemblyai.prerecorded.v2 import Transcriber

            transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])

            page = transcriber.list_transcripts(
                ListTranscriptParameters(limit=5, status="completed")
            )

            for item in page.transcripts:
                print(item.id, item.status)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const page = await client.transcripts.list({ limit: 5, status: "completed" });

            for (const item of page.transcripts) {
              console.log(item.id, item.status);
            }
        - lang: bash
          label: cURL
          source: |
            curl -s "https://api.assemblyai.com/v2/transcript?limit=5&status=completed" \
              -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r '.transcripts[].id'
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.get(
                "https://api.assemblyai.com/v2/transcript",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
                params={"limit": 5, "status": "completed"},
            )

            response.raise_for_status()

            for item in response.json()["transcripts"]:
                print(item["id"], item["status"])
        - lang: javascript
          label: JavaScript
          source: |
            const params = new URLSearchParams({ limit: "5", status: "completed" });

            const response = await fetch(
              `https://api.assemblyai.com/v2/transcript?${params}`,
              { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
            );

            const page = await response.json();

            for (const item of page.transcripts) {
              console.log(item.id, item.status);
            }
      responses:
        "200":
          description: |
            A list of transcripts. Transcripts are sorted from newest to oldest. The previous URL always points to a page with older transcripts.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/TranscriptList"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

  /v2/transcript/{transcript_id}:
    get:
      tags:
        - transcript
      summary: Get transcript
      operationId: getTranscript
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: get
      x-fern-request-name: GetTranscriptParams
      description: |
        <Note>To retrieve your transcriptions on our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        Get the transcript resource. The transcript is ready when the "status" is "completed".
      parameters:
        - name: transcript_id
          x-label: Transcript ID
          in: path
          description: ID of the transcript
          required: true
          schema:
            type: string
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            import assemblyai
            from assemblyai.prerecorded.v2 import Transcript

            # `Transcript.get_by_id()` uses the default client, which reads the global settings.
            assemblyai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

            transcript = Transcript.get_by_id("<TRANSCRIPT_ID>")
            print(transcript.status, transcript.text)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const transcript = await client.transcripts.get("<TRANSCRIPT_ID>");
            console.log(transcript.status, transcript.text);
        - lang: bash
          label: cURL
          source: |
            curl -s "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>" \
              -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r '.status, .text'
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.get(
                "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
            )

            response.raise_for_status()

            transcript = response.json()
            print(transcript["status"], transcript["text"])
        - lang: javascript
          label: JavaScript
          source: |
            const response = await fetch(
              "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>",
              { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
            );

            const transcript = await response.json();
            console.log(transcript.status, transcript.text);
      responses:
        "200":
          description: The transcript resource
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Transcript"
          links:
            GetTranscriptById:
              $ref: "#/components/links/GetTranscriptById"
            GetTranscriptSentencesById:
              $ref: "#/components/links/GetTranscriptSentencesById"
            GetTranscriptParagraphsById:
              $ref: "#/components/links/GetTranscriptParagraphsById"
            GetTranscriptSubtitlesById:
              $ref: "#/components/links/GetTranscriptSubtitlesById"
            GetTranscriptRedactedAudioById:
              $ref: "#/components/links/GetTranscriptRedactedAudioById"
            WordSearchByTranscriptId:
              $ref: "#/components/links/WordSearchByTranscriptId"
            DeleteTranscriptById:
              $ref: "#/components/links/DeleteTranscriptById"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

    delete:
      tags:
        - transcript
      summary: Delete transcript
      description: |
        <Note>To delete your transcriptions on our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        Remove the data from the transcript and mark it as deleted.
        <Warning>Files uploaded via the `/upload` endpoint are immediately deleted alongside the transcript when you make a DELETE request, ensuring your data is removed from our systems right away.</Warning>
      operationId: deleteTranscript
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: delete
      parameters:
        - name: transcript_id
          x-label: Transcript ID
          in: path
          description: ID of the transcript
          required: true
          schema:
            type: string
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            import assemblyai
            from assemblyai.prerecorded.v2 import Transcript

            # `Transcript.delete_by_id()` uses the default client, which reads the global settings.
            assemblyai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

            transcript = Transcript.delete_by_id("<TRANSCRIPT_ID>")
            print(transcript.status)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const transcript = await client.transcripts.delete("<TRANSCRIPT_ID>");
            console.log(transcript.status);
        - lang: bash
          label: cURL
          source: |
            curl -s -X DELETE "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>" \
              -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r .status
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.delete(
                "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
            )

            response.raise_for_status()
            print(response.json()["status"])
        - lang: javascript
          label: JavaScript
          source: |
            const response = await fetch(
              "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>",
              {
                method: "DELETE",
                headers: { authorization: process.env.ASSEMBLYAI_API_KEY },
              },
            );

            const transcript = await response.json();
            console.log(transcript.status);
      responses:
        "200":
          description: The deleted transcript response
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Transcript"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

  /v2/transcript/{transcript_id}/{subtitle_format}:
    get:
      tags:
        - transcript
      summary: Get subtitles for transcript
      description: |
        <Note>To retrieve your transcriptions on our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        Export your transcript in SRT or VTT format to use with a video player for subtitles and closed captions.
      operationId: getSubtitles
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: getSubtitles
      x-fern-request-name: GetSubtitlesParams
      parameters:
        - name: transcript_id
          x-label: Transcript ID
          in: path
          description: ID of the transcript
          required: true
          schema:
            type: string
        - name: subtitle_format
          x-label: Subtitle format
          in: path
          description: The format of the captions
          required: true
          schema:
            $ref: "#/components/schemas/SubtitleFormat"
        - name: chars_per_caption
          x-label: Number of characters per caption
          in: query
          description: The maximum number of characters per caption
          schema:
            type: integer

      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            import assemblyai
            from assemblyai.prerecorded.v2 import Transcript

            # `Transcript.get_by_id()` uses the default client, which reads the global settings.
            assemblyai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

            transcript = Transcript.get_by_id("<TRANSCRIPT_ID>")

            srt = transcript.export_subtitles_srt(chars_per_caption=32)
            # vtt = transcript.export_subtitles_vtt(chars_per_caption=32)
            print(srt)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const srt = await client.transcripts.subtitles("<TRANSCRIPT_ID>", "srt", 32);
            // const vtt = await client.transcripts.subtitles("<TRANSCRIPT_ID>", "vtt", 32);
            console.log(srt);
        - lang: bash
          label: cURL
          source: |
            curl -s "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/srt?chars_per_caption=32" \
              -H "authorization: $ASSEMBLYAI_API_KEY"
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.get(
                "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/srt",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
                params={"chars_per_caption": 32},
            )

            response.raise_for_status()
            print(response.text)
        - lang: javascript
          label: JavaScript
          source: |
            const response = await fetch(
              "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/srt?chars_per_caption=32",
              { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
            );

            const srt = await response.text();
            console.log(srt);
      responses:
        "200":
          description: The exported captions as text
          content:
            text/plain:
              schema:
                type: string
                example: |
                  WEBVTT
                  00:12.340 --> 00:16.220
                  Last year I showed these two slides said that demonstrate
                  00:16.200 --> 00:20.040
                  that the Arctic ice cap which for most of the last 3,000,000 years has been the
                  00:20.020 --> 00:25.040
                  size of the lower 48 States has shrunk by 40% but this understates
              examples:
                srt:
                  $ref: "#/components/examples/SrtSubtitlesResponse"
                vtt:
                  $ref: "#/components/examples/VttSubtitlesResponse"
            text/html:
              schema:
                type: string
                example: |
                  WEBVTT
                  00:12.340 --> 00:16.220
                  Last year I showed these two slides said that demonstrate
                  00:16.200 --> 00:20.040
                  that the Arctic ice cap which for most of the last 3,000,000 years has been the
                  00:20.020 --> 00:25.040
                  size of the lower 48 States has shrunk by 40% but this understates
              examples:
                srt:
                  $ref: "#/components/examples/SrtSubtitlesResponse"
                vtt:
                  $ref: "#/components/examples/VttSubtitlesResponse"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          x-label: Not found
          description: |
            Returned when `subtitle_format` is not `srt` or `vtt`. The response body is a plain HTML 404 page — the literal string `404: Not Found` — not JSON. Unknown `transcript_id` lookups, by contrast, return a `400` with a JSON `Error` object.
          content:
            text/html:
              schema:
                type: string
              example: "404: Not Found"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

  /v2/transcript/{transcript_id}/sentences:
    get:
      tags:
        - transcript
      summary: Get sentences in transcript
      operationId: getTranscriptSentences
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: getSentences
      x-fern-request-name: GetSentencesParams
      description: |
        <Note>To retrieve your transcriptions on our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        Get the transcript split by sentences. The API will attempt to semantically segment the transcript into sentences to create more reader-friendly transcripts.
      parameters:
        - name: transcript_id
          x-label: Transcript ID
          in: path
          description: ID of the transcript
          required: true
          schema:
            type: string
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            import assemblyai
            from assemblyai.prerecorded.v2 import Transcript

            # `Transcript.get_by_id()` uses the default client, which reads the global settings.
            assemblyai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

            transcript = Transcript.get_by_id("<TRANSCRIPT_ID>")

            sentences = transcript.get_sentences()
            print(len(sentences), sentences[0].text)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const { sentences } = await client.transcripts.sentences("<TRANSCRIPT_ID>");
            console.log(sentences.length, sentences[0].text);
        - lang: bash
          label: cURL
          source: |
            curl -s "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/sentences" \
              -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r '.sentences[].text'
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.get(
                "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/sentences",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
            )

            response.raise_for_status()

            sentences = response.json()["sentences"]
            print(len(sentences), sentences[0]["text"])
        - lang: javascript
          label: JavaScript
          source: |
            const response = await fetch(
              "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/sentences",
              { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
            );

            const { sentences } = await response.json();
            console.log(sentences.length, sentences[0].text);
      responses:
        "200":
          description: Exported sentences
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/SentencesResponse"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

  /v2/transcript/{transcript_id}/paragraphs:
    get:
      tags:
        - transcript
      summary: Get paragraphs in transcript
      operationId: getTranscriptParagraphs
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: getParagraphs
      x-fern-request-name: GetParagraphsParams
      description: |
        <Note>To retrieve your transcriptions on our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        Get the transcript split by paragraphs. The API will attempt to semantically segment your transcript into paragraphs to create more reader-friendly transcripts.
      parameters:
        - name: transcript_id
          x-label: Transcript ID
          in: path
          description: ID of the transcript
          required: true
          schema:
            type: string
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            import assemblyai
            from assemblyai.prerecorded.v2 import Transcript

            # `Transcript.get_by_id()` uses the default client, which reads the global settings.
            assemblyai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

            transcript = Transcript.get_by_id("<TRANSCRIPT_ID>")

            paragraphs = transcript.get_paragraphs()
            print(len(paragraphs), paragraphs[0].text)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const { paragraphs } = await client.transcripts.paragraphs("<TRANSCRIPT_ID>");
            console.log(paragraphs.length, paragraphs[0].text);
        - lang: bash
          label: cURL
          source: |
            curl -s "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/paragraphs" \
              -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r '.paragraphs[].text'
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.get(
                "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/paragraphs",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
            )

            response.raise_for_status()

            paragraphs = response.json()["paragraphs"]
            print(len(paragraphs), paragraphs[0]["text"])
        - lang: javascript
          label: JavaScript
          source: |
            const response = await fetch(
              "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/paragraphs",
              { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
            );

            const { paragraphs } = await response.json();
            console.log(paragraphs.length, paragraphs[0].text);
      responses:
        "200":
          description: Exported paragraphs
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/ParagraphsResponse"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

  /v2/transcript/{transcript_id}/word-search:
    get:
      tags:
        - transcript
      summary: Search words in transcript
      description: |
        <Note>To search through a transcription created on our EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com`.</Note>
        Search through the transcript for keywords. You can search for individual words, numbers, or phrases containing up to five words or numbers.
      operationId: wordSearch
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: wordSearch
      x-fern-request-name: WordSearchParams
      parameters:
        - name: transcript_id
          x-label: Transcript ID
          in: path
          description: ID of the transcript
          required: true
          schema:
            type: string
        - name: words
          x-label: Words
          in: query
          description: Keywords to search for
          required: true
          style: form
          explode: false
          schema:
            type: array
            items:
              x-label: Word
              type: string

      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            import assemblyai
            from assemblyai.prerecorded.v2 import Transcript

            # `Transcript.get_by_id()` uses the default client, which reads the global settings.
            assemblyai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

            transcript = Transcript.get_by_id("<TRANSCRIPT_ID>")

            matches = transcript.word_search(["foo", "bar", "foo bar"])

            for match in matches:
                print(f"Found '{match.text}' {match.count} times in the transcript")
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const { matches } = await client.transcripts.wordSearch("<TRANSCRIPT_ID>", [
              "foo",
              "bar",
              "foo bar",
            ]);

            for (const match of matches) {
              console.log(`Found '${match.text}' ${match.count} times in the transcript`);
            }
        - lang: bash
          label: cURL
          source: |
            curl -s "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/word-search?words=foo,bar,foo%20bar" \
              -H "authorization: $ASSEMBLYAI_API_KEY" | jq -c '.matches[] | {text, count}'
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.get(
                "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/word-search",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
                params={"words": ",".join(["foo", "bar", "foo bar"])},
            )

            response.raise_for_status()

            for match in response.json()["matches"]:
                print(f"Found '{match['text']}' {match['count']} times in the transcript")
        - lang: javascript
          label: JavaScript
          source: |
            const words = ["foo", "bar", "foo bar"];
            const params = new URLSearchParams({ words: words.join(",") });

            const response = await fetch(
              `https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/word-search?${params}`,
              { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
            );

            const { matches } = await response.json();

            for (const match of matches) {
              console.log(`Found '${match.text}' ${match.count} times in the transcript`);
            }
      responses:
        "200":
          description: Word search response
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/WordSearchResponse"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

  /v2/transcript/{transcript_id}/redacted-audio:
    get:
      tags:
        - transcript
      summary: Get redacted audio
      description: |
        <Note>To retrieve the redacted audio on the EU server, replace `api.assemblyai.com` with `api.eu.assemblyai.com` in the `GET` request above.</Note>
        <Note>Redacted audio files are only available for 24 hours. Make sure to download the file within this time frame.</Note>
        Retrieve the redacted audio object containing the status and URL to the redacted audio.
      operationId: getRedactedAudio
      x-fern-sdk-group-name: transcripts
      x-fern-sdk-method-name: getRedactedAudio
      x-fern-request-name: GetRedactedAudioParams
      parameters:
        - name: transcript_id
          x-label: Transcript ID
          in: path
          description: ID of the transcript
          required: true
          schema:
            type: string

      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            import assemblyai
            from assemblyai.prerecorded.v2 import Transcript

            # `Transcript.get_by_id()` uses the default client, which reads the global settings.
            assemblyai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

            transcript = Transcript.get_by_id("<TRANSCRIPT_ID>")

            print(transcript.get_redacted_audio_url())
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const { status, redacted_audio_url } =
              await client.transcripts.redactedAudio("<TRANSCRIPT_ID>");
            console.log(status, redacted_audio_url);
        - lang: bash
          label: cURL
          source: |
            curl -s "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/redacted-audio" \
              -H "authorization: $ASSEMBLYAI_API_KEY" | jq -r '.status, .redacted_audio_url'
        - lang: python
          label: Python
          source: |
            import os

            import requests

            response = requests.get(
                "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/redacted-audio",
                headers={"authorization": os.environ["ASSEMBLYAI_API_KEY"]},
            )

            response.raise_for_status()

            redacted_audio = response.json()
            print(redacted_audio["status"], redacted_audio["redacted_audio_url"])
        - lang: javascript
          label: JavaScript
          source: |
            const response = await fetch(
              "https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID>/redacted-audio",
              { headers: { authorization: process.env.ASSEMBLYAI_API_KEY } },
            );

            const { status, redacted_audio_url } = await response.json();
            console.log(status, redacted_audio_url);
      responses:
        "200":
          description: The redacted audio object containing the status and URL to the redacted audio
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/RedactedAudioResponse"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "429":
          $ref: "#/components/responses/TooManyRequests"
        "500":
          $ref: "#/components/responses/InternalServerError"
        "503":
          $ref: "#/components/responses/ServiceUnavailable"
        "504":
          $ref: "#/components/responses/GatewayTimeout"

components:
  links:
    GetTranscriptById:
      operationId: getTranscript
      parameters:
        transcript_id: "$response.body#/id"
      description: The transcript ID can be used as the `transcript_id` parameter in `GET /v2/transcript/{transcript_id}`.
    GetTranscriptSentencesById:
      operationId: getSentences
      parameters:
        transcript_id: "$response.body#/id"
      description: The transcript ID can be used as the `transcript_id` parameter in `GET /v2/transcript/{transcript_id}/sentences`.
    GetTranscriptParagraphsById:
      operationId: getParagraphs
      parameters:
        transcript_id: "$response.body#/id"
      description: The transcript ID can be used as the `transcript_id` parameter in `GET /v2/transcript/{transcript_id}/paragraphs`.
    GetTranscriptSubtitlesById:
      operationId: getSubtitles
      parameters:
        transcript_id: "$response.body#/id"
      description: The transcript ID can be used as the `transcript_id` parameter in `GET /v2/transcript/{transcript_id}/{subtitle_format}`.
    GetTranscriptRedactedAudioById:
      operationId: getRedactedAudio
      parameters:
        transcript_id: "$response.body#/id"
      description: The transcript ID can be used as the `transcript_id` parameter in `GET /v2/transcript/{transcript_id}/redacted-audio`.
    WordSearchByTranscriptId:
      operationId: wordSearch
      parameters:
        transcript_id: "$response.body#/id"
      description: The transcript ID can be used as the `transcript_id` parameter in `GET /v2/transcript/{transcript_id}/word-search`.
    DeleteTranscriptById:
      operationId: deleteTranscript
      parameters:
        transcript_id: "$response.body#/id"
      description: The transcript ID can be used as the `transcript_id` parameter in `DELETE /v2/transcript/{transcript_id}`.
  schemas:
    TranscriptReadyNotification:
      description: The notification when the transcript status is completed or error.
      x-label: Transcript ready notification
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      properties:
        transcript_id:
          x-label: Transcript ID
          description: The ID of the transcript
          type: string
          format: uuid
        status:
          x-label: Transcript status
          description: The status of the transcript. Either completed or error.
          $ref: "#/components/schemas/TranscriptReadyStatus"
      required:
        - transcript_id
        - status
      example:
        {
          transcript_id: "9ea68fd3-f953-42c1-9742-976c447fb463",
          status: "completed",
        }

    RedactedAudioNotification:
      description: The notification when the redacted audio is ready.
      x-label: Redacted audio notification
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      allOf:
        - $ref: "#/components/schemas/RedactedAudioResponse"

    RedactedAudioResponse:
      type: object
      additionalProperties: false
      x-label: Redacted audio response
      x-fern-sdk-group-name: transcripts
      required:
        - status
        - redacted_audio_url
      properties:
        status:
          x-label: Status
          description: The status of the redacted audio
          $ref: "#/components/schemas/RedactedAudioStatus"
        redacted_audio_url:
          x-label: Redacted audio URL
          description: The URL of the redacted audio file
          type: string
          format: url
      example:
        {
          redacted_audio_url: "https://s3.us-west-2.amazonaws.com/api.assembly.ai.usw2/redacted-audio/785efd9e-0e20-45e1-967b-3db17770ed9f.wav?AWSAccessKeyId=aws-access-key0id&Signature=signature&x-amz-security-token=security-token&Expires=1698966551",
          status: "redacted_audio_ready",
        }

    RedactedAudioStatus:
      x-label: Redacted audio status
      description: The status of the redacted audio
      x-fern-sdk-group-name: transcripts
      type: string
      enum:
        - redacted_audio_ready
      x-aai-enum:
        redacted_audio_ready:
          label: Redacted audio is ready

    SubtitleFormat:
      x-label: Subtitle format
      description: Format of the subtitles
      x-fern-sdk-group-name: transcripts
      type: string
      enum:
        - srt
        - vtt
      x-aai-enum:
        srt:
          label: SRT
        vtt:
          label: VTT

    WordSearchResponse:
      x-label: Word search response
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      properties:
        id:
          x-label: Transcript ID
          description: The ID of the transcript
          type: string
          format: uuid
        total_count:
          x-label: Total count of matches
          description: The total count of all matched instances. For e.g., word 1 matched 2 times, and word 2 matched 3 times, `total_count` will equal 5.
          type: integer
        matches:
          x-label: Matches
          description: The matches of the search
          type: array
          items:
            x-label: Match
            $ref: "#/components/schemas/WordSearchMatch"
      required:
        - id
        - total_count
        - matches
      example:
        {
          id: "d5a3d302-066e-43fb-b63b-8f57baf185db",
          total_count: 10,
          matches:
            [
              {
                text: "smoke",
                count: 6,
                timestamps:
                  [
                    [250, 650],
                    [49168, 49398],
                    [55284, 55594],
                    [168888, 169118],
                    [215108, 215386],
                    [225944, 226170],
                  ],
                indexes: [0, 136, 156, 486, 652, 698],
              },
              {
                text: "wildfires",
                count: 4,
                timestamps:
                  [
                    [1668, 2346],
                    [33852, 34546],
                    [50118, 51110],
                    [231356, 232354],
                  ],
                indexes: [4, 90, 140, 716],
              },
            ],
        }

    WordSearchMatch:
      type: object
      x-label: Word search match
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      properties:
        text:
          x-label: Text
          description: The matched word
          type: string
        count:
          x-label: Count
          description: The total amount of times the word is in the transcript
          type: integer
        timestamps:
          x-label: Timestamps
          description: An array of timestamps
          type: array
          items:
            x-label: Timestamp
            $ref: "#/components/schemas/WordSearchTimestamp"
        indexes:
          x-label: Indexes
          description: An array of all index locations for that word within the `words` array of the completed transcript
          type: array
          items:
            x-label: Index
            type: integer
      required:
        - text
        - count
        - timestamps
        - indexes
      example:
        {
          text: "smoke",
          count: 6,
          timestamps:
            [
              [250, 650],
              [49168, 49398],
              [55284, 55594],
              [168888, 169118],
              [215108, 215386],
              [225944, 226170],
            ],
          indexes: [0, 136, 156, 486, 652, 698],
        }

    WordSearchTimestamp:
      x-label: Word search timestamp
      description: An array of timestamps structured as [`start_time`, `end_time`] in milliseconds
      x-fern-sdk-group-name: transcripts
      type: array
      items:
        x-label: Timestamp
        description: Timestamp in milliseconds
        type: integer
      example: [250, 650]

    Timestamp:
      x-label: Timestamp
      description: Timestamp containing a start and end property in milliseconds
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      properties:
        start:
          x-label: Start
          description: The start time in milliseconds
          type: integer
        end:
          x-label: End
          description: The end time in milliseconds
          type: integer
      required:
        - start
        - end
      example: { start: 3978, end: 5114 }

    # This type is used by the Transcriber
    TranscriptOptionalParams:
      x-label: Optional transcript parameters
      description: The parameters for creating a transcript
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      properties:
        audio_end_at:
          x-label: Audio end at
          description: The point in time, in milliseconds, to stop transcribing in your media file. See [Set the start and end of the transcript](https://www.assemblyai.com/docs/pre-recorded-audio/set-the-start-and-end-of-the-transcript) for more details.
          type: integer

        audio_start_from:
          x-label: Audio start from
          description: The point in time, in milliseconds, to begin transcribing in your media file. See [Set the start and end of the transcript](https://www.assemblyai.com/docs/pre-recorded-audio/set-the-start-and-end-of-the-transcript) for more details.
          type: integer

        auto_chapters:
          x-label: Auto chapters
          description: |
            Enable [Auto Chapters](https://www.assemblyai.com/docs/speech-understanding/auto-chapters), can be true or false. Requires `punctuate` to be `true`, and cannot be enabled together with `summarization`. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible chapter summaries. See the [updated Auto Chapters page](https://www.assemblyai.com/docs/speech-understanding/auto-chapters) for details.

            Note: This parameter is only supported for the Universal-2 model.
          type: boolean
          default: false
          deprecated: true

        auto_highlights:
          x-label: Key phrases
          description: Enable [Key Phrases](https://www.assemblyai.com/docs/speech-understanding/key-phrases), either true or false
          type: boolean
          default: false

        content_safety:
          x-label: Content Moderation
          description: Enable [Content Moderation](https://www.assemblyai.com/docs/content-moderation), can be true or false
          type: boolean
          default: false

        content_safety_confidence:
          x-label: Content Moderation confidence
          description: The confidence threshold for the [Content Moderation](https://www.assemblyai.com/docs/content-moderation) model. Values must be between 25 and 100. Requires `content_safety` to be enabled; otherwise it's ignored.
          type: integer
          default: 50
          minimum: 25
          maximum: 100

        custom_spelling:
          x-label: Custom spellings
          description: Customize how words are spelled and formatted using to and from values. Each `to` value must be a single word, and each `from` phrase can contain at most 5 words. See [Custom Spelling](https://www.assemblyai.com/docs/pre-recorded-audio/correct-spelling-of-terms) for more details.
          type: array
          items:
            x-label: Custom spelling
            $ref: "#/components/schemas/TranscriptCustomSpelling"

        disfluencies:
          x-label: Disfluencies
          description: Transcribe [Filler Words](https://www.assemblyai.com/docs/pre-recorded-audio/include-filler-words), like "umm", in your media file; can be true or false. Supported on Universal-3.5 Pro and Universal-2.
          type: boolean
          default: false

        domain:
          x-label: Domain
          description: |
            Enable domain-specific transcription models to improve accuracy for specialized terminology. Set to `"medical-v1"` to enable [Medical Mode](https://www.assemblyai.com/docs/pre-recorded-audio/medical-mode) for improved accuracy of medical terms such as medications, procedures, conditions, and dosages.

            Supported languages: English (`en`), Spanish (`es`), German (`de`), French (`fr`). If `medical-v1` is used with an unsupported language, the parameter is ignored and a warning is returned.
          type:
            - string
            - "null"
          enum:
            - medical-v1
            - null
          default: null

        entity_detection:
          x-label: Entity Detection
          description: Enable [Entity Detection](https://www.assemblyai.com/docs/speech-understanding/entity-detection), can be true or false
          type: boolean
          default: false

        filter_profanity:
          x-label: Filter profanity
          description: Filter profanity from the transcribed text, can be true or false. See [Profanity Filtering](https://www.assemblyai.com/docs/profanity-filtering) for more details.
          type: boolean
          default: false

        format_text:
          x-label: Format text
          description: Enable [Text Formatting](https://www.assemblyai.com/docs/pre-recorded-audio), can be true or false
          type: boolean
          default: true

        iab_categories:
          x-label: Topic Detection
          description: Enable [Topic Detection](https://www.assemblyai.com/docs/speech-understanding/topic-detection), can be true or false
          type: boolean
          default: false

        keyterms_prompt:
          x-label: Keyterms prompt
          description: |
            Improve accuracy with up to 200 (for Universal-2) or 1000 (for Universal-3.5 Pro) domain-specific words or phrases (maximum 6 words per phrase). See [Keyterms Prompting](https://www.assemblyai.com/docs/pre-recorded-audio/universal-3-5-pro/prompting#keyterms-prompting) for more details.
          type: array
          items:
            x-label: Keyterm
            type: string

        language_code:
          x-label: Language code
          description: |
            The language of your audio file. Possible values are found in [Supported Languages](https://www.assemblyai.com/docs/pre-recorded-audio/supported-languages).
            If you don't specify a language, it's detected automatically. Cannot be used together with `language_detection`.
          oneOf:
            - anyOf:
                - $ref: "#/components/schemas/TranscriptLanguageCode"
                - type: string
            - type: "null"
          x-ts-type: LiteralUnion<TranscriptLanguageCode, string> | null
          x-go-type: TranscriptLanguageCode

        language_codes:
          description: |
            The language codes of your audio file. Used for [Code switching](/speech-to-text/pre-recorded-audio/code-switching)
            One of the values specified must be `en`.
          type: [array, "null"]
          items:
            x-label: Language code
            $ref: "#/components/schemas/TranscriptLanguageCode"

        language_confidence_threshold:
          x-label: Language confidence threshold
          description: |
            The confidence threshold for the automatically detected language.
            An error will be returned if the language confidence is below this threshold.
            Defaults to 0. Can only be set when `language_detection` is enabled. See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
          type: number
          format: float
          minimum: 0
          maximum: 1
          default: 0

        language_detection:
          x-label: Language detection
          description: |
            [Automatic language detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) identifies the spoken language and routes the request to the best model. It's applied automatically when you don't specify a `language_code`. Set to `false` only together with a `language_code`; disabling it without specifying a language returns an error.
          type: boolean

        language_detection_options:
          x-label: Specify options for Automatic Language Detection.
          description: Specify options for [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection).
          type: object
          additionalProperties: false
          properties:
            expected_languages:
              x-label: Expected languages
              description: List of languages expected in the audio file. Defaults to `["all"]` when unspecified. See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
              type: array
              items:
                x-label: language
                type: string
            fallback_language:
              x-label: Fallback language
              description: |
                If the detected language of the audio file is not in the list of expected languages, the `fallback_language` is used. Specify `["auto"]` to let our model choose the fallback language from `expected_languages` with the highest confidence score. See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
              type: string
              default: "auto"
            code_switching:
              x-label: Code switching
              description: |
                Whether [code switching](/speech-to-text/pre-recorded-audio/code-switching) should be detected.
              type: boolean
              default: false
            code_switching_confidence_threshold:
              x-label: Code switching confidence threshold
              description: |
                The confidence threshold for [code switching](/speech-to-text/pre-recorded-audio/code-switching) detection. If the code switching confidence is below this threshold, the transcript will be processed in the language with the highest `language_detection_confidence` score.
              type: number
              minimum: 0
              maximum: 1
              default: 0.3
            localization:
              x-label: Localization
              description: |
                Render the transcript in a regional variant of the detected language. Supported values are `en_au` (Australian English) and `en_uk` (British English) — only English is supported today, and you can specify at most one locale per base language. Base or default-region codes such as `en` and `en_us` are not localization variants and return a `400`.

                When the detected language matches the requested locale's base language, the transcript uses that locale's spelling and `language_code` returns the region-aware code (for example `en_au`) instead of the base `en`. Otherwise, when the detected language isn't available as a locale, the option is ignored. See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
              type: array
              items:
                x-label: locale
                type: string

        multichannel:
          x-label: Multichannel
          description: Enable [Multichannel](https://www.assemblyai.com/docs/pre-recorded-audio/transcribe-multiple-audio-channels) transcription, can be true or false.
          type: boolean
          default: false

        prompt:
          x-label: Prompt
          description: |
            Provide natural language prompting of up to 1,500 words of contextual information to the model. See the [Prompting Guide](https://www.assemblyai.com/docs/pre-recorded-audio/prompting) for best practices.

            Note: This parameter is only supported for the Universal-3.5 Pro model.
          type: string

        punctuate:
          x-label: Punctuate
          description: Enable [Automatic Punctuation](https://www.assemblyai.com/docs/pre-recorded-audio), can be true or false
          type: boolean
          default: true

        redact_pii:
          x-label: Redact PII
          description: Redact PII from the transcribed text using the Redact PII model, can be true or false. Requires `format_text` to be `true`. See [PII Redaction](https://www.assemblyai.com/docs/pii-redaction) for more details.
          type: boolean
          default: false

        redact_pii_audio:
          x-label: Redact PII audio
          description: Generate a copy of the original media file with spoken PII "beeped" out, can be true or false. Requires `redact_pii` to be `true`. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction#request-for-redacted-audio) for more details.
          type: boolean
          default: false

        redact_pii_audio_options:
          x-label: Specify options for PII redacted audio files.
          description: Specify options for [PII redacted audio](https://www.assemblyai.com/docs/pii-redaction#request-for-redacted-audio) files.
          type: object
          additionalProperties: false
          properties:
            return_redacted_no_speech_audio:
              x-label: Return redacted no speech audio
              description: By default, audio redaction provides redacted audio URLs only when speech is detected. However, if your use-case specifically requires redacted audio files even for silent audio files without any dialogue, you can opt to receive these URLs by setting this parameter to `true`. Requires `redact_pii_audio` to be `true`.
              type: boolean
              default: false
            override_audio_redaction_method:
              x-label: Override audio redaction method
              description: Specify the method used to redact audio. By default, redacted audio uses a beep sound. Set to `silence` to replace PII with silence instead of a beep.
              type: string
              enum:
                - silence

        redact_pii_audio_quality:
          x-label: Redact PII audio quality
          description: Controls the filetype of the audio created by redact_pii_audio. Currently supports mp3 (default) and wav. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction#request-for-redacted-audio) for more details.
          default: mp3
          $ref: "#/components/schemas/RedactPiiAudioQuality"

        redact_pii_policies:
          x-label: Redact PII policies
          description: The list of PII Redaction policies to enable. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more details.
          type: array
          items:
            x-label: PII policy
            $ref: "#/components/schemas/PiiPolicy"

        redact_pii_sub:
          x-label: Redact PII substitution
          description: The replacement logic for detected PII, can be `entity_name` or `hash`. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more details.
          oneOf:
            - $ref: "#/components/schemas/SubstitutionPolicy"
            - type: "null"
          default: hash

        redact_pii_return_unredacted:
          x-label: Return unredacted transcript
          description: |
            When set to `true`, returns the original unredacted transcript alongside the redacted one in the same response. Requires `redact_pii` to be `true`, otherwise a 400 error is returned.

            When enabled, the response includes the additional fields `unredacted_text`, `unredacted_words`, and `unredacted_utterances`. The existing `text`, `words`, and `utterances` fields remain fully redacted. When disabled (default), the response is unchanged and contains only the redacted transcript. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more details.
          type: boolean
          default: false

        redact_static_entities:
          x-label: Redact static entities
          description: |
            A map of user-defined terms to redact, where each key is a redaction label and each value is a list of exact terms to match (e.g. `{ "INTERNAL_TOOL": ["Bearclaw", "Cubclaw"] }`). Each matching term in the transcript is redacted using the `redact_pii_sub` substitution, on top of standard PII Redaction. Useful for redacting specific, predefined terms (proprietary names, internal codenames) that aren't general PII categories.

            This is a literal find-and-replace (tolerant of casing, surrounding punctuation, and minor spacing/hyphenation), not a model — it does not generalize beyond the terms you provide. Requires `redact_pii` to be `true`, otherwise a 400 error is returned. When `redact_pii_audio` is enabled, matched terms are also redacted in the audio output. You can provide up to 100 labels, each with up to 200 terms of at most 200 characters; a label may contain only letters, numbers, spaces, underscores, and hyphens (max 80 characters). See [Static Entity Redaction](https://www.assemblyai.com/docs/guardrails/redact-pii-from-transcripts#static-entity-redaction) for more details.
          type: object
          additionalProperties:
            type: array
            items:
              type: string

        sentiment_analysis:
          x-label: Sentiment Analysis
          description: Enable [Sentiment Analysis](https://www.assemblyai.com/docs/speech-understanding/sentiment-analysis), can be true or false. Requires `punctuate` to be `true`.
          type: boolean
          default: false

        speaker_labels:
          x-label: Speaker labels
          description: Enable [Speaker diarization](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers), can be true or false. Requires `punctuate` to be `true`.
          type: boolean
          default: false

        speaker_options:
          x-label: Specify options for speaker diarization.
          description: Specify options for [Speaker diarization](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers#set-a-range-of-possible-speakers). Use this to set a range of possible speakers. Requires `speaker_labels` to be `true`, and cannot be used together with `speakers_expected`. When both bounds are set, `min_speakers_expected` must be less than or equal to `max_speakers_expected`.
          type: object
          additionalProperties: false
          properties:
            min_speakers_expected:
              x-label: Minimum speakers expected
              description: A hard lower limit on the number of speaker labels — the model won't return fewer speakers than this. See [Set a range of possible speakers](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers#set-a-range-of-possible-speakers) for more details.
              type: integer
            max_speakers_expected:
              x-label: Maximum speakers expected
              description: |
                <Warning>Setting this parameter too high may hurt model accuracy</Warning>
                A hard upper limit on the number of speaker labels. If more people speak than this value, the additional speakers are merged into existing labels. Setting it higher than the true number of speakers can cause the model to over-split and return more speakers than are actually present. The default depends on audio duration: no limit for 0-2 minutes, 10 for 2-10 minutes, and 30 for 10+ minutes. See [Set a range of possible speakers](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers#set-a-range-of-possible-speakers) for more details.
              type: integer

        speakers_expected:
          x-label: Speakers expected
          description: Tells the speaker label model how many speakers it should attempt to identify. Requires `speaker_labels` to be `true` and must be a positive integer; cannot be used together with `speaker_options`. See [Set number of speakers expected](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers#set-number-of-speakers-expected) for more details.
          type: [integer, "null"]
          default: null

        speech_models:
          x-label: Speech models
          description: |
            Optional. Supported values: `universal-3-5-pro`, `universal-2`. If omitted, defaults to `["universal-3-5-pro", "universal-2"]`. See [Model Selection](https://www.assemblyai.com/docs/pre-recorded-audio/select-the-speech-model) for available models and routing behavior.
          type: array
          items:
            x-label: Speech model
            $ref: "#/components/schemas/SpeechModel"
          default:
            - universal-3-5-pro
            - universal-2

        speech_threshold:
          x-label: Speech threshold
          description: |
            Reject audio files that contain less than this fraction of speech.
            Valid values are in the range [0, 1] inclusive. See [Speech Threshold](https://www.assemblyai.com/docs/speech-threshold) for more details.
          type: [number, "null"]
          format: float
          minimum: 0
          maximum: 1
          default: 0

        speech_understanding:
          x-label: Speech Understanding
          description: |
            Enable speech understanding tasks like [Translation](https://www.assemblyai.com/docs/speech-understanding/translation), [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification), and [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting). See the task-specific docs for available options and configuration.
          type: object
          properties:
            request:
              oneOf:
                - $ref: "#/components/schemas/TranslationRequestBody"
                - $ref: "#/components/schemas/SpeakerIdentificationRequestBody"
                - $ref: "#/components/schemas/CustomFormattingRequestBody"
                - $ref: "#/components/schemas/SummarizationRequestBody"
                - $ref: "#/components/schemas/ActionItemsRequestBody"
          required:
            - request
        summarization:
          x-label: Enable Summarization
          description: |
            Enable [Summarization](https://www.assemblyai.com/docs/speech-understanding/summarization), can be true or false. Requires both `punctuate` and `format_text` to be `true`, and cannot be enabled together with `auto_chapters`. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible summaries. See the [updated Summarization page](https://www.assemblyai.com/docs/speech-understanding/summarization) for details.

            Note: This parameter is only supported for the Universal-2 model.
          type: boolean
          default: false
          deprecated: true

        summary_model:
          x-label: Summary model
          description: The model to summarize the transcript. Must be set together with `summary_type`. Compatibility - `catchy` supports `gist` and `headline`; `informative` and `conversational` support `headline`, `paragraph`, `bullets`, and `bullets_verbose`. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible summaries. See the [updated Summarization page](https://www.assemblyai.com/docs/speech-understanding/summarization) for details.
          default: informative
          deprecated: true
          $ref: "#/components/schemas/SummaryModel"

        summary_type:
          x-label: Summary type
          description: The type of summary. Must be set together with `summary_model`; see `summary_model` for the supported model and type combinations. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible summaries. See the [updated Summarization page](https://www.assemblyai.com/docs/speech-understanding/summarization) for details.
          default: bullets
          deprecated: true
          $ref: "#/components/schemas/SummaryType"

        remove_audio_tags:
          x-label: Remove audio tags
          description: |
            Universal-3.5 Pro generates rich transcripts that can include inline annotations such as audio event markers and speaker cues. Set to `"all"` to remove all inline annotations, or `"speaker"` to remove only speaker cues while keeping other annotations. By default, all inline annotations are removed.

            Note: This parameter is only supported for the Universal-3.5 Pro model.
          oneOf:
            - type: string
              enum:
                - all
                - speaker
            - type: "null"
          default: all

        temperature:
          x-label: Temperature
          description: |
            Control the amount of randomness injected into the model's response. See the [Prompting Guide](https://www.assemblyai.com/docs/pre-recorded-audio/prompting) for more details.

            Note: This parameter only takes effect on the Universal-3.5 Pro model.
          type: number
          minimum: 0.0
          maximum: 1.0
          default: 0.0

        webhook_auth_header_name:
          x-label: Webhook auth header name
          description: The header name to be sent with the transcript completed or failed [webhook](https://www.assemblyai.com/docs/deployment/webhooks-for-pre-recorded-audio) requests. Must be 1-1000 characters and contain only ASCII letters, numbers, hyphens, and underscores. Requires `webhook_auth_header_value` and `webhook_url` to also be set.
          type: [string, "null"]
          default: null

        webhook_auth_header_value:
          x-label: Webhook auth header value
          description: The header value to send back with the transcript completed or failed [webhook](https://www.assemblyai.com/docs/deployment/webhooks-for-pre-recorded-audio) requests for added security. Must be 1-1000 characters and must not contain carriage returns or newlines. Requires `webhook_auth_header_name` and `webhook_url` to also be set.
          type: [string, "null"]
          default: null

        webhook_url:
          x-label: Webhook URL
          description: |
            The URL to which we send [webhook](https://www.assemblyai.com/docs/deployment/webhooks-for-pre-recorded-audio) requests.
          type: string
          format: url

        custom_topics:
          x-label: Enable custom topics
          description: This parameter does not currently have any functionality attached to it.
          type: boolean
          default: false
          deprecated: true

        speech_model:
          x-label: Speech model
          description: |
            This parameter has been replaced with the `speech_models` parameter, learn more about the `speech_models` parameter [here](https://www.assemblyai.com/docs/pre-recorded-audio/select-the-speech-model).
          oneOf:
            - $ref: "#/components/schemas/SpeechModel"
            - type: "null"
          deprecated: True

        topics:
          x-label: Custom topics
          description: This parameter does not currently have any functionality attached to it.
          type: array
          items:
            x-label: Topic
            type: string
          deprecated: true

      example:
        {
          speech_model: null,
          language_detection: true,
          language_confidence_threshold: 0.7,
          punctuate: true,
          format_text: true,
          multichannel: true,
          webhook_url: "https://your-webhook-url.tld/path",
          webhook_auth_header_name: "webhook-secret",
          webhook_auth_header_value: "webhook-secret-value",
          auto_highlights: true,
          audio_start_from: 10,
          audio_end_at: 280,
          filter_profanity: true,
          redact_pii: true,
          redact_pii_audio: true,
          redact_pii_audio_quality: "mp3",
          redact_pii_policies:
            ["us_social_security_number", "credit_card_number"],
          redact_pii_sub: "hash",
          speaker_labels: true,
          speakers_expected: 2,
          content_safety: true,
          iab_categories: true,
          custom_spelling: [],
          disfluencies: false,
          sentiment_analysis: true,
          auto_chapters: false,
          entity_detection: true,
          speech_threshold: 0.5,
          summarization: false,
          summary_model: null,
          summary_type: null,
          custom_topics: true,
          topics: [],
          remove_audio_tags: null,
          domain: null,
          speech_understanding:
            {
              request:
                {
                  translation:
                    {
                      target_languages: ["es", "de"],
                      formal: true,
                      match_original_utterance: true,
                    },
                },
            },
        }

    TranscriptParams:
      x-label: Transcript parameters
      description: The parameters for creating a transcript
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      allOf:
        - type: object
          required: [audio_url]
          properties:
            audio_url:
              x-label: Audio URL
              description: The URL of the audio or video file to transcribe.
              type: string
              format: url
          additionalProperties: false
        - $ref: "#/components/schemas/TranscriptOptionalParams"
      example:
        {
          speech_model: null,
          language_detection: true,
          language_confidence_threshold: 0.7,
          audio_url: "https://assembly.ai/wildfires.mp3",
          punctuate: true,
          format_text: true,
          multichannel: true,
          webhook_url: "https://your-webhook-url/path",
          webhook_auth_header_name: "webhook-secret",
          webhook_auth_header_value: "webhook-secret-value",
          auto_highlights: true,
          audio_start_from: 10,
          audio_end_at: 280,
          filter_profanity: true,
          redact_pii: true,
          redact_pii_audio: true,
          redact_pii_audio_quality: "mp3",
          redact_pii_policies:
            ["us_social_security_number", "credit_card_number"],
          redact_pii_sub: "hash",
          speaker_labels: true,
          speakers_expected: 2,
          content_safety: true,
          iab_categories: true,
          custom_spelling: [],
          disfluencies: false,
          sentiment_analysis: true,
          auto_chapters: false,
          entity_detection: true,
          speech_threshold: 0.5,
          summarization: false,
          summary_model: null,
          summary_type: null,
          custom_topics: true,
          topics: [],
          remove_audio_tags: null,
          domain: null,
          speech_understanding:
            {
              request:
                {
                  translation:
                    {
                      target_languages: ["es", "de"],
                      formal: true,
                      match_original_utterance: true,
                    },
                },
            },
        }

    SummaryModel:
      type: string
      x-label: Summary model
      description: The model to summarize the transcript
      x-fern-sdk-group-name: transcripts
      enum:
        - informative
        - conversational
        - catchy
      x-aai-enum:
        informative:
          label: Informative
        conversational:
          label: Conversational
        catchy:
          label: Catchy

    SummaryType:
      type: string
      x-label: Summary type
      description: The type of summary
      x-fern-sdk-group-name: transcripts
      enum:
        - bullets
        - bullets_verbose
        - gist
        - headline
        - paragraph
      x-aai-enum:
        bullets:
          label: Bullets
        bullets_verbose:
          label: Bullets verbose
        gist:
          label: Gist
        headline:
          label: Headline
        paragraph:
          label: Paragraph

    TranscriptCustomSpelling:
      x-label: Custom spelling
      description: Object containing words or phrases to replace, and the word or phrase to replace with
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      properties:
        from:
          x-label: From
          description: Words or phrases to replace
          type: array
          items:
            x-label: Word or phrase
            description: Word or phrase to replace
            type: string
        to:
          x-label: To
          description: Word to replace with
          type: string
      required: [from, to]
      example: { from: ["dicarlo"], to: "Decarlo" }

    TranscriptUtterance:
      type: object
      x-label: Utterance
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      properties:
        confidence:
          x-label: Confidence
          description: The confidence score for the transcript of this utterance
          type: number
          format: double
          minimum: 0
          maximum: 1
        start:
          x-label: Start
          description: The starting time, in milliseconds, of the utterance in the audio file
          type: integer
        end:
          x-label: End
          description: The ending time, in milliseconds, of the utterance in the audio file
          type: integer
        text:
          x-label: Text
          description: The text for this utterance
          type: string
        words:
          x-label: Words
          description: The words in the utterance.
          type: array
          items:
            x-label: Word
            $ref: "#/components/schemas/TranscriptWord"
        channel:
          x-label: Channel
          description: The channel of this utterance. The left and right channels are channels 1 and 2. Additional channels increment the channel number sequentially.
          type: [string, "null"]
        speaker:
          x-label: Speaker
          description: The speaker of this utterance, where each speaker is assigned a sequential capital letter - e.g. "A" for Speaker A, "B" for Speaker B, etc.
          type: string
        translated_texts:
          x-label: Translated texts
          description: 'Translations keyed by language code (e.g., `{"es": "Texto traducido", "de": "Übersetzter Text"}`). Only present when `match_original_utterance` is enabled with translation.'
          type: object
          additionalProperties:
            type: string
      required:
        - confidence
        - start
        - end
        - text
        - words
        - speaker
      example:
        {
          confidence: 0.9359033333333334,
          end: 26950,
          speaker: "A",
          start: 250,
          text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor.",
          words:
            [
              {
                text: "Smoke",
                start: 250,
                end: 650,
                confidence: 0.97503,
                speaker: "A",
              },
              {
                text: "from",
                start: 730,
                end: 1022,
                confidence: 0.99999,
                speaker: "A",
              },
              {
                text: "hundreds",
                start: 1076,
                end: 1418,
                confidence: 0.99843,
                speaker: "A",
              },
              {
                text: "of",
                start: 1434,
                end: 1614,
                confidence: 0.85,
                speaker: "A",
              },
              {
                text: "wildfires",
                start: 1652,
                end: 2346,
                confidence: 0.89657,
                speaker: "A",
              },
              {
                text: "in",
                start: 2378,
                end: 2526,
                confidence: 0.99994,
                speaker: "A",
              },
              {
                text: "Canada",
                start: 2548,
                end: 3130,
                confidence: 0.93864,
                speaker: "A",
              },
              {
                text: "is",
                start: 3210,
                end: 3454,
                confidence: 0.999,
                speaker: "A",
              },
              {
                text: "triggering",
                start: 3492,
                end: 3946,
                confidence: 0.75366,
                speaker: "A",
              },
              {
                text: "air",
                start: 3978,
                end: 4174,
                confidence: 1.0,
                speaker: "A",
              },
              {
                text: "quality",
                start: 4212,
                end: 4558,
                confidence: 0.87745,
                speaker: "A",
              },
              {
                text: "alerts",
                start: 4644,
                end: 5114,
                confidence: 0.94739,
                speaker: "A",
              },
              {
                text: "throughout",
                start: 5162,
                end: 5466,
                confidence: 0.99726,
                speaker: "A",
              },
              {
                text: "the",
                start: 5498,
                end: 5694,
                confidence: 0.79,
                speaker: "A",
              },
              {
                text: "US.",
                start: 5732,
                end: 6382,
                confidence: 0.88,
                speaker: "A",
              },
            ],
        }

    SubstitutionPolicy:
      x-label: Redact PII substitution
      type: string
      x-fern-sdk-group-name: transcripts
      description: The replacement logic for detected PII, can be `entity_name` or `hash`. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more details.
      enum:
        - entity_name
        - hash
      x-aai-enum:
        entity_name:
          label: Entity name
        hash:
          label: Hash

    RedactPiiAudioQuality:
      x-label: Redact PII audio quality
      type: string
      description: Controls the filetype of the audio created by redact_pii_audio. Currently supports mp3 (default) and wav. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction#request-for-redacted-audio) for more details.
      x-fern-sdk-group-name: transcripts
      enum:
        - mp3
        - wav
      x-fern-enum:
        mp3:
          description: MP3 audio format is lower quality and lower size than WAV.
          casing:
            camel: mp3
            snake: mp3
            pascal: Mp3
            screamingSnake: MP3
        wav:
          description: WAV audio format is the highest quality (no compression) and larger size than MP3.
      x-aai-enum:
        mp3:
          label: MP3
        wav:
          label: WAV
      example: "mp3"

    PiiPolicy:
      x-label: PII policy
      description: The type of PII to redact
      x-fern-sdk-group-name: transcripts
      type: string
      enum:
        - account_number
        - banking_information
        - blood_type
        - credit_card_cvv
        - credit_card_expiration
        - credit_card_number
        - date
        - date_interval
        - date_of_birth
        - drivers_license
        - drug
        - duration
        - email_address
        - event
        - filename
        - gender
        - gender_sexuality
        - healthcare_number
        - injury
        - ip_address
        - language
        - location
        - location_address
        - location_address_street
        - location_city
        - location_coordinate
        - location_country
        - location_state
        - location_zip
        - marital_status
        - medical_condition
        - medical_process
        - money_amount
        - nationality
        - number_sequence
        - occupation
        - organization
        - organization_medical_facility
        - passport_number
        - password
        - person_age
        - person_name
        - phone_number
        - physical_attribute
        - political_affiliation
        - religion
        - sexuality
        - statistics
        - time
        - url
        - us_social_security_number
        - username
        - vehicle_id
        - zodiac_sign
      x-fern-enum:
        account_number:
          description:
            "Customer account or membership identification number (e.g., Policy
            No. 10042992, Member ID: HZ-5235-001)"
        banking_information:
          description: Banking information, including account and routing numbers
        blood_type:
          description: Blood type (e.g., O-, AB positive)
        credit_card_cvv:
          description: "Credit card verification code (e.g., CVV: 080)"
        credit_card_expiration:
          description: Expiration date of a credit card
        credit_card_number:
          description: Credit card number
        date:
          description: Specific calendar date (e.g., December 18)
        date_interval:
          description:
            Broader time periods, including date ranges, months, seasons, years,
            and decades (e.g., 2020-2021, 5-9 May, January 1984)
        date_of_birth:
          description: "Date of birth (e.g., Date of Birth: March 7,1961)"
        drivers_license:
          description: Driver's license number. (e.g., DL# 356933-540)
        drug:
          description:
            Medications, vitamins, or supplements (e.g., Advil, Acetaminophen,
            Panadol)
        duration:
          description:
            Measurements of time expressed as a numerical value plus a unit (e.g.,
            8 months, 2 years)
        email_address:
          description: Email address (e.g., support@assemblyai.com)
        event:
          description: Name of an event or holiday (e.g., Olympics, Yom Kippur)
        filename:
          description:
            Names of computer files, including the extension or filepath (e.g.,
            Taxes/2012/brad-tax-returns.pdf)
        gender:
          description: Terms indicating gender identity (e.g., female, male, non-binary)
        gender_sexuality:
          description:
            Terms indicating gender identity or sexual orientation, including
            slang terms (e.g., female, bisexual, trans)
        healthcare_number:
          description:
            "Healthcare numbers and health plan beneficiary numbers (e.g.,
            Policy No.: 5584-486-674-YM)"
        injury:
          description: Bodily injury (e.g., I broke my arm, I have a sprained wrist)
        ip_address:
          description: Internet IP address, including IPv4 and IPv6 formats (e.g., 192.168.0.1)
        language:
          description: Name of a natural language (e.g., Spanish, French)
        location:
          description:
            Any Location reference including mailing address, postal code,
            city, state, province, country, or coordinates. (e.g., Lake Victoria, 145
            Windsor St., 90210)
        location_address:
          description: A complete mailing address (e.g., 145 Windsor St., Toronto, ON M5A 2P5)
        location_address_street:
          description: Street-level component of an address (e.g., 145 Windsor St.)
        location_city:
          description: City name (e.g., Toronto, San Francisco)
        location_coordinate:
          description: Geographic coordinates (e.g., 40.7128° N, 74.0060° W)
        location_country:
          description: Country name (e.g., Canada, United States)
        location_state:
          description: State, province, or region name (e.g., Ontario, California)
        location_zip:
          description: Postal or ZIP code (e.g., M5A 2P5, 90210)
        marital_status:
          description:
            Terms indicating marital status (e.g., Single, common-law, ex-wife,
            married)
        medical_condition:
          description:
            Name of a medical condition, disease, syndrome, deficit, or disorder
            (e.g., chronic fatigue syndrome, arrhythmia, depression)
        medical_process:
          description:
            Medical process, including treatments, procedures, and tests (e.g.,
            heart surgery, CT scan)
        money_amount:
          description: Name and/or amount of currency (e.g., 15 pesos, $94.50)
        nationality:
          description:
            Terms indicating nationality, ethnicity, or race (e.g., American,
            Asian, Caucasian)
        number_sequence:
          description:
            Numerical PII (including alphanumeric strings) that doesn't fall
            under other categories
        occupation:
          description: Job title or profession (e.g., professor, actors, engineer, CPA)
        organization:
          description:
            Name of an organization (e.g., CNN, McDonalds, University of Alaska,
            Northwest General Hospital)
        organization_medical_facility:
          description:
            Name of a medical facility (e.g., Mayo Clinic, Northwest General
            Hospital)
        passport_number:
          description: Passport numbers, issued by any country (e.g., PA4568332, NU3C6L86S12)
        password:
          description:
            Account passwords, PINs, access keys, or verification answers (e.g.,
            27%alfalfa, temp1234, My mother's maiden name is Smith)
        person_age:
          description: Number associated with an age (e.g., 27, 75)
        person_name:
          description: Name of a person (e.g., Bob, Doug Jones, Dr. Kay Martinez, MD)
        phone_number:
          description: Telephone or fax number
        physical_attribute:
          description: Distinctive bodily attributes, including race
            (e.g., I'm 190cm tall, He belongs to the Black students' association)
        political_affiliation:
          description:
            Terms referring to a political party, movement, or ideology (e.g.,
            Republican, Liberal)
        religion:
          description: Terms indicating religious affiliation (e.g., Hindu, Catholic)
        sexuality:
          description: Terms indicating sexual orientation (e.g., heterosexual, bisexual, gay)
        statistics:
          description: Medical statistics (e.g., 18%, 18 percent)
        time:
          description: Expressions indicating clock times (e.g., 19:37:28, 10pm EST)
        url:
          description: Internet addresses (e.g., https://www.assemblyai.com/)
        us_social_security_number:
          description: Social Security Number or equivalent
        username:
          description: Usernames, login names, or handles (e.g., @AssemblyAI)
        vehicle_id:
          description:
            Vehicle identification numbers (VINs), vehicle serial numbers,
            and license plate numbers (e.g., 5FNRL38918B111818, BIF7547)
        zodiac_sign:
          description: Names of Zodiac signs (e.g., Aries, Taurus)
      x-aai-enum:
        account_number:
          label: Account number
        banking_information:
          label: Banking information
        blood_type:
          label: Blood type
        credit_card_cvv:
          label: Credit card CVV
        credit_card_expiration:
          label: Credit card expiration
        credit_card_number:
          label: Credit card number
        date:
          label: Date
        date_interval:
          label: Date interval
        date_of_birth:
          label: Date of birth
        drivers_license:
          label: Driver's license
        drug:
          label: Drug
        duration:
          label: Duration
        email_address:
          label: Email address
        event:
          label: Event
        filename:
          label: Filename
        gender:
          label: Gender
        gender_sexuality:
          label: Gender sexuality
        healthcare_number:
          label: Healthcare number
        injury:
          label: Injury
        ip_address:
          label: IP address
        language:
          label: Language
        location:
          label: Location
        location_address:
          label: Location address
        location_address_street:
          label: Location address street
        location_city:
          label: Location city
        location_coordinate:
          label: Location coordinate
        location_country:
          label: Location country
        location_state:
          label: Location state
        location_zip:
          label: Location ZIP
        marital_status:
          label: Marital status
        medical_condition:
          label: Medical condition
        medical_process:
          label: Medical process
        money_amount:
          label: Money amount
        nationality:
          label: Nationality
        number_sequence:
          label: Number sequence
        occupation:
          label: Occupation
        organization:
          label: Organization
        organization_medical_facility:
          label: Organization medical facility
        passport_number:
          label: Passport number
        password:
          label: Password
        person_age:
          label: Person age
        person_name:
          label: Person name
        phone_number:
          label: Phone number
        physical_attribute:
          label: Physical attribute
        political_affiliation:
          label: Political affiliation
        religion:
          label: Religion
        sexuality:
          label: Sexuality
        statistics:
          label: Statistics
        time:
          label: Time
        url:
          label: URL
        us_social_security_number:
          label: US Social Security Number
        username:
          label: Username
        vehicle_id:
          label: Vehicle ID
        zodiac_sign:
          label: Zodiac sign

    SpeechModel:
      x-label: Speech model
      type: string
      description: The speech model to use for the transcription. See [Model Selection](https://www.assemblyai.com/docs/pre-recorded-audio/select-the-speech-model) for available models.
      enum:
        - universal-3-5-pro
        - universal-2
      x-fern-sdk-group-name: transcripts

    TranscriptLanguageCode:
      x-label: Language code
      type: string
      description: |
        The language of your audio file. Possible values are found in [Supported Languages](https://www.assemblyai.com/docs/pre-recorded-audio/supported-languages).
      x-fern-sdk-group-name: transcripts
      enum:
        - en
        - en_au
        - en_uk
        - en_us
        - es
        - fr
        - de
        - it
        - pt
        - nl
        - af
        - sq
        - am
        - ar
        - hy
        - as
        - az
        - ba
        - eu
        - be
        - bn
        - bs
        - br
        - bg
        - my
        - ca
        - zh
        - hr
        - cs
        - da
        - et
        - fo
        - fi
        - gl
        - ka
        - el
        - gu
        - ht
        - ha
        - haw
        - he
        - hi
        - hu
        - is
        - id
        - ja
        - jw
        - kn
        - kk
        - km
        - ko
        - lo
        - la
        - lv
        - ln
        - lt
        - lb
        - mk
        - mg
        - ms
        - ml
        - mt
        - mi
        - mr
        - mn
        - ne
        - no
        - nn
        - oc
        - pa
        - ps
        - fa
        - pl
        - ro
        - ru
        - sa
        - sr
        - sn
        - sd
        - si
        - sk
        - sl
        - so
        - su
        - sw
        - sv
        - tl
        - tg
        - ta
        - tt
        - te
        - th
        - bo
        - tr
        - tk
        - uk
        - ur
        - uz
        - vi
        - cy
        - yi
        - yo
      x-aai-enum:
        en:
          label: English (global)
        en_au:
          label: English (Australian)
        en_uk:
          label: English (British)
        en_us:
          label: English (US)
        es:
          label: Spanish
        fr:
          label: French
        de:
          label: German
        it:
          label: Italian
        pt:
          label: Portuguese
        nl:
          label: Dutch
        af:
          label: Afrikaans
        sq:
          label: Albanian
        am:
          label: Amharic
        ar:
          label: Arabic
        hy:
          label: Armenian
        as:
          label: Assamese
        az:
          label: Azerbaijani
        ba:
          label: Bashkir
        eu:
          label: Basque
        be:
          label: Belarusian
        bn:
          label: Bengali
        bs:
          label: Bosnian
        br:
          label: Breton
        bg:
          label: Bulgarian
        my:
          label: Burmese
        ca:
          label: Catalan
        zh:
          label: Chinese
        hr:
          label: Croatian
        cs:
          label: Czech
        da:
          label: Danish
        et:
          label: Estonian
        fo:
          label: Faroese
        fi:
          label: Finnish
        gl:
          label: Galician
        ka:
          label: Georgian
        el:
          label: Greek
        gu:
          label: Gujarati
        ht:
          label: Haitian
        ha:
          label: Hausa
        haw:
          label: Hawaiian
        he:
          label: Hebrew
        hi:
          label: Hindi
        hu:
          label: Hungarian
        is:
          label: Icelandic
        id:
          label: Indonesian
        ja:
          label: Japanese
        jw:
          label: Javanese
        kn:
          label: Kannada
        kk:
          label: Kazakh
        km:
          label: Khmer
        ko:
          label: Korean
        lo:
          label: Lao
        la:
          label: Latin
        lv:
          label: Latvian
        ln:
          label: Lingala
        lt:
          label: Lithuanian
        lb:
          label: Luxembourgish
        mk:
          label: Macedonian
        mg:
          label: Malagasy
        ms:
          label: Malay
        ml:
          label: Malayalam
        mt:
          label: Maltese
        mi:
          label: Maori
        mr:
          label: Marathi
        mn:
          label: Mongolian
        ne:
          label: Nepali
        no:
          label: Norwegian
        nn:
          label: Norwegian Nynorsk
        oc:
          label: Occitan
        pa:
          label: Panjabi
        ps:
          label: Pashto
        fa:
          label: Persian
        pl:
          label: Polish
        ro:
          label: Romanian
        ru:
          label: Russian
        sa:
          label: Sanskrit
        sr:
          label: Serbian
        sn:
          label: Shona
        sd:
          label: Sindhi
        si:
          label: Sinhala
        sk:
          label: Slovak
        sl:
          label: Slovenian
        so:
          label: Somali
        su:
          label: Sundanese
        sw:
          label: Swahili
        sv:
          label: Swedish
        tl:
          label: Tagalog
        tg:
          label: Tajik
        ta:
          label: Tamil
        tt:
          label: Tatar
        te:
          label: Telugu
        th:
          label: Thai
        bo:
          label: Tibetan
        tr:
          label: Turkish
        tk:
          label: Turkmen
        uk:
          label: Ukrainian
        ur:
          label: Urdu
        uz:
          label: Uzbek
        vi:
          label: Vietnamese
        cy:
          label: Welsh
        yi:
          label: Yiddish
        yo:
          label: Yoruba

    TranscriptStatus:
      x-label: Status
      type: string
      description: The status of your transcript. Possible values are queued, processing, completed, or error.
      x-fern-sdk-group-name: transcripts
      enum:
        - queued
        - processing
        - completed
        - error
      x-fern-enum:
        queued:
          description: The audio file is in the queue to be processed by the API.
        processing:
          description: The audio file is being processed by the API.
        completed:
          description: The transcript job has been completed successfully.
        error:
          description: An error occurred while processing the audio file.
      x-aai-enum:
        queued:
          label: Queued
        processing:
          label: Processing
        completed:
          label: Completed
        error:
          label: Error

    TranscriptReadyStatus:
      x-label: Status
      type: string
      description: The status of the transcript. Either completed or error.
      x-fern-sdk-group-name: transcripts
      enum:
        - completed
        - error
      x-fern-enum:
        completed:
          description: The transcript job has been completed successfully.
        error:
          description: An error occurred while processing the audio file.
      x-aai-enum:
        completed:
          label: Completed
        error:
          label: Error

    Transcript:
      x-label: Transcript
      description: A transcript object
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      properties:
        audio_channels:
          x-label: Audio channels
          description: The number of audio channels in the audio file. This is only present when [multichannel](https://www.assemblyai.com/docs/pre-recorded-audio/transcribe-multiple-audio-channels) is enabled.
          type: integer

        audio_duration:
          x-label: Audio duration
          description: The duration of this transcript object's media file, in seconds
          type: [integer, "null"]

        audio_end_at:
          x-label: Audio end at
          description: The point in time, in milliseconds, in the file at which the transcription was terminated. See [Set the start and end of the transcript](https://www.assemblyai.com/docs/pre-recorded-audio/set-the-start-and-end-of-the-transcript) for more details.
          type: [integer, "null"]

        audio_start_from:
          x-label: Audio start from
          description: The point in time, in milliseconds, in the file at which the transcription was started. See [Set the start and end of the transcript](https://www.assemblyai.com/docs/pre-recorded-audio/set-the-start-and-end-of-the-transcript) for more details.
          type: [integer, "null"]

        audio_url:
          x-label: Audio URL
          description: The URL of the media that was transcribed
          type: string
          format: url

        auto_chapters:
          x-label: Auto Chapters enabled
          description: |
            Whether [Auto Chapters](https://www.assemblyai.com/docs/speech-understanding/auto-chapters) is enabled, can be true or false. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible chapter summaries. See the [updated Auto Chapters page](https://www.assemblyai.com/docs/speech-understanding/auto-chapters) for details.

            Note: This parameter is only supported for the Universal-2 model.
          type: [boolean, "null"]
          deprecated: true

        auto_highlights:
          x-label: Key Phrases
          description: Whether [Key Phrases](https://www.assemblyai.com/docs/speech-understanding/key-phrases) is enabled, either true or false
          type: boolean

        auto_highlights_result:
          x-label: Key Phrases result
          description: |
            An array of results for the Key Phrases model, if it is enabled.
            See [Key Phrases](https://www.assemblyai.com/docs/speech-understanding/key-phrases) for more information.
          oneOf:
            - $ref: "#/components/schemas/AutoHighlightsResult"
            - type: "null"

        chapters:
          x-label: Chapters
          description: An array of temporally sequential chapters for the audio file. See [Auto Chapters](https://www.assemblyai.com/docs/speech-understanding/auto-chapters) for more information.
          type: [array, "null"]
          items:
            x-label: Chapter
            $ref: "#/components/schemas/Chapter"

        confidence:
          x-label: Confidence
          description: The confidence score for the transcript, between 0.0 (low confidence) and 1.0 (high confidence)
          type: [number, "null"]
          format: double
          minimum: 0
          maximum: 1

        content_safety:
          x-label: Content Moderation
          description: Whether [Content Moderation](https://www.assemblyai.com/docs/content-moderation) is enabled, can be true or false
          type: [boolean, "null"]

        content_safety_labels:
          x-label: Content Moderation labels
          description: |
            An array of results for the Content Moderation model, if it is enabled.
            See [Content moderation](https://www.assemblyai.com/docs/content-moderation) for more information.
          oneOf:
            - $ref: "#/components/schemas/ContentSafetyLabelsResult"
            - type: "null"

        custom_spelling:
          x-label: Custom spellings
          description: Customize how words are spelled and formatted using to and from values. Each `to` value must be a single word, and each `from` phrase can contain at most 5 words. See [Custom Spelling](https://www.assemblyai.com/docs/pre-recorded-audio/correct-spelling-of-terms) for more details.
          type: [array, "null"]
          items:
            x-label: Custom spelling
            $ref: "#/components/schemas/TranscriptCustomSpelling"

        disfluencies:
          x-label: Disfluencies
          description: Transcribe [Filler Words](https://www.assemblyai.com/docs/pre-recorded-audio/include-filler-words), like "umm", in your media file; can be true or false. Supported on Universal-3.5 Pro and Universal-2.
          type: [boolean, "null"]

        domain:
          x-label: Domain
          description: |
            The domain-specific model applied to the transcript. When set to `"medical-v1"`, [Medical Mode](https://www.assemblyai.com/docs/pre-recorded-audio/medical-mode) was used to improve accuracy for medical terminology.
          type: [string, "null"]

        entities:
          x-label: Entities
          description: |
            An array of results for the Entity Detection model, if it is enabled.
            See [Entity detection](https://www.assemblyai.com/docs/speech-understanding/entity-detection) for more information.
          type: [array, "null"]
          items:
            x-label: Entity
            $ref: "#/components/schemas/Entity"

        entity_detection:
          x-label: Entity Detection
          description: Whether [Entity Detection](https://www.assemblyai.com/docs/speech-understanding/entity-detection) is enabled, can be true or false
          type: [boolean, "null"]

        error:
          x-label: Error
          description: Error message of why the transcript failed
          type: string

        filter_profanity:
          x-label: Filter profanity
          description: Whether [Profanity Filtering](https://www.assemblyai.com/docs/profanity-filtering) is enabled, either true or false
          type: [boolean, "null"]

        format_text:
          x-label: Format text
          description: Whether [Text Formatting](https://www.assemblyai.com/docs/pre-recorded-audio) is enabled, either true or false
          type: [boolean, "null"]

        iab_categories:
          x-label: Topic Detection
          description: Whether [Topic Detection](https://www.assemblyai.com/docs/speech-understanding/topic-detection) is enabled, can be true or false
          type: [boolean, "null"]

        iab_categories_result:
          x-label: Topic Detection result
          description: |
            The result of the Topic Detection model, if it is enabled.
            See [Topic Detection](https://www.assemblyai.com/docs/speech-understanding/topic-detection) for more information.
          oneOf:
            - $ref: "#/components/schemas/TopicDetectionModelResult"
            - type: "null"

        id:
          x-label: ID
          description: The unique identifier of your transcript
          type: string
          format: uuid

        keyterms_prompt:
          x-label: Keyterms prompt
          description: |
            Improve accuracy with up to 200 (for Universal-2) or 1000 (for Universal-3.5 Pro) domain-specific words or phrases (maximum 6 words per phrase). See [Keyterms Prompting](https://www.assemblyai.com/docs/pre-recorded-audio/universal-3-5-pro/prompting#keyterms-prompting) for more details.
          type: array
          items:
            x-label: Keyterm
            type: string

        language_code:
          x-label: Language code
          description: |
            The language of your audio file.
            Possible values are found in [Supported Languages](https://www.assemblyai.com/docs/pre-recorded-audio/supported-languages).
          anyOf:
            - $ref: "#/components/schemas/TranscriptLanguageCode"
            - type: string
          x-ts-type: LiteralUnion<TranscriptLanguageCode, string>
          x-go-type: TranscriptLanguageCode

        language_codes:
          description: |
            The language codes of your audio file. Used for [Code switching](/speech-to-text/pre-recorded-audio/code-switching)
            One of the values specified must be `en`.
          type: [array, "null"]
          items:
            x-label: Language code
            $ref: "#/components/schemas/TranscriptLanguageCode"

        language_confidence:
          x-label: Language confidence
          description: The confidence score for the detected language, between 0.0 (low confidence) and 1.0 (high confidence). See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
          type: [number, "null"]
          format: double
          minimum: 0
          maximum: 1

        language_confidence_threshold:
          x-label: Language confidence threshold
          description: |
            The confidence threshold for the automatically detected language.
            An error will be returned if the language confidence is below this threshold.
            See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
          type: [number, "null"]
          format: float
          minimum: 0
          maximum: 1

        language_detection:
          x-label: Language detection
          description: Whether [Automatic language detection](/pre-recorded-audio/automatic-language-detection) is enabled, either true or false
          type: [boolean, "null"]

        language_detection_options:
          x-label: Specify options for Automatic Language Detection.
          description: Specify options for [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection).
          type: object
          additionalProperties: false
          properties:
            expected_languages:
              x-label: Expected languages
              description: List of languages expected in the audio file. Defaults to `["all"]` when unspecified. See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
              type: array
              items:
                x-label: language
                type: string
            fallback_language:
              x-label: Fallback language
              description: |
                If the detected language of the audio file is not in the list of expected languages, the `fallback_language` is used. Specify `["auto"]` to let our model choose the fallback language from `expected_languages` with the highest confidence score. See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
              type: string
              default: "auto"
            code_switching:
              x-label: Code switching
              description: |
                Whether [code switching](/speech-to-text/pre-recorded-audio/code-switching) should be detected.
              type: boolean
              default: false
            code_switching_confidence_threshold:
              x-label: Code switching confidence threshold
              description: |
                The confidence threshold for [code switching](/speech-to-text/pre-recorded-audio/code-switching) detection. If the code switching confidence is below this threshold, the transcript will be processed in the language with the highest `language_detection_confidence` score.
              type: number
              minimum: 0
              maximum: 1
              default: 0.3
            localization:
              x-label: Localization
              description: |
                Render the transcript in a regional variant of the detected language. Supported values are `en_au` (Australian English) and `en_uk` (British English) — only English is supported today, and you can specify at most one locale per base language. Base or default-region codes such as `en` and `en_us` are not localization variants and return a `400`.

                When the detected language matches the requested locale's base language, the transcript uses that locale's spelling and `language_code` returns the region-aware code (for example `en_au`) instead of the base `en`. Otherwise, when the detected language isn't available as a locale, the option is ignored. See [Automatic Language Detection](https://www.assemblyai.com/docs/pre-recorded-audio/language-detection) for more details.
              type: array
              items:
                x-label: locale
                type: string

        multichannel:
          x-label: Multichannel
          description: Whether [Multichannel transcription](https://www.assemblyai.com/docs/pre-recorded-audio/transcribe-multiple-audio-channels) was enabled in the transcription request, either true or false
          type: [boolean, "null"]

        prompt:
          x-label: Prompt
          description: |
            Provide natural language prompting of up to 1,500 words of contextual information to the model. See the [Prompting Guide](https://www.assemblyai.com/docs/pre-recorded-audio/prompting) for best practices.

            Note: This parameter is only supported for the Universal-3.5 Pro model.
          type: string

        punctuate:
          x-label: Punctuate
          description: Whether [Automatic Punctuation](https://www.assemblyai.com/docs/pre-recorded-audio) is enabled, either true or false
          type: [boolean, "null"]

        redact_pii:
          x-label: Redact PII
          description: Whether [PII Redaction](https://www.assemblyai.com/docs/pii-redaction) is enabled, either true or false
          type: boolean

        redact_pii_audio:
          x-label: Redact PII audio
          description: |
            Whether a redacted version of the audio file was generated,
            either true or false. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction#request-for-redacted-audio) for more information.
          type: [boolean, "null"]

        redact_pii_audio_options:
          x-label: Redact PII audio options
          description: |
            The options for PII-redacted audio, if redact_pii_audio is enabled.
            See [PII redaction](https://www.assemblyai.com/docs/pii-redaction#request-for-redacted-audio) for more information.
          type: object
          additionalProperties: false
          properties:
            return_redacted_no_speech_audio:
              x-label: Return redacted no speech audio
              description: By default, audio redaction provides redacted audio URLs only when speech is detected. However, if your use-case specifically requires redacted audio files even for silent audio files without any dialogue, you can opt to receive these URLs by setting this parameter to `true`. Requires `redact_pii_audio` to be `true`.
              type: boolean
              default: false
            override_audio_redaction_method:
              x-label: Override audio redaction method
              description: Specify the method used to redact audio. By default, redacted audio uses a beep sound. Set to `silence` to replace PII with silence instead of a beep.
              type: string
              enum:
                - silence

        redact_pii_audio_quality:
          x-label: Redact PII audio quality
          description: |
            The audio quality of the PII-redacted audio file, if redact_pii_audio is enabled.
            See [PII redaction](https://www.assemblyai.com/docs/pii-redaction#request-for-redacted-audio) for more information.
          oneOf:
            - $ref: "#/components/schemas/RedactPiiAudioQuality"
            - type: "null"

        redact_pii_policies:
          x-label: Redact PII policies
          description: |
            The list of PII Redaction policies that were enabled, if PII Redaction is enabled.
            See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more information.
          type: [array, "null"]
          items:
            x-label: PII policy
            $ref: "#/components/schemas/PiiPolicy"

        redact_pii_sub:
          x-label: Redact PII substitution
          description: The replacement logic for detected PII, can be `entity_name` or `hash`. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more details.
          $ref: "#/components/schemas/SubstitutionPolicy"

        redact_pii_return_unredacted:
          x-label: Return unredacted transcript
          description: |
            Whether the original unredacted transcript was also returned alongside the redacted one. When `true`, the response includes `unredacted_text`, `unredacted_words`, and `unredacted_utterances`. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more information.
          type: [boolean, "null"]

        sentiment_analysis:
          x-label: Sentiment Analysis
          description: Whether [Sentiment Analysis](https://www.assemblyai.com/docs/speech-understanding/sentiment-analysis) is enabled, can be true or false
          type: [boolean, "null"]

        sentiment_analysis_results:
          x-label: Sentiment Analysis results
          description: |
            An array of results for the Sentiment Analysis model, if it is enabled.
            See [Sentiment Analysis](https://www.assemblyai.com/docs/speech-understanding/sentiment-analysis) for more information.
          type: [array, "null"]
          items:
            x-label: Sentiment Analysis result
            $ref: "#/components/schemas/SentimentAnalysisResult"

        speaker_labels:
          x-label: Speaker labels
          description: Whether [Speaker diarization](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers) is enabled, can be true or false
          type: [boolean, "null"]

        speakers_expected:
          x-label: Speakers expected
          description: Tell the speaker label model how many speakers it should attempt to identify. See [Set number of speakers expected](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers#set-number-of-speakers-expected) for more details.
          type: [integer, "null"]

        speech_model_used:
          x-label: Speech model used
          description: The speech model that was actually used for the transcription. See [Model Selection](https://www.assemblyai.com/docs/pre-recorded-audio/select-the-speech-model) for available models.
          type: string
          $ref: "#/components/schemas/SpeechModel"

        speech_models:
          x-label: Speech models
          description: |
            List of speech models that were used (in priority order) to transcribe the audio. If not specified in the request, this defaults to `["universal-3-5-pro", "universal-2"]`. See [Model Selection](https://www.assemblyai.com/docs/pre-recorded-audio/select-the-speech-model) for available models and routing behavior.
          type: [array, "null"]
          items:
            x-label: Speech model
            $ref: "#/components/schemas/SpeechModel"

        speech_threshold:
          x-label: Speech threshold
          description: |
            Defaults to null. Reject audio files that contain less than this fraction of speech.
            Valid values are in the range [0, 1] inclusive. See [Speech Threshold](https://www.assemblyai.com/docs/speech-threshold) for more details.
          type: [number, "null"]
          minimum: 0
          maximum: 1
          format: float

        speech_understanding:
          x-label: Speech Understanding
          description: |
            Speech understanding tasks like [Translation](https://www.assemblyai.com/docs/speech-understanding/translation), [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification), and [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting). See the task-specific docs for available options and configuration.
          type: object
          properties:
            request:
              oneOf:
                - $ref: "#/components/schemas/TranslationRequestBody"
                - $ref: "#/components/schemas/SpeakerIdentificationRequestBody"
                - $ref: "#/components/schemas/CustomFormattingRequestBody"
                - $ref: "#/components/schemas/SummarizationRequestBody"
                - $ref: "#/components/schemas/ActionItemsRequestBody"
            response:
              oneOf:
                - $ref: "#/components/schemas/TranslationResponse"
                - $ref: "#/components/schemas/SpeakerIdentificationResponse"
                - $ref: "#/components/schemas/CustomFormattingResponse"
                - $ref: "#/components/schemas/SummarizationResponse"
                - $ref: "#/components/schemas/ActionItemsResponse"

        status:
          x-label: Status
          description: The status of your transcript. Possible values are queued, processing, completed, or error.
          $ref: "#/components/schemas/TranscriptStatus"

        summarization:
          x-label: Summarization enabled
          description: |
            Whether [Summarization](https://www.assemblyai.com/docs/speech-understanding/summarization) is enabled, either true or false. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible summaries. See the [updated Summarization page](https://www.assemblyai.com/docs/speech-understanding/summarization) for details.

            Note: This parameter is only supported for the Universal-2 model.
          type: boolean
          deprecated: true

        summary:
          x-label: Summary
          description: The generated summary of the media file, if [Summarization](https://www.assemblyai.com/docs/speech-understanding/summarization) is enabled. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible summaries. See the [updated Summarization page](https://www.assemblyai.com/docs/speech-understanding/summarization) for details.
          type: [string, "null"]
          deprecated: true

        summary_model:
          x-label: Summary model
          description: |
            The Summarization model used to generate the summary,
            if [Summarization](https://www.assemblyai.com/docs/speech-understanding/summarization) is enabled. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible summaries. See the [updated Summarization page](https://www.assemblyai.com/docs/speech-understanding/summarization) for details.
          type: [string, "null"]
          deprecated: true

        summary_type:
          x-label: Summary type
          description: The type of summary generated, if [Summarization](https://www.assemblyai.com/docs/speech-understanding/summarization) is enabled. Deprecated - use [LLM Gateway](https://www.assemblyai.com/docs/llm-gateway/quickstart) instead for more flexible summaries. See the [updated Summarization page](https://www.assemblyai.com/docs/speech-understanding/summarization) for details.
          type: [string, "null"]
          deprecated: true

        remove_audio_tags:
          x-label: Remove audio tags
          description: |
            Universal-3.5 Pro generates rich transcripts that can include inline annotations such as audio event markers and speaker cues. Set to `"all"` to remove all inline annotations, or `"speaker"` to remove only speaker cues while keeping other annotations. By default, all inline annotations are removed.

            Note: This parameter is only supported for the Universal-3.5 Pro model.
          oneOf:
            - type: string
              enum:
                - all
                - speaker
            - type: "null"

        temperature:
          x-label: Temperature
          description: |
            The temperature that was used for the model's response. See the [Prompting Guide](https://www.assemblyai.com/docs/pre-recorded-audio/prompting) for more details.

            Note: This parameter only takes effect on the Universal-3.5 Pro model.
          type: [number, "null"]
          minimum: 0.0
          maximum: 1.0

        text:
          x-label: Text
          description: The textual transcript of your media file
          type: [string, "null"]

        unredacted_text:
          x-label: Unredacted text
          description: |
            The original textual transcript of your media file before PII redaction was applied. Only returned when `redact_pii_return_unredacted` was set to `true` on the transcription request, otherwise this field is omitted and the `text` field remains fully redacted. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more information.
          type: [string, "null"]

        throttled:
          x-label: Throttled
          description: True while a request is throttled and false when a request is no longer throttled
          type: [boolean, "null"]

        utterances:
          x-label: Utterances
          description: |
            When multichannel or speaker_labels is enabled, a list of turn-by-turn utterance objects.
            See [Speaker diarization](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers) and [Multichannel transcription](https://www.assemblyai.com/docs/pre-recorded-audio/transcribe-multiple-audio-channels) for more information.
          type: [array, "null"]
          items:
            x-label: Utterance
            $ref: "#/components/schemas/TranscriptUtterance"

        unredacted_utterances:
          x-label: Unredacted utterances
          description: |
            The original turn-by-turn utterance objects before PII redaction was applied. Same shape as `utterances`. Only returned when `redact_pii_return_unredacted` was set to `true` on the transcription request, otherwise this field is omitted and the `utterances` field remains fully redacted. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more information.
          type: [array, "null"]
          items:
            x-label: Utterance
            $ref: "#/components/schemas/TranscriptUtterance"

        webhook_auth:
          x-label: Webhook auth enabled
          description: Whether [webhook](https://www.assemblyai.com/docs/deployment/webhooks-for-pre-recorded-audio) authentication details were provided
          type: boolean

        webhook_auth_header_name:
          x-label: Webhook auth header name
          description: The header name to be sent with the transcript completed or failed [webhook](https://www.assemblyai.com/docs/deployment/webhooks-for-pre-recorded-audio) requests. Must be 1-1000 characters and contain only ASCII letters, numbers, hyphens, and underscores. Requires `webhook_auth_header_value` and `webhook_url` to also be set.
          type: [string, "null"]

        webhook_status_code:
          x-label: Webhook HTTP status code
          description: The status code we received from your server when delivering the transcript completed or failed [webhook](https://www.assemblyai.com/docs/deployment/webhooks-for-pre-recorded-audio) request, if a webhook URL was provided
          type: [integer, "null"]

        webhook_url:
          x-label: Webhook URL
          description: |
            The URL to which we send [webhook](https://www.assemblyai.com/docs/deployment/webhooks-for-pre-recorded-audio) requests.
          type: [string, "null"]
          format: url

        words:
          x-label: Words
          description: |
            An array of temporally-sequential word objects, one for each word in the transcript.
          type: [array, "null"]
          items:
            x-label: Word
            $ref: "#/components/schemas/TranscriptWord"

        unredacted_words:
          x-label: Unredacted words
          description: |
            The original temporally-sequential word objects before PII redaction was applied. Same shape as `words`. Only returned when `redact_pii_return_unredacted` was set to `true` on the transcription request, otherwise this field is omitted and the `words` field remains fully redacted. See [PII redaction](https://www.assemblyai.com/docs/pii-redaction) for more information.
          type: [array, "null"]
          items:
            x-label: Word
            $ref: "#/components/schemas/TranscriptWord"

        acoustic_model:
          x-label: Acoustic model
          description: This parameter does not currently have any functionality attached to it.
          type: string
          deprecated: true

        custom_topics:
          x-label: Custom topics enabled
          description: This parameter does not currently have any functionality attached to it.
          type: [boolean, "null"]
          deprecated: true

        language_model:
          x-label: Language model
          description: This parameter does not currently have any functionality attached to it.
          type: string
          deprecated: true

        speech_model:
          x-label: Speech model
          description: |
            This parameter has been replaced with the `speech_models` parameter, learn more about the `speech_models` parameter [here](https://www.assemblyai.com/docs/pre-recorded-audio/select-the-speech-model).
          oneOf:
            - $ref: "#/components/schemas/SpeechModel"
            - type: "null"
          deprecated: True

        speed_boost:
          x-label: Speed boost
          description: This parameter does not currently have any functionality attached to it.
          type: [boolean, "null"]
          deprecated: true

        topics:
          x-label: Topics
          description: This parameter does not currently have any functionality attached to it.
          type: array
          items:
            x-label: Topic
            type: string
          deprecated: true

        translated_texts:
          # x-label: Translated text
          type: object
          description: Translated text keyed by language code. See [Translation](https://www.assemblyai.com/docs/speech-understanding/translation) for more details.
          properties:
            language_code:
              x-label: Language code
              type: string
              description: Translated text for this language code

      required:
        - id
        - speech_model
        - language_model
        - acoustic_model
        - status
        - audio_url
        - webhook_auth
        - auto_highlights
        - redact_pii
        - summarization
        - language_confidence_threshold
        - language_confidence
      example:
        {
          id: "9ea68fd3-f953-42c1-9742-976c447fb463",
          speech_model: null,
          language_model: "assemblyai_default",
          acoustic_model: "assemblyai_default",
          language_code: "en_us",
          language_detection: true,
          language_confidence_threshold: 0.7,
          language_confidence: 0.9959,
          status: "completed",
          audio_url: "https://assembly.ai/wildfires.mp3",
          text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor. Good morning. What is it about the conditions right now that have caused this round of wildfires to affect so many people so far away? Well, there's a couple of things. The season has been pretty dry already. And then the fact that we're getting hit in the US. Is because there's a couple of weather systems that are essentially channeling the smoke from those Canadian wildfires through Pennsylvania into the Mid Atlantic and the Northeast and kind of just dropping the smoke there. So what is it in this haze that makes it harmful? And I'm assuming it is harmful. It is. The levels outside right now in Baltimore are considered unhealthy. And most of that is due to what's called particulate matter, which are tiny particles, microscopic smaller than the width of your hair that can get into your lungs and impact your respiratory system, your cardiovascular system, and even your neurological your brain. What makes this particularly harmful? Is it the volume of particulant? Is it something in particular? What is it exactly? Can you just drill down on that a little bit more? Yeah. So the concentration of particulate matter I was looking at some of the monitors that we have was reaching levels of what are, in science, big 150 micrograms per meter cubed, which is more than ten times what the annual average should be and about four times higher than what you're supposed to have on a 24 hours average. And so the concentrations of these particles in the air are just much, much higher than we typically see. And exposure to those high levels can lead to a host of health problems. And who is most vulnerable? I noticed that in New York City, for example, they're canceling outdoor activities. And so here it is in the early days of summer, and they have to keep all the kids inside. So who tends to be vulnerable in a situation like this? It's the youngest. So children, obviously, whose bodies are still developing. The elderly, who are their bodies are more in decline and they're more susceptible to the health impacts of breathing, the poor air quality. And then people who have preexisting health conditions, people with respiratory conditions or heart conditions can be triggered by high levels of air pollution. Could this get worse? That's a good question. In some areas, it's much worse than others. And it just depends on kind of where the smoke is concentrated. I think New York has some of the higher concentrations right now, but that's going to change as that air moves away from the New York area. But over the course of the next few days, we will see different areas being hit at different times with the highest concentrations. I was going to ask you about more fires start burning. I don't expect the concentrations to go up too much higher. I was going to ask you how and you started to answer this, but how much longer could this last? Or forgive me if I'm asking you to speculate, but what do you think? Well, I think the fires are going to burn for a little bit longer, but the key for us in the US. Is the weather system changing. And so right now, it's kind of the weather systems that are pulling that air into our mid Atlantic and Northeast region. As those weather systems change and shift, we'll see that smoke going elsewhere and not impact us in this region as much. And so I think that's going to be the defining factor. And I think the next couple of days we're going to see a shift in that weather pattern and start to push the smoke away from where we are. And finally, with the impacts of climate change, we are seeing more wildfires. Will we be seeing more of these kinds of wide ranging air quality consequences or circumstances? I mean, that is one of the predictions for climate change. Looking into the future, the fire season is starting earlier and lasting longer, and we're seeing more frequent fires. So, yeah, this is probably something that we'll be seeing more frequently. This tends to be much more of an issue in the Western US. So the eastern US. Getting hit right now is a little bit new. But yeah, I think with climate change moving forward, this is something that is going to happen more frequently. That's Peter De Carlo, associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University. Sergeant Carlo, thanks so much for joining us and sharing this expertise with us. Thank you for having me.",
          words:
            [
              {
                text: "Smoke",
                start: 250,
                end: 650,
                confidence: 0.97465,
                speaker: null,
              },
              {
                text: "from",
                start: 730,
                end: 1022,
                confidence: 0.99999,
                speaker: null,
              },
              {
                text: "hundreds",
                start: 1076,
                end: 1418,
                confidence: 0.99844,
                speaker: null,
              },
              {
                text: "of",
                start: 1434,
                end: 1614,
                confidence: 0.84,
                speaker: null,
              },
              {
                text: "wildfires",
                start: 1652,
                end: 2346,
                confidence: 0.89572,
                speaker: null,
              },
              {
                text: "in",
                start: 2378,
                end: 2526,
                confidence: 0.99994,
                speaker: null,
              },
              {
                text: "Canada",
                start: 2548,
                end: 3130,
                confidence: 0.93953,
                speaker: null,
              },
              {
                text: "is",
                start: 3210,
                end: 3454,
                confidence: 0.999,
                speaker: null,
              },
              {
                text: "triggering",
                start: 3492,
                end: 3946,
                confidence: 0.74794,
                speaker: null,
              },
              {
                text: "air",
                start: 3978,
                end: 4174,
                confidence: 1.0,
                speaker: null,
              },
              {
                text: "quality",
                start: 4212,
                end: 4558,
                confidence: 0.88077,
                speaker: null,
              },
              {
                text: "alerts",
                start: 4644,
                end: 5114,
                confidence: 0.94814,
                speaker: null,
              },
              {
                text: "throughout",
                start: 5162,
                end: 5466,
                confidence: 0.99726,
                speaker: null,
              },
              {
                text: "the",
                start: 5498,
                end: 5694,
                confidence: 0.79,
                speaker: null,
              },
              {
                text: "US.",
                start: 5732,
                end: 6382,
                confidence: 0.89,
                speaker: null,
              },
            ],
          utterances:
            [
              {
                confidence: 0.9359033333333334,
                end: 26950,
                speaker: "A",
                start: 250,
                text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor.",
                words:
                  [
                    {
                      text: "Smoke",
                      start: 250,
                      end: 650,
                      confidence: 0.97503,
                      speaker: "A",
                    },
                    {
                      text: "from",
                      start: 730,
                      end: 1022,
                      confidence: 0.99999,
                      speaker: "A",
                    },
                    {
                      text: "hundreds",
                      start: 1076,
                      end: 1418,
                      confidence: 0.99843,
                      speaker: "A",
                    },
                    {
                      text: "of",
                      start: 1434,
                      end: 1614,
                      confidence: 0.85,
                      speaker: "A",
                    },
                    {
                      text: "wildfires",
                      start: 1652,
                      end: 2346,
                      confidence: 0.89657,
                      speaker: "A",
                    },
                    {
                      text: "in",
                      start: 2378,
                      end: 2526,
                      confidence: 0.99994,
                      speaker: "A",
                    },
                    {
                      text: "Canada",
                      start: 2548,
                      end: 3130,
                      confidence: 0.93864,
                      speaker: "A",
                    },
                    {
                      text: "is",
                      start: 3210,
                      end: 3454,
                      confidence: 0.999,
                      speaker: "A",
                    },
                    {
                      text: "triggering",
                      start: 3492,
                      end: 3946,
                      confidence: 0.75366,
                      speaker: "A",
                    },
                    {
                      text: "air",
                      start: 3978,
                      end: 4174,
                      confidence: 1.0,
                      speaker: "A",
                    },
                    {
                      text: "quality",
                      start: 4212,
                      end: 4558,
                      confidence: 0.87745,
                      speaker: "A",
                    },
                    {
                      text: "alerts",
                      start: 4644,
                      end: 5114,
                      confidence: 0.94739,
                      speaker: "A",
                    },
                    {
                      text: "throughout",
                      start: 5162,
                      end: 5466,
                      confidence: 0.99726,
                      speaker: "A",
                    },
                    {
                      text: "the",
                      start: 5498,
                      end: 5694,
                      confidence: 0.79,
                      speaker: "A",
                    },
                    {
                      text: "US.",
                      start: 5732,
                      end: 6382,
                      confidence: 0.88,
                      speaker: "A",
                    },
                  ],
              },
            ],
          confidence: 0.9404651451800253,
          audio_duration: 281,
          punctuate: true,
          format_text: true,
          multichannel: false,
          webhook_url: "https://your-webhook-url.tld/path",
          webhook_status_code: 200,
          webhook_auth: true,
          webhook_auth_header_name: "webhook-secret",
          auto_highlights_result:
            {
              status: "success",
              results:
                [
                  {
                    count: 1,
                    rank: 0.08,
                    text: "air quality alerts",
                    timestamps: [{ start: 3978, end: 5114 }],
                  },
                  {
                    count: 1,
                    rank: 0.08,
                    text: "wide ranging air quality consequences",
                    timestamps: [{ start: 235388, end: 238694 }],
                  },
                  {
                    count: 1,
                    rank: 0.07,
                    text: "more wildfires",
                    timestamps: [{ start: 230972, end: 232354 }],
                  },
                  {
                    count: 1,
                    rank: 0.07,
                    text: "air pollution",
                    timestamps: [{ start: 156004, end: 156910 }],
                  },
                  {
                    count: 3,
                    rank: 0.07,
                    text: "weather systems",
                    timestamps:
                      [
                        { start: 47344, end: 47958 },
                        { start: 205268, end: 205818 },
                        { start: 211588, end: 213434 },
                      ],
                  },
                  {
                    count: 2,
                    rank: 0.06,
                    text: "high levels",
                    timestamps:
                      [
                        { start: 121128, end: 121646 },
                        { start: 155412, end: 155866 },
                      ],
                  },
                  {
                    count: 1,
                    rank: 0.06,
                    text: "health conditions",
                    timestamps: [{ start: 152138, end: 152666 }],
                  },
                  {
                    count: 2,
                    rank: 0.06,
                    text: "Peter de Carlo",
                    timestamps:
                      [
                        { start: 18948, end: 19930 },
                        { start: 268298, end: 269194 },
                      ],
                  },
                  {
                    count: 1,
                    rank: 0.06,
                    text: "New York City",
                    timestamps: [{ start: 125768, end: 126274 }],
                  },
                  {
                    count: 1,
                    rank: 0.05,
                    text: "respiratory conditions",
                    timestamps: [{ start: 152964, end: 153786 }],
                  },
                  {
                    count: 3,
                    rank: 0.05,
                    text: "New York",
                    timestamps:
                      [
                        { start: 125768, end: 126034 },
                        { start: 171448, end: 171938 },
                        { start: 176008, end: 176322 },
                      ],
                  },
                  {
                    count: 3,
                    rank: 0.05,
                    text: "climate change",
                    timestamps:
                      [
                        { start: 229548, end: 230230 },
                        { start: 244576, end: 245162 },
                        { start: 263348, end: 263950 },
                      ],
                  },
                  {
                    count: 1,
                    rank: 0.05,
                    text: "Johns Hopkins University Varsity",
                    timestamps: [{ start: 23972, end: 25490 }],
                  },
                  {
                    count: 1,
                    rank: 0.05,
                    text: "heart conditions",
                    timestamps: [{ start: 153988, end: 154506 }],
                  },
                  {
                    count: 1,
                    rank: 0.05,
                    text: "air quality warnings",
                    timestamps: [{ start: 12308, end: 13434 }],
                  },
                ],
            },
          auto_highlights: true,
          audio_start_from: 10,
          audio_end_at: 280,
          filter_profanity: true,
          redact_pii: true,
          redact_pii_audio: true,
          redact_pii_audio_quality: "mp3",
          redact_pii_policies:
            ["us_social_security_number", "credit_card_number"],
          redact_pii_sub: "hash",
          speaker_labels: true,
          content_safety: true,
          iab_categories: true,
          content_safety_labels:
            {
              status: "success",
              results:
                [
                  {
                    text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor. Good morning.",
                    labels:
                      [
                        {
                          label: "disasters",
                          confidence: 0.8142836093902588,
                          severity: 0.4093044400215149,
                        },
                      ],
                    sentences_idx_start: 0,
                    sentences_idx_end: 5,
                    timestamp: { start: 250, end: 28840 },
                  },
                ],
              summary:
                {
                  disasters: 0.9940800441842205,
                  health_issues: 0.9216489289040967,
                },
              severity_score_summary:
                {
                  disasters:
                    {
                      low: 0.5733263024656846,
                      medium: 0.42667369753431533,
                      high: 0.0,
                    },
                  health_issues:
                    {
                      low: 0.22863814977924785,
                      medium: 0.45014154926938227,
                      high: 0.32122030095136983,
                    },
                },
            },
          iab_categories_result:
            {
              status: "success",
              results:
                [
                  {
                    text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor. Good morning.",
                    labels:
                      [
                        {
                          relevance: 0.988274097442627,
                          label: "Home&Garden>IndoorEnvironmentalQuality",
                        },
                        {
                          relevance: 0.5821335911750793,
                          label: "NewsAndPolitics>Weather",
                        },
                        {
                          relevance: 0.0042327106930315495,
                          label: "MedicalHealth>DiseasesAndConditions>LungAndRespiratoryHealth",
                        },
                        {
                          relevance: 0.0033971222583204508,
                          label: "NewsAndPolitics>Disasters",
                        },
                        {
                          relevance: 0.002469958271831274,
                          label: "BusinessAndFinance>Business>GreenSolutions",
                        },
                        {
                          relevance: 0.0014376690378412604,
                          label: "MedicalHealth>DiseasesAndConditions>Cancer",
                        },
                        {
                          relevance: 0.0014294233405962586,
                          label: "Science>Environment",
                        },
                        {
                          relevance: 0.001234519761055708,
                          label: "Travel>TravelLocations>PolarTravel",
                        },
                        {
                          relevance: 0.0010231725173071027,
                          label: "MedicalHealth>DiseasesAndConditions>ColdAndFlu",
                        },
                        {
                          relevance: 0.0007445293595083058,
                          label: "BusinessAndFinance>Industries>PowerAndEnergyIndustry",
                        },
                      ],
                    timestamp: { start: 250, end: 28840 },
                  },
                ],
              summary:
                {
                  "NewsAndPolitics>Weather": 1.0,
                  "Home&Garden>IndoorEnvironmentalQuality": 0.9043831825256348,
                  "Science>Environment": 0.16117265820503235,
                  "BusinessAndFinance>Industries>EnvironmentalServicesIndustry": 0.14393523335456848,
                  "MedicalHealth>DiseasesAndConditions>LungAndRespiratoryHealth": 0.11401086300611496,
                  "BusinessAndFinance>Business>GreenSolutions": 0.06348437070846558,
                  "NewsAndPolitics>Disasters": 0.05041387677192688,
                  "Travel>TravelLocations>PolarTravel": 0.01308488193899393,
                  HealthyLiving: 0.008222488686442375,
                  "MedicalHealth>DiseasesAndConditions>ColdAndFlu": 0.0022315620444715023,
                  "MedicalHealth>DiseasesAndConditions>HeartAndCardiovascularDiseases": 0.00213034451007843,
                  "HealthyLiving>Wellness>SmokingCessation": 0.001540527562610805,
                  "MedicalHealth>DiseasesAndConditions>Injuries": 0.0013950627762824297,
                  "BusinessAndFinance>Industries>PowerAndEnergyIndustry": 0.0012570273829624057,
                  "MedicalHealth>DiseasesAndConditions>Cancer": 0.001097781932912767,
                  "MedicalHealth>DiseasesAndConditions>Allergies": 0.0010148967849090695,
                  "MedicalHealth>DiseasesAndConditions>MentalHealth": 0.000717321818228811,
                  "Style&Fashion>PersonalCare>DeodorantAndAntiperspirant": 0.0006022014422342181,
                  "Technology&Computing>Computing>ComputerNetworking": 0.0005461975233629346,
                  "MedicalHealth>DiseasesAndConditions>Injuries>FirstAid": 0.0004885646631009877,
                },
            },
          custom_spelling: null,
          throttled: null,
          auto_chapters: false,
          summarization: false,
          summary_type: null,
          summary_model: null,
          custom_topics: true,
          topics: [],
          speech_threshold: 0.5,
          remove_audio_tags: null,
          domain: null,
          disfluencies: false,
          sentiment_analysis: true,
          sentiment_analysis_results: null,
          entity_detection: true,
          entities:
            [
              {
                entity_type: "location",
                text: "Canada",
                start: 2548,
                end: 3130,
              },
              {
                entity_type: "location",
                text: "the US",
                start: 5498,
                end: 6382,
              },
              {
                entity_type: "location",
                text: "Maine",
                start: 7492,
                end: 7914,
              },
              {
                entity_type: "location",
                text: "Maryland",
                start: 8212,
                end: 8634,
              },
              {
                entity_type: "location",
                text: "Minnesota",
                start: 8932,
                end: 9578,
              },
              {
                entity_type: "person_name",
                text: "Peter de Carlo",
                start: 18948,
                end: 19930,
              },
              {
                entity_type: "occupation",
                text: "associate professor",
                start: 20292,
                end: 21194,
              },
              {
                entity_type: "organization",
                text: "Department of Environmental Health and Engineering",
                start: 21508,
                end: 23706,
              },
              {
                entity_type: "organization",
                text: "Johns Hopkins University Varsity",
                start: 23972,
                end: 25490,
              },
              {
                entity_type: "occupation",
                text: "professor",
                start: 26076,
                end: 26950,
              },
              {
                entity_type: "location",
                text: "the US",
                start: 45184,
                end: 45898,
              },
              {
                entity_type: "nationality",
                text: "Canadian",
                start: 49728,
                end: 50086,
              },
            ],
          chapters: null,
          summary: null,
          speakers_expected: 2,
        }

    TopicDetectionModelResult:
      x-label: Topic Detection result
      description: |
        The result of the Topic Detection model, if it is enabled.
        See [Topic Detection](https://www.assemblyai.com/docs/speech-understanding/topic-detection) for more information.
      x-fern-sdk-group-name: transcripts
      type: object
      required:
        - status
        - results
        - summary
      properties:
        status:
          x-label: Status
          description: The status of the Topic Detection model. Either success, or unavailable in the rare case that the model failed.
          $ref: "#/components/schemas/AudioIntelligenceModelStatus"
        results:
          x-label: Results
          description: An array of results for the Topic Detection model
          type: array
          items:
            x-label: Topic Detection result
            $ref: "#/components/schemas/TopicDetectionResult"
        summary:
          x-label: Summary
          description: The overall relevance of topic to the entire audio file
          type: object
          additionalProperties:
            type: number
            format: double
            minimum: 0
            maximum: 1
      example:
        {
          status: "success",
          results:
            [
              {
                text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor. Good morning.",
                labels:
                  [
                    {
                      relevance: 0.988274097442627,
                      label: "Home&Garden>IndoorEnvironmentalQuality",
                    },
                    {
                      relevance: 0.5821335911750793,
                      label: "NewsAndPolitics>Weather",
                    },
                    {
                      relevance: 0.0042327106930315495,
                      label: "MedicalHealth>DiseasesAndConditions>LungAndRespiratoryHealth",
                    },
                    {
                      relevance: 0.0033971222583204508,
                      label: "NewsAndPolitics>Disasters",
                    },
                    {
                      relevance: 0.002469958271831274,
                      label: "BusinessAndFinance>Business>GreenSolutions",
                    },
                    {
                      relevance: 0.0014376690378412604,
                      label: "MedicalHealth>DiseasesAndConditions>Cancer",
                    },
                    {
                      relevance: 0.0014294233405962586,
                      label: "Science>Environment",
                    },
                    {
                      relevance: 0.001234519761055708,
                      label: "Travel>TravelLocations>PolarTravel",
                    },
                    {
                      relevance: 0.0010231725173071027,
                      label: "MedicalHealth>DiseasesAndConditions>ColdAndFlu",
                    },
                    {
                      relevance: 0.0007445293595083058,
                      label: "BusinessAndFinance>Industries>PowerAndEnergyIndustry",
                    },
                  ],
                timestamp: { start: 250, end: 28840 },
              },
            ],
          summary:
            {
              "NewsAndPolitics>Weather": 1.0,
              "Home&Garden>IndoorEnvironmentalQuality": 0.9043831825256348,
              "Science>Environment": 0.16117265820503235,
              "BusinessAndFinance>Industries>EnvironmentalServicesIndustry": 0.14393523335456848,
              "MedicalHealth>DiseasesAndConditions>LungAndRespiratoryHealth": 0.11401086300611496,
              "BusinessAndFinance>Business>GreenSolutions": 0.06348437070846558,
              "NewsAndPolitics>Disasters": 0.05041387677192688,
              "Travel>TravelLocations>PolarTravel": 0.01308488193899393,
              HealthyLiving: 0.008222488686442375,
              "MedicalHealth>DiseasesAndConditions>ColdAndFlu": 0.0022315620444715023,
              "MedicalHealth>DiseasesAndConditions>HeartAndCardiovascularDiseases": 0.00213034451007843,
              "HealthyLiving>Wellness>SmokingCessation": 0.001540527562610805,
              "MedicalHealth>DiseasesAndConditions>Injuries": 0.0013950627762824297,
              "BusinessAndFinance>Industries>PowerAndEnergyIndustry": 0.0012570273829624057,
              "MedicalHealth>DiseasesAndConditions>Cancer": 0.001097781932912767,
              "MedicalHealth>DiseasesAndConditions>Allergies": 0.0010148967849090695,
              "MedicalHealth>DiseasesAndConditions>MentalHealth": 0.000717321818228811,
              "Style&Fashion>PersonalCare>DeodorantAndAntiperspirant": 0.0006022014422342181,
              "Technology&Computing>Computing>ComputerNetworking": 0.0005461975233629346,
              "MedicalHealth>DiseasesAndConditions>Injuries>FirstAid": 0.0004885646631009877,
            },
        }

    ContentSafetyLabelsResult:
      x-label: Content Moderation labels result
      description: |
        An array of results for the Content Moderation model, if it is enabled.
        See [Content moderation](https://www.assemblyai.com/docs/content-moderation) for more information.
      x-fern-sdk-group-name: transcripts
      type: object
      required:
        - status
        - results
        - summary
        - severity_score_summary
      properties:
        status:
          x-label: Status
          description: The status of the Content Moderation model. Either success, or unavailable in the rare case that the model failed.
          $ref: "#/components/schemas/AudioIntelligenceModelStatus"
        results:
          x-label: Results
          description: An array of results for the Content Moderation model
          type: array
          items:
            x-label: Content Moderation label result
            $ref: "#/components/schemas/ContentSafetyLabelResult"
        summary:
          x-label: Summary
          description: A summary of the Content Moderation confidence results for the entire audio file
          type: object
          additionalProperties:
            description: A confidence score for the presence of the sensitive topic "topic" across the entire audio file
            type: number
            format: double
            minimum: 0
            maximum: 1
        severity_score_summary:
          x-label: Severity score summary
          description: A summary of the Content Moderation severity results for the entire audio file
          type: object
          additionalProperties:
            $ref: "#/components/schemas/SeverityScoreSummary"
      example:
        {
          status: "success",
          results:
            [
              {
                text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor. Good morning.",
                labels:
                  [
                    {
                      label: "disasters",
                      confidence: 0.8142836093902588,
                      severity: 0.4093044400215149,
                    },
                  ],
                sentences_idx_start: 0,
                sentences_idx_end: 5,
                timestamp: { start: 250, end: 28840 },
              },
            ],
          summary:
            {
              disasters: 0.9940800441842205,
              health_issues: 0.9216489289040967,
            },
          severity_score_summary:
            {
              disasters:
                {
                  low: 0.5733263024656846,
                  medium: 0.42667369753431533,
                  high: 0.0,
                },
              health_issues:
                {
                  low: 0.22863814977924785,
                  medium: 0.45014154926938227,
                  high: 0.32122030095136983,
                },
            },
        }

    Chapter:
      x-label: Chapter
      description: Chapter of the audio file
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      required:
        - gist
        - headline
        - summary
        - start
        - end
      properties:
        gist:
          x-label: Gist
          description: An ultra-short summary (just a few words) of the content spoken in the chapter
          type: string
        headline:
          x-label: Headline
          description: A single sentence summary of the content spoken during the chapter
          type: string
        summary:
          x-label: Summary
          description: A one paragraph summary of the content spoken during the chapter
          type: string
        start:
          x-label: Start
          description: The starting time, in milliseconds, for the chapter
          type: integer
        end:
          x-label: End
          description: The starting time, in milliseconds, for the chapter
          type: integer
      example:
        {
          summary: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. In some places, the air quality warnings include the warning to stay inside.",
          gist: "Smoggy air quality alerts across US",
          headline: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts across US",
          start: 250,
          end: 28840,
        }

    Entity:
      x-label: Entity
      description: A detected entity
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      required:
        - entity_type
        - text
        - start
        - end
      properties:
        entity_type:
          x-label: Entity type
          description: The type of entity for the detected entity
          $ref: "#/components/schemas/EntityType"
        text:
          x-label: Text
          description: The text for the detected entity
          type: string
        start:
          x-label: Start
          description: The starting time, in milliseconds, at which the detected entity appears in the audio file
          type: integer
        end:
          x-label: End
          description: The ending time, in milliseconds, for the detected entity in the audio file
          type: integer
      example:
        { entity_type: "location", text: "Canada", start: 2548, end: 3130 }

    EntityType:
      x-label: Entity type
      description: The type of entity for the detected entity. Entity Detection returns every location reference under the single `location` value; granular location subtypes (`location_address`, `location_city`, etc.) are supported only by PII redaction.
      x-fern-sdk-group-name: transcripts
      type: string
      enum:
        - account_number
        - banking_information
        - blood_type
        - credit_card_cvv
        - credit_card_expiration
        - credit_card_number
        - date
        - date_interval
        - date_of_birth
        - drivers_license
        - drug
        - duration
        - email_address
        - event
        - filename
        - gender
        - gender_sexuality
        - healthcare_number
        - injury
        - ip_address
        - language
        - location
        - marital_status
        - medical_condition
        - medical_process
        - money_amount
        - nationality
        - number_sequence
        - occupation
        - organization
        - organization_medical_facility
        - passport_number
        - password
        - person_age
        - person_name
        - phone_number
        - physical_attribute
        - political_affiliation
        - religion
        - sexuality
        - statistics
        - time
        - url
        - us_social_security_number
        - username
        - vehicle_id
        - zodiac_sign
      x-fern-enum:
        account_number:
          description:
            "Customer account or membership identification number (e.g., Policy
            No. 10042992, Member ID: HZ-5235-001)"
        banking_information:
          description: Banking information, including account and routing numbers
        blood_type:
          description: Blood type (e.g., O-, AB positive)
        credit_card_cvv:
          description: "Credit card verification code (e.g., CVV: 080)"
        credit_card_expiration:
          description: Expiration date of a credit card
        credit_card_number:
          description: Credit card number
        date:
          description: Specific calendar date (e.g., December 18)
        date_interval:
          description:
            Broader time periods, including date ranges, months, seasons, years,
            and decades (e.g., 2020-2021, 5-9 May, January 1984)
        date_of_birth:
          description: "Date of birth (e.g., Date of Birth: March 7,1961)"
        drivers_license:
          description: Driver's license number. (e.g., DL# 356933-540)
        drug:
          description:
            Medications, vitamins, or supplements (e.g., Advil, Acetaminophen,
            Panadol)
        duration:
          description:
            Measurements of time expressed as a numerical value plus a unit (e.g.,
            8 months, 2 years)
        email_address:
          description: Email address (e.g., support@assemblyai.com)
        event:
          description: Name of an event or holiday (e.g., Olympics, Yom Kippur)
        filename:
          description:
            Names of computer files, including the extension or filepath (e.g.,
            Taxes/2012/brad-tax-returns.pdf)
        gender:
          description: Terms indicating gender identity (e.g., female, male, non-binary)
        gender_sexuality:
          description:
            Terms indicating gender identity or sexual orientation, including
            slang terms (e.g., female, bisexual, trans)
        healthcare_number:
          description:
            "Healthcare numbers and health plan beneficiary numbers (e.g.,
            Policy No.: 5584-486-674-YM)"
        injury:
          description: Bodily injury (e.g., I broke my arm, I have a sprained wrist)
        ip_address:
          description: Internet IP address, including IPv4 and IPv6 formats (e.g., 192.168.0.1)
        language:
          description: Name of a natural language (e.g., Spanish, French)
        location:
          description:
            Any Location reference including mailing address, postal code,
            city, state, province, country, or coordinates. (e.g., Lake Victoria, 145
            Windsor St., 90210)
        marital_status:
          description:
            Terms indicating marital status (e.g., Single, common-law, ex-wife,
            married)
        medical_condition:
          description:
            Name of a medical condition, disease, syndrome, deficit, or disorder
            (e.g., chronic fatigue syndrome, arrhythmia, depression)
        medical_process:
          description:
            Medical process, including treatments, procedures, and tests (e.g.,
            heart surgery, CT scan)
        money_amount:
          description: Name and/or amount of currency (e.g., 15 pesos, $94.50)
        nationality:
          description:
            Terms indicating nationality, ethnicity, or race (e.g., American,
            Asian, Caucasian)
        number_sequence:
          description:
            Numerical PII (including alphanumeric strings) that doesn't fall
            under other categories
        occupation:
          description: Job title or profession (e.g., professor, actors, engineer, CPA)
        organization:
          description:
            Name of an organization (e.g., CNN, McDonalds, University of Alaska,
            Northwest General Hospital)
        organization_medical_facility:
          description:
            Name of a medical facility (e.g., Mayo Clinic, Northwest General
            Hospital)
        passport_number:
          description: Passport numbers, issued by any country (e.g., PA4568332, NU3C6L86S12)
        password:
          description:
            Account passwords, PINs, access keys, or verification answers (e.g.,
            27%alfalfa, temp1234, My mother's maiden name is Smith)
        person_age:
          description: Number associated with an age (e.g., 27, 75)
        person_name:
          description: Name of a person (e.g., Bob, Doug Jones, Dr. Kay Martinez, MD)
        phone_number:
          description: Telephone or fax number
        physical_attribute:
          description: Distinctive bodily attributes, including race
            (e.g., I'm 190cm tall, He belongs to the Black students' association)
        political_affiliation:
          description:
            Terms referring to a political party, movement, or ideology (e.g.,
            Republican, Liberal)
        religion:
          description: Terms indicating religious affiliation (e.g., Hindu, Catholic)
        sexuality:
          description: Terms indicating sexual orientation (e.g., heterosexual, bisexual, gay)
        statistics:
          description: Medical statistics (e.g., 18%, 18 percent)
        time:
          description: Expressions indicating clock times (e.g., 19:37:28, 10pm EST)
        url:
          description: Internet addresses (e.g., https://www.assemblyai.com/)
        us_social_security_number:
          description: Social Security Number or equivalent
        username:
          description: Usernames, login names, or handles (e.g., @AssemblyAI)
        vehicle_id:
          description:
            Vehicle identification numbers (VINs), vehicle serial numbers,
            and license plate numbers (e.g., 5FNRL38918B111818, BIF7547)
        zodiac_sign:
          description: Names of Zodiac signs (e.g., Aries, Taurus)
      x-aai-enum:
        account_number:
          label: Account number
        banking_information:
          label: Banking information
        blood_type:
          label: Blood type
        credit_card_cvv:
          label: Credit card CVV
        credit_card_expiration:
          label: Credit card expiration
        credit_card_number:
          label: Credit card number
        date:
          label: Date
        date_interval:
          label: Date interval
        date_of_birth:
          label: Date of birth
        drivers_license:
          label: Driver's license
        drug:
          label: Drug
        duration:
          label: Duration
        email_address:
          label: Email address
        event:
          label: Event
        filename:
          label: Filename
        gender:
          label: Gender
        gender_sexuality:
          label: Gender sexuality
        healthcare_number:
          label: Healthcare number
        injury:
          label: Injury
        ip_address:
          label: IP address
        language:
          label: Language
        location:
          label: Location
        marital_status:
          label: Marital status
        medical_condition:
          label: Medical condition
        medical_process:
          label: Medical process
        money_amount:
          label: Money amount
        nationality:
          label: Nationality
        number_sequence:
          label: Number sequence
        occupation:
          label: Occupation
        organization:
          label: Organization
        organization_medical_facility:
          label: Organization medical facility
        passport_number:
          label: Passport number
        password:
          label: Password
        person_age:
          label: Person age
        person_name:
          label: Person name
        phone_number:
          label: Phone number
        physical_attribute:
          label: Physical attribute
        political_affiliation:
          label: Political affiliation
        religion:
          label: Religion
        sexuality:
          label: Sexuality
        statistics:
          label: Statistics
        time:
          label: Time
        url:
          label: URL
        us_social_security_number:
          label: US Social Security Number
        username:
          label: Username
        vehicle_id:
          label: Vehicle ID
        zodiac_sign:
          label: Zodiac sign

    SentimentAnalysisResult:
      x-label: Sentiment Analysis result
      description: The result of the Sentiment Analysis model
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      required:
        - text
        - start
        - end
        - sentiment
        - confidence
        - speaker
      properties:
        text:
          x-label: Text
          description: The transcript of the sentence
          type: string
        start:
          x-label: Start
          description: The starting time, in milliseconds, of the sentence
          type: integer
        end:
          x-label: End
          description: The ending time, in milliseconds, of the sentence
          type: integer
        sentiment:
          x-label: Sentiment
          description: The detected sentiment for the sentence, one of POSITIVE, NEUTRAL, NEGATIVE
          $ref: "#/components/schemas/Sentiment"
        confidence:
          x-label: Confidence
          description: The confidence score for the detected sentiment of the sentence, from 0 to 1
          type: number
          format: double
          minimum: 0
          maximum: 1
        channel:
          x-label: Channel
          description: The channel of this utterance. The left and right channels are channels 1 and 2. Additional channels increment the channel number sequentially.
          type: [string, "null"]
        speaker:
          x-label: Speaker
          description: The speaker of the sentence if [Speaker Diarization](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers) is enabled, else null
          type: [string, "null"]
      example:
        {
          text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US.",
          start: 250,
          end: 6350,
          sentiment: "NEGATIVE",
          confidence: 0.8181032538414001,
          speaker: null,
        }

    Sentiment:
      x-label: Sentiment
      x-fern-sdk-group-name: transcripts
      enum:
        - POSITIVE
        - NEUTRAL
        - NEGATIVE
      x-aai-enum:
        POSITIVE:
          label: Positive
        NEUTRAL:
          label: Neutral
        NEGATIVE:
          label: Negative

    TopicDetectionResult:
      x-label: Topic detection result
      description: The result of the topic detection model
      x-fern-sdk-group-name: transcripts
      type: object
      additionalProperties: false
      required:
        - text
      properties:
        text:
          x-label: Text
          description: The text in the transcript in which a detected topic occurs
          type: string
        labels:
          x-label: Labels
          description: An array of detected topics in the text
          type: array
          items:
            x-label: Label
            type: object
            additionalProperties: false
            required:
              - relevance
              - label
            properties:
              relevance:
                x-label: Relevance
                description: How relevant the detected topic is of a detected topic
                type: number
                format: double
                minimum: 0
                maximum: 1
              label:
                x-label: Label
                description: The IAB taxonomical label for the label of the detected topic, where > denotes supertopic/subtopic relationship
                type: string
        timestamp:
          x-label: Timestamp
          $ref: "#/components/schemas/Timestamp"
      example:
        {
          text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor. Good morning.",
          labels:
            [
              {
                relevance: 0.988274097442627,
                label: "Home&Garden>IndoorEnvironmentalQuality",
              },
              {
                relevance: 0.5821335911750793,
                label: "NewsAndPolitics>Weather",
              },
              {
                relevance: 0.0042327106930315495,
                label: "MedicalHealth>DiseasesAndConditions>LungAndRespiratoryHealth",
              },
              {
                relevance: 0.0033971222583204508,
                label: "NewsAndPolitics>Disasters",
              },
              {
                relevance: 0.002469958271831274,
                label: "BusinessAndFinance>Business>GreenSolutions",
              },
              {
                relevance: 0.0014376690378412604,
                label: "MedicalHealth>DiseasesAndConditions>Cancer",
              },
              {
                relevance: 0.0014294233405962586,
                label: "Science>Environment",
              },
              {
                relevance: 0.001234519761055708,
                label: "Travel>TravelLocations>PolarTravel",
              },
              {
                relevance: 0.0010231725173071027,
                label: "MedicalHealth>DiseasesAndConditions>ColdAndFlu",
              },
              {
                relevance: 0.0007445293595083058,
                label: "BusinessAndFinance>Industries>PowerAndEnergyIndustry",
              },
            ],
          timestamp: { start: 250, end: 28840 },
        }

    ContentSafetyLabel:
      x-label: Content Moderation label
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - label
        - confidence
        - severity
      properties:
        label:
          x-label: Label
          description: The label of the sensitive topic
          type: string
        confidence:
          x-label: Confidence
          description: The confidence score for the topic being discussed, from 0 to 1
          type: number
          format: double
          minimum: 0
          maximum: 1
        severity:
          x-label: Severity
          description: How severely the topic is discussed in the section, from 0 to 1
          type: number
          format: double
          minimum: 0
          maximum: 1
      example:
        {
          label: "disasters",
          confidence: 0.8142836093902588,
          severity: 0.4093044400215149,
        }

    ContentSafetyLabelResult:
      x-label: Content Moderation label result
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - text
        - labels
        - sentences_idx_start
        - sentences_idx_end
        - timestamp
      properties:
        text:
          x-label: Text
          description: The transcript of the section flagged by the Content Moderation model
          type: string
        labels:
          x-label: Labels
          description: An array of safety labels, one per sensitive topic that was detected in the section
          type: array
          items:
            x-label: Label
            $ref: "#/components/schemas/ContentSafetyLabel"
        sentences_idx_start:
          x-label: Sentence index start
          description: The sentence index at which the section begins
          type: integer
        sentences_idx_end:
          x-label: Sentence index end
          description: The sentence index at which the section ends
          type: integer
        timestamp:
          x-label: Timestamp
          description: Timestamp information for the section
          $ref: "#/components/schemas/Timestamp"
      example:
        {
          text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter de Carlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University Varsity. Good morning, professor. Good morning.",
          labels:
            [
              {
                label: "disasters",
                confidence: 0.8142836093902588,
                severity: 0.4093044400215149,
              },
            ],
          sentences_idx_start: 0,
          sentences_idx_end: 5,
          timestamp: { start: 250, end: 28840 },
        }

    SeverityScoreSummary:
      x-label: Severity score summary
      type: object
      x-fern-sdk-group-name: transcripts
      required:
        - low
        - medium
        - high
      properties:
        low:
          x-label: Low
          type: number
          format: double
          minimum: 0
          maximum: 1
        medium:
          x-label: Medium
          type: number
          format: double
          minimum: 0
          maximum: 1
        high:
          x-label: High
          type: number
          format: double
          minimum: 0
          maximum: 1
      example:
        { low: 0.5733263024656846, medium: 0.42667369753431533, high: 0.0 }

    AutoHighlightsResult:
      x-label: Auto highlights result
      description: |
        An array of results for the Key Phrases model, if it is enabled.
        See [Key phrases](https://www.assemblyai.com/docs/speech-understanding/key-phrases) for more information.
      x-fern-sdk-group-name: transcripts
      type: object
      required:
        - status
        - results
      properties:
        status:
          x-label: Status
          description: The status of the Key Phrases model. Either success, or unavailable in the rare case that the model failed.
          $ref: "#/components/schemas/AudioIntelligenceModelStatus"
        results:
          x-label: Results
          description: A temporally-sequential array of Key Phrases
          type: array
          items:
            x-label: Auto highlight result
            $ref: "#/components/schemas/AutoHighlightResult"
      example:
        {
          status: "success",
          results:
            [
              {
                count: 1,
                rank: 0.08,
                text: "air quality alerts",
                timestamps: [{ start: 3978, end: 5114 }],
              },
              {
                count: 1,
                rank: 0.08,
                text: "wide ranging air quality consequences",
                timestamps: [{ start: 235388, end: 238694 }],
              },
              {
                count: 1,
                rank: 0.07,
                text: "more wildfires",
                timestamps: [{ start: 230972, end: 232354 }],
              },
              {
                count: 1,
                rank: 0.07,
                text: "air pollution",
                timestamps: [{ start: 156004, end: 156910 }],
              },
              {
                count: 3,
                rank: 0.07,
                text: "weather systems",
                timestamps:
                  [
                    { start: 47344, end: 47958 },
                    { start: 205268, end: 205818 },
                    { start: 211588, end: 213434 },
                  ],
              },
              {
                count: 2,
                rank: 0.06,
                text: "high levels",
                timestamps:
                  [
                    { start: 121128, end: 121646 },
                    { start: 155412, end: 155866 },
                  ],
              },
              {
                count: 1,
                rank: 0.06,
                text: "health conditions",
                timestamps: [{ start: 152138, end: 152666 }],
              },
              {
                count: 2,
                rank: 0.06,
                text: "Peter de Carlo",
                timestamps:
                  [
                    { start: 18948, end: 19930 },
                    { start: 268298, end: 269194 },
                  ],
              },
              {
                count: 1,
                rank: 0.06,
                text: "New York City",
                timestamps: [{ start: 125768, end: 126274 }],
              },
              {
                count: 1,
                rank: 0.05,
                text: "respiratory conditions",
                timestamps: [{ start: 152964, end: 153786 }],
              },
              {
                count: 3,
                rank: 0.05,
                text: "New York",
                timestamps:
                  [
                    { start: 125768, end: 126034 },
                    { start: 171448, end: 171938 },
                    { start: 176008, end: 176322 },
                  ],
              },
              {
                count: 3,
                rank: 0.05,
                text: "climate change",
                timestamps:
                  [
                    { start: 229548, end: 230230 },
                    { start: 244576, end: 245162 },
                    { start: 263348, end: 263950 },
                  ],
              },
              {
                count: 1,
                rank: 0.05,
                text: "Johns Hopkins University Varsity",
                timestamps: [{ start: 23972, end: 25490 }],
              },
              {
                count: 1,
                rank: 0.05,
                text: "heart conditions",
                timestamps: [{ start: 153988, end: 154506 }],
              },
              {
                count: 1,
                rank: 0.05,
                text: "air quality warnings",
                timestamps: [{ start: 12308, end: 13434 }],
              },
            ],
        }

    AutoHighlightResult:
      x-label: Auto highlight result
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - count
        - rank
        - text
        - timestamps
      properties:
        count:
          x-label: Count
          description: The total number of times the key phrase appears in the audio file
          type: integer
        rank:
          x-label: Rank
          description: The total relevancy to the overall audio file of this key phrase - a greater number means more relevant
          type: number
          format: float
          minimum: 0
          maximum: 1
        text:
          x-label: Text
          description: The text itself of the key phrase
          type: string
        timestamps:
          x-label: Timestamps
          description: The timestamp of the of the key phrase
          type: array
          items:
            x-label: Timestamp
            $ref: "#/components/schemas/Timestamp"
      example:
        {
          count: 1,
          rank: 0.08,
          text: "air quality alerts",
          timestamps: [{ start: 3978, end: 5114 }],
        }

    TranscriptWord:
      x-label: Word
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - confidence
        - start
        - end
        - text
        - speaker
      properties:
        confidence:
          x-label: Confidence
          description: The confidence score for the transcript of this word
          type: number
          format: double
          minimum: 0
          maximum: 1
        start:
          x-label: Start
          description: The starting time, in milliseconds, for the word
          type: integer
        end:
          x-label: End
          description: The ending time, in milliseconds, for the word
          type: integer
        text:
          x-label: Text
          description: The text of the word
          type: string
        channel:
          x-label: Channel
          description: The channel of the word. The left and right channels are channels 1 and 2. Additional channels increment the channel number sequentially.
          type: [string, "null"]
        speaker:
          x-label: Speaker
          description: The speaker of the word if [Speaker Diarization](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers) is enabled, else null
          type: [string, "null"]
      example:
        {
          text: "Smoke",
          start: 250,
          end: 650,
          confidence: 0.97465,
          channel: null,
          speaker: null,
        }

    TranscriptSentence:
      x-label: Sentence
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - text
        - start
        - end
        - confidence
        - words
        - speaker
      properties:
        text:
          x-label: Text
          description: The transcript of the sentence
          type: string
        start:
          x-label: Start
          description: The starting time, in milliseconds, for the sentence
          type: integer
        end:
          x-label: End
          description: The ending time, in milliseconds, for the sentence
          type: integer
        confidence:
          x-label: Confidence
          description: The confidence score for the transcript of this sentence
          type: number
          format: double
          minimum: 0
          maximum: 1
        words:
          x-label: Words
          description: An array of words in the sentence
          type: array
          items:
            x-label: Word
            $ref: "#/components/schemas/TranscriptWord"
        channel:
          x-label: Channel
          description: The channel of the sentence. The left and right channels are channels 1 and 2. Additional channels increment the channel number sequentially.
          type: [string, "null"]
        speaker:
          x-label: Speaker
          description: The speaker of the sentence if [Speaker Diarization](https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers) is enabled, else null
          type: [string, "null"]
      example:
        {
          text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US.",
          start: 250,
          end: 6350,
          confidence: 0.72412,
          words:
            [
              {
                text: "Smoke",
                start: 250,
                end: 650,
                confidence: 0.72412,
                speaker: null,
              },
              {
                text: "from",
                start: 730,
                end: 1022,
                confidence: 0.99996,
                speaker: null,
              },
              {
                text: "hundreds",
                start: 1076,
                end: 1466,
                confidence: 0.99992,
                speaker: null,
              },
              {
                text: "of",
                start: 1498,
                end: 1646,
                confidence: 1,
                speaker: null,
              },
            ],
          speaker: null,
        }

    SentencesResponse:
      x-label: Sentences response
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - id
        - confidence
        - audio_duration
        - sentences
      properties:
        id:
          x-label: Transcript ID
          description: The unique identifier for the transcript
          type: string
          format: uuid
        confidence:
          x-label: Confidence
          description: The confidence score for the transcript
          type: number
          format: double
          minimum: 0
          maximum: 1
        audio_duration:
          x-label: Audio duration
          description: The duration of the audio file in seconds
          type: number
        sentences:
          x-label: Sentences
          description: An array of sentences in the transcript
          type: array
          items:
            x-label: Sentence
            $ref: "#/components/schemas/TranscriptSentence"
      example:
        {
          sentences:
            [
              {
                text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US.",
                start: 250,
                end: 6350,
                confidence: 0.72412,
                words:
                  [
                    {
                      text: "Smoke",
                      start: 250,
                      end: 650,
                      confidence: 0.72412,
                      speaker: null,
                    },
                    {
                      text: "from",
                      start: 730,
                      end: 1022,
                      confidence: 0.99996,
                      speaker: null,
                    },
                    {
                      text: "hundreds",
                      start: 1076,
                      end: 1466,
                      confidence: 0.99992,
                      speaker: null,
                    },
                    {
                      text: "of",
                      start: 1498,
                      end: 1646,
                      confidence: 1,
                      speaker: null,
                    },
                  ],
                speaker: null,
              },
              {
                text: "Skylines from Maine to Maryland to Minnesota are gray and smoggy.",
                start: 6500,
                end: 11050,
                confidence: 0.99819,
                words:
                  [
                    {
                      text: "Skylines",
                      start: 6500,
                      end: 7306,
                      confidence: 0.99819,
                      speaker: null,
                    },
                    {
                      text: "from",
                      start: 7338,
                      end: 7534,
                      confidence: 0.99987,
                      speaker: null,
                    },
                    {
                      text: "Maine",
                      start: 7572,
                      end: 7962,
                      confidence: 0.9972,
                      speaker: null,
                    },
                    {
                      text: "to",
                      start: 8026,
                      end: 8206,
                      confidence: 1,
                      speaker: null,
                    },
                    {
                      text: "Maryland",
                      start: 8228,
                      end: 8650,
                      confidence: 0.5192,
                      speaker: null,
                    },
                    {
                      text: "to",
                      start: 8730,
                      end: 8926,
                      confidence: 1,
                      speaker: null,
                    },
                  ],
                speaker: null,
              },
            ],
          id: "d5a3d302-066e-43fb-b63b-8f57baf185db",
          confidence: 0.9579390654205628,
          audio_duration: 281,
        }

    TranscriptParagraph:
      x-label: Paragraph
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - text
        - start
        - end
        - confidence
        - words
      properties:
        text:
          x-label: Text
          description: The transcript of the paragraph
          type: string
        start:
          x-label: Start
          description: The starting time, in milliseconds, of the paragraph
          type: integer
        end:
          x-label: End
          description: The ending time, in milliseconds, of the paragraph
          type: integer
        confidence:
          x-label: Confidence
          description: The confidence score for the transcript of this paragraph
          type: number
          format: double
          minimum: 0
          maximum: 1
        words:
          x-label: Words
          description: An array of words in the paragraph
          type: array
          items:
            x-label: Word
            $ref: "#/components/schemas/TranscriptWord"
      example:
        {
          text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter Decarlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University. Good morning, professor.",
          start: 250,
          end: 26950,
          confidence: 0.73033,
          words:
            [
              {
                text: "Smoke",
                start: 250,
                end: 650,
                confidence: 0.73033,
                speaker: null,
              },
              {
                text: "from",
                start: 730,
                end: 1022,
                confidence: 1,
                speaker: null,
              },
              {
                text: "hundreds",
                start: 1076,
                end: 1466,
                confidence: 0.99992,
                speaker: null,
              },
              {
                text: "of",
                start: 1498,
                end: 1646,
                confidence: 1,
                speaker: null,
              },
            ],
        }

    ParagraphsResponse:
      x-label: Paragraphs response
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - id
        - confidence
        - audio_duration
        - paragraphs
      properties:
        id:
          x-label: Transcript ID
          description: The unique identifier of your transcript
          type: string
          format: uuid
        confidence:
          x-label: Confidence
          description: The confidence score for the transcript
          type: number
          format: double
          minimum: 0
          maximum: 1
        audio_duration:
          x-label: Audio duration
          description: The duration of the audio file in seconds
          type: number
        paragraphs:
          x-label: Paragraphs
          description: An array of paragraphs in the transcript
          type: array
          items:
            x-label: Paragraph
            $ref: "#/components/schemas/TranscriptParagraph"
      example:
        {
          paragraphs:
            [
              {
                text: "Smoke from hundreds of wildfires in Canada is triggering air quality alerts throughout the US. Skylines from Maine to Maryland to Minnesota are gray and smoggy. And in some places, the air quality warnings include the warning to stay inside. We wanted to better understand what's happening here and why, so we called Peter Decarlo, an associate professor in the Department of Environmental Health and Engineering at Johns Hopkins University. Good morning, professor.",
                start: 250,
                end: 26950,
                confidence: 0.73033,
                words:
                  [
                    {
                      text: "Smoke",
                      start: 250,
                      end: 650,
                      confidence: 0.73033,
                      speaker: null,
                    },
                    {
                      text: "from",
                      start: 730,
                      end: 1022,
                      confidence: 1,
                      speaker: null,
                    },
                    {
                      text: "hundreds",
                      start: 1076,
                      end: 1466,
                      confidence: 0.99992,
                      speaker: null,
                    },
                    {
                      text: "of",
                      start: 1498,
                      end: 1646,
                      confidence: 1,
                      speaker: null,
                    },
                  ],
              },
              {
                text: "Good morning. So what is it about the conditions right now that have caused this round of wildfires to affect so many people so far away? Well, there's a couple of things. The season has been pretty dry already, and then the fact that we're getting hit in the US. Is because there's a couple of weather systems that are essentially channeling the smoke from those Canadian wildfires through Pennsylvania into the Mid Atlantic and the Northeast and kind of just dropping the smoke there.",
                start: 27850,
                end: 56190,
                confidence: 0.99667,
                words:
                  [
                    {
                      text: "Good",
                      start: 27850,
                      end: 28262,
                      confidence: 0.99667,
                      speaker: null,
                    },
                    {
                      text: "morning.",
                      start: 28316,
                      end: 28920,
                      confidence: 0.99742,
                      speaker: null,
                    },
                    {
                      text: "So",
                      start: 29290,
                      end: 29702,
                      confidence: 0.94736,
                      speaker: null,
                    },
                  ],
              },
            ],
          id: "d5a3d302-066e-43fb-b63b-8f57baf185db",
          confidence: 0.9578730257009361,
          audio_duration: 281,
        }

    PageDetails:
      x-label: Page details
      description: Details of the transcript page. Transcripts are sorted from newest to oldest. The previous URL always points to a page with older transcripts.
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - limit
        - result_count
        - current_url
        - prev_url
        - next_url
      properties:
        limit:
          x-label: Limit
          description: The number of results this page is limited to
          type: integer
        result_count:
          x-label: Result count
          description: The actual number of results in the page
          type: integer
        current_url:
          x-label: Current URL
          description: The URL used to retrieve the current page of transcripts
          type: string
          format: url
        prev_url:
          x-label: Previous URL
          description: The URL to the next page of transcripts. The previous URL always points to a page with older transcripts.
          type: [string, "null"]
          format: url
        next_url:
          x-label: Next URL
          description: The URL to the next page of transcripts. The next URL always points to a page with newer transcripts.
          type: [string, "null"]
          format: url
      example:
        {
          limit: 10,
          result_count: 10,
          current_url: "https://api.assemblyai.com/v2/transcript?limit=10",
          prev_url: "https://api.assemblyai.com/v2/transcript?limit=10&before_id=62npeahu2b-a8ea-4112-854c-69542c20d90c",
          next_url: "https://api.assemblyai.com/v2/transcript?limit=10&after_id=62nfw3mlar-01ad-4631-92f6-629929496eed",
        }

    TranscriptListItem:
      x-label: Transcript list item
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - id
        - resource_url
        - status
        - created
        - completed
        - audio_url
        - error
      properties:
        id:
          x-label: ID
          description: The unique identifier for the transcript
          type: string
          format: uuid
        resource_url:
          x-label: Resource URL
          description: The URL to retrieve the transcript
          type: string
          format: url
        status:
          x-label: Status
          description: The status of the transcript
          $ref: "#/components/schemas/TranscriptStatus"
        created:
          x-label: Created
          description: The date and time the transcript was created
          type: string
          pattern: '^(?:(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}(?:\.\d+)?))$'
          x-fern-type: datetime
          x-ts-type: Date
        completed:
          x-label: Completed
          description: The date and time the transcript was completed
          type: [string, "null"]
          pattern: '^(?:(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}(?:\.\d+)?))$'
          x-fern-type: optional<datetime>
          x-ts-type: Date | null
        audio_url:
          x-label: Audio URL
          description: The URL to the audio file
          type: string
          format: url
        error:
          x-label: Error
          description: Error message of why the transcript failed
          type: [string, "null"]
      example:
        {
          id: "9ea68fd3-f953-42c1-9742-976c447fb463",
          resource_url: "https://api.assemblyai.com/v2/transcript/9ea68fd3-f953-42c1-9742-976c447fb463",
          status: "completed",
          created: "2023-11-02T21:49:25.586965",
          completed: "2023-11-02T21:49:25.586965",
          audio_url: "https://assembly.ai/wildfires.mp3",
          error: null,
        }

    TranscriptList:
      x-label: Transcript list
      description: A list of transcripts. Transcripts are sorted from newest to oldest. The previous URL always points to a page with older transcripts.
      type: object
      x-fern-sdk-group-name: transcripts
      additionalProperties: false
      required:
        - page_details
        - transcripts
      properties:
        page_details:
          x-label: Page details
          description: Details of the transcript page
          $ref: "#/components/schemas/PageDetails"
        transcripts:
          x-label: Transcripts
          description: An array of transcripts
          type: array
          items:
            x-label: Transcript list item
            $ref: "#/components/schemas/TranscriptListItem"
      example:
        {
          page_details:
            {
              limit: 3,
              result_count: 3,
              current_url: "https://api.assemblyai.com/v2/transcript?limit=3",
              prev_url: "https://api.assemblyai.com/v2/transcript?limit=3&before_id=28a73d01-98db-41dd-9e98-2533ba0af117",
              next_url: "https://api.assemblyai.com/v2/transcript?limit=3&after_id=b33f4691-85b7-4f31-be12-a87cef1c1229",
            },
          transcripts:
            [
              {
                id: "b33f4691-85b7-4f31-be12-a87cef1c1229",
                resource_url: "https://api.assemblyai.com/v2/transcript/b33f4691-85b7-4f31-be12-a87cef1c1229",
                status: "completed",
                created: "2024-03-11T21:29:59.936851",
                completed: "2024-03-11T21:30:07.314223",
                audio_url: "http://deleted_by_user",
                error: null,
              },
              {
                id: "ce522f10-d204-42e8-a838-6b95098145cc",
                resource_url: "https://api.assemblyai.com/v2/transcript/ce522f10-d204-42e8-a838-6b95098145cc",
                status: "error",
                created: "2024-03-11T21:23:59.979420",
                completed: null,
                audio_url: "https://storage.googleapis.com/client-docs-samples/nbc.oopsie",
                error: "Download error, unable to download https://storage.googleapis.com/client-docs-samples/nbc.oopsie. Please make sure the file exists and is accessible from the internet.",
              },
              {
                id: "28a73d01-98db-41dd-9e98-2533ba0af117",
                resource_url: "https://api.assemblyai.com/v2/transcript/28a73d01-98db-41dd-9e98-2533ba0af117",
                status: "completed",
                created: "2024-03-11T21:12:57.372215",
                completed: "2024-03-11T21:13:03.267020",
                audio_url: "https://assembly.ai/nbc.mp3",
                error: null,
              },
            ],
        }

    UploadedFile:
      x-label: Uploaded file
      type: object
      x-fern-sdk-group-name: files
      additionalProperties: false
      required:
        - upload_url
      properties:
        upload_url:
          x-label: Uploaded file URL
          description: |
            A URL that points to your audio file, accessible only by AssemblyAI's servers
          type: string
          format: url
      example:
        {
          upload_url: "https://cdn.assemblyai.com/upload/f756988d-47e2-4ca3-96ce-04bb168f8f2a",
        }

    AudioIntelligenceModelStatus:
      x-label: Audio intelligence model status
      x-fern-sdk-group-name: transcripts
      type: string
      description: Either success, or unavailable in the rare case that the model failed
      enum:
        - success
        - unavailable
      x-aai-enum:
        success:
          label: Success
        unavailable:
          label: Unavailable

    Error:
      x-label: Error
      type: object
      additionalProperties: true
      required: [error]
      properties:
        error:
          x-label: Error message
          description: Error message
          type: string
        status:
          x-label: Status
          type: string
          const: error
      example:
        { error: "Transcript creation error, audio_url not found" }

        # Speech Understanding Request Schemas
    TranslationRequestBody:
      x-label: Translation request body
      description: Request body for [Translation](https://www.assemblyai.com/docs/speech-understanding/translation).
      type: object
      properties:
        translation:
          type: object
          properties:
            target_languages:
              type: array
              items:
                type: string
              description: List of target language codes (e.g., `["es", "de"]`). See [Translation](https://www.assemblyai.com/docs/speech-understanding/translation) for supported languages.
            formal:
              type: boolean
              description: Use formal language style. See [Translation](https://www.assemblyai.com/docs/speech-understanding/translation) for more details.
              default: true
            match_original_utterance:
              type: boolean
              description: When enabled with Speaker Labels, returns translated text in the utterances array. Each utterance will include a `translated_texts` key containing translations for each target language.
              default: false
          required:
            - target_languages
      required:
        - translation

    SpeakerIdentificationRequestBody:
      type: object
      description: Request body for [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification).
      properties:
        speaker_identification:
          type: object
          properties:
            speaker_type:
              type: string
              enum: [role, name]
              description: Type of speaker identification. See [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification) for details on each type.
            known_values:
              type: array
              items:
                type: string
              description: Required if speaker_type is "role". Each value must be 35 characters or less.
            speakers:
              type: array
              description: An array of speaker objects with metadata to improve identification accuracy. Each object should include a `role` or `name` (depending on `speaker_type`) and an optional `description` to help the model identify the speaker. You can also include any additional custom properties (e.g., `company`, `title`) to provide more context. Use this as an alternative to `known_values` when you want to provide additional context about each speaker.
              items:
                type: object
                properties:
                  role:
                    type: string
                    description: The role of the speaker. Required when `speaker_type` is "role".
                  name:
                    type: string
                    description: The name of the speaker. Required when `speaker_type` is "name".
                  description:
                    type: string
                    description: A description of the speaker to help the model identify them based on conversational context.
                additionalProperties: true
          required:
            - speaker_type
      required:
        - speaker_identification

    CustomFormattingRequestBody:
      type: object
      description: Request body for [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting).
      properties:
        custom_formatting:
          type: object
          properties:
            date:
              type: string
              description: Date format pattern (e.g., `"mm/dd/yyyy"`). See [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting) for more details.
            phone_number:
              type: string
              description: Phone number format pattern (e.g., `"(xxx)xxx-xxxx"`). See [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting) for more details.
            email:
              type: string
              description: Email format pattern (e.g., `"username@domain.com"`). See [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting) for more details.
      required:
        - custom_formatting

    SummarizationRequestBody:
      type: object
      description: Request body for [Summarization](https://www.assemblyai.com/docs/speech-understanding/summarization).
      properties:
        summarization:
          type: object
          properties:
            summary_type:
              type: string
              enum: [ paragraph, bullets ]
              description: Type of summary. Bullets returns short bullet point style summaries, paragraph is generally more verbose and detailed.
            effort:
              type: string
              enum: [ low, medium ]
          required:
            - summary_type
      required:
        - summarization
      
    ActionItemsRequestBody:
      type: object
      description: Request body for [Action Items](https://www.assemblyai.com/docs/speech-understanding/action-items).
      properties:
        action_items:
          type: object
          properties:
            include_decisions:
              type: bool
              description: Option to include decision making in action items.
            effort:
              type: string
              enum: [ low, medium ]
      required:
        - action_items

    TranslationResponse:
      type: object
      properties:
        translation:
          type: object
          properties:
            status:
              type: string

    SpeakerIdentificationResponse:
      type: object
      properties:
        speaker_identification:
          type: object
          properties:
            mapping:
              type: object
              description: A mapping of the original generic speaker labels (e.g., "A", "B") to the identified speaker names or roles.
              additionalProperties:
                type: string
            status:
              type: string

    CustomFormattingResponse:
      type: object
      properties:
        custom_formatting:
          type: object
          properties:
            mapping:
              type: object
              additionalProperties:
                type: string
            formatted_text:
              type: string

    SummarizationResponse:
      type: object
      properties:
        summarization:
          type: object
          properties:
            summary:
              type: array
              description: Array of summaries.
              items:
                type: object
            status:
              type: string
          required:
            - summary
            - status
              
    ActionItemsResponse:
      type: object
      properties:
        action_items:
          type: object
          properties:
            items:
              type: array
              description: Array of action items.
              items:
                type: object
            status:
              type: string
          required:
            - action_items
            - status

  examples:
    VttSubtitlesResponse:
      value: |
        WEBVTT
        00:12.340 --> 00:16.220
        Last year I showed these two slides said that demonstrate
        00:16.200 --> 00:20.040
        that the Arctic ice cap which for most of the last 3,000,000 years has been the
        00:20.020 --> 00:25.040
        size of the lower 48 States has shrunk by 40% but this understates
    SrtSubtitlesResponse:
      value: |
        1
        00:00:13,160 --> 00:00:16,694
        Last year I showed these two slides that demonstrate that the Arctic

        2
        00:00:16,734 --> 00:00:20,214
        ice cap, which for most of the last 3 million years has been the size

        3
        00:00:20,254 --> 00:00:23,274
        of the lower 48 states, has shrunk by 40%.

  responses:
    BadRequest:
      x-label: Bad request
      description: Bad request
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
          example: { "error": "This is a sample error message" }
    Unauthorized:
      x-label: Unauthorized
      description: Unauthorized
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
          example:
            { "error": "Authentication error, API token missing/invalid" }
    CannotAccessUploadedFile:
      x-label: Cannot access uploaded file
      description: Cannot access uploaded file
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
          example: { "error": "Cannot access uploaded file" }
    NotFound:
      x-label: Not found
      description: Not found
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
          example: { "error": "Not found" }
    TooManyRequests:
      x-label: Too many requests
      description: Too many requests
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
          example: { "error": "Too Many Requests" }
      headers:
        Retry-After:
          description: The number of seconds to wait before retrying the request
          schema:
            type: integer
    InternalServerError:
      x-label: Internal server error
      description: An error occurred while processing the request
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
          example:
            {
              "error": "Internal server error. Please retry your request. If the error persists, please contact support@assemblyai.com for more information.",
            }
    ServiceUnavailable:
      x-label: Service unavailable
      description: Service unavailable
    GatewayTimeout:
      x-label: Gateway timeout
      description: Gateway timeout

  securitySchemes:
    ApiKey:
      type: apiKey
      in: header
      name: Authorization
```

## docs/api-reference/specs/llm-gateway.yaml

URL: https://www.assemblyai.com/docs/api-reference/specs/llm-gateway.yaml

```yaml
openapi: 3.1.0
info:
  title: AAI Chat Completions API
  description: API for generating text with various language models.
  version: 1.0.0
servers:
  - url: https://llm-gateway.assemblyai.com/v1
    description: Production Server
    x-fern-server-name: Production

security:
  - ApiKey: []

paths:
  /chat/completions:
    post:
      x-fern-sdk-group-name: LLM Gateway
      x-fern-display-name: Chat completions
      summary: Create a chat completion
      description: "\n\n<Note>To use our EU server for LLM Gateway, replace `llm-gateway.assemblyai.com` with `llm-gateway.eu.assemblyai.com`.</Note>\n\nGenerates a response from a model given a prompt or a series of messages."
      operationId: createChatCompletion
      requestBody:
        description: Request body for creating a chat completion.
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/LLMGatewayRequest"
            examples:
              messages_example:
                summary: Chat with messages
                value:
                  model: "qwen3.5-4b-32k-fast"
                  messages:
                    - role: "user"
                      content: "Hello, how are you?"
                  max_tokens: 100
                  temperature: 0.7
              prompt_example:
                summary: Simple prompt
                value:
                  model: "qwen3.5-4b-32k-fast"
                  prompt: "Write a haiku about coding"
                  max_tokens: 50
                  temperature: 0.5
              transcript_id_example:
                summary: Inject a transcript by ID
                value:
                  model: "gemini-2.5-flash-lite"
                  messages:
                    - role: "user"
                      content: "hi there"
                    - role: "assistant"
                      content: "Hi! How can I help?"
                    - role: "user"
                      content: "Here is a transcript: {{ transcript }}. Return the text verbatim."
                  transcript_id: "065a71ac-dc3e-4e38-9374-e54c0bea564f"
              structured_output_example:
                summary: Structured output with JSON schema
                value:
                  model: "gemini-2.5-flash-lite"
                  messages:
                    - role: "system"
                      content: "You are a helpful math tutor. Guide the user through the solution step by step."
                    - role: "user"
                      content: "how can I solve 8x + 7 = -23"
                  response_format:
                    type: "json_schema"
                    json_schema:
                      name: "math_reasoning"
                      schema:
                        type: "object"
                        properties:
                          steps:
                            type: "array"
                            items:
                              type: "object"
                              properties:
                                explanation:
                                  type: "string"
                                output:
                                  type: "string"
                              required: ["explanation", "output"]
                              additionalProperties: false
                          final_answer:
                            type: "string"
                        required: ["steps", "final_answer"]
                        additionalProperties: false
                      strict: true
              post_processing_example:
                summary: JSON repair post-processing
                value:
                  model: "gemini-2.5-flash-lite"
                  messages:
                    - role: "user"
                      content: "Extract the user name and return as JSON"
                  post_processing_steps:
                    - type: "json-repair"
      responses:
        "200":
          description: Successful response containing the model's choices.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Response"
        default:
          description: An unexpected error occurred.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/ErrorResponse"

  /understanding:
    post:
      x-fern-sdk-group-name: LLM Gateway
      x-fern-display-name: Speech Understanding
      summary: Process speech understanding
      description: "\n\n<Note>To use our EU server for LLM Gateway, replace `llm-gateway.assemblyai.com` with `llm-gateway.eu.assemblyai.com`.</Note>\n\nPerform various speech understanding tasks like [Translation](https://www.assemblyai.com/docs/speech-understanding/translation), [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification), and [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting)."
      operationId: createSpeechUnderstanding
      requestBody:
        description: Request body for speech understanding tasks.
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/UnderstandingRequest"
            examples:
              translation_example:
                summary: Translation request
                value:
                  transcript_id: "12345"
                  speech_understanding:
                    request:
                      translation:
                        target_languages: ["es", "de"]
                        formal: true
                        match_original_utterance: true
              speaker_identification_example:
                summary: Speaker identification request with known_values
                value:
                  transcript_id: "12345"
                  speech_understanding:
                    request:
                      speaker_identification:
                        speaker_type: "role"
                        known_values: ["interviewer", "candidate"]
              speaker_identification_with_speakers_example:
                summary: Speaker identification request with speakers metadata
                value:
                  transcript_id: "12345"
                  speech_understanding:
                    request:
                      speaker_identification:
                        speaker_type: "name"
                        speakers:
                          - name: "Michel Martin"
                            description: "Hosts the program and interviews the guests"
                            company: "NPR"
                            title: "Host Morning Edition"
                          - name: "Peter DeCarlo"
                            description: "Answers questions from the interview"
                            company: "Johns Hopkins University"
                            title: "Professor and Vice Chair of Environmental Health and Engineering"
              custom_formatting_example:
                summary: Custom formatting request
                value:
                  transcript_id: "12345"
                  speech_understanding:
                    request:
                      custom_formatting:
                        date: "mm/dd/yyyy"
                        phone_number: "(xxx)xxx-xxxx"
                        email: "username@domain.com"
      responses:
        "200":
          description: Successful response containing the speech understanding results.
          content:
            application/json:
              schema:
                oneOf:
                  - $ref: "#/components/schemas/LLMGatewayTranslationResponse"
                  - $ref: "#/components/schemas/LLMGatewaySpeakerIdentificationResponse"
                  - $ref: "#/components/schemas/LLMGatewayCustomFormattingResponse"
                  - $ref: "#/components/schemas/LLMGatewaySummarizationResponse"
              examples:
                complete_response:
                  summary: Complete response example
                  value:
                    speech_understanding:
                      request:
                        translation:
                          target_languages: ["es", "de"]
                          formal: true
                          match_original_utterance: true
                        speaker_identification:
                          speaker_type: "name"
                          speakers:
                            - name: "Michel Martin"
                              description: "Hosts the program and interviews the guests"
                              company: "NPR"
                              title: "Host Morning Edition"
                            - name: "Peter DeCarlo"
                              description: "Answers questions from the interview"
                              company: "Johns Hopkins University"
                              title: "Professor and Vice Chair of Environmental Health and Engineering"
                        custom_formatting:
                          date: "mm/dd/yyyy"
                          phone_number: "(xxx)xxx-xxxx"
                          email: "username@domain.com"
                        summarization:
                          summary_type: "bullets"
                      response:
                        summarization:
                          status: "success"
                          effort: "low"
                          summary:
                            - start: 0
                              end: 1000
                              bullets:
                                - "Host introduces Dr Peter DeCarlo"
                                - "Asks for a quick intro"
                            - start: 1050
                              end: 3500
                              bullets:
                                - "Identifies key issues with the climate"
                        translation:
                          status: "success"
                        speaker_identification:
                          mapping:
                            "A": "Michel Martin"
                            "B": "Peter DeCarlo"
                          status: "success"
                        custom_formatting:
                          status: "success"
                          mapping:
                            "2024-12-25": "12/25/2024"
                            "555-1234-5678": "(555)123-45678"
                          formatted_text: "Call me at (555)123-45678 on 12/25/2024"
                          formatted_utterances:
                            - confidence: 0.92
                              start: 0
                              end: 2500
                              text: "Hi, I'm the interviewer. Call me at (555)123-45678 on 12/25/2024"
                              speaker: "interviewer"
                            - confidence: 0.95
                              start: 2500
                              end: 5000
                              text: "Thanks! I'll reach out then."
                              speaker: "candidate"
                    translated_texts:
                      es: "Hola, soy el entrevistador. Llámame al cinco cinco cinco uno dos tres cuatro cinco seis siete ocho el veinticinco de diciembre de dos mil veinticuatro. ¡Gracias! Me pondré en contacto entonces."
                      de: "Hallo, ich bin der Interviewer. Rufen Sie mich an unter fünf fünf fünf eins zwei drei vier fünf sechs sieben acht am fünfundzwanzigsten Dezember zweitausendvierundzwanzig. Danke! Ich werde mich dann melden."
                    utterances:
                      - confidence: 0.92
                        start: 0
                        end: 2500
                        text: "Hi, I'm the interviewer. Call me at five five five one two three four five six seven eight on December twenty fifth twenty twenty four"
                        speaker: "interviewer"
                        translated_texts:
                          es: "Hola, soy el entrevistador. Llámame al cinco cinco cinco uno dos tres cuatro cinco seis siete ocho el veinticinco de diciembre de dos mil veinticuatro"
                          de: "Hallo, ich bin der Interviewer. Rufen Sie mich an unter fünf fünf fünf eins zwei drei vier fünf sechs sieben acht am fünfundzwanzigsten Dezember zweitausendvierundzwanzig"
                      - confidence: 0.95
                        start: 2500
                        end: 5000
                        text: "Thanks! I'll reach out then."
                        speaker: "candidate"
                        translated_texts:
                          es: "¡Gracias! Me pondré en contacto entonces."
                          de: "Danke! Ich werde mich dann melden."
                    words: []
        default:
          description: An unexpected error occurred.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/ErrorResponse"

  /models:
    get:
      x-fern-sdk-group-name: LLM Gateway
      x-fern-display-name: List available models
      summary: List available models
      description: "\n\n<Note>To use our EU server for LLM Gateway, replace `llm-gateway.assemblyai.com` with `llm-gateway.eu.assemblyai.com`.</Note>\n\nReturns the list of models available through LLM Gateway, including the supported parameters, context length, pricing, and regional availability of each model. This endpoint doesn't require authentication."
      operationId: listAvailableModels
      security: []
      responses:
        "200":
          description: Successful response containing the list of available models.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/ListAvailableModelsResponse"
              examples:
                models_example:
                  summary: List of available models
                  value:
                    data:
                      - id: "qwen3.5-4b-32k-fast"
                        name: "Qwen3.5 4B Fast"
                        description: ""
                        creator: "Qwen"
                        context_length: 32768
                        supported_parameters:
                          - "max_tokens"
                          - "temperature"
                          - "stream"
                        default_parameters:
                          temperature: null
                          top_p: null
                          frequency_penalty: null
                        top_provider:
                          is_moderated: false
                          context_length: 32768
                          max_completion_tokens: 8192
                        pricing:
                          global:
                            prompt: 0.1
                            completions: 0.5
                        retirement_date: 0
                        available_regions: ["us", "eu"]
                      - id: "claude-sonnet-4-6"
                        name: "Sonnet 4.6"
                        description: ""
                        creator: "Claude"
                        context_length: 200000
                        supported_parameters:
                          - "max_tokens"
                          - "temperature"
                          - "tools"
                          - "tool_choice"
                          - "response_format"
                          - "stream"
                        default_parameters:
                          temperature: null
                          top_p: null
                          frequency_penalty: null
                        top_provider:
                          is_moderated: false
                          context_length: 200000
                          max_completion_tokens: 128000
                        pricing:
                          global:
                            prompt: 3
                            completions: 15
                            input_cache_read: 0.3
                            input_cache_write: 3.75
                            input_cache_write_1h: 6
                          regional_increase_percent: 0.1
                        retirement_date: 0
                        available_regions: ["us", "eu", "global"]
                      - id: "gemini-2.5-flash-lite"
                        name: "Gemini 2.5 Flash Lite"
                        description: ""
                        creator: "Google"
                        context_length: 1048576
                        supported_parameters:
                          - "max_tokens"
                          - "response_format"
                          - "temperature"
                          - "tools"
                          - "tool_choice"
                          - "stream"
                        default_parameters:
                          temperature: null
                          top_p: null
                          frequency_penalty: null
                        top_provider:
                          is_moderated: false
                          context_length: 1048576
                          max_completion_tokens: 65535
                        pricing:
                          global:
                            prompt: 0.1
                            completions: 0.4
                            input_cache_read: 0.01
                          regional_increase_percent: 0.1
                        retirement_date: 0
                        available_regions: ["us", "eu", "global"]
        default:
          description: An unexpected error occurred.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/ErrorResponse"

components:
  schemas:
    # Request Schemas
    LLMGatewayRequest:
      type: object
      description: The main request body for the chat completions endpoint.
      properties:
        messages:
          type: array
          items:
            $ref: "#/components/schemas/Message"
          description: A list of messages comprising the conversation so far.
        prompt:
          type: string
          description: A simple string prompt. The API will automatically convert this into a user message.
        transcript_id:
          type: string
          description: >
            Optional. The ID of an AssemblyAI transcript whose text replaces the first `{{ transcript }}`
            tag in the prompt. See [Inject a transcript by ID](https://www.assemblyai.com/docs/llm-gateway/chat-completions#inject-a-transcript-by-id)
            for substitution rules and edge cases.
          examples: ["065a71ac-dc3e-4e38-9374-e54c0bea564f"]
        model:
          type: string
          description: The ID of the model to use for this request. See [LLM Gateway Overview](https://www.assemblyai.com/docs/llm-gateway/quickstart#available-models) for available models.
          examples: ["claude-sonnet-4-5-20250929"]
        model_region:
          type: string
          enum: [global]
          description: >
            Optional. Routes the request to the provider's global (non-region) endpoints
            for lower-cost processing. The only accepted value is `global`; omit this field
            for default in-region processing. Currently live for Anthropic Claude models,
            with Google Gemini 3 series coming soon.
          examples: ["global"]
        max_tokens:
          type: integer
          minimum: 1
          default: 1000
          description: The maximum number of tokens to generate in the completion. Default is 1000.
        temperature:
          type: number
          format: float
          minimum: 0.0
          maximum: 2.0
          description: Controls randomness. Lower values produce more deterministic results.
        stream:
          type: boolean
          default: false
          description: When true, responses are streamed as server-sent events (SSE). Supported on OpenAI models only.
        tools:
          type: array
          items:
            $ref: "#/components/schemas/Tool"
          description: A list of tools the model may call.
        tool_choice:
          $ref: "#/components/schemas/ToolChoice"
        response_format:
          $ref: "#/components/schemas/ResponseFormat"
          description: Specifies the format of the model's response. Use this to constrain the model to output valid JSON matching a schema. Supported by OpenAI (GPT-4.1, GPT-5.x), Gemini, and Claude models. Not supported by gpt-oss models.
        fallbacks:
          type: array
          items:
            $ref: "#/components/schemas/FallbackObject"
          description: An array of fallback objects. Each object must include a `model` and can optionally override any field from the original request. If the primary model fails, the LLM Gateway tries each fallback in order until one succeeds. See [Specify fallback models](https://www.assemblyai.com/docs/llm-gateway/fallback) for more details.
        fallback_config:
          $ref: "#/components/schemas/FallbackConfig"
          description: Configuration for fallback behavior, including retry and depth settings. See [Specify fallback models](https://www.assemblyai.com/docs/llm-gateway/fallback) for more details.
        post_processing_steps:
          type: array
          items:
            $ref: "#/components/schemas/PostProcessingStep"
          description: >
            An ordered list of post-processing steps to apply to the model's response after generation.
            Currently supports `json-repair`, which automatically fixes malformed JSON in
            LLM Gateway content responses.
            See [Post-processing](https://www.assemblyai.com/docs/llm-gateway/structured-outputs#post-processing) for details.
      required:
        - model

    Message:
      oneOf:
        - $ref: "#/components/schemas/UserAssistantSystemMessage"
        - $ref: "#/components/schemas/ToolMessage"
      discriminator:
        propertyName: role
        mapping:
          user: "#/components/schemas/UserAssistantSystemMessage"
          assistant: "#/components/schemas/UserAssistantSystemMessage"
          system: "#/components/schemas/UserAssistantSystemMessage"
          tool: "#/components/schemas/ToolMessage"

    UserAssistantSystemMessage:
      type: object
      properties:
        role:
          type: string
          enum: [user, assistant, system]
        content:
          oneOf:
            - type: string
            - type: array
              items:
                $ref: "#/components/schemas/ContentPart"
          description: The content of the message. Can be a string or an array of content parts (for user messages).
        name:
          type: string
          description: An optional name for the participant.
      required:
        - role
        - content

    ToolMessage:
      type: object
      properties:
        role:
          type: string
          enum: [tool]
        content:
          type: string
          description: The result of the tool call.
        tool_call_id:
          type: string
          description: The ID of the tool call that this message is responding to.
      required:
        - role
        - content
        - tool_call_id

    ContentPart:
      description: Currently only supports text content parts.
      $ref: "#/components/schemas/TextContent"

    TextContent:
      type: object
      properties:
        type:
          type: string
          enum: [text]
        text:
          type: string
      required:
        - type
        - text

    Tool:
      type: object
      properties:
        type:
          type: string
          enum: [function]
        function:
          $ref: "#/components/schemas/FunctionDescription"
      required:
        - type
        - function

    FunctionDescription:
      type: object
      properties:
        name:
          type: string
          description: The name of the function to be called.
        description:
          type: string
          description: A description of what the function does.
        parameters:
          type: object
          description: A JSON Schema object describing the parameters the function accepts.
          additionalProperties: true
      required:
        - name
        - parameters

    ToolChoice:
      oneOf:
        - type: string
          enum: [none, auto]
        - type: object
          properties:
            type:
              type: string
              enum: [function]
            function:
              type: object
              properties:
                name:
                  type: string
              required:
                - name
          required:
            - type
            - function
      description: Controls which (if any) function is called by the model.

    ResponseFormat:
      type: object
      description: Specifies the format of the model's response. Use `json_schema` type to constrain the model to output valid JSON matching a schema.
      properties:
        type:
          type: string
          enum: [json_schema]
          description: The type of response format. Use `json_schema` for structured outputs.
        json_schema:
          $ref: "#/components/schemas/JsonSchemaConfig"
          description: The JSON schema configuration object.
      required:
        - type
        - json_schema

    JsonSchemaConfig:
      type: object
      description: Configuration for JSON schema-based structured outputs.
      properties:
        name:
          type: string
          description: A name for the schema. Used for identification purposes.
        schema:
          type: object
          description: A valid JSON Schema object that defines the structure of the expected response.
          additionalProperties: true
        strict:
          type: boolean
          description: When `true`, the model will strictly adhere to the schema. Recommended for reliable parsing.
          default: false
      required:
        - name
        - schema

    FallbackObject:
      type: object
      description: A fallback model configuration. Each object must include a `model` and can optionally override any field from the original request. Fields not specified in the fallback inherit the values from the original request. See [Specify fallback models](https://www.assemblyai.com/docs/llm-gateway/fallback) for more details.
      properties:
        model:
          type: string
          description: The fallback model to use. See [LLM Gateway Overview](https://www.assemblyai.com/docs/llm-gateway/quickstart#available-models) for available models.
          examples: ["claude-sonnet-4-6"]
        messages:
          type: array
          items:
            $ref: "#/components/schemas/Message"
          description: Override the messages for the fallback request.
        max_tokens:
          type: integer
          minimum: 1
          description: Override the maximum number of tokens to generate in the fallback completion.
        temperature:
          type: number
          format: float
          minimum: 0.0
          maximum: 2.0
          description: Override the temperature for the fallback request.
      additionalProperties: true
      required:
        - model

    FallbackConfig:
      type: object
      description: Configuration for fallback behavior. See [Specify fallback models](https://www.assemblyai.com/docs/llm-gateway/fallback) for more details.
      properties:
        retry:
          type: boolean
          default: true
          description: Whether to automatically retry the request once after 500ms on failure. Defaults to `true`.
        depth:
          type: integer
          minimum: 1
          maximum: 2
          default: 1
          description: The maximum number of fallbacks to traverse. Defaults to `1`, with a maximum of `2`.

    PostProcessingStep:
      type: object
      description: A single post-processing operation to apply to the model's response.
      properties:
        type:
          type: string
          enum: [json-repair]
          description: The type of post-processing to apply. Currently `json-repair` is supported.
      required:
        - type

    # Response Schemas
    Response:
      type: object
      properties:
        request_id:
          type: string
          format: uuid
        choices:
          type: array
          items:
            $ref: "#/components/schemas/Choice"
        request:
          type: object
          description: A copy of the original request, excluding `prompt` and `messages`.
          properties:
            model:
              type: string
            max_tokens:
              type: integer
            temperature:
              type: number
            tools:
              type: array
              items:
                $ref: "#/components/schemas/Tool"
            tool_choice:
              $ref: "#/components/schemas/ToolChoice"
        usage:
          $ref: "#/components/schemas/Usage"
        http_status_code:
          type: integer
          description: The HTTP status code of the response
          example: 200
        response_time:
          type: integer
          description: The response time in nanoseconds
          example: 275510459
        llm_status_code:
          type: integer
          description: The status code from the LLM provider
          example: 200

    Choice:
      type: object
      properties:
        message:
          $ref: "#/components/schemas/ResponseMessage"
        finish_reason:
          type: string
          description: The reason the model stopped generating tokens.
          examples: ["stop"]

    ResponseMessage:
      type: object
      properties:
        role:
          type: string
        content:
          type: string
          nullable: true
          description: "The text content of the model's response. Null or empty when the model is only emitting tool_calls."
        tool_calls:
          type: array
          items:
            $ref: "#/components/schemas/FunctionToolCall"

    FunctionToolCall:
      type: object
      properties:
        id:
          type: string
        type:
          type: string
          enum: [function]
        function:
          $ref: "#/components/schemas/FunctionCall"
      required:
        - id
        - type
        - function

    FunctionCall:
      type: object
      properties:
        name:
          type: string
        arguments:
          type: string
          description: The arguments to call the function with, as a JSON-formatted string.
      required:
        - name
        - arguments

    Usage:
      type: object
      properties:
        input_tokens:
          type: integer
        output_tokens:
          type: integer
        total_tokens:
          type: integer
      required:
        - input_tokens
        - output_tokens
        - total_tokens

    # Understanding Request/Response Schemas
    UnderstandingRequest:
      type: object
      properties:
        transcript_id:
          type: string
          description: The ID of the transcript to process.
        speech_understanding:
          type: object
          description: "The speech understanding task to perform. Supports [Translation](https://www.assemblyai.com/docs/speech-understanding/translation), [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification), and [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting). Click into the request object below to see the available options."
          properties:
            request:
              oneOf:
                - $ref: "#/components/schemas/TranslationRequestBody"
                - $ref: "#/components/schemas/SpeakerIdentificationRequestBody"
                - $ref: "#/components/schemas/CustomFormattingRequestBody"
                - $ref: "#/components/schemas/SummarizationRequestBody"
                - $ref: "#/components/schemas/ActionItemsRequestBody"
          required:
            - request
      required:
        - transcript_id
        - speech_understanding

    TranslationRequestBody:
      type: object
      description: Request body for [Translation](https://www.assemblyai.com/docs/speech-understanding/translation).
      properties:
        translation:
          type: object
          properties:
            target_languages:
              type: array
              items:
                type: string
              description: List of target language codes (e.g., `["es", "de"]`). See [Translation](https://www.assemblyai.com/docs/speech-understanding/translation) for supported languages.
            formal:
              type: boolean
              description: Use formal language style. See [Translation](https://www.assemblyai.com/docs/speech-understanding/translation) for more details.
              default: true
            match_original_utterance:
              type: boolean
              description: When enabled with Speaker Labels, returns translated text in the utterances array. Each utterance will include a `translated_texts` key containing translations for each target language.
              default: false
          required:
            - target_languages
      required:
        - translation

    SpeakerIdentificationRequestBody:
      type: object
      description: Request body for [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification).
      properties:
        speaker_identification:
          type: object
          properties:
            speaker_type:
              type: string
              enum: [role, name]
              description: Type of speaker identification. See [Speaker Identification](https://www.assemblyai.com/docs/speech-understanding/speaker-identification) for details on each type.
            known_values:
              type: array
              items:
                type: string
              description: Required if speaker_type is "role". Each value must be 35 characters or less.
            speakers:
              type: array
              description: An array of speaker objects with metadata to improve identification accuracy. Each object should include a `role` or `name` (depending on `speaker_type`) and an optional `description` to help the model identify the speaker. You can also include any additional custom properties (e.g., `company`, `title`) to provide more context. Use this as an alternative to `known_values` when you want to provide additional context about each speaker.
              items:
                type: object
                properties:
                  role:
                    type: string
                    description: The role of the speaker. Required when `speaker_type` is "role".
                  name:
                    type: string
                    description: The name of the speaker. Required when `speaker_type` is "name".
                  description:
                    type: string
                    description: A description of the speaker to help the model identify them based on conversational context.
                additionalProperties: true
          required:
            - speaker_type
      required:
        - speaker_identification

    CustomFormattingRequestBody:
      type: object
      description: Request body for [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting).
      properties:
        custom_formatting:
          type: object
          properties:
            date:
              type: string
              description: Date format pattern (e.g., `"mm/dd/yyyy"`). See [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting) for more details.
            phone_number:
              type: string
              description: Phone number format pattern (e.g., `"(xxx)xxx-xxxx"`). See [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting) for more details.
            email:
              type: string
              description: Email format pattern (e.g., `"username@domain.com"`). See [Custom Formatting](https://www.assemblyai.com/docs/speech-understanding/custom-formatting) for more details.
      required:
        - custom_formatting

    SummarizationRequestBody:
      type: object
      description: Request body for [Summarization](https://www.assemblyai.com/docs/speech-understanding/summarization).
      properties:
        summarization:
          type: object
          properties:
            summary_type:
              type: string
              enum: [ paragraph, bullets ]
              description: Type of summary. Bullets returns short bullet point style summaries, paragraph is generally more verbose and detailed.
            effort:
              type: string
              enum: [ low, medium ]
          required:
            - summary_type
      required:
        - summarization

    ActionItemsRequestBody:
      type: object
      description: Request body for [Action Items](https://www.assemblyai.com/docs/speech-understanding/action-items).
      properties:
        action_items:
          type: object
          properties:
            include_decisions:
              type: bool
              description: Option to include decision making in action items.
            effort:
              type: string
              enum: [ low, medium ]
      required:
        - action_items

    LLMGatewayTranslationResponse:
      type: object
      properties:
        speech_understanding:
          type: object
          properties:
            request:
              $ref: "#/components/schemas/TranslationRequestBody"
            response:
              type: object
              properties:
                translation:
                  type: object
                  properties:
                    status:
                      type: string
        translated_texts:
          type: object
          description: 'Translated text keyed by language code (e.g., `{"es": "Texto traducido"}`)'
          additionalProperties:
            type: string

        utterances:
          type: array
          description: Array of utterances with translations (when `match_original_utterance` is true)
          items:
            type: object
            properties:
              translated_texts:
                type: object
                description: 'Translations keyed by language code (e.g., `{"es": "Texto traducido", "de": "Übersetzter Text"}`). Only present when `match_original_utterance` is enabled.'
                additionalProperties:
                  type: string
        words:
          type: array
          items:
            type: object

    LLMGatewaySpeakerIdentificationResponse:
      type: object
      properties:
        speech_understanding:
          type: object
          properties:
            request:
              $ref: "#/components/schemas/SpeakerIdentificationRequestBody"
            response:
              type: object
              properties:
                speaker_identification:
                  type: object
                  properties:
                    mapping:
                      type: object
                      description: A mapping of the original generic speaker labels (e.g., "A", "B") to the identified speaker names or roles.
                      additionalProperties:
                        type: string
                    status:
                      type: string
                  required:
                    - status
        utterances:
          type: array
          items:
            type: object
        words:
          type: array
          items:
            type: object

    LLMGatewayCustomFormattingResponse:
      type: object
      properties:
        speech_understanding:
          type: object
          properties:
            request:
              $ref: "#/components/schemas/CustomFormattingRequestBody"
            response:
              type: object
              properties:
                custom_formatting:
                  type: object
                  properties:
                    mapping:
                      type: object
                      additionalProperties:
                        type: string
                    formatted_text:
                      type: string
                    formatted_utterances:
                      type: array
                      description: Array of formatted utterances. Only included when utterances exist and formatting was applied.
                      items:
                        type: object
                    status:
                      type: string
                  required:
                    - mapping
                    - status
        utterances:
          type: array
          items:
            type: object

    LLMGatewaySummarizationResponse:
      type: object
      properties:
        speech_understanding:
          type: object
          properties:
            request:
              $ref: "#/components/schemas/SummarizationRequestBody"
            response:
              type: object
              properties:
                summarization:
                  type: object
                  properties:
                    summary:
                      type: array
                      description: Array of summaries.
                      items:
                        type: object
                    status:
                      type: string
                  required:
                    - summary
                    - status

    LLMGatewayActionItemsResponse:
      type: object
      properties:
        speech_understanding:
          type: object
          properties:
            request:
              $ref: "#/components/schemas/ActionItemsRequestBody"
            response:
              type: object
              properties:
                action_items:
                  type: object
                  properties:
                    items:
                      type: array
                      description: Array of action items.
                      items:
                        type: object
                    status:
                      type: string
                  required:
                    - action_items
                    - status


    # Models Endpoint Schemas
    ListAvailableModelsResponse:
      type: object
      description: The response returned by the models endpoint.
      properties:
        data:
          type: array
          items:
            $ref: "#/components/schemas/ModelDetails"
          description: The list of models available through LLM Gateway.
      required:
        - data

    ModelDetails:
      type: object
      description: Details about a model available through LLM Gateway.
      properties:
        id:
          type: string
          description: The model ID. Use this value in the `model` field of a chat completion request.
          examples: ["qwen3.5-4b-32k-fast"]
        name:
          type: string
          description: The human-readable name of the model.
          examples: ["Qwen3.5 4B Fast"]
        description:
          type: string
          description: A description of the model.
        creator:
          type: string
          description: The creator of the model.
          examples: ["Qwen"]
        context_length:
          type: integer
          description: The maximum context length of the model, in tokens.
          examples: [32768]
        supported_parameters:
          type: array
          items:
            type: string
          description: The request parameters and features supported by the model (e.g., `max_tokens`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream`).
        default_parameters:
          $ref: "#/components/schemas/DefaultParameters"
          description: Default parameter values applied to requests for this model. A `null` value means no default is set.
        top_provider:
          $ref: "#/components/schemas/TopProvider"
          description: Information about the best provider for this model. Not yet supported.
        pricing:
          $ref: "#/components/schemas/ModelPricing"
          description: Pricing details for the model, in USD per 1 million tokens.
        retirement_date:
          type: integer
          format: int64
          description: The Unix timestamp of the model's retirement date. `0` if no retirement date is set.
        available_regions:
          type: array
          items:
            type: string
          description: The regions where the model is available (`us`, `eu`, and/or `global`). See [Cloud endpoints & data residency](https://www.assemblyai.com/docs/llm-gateway/cloud-endpoints-and-data-residency) for more details.

    DefaultParameters:
      type: object
      description: Default parameter values applied to requests for this model. A `null` value means no default is set.
      properties:
        temperature:
          type: number
          format: float
          nullable: true
          description: The default temperature for this model, if any.
        top_p:
          type: number
          format: float
          nullable: true
          description: The default top_p for this model, if any.
        frequency_penalty:
          type: integer
          nullable: true
          description: The default frequency penalty for this model, if any.

    TopProvider:
      type: object
      description: Information about the best provider for this model. Not yet supported.
      properties:
        is_moderated:
          type: boolean
          description: Whether the provider moderates requests.
        context_length:
          type: integer
          description: The maximum context length supported by the provider, in tokens.
        max_completion_tokens:
          type: integer
          description: The maximum number of completion tokens supported by the provider.

    ModelPricing:
      type: object
      description: Pricing details for the model, in USD per 1 million tokens.
      properties:
        global:
          $ref: "#/components/schemas/ModelPricingData"
          description: Pricing for global processing. See [Global routing](https://www.assemblyai.com/docs/llm-gateway/cloud-endpoints-and-data-residency#global-routing) for more details.
        us:
          $ref: "#/components/schemas/ModelPricingData"
          description: US-specific pricing. Only present when the model has pricing distinct from global pricing in the US region.
        eu:
          $ref: "#/components/schemas/ModelPricingData"
          description: EU-specific pricing. Only present when the model has pricing distinct from global pricing in the EU region.
        regional_increase_percent:
          type: number
          format: float
          description: The fractional price increase applied to in-region (US or EU) processing relative to global pricing (e.g., `0.1` means 10%).
      required:
        - global

    ModelPricingData:
      type: object
      description: Prices in USD per 1 million tokens.
      properties:
        prompt:
          type: number
          format: float
          description: The price per 1 million input (prompt) tokens.
        completions:
          type: number
          format: float
          description: The price per 1 million output (completion) tokens.
        input_cache_read:
          type: number
          format: float
          description: The price per 1 million cached input tokens read. Only present for models that support prompt caching.
        input_cache_write:
          type: number
          format: float
          description: The price per 1 million input tokens written to the cache. Only present for models that support prompt caching.
        input_cache_write_1h:
          type: number
          format: float
          description: The price per 1 million input tokens written to the cache with a 1-hour TTL. Only present for models that support prompt caching.

    # Error Schema
    ErrorResponse:
      type: object
      properties:
        code:
          type: integer
          format: int32
          description: HTTP status code for the error.
        message:
          type: string
          description: A human-readable description of the error.
        request_id:
          type: string
          format: uuid
          description: Unique identifier for the request.
        metadata:
          type: object
          description: Optional. Present on 400 responses with per-field validation details.
          properties:
            errors:
              type: array
              items:
                type: string
              description: List of specific validation failure messages.
      required:
        - code
        - message
        - request_id

  securitySchemes:
    ApiKey:
      type: apiKey
      in: header
      name: Authorization
```

## docs/api-reference/specs/streaming-token.yaml

URL: https://www.assemblyai.com/docs/api-reference/specs/streaming-token.yaml

```yaml
openapi: 3.0.3

info:
  title: AssemblyAI Streaming Token API
  description: API for generating temporary authentication tokens for streaming services
  version: 1.0.0
  termsOfService: https://www.assemblyai.com/legal/terms-of-service
  contact:
    name: API Support
    email: support@assemblyai.com
    url: https://www.assemblyai.com/docs/

servers:
  - url: https://streaming.assemblyai.com
    description: Production server
    x-fern-server-name: Default

paths:
  /v3/token:
    get:
      summary: Generate temporary streaming token
      description: |
        Generates a temporary authentication token for use with streaming services. See [Authenticate with a temporary token](https://www.assemblyai.com/docs/streaming/authenticate-with-a-temporary-token) for more details.
      operationId: generateStreamingToken
      x-fern-sdk-group-name: Streaming API
      x-fern-sdk-method-name: generateStreamingToken
      security:
        - apiKey: []
      parameters:
        - name: expires_in_seconds
          in: query
          description: |
            Token redemption window, in seconds. The token must be used to open a WebSocket connection within this window. If the window has elapsed when the WebSocket is opened, the server rejects the session on the first frame. This does NOT cap the duration of the resulting streaming session — once the WebSocket is open the session runs for up to `max_session_duration_seconds`.
          required: true
          schema:
            type: integer
            minimum: 1
            maximum: 600
            example: 60
        - name: max_session_duration_seconds
          in: query
          description: |
            Maximum duration of the streaming session started with this token, in seconds. If you need to finalize work before the session ends, run a client-side timer using the value you passed here.
          required: false
          schema:
            type: integer
            minimum: 60
            maximum: 10800
            example: 600
            default: 10800
      x-codeSamples:
        - lang: python
          label: Python SDK
          source: |
            import os

            from assemblyai.streaming.v3 import RealTimeTranscriber

            client = RealTimeTranscriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])

            token = client.create_temporary_token(expires_in_seconds=60)
            print(token)
        - lang: javascript
          label: JavaScript SDK
          source: |
            import { AssemblyAI } from "assemblyai";

            const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

            const token = await client.streaming.createTemporaryToken({
              expires_in_seconds: 60,
            });
            console.log(token);
        - lang: bash
          label: cURL
          source: |
            curl -s -G https://streaming.assemblyai.com/v3/token \
              -H "authorization: $ASSEMBLYAI_API_KEY" \
              -d expires_in_seconds=60 | jq -r .token
        - lang: python
          label: Python
          source: |
            import os

            import requests

            headers = {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}

            response = requests.get(
                "https://streaming.assemblyai.com/v3/token",
                headers=headers,
                params={"expires_in_seconds": 60},
            )

            response.raise_for_status()
            print(response.json()["token"])
        - lang: javascript
          label: JavaScript
          source: |
            const url = new URL("https://streaming.assemblyai.com/v3/token");
            url.searchParams.set("expires_in_seconds", "60");

            const response = await fetch(url, {
              headers: { authorization: process.env.ASSEMBLYAI_API_KEY },
            });

            const { token } = await response.json();
            console.log(token);
      responses:
        "200":
          description: Successfully generated temporary token
          content:
            application/json:
              schema:
                type: object
                required:
                  - token
                  - expires_in_seconds
                properties:
                  token:
                    type: string
                    description: The temporary authentication token
                    example: "your-temporary-token"
                  expires_in_seconds:
                    type: integer
                    description: |
                      The token redemption window in seconds — the time the client has to use this token to open a WebSocket before it expires unused. This is not the session duration; see `max_session_duration_seconds` for that.
                    example: 60
        "401":
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        "400":
          description: Bad Request - Invalid parameters
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        "429":
          description: Too Many Requests - Rate limit exceeded
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        "500":
          description: Internal Server Error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"

components:
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: Authorization
      description: Your AssemblyAI API key
  schemas:
    Error:
      type: object
      required:
        - error
      properties:
        error:
          type: string
          description: Error message describing what went wrong
        code:
          type: string
          description: Error code for programmatic handling
        details:
          type: object
          description: Additional error details if available
```

## docs/api-reference/specs/voice-agent-token.yaml

URL: https://www.assemblyai.com/docs/api-reference/specs/voice-agent-token.yaml

```yaml
openapi: 3.0.3

info:
  title: AssemblyAI Voice Agent Token API
  description: API for generating temporary authentication tokens for the Voice Agent API.
  version: 1.0.0
  termsOfService: https://www.assemblyai.com/legal/terms-of-service
  contact:
    name: API Support
    email: support@assemblyai.com
    url: https://www.assemblyai.com/docs/

servers:
  - url: https://agents.assemblyai.com
    description: Production server
    x-fern-server-name: Default

paths:
  /v1/token:
    get:
      summary: Generate temporary Voice Agent token
      description: |
        Generate a temporary authentication token for the Voice Agent API. Use this to
        connect browser or other client-side applications to the Voice Agent API without
        exposing your permanent API key. See
        [Browser integration](/voice-agents/voice-agent-api/browser-integration)
        for the full flow.

        Each token is one-time use and can only be used for a single session. All usage
        is attributed to the API key that generated the token.
      operationId: generateVoiceAgentToken
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: generateVoiceAgentToken
      security:
        - apiKey: []
      parameters:
        - name: expires_in_seconds
          in: query
          description: |
            Token redemption window, in seconds. The token must be used to open a WebSocket connection within this window. If the window has elapsed when the WebSocket is opened, the server returns a `session.error` with code `unauthorized` on the first frame instead of `session.ready`. This does NOT cap the duration of the resulting voice agent session — once the WebSocket is open the session runs for up to `max_session_duration_seconds`.
          required: true
          schema:
            type: integer
            minimum: 1
            maximum: 600
            example: 300
        - name: max_session_duration_seconds
          in: query
          description: |
            Maximum duration of the Voice Agent session started with this token, in seconds. There is no "closing soon" warning event before the session ends, so if you need to finalize gracefully (e.g. play a wrap-up message, save state), run a client-side timer using the value you passed here.
          required: false
          schema:
            type: integer
            minimum: 60
            maximum: 10800
            example: 8640
            default: 10800
      responses:
        "200":
          description: Successfully generated a temporary token.
          content:
            application/json:
              schema:
                type: object
                required:
                  - token
                  - expires_in_seconds
                properties:
                  token:
                    type: string
                    description: The temporary authentication token. Pass as the `token` query parameter when opening the WebSocket.
                    example: "your-temporary-token"
                  expires_in_seconds:
                    type: integer
                    description: The token redemption window in seconds — the time the client has to use this token to open a WebSocket before it expires unused. This is not the session duration; see `max_session_duration_seconds` for that.
                    example: 300
        "400":
          description: Bad request — invalid parameters.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        "401":
          description: Unauthorized — invalid or missing API key.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        "429":
          description: Too Many Requests — rate limit exceeded.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        "500":
          description: Internal Server Error.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"

components:
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: Authorization
      description: "Your AssemblyAI API key passed as a Bearer token (e.g. `Authorization: Bearer YOUR_API_KEY`)."
  schemas:
    Error:
      type: object
      required:
        - error
      properties:
        error:
          type: string
          description: Error message describing what went wrong.
        code:
          type: string
          description: Error code for programmatic handling.
        details:
          type: object
          description: Additional error details if available.
```

## docs/api-reference/specs/agents.yaml

URL: https://www.assemblyai.com/docs/api-reference/specs/agents.yaml

```yaml
openapi: 3.0.3

info:
  title: AssemblyAI Voice Agents API
  description: |
    REST API for creating and managing reusable voice agents. An agent stores its
    system prompt, greeting, voice, and tools server-side, so you can deploy the same
    agent across the WebSocket API, a browser, or Twilio by referencing its `id`. See
    [Create an agent](/voice-agents/voice-agent-api/create-agent)
    for a guided walkthrough.
  version: 1.0.0
  termsOfService: https://www.assemblyai.com/legal/terms-of-service
  contact:
    name: API Support
    email: support@assemblyai.com
    url: https://www.assemblyai.com/docs/

servers:
  - url: https://agents.assemblyai.com
    description: Production server
    x-fern-server-name: Default

paths:
  /v1/agents:
    post:
      summary: Create an agent
      description: |
        Create a reusable voice agent. Returns the full agent record, including a
        generated `id` you use to connect over the WebSocket (`session.update` with
        `agent_id`).
      operationId: createAgent
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: createAgent
      security:
        - apiKey: []
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/AgentCreateRequest"
      responses:
        "201":
          description: Agent created.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Agent"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "422":
          $ref: "#/components/responses/UnprocessableEntity"
    get:
      summary: List agents
      description: List the agents for the authenticated account. Returns lightweight records (no tools or prompt), newest first.
      operationId: listAgents
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: listAgents
      security:
        - apiKey: []
      responses:
        "200":
          description: A list of agents.
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/AgentListItem"
        "401":
          $ref: "#/components/responses/Unauthorized"

  /v1/agents/{agent_id}:
    parameters:
      - name: agent_id
        in: path
        required: true
        description: The agent's unique ID.
        schema:
          type: string
          example: "7ad24396-b822-4dca-871a-be9cc4781cf9"
    get:
      summary: Retrieve an agent
      description: Retrieve a single agent by ID. Tool header **values** are masked as `"***"` in the response.
      operationId: getAgent
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: getAgent
      security:
        - apiKey: []
      responses:
        "200":
          description: The agent record.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Agent"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
    put:
      summary: Update an agent
      description: |
        Update an agent. Every field is optional. Send only the fields you want to
        change. Returns the updated record.
      operationId: updateAgent
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: updateAgent
      security:
        - apiKey: []
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/AgentUpdateRequest"
      responses:
        "200":
          description: The updated agent record.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Agent"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
        "422":
          $ref: "#/components/responses/UnprocessableEntity"
    delete:
      summary: Delete an agent
      description: Delete an agent. Returns `204` with no body.
      operationId: deleteAgent
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: deleteAgent
      security:
        - apiKey: []
      responses:
        "204":
          description: Agent deleted.
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"

  /v1/sessions:
    get:
      summary: List sessions
      description: |
        List sessions for the authenticated account, newest first. Page forward by
        passing the previous response's `response_metadata.next_cursor` back as `cursor`.
      operationId: listSessions
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: listSessions
      security:
        - apiKey: []
      parameters:
        - name: limit
          in: query
          description: Maximum sessions per page.
          schema:
            type: integer
            default: 50
            minimum: 1
            maximum: 200
        - name: cursor
          in: query
          description: Pagination cursor from the previous response's `response_metadata.next_cursor`.
          schema:
            type: string
        - name: status
          in: query
          description: Return only sessions with this status.
          schema:
            type: string
        - name: agent_id
          in: query
          description: Return only sessions for this agent.
          schema:
            type: string
      responses:
        "200":
          description: A page of sessions.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/SessionListResponse"
        "401":
          $ref: "#/components/responses/Unauthorized"

  /v1/sessions/{session_id}:
    parameters:
      - name: session_id
        in: path
        required: true
        description: The session's unique ID.
        schema:
          type: string
          example: "sess_9a648a2ab75747a9a597ba046ced3e13"
    get:
      summary: Retrieve a session
      description: |
        Retrieve a single session by ID, including its artifacts (recording, conversation
        timeline, and metadata) as short-lived pre-signed download URLs.
      operationId: getSession
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: getSession
      security:
        - apiKey: []
      responses:
        "200":
          description: The session record with artifacts.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/SessionResponse"
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"
    delete:
      summary: Delete a session
      description: Soft-delete a session. Returns `204` with no body.
      operationId: deleteSession
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: deleteSession
      security:
        - apiKey: []
      responses:
        "204":
          description: Session deleted.
        "401":
          $ref: "#/components/responses/Unauthorized"
        "404":
          $ref: "#/components/responses/NotFound"

components:
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: Authorization
      description: "Your AssemblyAI API key in the `Authorization` header. The raw key works directly; a `Bearer ` prefix is also accepted."

  responses:
    BadRequest:
      description: Bad request. A field failed validation (e.g. invalid voice, non-https tool URL, timeout out of range).
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
    Unauthorized:
      description: Unauthorized. Invalid or missing API key, or the key is not entitled for Voice Agents.
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
    NotFound:
      description: The agent was not found.
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
    UnprocessableEntity:
      description: Unprocessable entity. Malformed JSON or a field of the wrong type.
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"

  schemas:
    AgentCreateRequest:
      type: object
      required:
        - name
        - system_prompt
        - voice
      properties:
        name:
          type: string
          description: Display name for the agent.
          example: "Support Assistant"
        system_prompt:
          type: string
          description: The agent's instructions. Write voice-first.
          example: "You are a friendly support agent. Keep responses under two sentences."
        voice:
          $ref: "#/components/schemas/VoiceConfig"
        greeting:
          type: string
          nullable: true
          description: Spoken on connect; sent straight to TTS. Omit to listen first.
          example: "Hi, how can I help?"
        input:
          $ref: "#/components/schemas/AudioInput"
        output:
          $ref: "#/components/schemas/AudioOutput"
        tools:
          type: array
          description: Tools the agent can call.
          items:
            $ref: "#/components/schemas/ToolDefinition"
        llm:
          type: array
          maxItems: 1
          description: |
            Connect your own OpenAI-compatible LLM instead of AssemblyAI's
            managed model. Provide at most one config; omit to use the managed
            LLM. Multiple configs (fallbacks) are not yet supported.
          items:
            $ref: "#/components/schemas/LlmConfig"

    AgentUpdateRequest:
      type: object
      description: Same shape as the create request, but every field is optional. Send only what changes.
      properties:
        name:
          type: string
        system_prompt:
          type: string
        voice:
          $ref: "#/components/schemas/VoiceConfig"
        greeting:
          type: string
          nullable: true
        input:
          $ref: "#/components/schemas/AudioInput"
        output:
          $ref: "#/components/schemas/AudioOutput"
        tools:
          type: array
          items:
            $ref: "#/components/schemas/ToolDefinition"
        llm:
          type: array
          maxItems: 1
          items:
            $ref: "#/components/schemas/LlmConfig"

    Agent:
      type: object
      description: A stored voice agent.
      properties:
        id:
          type: string
          example: "7ad24396-b822-4dca-871a-be9cc4781cf9"
        name:
          type: string
          example: "Support Assistant"
        system_prompt:
          type: string
        greeting:
          type: string
          nullable: true
        voice:
          $ref: "#/components/schemas/VoiceConfig"
        input:
          $ref: "#/components/schemas/AudioInput"
        output:
          $ref: "#/components/schemas/AudioOutput"
        tools:
          type: array
          items:
            $ref: "#/components/schemas/ToolResponse"
        llm:
          type: array
          description: Connected LLM, if any. The api_key is write-only and never returned.
          items:
            $ref: "#/components/schemas/LlmConfigResponse"
        created_at:
          type: string
          format: date-time
        updated_at:
          type: string
          format: date-time

    AgentListItem:
      type: object
      description: Lightweight agent record returned by the list endpoint.
      properties:
        id:
          type: string
        name:
          type: string
        deleted_at:
          type: string
          format: date-time
          nullable: true
        created_at:
          type: string
          format: date-time
        updated_at:
          type: string
          format: date-time

    VoiceConfig:
      type: object
      required:
        - voice_id
      properties:
        voice_id:
          type: string
          description: A voice from the voice catalog. An invalid id returns 400 with the list of valid voices.
          example: "alba"

    LlmConfig:
      type: object
      required:
        - base_url
        - model
        - api_key
      description: |
        An OpenAI-compatible chat-completions endpoint used as the agent's
        conversational LLM in place of AssemblyAI's managed model.
      properties:
        base_url:
          type: string
          description: HTTPS base URL of the OpenAI-compatible endpoint. Must be https and resolve to a public host.
          example: "https://api.openai.com/v1"
        model:
          type: string
          description: Model name sent in the chat-completions request body.
          example: "gpt-4o-mini"
        api_key:
          type: string
          description: API key for the endpoint. Write-only — encrypted at rest and never returned in any response.

    LlmConfigResponse:
      type: object
      description: A connected LLM as returned in an agent record. The api_key is never included.
      properties:
        base_url:
          type: string
          example: "https://api.openai.com/v1"
        model:
          type: string
          example: "gpt-4o-mini"

    AudioInput:
      type: object
      description: Audio input configuration. Defaults to PCM at 24 kHz if omitted.
      properties:
        type:
          type: string
          default: "audio"
        format:
          $ref: "#/components/schemas/AudioFormat"
        turn_detection:
          type: object
          nullable: true
          description: VAD tuning, or null for adaptive defaults.
        keyterms:
          type: array
          nullable: true
          description: Up to 100 transcription-bias strings, or null.
          items:
            type: string

    AudioOutput:
      type: object
      description: Audio output configuration. Defaults to PCM at 24 kHz with the agent's voice.
      properties:
        type:
          type: string
          default: "audio"
        voice:
          type: string
          example: "alba"
        format:
          $ref: "#/components/schemas/AudioFormat"
        volume:
          type: number
          nullable: true
          description: Playback volume 0–100, or null for native level.

    AudioFormat:
      type: object
      properties:
        encoding:
          type: string
          enum: [audio/pcm, audio/pcmu, audio/pcma]
          default: audio/pcm
        sample_rate:
          type: integer
          example: 24000

    ToolDefinition:
      type: object
      required:
        - name
        - description
      properties:
        name:
          type: string
          description: Tool name the model calls.
          example: "get_weather"
        description:
          type: string
          description: When to call the tool. The model's main signal.
          example: "Get current weather for a location. Use whenever the user asks about weather."
        parameters:
          $ref: "#/components/schemas/ToolParameters"
        http:
          $ref: "#/components/schemas/HttpToolConfig"
        timeout_seconds:
          type: integer
          minimum: 1
          maximum: 300
          default: 120
        execution_mode:
          type: string
          enum: [interactive, hold]
          default: interactive
          description: How the agent behaves while the tool runs.

    ToolParameters:
      type: object
      description: |
        A JSON Schema describing the tool's arguments. The same form for HTTP and
        client-side tools. Each property accepts standard JSON-Schema keywords; beyond
        `type` and `description`, use `enum`, `examples`, `pattern`, and `format` to
        sharpen tool-calling and turn-detection accuracy.
      properties:
        type:
          type: string
          enum: [object]
          default: object
        properties:
          type: object
          description: Map of argument name to its JSON-Schema definition.
          additionalProperties: true
        required:
          type: array
          description: Names of the required arguments.
          items:
            type: string
      example:
        type: object
        properties:
          order_id:
            type: string
            description: The customer's order ID.
            examples: ["AB-12345"]
            pattern: "[A-Z]{2}-\\d{5}"
        required: ["order_id"]

    HttpToolConfig:
      type: object
      required:
        - url
      description: Makes the tool a server-side HTTP tool. AssemblyAI calls this endpoint when the model invokes the tool.
      properties:
        url:
          type: string
          description: Must be https and resolve to a public host. Max 2048 chars.
          example: "https://api.example.com/weather"
        http_method:
          type: string
          enum: [GET, POST, PUT, PATCH, DELETE]
          default: POST
          description: GET/DELETE send arguments as query params; POST/PUT/PATCH send them as a JSON body.
        headers:
          type: object
          nullable: true
          description: Sent on every call. Values are encrypted at rest and write-only (masked as "***" in responses).
          additionalProperties:
            type: string

    ToolResponse:
      type: object
      description: A tool as returned in an agent record. Header values are masked.
      properties:
        id:
          type: string
        name:
          type: string
        description:
          type: string
        http:
          $ref: "#/components/schemas/HttpToolConfigResponse"
        parameters:
          $ref: "#/components/schemas/ToolParameters"
        timeout_seconds:
          type: integer
        execution_mode:
          type: string
          enum: [interactive, hold]

    HttpToolConfigResponse:
      type: object
      properties:
        url:
          type: string
        http_method:
          type: string
          enum: [GET, POST, PUT, PATCH, DELETE]
        headers:
          type: object
          description: Header names with values masked as "***".
          additionalProperties:
            type: string
          example: { "Authorization": "***" }

    Error:
      type: object
      properties:
        detail:
          type: string
          description: Error message describing what went wrong.
          example: "Unauthorized"

    SessionListResponse:
      type: object
      required: [sessions]
      properties:
        sessions:
          type: array
          description: Page of sessions, newest first.
          items:
            $ref: "#/components/schemas/SessionListItem"
        has_more:
          type: boolean
          description: True when more results may be available; fetch the next page with `response_metadata.next_cursor`.
          example: true
        response_metadata:
          $ref: "#/components/schemas/ResponseMetadata"

    ResponseMetadata:
      type: object
      properties:
        next_cursor:
          type: string
          description: Cursor for the next page; pass it back as the `cursor` query parameter. Empty string when there are no further results.
          example: "c2Vzc185YTY0OGEyYWI3NTc0N2E5"

    SessionListItem:
      type: object
      required: [id, status]
      properties:
        id:
          type: string
          description: Unique ID of the session.
          example: "sess_9a648a2ab75747a9a597ba046ced3e13"
        agent_id:
          type: string
          nullable: true
          description: Agent the session ran with. Null when no stored agent was used.
          example: "7ad24396-b822-4dca-871a-be9cc4781cf9"
        status:
          type: string
          description: Lifecycle status of the session, e.g. `completed`.
          example: "completed"
        public_close_reason:
          type: string
          nullable: true
          description: Why the session ended, when known.
          example: "client_end"
        duration_seconds:
          type: number
          nullable: true
          description: Session length in seconds. Null while the session is active.
          example: 42.6
        created_at:
          type: string
          format: date-time
          nullable: true
          description: When the session was created.
        ended_at:
          type: string
          format: date-time
          nullable: true
          description: When the session ended. Null while the session is active.

    SessionResponse:
      type: object
      required: [id, status]
      properties:
        id:
          type: string
          description: Unique ID of the session.
          example: "sess_9a648a2ab75747a9a597ba046ced3e13"
        agent_id:
          type: string
          nullable: true
          description: Agent the session ran with. Null when no stored agent was used.
        status:
          type: string
          description: Lifecycle status of the session, e.g. `completed`.
          example: "completed"
        public_close_reason:
          type: string
          nullable: true
          description: Why the session ended, when known.
          example: "client_end"
        duration_seconds:
          type: number
          nullable: true
          description: Session length in seconds. Null while the session is active.
          example: 42.6
        config:
          type: object
          nullable: true
          description: Session configuration captured when the session started.
          additionalProperties: true
        created_at:
          type: string
          format: date-time
          nullable: true
          description: When the session was created.
        ended_at:
          type: string
          format: date-time
          nullable: true
          description: When the session ended. Null while the session is active.
        artifacts:
          type: array
          description: Recording, timeline, and metadata produced by the session. Empty until the session completes.
          items:
            $ref: "#/components/schemas/SessionArtifact"

    SessionArtifact:
      type: object
      required: [type, url, content_type]
      properties:
        type:
          type: string
          description: Artifact kind.
          enum: [audio, timeline, metadata]
        url:
          type: string
          description: Pre-signed download URL. Expires after a short TTL; re-fetch the session for a fresh link.
        content_type:
          type: string
          description: MIME type of the artifact.
          example: "audio/ogg"
```

## docs/api-reference/specs/webhooks.yaml

URL: https://www.assemblyai.com/docs/api-reference/specs/webhooks.yaml

```yaml
openapi: 3.0.3

info:
  title: AssemblyAI Voice Agent Webhooks API
  description: |
    Subscribe to session and call lifecycle events for your voice agents. AssemblyAI
    POSTs a signed JSON payload to your URL when each event fires. See
    [Webhooks](/voice-agents/voice-agent-api/webhooks).
  version: 1.0.0
  termsOfService: https://www.assemblyai.com/legal/terms-of-service
  contact:
    name: API Support
    email: support@assemblyai.com
    url: https://www.assemblyai.com/docs/

servers:
  - url: https://agents.assemblyai.com
    description: Production server
    x-fern-server-name: Default

paths:
  /v1/webhook-subscriptions:
    post:
      summary: Create a webhook subscription
      description: Subscribe a URL to one or more events. Scope to a single agent with agent_id, or omit it for account-wide events.
      operationId: createWebhookSubscription
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: createWebhookSubscription
      security:
        - apiKey: []
      requestBody:
        required: true
        content:
          application/json:
            schema: { $ref: "#/components/schemas/CreateWebhookSubscriptionRequest" }
      responses:
        "201":
          description: The created subscription.
          content:
            application/json:
              schema: { $ref: "#/components/schemas/WebhookSubscription" }
        "401": { $ref: "#/components/responses/Unauthorized" }
        "422": { description: Invalid request., content: { application/json: { schema: { $ref: "#/components/schemas/Error" } } } }
    get:
      summary: List webhook subscriptions
      operationId: listWebhookSubscriptions
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: listWebhookSubscriptions
      security:
        - apiKey: []
      parameters:
        - { name: limit, in: query, required: false, schema: { type: integer, minimum: 1, maximum: 100, default: 20 } }
        - { name: cursor, in: query, required: false, schema: { type: string } }
      responses:
        "200":
          description: A page of subscriptions.
          content:
            application/json:
              schema: { $ref: "#/components/schemas/WebhookSubscriptionList" }
        "401": { $ref: "#/components/responses/Unauthorized" }

  /v1/webhook-subscriptions/{subscription_id}:
    parameters:
      - name: subscription_id
        in: path
        required: true
        schema: { type: string, example: "0d75ebd7-937c-495c-b638-ba582ec05b39" }
    get:
      summary: Retrieve a webhook subscription
      operationId: getWebhookSubscription
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: getWebhookSubscription
      security:
        - apiKey: []
      responses:
        "200":
          description: The subscription.
          content:
            application/json:
              schema: { $ref: "#/components/schemas/WebhookSubscription" }
        "401": { $ref: "#/components/responses/Unauthorized" }
        "404": { $ref: "#/components/responses/NotFound" }
    patch:
      summary: Update a webhook subscription
      description: Update any of url, events, secret, or enabled. Sending a new secret rotates it and increments secret_version.
      operationId: updateWebhookSubscription
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: updateWebhookSubscription
      security:
        - apiKey: []
      requestBody:
        required: true
        content:
          application/json:
            schema: { $ref: "#/components/schemas/UpdateWebhookSubscriptionRequest" }
      responses:
        "200":
          description: The updated subscription.
          content:
            application/json:
              schema: { $ref: "#/components/schemas/WebhookSubscription" }
        "401": { $ref: "#/components/responses/Unauthorized" }
        "404": { $ref: "#/components/responses/NotFound" }
    delete:
      summary: Delete a webhook subscription
      operationId: deleteWebhookSubscription
      x-fern-sdk-group-name: Voice Agent API
      x-fern-sdk-method-name: deleteWebhookSubscription
      security:
        - apiKey: []
      responses:
        "204": { description: The subscription was deleted. }
        "401": { $ref: "#/components/responses/Unauthorized" }
        "404": { $ref: "#/components/responses/NotFound" }

components:
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: Authorization
      description: "Your AssemblyAI API key in the Authorization header (raw key; a Bearer prefix is also accepted)."
  responses:
    Unauthorized:
      description: Unauthorized. Invalid or missing API key.
      content: { application/json: { schema: { $ref: "#/components/schemas/Error" } } }
    NotFound:
      description: The subscription was not found.
      content: { application/json: { schema: { $ref: "#/components/schemas/Error" } } }
  schemas:
    WebhookEvent:
      type: string
      enum: [session.started, session.completed, call.connected, call.ended, call.failed]
    CreateWebhookSubscriptionRequest:
      type: object
      required: [url, events, secret]
      properties:
        url:      { type: string, description: "Delivery URL. Must be https and a public host.", example: "https://your-app.com/webhooks/assemblyai" }
        events:
          type: array
          minItems: 1
          items: { $ref: "#/components/schemas/WebhookEvent" }
        secret:   { type: string, description: "Signing secret, 32–256 printable ASCII chars (no whitespace). Write-only; never returned.", minLength: 32, maxLength: 256 }
        agent_id: { type: string, description: "Scope to one agent. Omit for account-wide events.", nullable: true }
        enabled:  { type: boolean, default: true }
    UpdateWebhookSubscriptionRequest:
      type: object
      description: All fields optional. A new secret rotates it and increments secret_version.
      properties:
        url:     { type: string }
        events:
          type: array
          items: { $ref: "#/components/schemas/WebhookEvent" }
        secret:  { type: string, minLength: 32, maxLength: 256 }
        enabled: { type: boolean }
    WebhookSubscription:
      type: object
      description: A subscription. The signing secret is never returned, only its version.
      properties:
        id:             { type: string }
        agent_id:       { type: string, nullable: true }
        url:            { type: string }
        events:
          type: array
          items: { $ref: "#/components/schemas/WebhookEvent" }
        enabled:        { type: boolean }
        secret_version: { type: integer, example: 1 }
        created_at:     { type: string, format: date-time }
        updated_at:     { type: string, format: date-time }
    WebhookSubscriptionList:
      type: object
      properties:
        subscriptions:
          type: array
          items: { $ref: "#/components/schemas/WebhookSubscription" }
        has_more: { type: boolean }
        response_metadata:
          type: object
          properties:
            next_cursor: { type: string }
    Error:
      type: object
      properties:
        detail: { type: string, example: "Unauthorized" }
```

## docs/api-reference/specs/streaming.yaml

URL: https://www.assemblyai.com/docs/api-reference/specs/streaming.yaml

```yaml
asyncapi: 3.0.0

info:
  title: AssemblyAI Streaming WebSocket API
  version: 1.0.0
  description: |
    Real-time speech-to-text transcription API over WebSocket.
    Clients send audio data and configuration messages and receive transcription events.

servers:
  production:
    host: streaming.assemblyai.com
    protocol: wss
    description: Global, latency-optimized endpoint. Automatically routes each connection to the nearest region.
  usDataResidency:
    host: streaming.us.assemblyai.com
    protocol: wss
    description: Dedicated US endpoint for data residency requirements. Audio and transcription data stays in the US.
  euDataResidency:
    host: streaming.eu.assemblyai.com
    protocol: wss
    description: Dedicated EU endpoint for data residency requirements. Audio and transcription data stays in the EU.

channels:
  streaming:
    address: /v3/ws
    description: |
      Stream audio and receive real-time transcription results.

      ### Endpoints

      | WebSocket URL | Description |
      | --- | --- |
      | `wss://streaming.assemblyai.com/v3/ws` | **Global (default).** Latency-optimized — automatically routes each connection to the nearest region. |
      | `wss://streaming.us.assemblyai.com/v3/ws` | **US data residency.** Audio and transcription data never leaves the US. |
      | `wss://streaming.eu.assemblyai.com/v3/ws` | **EU data residency.** Audio and transcription data never leaves the EU. |

      Use a data residency endpoint for compliance requirements; see [Cloud endpoints & data residency](https://www.assemblyai.com/docs/streaming/endpoints-and-data-zones).

      ### Headers

      | Header | Description |
      | --- | --- |
      | `Authorization` | **Required.** Your AssemblyAI API key (no `Bearer` prefix). Required on every connection unless you authenticate with the `token` query parameter instead. |
      | `AssemblyAI-Version` | **Optional.** API version pin. Defaults to the latest version. |

      Browsers cannot set headers on a WebSocket connection. In that case, [generate a temporary token](https://www.assemblyai.com/docs/streaming/authenticate-with-a-temporary-token) server-side and pass it via the `token` query parameter instead. Never expose your permanent API key in a URL or in client-side code.

      ### Query parameters

      All query parameters below are optional — a connection with only authentication set uses the default speech model (`universal-3-5-pro`) at 16 kHz `pcm_s16le`. Parameters apply to every model unless their description says otherwise.
    servers:
      - $ref: "#/servers/production"
      - $ref: "#/servers/usDataResidency"
      - $ref: "#/servers/euDataResidency"
    parameters:
      token:
        description: Authenticate with a [temporary token](/api-reference/streaming-api/generate-streaming-token) via query parameter. Use this where request headers cannot be set — for example, browser WebSocket connections — and prefer the `Authorization` header everywhere else. Generate the temporary token server-side; never expose your permanent API key in a URL.
        location: $message.payload#/token

      speech_model:
        description: The speech model to use for the session. See [Select the speech model](https://www.assemblyai.com/docs/streaming/select-the-speech-model) for the differences between models.
        location: $message.payload#/speech_model
        enum: ["universal-3-5-pro", "universal-streaming-english", "universal-streaming-multilingual"]

      language_codes:
        description: >-
          Steers transcription toward a set of languages by biasing output toward them on a per-token basis while still allowing native code-switching among them. Pass a list of language codes for the languages you expect (for example, `["en", "es"]`), or a single-element list (for example, `["es"]`) for a monolingual session. When unset, no steering is applied and the model code-switches natively across all of its supported languages. This is distinct from `language_detection`, which only controls whether the detected language is reported. Accepted codes: `en`, `es`, `fr`, `de`, `it`, `pt`, `tr`, `nl`, `sv`, `no`, `da`, `fi`, `hi`, `vi`, `ar`, `he`, `ja`, `zh`. Universal-3.5 Pro Streaming only.
        location: $message.payload#/language_codes
        examples:
          - '["en", "es"]'

      language_detection:
        description: Whether to return `language_code` and `language_confidence` in turn messages. Available on Universal-3.5 Pro Streaming (which natively code-switches across all of its supported languages by default) and on Universal Streaming Multilingual only.
        location: $message.payload#/language_detection
        enum: ["true", "false"]
        default: "false"

      domain:
        description: >-
          Enable domain-specific transcription models to improve accuracy for specialized terminology.
          Set to `"medical-v1"` to enable [Medical Mode](https://www.assemblyai.com/docs/streaming/medical-mode) for improved accuracy of medical terms such as medications, procedures, conditions, and dosages.
          Supported languages: English (`en`), Spanish (`es`), German (`de`), French (`fr`). If used with an unsupported language, the parameter is ignored and a warning is returned.
        location: $message.payload#/domain
        enum: ["medical-v1"]

      mode:
        description: Latency and accuracy preset that controls the model's turn-detection and partial-emission defaults. `max_accuracy` favors transcription quality, `min_latency` favors speed, and `balanced` trades off between the two. When omitted, the server applies its own preset, which determines the defaults for the mode-dependent fields `min_turn_silence` and `interruption_delay`. Universal-3.5 Pro Streaming only.
        location: $message.payload#/mode
        enum: ["max_accuracy", "min_latency", "balanced"]

      session_heartbeat:
        description: When enabled, the server sends a `Heartbeat` message every 5 seconds of wall-clock time for the life of the session, during both speech and silence. Use it to confirm the session is alive and to compare how fast the client is delivering audio against real time. Can also be toggled mid-session via `UpdateConfiguration`.
        location: $message.payload#/session_heartbeat
        enum: ["true", "false"]
        default: "false"

      encoding:
        description: >-
          Encoding of the audio stream. `pcm_s16le` and `pcm_mulaw` are raw
          PCM formats. `opus` is raw Opus packets, where each binary WebSocket
          message must contain exactly one Opus packet. `ogg_opus` is an
          Ogg-encapsulated Opus byte stream (the format produced by ffmpeg,
          gstreamer, opusenc, and browser MediaRecorder); binary WebSocket
          messages can be arbitrary chunks of the stream. `aac` is an AAC
          stream in ADTS framing; raw AAC packets and MP4/M4A containers are
          not supported. Binary WebSocket messages can be arbitrary chunks of
          the stream, and the server buffers partial frames. For `aac` and both
          Opus encodings, `sample_rate` is ignored because each stream
          describes its own sample rate. Audio is decoded to PCM server side
          and transcription behavior is otherwise identical.
        location: $message.payload#/encoding
        enum: ["pcm_s16le", "pcm_mulaw", "opus", "ogg_opus", "aac"]
        default: "pcm_s16le"

      sample_rate:
        description: Sample rate of the audio stream in Hz. Accepts any integer from 8000 to 96000. Ignored for Opus and AAC encodings (`opus`, `ogg_opus`, `aac`), which describe their own sample rate.
        location: $message.payload#/sample_rate
        default: "16000"
        examples:
          - "16000"
          - "48000"

      min_turn_silence:
        description: Silence duration in milliseconds before a speculative end-of-turn check. On Universal-3.5 Pro Streaming the check is punctuation-based (if terminal punctuation is found the turn ends, otherwise a partial is emitted and the turn continues) and the default is mode-dependent, set by the `mode` preset. On Universal Streaming the check is confidence-based and the default is 400 ms. Clamped to the range 50 to 10000 ms. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details.
        location: $message.payload#/min_turn_silence
        examples:
          - "100"
          - "200"

      max_turn_silence:
        description: Maximum silence in milliseconds before the turn is forced to end, regardless of punctuation. Defaults are 1536 ms on Universal-3.5 Pro Streaming (768 ms when `speaker_labels` is enabled) and 1280 ms on Universal Streaming. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details.
        location: $message.payload#/max_turn_silence
        examples:
          - "700"
          - "1000"

      end_of_turn_confidence_threshold:
        description: >
          The confidence threshold (0.0 to 1.0) to use when determining if the end of a turn has been reached. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details. Universal Streaming (English and Multilingual) only.
        location: $message.payload#/end_of_turn_confidence_threshold
        default: "0.4"
        examples:
          - "0.4"
          - "0.7"

      vad_threshold:
        description: The confidence threshold (0.0 to 1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent. Increase for noisy environments to reduce false speech detection. Defaults are 0.2 on Universal-3.5 Pro Streaming (0.5 when `speaker_labels` is enabled) and 0.4 on Universal Streaming.
        location: $message.payload#/vad_threshold

      interruption_delay:
        description: How soon the first partial is emitted in milliseconds. Accepts a value from 0 to 1000. Useful for tuning voice agent barge-in responsiveness or allowing earlier partials for early LLM inference. Larger values are more confident on interruptions, smaller values result in faster time to first partial. The server adds a fixed 256 ms on top of this value, so `0` yields an effective delay of 256 ms and `500` yields 756 ms. The default is mode-dependent and set by the server based on the `mode` preset. Universal-3.5 Pro Streaming only.
        location: $message.payload#/interruption_delay
        examples:
          - "0"
          - "500"

      continuous_partials:
        description: Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled (default), additional partials covering the full turn transcript are emitted approximately every 3 seconds while speech continues. When disabled, only one early partial is emitted near turn start. The first partial (timed by `interruption_delay`) is unaffected. When `speaker_labels` is enabled the server disables continuous partials by default. Universal-3.5 Pro Streaming only.
        location: $message.payload#/continuous_partials
        default: "true"

      include_partial_turns:
        description: Whether to emit partial transcripts during the turn. When enabled (default), partial transcripts are forwarded as speech is still in progress alongside final turns. When disabled, only final turns (with end_of_turn true) are sent. Defaults to false when redact_pii is enabled, to prevent unredacted partial transcripts from reaching the client; set explicitly to true to override.
        location: $message.payload#/include_partial_turns
        default: "true"

      format_turns:
        description: Whether to return formatted final transcripts. Universal Streaming (English and Multilingual) only.
        location: $message.payload#/format_turns
        enum: ["true", "false"]
        default: "false"

      prompt:
        description: A contextual prompt describing what the audio is about, such as its domain, scenario, or full conversation details. The model uses this context to better recognize the vocabulary it makes likely, for example priming medical terminology for a cardiology call. It carries context about your audio, not transcription instructions, so formatting or behavioral commands (such as punctuation rules) are not supported. Maximum 1750 characters. Universal-3.5 Pro Streaming only. See [Prompting and keyterms](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms) for details.
        location: $message.payload#/prompt

      keyterms_prompt:
        description: A list of words and phrases to improve recognition accuracy for. Maximum 100 terms. See [Keyterms Prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) for more details.
        location: $message.payload#/keyterms_prompt

      agent_context:
        description: Your voice agent's spoken text (TTS reply). The model uses this as context for the next user turn, which improves accuracy on short or ambiguous replies and on spelled-out entities like emails or IDs. Set at connection time to seed the model with your agent's opening greeting, and/or update mid-stream via `UpdateConfiguration` after each agent reply. Each `UpdateConfiguration` replaces the previously set value. Maximum 1750 characters per value. Universal-3.5 Pro Streaming only. See [Context Carryover](https://www.assemblyai.com/docs/streaming/universal-3-5-pro/context-carryover).
        location: $message.payload#/agent_context
        examples:
          - "Welcome to the Krusty Krab, home of the Krabby Patty, may I take your order?"

      previous_context_n_turns:
        description: Advanced. Maximum number of prior conversation entries (user transcripts and any `agent_context` values) carried forward as context for each transcription. Range 0 to 100. Set to `0` to disable automatic context carryover entirely. When unset, the server default applies (currently 5 turns). Most integrations should leave this unset. Universal-3.5 Pro Streaming only.
        location: $message.payload#/previous_context_n_turns

      speaker_labels:
        description: Whether to enable [Streaming Speaker Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels). When enabled, each Turn event will include a `speaker_label` field and each final word in the `words` array will include a `speaker` field for word-level speaker attribution.
        location: $message.payload#/speaker_labels
        enum: ["true", "false"]
        default: "false"

      max_speakers:
        description: A hard cap on the number of speaker labels in the audio stream (integer, 1-10). This is a strict limit, not a hint — once it is reached, any additional speakers are merged into the closest existing label rather than given a new one. Give the model a little headroom above the number of speakers you expect; setting it too high can cause over-splitting and return more speakers than are actually present. Only used when `speaker_labels` is enabled. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details.
        location: $message.payload#/max_speakers
        examples:
          - "2"
          - "4"

      voice_focus:
        description: Enable [Voice Focus](https://www.assemblyai.com/docs/streaming/voice-focus) to isolate the primary voice and suppress background noise before transcription. Set to `near-field` for close-talking microphones (for example, headsets or phones) or `far-field` for distant microphones (for example, conference rooms). When unset, Voice Focus is off.
        location: $message.payload#/voice_focus
        enum: ["near-field", "far-field"]

      voice_focus_threshold:
        description: Controls how aggressively Voice Focus suppresses background audio, from `0.0` (least) to `1.0` (most). Higher values are more aggressive. Requires `voice_focus` to be set, otherwise a validation error is returned.
        location: $message.payload#/voice_focus_threshold
        default: "0.7"

      redact_pii:
        description: Redact PII from the transcribed text using the Redact PII model, can be true or false. Only applies to final turns. See [PII Redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details.
        location: $message.payload#/redact_pii
        enum: ["true", "false"]
        default: "false"

      redact_pii_policies:
        description: The list of PII Redaction policies to enable. Requires `redact_pii` to be `true`. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details.
        location: $message.payload#/redact_pii_policies

      redact_pii_sub:
        description: The replacement logic for detected PII, can be `entity_name` or `hash`. Requires `redact_pii` to be `true`. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details.
        location: $message.payload#/redact_pii_sub
        enum: ["entity_name", "hash"]
        default: "hash"

      filter_profanity:
        description: Filter profanity from the transcribed text, can be true or false. See [Profanity Filtering](https://www.assemblyai.com/docs/streaming/filter-profanity-from-transcripts) for more details.
        location: $message.payload#/filter_profanity
        enum: ["true", "false"]
        default: "false"

      llm_gateway:
        description: >-
          JSON-stringified LLM Gateway configuration that processes each finalized turn.
          Follows the same interface as the [Chat Completions](/llm-gateway/chat-completions) endpoint
          and accepts `model`, `messages`, `tools`, `tool_choice`, `post_processing_steps`, and `max_tokens`.
          See [Apply LLM Gateway to Streaming](https://www.assemblyai.com/docs/guides/real_time_llm_gateway) for the full schema and examples.
        location: $message.payload#/llm_gateway

      inactivity_timeout:
        description: Optional time in seconds of inactivity before session is terminated (integer, minimum 5, maximum 3600). If not set, no inactivity timeout is applied.
        location: $message.payload#/inactivity_timeout
        examples:
          - "5"
          - "3600"

    messages:
      updateConfiguration:
        $ref: "#/components/messages/UpdateConfiguration"
      forceEndpoint:
        $ref: "#/components/messages/ForceEndpoint"
      sessionTermination:
        $ref: "#/components/messages/SessionTermination"
      audioChunk:
        $ref: "#/components/messages/AudioChunk"
      keepAlive:
        $ref: "#/components/messages/KeepAlive"

      sessionBegins:
        $ref: "#/components/messages/SessionBegins"
      speechStarted:
        $ref: "#/components/messages/SpeechStarted"
      turn:
        $ref: "#/components/messages/Turn"
      speakerRevision:
        $ref: "#/components/messages/SpeakerRevision"
      termination:
        $ref: "#/components/messages/Termination"
      heartbeat:
        $ref: "#/components/messages/Heartbeat"
      llmGatewayResponse:
        $ref: "#/components/messages/LLMGatewayResponse"
operations:
  audio:
    action: receive
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/audioChunk"
    description: Send audio data chunks for transcription. The payload must be of type bytes and contain audio data between 50ms and 1000ms in length. When streaming from a pre-recorded file, pace the chunks at approximately real-time (for example, sleep for the chunk's duration between sends) — sending chunks in a tight loop can produce inconsistent Turn messages. See the [Streaming quickstart](https://www.assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio) to get started.

  updateConfiguration:
    action: receive
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/updateConfiguration"
    description: Update streaming configuration parameters during an active session. You can update `prompt`, `keyterms_prompt`, `min_turn_silence`, `max_turn_silence`, `end_of_turn_confidence_threshold`, `continuous_partials`, `vad_threshold`, `interruption_delay`, `mode`, `language_codes`, `session_heartbeat`, and `agent_context`.

  forceEndpoint:
    action: receive
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/forceEndpoint"
    description: Manually force an endpoint in the transcription.

  terminateSession:
    action: receive
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/sessionTermination"
    description: Gracefully terminate the streaming session.

  keepAlive:
    action: receive
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/keepAlive"
    description: Send a keep-alive message to reset the `inactivity_timeout` timer. This is not necessary by default — sessions remain open until explicitly terminated or until the 3-hour maximum session duration is reached. This message is only needed if you have set `inactivity_timeout` and want to keep the session open during periods where no audio is being sent.

  sessionBegins:
    action: send
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/sessionBegins"
    description: Receive confirmation that the streaming session has successfully started.

  speechStarted:
    action: send
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/speechStarted"
    description: Receive a notification that speech has been detected. This event is only emitted when the model produces a transcript. Every `SpeechStarted` is guaranteed to be followed by one or more Turn messages.

  turn:
    action: send
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/turn"
    description: Receive a turn-based transcription result (partial or final).

  speakerRevision:
    action: send
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/speakerRevision"
    description: Receive revised speaker labels at the end of a session. Emitted as a single message immediately before `Termination` when `speaker_labels` is enabled. Contains a `revisions` array with only the turns whose labels changed — unchanged turns are omitted. Text and word timestamps are never changed — only speaker assignments. Adds approximately 400ms of latency at session close. See [Revised speaker labels](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels#revised-speaker-labels) for details.

  sessionTermination:
    action: send
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/termination"
    description: Receive confirmation that the session has been terminated by the server.

  heartbeat:
    action: send
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/heartbeat"
    description: Receive a periodic liveness message. Emitted every 5 seconds of wall-clock time for the life of the session when `session_heartbeat` is enabled, during both speech and silence.

  llmGatewayResponse:
    action: send
    channel:
      $ref: "#/channels/streaming"
    messages:
      - $ref: "#/channels/streaming/messages/llmGatewayResponse"
    description: Receive an LLM Gateway response for a finalized turn. Emitted once per turn when `llm_gateway` is configured on the connection.

components:
  messages:
    UpdateConfiguration:
      name: UpdateConfiguration
      title: Update Streaming Configuration
      summary: Client message to update streaming configuration parameters during an active session.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/UpdateConfigurationPayload"
      examples:
        - payload:
            type: UpdateConfiguration
            prompt: Transcribe product names accurately.
            keyterms_prompt:
              - AssemblyAI
              - Krabby Patty
            min_turn_silence: 700
            max_turn_silence: 1600
            agent_context: Sure — what date would you like to book?

    ForceEndpoint:
      name: ForceEndpoint
      title: Force Endpoint
      summary: Client message to manually force an endpoint in the transcription.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/ForceEndpointPayload"
      examples:
        - payload:
            type: ForceEndpoint

    SessionTermination:
      name: SessionTermination
      title: Terminate Session (Client Initiated)
      summary: Client message to gracefully terminate the streaming session.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionTerminationPayload"

    AudioChunk:
      name: AudioChunk
      title: Audio Data Chunk
      summary: Client sends audio data as raw binary.
      contentType: application/octet-stream
      payload:
        type: string
        format: binary
      examples:
        - payload: '\x10\x00\x20\x00\x30\x00\x40\x00\x30\x00\x20\x00\x10\x00\x00\x00\xf0\xff\xe0\xff\xd0\xff\xc0\xff'

    SessionBegins:
      name: SessionBegins
      title: Session Begins Confirmation
      summary: Server message indicating the streaming session has successfully started.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionBeginsPayload"
      examples:
        - payload:
            type: Begin
            id: 3207b601-2054-48df-ba77-8784dfcf9fb8
            expires_at: 1772570132
            configuration:
              model: universal-3-5-pro
              mode: balanced
              api_version: "2025-05-12"
              speaker_labels: false
              redact_pii: false
              filter_profanity: false
              domain: null
              voice_focus: null

    KeepAlive:
      name: KeepAlive
      title: Keep Alive
      summary: Client message to reset the inactivity timeout timer. This is not necessary by default — sessions remain open until explicitly terminated or until the 3-hour maximum session duration is reached. This message is only needed if you have set `inactivity_timeout` and want to keep the session open during periods where no audio is being sent.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/KeepAlivePayload"
      examples:
        - payload:
            type: KeepAlive

    SpeechStarted:
      name: SpeechStarted
      title: Speech Started
      summary: Server message indicating that speech has been detected.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SpeechStartedPayload"

    Turn:
      name: Turn
      title: Turn Transcript
      summary: Server message containing a turn-based transcription result.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/TurnPayload"
      examples:
        - payload:
            type: Turn
            turn_order: 0
            turn_is_formatted: true
            end_of_turn: true
            transcript: Hello world.
            end_of_turn_confidence: 1
            words:
              - text: Hello
                start: 0
                end: 500
                confidence: 0.99
              - text: world.
                start: 500
                end: 1000
                confidence: 0.98

    SpeakerRevision:
      name: SpeakerRevision
      title: Revised Speaker Labels
      summary: Server message containing corrected speaker labels for any turns that changed. Emitted as a single message after the client sends `Terminate`, when `speaker_labels` is enabled. A session may produce zero or one `SpeakerRevision` message; if sent, only changed turns are included in the `revisions` array. See [Revised speaker labels](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels#revised-speaker-labels) for details.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SpeakerRevisionPayload"
      examples:
        - payload:
            type: SpeakerRevision
            revisions:
              - turn_order: 3
                speaker_label: B
                words:
                  - text: Hello
                    speaker: B
                    start: 1200
                    end: 1450
                  - text: there.
                    speaker: B
                    start: 1450
                    end: 1780
              - turn_order: 7
                speaker_label: A
                words:
                  - text: Got it.
                    speaker: A
                    start: 4100
                    end: 4520

    Termination:
      name: Termination
      title: Session Terminated (Server Confirmation)
      summary: Server message confirming session termination with session statistics.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/TerminationPayload"

    Heartbeat:
      name: Heartbeat
      title: Heartbeat
      summary: Server liveness message sent at a fixed interval when `session_heartbeat` is enabled.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/HeartbeatPayload"
      examples:
        - payload:
            type: Heartbeat
            total_audio_received_ms: 45000
            total_duration_ms: 45205
            realtime_factor: 0.9964
            max_speech_probability: 0.999954

    LLMGatewayResponse:
      name: LLMGatewayResponse
      title: LLM Gateway Response
      summary: Server message containing an LLM Gateway response for a finalized turn.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/LLMGatewayResponsePayload"
      examples:
        - payload:
            type: LLMGatewayResponse
            turn_order: 0
            transcript: Hello world.
            data:
              request_id: 3c90c3cc-0d44-4b50-8888-8dd25736052a
              choices:
                - index: 0
                  message:
                    role: assistant
                    content: Hello! How can I help?
                  finish_reason: stop
              usage:
                input_tokens: 12
                output_tokens: 8
                total_tokens: 20
                prompt_tokens_details: {}
                completion_tokens_details: {}
              request: {}
              response_time: 123456789

  schemas:
    UpdateConfigurationPayload:
      type: object
      properties:
        type:
          type: string
          const: UpdateConfiguration
        prompt:
          type: string
          description: A contextual prompt describing what the audio is about, such as its domain, scenario, or full conversation details. The model uses this context to better recognize the vocabulary it makes likely. It carries context about your audio, not transcription instructions, so formatting or behavioral commands are not supported. Maximum 1750 characters. See [Prompting and keyterms](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms) for details.
        keyterms_prompt:
          type: array
          items:
            type: string
          description: A list of words and phrases to boost recognition for. See [Keyterms Prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) for more details.
        min_turn_silence:
          type: integer
          description: Silence duration in milliseconds before a speculative end-of-turn check. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details.
        max_turn_silence:
          type: integer
          description: Maximum silence in milliseconds before the turn is forced to end, regardless of punctuation. See [Configuring Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details.
        continuous_partials:
          type: boolean
          description: Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled (default), additional partials covering the full turn transcript are emitted approximately every 3 seconds while speech continues. When disabled, only one early partial is emitted near turn start. The first partial (timed by `interruption_delay`) is unaffected.
        vad_threshold:
          type: number
          description: The confidence threshold (0.0 to 1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent. Increase for noisy environments to reduce false speech detection.
        interruption_delay:
          type: integer
          minimum: 0
          maximum: 1000
          description: How soon the first partial is emitted in milliseconds. Useful for tuning voice agent barge-in responsiveness or allowing earlier partials for early LLM inference. Larger values are more confident on interruptions, smaller values result in faster time to first partial.
        agent_context:
          type: string
          description: Your voice agent's most recent spoken reply (TTS text). The model uses this as context for the next user turn, which improves accuracy on short or ambiguous replies and on spelled-out entities like emails or IDs. Each `UpdateConfiguration` replaces the previously set value. Maximum 1750 characters. See [Context Carryover](https://www.assemblyai.com/docs/streaming/universal-3-5-pro/context-carryover#passing-your-agents-reply-as-context).
        mode:
          type: string
          enum: [max_accuracy, min_latency, balanced]
          description: Switch the latency and accuracy preset mid-session. See the `mode` connection parameter for the behavior of each preset. Universal-3.5 Pro Streaming only.
        end_of_turn_confidence_threshold:
          type: number
          description: The confidence threshold (0.0 to 1.0) to use when determining if the end of a turn has been reached. Universal Streaming (English and Multilingual) only. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for configuration details.
        language_codes:
          type: array
          items:
            type: string
          description: Update the set of languages transcription is steered toward. Takes effect from the next turn. See the `language_codes` connection parameter for accepted codes and behavior. Pass an empty list to clear steering and return to native code-switching across all supported languages. Universal-3.5 Pro Streaming only.
        session_heartbeat:
          type: boolean
          description: Turn periodic `Heartbeat` liveness messages on or off mid-session. See the `session_heartbeat` connection parameter for behavior.
      required:
        - type

    ForceEndpointPayload:
      type: object
      properties:
        type:
          type: string
          const: ForceEndpoint
      required:
        - type

    KeepAlivePayload:
      type: object
      properties:
        type:
          type: string
          const: KeepAlive
      required:
        - type

    SessionTerminationPayload:
      type: object
      properties:
        type:
          type: string
          const: Terminate
      required:
        - type

    SessionBeginsPayload:
      type: object
      properties:
        type:
          type: string
          const: Begin
          description: Identifies the type of the message.
        id:
          type: string
          format: uuid
          description: Unique identifier for the streaming session.
        expires_at:
          type: integer
          description: Unix timestamp indicating when the session will expire.
        configuration:
          type: object
          description: Echo of the configuration the server applied to the session. Always check that `model` matches the `speech_model` you requested — unrecognized or misspelled query parameters are ignored rather than rejected. Optional settings that were not set are null.
          properties:
            model:
              type: string
              description: The speech model applied to the session.
            mode:
              type: string
              nullable: true
              description: The latency/accuracy preset in effect. Universal-3.5 Pro Streaming only; null otherwise.
            api_version:
              type: string
              description: The API version in effect for the session (see the `AssemblyAI-Version` header).
            speaker_labels:
              type: boolean
              description: Whether speaker diarization is enabled.
            redact_pii:
              type: boolean
              description: Whether PII redaction is enabled.
            filter_profanity:
              type: boolean
              description: Whether profanity filtering is enabled.
            domain:
              type: string
              nullable: true
              description: The domain specialization in effect (for example `medical-v1`), or null.
            voice_focus:
              type: string
              nullable: true
              description: The Voice Focus model in effect (`near-field` or `far-field`), or null.
      required:
        - type
        - id
        - expires_at

    SpeechStartedPayload:
      type: object
      properties:
        type:
          type: string
          const: SpeechStarted
          description: Identifies the type of the message.
        timestamp:
          type: integer
          description: The timestamp in milliseconds when speech was detected, relative to the beginning of the audio stream.
          format: ms
        confidence:
          type: number
          format: float
          description: The confidence score that speech has started.
      required:
        - type
        - timestamp
        - confidence

    Word:
      type: object
      properties:
        text:
          type: string
          description: The transcribed word.
        start:
          type: integer
          description: Start time in milliseconds relative to the beginning of the audio stream.
          format: ms
        end:
          type: integer
          description: End time in milliseconds relative to the beginning of the audio stream.
          format: ms
        confidence:
          type: number
          format: float
          minimum: 0
          maximum: 1
          description: Confidence score for the word (0.0 to 1.0).
        word_is_final:
          type: boolean
          description: Whether the word is final.
        speaker:
          type: string
          description: The speaker label for this word (e.g. `A`, `B`, `UNKNOWN`). Only present on final words (`word_is_final` is `true`) when `speaker_labels` is enabled. May be absent on individual words even when diarization is enabled — treat absent as unattributed and fall back to the turn-level `speaker_label` if needed. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details.
      required:
        - text
        - start
        - end
        - confidence
        - word_is_final

    TurnPayload:
      type: object
      properties:
        type:
          type: string
          const: Turn
        turn_order:
          type: integer
          description: Order of this turn in the conversation.
        turn_is_formatted:
          type: boolean
          description: Whether this turn has been formatted. For Universal-3.5 Pro Streaming, this always matches `end_of_turn`.
        end_of_turn:
          type: boolean
          description: Whether this marks the end of a turn. See [Turn Detection](https://www.assemblyai.com/docs/streaming/getting-started/optimizing-accuracy-and-latency) for more information.
        transcript:
          type: string
          description: Transcript of all finalized words in the turn.
        utterance:
          type: string
          description: Finalized transcript of the turn, populated only on end_of_turn messages. Empty string on all other Turn messages. Equivalent to transcript when populated.
        language_code:
          type: string
          description: The language of the turn. Only populated when language detection is enabled and an utterance is complete or turn is final.
        language_confidence:
          type: number
          format: float
          minimum: 0
          maximum: 1
          description: The confidence score for the detected language, between 0 (low confidence) and 1 (high confidence). Only populated when language detection is enabled and an utterance is complete or turn is final.
        speaker_label:
          type: string
          description: The speaker label for this turn (e.g. `A`, `B`). Only present when `speaker_labels` is enabled. Short turns with less than approximately 1 second of audio will have the label `UNKNOWN`. See [Streaming Diarization](https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels) for more details.
        end_of_turn_confidence:
          type: number
          format: float
          minimum: 0
          maximum: 1
          description: The confidence score that this is the end of a turn, between 0.0 (low confidence) and 1.0 (high confidence). For Universal-3.5 Pro Streaming, this is 1.0 when `end_of_turn` is true and 0.0 otherwise.
        words:
          type: array
          items:
            $ref: "#/components/schemas/Word"
          description: Array of word-level details for this turn.
      required:
        - type
        - turn_order
        - turn_is_formatted
        - end_of_turn
        - transcript
        - end_of_turn_confidence
        - words

    SpeakerRevisionPayload:
      type: object
      description: Sent at the end of a streaming session when `speaker_labels` is enabled. Contains corrections for any turns whose speaker labels changed. Text content and word timestamps are never changed — only speaker assignments.
      properties:
        type:
          type: string
          const: SpeakerRevision
          description: Identifies the type of the message.
        revisions:
          type: array
          description: Array of turn corrections. Only turns whose speaker labels changed are included.
          items:
            type: object
            properties:
              turn_order:
                type: integer
                description: Matches the `turn_order` of the original Turn message being corrected.
              speaker_label:
                type: string
                nullable: true
                description: The corrected turn-level speaker label (e.g. `A`, `B`, `UNKNOWN`). Replaces the `speaker_label` originally delivered on the matching Turn message.
              words:
                type: array
                description: Words with corrected per-word `speaker` assignments. Text and timestamps match the original Turn — only the `speaker` field may change.
                items:
                  type: object
                  properties:
                    text:
                      type: string
                      description: The word text (unchanged from the original Turn).
                    speaker:
                      type: string
                      description: The revised speaker label for this word (e.g. `A`, `B`, `UNKNOWN`).
                    start:
                      type: integer
                      description: Start time of the word in milliseconds from the start of the session.
                    end:
                      type: integer
                      description: End time of the word in milliseconds from the start of the session.
                  required:
                    - text
                    - speaker
                    - start
                    - end
            required:
              - turn_order
              - speaker_label
              - words
      required:
        - type
        - revisions

    TerminationPayload:
      type: object
      properties:
        type:
          type: string
          const: Termination
          description: Indicates the session has been terminated.
        audio_duration_seconds:
          type: integer
          description: Duration of the audio in seconds.
        session_duration_seconds:
          type: integer
          description: Duration of the session in seconds.
      required:
        - type
        - audio_duration_seconds
        - session_duration_seconds

    HeartbeatPayload:
      type: object
      properties:
        type:
          type: string
          const: Heartbeat
          description: Identifies the type of the message.
        total_audio_received_ms:
          type: integer
          format: ms
          description: Total audio received since the session started, in milliseconds.
        total_duration_ms:
          type: integer
          format: ms
          description: Wall-clock time since the session started, in milliseconds.
        realtime_factor:
          type: number
          format: float
          description: Ratio of audio received to wall-clock time over the window since the previous heartbeat. Around 1.0 when the client streams in real time, and below 1.0 when the client is under-delivering audio.
        max_speech_probability:
          type: number
          format: float
          minimum: 0
          maximum: 1
          description: Maximum voice-activity speech probability observed since the previous heartbeat. Near 1.0 during speech and near 0.0 during silence.
      required:
        - type
        - total_audio_received_ms
        - total_duration_ms
        - realtime_factor
        - max_speech_probability

    LLMGatewayResponsePayload:
      type: object
      properties:
        type:
          type: string
          const: LLMGatewayResponse
          description: Identifies the type of the message.
        turn_order:
          type: integer
          description: The order of the finalized turn that triggered the LLM Gateway call.
        transcript:
          type: string
          description: The finalized turn transcript that triggered the LLM Gateway call.
        data:
          type: object
          description: The chat completions response from the LLM Gateway.
          required: []
          properties:
            request_id:
              type: string
              format: uuid
              description: Unique identifier for the LLM Gateway request.
            choices:
              type: array
              items:
                type: object
                required: []
                properties:
                  index:
                    type: integer
                    description: The index of the choice in the response.
                  message:
                    type: object
                    required: []
                    properties:
                      role:
                        type: string
                      content:
                        type: string
                        description: The text content of the model's response. Null when only tool_calls are present.
                      tool_calls:
                        type: array
                        items:
                          type: object
                          properties:
                            id:
                              type: string
                              description: Unique identifier for the tool call.
                            type:
                              type: string
                              enum: [function]
                            function:
                              type: object
                              properties:
                                name:
                                  type: string
                                  description: The name of the function to call.
                                arguments:
                                  type: string
                                  description: The arguments to call the function with, as a JSON-formatted string.
                              required:
                                - name
                                - arguments
                          required:
                            - id
                            - type
                            - function
                        description: Tool calls requested by the model. Present when the model invokes tools.
                  finish_reason:
                    type: string
                    description: The reason the model stopped generating tokens.
              description: Array of completion choices returned by the model.
            usage:
              type: object
              description: Token usage statistics for the request.
              required: []
              properties:
                input_tokens:
                  type: integer
                  description: Number of tokens in the prompt.
                output_tokens:
                  type: integer
                  description: Number of tokens in the completion.
                total_tokens:
                  type: integer
                  description: Total tokens used (prompt + completion).
                prompt_tokens_details:
                  type: object
                  additionalProperties: true
                  required: []
                  properties: {}
                  description: Detailed breakdown of prompt token usage.
                completion_tokens_details:
                  type: object
                  additionalProperties: true
                  required: []
                  properties: {}
                  description: Detailed breakdown of completion token usage.
            request:
              type: object
              additionalProperties: true
              required: []
              properties: {}
              description: A copy of the original request, excluding `prompt` and `messages`.
            response_time:
              type: integer
              description: The response time in nanoseconds.
            llm_status_code:
              type: integer
              description: The status code from the LLM provider.
      required:
        - type
        - turn_order
        - transcript
        - data

    SubstitutionPolicy:
      type: string
      description: The replacement logic for detected PII, can be `entity_name` or `hash`. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for more details.
      enum:
        - entity_name
        - hash

    PiiPolicy:
      type: string
      description: The type of PII to redact. See [PII redaction](https://www.assemblyai.com/docs/streaming/pii-redaction) for the full list of policies and their descriptions.
      enum:
        - account_number
        - banking_information
        - blood_type
        - credit_card_cvv
        - credit_card_expiration
        - credit_card_number
        - date
        - date_interval
        - date_of_birth
        - drivers_license
        - drug
        - duration
        - email_address
        - event
        - filename
        - gender
        - gender_sexuality
        - healthcare_number
        - injury
        - ip_address
        - language
        - location
        - location_address
        - location_address_street
        - location_city
        - location_coordinate
        - location_country
        - location_state
        - location_zip
        - marital_status
        - medical_condition
        - medical_process
        - money_amount
        - nationality
        - number_sequence
        - occupation
        - organization
        - organization_medical_facility
        - passport_number
        - password
        - person_age
        - person_name
        - phone_number
        - physical_attribute
        - political_affiliation
        - religion
        - sexuality
        - statistics
        - time
        - url
        - us_social_security_number
        - username
        - vehicle_id
        - zodiac_sign
```

## docs/api-reference/specs/voice-agent-api.yaml

URL: https://www.assemblyai.com/docs/api-reference/specs/voice-agent-api.yaml

```yaml
asyncapi: 3.0.0

# Trigger redeploy
info:
  title: AssemblyAI Voice Agent API
  version: 1.0.0
  description: |
    Real-time voice conversation API over a single WebSocket. The agent listens to the user,
    transcribes, reasons, and speaks a response back — all in one connection. Clients stream
    PCM16 audio and receive audio, transcripts, and tool calls in return.

servers:
  production:
    host: agents.assemblyai.com
    protocol: wss
    description: Production Voice Agent WebSocket endpoint

channels:
  voiceAgent:
    address: /v1/ws
    description: |
      
      Connect to the Voice Agent API to run a real-time voice conversation. The client streams
      PCM16 audio to the server and receives the agent's spoken response (also PCM16), along with
      transcripts, tool calls, and lifecycle events.

      After the WebSocket opens, send a [`session.update`](#sendSessionUpdate) as your
      first message. You have two ways to configure the agent:

      - **Stored agent.** Send `{ "agent_id": "<id>" }` as the only field in `session`
        to bind to a reusable agent created via the
        [Agents REST API](https://www.assemblyai.com/docs/api-reference/voice-agent-api/create-agent). The stored
        `system_prompt`, `greeting`, `tools`, `input`, and `output` are applied
        server-side.
      - **Inline configuration.** Omit `agent_id` and send `system_prompt`, `greeting`,
        `tools`, `input`, and `output` directly. Useful for one-off or fully dynamic
        agents.

      The two modes are mutually exclusive. See
      [Deploy your agent](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/deploy) and
      [Inline session configuration](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/session-configuration)
      for details, or jump to the [Voice Agent API overview](https://www.assemblyai.com/docs/voice-agents/voice-agent-api)
      for the full event flow and a runnable quickstart.
    servers:
      - $ref: "#/servers/production"
    parameters:
      ApiKey:
        description: >-
          Pass your API key as a Bearer token in the `Authorization` header on the
          WebSocket upgrade request. For browser apps (which can't set custom headers
          on WebSockets), generate a
          [temporary token](https://www.assemblyai.com/docs/api-reference/voice-agent-api/generate-voice-agent-token)
          and pass it via the `token` query parameter instead. See
          [Browser integration](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/browser-integration).
        location: $message.header#/Authorization
        examples:
          - Bearer YOUR_ASSEMBLYAI_API_KEY

      token:
        description: >-
          Temporary authentication token for client-side connections. Generate one with
          [`GET /v1/token`](https://www.assemblyai.com/docs/api-reference/voice-agent-api/generate-voice-agent-token)
          on your server and pass it here so you don't expose your permanent API key in
          the browser. Each token is one-time use.
        location: $message.payload#/token

    messages:
      sessionUpdate:
        $ref: "#/components/messages/SessionUpdate"
      sessionResume:
        $ref: "#/components/messages/SessionResume"
      sessionEnd:
        $ref: "#/components/messages/SessionEnd"
      inputAudio:
        $ref: "#/components/messages/InputAudio"
      toolResult:
        $ref: "#/components/messages/ToolResult"
      replyCreate:
        $ref: "#/components/messages/ReplyCreate"

      sessionReady:
        $ref: "#/components/messages/SessionReady"
      sessionUpdated:
        $ref: "#/components/messages/SessionUpdated"
      sessionEnded:
        $ref: "#/components/messages/SessionEnded"
      sessionError:
        $ref: "#/components/messages/SessionError"
      inputSpeechStarted:
        $ref: "#/components/messages/InputSpeechStarted"
      inputSpeechStopped:
        $ref: "#/components/messages/InputSpeechStopped"
      transcriptUserDelta:
        $ref: "#/components/messages/TranscriptUserDelta"
      transcriptUser:
        $ref: "#/components/messages/TranscriptUser"
      replyStarted:
        $ref: "#/components/messages/ReplyStarted"
      replyAudio:
        $ref: "#/components/messages/ReplyAudio"
      transcriptAgent:
        $ref: "#/components/messages/TranscriptAgent"
      replyDone:
        $ref: "#/components/messages/ReplyDone"
      toolCall:
        $ref: "#/components/messages/ToolCall"

operations:
  sendSessionUpdate:
    action: send
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/sessionUpdate"
    description: |
      Configure the session. Send immediately on connect — before `session.ready` — to either:

      - **Bind to a stored agent** by sending `{ "agent_id": "<id>" }` as the only field in
        `session`. The agent's stored `system_prompt`, `greeting`, `tools`, `input`, and
        `output` are loaded server-side. Create stored agents with
        [`POST /v1/agents`](https://www.assemblyai.com/docs/api-reference/voice-agent-api/create-agent).
      - **Configure inline** by sending any combination of `system_prompt`, `greeting`,
        `tools`, `input`, and `output` (omit `agent_id`).

      `agent_id` is mutually exclusive with the inline fields; sending both raises a
      validation error.

      Can also be sent mid-conversation to update mutable fields (e.g. `system_prompt`,
      `input.turn_detection`). `greeting` and `output` are immutable after `session.ready`
      and changing them returns `immutable_field`.

  sendSessionResume:
    action: send
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/sessionResume"
    description: |
      Resume a previous session using the `session_id` from a prior `session.ready`. Preserves
      conversation context across dropped connections. Sessions are held for 30 seconds after
      every disconnection.

  sendSessionEnd:
    action: send
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/sessionEnd"
    description: |
      Cleanly end the session. The server emits a final `session.ended` and closes the
      WebSocket; the `session_id` is dead immediately and cannot be resumed. Use this
      instead of just closing the socket when the call is over — closing the socket without
      sending `session.end` leaves the session resumable (and billable) for 30 seconds.

  sendInputAudio:
    action: send
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/inputAudio"
    description: |
      Stream a chunk of user audio to the agent. Only send `input.audio` after `session.ready`.
      See [Audio format](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/audio-format) for the expected
      encoding (base64-encoded, mono). Supports `audio/pcm` (24 kHz), `audio/pcmu` (8 kHz), and `audio/pcma` (8 kHz).

  sendToolResult:
    action: send
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/toolResult"
    description: |
      Return a tool result to the agent. Send this inside your `reply.done` handler — not
      immediately on `tool.call`. See
      [Tool calling](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/overview).

  sendReplyCreate:
    action: send
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/replyCreate"
    description: |
      Ask the agent to generate a reply right now, optionally with one-shot
      `instructions`. Primary use case: deliver status updates while a `hold`-mode
      tool call is in flight. See
      [Tool calling — hold mode](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/overview#hold).

  receiveSessionReady:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/sessionReady"
    description: Session is established. Save `session_id` for reconnection and start streaming audio.

  receiveSessionUpdated:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/sessionUpdated"
    description: Sent after a `session.update` is applied successfully.

  receiveSessionEnded:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/sessionEnded"
    description: |
      Final event emitted on every clean teardown, right before the WebSocket closes.
      Sent when the client sends `session.end`, when the session hits
      `max_session_duration_seconds`, when the server hits an unrecoverable error, or
      when the 30-second grace window after a disconnect expires.

  receiveSessionError:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/sessionError"
    description: A session- or protocol-level error occurred.

  receiveInputSpeechStarted:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/inputSpeechStarted"
    description: Turn detection determined the user has started speaking.

  receiveInputSpeechStopped:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/inputSpeechStopped"
    description: Turn detection determined the user has stopped speaking.

  receiveTranscriptUserDelta:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/transcriptUserDelta"
    description: Partial transcript of the user's utterance, updating in real-time.

  receiveTranscriptUser:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/transcriptUser"
    description: Final transcript of the user's utterance.

  receiveReplyStarted:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/replyStarted"
    description: Agent has begun generating a response.

  receiveReplyAudio:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/replyAudio"
    description: |
      A chunk of the agent's spoken response (base64 PCM16). Decode and play immediately.
      See [Audio format](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/audio-format#playing-output-audio)
      for playback guidance.

  receiveTranscriptAgent:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/transcriptAgent"
    description: Full text of the agent's response, delivered after all audio for the reply has been sent.

  receiveReplyDone:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/replyDone"
    description: Agent has finished speaking. Send any accumulated `tool.result` events here.

  receiveToolCall:
    action: receive
    channel:
      $ref: "#/channels/voiceAgent"
    messages:
      - $ref: "#/channels/voiceAgent/messages/toolCall"
    description: |
      Agent wants to invoke a registered tool. Execute the tool, then send the result with
      `tool.result` after `reply.done` fires.

components:
  messages:
    SessionUpdate:
      name: SessionUpdate
      title: Update Session
      summary: |
        Client message to configure the session. Either bind to a stored agent by sending
        `agent_id`, or configure an inline agent with `system_prompt`, `greeting`, `input`,
        `output`, and `tools`. The two modes are mutually exclusive.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionUpdatePayload"
      examples:
        - name: BindStoredAgent
          summary: Bind the session to a stored agent by `agent_id`.
          payload:
            type: session.update
            session:
              agent_id: 7ad24396-b822-4dca-871a-be9cc4781cf9
        - name: InlineConfiguration
          summary: Configure the agent inline at connect time.
          payload:
            type: session.update
            session:
              system_prompt: You are a concise assistant.
              greeting: Hi — how can I help?
              input:
                format:
                  encoding: audio/pcm
                turn_detection:
                  vad_threshold: 0.5
              output:
                voice: anna
                format:
                  encoding: audio/pcm
                volume: 100
              tools:
                - type: function
                  name: get_weather
                  description: Get weather for a city
                  parameters:
                    type: object
                    properties:
                      city:
                        type: string
                    required:
                      - city

    SessionResume:
      name: SessionResume
      title: Resume Session
      summary: Client message to resume a previous session by `session_id`.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionResumePayload"
      examples:
        - payload:
            type: session.resume
            session_id: sess_abc123

    SessionEnd:
      name: SessionEnd
      title: End Session
      summary: |
        Client message to cleanly end the session. The server emits a final `session.ended`
        and closes the WebSocket; the `session_id` is dead immediately and cannot be resumed.
        Use this instead of just closing the socket to stop billing right away — closing the
        socket without `session.end` leaves the session resumable (and billable) for 30 seconds.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionEndPayload"
      examples:
        - payload:
            type: session.end

    InputAudio:
      name: InputAudio
      title: Input Audio Chunk
      summary: Client streams a chunk of PCM16 audio as base64.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/InputAudioPayload"
      examples:
        - payload:
            type: input.audio
            audio: EAAgADAAQAAwACAAEAAAAPD/4P/Q/8D/

    ToolResult:
      name: ToolResult
      title: Tool Result
      summary: Client returns the result of a tool invocation to the agent.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/ToolResultPayload"
      examples:
        - payload:
            type: tool.result
            call_id: call_abc123
            result: '{"temp_c": 22, "description": "Sunny"}'

    ReplyCreate:
      name: ReplyCreate
      title: Reply Create
      summary: Client asks the agent to generate a reply now, optionally with one-shot instructions.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/ReplyCreatePayload"
      examples:
        - payload:
            type: reply.create
            instructions: Let the customer know we're still processing the transfer.

    SessionReady:
      name: SessionReady
      title: Session Ready
      summary: Server confirms the session is established and ready for audio.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionReadyPayload"
      examples:
        - payload:
            type: session.ready
            session_id: sess_abc123

    SessionUpdated:
      name: SessionUpdated
      title: Session Updated
      summary: Server acknowledges that a `session.update` was applied successfully.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionUpdatedPayload"
      examples:
        - payload:
            type: session.updated

    SessionEnded:
      name: SessionEnded
      title: Session Ended
      summary: |
        Final event emitted on every clean teardown, right before the WebSocket closes. Sent
        when the client sends `session.end`, the session hits `max_session_duration_seconds`,
        the server hits an unrecoverable error, or the 30-second grace window after a
        disconnect expires.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionEndedPayload"
      examples:
        - payload:
            type: session.ended
            session_duration_seconds: 42.7
            audio_duration_seconds: 38.2
            timestamp: 1717180000.123

    SessionError:
      name: SessionError
      title: Session Error
      summary: Server reports a session- or protocol-level error.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/SessionErrorPayload"
      examples:
        - payload:
            type: session.error
            code: invalid_format
            message: Invalid message format

    InputSpeechStarted:
      name: InputSpeechStarted
      title: User Started Speaking
      summary: Server signals that turn detection determined the user started speaking.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/InputSpeechStartedPayload"
      examples:
        - payload:
            type: input.speech.started

    InputSpeechStopped:
      name: InputSpeechStopped
      title: User Stopped Speaking
      summary: Server signals that turn detection determined the user stopped speaking.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/InputSpeechStoppedPayload"
      examples:
        - payload:
            type: input.speech.stopped

    TranscriptUserDelta:
      name: TranscriptUserDelta
      title: User Transcript Delta
      summary: Partial transcript of the user's current utterance.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/TranscriptUserDeltaPayload"
      examples:
        - payload:
            type: transcript.user.delta
            text: What's the weather in

    TranscriptUser:
      name: TranscriptUser
      title: User Transcript
      summary: Final transcript of the user's utterance.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/TranscriptUserPayload"
      examples:
        - payload:
            type: transcript.user
            text: What's the weather in Tokyo?
            item_id: item_abc123

    ReplyStarted:
      name: ReplyStarted
      title: Reply Started
      summary: Agent has begun generating a reply.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/ReplyStartedPayload"
      examples:
        - payload:
            type: reply.started
            reply_id: reply_abc123

    ReplyAudio:
      name: ReplyAudio
      title: Reply Audio Chunk
      summary: A chunk of the agent's spoken response as base64 PCM16.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/ReplyAudioPayload"
      examples:
        - payload:
            type: reply.audio
            data: EAAgADAAQAAwACAAEAAAAPD/4P/Q/8D/

    TranscriptAgent:
      name: TranscriptAgent
      title: Agent Transcript
      summary: Text of the agent's response, sent after all reply audio has been delivered.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/TranscriptAgentPayload"
      examples:
        - payload:
            type: transcript.agent
            text: It's currently 22°C and sunny in Tokyo.
            reply_id: reply_abc123
            item_id: item_abc123
            interrupted: false

    ReplyDone:
      name: ReplyDone
      title: Reply Done
      summary: |
        Agent has finished speaking. If the user barged in, `status` is `"interrupted"`. Send
        accumulated `tool.result` events on this event.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/ReplyDonePayload"
      examples:
        - payload:
            type: reply.done

    ToolCall:
      name: ToolCall
      title: Tool Call
      summary: Agent wants to invoke a registered tool.
      contentType: application/json
      payload:
        $ref: "#/components/schemas/ToolCallPayload"
      examples:
        - payload:
            type: tool.call
            call_id: call_abc123
            name: get_weather
            arguments:
              location: Tokyo

  schemas:
    SessionConfig:
      type: object
      description: |
        Session configuration fields. All fields are optional — only include the ones you want to change.

        There are two ways to configure a session:

        - **Bind to a stored agent.** Send `agent_id` (and only `agent_id`) in your first
          `session.update` to load a reusable agent created via the
          [Agents REST API](https://www.assemblyai.com/docs/api-reference/voice-agent-api/create-agent). The agent's
          stored `system_prompt`, `greeting`, `tools`, `input`, and `output` are
          applied server-side.
        - **Configure inline.** Omit `agent_id` and send any combination of
          `system_prompt`, `greeting`, `tools`, `input`, and `output` directly.

        `agent_id` is mutually exclusive with the inline fields — sending both in the
        same `session.update` is rejected. See
        [Deploy your agent](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/deploy) for details.
      required: []
      properties:
        agent_id:
          type: string
          description: |
            ID of a stored agent (from
            [`POST /v1/agents`](https://www.assemblyai.com/docs/api-reference/voice-agent-api/create-agent))
            to bind this session to. When set, the agent's stored configuration is
            applied and you must not send `system_prompt`, `greeting`, `tools`,
            `input`, or `output` in the same `session.update`. Can only be set in
            the first `session.update`, before `session.ready`.
          examples:
            - 7ad24396-b822-4dca-871a-be9cc4781cf9
        system_prompt:
          type: string
          description: The agent's personality and context. Can be updated mid-session. Mutually exclusive with `agent_id`.
        greeting:
          type: string
          description: |
            What the agent says at the start of the conversation. Sent directly to
            the TTS engine and spoken verbatim. It is NOT run through the LLM,
            so write the exact words you want the user to hear. Immutable after
            `session.ready`. Omit to have the agent wait silently for the user to
            speak first.
        input:
          $ref: "#/components/schemas/SessionInputConfig"
        output:
          $ref: "#/components/schemas/SessionOutputConfig"
        tools:
          type: array
          description: Tool definitions. See [Tool calling](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/overview).
          items:
            $ref: "#/components/schemas/ToolDefinition"

    SessionInputConfig:
      type: object
      description: Configuration for the user audio input stream.
      required: []
      properties:
        format:
          $ref: "#/components/schemas/AudioFormat"
        keyterms:
          type: array
          items:
            type: string
          description: List of rare or domain-specific terms to boost in transcription (e.g. names, brands, jargon). See [Key terms](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/session-configuration#key-terms).
        turn_detection:
          $ref: "#/components/schemas/TurnDetection"

    SessionOutputConfig:
      type: object
      description: Configuration for the agent audio output stream.
      required: []
      properties:
        voice:
          type: string
          description: Voice used for the agent's speech. See [Voices](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/voices).
          default: anna
          examples:
            - anna
        format:
          $ref: "#/components/schemas/AudioFormat"
        volume:
          type: number
          minimum: 0
          maximum: 100
          description: Playback volume for the agent's speech. `0` is silent, `100` is loudest. If omitted, the voice plays at its native level. Mutable mid-session.
          examples:
            - 100

    AudioFormat:
      type: object
      description: Audio format configuration. The encoding determines the sample rate. See [Audio format](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/audio-format).
      required: []
      properties:
        encoding:
          type: string
          enum:
            - audio/pcm
            - audio/pcmu
            - audio/pcma
          default: audio/pcm
          description: "Audio encoding. `audio/pcm` (PCM16, 16-bit little-endian, 24 kHz), `audio/pcmu` (G.711 μ-law, 8 kHz), or `audio/pcma` (G.711 A-law, 8 kHz)."

    ToolDefinition:
      type: object
      description: A single tool the agent can call.
      properties:
        type:
          type: string
          const: function
          description: Always `function`.
        name:
          type: string
          description: Name of the tool, referenced by `tool.call`.
        description:
          type: string
          description: Human-readable description of what the tool does.
        parameters:
          type: object
          description: |
            JSON Schema describing the tool's arguments. Must be a JSON Schema object
            (`{"type": "object", "properties": {...}, "required": [...]}`).
            Passed to the agent's LLM verbatim. The server does NOT validate the schema
            at `session.update` time, so a malformed schema is accepted but produces
            unpredictable tool calls. Validate your schema before sending.
            Every property should have a `description`. That string is how the model
            extracts argument values from user speech.
          additionalProperties: true
          required: []
          properties: {}
        execution_mode:
          type: string
          enum:
            - interactive
            - hold
          default: interactive
          description: |
            How the agent waits for the result.
            `"interactive"` (default): the agent speaks a short transition phrase
            (e.g. "let me check that") while the tool runs, then delivers the result
            conversationally. Use for short tools (DB lookups, REST calls).
            `"hold"`: the agent stays silent while the tool runs and suppresses replies
            triggered by user speech. Send `reply.create` to deliver status updates
            during the hold. When `tool.result` arrives, the agent gives a brief,
            direct delivery of the result. Use for long-running tools (transfers,
            escalations, async jobs). See
            [Tool calling — execution modes](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/overview#execution-modes).
        timeout_seconds:
          type: number
          format: float
          minimum: 1
          maximum: 300
          default: 120
          description: How long to wait for the matching `tool.result` before timing out.
      required:
        - type
        - name
        - description
        - parameters

    TurnDetection:
      type: object
      description: Configures turn detection sensitivity, end-of-turn detection, and barge-in behavior. Lives under `session.input.turn_detection`.
      required: []
      properties:
        vad_threshold:
          type: number
          format: float
          minimum: 0
          maximum: 1
          default: 0.5
          description: Speech detection sensitivity (0.0–1.0). Lower = more sensitive to speech.
        min_silence:
          type: integer
          minimum: 50
          maximum: 10000
          default: 1000
          description: Minimum silence to consider a confident end-of-turn, in milliseconds. Must be less than `max_silence`.
        max_silence:
          type: integer
          minimum: 50
          maximum: 10000
          default: 3000
          description: Maximum silence before forcing end-of-turn, in milliseconds. Must be greater than `min_silence`.
        interrupt_response:
          type: boolean
          default: true
          description: Whether user speech interrupts the agent. Set `false` to disable barge-in.

    SessionUpdatePayload:
      type: object
      properties:
        type:
          type: string
          const: session.update
        session:
          $ref: "#/components/schemas/SessionConfig"
      required:
        - type
        - session

    SessionResumePayload:
      type: object
      properties:
        type:
          type: string
          const: session.resume
        session_id:
          type: string
          description: The `session_id` from a previous `session.ready` event.
      required:
        - type
        - session_id

    SessionEndPayload:
      type: object
      properties:
        type:
          type: string
          const: session.end
      required:
        - type

    InputAudioPayload:
      type: object
      properties:
        type:
          type: string
          const: input.audio
        audio:
          type: string
          description: Base64-encoded audio chunk in the configured input encoding.
      required:
        - type
        - audio

    ToolResultPayload:
      type: object
      properties:
        type:
          type: string
          const: tool.result
        call_id:
          type: string
          description: The `call_id` from the `tool.call` event you are responding to.
        result:
          type: string
          description: |
            JSON-encoded string containing the tool result. Always a string,
            not a nested object. Use `json.dumps(...)` (Python) or `JSON.stringify(...)` (JS)
            on the payload before sending.
      required:
        - type
        - call_id
        - result

    ReplyCreatePayload:
      type: object
      properties:
        type:
          type: string
          const: reply.create
        instructions:
          type: string
          description: |
            Optional one-shot instructions the agent uses to compose this reply.
            Does not modify `system_prompt`. Useful for status updates during a
            `hold`-mode tool call. See
            [Tool calling — execution modes](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/tools/overview#execution-modes).
      required:
        - type

    SessionReadyPayload:
      type: object
      properties:
        type:
          type: string
          const: session.ready
        session_id:
          type: string
          description: Unique identifier for this session. Save this to reconnect with `session.resume`.
      required:
        - type
        - session_id

    SessionUpdatedPayload:
      type: object
      properties:
        type:
          type: string
          const: session.updated
      required:
        - type

    SessionEndedPayload:
      type: object
      properties:
        type:
          type: string
          const: session.ended
        session_duration_seconds:
          type: number
          description: Total wall-clock duration of the session, in seconds.
        audio_duration_seconds:
          type: number
          nullable: true
          description: Total audio streamed in by the client, in seconds. `null` if no audio was streamed.
        timestamp:
          type: number
          description: Unix epoch seconds when the server emitted the event.
      required:
        - type
        - session_duration_seconds
        - timestamp

    SessionErrorPayload:
      type: object
      properties:
        type:
          type: string
          enum:
            - session.error
            - error
          description: |
            `session.error` for session/protocol errors and `error` for connection-level errors.
        code:
          type: string
          enum:
            - UNAUTHORIZED
            - FORBIDDEN
            - INTERNAL_ERROR
            - server_error
            - session_not_found
            - session_forbidden
            - session_expired
            - agent_init_failed
            - agent_timeout
            - invalid_format
            - invalid_audio
            - invalid_value
            - immutable_field
            - invalid_config
          description: |
            Machine-readable error code. See [error codes](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/events-reference#sessionerror)
            for the full table grouped by lifecycle stage.
        message:
          type: string
          description: Human-readable error description.
        timestamp:
          type: string
          format: date-time
          description: ISO-8601 timestamp of the error.
        param:
          type: string
          description: Name of the offending field when applicable (e.g. on `session.update` validation failures).
      required:
        - type
        - code
        - message

    InputSpeechStartedPayload:
      type: object
      properties:
        type:
          type: string
          const: input.speech.started
      required:
        - type

    InputSpeechStoppedPayload:
      type: object
      properties:
        type:
          type: string
          const: input.speech.stopped
      required:
        - type

    TranscriptUserDeltaPayload:
      type: object
      properties:
        type:
          type: string
          const: transcript.user.delta
        text:
          type: string
          description: Partial transcript of what the user is saying.
      required:
        - type
        - text

    TranscriptUserPayload:
      type: object
      properties:
        type:
          type: string
          const: transcript.user
        text:
          type: string
          description: Final transcript of the user's utterance.
        item_id:
          type: string
          description: Conversation item ID.
      required:
        - type
        - text
        - item_id

    ReplyStartedPayload:
      type: object
      properties:
        type:
          type: string
          const: reply.started
        reply_id:
          type: string
          description: ID of this reply.
      required:
        - type
        - reply_id

    ReplyAudioPayload:
      type: object
      properties:
        type:
          type: string
          const: reply.audio
        data:
          type: string
          description: Base64-encoded audio chunk in the configured output encoding.
      required:
        - type
        - data

    TranscriptAgentPayload:
      type: object
      properties:
        type:
          type: string
          const: transcript.agent
        text:
          type: string
          description: What the agent said. If interrupted, trimmed to the point of interruption.
        reply_id:
          type: string
          description: ID of the reply this transcript belongs to.
        item_id:
          type: string
          description: Conversation item ID.
        interrupted:
          type: boolean
          description: Whether the user interrupted the agent mid-response.
      required:
        - type
        - text
        - reply_id
        - item_id
        - interrupted

    ReplyDonePayload:
      type: object
      properties:
        type:
          type: string
          const: reply.done
        status:
          type: string
          enum:
            - completed
            - interrupted
          description: |
            `"completed"` for normal completion, `"interrupted"` if the user barged in.
      required:
        - type
        - status

    ToolCallPayload:
      type: object
      properties:
        type:
          type: string
          const: tool.call
        call_id:
          type: string
          description: Include this value in the corresponding `tool.result`.
        name:
          type: string
          description: Name of the tool the agent is invoking.
        arguments:
          type: object
          description: Arguments to pass to the tool, as a dictionary.
          additionalProperties: true
          required: []
          properties: {}
      required:
        - type
        - call_id
        - name
        - arguments
```