Core

Speech and Voice AI: ASR, TTS, and Voice Agents

Voice agents chain three systems: speech-to-text (ASR), an LLM, and text-to-speech (TTS), all under a hard real-time latency budget that text chat never faces. This page covers acoustic modeling and CTC basics, the cascade-versus-end-to-end tradeoff, and the conversational mechanics that actually break demos: turn-taking, barge-in, and the sub-second response budget. Applied AI interviews probe it because voice exposes whether you can reason about streaming, latency accounting, and a distinct class of failure modes.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

Multimodal Models and VLMs Latency Budgets and Streaming The Transformer Architecture Streaming and Backpressure

PRACTICE THIS IN REAL QUESTIONS

System Design for AI in ProductionDesign a voice assistant architecture (speech in, speech out) with low latency.→System Design for AI in ProductionDesign a real-time transcription system for thousands of concurrent audio streams.→Machine Learning & Data ScienceHow does speech-to-text (Whisper) work, and what matters when building voice AI (STT + TTS)?→MLOps & ML EngineeringYour online features are stale, and predictions suffer for it. How do you guarantee feature freshness?→System Design for AI in ProductionDesign an AI writing assistant (Grammarly/Notion-style) for rewriting, grammar, and tone.→System Design for AI in ProductionDesign a real-time speech translation system (speak in one language, hear another) with low latency.→

COMPANIES THAT ASSUME THIS

OpenAI Anthropic Google Microsoft Scale AI Decagon

NEXT IN FOUNDATIONS OF LLMS & GENAIContext Rot and Long-Context Failure Modes→