Speech & Dialogue
Language as sound and conversation. Representing speech as spectrograms, automatic speech recognition (CTC and seq2seq), text-to-speech, dialogue state tracking and turn-taking, voice agents — and the ethical stakes of synthetic voices and always-listening systems.
01 · Signal
Speech as a signal
Spoken language adds a layer the text pipeline never had: a continuous, noisy acoustic signal. Raw audio is a sequence of tens of thousands of amplitude samples per second — too long and too unstructured to feed a model directly. The first job is to turn sound into a representation that exposes linguistic structure.
02 · Representation
Spectrograms — STFT and the mel scale
The standard representation is the spectrogram. Slice the waveform into short overlapping frames (~25 ms, hopping ~10 ms), and take the Short-Time Fourier Transform of each — the energy at every frequency in that frame. Stacking frames gives a time–frequency image. Then warp the frequency axis to the mel scale, which spaces frequencies the way human hearing does (fine resolution low, coarse high). The result — a mel spectrogram — is what almost every speech model consumes.
Mel spectrogram
A spectrogram shows energy at each frequency (rows, mel-warped) over time (columns). It is the image almost every speech model actually reads. Switch sounds to see steady formant bands, a rising chirp, or broadband hiss.
time →
Why a spectrogram, not raw audio
Raw audio is enormously long (16 000 samples/second) and the linguistically relevant structure (formants, pitch, fricative noise) lives in the frequency domain. The STFT exposes that structure as a compact time–frequency image; the mel scale matches human perception. It is feature engineering that makes the learning problem tractable — the audio analogue of tokenisation.
03 · Speech → text
Automatic speech recognition
ASR maps an audio spectrogram to a text transcript. The hard part is alignment: which audio frames correspond to which output characters is unknown and variable (people speak at different speeds).
CTC
Connectionist Temporal Classification predicts a character (or blank) per frame, then collapses repeats and blanks. It sums over all valid alignments with a dynamic-programming loss, so no frame-level labels are needed — alignment is learned implicitly.
Seq2seq / attention (Whisper)
An encoder reads the spectrogram; an attention decoder generates the transcript token by token, learning the alignment as attention weights — the same architecture as machine translation, with audio as the source.
The alignment problem, and how CTC solves it
Audio and text have different, unknown lengths with no given correspondence. CTC introduces a blank symbol and defines a many-to-one collapse (merge repeats, drop blanks), then trains by summing the probability of every frame-labelling that collapses to the target — a forward-backward DP. This avoids needing hand-aligned data, the same reason Viterbi-style DP is everywhere in sequence modelling.
04 · Text → speech
Text-to-speech
TTS runs the pipeline backwards: text → mel spectrogram → waveform. A first model (e.g. Tacotron, FastSpeech) predicts a mel spectrogram from text, including prosody — rhythm, stress, intonation — which carries meaning (“really?” vs “really.”). A second model, the neural vocoder (WaveNet, HiFi-GAN), turns the spectrogram into an audible waveform. Modern systems clone a voice from seconds of reference audio.
05 · Conversation
Dialogue systems
A conversational system must track context across turns. Dialogue state tracking (DST) maintains a structured summary of what the user wants so far (the slots: destination, date, party size for a booking) and updates it each turn. Turn-taking decides when the system should speak — detecting that the user has finished, handling interruptions and back-channels (“mm-hm”) — a problem text chat never has. Task-oriented dialogue adds a policy that chooses the next action given the state; open-domain chat is increasingly just a prompted LLM.
06 · Voice agents
Voice agents — closing the loop
A voice assistant chains everything: ASR (speech → text) → an NLU / LLM (understand, decide, with DST and tools as in Ch. 10) → TTS (text → speech), all under a latency budget tight enough to feel conversational, and with turn-taking to know when to listen versus speak. End-to-end speech-to-speech models are starting to collapse this cascade into one network.
07 · Stakes
Ethics and risks
Speech technology raises stakes text does not:
- Voice cloning & deepfakes — synthesising someone’s voice from seconds of audio enables fraud and impersonation; provenance and watermarking are open problems.
- Always-on listening — voice assistants imply a microphone in private space; consent and on-device processing matter.
- Bias & accessibility — ASR accuracy varies by accent, dialect, and speech difference, so systems can systematically fail some users.
- Consent — using recorded voices for training or cloning without permission is an ethical and increasingly legal line.
Why speech ethics is sharper than text ethics
Voice is biometric and personal — it identifies a specific human and conveys emotion and identity. A cloned voice can defraud a relative; a biased ASR can lock someone out of a service they depend on; an always-listening device sits in the bedroom. The capability is the easy part; deploying it responsibly is the hard part.
08 · Self-check
Questions before you move on
Why do speech models operate on spectrograms rather than raw audio samples?
What problem does CTC solve in speech recognition?
A text-to-speech pipeline typically goes:
Dialogue state tracking (DST) maintains:
09 · Recap
One-screen summary
Chapter 11 — load-bearing ideas
- Speech is a signal; we convert it to a spectrogram via the STFT and warp frequency to the mel scale — the image speech models actually read.
- ASR maps spectrogram → text; CTC solves alignment with a blank symbol and a forward-backward loss; seq2seq/attention (Whisper) learns alignment as attention.
- TTS runs backwards: text → mel spectrogram (with prosody) → waveform via a neural vocoder.
- Dialogue needs state tracking (slots) and turn-taking (when to speak) that text chat never required.
- Voice agents chain ASR → LLM/NLU (+ tools) → TTS under tight latency; end-to-end speech models are emerging.
- Ethics is sharper for speech: voice cloning, always-on listening, accent bias, and consent are first-order concerns.
10 · Exam · past papers
Past-paper questions
Q-SP1A spectrogram is produced from a waveform by:
Q-SP2The mel scale is used because it:
Q-SP3CTC enables ASR training without frame-level alignment by:
Q-SP4In a text-to-speech system, the neural vocoder:
Q-SP5Turn-taking is a problem unique to spoken dialogue because the system must:
Q-SP6A first-order ethical risk specific to speech technology is: