Adding TTS Audio to a Feature

This skill walks you through adding text-to-speech audio to a feature in the app. The TTS system plays audio via a voice chain (pregenerated mp3 → on-demand generation via OpenAI → browser SpeechSynthesis → subtitles). Clips are collected at runtime, persisted to the database, and generated as high-quality OpenAI TTS mp3s — either on-the-fly during playback (if the

generate

Before You Start

Read the integration guide:

apps/web/.claude/reference/tts-audio-system.md

It contains the full API reference, patterns, anti-patterns, and existing implementations.

Your Job

1. Understand what text the feature needs spoken and when
2. Create a feature-specific audio hook
3. Wire it into the component
4. Verify with TypeScript

Step 1: Design the Utterances

For each piece of audio the feature needs, determine:

• What text to speak (static string or dynamic from state/props)
• When to speak it (on mount, on state change, on user action, on completion)
• How it should sound (tone — write as voice-actor stage directions)

Step 2: Create a Feature Audio Hook

Create a hook in the feature's

hooks/

Read the reference implementation first:

apps/web/src/components/practice/hooks/usePracticeAudioHelp.ts

Key rules:

1. Tone strings must be module-level constants — never compute them dynamically per render
2. Always clean up on unmount — call

   stop()

   in a cleanup effect
3. Use refs to track previous values — prevents re-playing on every render
4. Guard with

   isEnabled

   — respect the user's audio toggle

Template:

'use client'

import { useEffect, useRef } from 'react'
import { useTTS } from '@/hooks/useTTS'
import { useAudioManager } from '@/hooks/useAudioManager'

// Stable tone constants — changing these creates new clips
const INSTRUCTION_TONE =
  'Patiently guiding a young child. Clear, slow, friendly.'
const CELEBRATION_TONE =
  'Warmly congratulating a child. Genuinely encouraging and happy.'

interface UseMyFeatureAudioHelpOptions {
  currentStep: string
  isComplete: boolean
}

export function useMyFeatureAudioHelp({
  currentStep,
  isComplete,
}: UseMyFeatureAudioHelpOptions) {
  const { isEnabled, stop } = useAudioManager()

  // Declare utterances
  const sayInstruction = useTTS(currentStep, { tone: INSTRUCTION_TONE })
  const sayCelebration = useTTS(
    isComplete ? 'Well done!' : '',
    { tone: CELEBRATION_TONE },
  )

  // Auto-play when step changes
  const prevStepRef = useRef<string>('')
  useEffect(() => {
    if (!isEnabled || !currentStep || currentStep === prevStepRef.current) return
    prevStepRef.current = currentStep
    sayInstruction()
  }, [isEnabled, currentStep, sayInstruction])

  // Auto-play celebration on completion
  useEffect(() => {
    if (!isEnabled || !isComplete) return
    sayCelebration()
  }, [isEnabled, isComplete, sayCelebration])

  // Stop audio on unmount
  useEffect(() => {
    return () => stop()
  }, [stop])

  return { replay: sayInstruction }
}

Step 3: Wire Into the Component

import { useMyFeatureAudioHelp } from './hooks/useMyFeatureAudioHelp'
import { useAudioManager } from '@/hooks/useAudioManager'

function MyFeature() {
  const { isEnabled, isPlaying } = useAudioManager()
  const { replay } = useMyFeatureAudioHelp({
    currentStep: 'Tap the bead to move it up',
    isComplete: false,
  })

  return (
    <div>
      {isEnabled && (
        <button onClick={replay} disabled={isPlaying}>
          {isPlaying ? 'Speaking...' : 'Replay'}
        </button>
      )}
    </div>
  )
}

Step 4: Verify

cd apps/web && npx tsc --noEmit

Common Patterns

Dynamic text from state

const text = useMemo(
  () => (terms ? termsToSentence(terms) : ''),
  [terms],
)
const sayProblem = useTTS(text, { tone: MATH_TONE })

One-shot playback (play once, don't repeat)

const playedRef = useRef(false)
useEffect(() => {
  if (!shouldPlay || playedRef.current) return
  playedRef.current = true
  sayIt()
}, [shouldPlay, sayIt])

// Reset when trigger resets
useEffect(() => {
  if (!shouldPlay) playedRef.current = false
}, [shouldPlay])

Multiple utterances — play the right one

const sayStep1 = useTTS('First, look at the abacus', { tone: INST })
const sayStep2 = useTTS('Now tap the bead', { tone: INST })

// speak() stops previous before starting
if (step === 0) sayStep1()
if (step === 1) sayStep2()

Tone String Guidelines

Write tones as voice-actor stage directions. Be specific about emotion, pace, and audience.

Good examples:

• 'Speaking clearly and steadily, reading a math problem to a young child. Pause slightly between each number and operator.'

• 'Warmly congratulating a child. Genuinely encouraging and happy.'

• 'Gently guiding a child after a wrong answer. Kind, not disappointed.'

• 'Patiently guiding a young child through an abacus tutorial. Clear, slow, friendly.'

Bad examples:

• 'Read this text'

  — too vague

• `Speaking ${mood}`

  — dynamic per render, creates new clips every time

Anti-Patterns to Avoid

1. Never use raw

   speechSynthesis

   — always go through

   useTTS

   so the voice chain and collection work
2. Never forget cleanup — always

   useEffect(() => () => stop(), [stop])

3. Never use dynamic tone strings — keep them as module-level constants
4. Never call

   speak()

   unconditionally in render — always guard with refs and

   isEnabled

Key Files

src/hooks/useTTS.ts

src/hooks/useAudioManager.ts

src/lib/audio/TtsAudioManager.ts

src/lib/audio/voiceSource.ts

generate()

src/contexts/AudioManagerContext.tsx

src/lib/audio/termsToSentence.ts

[5, 3]

"five plus three"

src/lib/audio/buildFeedbackText.ts

src/lib/audio/numberToEnglish.ts

42

"forty two"

Voice Chain

Audio plays through the voice chain in order. The typical chain is:

pregenerated voice (nova) → auto-generate → browser TTS → subtitles

• Pregenerated: instant playback from pre-generated mp3 on disk
• Auto-generate: calls OpenAI on-the-fly if the pregenerated mp3 is missing, caches result
• Browser TTS: uses the browser's built-in speech synthesis
• Subtitles: shows text on screen with a reading-time timer

You don't need to think about this when adding TTS to a feature — just use

useTTS()

/admin/audio

Reference Implementations

usePracticeAudioHelp

src/components/practice/hooks/

useTutorialAudioHelp

src/components/tutorial/hooks/

Follow

usePracticeAudioHelp