LiveKit Self-Hosted STT Plugin

Build self-hosted speech-to-text APIs and LiveKit voice agent plugins using Hugging Face models.

Overview

This skill provides templates and guidance for:

1. Building a self-hosted STT API server using FastAPI + Whisper/HF models
2. Creating a LiveKit plugin that connects to your self-hosted API
3. Deploying and scaling in production

Quick Start

Option 1: Build Both (API + Plugin)

When user wants complete setup:

1. Create API Server:

python scripts/setup_api_server.py my-stt-server --model openai/whisper-medium
cd my-stt-server
pip install -r requirements.txt
python main.py

2. Create Plugin:

python scripts/setup_plugin.py custom-stt
cd livekit-plugins-custom-stt
pip install -e .

3. Use in LiveKit Agent:

from livekit.plugins import custom_stt

stt=custom_stt.STT(api_url="ws://localhost:8000/ws/transcribe")

Option 2: API Server Only

When user only needs the API server:

• Use

  scripts/setup_api_server.py

  with desired model
• See

  references/api_server_guide.md

  for implementation details
• Template in

  assets/api-server/

Option 3: Plugin Only

When user has existing API and needs LiveKit plugin:

• Use

  scripts/setup_plugin.py

  with plugin name
• See

  references/plugin_implementation.md

  for details
• Template in

  assets/plugin-template/

Model Selection

Help user choose the right model:

openai/whisper-large-v3

openai/whisper-medium

openai/whisper-small

openai/whisper-tiny

facebook/wav2vec2-large-960h

For detailed comparison and optimization tips, see

references/models_comparison.md

Implementation Workflow

Building the API Server

1. Use the template: Start with

   assets/api-server/main.py

2. Key components:

   • FastAPI app with WebSocket endpoint
   • Model loading at startup (kept in memory)
   • Audio buffer management
   • WebSocket protocol for streaming

3. Customization points:

   • Model selection (change

     MODEL_ID

     in .env)
   • Audio processing parameters
   • Batch size and optimization
   • Error handling

For complete implementation guide, see

references/api_server_guide.md

Building the LiveKit Plugin

1. Use the template: Start with

   assets/plugin-template/

2. Required implementations:

   • _recognize_impl()

     - Non-streaming recognition

   • stream()

     - Return SpeechStream instance

   • SpeechStream

     class - Handle streaming

3. Key considerations:

   • Audio format conversion (16kHz, mono, 16-bit PCM)
   • WebSocket connection management
   • Event emission (interim/final transcripts)
   • Error handling and cleanup

For complete implementation guide, see

references/plugin_implementation.md

Deployment

Development

# API Server
uvicorn main:app --host 0.0.0.0 --port 8000 --reload

# Test WebSocket
ws://localhost:8000/ws/transcribe

Production

Docker (Recommended):

docker-compose up

Kubernetes: Use manifests in deployment guide

Cloud Platforms: AWS ECS, GCP Cloud Run, Azure Container Instances

For complete deployment guide including scaling, monitoring, and security, see

references/deployment.md

WebSocket Protocol

Client → Server

• Audio: Binary (16-bit PCM, 16kHz)
• Config:

  {"type": "config", "language": "en"}

• End:

  {"type": "end"}

Server → Client

• Interim:

  {"type": "interim", "text": "..."}

• Final:

  {"type": "final", "text": "...", "language": "en"}

• Error:

  {"type": "error", "message": "..."}

Common Tasks

Change Model

Edit

.env

MODEL_ID=openai/whisper-small  # Faster model

Add Language Support

In plugin usage:

stt=custom_stt.STT(language="es")  # Spanish
stt=custom_stt.STT(detect_language=True)  # Auto-detect

Enable GPU

In API server:

DEVICE=cuda:0  # Use GPU

Scale Horizontally

Deploy multiple API server instances behind load balancer. See

references/deployment.md

Troubleshooting

Out of Memory

• Use smaller model (

  whisper-small

  or

  whisper-tiny

  )
• Reduce

  batch_size

  in pipeline
• Enable

  low_cpu_mem_usage=True

Slow Transcription

• Ensure GPU is enabled (

  DEVICE=cuda:0

  )
• Use FP16 precision (automatic on GPU)
• Increase

  batch_size

• Use smaller model

Connection Issues

• Verify WebSocket support in load balancer
• Check firewall rules
• Increase timeout settings

Scripts

• scripts/setup_api_server.py

  - Generate API server from template

• scripts/setup_plugin.py

  - Generate LiveKit plugin from template

References

Load these as needed for detailed information:

• references/api_server_guide.md

  - Complete API implementation guide

• references/plugin_implementation.md

  - LiveKit plugin development

• references/models_comparison.md

  - Model selection and optimization

• references/deployment.md

  - Production deployment best practices

Assets

Ready-to-use templates:

• assets/api-server/

  - Complete FastAPI server with Whisper

• assets/plugin-template/

  - LiveKit STT plugin structure

Best Practices

1. Keep models in memory - Load once at startup, not per request
2. Use appropriate model size - Balance quality vs. speed for your use case
3. Process audio in chunks - 1-second chunks work well for streaming
4. Implement proper cleanup - Close WebSocket connections gracefully
5. Monitor metrics - Track latency, throughput, GPU utilization
6. Use Docker - Ensures consistent deployments
7. Enable authentication - Secure production APIs
8. Scale horizontally - Use load balancer for high availability