Llamafile

Configure and manage Mozilla Llamafile - a cross-platform executable distribution format that runs LLMs locally with an OpenAI-compatible API.

When to Use This Skill

Use this skill when:

• Installing llamafile binary and GGUF model files
• Starting llamafile server with optimal configuration
• Integrating llamafile with LiteLLM or OpenAI SDK
• Configuring llamafile for different performance profiles (GPU, CPU, network access)
• Troubleshooting llamafile server startup or API connection issues
• Building applications requiring local LLM inference
• Setting up commit message tools, code review systems, or other developer tools with local AI
• Managing llamafile as a background service
• Selecting and downloading appropriate GGUF models
• Validating OpenAI-compatible API responses

Core Capabilities

What Llamafile Provides

Llamafile combines llama.cpp with Cosmopolitan Libc to create single-file executables that:

• Run on macOS, Windows, Linux, FreeBSD, OpenBSD, NetBSD
• Support AMD64 and ARM64 architectures
• Serve OpenAI-compatible HTTP API on localhost
• Load GGUF model files for inference
• Provide

  /health

  endpoint for monitoring
• Support GPU acceleration (CUDA, Metal, Vulkan)
• Enable embeddings generation with

  --embedding

  flag

API Compatibility

Llamafile exposes these OpenAI-compatible endpoints when running with

--server

http://localhost:8080/v1/chat/completions

http://localhost:8080/v1/completions

http://localhost:8080/v1/embeddings

--embedding

http://localhost:8080/health

Critical Detail: All OpenAI-compatible endpoints require

/v1

Installation

Download Llamafile Binary

# Download llamafile v0.9.3 binary
curl -L -o llamafile https://github.com/mozilla-ai/llamafile/releases/download/0.9.3/llamafile-0.9.3

# Make executable
chmod 755 llamafile

# Verify version
./llamafile --version

Alternative download sources:

• GitHub Release:

  https://github.com/mozilla-ai/llamafile/releases/download/0.9.3/llamafile-0.9.3

• SourceForge Mirror:

  https://sourceforge.net/projects/llamafile.mirror/files/0.9.3/

Download GGUF Model

Llamafile requires GGUF format models. Download from Hugging Face:

# Recommended: Gemma 3 3B (balanced speed/quality, ~2GB)
curl -L -o gemma-3-3b.gguf \
  https://huggingface.co/Mozilla/gemma-3-3b-it-gguf/resolve/main/gemma-3-3b-it-Q4_K_M.gguf

# Alternative: Pre-packaged llamafile with embedded model
curl -LO https://huggingface.co/Mozilla/llava-v1.5-7b-llamafile/resolve/main/llava-v1.5-7b-q4.llamafile
chmod +x llava-v1.5-7b-q4.llamafile

Recommended models by use case:

Quantization recommendation: Use Q4_K_M quantized models for optimal balance of quality and performance.

Server Configuration

Basic Server Command

Start llamafile server for local API access:

./llamafile --server \
    -m /path/to/model.gguf \
    --nobrowser \
    --port 8080 \
    --host 127.0.0.1

Critical flags explained:

• --server

  : Required to enable HTTP API endpoints

• -m

  : Path to GGUF model file (required)

• --nobrowser

  : Prevents auto-opening browser on startup

• --port 8080

  : Default port (note: NOT 8000)

• --host 127.0.0.1

  : Localhost only (secure default)

Performance-Optimized Configuration

For GPU-accelerated inference with higher throughput:

./llamafile --server \
    -m /path/to/model.gguf \
    --nobrowser \
    --port 8080 \
    --host 127.0.0.1 \
    --ctx-size 4096 \
    --n-gpu-layers 99 \
    --threads 8 \
    --cont-batching \
    --parallel 4

Advanced flags:

--ctx-size

--n-gpu-layers

--threads

--threads-batch

--threads

--cont-batching

--parallel

--mlock

--embedding

/v1/embeddings

Network-Accessible Configuration

To allow connections from other machines (development/testing only):

./llamafile --server \
    -m /path/to/model.gguf \
    --nobrowser \
    --host 0.0.0.0 \
    --port 8080

Security warning: Binding to

0.0.0.0

API Integration

Using LiteLLM (Recommended)

LiteLLM provides unified interface for llamafile and cloud LLM providers.

import litellm

response = litellm.completion(
    model="llamafile/gemma-3-3b",  # MUST use llamafile/ prefix
    messages=[{"role": "user", "content": "Hello, world!"}],
    api_base="http://localhost:8080/v1",  # MUST include /v1 suffix
    temperature=0.3,
    max_tokens=200
)

print(response.choices[0].message.content)

Critical requirements for LiteLLM:

1. Model name MUST use

   llamafile/

   prefix for routing

2. api_base

   MUST include

   /v1

   suffix
3. No API key required (any placeholder value works)

Related skill: For comprehensive LiteLLM configuration, activate the litellm skill:

Skill(command: "litellm")

Using OpenAI Python SDK

Direct integration with OpenAI SDK for llamafile endpoints:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",  # MUST include /v1
    api_key="sk-no-key-required"  # Any value works
)

response = client.chat.completions.create(
    model="local-model",  # Model name is flexible
    messages=[
        {"role": "user", "content": "Hello, world!"}
    ],
    temperature=0.3,
    max_tokens=200
)

print(response.choices[0].message.content)

Using curl for Testing

Verify llamafile server is responding correctly:

# Health check
curl http://localhost:8080/health

# Chat completions
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0.3,
    "max_tokens": 200
  }'

# Embeddings (requires --embedding flag on server)
curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "input": ["Hello world"]
  }'

Server Management

Process Management Script

Python script to start llamafile as background process with health checking:

import subprocess
import time
import httpx

def start_llamafile(
    llamafile_path: str,
    model_path: str,
    port: int = 8080,
    host: str = "127.0.0.1"
) -> subprocess.Popen:
    """Start llamafile server as background process."""
    cmd = [
        llamafile_path,
        "--server",
        "-m", model_path,
        "--nobrowser",
        "--port", str(port),
        "--host", host,
    ]
    process = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    _wait_for_server(host, port)
    return process


def _wait_for_server(host: str, port: int, timeout: int = 30) -> None:
    """Wait for server to respond to health checks."""
    url = f"http://{host}:{port}/health"
    start = time.time()
    while time.time() - start < timeout:
        try:
            response = httpx.get(url, timeout=2)
            if response.status_code == 200:
                return
        except httpx.RequestError:
            pass
        time.sleep(0.5)
    raise TimeoutError(f"Server did not start within {timeout} seconds")

Configuration File Pattern

Example TOML configuration for applications using llamafile:

# ~/.config/app-name/config.toml
[ai]
model = "llamafile/gemma-3-3b"  # Must use llamafile/ prefix
temperature = 0.3
max_tokens = 200

[llamafile]
path = "/home/user/.local/bin/llamafile"
model_path = "/home/user/.local/share/app-name/models/gemma-3-3b.gguf"
api_base = "http://127.0.0.1:8080/v1"  # Include /v1 suffix

Troubleshooting

Server Fails to Start

Check if port is already in use:

# Find process using port 8080
lsof -i :8080

# Kill existing process
kill $(lsof -t -i :8080)

Verify model file exists and is readable:

ls -lh /path/to/model.gguf

Check llamafile binary permissions:

ls -la /path/to/llamafile
# Should show: -rwxr-xr-x (executable)

# Fix permissions if needed
chmod 755 /path/to/llamafile

Connection Refused Errors

Verify server is running:

# Check health endpoint
curl http://localhost:8080/health

# Check server is listening
netstat -tlnp | grep 8080
# or
lsof -i :8080

Common causes:

1. Server not started with

   --server

   flag
2. Wrong port number (8080 vs 8000)
3. Missing

   /v1

   in API URL path
4. Server bound to

   127.0.0.1

   but accessing from another machine

API Errors

Test basic connectivity:

# Verbose health check
curl -v http://localhost:8080/health

# Test chat completions with verbose output
curl -v http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"test","messages":[{"role":"user","content":"Hi"}]}'

Common API issues:

/v1

/v1

--server

-m

Performance Issues

Optimize inference speed:

1. Use quantized models (Q4_K_M recommended)
2. Enable GPU acceleration:

   --n-gpu-layers 99

3. Increase threads:

   --threads 8

4. Enable continuous batching:

   --cont-batching

5. Reduce context size if not needed:

   --ctx-size 2048

Check GPU availability:

# NVIDIA GPU
nvidia-smi

# AMD GPU
rocm-smi

# Apple Metal (check activity monitor)

Common Pitfalls

Avoid these frequent errors when using llamafile:

1. Port 8000 vs 8080: Llamafile defaults to port 8080, not 8000
2. Missing

   /v1

   in API URL: Always include

   /v1

   suffix for OpenAI-compatible endpoints
3. LiteLLM prefix: Must use

   llamafile/

   prefix in model name for proper routing
4. API key confusion: No real API key needed, but some clients require placeholder value
5. Starting server from hooks: Application hooks should check if server is running, not start it
6. Model path issues: Ensure GGUF file exists and is readable before starting server
7. Binary permissions: Llamafile must be executable (

   chmod 755

   )
8. GPU layers on CPU: Setting

   --n-gpu-layers

   on CPU-only systems causes errors

Version Information

Current stable version: 0.9.3 (May 14, 2025)

Version constants:

LLAMAFILE_MAJOR = 0
LLAMAFILE_MINOR = 9
LLAMAFILE_PATCH = 3

Recent changes in 0.9.3:

• Added Phi4 model support
• Added Qwen3 model support
• Respects NO_COLOR environment variable
• Fixed URL handling in JavaScript (preserves path when building relative URLs)
• Added Plaintext output option to LocalScore

Related Skills and Tools

Skills to activate:

• litellm

  - For unified LLM provider interface and routing

  Skill(command: "litellm")
  

External tools:

• LiteLLM - Unified interface for multiple LLM providers
• OpenAI Python SDK - Direct OpenAI-compatible API access
• llama.cpp - Underlying inference engine
• GGUF format - Model format specification

References

Official Documentation

• Mozilla llamafile GitHub - Primary repository and source code
• Mozilla llamafile Documentation - Official documentation site
• LiteLLM llamafile Provider - LiteLLM integration guide
• llama.cpp Server Documentation - Underlying server implementation
• Releases Page - Binary downloads and changelog

Model Resources

• Hugging Face Mozilla Models - Official Mozilla GGUF models
• GGUF Format Specification - Model file format details

Related Technologies

• Cosmopolitan Libc - Cross-platform binary format
• llama.cpp - LLM inference engine
• OpenAI API Reference - API compatibility reference