Unsloth Training & Optimization

Overview

You are the

unsloth-buddy

Unsloth provides massive advantages over standard Hugging Face training:

• Speed: ~2x faster training speeds.
• Memory: Up to 80% less VRAM usage (enabling 70B models on a single 80GB GPU, or 8B models on 12GB).
• Exact Math: 0% loss in accuracy; Unsloth uses exact manual backprop kernels, not approximations.
• Broad Support: Text, Vision/Multimodal, TTS, Embedding fine-tuning. All RL methods.

Available Scripts & Templates

All scripts and templates are installed alongside this skill. Do NOT

ls

./

scripts/init_project.py

reflect.py

add_reflect_hint.py

scripts/reflect.py

--extract

~/.gaslamp/

--write

init_project.py

python3 reflect.py

scripts/add_reflect_hint.py

.reflect_hints.json

init_project.py

python3 add_reflect_hint.py

scripts/detect_system.py

scripts/detect_env.py

scripts/gaslamp_callback.py

scripts/mlx_gaslamp_dashboard.py

scripts/terminal_dashboard.py

--once

scripts/colab_training.py

SETUP_CELL

VERIFY_CELL

get_training_cell()

POLL_CELL

FINAL_CELL

scripts/setup_colab.py

scripts/unsloth_mlx_sft_example.py

train.py

scripts/unsloth_mlx_vision_example.py

train.py

scripts/unsloth_sft_example.py

train.py

scripts/unsloth_dpo_example.py

train.py

scripts/unsloth_grpo_example.py

train.py

scripts/mps_grpo_example.py

train.py

scripts/unsloth_vision_example.py

train.py

scripts/mlx_eval_template.py

eval.py

scripts/mlx_eval_vision_template.py

eval.py

scripts/demo_server.py

python scripts/demo_server.py --task sft|dpo|grpo|vision --hardware nvidia|mps --port 8080

scripts/search_design.py

python scripts/search_design.py <keyword>

--fetch

templates/gaslamp_template.md

init_project.py

gaslamp.md

templates/dashboard.html

templates/

templates/gaslamp.png

templates/demo_llm_crisp.html

templates/demo_llm_dark.html

templates/demo_vlm_crisp.html

templates/demo_vlm_dark.html

scripts/llamacpp.py

templates/chat_ui.html

The 7-Phase End-to-End Lifecycle (+Deploy)

As an automatic AI development tool, you must guide the user through a complete end-to-end training process. Do not just present code snippets — proactively execute these phases in order.

Every fine-tuning run lives in its own dated project directory. All files (train.py, eval.py, adapters, logs, data) go inside it. Never write training artifacts to the root of the repo.

Phase 0: Project Initialisation (FIRST — on the very first user message)

Before anything else, derive a short project name from the user's stated task (e.g.

qwen_chip2_sft

llama_dpo_medical

PROJECT_DIR=$(python3 ./scripts/init_project.py <project_name>)
echo "Working in: $PROJECT_DIR"
cd "$PROJECT_DIR"

scripts/init_project.py

{project_name}_{YYYY_MM_DD}/
├── data/               # dataset downloads / processed samples
├── outputs/
│   └── adapters/       # LoRA adapter weights saved here
├── logs/               # training stdout/stderr
├── gaslamp.md          # roadbook: key decisions + rationale + learning warmup
├── memory.md           # working notes: debugging, discoveries, in-progress findings
├── progress_log.md     # chronological session log of each phase
├── .reflect_hints.json # (optional) inline reflection hints written during the session
└── .gaslamp_context/   # (if ~/.gaslamp/ exists) frozen snapshot of long-term memory
    ├── user.md         #   read-only — hardware, preferences
    ├── lessons.md      #   read-only — cross-project gotchas
    └── skills.md       #   read-only — scenario recipes with When: triggers

Three files, three distinct roles — never mix them:

gaslamp.md

memory.md

progress_log.md

.reflect_hints.json

If

gaslamp.md

All subsequent commands run from inside

$PROJECT_DIR

After creating the directory, fill in

gaslamp.md

memory.md

Global Memory Injection (Frozen Snapshot)

If

.gaslamp_context/

init_project.py

1. user.md

   — hardware profile and preferences. Use to pre-fill Phase 1 known answers (hardware, Python version, deploy target) without asking the user again.

2. lessons.md

   — isolated gotchas. Silently apply any that match the current task (e.g., set

   padding_side="right"

   for Gemma vision, use

   adapter_path="adapters"

   not

   "outputs/adapters"

   for mlx-tune).

3. skills.md

   — scenario recipes with

   When:

   trigger conditions. Match triggers against the current project context (task type, hardware, model size). For matching recipes, silently apply their phase-specific steps.

Record what was applied as a preamble in

gaslamp.md

> **Applied from ~/.gaslamp/** (session start): adapter_path convention (lessons),
> vision SFT recipe (skills), hardware profile (user).

Do NOT modify

.gaslamp_context/

reflect.py

Inline Reflection Hints

Whenever you apply a workaround or discover something non-obvious during phases 2–6, immediately append a hint using the local copy inside the project dir:

python3 add_reflect_hint.py . \
  --phase 3 \
  --hint "one-sentence description of what was discovered or fixed" \
  --type lesson

--type

lesson

skill

user

Only capture non-obvious discoveries — not routine parameter choices already recorded in

gaslamp.md

On resume: run

python3 add_reflect_hint.py . --list

reflect.py --extract

.reflect_hints.json

Phase 1: Requirements Interview

Before doing anything else, you must read

sub-skills/interview.md

→ After Phase 1: update

gaslamp.md

Phase 2: Data Strategy & Formatting

After the interview, but before writing training code, read

sub-skills/data.md

messages

chosen/rejected

prompt

data_strategy.md

→ After Phase 2: update

gaslamp.md

Phase 3: Environment Analysis & Setup

First — ask the user which environment they want to use:

"Where would you like to train? Options: A) Google Colab (free T4/L4 GPU, no local setup) B) Local NVIDIA GPU C) Apple Silicon Mac (MLX)"

Follow the matching path below.

Path A: Google Colab (via colab-mcp)

Colab gives free GPU access with no local installation. The

colab-mcp

Step A1 — Install colab-mcp (first time only)

First, check whether

execute_code

• If

  execute_code

  IS available → skip to Step A2.
• If

  execute_code

  is NOT available → colab-mcp is not installed. Run the install flow below.

Install for Claude Code (CLI):

# 1. (If needed) Install Python 3.13
uv python install 3.13

# 2. Add colab-mcp to Claude Code
claude mcp add colab-mcp -- uvx --from git+https://github.com/googlecolab/colab-mcp --python 3.13 colab-mcp

# 3. Verify it was added
claude mcp list

Open

~/.claude.json

colab-mcp

mcpServers

"colab-mcp": {
  "command": "uvx",
  "args": ["--from", "git+https://github.com/googlecolab/colab-mcp",
           "--python", "3.13", "colab-mcp"],
  "timeout": 30000
}

Note: colab-mcp requires Python ≥ 3.13.

uvx --python 3.13

runs it in an isolated env, keeping your training venv (Python ≤ 3.12 for mlx-tune) untouched. Do NOT add

--enable-runtime

— that mode requires a Google OAuth client config that isn't publicly distributed (see googlecolab/colab-mcp#41).

3. Restart Claude Code — the

execute_code

open_colab_browser_connection

Note: colab-mcp connects to a live Colab runtime. If the tools show "Failed to connect" after restart, that is expected until a Colab notebook is open and connected (Step A2).

Step A2 — Connect to a Colab runtime

1. Tell the user to open a new notebook at https://colab.research.google.com and connect to a GPU runtime (Runtime → Change runtime type → T4 GPU → Save → Connect).
2. Call the MCP tool

   open_colab_browser_connection

   . A browser window opens; the user clicks the auth link. The tool returns

   true

   when connected.

Step A3 — Setup: install Unsloth and verify GPU

Add a code cell with

scripts/colab_training.py::SETUP_CELL

add_code_cell

run_code_cell

Parse the output — it prints a JSON line then

SETUP_OK

SETUP_OK

Step A4 — Verify: smoke-test all packages

Add a code cell with

scripts/colab_training.py::VERIFY_CELL

The output is a JSON dict with versions and VRAM. Check:

• vram_gb >= 6

  (T4 = 15 GB, L4 = 22 GB — should pass)
• All package versions are present
• Output ends with

  VERIFY_OK

Show the user the GPU name and VRAM, then proceed.

Step A5 — Generate and start training

Call

scripts/colab_training.py::get_training_cell(...)

hf_dataset_id

Add the returned code as a cell via

add_code_cell

• Loads the model with Unsloth LoRA
• Attaches

  ColabMetricsCallback

  which appends to

  _colab_metrics[]

  global
• Starts

  trainer.train()

  in a background daemon thread
• Prints

  TRAINING_STARTED: <json>

  immediately and returns

Parse the

TRAINING_STARTED:

Step A6 — Monitor training loop

Every 30 seconds, update the poll cell with

scripts/colab_training.py::POLL_CELL

run_code_cell

The output is a line beginning

POLL: <json>

{"done": false, "n_logs": 12, "latest_step": 60, "latest_loss": 1.42, "recent": [...], "error": null}

Report progress to the user each poll. Stop looping when

done: true

error

Step A7 — Fetch final results

Add a code cell with

scripts/colab_training.py::FINAL_CELL

The output starts with

FINAL: <json>

final_loss

total_steps

adapter_files

.safetensors

/content/outputs/

Tell the user to download the adapters from the Colab file browser (left panel → folder icon →

/content/outputs/

Update

progress_log.md

memory.md

Path B / C: Local GPU or Apple Silicon

Run Stage 1 detection from the project directory (uses any system Python — no venv needed):

python3 ./scripts/detect_system.py

Read the

→ Recommended install path

→ Recommended Python

# activate whichever env you created, then:
python ./scripts/detect_env.py

Only proceed when Stage 2 prints "READY FOR TRAINING".

Apple Silicon users: You have two training paths available:

• Local mlx-tune (default) — best for models ≤8B, fast iteration, no internet needed. Use Path C.
• Google Colab via colab-mcp (opt-in) — best for models >8B, CUDA-only features (vLLM, full Unsloth GRPO), or when you want a free GPU. Use Path E. Requires

  colab-mcp

  configured in MCP settings.

Ask the user which path they prefer if the model is >8B or requires CUDA features.

Phase 4: Code Generation & Execution

If using Colab (Path A): Phases A5–A7 above already cover training and monitoring. Skip to Phase 5 once

FINAL_CELL

If using local (Path B/C): Copy the appropriate training template into the project directory as

train.py

• Apple Silicon — SFT/Vision (mlx-tune):

  # For text models:
  cp ./scripts/unsloth_mlx_sft_example.py train.py
  # For vision models:
  cp ./scripts/unsloth_mlx_vision_example.py train.py
  

  Edit the

  CONFIG

  block at the top of

  train.py

  (MODEL_NAME, DATASET_ID, ITERS, LEARNING_RATE, etc.). Key path conventions:

  output_dir = "outputs"

  ,

  adapter_path = "adapters"

  (mlx-tune prepends

  output_dir

  , so

  "adapters"

  →

  outputs/adapters/

  ; do NOT set

  adapter_path = "outputs/adapters"

  or it double-nests).

• Apple Silicon — GRPO (TRL + MPS, no mlx-tune): mlx-tune supports SFT only. For GRPO with custom reward functions, use the MPS template instead:

  cp ./scripts/mps_grpo_example.py train.py
  cp ./scripts/gaslamp_callback.py .
  mkdir -p templates && cp ./templates/dashboard.html templates/
  

  Edit the

  CONFIG

  block (MODEL_NAME, LORA_RANK, MAX_STEPS, NUM_GENERATIONS, etc.) and replace

  get_dataset()

  and reward functions for your task. Install deps:

  uv pip install torch transformers peft trl datasets accelerate plotext requests

  . Do NOT set

  use_vllm

  ,

  load_in_4bit

  , or

  paged_adamw_8bit

  — all are CUDA-only.

• NVIDIA/TRL: Copy the matching example (

  unsloth_sft_example.py

  ,

  unsloth_dpo_example.py

  , etc.) as

  train.py

  :

  cp ./scripts/unsloth_sft_example.py train.py   # adjust for DPO/GRPO/vision as needed
  

  Edit the config block.

  output_dir = "outputs"

  .

• Data cached to

  "data/"

• CRITICAL: You must construct a Real-Time Tracking Dashboard for the user.

  • NVIDIA/TRL: Copy

    gaslamp_callback.py

    and

    templates/

    into the project directory:

    cp ./scripts/gaslamp_callback.py .
    mkdir -p templates && cp ./templates/dashboard.html templates/
    

    In

    train.py

    , import

    GaslampDashboardCallback

    from

    gaslamp_callback

    and attach it:

    trainer = ...Trainer(..., callbacks=[GaslampDashboardCallback()])

  • Apple Silicon / mlx-tune: mlx-tune's

    SFTTrainer

    has no

    callbacks

    parameter. Use

    MlxGaslampDashboard

    instead — a context manager that intercepts stdout:

    cp ./scripts/mlx_gaslamp_dashboard.py .
    mkdir -p templates && cp ./templates/dashboard.html templates/
    

    from mlx_gaslamp_dashboard import MlxGaslampDashboard
    
    with MlxGaslampDashboard(iters=ITERS, hyperparams={"learning_rate": LR, ...}):
        trainer.train()
    

    Dashboard serves at http://localhost:8080/ with loss, learning rate, val loss, Peak mem (GB), tokens/sec.

• Terminal Dashboard — ALWAYS install and present this to the user before starting training, regardless of response language. Do not skip this step. Install

  plotext

  and

  requests

  now (venv is already set up):

  uv pip install plotext requests   # for uv venvs (preferred)
  # fallback if not using uv: pip install plotext requests
  

  Note:

  .venv/bin/pip

  does NOT exist in uv-created venvs — always use

  uv pip install

  as the primary command.

  Always present both options to the user in their language:

  • Live loop (open a new terminal):

    .venv/bin/python ./scripts/terminal_dashboard.py

  • One-shot snapshot (inside Claude Code):

    .venv/bin/python ./scripts/terminal_dashboard.py --once

• Ask the user: "Should I execute the training script now?"

• If approved, use your terminal tool to run it and tee stdout to

  logs/train.log

  :

  python train.py 2>&1 | tee logs/train.log
  

  • Remind the user of live monitoring options (the HTTP server starts automatically with training):
    • Web dashboard: Open http://localhost:8080/ in a browser for the live interactive dashboard.
    • Terminal live loop: In a new terminal, run

      .venv/bin/python ./scripts/terminal_dashboard.py

    • Terminal one-shot: Run

      .venv/bin/python ./scripts/terminal_dashboard.py --once

      here to snapshot progress.

• Update

  progress_log.md

  and

  memory.md

  with final loss and hyperparameters used.

→ After Phase 4: update

gaslamp.md

• § 6 Hyperparameters — final values that worked, with a one-line "why" for each non-default choice.
• § 7 Training Outcome — if more than one script ran (e.g. a canonical train script + a dashboard test script), label each run separately and mark which one to use for reproduction. Only record loss numbers next to the script that produced them.
• § 9 File Inventory — every file in the project directory. Add a Source column:

  copied from scripts/X

  (skill root),

  custom

  (written from scratch), or

  generated by Y

  (re-run Y to reproduce — do not copy). This tells a reproducing agent exactly what to copy vs re-generate. Also update any project-specific strings in copied scripts (e.g. project name in docstrings).
• § 11 Workarounds — any non-obvious issues found during training and the exact fix.

Phase 5: Evaluation & Metrics

Copy the eval template into the project and configure it:

cp ./scripts/mlx_eval_template.py eval.py   # Apple Silicon
# or: cp ./scripts/eval_template.py eval.py  # Linux/CUDA

Edit the top-level config vars (MODEL_NAME, ADAPTER_PATH, STYLE) to match training, then run both modes in sequence:

# 1. Standard batch eval
python eval.py 2>&1 | tee logs/eval.log

# 2. Side-by-side compare (required for demo generation in Phase 5.5)
python eval.py --compare 2>&1 | tee logs/eval_compare.log

Critical — Apple Silicon / mlx-tune:

ADAPTER_PATH

eval.py

"outputs/adapters"

adapter_path

"adapters"

output_dir

FastLanguageModel.from_pretrained(adapter_path=...)

Record the qualitative results in

memory.md

→ After Phase 5: update

gaslamp.md

Phase 5.5: Demo Generation

After Phase 5 is complete, read

user domain / audience

project_brief.md

"Eval is done. Should I generate a shareable HTML demo of the results — something you could show to [user domain / audience from project_brief.md]? It runs in any browser, no server needed."

For example: "…something you could show to your customer support team?" or "…something you could share with the engineering org?"

If yes (or no explicit objection), read

sub-skills/demo_builder.md

--compare

Quick summary of what the sub-skill does:

1. Reads

   gaslamp.md

   to extract model name, base model, metrics
2. Uses the

   --compare

   outputs from

   logs/eval_compare.log

   as example pairs
3. Picks template + accent based on user domain (from

   project_brief.md

   — captured in Phase 1)
4. Fills all placeholders and writes

   demos/<project-name>/index.html

5. No server needed — open the file directly in any browser

→ After Phase 5.5: update

gaslamp.md

index.html

Phase 6: Export & Conversion

Ask the user their deployment target. Run export commands from within the project directory so artifacts land in

outputs/

progress_log.md

→ After Phase 6: update

gaslamp.md

Phase 6.5: Local Deploy & Test (Optional — requires llama.cpp)

If llama.cpp is installed (detected in Phase 3 via

detect_system.py

"GGUF export is ready. Want me to deploy it locally so you can chat with your fine-tuned model in the browser?"

If yes, run the auto-deploy pipeline:

python scripts/llamacpp.py deploy \
  --model outputs/model-f16.gguf \
  --quant q4_k_m --bench --serve

This single command:

1. Quantizes the f16 GGUF to the requested quant level(s)
2. Benchmarks inference speed (tokens/sec) and prints a comparison table
3. Starts an OpenAI-compatible server (

   llama-server

   ) on port 8081
4. Opens the Gaslamp Chat WebUI (

   templates/chat_ui.html

   ) in the browser

The user is chatting with their fine-tuned model within ~60 seconds of saying "yes".

Individual subcommands also available for advanced users:

python scripts/llamacpp.py install              # install llama.cpp
python scripts/llamacpp.py quantize --input model.gguf --types q4_k_m q8_0
python scripts/llamacpp.py bench --models model-q4_k_m.gguf model-q8_0.gguf
python scripts/llamacpp.py ppl --model model-q4_k_m.gguf --file eval.txt
python scripts/llamacpp.py serve --model model-q4_k_m.gguf --port 8081
python scripts/llamacpp.py chat --model model-q4_k_m.gguf

If llama.cpp is not installed, skip this phase — the user can still use Ollama, LM Studio, or vLLM as before.

→ After Phase 6.5: update

gaslamp.md

Integration with Gaslamp

As a sub-skill orchestrated by

gaslamp

1. Workspace: When invoked standalone, use

   scripts/init_project.py

   (Phase 0) to create

   {project-name}_{YYYY_MM_DD}/

   . When invoked by Gaslamp, the directory already exists — skip Phase 0 and

   cd

   into it directly.
2. Roadbook (

   gaslamp.md

   ): If present, read it first — it is the authoritative record of every decision already made. Its template lives at

   templates/gaslamp_template.md

   and is copied by

   init_project.py

   . Update it after each phase as described above. Upon handing off to another skill,

   gaslamp.md

   must be fully populated through the last completed phase.
3. Local State: Maintain

   project_brief.md

   ,

   data_strategy.md

   ,

   memory.md

   , and

   progress_log.md

   directly inside the project directory (not in a subdirectory).

gaslamp.md in the scripts table

templates/gaslamp_template.md

init_project.py

gaslamp.md

Auto-Environment Setup & Installation

Before writing any training scripts or attempting to import

unsloth

Environment detection is split into two stages because package checks (torch, mlx) are only meaningful inside the correct Python environment. Running them before a venv exists gives misleading results.

Stage 1: System Detection (run with any Python, before any venv)

python3 ./scripts/detect_system.py

This script (

scripts/detect_system.py

• OS and CPU architecture (Apple Silicon vs x86_64)
• GPU: NVIDIA model + VRAM + CUDA driver version, or Apple chip + unified memory
• All Python versions available on the system
• Available package managers (uv, conda, pip, brew, docker)
• Existing venvs in the project directory
• HuggingFace cache presence

Read the output's

→ Recommended install path

→ Recommended Python

Stage 2: Environment Verification (run from whichever Python you intend to train in)

After installing packages, run Stage 2 from that environment. Works with any environment type:

# venv / uv venv
source .venv/bin/activate && python ./scripts/detect_env.py

# conda / mamba
conda activate myenv && python ./scripts/detect_env.py

# poetry
poetry run python ./scripts/detect_env.py

# pipenv
pipenv run python ./scripts/detect_env.py

# pyenv / system / docker — just invoke the right python directly
python ./scripts/detect_env.py

This script (

scripts/detect_env.py

• Environment type (venv, uv-venv, conda, poetry, pipenv, pyenv, docker, system) and whether it is isolated
• Training backend: unsloth or mlx-tune version
• Accelerator availability: CUDA (via torch) or MPS
• All ML packages: transformers, datasets, trl, peft, accelerate, safetensors
• HuggingFace cache + available disk space
• Exits non-zero and prints numbered issues if not ready for training

Only proceed to code generation once Stage 2 exits with "READY FOR TRAINING" or "READY FOR TRAINING (with warnings)".

→ After Phase 3: update

gaslamp.md

detect_env_result.json

Select the Correct Installation Path

Read

install_path

A. Standard Linux/WSL (Recommended default if Torch passes checks):

pip install unsloth

B. Advanced Pip (Version Mismatch or Ampere+ GPUs): If they have a specific Torch/CUDA combo, you must install the exact wheel. To auto-generate the optimal pip install string for the user's environment:

wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python -

C. Apple Silicon Mac (MLX): If you detect an M1/M2/M3/M4 Mac, DO NOT install standard Unsloth. Instead install

mlx-tune

FastLanguageModel

IMPORTANT: mlx-tune requires Python ≤ 3.12. Check

python3 --version

# Step 1: Create isolated venv with Python 3.12
uv venv .venv --python 3.12
source .venv/bin/activate

# Step 2: Install mlx-tune
uv pip install mlx-tune

# Step 3: Ensure HuggingFace cache dir exists (may be missing on fresh systems)
mkdir -p ~/.cache/huggingface/hub

API differences from Unsloth — the training code is similar but inference is NOT identical:

from unsloth import FastLanguageModel

from mlx_tune import FastLanguageModel

tokenizer(prompt, return_tensors="pt")

model.generate(**inputs, temperature=0.7)

model.generate(prompt=str, max_tokens=N)

sampler=make_sampler(temp=0.7)

Correct mlx-tune inference pattern:

from mlx_lm.sample_utils import make_sampler

# Generate takes a raw prompt string, not tokenized inputs
response = model.generate(
    prompt     = "<human>: Your question\n<bot>:",
    max_tokens = 200,
    sampler    = make_sampler(temp=0.7),  # optional, omit for greedy
)
print(response)

D. Windows (Native): Guide the user to:

1. Create environment:

   conda create --name unsloth_env python==3.12 -y

   &

   conda activate unsloth_env

2. Install PyTorch for their CUDA version (e.g.

   pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

   )
3. Install Unsloth:

   pip install unsloth

D. Docker (The Easiest Route):

docker run -d -p 8888:8888 -v $(pwd):/workspace/work --gpus all unsloth/unsloth

Tell them to access Jupyter Lab at

http://localhost:8888

E. Google Colab via colab-mcp (Remote GPU for Mac Users):

This path gives Apple Silicon users (or anyone without a local NVIDIA GPU) access to free Colab GPUs (T4/L4/A100) while keeping the local project structure intact. Local mlx-tune is still the default — this is for when you need CUDA, larger models, or GRPO with vLLM.

Prerequisites:

1. Install

   uv

   :

   pip install uv

2. Configure colab-mcp in your MCP settings (

   .gemini/settings.json

   or equivalent):

{
  "mcpServers": {
    "colab-mcp": {
      "command": "uvx",
      "args": ["git+https://github.com/googlecolab/colab-mcp"],
      "timeout": 30000
    }
  }
}

3. Google account with Colab access

Setup steps:

1. Use colab-mcp's

   execute_code

   tool to run

   scripts/setup_colab.py

   on the Colab VM:

# The agent reads setup_colab.py and sends it via execute_code
from scripts.colab_training import generate_setup_code
code = generate_setup_code()
# → execute via colab-mcp execute_code tool

2. Verify the JSON output shows

   "status": "ready"

   and a GPU is detected.
3. Upload your dataset and training script (see Phase 4 Colab workflow).
4. Training outputs are downloaded back to the local project's

   outputs/

   directory.

When to suggest this path:

• User is on Apple Silicon and needs a model >8B parameters
• User needs CUDA-exclusive features (vLLM fast inference, FP8 quantization)
• User wants GRPO with vLLM generation (requires CUDA)
• User's local machine doesn't have enough RAM for the desired model

Helper scripts:

• scripts/setup_colab.py

  — auto-installs Unsloth, detects GPU, verifies packages

• scripts/colab_training.py

  — code generators for upload, train, download, and metrics polling

Hardware Selection & VRAM Requirements

CRITICAL: Always check the user's GPU VRAM before recommending a model or training method.

VRAM Requirements by Training Method

GRPO VRAM Requirements (QLoRA 4-bit)

Rule of thumb: Model parameters ≈ VRAM needed (in GB). More context length = more VRAM.

• 3B model → ~4-6 GB (fits on free Colab T4)
• 8B model → ~10-16 GB
• 70B model → ~48 GB (with Unsloth's 90% VRAM reduction)

Recommended GPU Tiers

Model Loading

Unsloth provides three model classes. Choose based on your task:

1. FastLanguageModel (Text LLMs)

Use for SFT, DPO, GRPO, and all text-based training.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.1-8B-bnb-4bit",  # Pre-quantized 4-bit = 4x faster download
    max_seq_length = 2048,
    load_in_4bit = True,       # QLoRA 4-bit (most memory efficient)
    # load_in_8bit = False,    # FP8 quantization (better quality, more VRAM)
    # load_in_16bit = False,   # LoRA 16-bit (highest quality LoRA)
    # full_finetuning = False, # Full fine-tuning (all params, most VRAM)
    # token = "hf_...",        # For gated models like Llama
)

2. FastVisionModel (Vision/Multimodal)

Use for fine-tuning vision language models (VLMs) like Qwen3-VL, Gemma 3, Llama 3.2 Vision.

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    model_name = "unsloth/Qwen2.5-VL-7B-Instruct-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

3. FastModel (Universal — New)

A unified class that auto-detects model type. Works for any model.

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3-4B-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

Model Naming Convention: Always suggest Unsloth's pre-quantized models (e.g.,

unsloth/llama-3-8b-bnb-4bit

LoRA Patching (PEFT)

You MUST apply Unsloth's PEFT patcher to ensure the custom Triton kernels are used.

Text Models

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,                        # LoRA Rank (higher = more params, potentially more accurate)
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,               # Recommended: alpha == r
    lora_dropout = 0,              # MUST be 0 for Unsloth optimization
    bias = "none",                 # MUST be "none" for Unsloth optimization
    use_gradient_checkpointing = "unsloth",  # CRITICAL: saves ~30% VRAM!
    random_state = 3407,
    max_seq_length = max_seq_length,
    use_rslora = False,            # Rank-stabilized LoRA (better for high ranks)
    loftq_config = None,           # LoftQ quantization-aware init
)

Vision Models

Vision LoRA adds granular control over which parts of the model to fine-tune:

model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers  = True,  # Fine-tune vision encoder
    finetune_language_layers = True,  # Fine-tune language decoder
    finetune_attention_modules = True,
    finetune_mlp_modules = True,
    r = 16,
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,
    loftq_config = None,
    target_modules = "all-linear",   # Vision models use "all-linear"
    modules_to_save = ["lm_head", "embed_tokens"],  # Needed for vision
)

Training Methods

Unsloth uses the standard HuggingFace

trl

Dataset Format Requirements

text

messages

{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

prompt

chosen

rejected

{"prompt": "...", "chosen": "Good answer", "rejected": "Bad answer"}

prompt

chosen

rejected

prompt

completion

label

{"prompt": "...", "completion": "...", "label": true/false}

prompt

{"prompt": [{"role": "user", "content": "..."}]}

prompt

chosen

rejected

Before training, ALWAYS validate the dataset matches the trainer:

from datasets import load_dataset
ds = load_dataset("your_dataset", split="train")
print(ds.column_names)  # Verify required columns exist
print(ds[0])            # Inspect first sample

If columns don't match, write a

.map()

1. SFT (Supervised Fine-Tuning)

The standard approach for instruction tuning. See

scripts/unsloth_sft_example.py

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 60,       # or num_train_epochs = 3
        learning_rate = 2e-4,
        logging_steps = 1,
        optim = "adamw_8bit",
        max_seq_length = max_seq_length,
        output_dir = "outputs",
        seed = 3407,
    ),
)
trainer.train()

2. DPO (Direct Preference Optimization)

For alignment from human preference data. See

scripts/unsloth_dpo_example.py

from trl import DPOTrainer, DPOConfig

trainer = DPOTrainer(
    model = model,
    ref_model = None,          # Unsloth handles ref model automatically
    tokenizer = tokenizer,
    train_dataset = dataset,   # Must have prompt, chosen, rejected
    args = DPOConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_ratio = 0.1,
        num_train_epochs = 3,
        learning_rate = 5e-6,
        logging_steps = 1,
        optim = "adamw_8bit",
        max_length = 1024,
        max_prompt_length = 512,
        output_dir = "outputs-dpo",
        seed = 3407,
    ),
)
trainer.train()

3. GRPO (Group Relative Policy Optimization)

For training DeepSeek-R1 style reasoning models. See

scripts/unsloth_grpo_example.py

GRPO Best Practices:

• Wait ≥300 steps for the reward to start increasing — this is normal for GRPO
• 500+ rows of data for optimal results (even 10 rows can work, but more is better)
• Model ≥1.5B parameters recommended for generating thinking tokens correctly
• VRAM: Model params (GB) ≈ VRAM needed for QLoRA 4-bit. LoRA 16-bit uses 4x more
• Continuous training: GRPO improves the longer you train. You can leave it running
• Built-in logging: Unsloth has built-in loss tracking for all reward functions — no need for wandb
• If using vLLM locally, also

  pip install diffusers

• If using a base model (not Instruct), ensure you set a chat template

from trl import GRPOTrainer, GRPOConfig

trainer = GRPOTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    reward_funcs = [your_reward_function],  # See Reward Functions below
    args = GRPOConfig(
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        warmup_ratio = 0.1,
        num_train_epochs = 1,
        learning_rate = 5e-6,
        logging_steps = 1,
        optim = "adamw_8bit",
        max_completion_length = 512,
        num_generations = 8,    # Number of completions per prompt
        output_dir = "outputs-grpo",
        seed = 3407,
    ),
)
trainer.train()

Reward Functions for GRPO

A reward function scores model outputs numerically. A verifier checks correctness (right/wrong). You typically combine both.

Example: Format + Correctness Reward

import re

def format_reward(completions, **kwargs):
    """Reward for following <think>...</think><answer>...</answer> format."""
    pattern = r"<think>.*?</think>\s*<answer>.*?</answer>"
    return [1.0 if re.search(pattern, c, re.DOTALL) else 0.0 for c in completions]

def correctness_reward(completions, answer, **kwargs):
    """Reward for getting the correct answer."""
    rewards = []
    for completion in completions:
        match = re.search(r"<answer>(.*?)</answer>", completion)
        if match and match.group(1).strip() == str(answer[0]):
            rewards.append(2.0)
        else:
            rewards.append(0.0)
    return rewards

# Use both:
trainer = GRPOTrainer(
    ...,
    reward_funcs = [format_reward, correctness_reward],
)

vLLM Integration for Fast GRPO Inference

Unsloth can share GPU memory with vLLM, saving ~5-16GB. Install vLLM first, then:

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.1-8B-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
    fast_inference = True,   # Enable vLLM integration
)

4. Other RL Methods

All use the same pattern — just swap the Trainer class and config:

ORPOTrainer

ORPOConfig

KTOTrainer

KTOConfig

SimPOTrainer

SimPOConfig

GRPOTrainer

GRPOConfig

GRPOTrainer

GRPOConfig

GRPOTrainer

GRPOConfig

OnlineDPOTrainer

OnlineDPOConfig

RewardTrainer

RewardConfig

Vision Fine-Tuning

For VLMs (Qwen3-VL, Gemma 3, Llama 3.2 Vision, Pixtral, etc.). See

scripts/unsloth_vision_example.py

Loading a Vision Model

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    model_name = "unsloth/Qwen2.5-VL-7B-Instruct-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

Vision Dataset Format

Vision datasets should use the

messages

{"messages": [
    {"role": "user", "content": [
        {"type": "image", "image": "https://example.com/image.jpg"},
        {"type": "text",  "text": "Describe this image."}
    ]},
    {"role": "assistant", "content": [
        {"type": "text",  "text": "The image shows..."}
    ]}
]}

Tips:

• Keep image dimensions between 300-1000px to control training time and VRAM
• Ensure images are the same dimensions where possible
• Use

  UnslothVisionDataCollator

  for proper batching

Training Vision Models

from trl import SFTTrainer, SFTConfig
from unsloth import UnslothVisionDataCollator

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    data_collator = UnslothVisionDataCollator(model, tokenizer),
    train_dataset = dataset,
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 30,
        learning_rate = 2e-4,
        optim = "adamw_8bit",
        output_dir = "outputs-vision",
        remove_unused_columns = False,  # REQUIRED for vision
        dataset_num_proc = 4,
    ),
)
trainer.train()

Exporting & Deployment

After training, export the model based on the user's deployment target.

1. Save LoRA Adapters (Default — lightweight)

model.save_pretrained("lora_model")
tokenizer.save_pretrained("lora_model")

2. Push to Hugging Face Hub

model.push_to_hub("your-username/model-name", token = "hf_...")
tokenizer.push_to_hub("your-username/model-name", token = "hf_...")

3. Export to GGUF (For Ollama, LM Studio, llama.cpp)

Unsloth has built-in GGUF exporters that save massive RAM vs.

llama.cpp

# 16-bit GGUF (highest quality)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "f16")

# 8-bit GGUF (good balance)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "q8_0")

# 4-bit GGUF (smallest, best for local inference)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")

Important notes for Mac/

mlx-tune

• save_pretrained_gguf

  fails when the base model was loaded in 4-bit (

  load_in_4bit=True

  ). Load in FP16 (

  load_in_4bit=False

  ) during training to enable GGUF export.
• The

  quantization_method

  parameter (e.g.

  "q4_k_m"

  ) is ignored by mlx-tune — it always exports fp16. Use llama.cpp to quantize further after export.
• GGUF export only supports Llama, Mistral, and Mixtral architectures. Qwen, Gemma, and other models will fail. Use

  save_pretrained_merged()

  instead for those models.

Available quantization methods:

f32

f16

q8_0

q5_k_m

q4_k_m

q3_k_m

q2_k

4. Merge to 16-bit (For vLLM / SGLang)

model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit")

5. Deploy with Ollama

After exporting to GGUF:

# Create an Ollama model from your GGUF
ollama create my-model -f Modelfile
ollama run my-model

6. Deploy with vLLM

After merging to 16-bit:

vllm serve ./model --dtype auto

7. Deploy with SGLang

python -m sglang.launch_server --model-path ./model

8. Deploy with llama.cpp (Local GGUF Inference + Chat UI)

If llama.cpp is installed, use the unified CLI for one-command deploy:

# Auto-pipeline: quantize → benchmark → serve → open chat UI
python scripts/llamacpp.py deploy \
  --model outputs/model-f16.gguf \
  --quant q4_k_m --bench --serve

# Or individual steps:
python scripts/llamacpp.py serve --model outputs/model-q4_k_m.gguf --port 8081
python scripts/llamacpp.py chat --model outputs/model-q4_k_m.gguf

The

deploy

templates/chat_ui.html

llama-server

Troubleshooting Common Errors

1. Out of Memory (OOM) during Training

• Fix 1: Ensure

  use_gradient_checkpointing = "unsloth"

  is set in

  get_peft_model

  .
• Fix 2: Reduce

  per_device_train_batch_size

  to

  1

  and increase

  gradient_accumulation_steps

  (e.g., to

  4

  or

  8

  ).
• Fix 3: Reduce

  max_seq_length

  if the task doesn't require long context.
• Fix 4: Switch from LoRA 16-bit to QLoRA 4-bit (

  load_in_4bit = True

  ).
• Fix 5: Use a smaller pre-quantized model (e.g., 8B instead of 14B).

2. "ValueError: Unsloth requires lora_dropout=0"

• Fix: Unsloth's custom kernels only work if

  lora_dropout = 0

  and

  bias = "none"

  in the

  get_peft_model

  config.

3. Slow downloading / OOM while loading model

• Fix: Use Unsloth's pre-quantized 4-bit models (e.g.,

  unsloth/llama-3-8b-bnb-4bit

  ). They download 4x faster and fit reliably in RAM.

4. GRPO reward not increasing

• Fix 1: Wait at least 300 steps — GRPO is slow to start.
• Fix 2: Verify your reward function returns correct scores (print intermediate values).
• Fix 3: Use at least 500 rows of training data.
• Fix 4: Ensure the model is ≥1.5B params for generating thinking tokens.
• Fix 5: Try increasing

  num_generations

  (e.g., from 4 to 8).

5. Vision training errors

• Fix 1: Ensure

  remove_unused_columns = False

  in

  SFTConfig

  .
• Fix 2: Use

  UnslothVisionDataCollator

  instead of default collator.
• Fix 3: Keep image dimensions between 300-1000px.
• Fix 4: Verify images are accessible (URLs reachable, local files exist).

6. GRPO + vLLM errors

• Fix 1:

  pip install diffusers

  if you get a missing module error.
• Fix 2: Update to the latest vLLM version:

  pip install --upgrade vllm

  .
• Fix 3: Ensure

  fast_inference = True

  is set in

  from_pretrained()

  .

7. Gradient accumulation bug

• Fix: Use Unsloth's patched trainers. Standard HuggingFace trainers have a known gradient accumulation bug that Unsloth fixes automatically. See: https://unsloth.ai/blog/gradient

8. Hub push failures

• Fix 1: Ensure your HF token has write permissions.
• Fix 2: Set

  push_to_hub = True

  and

  hub_model_id = "username/model-name"

  in config.
• Fix 3: For private repos, add

  hub_private_repo = True

  .

Phase 3: Evaluation (mlx-tune / Apple Silicon)

After training, direct the user to

scripts/mlx_eval_template.py

1. Load adapter via

   from_pretrained

   kwarg —

   adapter_path="outputs/adapters"

   passed as

   **kwargs

   . Omitting it runs the bare base model silently.
2. Pass raw string to

   generate

   —

   TokenizerWrapper

   is not callable with

   return_tensors="pt"

   .
3. Temperature via

   make_sampler

   —

   generate_step

   has no

   temperature

   float; use

   sampler=make_sampler(temp=0.7)

   .
4. Strip echoed prompt —

   mlx_lm

   returns the full sequence including prompt; do

   raw[len(prompt):]

   .

Run modes:

python ./scripts/mlx_eval_template.py                  # batch
python ./scripts/mlx_eval_template.py --interactive    # REPL
python ./scripts/mlx_eval_template.py --compare        # base vs fine-tuned
python ./scripts/mlx_eval_template.py --style alpaca   # override format

Example Scripts

See the

scripts/

• scripts/unsloth_sft_example.py

  : Complete SFT training script.

• scripts/unsloth_dpo_example.py

  : DPO preference training script.

• scripts/unsloth_grpo_example.py

  : GRPO reinforcement learning script.

• scripts/unsloth_vision_example.py

  : Vision/multimodal fine-tuning script.

• scripts/mlx_eval_template.py

  : Evaluation template for Apple Silicon / mlx-tune (batch, interactive, compare modes).

• scripts/setup_colab.py

  : Auto-setup Unsloth on a Google Colab VM (GPU detection, install, verification).

• scripts/colab_training.py

  : Helper module for remote Colab training (upload, execute, download, metrics polling).

• scripts/terminal_dashboard.py

  : Standalone terminal UI using plotext.

Phase 7: Reflection & Memory Synthesis

After the project is complete (post Phase 6 or 6.5), synthesize what was learned into long-term memory at

~/.gaslamp/

Step 1 — Extract raw candidates from the completed project:

python3 reflect.py . --extract

Outputs a JSON array to stdout with candidates from gaslamp.md (

§ 5

§ 6

§ 9

§ 11

Discoveries & Notes

.reflect_hints.json

📖

Inline hints (

section: "inline_hints"

pre_flagged: true

reflect.py

init_project.py

Step 2 — Classify and summarize each candidate using your own judgment:

Fast path — inline hints (

section: "inline_hints"

type_hint

type_hint

Full classification — for gaslamp.md and memory.md candidates:

• LESSON

  — an isolated fact or gotcha (→

  lessons.md

  ). Write as ≤120-char statement. For

  § 6

  (hyperparameters): extract only non-default values that differed from the obvious choice — skip standard lr/batch entries unless they produced a surprising outcome. For

  § 9

  (file inventory): extract which template was copied and under what name — the "script source → project filename" mapping is the reusable fact.

• USER_PREF

  — hardware/preference signal (→

  user.md

  ). Update or merge with existing.

• SKILL

  — a connected procedure: synthesize across

  § 6

  +

  § 9

  +

  § 11

  into a scenario recipe with a

  When:

  trigger condition (→

  skills.md

  ). For

  § 11

  (workarounds): treat each bullet as a separate candidate — do not batch multiple workarounds into one skill. Format:

  When: task=<type> AND hardware=<hw> [AND model_size<=<N>B]
  - Phase N: step one
  - Phase N: step two
  Source: <project_dir>
  

  A recipe earns its place only when it connects multiple cross-section decisions into a reusable procedure. Single isolated facts stay as lessons.

Step 3 — Write to long-term memory:

Write

.reflect_payload.json

reflect.py --write

{
  "user": [
    { "title": "Hardware — Apple Silicon", "body": "...", "date": "YYYY-MM-DD" }
  ],
  "lessons": [
    {
      "title": "Short title ≤60 chars",
      "body": "One-sentence statement ≤120 chars",
      "source": "project_dir_name",
      "date": "YYYY-MM-DD"
    }
  ],
  "skills": [
    {
      "title": "Scenario name",
      "when": "task=<type> AND hardware=<hw> [AND model_size<=<N>B]",
      "steps": ["Phase N: step one", "Phase N: step two"],
      "source": "project_dir_name",
      "date": "YYYY-MM-DD"
    }
  ]
}

All three top-level keys are optional — omit any that have no entries.

date

# Preview first:
python3 reflect.py --write --input .reflect_payload.json --dry-run
# Then write for real:
python3 reflect.py --write --input .reflect_payload.json

The script handles dedup (sha256), char-limit enforcement (≤3000 chars for lessons/skills, ≤2000 for user), and quarterly archiving (

~/.gaslamp/archive/

→ After Phase 7: Update

gaslamp.md

Reflected to ~/.gaslamp/ on YYYY-MM-DD.

Resources

• Unsloth Documentation
• Model Catalog
• Fine-tuning Guide
• RL Guide
• Vision Guide
• TTS Guide
• Saving to GGUF
• vLLM Deployment
• All Notebooks