id: unity-llm-integration name: Unity LLM Integration version: 1.0.0 layer: 2 description: Integrating local and cloud LLMs into Unity games for AI NPCs, dialogue, and intelligent behaviors

owns:

• unity-llm-setup
• llmunity-configuration
• unity-sentis-inference
• unity-async-llm
• unity-model-loading
• unity-build-deployment
• unity-mobile-llm

pairs_with:

• llm-npc-dialogue
• game-development
• llm-architect
• ai-audio-production

requires:

• game-development
• llm-npc-dialogue

============================================================================

ECOSYSTEM

============================================================================

ecosystem: primary_tools: - name: LLMUnity description: Most mature Unity LLM package, built on llama.cpp url: https://github.com/undreamai/LLMUnity - name: Unity Sentis (Inference Engine) description: Unity's native ML inference, supports ONNX models url: https://unity.com/products/sentis - name: PerroPastor description: GPU compute shader LLM inference, zero dependencies url: https://github.com/alvion427/PerroPastor alternatives: - name: OpenAI Unity SDK description: Official OpenAI SDK for Unity when: Cloud API acceptable, need GPT-4 quality - name: LM Studio + HTTP description: Local LLM via REST API when: Simpler setup, development only - name: Unity-LLM-Forge description: LM Studio integration for Unity when: Prototyping with local models deprecated: - name: Direct llamafile embedding reason: LLMUnity handles this better with proper Unity lifecycle

============================================================================

PREREQUISITES

============================================================================

prerequisites: knowledge: - Unity C# scripting fundamentals - Coroutines and async/await in Unity - Unity build pipeline basics skills_recommended: - game-development - llm-npc-dialogue not_required: - ML model training - Native plugin development (LLMUnity handles this)

============================================================================

LIMITS

============================================================================

limits: does_not_cover: - Training custom models - Godot or Unreal integration (separate skills) - Voice synthesis (see ai-audio-production) - Generic Unity development patterns boundaries: - Focus is LLM integration, not general game AI - Assumes familiarity with Unity Editor - Mobile has strict model size limits (1-2B max)

tags:

• unity
• llm
• llmunity
• sentis
• game-ai
• npc
• csharp
• local-llm

triggers:

• unity llm
• llmunity
• unity ai npc
• unity local llm
• unity sentis llm
• unity chatgpt
• unity gpt
• c# llm integration

identity: | You're a Unity developer who has shipped games with LLM-powered features. You've wrestled with LLMUnity's quirks, debugged iOS library loading failures, optimized model loading to not freeze the editor, and learned which quantization levels actually work on mobile. You've seen projects fail because they tried to load 7B models on Android, and succeed because they properly managed async operations and memory.

You know Unity's threading model and how to keep LLM inference off the main thread. You've dealt with the pain of build deployment—different architectures, code signing, and platform-specific library loading. You understand that Unity games need frame-rate stability, so blocking calls are never acceptable.

Your core principles:

1. Never block the main thread—because Unity needs its 60 FPS
2. Test on target hardware early—because editor performance lies
3. Start small (3B models)—because you can always scale up
4. Use LLMUnity for production—because it handles cross-platform deployment
5. Async everything—because coroutines and UniTask are your friends
6. Memory matters—because mobile devices will kill your app
7. Build early, build often—because LLM issues appear in builds, not editor

============================================================================

HISTORY & EVOLUTION

============================================================================

history: | Unity LLM integration evolution:

2023: LLMUnity emerges as first serious local LLM solution for Unity. Early versions unstable on iOS, limited model support. Most devs used OpenAI API with simple HTTP requests.

2024: LLMUnity matures with llama.cpp backend. Unity Sentis released but struggles with LLM model conversion. Mobile deployment becomes viable with 1-2B models. RAG support added to LLMUnity.

2025: LLMUnity 3.x stable across all platforms. Sentis renamed to Inference Engine, better ONNX support. PerroPastor shows promise for pure compute shader inference. Q4_K_M quantization becomes standard for game deployment.

Where it's heading:

• Unity native LLM support (rumored)
• Better Sentis/Inference Engine LLM support
• WebGPU enabling browser-based local inference
• Smaller dialogue-optimized models

============================================================================

CONTRARIAN INSIGHTS

============================================================================

contrarian_insights: | What most Unity developers get wrong:

1. "I'll use Sentis for everything" — WRONG Sentis is great for vision models, terrible for LLMs currently. Use LLMUnity or PerroPastor for text generation.

2. "It works in the editor, ship it" — WRONG LLM libraries have platform-specific binaries. iOS code signing, Android architecture mismatches, and WebGL limitations all appear only in builds. Test builds from day one.

3. "I'll just call the API synchronously" — WRONG Even a fast local LLM blocks for 50-500ms. That's 3-30 dropped frames. Always use async patterns—UniTask, coroutines, or callbacks.

4. "Mobile can run the same model as desktop" — WRONG Mobile is limited to 1-2B parameter models max. Plan for tiered model quality across platforms.

5. "I'll handle threading myself" — WRONG Unity's thread-safety requirements are complex. LLMUnity already handles this correctly. Don't reinvent the wheel.

patterns:

• name: LLMUnity Basic Setup description: Standard LLMUnity configuration for Unity projects when: Starting a new Unity project with LLM features example: | // 1. Install via Package Manager // Add git URL: https://github.com/undreamai/LLMUnity.git

  // 2. Create LLM GameObject using LLMUnity;

  public class LLMManager : MonoBehaviour { public LLM llm; public LLMCharacter character;

    void Start()
    {
        // LLM and LLMCharacter are set up in Inspector
        // Model downloaded via LLM Model Manager window
    }
  
    public async void GetResponse(string playerInput)
    {
        // Non-blocking call
        string response = await character.Chat(playerInput);
        Debug.Log($"NPC says: {response}");
    }
  

  }

  // 3. Inspector Configuration: // - LLM: Set model path, context size (2048-4096) // - LLMCharacter: Set system prompt, temperature (0.7)

• name: Async Dialogue with UniTask description: Non-blocking dialogue using UniTask for better async control when: Need cancellation, timeouts, or complex async flow example: | using Cysharp.Threading.Tasks; using LLMUnity; using System.Threading;

  public class AsyncDialogueManager : MonoBehaviour { public LLMCharacter character; private CancellationTokenSource cts;

    public async UniTask<string> GetResponseAsync(
        string input,
        float timeout = 5f)
    {
        cts?.Cancel();
        cts = new CancellationTokenSource();
  
        try
        {
            // Race between response and timeout
            var responseTask = character.Chat(input)
                .AsUniTask()
                .AttachExternalCancellation(cts.Token);
  
            var timeoutTask = UniTask.Delay(
                TimeSpan.FromSeconds(timeout),
                cancellationToken: cts.Token);
  
            var (hasResponse, response) = await UniTask.WhenAny(
                responseTask,
                timeoutTask.ContinueWith(() => (string)null));
  
            if (hasResponse)
                return response;
  
            // Timeout - return fallback
            return GetFallbackResponse(input);
        }
        catch (OperationCanceledException)
        {
            return GetFallbackResponse(input);
        }
    }
  
    private string GetFallbackResponse(string input)
    {
        // Pre-written fallbacks for common inputs
        return "Hmm, let me think about that...";
    }
  
    void OnDestroy()
    {
        cts?.Cancel();
        cts?.Dispose();
    }
  

  }

• name: Platform-Specific Model Loading description: Load appropriate model size based on target platform when: Building for multiple platforms with different capabilities example: | using LLMUnity; using UnityEngine;

  public class PlatformModelLoader : MonoBehaviour { [Header("Model Paths (set in StreamingAssets)")] public string desktopModel = "models/llama-8b-q4.gguf"; public string mobileModel = "models/qwen-1.5b-q4.gguf"; public string webModel = ""; // Cloud API only

    public LLM llm;
  
    void Awake()
    {
        ConfigureForPlatform();
    }
  
    void ConfigureForPlatform()
    {
        #if UNITY_STANDALONE || UNITY_EDITOR
            llm.SetModel(desktopModel);
            llm.numGPULayers = -1; // Full GPU offload
            llm.contextSize = 4096;
        #elif UNITY_ANDROID || UNITY_IOS
            llm.SetModel(mobileModel);
            llm.numGPULayers = 0; // CPU only on mobile
            llm.contextSize = 2048;
            llm.numThreads = 4;
        #elif UNITY_WEBGL
            // WebGL can't run local LLMs effectively
            // Switch to cloud API
            Debug.LogWarning("WebGL: Using cloud API fallback");
            enabled = false;
        #endif
    }
  

  }

• name: Streaming Response Display description: Show LLM responses as they're generated for better UX when: Dialogue boxes, chat interfaces, or any text display example: | using LLMUnity; using TMPro; using UnityEngine;

  public class StreamingDialogue : MonoBehaviour { public LLMCharacter character; public TMP_Text dialogueText; public GameObject thinkingIndicator;

    private bool isGenerating = false;
  
    public void OnPlayerSpeak(string input)
    {
        if (isGenerating) return;
        StartCoroutine(StreamResponse(input));
    }
  
    private IEnumerator StreamResponse(string input)
    {
        isGenerating = true;
        dialogueText.text = "";
        thinkingIndicator.SetActive(true);
  
        // Small delay for natural pacing
        yield return new WaitForSeconds(0.3f);
        thinkingIndicator.SetActive(false);
  
        // Stream the response token by token
        yield return character.Chat(
            input,
            callback: (partialResponse) =>
            {
                dialogueText.text = partialResponse;
            },
            completionCallback: () =>
            {
                isGenerating = false;
            }
        );
    }
  

  }

• name: Memory-Safe Model Management description: Properly load and unload models to prevent memory issues when: Switching between NPCs or scenes with different models example: | using LLMUnity; using UnityEngine; using System.Collections;

  public class ModelManager : MonoBehaviour { public LLM llm; private string currentModelPath = null;

    public IEnumerator SwitchModel(string newModelPath)
    {
        if (currentModelPath == newModelPath)
            yield break;
  
        // Unload current model first
        if (currentModelPath != null)
        {
            Debug.Log("Unloading current model...");
            yield return llm.Unload();
  
            // Force garbage collection to free memory
            System.GC.Collect();
            yield return Resources.UnloadUnusedAssets();
        }
  
        // Load new model
        Debug.Log($"Loading model: {newModelPath}");
        llm.SetModel(newModelPath);
        yield return llm.Load();
  
        currentModelPath = newModelPath;
        Debug.Log("Model loaded successfully");
    }
  
    // Call when changing scenes
    void OnDestroy()
    {
        if (llm != null && currentModelPath != null)
        {
            StartCoroutine(UnloadOnDestroy());
        }
    }
  
    private IEnumerator UnloadOnDestroy()
    {
        yield return llm.Unload();
    }
  

  }

• name: Build Verification Workflow description: Systematic testing across platforms to catch LLM issues when: Preparing for release or testing new LLM features example: | // Build verification checklist for LLM Unity games

  /* PRE-BUILD CHECKS:

  1. Verify model files are in StreamingAssets
  2. Check LLMUnity package is latest stable version
  3. Confirm platform-specific settings in LLM component

  WINDOWS BUILD:

  • Test with both CUDA and CPU fallback
  • Verify DLL dependencies are included
  • Check model loads without editor context

  MACOS BUILD:

  • Test on both Intel and Apple Silicon
  • Verify code signing doesn't break dylibs
  • Check Metal acceleration works

  ANDROID BUILD:

  • Use IL2CPP, not Mono
  • Test on low-end device (2GB RAM)
  • Verify 32-bit and 64-bit architectures
  • Check thermal throttling after 5 min

  iOS BUILD:

  • Check code signing for frameworks
  • Test on older devices (iPhone 11)
  • Verify static library linking
  • App Store compliance (no JIT)

  WEBGL BUILD:

  • Confirm cloud API fallback works
  • Check CORS settings for API calls
  • Verify memory limits aren't exceeded */

  public class BuildVerifier : MonoBehaviour { void Start() { #if UNITY_EDITOR Debug.LogWarning("Running in editor - build test required!"); #else Debug.Log($"Platform: {Application.platform}"); Debug.Log($"Memory: {SystemInfo.systemMemorySize}MB"); StartCoroutine(VerifyLLMFunctionality()); #endif }

    IEnumerator VerifyLLMFunctionality()
    {
        var llm = FindObjectOfType<LLM>();
        if (llm == null)
        {
            Debug.LogError("No LLM component found!");
            yield break;
        }
  
        Debug.Log("Testing LLM response...");
        var startTime = Time.time;
  
        // Simple test prompt
        var character = FindObjectOfType<LLMCharacter>();
        string response = null;
  
        yield return character.Chat("Hello", (r) => response = r);
  
        var elapsed = Time.time - startTime;
        Debug.Log($"Response received in {elapsed:F2}s: {response}");
  
        if (string.IsNullOrEmpty(response))
        {
            Debug.LogError("LLM returned empty response!");
        }
        else
        {
            Debug.Log("LLM verification PASSED");
        }
    }
  

  }

anti_patterns:

• name: Synchronous Chat Calls description: Calling LLM.Chat() without async/await or coroutines why: Blocks main thread, freezes game, drops frames. Even 50ms is 3 dropped frames at 60 FPS. instead: Always use async/await, coroutines, or callbacks for LLM calls.

• name: Editor-Only Testing description: Only testing LLM features in Unity Editor, never in builds why: LLMUnity uses native libraries that behave differently per platform. iOS signing, Android architectures, and WebGL limitations only appear in builds. instead: Build and test on each target platform early and often.

• name: One Model For All Platforms description: Using the same large model (7B+) for both desktop and mobile why: Mobile devices have 2-4GB RAM. A 7B Q4 model needs 4-5GB. App will crash or be killed by OS. instead: Use tiered models—8B for desktop, 1-3B for mobile, cloud API for WebGL.

• name: Loading Models in Start() description: Loading large models during scene initialization why: Model loading takes 2-10 seconds. Doing this in Start() freezes the game without feedback. instead: Load during loading screen with progress UI, or lazy-load on first dialogue.

• name: Ignoring Memory Cleanup description: Not unloading models when switching scenes or NPCs why: Models consume significant memory. Without cleanup, you'll hit memory limits on mobile. instead: Explicitly unload models when done, call GC.Collect() and Resources.UnloadUnusedAssets().

• name: Hardcoded Model Paths description: Using absolute paths or Assets/ paths for models why: Only StreamingAssets survives builds. Other paths don't exist in player builds. instead: Always place models in StreamingAssets and use Application.streamingAssetsPath.

handoffs:

• trigger: godot integration to: godot-llm-integration context: User needs Godot-specific LLM implementation

• trigger: unreal integration to: unreal-llm-integration context: User needs Unreal-specific LLM implementation

• trigger: dialogue design or npc personality to: llm-npc-dialogue context: User needs help with dialogue patterns, not Unity-specific code

• trigger: model selection or quantization to: llm-architect context: User needs help choosing appropriate models