ChatKit Actions Skill

Overview

This skill unlocks the full power of ChatKit's agentic UI capabilities - where AI can render interactive widgets, users can click buttons that trigger both client and server actions, and the conversation becomes a two-way interactive experience.

Core Concepts

Action Handler Types

Widgets can specify where actions are handled:

"client"

onAction

"server"

action()

Widget Lifecycle

1. Agent tool generates widget → yield WidgetItem
2. Widget renders in chat with action buttons
3. User clicks action → action dispatched
4. Handler processes action:
   - client: onAction callback in frontend
   - server: action() method in ChatKitServer
5. Optional: Widget replaced with updated state

Implementation Patterns

Pattern 1: Widget Templates (.widget files)

When: Define reusable widget layouts with dynamic data

Widget Template Format:

{
  "version": "1.0",
  "name": "task_list",
  "template": "{\"type\":\"ListView\",\"children\":[...jinja template...]}",
  "jsonSchema": {
    "type": "object",
    "properties": {
      "tasks": { "type": "array", "items": {...} }
    }
  }
}

Widget Components Available:

• Layout:

  ListView

  ,

  ListViewItem

  ,

  Row

  ,

  Col

  ,

  Box

• Content:

  Text

  ,

  Title

  ,

  Image

  ,

  Icon

• Interactive:

  Button

  (with

  onClickAction

  )
• Styling:

  gap

  ,

  padding

  ,

  background

  ,

  border

  ,

  radius

Example - Task List Widget:

{
  "type": "ListView",
  "children": [
    {
      "type": "ListViewItem",
      "key": "task-1",
      "onClickAction": {
        "type": "task.select",
        "handler": "client",
        "payload": { "taskId": "task-1" }
      },
      "children": [
        {
          "type": "Row",
          "gap": 3,
          "children": [
            { "type": "Icon", "name": "check", "color": "success" },
            { "type": "Text", "value": "Complete review", "weight": "semibold" }
          ]
        }
      ]
    }
  ]
}

Python - Loading Templates:

from chatkit.widgets import WidgetTemplate, WidgetRoot

# Load template from file
task_list_template = WidgetTemplate.from_file("task_list.widget")

def build_task_list_widget(tasks: list[Task]) -> WidgetRoot:
    return task_list_template.build(
        data={
            "tasks": [task.model_dump() for task in tasks],
            "selected": None,
        }
    )

Evidence:

cat-lounge/backend/app/widgets/cat_name_suggestions.widget

Pattern 2: Client-Handled Actions

When: Actions that update local state, navigate, or send follow-up messages

Widget Definition (handler: "client"):

{
  "type": "Button",
  "label": "View Article",
  "onClickAction": {
    "type": "open_article",
    "handler": "client",
    "payload": { "id": "article-123" }
  }
}

Frontend Handler:

import { useChatKit, type Widgets } from "@openai/chatkit-react";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (
      action: { type: string; payload?: Record<string, unknown> },
      widgetItem: { id: string; widget: Widgets.Card | Widgets.ListView }
    ) => {
      switch (action.type) {
        case "open_article":
          // Navigate to article
          navigate(`/article/${action.payload?.id}`);
          break;

        case "more_suggestions":
          // Send follow-up message
          await chatkit.sendUserMessage({ text: "More suggestions, please" });
          break;

        case "select_option":
          // Update local state
          setSelectedOption(action.payload?.optionId);
          break;
      }
    },
  },
});

Evidence:

news-guide/frontend/src/components/ChatKitPanel.tsx:55-79

Pattern 3: Server-Handled Actions

When: Actions that mutate data, update widgets, or require backend processing

Widget Definition (handler: "server"):

{
  "type": "ListViewItem",
  "onClickAction": {
    "type": "line.select",
    "handler": "server",
    "payload": { "id": "blue-line" }
  }
}

Backend Handler:

from chatkit.server import ChatKitServer
from chatkit.types import (
    Action,
    WidgetItem,
    ThreadItemReplacedEvent,
    ThreadItemDoneEvent,
    AssistantMessageItem,
    HiddenContextItem,
    ClientEffectEvent,
)

class MyServer(ChatKitServer[dict]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:

        if action.type == "line.select":
            line_id = action.payload["id"]

            # 1. Update widget with selection
            updated_widget = build_line_selector_widget(
                lines=self.lines,
                selected=line_id,
            )
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Add hidden context for future agent input
            await self.store.add_thread_item(
                thread.id,
                HiddenContextItem(
                    id=self.store.generate_item_id("ctx", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=f"<LINE_SELECTED>{line_id}</LINE_SELECTED>",
                ),
                context=context,
            )

            # 3. Stream assistant message
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {line_id}. Where to add station?"}],
                )
            )

            # 4. Trigger client effect
            yield ClientEffectEvent(
                name="location_select_mode",
                data={"lineId": line_id},
            )

Evidence:

metro-map/backend/app/server.py

Pattern 4: Client-to-Server Action Forwarding

When: Client handles action locally, then notifies server for persistence/widget update

Frontend - Send Custom Action:

widgets: {
  onAction: async (action, widgetItem) => {
    if (action.type === "select_name") {
      // 1. Forward to server for processing
      await chatkit.sendCustomAction(action, widgetItem.id);

      // 2. Optionally refresh local state after server processes
      const data = await refreshCatStatus();
      if (data) {
        handleStatusUpdate(data, `Now called ${data.name}`);
      }
    }
  },
}

Evidence:

cat-lounge/frontend/src/components/ChatKitPanel.tsx:68-80

Pattern 5: Entity Tagging (@mentions)

When: Allow users to @mention entities (users, articles, tasks) in messages

Frontend - Entity Configuration:

import { useChatKit, type Entity } from "@openai/chatkit-react";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    // Search for entities as user types @...
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await fetch(`/api/search?q=${query}`).then(r => r.json());

      return results.map((item) => ({
        id: item.id,
        title: item.name,
        icon: item.type === "person" ? "profile" : "document",
        group: item.type === "People" ? "People" : "Articles",
        interactive: true,
        data: {
          type: item.type,
          article_id: item.id,
        },
      }));
    },

    // Handle entity click (e.g., navigate)
    onClick: (entity: Entity) => {
      if (entity.data?.article_id) {
        navigate(`/article/${entity.data.article_id}`);
      }
    },

    // Render entity preview on hover
    onRequestPreview: async (entity: Entity) => {
      const details = await fetch(`/api/entity/${entity.id}`).then(r => r.json());

      return {
        preview: {
          type: "Card",
          children: [
            { type: "Text", value: entity.title, weight: "bold" },
            { type: "Text", value: details.description, color: "tertiary" },
          ],
        },
      };
    },
  },
});

Backend - Converting Entity Tags:

# thread_item_converter.py
class EntityAwareConverter(BasicThreadItemConverter):
    """Convert entity tags to model-readable markers."""

    async def to_agent_input(self, items: list[ThreadItem]) -> list:
        result = []
        for item in items:
            if isinstance(item, UserMessageItem):
                content = item.content
                # Convert entity tags to XML markers
                for entity in item.entities or []:
                    if entity.type == "article":
                        content = content.replace(
                            f"@{entity.title}",
                            f"<ARTICLE_REFERENCE id='{entity.id}'>{entity.title}</ARTICLE_REFERENCE>"
                        )
                result.append({"role": "user", "content": content})
        return result

Evidence:

• news-guide/frontend/src/components/ChatKitPanel.tsx:122-126

• news-guide/backend/app/thread_item_converter.py

• metro-map/frontend/src/components/ChatKitPanel.tsx:73-117

Pattern 6: Composer Tools (Mode Selection)

When: Let users select different AI modes/tools from the composer

Frontend - Tool Configuration:

const TOOL_CHOICES = [
  {
    id: "general",
    label: "Chat",
    shortLabel: "Chat",
    icon: "sparkle",
    placeholderOverride: "Ask anything...",
    pinned: true,
  },
  {
    id: "event_finder",
    label: "Find Events",
    shortLabel: "Events",
    icon: "calendar",
    placeholderOverride: "What events are you looking for?",
    pinned: true,
  },
  {
    id: "puzzle",
    label: "Word Puzzle",
    shortLabel: "Puzzle",
    icon: "bolt",
    placeholderOverride: "Ready for today's puzzle?",
    pinned: false,
  },
];

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },
  composer: {
    placeholder: "What would you like to do?",
    tools: TOOL_CHOICES,
  },
});

Backend - Routing by Tool Choice:

# server.py
async def respond(self, thread, item, context):
    tool_choice = context.get("tool_choice")

    if tool_choice == "event_finder":
        agent = self.event_finder_agent
    elif tool_choice == "puzzle":
        agent = self.puzzle_agent
    else:
        agent = self.general_agent

    # Run selected agent
    result = Runner.run_streamed(agent, input_items, context=agent_context)
    async for event in stream_agent_response(agent_context, result):
        yield event

Evidence:

news-guide/frontend/src/lib/config.ts

Pattern 7: Thread Item Actions (Feedback/Retry/Share)

When: Enable built-in actions on AI messages

Frontend Configuration:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  threadItemActions: {
    feedback: true,   // Thumbs up/down
    retry: true,      // Regenerate response
    share: true,      // Share message
  },

  onLog: ({ name, data }) => {
    if (name === "message.feedback") {
      // Track feedback analytics
      fetch("/api/analytics/feedback", {
        method: "POST",
        body: JSON.stringify(data),
      });
    }
    if (name === "message.share") {
      // Track share events
      fetch("/api/analytics/share", {
        method: "POST",
        body: JSON.stringify(data),
      });
    }
  },
});

Pattern 8: Widget Streaming from Tools

When: Agent tool generates a widget as part of response

Backend - Tool with Widget Output:

from chatkit.types import WidgetItem
from agents import function_tool

@function_tool
async def show_article_list(ctx: AgentContext, query: str) -> str:
    """Show a list of articles matching the query."""

    articles = await article_store.search(query)

    # Build widget
    widget = build_article_list_widget(articles)

    # Yield widget item
    widget_item = WidgetItem(
        id=ctx.store.generate_item_id("widget", ctx.thread, ctx.request_context),
        thread_id=ctx.thread.id,
        created_at=datetime.now(),
        widget=widget,
    )

    # Save to store
    await ctx.store.add_thread_item(ctx.thread.id, widget_item, ctx.request_context)

    # Yield as event
    yield ThreadItemDoneEvent(item=widget_item)

    return f"Showing {len(articles)} articles"

Evidence:

news-guide/backend/app/agents/news_agent.py

Widget Component Reference

Layout Components

ListView

children

ListViewItem

key

onClickAction

children

Row

gap

align

justify

children

Col

gap

align

justify

flex

padding

children

Box

size

radius

background

border

padding

Content Components

Text

value

size

weight

color

maxLines

Title

value

size

weight

Image

src

alt

width

height

fit

radius

Icon

name

size

color

Interactive Components

Button

label

variant

color

size

pill

block

iconStart

iconEnd

onClickAction

disabled

Action Structure

interface Action {
  type: string;           // Action identifier
  handler: "client" | "server";
  payload?: Record<string, unknown>;
}

Common Patterns Summary

onAction

onAction

sendCustomAction

action()

ThreadItemReplacedEvent

sendCustomAction

action()

entities.onTagSearch

ThreadItemConverter

composer.tools

Critical Implementation Details

Action Object Structure

IMPORTANT: The

Action

payload

arguments

# ❌ WRONG - Will cause AttributeError
action.arguments  # 'Action' object has no attribute 'arguments'

# ✅ CORRECT
action.payload    # Access action data via .payload

Action Type Definition:

from chatkit.types import Action

# Action[str, Any] has these fields:
action.type      # str - action identifier (e.g., "task.start")
action.payload   # dict[str, Any] - action data
action.handler   # "client" | "server" - where action is processed

Server Action Handler Signature

CRITICAL: The

context

RequestContext

dict[str, Any]

# Type annotation vs runtime reality mismatch
async def action(
    self,
    thread: ThreadMetadata,
    action: Action[str, Any],
    sender: WidgetItem | None,
    context: dict[str, Any],  # ⚠️ Type hint says dict, but runtime is RequestContext!
) -> AsyncIterator[ThreadStreamEvent]:

    # ❌ WRONG - Tries to wrap RequestContext inside RequestContext
    request_context = RequestContext(metadata=context)

    # ✅ CORRECT - Use context directly, it's already RequestContext
    user_id = context.user_id
    metadata = context.metadata

Why this happens: ChatKit SDK passes

RequestContext

dict

context

UserMessageItem Required Fields

When creating synthetic user messages from actions, ALL these fields are required:

from chatkit.types import UserMessageItem, UserMessageTextContent
from datetime import datetime

# ❌ WRONG - Missing required fields causes ValidationError
synthetic_message = UserMessageItem(
    content=[UserMessageTextContent(type="text", text=message_text)]
)

# ✅ CORRECT - Include all required fields
synthetic_message = UserMessageItem(
    id=self.store.generate_item_id("message", thread, context),
    thread_id=thread.id,
    created_at=datetime.now(),
    content=[UserMessageTextContent(type="input_text", text=message_text)],
    inference_options={},
)

Required fields:

• id

  : Generate via

  store.generate_item_id("message", thread, context)

• thread_id

  : From

  thread.id

  parameter

• created_at

  : Current timestamp via

  datetime.now()

• content

  : List of content blocks (UserMessageTextContent)

• inference_options

  : Empty dict

  {}

  if no special options

UserMessageTextContent type values:

• ✅

  type="input_text"

  - User text input (correct)
• ❌

  type="text"

  - Invalid for UserMessageTextContent (causes ValidationError)

Local Tool Wrappers for Widget Streaming

Problem: Agent calls MCP tool successfully, but widget doesn't appear in UI.

Root Cause: Widgets stream via

RunHooks

Solution Pattern:

# 1. Create local tool wrapper
from agents import function_tool

@function_tool
async def show_task_form(
    ctx: RunContextWrapper[TaskFlowAgentContext],
) -> str:
    """Show interactive task creation form widget."""

    agent_ctx = ctx.context
    mcp_url = agent_ctx.mcp_server_url

    # Call MCP tool via HTTP
    result = await _call_mcp_tool(
        mcp_url,
        "taskflow_show_task_form",
        arguments={"params": {"user_id": agent_ctx.user_id}},
        access_token=agent_ctx.access_token,
    )

    # Return result - RunHooks will intercept and stream widget
    return json.dumps(result)

# 2. Register local wrapper with agent
agent = Agent(
    name="TaskFlow Assistant",
    tools=[
        show_task_form,  # Local wrapper - triggers RunHooks
        # ... other local wrappers
    ],
)

# 3. In RunHooks.on_tool_end() - Stream widget
async def on_tool_end(self, output: str | None, tool_name: str) -> None:
    if tool_name == "show_task_form":
        result = json.loads(output)
        if result.get("action") == "show_form":
            widget = build_task_form_widget()
            yield WidgetItem(...)

Key insight: Direct MCP tools → no widgets. Local wrappers → RunHooks → widgets streamed.

Common Pydantic Validation Errors

Error 1: 'Action' object has no attribute 'arguments'

AttributeError: 'Action[str, Any]' object has no attribute 'arguments'

Fix: Use

action.payload

action.arguments

Error 2: UserMessageTextContent type mismatch

ValidationError: Input should be 'input_text' [type=literal_error, input_value='text']

Fix: Use

type="input_text"

type="text"

Error 3: UserMessageItem missing required fields

4 validation errors for UserMessageItem
- id: Field required
- thread_id: Field required
- created_at: Field required
- inference_options: Field required

Fix: Include all required fields when creating UserMessageItem (see pattern above)

Error 4: RequestContext wrapping issue

2 validation errors for RequestContext
user_id: Field required
metadata: Input should be a valid dictionary [input_value=RequestContext(...)]

Fix: Don't wrap

context

Widget Action Testing Checklist

Before claiming widget actions are complete, test:

• Widget renders with correct data
• All buttons have clear labels (not just icons)
• Client actions navigate/update UI correctly
• Server actions call backend successfully
• Action payload contains all required data
• Widget updates after server action completes
• No AttributeError on action.payload access
• No ValidationError on UserMessageItem creation
• Local tool wrappers trigger widget streaming
• All status transitions have appropriate buttons
• Test with real user session (not mock data)
• Check browser console for errors
• Verify backend logs show action processing
• Test error cases (network failure, invalid data)

Anti-Patterns to Avoid

1. Mixing handlers - Don't handle same action in both client and server
2. Missing payload - Always include necessary data in action payload
3. Forgetting widget ID -

   sendCustomAction

   requires widget ID for updates
4. Not updating widget - Server actions should yield

   ThreadItemReplacedEvent

5. Blocking in onAction - Keep client handlers fast, offload to server
6. Using action.arguments - Use

   action.payload

   (arguments doesn't exist)
7. Wrapping RequestContext - Context is already RequestContext, don't wrap it
8. Missing UserMessageItem fields - Include id, thread_id, created_at, inference_options
9. Wrong content type - Use

   type="input_text"

   for user messages
10. No local tool wrappers - MCP tools alone don't stream widgets
11. Not testing thoroughly - Test all actions with real data before claiming done
12. Assuming type hints are correct - ChatKit has type annotation vs runtime mismatches

References

Documentation

• references/widget-templates.md

  - Widget template syntax

• references/client-vs-server-actions.md

  - Action routing guide

• references/entity-tagging.md

  - @mention implementation

• references/composer-tools.md

  - Tool choice patterns

• references/server-action-handler.py

  - Complete backend action handler pattern

Widget Template Assets

• assets/line-select.widget

  - Server action selection list (metro-map pattern)

• assets/name-suggestions.widget

  - Client action with "more" button (cat-lounge pattern)

• assets/article-list.widget

  - Rich card layout with images (news-guide pattern)

Evidence Sources

All patterns derived from OpenAI ChatKit advanced samples:

• blueprints/openai-chatkit-advanced-samples-main/examples/cat-lounge/

• blueprints/openai-chatkit-advanced-samples-main/examples/metro-map/

• blueprints/openai-chatkit-advanced-samples-main/examples/news-guide/