Building Chat Widgets

Create interactive widgets for AI chat with actions and entity tagging.

Quick Start

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      if (action.type === "view_details") {
        navigate(`/details/${action.payload.id}`);
      }
    },
  },
});

Action Handler Types

"client"

onAction

"server"

action()

Widget Lifecycle

1. Agent tool generates widget → yield WidgetItem
2. Widget renders in chat with action buttons
3. User clicks action → action dispatched
4. Handler processes action:
   - client: onAction callback in frontend
   - server: action() method in ChatKitServer
5. Optional: Widget replaced with updated state

Core Patterns

1. Widget Templates

Define reusable widget layouts with dynamic data:

{
  "type": "ListView",
  "children": [
    {
      "type": "ListViewItem",
      "key": "item-1",
      "onClickAction": {
        "type": "item.select",
        "handler": "client",
        "payload": { "itemId": "item-1" }
      },
      "children": [
        {
          "type": "Row",
          "gap": 3,
          "children": [
            { "type": "Icon", "name": "check", "color": "success" },
            { "type": "Text", "value": "Item title", "weight": "semibold" }
          ]
        }
      ]
    }
  ]
}

2. Client-Handled Actions

Actions that update local state, navigate, or send follow-up messages:

Widget Definition:

{
  "type": "Button",
  "label": "View Article",
  "onClickAction": {
    "type": "open_article",
    "handler": "client",
    "payload": { "id": "article-123" }
  }
}

Frontend Handler:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      switch (action.type) {
        case "open_article":
          navigate(`/article/${action.payload?.id}`);
          break;

        case "more_suggestions":
          await chatkit.sendUserMessage({ text: "More suggestions, please" });
          break;

        case "select_option":
          setSelectedOption(action.payload?.optionId);
          break;
      }
    },
  },
});

3. Server-Handled Actions

Actions that mutate data, update widgets, or require backend processing:

Widget Definition:

{
  "type": "ListViewItem",
  "onClickAction": {
    "type": "line.select",
    "handler": "server",
    "payload": { "id": "blue-line" }
  }
}

Backend Handler:

from chatkit.types import (
    Action, WidgetItem, ThreadItemReplacedEvent,
    ThreadItemDoneEvent, AssistantMessageItem, ClientEffectEvent,
)

class MyServer(ChatKitServer[dict]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: RequestContext,  # Note: Already RequestContext, not dict
    ) -> AsyncIterator[ThreadStreamEvent]:

        if action.type == "line.select":
            line_id = action.payload["id"]  # Use .payload, not .arguments

            # 1. Update widget with selection
            updated_widget = build_selector_widget(selected=line_id)
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Stream assistant message
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {line_id}"}],
                )
            )

            # 3. Trigger client effect
            yield ClientEffectEvent(
                name="selection_changed",
                data={"lineId": line_id},
            )

4. Entity Tagging (@mentions)

Allow users to @mention entities in messages:

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await fetch(`/api/search?q=${query}`).then(r => r.json());

      return results.map((item) => ({
        id: item.id,
        title: item.name,
        icon: item.type === "person" ? "profile" : "document",
        group: item.type === "People" ? "People" : "Articles",
        interactive: true,
        data: { type: item.type, article_id: item.id },
      }));
    },

    onClick: (entity: Entity) => {
      if (entity.data?.article_id) {
        navigate(`/article/${entity.data.article_id}`);
      }
    },
  },
});

5. Composer Tools (Mode Selection)

Let users select different AI modes from the composer:

const TOOL_CHOICES = [
  {
    id: "general",
    label: "Chat",
    icon: "sparkle",
    placeholderOverride: "Ask anything...",
    pinned: true,
  },
  {
    id: "event_finder",
    label: "Find Events",
    icon: "calendar",
    placeholderOverride: "What events are you looking for?",
    pinned: true,
  },
];

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },
  composer: {
    placeholder: "What would you like to do?",
    tools: TOOL_CHOICES,
  },
});

Backend Routing:

async def respond(self, thread, item, context):
    tool_choice = context.metadata.get("tool_choice")

    if tool_choice == "event_finder":
        agent = self.event_finder_agent
    else:
        agent = self.general_agent

    result = Runner.run_streamed(agent, input_items)
    async for event in stream_agent_response(context, result):
        yield event

Widget Component Reference

Layout Components

ListView

children

ListViewItem

key

onClickAction

children

Row

gap

align

justify

children

Col

gap

padding

children

Box

size

radius

background

padding

Content Components

Text

value

size

weight

color

Title

value

size

weight

Image

src

alt

width

height

Icon

name

size

color

Interactive Components

Button

label

variant

onClickAction

Critical Implementation Details

Action Object Structure

IMPORTANT: Use

action.payload

action.arguments

# WRONG - Will cause AttributeError
action.arguments

# CORRECT
action.payload

Context Parameter

The

context

RequestContext

dict

# WRONG - Tries to wrap RequestContext
request_context = RequestContext(metadata=context)

# CORRECT - Use directly
user_id = context.user_id

UserMessageItem Required Fields

When creating synthetic user messages:

from chatkit.types import UserMessageItem, UserMessageTextContent

# Include ALL required fields
synthetic_message = UserMessageItem(
    id=self.store.generate_item_id("message", thread, context),
    thread_id=thread.id,
    created_at=datetime.now(),
    content=[UserMessageTextContent(type="input_text", text=message_text)],
    inference_options={},
)

Anti-Patterns

1. Mixing handlers - Don't handle same action in both client and server
2. Missing payload - Always include data in action payload
3. Using action.arguments - Use

   action.payload

4. Wrapping RequestContext - Context is already RequestContext
5. Missing UserMessageItem fields - Include id, thread_id, created_at
6. Wrong content type - Use

   type="input_text"

   for user messages

Verification

Run:

python3 scripts/verify.py

Expected:

✓ building-chat-widgets skill ready

If Verification Fails

1. Check: references/ folder has widget-patterns.md
2. Stop and report if still failing

References

• references/widget-patterns.md - Complete widget patterns
• references/server-action-handler.md - Backend action handling