Build a New AI Agent Project with Olakai

This skill guides you through creating a new AI agent that is fully integrated with Olakai for analytics, KPI tracking, and governance.

Prerequisites

Before starting, ensure:

1. Olakai CLI installed:

   npm install -g olakai-cli

2. CLI authenticated:

   olakai login

3. API key for SDK (generated per-agent via CLI — see Step 2.2)

Why Custom KPIs Are Essential

Olakai's core value is tracking business-specific KPIs for your AI agents. Without KPIs, you're tracking events without gaining actionable insights.

What you can measure with KPIs:

• Business outcomes (items processed, success rates, revenue impact)
• Operational data (step counts, retry rates, execution time)
• Quality indicators (error rates, user satisfaction signals)

Without KPIs configured:

• No dashboard KPIs beyond basic token counts
• No aggregated performance views
• No alerting thresholds
• No ROI calculations

Every agent should have 2-4 KPIs that answer: "How do I know this agent is performing well?"

KPIs created here belong to this specific agent only. If you later create additional agents, each one needs its own KPI definitions — KPIs cannot be shared or reused across agents.

Understanding the customData to KPI Pipeline

Before diving into implementation, understand how data flows through Olakai:

SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiData

How It Works

1. customData (SDK): Raw JSON you send with each event
2. CustomDataConfig (Platform): Schema defining which fields are processed
3. Context Variables: CustomDataConfig fields become available for formulas
4. KPI Formula: Expression that computes a value (e.g.,

   SuccessRate * 100

   )
5. kpiData (Response): Computed KPI values returned with each event

Critical Rules

stepCount

STEPCOUNT

StepCount

"5"

5

Built-in Context Variables (Always Available)

Prompt

Response

Documents count

PII detected

PHI detected

CODE detected

SECRET detected

Step 1: Design the Agent Architecture

1.1 Determine Agent Type

Agentic AI (Multi-step autonomous workflows):

• Research agents, document processors, data pipelines
• Track as SINGLE events aggregating all internal LLM calls
• Focus on workflow-level KPIs (total tokens, total time, success/failure)

Assistive AI (Interactive chatbots/copilots):

• Customer support agents, coding assistants, Q&A systems
• Track EACH interaction as separate events
• Focus on conversation-level KPIs (per-message tokens, response quality)

1.2 Design Your KPI Schema (CRITICAL)

Design your KPIs BEFORE writing any SDK code. This ensures only meaningful data is sent and tracked.

Step A: Identify Business Questions

What do stakeholders need to know about this agent?

• "How many items does it process per run?"
• "What's the success/failure rate?"
• "How efficient is each execution?"

Step B: Map Questions to Data Fields

ItemsProcessed

SuccessRate * 100

IF(SuccessRate < 1, 1, 0)

Step C: Plan Your customData Structure

// ONLY include fields you'll register as CustomDataConfigs
customData: {
  // Business KPIs
  ItemsProcessed: number,  // Count of items handled
  SuccessRate: number,     // 0-1 success ratio

  // Performance KPIs
  StepCount: number,       // Number of workflow steps

  // Identification (for filtering, not KPIs)
  ExecutionId: string,     // Correlation ID
}

IMPORTANT: Only include fields you will register as CustomDataConfigs. Unregistered fields are stored but cannot be used in KPIs.

What NOT to Include in customData

The Olakai platform automatically tracks these — do NOT duplicate them:

sessionId

agentId

userEmail

email

userEmail

timestamp

createdAt

requestTime

duration

latency

tokens

tokenCount

model

modelName

provider

customData is ONLY for:

1. KPI variables — Fields you'll use in formula calculations
2. Tagging/filtering — Fields you'll filter by in queries

Step 2: Configure Olakai Platform

2.1 Create a Workflow (Required)

Every agent MUST belong to a workflow, even if it's the only agent in that workflow.

olakai workflows create --name "Your Workflow Name" --json
# Output: { "id": "wfl_xxx...", "name": "Your Workflow Name" }

2.2 Create the Agent in Olakai

olakai agents create \
  --name "Your Agent Name" \
  --description "What this agent does" \
  --workflow WORKFLOW_ID \
  --with-api-key \
  --json

# Returns agent details including apiKey:
# {
#   "id": "cmkbteqn501kyjy4yu6p6xrrx",
#   "name": "Your Agent Name",
#   "workflowId": "wfl_xxx...",
#   "apiKey": "sk_agent_xxxxx..."   <-- Use this in your SDK
# }

Agent-Workflow Hierarchy:

Workflow: "Customer Support Pipeline"
├── Agent: "Ticket Classifier"
├── Agent: "Response Generator"
└── Agent: "Quality Checker"

Workflow: "Document Processing"
└── Agent: "Document Summarizer"  ← single-agent workflows are valid

2.3 Create Custom Data Configurations (BEFORE Writing SDK Code)

This step MUST be completed before Step 3 (SDK Integration). Only fields registered here can be used in KPI formulas.

ONLY create configs for data you'll use in KPIs or for filtering. Don't create configs for data already tracked automatically.

# For numeric fields (can be used in KPI calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ItemsProcessed" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "SuccessRate" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "StepCount" --type NUMBER

# For string fields (for filtering/grouping, not calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ExecutionId" --type STRING

# Verify all configs are created
olakai custom-data list --agent-id YOUR_AGENT_ID

2.4 Create KPI Definitions

Quick Start with Templates

Instead of writing formulas from scratch, use predefined classifier templates:

# List available templates
olakai kpis templates

# Create a classifier KPI from a template
olakai kpis create --name "User Satisfaction" \
  --calculator-id classifier --template-id sentiment_scorer \
  --scope CHAT --agent-id $AGENT_ID

# Create a time-saved estimator
olakai kpis create --name "Time Saved" \
  --calculator-id classifier --template-id time_saved_estimator \
  --scope CHAT --agent-id $AGENT_ID

Custom Formula KPIs

# Variable passthrough
olakai kpis create \
  --name "Items Processed" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "ItemsProcessed" \
  --unit "items" \
  --aggregation SUM

# Percentage calculation
olakai kpis create \
  --name "Success Rate" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "SuccessRate * 100" \
  --unit "%" \
  --aggregation AVERAGE

# Conditional counting
olakai kpis create \
  --name "Error Count" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "IF(SuccessRate < 1, 1, 0)" \
  --unit "errors" \
  --aggregation SUM

# Validate formulas before creating
olakai kpis validate --formula "ItemsProcessed" --agent-id YOUR_AGENT_ID

Step 3: Implement SDK Integration

3.1 TypeScript Implementation

Install dependencies:

npm install @olakai/sdk openai

Initialize and track:

import { olakaiConfig, olakai } from "@olakai/sdk";
import OpenAI from "openai";

// Initialize Olakai
olakaiConfig({
  apiKey: process.env.OLAKAI_API_KEY!,
  debug: process.env.NODE_ENV === "development",
});

// Create LLM client
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// Use wrapped client — monitoring happens automatically
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: userPrompt }],
});

Agentic workflow with manual event tracking:

taskExecutionId

— Cross-Agent Task Correlation. Generate ONE

taskExecutionId

per task and share it across all agents in a multi-agent workflow. This links events from different agents into a single logical task for analytics.

async function runAgent(input: string): Promise<string> {
  const startTime = Date.now();
  const executionId = crypto.randomUUID();
  const taskExecutionId = crypto.randomUUID();
  let totalTokens = 0;
  let stepCount = 0;
  let itemsProcessed = 0;

  try {
    // Step 1: Planning
    stepCount++;
    const plan = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: `Plan: ${input}` }],
    });
    totalTokens += plan.usage?.total_tokens ?? 0;

    // Step 2: Process items
    const items = parseItems(plan.choices[0].message.content);
    for (const item of items) {
      stepCount++;
      const result = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: `Process: ${item}` }],
      });
      totalTokens += result.usage?.total_tokens ?? 0;
      itemsProcessed++;
    }

    // Step 3: Summarize
    stepCount++;
    const summary = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: "Summarize results" }],
    });
    totalTokens += summary.usage?.total_tokens ?? 0;

    const finalResponse = summary.choices[0].message.content ?? "";

    // Track the complete workflow as a single event
    // Only send fields that have CustomDataConfigs (from Step 2.3)
    olakai("event", "ai_activity", {
      prompt: input,
      response: finalResponse,
      tokens: totalTokens,
      requestTime: Date.now() - startTime,
      taskExecutionId,
      task: "Data Processing & Analysis",
      customData: {
        ExecutionId: executionId,
        StepCount: stepCount,
        ItemsProcessed: itemsProcessed,
        SuccessRate: 1.0,
      },
    });

    return finalResponse;
  } catch (error) {
    // Track failed execution — same fields, different values
    olakai("event", "ai_activity", {
      prompt: input,
      response: `Error: ${error instanceof Error ? error.message : "Unknown"}`,
      tokens: totalTokens,
      requestTime: Date.now() - startTime,
      taskExecutionId,
      task: "Data Processing & Analysis",
      customData: {
        ExecutionId: executionId,
        StepCount: stepCount,
        ItemsProcessed: itemsProcessed,
        SuccessRate: 0,
      },
    });
    throw error;
  }
}

3.2 Python Implementation

Install dependencies:

pip install olakai-sdk openai

Initialize and track:

import os
from olakaisdk import olakai_config, olakai, OlakaiEventParams
from openai import OpenAI

# Initialize Olakai
olakai_config(os.getenv("OLAKAI_API_KEY"))

# Create OpenAI client
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

Agentic workflow:

import time
import uuid

def run_agent(input_text: str) -> str:
    start_time = time.time()
    execution_id = str(uuid.uuid4())
    task_execution_id = str(uuid.uuid4())
    total_tokens = 0
    step_count = 0
    items_processed = 0

    try:
        # Your workflow steps here...
        step_count += 1
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": input_text}]
        )
        total_tokens += response.usage.total_tokens
        final_response = response.choices[0].message.content

        # Track successful execution
        olakai("event", "ai_activity", OlakaiEventParams(
            prompt=input_text,
            response=final_response,
            tokens=total_tokens,
            requestTime=int((time.time() - start_time) * 1000),
            taskExecutionId=task_execution_id,
            task="Data Processing & Analysis",
            customData={
                "ExecutionId": execution_id,
                "StepCount": step_count,
                "ItemsProcessed": items_processed,
                "SuccessRate": 1.0,
            }
        ))

        return final_response

    except Exception as e:
        olakai("event", "ai_activity", OlakaiEventParams(
            prompt=input_text,
            response=f"Error: {str(e)}",
            tokens=total_tokens,
            requestTime=int((time.time() - start_time) * 1000),
            taskExecutionId=task_execution_id,
            task="Data Processing & Analysis",
            customData={
                "ExecutionId": execution_id,
                "StepCount": step_count,
                "ItemsProcessed": items_processed,
                "SuccessRate": 0,
            }
        ))
        raise

3.3 REST API Direct Integration

For other languages or custom integrations:

curl -X POST "https://app.olakai.ai/api/monitoring/prompt" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "prompt": "User input here",
    "response": "Agent response here",
    "app": "your-agent-name",
    "task": "Data Processing & Analysis",
    "tokens": 1500,
    "requestTime": 5000,
    "customData": {
      "ExecutionId": "abc-123",
      "StepCount": 5,
      "ItemsProcessed": 10,
      "SuccessRate": 1.0
    }
  }'

Step 4: Test-Validate-Iterate Cycle

Always validate your implementation by running a test and inspecting the actual event data.

4.1 Run Your Agent

Execute your agent with test data to generate at least one event.

4.2 Fetch and Inspect the Event

olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json
olakai activity get EVENT_ID --json

4.3 Validate Each Component

Check customData is present and correct:

olakai activity get EVENT_ID --json | jq '.customData'

Check KPIs are numeric (not strings):

olakai activity get EVENT_ID --json | jq '.kpiData'

CORRECT — numeric values:

{
  "Items Processed": 10,
  "Success Rate": 100
}

WRONG — string values (broken formula):

{
  "Items Processed": "itemsProcessed"
}

Fix:

olakai kpis update KPI_ID --formula "YourVariable"

WRONG — null values: Fix by verifying:

1. CustomDataConfig exists:

   olakai custom-data list --agent-id ID

2. Field name case matches exactly (case-sensitive)
3. SDK actually sends the field in customData

4.4 Validation Flow

1. Run agent (generate event)
           ↓
2. Fetch event: olakai activity get ID --json
           ↓
3. Check customData present? NO → Fix SDK code
           ↓
4. Check kpiData numeric? NO → Fix formula
           ↓
5. Check kpiData not null? NO → Create CustomDataConfig or fix field name
           ↓
✅ All validations pass — implementation complete

Step 5: Production Checklist

Before deploying to production:

• API key stored securely in environment variables
• Error handling wraps all LLM calls
• Failed executions still report events (with SuccessRate: 0)
• All custom data fields have corresponding CustomDataConfig entries
• KPI formulas validated and showing numeric values (not strings)
• SDK configured with appropriate retries and timeouts
• Sensitive data redaction enabled if needed

KPI Formula Reference

Supported Operators

+

-

*

/

<

<=

=

<>

>=

>

AND

OR

NOT

IF(condition, true_val, false_val)

MAP(value, match1, out1, default)

ABS

MAX

MIN

AVERAGE

TRUNC

ISNA(value)

ISDEFINED(value)

NA()

Common Formula Patterns

--formula "ItemsProcessed"                              # passthrough
--formula "SuccessRate * 100"                            # percentage (0-1 to 0-100)
--formula "IF(SuccessRate < 1, 1, 0)"                    # conditional counting
--formula "IF(PII detected, 1, 0)"                       # built-in variable
--formula "IF(ISDEFINED(MyField), MyField, 0)"           # null-safe
--formula "IF(AND(StepCount > 5, SuccessRate < 0.9), 1, 0)"  # compound conditions

Aggregation Types

SUM

AVERAGE

Task Categories Reference

Use these predefined task categories for the

task

Quick Reference

# CLI Commands
olakai login                                              # Authenticate
olakai workflows create --name "Name" --json              # Create workflow
olakai agents create --name "Name" --workflow ID --with-api-key  # Register agent
olakai custom-data create --agent-id ID --name X --type NUMBER   # Create custom field
olakai kpis create --formula "X" --agent-id ID            # Create KPI
olakai activity list --agent-id ID                        # View events

// TypeScript SDK
import { olakaiConfig, olakai } from "@olakai/sdk";
olakaiConfig({ apiKey: process.env.OLAKAI_API_KEY });

olakai("event", "ai_activity", {
  prompt: "input",
  response: "output",
  tokens: 1500,
  task: "Data Processing & Analysis",
  customData: { StepCount: 3, Success: 1 },
});

# Python SDK
from olakaisdk import olakai_config, olakai, OlakaiEventParams
olakai_config(os.getenv("OLAKAI_API_KEY"))

olakai("event", "ai_activity", OlakaiEventParams(
    prompt="input",
    response="output",
    tokens=1500,
    task="Data Processing & Analysis",
    customData={"StepCount": 3, "Success": 1},
))