WizTelemetry Data Pipeline (Vector)

Overview

WizTelemetry Data Pipeline is an extension based on vector (https://vector.dev/) that provides the ability to collect, transform, and route observability data. It is a core dependency for other WizTelemetry extensions like Logging, Auditing, Events, and Notification.

When to Use

• Installing or configuring the WizTelemetry Data Pipeline extension
• Setting up data collection for logs, auditing, events, and notifications
• Configuring Vector sinks (OpenSearch)
• Managing Vector agent components

Installation

Prerequisites

REQUIRED: Complete all steps in order before generating InstallPlan.

Step 1: Get Available Clusters and Confirm Target

⚠️ CRITICAL: DO NOT proceed until target clusters are determined.

Step 1.1: Get available clusters

kubectl get clusters -o jsonpath='{.items[*].metadata.name}'

Step 1.2: Determine target clusters

• If user explicitly specified target clusters in the request → Use those clusters directly, proceed to Step 2
• If user did NOT specify target clusters → Ask user to confirm which clusters to deploy to, then proceed to Step 2

Ask user (if not specified):

Available clusters: host, dev
Which clusters do you want to deploy Vector to?

Step 2: Get OpenSearch Endpoint and Credentials (MUST DO)

• If user already provided OpenSearch endpoint and credentials in the request → Use those directly, proceed to Step 3
• If user did NOT provide → You MUST ask user for OpenSearch endpoint and credentials

Ask user for (if not provided):

1. OpenSearch endpoint URL (required)

   • Example:

     http://<node-ip>:30920

     or

     https://opensearch.example.com:9200

2. OpenSearch credentials (required)

   • Username (default:

     admin

     )
   • Password

DO NOT proceed to Step 3 until user provides both endpoint and credentials.

Step 3: Get Latest Vector Version (if not provided by user)

MUST do this to get the latest version:

kubectl get extensionversions -n kubesphere-system -l kubesphere.io/extension-ref=vector -o jsonpath='{range .items[*]}{.spec.version}{"\n"}{end}' | sort -V | tail -1

This outputs the latest version (e.g.,

1.1.4

Install Vector Extension

⚠️ IMPORTANT: Complete prerequisite steps (1-3) BEFORE this step.

⚠️ CRITICAL: InstallPlan

metadata.name

vector

Based on your selections:

• Target clusters: Use the user-confirmed cluster names
• OpenSearch endpoint: User-provided endpoint
• OpenSearch credentials: User-provided username and password

⚠️ CRITICAL:

config

• Use the config structure exactly as shown in the template
• DO NOT add configuration fields that are not shown in the template
• DO NOT modify the structure or hierarchy

⚠️ CRITICAL: All placeholders MUST be replaced with actual values. DO NOT leave them as placeholders.

Template

apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
  namespace: kubesphere-system
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>  # From Step 3
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Replace placeholders:

• <VECTOR_VERSION>

  : From Step 2 (e.g.,

  1.1.4

  )

• <OPENSEARCH_ENDPOINT>

  : User-provided endpoint (e.g.,

  http://<node-ip>:30920

  )

• <OPENSEARCH_USER>

  : User-provided username (default:

  admin

  )

• <OPENSEARCH_PASSWORD>

  : User-provided password

• <TARGET_CLUSTERS>

  : User-confirmed cluster names

⚠️ DO NOT generate InstallPlan until all placeholders have real values.

Wait for Deployment

After applying InstallPlan, you MUST wait for deployment to complete:

# Wait for Vector pods to be ready (on each cluster)
kubectl wait --for=condition=Ready pods -n kubesphere-logging-system -l app.kubernetes.io/instance=vector --timeout=300s

# Verify deployment status
kubectl get pods -n kubesphere-logging-system -l app.kubernetes.io/instance=vector

Show deployment summary to user:

• Which clusters Vector was deployed to
• OpenSearch endpoint used
• Pod status (Ready/Total)

Enable Metrics Export

apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
  namespace: kubesphere-system
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
      exportMetrics:
        enabled: true
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Configuration Parameters

Agent Parameters

agent.role

agent.image.tag

agent.resources.requests.cpu

agent.resources.requests.memory

agent.resources.limits.cpu

agent.resources.limits.memory

agent.service.ports

agent.exportMetrics.enabled

Agent Sinks OpenSearch Parameters

agent.sinks.opensearch.endpoints

agent.sinks.opensearch.auth.strategy

basic

agent.sinks.opensearch.auth.user

agent.sinks.opensearch.auth.password

agent.sinks.opensearch.tls.verify

Example:

agent:
  sinks:
    opensearch:
      endpoints:
        - http://<node-ip>:30920
      auth:
        strategy: basic
        user: admin
        password: admin
      tls:
        verify: false

Docker Root Directory Configuration

If Docker root directory is not

/var/lib

agent:
  extraVolumes:
    - name: docker-root
      hostPath:
        path: /path/to/docker
        type: ''
  extraVolumeMounts:
    - name: docker-root
      mountPath: /path/to/docker

Extension Operations

Check Extension Status

# View extension installation status
kubectl get installplan -n kubesphere-system -l extension.kubesphere.io/name=vector

# View extension version
kubectl get extensionversions -n kubesphere-system vector

Check Pod Status

# View all Vector pods
kubectl get pods -n kubesphere-logging-system -l app.kubernetes.io/name=vector

# View agent pods
kubectl get pods -n kubesphere-logging-system -l app.kubernetes.io/name=vector,app.kubernetes.io/component=agent

View Logs

# View agent logs
kubectl logs -n kubesphere-logging-system -l app.kubernetes.io/name=vector,app.kubernetes.io/component=agent --tail=100

Update Configuration

apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
  namespace: kubesphere-system
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Uninstall Extension

Uninstall from all clusters:

kubectl delete installplan -n kubesphere-system vector

Uninstall from specific cluster:

To remove Vector from a specific cluster, update the InstallPlan by removing that cluster from

clusterScheduling.placement.clusters

apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
  namespace: kubesphere-system
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
  clusterScheduling:
    placement:
      clusters:
        - <REMAINING_CLUSTERS>  # Remove the cluster you want to uninstall from

Important Notes

1. Dependency: Vector is a core dependency for WizTelemetry extensions. Install it first before installing Logging, Auditing, Events, or Notification.
2. OpenSearch Required: User must provide OpenSearch endpoint and credentials.
3. Multicluster: The extension uses

   installationMode: Multicluster

   :

   • agent

     (tag: agent) is deployed to all selected member clusters
4. Agent Scheduling: Agent pods have affinity to avoid edge nodes and tolerate all taints.
5. Cross-cluster Access: Ensure OpenSearch endpoint is accessible from all Vector clusters.

Troubleshooting

Check Vector Configuration

# View Vector configmap
kubectl get configmap -n kubesphere-logging-system -l app.kubernetes.io/name=vector

# View specific config
kubectl get configmap -n kubesphere-logging-system vector-config -o yaml

Verify Sinks

# Check if sinks are configured correctly
kubectl get secret -n kubesphere-logging-system vector-sinks -o yaml

Common Issues