Advanced Kubernetes: Operators & CRDs

Level 1: Quick Reference

Core Concepts at a Glance

Custom Resource Definitions (CRDs) extend Kubernetes API with custom object types. Operators are controllers that manage these custom resources using domain-specific logic.

CRD vs ConfigMap Comparison:

Aspect	CRD	ConfigMap
API Integration	Full Kubernetes API support (CRUD, watch, RBAC)	Simple key-value storage
Validation	OpenAPI v3 schema validation, admission webhooks	No built-in validation
Versioning	Multiple versions with conversion webhooks	Single version only
Use Case	Complex application state, declarative APIs	Configuration data, environment variables
Controller Support	Reconciliation loops, status tracking	Manual polling required
Example	Database instances, ML workflows, backup policies	App config files, feature flags

Operator Pattern Overview

┌─────────────────────────────────────────────────────────┐
│                    Kubernetes API Server                 │
│  (stores desired state in etcd)                         │
└────────────┬────────────────────────────┬───────────────┘
             │                            │
             │ Watch                      │ Update Status
             ↓                            ↑
    ┌────────────────┐         ┌─────────────────────┐
    │   Controller   │────────→│  External Resources │
    │  (Reconcile)   │  Manage │  (DBs, APIs, etc.)  │
    └────────────────┘         └─────────────────────┘
         ↑
         │ Compare
         │
    ┌────┴─────┐
    │ Desired  │
    │ vs Actual│
    └──────────┘

Reconciliation Loop:

1. Watch - Controller watches for changes to custom resources
2. Compare - Reconcile function compares desired vs actual state
3. Act - Controller takes actions to align actual state with desired
4. Update Status - Controller updates resource status with current state
5. Requeue - Schedule next reconciliation (periodic or event-driven)

Controller Reconciliation Logic

// Simplified reconciliation pattern
func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. Fetch the custom resource
    obj := &myapi.MyResource{}
    if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Handle deletion (finalizers)
    if !obj.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, obj)
    }

    // 3. Reconcile external state
    if err := r.reconcileExternal(ctx, obj); err != nil {
        return ctrl.Result{}, err
    }

    // 4. Update status
    obj.Status.Ready = true
    if err := r.Status().Update(ctx, obj); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil // Success, no requeue
}

Essential Checklist

Prerequisites:

• Kubernetes cluster (v1.25+) - local (kind, minikube) or remote
• kubectl configured with admin access
• Go 1.21+ installed
• Docker/Podman for building operator images

Development Tools:

• kubebuilder

  (v3.12+) - scaffolding and code generation

• operator-sdk

  (optional) - alternative framework

• controller-gen

  - generates CRDs, RBACs, webhooks

• kustomize

  - manages Kubernetes manifests

Testing Tools:

• envtest

  - runs API server locally for unit tests

• kind

  - Kubernetes in Docker for integration tests

• ginkgo

  - BDD testing framework (optional)

Key Files in Operator Project:

my-operator/
├── api/v1/           # CRD definitions (Go structs)
├── config/
│   ├── crd/          # Generated CRD YAML
│   ├── rbac/         # Generated RBAC YAML
│   ├── manager/      # Operator deployment
│   └── webhook/      # Webhook configurations
├── controllers/      # Reconciliation logic
├── main.go           # Entrypoint (manager setup)
└── Dockerfile        # Container image build

Quick Commands:

# Initialize operator project
kubebuilder init --domain example.com --repo github.com/myorg/my-operator

# Create CRD + controller
kubebuilder create api --group apps --version v1 --kind MyApp

# Generate manifests
make manifests

# Run locally (connects to current kubeconfig cluster)
make install run

# Run tests
make test

# Build and deploy
make docker-build docker-push deploy IMG=myregistry/my-operator:v1.0.0

Common Pitfalls:

• ❌ Forgetting to update CRD when changing API structs → run

  make manifests

• ❌ Infinite reconciliation loops → use

  ctrl.Result{RequeueAfter: time.Minute}

• ❌ Not handling deletion properly → implement finalizers
• ❌ Blocking operations in reconcile → use background workers for long tasks
• ❌ Not setting owner references → orphaned resources on deletion

When to Use Operators:

• ✅ Managing complex stateful applications (databases, message queues)
• ✅ Automating operational tasks (backups, upgrades, scaling)
• ✅ Integrating with external systems (cloud APIs, SaaS platforms)
• ✅ Enforcing organizational policies (cost controls, security standards)
• ❌ Simple deployments (use Helm or plain manifests)
• ❌ One-time configuration changes (use Jobs or manual kubectl)

---

Level 2: Implementation Guide

📚 Complete Examples: See REFERENCE.md for full controller implementations, webhook code, test suites, and production-ready patterns.

2.1 Custom Resource Definitions (CRDs)

CRDs extend Kubernetes API with custom object types validated by OpenAPI v3 schemas.

Key Components:

• Spec - Desired state (user input)
• Status - Observed state (controller output, separate subresource)
• Validation - Markers like

  +kubebuilder:validation:Minimum=1

• Versions - Support multiple API versions with conversion webhooks

Essential Kubebuilder Markers:

// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=10
Size int32 `json:"size"`

// +kubebuilder:validation:Pattern=`^[a-z0-9.-]+/[a-z0-9.-]+:[a-z0-9.-]+$`
Image string `json:"image"`

// +optional
Port int32 `json:"port,omitempty"`

Printcolumns for

kubectl get

:

// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`

Subresources:

• +kubebuilder:subresource:status

  - Separate status endpoint

• +kubebuilder:subresource:scale

  - Enable

  kubectl scale

Generate CRDs:

make manifests

→ outputs to

config/crd/bases/

See REFERENCE.md for complete CRD definition, versioning, and conversion webhooks.

---

2.2 Operators and Controllers

Reconciliation Loop:

1. Watch - Controller watches for resource changes (via informers/caches)
2. Compare - Reconcile compares desired vs actual state
3. Act - Create/update/delete Kubernetes resources to match desired state
4. Update Status - Set status conditions (

   Ready

   ,

   Progressing

   ,

   Degraded

   )
5. Requeue - Schedule next reconciliation (event-driven or periodic)

Controller Pattern:

func (r *Reconciler) Reconcile(ctx, req) (ctrl.Result, error) {
    // 1. Fetch custom resource
    obj := &MyResource{}
    if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Handle deletion (finalizers)
    if !obj.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, obj)
    }

    // 3. Reconcile external state
    if err := r.reconcileDeployment(ctx, obj); err != nil {
        return ctrl.Result{RequeueAfter: 30*time.Second}, err
    }

    // 4. Update status
    obj.Status.Ready = true
    return ctrl.Result{}, r.Status().Update(ctx, obj)
}

Key Functions:

• controllerutil.CreateOrUpdate()

  - Idempotent create/update

• controllerutil.SetControllerReference()

  - Automatic garbage collection

• controllerutil.AddFinalizer()

  - Cleanup before deletion

Error Handling:

• Transient errors - Requeue with delay:

  ctrl.Result{RequeueAfter: 30s}

• Permanent errors - Set degraded condition, don't requeue
• Unknown errors - Return error for exponential backoff

See REFERENCE.md for complete controller implementation with finalizers, owner references, and error handling.

---

2.3 Admission Webhooks

Webhooks intercept API requests before persistence for validation/mutation.

Types:

• Validating - Accept/reject requests (JWT validation, cross-field checks)
• Mutating - Modify requests (inject sidecars, set defaults)

Implementation:

// Validating webhook
func (r *MyApp) ValidateCreate() (admission.Warnings, error) {
    if r.Spec.Size < 1 || r.Spec.Size > 100 {
        return nil, fmt.Errorf("size must be 1-100")
    }
    return nil, nil
}

// Mutating webhook (Defaulter)
func (r *MyApp) Default() {
    if r.Spec.Port == 0 {
        r.Spec.Port = 8080
    }
}

Setup:

1. Implement

   webhook.Validator

   or

   webhook.Defaulter

   interface
2. Add kubebuilder marker:

   // +kubebuilder:webhook:path=/validate-...,mutating=false,...

3. make manifests

   generates webhook config
4. Deploy with cert-manager for TLS certificates

Requirements:

• TLS certificates (use cert-manager)
• Service routing webhook traffic to operator

• failurePolicy: fail

  (default) - reject on webhook errors

See REFERENCE.md for complete webhook examples, cert-manager setup, and validation patterns.

---

2.4 Leader Election & High Availability

Leader election ensures only one controller instance reconciles at a time (prevents race conditions).

Configuration:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    LeaderElection:          true,
    LeaderElectionID:        "myapp-controller.example.com",
    LeaderElectionNamespace: "myapp-system",
})

How It Works:

• Uses Kubernetes

  Lease

  resource for coordination
• One replica acquires lease, becomes leader
• Other replicas standby, ready to take over on leader failure
• Leader renews lease every 10s (default)

Deployment:

spec:
  replicas: 3  # High availability
  containers:
  - args:
    - --leader-elect

See REFERENCE.md for RBAC requirements and lease configuration tuning.

---

2.5 Testing Operators

Unit Testing with envtest:

• Runs local API server (no kubelet, no containers)
• Fast tests (milliseconds per test)
• Full CRD validation

Setup:

testEnv = &envtest.Environment{
    CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
}
cfg, _ := testEnv.Start()
k8sClient, _ = client.New(cfg, client.Options{Scheme: scheme.Scheme})

Test Pattern:

It("Should create Deployment", func() {
    myApp := &MyApp{...}
    Expect(k8sClient.Create(ctx, myApp)).Should(Succeed())

    deployment := &Deployment{}
    Eventually(func() error {
        return k8sClient.Get(ctx, namespacedName, deployment)
    }, timeout, interval).Should(Succeed())

    Expect(*deployment.Spec.Replicas).To(Equal(int32(3)))
})

Integration Testing with kind:

kind create cluster
make docker-build docker-push deploy IMG=operator:test
kubectl wait --for=condition=available deployment/operator
kubectl apply -f test-cr.yaml

See REFERENCE.md for complete test suites, ginkgo patterns, and E2E test scripts.

---

2.6 Best Practices & Anti-Patterns

✅ Best Practices:

• Idempotent reconciliation - Same result on multiple calls
• Use

  CreateOrUpdate

  - Simplifies create/update logic
• Set owner references - Automatic garbage collection
• Finalizers for cleanup - External resources (cloud APIs, databases)
• Status conditions -

  Ready

  ,

  Progressing

  ,

  Degraded

  with detailed messages
• Structured logging - JSON format with consistent key-value pairs

❌ Anti-Patterns:

• Blocking operations - Don't make sync calls that block reconcile
• Infinite loops - Updating spec in reconcile triggers another reconcile
• Hardcoded values - Use env vars/ConfigMaps
• Missing watches - Ensure RBAC allows watching dependent resources
• No health checks - Implement

  /healthz

  and

  /readyz

  endpoints

Requeue Strategies:

// Immediate requeue (rate-limited)
return ctrl.Result{Requeue: true}, nil

// Requeue after delay
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

// No requeue (wait for watch event)
return ctrl.Result{}, nil

// Error (exponential backoff)
return ctrl.Result{}, fmt.Errorf("transient error")

See REFERENCE.md for advanced patterns, multi-cluster operators, and OLM integration.

---

Level 3: Deep Dive Resources

Advanced Operator Patterns

State Machine Operators

• Model complex workflows as finite state machines

• Use status phases to track progression through states

• Implement state transition validations and guards

Multi-Tenancy Operators

• Namespace isolation strategies

• Shared vs dedicated operator deployments

• RBAC scoping for tenant-specific resources

GitOps Integration

• Reconcile against Git repository state

• Implement drift detection and auto-remediation

• Use annotations to track source commits

External Secret Management

• Integrate with Vault, AWS Secrets Manager, or Azure Key Vault

• Implement secret rotation without downtime

• Use external-secrets operator pattern

Multi-Cluster Operators

Architecture Patterns:

1. Hub-Spoke Model - Central operator manages multiple clusters

2. Federated Model - Operators in each cluster coordinate via shared state

3. Active-Active - Operators in multiple clusters handle same resources

Implementation Considerations:

• Use cluster-api for cluster lifecycle management

• Implement cross-cluster service discovery (e.g., Submariner)

• Handle network partitions and split-brain scenarios

• Use consensus protocols for distributed state

Tools:

• KubeFed (deprecated) - Kubernetes Federation v2

• OCM (Open Cluster Management) - CNCF sandbox project

• Argo CD ApplicationSet - Multi-cluster GitOps

• Crossplane - Universal control plane for multi-cloud

Operator Lifecycle Manager (OLM)

What is OLM?

• Package manager for Kubernetes operators

• Handles installation, upgrades, and dependency management

• Used by OpenShift and available as CNCF project

OLM Components:

• Catalog - Repository of operator metadata (CSV, CRD)

• Subscription - Declarative operator installation

• InstallPlan - Execution plan for operator installation

• ClusterServiceVersion (CSV) - Operator metadata and deployment info

Creating an OLM Bundle:

# Generate bundle manifests

operator-sdk generate bundle --version 1.0.0



# Validate bundle

operator-sdk bundle validate ./bundle



# Build and push bundle image

docker build -f bundle.Dockerfile -t myregistry/myapp-operator-bundle:v1.0.0 .

docker push myregistry/myapp-operator-bundle:v1.0.0



# Add to catalog

opm index add --bundles myregistry/myapp-operator-bundle:v1.0.0 \

  --tag myregistry/myapp-catalog:latest

OLM Best Practices:

• Define proper upgrade paths in CSV

• Test upgrade scenarios (skip versions, downgrades)

• Use semantic versioning

• Document breaking changes in release notes

Advanced Testing Strategies

Property-Based Testing:

• Use tools like

  gopter

  for property-based tests

• Test invariants across state transitions

• Generate random valid/invalid inputs

Chaos Testing:

• Use Chaos Mesh or Litmus to inject failures

• Test operator resilience to node failures, network partitions

• Verify recovery from partial updates

Performance Testing:

• Benchmark reconciliation loop latency

• Test with 1000+ custom resources

• Measure memory/CPU usage under load

• Use profiling tools (pprof) for bottleneck analysis

Production Readiness Checklist

Observability:

• Metrics exported via Prometheus endpoint

• Structured logging with levels (info, warn, error)

• Distributed tracing (OpenTelemetry)

• Custom metrics for business logic (e.g., backup success rate)

Security:

• RBAC follows least-privilege principle

• Secrets encrypted at rest and in transit

• Pod Security Standards enforced

• Network policies restrict traffic

• Image vulnerability scanning in CI/CD

Reliability:

• Leader election enabled for HA

• Graceful shutdown with finalizers

• Rate limiting to prevent API server overload

• Circuit breakers for external dependencies

• Backup/restore procedures documented

Operational:

• Runbooks for common failure scenarios

• SLO/SLI definitions (e.g., 99.9% reconciliation success)

• Alerting rules for critical conditions

• Upgrade/rollback procedures tested

• Capacity planning documented

Resources and Further Learning

Official Documentation:

• Kubebuilder Book

• Operator SDK Documentation

• Kubernetes Controller Runtime

Bundled Resources in This Directory:

1. templates/crd-definition.yaml

   - Complete CRD with OpenAPI schema

2. templates/operator-scaffold.go

   - Controller with reconcile logic

3. templates/webhook.go

   - Validating and mutating webhooks

4. templates/rbac.yaml

   - RBAC manifests for operator deployment

5. scripts/setup-operator-dev.sh

   - Development environment setup

6. resources/operator-patterns.md

   - Common patterns and anti-patterns

Community Resources:

• CNCF Operator White Paper

• Awesome Operators

• Kubernetes Slack #kubebuilder

Example Production Operators:

• etcd-operator

• prometheus-operator

• cert-manager

• strimzi-kafka-operator

Next Steps

1. Build a Simple Operator - Start with a basic CRD and controller

2. Add Validation - Implement admission webhooks

3. Test Thoroughly - Write unit tests with envtest, integration tests with kind

4. Observe in Production - Deploy with metrics, logging, and tracing

5. Iterate - Add features based on operational experience

Advanced Topics to Explore:

• Custom admission plugins

• API aggregation and extension API servers

• Operator Hub and OLM

• Multi-cluster federation

• Operator performance optimization