GitOps Patterns

Best practices for operating infrastructure and applications through Git as the single source of truth. Covers ArgoCD, Flux, Argo Rollouts, environment promotion, secret management, and progressive delivery.

GitOps Principles

GitOps extends Infrastructure as Code by adding a reconciliation loop that continuously ensures the live system matches the declared state in Git.

git revert

Repository Structure

Separate application source code from deployment manifests. This is the most critical architectural decision in GitOps.

# Deployment repo (gitops-deploy) -- separate from application source
gitops-deploy/
  apps/
    backend/
      base/
        deployment.yaml
        service.yaml
        kustomization.yaml
      overlays/
        dev/
          kustomization.yaml       # image: backend:dev-abc1234
        staging/
          kustomization.yaml       # image: backend:staging-def5678
        prod/
          kustomization.yaml       # image: backend:v1.2.3
          patches/
            replicas.yaml
            hpa.yaml
    frontend/
      base/
      overlays/
  infrastructure/
    cert-manager/
    ingress-nginx/
    monitoring/
  clusters/
    dev/
      kustomization.yaml          # References apps/*/overlays/dev
    staging/
      kustomization.yaml
    prod/
      kustomization.yaml

ArgoCD Application Manifest

ArgoCD watches Git repositories and synchronizes Kubernetes cluster state to match.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: backend-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: production

  source:
    repoURL: https://github.com/myorg/gitops-deploy.git
    targetRevision: main
    path: apps/backend/overlays/prod
    kustomize:
      images:
        - backend=ghcr.io/myorg/backend:v1.2.3

  destination:
    server: https://kubernetes.default.svc
    namespace: backend

  syncPolicy:
    automated:
      prune: true          # Delete resources removed from Git
      selfHeal: true       # Revert manual changes in cluster
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas   # Ignore HPA-managed replica count

ArgoCD ApplicationSet for Multi-Environment

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: backend
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - env: dev
            cluster: https://dev-cluster.example.com
            autoSync: true
          - env: staging
            cluster: https://staging-cluster.example.com
            autoSync: true
          - env: prod
            cluster: https://prod-cluster.example.com
            autoSync: false   # Prod requires manual sync
  template:
    metadata:
      name: 'backend-{{env}}'
    spec:
      project: '{{env}}'
      source:
        repoURL: https://github.com/myorg/gitops-deploy.git
        targetRevision: main
        path: 'apps/backend/overlays/{{env}}'
      destination:
        server: '{{cluster}}'
        namespace: backend
      syncPolicy:
        automated:
          prune: '{{autoSync}}'
          selfHeal: '{{autoSync}}'

Flux Kustomization

Flux uses native Kubernetes custom resources and supports multi-tenancy out of the box.

# Source: Git repository to watch
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: gitops-deploy
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/gitops-deploy.git
  ref:
    branch: main
  secretRef:
    name: gitops-deploy-auth

---
# Kustomization: what to deploy from that source
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: backend-prod
  namespace: flux-system
spec:
  interval: 5m
  retryInterval: 2m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: gitops-deploy
  path: ./apps/backend/overlays/prod
  prune: true
  targetNamespace: backend
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: backend
      namespace: backend
  dependsOn:
    - name: infrastructure
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-config

---
# Image automation: watch for new tags, update Git
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: backend
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: backend
  policy:
    semver:
      range: '>=1.0.0'

Progressive Delivery with Argo Rollouts

Argo Rollouts extends Kubernetes Deployments with canary releases, blue-green deployments, and automated analysis.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: backend
spec:
  replicas: 10
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: ghcr.io/myorg/backend:v1.2.3
          ports:
            - containerPort: 8080
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits: { cpu: 500m, memory: 512Mi }

  strategy:
    canary:
      canaryService: backend-canary
      stableService: backend-stable
      trafficRouting:
        istio:
          virtualServices:
            - name: backend-vsvc
              routes: [primary]
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

---
# AnalysisTemplate -- automated success criteria
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 30s
      count: 5
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))
    - name: p99-latency
      interval: 30s
      count: 5
      successCondition: result[0] <= 500
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_ms_bucket{service="{{args.service-name}}"}[2m])) by (le)
            )

Reconciliation Loop

The reconciliation loop is the core mechanism that makes GitOps self-healing.

         +----------------+
         |   Git Repo     |
         | (desired state)|
         +-------+--------+
                 |  poll / webhook
         +-------+--------+
         |  GitOps Agent   |
         | (ArgoCD / Flux) |
         +-------+--------+
                 |  compare desired vs actual
         +-------+--------+
         |   Diff Engine   |----> No diff? Sleep, poll again.
         +-------+--------+
                 |  diff detected
         +-------+--------+
         |  Apply Changes  |
         +-------+--------+
                 |
         +-------+--------+
         | Health Checks   |----> Unhealthy? Retry or alert.
         +-------+--------+
                 |  healthy
         +-------+--------+
         |  Mark Synced    |
         +----------------+

spec.interval

selfHeal: true

spec.force: true

prune: true

spec.prune: true

retry.limit

retryInterval

Environment Promotion Strategies

PR-Based Promotion Pipeline

name: Promote
on:
  workflow_dispatch:
    inputs:
      image_tag:
        description: 'Image tag to promote'
        required: true
      target_env:
        type: choice
        options: [staging, prod]

jobs:
  promote:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          repository: myorg/gitops-deploy
          token: ${{ secrets.GITOPS_PAT }}

      - name: Update image tag
        run: |
          cd apps/backend/overlays/${{ inputs.target_env }}
          kustomize edit set image \
            backend=ghcr.io/myorg/backend:${{ inputs.image_tag }}

      - uses: peter-evans/create-pull-request@v6
        with:
          title: "promote: backend ${{ inputs.image_tag }} to ${{ inputs.target_env }}"
          branch: "promote/backend-${{ inputs.target_env }}-${{ inputs.image_tag }}"
          labels: promotion,${{ inputs.target_env }}

Secret Management in GitOps

Secrets cannot be stored in plain text in Git. Multiple solutions exist with different trade-offs.

Sealed Secrets

# Create a regular secret, then seal it for safe Git storage
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=s3cret \
  --dry-run=client -o yaml | \
  kubeseal --format yaml \
    --controller-name sealed-secrets \
    --controller-namespace flux-system \
    > apps/backend/overlays/prod/sealed-db-credentials.yaml

External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: backend
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/backend/db
        property: username
    - secretKey: password
      remoteRef:
        key: prod/backend/db
        property: password

SOPS with Flux Decryption

# Flux Kustomization with SOPS decryption
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: backend-prod
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: gitops-deploy
  path: ./apps/backend/overlays/prod
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age-key   # Secret containing the age private key

Anti-Patterns

kubectl apply

kubectl apply

selfHeal: true

prune: true

kubectl

latest

GitOps Readiness Checklist

• All application manifests stored in a Git repository (not applied manually)
• Deployment repository is separate from application source code
• Directory-per-environment structure with Kustomize overlays or Helm values
• ArgoCD or Flux installed and configured to watch the deployment repo
• Auto-sync enabled with self-heal and prune for all environments
• Health checks configured for all synced applications
• Secret management solution deployed (Sealed Secrets, SOPS, or External Secrets)
• No plain-text secrets committed to any Git repository
• Environment promotion workflow defined (PR-based or automated)
• Image tags are immutable (semver, SHA, or build ID -- never

  latest

  )
• RBAC configured: AppProjects (ArgoCD) or namespaced Kustomizations (Flux)
• Notifications configured for sync failures and degraded health

• kubectl

  write access restricted in GitOps-managed clusters
• Rollback tested:

  git revert

  triggers successful rollback deployment
• Disaster recovery plan: GitOps agent recovery, state rebuild from Git
• Progressive delivery configured for production (canary or blue-green)