Advanced Kubernetes Topics

Explore advanced Kubernetes concepts including multi-cluster management, Custom Resource Definitions (CRDs), CI/CD integration, and advanced scheduling techniques.

Table of Contents

1. Multi-Cluster Management

Managing multiple Kubernetes clusters is common in enterprise environments for:

  • High availability and disaster recovery
  • Geographic distribution
  • Environment separation (dev, staging, prod)
  • Compliance and data residency

1.1 Multi-Cluster Strategies

  • Federation: Kubernetes Cluster Federation (deprecated)
  • Service Mesh: Istio Multi-Cluster
  • GitOps: ArgoCD Multi-Cluster
  • Kubefed: Kubernetes Federation v2

1.2 ArgoCD Multi-Cluster

# Add cluster to ArgoCD
argocd cluster add <context-name>

# List clusters
argocd cluster list

# Create application targeting multiple clusters
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  project: default
  source:
    repoURL: https://github.com/example/repo
    targetRevision: HEAD
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

1.3 Cluster Context Management

# List contexts
kubectl config get-contexts

# Switch context
kubectl config use-context <context-name>

# Set default namespace for context
kubectl config set-context <context-name> --namespace=production

# Use context for single command
kubectl get pods --context=<context-name>

2. Custom Resource Definitions

Custom Resource Definitions (CRDs) extend the Kubernetes API to add custom resource types.

2.1 Creating CRDs

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              databaseName:
                type: string
              replicas:
                type: integer
                minimum: 1
                maximum: 10
          status:
            type: object
            properties:
              phase:
                type: string
                enum: ["Pending", "Running", "Failed"]
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames:
    - db

2.2 CRD Validation

Use OpenAPI schema for validation:

  • Type validation
  • Required fields
  • Enum values
  • Min/max constraints
  • Pattern matching

2.3 CRD Versioning

Support multiple versions with conversion: 

spec:
  group: example.com
  versions:
  - name: v1
    served: true
    storage: false  # Not stored, converted from v2
    schema:
      # v1 schema
  - name: v2
    served: true
    storage: true  # Storage version
    schema:
      # v2 schema
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          namespace: default
          name: conversion-webhook

3. CI/CD Pipelines

Integrating Kubernetes with CI/CD pipelines enables automated deployment:

3.1 GitOps Approach

GitOps uses Git as the source of truth:

  • Configuration stored in Git
  • Automated synchronization
  • Declarative approach
  • Audit trail

3.2 ArgoCD

# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Get admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Port forward
kubectl port-forward svc/argocd-server -n argocd 8080:443

3.3 Tekton Pipelines

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: build-and-deploy
spec:
  tasks:
  - name: build
    taskRef:
      name: buildah
    params:
    - name: IMAGE
      value: myapp:latest
  - name: deploy
    runAfter:
    - build
    taskRef:
      name: kubectl-apply
    params:
    - name: MANIFEST
      value: k8s/deployment.yaml

3.4 Jenkins X

Jenkins X provides automated CI/CD for Kubernetes:

  • Automated pipeline creation
  • Preview environments
  • Promotion between environments
  • GitOps integration

4. Advanced Scheduling

4.1 Node Affinity

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: zone
            operator: In
            values:
            - us-west-1a
  containers:
  - name: app
    image: myapp:1.0

4.2 Pod Affinity/Anti-Affinity

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - cache
        topologyKey: zone
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - web
          topologyKey: kubernetes.io/hostname
  containers:
  - name: app
    image: myapp:1.0

4.3 Taints and Tolerations

# Taint a node
kubectl taint nodes node1 key=value:NoSchedule

# Remove taint
kubectl taint nodes node1 key:NoSchedule-

# Pod with toleration
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  tolerations:
  - key: "key"
    operator: "Equal"
    value: "value"
    effect: "NoSchedule"
  containers:
  - name: app
    image: myapp:1.0

5. Admission Controllers

Admission Controllers intercept requests to the API Server and can modify or reject them.

5.1 Types of Admission Controllers

  • Validating: Validate requests (can reject)
  • Mutating: Modify requests before persistence

5.2 ValidatingAdmissionWebhook

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy
webhooks:
- name: validate-pod.example.com
  clientConfig:
    service:
      name: webhook-service
      namespace: default
      path: "/validate"
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE", "UPDATE"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None

5.3 OPA Gatekeeper

OPA Gatekeeper provides policy enforcement: 

# Install Gatekeeper
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.14/deploy/gatekeeper.yaml

# Example ConstraintTemplate
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          required := input.parameters.labels
          provided := input.review.object.metadata.labels
          missing := required[_]
          not provided[missing]
          msg := sprintf("Missing required label: %v", [missing])
        }

6. Resource Quotas

Resource Quotas limit resource consumption per namespace:

6.1 ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: production
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    persistentvolumeclaims: "10"
    pods: "20"
    services: "10"

6.2 LimitRange

LimitRange sets default resource limits for Pods: 

apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
  namespace: default
spec:
  limits:
  - default:
      memory: "512Mi"
      cpu: "500m"
    defaultRequest:
      memory: "256Mi"
      cpu: "250m"
    type: Container

7. Custom Controllers

Custom controllers watch and reconcile custom resources:

7.1 Controller Pattern

Controllers follow this pattern:

  1. Watch for resource changes
  2. Compare current state with desired state
  3. Take action to reconcile
  4. Update status
  5. Repeat

7.2 Using Controller Runtime

Use controller-runtime library to build controllers: 

// Example controller
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1.Database{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

8. Best Practices

8.1 Use GitOps

Adopt GitOps for configuration management. Use tools like ArgoCD or Flux.

8.2 Implement Resource Quotas

Set ResourceQuotas and LimitRanges to prevent resource exhaustion.

8.3 Use Admission Controllers

Implement policies using admission controllers (OPA Gatekeeper, Kyverno).

8.4 Plan Multi-Cluster Strategy

Plan your multi-cluster approach early. Consider federation, service mesh, or GitOps.

8.5 Version CRDs

Plan for CRD versioning and conversion from the start.

8.6 Test Controllers Thoroughly

Write comprehensive tests for custom controllers and CRDs.

8.7 Monitor Custom Resources

Monitor custom resources and controller health. Set up alerts.

Summary: Advanced Kubernetes topics include multi-cluster management, CRDs, CI/CD integration, advanced scheduling, admission controllers, and custom controllers. Use GitOps for configuration management, implement resource quotas, and plan for scalability from the start.

Tags:

Post a Comment

0 Comments