Unit 5.3: GitOps and Deployment Operations

Introduction

You have a working TaskBoard deployment on Managed Kubernetes, a CI/CD pipeline that builds and pushes images, and Terraform that provisions the cluster. The problem is drift. Someone runs kubectl edit to hotfix a production deployment, the live state diverges from what is in Git, and the next pipeline run silently reverts it or, worse, fails. GitOps fixes this by making a Git repository the single source of truth and running a controller inside the cluster that continuously pulls the desired state and reconciles the live state to match it.

In this unit you set up ArgoCD on your MKS cluster, wire it to TaskBoard's manifest repository, and watch it self-heal manual changes. You separate the repository that Terraform owns (the cluster, networking, databases) from the repository ArgoCD owns (Kubernetes manifests), so the two controllers never overwrite each other. Finally you automate branch-environment teardown so a feature branch spins up its own namespace and tears it down on merge or close.

1. GitOps Principles and the Pull-Based Reconciliation Loop

GitOps inverts the usual deployment direction. Instead of a CI runner pushing kubectl apply into the cluster from the outside (push model), a controller running inside the cluster pulls the desired state from Git and applies it (pull model). The Git commit is the deployment trigger, the audit log, and the rollback mechanism all at once. To roll back, you revert the commit and the controller reconciles back to the previous state.

Three properties define a GitOps system: the entire desired state is declarative and stored in Git, the controller continuously compares desired state against live state, and any divergence (drift) is either reported or automatically corrected. On IONOS MKS this matters because the cluster already runs reconciliation loops of its own during the weekly maintenance window, so your application reconciliation has to coexist with platform-driven changes.

1.1 Declarative State in Git

Everything ArgoCD manages is plain Kubernetes YAML committed to a repository. There is no imperative kubectl run in a GitOps workflow. A minimal TaskBoard deployment manifest looks like this:

# apps/taskboard/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: taskboard-api
  namespace: taskboard
spec:
  replicas: 2
  selector:
    matchLabels:
      app: taskboard-api
  template:
    metadata:
      labels:
        app: taskboard-api
    spec:
      imagePullSecrets:
        - name: cr-pull-secret
      containers:
        - name: api
          image: my-registry.cr.de-fra.ionos.com/taskboard-api:GIT_SHA
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5

The image tag carries a git SHA rather than latest. This is the promotion lever: changing the tag in Git is what triggers a new rollout, and the tag tells you exactly which commit is running.

1.2 Drift Detection and Self-Heal

When someone runs kubectl scale deployment/taskboard-api --replicas=5 directly against the cluster, the live state no longer matches Git, which still says replicas: 2. A GitOps controller in self-heal mode detects this difference on its next sync cycle and scales back down to 2. The lesson for your team is operational: the cluster is read-only for humans. All changes go through a pull request.

# This manual change will be reverted by ArgoCD self-heal
kubectl -n taskboard scale deployment/taskboard-api --replicas=5

# Within the sync interval, ArgoCD reports OutOfSync, then heals back to 2
argocd app get taskboard --refresh

2. Installing and Configuring ArgoCD on MKS

ArgoCD is a CNCF project, not an IONOS product, so it installs onto MKS the same way it installs onto any conformant Kubernetes cluster. The IONOS-specific parts are how you obtain the kubeconfig, how the ArgoCD server gets exposed (which depends on the MKS LoadBalancer model covered in Section 4), and how image pulls authenticate against IONOS Container Registry.

2.1 Retrieving the kubeconfig and Installing ArgoCD

You provision the cluster with Terraform and retrieve the kubeconfig before installing anything. IONOS documents three configuration-management paths for retrieving the kubeconfig file: the ionosctl CLI, Ansible, and Terraform. With Terraform, the ionoscloud_k8s_cluster data source exposes the kubeconfig as an attribute you can write to a file.

data "ionoscloud_k8s_cluster" "taskboard" {
  id = ionoscloud_k8s_cluster.taskboard.id
}

resource "local_file" "kubeconfig" {
  content         = data.ionoscloud_k8s_cluster.taskboard.kube_config
  filename        = "${path.module}/kubeconfig.yaml"
  file_permission = "0600"
}

With the kubeconfig in place, install ArgoCD into its own namespace:

export KUBECONFIG=./kubeconfig.yaml

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for the API server to come up on the worker nodes
kubectl -n argocd rollout status deployment/argocd-server

The ArgoCD pods schedule onto your MKS worker nodes. The control plane components that IONOS manages (the K8s API server, CSI, CCM, Calico, and CoreDNS) are updated by IONOS during maintenance and are not something ArgoCD touches.

2.2 Defining the Application CRD

ArgoCD's core object is the Application custom resource. It points at a Git repository, a path within it, and a destination cluster and namespace. automated sync with prune and selfHeal is what turns it into a true GitOps loop: prune deletes resources removed from Git, and selfHeal reverts manual cluster changes.

# argocd/taskboard-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: taskboard
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/taskboard-manifests.git
    targetRevision: main
    path: overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: taskboard
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Apply it with kubectl apply -n argocd -f argocd/taskboard-app.yaml. From this point, every commit to the overlays/prod path triggers reconciliation.

3. Repository Topology: Separating Infrastructure from Applications

The most common GitOps mistake on IONOS is mixing Terraform-managed infrastructure and ArgoCD-managed applications in one place, then watching them fight. The clean pattern is two repositories with a clear ownership boundary.

The infrastructure repository holds Terraform: ionoscloud_k8s_cluster, ionoscloud_k8s_node_pool, networking, databases, and the Container Registry. It applies on merge to main through your CI pipeline. The application repository holds Kubernetes manifests and Kustomize overlays, and ArgoCD watches it.

3.1 Ownership Boundary

The rule is simple: if a resource is created by terraform apply, ArgoCD must never manage it, and if a resource is created by ArgoCD sync, Terraform must never manage it. The handoff happens through outputs. Terraform produces the cluster and the registry pull secret; ArgoCD consumes the cluster and deploys into it.

# infrastructure repo: outputs that the app layer consumes
output "k8s_cluster_id" {
  value = ionoscloud_k8s_cluster.taskboard.id
}

output "registry_hostname" {
  value     = ionoscloud_container_registry.taskboard.hostname
  sensitive = false
}

The following table sets the boundary explicitly for TaskBoard.

As shown above, the controllers never share a resource type, so neither reverts the other.

3.2 The imagePullSecret Handoff

The Container Registry on IONOS uses token-based docker login only; there is no RBAC and no per-team repositories. The pull secret is the one credential that crosses the infrastructure-to-application boundary. Create it once from the registry token, then reference it in manifests that ArgoCD syncs.

kubectl create secret docker-registry cr-pull-secret \
  --docker-server=my-registry.cr.de-fra.ionos.com \
  --docker-username='<token-name>' \
  --docker-password='<registry-token>' \
  --namespace=taskboard

Because this secret contains a credential, do not commit it in plaintext. Either create it imperatively as above (one time, outside Git) or use a sealed-secrets or external-secrets operator so ArgoCD can manage an encrypted form.

4. IONOS-Specific Reconciliation Hazards

GitOps assumes the cluster behaves predictably. Two MKS behaviors break that assumption if you do not plan for them: nodes are immutable and rebuilt rather than patched, and a LoadBalancer Service does not provision a true external load balancer.

4.1 Immutable Nodes and the Maintenance Window

MKS nodes are immutable. Any node pool upgrade rebuilds every node belonging to the pool rather than patching it in place, and upgrades generally happen automatically during the weekly maintenance window. That maintenance window is limited to a maximum of 4 hours. During this window IONOS updates all in-cluster components, including the control plane, CSI, CCM, Calico, and CoreDNS.

The GitOps implication is that pods get evicted and rescheduled on freshly built nodes on a schedule you do not control. Your manifests must tolerate this. Set readiness probes so ArgoCD and Services route traffic only to ready pods, and use a PodDisruptionBudget so the rebuild does not take all replicas down at once.

# Survive node rebuilds during the maintenance window
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: taskboard-api
  namespace: taskboard
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: taskboard-api

A rebuild also requires server quota headroom: a node rebuild provisions a new node before removing the old one, so if you have exhausted your contract's server quota the rebuild cannot complete. Keep quota slack equal to at least one node per pool.

4.2 The LoadBalancer Service Trap

This is the hazard that surprises most teams exposing ArgoCD or TaskBoard. On MKS, a Service of type: LoadBalancer is not a true external load balancer. IONOS Cloud reserves a static public IP and assigns it as a secondary IP to one worker node, which acts as the ingress node, and kube-proxy NATs traffic to the target pod. The source IP is lost unless you set externalTrafficPolicy: Local, and throughput is capped at that single ingress node's 2 Gbit/s public ceiling.

apiVersion: v1
kind: Service
metadata:
  name: taskboard-api-lb
  namespace: taskboard
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local   # preserve client source IP
  selector:
    app: taskboard-api
  ports:
    - port: 443
      targetPort: 8080

To exceed the single-node ceiling, scale horizontally across multiple LoadBalancer IPs and ingress nodes using DNS load balancing, or front the service with a separately provisioned Managed ALB (provisioned in the infrastructure repo via Terraform, never auto-created from a manifest). For production TaskBoard traffic, the ALB path is the right answer and keeps the ingress concern in the Terraform-owned layer.

5. Environment Promotion and Ephemeral Teardown

Promotion across environments is a Git operation, not a deploy script. With Kustomize, a shared base holds the manifests and each environment is an overlay that patches replica counts, resource limits, and the image tag.

5.1 Kustomize Overlays

The base defines TaskBoard once; overlays differentiate per environment. Promoting a build from staging to production is a one-line image-tag change in the prod overlay, committed through a pull request.

# overlays/prod/kustomization.yaml
resources:
  - ../../base
namespace: taskboard
images:
  - name: my-registry.cr.de-fra.ionos.com/taskboard-api
    newTag: 1a2b3c4   # promoted git SHA
patches:
  - path: replicas-patch.yaml

Each ArgoCD Application points at a different overlay path (overlays/dev, overlays/staging, overlays/prod), so the same repository drives all three environments and the diff between them is auditable in Git.

5.2 Ephemeral Branch Environments and Teardown

For feature branches you spin up a throwaway environment and tear it down on merge or branch close. Terraform-provisioned resources incur charges from the moment of apply, so automated teardown is cost governance, not a nicety. The CI job calls terraform destroy for the branch's stack.

# .github/workflows/teardown.yml (triggered on PR close)
name: teardown-branch-env
on:
  pull_request:
    types: [closed]
jobs:
  destroy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Terraform destroy branch stack
        env:
          IONOS_TOKEN: ${{ secrets.IONOS_TOKEN }}
        run: |
          terraform init -backend-config="key=env/pr-${{ github.event.number }}.tfstate"
          terraform destroy -auto-approve -var="env_name=pr-${{ github.event.number }}"

One cleanup gotcha is specific to MKS storage: IONOS volumes are represented as PersistentVolume resources in Kubernetes, and the PV reclaim policy determines what happens to the underlying volume when the claim is deleted. If the reclaim policy is Retain, deleting the namespace or running terraform destroy on the cluster leaves orphaned IONOS volumes in your VDC that keep billing. Either set the reclaim policy to Delete for ephemeral environments or explicitly clean up leftover volumes after teardown.

API Reference Quick Card

Key API endpoints for GitOps and deployment operations on MKS:

Base URL: https://api.ionos.com/cloudapi/v6 Authentication: Authorization: Bearer <token>

Code Lab

Objective: Set up ArgoCD auto-sync for TaskBoard manifests on MKS, push a manifest change and watch reconciliation heal it, then tear down a branch environment.

Prerequisites:

• IONOS Cloud account with API token
• A running MKS cluster provisioned via Terraform (from Unit 3.2)
• kubectl, argocd CLI, and Terraform installed locally
• A Git repository containing TaskBoard manifests

Step 1: Retrieve the kubeconfig from Terraform

terraform output -raw kubeconfig > kubeconfig.yaml
export KUBECONFIG=./kubeconfig.yaml
kubectl get nodes

Expected output:

NAME                STATUS   ROLES    AGE   VERSION
taskboard-pool-1    Ready    <none>   12m   v1.34.x
taskboard-pool-2    Ready    <none>   12m   v1.34.x

Step 2: Install ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deployment/argocd-server

Expected output:

deployment "argocd-server" successfully rolled out

Step 3: Create the Application CRD pointing at TaskBoard manifests

kubectl apply -n argocd -f argocd/taskboard-app.yaml
argocd app get taskboard

Expected output:

Name:               argocd/taskboard
Sync Status:        Synced to main (1a2b3c4)
Health Status:      Healthy

Step 4: Introduce drift manually

kubectl -n taskboard scale deployment/taskboard-api --replicas=5
kubectl -n taskboard get deploy taskboard-api

Expected output:

NAME            READY   UP-TO-DATE   AVAILABLE
taskboard-api   5/5     5            5

Step 5: Watch ArgoCD self-heal back to the Git-declared state

argocd app get taskboard --refresh
sleep 30
kubectl -n taskboard get deploy taskboard-api

Expected output:

NAME            READY   UP-TO-DATE   AVAILABLE
taskboard-api   2/2     2            2

Step 6: Promote a new image via Git commit

# In the manifests repo, update overlays/prod/kustomization.yaml newTag
git commit -am "promote taskboard-api to 9f8e7d6"
git push origin main
argocd app wait taskboard --sync

Expected output:

taskboard   Synced   Healthy

Step 7: Tear down a branch environment

terraform init -backend-config="key=env/pr-42.tfstate"
terraform destroy -auto-approve -var="env_name=pr-42"

Expected output:

Destroy complete! Resources: 7 destroyed.

Step 8: Verify no orphaned volumes remain

ionosctl volume list --datacenter-id $DC_ID

Expected output:

No volumes found  (or: only volumes belonging to retained environments)

Validation Checklist:

• [ ] ArgoCD reports Synced and Healthy for TaskBoard
• [ ] Manual scale change is reverted by self-heal within the sync interval
• [ ] Image-tag commit triggers a rollout without any kubectl apply
• [ ] terraform destroy removes the branch stack and leaves no billing volumes

Cleanup:

kubectl delete -n argocd -f argocd/taskboard-app.yaml
kubectl delete namespace argocd
terraform destroy -auto-approve

Common Pitfalls

Developer mistakes to avoid with GitOps and deployment operations on IONOS:

1. Terraform and ArgoCD fighting over the same resource

   • Problem: Terraform manages a Kubernetes Service while ArgoCD also syncs it from Git. Each apply and each sync reverts the other, producing an endless fight-loop and flapping resources.
   • Why it happens: No clear ownership boundary between the infrastructure repo and the application repo.
   • Fix: Make Terraform own only IONOS infrastructure (cluster, node pools, registry, databases) and ArgoCD own only in-cluster manifests. Pass the cluster and registry between them through Terraform outputs, never by both managing the same object.

2. Exposing ArgoCD or TaskBoard with type: LoadBalancer and expecting a real LB

   • Problem: Traffic all lands on one worker node, the client source IP is lost, and throughput plateaus around 2 Gbit/s with no obvious cause.
   • Why it happens: An MKS LoadBalancer Service does not provision an external load balancer. IONOS assigns a static public IP as a secondary IP to a single ingress node and kube-proxy NATs to the pod.
   • Fix: Set externalTrafficPolicy: Local to preserve source IP, and for production front the service with a separately provisioned Managed ALB from the Terraform layer rather than relying on the single-node Service.

3. Orphaned IONOS volumes after branch-environment teardown

   • Problem: You terraform destroy a branch environment but keep getting billed, and stray volumes linger in the VDC.
   • Why it happens: IONOS volumes back PersistentVolumes whose reclaim policy is Retain, so deleting the claim or namespace leaves the underlying volume behind.
   • Fix: Use a StorageClass with reclaim policy Delete for ephemeral environments, or add a teardown step that lists and removes leftover volumes with ionosctl volume list and ionosctl volume delete.

Summary

You can now run a true pull-based GitOps workflow on IONOS Managed Kubernetes. ArgoCD reconciles TaskBoard's declared state from Git, self-heals manual drift, and promotes builds through environments with a single image-tag commit. You keep Terraform and ArgoCD in separate repositories with a clean ownership boundary, so the two controllers never overwrite each other. You also know the two MKS behaviors that break naive GitOps assumptions and how to design around them.

The operational payoff is that production is now read-only for humans, every change is a reviewable commit, and rollback is a git revert. Branch environments are disposable and tear themselves down, and you avoid the orphaned-volume billing trap by setting the right reclaim policy.

Key Points:

• GitOps uses a pull model: a controller inside the cluster reconciles Git's desired state, making commits the deployment trigger and rollback mechanism
• Separate the Terraform-owned infrastructure repo from the ArgoCD-owned application repo to prevent reconciliation fight-loops
• MKS nodes are immutable and rebuilt during the weekly maintenance window (max 4 hours); use PodDisruptionBudgets and readiness probes so deployments survive rebuilds
• An MKS LoadBalancer Service is not a true external LB: one ingress node, lost source IP unless externalTrafficPolicy: Local, and a 2 Gbit/s ceiling; use a Managed ALB for production
• Automate terraform destroy for branch environments and set PV reclaim policy to Delete to avoid orphaned, still-billing volumes

Important Terminology:

• GitOps: An operating model where Git is the single source of truth and an in-cluster controller continuously reconciles live state to match the committed declarative state
• Drift: Divergence between the live cluster state and the desired state declared in Git, caused by manual changes or external processes
• Self-heal: An ArgoCD sync mode that automatically reverts live changes back to the Git-declared state
• Reclaim policy: The Kubernetes PersistentVolume setting (Retain or Delete) that determines whether the backing IONOS volume is removed when its claim is deleted
• Ingress node: The single MKS worker node that receives a LoadBalancer Service's reserved public IP as a secondary IP and NATs traffic to target pods

Next Steps

Continue Learning: Unit 5.4: Knowledge Check - Production Operations

Related Topics:

• Unit 3.3: CI/CD Pipelines for IONOS
• Unit 5.2: Security Automation