Unit 5.1: Observability and Debugging

Introduction

You shipped TaskBoard to Managed Kubernetes in Module 3 and wired it to PostgreSQL, Redis, Object Storage, and Kafka in Module 4. Now it is running in production, and at some point it will misbehave: API latency spikes, a pod crash-loops, a database connection silently times out, or traffic stops reaching a tier and nobody knows why. Without observability wired in, you are debugging blind.

This unit shows you how to instrument and debug TaskBoard programmatically on IONOS Cloud. You will push metrics to the Monitoring Service, forward logs to the Logging Service, audit infrastructure changes through Activity Logs, and capture network traffic with Flow Logs. Each tool covers a different layer of the stack, and the unit closes with a repeatable Kubernetes debugging workflow that ties them together. Critically, you will also learn the one IONOS constraint that catches most teams: Kubernetes control-plane events do not flow through the Logging Service automatically, so you must forward cluster logs yourself.

1. Metrics with the Monitoring Service

The Monitoring Service ingests metrics over a pipeline that you create through the REST API. Each pipeline exposes an HTTP push endpoint that accepts Prometheus-format metrics (Counter, Gauge, Histogram, Summary), and visualization happens in a managed Grafana instance provisioned per contract and per region. The alternative ingest format is JSON, which must be compressed using Snappy.

You create a pipeline by POSTing to /pipelines on the regional endpoint. The endpoint follows the template https://monitoring.<region-slug>.ionos.com, for example https://monitoring.de-txl.ionos.com/pipelines for Berlin or https://monitoring.de-fra.ionos.com/pipelines for Frankfurt. The supported push agents are Prometheus, Grafana Agent, OpenTelemetry, and Fluent Bit.

1.1 Creating a Metric Pipeline

Create the pipeline with a Bearer token. The response returns the per-pipeline ingest key in metadata.key, but only on creation. For security reasons the key is never returned in later responses, so capture it immediately and store it in your secret manager.

curl -s -X POST "https://monitoring.de-txl.ionos.com/pipelines" \
  -H "Authorization: Bearer $IONOS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "properties": {
          "name": "taskboard-prod"
        }
      }'

Expected output (truncated):

{
  "id": "a1b2c3d4-...",
  "metadata": {
    "key": " exporter-api-key-shown-once ",
    "grafanaEndpoint": "https://a1b2c3d4-...grafana.de-txl.ionos.com"
  },
  "properties": { "name": "taskboard-prod", "status": "PROVISIONING" }
}

The ingest host pattern is <id>-metrics.<id>.monitoring.<region>.ionos.com, and the push path is /api/v1/push over outbound port 443. The full push URL pattern is <httpEndpoint>/api/v1/push.

1.2 Pushing Metrics from TaskBoard

Configure your agent to push to the pipeline endpoint. The Grafana Agent, Prometheus, and OpenTelemetry all authenticate by setting the header APIKEY: <key>. The Fluent Bit example instead uses Authorization: Bearer <apiKey>. The default push interval is 1 minute and is configurable.

# grafana-agent.yaml - remote_write block for TaskBoard API pods
metrics:
  configs:
    - name: taskboard
      remote_write:
        - url: https://a1b2c3d4-metrics.a1b2c3d4.monitoring.de-txl.ionos.com/api/v1/push
          headers:
            APIKEY: ${MONITORING_PIPELINE_KEY}
      scrape_configs:
        - job_name: taskboard-api
          static_configs:
            - targets: ["taskboard-api:8080"]

Once data lands, build dashboards and alerts in the managed Grafana instance at the grafanaEndpoint from the create response. You define alert conditions, thresholds, and notification preferences directly in Grafana, for example an alert when TaskBoard API p95 request latency exceeds a threshold over a 5-minute window. Access to the Monitoring Service is gated by the IAM privilege named Access and manage Monitoring.

2. Centralized Logs with the Logging Service

The Logging Service collects logs over its own pipeline, created with POST /pipelines on the regional endpoint https://logging.<region-slug>.ionos.com. A new pipeline returns status PROVISIONING and becomes usable once it is Ready. The supported log sources are Kubernetes, Docker, Linux Systemd, HTTP (JSON REST API), and Generic. Default retention is 30 days, and the allowed retention values are 7, 14, 30, or unlimited. Each pipeline allows up to 5 log streams.

curl -s -X POST "https://logging.de-txl.ionos.com/pipelines" \
  -H "Authorization: Bearer $IONOS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "properties": {
          "name": "taskboard-logs",
          "logs": [
            {"source": "kubernetes", "tag": "taskboard", "protocol": "tcp", "retentionInDays": 30}
          ]
        }
      }'

Logs are queryable in the Logging Service Grafana over a 30-day query window. Access requires the IAM privilege Access and manage Logging Service.

2.1 Forwarding Logs with Fluent Bit

Fluent Bit is the supported log agent. Each log source requires the pipeline endpoint and a key, both obtained from the REST API response. For Kubernetes, install the Fluent Bit package for your distribution, then point its forward output at the pipeline's tcpAddress with the shared key.

# fluent-bit.conf - forward TaskBoard pod logs to the Logging Service
[OUTPUT]
    Name      forward
    Match     *
    Host      <tcpAddress-host>
    Port      9000
    Tag       taskboard
    tls       on
    SharedKey ${LOGGING_PIPELINE_KEY}

Note that the HTTP REST source only accepts JSON-formatted log records, so structure your application logs as JSON when you use the HTTP path instead of the forward protocol.

2.2 The Kubernetes Control-Plane Constraint

This is the IONOS-specific trap. Kubernetes control-plane events do NOT flow through the IONOS Logging Service automatically. Deploying a workload to Managed Kubernetes does not give you cluster log forwarding for free. You must configure cluster-level log forwarding separately, which in practice means running Fluent Bit as a DaemonSet in your cluster to ship node and pod logs to your pipeline.

# Excerpt: Fluent Bit DaemonSet output for an MKS cluster
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  template:
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:latest
          env:
            - name: LOGGING_PIPELINE_KEY
              valueFrom:
                secretKeyRef:
                  name: logging-pipeline
                  key: shared-key

For integrated forwarding without each product running its own ingestion stack, the Logging Service also offers Central Logging, a contract-level, per-region capability activated with PUT /central (body {"properties":{"enabled":true}}). Only contract administrators, owners, and users with the Access and manage Logging Service privilege can toggle it.

3. Auditing Changes with Activity Logs

When an incident traces back to a configuration change, Activity Logs answer who changed what and when. The API base URL is https://api.ionos.com/activitylog/v1, and the service is read-only by design: every call is a GET. It records user logins, resource provisioning, configuration changes, data access, resource retrievals, resource modifications, and resource deletions. Each entry also tracks the source of an action, the resources affected, and the timeline of events. Responses are JSON, and authentication accepts Basic Authentication or a Bearer Token.

3.1 Querying the Activity Log

Filter by a date start and end range to scope an investigation window. Pagination is controlled with a limit that caps the number of response elements per page and an offset to walk subsequent pages.

# Find all changes during the incident window
curl -s "https://api.ionos.com/activitylog/v1/contracts/${CONTRACT_NUMBER}?startDate=2026-06-04&endDate=2026-06-05&limit=50" \
  -H "Authorization: Bearer $IONOS_TOKEN"

(replace ${CONTRACT_NUMBER} with your contract number; the base /activitylog/v1 path with no /contracts/{contractNumber} segment only returns API version info, not log entries.)

import requests

def find_changes(token, contract_number, start, end):
    url = f"https://api.ionos.com/activitylog/v1/contracts/{contract_number}"
    headers = {"Authorization": f"Bearer {token}"}
    offset, limit = 0, 100
    while True:
        params = {"startDate": start, "endDate": end, "limit": limit, "offset": offset}
        items = requests.get(url, headers=headers, params=params, timeout=30).json()["hits"]["hits"]
        if not items:
            break
        for entry in items:
            yield entry["_source"]
        offset += limit

The retention period for Activity Logs is 35 days. For anything you need beyond that, download the data and store it elsewhere. IONOS Cloud Object Storage is the explicitly recommended target for long-term persistence, so a scheduled export job into a bucket gives you an audit trail that outlives the built-in window.

4. Network Debugging with Flow Logs

When connectivity breaks between TaskBoard tiers and the application logs are silent, the problem is usually at the network layer: a firewall rule, a routing gap, or a misconfigured NIC. Flow Logs capture the traffic so you can see exactly what is being accepted and rejected. Supported resources are the VM NIC, Managed Network Load Balancer, Managed Application Load Balancer, and Managed NAT Gateway. Capture actions are Rejected, Accepted, or Any, and capture directions are Ingress, Egress, or Bidirectional. Both IPv4 and IPv6 are supported.

Flow Logs publish to an IONOS Cloud Object Storage bucket as .log.gz (gzip-compressed text), rotated every 10 minutes. Two constraints matter for automation: the configuration is immutable after creation, and there is one flow log per resource. To change a setting you delete and recreate. Creating flow logs requires the Create Flow logs DCD group privilege.

4.1 Reading Flow Log Records

Each record is a fixed, immutable format (version 2). The most useful fields for debugging are srcaddr, dstaddr, srcport, dstport, protocol, and action, where action is ACCEPT (allowed by firewall) or REJECT (blocked by firewall). A burst of REJECT entries on the port your app uses points straight at an NSG rule.

import boto3, gzip

# Flow logs land in an Object Storage bucket as .log.gz objects
s3 = boto3.client("s3",
    endpoint_url="https://s3-eu-central-1.ionoscloud.com",
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY)

obj = s3.get_object(Bucket="taskboard-flowlogs", Key="2026/06/05/flow-0001.log.gz")
for line in gzip.decompress(obj["Body"].read()).decode().splitlines():
    f = line.split()
    # version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
    if f and f[-2] == "REJECT":
        print("BLOCKED:", f[3], "->", f[4], "port", f[6])

The log-status field is OK for a normal interval or SKIPDATA when records were skipped during the interval. Seeing SKIPDATA means you are not getting the full picture for that window.

5. The Kubernetes Debugging Workflow

Most TaskBoard production incidents surface in Kubernetes first. Work the layers in order rather than jumping straight to platform tools, because the cheapest signal is closest to the pod.

5.1 Escalation Path

Start with kubectl and escalate to platform observability only when the in-cluster signal runs out:

# 1. What is the pod actually doing?
kubectl get pods -n taskboard
kubectl logs -n taskboard deploy/taskboard-api --tail=100

# 2. Why won't it start or stay healthy?
kubectl describe pod -n taskboard <pod-name>
kubectl get events -n taskboard --sort-by=.lastTimestamp

# 3. Is it a resource or latency problem? -> Monitoring Service metrics in Grafana
# 4. Need historical/aggregated logs across pods? -> Logging Service search

Remember the constraint from Section 2: kubectl logs shows you live container output, but it does not persist and control-plane events do not reach the Logging Service on their own. The Logging Service search is only useful if your Fluent Bit DaemonSet is already forwarding. Wire observability in before the incident, not during it.

5.2 Health Checks

Liveness and readiness probes are your first line of automated recovery. A readiness probe that fails removes the pod from Service endpoints so traffic stops flowing to a broken instance, while a liveness probe restarts a hung container.

# TaskBoard API deployment probes
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 10
  periodSeconds: 15
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  initialDelaySeconds: 5
  periodSeconds: 10

Because Managed Application Load Balancer is provisioned separately from your K8s manifests, configure its health check to match the same /readyz path so the ALB and Kubernetes agree on which backends are healthy.

API Reference Quick Card

Key endpoints for this unit's topic:

Base URL (CloudAPI): https://api.ionos.com/cloudapi/v6 Authentication: Authorization: Bearer <token> (metric push uses APIKEY: <key>)

Code Lab

Objective: Stand up metrics and centralized logging for TaskBoard, then debug a simulated connectivity outage using Activity Logs and Flow Logs.

Prerequisites:

• IONOS Cloud account with API token (IONOS_TOKEN exported)
• TaskBoard running on Managed Kubernetes (Unit 3.2)
• kubectl configured against your MKS cluster
• An Object Storage bucket and Access Key + Secret Key for Flow Logs

Step 1: Create the monitoring pipeline

curl -s -X POST "https://monitoring.de-txl.ionos.com/pipelines" \
  -H "Authorization: Bearer $IONOS_TOKEN" -H "Content-Type: application/json" \
  -d '{"properties":{"name":"taskboard-prod"}}' | tee pipeline.json

Expected output:

{"id":"...","metadata":{"key":"...","grafanaEndpoint":"https://...grafana.de-txl.ionos.com"},...}

Step 2: Capture the ingest key

export MON_KEY=$(python3 -c "import json;print(json.load(open('pipeline.json'))['metadata']['key'])")
echo "Stored key length: ${#MON_KEY}"

Expected output:

Stored key length: 44

Step 3: Create the logging pipeline

curl -s -X POST "https://logging.de-txl.ionos.com/pipelines" \
  -H "Authorization: Bearer $IONOS_TOKEN" -H "Content-Type: application/json" \
  -d '{"properties":{"name":"taskboard-logs","logs":[{"source":"kubernetes","tag":"taskboard","protocol":"tcp","retentionInDays":30}]}}'

Expected output:

{"id":"...","properties":{"name":"taskboard-logs","status":"PROVISIONING"}}

Step 4: Deploy the Fluent Bit DaemonSet to forward cluster logs (control-plane logs will NOT arrive otherwise)

kubectl create namespace logging
kubectl create secret generic logging-pipeline -n logging --from-literal=shared-key="$LOGGING_PIPELINE_KEY"
kubectl apply -f fluent-bit-daemonset.yaml

Expected output:

daemonset.apps/fluent-bit created

Step 5: Simulate an outage by adding an NSG rule that blocks the app tier port, then watch traffic fail

kubectl get pods -n taskboard   # pods Running, but requests time out

Step 6: Find the change in Activity Logs

curl -s "https://api.ionos.com/activitylog/v1/contracts/${CONTRACT_NUMBER}?startDate=2026-06-05&endDate=2026-06-05&limit=20" \
  -H "Authorization: Bearer $IONOS_TOKEN"

Expected output:

{"hits":{"total":1,"hits":[{"_source":{"action":"configuration changes","resource":"firewallrule/...","user":"...","time":"..."}}]}}

Step 7: Confirm at the network layer with Flow Logs

python3 read_flowlogs.py | grep BLOCKED

Expected output:

BLOCKED: 10.0.2.5 -> 10.0.3.7 port 8080

Validation Checklist:

• [ ] Monitoring pipeline created and key captured before any second API call
• [ ] Logging pipeline reaches Ready and Fluent Bit DaemonSet is running
• [ ] Activity Logs query returns the offending configuration change
• [ ] Flow Logs show REJECT records on the blocked port

Cleanup:

curl -s -X DELETE "https://monitoring.de-txl.ionos.com/pipelines/$MON_ID" -H "Authorization: Bearer $IONOS_TOKEN"
curl -s -X DELETE "https://logging.de-txl.ionos.com/pipelines/$LOG_ID" -H "Authorization: Bearer $IONOS_TOKEN"
kubectl delete namespace logging

Common Pitfalls

Developer mistakes to avoid with observability on IONOS Cloud:

1. Expecting Kubernetes logs without forwarding

   • Problem: You open the Logging Service Grafana after deploying to MKS and see no cluster or control-plane logs.
   • Why it happens: Kubernetes control-plane events do not flow through the IONOS Logging Service automatically. The platform does not ship them for you.
   • Fix: Run Fluent Bit as a DaemonSet (or configure Central Logging) and point it at your logging pipeline's tcpAddress with the shared key before you need the logs.

2. Losing the metric pipeline key

   • Problem: Your agent gets 401/403 on push and you cannot find the key to fix it.
   • Why it happens: The ingest key is returned only once, in metadata.key on pipeline creation, and never appears in later responses for security reasons.
   • Fix: Capture the key into your secret store at creation time. If it is lost, you cannot retrieve it; provision a fresh pipeline or rotate the key.

3. Trying to edit a Flow Log in place

   • Problem: You update a Flow Log's capture direction via API and the change does not take effect.
   • Why it happens: Flow Log configuration is immutable after creation, and there is one flow log per resource.
   • Fix: Delete the existing flow log and create a new one with the desired settings. Encode this delete-then-create pattern in your Terraform/automation rather than expecting an in-place update.

Summary

You can now instrument TaskBoard end to end on IONOS Cloud. Metrics flow to a Monitoring Service pipeline and render in managed Grafana with alerts; logs flow to a Logging Service pipeline through Fluent Bit; infrastructure changes are auditable through the read-only Activity Logs API; and network-level failures are visible in Flow Logs written to Object Storage. With probes and matching ALB health checks in place, broken instances are removed from rotation automatically. Most importantly, you know the IONOS constraint that bites teams in production: cluster logs and control-plane events must be forwarded by you, not the platform.

Key Points:

• Monitoring pipelines accept Prometheus-format metrics at /api/v1/push; the ingest key is shown only once on creation
• Logging pipelines support Kubernetes, Docker, Systemd, HTTP, and Generic sources with 7/14/30/unlimited retention; Fluent Bit is the supported agent
• Kubernetes control-plane events do NOT flow through the Logging Service automatically; forward them yourself with a Fluent Bit DaemonSet or Central Logging
• Activity Logs is read-only (GET only), filterable by date, with 35-day retention; export to Object Storage for longer audit trails
• Flow Logs capture ACCEPT/REJECT per NIC, NLB, ALB, and NAT Gateway into .log.gz objects; configuration is immutable and one-per-resource

Important Terminology:

• Metric pipeline: A Monitoring Service instance created via POST /pipelines that exposes an HTTP push endpoint for Prometheus-format metrics.
• Ingest key: The per-pipeline credential returned once in metadata.key; agents send it as the APIKEY header (or Bearer for Fluent Bit).
• Central Logging: A contract-level, per-region toggle (PUT /central) that lets integrated products forward logs to the Logging Service without each running its own ingestion.
• Activity Logs: The read-only audit API at /activitylog/v1 recording who changed which resource and when, retained for 35 days.
• Flow Logs: Immutable, per-resource network captures published as gzip text to Object Storage, used to see accepted and rejected traffic.

Next Steps

Continue Learning: Unit 5.2: Security Automation

Related Topics:

• Unit 3.2: Kubernetes Deployment and Operations
• Unit 4.2: Object Storage Integration