Unit 5.6: Event Streaming (Managed Kafka)

Introduction

The load-bearing decision in event streaming is not provisioning the cluster; it is the shape of the topic. Partition count fixes both the ordering guarantee and the ceiling on consumer parallelism for the life of that topic, and replication factor and retention together decide how much you pay to keep data safe and for how long. Get the topic shape right and the cluster is almost incidental. Get it wrong and you discover the limit in production, where partitions cannot be reduced and a re-shape means a new topic and a migration.

This unit treats Event Streams for Apache Kafka as the streaming backbone for FinCorp, the German financial-services firm carrying GDPR and BSI obligations through its migration and its new AI capability. We open at the topic-design decision, explain how partitions, consumer groups, replication, and retention interact, show where the stream sits in the wider architecture, and then build a cluster and a correctly shaped topic in the Data Center Designer.

1. The Topic Is the Architecture: Partitions, Ordering, and Parallelism

A partition is the unit of both ordering and parallelism, and those two properties pull in the same direction, which is what makes partition count the decision that matters most. Kafka maintains the order of messages within a partition; it makes no ordering promise across partitions. So if FinCorp must process all events for a given account in the order they occurred, every event for that account must land on the same partition. The standard way to guarantee that is to set a partition key (the account identifier) so the producer hashes it to a fixed partition. Ordering is therefore a per-key property, not a per-topic one: you get strict order within each key and concurrency across keys, which is exactly what a high-volume transaction feed wants.

Parallelism is the other side of the same coin. Within a consumer group, a partition is consumed by at most one member at a time, so the partition count is the hard ceiling on how many consumers can work that topic in parallel. Three partitions means at most three active consumers; a fourth sits idle. More partitions enable better load balancing across consumers and improve throughput, but they also multiply open file handles, replication traffic, and rebalance time, so the count is sized to the throughput you actually expect, not inflated speculatively. The asymmetry to internalise is that you can usually add partitions later, but adding them rehashes the key-to-partition mapping and breaks the per-key ordering guarantee for in-flight keys, so for an ordered feed you size partitions up front and leave them alone.

IONOS gives a concrete sizing rule that ties partition count to the cluster's broker topology. Every Event Streams for Apache Kafka cluster runs three brokers regardless of size template. The guidance is that the number of partitions should be equal to or greater than the number of brokers, and that you should use only multiples of the broker count (3, 6, 9, 12, and so on) to avoid an uneven distribution of partitions across brokers. An uneven spread leaves one broker hotter than the others and wastes the capacity you paid for. If high throughput is a priority, the documentation suggests 2x or 3x the broker count or more. For FinCorp's account-event topic, that means starting at 3 partitions and stepping to 6 or 9 only when measured throughput justifies it, always staying on a multiple of three.

Consumer groups and offsets are how this design survives restarts and scales out. A consumer group is a set of consumers that cooperatively divide a topic's partitions among themselves; Kafka tracks, per group and per partition, the offset of the last processed message. Because the offset is committed back to the cluster, a consumer that crashes and restarts resumes from where the group left off rather than from the beginning, and adding a consumer to the group triggers a rebalance that redistributes partitions. This is why FinCorp can run one consumer group for real-time fraud scoring and a second, independent group for the analytics warehouse against the same topic: each group keeps its own offsets and reads the full stream at its own pace, and neither blocks the other.

2. Replication and Retention as Cost-and-Reliability Levers

Two topic-level settings translate directly into your storage bill and your durability posture, and both are decisions, not defaults to accept blindly.

Replication factor is the number of copies of each message kept on distinct brokers. IONOS recommends a replication factor of 3, which on a three-broker cluster means a full copy of the partition on every broker, so the topic survives the loss of brokers without data loss and the cluster's automatic failover and replication keep the pipeline flowing. The cost is multiplicative: at replication factor 3, the storage a topic consumes is three times its raw message volume. The IONOS sizing note makes this explicit, warning that final storage consumption can be several times the incoming data volume depending on the configured replication factor. A lower factor saves storage but trades away fault tolerance; for FinCorp's regulated transaction data, 3 is the right floor.

Retention decides how long messages stay before they are purged, and it is the other big storage lever. Retention time is set in milliseconds, defaults to 604800000 (7 days), and a value of -1 means no time limit. There is a companion retention segment size in bytes (the size at which a log segment is rolled, default 1073741824, which is 1 GB, with a documented floor of 14 bytes), and a retention size cap that purges the oldest messages once the topic's total size limit is reached. Sizing the cluster is therefore a retention arithmetic problem. The documentation gives a worked example: three topics, each retaining seven days at 500 KB per second, need a minimum of roughly 866 GB of total storage before replication is even counted. The cluster templates fix the storage you get:

Size	Cores per broker	RAM per broker	Storage per broker	Total cluster storage
XS	1	2 GB	195 GB	585 GB
S	2	4 GB	250 GB	750 GB
M	2	8 GB	400 GB	1200 GB
L	4	16 GB	800 GB	2400 GB
XL	8	32 GB	1500 GB	4500 GB

Read this table as a budget, not a menu of speed grades: you select the size by the throughput and retention your topics demand, then check that total cluster storage comfortably exceeds raw-volume times replication factor over the retention window. FinCorp's seven-day, replication-factor-3 transaction feed pushes it past the S template even at modest ingest rates, so M is the realistic starting point with headroom for a second topic.

One retention nuance worth naming because the platform supports it: Kafka log compaction retains only the latest value for a given key rather than purging by age, which suits a topic that represents current state (the latest balance per account, say) rather than an event history. It is a different retention mode from time or size based purging; reach for it only when the topic genuinely models keyed state, and do not assume it is on by default.

3. The Stream as Substitute and Spine

This is where event streaming earns its place in the platform's native-substitution model. IONOS managed databases do not expose a change-data-capture stream you can subscribe to, so you cannot tail the database transaction log to drive downstream consumers. The native pattern is to invert the dependency: instead of mining the database for changes after the fact, the application publishes a domain event to a Kafka topic at the moment it makes the change, and every downstream system (the analytics warehouse, the fraud model, a search index, an audit archive) consumes that topic. The stream becomes the source of truth for change propagation, the database becomes one more consumer-side projection, and FinCorp gets the fan-out a CDC stream would have given it without a feature the platform does not sell. The discipline this demands is that the write to the database and the publish to the topic are treated as one logical operation in the application, since the platform offers no automatic bridge between them.

The same stream is the ingestion spine into the AI tier. FinCorp's models do not call the transactional database directly; they consume the event topics, which gives the AI workloads a decoupled, replayable feed they can re-read from a committed offset whenever a model is retrained. Two endpoints close the loop and both lean on Object Storage. First, archive: bulk data export is available but is not self-service, it is a support-ticket process (contract number, support PIN, and a PGP key, after which IONOS delivers an encrypted archive to your Object Storage bucket), so the long tail of events that has aged past the hot retention window is exported to Object Storage only after that request, and short cluster retention is a workable design only if the archive cadence accounts for that manual lead time. Second, the dead-letter tail: messages a consumer cannot process are routed to a separate dead-letter topic and, for durable forensic retention, drained to Object Storage where object lock makes them tamper-evident. The brokers stay lean for live traffic; the slow, cheap, immutable storage absorbs the archive and the failures.

Security on the data plane is mutual TLS, and the model is strict. Communication with the cluster is TLS secured with both sides authenticating: the client verifies the cluster's certificate and the cluster verifies the client's certificate. Because the cluster does not use publicly signed certificates, clients validate the server against the cluster's own certificate authority, and the cluster maintains a client CA that signs each authenticated user's certificate. Each certificate is valid for 365 days, and a renewed certificate becomes available for retrieval from the user-credentials API 30 days before expiry, which is the window your rotation runbook targets. The management API, by contrast, is authenticated with a Bearer token and is reachable publicly at a per-region endpoint (kafka.<region>.ionos.com); only the Kafka data-plane protocol on the brokers is private-LAN only with no public endpoint, so the security perimeter for the data plane is the private network plus the mTLS handshake, while the management API's perimeter is Bearer-token control (scoped IAM privilege, token handling and rotation).

DCD Implementation Walkthrough

You will provision a three-broker Event Streams for Apache Kafka cluster on FinCorp's private data-tier LAN and create one correctly shaped topic for the account-event feed. The architecture goal is the ingestion spine from Section 3: a private, mTLS-secured stream sized for a seven-day, replication-factor-3 retention window, with a partition count that honours both ordering and the broker topology. The cluster has no public endpoint, so the work is entirely inside the VDC you built earlier in the course.

Build goal: Create a cluster and a well-shaped topic.

Prerequisites: A provisioned Virtual Data Center containing at least one VM in a private LAN; that VM accesses the cluster over the private LAN and counts against your contract quota. Decide the broker IP addresses in advance: they must belong to the same subnet as the LAN you select.

Steps (in the Data Center Designer):

Open the Event Streams for Apache Kafka section and click Create cluster.
Set Cluster Name to a unique, identifiable name, and select Kafka Version (version 4.0.0 is the supported version; 3.9.0 and 3.9.1 are deprecated, and existing clusters on those versions keep running).
Choose the Cluster Size (XS through XL) by your throughput and retention requirements. For FinCorp's seven-day, replication-factor-3 feed, size from the storage table in Section 2 rather than guessing; M is the realistic floor.
Select the Location (region) for the cluster. The region fixes geographical placement, latency, and regulatory residency, so pick the German data center for FinCorp's regulated data.
Choose the Datacenter and the Datacenter LAN. Select the private data-tier LAN; the cluster will be reachable only on this network.
Configure the broker addresses clients will use to connect. The IP addresses must belong to the same subnet as the LAN selected in the previous step, so that routing within the cluster works. Use the DCD screen's subnet hint to confirm the range.
Review the estimated costs shown for the input (an estimate that excludes variables such as traffic), then click Save to deploy. Monitor progress until the cluster reaches the Available state.
With the cluster Available, open it, select the Topics tab, and click Create topic.
In the Create Topic dialog set: Name; Replication Factor (use 3); Number of Partitions (a multiple of the three brokers: 3 to start, 6 or 9 if throughput demands); Retention Time (ms) (604800000 for seven days, or -1 for no time limit); and Retention Segment Size (B) (default 1073741824, with a minimum of 14 bytes). Click Create.

To use the cluster afterwards, retrieve the per-user client certificate from the credentials API and validate the broker against the cluster CA; the API returns the certificate as PEM, for example:

curl --location https://kafka.<region>.ionos.com/clusters/${clusterId}/users/${userId}/access \
  --header "Authorization: Bearer ${personalToken}" | yq -r '.metadata.certificate' > client-cert.pem

This single call illustrates the architectural boundary, not a configuration lab: the data plane is mTLS, so a client without a CA-signed certificate cannot connect at all, and certificate lifecycle (365-day validity, renewal available 30 days out) is an operational task the team owns.

Common mistakes:

Setting a partition count that is not a multiple of the broker count. Use multiples of three (3, 6, 9, 12) so partitions distribute evenly; an uneven spread overloads one broker and wastes paid capacity.
Treating partition count as adjustable later. Adding partitions rehashes keys and breaks per-key ordering for in-flight keys, so size an ordered feed's partitions up front.
Under-sizing the cluster against retention. Multiply raw ingest by the retention window and then by the replication factor; the 7-day, replication-factor-3 case can be several times the raw volume, and once the storage limit is hit the oldest messages are purged.
Choosing the broker IP addresses outside the selected LAN's subnet. They must be in the same subnet or the brokers cannot route to clients.
Forgetting the private-LAN prerequisite: a VDC with a VM on a private LAN must exist first, and there is no public endpoint, so connectivity and the mTLS certificate path are the only way in.
Letting client certificates lapse. Each is valid for 365 days with a renewal available 30 days before expiry; bake retrieval and rotation into the runbook.

Summary

The shape of the topic is the architecture: partition count fixes per-key ordering and caps consumer-group parallelism, and on IONOS it should be a multiple of the three brokers; replication factor (recommend 3) and retention (default 7 days, -1 for unlimited) are the cost-and-reliability levers that size the cluster. The stream is also the platform's native answer to two gaps, standing in for a database change-data-capture stream through application-level event publishing and serving as the replayable ingestion spine into the AI tier, with Object Storage carrying the aged archive and the dead-letter tail. The build is a private, mTLS-secured cluster on the data-tier LAN plus a single well-shaped topic, sized from the storage table rather than by guesswork.

Key Points:

A partition is the unit of both ordering (per key) and consumer parallelism (one consumer per partition per group); size partitions as a multiple of the three brokers and up front for ordered feeds.
Replication factor 3 is the recommended durability floor and multiplies storage roughly threefold; retention time (default 604800000 ms / 7 days, -1 for no limit) and segment size drive the rest of the storage bill.
Consumer groups track committed offsets per partition, so independent groups read the same topic at their own pace and survive restarts without reprocessing.
Application-level event publishing is the native substitute for the absent database change-data-capture stream, and the same topics are the replayable feed into the AI tier.
Object Storage is the archive (via bulk data export) and the durable dead-letter tail; the brokers stay lean for live traffic.
The data plane is mutual TLS against the cluster's own CA with 365-day certificates (renewal 30 days out); the management API uses a Bearer token; the cluster is private-LAN only.

Important Terminology:

Partition: The unit of ordering and parallelism within a topic. Order is guaranteed within a partition only; a consumer group assigns each partition to at most one member.
Consumer group: A set of cooperating consumers that divide a topic's partitions and share committed offsets, enabling parallel, resumable, independent consumption.
Replication factor: The number of broker-resident copies of each message; recommended 3, and a direct multiplier on storage consumption.
Retention: The policy (time, size, or log compaction) governing how long messages persist before being purged or compacted.
mTLS: Mutual TLS where client and cluster authenticate each other against the cluster's certificate authority; the only path onto the private data plane.

Learning Objectives