Kafka

Managed Apache Kafka: a distributed event-streaming platform with single-broker or multi-broker high availability, a built-in catalog of source and sink connectors, topic and consumer tooling, and Prometheus metrics.

Kafka is not a traditional database. It is a distributed, append-only event log. Producers write messages to named topics, each topic is split into partitions for parallelism, and consumers read at their own pace. Messages are retained on disk so multiple consumers can replay the same stream independently. StackBlaze runs Kafka with the Strimzi operator in KRaft mode (no ZooKeeper) and adds a managed layer on top: a connector catalog, topic and consumer-group tooling, a message viewer, and metrics.

Creating a Kafka cluster

From the dashboard, click New service, then Database, then Kafka. Pick a plan, a storage size, and whether you want a single broker (Standard) or a multi-broker cluster (high availability). Connecting the cluster to a service injects KAFKA_BOOTSTRAP_SERVERS into that service, so there is nothing to hardcode.

Connecting

Kafka clients connect through bootstrap servers. A client contacts a bootstrap address, discovers the full set of brokers, then talks directly to whichever broker hosts the partitions it needs. The internal plaintext listener is on port 9092, and a TLS listener is available on port 9093.

[service-name]:9092

Producing and consuming

Any standard Kafka client works. This example uses kafkajs for Node.js, reading brokers from the injected variable:

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: process.env.KAFKA_BOOTSTRAP_SERVERS.split(','),
})

async function produce() {
  const producer = kafka.producer()
  await producer.connect()
  await producer.send({
    topic: 'events',
    messages: [{ value: 'hello kafka' }],
  })
  await producer.disconnect()
}

async function consume() {
  const consumer = kafka.consumer({ groupId: 'my-group' })
  await consumer.connect()
  await consumer.subscribe({ topic: 'events', fromBeginning: true })
  await consumer.run({
    eachMessage: async ({ message }) => console.log(message.value.toString()),
  })
}

produce().then(consume)

Standard vs high availability

You choose the topology when you create the cluster. Both modes run in KRaft mode, where each broker is a combined controller and broker (there is no separate ZooKeeper to operate).

In HA, each partition has a leader and in-sync replicas on other brokers. Lose the leader, and a replica is promoted automatically.

Standard

A single broker with a replication factor of 1 and min.insync.replicas=1. It is the simplest, lowest-cost option and a good fit for development and lower-stakes workloads. There is no replication, so a broker restart briefly interrupts the stream.

High availability

Multiple brokers, each on its own node with its own persistent volume. Topics use a replication factor greater than 1 (3 by default), so every partition has a leader plus in-sync replicas on other brokers. Writes are acknowledged once they reach the minimum in-sync replicas (min.insync.replicas=2 by default), which protects against data loss. If the broker holding a partition leader fails, Kafka elects a new leader from the in-sync replicas automatically, and producers and consumers continue without intervention.

	Standard	High availability
Brokers	Single broker	Multiple brokers, one per node
Replication factor	1	3 (configurable)
min.insync.replicas	1	2
Leader election	Not applicable	Automatic on broker loss
Tolerates broker loss	No	Yes, no data loss
Best for	Development and testing	Production and resilient workloads

Scaling from Standard to HA raises the broker count and replication factor; StackBlaze performs a rolling change. The internal topics that back consumer offsets and transactions are replicated to match.

Kafka Connect: source and sink connectors

Most data does not start or end in Kafka. Kafka Connect moves it in and out without writing custom producers or consumers. Source connectors stream changes from another system into topics, and sink connectors stream topic records out to another system. StackBlaze ships a curated catalog with the connector plugins pre-baked into the runtime image, so there is nothing to install.

Source connectors stream changes in; sink connectors stream records out. Connection details auto-fill from a paired add-on.

Connector catalog

Connector	Type	What it does
Postgres CDC (Debezium)	Source	Capture row-level changes (inserts, updates, deletes) from a PostgreSQL add-on into topics, in real time
MySQL / MariaDB CDC (Debezium)	Source	Capture changes from a MySQL or MariaDB add-on into topics via the binlog
JDBC sink	Sink	Write topic records into a SQL database (insert or upsert), with optional auto-create and schema evolution
S3 sink (RustFS)	Sink	Archive topics to S3 object storage as partitioned, compressed JSON objects
MongoDB sink	Sink	Write topic records into a MongoDB or DocumentDB collection
ClickHouse sink	Sink	Stream topics into ClickHouse for real-time analytics

Zero-config setup

Each connector is paired with a sibling add-on in your project. When you pick, for example, the Postgres CDC source and point it at one of your PostgreSQL add-ons, StackBlaze auto-fills the host, port, database, and credentials from that add-on. The S3 sink auto-fills the endpoint and access keys from a paired RustFS add-on. You only provide the connector-specific fields: the topic prefix or topic list, and which tables or collections to include. A test step validates the connection before the connector is created.

A common pattern is change data capture: a Debezium source streams every change from your Postgres add-on into topics, and a sink fans those records out to analytics (ClickHouse), to a search index, or to object storage for archival, all from the same stream.

Topics, consumer groups, and message inspection

The Kafka add-on page is a full management console, so you rarely need a separate desktop tool.

Area	What you can do
Topics	Create, delete, and describe topics; set partitions and replication factor; view per-partition offsets and message counts
Consumer groups	List groups and inspect per-partition lag, offsets, and members to see how far behind your consumers are
Messages	Peek messages from the earliest or latest offset on a chosen partition, with key, value, and timestamp, like a built-in viewer
Topology	A live flow diagram of brokers, topics, consumer groups, and connectors, with lag and connector state

Metrics and monitoring

Broker and connector metrics are exported from JMX to Prometheus automatically. You get broker health, per-topic message and partition counts, and per-partition consumer lag, plus connector throughput and error rates for both source and sink tasks. Lag and failed connector tasks surface directly on the topology view so problems are easy to spot.

Security

Traffic inside your project uses the private network. A TLS listener is available on port 9093, and the Strimzi user operator manages SCRAM credentials for authenticated clients. You can also drive topics and messages from the chat assistant, which talks to your cluster over an authenticated connection.

Backups and retention

Kafka retains messages on disk per each topic’s retention settings, so consumers can replay history within that window. The persistent volumes backing the brokers follow the standard platform backup policy. For schedules and restores, see Backups & recovery.