Storage concepts in Kafka

Main components of Kafka

The components that form the basis of the Kafka storage concepts:

  • Message — a record of an event that has occurred, including the state of the object, the value of a physical quantity, or any other parameter that needs to be tracked, stored, or transferred to another system.

  • Producer — a client application that publishes (writes) messages to Kafka.

  • Consumer — a client application that subscribes (reads and processes) messages.

  • Topic — a category in which messages are published in Kafka. In a topic messages are written to the commit log.

  • Commit log — an ordered structure of messages that is write-only. Commit log messages are not editable or deletable. Large logs are divided into partitions. If the log consists of one partition, then it is a partition.

  • Partition — an ordered immutable sequence of records that is constantly added to the topic log.

  • Broker — a server where partitions are stored.

  • Segment — a part of the partition stored in a separate file on disk.

  • Consumer group — multiple consumers grouped together to receive messages from one or more topics. Partitions of one topic are automatically evenly distributed among consumers in the same group. Each partition is available to only one consumer from the group.

Basic settings when working with Kafka

When writing messages, the producer specifies:

  • topic to which the data will be written;

  • partition where messages with the same key will be written (if necessary for structuring the data in the topic);

  • retention period of messages in Kafka;

  • necessary compression;

  • required level of confirmation about writing a message to the partition and its replicas;

  • partition replication factor.

 
The consumer for reading messages specifies:

  • number of unique consumer groups;

  • group_id for each of the consumer groups, which determines where the group will read messages from;

  • maximum time the client has to wait for a response to a request;

  • required level of message delivery confirmation (semantics).

 
The message has the following parameters:

  • data;

  • key to determine whether messages are written to the same partition if necessary (for example, client identifier);

  • timestamp and others.

Sequence of writing and reading messages

The figure below shows the sequence of message processing within a partition when writing and reading.

Writing and reading messages in Kafka
Writing and reading messages in Kafka
Writing and reading messages in Kafka
Writing and reading messages in Kafka
  1. When a new message is written to a topic, it is actually added to one of the topic’s partitions.

  2. Every message in a partition is assigned a sequence number id, called an offset, which uniquely identifies every message.

  3. The producer obtains acknowledgment of a written message using the acks configuration parameter, which specifies under what condition the message is considered written to the partition. Options for the acks parameter:

    • 0 — the producer considers the records successfully delivered after they are sent to the broker;

    • 1 — the producer waits for confirmation from the leader’s broker that it has logged the record;

    • all — the producer is waiting for confirmation from the leader broker and replicas.

  4. This message is then made available for the consumer to read.

  5. The consumer instance connects with its own group_id to the topic. According to group_id it is assigned the partition to read and the offset in that partition, if any. If they are not specified, the reading will start from the message with the lower unread offset (earliest written).

  6. The consumer requests messages from the topic, starting at the offset assigned to it.

  7. Messages are read sequentially from the log.

  8. When reading from a partition, the reader makes a commit offset — a record about the read message in the Kafka service topic __consumer_offsets. The offset commit is performed according to the message read semantics adopted by the consumer application. Value options:

    • at most once — the offset is accepted immediately after the message is received (if no processing occurs, the message is lost);

    • at least once — the offset is only accepted after the message has been processed (if no processing occurs, the message is read again);

    • exactly once — all messages are delivered once.

  9. The message with the next offset becomes available for reading.

 

The metadata of each message is its offset or position in the log. The offset to start reading the message can be set by the consumer:

  • linear advancement of the offset when reading the record (by default);

  • read the record in any order if necessary (revert to an older offset or jump to the most recent actual record).

Replication and distribution of partitions in the broker

Replication

To make messages fault-tolerant and highly available, each topic with its partitions can be replicated. This means to create a copy and put it on the server (write it to disk).

Replication factor — a parameter that determines how many copies of each partition will be distributed on brokers. The recommended value for the replication factor (default value) is 3.

Each partition replica is written on a separate broker. This means that if the broker on which one of the replicas is located fails, there will be two more copies of this partition left. This ensures that data is always available, even during broker maintenance.

Brokers break each partition into segments. Each segment is stored in a separate data file on a disk connected to the broker. By default, each segment contains either 1 GB of data or a week worth of data, whichever limit is reached first. When the segment limit is reached, the broker closes the file and creates a new file to record new messages. Active segment is the file that is currently being written to.

Replica synchronization

The replication format is called InSync. This means that of all the replicas, one is the leader, and the others are followers. The producer can control the synchronous/asynchronous replication with the acks(acknowledgment) option.

Writing and reading messages is carried out through the leader replica of the partition. The follower that contains all the messages held by the leader is called an in-sync replica.

Brokers

To ensure fault tolerance in a Kafka cluster, partition leaders are automatically distributed among different brokers. Each broker can be a leader for some partitions and a follower for others.

If a broker acting as a leader for a partition goes down, any broker that is up to date or synchronized for that partition can take over the leader role.

The figure below shows how partitions can be distributed with a given cluster configuration:

  • number of brokers — 3;

  • number of topics — 2;

  • number of partitions in Topic 1 — 2;

  • number of partitions in Topic 2 — 1;

  • replication factor — 3.

Distribution of partitions between Kafka brokers
Distribution of partitions between Kafka brokers
Distribution of partitions between Kafka brokers
Distribution of partitions between Kafka brokers

Consumer groups

It is possible to create a large number of consumer groups configurations for reading from Kafka topics, but they are all based on two basic messaging schemes — point-to-point and publish-subscribe.

Consumer groups are managed by a consumer group coordinator.

Consumer group coordinator — one of the brokers that manage all consumers in the consumer group. If a consumer stops sending heartbeats, the coordinator initiates consumer rebalancing.

Point to point

The figure below shows how one consumer group reads messages from one topic.

Reading messages with a point-to-point pattern
Reading messages with a point-to-point pattern
Reading messages with a point-to-point pattern
Reading messages with a point-to-point pattern

This is an ideal load distribution option.

The consumer group has a group_id that contains information about which topic to connect to and where to start reading from, and Kafka evenly distributes all partitions among the consumers of the same group.

Kafka gives access to a partition to only one consumer (from a group of consumers) and each of them reads data from one partition that has the same key (k1, k2, k3).

In this configuration, the Kafka broker only passes messages sequentially to the consumer when it requests them. This requires a minimum of additional settings.

When one consumer from the group is disconnected, its load (reading from its partition) will be redistributed evenly to other consumers.

Publish-subscribe

The figure below shows how two groups of consumers read messages from the same topic.

Reading messages with a publish-subscribe pattern
Reading messages with a publish-subscribe pattern
Reading messages with a publish-subscribe pattern
Reading messages with a publish-subscribe pattern

Every consumer group using the parameters specified in its group_id can subscribe to any topic or partition with the desired key (k1, k2, k3).

At the same time, consumer groups do not depend on each other.

Kafka acts like a normal pub-sub topic to which multiple consumers are subscribed, but Kafka has the advantage that all messages are stored and can be processed multiple times.

Most often, topics have a small number of consumer groups, one for each logical subscriber.

When one consumer from the group is disconnected, its load (reading from its partition) will be redistributed evenly to other consumers (same as in the point-to-point reading scheme).

Found a mistake? Seleсt text and press Ctrl+Enter to report it