Mirror Maker 2 in ADS Control

This article describes setting up topic replication to a remote cluster via the Mirror Maker 2 mechanism using the ADS Control.

The main concepts used in this article:

  • Source cluster — the cluster from which topics are replicated.

  • Target cluster — the cluster to which topics are replicated.

  • Source topic — a topic created on the source cluster and intended to be replicated to the target cluster.

  • Replica topic — a topic automatically created on the target cluster and having the same content and metadata as the source topic.

Note
  • Replication is shown based on the Active/Standby architecture when the target cluster is in passive mode (has no consumers and producers connected to it) and receives only replicated data.

  • Before configuring Mirror Maker 2, the Kafka Connect service must be installed and configured on the source and destination clusters. Kafka Connect is available for installation in ADS starting from version 1.7.1.

Create connectors for Mirror Maker 2

To perform a simple replication of a topic to a remote cluster, use the following connectors created on the basis of Kafka Connect:

  • MirrorSourceConnector

  • MirrorCheckpointConnector

To create connectors using the ADS Control, you need to:

  1. In the ADS Control user interface, select the required ADS cluster.

    Cluster selection in the ADS Control interface
    Cluster selection in the ADS Control interface
    Cluster selection in the ADS Control interface
    Cluster selection in the ADS Control interface
  2. On the page that opens, select a ADS cluster from the list of Kafka Connects.

    Opening the page with Kafka Connect connectors
    Opening the page with Kafka Connect connectors
    Opening the page with Kafka Connect connectors
    Opening the page with Kafka Connect connectors
  3. On the page that opens, click Create Connector.

    Creating a Kafka Connect Connector
    Creating a Kafka Connect Connector
    Creating a Kafka Connect Connector
    Creating a Kafka Connect Connector
  4. Select the desired connector to create.

    Selecting the Kafka Connect connector to create
    Selecting the Kafka Connect connector to create
    Selecting the Kafka Connect connector to create
    Selecting the Kafka Connect connector to create
  5. Fill in the connector configuration parameters. If necessary, refer to the Kafka Connect service configuration information in ADS configuration parameters. You can use filling in the configuration in the form of a JSON file. To do this, enable the corresponding switch.

    Connector configuration
    Connector configuration
    Connector configuration
    Connector configuration
    Сonnector JSON configuration file
    Сonnector JSON configuration file
    Сonnector JSON configuration file
    Сonnector JSON configuration file
    Example JSON file content for simple connector configuration for Active/Standby architecture
    {
        "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
        "source.cluster.alias": "source",
        "target.cluster.alias": "dest",
        "name": "<connector_name>",
        "target.cluster.bootstrap.servers": "<dest_cluster_bootstrap>",
        "tasks.max": "4",
        "topics": "<test_topic_name>",
        "source.cluster.bootstrap.servers": "<source_cluster_bootstrap>"
    }
    Attribute Description

    connector.class

    The name of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector

    source.cluster.alias

    The name of the source cluster to be used in the replication

    target.cluster.alias

    The name of the target cluster to be used in the replication

    name

    The name of the connector to be used in the replication

    target.cluster.bootstrap.servers

    Bootstrap-servers of the target cluster, used with port 9092

    tasks.max

    The maximum number of tasks that can be used for a connector

    topics

    The comma-separated list of source topics to use

    source.cluster.bootstrap.servers

    Bootstrap-servers of the source cluster, used on port 9092

  6. After filling, click Save and get a message about the successful creation of the connector.

    Message about the successful creation of the connector
    Message about the successful creation of the connector
    Message about the successful creation of the connector
    Message about the successful creation of the connector
  7. Check that the Connectors for ads page displays the created connectors in the working status. The status is determined depending on the indicator in front of the connector name:

    • green — the connector/task is running;

    • yellow — the connector/task has been administratively paused;

    • red — the connector/task has failed (usually by raising an exception, which is reported in the status output);

    • unassigned — the connector/task has not yet been assigned to a worker.

      Created connectors
      Created connectors
      Created connectors
      Created connectors

Topic replication check

After successful completion of the connector configuration, a service topic mm2-offset-syncs.<target_cluster_alias>.internal is created on the side of the source cluster, where <target_cluster_alias> corresponds to the target cluster designation entered during the connector configuration.

The mm2-offset-syncs topic is populated by the MirrorSourceConnector and then used by the MirrorCheckpointConnector to convert consumer group offsets.

In order to check for the presence of the service topic mm2-offset-syncs, enter the command on any broker of the source cluster to display the list of topics:

$ /usr/lib/kafka/bin/kafka-topics.sh --list --bootstrap-server hostname:9092

The list of topics that appears should contain the desired topic:

__consumer_offsets
mm-connect-configs
mm-connect-offsets
mm-connect-status
mm2-offset-syncs.dest.internal

The creation of a replicated source topic and the writing of messages to it occurs on the side of the source cluster.

To create an source topic, enter the command:

$ /usr/lib/kafka/bin/kafka-topics.sh --create --topic <test_topic_name> --bootstrap-server hostname:9092

where <test_topic_name> — the name of the source topic, as entered when configuring the connector.

To write messages to the original topic, enter the command:

$ /usr/lib/kafka/bin/kafka-console-producer.sh --topic <test_topic_name> --bootstrap-server hostname:9092

After writing messages to the source topic on the Connectors for ads page in the ADS Control user interface, the name of the replica topic appears in the line of the MirrorSourceConnector connector in the <source_cluster_alias>.<test_topic_name> format, where:

  • <source_cluster_alias> — the corresponding source cluster alias entered during connector configuration.

  • <test_topic_name> — the name of the source topic entered during connector configuration.

List of connectors indicating the replica topic
List of connectors indicating the replica topic
List of connectors indicating the replica topic
List of connectors indicating the replica topic

The replica topic also appears in the list of topics on the target cluster. In order to read messages from the replica topic on the target cluster, enter the command:

$ /usr/lib/kafka/bin/kafka-console-consumer.sh --topic <source_cluster_alias>.<test_topic_name> --from-beginning --bootstrap-server hostname:9092

Read messages in the target cluster completely repeat those written in the source cluster.

Found a mistake? Seleсt text and press Ctrl+Enter to report it