ADQM Spark 3 Connector overview

According to shard_id. Data from ADQM tables is read and distributed across Spark 3 partitions depending on a shard number. The number of Spark 3 partitions corresponds to the number of active ADQM shards. This partitioning mode does not require additional parameters and is used by default.

According to the specified column and number of partitions. Data is divided into Spark 3 partitions according to value ranges of the specified column by the specified number of partitions. The number of Spark 3 partitions corresponds to the number of ranges. It is necessary to set the spark.adqm.partition.column and spark.adqm.partition.count parameters (see ADQM Spark 3 Connector options). This mode has limitations: only an integer or date/time table field can be used as a column.

According to the specified column. Data is divided into Spark 3 partitions depending on unique values of the specified column. The number of Spark 3 partitions corresponds to the number of unique values in the specified column. You should set the spark.adqm.partition.column option (see ADQM Spark 3 Connector options). This mode is recommended for small and limited sets of values.

append. Additional data is written to the target table. If the spark.adqm.cluster.name option is specified, the connector attempts to write data to each shard of the cluster, otherwise — only to shards specified in spark.adqm.url. If any of the shards does not have a table to be appended to, the loading process fails. In this mode, a target table must be pre-created on all shards.

errorIfExists. If the target table exists, an error occurs. This mode assumes that the target table does not exist, it is created during the loading process. For this mode, you should use the spark.adqm.create.table.engine option to specify an engine of the target table to be created. If the spark.adqm.cluster.name option is specified, then the target table with the specified engine is created on each shard of the cluster, otherwise — the target table is created on shards specified in the spark.adqm.url option.

overwrite. Table data is overwritten. Depending on spark.adqm.table.truncate, the target table is either completely deleted, or all data is cleaned up before the writing process. In the case of a complete deletion of a table, the writing process is similar to the errorIfExists mode, in the case of a cleanup, it is similar to the append mode.

MergeTree engine family. These engines are designed to quickly record a huge amount of data in parts according to the specified rules. The main MergeTree engine is supported, as well as its special versions:

Distributed engine. It does not store data, but allows you to process queries distributed across multiple servers, automatically performing parallel reading (read more about creating and using Distributed tables in the Sharding article of the ADQM documentation).