HBase data model

First, consider the data model terminology used in HBase:

At the logical level the data in HBase is organized into tables, indexed by the primary row key. You can store an unlimited set of columns for each row key. Columns are organized into column groups, called column families. As a rule, each column family combines the columns with the same pattern of usage and storage.

Columns are used for storing pieces of data, called values. Each column is defined by column family name and its own unique name, called column qualifier. A column can store several versions of one value. Different value versions have different timestamps.

Suppose that there is a table with two column families and two rows identified by their row keys. For each combination of the row key and the column family, you can define different column qualifiers. You also can add new column qualifiers on the fly.

The approximate logical view of such table is shown below (where ts is an abbreviation of "timestamp").

You can see sparse data in the table, shown above, as filling of columns for each row key is optional. You don’t even have to fill each column family for your rows. If the column is missing, it does not cause overhead for storing empty values, because at the physical level the data are stored differently, than at the logical one.

Physically all the data values in HBase are stored in the row key-sorted order. Blocks of nearby row keys form Regions described in the HBase architecture article. The data, corresponding to different column families, are stored separately — in different HFiles. Each column family uses at least one HFile. It allows, if necessary, to read data only from the desired column family. Columns that correspond to the same column family and the same row key are also stored as a list, sorted alphabetically.

Any column can be missing or present for each row key — empty values are never stored in the database.

When you delete a certain column value, it is not physically deleted immediately — it is marked with a special flag tombstone. The physical deletion of the data will occur later, while performing the Major compaction operation.

In addition to manual deletion of old values versions, you can configure automatic one. At the column family level, you can define such parameters as time to live (TTL) and the maximum number of stored versions. If the difference between the timestamp for a particular version and the current time is greater than TTL, this record is marked for deletion. If the number of versions for a certain column exceeds the maximum number of stored versions, the record is also marked for deletion.

The approximate physical view of our example table is shown below. You can see that all the data is stored in one Region, and different column families belong to different HFiles inside of this Region.

This section describes the HBase data model on the particular example.

Suppose there is a need to store text articles. Each article can have basic attributes, which are changed very rarely (author name, article header, creation datetime, etc.), and tags, which describe the article content and help to perform text search among all articles (arch, concepts, tutorials, ref, etc.). The tags are modified more often.

In the given example, we could create the table articles with two column families: basic — for storing rarely modified data, and tags — for storing document tags. The following JSON illustrates the possible data model in this case.

Please, do not consider this code as illustration of how data are physically stored: this is only a mock-up used to understand the basic data objects of HBase. JSON is more suitable for describing mappings, used in HBase, than a regular two-dimensional table.

So, to get a specific data value stored in HBase, you should know the following path: Table → Row Key → Column Family → Column Qualifier → Timestamp. Without defining timestamps, you get the most recent data. For example, if you try to get the value of the row with the key article1 and the column basic:header, you get the most recent value Test article. Version 3. But if you additionally define the timestamp 1637054560118 — you get the value Test article.

The main commands, based on the data model showed in this example, are described in the Quick start with HBase shell article.