Running Kafka & Clickhouse with Analytics and Events Source Configuration

This document provides instructions on how to run Kafka and Clickhouse using Docker Compose, and how to configure the analytics and events source.

Architecture

 +------------------------+
 |       Hyperswitch      |
 +------------------------+
            |
            |
            v
 +------------------------+
 |         Kafka          |
 |  (Event Stream Broker) |
 +------------------------+
            |
            |
            v
 +------------------------+
 |  ClickHouse            |
 |  +------------------+  |
 |  | Kafka Engine     |  |
 |  |    Table         |  |
 |  +------------------+  |
 |            |           |
 |            v           |
 |  +------------------+  |
 |  | Materialized     |  |
 |  |    View (MV)     |  |
 |  +------------------+  |
 |            |           |
 |            v           |
 |  +------------------+  |
 |  | Storage Table    |  |
 |  +------------------+  |
 +------------------------+

Starting the Containers

Docker Compose can be used to start all the components.

Run the following command:

docker compose --profile olap up -d

This will spawn up the following services

kafka
clickhouse
opensearch

Setting up Kafka

Kafka-UI is a visual tool for inspecting Kafka and it can be accessed at localhost:8090 to view topics, partitions, consumers & generated events.

Setting up Clickhouse

Once Clickhouse is up and running, you can interact with it via web.

You can either visit the URL (http://localhost:8123/play) where the Clickhouse server is running to get a playground, or you can bash into the Clickhouse container and execute commands manually.

Run the following commands:

# On your local terminal
docker compose exec clickhouse-server bash

# Inside the clickhouse-server container shell
clickhouse-client --user default

# Inside the clickhouse-client shell
SHOW TABLES;

Configuring Analytics and Events Source

To use Clickhouse and Kafka, you need to enable the analytics.source and update the events.source in the configuration file.

You can do this in either the config/development.toml or config/docker_compose.toml file.

Here's an example of how to do this:

[analytics]
source = "clickhouse"

[events]
source = "kafka"

After making this change, save the file and restart your application for the changes to take effect.

Enabling Data Features in Dashboard

To check the data features in the dashboard, you need to enable them in the config/dashboard.toml configuration file.

Here's an example of how to do this:

[default.features]
audit_trail=true
system_metrics=true