mirror of
https://github.com/akvorado/akvorado.git
synced 2025-12-11 22:14:02 +01:00
orchestrator/clickhouse: add support for distributed/replicated tables
Fix #605 All MergeTree tables are now replicated. For some tables, a `_local` variant is added and the non-`_local` variant is now distributed. The distributed tables are the `flows` table, the `flows_DDDD` tables (where `DDDD` is a duration), as well as the `flows_raw_errors` table. The `exporters` table is not distributed and stays local. The data is following this schema: - data is coming from `flows_HHHH_raw` table, using the Kafka engine - the `flows_HHHH_raw_consumer` reads data from `flows_HHHH_raw` (local) and sends it to `flows` (distributed) when there is no error - the `flows_raw_errors_consumer` reads data from `flows_HHHH_raw` (local) and sends it to `flows_raw_errors` (distributed) - the `flows_DDDD_consumer` reads fata from `flows_local` (local) and sends it to `flow_DDDD_local` (local) - the `exporters_consumer` reads data from `flows` (distributed) and sends it to `exporters` (local) The reason for `flows_HHHH_raw_consumer` to send data to the distributed `flows` table, and not the local one is to ensure flows are balanced (for example, if there is not enough Kafka partitions). But sending it to `flows_local` would have been possible. On the other hand, it is important for `flows_DDDD_consumer` to read from local to avoid duplication. It could have sent to distributed, but the data is now balanced correctly and we just send it to local instead for better performance. The `exporters_consumer` is allowed to read from the distributed `flows` table because it writes the result to the local `exporters` table.
This commit is contained in:
@@ -21,7 +21,9 @@ SELECT name
|
||||
FROM system.tables
|
||||
WHERE database=currentDatabase()
|
||||
AND table LIKE 'flows%'
|
||||
AND engine LIKE '%MergeTree'
|
||||
AND table NOT LIKE '%_local'
|
||||
AND table != 'flows_raw_errors'
|
||||
AND (engine LIKE '%MergeTree' OR engine = 'Distributed')
|
||||
`).
|
||||
Return(nil).
|
||||
SetArg(1, []struct {
|
||||
|
||||
Reference in New Issue
Block a user