Commit Graph

18 Commits

Author SHA1 Message Date
Vincent Bernat
fb3f5f976b common: use slices from standard library instead of x/exp/slices 2025-06-15 13:58:30 +02:00
Vincent Bernat
297e04b95c common: clickHouse → clickhouse
Let's say that we use "ClickHouse" and "clickhouse".
2024-06-09 14:59:09 +02:00
Vincent Bernat
cec8661387 chore: capitalize comments 2024-01-22 20:34:08 +01:00
Vincent Bernat
6dc0b512c6 console/filter: add filtering support for custom columns
Some of the code is based on #870.
2023-09-16 17:19:12 +02:00
Marvin Gaube
5efa368e79 feat: add option for materialized types & improve filter performance for materialized Prefixes 2023-09-08 20:54:27 +02:00
Marvin Gaube
e6effd1335 feat: add custom dictionaries for additional, customized flow hydration 2023-08-25 22:10:30 +02:00
Vincent Bernat
23b453f573 common/schema: rename generate to materialized
See #588
2023-03-30 22:47:12 +02:00
Marvin Gaube
00026beabd common/schema: make alias/generate configureable 2023-03-27 15:16:45 +02:00
Vincent Bernat
65e3e1783a common/schema: check for dependencies between columns 2023-01-30 06:48:16 +01:00
Vincent Bernat
d3b1ae7ad2 common/schema: allow to move some data in or out the main table 2023-01-26 00:33:57 +01:00
Vincent Bernat
9c51b22845 common/schema: group some columns to skip them quickly when not enabled 2023-01-21 11:19:13 +01:00
Vincent Bernat
72d51d0512 common/schema: make enabled/disabled columns configurable 2023-01-19 18:53:21 +01:00
Vincent Bernat
c6a9319b57 common/schema: turns into a component
This is a first step to make it accept configuration. Most of the
changes are quite trivial, but I also ran into some difficulties with
query columns and filters. They need the schema for parsing, but parsing
happens before dependencies are instantiated (and even if it was not the
case, parsing is stateless). Therefore, I have added a `Validate()`
method that must be called after instantiation. Various bits `panic()`
if not validated to ensure we catch all cases.

The alternative to make the component manages a global state would have
been simpler but it would break once we add the ability to add or
disable columns.
2023-01-18 12:22:10 +01:00
Vincent Bernat
727807b937 common/schema: use a symbol to identify columns 2023-01-17 20:53:00 +01:00
Vincent Bernat
e88d2a2974 common/schema: namespace column members by subsystem
Otherwise, it will be a mess once we introduce IPFIX/sFlow specific fields
2023-01-17 20:53:00 +01:00
Vincent Bernat
a30024cfa1 console: use common/schema for dimensions
This is a bit less type-safe. We could keep type safety by redefining
all the consts in `query_consts.go` in `common/schema`, but this is
pointless as the goal is to have arbitrary dimensions at some point.
2023-01-03 18:40:19 +01:00
Vincent Bernat
aae92e7fe0 common/schema: use an ordered map for columns
For ClickHouse, we can iterate, but for other aspects, we need a direct
access.
2023-01-03 17:45:50 +01:00
Vincent Bernat
7d1ba478a1 orchestrator/clickhouse: rework migrations to use an abstract schema
We introduce an leaky abstraction for flows schema and use it for
migrations as a first step.

For views and dictionaries, we stop relying on a hash to know if they
need to be recreated, but we compare the select statements with our
target statement. This is a bit fragile, but strictly better than the
hash.

For data tables, we add the missing columns.

We give up on the abstraction of a migration step and just rely on
helper functions to get the same result. The migration code is now
shorter and we don't need to update it when adding new columns.

This is a preparatory work for #211 to allow a user to specify
additional fields to collect.
2023-01-02 23:42:05 +01:00