This change split the inlet component into a simpler inlet and a new
outlet component. The new inlet component receive flows and put them in
Kafka, unparsed. The outlet component takes them from Kafka and resume
the processing from here (flow parsing, enrichment) and puts them in
ClickHouse.
The main goal is to ensure the inlet does a minimal work to not be late
when processing packets (and restart faster). It also brings some
simplification as the number of knobs to tune everything is reduced: for
inlet, we only need to tune the queue size for UDP, the number of
workers and a few Kafka parameters; for outlet, we need to tune a few
Kafka parameters, the number of workers and a few ClickHouse parameters.
The outlet component features a simple Kafka input component. The core
component becomes just a callback function. There is also a new
ClickHouse component to push data to ClickHouse using the low-level
ch-go library with batch inserts.
This processing has an impact on the internal representation of a
FlowMessage. Previously, it was tailored to dynamically build the
protobuf message to be put in Kafka. Now, it builds the batch request to
be sent to ClickHouse. This makes the FlowMessage structure hides the
content of the next batch request and therefore, it should be reused.
This also changes the way we decode flows as they don't output
FlowMessage anymore, they reuse one that is provided to each worker.
The ClickHouse tables are slightly updated. Instead of using Kafka
engine, the Null engine is used instead.
Fix#1122
Instead, just map configuration files inside the container. As we don't
have to push the schema anymore, pushing some arbitrary configuration
does not seem to be our job.
* Add limit type field selection
* Take into account LimitType to generate SQL request for Line graph
Also, add LimitType in default configuration for console
* Take into account LimitType to generate SQL request for Sankey graph
* Refactor on SQL query used by line and sankey
* Add limitType description in doc
* Order by max in graphLine when limitType max is used
* Fix query when using top max
Revert some modifications, as they were no longer relevant with the query fixed.
* Rework way to sort by max in line graph type
* Add configuration validation on LimitType
---------
Co-authored-by: Dimitri Baudrier <github.52grm@simplelogin.com>
This is useful to detect interfaces that are close to saturation
quickly. It would usually require to group by exporter name and
interface name and it may not make sense for some graph types (like
stacked 100%). It is useful with Lines and Grid.
This displays a line for the previous period on stacked graphs.
Previous period depends on the current period. It could be hour, day,
week, month, or year.
Also propagate this rename to configuration and code. It makes easier
to understand the purpose of such a command in the provided
`docker-compose` file.
