outlet/flow: persist decoders' state

Currently, for NetFlow, persist data/options templates and options. This
should allow the outlet to restart without loosing any message.

Fix #2100
This commit is contained in:
Vincent Bernat
2025-11-17 18:01:17 +01:00
parent 7c0a8db76d
commit cb92fc7199
17 changed files with 533 additions and 31 deletions

View File

@@ -608,7 +608,7 @@ exporter-classifiers:
### ClickHouse
The ClickHouse component pushes data to ClickHouse. There are two settings that
The ClickHouse component pushes data to ClickHouse. There are three settings that
are configurable:
- `maximum-batch-size` defines how many flows to send to ClickHouse in a single batch at most
@@ -621,6 +621,14 @@ send a batch of size at most `maximum-batch-size` at least every
The default value is 100 000 and allows ClickHouse to handle incoming flows
efficiently.
### Flow
The flow component decodes flows received from Kafka. There is only one setting:
- `state-persist-file` defines the location of the file to save the state of the
flow decoders and read it back on startup. It is used to store IPFIX/NetFlow
templates and options.
## Orchestrator service
The three main components of the orchestrator service are `schema`,

View File

@@ -22,6 +22,7 @@ identified with a specific icon:
- 🩹 *outlet*: accept flows where interface names or descriptions are missing
- 🩹 *docker*: update Traefik to 3.6.1 (for compatibility with Docker Engine 29)
- 🌱 *common*: enable block and mutex profiling
- 🌱 *outlet*: save IPFIX decoder state to a file to prevent discarding flows on start
- 🌱 *config*: rename `verify` to `skip-verify` in TLS configurations for
ClickHouse, Kafka and remote data sources (with inverted logic)
- 🌱 *config*: remote data sources accept a specific TLS configuration