This should be netip.To6() but it does not exist and it was rejected.
There is a benchmark showing the improvment of such optimisation:
BenchmarkNetIPTo6/safe_v4-12 170152954 7.054 ns/op
BenchmarkNetIPTo6/unsafe_v4-12 764772190 1.553 ns/op
See https://github.com/golang/go/issues/54365.
This restores the previous performance of decoding flows. Moreover, we
usually don't need to allocate a large set, as most of the time, the
reverse presents are 1, 2, and 6.
Before:
```
goos: linux
goarch: amd64
pkg: akvorado/outlet/flow
cpu: AMD Ryzen 7 PRO 6850U with Radeon Graphics
BenchmarkDecodeNetFlow-16 100941 11637 ns/op
BenchmarkDecodeBidirNetFlow-16 102210 11369 ns/op
BenchmarkDecodeSflow-16 247066 4684 ns/op
```
After:
```
goos: linux
goarch: amd64
pkg: akvorado/outlet/flow
cpu: AMD Ryzen 7 PRO 6850U with Radeon Graphics
BenchmarkDecodeNetFlow-16 223741 5243 ns/op
BenchmarkDecodeBidirNetFlow-16 220218 5486 ns/op
BenchmarkDecodeSflow-16 220261 5101 ns/op
```
Also, don't decode IPv4/IPv6 addresses when they are 0 (some templates
will include both). Also decode dot1VlanId and postDot1qVlanId but
prefer vlanId and postVlanId if they are present.
Fix#1621
For example:
```
17:35 ❱ curl -s 127.0.0.1:8080/api/v0/outlet/metrics | promtool check metrics
akvorado_outlet_core_classifier_exporter_cache_size_items counter metrics should have "_total" suffix
akvorado_outlet_core_classifier_interface_cache_size_items counter metrics should have "_total" suffix
akvorado_outlet_flow_decoder_netflow_flowset_records_sum counter metrics should have "_total" suffix
akvorado_outlet_flow_decoder_netflow_flowset_records_sum non-histogram and non-summary metrics should not have "_sum" suffix
akvorado_outlet_flow_decoder_netflow_flowset_sum counter metrics should have "_total" suffix
akvorado_outlet_flow_decoder_netflow_flowset_sum non-histogram and non-summary metrics should not have "_sum" suffix
akvorado_outlet_kafka_buffered_fetch_records_total non-counter metrics should not have "_total" suffix
akvorado_outlet_kafka_buffered_produce_records_total non-counter metrics should not have "_total" suffix
akvorado_outlet_metadata_cache_refreshs counter metrics should have "_total" suffix
akvorado_outlet_routing_provider_bmp_peers_total non-counter metrics should not have "_total" suffix
akvorado_outlet_routing_provider_bmp_routes_total non-counter metrics should not have "_total" suffix
```
Also ensure metrics using errors as label don't have a too great
cardinality by using constants for error messages used.
This change split the inlet component into a simpler inlet and a new
outlet component. The new inlet component receive flows and put them in
Kafka, unparsed. The outlet component takes them from Kafka and resume
the processing from here (flow parsing, enrichment) and puts them in
ClickHouse.
The main goal is to ensure the inlet does a minimal work to not be late
when processing packets (and restart faster). It also brings some
simplification as the number of knobs to tune everything is reduced: for
inlet, we only need to tune the queue size for UDP, the number of
workers and a few Kafka parameters; for outlet, we need to tune a few
Kafka parameters, the number of workers and a few ClickHouse parameters.
The outlet component features a simple Kafka input component. The core
component becomes just a callback function. There is also a new
ClickHouse component to push data to ClickHouse using the low-level
ch-go library with batch inserts.
This processing has an impact on the internal representation of a
FlowMessage. Previously, it was tailored to dynamically build the
protobuf message to be put in Kafka. Now, it builds the batch request to
be sent to ClickHouse. This makes the FlowMessage structure hides the
content of the next batch request and therefore, it should be reused.
This also changes the way we decode flows as they don't output
FlowMessage anymore, they reuse one that is provided to each worker.
The ClickHouse tables are slightly updated. Instead of using Kafka
engine, the Null engine is used instead.
Fix#1122
