Commit Graph

10 Commits

Author SHA1 Message Date
Vincent Bernat
d2ebd76a5d common/kafka: switch to github.com/IBM/sarama 2023-07-18 08:02:49 +02:00
Vincent Bernat
c6a9319b57 common/schema: turns into a component
This is a first step to make it accept configuration. Most of the
changes are quite trivial, but I also ran into some difficulties with
query columns and filters. They need the schema for parsing, but parsing
happens before dependencies are instantiated (and even if it was not the
case, parsing is stateless). Therefore, I have added a `Validate()`
method that must be called after instantiation. Various bits `panic()`
if not validated to ensure we catch all cases.

The alternative to make the component manages a global state would have
been simpler but it would break once we add the ability to add or
disable columns.
2023-01-18 12:22:10 +01:00
Vincent Bernat
e352202631 inlet: make use of schema for inlet
This is a huge change to make the various subcomponents of the inlet use
the schema to generate the protobuf. For it to make sense, we also
modify the way we parse flows to directly serialize non-essential fields
to Protobuf.

The performance is mostly on par with the previous commit. We are a bit
less efficient because we don't have a fixed structure, but we avoid
loosing too much performance by not relying on reflection and keeping
the production of messages as code. We use less of Goflow2: raw flow
parsing is still done by Goflow2, but we don't use the producer part
anymore. This helps a bit with the performance as we parse less.
Overall, we are 20% than the previous commit and twice faster than the
1.6.4!

```
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
BenchmarkDecodeEncodeNetflow
BenchmarkDecodeEncodeNetflow/with_encoding
BenchmarkDecodeEncodeNetflow/with_encoding-12             151484              7789 ns/op            8272 B/op        143 allocs/op
BenchmarkDecodeEncodeNetflow/without_encoding
BenchmarkDecodeEncodeNetflow/without_encoding-12          162550              7133 ns/op            8272 B/op        143 allocs/op
BenchmarkDecodeEncodeSflow
BenchmarkDecodeEncodeSflow/with_encoding
BenchmarkDecodeEncodeSflow/with_encoding-12                94844             13193 ns/op            9816 B/op        295 allocs/op
BenchmarkDecodeEncodeSflow/without_encoding
BenchmarkDecodeEncodeSflow/without_encoding-12             92569             12456 ns/op            9816 B/op        295 allocs/op
```

There was a tentative to parse sFlow packets with gopackets, but the
adhoc parser used here is more performant.
2023-01-17 20:53:00 +01:00
Vincent Bernat
cd8cf04d03 inlet: don't hard code schema version in some tests
Otherwise, this is painful to update
2022-09-17 23:30:53 +02:00
Vincent Bernat
e5e63be586 inlet/kafka: use a 4-byte random value for key
Using the exporter IP address is not helpful at all as we may not have
many of them and it would make big exporters difficult to scale if one
thread is not enough to ingest on the ClickHouse side.

Fix #75
2022-08-09 15:20:52 +02:00
Vincent Bernat
1e147704c7 inlet: classify exporters to group, role, site, region, and tenant
Previously, this was done only for groups. Encoding everything into
groups is a bit restrictive. The same should be done for IP networks.
2022-07-18 11:01:30 +02:00
Vincent Bernat
8be1bca4fd license: AGPL-3.0-only
```
git ls-files \*.js \*.go \
  | xargs sed -i '1i // SPDX-FileCopyrightText: 2022 Free Mobile\n// SPDX-License-Identifier: AGPL-3.0-only\n'
git ls-files \*.vue \
  | xargs sed -i '1i <!-- SPDX-FileCopyrightText: 2022 Free Mobile -->\n<!-- SPDX-License-Identifier: AGPL-3.0-only -->\n'
```
2022-06-29 11:42:28 +02:00
Vincent Bernat
ce9bd6a4da tests: add an helper to start/stop components 2022-04-13 17:02:14 +02:00
Vincent Bernat
93da599adf cmd: take configuration as a mandatory argument (+ other changes)
The other changes are:
 - rename configure service to orchestrator service
 - turn DefaultConfiguration variables into functions
2022-04-10 15:14:39 +02:00
Vincent Bernat
1dc253764d global: split Akvorado into 3 services 2022-04-01 20:21:53 +02:00