Commit Graph

9 Commits

Author SHA1 Message Date
Vincent Bernat
756e4a8fbd */kafka: switch to franz-go
The concurrency of this library is easier to handle than Sarama.
Notably, it is more compatible with the new model of "almost share
nothing" we use for the inlet and the outlet. The lock for workers in
outlet is removed. We can now use sync.Pool to allocate slice of bytes
in inlet.

It may also be more performant.

In the future, we may want to commit only when pushing data to
ClickHouse. However, this does not seem easy when there is a rebalance.
In case of rebalance, we need to do something when a partition is
revoked to avoid duplicating data. For example, we could flush the
current batch to ClickHouse. Have a look at the
`example/mark_offsets/main.go` file in franz-go repository for a
possible approach. In the meantime, we rely on autocommit.

Another contender could be https://github.com/segmentio/kafka-go. Also
see https://github.com/twmb/franz-go/pull/1064.
2025-07-27 21:44:28 +02:00
Vincent Bernat
ad59598831 inlet/kafka: move metric handling into common/kafka
This will be used for consumer as well.
2025-07-27 21:44:28 +02:00
Vincent Bernat
39882fd98e common: handle per-candidate ports for external service checks 2024-01-20 09:25:31 +01:00
Vincent Bernat
b49abde028 common: use 127.0.0.1 instead of localhost for external test services 2024-01-20 08:29:32 +01:00
Vincent Bernat
5a5820a51a common/kafka: don't try to fiddle with brokers
Assume that if we can refresh metadata we are OK.
2023-09-17 08:56:49 +02:00
Vincent Bernat
4fa05560b8 common/kafka: pace Kafka broker setup tentatives 2023-09-17 08:20:41 +02:00
Vincent Bernat
d2ebd76a5d common/kafka: switch to github.com/IBM/sarama 2023-07-18 08:02:49 +02:00
Vincent Bernat
8be1bca4fd license: AGPL-3.0-only
```
git ls-files \*.js \*.go \
  | xargs sed -i '1i // SPDX-FileCopyrightText: 2022 Free Mobile\n// SPDX-License-Identifier: AGPL-3.0-only\n'
git ls-files \*.vue \
  | xargs sed -i '1i <!-- SPDX-FileCopyrightText: 2022 Free Mobile -->\n<!-- SPDX-License-Identifier: AGPL-3.0-only -->\n'
```
2022-06-29 11:42:28 +02:00
Vincent Bernat
1dc253764d global: split Akvorado into 3 services 2022-04-01 20:21:53 +02:00