Vincent Bernat
8886f4a212
Revert "common/kafka: disable client telemetry"
...
CI / 🤖 Check dependabot status (push) Has been cancelled
CI / 🐧 Test on Linux (${{ github.ref_type == 'tag' }}, misc) (push) Has been cancelled
CI / 🐧 Test on Linux (coverage) (push) Has been cancelled
CI / 🐧 Test on Linux (regular) (push) Has been cancelled
CI / ❄️ Build on Nix (push) Has been cancelled
CI / 🍏 Build and test on macOS (push) Has been cancelled
CI / 🧪 End-to-end testing (push) Has been cancelled
CI / 🔍 Upload code coverage (push) Has been cancelled
CI / 🔬 Test only Go (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 20) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 22) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 24) (push) Has been cancelled
CI / ⚖️ Check licenses (push) Has been cancelled
CI / 🐋 Build Docker images (push) Has been cancelled
CI / 🐋 Tag Docker images (push) Has been cancelled
CI / 🚀 Publish release (push) Has been cancelled
This reverts commit c2b7754c15
2025-11-24 18:13:13 +01:00
Vincent Bernat
c2b7754c15
common/kafka: disable client telemetry
...
CI / 🤖 Check dependabot status (push) Has been cancelled
CI / 🐧 Test on Linux (${{ github.ref_type == 'tag' }}, misc) (push) Has been cancelled
CI / 🐧 Test on Linux (coverage) (push) Has been cancelled
CI / 🐧 Test on Linux (regular) (push) Has been cancelled
CI / ❄️ Build on Nix (push) Has been cancelled
CI / 🍏 Build and test on macOS (push) Has been cancelled
CI / 🧪 End-to-end testing (push) Has been cancelled
CI / 🔍 Upload code coverage (push) Has been cancelled
CI / 🔬 Test only Go (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 20) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 22) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 24) (push) Has been cancelled
CI / ⚖️ Check licenses (push) Has been cancelled
CI / 🐋 Build Docker images (push) Has been cancelled
CI / 🐋 Tag Docker images (push) Has been cancelled
CI / 🚀 Publish release (push) Has been cancelled
Temporarily disable client telemetry (not enabled by default in the
Docker Compose setup). This is until
https://github.com/twmb/franz-go/pull/1174 is merged.
2025-11-19 09:59:48 +01:00
Vincent Bernat
6b27794dd2
common/kafka: fix logger for kfake
2025-11-17 21:54:45 +01:00
Vincent Bernat
abca5e983d
chore: modernize some code
2025-11-14 23:22:02 +01:00
Vincent Bernat
a3507a3ff2
commong: move logging functions into */logs.go
...
The idea is that we should be able to skip it to find the true call
site. However, in some case, there is no true call site as we are in an
internal goroutine (like for Kafka). We can't do too complex things as
it would cost more CPU.
Here is the tentative. We should keep the last valid caller.
```go
// Run adds more context to an event, including "module" and "caller".
func (h contextHook) Run(e *zerolog.Event, _ zerolog.Level, _ string) {
callStack := stack.Callers()
callStack = callStack[3:] // Trial and error, there is a test to check it works.
// We want to get the next caller that is in our own module but that is not logs.go.
for _, call := range callStack {
module := call.FunctionName()
if !strings.HasPrefix(module, stack.ModuleName) || strings.HasSuffix(call.FileName(), "/logs.go") {
continue
}
caller := callStack[0].SourceFile(true)
e.Str("caller", caller)
module = strings.SplitN(module, ".", 2)[0]
e.Str("module", module)
break
}
}
```
2025-11-05 08:22:33 +01:00
Vincent Bernat
e68b2de72c
common/helpers: migrate from verify to skip-verify in TLS config
...
CI / 🤖 Check dependabot status (push) Has been cancelled
CI / 🐧 Test on Linux (${{ github.ref_type == 'tag' }}, misc) (push) Has been cancelled
CI / 🐧 Test on Linux (coverage) (push) Has been cancelled
CI / 🐧 Test on Linux (regular) (push) Has been cancelled
CI / ❄️ Build on Nix (push) Has been cancelled
CI / 🍏 Build and test on macOS (push) Has been cancelled
CI / 🧪 End-to-end testing (push) Has been cancelled
CI / 🔍 Upload code coverage (push) Has been cancelled
CI / 🔬 Test only Go (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 20) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 22) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 24) (push) Has been cancelled
CI / ⚖️ Check licenses (push) Has been cancelled
CI / 🐋 Build Docker images (push) Has been cancelled
CI / 🐋 Tag Docker images (push) Has been cancelled
CI / 🚀 Publish release (push) Has been cancelled
Otherwise, the default is "false" for verify. This is a breaking change.
Fix #2055 .
2025-10-30 08:31:27 +01:00
Vincent Bernat
801f3f1676
common/kafka: also logs output of kfake cluster
2025-09-23 07:06:58 +02:00
Vincent Bernat
98eb1bdba5
chore: make a run of gofumpt
2025-08-05 06:21:34 +02:00
Vincent Bernat
03b947e3c5
chore: fix many staticcheck warnings
...
The most important ones were fixed in the two previous commit.
2025-08-02 20:54:49 +02:00
Vincent Bernat
18beb310ee
chore: replace interface{} with any
2025-07-29 07:42:49 +02:00
Vincent Bernat
76151bea66
common/helpers: make some mapstructure hooks work with embedded structs
...
When using `mapstructure:",squash"`, most structure-specific hook don't
dive into the structure as they are provided with the parent structure.
Add an helper to make them work on the embedded structure as well and
use it for the generic "deprecated fields" hook, but also for the hook
for the common Kafka configuration.
This is a bit brittle. There are other use cases, but they may not need
this change.
2025-07-27 21:44:28 +02:00
Vincent Bernat
756e4a8fbd
*/kafka: switch to franz-go
...
The concurrency of this library is easier to handle than Sarama.
Notably, it is more compatible with the new model of "almost share
nothing" we use for the inlet and the outlet. The lock for workers in
outlet is removed. We can now use sync.Pool to allocate slice of bytes
in inlet.
It may also be more performant.
In the future, we may want to commit only when pushing data to
ClickHouse. However, this does not seem easy when there is a rebalance.
In case of rebalance, we need to do something when a partition is
revoked to avoid duplicating data. For example, we could flush the
current batch to ClickHouse. Have a look at the
`example/mark_offsets/main.go` file in franz-go repository for a
possible approach. In the meantime, we rely on autocommit.
Another contender could be https://github.com/segmentio/kafka-go . Also
see https://github.com/twmb/franz-go/pull/1064 .
2025-07-27 21:44:28 +02:00
Vincent Bernat
ad59598831
inlet/kafka: move metric handling into common/kafka
...
This will be used for consumer as well.
2025-07-27 21:44:28 +02:00
Vincent Bernat
a70744429a
common/kafka: ability to specify OAuth scopes
2025-05-02 06:55:47 +02:00
Vincent Bernat
55b74a1954
common/kafka: rely on mechanism to enable or disable SASL
...
CI / 🤖 Check dependabot status (push) Waiting to run
CI / 🐧 Build and test on Linux (push) Blocked by required conditions
CI / 🍏 Build and test on macOS (push) Blocked by required conditions
CI / 🔍 Upload code coverage (push) Blocked by required conditions
CI / 🔭 Build Go backend (1.24) (push) Blocked by required conditions
CI / 🔭 Build JS frontend (18) (push) Blocked by required conditions
CI / 🔭 Build JS frontend (20) (push) Blocked by required conditions
CI / 🔭 Build JS frontend (22) (push) Blocked by required conditions
CI / ⚖️ Check licenses (push) Waiting to run
CI / 🐋 Build Docker images (push) Blocked by required conditions
CI / 🚀 Publish release (push) Blocked by required conditions
Update Nix dependencies / Update Nix lockfile (asn2org) (push) Waiting to run
Update Nix dependencies / Update Nix lockfile (nixpkgs) (push) Waiting to run
Update Nix dependencies / Update dependency hashes (push) Waiting to run
Instead of username. This should be the same, but the code is more
correct this way.
2025-05-01 20:16:03 +02:00
Vincent Bernat
f672ac98d9
common/kafka: add support for OAuth2
...
The support is still pretty basic. Notably, scopes are not
configurable (waiting for someone to request them) and maybe there
client ID and secrets should not be provided as username/password.
Fix #1714
2025-05-01 19:37:06 +02:00
Vincent Bernat
113df1995f
common/kafka: put SASL parameters in their own section
2025-05-01 19:37:06 +02:00
Vincent Bernat
4332750edb
common/kafka: use enumer for SASLMechanism
2024-11-23 23:48:02 +01:00
Vincent Bernat
a6ce62dda0
chore: fix inline comments in exported const blocks
...
They should not be inline, as this is not supported by godoc. This is
reported by revive 1.5.0.
2024-11-11 07:03:52 +01:00
Vincent Bernat
102cd7fe9c
common/kafka: set Kafka ClientID to akvorado instead of sarama
2024-07-21 16:18:01 +02:00
Vincent Bernat
f321e8fa64
common/helpers: add a way to test Marshal/Unmarshal for bimaps
2024-01-22 21:53:26 +01:00
Vincent Bernat
39882fd98e
common: handle per-candidate ports for external service checks
2024-01-20 09:25:31 +01:00
Vincent Bernat
b49abde028
common: use 127.0.0.1 instead of localhost for external test services
2024-01-20 08:29:32 +01:00
netixx
e2dc6e4f53
Add TLS support for ClickhouseDB
2024-01-17 11:25:15 +01:00
Vincent Bernat
5a5820a51a
common/kafka: don't try to fiddle with brokers
...
Assume that if we can refresh metadata we are OK.
2023-09-17 08:56:49 +02:00
Vincent Bernat
4fa05560b8
common/kafka: pace Kafka broker setup tentatives
2023-09-17 08:20:41 +02:00
Vincent Bernat
d2ebd76a5d
common/kafka: switch to github.com/IBM/sarama
2023-07-18 08:02:49 +02:00
Vincent Bernat
a912da7fa1
build: use gofumpt
...
Undecided if we need to use it. I think it's nice.
2023-02-11 10:03:45 +01:00
Vincent Bernat
24cfabb682
common/helpers: move bimap and intern to separate packages
...
subnetmap would be a candidate too, but there are cyclic dependencies
because we want diff to handle it correctly.
2022-11-28 15:22:59 +01:00
Vincent Bernat
fa7e4b6ac7
inlet/kafka: add TLS support to Kafka
...
Fix #277
2022-11-25 01:00:21 +01:00
Vincent Bernat
18ee3eac6f
build: switch to Go 1.19
...
This enables us to use atomic.Pointer!
2022-08-19 17:46:49 +02:00
Vincent Bernat
574ec7e79e
inlet/snmp: add proper support for SNMPv3
2022-08-14 10:20:19 +02:00
Vincent Bernat
61c8143fe8
build: fix linting issues introduced by latest revive release
2022-08-07 23:54:24 +02:00
Vincent Bernat
6121aaea15
config: use a validator for better configuration validation
2022-06-30 01:23:29 +02:00
Vincent Bernat
8be1bca4fd
license: AGPL-3.0-only
...
```
git ls-files \*.js \*.go \
| xargs sed -i '1i // SPDX-FileCopyrightText: 2022 Free Mobile\n// SPDX-License-Identifier: AGPL-3.0-only\n'
git ls-files \*.vue \
| xargs sed -i '1i <!-- SPDX-FileCopyrightText: 2022 Free Mobile -->\n<!-- SPDX-License-Identifier: AGPL-3.0-only -->\n'
```
2022-06-29 11:42:28 +02:00
Vincent Bernat
93da599adf
cmd: take configuration as a mandatory argument (+ other changes)
...
The other changes are:
- rename configure service to orchestrator service
- turn DefaultConfiguration variables into functions
2022-04-10 15:14:39 +02:00
Vincent Bernat
1dc253764d
global: split Akvorado into 3 services
2022-04-01 20:21:53 +02:00
