Vincent Bernat
e68b2de72c
common/helpers: migrate from verify to skip-verify in TLS config
...
CI / 🤖 Check dependabot status (push) Has been cancelled
CI / 🐧 Test on Linux (${{ github.ref_type == 'tag' }}, misc) (push) Has been cancelled
CI / 🐧 Test on Linux (coverage) (push) Has been cancelled
CI / 🐧 Test on Linux (regular) (push) Has been cancelled
CI / ❄️ Build on Nix (push) Has been cancelled
CI / 🍏 Build and test on macOS (push) Has been cancelled
CI / 🧪 End-to-end testing (push) Has been cancelled
CI / 🔍 Upload code coverage (push) Has been cancelled
CI / 🔬 Test only Go (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 20) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 22) (push) Has been cancelled
CI / 🔬 Test only JS (${{ needs.dependabot.outputs.package-ecosystem }}, 24) (push) Has been cancelled
CI / ⚖️ Check licenses (push) Has been cancelled
CI / 🐋 Build Docker images (push) Has been cancelled
CI / 🐋 Tag Docker images (push) Has been cancelled
CI / 🚀 Publish release (push) Has been cancelled
Otherwise, the default is "false" for verify. This is a breaking change.
Fix #2055 .
2025-10-30 08:31:27 +01:00
Vincent Bernat
18beb310ee
chore: replace interface{} with any
2025-07-29 07:42:49 +02:00
Vincent Bernat
756e4a8fbd
*/kafka: switch to franz-go
...
The concurrency of this library is easier to handle than Sarama.
Notably, it is more compatible with the new model of "almost share
nothing" we use for the inlet and the outlet. The lock for workers in
outlet is removed. We can now use sync.Pool to allocate slice of bytes
in inlet.
It may also be more performant.
In the future, we may want to commit only when pushing data to
ClickHouse. However, this does not seem easy when there is a rebalance.
In case of rebalance, we need to do something when a partition is
revoked to avoid duplicating data. For example, we could flush the
current batch to ClickHouse. Have a look at the
`example/mark_offsets/main.go` file in franz-go repository for a
possible approach. In the meantime, we rely on autocommit.
Another contender could be https://github.com/segmentio/kafka-go . Also
see https://github.com/twmb/franz-go/pull/1064 .
2025-07-27 21:44:28 +02:00
Vincent Bernat
a70744429a
common/kafka: ability to specify OAuth scopes
2025-05-02 06:55:47 +02:00
Vincent Bernat
55b74a1954
common/kafka: rely on mechanism to enable or disable SASL
...
CI / 🤖 Check dependabot status (push) Waiting to run
CI / 🐧 Build and test on Linux (push) Blocked by required conditions
CI / 🍏 Build and test on macOS (push) Blocked by required conditions
CI / 🔍 Upload code coverage (push) Blocked by required conditions
CI / 🔭 Build Go backend (1.24) (push) Blocked by required conditions
CI / 🔭 Build JS frontend (18) (push) Blocked by required conditions
CI / 🔭 Build JS frontend (20) (push) Blocked by required conditions
CI / 🔭 Build JS frontend (22) (push) Blocked by required conditions
CI / ⚖️ Check licenses (push) Waiting to run
CI / 🐋 Build Docker images (push) Blocked by required conditions
CI / 🚀 Publish release (push) Blocked by required conditions
Update Nix dependencies / Update Nix lockfile (asn2org) (push) Waiting to run
Update Nix dependencies / Update Nix lockfile (nixpkgs) (push) Waiting to run
Update Nix dependencies / Update dependency hashes (push) Waiting to run
Instead of username. This should be the same, but the code is more
correct this way.
2025-05-01 20:16:03 +02:00
Vincent Bernat
f672ac98d9
common/kafka: add support for OAuth2
...
The support is still pretty basic. Notably, scopes are not
configurable (waiting for someone to request them) and maybe there
client ID and secrets should not be provided as username/password.
Fix #1714
2025-05-01 19:37:06 +02:00
Vincent Bernat
113df1995f
common/kafka: put SASL parameters in their own section
2025-05-01 19:37:06 +02:00
Vincent Bernat
4332750edb
common/kafka: use enumer for SASLMechanism
2024-11-23 23:48:02 +01:00
Vincent Bernat
f321e8fa64
common/helpers: add a way to test Marshal/Unmarshal for bimaps
2024-01-22 21:53:26 +01:00
netixx
e2dc6e4f53
Add TLS support for ClickhouseDB
2024-01-17 11:25:15 +01:00
Vincent Bernat
d2ebd76a5d
common/kafka: switch to github.com/IBM/sarama
2023-07-18 08:02:49 +02:00
Vincent Bernat
a912da7fa1
build: use gofumpt
...
Undecided if we need to use it. I think it's nice.
2023-02-11 10:03:45 +01:00
Vincent Bernat
fa7e4b6ac7
inlet/kafka: add TLS support to Kafka
...
Fix #277
2022-11-25 01:00:21 +01:00
Vincent Bernat
6121aaea15
config: use a validator for better configuration validation
2022-06-30 01:23:29 +02:00
