Vincent Bernat
92ee2e05b7
outlet/routing: use gaissmai/bart instead of kentik/patricia
...
For each prefix, list of routes is stored into a map (like what is done
with kentik/patricia). The benchmark shows an improvement, both in
insertion time and in memory.
Before:
```
goos: linux
goarch: amd64
pkg: akvorado/outlet/routing/provider/bmp
cpu: AMD Ryzen 5 5600X 6-Core Processor
BenchmarkRandomRealWorldRoutes4-12 10000 106.4 ns/route
BenchmarkRIBInsertion/1000_routes,_1_peers-12 2504 100.0 %ins 361.4 bytes/route 515.6 ns/route
BenchmarkRIBInsertion/1000_routes,_2_peers-12 1234 100.0 %ins 337.5 bytes/route 545.7 ns/route
BenchmarkRIBInsertion/1000_routes,_5_peers-12 412 100.0 %ins 377.6 bytes/route 646.9 ns/route
BenchmarkRIBInsertion/10000_routes,_1_peers-12 170 99.98 %ins 373.0 bytes/route 780.7 ns/route
BenchmarkRIBInsertion/10000_routes,_2_peers-12 52 99.99 %ins 373.2 bytes/route 1136 ns/route
BenchmarkRIBInsertion/10000_routes,_5_peers-12 13 99.99 %ins 299.8 bytes/route 1877 ns/route
BenchmarkRIBInsertion/100000_routes,_1_peers-12 4 99.84 %ins 300.0 bytes/route 2918 ns/route
BenchmarkRIBInsertion/100000_routes,_2_peers-12 2 99.83 %ins 300.2 bytes/route 5220 ns/route
BenchmarkRIBInsertion/100000_routes,_5_peers-12 1 99.81 %ins 340.4 bytes/route 22259 ns/route
BenchmarkRIBLookup/1000_routes,_1_peers-12 56382 214.2 ns/op
BenchmarkRIBLookup/1000_routes,_2_peers-12 52376 227.3 ns/op
BenchmarkRIBLookup/1000_routes,_5_peers-12 46570 257.8 ns/op
BenchmarkRIBLookup/10000_routes,_1_peers-12 4084 277.2 ns/op
BenchmarkRIBLookup/10000_routes,_2_peers-12 3552 295.2 ns/op
BenchmarkRIBLookup/10000_routes,_5_peers-12 3586 340.0 ns/op
BenchmarkRIBLookup/100000_routes,_1_peers-12 300 382.2 ns/op
BenchmarkRIBLookup/100000_routes,_2_peers-12 240 474.1 ns/op
BenchmarkRIBLookup/100000_routes,_5_peers-12 156 752.9 ns/op
BenchmarkRIBFlush/1000_routes,_1_peers-12 8642 0.1422 ms/op
BenchmarkRIBFlush/1000_routes,_2_peers-12 4234 0.2829 ms/op
BenchmarkRIBFlush/1000_routes,_5_peers-12 1995 0.5927 ms/op
BenchmarkRIBFlush/10000_routes,_1_peers-12 807 1.411 ms/op
BenchmarkRIBFlush/10000_routes,_2_peers-12 360 3.341 ms/op
BenchmarkRIBFlush/10000_routes,_5_peers-12 166 7.186 ms/op
BenchmarkRIBFlush/100000_routes,_1_peers-12 58 20.85 ms/op
BenchmarkRIBFlush/100000_routes,_2_peers-12 22 51.13 ms/op
BenchmarkRIBFlush/100000_routes,_5_peers-12 8 135.5 ms/op
```
After:
```
goos: linux
goarch: amd64
pkg: akvorado/outlet/routing/provider/bmp
cpu: AMD Ryzen 5 5600X 6-Core Processor
BenchmarkRandomRealWorldRoutes4-12 10000 110.2 ns/route
BenchmarkRIBInsertion/1000_routes,_1_peers-12 2299 100.0 %ins 348.7 bytes/route 578.4 ns/route
BenchmarkRIBInsertion/1000_routes,_2_peers-12 1112 100.0 %ins 328.7 bytes/route 579.0 ns/route
BenchmarkRIBInsertion/1000_routes,_5_peers-12 432 100.0 %ins 279.7 bytes/route 615.6 ns/route
BenchmarkRIBInsertion/10000_routes,_1_peers-12 182 99.98 %ins 278.1 bytes/route 722.5 ns/route
BenchmarkRIBInsertion/10000_routes,_2_peers-12 61 99.99 %ins 273.0 bytes/route 1013 ns/route
BenchmarkRIBInsertion/10000_routes,_5_peers-12 14 99.99 %ins 232.4 bytes/route 1717 ns/route
BenchmarkRIBInsertion/100000_routes,_1_peers-12 4 99.84 %ins 228.3 bytes/route 2857 ns/route
BenchmarkRIBInsertion/100000_routes,_2_peers-12 2 99.83 %ins 214.3 bytes/route 4944 ns/route
BenchmarkRIBInsertion/100000_routes,_5_peers-12 1 99.81 %ins 265.4 bytes/route 22098 ns/route
BenchmarkRIBLookup/1000_routes,_1_peers-12 61369 190.1 ns/op
BenchmarkRIBLookup/1000_routes,_2_peers-12 64584 186.5 ns/op
BenchmarkRIBLookup/1000_routes,_5_peers-12 63253 190.2 ns/op
BenchmarkRIBLookup/10000_routes,_1_peers-12 5934 188.7 ns/op
BenchmarkRIBLookup/10000_routes,_2_peers-12 5386 207.7 ns/op
BenchmarkRIBLookup/10000_routes,_5_peers-12 5348 220.3 ns/op
BenchmarkRIBLookup/100000_routes,_1_peers-12 516 227.1 ns/op
BenchmarkRIBLookup/100000_routes,_2_peers-12 477 241.7 ns/op
BenchmarkRIBLookup/100000_routes,_5_peers-12 428 264.2 ns/op
BenchmarkRIBFlush/1000_routes,_1_peers-12 5246 0.2294 ms/op
BenchmarkRIBFlush/1000_routes,_2_peers-12 2984 0.3965 ms/op
BenchmarkRIBFlush/1000_routes,_5_peers-12 1406 0.8498 ms/op
BenchmarkRIBFlush/10000_routes,_1_peers-12 578 2.084 ms/op
BenchmarkRIBFlush/10000_routes,_2_peers-12 295 3.988 ms/op
BenchmarkRIBFlush/10000_routes,_5_peers-12 100 10.15 ms/op
BenchmarkRIBFlush/100000_routes,_1_peers-12 33 30.82 ms/op
BenchmarkRIBFlush/100000_routes,_2_peers-12 18 61.41 ms/op
BenchmarkRIBFlush/100000_routes,_5_peers-12 7 158.4 ms/op
```
This is a 20% improvement on insertion, 30% on lookups, but 36%
degradation for flushing.
Fix #253
Next steps:
- test lockless updates (with *Persist functions)
2025-08-16 17:06:36 +02:00
Vincent Bernat
ac68c5970e
inlet: split inlet into new inlet and outlet
...
This change split the inlet component into a simpler inlet and a new
outlet component. The new inlet component receive flows and put them in
Kafka, unparsed. The outlet component takes them from Kafka and resume
the processing from here (flow parsing, enrichment) and puts them in
ClickHouse.
The main goal is to ensure the inlet does a minimal work to not be late
when processing packets (and restart faster). It also brings some
simplification as the number of knobs to tune everything is reduced: for
inlet, we only need to tune the queue size for UDP, the number of
workers and a few Kafka parameters; for outlet, we need to tune a few
Kafka parameters, the number of workers and a few ClickHouse parameters.
The outlet component features a simple Kafka input component. The core
component becomes just a callback function. There is also a new
ClickHouse component to push data to ClickHouse using the low-level
ch-go library with batch inserts.
This processing has an impact on the internal representation of a
FlowMessage. Previously, it was tailored to dynamically build the
protobuf message to be put in Kafka. Now, it builds the batch request to
be sent to ClickHouse. This makes the FlowMessage structure hides the
content of the next batch request and therefore, it should be reused.
This also changes the way we decode flows as they don't output
FlowMessage anymore, they reuse one that is provided to each worker.
The ClickHouse tables are slightly updated. Instead of using Kafka
engine, the Null engine is used instead.
Fix #1122
2025-07-27 21:44:28 +02:00
