Vincent Bernat
1dc253764d
global: split Akvorado into 3 services
2022-04-01 20:21:53 +02:00
Vincent Bernat
a336370c05
web: frontend using Bootstrap
2022-03-31 20:56:14 +02:00
Vincent Bernat
de88cd92ae
core: do not make "no more flows available an error"
2022-03-31 08:14:34 +02:00
Vincent Bernat
9a8916b64e
clickhouse: move template rendering to Open()
...
This way, we got proper error reports. Read() cannot fail anymore.
2022-03-29 21:48:37 +02:00
Vincent Bernat
aa80787f53
build: update go.mod/go.sum
2022-03-29 21:43:47 +02:00
Vincent Bernat
38e3b790bc
clickhouse: enable logger to correctly log Clickhouse errors
2022-03-29 21:40:20 +02:00
Vincent Bernat
b8755808f1
clickhouse: fix typo in migration template
...
It seems go-migrate does not check for errors when reading files...
2022-03-29 21:40:20 +02:00
Vincent Bernat
e5c8988b43
http: add metrics and logs
2022-03-29 21:40:20 +02:00
Vincent Bernat
741543ddf4
clickhouse: use one kafka consumer per partition
2022-03-29 13:52:14 +02:00
Vincent Bernat
b38c974d0f
snmp: make refresh test more reliable by not jumping clock too fast
2022-03-29 13:30:49 +02:00
Vincent Bernat
b6a6a0392d
build: fix build on Gitlab
2022-03-29 13:22:49 +02:00
Vincent Bernat
6545ea91f6
clickhouse: fix 9th migration
2022-03-29 13:02:25 +02:00
Vincent Bernat
41d3ee538f
global: rename sampler to exporter
...
This is the terminology for IPFix. With Netflow, sampler is ambiguous
as it can mean the exporter or the sampler inside the exporter.
2022-03-29 11:27:54 +02:00
Vincent Bernat
15627fe63b
web: add option to serve live filesystem instead of embedded one
2022-03-28 22:32:12 +02:00
Vincent Bernat
952cfabcbc
helpers: remove unneeded files
2022-03-28 22:11:40 +02:00
Vincent Bernat
ee6aeadbf1
clickhouse: alter codec of the TimeReceived column
...
This improves compression which should also improve reading time due
to less IO.
2022-03-28 19:51:22 +02:00
Vincent Bernat
6d716e3e5b
doc: update documentation with a troubleshooting section
2022-03-28 12:52:51 +02:00
Vincent Bernat
b1890991cc
flow: make input overflow more reproducible
2022-03-28 11:49:52 +02:00
Vincent Bernat
3b0f6b5a27
flow: add a counter to monitor dropped packets
2022-03-28 09:52:17 +02:00
Vincent Bernat
ba99a50b43
flow: add option to set UDP receive buffer size
2022-03-28 09:11:48 +02:00
Vincent Bernat
120d9d3bd3
flow: don't use go-reuseport
...
It's a few lines to do that ourselves. This enables us to get control
of the socket.
2022-03-27 14:32:03 +02:00
Vincent Bernat
b086c9478a
flow: add IPFIX support
2022-03-26 21:25:19 +01:00
Vincent Bernat
1b4ba973a7
flow: make input decode flows
...
We cannot pass the payload around as we want to reuse it. We should
pass a copy, but this is a bit of a waste. Just do decoding during
input ingestion. The CPU usage should be small, this helps locality
and it avoids one channel.
2022-03-26 15:18:52 +01:00
Vincent Bernat
176e4999b9
build: speed up linting by always using ./...
...
This is quite odd, but revive is super slow when providing many
directories. I suppose they are evaluated several times.
2022-03-26 15:15:25 +01:00
Vincent Bernat
1a022ce94d
snmp: use a per-sampler breaker logger
...
Otherwise, we may think there is an issue with only one sampler.
2022-03-26 13:20:41 +01:00
Vincent Bernat
794f9bf6e4
poller: specify number of OIDs we tried to get
2022-03-26 12:49:15 +01:00
Vincent Bernat
a5a8d8a55c
snmp: remove useless metrics
2022-03-26 12:46:54 +01:00
Vincent Bernat
47fe13a617
core: remove useless metrics
2022-03-26 12:46:53 +01:00
Vincent Bernat
cb35ebdcd7
snmp: remove debug code
2022-03-26 12:24:41 +01:00
Vincent Bernat
c44434a8fa
cmd: allow to set arrays using environment variables
2022-03-26 12:21:03 +01:00
Vincent Bernat
43f58bde60
http: optionally enable profiler
2022-03-26 11:51:13 +01:00
Vincent Bernat
6899d65f15
flow: add a default value for inputs
2022-03-26 11:45:10 +01:00
Vincent Bernat
41fc4cb336
snmp: fix flaky test around coaelescing
...
We didn't block the dispatcher, but the poller. One more message was
needed to block the dispatcher. Let's simplify all that by
instrumenting the dispatcher to be blocked on demand.
2022-03-26 08:47:22 +01:00
Vincent Bernat
4eeb775b27
cmd: start flow component last to speedup stop and avoid clogging start
2022-03-26 08:01:21 +01:00
Vincent Bernat
d5ecc4d792
tests: use current source file to find testdata for helpers
2022-03-26 07:50:50 +01:00
Vincent Bernat
050ad03241
build: don't display color on dumb terminal
2022-03-26 07:50:50 +01:00
Vincent Bernat
c0fe3c5b75
doc: split design into two files
...
Most users won't care about the programming design.
2022-03-26 07:50:50 +01:00
Vincent Bernat
c54ecadae2
kafka: add a setting to tune internal queue sizes
2022-03-26 07:50:50 +01:00
Vincent Bernat
c256d3e097
reporter: use zerolog own rate limiter
2022-03-26 07:50:50 +01:00
Vincent Bernat
0abf71224c
flow: make inputs modular
2022-03-25 23:35:41 +01:00
Vincent Bernat
7aa75269bc
reporter: allow to declare several times the same metric
...
This will simplify the way we handle inputs.
2022-03-24 23:57:55 +01:00
Vincent Bernat
bdda4e3be3
flow: adopt a static approach to register decoders
...
This makes little sense to have something dynamic if all our decoders
are in the same codebase. Just register all decoders into a struct.
2022-03-24 22:41:42 +01:00
Vincent Bernat
537065db52
flow: make flow decoding modular
...
This is a preparation step to support multiple inputs and decoders.
2022-03-24 22:23:00 +01:00
Vincent Bernat
844e0b4b52
web: log HTTP proxy errors with zerolog
2022-03-24 18:44:04 +01:00
Vincent Bernat
41a704c044
snmp: try harder to ensure packets get queued for coalescing test
2022-03-24 16:21:06 +01:00
Vincent Bernat
ab6888d3e5
flow: don't reuse IP fields from the original protobuf message
...
For some reason, they may get modified. This is a bit unclear how and
when since the structure is not serialized at any point, but
otherwise, src/destination IPs may get randomly corrupted.
2022-03-24 15:21:45 +01:00
Vincent Bernat
50ad351e6a
doc: update SNMP design with latest commits
2022-03-23 18:22:13 +01:00
Vincent Bernat
29471c7073
snmp: lower even more the default value fo coalescing
...
If we assume a MTU of 1500, we may run into trouble with long
descriptions. Assume 50 bytes for description, 30 bytes for interface
name, plus 30 bytes for OIDs, it means we can only put 13 results in a
standard MTU.
At some point, we may try to be smarter and have per-sampler values?
2022-03-23 18:14:43 +01:00
Vincent Bernat
064d734d23
snmp: fix default value of poller-coalesce
...
Just one unit too big!
2022-03-23 18:03:13 +01:00
Vincent Bernat
3d6d28f15f
build: try twice test-race and test-coverage when test is successful
2022-03-23 16:39:12 +01:00
