Vincent Bernat
1dc253764d
global: split Akvorado into 3 services
2022-04-01 20:21:53 +02:00
Vincent Bernat
a336370c05
web: frontend using Bootstrap
2022-03-31 20:56:14 +02:00
Vincent Bernat
41d3ee538f
global: rename sampler to exporter
...
This is the terminology for IPFix. With Netflow, sampler is ambiguous
as it can mean the exporter or the sampler inside the exporter.
2022-03-29 11:27:54 +02:00
Vincent Bernat
6d716e3e5b
doc: update documentation with a troubleshooting section
2022-03-28 12:52:51 +02:00
Vincent Bernat
b086c9478a
flow: add IPFIX support
2022-03-26 21:25:19 +01:00
Vincent Bernat
1b4ba973a7
flow: make input decode flows
...
We cannot pass the payload around as we want to reuse it. We should
pass a copy, but this is a bit of a waste. Just do decoding during
input ingestion. The CPU usage should be small, this helps locality
and it avoids one channel.
2022-03-26 15:18:52 +01:00
Vincent Bernat
c0fe3c5b75
doc: split design into two files
...
Most users won't care about the programming design.
2022-03-26 07:50:50 +01:00
Vincent Bernat
c54ecadae2
kafka: add a setting to tune internal queue sizes
2022-03-26 07:50:50 +01:00
Vincent Bernat
c256d3e097
reporter: use zerolog own rate limiter
2022-03-26 07:50:50 +01:00
Vincent Bernat
0abf71224c
flow: make inputs modular
2022-03-25 23:35:41 +01:00
Vincent Bernat
ab6888d3e5
flow: don't reuse IP fields from the original protobuf message
...
For some reason, they may get modified. This is a bit unclear how and
when since the structure is not serialized at any point, but
otherwise, src/destination IPs may get randomly corrupted.
2022-03-24 15:21:45 +01:00
Vincent Bernat
50ad351e6a
doc: update SNMP design with latest commits
2022-03-23 18:22:13 +01:00
Vincent Bernat
29471c7073
snmp: lower even more the default value fo coalescing
...
If we assume a MTU of 1500, we may run into trouble with long
descriptions. Assume 50 bytes for description, 30 bytes for interface
name, plus 30 bytes for OIDs, it means we can only put 13 results in a
standard MTU.
At some point, we may try to be smarter and have per-sampler values?
2022-03-23 18:14:43 +01:00
Vincent Bernat
913e7514b2
snmp: add a breaker for too samplers spewing errors
2022-03-23 16:10:36 +01:00
Vincent Bernat
1a0a7184c2
snmp: make retries and timeouts configurable
...
And reduce the number of retries.
2022-03-23 09:45:16 +01:00
Vincent Bernat
443a16bc6f
snmp: adapt cache default timings so that they make sense
...
If we refresh SNMP cache sooner than the expiration delay, expiration
will never take place.
2022-03-23 08:11:52 +01:00
Vincent Bernat
bb1ce0fde5
flow: add a metric when the internal queue is full
...
This gives a good hint if there is a problem. Maybe we don't need to
optimize more.
2022-03-22 23:21:08 +01:00
Vincent Bernat
cf1fe53511
flow: rename buffer-length option to buffer-size
...
It is a better match. I was worried people would understand the size
as in bytes, but we also say a channel size.
2022-03-22 22:43:36 +01:00
Vincent Bernat
62c6fbbe6f
clickhouse: add a config setting to override URL to reach HTTP server
2022-03-22 16:20:24 +01:00
Vincent Bernat
5f5479524f
reporter: factorize how we use channels for healthchecking
...
Add two to the snmp component. Other components are not interesting.
2022-03-22 13:43:30 +01:00
Vincent Bernat
15483ee8f6
reporter: add a generic healthcheck mechanism, usable accross the app
...
Convert the one from core.
2022-03-22 12:52:43 +01:00
Vincent Bernat
898c23713d
docs: add some additional information about ClickHouse
2022-03-22 07:47:25 +01:00
Vincent Bernat
c970850d7a
clickhouse: complete integration of ClickHouse by populating the database
...
We use go-migrate to handle migrations. Only the "up" direction is
done for now. As migrations are handled inside the application,
writing downgrades is not really useful as it would be needed when
downgrading an app, but then we cannot downgrade. Moreover, the way
the flow topics are versioned, you can downgrade without much dommage.
2022-03-21 22:45:14 +01:00
Vincent Bernat
1db159fd1f
cmd: ability to override settings with environment variables
2022-03-21 11:28:05 +01:00
Vincent Bernat
e25652c1c4
cmd: directly use mapstructure for configuration parsing
...
No point in using Viper as it mostly work when binding explicitely
each setting or when requesting each setting through its methods.
2022-03-21 01:52:26 +01:00
Vincent Bernat
41131fca96
flow: introduce versioned flows
...
We need to version flow schemas. Otherwise, this won't be manageable.
Confluent is pushing for a registry, but it seems the ecosystem is
still too young. Let's version on our side with a topic for each
version.
2022-03-20 22:01:31 +01:00
Vincent Bernat
be0d510b12
clickhouse: add a new Clickhouse component to help setting Clickhouse
2022-03-20 22:01:31 +01:00
Vincent Bernat
b691e299c6
kafka: add topic creation
2022-03-20 22:01:31 +01:00
Vincent Bernat
e3c609b290
cmd: accept snake case for configuration keys
2022-03-20 22:01:31 +01:00
Vincent Bernat
417670da80
all: move most endpoints to /api/v0
2022-03-19 17:34:14 +01:00
Vincent Bernat
ef508dbaa6
core: detect regex error early
2022-03-19 12:21:01 +01:00
Vincent Bernat
4118bc2a45
doc: remove item on the todo list
2022-03-18 16:20:55 +01:00
Vincent Bernat
7728c3089d
core: add classification support
2022-03-18 13:36:27 +01:00
Vincent Bernat
845fc7c18c
docs: more documentation for design
2022-03-18 08:30:27 +01:00
Vincent Bernat
ae31596423
common: rename augment to hydrate
...
Funnier word.
2022-03-18 07:50:03 +01:00
Vincent Bernat
7c5a7bdbef
core: move augmentation function outside of root.go
...
Also enrich → augment. Not sure. Or hydrate?
2022-03-18 07:47:49 +01:00
Vincent Bernat
ad14826f6f
web: add a reverse proxy to Grafana
2022-03-17 22:58:19 +01:00
Vincent Bernat
91e3d7661f
docs: add documentation served directly by a dedicated component
2022-03-17 22:16:05 +01:00
