And also add documentation on how to use IPv6. The proposed setup relies
on NAT66, which is not good, but it works on any host with IPv6
connectivity. The documentation explains how to configure routed IPv6.
By using an IPv4 subnet in class E, we ensure that it is very unlikely
users will have overlap between their Docker setup and their production
network. This way, no need to change the Docker daemon configuration.
The advice was not true. An active part is not one that should be
actively merged, it's one that is used (and not to be deleted).
ClickHouse is good with more than 10k parts.
And as a temporary workaround, reduce the number of Kafka workers to 1.
This should be more dynamic to keep the number of inserted rows around
100k. Another strategy would be buffer tables or async inserts.
This does not seem to survive a restart. There is no indication in the
documentation this is the right way. One should modify settings
directly. I need to investigate how to do this properly with Docker.
* Add functionality for overwriting the exporter address with flow source ip
* Remove "agent-id-src-addr-overwrite" from default config
* Improve use-src-addr-for-exporter-addr documentation
* Rename to UseSrcAddrForExporterAddr
* Fix use-src-addr-for-exporter-addr key in example config
* Add UseSrcAddrForExporterAddr to configuration test
Co-authored-by: Marvin Gaube <marvin.gaube@exaring.de>
At first, there was a tentative to use BMP collector implementation
from bio-rd. However, this current implementation is using GoBGP
instead:
- BMP is very simple from a protocol point of view. The hard work is
mostly around decoding. Both bio-rd and GoBGP can decode, but for
testing, GoBGP is able to generate messages as well (this is its
primary purpose, I suppose parsing was done for testing purpose).
Using only one library is always better. An alternative would be
GoBMP, but it also only do parsing.
- Logging and metrics can be customized easily (but the work was done
for bio-rd, so not a real argument).
- bio-rd is an application and there is no API stability (and I did
that too)
- GoBGP supports FlowSpec, which may be useful in the future for the
DDoS part. Again, one library for everything is better (but
honestly, GoBGP as a lib is not the best part of it, maybe
github.com/jwhited/corebgp would be a better fit while keeping GoBGP
for decoding/encoding).
There was a huge effort around having a RIB which is efficient
memory-wise (data are interned to save memory), performant during
reads, while being decent during insertions. We rely on a patched
version of Kentik's Patricia trees to be able to apply mutations to
the tree.
There was several tentatives to implement some kind of graceful
restart, but ultimetaly, the design is kept simple: when a BMP
connection goes down, routes will be removed after a configurable
time. If the connection comes back up, then it is just considered new.
It would have been ideal to rely on EoR markers, but the RFC is
unclear about them, and they are likely to be per peer, making it
difficult to know what to do if one peer is back, but not the other.
Remaining tasks:
- [ ] Confirm support for LocRIB
- [ ] Import data in ClickHouse
- [ ] Make data available in the frontend
Fix#52
