This is a first step to make it accept configuration. Most of the
changes are quite trivial, but I also ran into some difficulties with
query columns and filters. They need the schema for parsing, but parsing
happens before dependencies are instantiated (and even if it was not the
case, parsing is stateless). Therefore, I have added a `Validate()`
method that must be called after instantiation. Various bits `panic()`
if not validated to ensure we catch all cases.
The alternative to make the component manages a global state would have
been simpler but it would break once we add the ability to add or
disable columns.
This is a huge change to make the various subcomponents of the inlet use
the schema to generate the protobuf. For it to make sense, we also
modify the way we parse flows to directly serialize non-essential fields
to Protobuf.
The performance is mostly on par with the previous commit. We are a bit
less efficient because we don't have a fixed structure, but we avoid
loosing too much performance by not relying on reflection and keeping
the production of messages as code. We use less of Goflow2: raw flow
parsing is still done by Goflow2, but we don't use the producer part
anymore. This helps a bit with the performance as we parse less.
Overall, we are 20% than the previous commit and twice faster than the
1.6.4!
```
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
BenchmarkDecodeEncodeNetflow
BenchmarkDecodeEncodeNetflow/with_encoding
BenchmarkDecodeEncodeNetflow/with_encoding-12 151484 7789 ns/op 8272 B/op 143 allocs/op
BenchmarkDecodeEncodeNetflow/without_encoding
BenchmarkDecodeEncodeNetflow/without_encoding-12 162550 7133 ns/op 8272 B/op 143 allocs/op
BenchmarkDecodeEncodeSflow
BenchmarkDecodeEncodeSflow/with_encoding
BenchmarkDecodeEncodeSflow/with_encoding-12 94844 13193 ns/op 9816 B/op 295 allocs/op
BenchmarkDecodeEncodeSflow/without_encoding
BenchmarkDecodeEncodeSflow/without_encoding-12 92569 12456 ns/op 9816 B/op 295 allocs/op
```
There was a tentative to parse sFlow packets with gopackets, but the
adhoc parser used here is more performant.
This will serve as a base for converting this to a one-step conversion
to Protobuf. The main goal is not to be faster, but we don't want to be
slower and faster would be a nice bonus.
```
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
BenchmarkDecodeEncodeNetflow
BenchmarkDecodeEncodeNetflow-12 39586 29199 ns/op
BenchmarkDecodeEncodeSflow
BenchmarkDecodeEncodeSflow-12 24349 48914 ns/op
ok akvorado/inlet/flow 3.167s
DONE 0 tests in 3.636s
```
This is a bit less type-safe. We could keep type safety by redefining
all the consts in `query_consts.go` in `common/schema`, but this is
pointless as the goal is to have arbitrary dimensions at some point.
We introduce an leaky abstraction for flows schema and use it for
migrations as a first step.
For views and dictionaries, we stop relying on a hash to know if they
need to be recreated, but we compare the select statements with our
target statement. This is a bit fragile, but strictly better than the
hash.
For data tables, we add the missing columns.
We give up on the abstraction of a migration step and just rely on
helper functions to get the same result. The migration code is now
shorter and we don't need to update it when adding new columns.
This is a preparatory work for #211 to allow a user to specify
additional fields to collect.
In our case, this shouldn't matter. However, performance hit should be
low and maybe at some point this middleware could be used for more
sensitive stuff.
validate is only able to validate non-struct types (or recurse inside
struct). So, if we want to use "required" on some of them, we need a
custom type.
Fix#263
RIB updates are handled by a single goroutine accepting update requests
through a channel receiving functions to execute on the state (RIB +
peer state).
For lookups, we have 3 options (better to lower performance, higher to
lower memory usage):
1. have a read-only copy updated "atomically" at regular interval,
2. have a read-only copy updated behind a lock at regular interval,
3. handle lookups by the worker through a high priority channel.
This commit implements option 3. It may be a regression in latency
compared to the previous design because long updates (flushing peers)
may prevent answering lookup requests. This will be addressed in the
next commit.
This could help performance as we will skip removing a prefix if we
don't have the associated NLRI. However, this is an unlikely corner
case (all routes we have should have been added first).
When the RIB is locked for too long, inlet is hung. Try to ensure give a
bit of time for the inlet to move forward between two flush of the RIB.
There are various knobs not documnted yet until we get better defaults:
- `inlet.bmp.peer-removal-max-time`: how long to keep the lock
- `inlet.bmp.peer-removal-sleep-interval`: how long to sleep between two
runs if we were unable to flush the whole peer
- `inlet.bmp.peer-removal-max-queue`: maximum number of flush requests
- `inlet.bmp.peer-removal-min-routes`: minimum number of routes to flush
before yielding
May fix#253
At first, there was a tentative to use BMP collector implementation
from bio-rd. However, this current implementation is using GoBGP
instead:
- BMP is very simple from a protocol point of view. The hard work is
mostly around decoding. Both bio-rd and GoBGP can decode, but for
testing, GoBGP is able to generate messages as well (this is its
primary purpose, I suppose parsing was done for testing purpose).
Using only one library is always better. An alternative would be
GoBMP, but it also only do parsing.
- Logging and metrics can be customized easily (but the work was done
for bio-rd, so not a real argument).
- bio-rd is an application and there is no API stability (and I did
that too)
- GoBGP supports FlowSpec, which may be useful in the future for the
DDoS part. Again, one library for everything is better (but
honestly, GoBGP as a lib is not the best part of it, maybe
github.com/jwhited/corebgp would be a better fit while keeping GoBGP
for decoding/encoding).
There was a huge effort around having a RIB which is efficient
memory-wise (data are interned to save memory), performant during
reads, while being decent during insertions. We rely on a patched
version of Kentik's Patricia trees to be able to apply mutations to
the tree.
There was several tentatives to implement some kind of graceful
restart, but ultimetaly, the design is kept simple: when a BMP
connection goes down, routes will be removed after a configurable
time. If the connection comes back up, then it is just considered new.
It would have been ideal to rely on EoR markers, but the RFC is
unclear about them, and they are likely to be per peer, making it
difficult to know what to do if one peer is back, but not the other.
Remaining tasks:
- [ ] Confirm support for LocRIB
- [ ] Import data in ClickHouse
- [ ] Make data available in the frontend
Fix#52
Raw data files can be converted with Scapy:
```python
from scapy.all import *
wrpcap("data-1140.pcap",
Ether(src="00:53:00:11:22:33",dst="00:53:00:44:55:66")/
IP(src="192.0.2.100", dst="192.0.2.101")/
UDP(sport=47873,dport=6343)/
open("data-1140.data", "rb").read())
```
While there is more helpful information in a panic, this is confusing
to the user. With the amount of code using reflection, it seems better
to have clearer messages to help the user find the faulty section if
any.
For each case, we test from native map and from YAML. This should
capture all the cases we are interested.
Also, simplify pretty diff by using stringer for everything. I don't
remember why this wasn't the case. Maybe IP addresses? It's possible
to opt out by overriding formatters.
