As the file may change, even the latest tagged version of flake.nix
won't work to build Akvorado. Instead, rely on an unofficial Git
repository.
Fix#2043
It should be a bit more secure to not install scripts by default and to
allow one to update dependencies with a delay. Also, it is faster. The
downside is that it is not usually shipped with npm, but we can download
it through corepack (which is shipped with node). It also has more
builtin features, including patching packages (but we don't need that
anymore).
Some of the files were quite big:
- asns.csv ~ 3 MB
- index.js ~ 1.5 MB
- *.svg ~ 2 MB
Use a ZIP archive to put them all and embed it. This reduce the binary
size from 89 MB to 82 MB. 🤯
This also pulls some code modernization (use of http.ServeFileFS).
While I was relunctant to let Go download the right toolchain if we
didn't have one, this makes everything simpler. The Go version is now
fully controlled by `go.mod`. It also a nice for people wanting to build
on older distributions.
For Nix, GOTOOLCHAIN is set to local, so we rely on `go_latest` being
up-to-date enough. But they are usually quite fast to update, so it
should be OK.
In addition to the regular linux/amd64. Both "docker image pull" and
"docker compose" will use linux/amd64/v3 if the architecture allows it.
This enables some optimizations at Go level.
I would have wished that Go would have been able to automatically
support fat binaries. This is in discussion, but mostly for different
architectures, not for different microarchitectures. It exists some
naive approach, like https://github.com/CAFxX/mgo/, but it does not play
well with embeds (the ASN database would make the binary image too big).
The Docker approach should be good enough for most people.
There is still room for improvement. For inlet, it would require to know
when Kafka has sent the message (so enabling successes return). For
outlet, it should be possible to reuse the same flow (with a ResetVT
between each use).
This change split the inlet component into a simpler inlet and a new
outlet component. The new inlet component receive flows and put them in
Kafka, unparsed. The outlet component takes them from Kafka and resume
the processing from here (flow parsing, enrichment) and puts them in
ClickHouse.
The main goal is to ensure the inlet does a minimal work to not be late
when processing packets (and restart faster). It also brings some
simplification as the number of knobs to tune everything is reduced: for
inlet, we only need to tune the queue size for UDP, the number of
workers and a few Kafka parameters; for outlet, we need to tune a few
Kafka parameters, the number of workers and a few ClickHouse parameters.
The outlet component features a simple Kafka input component. The core
component becomes just a callback function. There is also a new
ClickHouse component to push data to ClickHouse using the low-level
ch-go library with batch inserts.
This processing has an impact on the internal representation of a
FlowMessage. Previously, it was tailored to dynamically build the
protobuf message to be put in Kafka. Now, it builds the batch request to
be sent to ClickHouse. This makes the FlowMessage structure hides the
content of the next batch request and therefore, it should be reused.
This also changes the way we decode flows as they don't output
FlowMessage anymore, they reuse one that is provided to each worker.
The ClickHouse tables are slightly updated. Instead of using Kafka
engine, the Null engine is used instead.
Fix#1122
They are all devDependencies, except the LSP ones that we move to
optionalDependencies. Also, exclude optionalDependencies from lockfile
when building for Nix as there is a dependency that Nix does not like.
I don't remember why I did this. This is a pointless as the apps do not
contribute to the inputs of the packages. This was added in a6139b4e59
without any explanation...
This PR reuse some of the code from #653. Two major changes:
- add a script to easily update hashs (in the future, this should
include nixpkgs)
- do not duplicate the content of Makefile
This is required if we want to update vue-tsc which is abandoning ES2016
for ES2021. We also use Node 18 by default for our builds. Moreover,
Node 14 is EoL since recently.
Alternatively, we could bundle the CA certificate path (at compile time)
inside Akvorado binary and it would be pulled automatically. This seems
like a hurdle.
