The goal is to be able to reenable full dead code elimination that is
prevented by non-constant calls to reflect.MethodByName(). There are
many other culprits for this:
```
08:04 ❱ go build -o /dev/null -tags release,grpcnotrace -ldflags=-dumpdep . |& go run github.com/aarzilli/whydeadcode@latest
github.com/go-playground/validator/v10.tryCallValidateFn reachable from:
github.com/go-playground/validator/v10.isValidateFn
github.com/go-playground/validator/v10.isValidateFn·f
github.com/go-playground/validator/v10.map.init.11
github.com/go-playground/validator/v10.bakedInValidators
github.com/go-playground/validator/v10.New
akvorado/common/helpers.init.1
akvorado/common/helpers..inittask
go:main.inittasks
_
github.com/expr-lang/expr/builtin.get reachable from:
github.com/expr-lang/expr/builtin.get·f
github.com/expr-lang/expr/builtin..stmp_53
github.com/expr-lang/expr/builtin..stmp_0
github.com/expr-lang/expr/builtin.init
github.com/expr-lang/expr/builtin..inittask
go:main.inittasks
_
github.com/expr-lang/expr/checker/nature.Nature.All reachable from:
type:github.com/expr-lang/expr/checker/nature.Nature
type:func() github.com/expr-lang/expr/checker/nature.Nature
type:github.com/expr-lang/expr/ast.Node
type:github.com/expr-lang/expr/vm.Program
type:*github.com/expr-lang/expr/vm.Program
type:akvorado/outlet/core.ExporterClassifierRule
type:[]akvorado/outlet/core.ExporterClassifierRule
type:akvorado/outlet/core.Configuration
type:akvorado/cmd.OutletConfiguration
type:[]akvorado/cmd.OutletConfiguration
type:akvorado/cmd.OrchestratorConfiguration
type:*akvorado/cmd.OrchestratorConfiguration
internal/abi..dict.TypeFor[akvorado/cmd.OrchestratorConfiguration]
reflect..dict.TypeFor[akvorado/cmd.OrchestratorConfiguration]
akvorado/cmd.init.6.orchestratorClickHouseMigrationHook.func2
akvorado/cmd.init.6.orchestratorClickHouseMigrationHook.func2·f
akvorado/cmd.init.6
akvorado/cmd..inittask
go:main.inittasks
_
text/template.(*state).evalField reachable from:
text/template.(*state).evalFieldChain
text/template.(*state).evalCommand
text/template.(*state).evalPipeline
text/template.(*state).walk
text/template.(*Template).execute
akvorado/console/authentication.(*Component).UserAuthentication.func1
akvorado/console/authentication.(*Component).UserAuthentication
type:*akvorado/console/authentication.Component
akvorado/cmd.consoleStart
akvorado/cmd.init.func3
akvorado/cmd.init.func3·f
akvorado/cmd..stmp_2
akvorado/cmd.init
akvorado/cmd..inittask
go:main.inittasks
_
gorm.io/gorm/schema.ParseWithSpecialTableName reachable from:
gorm.io/gorm.(*DB).assignInterfacesToValue
gorm.io/gorm.(*DB).FirstOrInit
type:*gorm.io/gorm.DB
type:akvorado/console/database.Component
akvorado/cmd.consoleStart
akvorado/cmd.init.func3
akvorado/cmd.init.func3·f
akvorado/cmd..stmp_2
akvorado/cmd.init
akvorado/cmd..inittask
go:main.inittasks
_
gorm.io/gorm/schema.ParseWithSpecialTableName reachable from:
gorm.io/gorm.(*DB).assignInterfacesToValue
gorm.io/gorm.(*DB).FirstOrInit
type:*gorm.io/gorm.DB
type:akvorado/console/database.Component
type:*akvorado/console/database.Component
akvorado/cmd.consoleStart
akvorado/cmd.init.func3
akvorado/cmd.init.func3·f
akvorado/cmd..stmp_2
akvorado/cmd.init
akvorado/cmd..inittask
go:main.inittasks
_
github.com/expr-lang/expr/vm/runtime.FetchMethod reachable from:
github.com/expr-lang/expr/vm.(*VM).Run
github.com/expr-lang/expr/vm.Run
akvorado/outlet/core.(*ExporterClassifierRule).exec
akvorado/outlet/core.(*Component).classifyExporter
akvorado/outlet/core.(*worker).enrichFlow
akvorado/outlet/core.(*worker).processIncomingFlow.func1
akvorado/outlet/core.(*worker).processIncomingFlow
akvorado/outlet/core.(*worker).processIncomingFlow-fm
akvorado/outlet/core.(*Component).newWorker
akvorado/outlet/core.(*Component).newWorker-fm
akvorado/outlet/core.(*Component).Start
type:*akvorado/outlet/core.Component
akvorado/cmd.outletStart
akvorado/cmd.init.func7
akvorado/cmd.init.func7·f
akvorado/cmd..stmp_6
akvorado/cmd.init
akvorado/cmd..inittask
go:main.inittasks
_
```
For gorm, we have https://github.com/go-gorm/gorm/pull/7643. We could
replace text/template by something else as our usage is quite light. For
validator, it may be addressable upstream by disallowing calls to
configurable validator function through a tag. For expr, it looks more
complex.
Then, regression should be detected by CI.
`make docker` builds the context from a tarball. We don't use git
archive as we want to allow local modifications without committing them.
This is not unlike nix work.
For the other Docker files, use a tailored `.dockerignore`:
- dev, ignore everything except bin/akvorado
- ipinfo-geoipupdate, ignore everything except the update script
- nix, let nix handle it from the Git repository
By default, the 5-tuple is used to load balance flows. Exporters with
many flows are bound to a specific worker. Use eBPF to do a per-packet
load-balancing.
Currently, this is done randomly, but we will use a percpu counter in
the next commit. This will make the test easier too, maybe?
This should also enable graceful restart but not with the current
Docker Compose setup, we would need to use mode host or spawn a new one
in the same network namespace than the old one. This does not look like
very complex:
- spawn a new inlet in the same network namespace, but listening to a
different HTTP port
- stop the previous inlet
- spawn a new inlet in the same network namespace
- stop the previous inlet
Alternatively, we could use SO_REUSEPORT for the HTTP socket too!
It should be a bit more secure to not install scripts by default and to
allow one to update dependencies with a delay. Also, it is faster. The
downside is that it is not usually shipped with npm, but we can download
it through corepack (which is shipped with node). It also has more
builtin features, including patching packages (but we don't need that
anymore).
Some of the files were quite big:
- asns.csv ~ 3 MB
- index.js ~ 1.5 MB
- *.svg ~ 2 MB
Use a ZIP archive to put them all and embed it. This reduce the binary
size from 89 MB to 82 MB. 🤯
This also pulls some code modernization (use of http.ServeFileFS).
While I was relunctant to let Go download the right toolchain if we
didn't have one, this makes everything simpler. The Go version is now
fully controlled by `go.mod`. It also a nice for people wanting to build
on older distributions.
For Nix, GOTOOLCHAIN is set to local, so we rely on `go_latest` being
up-to-date enough. But they are usually quite fast to update, so it
should be OK.
gocov and gocovxml are unmaintained. There is
https://github.com/boumenot/gocover-cobertura which is linked from
Gitlab, but it is missing some lines during the conversion (code defined
in callbacks called from var, see hellogopher as an example), so it is
not reliable.
And cache it with Docker. This way, it only needs to be built once. And
remove the all_js target. There is another method when working on JS in
CONTRIBUTING.md.
This should be faster to build multiarch images, as Nix requires
emulation, while we can build everything natively using builtin Go
cross-compilation support. Moreover, caching should be better.
Also, add Nix build to CI.
Also, build more architectures (linux/arm/v7).
There is still room for improvement. For inlet, it would require to know
when Kafka has sent the message (so enabling successes return). For
outlet, it should be possible to reuse the same flow (with a ResetVT
between each use).
This change split the inlet component into a simpler inlet and a new
outlet component. The new inlet component receive flows and put them in
Kafka, unparsed. The outlet component takes them from Kafka and resume
the processing from here (flow parsing, enrichment) and puts them in
ClickHouse.
The main goal is to ensure the inlet does a minimal work to not be late
when processing packets (and restart faster). It also brings some
simplification as the number of knobs to tune everything is reduced: for
inlet, we only need to tune the queue size for UDP, the number of
workers and a few Kafka parameters; for outlet, we need to tune a few
Kafka parameters, the number of workers and a few ClickHouse parameters.
The outlet component features a simple Kafka input component. The core
component becomes just a callback function. There is also a new
ClickHouse component to push data to ClickHouse using the low-level
ch-go library with batch inserts.
This processing has an impact on the internal representation of a
FlowMessage. Previously, it was tailored to dynamically build the
protobuf message to be put in Kafka. Now, it builds the batch request to
be sent to ClickHouse. This makes the FlowMessage structure hides the
content of the next batch request and therefore, it should be reused.
This also changes the way we decode flows as they don't output
FlowMessage anymore, they reuse one that is provided to each worker.
The ClickHouse tables are slightly updated. Instead of using Kafka
engine, the Null engine is used instead.
Fix#1122
