Notably when the main table is required, but also on rare conditions
when another table would be selected because of the interval selection.
This is not perfect as sometimes, we won't have the data.
Fix#605
All MergeTree tables are now replicated.
For some tables, a `_local` variant is added and the non-`_local`
variant is now distributed. The distributed tables are the `flows`
table, the `flows_DDDD` tables (where `DDDD` is a duration), as well as
the `flows_raw_errors` table. The `exporters` table is not distributed
and stays local.
The data is following this schema:
- data is coming from `flows_HHHH_raw` table, using the Kafka engine
- the `flows_HHHH_raw_consumer` reads data from `flows_HHHH_raw` (local)
and sends it to `flows` (distributed) when there is no error
- the `flows_raw_errors_consumer` reads data from
`flows_HHHH_raw` (local) and sends it to
`flows_raw_errors` (distributed)
- the `flows_DDDD_consumer` reads fata from `flows_local` (local) and
sends it to `flow_DDDD_local` (local)
- the `exporters_consumer` reads data from `flows` (distributed) and
sends it to `exporters` (local)
The reason for `flows_HHHH_raw_consumer` to send data to the distributed
`flows` table, and not the local one is to ensure flows are
balanced (for example, if there is not enough Kafka partitions). But
sending it to `flows_local` would have been possible.
On the other hand, it is important for `flows_DDDD_consumer` to read
from local to avoid duplication. It could have sent to distributed, but
the data is now balanced correctly and we just send it to local instead
for better performance.
The `exporters_consumer` is allowed to read from the distributed `flows`
table because it writes the result to the local `exporters` table.
This is useful to detect interfaces that are close to saturation
quickly. It would usually require to group by exporter name and
interface name and it may not make sense for some graph types (like
stacked 100%). It is useful with Lines and Grid.
This is a first step to make it accept configuration. Most of the
changes are quite trivial, but I also ran into some difficulties with
query columns and filters. They need the schema for parsing, but parsing
happens before dependencies are instantiated (and even if it was not the
case, parsing is stateless). Therefore, I have added a `Validate()`
method that must be called after instantiation. Various bits `panic()`
if not validated to ensure we catch all cases.
The alternative to make the component manages a global state would have
been simpler but it would break once we add the ability to add or
disable columns.
When requesting a too small resolution for data that are too far in the
past, prefer to select a table with data than a table without.
Previously, the resolution was a hard requirement.
But not using it as some linters are either plain incorrect (the one
suggesting to not use nil for `c.t.Context()`) or just
debatable (checking for err value is a good practice, but there are
good reasons to opt out in some cases).
This displays a line for the previous period on stacked graphs.
Previous period depends on the current period. It could be hour, day,
week, month, or year.
This is needed if we want to be able to mix use of several tables
inside a single query (for example, flows_1m0s for a part of the query
and flows_5m0s for another part to overlay historical data).
Also, the way we handle time buckets is now cleaner. The previous way
had two stages of rounding and was incorrect. We were discarding the
first and last value for this reason. The new way only has one stage
of rounding and is correct. It tries hard to align the buckets at the
specified start time. We don't need to discard these values anymore.
We still discard the last one because it could be incomplete (when end
is "now").
