Inserting into ClickHouse should be done in large batches to minimize
the number of parts created. This would require the user to tune the
number of Kafka workers to match a target of around 50k-100k rows. Instead,
we dynamically tune the number of workers depending on the load to reach
this target.
We keep using async if we are too low in number of flows.
It is still possible to do better by consolidating batches from various
workers, but that's something I wanted to avoid.
Also, increase the maximum wait time to 5 seconds. It should be good
enough for most people.
Fix#1885
