Kafka REST Client: avoid clashes between clients

When multiple receivers are listening to different topics
they need to avoid clashing by using different group ids.

With REST Client API based receivers, two receivers on the
same group would deadlock with each other.

Also set explicitly the auto-commit and consumer.request.timeout.ms,
the receive loops are blocking in the Kafka REST Proxy.

Staying in a receive loop too much is risky, because if the Gerrit
receiver becomes unresponsive or disconnects for whatever reason,
the receiver would be considered as failed and would start a
5 mins rebalancing loop.

Change-Id: I774c295276441365b8939e36ffd93fbcd8fd60bc
2 files changed
tree: 71de48fb7b036a334f38ea346ab9a1aaae4c8ba3
  1. src/
  2. BUILD
  3. external_plugin_deps.bzl
  4. Jenkinsfile
  5. LICENSE
  6. README.md
README.md

Kafka: Gerrit event producer for Apache Kafka

Build Status

Synopsis

This plugins allows to define a distributed stream of events published by Gerrit.

Events can be anything, from the traditional stream events to the Gerrit metrics.

This plugin requires Gerrit 2.13 or laster.

Environments

  • linux
  • java-1.8
  • Bazel

Build

Kafka plugin can be build as a regular ‘in-tree’ plugin. That means that is required to clone a Gerrit source tree first and then to have the Kafka plugin source directory into the /plugins path. Additionally, the plugins/external_plugin_deps.bzl file needs to be updated to match the Kafka plugin one.

git clone --recursive https://gerrit.googlesource.com/gerrit
git clone https://gerrit.googlesource.com/plugins/events-kafka gerrit/plugins/events-kafka
cd gerrit
rm plugins/external_plugin_deps.bzl
ln -s ./events-kafka/external_plugin_deps.bzl plugins/.

To build the events-kafka plugins, issue the command from the Gerrit source path:

bazel build plugins/events-kafka

The output is created in

bazel-genfiles/plugins/events-kafka/events-kafka.jar

Minimum Configuration

Assuming a running Kafka broker on the same Gerrit host, add the following settings to gerrit.config:

  [plugin "events-kafka"]
    bootstrapServers = localhost:9092