Remora Versions Save

Breaking changes

• Change introduced in the format of metrics exported to AWS CloudWatch.
• Moved team name (ownership) to 'Buffalo' in Docker / Pierone. Images are now published to registry-write.opensource.zalan.do/buffalo instead of registry-write.opensource.zalan.do/machina.

Other Changes

• Added support for Kafka 3.1.0.
• Upgraded Docker base image to use Java 17.
• Upgraded all dependencies caused by migration from SBT (v0.13 to v1.6), and Scala (v2.11 to v2.13).

What's Changed

• Allow to override root log level by @dtrsan in https://github.com/zalando-incubator/remora/pull/81
• Allow configuration of AdminClient in KafkaSettings by @soceanainn in https://github.com/zalando-incubator/remora/pull/83
• Add cloudwatch metric name whitelisting feature [Fixes #71] by @jaelee-branch in https://github.com/zalando-incubator/remora/pull/85
• Refactor CloudWatch reporting by adding dedicated reporter class by @soceanainn in https://github.com/zalando-incubator/remora/pull/92

Full Changelog: https://github.com/zalando-incubator/remora/compare/v1.2.0...v1.3.0

• #78 - Kafka client 2.2.1

There will be a breaking change for the payload returned on the healthcheck. Please see the README

Issues

• 45 - HTTP Gereral Counters
• 49 - Assembly errors due to dependency issues
• Upgrade to use kafka 1.1.1

See readme for datadog integration.

Works with Kafka 1.0.0

The API need to change, the kafka clients are not backwards compatible and so it was best to just increase and fix up the code.

Show Active Consumers

http://localhost:9000/consumers

which gives

["consumer-1", "consumer-2", "consumer-3"]

Show specific consumer info

curl http://localhost:9000/consumers/<ConsumerGroupId>

State can be "Empty", "Dead", "Stable", "PreparingRebalance", "AwaitingSync"

{  
   "state":"Empty",
   "partition_assignment":[  
      {  
         "group":"console-consumer-20891",
         "coordinator":{  
            "id":0,
            "id_string":"0",
            "host":"foo.company.com",
            "port":9092
         },
         "topic":"products-in",
         "partition":1,
         "offset":3,
         "lag":0,
         "consumer_id":"-",
         "host":"-",
         "client_id":"-",
         "log_end_offset":3
      },
      {  
         "group":"console-consumer-20891",
         "coordinator":{  
            "id":0,
            "id_string":"0",
            "host":"foo.company.com",
            "port":9092
         },
         "topic":"products-in",
         "partition":0,
         "offset":3,
         "lag":0,
         "consumer_id":"consumer-1-7baba9b9-0ec3-4241-9433-f36255dd4708",
         "host":"/10.170.2.103",
         "client_id":"consumer-1",
         "log_end_offset":3
      }
   ]
}

basic application showing consumer offsets stored in kafka. Tested with kafka 0.10.0.1

Show Active Consumers

http://localhost:9000/consumers

which gives

[
    {
      "protocolType": "consumer",
      "groupId": "consumer-1"
    }
    {
      "protocolType": "consumer",
      "groupId": "consumer-2"
    }
    {
      "protocolType": "consumer",
      "groupId": "consumer-3"
    }
]

Show specific consumer info

curl http://localhost:9000/consumers/<ConsumerGroupId>

[
    {
      "owner": "consumer-2_/132.34.134.12",
      "lag": 155758,
      "log_end_offset": 2580124,
      "offset": 2424366,
      "partition": 1,
      "topic": "foobar",
      "group": "consumer-1"
    }
    {
      "owner": "consumer-2_/132.34.134.12",
      "lag": 155758,
      "log_end_offset": 2580124,
      "offset": 2424366,
      "partition": 2,
      "topic": "foobar",
      "group": "consumer-1"
    }
    {
      "owner": "consumer-2_/132.34.134.12",
      "lag": 155758,
      "log_end_offset": 2580124,
      "offset": 2424366,
      "partition": 3,
      "topic": "foobar",
      "group": "consumer-1"
    }
]