depot/third_party/nixpkgs/nixos/modules/services/misc/kafka.md

Apache Kafka

Apache Kafka is an open-source distributed event streaming platform

Basic Usage

The Apache Kafka service is configured almost exclusively through its settings option, with each attribute corresponding to the upstream configuration manual broker settings.

KRaft

Unlike in Zookeeper mode, Kafka in KRaft mode requires each log dir to be "formatted" (which means a cluster-specific a metadata file must exist in each log dir)

The upstream intention is for users to execute the storage tool to achieve this, but this module contains a few extra options to automate this:

Migrating to settings

Migrating a cluster to the new settings-based changes requires adapting removed options to the corresponding upstream settings.

This means that the upstream Broker Configs documentation should be followed closely.

Note that dotted options in the upstream docs do not correspond to nested Nix attrsets, but instead as quoted top level settings attributes, as in services.apache-kafka.settings."broker.id", NOT services.apache-kafka.settings.broker.id.

Care should be taken, especially when migrating clusters from the old module, to ensure that the same intended configuration is reproduced faithfully via settings.

To assist in the comparison, the final config can be inspected by building the config file itself, ie. with: nix-build <nixpkgs/nixos> -A config.services.apache-kafka.configFiles.serverProperties.

Notable changes to be aware of include:

• Removal of services.apache-kafka.extraProperties and services.apache-kafka.serverProperties

  • Translate using arbitrary properties using
  • Upstream docs
  • The intention is for all broker properties to be fully representable via .
  • If this is not the case, please do consider raising an issue.
  • Until it can be remedied, you can bail out by using to the path of a fully rendered properties file.

• Removal of services.apache-kafka.hostname and services.apache-kafka.port

  • Translate using: services.apache-kafka.settings.listeners
  • Upstream docs

• Removal of services.apache-kafka.logDirs

  • Translate using: services.apache-kafka.settings."log.dirs"
  • Upstream docs

• Removal of services.apache-kafka.brokerId

  • Translate using: services.apache-kafka.settings."broker.id"
  • Upstream docs

• Removal of services.apache-kafka.zookeeper

  • Translate using: services.apache-kafka.settings."zookeeper.connect"
  • Upstream docs