Skip to content

Expand DocC: config-type reference, rebalances/events, error handling, observability, and migration#248

Open
ssikka100 wants to merge 3 commits into
swift-server:mainfrom
ssikka100:docc-current-client
Open

Expand DocC: config-type reference, rebalances/events, error handling, observability, and migration#248
ssikka100 wants to merge 3 commits into
swift-server:mainfrom
ssikka100:docc-current-client

Conversation

@ssikka100

@ssikka100 ssikka100 commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Motivation

The DocC catalog added in #240 covers the core produce/consume/secure flow well, but several capabilities the client actually ships had no narrative documentation, and the three generated configuration types had no type-level abstracts.

Modifications

  • Configuration type reference — DocC extension files give abstracts/overviews to KafkaConfig, KafkaConsumerConfig, and KafkaProducerConfig
  • Consumer rebalances + events — a new section in ConsumingMessages demonstrates makeConsumerWithEvents(config:logger:) and iterating KafkaConsumerEvents (previously the events sequence was only linked, never shown), covering assign/revoke/error and cooperative vs eager assignment.
  • Error handling — a new article on where KafkaError surfaces (thrown, event sequences, delivery reports) and how to classify it with isFatal / isRetriable / rdKafkaCode.
  • Observability — a new article covering swift-metrics gauges (ConsumerMetrics / ProducerMetrics) and the structured logging metadata the client attaches (kafka.client.id, kafka.client.type, kafka.group.id). Delivers the observability integration the landing page advertises.
  • Migration guide — a new article mapping the deprecated Configuration types and commitSync to the current Config types and commit.

Result

The DocC build is warning-free, and the advertised-but-undocumented features (events sequence, error semantics, metrics, logging) now have guides.

Add documentation-extension files providing abstracts and overviews for KafkaConfig, KafkaConsumerConfig, and KafkaProducerConfig, which are generated from .gyb templates and had no type-level doc comments. Documents them without touching the generated sources.
Add an 'Observe rebalances' section to ConsumingMessages demonstrating makeConsumerWithEvents and iterating KafkaConsumerEvents (previously the events sequence was only linked, never shown), and curate the events entry point in Topics.
Add three standalone articles: HandlingErrors (where KafkaError surfaces and how to classify it), Observability (enabling swift-metrics gauges and the structured logging metadata the client attaches), and Migrating (moving off the deprecated Configuration types and commitSync). Link them from the landing page.
@ssikka100 ssikka100 added the semver/none No version bump required. label Jul 11, 2026
@ssikka100 ssikka100 requested a review from FranzBusch July 13, 2026 13:52
@@ -0,0 +1,51 @@
# Migrating to the current API

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if we need this in the docs once we tag the real 1.0.0 release or if this would be better put into the release notes

@FranzBusch FranzBusch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM

@FranzBusch

Copy link
Copy Markdown
Contributor

@heckj Could you take a look at this as well?

@@ -0,0 +1,7 @@
# ``Kafka/KafkaConsumerConfig``

Configures a ``KafkaConsumer``.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We generally don't want symbols (or code-voice - monospaced content) in the abstract for a symbol, because it's populated in lots of places - some of which can't or don't support showing symbol links (navigation indices in particular). I'd recommend keeping the abstract as narrative prose.

Suggested change
Configures a ``KafkaConsumer``.
Configures a Kafka consumer.

@@ -0,0 +1,7 @@
# ``Kafka/KafkaProducerConfig``

Configures a ``KafkaProducer``.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We generally don't want symbols (or code-voice - monospaced content) in the abstract for a symbol, because it's populated in lots of places - some of which can't or don't support showing symbol links (navigation indices in particular). I'd recommend keeping the abstract as narrative prose.

Suggested change
Configures a ``KafkaProducer``.
Configures a Kafka producer.

@@ -0,0 +1,62 @@
# Handling errors

Learn where Kafka errors surface and how to classify them to decide whether to retry, recreate the client, or fail.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would you think to making this more active:

Suggested change
Learn where Kafka errors surface and how to classify them to decide whether to retry, recreate the client, or fail.
Find where Kafka errors surface and classify them to decide whether to retry, recreate the client, or fail.


The gauges you assign are ordinary [swift-metrics](https://github.com/apple/swift-metrics) types, so the values reach whatever metrics backend you bootstrap through `MetricsSystem`.

## Structured logging

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Structured logging
## Emit structured logs

(this keeps the headings parallel - both imperative vs. noun phrases)

@@ -0,0 +1,66 @@
# Observability

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the sibling articles are gerunds (Producing... Consuming... etc) - so what do you think about:

Suggested change
# Observability
# Observing Kafka clients

(observing may be too overloaded in the context of queues here, so this is a weak suggestion - Observability is pretty well understood - more it just visually stood out as awkward when I scanned it with it's neighbors)

let consumer = try KafkaConsumer(config: config, logger: logger)
```

Metrics are emitted only when ``KafkaConfiguration/ConsumerMetrics/updateInterval`` is set **and** at least one gauge is assigned; otherwise the client skips statistics collection entirely. The producer exposes the same pattern through ``KafkaProducerConfig/metrics`` with producer-specific gauges such as ``KafkaConfiguration/ProducerMetrics/queuedProducerMessages`` and ``KafkaConfiguration/ProducerMetrics/totalKafkaBrokerMessagesSent``.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Metrics are emitted only when ``KafkaConfiguration/ConsumerMetrics/updateInterval`` is set **and** at least one gauge is assigned; otherwise the client skips statistics collection entirely. The producer exposes the same pattern through ``KafkaProducerConfig/metrics`` with producer-specific gauges such as ``KafkaConfiguration/ProducerMetrics/queuedProducerMessages`` and ``KafkaConfiguration/ProducerMetrics/totalKafkaBrokerMessagesSent``.
The client emits metrics only when ``KafkaConfiguration/ConsumerMetrics/updateInterval`` is set **and** at least one gauge is assigned; otherwise the client skips statistics collection entirely. The producer exposes the same pattern through ``KafkaProducerConfig/metrics`` with producer-specific gauges such as ``KafkaConfiguration/ProducerMetrics/queuedProducerMessages`` and ``KafkaConfiguration/ProducerMetrics/totalKafkaBrokerMessagesSent``.

(passive voice -> active voice)


## Emit metrics

The client periodically samples librdkafka's internal statistics and records them into [swift-metrics](https://github.com/apple/swift-metrics) gauges that you supply. Configure this through the ``KafkaConsumerConfig/metrics`` (or ``KafkaProducerConfig/metrics``) property: set an update interval and assign a `Gauge` to each statistic you want to track.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The client periodically samples librdkafka's internal statistics and records them into [swift-metrics](https://github.com/apple/swift-metrics) gauges that you supply. Configure this through the ``KafkaConsumerConfig/metrics`` (or ``KafkaProducerConfig/metrics``) property: set an update interval and assign a `Gauge` to each statistic you want to track.
The client periodically samples internal statistics and records them into [swift-metrics](https://github.com/apple/swift-metrics) gauges that you supply. Configure this through the ``KafkaConsumerConfig/metrics`` (or ``KafkaProducerConfig/metrics``) property: set an update interval and assign a `Gauge` to each statistic you want to track.

Is it OK if we drop the specifics of librdkafka here - or is it meaningful that it's specifically that library? It adds in a heavy jargon bit here that's technically accurate, but kind of reads like an exposed implementation detail that doesn't benefit the reader to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver/none No version bump required.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants