Expand DocC: config-type reference, rebalances/events, error handling, observability, and migration#248
Expand DocC: config-type reference, rebalances/events, error handling, observability, and migration#248ssikka100 wants to merge 3 commits into
Conversation
Add documentation-extension files providing abstracts and overviews for KafkaConfig, KafkaConsumerConfig, and KafkaProducerConfig, which are generated from .gyb templates and had no type-level doc comments. Documents them without touching the generated sources.
Add an 'Observe rebalances' section to ConsumingMessages demonstrating makeConsumerWithEvents and iterating KafkaConsumerEvents (previously the events sequence was only linked, never shown), and curate the events entry point in Topics.
Add three standalone articles: HandlingErrors (where KafkaError surfaces and how to classify it), Observability (enabling swift-metrics gauges and the structured logging metadata the client attaches), and Migrating (moving off the deprecated Configuration types and commitSync). Link them from the landing page.
| @@ -0,0 +1,51 @@ | |||
| # Migrating to the current API | |||
There was a problem hiding this comment.
I am not sure if we need this in the docs once we tag the real 1.0.0 release or if this would be better put into the release notes
|
@heckj Could you take a look at this as well? |
| @@ -0,0 +1,7 @@ | |||
| # ``Kafka/KafkaConsumerConfig`` | |||
|
|
|||
| Configures a ``KafkaConsumer``. | |||
There was a problem hiding this comment.
We generally don't want symbols (or code-voice - monospaced content) in the abstract for a symbol, because it's populated in lots of places - some of which can't or don't support showing symbol links (navigation indices in particular). I'd recommend keeping the abstract as narrative prose.
| Configures a ``KafkaConsumer``. | |
| Configures a Kafka consumer. |
| @@ -0,0 +1,7 @@ | |||
| # ``Kafka/KafkaProducerConfig`` | |||
|
|
|||
| Configures a ``KafkaProducer``. | |||
There was a problem hiding this comment.
We generally don't want symbols (or code-voice - monospaced content) in the abstract for a symbol, because it's populated in lots of places - some of which can't or don't support showing symbol links (navigation indices in particular). I'd recommend keeping the abstract as narrative prose.
| Configures a ``KafkaProducer``. | |
| Configures a Kafka producer. |
| @@ -0,0 +1,62 @@ | |||
| # Handling errors | |||
|
|
|||
| Learn where Kafka errors surface and how to classify them to decide whether to retry, recreate the client, or fail. | |||
There was a problem hiding this comment.
What would you think to making this more active:
| Learn where Kafka errors surface and how to classify them to decide whether to retry, recreate the client, or fail. | |
| Find where Kafka errors surface and classify them to decide whether to retry, recreate the client, or fail. |
|
|
||
| The gauges you assign are ordinary [swift-metrics](https://github.com/apple/swift-metrics) types, so the values reach whatever metrics backend you bootstrap through `MetricsSystem`. | ||
|
|
||
| ## Structured logging |
There was a problem hiding this comment.
| ## Structured logging | |
| ## Emit structured logs |
(this keeps the headings parallel - both imperative vs. noun phrases)
| @@ -0,0 +1,66 @@ | |||
| # Observability | |||
There was a problem hiding this comment.
All the sibling articles are gerunds (Producing... Consuming... etc) - so what do you think about:
| # Observability | |
| # Observing Kafka clients |
(observing may be too overloaded in the context of queues here, so this is a weak suggestion - Observability is pretty well understood - more it just visually stood out as awkward when I scanned it with it's neighbors)
| let consumer = try KafkaConsumer(config: config, logger: logger) | ||
| ``` | ||
|
|
||
| Metrics are emitted only when ``KafkaConfiguration/ConsumerMetrics/updateInterval`` is set **and** at least one gauge is assigned; otherwise the client skips statistics collection entirely. The producer exposes the same pattern through ``KafkaProducerConfig/metrics`` with producer-specific gauges such as ``KafkaConfiguration/ProducerMetrics/queuedProducerMessages`` and ``KafkaConfiguration/ProducerMetrics/totalKafkaBrokerMessagesSent``. |
There was a problem hiding this comment.
| Metrics are emitted only when ``KafkaConfiguration/ConsumerMetrics/updateInterval`` is set **and** at least one gauge is assigned; otherwise the client skips statistics collection entirely. The producer exposes the same pattern through ``KafkaProducerConfig/metrics`` with producer-specific gauges such as ``KafkaConfiguration/ProducerMetrics/queuedProducerMessages`` and ``KafkaConfiguration/ProducerMetrics/totalKafkaBrokerMessagesSent``. | |
| The client emits metrics only when ``KafkaConfiguration/ConsumerMetrics/updateInterval`` is set **and** at least one gauge is assigned; otherwise the client skips statistics collection entirely. The producer exposes the same pattern through ``KafkaProducerConfig/metrics`` with producer-specific gauges such as ``KafkaConfiguration/ProducerMetrics/queuedProducerMessages`` and ``KafkaConfiguration/ProducerMetrics/totalKafkaBrokerMessagesSent``. |
(passive voice -> active voice)
|
|
||
| ## Emit metrics | ||
|
|
||
| The client periodically samples librdkafka's internal statistics and records them into [swift-metrics](https://github.com/apple/swift-metrics) gauges that you supply. Configure this through the ``KafkaConsumerConfig/metrics`` (or ``KafkaProducerConfig/metrics``) property: set an update interval and assign a `Gauge` to each statistic you want to track. |
There was a problem hiding this comment.
| The client periodically samples librdkafka's internal statistics and records them into [swift-metrics](https://github.com/apple/swift-metrics) gauges that you supply. Configure this through the ``KafkaConsumerConfig/metrics`` (or ``KafkaProducerConfig/metrics``) property: set an update interval and assign a `Gauge` to each statistic you want to track. | |
| The client periodically samples internal statistics and records them into [swift-metrics](https://github.com/apple/swift-metrics) gauges that you supply. Configure this through the ``KafkaConsumerConfig/metrics`` (or ``KafkaProducerConfig/metrics``) property: set an update interval and assign a `Gauge` to each statistic you want to track. |
Is it OK if we drop the specifics of librdkafka here - or is it meaningful that it's specifically that library? It adds in a heavy jargon bit here that's technically accurate, but kind of reads like an exposed implementation detail that doesn't benefit the reader to me.
Motivation
The DocC catalog added in #240 covers the core produce/consume/secure flow well, but several capabilities the client actually ships had no narrative documentation, and the three generated configuration types had no type-level abstracts.
Modifications
KafkaConfig,KafkaConsumerConfig, andKafkaProducerConfigConsumingMessagesdemonstratesmakeConsumerWithEvents(config:logger:)and iteratingKafkaConsumerEvents(previously the events sequence was only linked, never shown), covering assign/revoke/error and cooperative vs eager assignment.KafkaErrorsurfaces (thrown, event sequences, delivery reports) and how to classify it withisFatal/isRetriable/rdKafkaCode.ConsumerMetrics/ProducerMetrics) and the structured logging metadata the client attaches (kafka.client.id,kafka.client.type,kafka.group.id). Delivers the observability integration the landing page advertises.Configurationtypes andcommitSyncto the currentConfigtypes andcommit.Result
The DocC build is warning-free, and the advertised-but-undocumented features (events sequence, error semantics, metrics, logging) now have guides.