Support mTLS on the Kafka consumer - #2813
Open
delthas wants to merge 2 commits into
Open
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
@@ Coverage Diff @@
## improvement/BB-848/pin-fromoffset-mongo-processor #2813 +/- ##
=====================================================================================
+ Coverage 75.61% 75.62% +0.01%
=====================================================================================
Files 200 200
Lines 13922 13929 +7
=====================================================================================
+ Hits 10527 10534 +7
Misses 3385 3385
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
delthas
force-pushed
the
improvement/BB-850/support-mtls-kafka-consumer
branch
from
August 19, 2026 09:53
4d5707f to
e5ea0dd
Compare
delthas
force-pushed
the
improvement/BB-850/support-mtls-kafka-consumer
branch
2 times, most recently
from
August 19, 2026 16:24
f69017b to
c8c7f33
Compare
delthas
force-pushed
the
improvement/BB-850/support-mtls-kafka-consumer
branch
from
August 19, 2026 16:58
c8c7f33 to
35fcfd6
Compare
delthas
force-pushed
the
improvement/BB-850/support-mtls-kafka-consumer
branch
2 times, most recently
from
August 19, 2026 17:12
34d3bb0 to
761dcde
Compare
Move the params built inline in _initConsumer into consumerConfig and topicConfig getters, so they can be asserted in a unit test. Mirrors BackbeatProducer, which already exposes producerConfig and topicConfig. The "follower fetching enabled" log stays in _initConsumer, keeping the getters side-effect free as the producer's are. No behaviour change. Issue: BB-850
BackbeatConsumer built its rdkafka params with no way to pass security.protocol or ssl.*, so it could not reach a broker requiring client certificates. Producers already had this: BackbeatProducer spreads producerParams, and kafka.producerParams is an existing config field. Add kafka.consumerParams alongside it with the same shape -- .unknown(true), and the keys BackbeatConsumer sets itself forbidden, so a caller gets a validation error rather than a silently ignored setting. Three keys it does not set are forbidden too: enable.auto.commit and auto.commit.interval.ms would silently disable the periodic offset commits the consumer relies on, and auto.offset.reset is topic scope, so librdkafka accepts it into a default_topic_conf that node-rdkafka then replaces wholesale -- a caller could otherwise believe they had overridden the offset reset a processor pins. The schema is defined once in config.joi and reused by BackbeatConsumer's own joi, so a programmatic caller is validated like a config file. Thread it through every consumer rather than only the one with a remote-broker use case today, mirroring how producerParams rolled out in efc9c6a: one line per call site, so the knob exists wherever a broker might require certificates. BackbeatConsumer's own internal producers -- the canary and the test-only bootstrap -- take producerParams as well. Its kafka block already accepted compressionType and requiredAcks for them, but not producerParams, so the replication queue processor (the one consumer with canary: true) could not have completed its canary against an mTLS broker. librdkafka in node-rdkafka 3.0.1 already reports ssl and sasl in its features, so no dependency change is needed. Issue: BB-850
delthas
force-pushed
the
improvement/BB-850/support-mtls-kafka-consumer
branch
from
August 25, 2026 18:40
761dcde to
3a3db1b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BackbeatConsumerbuilds its rdkafka params with no way to passsecurity.protocolorssl.*, so it cannot reach a broker that requires client certificates. Producers already have this:BackbeatProducerspreadsproducerParams, andkafka.producerParamsis an existing config field thatMetricsProducerforwards — so TLS on the producer side is already configuration-only. Consumers were the gap.This adds
kafka.consumerParamsalongside it with the same shape:.unknown(true), and the keysBackbeatConsumersets itself forbidden, so a caller gets a validation error rather than a silently ignored setting. The schema is defined once inconfig.joiand reused byBackbeatConsumer's own joi, so a programmatic caller is validated like a config file.Threaded through the mongo-processor only — the one consumer with a remote-broker use case. The D/R metadata sink (ZKOP-562) consumes from brokers on the production site, whose exposed listener is configured with
SSLClientAuth: SSLClientAuthRequired. The other eightBackbeatConsumercall sites are one line each whenever they need it.The operator half is ZKOP-577.
The first commit extracts
consumerConfigandtopicConfiggetters, mirroringBackbeatProducer, so the rdkafka config can be asserted without stubbing.Issue: BB-850