Perhaps this helps with readability: req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte