Hello, configuring bucket notification for Kafka. I have the ca.pem, the user.pem and the user.key file. Running openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -cert user.pem -key user.key from inside the RGW container everything is fine. I added the three files via - mount_path: ... to the spec file for the RGW service. Inside the container I ran update-ca-trust kafkacat works with the three files as producer. However when uploading objects to the test bucket the RGW shows the following (debug is set to 20): Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down I can only add ca-location to the topic described here: https://docs.ceph.com/en/quincy/radosgw/notifications/ But the notification service needs all infos from all three files. A certificate chain did not work. I browsed rgw_kafka.cc and did not find anything like ca-cert or ca-key. How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker? Best Malte