Kafka notification, bad certificate
Hello, configuring bucket notification for Kafka. I have the ca.pem, the user.pem and the user.key file. Running openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -cert user.pem -key user.key from inside the RGW container everything is fine. I added the three files via - mount_path: ... to the spec file for the RGW service. Inside the container I ran update-ca-trust kafkacat works with the three files as producer. However when uploading objects to the test bucket the RGW shows the following (debug is set to 20): Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down I can only add ca-location to the topic described here: https://docs.ceph.com/en/quincy/radosgw/notifications/ But the notification service needs all infos from all three files. A certificate chain did not work. I browsed rgw_kafka.cc and did not find anything like ca-cert or ca-key. How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker? Best Malte
Perhaps this helps with readability: req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte
I somehow have the feeling this is related to: https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581... Because we have a three node Kafka cluster. On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push- endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka- testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte
Hi Malte, Based on the documentation link you shared in your first message, you must be using Quincy. Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL. It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out. Regards, Frédéric. [1] https://tracker.ceph.com/issues/68033 ----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@gmail.com a écrit :
I somehow have the feeling this is related to:
https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581...
Because we have a three node Kafka cluster.
On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push- endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka- testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frédéric, thanks, but no, my mistake. We use Reef. However, I think my problem is regarding Kafka using mTLS for authentication. If I am correct this will be implemented somewhere in the future. See the Github link I posted. I got a ca.crt, a user.crt and a user.key file for authentication to the Kafka broker. Using a simple Python script this works fine. Also "kafkacat" works good. And of course openssl if I add these three files. openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.crt -cert user.crt -key user.key I can send notifications. The RGW just does not like this and always says: RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909:error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) It somehow does not use these three files or at least the two files user.crt and user.key. Best, Malte On 19.03.25 17:22, Frédéric Nass wrote:
Hi Malte,
Based on the documentation link you shared in your first message, you must be using Quincy.
Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL. It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/68033
----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@gmail.com a écrit :
I somehow have the feeling this is related to:
https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581...
Because we have a three node Kafka cluster.
On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push- endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka- testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Malte, Yeah, I just wanted to make you aware of this separate Kafka bug in Quincy and Reef v18.2.4. Regarding your issue, if your Kafka installation requires mTLS and mTLS hasn't been coded in RGW yet (as you pointed [1]), I think you figured it out. Also, the logs showing a first successful TLS negotiation ("successfully configured SSL security") followed by a second one that fails ("alert bad certificate: SSL alert number 42" [2]) seems to support the hypothesis that the client sent a message instead of a valid certificate as expected by the broker to validate the second step of mTLS [3]. Regards, Frédéric. [1] https://github.com/ceph/ceph/pull/61572 [2] https://serverfault.com/questions/806141/is-the-alert-ssl3-read-bytessslv3-a... [3] https://medium.com/lydtech-consulting/securing-kafka-with-mutual-tls-and-acl... ----- Le 19 Mar 25, à 23:00, Malte Stroem malte.stroem@gmail.com a écrit :
Hi Frédéric,
thanks, but no, my mistake. We use Reef.
However, I think my problem is regarding Kafka using mTLS for authentication.
If I am correct this will be implemented somewhere in the future.
See the Github link I posted.
I got a ca.crt, a user.crt and a user.key file for authentication to the Kafka broker.
Using a simple Python script this works fine. Also "kafkacat" works good. And of course openssl if I add these three files.
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.crt -cert user.crt -key user.key
I can send notifications.
The RGW just does not like this and always says:
RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909:error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY)
It somehow does not use these three files or at least the two files user.crt and user.key.
Best, Malte
On 19.03.25 17:22, Frédéric Nass wrote:
Hi Malte,
Based on the documentation link you shared in your first message, you must be using Quincy.
Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL. It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/68033
----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@gmail.com a écrit :
I somehow have the feeling this is related to:
https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581...
Because we have a three node Kafka cluster.
On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push- endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka- testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, As noted above, I already started implementing mtls support. Currently blocked on adding an mtls test (would appreciate any help there!). Once I'll have time to write the test it would be easy to backport (it's a simple fix). Yuval On Thu, Mar 20, 2025 at 11:00 AM Frédéric Nass < frederic.nass@univ-lorraine.fr> wrote:
Hi Malte,
Yeah, I just wanted to make you aware of this separate Kafka bug in Quincy and Reef v18.2.4.
Regarding your issue, if your Kafka installation requires mTLS and mTLS hasn't been coded in RGW yet (as you pointed [1]), I think you figured it out. Also, the logs showing a first successful TLS negotiation ("successfully configured SSL security") followed by a second one that fails ("alert bad certificate: SSL alert number 42" [2]) seems to support the hypothesis that the client sent a message instead of a valid certificate as expected by the broker to validate the second step of mTLS [3].
Regards, Frédéric.
[1] https://github.com/ceph/ceph/pull/61572 [2] https://serverfault.com/questions/806141/is-the-alert-ssl3-read-bytessslv3-a... [3] https://medium.com/lydtech-consulting/securing-kafka-with-mutual-tls-and-acl...
----- Le 19 Mar 25, à 23:00, Malte Stroem malte.stroem@gmail.com a écrit :
Hi Frédéric,
thanks, but no, my mistake. We use Reef.
However, I think my problem is regarding Kafka using mTLS for authentication.
If I am correct this will be implemented somewhere in the future.
See the Github link I posted.
I got a ca.crt, a user.crt and a user.key file for authentication to the Kafka broker.
Using a simple Python script this works fine. Also "kafkacat" works good. And of course openssl if I add these three files.
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.crt -cert user.crt -key user.key
I can send notifications.
The RGW just does not like this and always says:
RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909:error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY)
It somehow does not use these three files or at least the two files user.crt and user.key.
Best, Malte
On 19.03.25 17:22, Frédéric Nass wrote:
Hi Malte,
Based on the documentation link you shared in your first message, you must be using Quincy.
Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL. It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/68033
----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@gmail.com a écrit :
I somehow have the feeling this is related to:
https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581...
Because we have a three node Kafka cluster.
On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name:
Version
val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push- endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl:// kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap : Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka- testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap ]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap ]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Yuval, yes, I would really like to help here. We're running Reef but can upgrade immediately. Contact me if you need the help. Best, Malte On 22.03.25 18:46, Yuval Lifshitz wrote:
Hi, As noted above, I already started implementing mtls support. Currently blocked on adding an mtls test (would appreciate any help there!). Once I'll have time to write the test it would be easy to backport (it's a simple fix).
Yuval
On Thu, Mar 20, 2025 at 11:00 AM Frédéric Nass < frederic.nass@univ-lorraine.fr> wrote:
Hi Malte,
Yeah, I just wanted to make you aware of this separate Kafka bug in Quincy and Reef v18.2.4.
Regarding your issue, if your Kafka installation requires mTLS and mTLS hasn't been coded in RGW yet (as you pointed [1]), I think you figured it out. Also, the logs showing a first successful TLS negotiation ("successfully configured SSL security") followed by a second one that fails ("alert bad certificate: SSL alert number 42" [2]) seems to support the hypothesis that the client sent a message instead of a valid certificate as expected by the broker to validate the second step of mTLS [3].
Regards, Frédéric.
[1] https://github.com/ceph/ceph/pull/61572 [2] https://serverfault.com/questions/806141/is-the-alert-ssl3-read-bytessslv3-a... [3] https://medium.com/lydtech-consulting/securing-kafka-with-mutual-tls-and-acl...
----- Le 19 Mar 25, à 23:00, Malte Stroem malte.stroem@gmail.com a écrit :
Hi Frédéric,
thanks, but no, my mistake. We use Reef.
However, I think my problem is regarding Kafka using mTLS for authentication.
If I am correct this will be implemented somewhere in the future.
See the Github link I posted.
I got a ca.crt, a user.crt and a user.key file for authentication to the Kafka broker.
Using a simple Python script this works fine. Also "kafkacat" works good. And of course openssl if I add these three files.
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.crt -cert user.crt -key user.key
I can send notifications.
The RGW just does not like this and always says:
RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909:error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY)
It somehow does not use these three files or at least the two files user.crt and user.key.
Best, Malte
On 19.03.25 17:22, Frédéric Nass wrote:
Hi Malte,
Based on the documentation link you shared in your first message, you must be using Quincy.
Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL. It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/68033
----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@gmail.com a écrit :
I somehow have the feeling this is related to:
https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581...
Because we have a three node Kafka cluster.
On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name:
Version
val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push- endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl:// kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap : Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote: > Hello, > > configuring bucket notification for Kafka. > > I have the ca.pem, the user.pem and the user.key file. > > Running > > openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - > cert user.pem -key user.key > > from inside the RGW container everything is fine. > > I added the three files via > > - mount_path: ... > > to the spec file for the RGW service. > > Inside the container I ran > > update-ca-trust > > kafkacat works with the three files as producer. > > However when uploading objects to the test bucket the RGW shows the > following (debug is set to 20): > > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: persistent val: false > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// > kafka-testbroker.org:9094 > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: security.protocol val: ssl > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ > pki/ ca-trust/source/anchors/kafka-rgw-ca.pem > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: ssl.certificate.location > val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ > pki/ ca-trust/source/anchors/kafka-rgw-user.key > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: use-ssl val: true > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 > 0.024000837s s3:complete_multipart name: verify-ssl val: true > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new > connection is created. Total connections: 1 > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: > successfully configured SSL security > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using > default CA location > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: > successfully configured security > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: > successfully created new producer > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 > 0.040001396s INFO: push endpoint created: kafka://kafka- > testbroker.org:9094 > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: > successfully created topic: ktest > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with > callback, tag=1): OK. Queue has: 1 callbacks > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: > rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap ]: > ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ > rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad > certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll > error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive > failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL > routines::ssl/tls alert bad certificate: SSL alert number 42 (after > 0ms in state APIVERSION_QUERY) > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll > error(-187): 1/1 brokers are down > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: > rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap ]: > ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ > rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad > certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, > 1 identical error(s) suppressed) > ar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll > error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive > failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL > routines::ssl/tls alert bad certificate: SSL alert number 42 (after > 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll > error(-187): 1/1 brokers are down > Mar 12 16:52:30 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: > RGWDataChangesLog::ChangesRenewThread: start > Mar 12 16:52:31 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll > error(-187): 1/1 brokers are down > Mar 12 16:52:34 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync > user=kuser > bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) > Mar 12 16:52:34 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache > get: name=default.rgw.log++script.background. : hit (negative entry) > Mar 12 16:52:35 rgwnode1000 bash[811097]: debug > 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll > error(-187): 1/1 brokers are down > > I can only add ca-location to the topic described here: > > https://docs.ceph.com/en/quincy/radosgw/notifications/ > > But the notification service needs all infos from all three files. > > A certificate chain did not work. > > I browsed rgw_kafka.cc and did not find anything like > > ca-cert or ca-key. > > How to add the full chain or all three files to the topic so the RGW > can connect to the Kafka broker? > > Best > Malte
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frédéric, merci beaucoup. Yes, that's what I saw, too. Thank you for your feedback and acknowledgement. Best, Malte On 20.03.25 10:00, Frédéric Nass wrote:
Hi Malte,
Yeah, I just wanted to make you aware of this separate Kafka bug in Quincy and Reef v18.2.4.
Regarding your issue, if your Kafka installation requires mTLS and mTLS hasn't been coded in RGW yet (as you pointed [1]), I think you figured it out. Also, the logs showing a first successful TLS negotiation ("successfully configured SSL security") followed by a second one that fails ("alert bad certificate: SSL alert number 42" [2]) seems to support the hypothesis that the client sent a message instead of a valid certificate as expected by the broker to validate the second step of mTLS [3].
Regards, Frédéric.
[1] https://github.com/ceph/ceph/pull/61572 [2] https://serverfault.com/questions/806141/is-the-alert-ssl3-read-bytessslv3-a... [3] https://medium.com/lydtech-consulting/securing-kafka-with-mutual-tls-and-acl...
----- Le 19 Mar 25, à 23:00, Malte Stroem malte.stroem@gmail.com a écrit :
Hi Frédéric,
thanks, but no, my mistake. We use Reef.
However, I think my problem is regarding Kafka using mTLS for authentication.
If I am correct this will be implemented somewhere in the future.
See the Github link I posted.
I got a ca.crt, a user.crt and a user.key file for authentication to the Kafka broker.
Using a simple Python script this works fine. Also "kafkacat" works good. And of course openssl if I add these three files.
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.crt -cert user.crt -key user.key
I can send notifications.
The RGW just does not like this and always says:
RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909:error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY)
It somehow does not use these three files or at least the two files user.crt and user.key.
Best, Malte
On 19.03.25 17:22, Frédéric Nass wrote:
Hi Malte,
Based on the documentation link you shared in your first message, you must be using Quincy.
Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL. It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/68033
----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@gmail.com a écrit :
I somehow have the feeling this is related to:
https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581...
Because we have a three node Kafka cluster.
On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- ack-level val: broker req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false req 4130898986969377541 0.024000837s s3:complete_multipart name: push- endpoint val: kafka://kafka-testbroker.org:9094 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- rgw-user.pem req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- ssl val: true Kafka connect: new connection is created. Total connections: 1 Kafka connect: successfully configured SSL security Kafka connect: using default CA location Kafka connect: successfully configured security Kafka connect: successfully created new producer req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Kafka publish: successfully created topic: ktest Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Kafka run: poll error(-187): 1/1 brokers are down RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Kafka run: poll error(-187): 1/1 brokers are down rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Kafka run: poll error(-187): 1/1 brokers are down BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ pki/ ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka- testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker?
Best Malte
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Frédéric Nass
-
Malte Stroem
-
Yuval Lifshitz