- ceph-users - lists.ceph.io

Kafka notification, bad certificate
by Malte Stroem 25 Mar '25

25 Mar '25

Hello, configuring bucket notification for Kafka. I have the ca.pem, the user.pem and the user.key file. Running openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -cert user.pem -key user.key from inside the RGW container everything is fine. I added the three files via - mount_path: ... to the spec file for the RGW service. Inside the container I ran update-ca-trust kafkacat works with the three files as producer. However when uploading objects to the test bucket the RGW shows the following (debug is set to 20): Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: persistent val: false Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: push-endpoint val: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: security.protocol val: ssl Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-ssl val: true Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new connection is created. Total connections: 1 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured SSL security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using default CA location Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully configured security Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully created new producer Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094 Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully created topic: ktest Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) ar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:30 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start Mar 12 16:52:31 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) Mar 12 16:52:34 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get: name=default.rgw.log++script.background. : hit (negative entry) Mar 12 16:52:35 rgwnode1000 bash[811097]: debug 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll error(-187): 1/1 brokers are down I can only add ca-location to the topic described here: https://docs.ceph.com/en/quincy/radosgw/notifications/ But the notification service needs all infos from all three files. A certificate chain did not work. I browsed rgw_kafka.cc and did not find anything like ca-cert or ca-key. How to add the full chain or all three files to the topic so the RGW can connect to the Kafka broker? Best Malte

3 8

Downgrading the osdmap
by Marek Szuba 24 Mar '25

24 Mar '25

Dear fellow Ceph users, I run a Ceph cluster providing CephFS to a medium-sized Linux server farm. Originally we used the kernel driver (which on the distro we use shows itself to the cluster as a Luminous client) to mount the file system, however at the time of the upgrade to Squid we became aware of the data-corruption bug associated with the use of root squash and subsequently switched to (the Squid version of) ceph-fuse. Furthermore, I took advantage of the switch to bump the OSD require-min-compat-client to Reef and subsequently switch the balancer mode to upmap-read. Weeks passed and I became aware that despite all the attempted tuning, certain work loads which performed fine using the kernel driver perform _extremely_ poorly using ceph-fuse. A decision has therefore been made to relocate the servers which absolutely must have root squash to a different network-storage solution, switch root squash off for all CephFS clients and revert to the kernel driver. Unfortunately while removing client_mds_auth_caps from CephFS required_client_features and switching the balancer mode back to upmap went without any problems, "ceph osd set-require-min-compat-client luminous" fails with Error EPERM: osdmap current utilizes features that require reef; cannot set require_min_compat_client below that to luminous Is there a way of making the osdmap Luminous-compatible again without losing any data stored on the cluster ()? Thank you in advance for your help! -- MS

2 3

Reper Ceph Cluster only OSDs intakt
by filip Mutterer 24 Mar '25

24 Mar '25

Playing around with Ceph I did a Big mistake. My root filesystem uses snapshot and only excluding /home & /root. When I switched back to an older Snapshot I noticed Ceph is complaining all OSDs are not working. How can I Fix this? Rebuild cluster and then add the OSDs back in? Is there a guide? Which folders had to be excluded for snapshotting not affecting Ceph?

1 0

Rogue EXDEV errors when hardlinking
by Domhnall McGuigan 24 Mar '25

24 Mar '25

Hi all, we've been seeing persistent problems when trying to create hardlinks on cephfs; it's returning EXDEV in a way that makes no sense given typical POSIX behaviour and ceph documentation. Here's a typical strace of the problem: 78 13:47:26.572435 link("/data/db/hdb/data/2023.08.06/table1.0/column1", "/data/db/hdb/data/2023.08.06/table1.1/column1") = -1 EXDEV (Invalid cross-device link) 78 13:47:26.577661 write(1, "{\"time\":\"2025-03-03T13:47:26.577z\",\"component\":\"MSVC\",\"level\":\"INFO\",\"message\":\"[eoi-78] Retrying in 500 milliseconds\",\"service\":\"eoi\"}\n", 136) = 136 78 13:47:26.577762 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=500000000}, NULL) = 0 78 13:47:27.078037 link("/data/db/hdb/data/2023.08.06/table1.0/column1", "/data/db/hdb/data/2023.08.06/table1.1/column1") = 0 We try creating a link, get EXDEV, wait 500 milliseconds, then try the same operation again and it succeeds. The link and its target are both on the same cephfs mount (/data/db/hdb in this case), so the normal POSIX 'linking between filesystems' explanation doesn't apply. I've looked through the ceph client and server code and from what I've seen EXDEV is only returned in a couple of other situations: linking between snapshots, and linking across quotas. Neither snapshots nor quotas were in use here, and if they were the culprit it seems unlikely the automatic retry would have worked. Web searches on EXDEV errors in ceph have also proven to be a dead end. My best guess, although it's not a very good one, is that stale MDS cache data is somehow involved -- in one case the issue reportedly got much worse after increasing (!) the MDS memory limit. This error has been occurring for a particular client for upwards of 9 months and has proven stubbornly resistant to reproduction elsewhere (we are working on migrating them to a more recent ceph version to see if the error remains), so our technical investigations haven't got particularly far. I was hoping someone here on ceph-users would have seen similar EXDEV errors in the wild or in development and have some insight into what could be causing them. Regards, Domhnall *********************************************************************************************************************************************************************** This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. If you are not the named addressee or if you have received this email in error, you may not, without the consent of KX, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. Unless separately agreed, KX does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of KX and KX shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed. ***********************************************************************************************************************************************************************

3 4

Ceph MCP Server
by Anthony Fecarotta 23 Mar '25

23 Mar '25

Has the idea been floated to create a Ceph MCP server? I can't remember if it was the last Ceph Days or Cephlacon, but someone mentioned the concept of a Ceph chatbot trained on Ceph data, which would be awesome, but an MCP server would be a quick/easy way to get up and running in the LLM vertical. Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

1 0

Osd won't restart ceph 17.2.7
by xadhoom76＠gmail.com 22 Mar '25

22 Mar '25

Hi, i try to restart some osds after have enabled compact ceph config set osd osd_compact_on_start true running ceph orch daemon restart osd.4 looking to journaclt of that osd i got this received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 mar 22 03:10:02 ceph03s3.itservicenet.net ceph-aed2e978-85f2-11ed-b955-0cc47aa3fdbc-osd-4[8893]: 2025-03-22T07:10:02.068+0000 7fb232211700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Why does not work ? Best regards

1 1

19.2.1 dashboard OSD column sorts do nothing?
by Harry G Coin 21 Mar '25

21 Mar '25

Has anyone else tried to change the sort order of columns in the cluster/osd display on 19.2.1? While the header changes to indicate 'increasing/descending' and 'selected', the rows stay fixed on an ascending order by ID. ???

2 1

Reef: Dashboard bucket edit fails in get_bucket_versioning
by Eugen Block 21 Mar '25

21 Mar '25

Hi, I just created a new tracker issue [0]. Trying to modify a bucket via dashboard results in a stack trace for the mgr: ---snip--- 2025-03-13T11:23:15.077+0000 7efe02290640 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 357, in get versioning = self._get_versioning(result['owner'], daemon_name, bucket_name) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 246, in _get_versioning return rgw_client.get_bucket_versioning(bucket_name) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 537, in func_wrapper return func( File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 617, in get_bucket_versioning if 'Status' not in result: TypeError: argument of type 'NoneType' is not iterable 2025-03-13T11:23:15.077+0000 7efe02290640 0 [dashboard ERROR request] [::ffff:IP:57676] [GET] [500] [0.034s] [admin] [513.0B] /api/rgw/bucket/bucket2 2025-03-13T11:23:15.077+0000 7efe02290640 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "d574a812-310f-4a51-beba-a677e596c5bf"} ---snip--- I couldn't find anything existing in tracker yet. I don't have versioning enabled. Has anyone seen the same? Thanks! Eugen [0] https://tracker.ceph.com/issues/70449

2 3

One host down osd status error
by Marcus 21 Mar '25

21 Mar '25

Hi all, We are running a ceph cluster with filesystem that contains 5 servers. Ceph version: 19.2.0 squid If I run: ceph osd status when all hosts are online and in the output is the way it should and it prints status for all osds. If just a couple of osds are down status is printed and specific osds are stated as down. One of the servers went down and we ended up with a health warning. If I run: ceph osd stat I get the information that 64 out of 80 osd is in. If I try to run: ceph osd status I get an python error: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1864, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/status/module.py", line 337, in handle_osd_status assert metadata AssertionError I suppose this is some type of bug when one host is down? Thanks! Marcus

3 4

March Ceph Science Virtual User Group
by Enrico Bocchi 20 Mar '25

20 Mar '25

Dear All, We will be having a Ceph science/research/big cluster call on Tuesday March 25th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We are trying to start each meeting with a brief 10-15 minutes talk given by a community member. While there is none schedule for the upcoming meeting, we welcome contributions. If you are interested in signing up to give a short talk for a meeting please let me know! Virtual event details: March 25th, 2025 10am Central US 15:00 UTC 4pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ Go to Mar 25 and you can quickly add it to your calendar as well. See you there! Enrico

1 1