- ceph-users - lists.ceph.io

restart dashboard ?
by lejeczek 01 Aug '25

01 Aug '25

Hi guys. A newbie here - go easy on me. I'm following docs on adding certificates to dashboard - from an external CA. I do 'ceph mgr module disable/enable' & 'ceph mgr fail' but those do not seem to get desired effect. How to get ceph take certs changes without disrupting ceph? many thanks, L.

2 1

Two Quick Reminders
by Anthony Middleton 31 Jul '25

31 Jul '25

The deadline for the Cephalocon 2025 T-shirt design contest <https://ceph.io/en/news/blog/2025/cephalocon25-tshirt-contest/> is tomorrow at 11:59 pm PDT. Please send your designs to me at community-manager(a)ceph.io. Feel free to reach out to me with any questions you might have. The Ceph Community Survey is ongoing. I'm reviewing the data, but there's still time to participate. Please spend 2-3 minutes to share your feedback at https://forms.gle/uv57ahnQkjbu6VSJ8. Best, Anthony Middleton Ceph Community Manager

1 0

squid 19.2.2 deployed with cephadmin - no grafana data on some dashboards ( RGW, MDS)
by Steven Vacaroaia 31 Jul '25

31 Jul '25

Hi, Any of you encounter this issue ? Most of the dashboards works but MDS and RGW specific ones have no data Prometheus ( http:server:9283) does not show anything with RGW or MDS in it so, I am guessing, grafana cannot show what it is not collected :-) ceph-exporter is running Any suggestions/ ideas for how to troubleshoot /fix this ? Many thanks Steven

3 3

bdev_ioring -- true - Drives failing
by Devender Singh 31 Jul '25

31 Jul '25

Hello all Using 19.2.2, when enabling bdev_ioring and rebooting host then osd never comes online as below…. Why? Also Dasboard not matching with cli? Host output in dashboard… showing running. Tried failing manager too but it same. Dashboard showing down # ceph health detail HEALTH_WARN 4 osds down; 1 host (4 osds) down; Degraded data redundancy: 12998/38994 objects degraded (33.333%), 287 pgs degraded, 801 pgs undersized [WRN] OSD_DOWN: 4 osds down osd.4 (root=default,host= host07n) is down osd.5 (root=default,host= host07n) is down osd.6 (root=default,host= host07n) is down osd.7 (root=default,host= host07n) is down [WRN] OSD_HOST_DOWN: 1 host (4 osds) down host host07n (root=default) (4 osds) is down # ceph orch ps |grep -v running. ————> not showing anything down…. NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.4.service loaded activating auto-restart Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.5.service loaded activating auto-restart Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.6.service loaded activating auto-restart Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.7.service loaded activating auto-restart Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Came to running ….. # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.4.service loaded active running Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.5.service loaded active running Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.6.service loaded active running Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.7.service loaded active running Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Failed ….. # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.4.service loaded failed failed Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.5.service loaded failed failed Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.6.service loaded failed failed Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.7.service loaded failed failed Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Regards Dev

2 4

Kafka bucket notifications with OAUTHBEARER on Ceph Reef/Squid
by Rao, Shreesha 31 Jul '25

31 Jul '25

Hi Ceph community, We are exploring setting up bucket notifications using Kafka on our Ceph cluster running Reef/Squid (P.S. versions output at the end of this email), and have verified that the Kafka library is integrated in the `radosgw-admin` binary. We attempted to use mTLS authentication by providing certificates and keys to `radosgw-admin`, but it fails to connect to the Kafka broker. From various threads and the open PR here (http://github.com/ceph/ceph/pull/61572) it seems mTLS authentication is not yet supported on the Ceph version we are using. Using username/password is not an option with our Kafka system. The documentation mentions support for the OAUTHBEARER mechanism; however, the payload example at https://docs.ceph.com/en/squid/radosgw/notifications/#create-a-topic does not include a token field. It also does not clarify how token refresh is handled when the token expires. Specifically: • Where can we configure the URL or method for RGW to refresh the auth token? • Are there any example configurations available demonstrating the correct setup and token refresh with OAUTHBEARER for Kafka notifications? Additionally, could you please clarify if these configurations must be done via `radosgw-admin` commands or through an S3 client? Examples of commands for either approach would be very helpful. If OAUTHBEARER and user-password is not an option, is there any other way to configure bucket notifications using Kafka? Please advise. Thank you very much in advance for your guidance. Best regards, Shreesha P.S. ubuntu@fr1obj001:~$ sudo cephadm shell Inferring fsid 877b08fe-f88a-4468-a224-7cf7980c0f47 Inferring config /var/lib/ceph/877b08fe-f88a-4468-a224-7cf7980c0f47/mon.fr1obj001/config root@fr1obj001:/# ceph -v ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) root@fr1obj001:/# ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 106 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 115 } } Sensitivity: Company-Internal ==================== This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message.

1 0

Is "ceph -v" command deprecated?
by Rao, Shreesha 31 Jul '25

31 Jul '25

Hi, We were running ceph cluster with reef version (18.2.4) and recently upgraded to the new squid version. However, “ceph -v” still shows the version as reef, even though the upgrade successfully went through without any errors. Are we missing something here? root@fr1obj001:/# ceph -v ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) root@fr1obj001:/# ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 106 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 115 } } Best, Shreesha Sensitivity: Company-Internal ==================== This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message.

2 2

Re: Getting SSL Certificate Verify failed Error while installation
by John Mulligan 30 Jul '25

30 Jul '25

On Tuesday, July 29, 2025 10:50:06 AM Eastern Daylight Time gagan tiwari wrote: > Hi John, > Thanks for reply! > > curl seems to be working fine. Here is output of the command :- > > curl https://download.ceph.com > <html> > <head><title>Index of /</title></head> > <body> > <h1>Index of /</h1><hr><pre><a href="../">../</a> > <a href="archive/">archive/</a> > 13-Nov-2024 03:25 - > <a href="calamari/">calamari/</a> > 29-Oct-2015 18:40 - > <a href="ceph-iscsi/">ceph-iscsi/</a> > 22-May-2024 03:18 - > <a href="cephmetrics/">cephmetrics/</a> > 29-Jun-2017 22:07 - > <a href="cloudinit/">cloudinit/</a> > 11-Aug-2020 14:47 - > <a href="debian-15.1.0/">debian-15.1.0/</a> > 13-Mar-2020 22:19 - > <a href="debian-15.1.1/">debian-15.1.1/</a> > 13-Mar-2020 22:21 - > <a href="debian-15.2.0/">debian-15.2.0/</a> > 23-Mar-2020 22:43 - > <a href="debian-15.2.1/">debian-15.2.1/</a> > 09-Apr-2020 16:11 - > <a href="debian-15.2.10/">debian-15.2.10/</a> > 18-Mar-2021 13:41 - > <a href="debian-15.2.11/">debian-15.2.11/</a> > 19-Apr-2021 22:17 - > <a href="debian-15.2.12/">debian-15.2.12/</a> > 13-May-2021 23:06 - > <a href="debian-15.2.13/">debian-15.2.13/</a> > 26-May-2021 22:26 - > <a href="debian-15.2.14/">debian-15.2.14/</a> > 05-Aug-2021 21:32 - > <a href="debian-15.2.15/">debian-15.2.15/</a> > 20-Oct-2021 18:33 - > <a href="debian-15.2.16/">debian-15.2.16/</a> > 01-Mar-2022 12:18 - > <a href="debian-15.2.17/">debian-15.2.17/</a> > 09-Aug-2022 20:59 > > > I have tried to install the latest version of squid 19.2.3 and 19.2.2. > Facing the same SSL issue with both versions. > > Thanks, > Gagan > It dawns on me that curl might not be the best test as I think it can use a different cert bundle by default. Does this command fail the same way as cephadm? ``` python3 -c 'import urllib.request; print(len(urllib.request.urlopen("https://download.ceph.com").read()))' ``` PS. Please keep the list CC'ed. We want others to be able to help as well and keep a record of what was discussed.

4 4

MDS Client Request Load
by Devender Singh 30 Jul '25

30 Jul '25

Hello How to lower this load? Cephfs is on EC pool sitting over replicated data and meta.. What settings can improve it? Regards Dev

2 1

Upgrade from 19.2.2 to .3 pauses on 'phantom' duplicate osd?
by Harry G Coin 30 Jul '25

30 Jul '25

Need a clue about what appears to be a phantom duplicate osd automagically created/discovered via the upgrade process -- which blocks the upgrade. The upgrade process on a known-good 19.2.2 to 19.2.3 proceeded normally through the mgrs and mons. It upgraded most of the osds, then stopped with the complaint "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.1 on host noc3 failed." The roster in the "Daemon Versions" table on the dashboard looks normal except: There are two entries for 'osd.1' One of them has the correct version number, 19.2.2, the other is blank. The upgrade appears 'stuck'. An attempt to 'resume' resulted in the same error. The cluster operations are normal with all osds up and in. The cluster is ipv6. Oddly ceph -s reports: root@noc1:~# ceph -s cluster: id: 406xxxxxxx0f8 health: HEALTH_WARN Public/cluster network defined, but can not be found on any host Upgrading daemon osd.1 on host noc3 failed. services: mon: 5 daemons, quorum noc4,noc2,noc1,noc3,sysmon1 (age 39m) mgr: noc2.yhyuxd(active, since 4h), standbys: noc3.sybsfb, noc4.tvhgac, noc1.jtteqg mds: 1/1 daemons up, 3 standby osd: 27 osds: 27 up (since 3h), 27 in (since 10d) data: volumes: 1/1 healthy pools: 16 pools, 1809 pgs objects: 14.77M objects, 20 TiB usage: 52 TiB used, 58 TiB / 111 TiB avail pgs: 1808 active+clean 1 active+clean+scrubbing io: client: 835 KiB/s rd, 1.0 MiB/s wr, 24 op/s rd, 105 op/s wr progress: Upgrade to 19.2.3 (4h) [============................] (remaining: 4h) Related log entry: 29/7/25 02:40 PM[ERR]cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126d-01cb-40af-824a-881c130140f8-osd-1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd-1 Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 Reconfig daemon osd.1 ... Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3051, in command_deploy_from File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3086, in _common_deploy File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3106, in _deploy_daemon_container File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 1077, in deploy_daemon File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 765, in create_daemon_dirs File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/file_utils.py", line 52, in write_new IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config.new' -> '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config' Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1145, in _check_daemons self.mgr._daemon_action(daemon_spec, action=action) File "/usr/share/ceph/mgr/cephadm/module.py", line 2545, in _daemon_action return self.wait_async( File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/serve.py", line 1381, in _create_daemon out, err, code = await self._run_cephadm( File "/usr/share/ceph/mgr/cephadm/serve.py", line 1724, in _run_cephadm raise OrchestratorError( orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd-1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd-1 Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 Reconfig daemon osd.1 ... Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3051, in command_deploy_from File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3086, in _common_deploy File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3106, in _deploy_daemon_container File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 1077, in deploy_daemon File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 765, in create_daemon_dirs File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/file_utils.py", line 52, in write_new IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config.new' -> '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config'

2 1

Still Time to Take the Ceph Community Survey
by Anthony Middleton 29 Jul '25

29 Jul '25

Hey Ceph Users, I'm still collecting responses for the *Ceph Community Survey*, and your input would be greatly appreciated! This brief survey helps us understand: - How Ceph is used in production today - Why some organizations have moved away - - The value Ceph provides to your teams - Who might be interested in sharing their story at *Cephalocon 2025* in Vancouver I will begin reviewing submissions later this week, so please submit your feedback as soon as possible! Survey link: https://forms.gle/uQPiKkEbQAvjPfp46. Thanks to everyone who has contributed so far! Your insights are shaping the future of Ceph. Best, Anthony Middleton Ceph Community Manager

1 0