- ceph-users - lists.ceph.io

Join the Ceph New Users Workshop
by Anthony Middleton 11 Aug '25

11 Aug '25

Registration for the Ceph New Users Workshop is now open! This free workshop will be held on Monday, Oct 27, the day before Cephalocon, in Vancouver. RSVP here: https://events.clyso.tech/event/2/registrations/2/ Registration closes on Oct 24 at 11:59 p.m. PDT. Priority will be given to users who are registered for Cephalocon. If you have any questions, don't hesitate to get in touch with me at community-manager(a)ceph.io.

1 0

Best/Safest way to power off cluster
by gagan tiwari 11 Aug '25

11 Aug '25

Hi Guys, I have recently set-up a production ceph cluster which consists of 3 monitor nodes and 7 Osd nodes. There is power maintenance activity scheduled at the data centre coming weekend and due to that I need to power off all the devices. Can you please advise me on the safest way to power off all servers? Should I power off all 7 OSD servers one by one followed by all 3 monitor nodes or vice versa? Thanks, Gagan

6 14

Squid 19.2.3 rm-cluster does not zap OSDs
by Eugen Block 11 Aug '25

11 Aug '25

Hi *, I have a VM which I use frequently to test cephadm bootstrap operations as well as upgrades, it's a single node with a few devices attached. After successfully testing the upgrade to 19.2.3, I wanted to test the bootstrap again, but removing the cluster with the --zap-osds flag doesn't actually remove the VGs/LVs. This used to work just fine until 19.2.2. This is the command I used: ceph:~ # cephadm --image myregistry/ceph_v19.2.3 rm-cluster --fsid {FSID} --zap-osds --force There's not much to see in the cephadm.log after these sort of lines: 2025-08-01 09:04:02,103 7f3bf5206b80 DEBUG systemctl: Removed "/etc/systemd/system/ceph-6b501d0a-6ea3-11f0-a251-fa163e2ad8c5.target.wants/ceph-6b501d0a-6ea3-11f0-a251-fa163e2ad8c5(a)osd.0.service". But it looks like the inventory output in json format, it's quite lengthy, so I'll spare you the output here. I can add it to a tracker though, if there isn't one yet. Has this already been reported? Thanks, Eugen

1 1

Display ceph version on ceph -s output
by Frédéric Nass 11 Aug '25

11 Aug '25

Dear community, Some time ago, I wondered why the Ceph version didn't show up in the ceph status (ceph -s) output. Perhaps it was because a Ceph cluster could run multiple versions at once (during an upgrade for example) and this information wasn't centralized, before the 'ceph versions' command was introduced. Regardless, I think it would be a great addition to have the Ceph version displayed in the ceph status. It could look something like this below. On a cluster where all daemons are on version 18.2.7: $ ceph -s cluster: version: 18.2.7 id: f4147dec-4c5d-11f0-a310-bc24113b5ce1 health: HEALTH_OK On a cluster that is currently being upgraded from 18.2.4 to 18.2.7: $ ceph -s cluster: version: 18.2.7 (432), 18.2.4 (248) <----- Could be refined. For example (3 MONs, 3 MGRs, 426 OSDs) until all daemons are on the same version. id: f4147dec-4c5d-11f0-a310-bc24113b5ce1 health: HEALTH_WARN Let me know if you like the idea and I'll create a tracker. Best regards, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass(a)clyso.com

7 6

osd latencies and grafana dashboards, squid 19.2.2
by Christopher Durham 11 Aug '25

11 Aug '25

In my 19.2.2/squid cluster, (Rocky 9 Linux) I am trying to determine if I am having issues with OSD latency. The following URL: https://sysdig.com/blog/monitoring-ceph-prometheus/ states the following about prometheus metrics: * ceph_osd_op_r_latency_count: Returns the number of reading operations running. * ceph_osd_op_r_latency_sum: Returns the time, in milliseconds, taken by the reading operations. This metric includes the queue time. * ceph_osd_op_w_latency_count: Returns the number of writing operations running. * ceph_osd_op_w_latency_sum: Returns the time, in milliseconds, taken by the writing operations. This metric includes the queue time. and * ceph_osd_commit_latency_ms: Returns the time it takes OSD to read or write to the journal. * ceph_osd_apply_latency_ms: Returns the time it takes to write the journal to the physical disk. The first set states 'includes the queue time'. What exactly does this mean? Does this mean that this is the time waiting before writing to the journal while in the memory of the ceph-osd daemon? If so, does the latter two metrics mean that once the writes start, this is the time it takes to write to the journal or the disk? Does the first set of metrics *include* the latter? In other words, are the apply/commit latencies included in the *[r,w]_latency_sum? The URL above suggests that to calculate the write latency for a given OSD, you do the following: (rate(ceph_osd_op_w_latency_sum[5m]) / rate(ceph_osd_op_w_latency_count[5m]) >= 0) However, the grafana dashboard 'OSD Overview' (in ceph-dashboards-19.2.2 rpm), does something very similar for max,avg,quantile: max(rate(ceph_osd_op_w_max_latency_sum(cluster=\$cluster\,}[$__rate_interval])/ on (ceph_daemon) rate(ceph_osd_op_w_max_latency_count(cluster=\$cluster\,}[$__rate_interval]) * 1000)) The extra multiple of 1000 seems extraneous based on the fact that the *latency_count is already in milliseconds, and the graph itself shows 'ms'. and led me to think I have disk latency issues, as these numbers are high. (maybe I do, and maybe I misunderstand something) Other dashboards such as 'Ceph Cluster - Advanced' which, in the panel 'OSD Commit Latency Distribution' use similar promQL expressions but without the extra multiple of 1000, which looks alot better for evaluation of my latencies. So the question is, is the extra multiple of 1000 incorrect in the 'OSD Overview' dashboard? Or am I not understanding things correctly? Also, does 'ceph osd perf' just show the apply/commit sum/count metrics from above? Thanks for any assistance. -Chris

2 2

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

ceph 19.2.2 - adding new hard drives messed up the order of existing ones - OSD down
by Steven Vacaroaia 08 Aug '25

08 Aug '25

Hi, I needed to add more spinning HDD to my nodes ( SuperMicro SSG-641E-E1CR36L) and made the mistake of NOT setting up osd_auto_discovery to "false" so ceph created OSDs on all 5 new spinning HDD This was an issue as I wanted to configure the OSDs the same as existing ones ( i.e. with WAL/DB on NVME) when the other 37 drives arrive No big harm done though because I could zap them and reconfigure ( after running ceph orch apply osd --all-available-devices --unmaged=true) when I will receive the remaining 37 drives ( as I will add 6 drives on each server) The interesting part is that, for whatever reason, one of the existing SSD based OSD is down now because the SSD drive it used changed from /dev/sdp to /dev/sdu therefore there is no "block" entry under /var/lib/ceph/FSID/osd.XX I am not sure why adding spinning disks mess up the order/naming of sdd disks I would appreciate some advice regarding the best course of action to reconfigure the OSD that is down The cluster is healthy , not busy, with all the other OSDs working as expected Has 7 hosts with 12 SSD and 6 HDD drives each ( one of them is the one with issues), 2 EC 4+2 pool , 2 MDS and few metadata pools replicated on NVME There are also 3 NVME disks on each dedicated to pools Many thanks Steven

2 1

60/90 bays + 6 NVME Supermicro
by Manuel Rios - EDH 08 Aug '25

08 Aug '25

Hi Ceph , Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4 NVME for WALL? Newest models support 144cores in single socket and several TB of ram without issues. But as far as we understood in the technical notes it use SAS Expanders for connect all disk, from 2 to 4 SAS Expanders to connect the whole chasis. We're looking for next configuration : Intel Xeon 96cores @ 2.4 1TB RAM 2 SSD for OS. 4 NVME x 30TB NVME KIOXIA 90 HDD * 22 TB HGST 4x25 Gbps or 2x100 Gbps Main use RGW. Regards [cid:image001.jpg@01DC07CB.6CB5C6C0] MANUEL RIOS FERNANDEZ CEO - EasyDataHost Phone: 677677179 Web: www.easydatahost.com<http://www.easydatahost.com/> Email: mriosfer(a)easydatahost.com<mailto:mriosfer@easydatahost.com> [T?tulo: LinkedIn - Descripci?n: image of LinkedIn icon] <https://es.linkedin.com/in/manuel-rios-fernandez-14880949?original_referer=…> ADVERTENCIA LEGAL: Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. [Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Flor ecoTECH]No imprimas si no es necesario. Protejamos el Medio Ambiente.

5 6

How set pg number for pools
by Albert Shih 07 Aug '25

07 Aug '25

Hi, Stupid question but how can I set the number of pg for a pool when I got differents (1.7 To , 2.7T, 2.9To and 3.4To) size of ssd ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 07 août 2025 14:06:35 CEST

2 1

Pgs troubleshooting
by GLE, Vivien 05 Aug '25

05 Aug '25

Hi, After replacing 2 OSD (data corruption), this is the stats of my testing ceph cluster ceph pg stat 498 pgs: 37 peering, 1 active+remapped+backfilling, 1 active+clean+remapped, 1 active+recovery_wait+undersized+remapped, 1 backfill_unfound+undersized+degraded+remapped+peered, 1 remapped+peering, 12 active+clean+scrubbing+deep, 1 active+undersized, 442 active+clean, 1 active+recovering+undersized+remapped 34 GiB data, 175 GiB used, 6.2 TiB / 6.4 TiB avail; 1.7 KiB/s rd, 1 op/s; 31/39768 objects degraded (0.078%); 6/39768 objects misplaced (0.015%); 1/13256 objects unfound (0.008%) ceph osd stat 7 osds: 7 up (since 20h), 7 in (since 20h); epoch: e427538; 4 remapped pgs Anyone had an idea of where to start to get a healthy cluster ? Thanks ! Vivien

4 22