April 2025 - ceph-users - lists.ceph.io

ceph-ansible LARGE OMAP in RGW pool
by Danish Khan 04 Apr '25

04 Apr '25

Dear All, My ceph cluster is giving Large OMAP warning from approx 2-3 Months. I tried a few things like : *Deep scrub of PGs* *Compact OSDs* *Trim log* But these didn't work out. I guess the main issue is that 4 shards in replication site are always recovering from 2-3 months. Any suggestions are highly appreciated. Sync status: root@drhost1:~# radosgw-admin sync status realm e259e0a92 (object-storage) zonegroup 7a8606d2 (staas) zone c8022ad1 (repstaas) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 2072ee2d32 (masterstaas) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [7,90,100] oldest incremental change not applied: 2025-03-12T13:14:10.268469+0530 [7] 4 shards are recovering recovering shards: [31,41,55,80] Master site: 1. *root@master1:~# for obj in $(rados ls -p masterstaas.rgw.log); do echo "$(rados listomapkeys -p masterstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head -10* 1225387 data_log.91 1225065 data_log.86 1224662 data_log.87 1224448 data_log.92 1224018 data_log.89 1222156 data_log.93 1201489 data_log.83 1174125 data_log.90 363498 data_log.84 258709 data_log.6 2. *root@master1:~# for obj in data_log.91 data_log.86 data_log.87 data_log.92 data_log.89 data_log.93 data_log.83 data_log.90; do rados stat -p masterstaas.rgw.log $obj; done* masterstaas.rgw.log/data_log.91 mtime 2025-02-24T15:09:25.000000+0530, size 0 masterstaas.rgw.log/data_log.86 mtime 2025-02-24T15:01:25.000000+0530, size 0 masterstaas.rgw.log/data_log.87 mtime 2025-02-24T15:02:25.000000+0530, size 0 masterstaas.rgw.log/data_log.92 mtime 2025-02-24T15:11:01.000000+0530, size 0 masterstaas.rgw.log/data_log.89 mtime 2025-02-24T14:54:55.000000+0530, size 0 masterstaas.rgw.log/data_log.93 mtime 2025-02-24T14:53:25.000000+0530, size 0 masterstaas.rgw.log/data_log.83 mtime 2025-02-24T14:16:21.000000+0530, size 0 masterstaas.rgw.log/data_log.90 mtime 2025-02-24T15:05:25.000000+0530, size 0 *3. ceph cluster log :* 2025-02-22T04:18:27.324886+0530 osd.173 (osd.173) 19 : cluster [WRN] Large omap object found. Object: 124:b2ddf551:::data_log.93:head PG: 124.8aafbb4d (124.d) Key count: 1218170 Size (bytes): 297085860 2025-02-22T04:18:28.735886+0530 osd.65 (osd.65) 308 : cluster [WRN] Large omap object found. Object: 124:f2081d70:::data_log.92:head PG: 124.eb8104f (124.f) Key count: 1220420 Size (bytes): 295240028 2025-02-22T04:18:30.668884+0530 mon.master1 (mon.0) 7974038 : cluster [WRN] Health check update: 3 large omap objects (LARGE_OMAP_OBJECTS) 2025-02-22T04:18:31.127585+0530 osd.18 (osd.18) 224 : cluster [WRN] Large omap object found. Object: 124:d1061236:::data_log.86:head PG: 124.6c48608b (124.b) Key count: 1221047 Size (bytes): 295398274 2025-02-22T04:18:33.189561+0530 osd.37 (osd.37) 32665 : cluster [WRN] Large omap object found. Object: 124:9a2e04b7:::data_log.87:head PG: 124.ed207459 (124.19) Key count: 1220584 Size (bytes): 295290366 2025-02-22T04:18:35.007117+0530 osd.77 (osd.77) 135 : cluster [WRN] Large omap object found. Object: 124:6b9e929a:::data_log.89:head PG: 124.594979d6 (124.16) Key count: 1219913 Size (bytes): 295127816 2025-02-22T04:18:36.189141+0530 mon.master1 (mon.0) 7974039 : cluster [WRN] Health check update: 5 large omap objects (LARGE_OMAP_OBJECTS) 2025-02-22T04:18:36.340247+0530 osd.112 (osd.112) 259 : cluster [WRN] Large omap object found. Object: 124:0958bece:::data_log.83:head PG: 124.737d1a90 (124.10) Key count: 1200406 Size (bytes): 290280292 2025-02-22T04:18:38.523766+0530 osd.73 (osd.73) 1064 : cluster [WRN] Large omap object found. Object: 124:fddd971f:::data_log.91:head PG: 124.f8e9bbbf (124.3f) Key count: 1221183 Size (bytes): 295425320 2025-02-22T04:18:42.619926+0530 osd.92 (osd.92) 285 : cluster [WRN] Large omap object found. Object: 124:7dc404fa:::data_log.90:head PG: 124.5f2023be (124.3e) Key count: 1169895 Size (bytes): 283025576 2025-02-22T04:18:44.242655+0530 mon.master1 (mon.0) 7974043 : cluster [WRN] Health check update: 8 large omap objects (LARGE_OMAP_OBJECTS) Replica site: 1. *for obj in $(rados ls -p repstaas.rgw.log); do echo "$(rados listomapkeys -p repstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head -10* 432850 data_log.91 432384 data_log.87 432323 data_log.93 431783 data_log.86 431510 data_log.92 427959 data_log.89 414522 data_log.90 407571 data_log.83 151015 data_log.84 109790 data_log.4 2. *ceph cluster log:* grep -ir "Large omap object found" /var/log/ceph/ /var/log/ceph/ceph-mon.drhost1.log:2025-03-12T14:49:59.997+0530 7fc4ad544700 0 log_channel(cluster) log [WRN] : Search the cluster log for 'Large omap object found' for more details. /var/log/ceph/ceph.log:2025-03-12T14:49:02.078108+0530 osd.10 (osd.10) 21 : cluster [WRN] Large omap object found. Object: 6:b2ddf551:::data_log.93:head PG: 6.8aafbb4d (6.d) Key count: 432323 Size (bytes): 105505884 /var/log/ceph/ceph.log:2025-03-12T14:49:02.389288+0530 osd.48 (osd.48) 37 : cluster [WRN] Large omap object found. Object: 6:d1061236:::data_log.86:head PG: 6.6c48608b (6.b) Key count: 431782 Size (bytes): 104564674 /var/log/ceph/ceph.log:2025-03-12T14:49:07.166954+0530 osd.24 (osd.24) 24 : cluster [WRN] Large omap object found. Object: 6:0958bece:::data_log.83:head PG: 6.737d1a90 (6.10) Key count: 407571 Size (bytes): 98635522 /var/log/ceph/ceph.log:2025-03-12T14:49:09.100110+0530 osd.63 (osd.63) 5 : cluster [WRN] Large omap object found. Object: 6:9a2e04b7:::data_log.87:head PG: 6.ed207459 (6.19) Key count: 432384 Size (bytes): 104712350 /var/log/ceph/ceph.log:2025-03-12T14:49:08.703760+0530 osd.59 (osd.59) 11 : cluster [WRN] Large omap object found. Object: 6:6b9e929a:::data_log.89:head PG: 6.594979d6 (6.16) Key count: 427959 Size (bytes): 103773777 /var/log/ceph/ceph.log:2025-03-12T14:49:11.126132+0530 osd.40 (osd.40) 24 : cluster [WRN] Large omap object found. Object: 6:f2081d70:::data_log.92:head PG: 6.eb8104f (6.f) Key count: 431508 Size (bytes): 104520406 /var/log/ceph/ceph.log:2025-03-12T14:49:13.799473+0530 osd.43 (osd.43) 61 : cluster [WRN] Large omap object found. Object: 6:fddd971f:::data_log.91:head PG: 6.f8e9bbbf (6.1f) Key count: 432850 Size (bytes): 104418869 /var/log/ceph/ceph.log:2025-03-12T14:49:14.398480+0530 osd.3 (osd.3) 55 : cluster [WRN] Large omap object found. Object: 6:7dc404fa:::data_log.90:head PG: 6.5f2023be (6.1e) Key count: 414521 Size (bytes): 100396561 /var/log/ceph/ceph.log:2025-03-12T14:50:00.000484+0530 mon.drhost1 (mon.0) 207423 : cluster [WRN] Search the cluster log for 'Large omap object found' for more details. Regards, Danish

2 10

SSO for Dashboard
by Ed Krotee 04 Apr '25

04 Apr '25

Hello, We are on Ceph Squid (19.2.0) and have a question about enabling SS0 for the dashboard. In our org we use MS AD and Imprivata for SSO. We found an article from IBM on enabling it with either OAuth2 or SAML, but the article was for the older IBM supported release. Has anyone successfully enabled SSO in Squid? If so, can you offer any tips/steps? Thanks, Ed Krotee

2 1

Updating ceph to pacific and quince
by Iban Cabrillo 02 Apr '25

02 Apr '25

Dear cephers, We intend to begin the migration of our Ceph cluster from Octopus to Pacific and subsequently to Quincy. I have seen that from Pacific onwards, it is possible to automate installations with cephadm. One of the questions that arise is whether the clients (depending on the OpenStack version) in Octopus and the mons, mrgs, osds, mds etc. etc. in Pacific/Quincy will function correctly. The second is whether it is feasible, advisable to switch to cephadm/orch, since I have always performed updates manually for many, many years. And the third, if there is any 'official' guide available for these updates. Thanks in advance, I -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

4 9

reef 18.2.5 QE validation status
by Yuri Weinstein 02 Apr '25

02 Apr '25

Details of this release are summarized here: https://tracker.ceph.com/issues/70563#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - Laura approved? rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? rgw - Adam E approved? fs - Venky is fixing QA suite, will need to be added and rerun orch - Adam King approved? rbd - Ilya approved? krbd - Ilya approved? upgrade-clients:client-upgrade-octopus-reef-reef - Ilya please take a look. upgrade/pacific-x (reef) - can this be deprecated? Josh? Neha? upgrade/quincy-x (reef) - Laura, Prashant please take a look. ceph-volume - Guillaume is fixing it. TIA

12 22

Ceph Tentacle release - dev freeze timeline
by Yaarit Hatuka 02 Apr '25

02 Apr '25

Hi everyone, In previous discussions, the Ceph Steering Committee tentatively agreed on a Tentacle dev freeze around the end of March or mid-April. We would like to revisit this and check in with all the tech leads to assess the readiness level and ensure we're aligned on the timeline. Please provide your input on the current status and any potential concerns you may have regarding the freeze. Thanks, on behalf of the CSC.

1 1

reshard stale-instances
by Richard Bade 01 Apr '25

01 Apr '25

Hi Everyone, We've just completed undoing multisite to bring us back down to a single zone/site. Now I'm working through cleaning things up. Heads up, I'll probably have a few more questions over the next couple of weeks :) Firstly though, regarding "radosgw-admin reshard stale-instances list". This brings back a decent list mostly of some test buckets that were being deleted and recreated by an automated system. This has left over 150k stale instances for each of three buckets. I know these ones will be fine to delete as even if it messes up the bucket I can delete and re-create them. So my question is, how do I clean these up without using the big hammer "radosgw-admin reshard stale-instances delete" to remove all of them which would include some production buckets that I'm not quite sure how to verify are stale? Another thought that I had is maybe if I were to reshard all the buckets up to the next prime now that we're on single zone would that guarantee that all the stale instances are actually stale? Any thoughts and tips would be much appreciated. Thanks, Rich

2 3

Major version upgrades with CephADM
by Alex Petty 01 Apr '25

01 Apr '25

Hello, I'm seeking guidance on best practices for major version upgrades with Cephadm managed clusters. I have a production Pacfic 16.2.15 cluster that has had minor version upgrades with Cephadm, but always just a single step at a time. We plan to upgrade to Reef soon. Is it best to go straight to the most recent Reef release available, which is 18.2.4 at this time? Or could there be reasons to start at 18.2.0? Is the best source of documentation about major version upgrades the blog posts when a major release happens: https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/#upgrade ? Thanks, Alex

4 3

Prometheus anomaly in Reef
by Tim Holloway 01 Apr '25

01 Apr '25

I finally got brave and migrated from Pacific to Reef, did some banging and hammering and for the first time in a long time got a complete "HEALTH OK" status. However, the dashboard is still not happy. It cannot contact the Prometheus API on port 9095. I have redeployed Prometheus multiple times without result. I'm pretty sure that at one time there were no less than 3 different Prometheus containers running on one of the configured Prometheus servers, but now all I can get is the node-exporter. Worse, if I do: ceph orch reconfig prometheus I get: Error EINVAL: No daemons exist under service name "prometheus". View currently running services using "ceph orch ls" But if I do: ceph orch ls I get: prometheus ?:9095 0/1 - 116s ceph02.mousetech.com Suggestions?

2 40

[grafana] ceph-cluster-advanced: wrong title for object count
by Eugen Block 01 Apr '25

01 Apr '25

Hi Ankush, I found another (minor) Grafana issue. The "Ceph Cluster - Advanced" panel contains a graph for "object count", but its title is "OSD Type Count". I see that in Squid 19.2.0 (Grafana version 9.4.12) and 19.2.1 (Grafana 10.4.0), and the latest ceph-cluster-advanced.json [0] also still contains that mistake. Regards, Eugen [0] https://github.com/ceph/ceph/blob/89f21db321ddd404654b0064a61b0aa5428d6f7e/…

1 0

Purpose of the cephadm account
by dhivagar selvam 01 Apr '25

01 Apr '25

Hi, We have set up a ceph cluster via ceph-ansible. When we add a new osd or mds to the ceph cluster, the "cephadm" account is automatically created with "/bin/bash". 1. Is this account necessary? 2. Can I delete this account or do I need to change something in the ceph-ansible configuration? Thanks,

1 0