- ceph-users - lists.ceph.io

Recheck for "large omap objects" warning
by Jan Kasprzak 23 Sep '25

23 Sep '25

Hello, Ceph users, on my Ceph cluster with my custom rgw-like front-end client storing metadata to omap I got "1 large omap objects" warning. According to cluster log, the large object is osd.39 (osd.39) 2292 : cluster [WRN] Large omap object found. Object: 76:0bfe63c7:::idx.00000000004:head PG: 76.e3c67fd0 (76.10) Key count: 259173 Size (bytes): 12440304 I have increased the limit using ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 1000000 but the warning still remains. How can I get rid of the warning after increasing the size limit? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

radosgw multisite unable to initialize site config
by Kevin Hrpcek 22 Sep '25

22 Sep '25

Hey all, I've been having some problems with a relatively basic rgw multisite set up. I initially had it working a few weeks ago with a simple config but rebuilt it with ssl enabled for the endpoints and then started having some problems to work through. I eventually had that working but then tore it all down so I could verify the recreation steps but now I'm having problems with the multisite. There are a few different errors.... I set up the master zone without problems, but if I use the rgw module with a token and spec it fails to sync the sync user and the primary zone can't sync back from the secondary zone. A permission denied error for data syncing primary->secondary but secondary-> primary works because it has the credentials. So i used to be able to manually create the user on the secondary zone and get data syncing but the metadata wouldn't sync with the master showing an error message but when I check the mdlog on the primary it looks okay. 2025-09-11T15:15:18.771+0000 7fc29e96c640 1 req 14061042141473454829 0.001000005s failed to read mdlog history: (2) No such file or directory I ended up trying a full clean reset of rgw on both clusters so I deleted the daemons and all related pools. The primary starts up but I still have the problem of the sync user not being created on the secondary and now when I try to create a user on the secondary for syncing or just for the dashboard to work I get errors that I've tracked down to https://github.com/ceph/ceph/blob/62bcf65e8c0995783bb3e368909716346874ad62/… but am not quite following whats all going on in the source to determine why. # radosgw-admin user create --uid=dashboard --display-name=dashboard --system couldn't init storage provider # radosgw-admin user create --uid=sysuser-asdf --display-name=sysuser-asdf --access-key=xxxxxxxxxxxxx --secret-key=yyyyyyyyyyyyyyy --system --rgw-realm=asdf Unable to initialize site config. Any thoughts? It seems like I may have messed something up in the ceph config database from multiple tear downs and restarts of the multisite setup but I'm not seeing what it could be. ceph 19.2.1 using cephadm on alma 9 Thanks! Kevin

4 8

Experience with mixed-arch clusters?
by Alexander Schreiber 22 Sep '25

22 Sep '25

Hi, does anyone have experience with mixed-arch Ceph clusters? Specifically, mixing amd64 and aarch64 machines in the same cluster? I've got my hands one some NVidia Bluefield-2 DPUs: 2x 25G ports, 8x 64bit ARM cores, 16/32G ECC RAM. I'm pondering if sticking those into amd64 OSD machines and using them as non-storage (MDS, MON) nodes would make sense. Or would that end up mightily confusing the tooling, e.g. cephadm? Kind regards, Alex. -- "Opportunity is missed by most people because it is dressed in overalls and looks like work." -- Thomas A. Edison

2 2

SSL cert expired for download.ceph.com
by Boris 21 Sep '25

21 Sep '25

Hi, it looks like the SSL cert just expired.

1 0

advice needed for sizing of wal+db device for hdd
by Adrian Sevcenco 19 Sep '25

19 Sep '25

Hi! i'm trying to estimate the size requiremenent for a wal+db volume for 26 TB HDD while trying to minimize the administrative hassle for configuration (meaning that i would prefer a wal+db volume not separate db from wal) (RGW usage, 3 node cluster) so, for a jbod that can go up to 44 drives (not all drives present, at most up to 12) i have 2x 6.4TB NVME AFAIU for metadata (db) device, i need raid as losing metadata, means losing the associated OSD .. did i get this right? So, what would the the best practice to map db+wal to hdd OSD? should i do a mdraid from the 2 nvme and that to split in 12 partitions? should i split the nvme ssd in 12 namespaces and make individual mdraid for each and that map to osds? For a 6.4 TB NVME (divided to 12) and 26 TB OSD, the db/wal would be ~533 GB, so around 2.05% .. how terrible is this number for RGW usage (i get that the recommended is 4%) At what size the data is in danger if db is too small and when it become safe but only with performance degradation? Thanks a lot!!! Adrian

4 8

Upgrade CEPH 14.x -> 16.X + switch from Filestore to Bluestore = strange behavior
by Olivier Delcourt 19 Sep '25

19 Sep '25

Hi, After reading your posts for years, I feel compelled to ask for your help/advice. First, I need to explain the context of our CEPH cluster, the problems we have encountered, and finally, my questions. Thanks taking the time for reading me. Cheers, Olivier *** Background *** Our CEPH cluster was created in 2015 with version 0.94.x (Hammer), which has been upgraded over time to version 10.2.x (Jewel), then 12.x (Luminous) and then 14.x (Nautilus). The MONitors and CEPHstores have always run on Linux Debian, with versions updated according to the requirements for supporting the underlying hardware and/or CEPH releases. In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient. Each of the storage servers comes with HDDs whose size varies depending on the server generation, as well as SSDs whose size is more consistent but still varies (depending on price). The idea has always been to add HDD and SSD storage to the CEPH cluster when we add storage servers to expand it or replace old ones. At the OSD level, the basic rule has always been followed: one device = one OSD with metadatas (FileStore) on dedicated partitioned SSDs (up to 6 for 32 OSDs) and, for the past few years, on a partitioned NVMe RAID1 (MD). In total, we have: 100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB 84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.6 PiB 1.7 PiB 1.9 PiB 1.9 PiB 53.45 ssd 480 TiB 321 TiB 158 TiB 158 TiB 33.04 TOTAL 4.0 PiB 2.0 PiB 2.1 PiB 2.1 PiB 51.08 Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 3660.19751 root main_storage -11 1222.29883 datacenter DC1 -68 163.79984 host cephstore16 -280 109.09988 host cephstore28 -20 109.09988 host cephstore34 -289 109.09988 host cephstore31 -31 116.39990 host cephstore40 -205 116.39990 host cephstore37 -81 109.09988 host cephstore22 -71 163.79984 host cephstore19 -84 109.09988 host cephstore25 -179 116.39990 host cephstore43 -12 1222.29883 datacenter DC2 -69 163.79984 host cephstore17 -82 109.09988 host cephstore23 -295 109.09988 host cephstore32 -72 163.79984 host cephstore20 -283 109.09988 host cephstore29 -87 109.09988 host cephstore35 -85 109.09988 host cephstore26 -222 116.39990 host cephstore44 -36 116.39990 host cephstore41 -242 116.39990 host cephstore38 -25 1215.59998 datacenter DC3 -70 163.80000 host cephstore18 -74 163.80000 host cephstore21 -83 99.00000 host cephstore24 -86 110.00000 host cephstore27 -286 110.00000 host cephstore30 -298 99.00000 host cephstore33 -102 110.00000 host cephstore36 -304 120.00000 host cephstore39 -136 120.00000 host cephstore42 -307 120.00000 host cephstore45 -1 516.06305 root high-speed_storage -21 171.91544 datacenter xDC1 -62 16.84781 host xcephstore16 -259 14.00000 host xcephstore28 -3 14.00000 host xcephstore34 -268 14.00000 host xcephstore31 -310 14.30786 host xcephstore40 -105 14.30786 host xcephstore37 -46 30.68784 host xcephstore10 -75 11.67993 host xcephstore22 -61 16.09634 host xcephstore19 -78 11.67993 host xcephstore25 -322 14.30786 host xcephstore43 -15 171.16397 datacenter xDC2 -63 16.09634 host xcephstore17 -76 11.67993 host xcephstore23 -274 14.00000 host xcephstore32 -65 16.09634 host xcephstore20 -262 14.00000 host xcephstore29 -13 14.00000 host xcephstore35 -79 11.67993 host xcephstore26 -51 30.68784 host xcephstore11 -325 14.30786 host xcephstore44 -313 14.30786 host xcephstore41 -175 14.30786 host xcephstore38 -28 172.98364 datacenter xDC3 -56 30.68784 host xcephstore12 -64 16.09200 host xcephstore18 -67 16.09200 host xcephstore21 -77 12.00000 host xcephstore24 -80 12.00000 host xcephstore27 -265 14.39999 host xcephstore30 -277 14.39990 host xcephstore33 -17 14.39999 host xcephstore36 -204 14.30399 host xcephstore39 -319 14.30399 host xcephstore42 -328 14.30396 host xcephstore45 Our allocation rules are: # rules rule main_storage_ruleset { id 0 type replicated min_size 1 max_size 10 step take main_storage step chooseleaf firstn 0 type datacenter step emit } rule high-speed_storage_ruleset { id 1 type replicated min_size 1 max_size 10 step take high-speed_storage step chooseleaf firstn 0 type datacenter step emit } All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1). This CEPH cluster is currently only used for RBD. The volumes are used by our ~ 1,200 KVM VMs. *** Problems *** Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem). * First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start. We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore. * Second problem: Since our failure domain is a DC, we had to upgrade a DC and then wait for it to recover (~500 TB net). SSD storage recovery takes a few hours, while HDD storage recovery takes approximately three days. Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us. I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help). Once our VMs were shut down (crashed), the cluster completed its recovery (HDD-type OSDs) and, curiously, the SSD-type OSDs began to ‘empty’. The day after that, we began updating the storage servers in our second DC, and the phenomenon started again. We did not wait until we reached full_ratio to shut down our virtualisation environment and this time, the ‘SSD’ OSDs began to ‘empty’ after the following commands: ceph osd unset noscrub && ceph osd unset nodeep-scrub. In fact, we used to block scrubs and deep scrubs during massive upgrades and recoveries to save I/O. This never caused any problems in FileStore. It should be added that since we started using the CEPH Cluster (2015), scrubs have only been enabled at night so as not to impact production I/O, via the following options: osd_recovery_delay_start = 5, osd_scrub_begin_hour = 19, osd_scrub_end_hour = 7, osd_scrub_sleep = 0.1 (the latter may be removed since classes are now available) . After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer). *** Questions / Assumptions / Opinions *** Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution and is not desirable in the medium term, but we are dependent on recovery time (and, in addition, on the issue I am presenting to you here). Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ? I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case: root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’, This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up? (It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).) Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.

5 9

Scrub errors after OSD_TOO_MANY_REPAIRS: how to recover?
by Michel Jouvin 18 Sep '25

18 Sep '25

Hi, We experienced one week ago problems with a dying OSD resulting in OSD_TOO_MANY_REPAIRS, unfortunately unnoticed (it seems our monitoring system is not notifying these errors properly). When we realized the error, we removed the problematic OSD (ceph orch osd rm --replace), despite the scrubbing errors: the resulting backfills succeeded but did not fix the scrub errors. The collegue who took care of this problem decided to lauch a `ceph pg repair` on the 3 PGs with reported inconsistencies but it doesn't seem to converge. 'ceph -s' still reports : 3 active+clean+scrubbing+deep+inconsistent+repair after a few hours and for at least one of the PG, there is the following message every 3s: Sep 18 12:30:55 ceph-76212 ceph-mon[2506]: osd.72 pg 11.e2d Deep scrub errors, upgrading scrub to deep-scrub Not sure if it is the sign of a problem or just because the operation is ongoing. I'm looking for advices on what to do to move forward. There was not yet a report from users of an impact of this but it doesn't mean there is none... The affected pool is storing RBD volumes (from OpenStack Cinder). A side issue is that we have 'osd_scrub_auto_repair=true' so I'd expect the repair to start automatically but it doesn't seem it was the case... Thanks in advance for any advice. Best regards, Michel

3 3

Failing upgrade 18.2.7 to 19.2.3 - failing to activate via raw takes 5 minutes (before proceeding to lvm)
by Mikael Öhman 18 Sep '25

18 Sep '25

I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3. This time again the ceph-volume activate step is taking to long, triggering failures due to systemd service timing out so the orch daemon fails (though the osd does eventually come up, the daemon is still dead, and upgrade halts). I can also reproduce the slowdown of startup with cephadm ceph-volume raw list (I don't use raw devices, but the ceph-volume activation method hardcodes checking raw first https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/… ) That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 ! I have 42 spinning drives per host (with multipath). It's spending all of it's time in the new method: self.exclude_lvm_osd_devices() and the list of items to scan, given all the duplication from multipath + and mapper names, it ends up with 308 items to scan in my setup. With good old print debugging, i found that while the threadpool speeds things up a bit, it simply takes to long to construct all those Device() objects. In fact, just creating a single Device() object, since it needs to call disk.get_devices() at least once, since this list does not include all devices, it filters out things like "/dev/mapper/mpathxx" from the list, but the code always regenerates (the same) device list if the path isn't found: if not sys_info.devices.get(self.path): sys_info.devices = disk.get_devices() will now force it to re-generate this list >400 times (initial 32 times in parallel, followed by about 400 more which will never match the device name). In the end, it's again O(n^2) computational time to list these raw devices with ceph-volume. So with 32 threads in the pool, it's also now requires running heavy load for 5 minutes before completing this trivial task every time the deamon needs to start.

3 4

September Ceph Science User Group Virtual Meeting
by Enrico Bocchi 18 Sep '25

18 Sep '25

Dear All, We will be having a Ceph science/research/big cluster call on Tuesday, September 23rd, as per ceph.io calendar: https://ceph.io/en/community/meetups/ Kevin Hrpcek from UCAR will report on his recent experience migrating from the Ceph CSI to the new Ceph CSI Operator. This is an informal open call for community members primarily from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc. There is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific, this can be added to the Pad linked below. If you are interested in giving a short talk during one of the next meetings, please reach out to me. Virtual event details: Sept 23rd, 2025 7am PDT 9am Central US (CDT) 14:00 UTC 16:00 Central European Summer Time -- please double-check the meeting time for your time zone -- Join the meeting at https://meet.jit.si/ceph-science-wg The meeting will be recorded and posted to the Ceph YouTube channel. Event Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20250923 Main Pad for the Ceph Science User Group: https://pad.ceph.com/p/Ceph_Science_User_Group_Index See you next Tuesday! Enrico

1 0

Per-RBD-image stats
by William David Edwards 18 Sep '25

18 Sep '25

Hi, What do people use to store per-RBD-image I/O stats? To retrospectively see which VM was keeping a Ceph cluster busy in terms of I/O, I'd like to track `rbd perf image iostat` and `rbd perf image iotop`-like metrics. I've looked at the LibreNMS plugin [1], Prometheus metrics in the manager [2], and Ceph ingestion in Elasticsearch [3]. None of them seem to export per-RBD metrics. I can't find much information about this use case on the mailing list either, though it's hard for me to imagine I'm the only one with this want. How do other users handle this? William David Edwards [1]: https://github.com/librenms/librenms-agent/blob/master/agent-local/ceph [2]: https://docs.ceph.com/en/latest/mgr/prometheus/ [3]: https://www.elastic.co/docs/reference/integrations/ceph

5 4