- ceph-users - lists.ceph.io

write latency increase after upgrade from pacific to quincy
by Nima AbolhassanBeigi 15 May '25

15 May '25

Hi dear ceph community We have encountered an issue with our ceph cluster after upgrading from v16.2.13 to v17.2.7. The issue is that the write latency on OSDs has increased significantly and doesn't seem to plummet back down. The average write latency has almost doubled, and this has happened since we upgraded the OSDs. If anybody could help figure this out. Kind regards Nima

2 4

18.2.6 : OSD(s) experiencing slow operations in BlueStore
by Michel Jouvin 15 May '25

15 May '25

Hi, Since our upgrade to 18.2.6 2 days ago, our cluster is reporting the warning "1 OSD(s) experiencing slow operations in BlueStore": [root@dig-osd4 bluestore-slow-ops]# ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.247 observed slow operation indications in BlueStore I have never seen this warning before so I've the feeling it is somehow related to the upgrade and it doesn't seem related to the regression mentioned in another thread (that should result in an OSD crash). Googling quickly, I found this reported on 19.2.1 with SSD where in my case it is an HDD. I don't know if the workaround mentioned in the issue (bdev_xxx_discard=true) also applies to 18.2.6... Did somebody saw this in 18.2.x? Any recommandation? Our plan was, according to best practicies described recently in another thread to move from 18.2.2 to 18.2.6 and then from 18.2.6 to 19.2.2... Will 19.2.2 clear this issue (at the risk of others as it is probably not widely used)? Best regards, Michel

8 12

reef radosgw returns 503/slow down when creating a bucket with location
by Boris 14 May '25

14 May '25

Hi, we have a multi zonegroup setup and since after the upgrade from pacific to reef we encountered an issue with creating buckets. When you try to create a bucket in another zonegroup and set the `bucket_location` (Region to create bucket in) parameter in the s3cmd, the radosgw answer with (503 (ServiceUnavailable)) 'reason': 'Slow Down' After removing the bucket_location from the config it just works. I use s3cmd with a really bare bones config. [default] # bucket_location = zg-b host_base = zg-b.test.dev host_bucket = %(bucket)s.zg-b.test.dev access_key = xxx secret_key = yyy website_endpoint = https://%(bucket)s.s3-website-zg-b.test.dev Is this a bug or expected behavior. Cheers Boris

2 1

Re: check bucket index on large bucket leads to laggy index PGs
by Boris 13 May '25

13 May '25

> > resharding the bucket is indeed the solution. while resharding does > have to read all of the keys from the source index objects, it doesn't > read all of them at once. writing these keys to the target bucket > index objects is the more expensive part, but those are different > objects/pgs and should be better distributed Ah great. Will try that. Ensure the index pool is really on only SSDs. I’ve seen crush rules not > specifying device class. > Yes they are. On dedicated NVMEs that we use for the meta pools (the listing I've sent in slack). Do you have autoresharding disabled? Versioned objects? Can you do a > bilog trim? Could you preshard a new bucket and move the objects? > No autoresharding enabled. We wanted to test sharding in the multisite setting before we enable it (and reshard all the buckets that need it in a controlled way). Yes, the bucket has versioned objects. We are in the process of deleting it, but the deletion is now running since two days and the bucket index is still very large. I can try to do a bilog trim. Need to read up what it does and how to do it. I could move the data to a new preshareded pool, but it feels like the bucket is somehow broken, because deleting is now working as I expect it. I will try to reshard the bucket tonight and hope it will work out. The explanation from Casey sounds promising. As I have a lot more buckets with a lot more objects (according to the bucket index) this needs to be done anyway. Cheers Boris Am Fr., 9. Mai 2025 um 19:21 Uhr schrieb Anthony D'Atri < anthony.datri(a)gmail.com>: > Ensure the index pool is really on only SSDs. I’ve seen crush rules not > specifying device class. > > Do you have autoresharding disabled? Versioned objects? Can you do a > bilog trim? Could you preshard a new bucket and move the objects? > > > On May 9, 2025, at 12:54 PM, Boris <bb(a)kervyn.de> wrote: > > > > Hi, > > > > I have a bucket that got >20m index entries but only got 11 shards. > > > > When I try to run a radosgw-admin bucket check the PGs that hold the > index start to become laggy after a couple of seconds. I need to stop it > because it kills the whole object storage. > > > > This is a latest reef cluster and the master of a multisite which only > replicates the metadata (1 realm, multiple zonegroups, one zone per > zonegroup). > > > > Any ideas what I can do? > > I fear to reshard the bucket, because I am not sure if I can stop the > resharding if the PGs become laggy. > > > > Cheers > > Boris > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

CephPGImbalance warning for different OSD sizes
by Eugen Block 13 May '25

13 May '25

Hi *, since the question came up yesterday on this list, I decided to share our workaround. I created a tracker issue [0] as well as a blog post [1] with some more details. The current Prometheus module relies on uniform OSD sizes, which is not very common, at least not from our experience or the reports on this list. So we added a new metric to the mgr prometheus module and modified the alert expression which only compares OSDs of the same size (since the crush weight is calculated similarly, we just called the metric osd_crush_weight). This is a bit hacky, not persistent across updates etc., but it has worked great so far. It would be great if the mgr module could be improved. I'm sure there are more elegant ways to do that, but with this approach we didn't need to introduce anything new, just utilized what was already there. Best regards, Eugen [0] https://tracker.ceph.com/issues/71310 [1] https://heiterbiswolkig.blogs.nde.ag/2025/05/13/cephadm-pg-imbalance/

1 0

reef upgrade 2.2 to 2.7 - slow operations in bluestore
by Steven Vacaroaia 13 May '25

13 May '25

Hi, After a cephadm upgrade from 18.2.2 to 18.2.7 that worked perfectly, I am noticing lots ( 42 out of 161 OSDs ) "slow operations in bluestore" errors My cluster has all 3 types of OSDs ( NVME, SSD, HDD+ journaling) I found some articles mentioning below setting but it did not help Anyone else having this issue ? Thanks Steven bdev_enable_discard: "true" # quote bdev_async_discard_threads: "1" # quote

4 4

Ceph 19.2.2 ceph-exporter and lvm zap
by Kristaps Cudars 13 May '25

13 May '25

Hello, After upgrading to 19.2.2 ceph-exporter container stopped exposing date. As workaround implemented exclude_perf_counters true Is anyone else experiencing this? And it seems that this is true for 19.2.1/2 from cephadm shell > ceph-volume lvm zap is not working. If you install “dnf install ceph-volume” than from OS ceph-volume lvm zap is working. This is expected and must be performed from OS on 19.2 or is it a bug?

2 2

cephfs: Moving directories with snapshots
by Kyriazis, George 12 May '25

12 May '25

Hello, I have a question regarding how snapshots are being treated on a directory that is moved out of a snapshot’ed are. My example: Let’s say I have a directory structure as follows: A/B/C A/D I have created snapsho schedules on A/B, and I want to move (with ‘mv’) directory C so it becomes A/D/C. There are no snapshots on A/D (yet). Will the existing snapshots “move” with the directory, or will they stay under the A/B hierarchy? My concern is that the C directory is 35TB, and if I mv the directory under A/D I may end up duplicating space. Ie, will I end up with 35TB stored in the old snapshots under A/B, and the actual 35TB on the live data under A/D/C ? Thank you for the clarification! George

1 0

Slow osd + 30% deviates (toward higher pgs or lower pgs numbers on osd)
by Devender Singh 12 May '25

12 May '25

Hello all Seeking some help to solve below… 1. # ceph health detail HEALTH_WARN 2 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 2 OSD(s) experiencing slow operations in BlueStore osd.10 observed slow operation indications in BlueStore osd.21 observed slow operation indications in BlueStore Having some other OSD’s with same size with more PG’s but only complaining on below… # ceph osd df |sort +17|egrep -i "^10|^21" 21 ssd 6.98630 1.00000 7.0 TiB 2.7 TiB 2.6 TiB 17 MiB 27 GiB 4.3 TiB 38.22 0.85 103 up 10 ssd 6.98630 1.00000 7.0 TiB 4.0 TiB 4.0 TiB 21 MiB 35 GiB 2.9 TiB 57.97 1.28 123 up 2. What does deviate means towards? holding higher number of pgs or Lower? OSD osd.6 on host1 deviates by more than 30% from average PG count. OSD osd.5 onhost2 deviates by more than 30% from average PG count. # ceph osd df |sort +17|egrep -i "^ 5|^ 6” 5 ssd 6.98630 1.00000 7.0 TiB 2.8 TiB 2.8 TiB 11 MiB 32 GiB 4.2 TiB 39.91 0.88 97 up 6 ssd 6.98630 1.00000 7.0 TiB 3.2 TiB 3.1 TiB 19 MiB 48 GiB 3.8 TiB 45.65 1.01 103 up Regards Dev

2 1

Notes from Ceph Steering Committee Weekly 2025-05-12
by Patrick Donnelly 12 May '25

12 May '25

Hello, Here are the minutes from today's meeting. - [DavidG] Offer for Z hardware. Non-mandatory PR tests? - tl;dr - Sure if they're willing to manage it. Run on main branch once a day? Would need enough hardware to match what we have for ARM? If they want to build it, sure. IBM Cloud? Crunch numbers on bare minimum useful new architecture hardware. - [DavidG] GitHub Security Improvements - Enforcing 2FA - yes - Downgrade non to outside collaborators? - One of the recommendations here: https://docs.google.com/document/d/1tEPuFv_UDYUmYcejMNZgTgZALI7jTSSye4wlgHC… - Periodic user removal from GitHub org? What interval of non-contribution? - 1yr no PR or Review. Will need to craft some API calls. - [Patrick] new pending release notes format - https://tracker.ceph.com/issues/71198 - Will move forward with this. - [Yehuda] cephx changes update - [Matt] revisit ceph-dashboard tests, there appeared to be a majority opinion these could be moved to teuthology (where they already can run); but we learned last week that "it was decided" not to? could we get a formal vote on this? that would allow everyone a voice - Seeking a resolution on how to run tests of this nature in the future: - Pull out Github CI and make part of all component lead QA testing in teuthology. - Tabled for dedicated discussion/meetign with Ernesto and other stakeholders. There seems to be somewhat broad agreement that these tests would be better in the teuthology QA suite going forward. - [Yuri] next quincy release for bluestore bugfix - https://github.com/ceph/ceph/pull/63122 - TODO: archive current quincy HEAD and reset to last bug fix release. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0