- ceph-users - lists.ceph.io

Slow initial boot of OSDs in large cluster with unclean state
by Thomas Byrne - STFC UKRI 28 Jan '25

28 Jan '25

Hi all, On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today... Looking at the logs of a new OSD when it's starting: 2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs. I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling. The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me. Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers? (There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.) So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation. Thanks, Tom

8 21

Seeking Participation! Take the new Ceph User Stores Survey!
by Laura Flores 28 Jan '25

28 Jan '25

Hi all, The Ceph User Council is conducting a survey to gather insights from community members who actively use production Ceph clusters. We want to hear directly from you: *What is the use case of your production Ceph cluster?* Since its official Argonaut release in 2012, Ceph has grown significantly in features and user adoption. By learning about your use cases, we aim to understand Ceph’s strengths and limitations in performance, scalability, and usability. This feedback will help inform future improvements to Ceph. Our ultimate goal is to enhance the Ceph community's shared knowledge by publishing real-world user stories on our website. These stories will serve as valuable resources for both current and future users, demonstrating the diversity and potential of Ceph in production environments. All responses are anonymous unless the participant willingly shares their contact information. The Ceph User Council will contact participants who opt to share their contact information before anything is officially published. We have not yet determined when the survey will close, but we will give a heads up before doing so. Take the survey here! https://docs.google.com/forms/d/e/1FAIpQLSe66NedXh4gHLgk9G45eqP5V2wHlz4IKqR… Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

4 12

Mix NVME's in a single cluster
by Bruno Gomes Pessanha 28 Jan '25

28 Jan '25

I have a Ceph Reef cluster with 10 hosts with 16 nvme slots but only half occupied with 15TB (2400 KIOPS) drives. 80 drives in total. I want to add another 80 to fully populate the slots. The question: What would be the downside if I expand the cluster with 80 x 30TB (3300 KIOPS) drives? Thank you! Bruno

4 6

radosgw daemons with "stuck ops"
by Reid Guyett 28 Jan '25

28 Jan '25

Hello, We are experiencing slowdowns on one of our radosgw clusters. We restart the radosgw daemons every 2 hours and things start getting slow after an hour and a half. The avg get/put latencies go from 20ms/400ms to 1s/5s+ according to the metrics. When I stop traffic to one of the radosgw daemon by setting it to DRAIN in HAProxy and HAProxy reports 0 sessions to the daemon, I still see the objecter_requests still going up and down 15 min later. linger_ops seems to stay at a constant 10 the whole time. > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 46 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 211 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 427 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 26 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 190 > The requests say they are "call rgw.bucket_list in=234b" and mostly reference ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.N" where N is 1-149 (the shard count for the bucket). > { > "tid": 2583119, > "pg": "8.579482b0", > "osd": 208, > "object_id": > ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.124", > "object_locator": "@8", > "target_object_id": > ".dir.5e9bc383-f7bd-4fd1-b607-1e563bfe0011.886814949.1.124", > "target_object_locator": "@8", > "paused": 0, > "used_replica": 0, > "precalc_pgid": 0, > "last_sent": "1796975.496664s", > "age": 0.350012616, > "attempts": 1, > "snapid": "head", > "snap_context": "0=[]", > "mtime": "1970-01-01T00:00:00.000000+0000", > "osd_ops": [ > "call rgw.bucket_list in=234b" > ] > }, > I don't think it should be other rgw processes because we have this daemon's ceph.conf set to disable the other threads. > rgw enable gc threads = false > rgw enable lc threads = false > rgw dynamic resharding = false > When I restart the service while it is still in a DRAINED state in HAProxy, checking the objecter_requests yields 0 even a few minutes after it has been up. > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > [root@rgw1 ~]# ceph daemon client.rgw.rgw1 objecter_requests | jq '.ops | > length' > 0 > Any thoughts on why these ops appear to be stuck/recurring until restarting the daemon? I think this is related to our performance issues but I don't know what the fix is. The rgws are 18.2.4 running as containers in Podman on Debian 11. Our other clusters do not exhibit this behavior. Thanks! Reid

2 2

Understanding how crush works
by Andre Tann 28 Jan '25

28 Jan '25

Hi list, I have a problem understanding on how crush works when the crush map changes. Let's take a pool with some data in it, and a crush map that enables a client to calculate itself where a particular chunk is stored. Now we add more OSDs, which means, the crush map changes. Now most objects are misplaced, given the new crush map. If the clients wants a particular chunk, it takes the modified map, but as the chunk is misplaced, it won't find it where the crush algorithm points to. How can the client know which crush map to consider when doing the calculation? Do the clients keep several versions of the map, and try them one after the other? Thanks for some hints on this. -- Andre Tann

2 1

Ceph Steering Committee Notes 2025-01-27
by Dan van der Ster 28 Jan '25

28 Jan '25

Hi all, Here is a summary of the CSC meeting Jan 27, 2025. Full notes are available at https://pad.ceph.com/p/csc-weekly-minutes Component Leads Poll re: Workload and Bottlenecks Dan proposes an informal poll for component leads to identify workload distribution, bottlenecks, and areas where community contributions are most needed. Venky shares that the CephFS team uses GitHub's "Assigned" field to track PR reviews and suggests exploring good and bad aspects of roles to involve more community support. LRC On-Call Assistance DavidG seeks improved mechanisms to involve developers and leads during LRC outages. In a recent case, an outage was related (again) to iSCSI, which is missing a lead maintainer. It was agreed to have a follow up discussion on David's plans for modernizing the upstream infra in the next meeting. Tentacle Roadmap Tentacle development freeze and release candidate timelines are not finalized. Historically, dev freeze occurs by the end of January and RC by the end of March. Discussions are to continue at the next CSC meeting. BOTO3 Issue A BOTO3 change caused S3 client issues, with backports and fixes in progress. Relevant links and PRs for tracking the problem are shared. Strategy Working Group Yehuda proposes forming a strategy working group to address high-level Ceph planning over a 3–5 year horizon, as existing forums like CSC and Dev Summit are too technical for such discussions. Discussion to continue next meeting. Cephalocon 2024 Recordings Most recordings from Cephalocon 2024 are now available on YouTube. Announcements to the mailing list are planned.

1 0

Cephalocon 2024 Recordings Available
by Matt Vandermeulen 27 Jan '25

27 Jan '25

Hi folks! The Cephalocon 2024 recordings are available on the YouTube channel! - Channel: https://www.youtube.com/@Cephstorage/videos - Cephalocon 2024 playlist: https://www.youtube.com/watch?v=ECkgu2zZzeQ&list=PLrBUGiINAakPfVfFfPQ5wLMQJ… Thanks, Matt

1 0

unmatched rstat rbytes on single dirfrag
by Frank Schilder 27 Jan '25

27 Jan '25

Hi all, I see error messages like these in the logs every now and then: 10:14:44 [ERR] unmatched rstat rbytes on single dirfrag 0x615, inode has n(v2211 rc2038-01-18T21:22:13.000000+0100 b2506575730 9676264=3693+9672571), dirfrag has n(v2211 rc2025-01-24T10:14:44.628760+0100 b30517 102=3+99) 10:14:44 [ERR] unmatched fragstat size on single dirfrag 0x615, inode has f(v67 m2025-01-24T10:14:44.628760+0100 9676556=3819+9672737), dirfrag has f(v67 m2025-01-24T10:14:44.628760+0100 102=3+99) 10 /var/log/ceph/ceph.log-20250123:2025-01-22T17:56:18.060011+0100 mds.ceph-11 (mds.6) 4 : cluster [ERR] unmatched fragstat on 0x641, inode has f(v14 m2025-01-18T11:59:21.676346+0100 2801=913+1888), dirfrags have f(v0 m2025-01-18T11:59:21.676346+0100 427=9+418) /var/log/ceph/ceph.log-20250123:2025-01-22T17:56:18.076061+0100 mds.ceph-11 (mds.6) 5 : cluster [ERR] inconsistent rstat on inode 0x641, inode has n(v194 rc2038-01-07T22:42:17.000000+0100 b46230814357 2802=913+1889), directory fragments have n(v0 rc2032-04-29T16:39:38.000000+0200 b46204251164 428=9+419) How critical are these? Everything seems to work normal. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 5

Error ENOENT: Module not found
by Devender Singh 27 Jan '25

27 Jan '25

Hello all Any quick fix for … root@sea-devnode1:~# ceph orch ls Error ENOENT: Module not found Regards Dev

6 11

Watcher Issue
by Devender Singh 26 Jan '25

26 Jan '25

Hello Seeking some help if I can clean the client mounting my volume? rbd status pool/image Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088 Issue: pod is failing in init- state. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 96s (x508 over 24h) kubelet MountVolume.MountDevice failed for volume "pvc-3a2048f1" : rpc error: code = Internal desc = rbd image k8s-rgnl-disks/csi-vol-945c6a66-9129 is still being used It shows above client, but there is no such volume… Another similar issue… on dashboard… CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on sea-prod-host01 will be full in less than 5 days based on the 48 hour trailing fill rate. Whereas nothing is mounted, I mapped one image yesterday using red map and then unmapped and unmounted everything but it been more than 12hours now, still showing the message.. CEPH version: 18.2.4 Regards Dev

4 13