- ceph-users - lists.ceph.io

Ceph Steering Committee Meeting 2025-11-03
by Casey Bodley 10 Nov '25

10 Nov '25

we covered two topics today that generated a lot of good discussion, but neither reached a consensus: AI Code contributions * current lack of clear policy on submissions containing AI-generated code. do we want one? what are the concerns/risks? how do we enforce? * requires further discussion Tentacle updates? * merge conflicts in https://github.com/ceph/ceph/pull/66096 * lrc upgrade? ** very risky with pending lab move next week. after the move, lrc may not be available for upgrade until 2026? ** should this block the release or not? requires further discussion

4 4

Weekly Ceph Community Newsletter
by Anthony Middleton 07 Nov '25

07 Nov '25

Hey Ceph Friends, I hope you're all doing well after last week's exciting time in Vancouver for Cephalocon. Videos are coming soon! As a first-time attending Cephalocon, the highlight of the event was meeting so many wonderful community members in person. I felt welcomed into the community, and I look forward to continuing to be a part of it. I have a few updates and opportunities for you to get involved. Let's dive in! Your Voice Matters: Take Our Community Survey! At last week's Cephalocon, Matthew Leonard, Chair of the Ceph Foundation, delivered the Foundation Update covering the good, the bad, and the ugly. He shared insights into how the Foundation fosters innovation and why YOUR participation matters. We need your help to shape the future of the Foundation! Take just *3 minutes* to complete our Ceph Community Engagement Survey. Please share with us what's working, what's not, and what needs improvement. This is your Foundation, and it's only as strong as the people who build it together. Survey link: https://t.ly/Engagement-Survey25 New Slack Channel Alert: #go-ceph Calling all Go developers! We've launched a new *#go-ceph* Slack channel <https://join.slack.com/t/ceph-storage/shared_invite/zt-32hkefbs5-f6qZDZLd5U…>. This is the definitive way to interact with Ceph in Go, and it will only become more important as our Go ecosystem grows. Join the conversation and connect with fellow Gophers working with Ceph! *Do you have suggestions for other Slack channels? Message or email me anytime!* Help Us Build Our Reddit Community We're seeking moderators to help manage our new Ceph subreddit <https://www.reddit.com/r/ceph_storage/>! After our previous subreddit was unfortunately banned due to a bot attack, we're starting fresh and need passionate community members to help keep things running smoothly. Interested in being part of the mod team? *Reach out to me directly* and let's chat! Share Your Story: Ceph Contributors Intake Form Along with the new subreddit, there are many ways to get involved with the Foundation, and here's how to do it. If you're interested in becoming more engaged with the Ceph Community, we have an intake form to help start the process. This form is brief and will let us understand some of the community's needs. The success of our Foundation depends on the support and involvement of our community. Complete the intake form here: https://t.ly/contributors Quick Action Items: - ✅ Complete the Community Engagement Survey <https://t.ly/Engagement-Survey25> (3 minutes!) - ✅ Join the new *#go-ceph* Slack channel - ✅ Contact me to become a Reddit moderator - ✅ Fill out the Contributors Intake Form <https://t.ly/contributors> Thanks for being a fantastic part of the Ceph community! Together, we're building something special. Stay awesome, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph> *Questions, suggestions, or just want to chat? Don't hesitate to reach out!*

1 0

How important is the "default" data pool being replicated for CephFS
by Mikael Öhman 07 Nov '25

07 Nov '25

The recommendations for cephfs is to make a replicated default data pool, and adding any EC data pools using layouts: https://docs.ceph.com/en/latest/cephfs/createfs/ > If erasure-coded pools are planned for file system data, it is best to configure the default as a replicated pool to improve small-object write and read performance when updating backtraces. I have an cephfs that unfortunately wasn't set up like this: they just made an EC pool on the slow HDDs as the default, which sounds like the worst case scenario to me. I would like to add a NVME data pool to this ceph fs, but recommended gives me pause on if i should instead go through the hassle of creating a new cephfs and migrating all users. I've tried to run some mdtest with small 1k files to see if i could measure this difference, but speed is about the same in my relatively small tests so far. I'm also not sure what impact I should realistically expect here. I don't even know if creating files counts as "updating backtraces", so my testing might just be pointless. I guess my core question is; just how important is this suggestion to keep the default data pool on replicated NVME? Setup: 14 hosts x 42 HDD + 3 NVMEs for db/wal 2*2x25 GbitE bonds 12 hosts x 10 NVME. 2*2x100 GbitE bonds Old CephFS setup: - metadata: replicated NVME - data-pools: EC 10+2 on HDD (i plan to add a EC NVME pool here via layouts) New CephFS setup as recommended: - metadata: replicated NVME - data-pools: replicated NVME (default), EC 8+2 on HDD via layout, EC 8+2 on NVME via layout. Ceph 18.2.7 Best regards, Mikael

4 5

HDD replacements with shared NVME lvm for DB - what am I missing here
by Mikael Öhman 07 Nov '25

07 Nov '25

My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per NVME). It's all using ceph orch, containerized setup, using LVMs, so it's probably the most conventional HDD based setup one can do. I have the basic osd spec: --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: data_devices: rotational: 1 db_devices: rotational: 0 But unless the NVME is completely empty orch will just never pick it up (which I of course don't want to do, as it brings down 13 other OSDs). Instead orch just flat out ignores the requirement that db_devices must go on rotational: 0 and incorrectly suggests the broken setup: ceph orch apply -i osd_spec_osd02.yml --dry-run ... ################ OSDSPEC PREVIEWS ################ +---------+----------+-------------+---------------------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+-------------+---------------------+----+-----+ |osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau |- |- | +---------+----------+-------------+---------------------+----+-----+ which is the worst outcome, so I have to set all specs to be unmanaged, since otherwise they do the wrong thing automatically. So, most of https://docs.ceph.com/en/squid/cephadm/services/osd/ can just be ignored, since it won't work. Instead I'm stuck with this long complicated procedure, ensuring all specs are unmanaged, recreate the db lvm manually making up a UUID for it so that it matches the rest of the osds, then manually enter a shell, copy over the client.bootstrap-osd keyring, run this huge ceph-volume lvm prepare --bluestore --no-systemd --osd-id 12345 --data /dev/mapper/mpathxx --block.db /dev/ceph-2e401d48-931b-4529-88c0-d36424560xxx/osd-db-61264cc4-0f40-458b-a4d1-08709b919xxx manually adding the daemon, starting it. I don't love this procedure and it's hard to explain all the things that can go wrong to new staff. I really miss the basic step by step of how to perform the most common operation; replacing a hdd with a ssd db partition. I have tried things with --replace and --no-destroy (a flag that isn't documented at all), but I can't understand from the documentation what it wants from me when I have separate DB partitions. -------- A bit of a tangent, butI also don't understand "REJECT REASONS" from ceph orch device ls Insufficient space (<10 extents) on vgs, LVM detected Why even bother looking at extents on VGs if the presence of LVM by itself already rejects the drive. In my case, it does detect that nvme0n1 has sufficient space on the vgs: mimer-osd02 /dev/mapper/mpathau hdd 12.7T Yes 10m ago mimer-osd02 /dev/nvme0n1 ssd KCM61VUL800G_7170A009TM38 745G No 10m ago Has a FileSystem, LVM detected mimer-osd02 /dev/nvme1n1 ssd KCM61VUL800G_7170A004TM38 745G No 10m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected mimer-osd02 /dev/nvme2n1 ssd KCM61VUL800G_7170A007TM38 745G No 10m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected but, "LVM detected" already rejects it. Checking the code (src/ceph-volume/ceph_volume/util_device.py): self.available_lvm, self.rejected_reasons_lvm = self._check_lvm_reject_reasons() self.available_raw, self.rejected_reasons_raw = self._check_raw_reject_reasons() self.available = self.available_lvm and self.available_raw self.rejected_reasons = list(set(self.rejected_reasons_lvm + self.rejected_reasons_raw)) So apparently, the device must be available both as RAW (i.e. have no vgs) and also have enough space on its LVMs (i.e. have enough vgs). This code doesn't make any sense at all. In addition, is also checks disk.blkdid for any TYPE field @property def has_fs(self) -> bool: self.load_blkid_api() return 'TYPE' in self.blkid_api and for a device in lvm it would have something like {'UUID': '3OWtic-3RCI-G3g5-7M6D-pKpb-eMJe-X0zgif', 'TYPE': 'LVM2_member'} so again it doesn't make any sense to ever have _check_lvm_reject_reasons, since it must never be a LVM2_member anyway. Best regards, Mikael

4 7

[reef-18.2.7] radosgw token cache for keystone not working
by Boris 07 Nov '25

07 Nov '25

Hi, I am currently debugging a problem that the radosgw keystone token cache seems not to work properly. Or at all. I tried to debug it and attached the rgw_debug log set to 10. I've truncated to only show the part from "No stored secret string, cache miss" until the request is done. The failed request hits a rate limit on the keystone which currently takes around 2k answered requests per minute. Any ideas what I did wrong? * All requests were done within 10 seconds and were only an ls to show buckets. * This particular RGW only took my requests during testing. * We didn't set any timeouts or special cache configs in ceph * system time is correct First request worked instantly: req 8122732607072897744 0.106001295s s3:list_buckets No stored secret string, cache miss [4.0K blob data] req 8122732607072897744 0.315003842s s3:list_buckets s3 keystone: validated token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266594 req 8122732607072897744 0.315003842s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee : hit (negative entry) req 8122732607072897744 0.315003842s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 8122732607072897744 0.315003842s s3:list_buckets normalizing buckets and tenants req 8122732607072897744 0.315003842s s->object=<NULL> s->bucket= req 8122732607072897744 0.315003842s s3:list_buckets init permissions req 8122732607072897744 0.315003842s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 8122732607072897744 0.315003842s s3:list_buckets recalculating target req 8122732607072897744 0.315003842s s3:list_buckets reading permissions req 8122732607072897744 0.315003842s s3:list_buckets init op req 8122732607072897744 0.315003842s s3:list_buckets verifying op mask req 8122732607072897744 0.315003842s s3:list_buckets verifying op permissions req 8122732607072897744 0.315003842s s3:list_buckets verifying op params req 8122732607072897744 0.315003842s s3:list_buckets pre-executing req 8122732607072897744 0.315003842s s3:list_buckets check rate limiting req 8122732607072897744 0.315003842s s3:list_buckets executing req 8122732607072897744 0.315003842s s3:list_buckets completing req 8122732607072897744 0.315003842s cache get: name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) req 8122732607072897744 0.315003842s s3:list_buckets op status=0 req 8122732607072897744 0.315003842s s3:list_buckets http status=200 ====== req done req=0x74659e51b6f0 op status=0 http_status=200 latency=0.315003842s ====== 2nd request failed req 10422983006485317789 0.061000749s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.keys+05917cf2ee9d4fdea8baf6a3348ca33a : hit (negative entry) req 10422983006485317789 0.061000749s s3:list_buckets error reading user info, uid=05917cf2ee9d4fdea8baf6a3348ca33a can't authenticate req 10422983006485317789 0.061000749s s3:list_buckets Failed the auth strategy, reason=-5 failed to authorize request WARNING: set_req_state_err err_no=5 resorting to 500 req 10422983006485317789 0.061000749s cache get: name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) req 10422983006485317789 0.061000749s s3:list_buckets op status=0 req 10422983006485317789 0.061000749s s3:list_buckets http status=500 ====== req done req=0x74659e51b6f0 op status=0 http_status=500 latency=0.061000749s ====== 3rd requests went through again req 13123970335019889535 0.000000000s s3:list_buckets No stored secret string, cache miss [250B blob data] req 13123970335019889535 0.204002500s s3:list_buckets s3 keystone: validated token: 8144848695793469:user-9XGYcbFNUVTQ expires: 1762266602 req 13123970335019889535 0.204002500s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee$a13f0472be744104ad1f64bb2855cdee : hit (negative entry) req 13123970335019889535 0.204002500s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 13123970335019889535 0.204002500s s3:list_buckets normalizing buckets and tenants req 13123970335019889535 0.204002500s s->object=<NULL> s->bucket= req 13123970335019889535 0.204002500s s3:list_buckets init permissions req 13123970335019889535 0.204002500s s3:list_buckets cache get: name=eu-central-lz.rgw.meta+users.uid+a13f0472be744104ad1f64bb2855cdee : hit (requested=0x13, cached=0x13) req 13123970335019889535 0.204002500s s3:list_buckets recalculating target req 13123970335019889535 0.204002500s s3:list_buckets reading permissions req 13123970335019889535 0.204002500s s3:list_buckets init op req 13123970335019889535 0.204002500s s3:list_buckets verifying op mask req 13123970335019889535 0.204002500s s3:list_buckets verifying op permissions req 13123970335019889535 0.204002500s s3:list_buckets verifying op params req 13123970335019889535 0.204002500s s3:list_buckets pre-executing req 13123970335019889535 0.204002500s s3:list_buckets check rate limiting req 13123970335019889535 0.204002500s s3:list_buckets executing req 13123970335019889535 0.204002500s s3:list_buckets completing req 13123970335019889535 0.204002500s cache get: name=eu-central-lz.rgw.log++script.postrequest. : hit (negative entry) req 13123970335019889535 0.204002500s s3:list_buckets op status=0 req 13123970335019889535 0.204002500s s3:list_buckets http status=200 ====== req done req=0x74659e51b6f0 op status=0 http_status=200 latency=0.204002500s ====== -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 15

Snaptrim flood
by Lukasz Gomulka 06 Nov '25

06 Nov '25

Dear all, In the previous year, we were forced to migrate all data from one cluster due to a critical issue with snaptrimq length, which caused the cluster to become unstable. Unfortunately, we are now experiencing the same problem again. We've been flooded with a massive queue of snaptrims despite a massive lack of snapshot deletion. Within seconds, every PG received over 2 million snaps to delete. It looks like some kind of database corruption. 1.1 SNAPTRIMQ_LEN 2264067 Last Year it was Octopus cluster (15.2.17). Now it is cluster in Quincy (17.2.7). Originally cluster was created in version 14.2.16 and upgraded -> 15.2.17 -> 17.2.7. Below I attached background/history in chronological order from earliest. I have also attach logs from snaptrimming osd in text file. ---------- BACKGROUND: All was good: > 2025-10-16T21:30:09.776167+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47659 : cluster [DBG] pgmap v22191: 8193 pgs: 2 active+clean+scrubbing+deep, 8191 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 491 MiB/s rd, 359 MiB/s wr, 20.99k op/s First laggy pg: > 2025-10-16T21:30:11.789813+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47661 : cluster [DBG] pgmap v22192: 8193 pgs: 4 active+clean+laggy, 2 active+clean+scrubbing+deep, 8187 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 328 MiB/s rd, 247 MiB/s wr, 14.01k op/s First hearbeat problem: > 2025-10-16T21:30:21.893+0000 7f578efd7700 -1 osd.yyy 602287 heartbeat_check: no reply from xyx.xyz.xyz.xyz:6802 osd.xxx since back 2025-10-16T21:29:57.827954+0000 front 2025-10-16T21:29:57.828552+0000 (oldest deadline 2025-10-16T21:30:21.328590+0000) First report of failed osd: > 2025-10-16T21:30:21.841714+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235394 : cluster [DBG] osd.16 reported failed by osd.182 First failed osd: > 2025-10-16T21:30:21.841681+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235393 : cluster [INF] osd.15 failed (root=default,rack=rack123,host=host-123) (2 reporters from different host after 23.000420 >= grace 20.000000) Host down: > 2025-10-16T21:30:22.914804+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235451 : cluster [WRN] Health check failed: 1 host (24 osds) down (OSD_HOST_DOWN) Some osd up again: > 2025-10-16T21:30:22.892201+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235450 : cluster [DBG] osd.22 failure report canceled by osd.64 > 2025-10-16T21:30:21.904+0000 7f4c358ca700 1 mon.mon-04-<cluster-fsid>@1(peon).osd e602288 e602288: 283 total, 266 up, 283 in Cluster during recovering: > 2025-10-16T21:30:23.865481+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47668 : cluster [DBG] pgmap v22200: 8193 pgs: 34 peering, 648 stale+active+clean, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 676 active+clean+laggy, 60 active+undersized+degraded, 6773 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 4.7 MiB/s rd, 45 MiB/s wr, 1.28k op/s; 372351/152450688 objects degraded (0.244%) Again all down: > 2025-10-16T21:30:23.936587+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235453 : cluster [DBG] osdmap e602290: 283 total, 259 up, 283 in ... few osds were flapping up and down x few times (about 5) (259->266->259...->259) ... Finally problem was born: >2025-10-16T21:30:34.405+0000 7fe5613fc700 0 log_channel(cluster) log [WRN] : Health check failed: snap trim queue for 4133 pg(s) >= 1024 (mon_osd_snap_trim_queue_warn_on) (PG_SLOW_SNAP_TRIMMING) During problem our monitors with joy were compacting rocksdb: > 2025-10-16T21:30:26.224+0000 7f4c310c1700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-04-<cluster-fsid>.caas.ovh.net) > 2025-10-16T21:30:26.820+0000 7f0fefa75700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-05-<cluster-fsid>.caas.ovh.net) ---------- Of course we ended with 2M+ snaps to remove on every pg. Additionally after few minutes we saw such operation on mon: > 2025-10-16T21:37:11.434+0000 7fe5613fc700 -1 mon.mon-01-<cluster-fsid>@0(leader) e10 get_health_metrics reporting 1 slow ops, oldest is pool_op(delete unmanaged snap pool 1 tid 7113677 name v602484) Do you have any idea what might be the reason for this issue and how to address it? Have you encountered a similar problem elsewhere? Currently, we are running snaptrim, but it may take several weeks to complete, and during this time, the cluster is severely impacted. Maybe do You have any idea what to check next, where can we found a clue. Thank You for any help, Best Regards, Lukasz Lucki Gomulka

2 4

squid 19.2.3 - RGW datalog shard 36 missing on secondary
by Daan van Gorkum 05 Nov '25

05 Nov '25

Hi all, Recently I had to replace a whole cluster that was a secondary zone of the rgw realm and zonegroup "main-store". I removed the original zone and added the new one. The sync started (although initially very slow) and now I'm up to about 85% of data replicated to the new cluster. (Data) sync seems to have stalled now (and based on the logs of the primary/master it seems to request the same objects over and over) and the RGW sync instance is spitting out these errors: debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 -1 rgw data changes log: int rgw::cls::fifo::FIFO::push(const DoutPrefixProvider*, const std::vector<ceph::buffer::v15_2_0::list>&, optional_yield):1446 canceled too many times, giving up: tid=88894 debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 -1 rgw data changes log: virtual int RGWDataChangesFIFO::push(const DoutPrefixProvider*, int, RGWDataChangesBE::entries&&, optional_yield): unable to push to FIFO: data_log.36: (125) Operation canceled debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 -1 rgw data changes log: ERROR: svc.cls->timelog.add() returned -125 debug 2025-11-05T15:01:57.165+0000 7fd9d6929640 0 rgw data changes log: ERROR: RGWDataChangesLog::renew_entries returned error r=-125 and debug 2025-11-05T15:01:42.638+0000 7fd9e5146640 -1 rgw async rados processor: virtual int RGWDataChangesFIFO::push(const DoutPrefixProvider*, int, ceph::real_time, const string&, ceph::bu ffer::v15_2_0::list&&, optional_yield): unable to push to FIFO: data_log.36: (125) Operation canceled I figured out this has to do with datalog shard 36 and I can indeed not see the status: bash-5.1$ radosgw-admin datalog status --rgw-realm main-store --shard-id 36 [ { "marker": "", "last_update": "0.000000" } ] 2025-11-05T15:04:10.622+0000 7f3bbf941900 -1 int rgw::cls::fifo::{anonymous}::get_part_info(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, rados::cls::fifo::part_header*, uint64_t, optional_yield):328 fifo::op::GET_PART_INFO failed r=-2 tid=2 2025-11-05T15:04:10.622+0000 7f3bbf941900 -1 int rgw::cls::fifo::FIFO::get_part_info(const DoutPrefixProvider*, int64_t, rados::cls::fifo::part_header*, optional_yield):2023 get_part_info failed: r=-2 tid=2 2025-11-05T15:04:10.622+0000 7f3bbf941900 -1 virtual int RGWDataChangesFIFO::get_info(const DoutPrefixProvider*, int, RGWDataChangesLogInfo*, optional_yield): unable to get part info: data_log.36/6: (2) No such file or directory When checking the rados pool I do see a "data_log.36" object: bash-5.1$ rados -p backup-store-ams.rgw.log ls | grep -E 'data_log.36' data_log.36 And the size seems similar to a neighbor object that's working fine: bash-5.1$ rados -p backup-store-ams.rgw.log stat data_log.36 # Bad backup-store-ams.rgw.log/data_log.36 mtime 2025-10-25T23:47:04.000000+0000, size 156 bash-5.1$ rados -p backup-store-ams.rgw.log stat data_log.35 # Good backup-store-ams.rgw.log/data_log.35 mtime 2025-11-01T12:27:43.000000+0000, size 156 I should add that currently the cluster has a "13 large omap objects" warning activated, but I was thinking this was part of the full resync and sharding happens after completion, it's unclear of this error is related to this warning at all. I've been going through the ceph FIFO source but I've not been able to pinpoint it. The primary cluster is still running OMAP datalog type. Any help/insights would be greatly appreciated! Daan

1 0

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 05 Nov '25

05 Nov '25

Hi Eugen, I have already tried as the below method I stated before. I got the current osdmap and tried to set-osdmap via ceph-object-store-tool. But the command failed with error: ceph osd getmap 72555 > /tmp/osd_map_72555 CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-11/ --op set-osdmap --file /tmp/osd_map_72555 osdmap (#-1:9c8e9ef2:::osdmap.72555:0#) does not exist. https://www.mail-archive.com/ceph-users@ceph.io/msg11545.html BR, Huseyin hcotuk(a)gmail.com

4 4

crc errors on 4 node cluster
by Roland Giesler 05 Nov '25

05 Nov '25

From https://forum.proxmox.com/threads/hyperconverged-cluster-logging-seemingly-… We have 4 nodes (dual Xeon CPU's, 256G RAM, 4 NVMe SSD's, 4 HDD's and dual Melanox 25Gb/s sfp's) in a cluster. I have started noticing apparently random crc errors in the osd logs. Node B, osd.6 2025-10-23T10:32:59.808+0200 7f22a75bf700 0 bad crc in data 3330350463 != exp 677417498 from v1:192.168.131.4:0/3121668685 192.168.131.4 is node D Node B, osd.7 2025-10-23T09:35:12.995+0200 7fbcdbcd7700 0 bad crc in data 3922083958 != exp 3479198006 from v1:192.168.131.2:0/2732728486 192.168.131.2 is node B, which is the node osd.7 is on. and so there are others on other nodes and osd's. From what I understand this means that data copied from some other osd to this one that logs the error fails the crc test. However, I have taken (as a test) one of these ssd's out of the cluster and it tests just fine. I put it back and no crc errors are logged for it. Question: Can something else we causing this? A network connector? It seems pretty random so me, so how can I trace this sources of this? It was suggested to me by die vendor tech support that turning offloading off on the nic: /ethtool -K vmbr0 rx off tx off tso off gso off gro off lro off./ That doesn't seem to make a difference however. The next step would be to check for a MTU/jumbo mismatch. Some details: ceph version 17.2.7 # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 62.29883 root default -3 15.17824 host FT1-NodeA 2 hdd 1.86029 osd.2 up 1.00000 1.00000 3 hdd 1.86029 osd.3 up 1.00000 1.00000 4 hdd 1.86029 osd.4 up 1.00000 1.00000 5 hdd 1.86589 osd.5 up 1.00000 1.00000 0 ssd 0.93149 osd.0 up 1.00000 1.00000 28 ssd 3.30690 osd.28 up 1.00000 1.00000 29 ssd 3.49309 osd.29 up 1.00000 1.00000 -7 16.10413 host FT1-NodeB 10 hdd 1.86029 osd.10 up 1.00000 1.00000 11 hdd 1.86029 osd.11 up 1.00000 1.00000 26 hdd 1.86029 osd.26 up 1.00000 1.00000 27 hdd 1.86029 osd.27 up 1.00000 1.00000 6 ssd 0.93149 osd.6 up 1.00000 1.00000 7 ssd 0.93149 osd.7 up 1.00000 1.00000 25 ssd 3.49309 osd.25 up 1.00000 1.00000 41 ssd 3.30690 osd.41 up 1.00000 1.00000 -10 15.84383 host FT1-NodeC 14 hdd 1.59999 osd.14 up 1.00000 1.00000 15 hdd 1.86029 osd.15 up 1.00000 1.00000 16 hdd 1.86029 osd.16 up 1.00000 1.00000 17 hdd 1.86029 osd.17 up 1.00000 1.00000 8 ssd 0.93149 osd.8 up 1.00000 1.00000 9 ssd 0.93149 osd.9 up 1.00000 1.00000 24 ssd 3.49309 osd.24 up 1.00000 1.00000 43 ssd 3.30690 osd.43 up 1.00000 1.00000 -13 15.17264 host FT1-NodeD 20 hdd 1.86029 osd.20 up 1.00000 1.00000 21 hdd 1.86029 osd.21 up 1.00000 1.00000 22 hdd 1.86029 osd.22 up 1.00000 1.00000 23 hdd 1.86029 osd.23 up 1.00000 1.00000 12 ssd 3.30690 osd.12 up 1.00000 1.00000 13 ssd 0.93149 osd.13 up 1.00000 1.00000 19 ssd 3.49309 osd.19 up 1.00000 1.00000 The spinners have their RocksDB on the nvme's, for extra performance, but I have not noticed any crc errors prior to installing the 4TB Samsungs in each node. Suggestions are welcome. thanks Roland

3 3

PSA: pad.ceph.com may have lost updates since ~10/27
by Patrick Donnelly 04 Nov '25

04 Nov '25

It was discovered the CSC notes pad is missing recent updates from this week's meeting. Please share if you have also observed this in one of the pads you use. This is probably related to the recent migration to AWS. https://lists.ceph.io/hyperkitty/list/sepia@ceph.io/thread/ECODNJ2EJH35QWUJ… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0