February 2025 - ceph-users

Feb 3 Ceph Steering Committee meeting notes
by Casey Bodley 04 Feb '25

04 Feb '25

discussed timeline for Tentacle release: - debated freeze soon (spring) or later (summer) to match Squid's delayed release - polled dev leads: later release preferred for EC optimizations and bluestore improvements - potential conflict with sepia lab migration which needs to be finished by end of year - tentative consensus for delaying Tentacle release to early 2026. further emails to follow on recent disruption from windows pr check: - broke over the weekend, fixed promptly with https://tracker.ceph.com/issues/69768 https://github.com/ceph/ceph/pull/61627 - future of windows support is uncertain, but we'll keep the windows check 'required' as long as it's maintained 19.2.1 release status: - packages/containers built, LRC upgrade planned CDM this week: - Please add topics: https://tracker.ceph.com/projects/ceph/wiki/CDM_05-FEB-2025 question about policy for AI-generated code contributions: - Linux Foundation policy at https://www.linuxfoundation.org/legal/generative-ai - the Ceph project should discuss

2 1

Full-Mesh?
by Anthony Fecarotta 04 Feb '25

04 Feb '25

I read a Ceph Benchmark paper ( https://www.proxmox.com/en/downloads/proxmox-virtual-environment/documentat…) where they demonstrated, among other things, the performance of using a Full-Mesh Network Schema for Ceph on a three node cluster. > • FRR ! full-mesh routed (with fallback) > • RSTP ! full-mesh RSTP loop setup > • Routed: ! full-mesh simple routed setup with round-robin bonds as > underlying links Is this method used at an enterprise level? I am seriously considering it. Thank you, everyone. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

2 2

Squid: RGW overview freezes (this.dataArray[W] is undefined)
by Eugen Block 04 Feb '25

04 Feb '25

Hi, I couldn't find an existing tracker yet, so I'm asking here if this already has been reported (or even fixed). Until now, I've only been playing around with virtual test clusters on Squid, and I noticed the dashboard freezes when I'm on the /rgw/overview page. This is what I get in the mgr log: ---snip--- Jan 31 13:16:37 host2 ceph-mgr[2799]: [dashboard ERROR frontend.error] (https://IP:8443/#/rgw/overview) this.dataArray[W] is undefined updateChartData@https://IP:8443/main.5d76eec705c379ed.js:3:1175137 ngOnChanges@https://IP:8443/main.5d76eec705c379ed.js:3:1174390 ... 99140/ye/v.prototype.invoke@https://IP:8443/ ---snip--- The output is lengthy (around 100 lines), just wanted to give an impression. Clicking anywhere else doesn't really improve anything, only reloading the page lets you browse again. This seems to be related to "Multi-Site Sync Status" (which I don't have enabled), it seems to freeze while loading. But sometimes it does load successfully, showing the info ( Multi-site needs to be configured in order to see the multi-site sync status. Please consult the documentation on how to configure and enable the multi-site functionality.). But I thought, maybe that's my lab environment or something. Last week I upgraded a customer cluster to 19.2.0 as well, and it shows the same behavior. Should I create a tracker issue? Thanks! Eugen

3 3

[ceph-ansible][radosgw]: s3 key regenration issue
by Danish Khan 04 Feb '25

04 Feb '25

Dear All, I am getting 500 internal server error after deleting and creating new access and secret keys. I am able to access my buckets and and its data using new keys from S3cmd utility. but it seems ceph MGR keep cache and throwing 500 internal server error I am trying to get meta data from CEPH dashboard. But when I do ceph mgr fail and move the ceph dashboard to another MGR then it starts working fine again. Is there a way I can clear cache of MGR or is the any bug in ceph for regeneration of S3 access keys? Regards, Danish

2 4

Identify who is doing what on an osd
by Szabo, Istvan (Agoda) 04 Feb '25

04 Feb '25

Hi, Is there a way to pull some meaningful information out from the dump_ops_in_flight dump? I'd like to know which bucket this operation relates to. "description": "osd_op(client.3685135831.0:1525226003 26.144 26.4e892144 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e738686)", "initiated_at": "2025-02-04T10:35:10.009003+0700", "age": 0.018950328999999998, "duration": 0.022382503000000002, "type_data": { "flag_point": "queued for pg", "client_info": { "client": "client.3685135831", "client_addr": "10.118.199.10:0/3348448578", "tid": 1525226003 }, "events": [ { "event": "initiated", "time": "2025-02-04T10:35:10.009003+0700", "duration": 0 }, { "event": "throttled", "time": "2025-02-04T10:35:10.009003+0700", "duration": 0 }, Thank you

1 0

Confusing documentation about ceph osd pool set pg_num
by Mark Schouten 03 Feb '25

03 Feb '25

Hi, We need to increase the number of PG’s for a pool on a Pacific cluster. I know we can easily do that using: ceph osd pool set $name pg_num $amount However, the documentation [1] states (both for Pacific and for Squid) that you need to upgrade pgp_num as well. But, I seem to remember that in newer versions, pgp_num will be upgraded automatically. And this message claims that as well [2] So which one is true? :) [1]: https://docs.ceph.com/en/squid/rados/operations/placement-groups/#setting-t… [2]: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/F6PLHO7BGR… — Mark Schouten CTO, Tuxis B.V. +31 318 200208 / mark(a)tuxis.nl

3 5

CephFS: EC pool with "leftover" objects
by Robert Sander 03 Feb '25

03 Feb '25

Hi, there is an old cluster (9 years) that gets constantly upgraded and is currently running version 17.2.7. 3 years ago (when running version 16) a new EC pool was added to the existing CephFS to be used with the directory layout feature. Now it was decided to remove that pool again. On the filesystem level all files have been copied to the original replicated pool and then deleted. No files or directories have the EC pool in their extended attributes referenced. But still this EC pool has appr 242 million objects with a total of ca 600 TB data stored. This shows up in "ceph df" and "ceph pg dump". The objects can be listed with "rados ls" but a "rados stat" or "rados get" will yield an error: error stat-ing cephfs_data_ec/1001275b3fe.00000241: (2) No such file or directory How can this be? Are these artifacts from not properly removed snapshots? Is it really save to remove this pool from CephFS and delete it? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

4 6

cephadm: Move DB/WAL from HDD to SSD
by Alan Murrell 03 Feb '25

03 Feb '25

Hello, I posted this on the Reddit sub, but it was suggested I might get better responses here. I am running a 5-node Ceph cluster (v18.2.2) installed using "cephadm". I am trying to migrate the DB/WAL on our slower HDDs to NVME; I am following this article: https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-fo… I have a 1TB NVME in each node, and there are four HDDs. I have created the VG ("cephdbX", where "X" is the node number) and four equal-sized LVs ("cephdb1", "cephdb2", "cephdb3", "cephdb4"). On the node I am trying to move the DB/WAL first, I have stopped the systemd OSD service for the OSD I am doing this first to. I have switched into the cephadm shell so I can run the ceph-volume commands, but when I run: ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/cephdb1 I get the following error: --> Target path cephdb03/cephdb1 is not a Logical Volume Unable to attach new volume : cephdb03/cephdb1 If I run 'lvs' in the cephadm shell, I can see the LVs (sorry about the formatting; hopefully you can view it in a way that it isn't a big "jumble"): LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-f85a57a8-e2f5-4bda-bc3b-e99d8b70768b ceph-341561e6-da91-4678-b6c8-0f0281443945 -wi-ao---- <1.75t osd-block-f1fd3d53-4ed9-4492-82a0-4686231d57e1 ceph-65ebde73-28ac-4dac-b0cb-4cf8df18bd4b -wi-ao---- 16.37t osd-block-3571394c-3afa-4177-904a-17550f8e902c ceph-6c8de2ed-cae3-4dd9-9ea8-49c94b746878 -wi-a----- 16.37t osd-block-41d44327-3df7-4166-a675-d9630bde4867 ceph-703962c7-6f28-4d8b-b77f-a6eba39da6b2 -wi-ao---- <1.75t osd-block-438c7681-ee6b-4d29-91f5-d487377c3ac9 ceph-71cc35c4-436d-42b7-a704-b21c2d22b43b -wi-ao---- 16.37t osd-block-2ebf78e8-1de1-464e-9125-14a8b7e6796f ceph-7c1fe149-8500-4a41-9052-64f27b2cb70b -wi-ao---- <1.75t osd-block-ca347144-eb84-4e9f-bfb5-81d60659f417 ceph-92595dfe-dc70-47c7-bcab-65b26d84448c -wi-ao---- 16.37t osd-block-2d338a42-83ce-4281-9762-b268e74f83b3 ceph-e9b51fa2-2be1-40f3-b96d-fb0844740afa -wi-ao---- <1.75t cephdb1 cephdb03 -wi-a----- 232.00g cephdb2 cephdb03 -wi-a----- 232.00g cephdb3 cephdb03 -wi-a----- 232.00g cephdb4 cephdb03 -wi-a----- 232.00g lv_root cephnode03-20240110 -wi-ao---- <468.36g lv_swap cephnode03-20240110 -wi-ao---- <7.63g All the official docs I read about it seem to assume the Ceph components are installed directly on the host, rather than in containers (which is what 'cephadm' does) Any advice for migrating the DB/WAL to the SSDs when using 'cephadm'? (I could probably destroy the OSD and manually re-create it with the options for pointing the DB/WAL to the SSD, but I would rather do it without forcing a data migration, otherwise I would have to wait for that with each OSD I am migrating) Followup question: if I do seem to have to go the route of destroying the OSD and re-creating it with the flags for point the DB/WAL to the SSD partition, the syntax I have seen is this: ceph-volume lvm prepare --bluestore --block.db --block.wal --data VOLUME_GROUP/LOGICAL_VOLUME but the above command seems to only specify the location for the data? Should the syntax be more like: ceph-volume lvm prepare --bluestore --block.db VG2/LV1 --block.wal VG2/LV1 --data /dev/sda (assuming my HDD is /dev/sda)? Thanks! :-)

3 2

Suggestions
by Devender Singh 01 Feb '25

01 Feb '25

Hello all Few more suggestions.. if can be added to further releases. 1. We faced some issue, can we add more command to control clients using watcher, rbd status pool/image Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088 Some commands to control watcher and kill client.id <http://client.id/>. something like rbd lock remove <pool-name>/<image-name> <client_id> Or rbd watchers <pool-name>/<image-name> Or something rbd check <pool-name>/<image-name> Or rbd list watchers pool-name or pool/image 2. Also, as we have multiple ceph clusters, so on dashboard every time by going to hosts we are able to see hosts names to identify the nodes and cluster type lets say, dev or prod. Can we have a variable on dashboard to see “Name: Location-Dev”?, I think we have enough space to list name in this are. 3. Seems dashboard/mgr is not cleaning itself. Most of time we need to fail to manager to clear such errors, But it seems a similar issue as in step1 above. I mounted this volume and unmounted and clean everything, even mount point. But for last three days this alert is active, I have tried failing back to different mgrs. CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on prod-host1 will be full in less than 5 days based on the 48 hour trailing fill rate. 4. We need more command to control pool repair. If we have started a pool repair command, how we can stop it? Regards Dev

2 1

Error ENOENT: Module not found
by Devender Singh 01 Feb '25

01 Feb '25

Hello Need some help.. Tried draining host but it got stuck and now orchestrator is not running. But cluster health is OK , also I adde host back. # ceph health detail HEALTH_OK Tried adding blank queue file,, ceph config-key set mgr/cephadm/osd_remove_queue -I osd_remove_queue_blank.json and mgr fail but did not work.. # ceph config-key get mgr/cephadm/osd_remove_queue [{"osd_id": 27, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.100700Z"}, {"osd_id": 31, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.790403Z"}, {"osd_id": 35, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.857252Z"}, {"osd_id": 38, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.192332Z"}, {"osd_id": 42, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.236171Z"}, {"osd_id": 44, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.000889Z"}, {"osd_id": 49, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.027052Z"}, {"osd_id": 54, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.974898Z"}, {"osd_id": 58, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.834649Z"}, {"osd_id": 62, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.812439Z"}, {"osd_id": 66, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.902356Z"}, {"osd_id": 70, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.726199Z"}, {"osd_id": 74, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.881018Z"}, {"osd_id": 78, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:05:02.498519Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.078261Z"}, {"osd_id": 82, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:03:48.916299Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.948462Z"}, {"osd_id": 86, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:02:28.907365Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.148633Z"}, {"osd_id": 89, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:01:15.727422Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.117332Z"}, {"osd_id": 94, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:59:54.173088Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.768943Z"}, {"osd_id": 98, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:57:24.192595Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.053079Z"}, {"osd_id": 102, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:57:25.201520Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.927255Z"}, {"osd_id": 106, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", ""drain_started_at": "2025-01-31T23:57:09.422738Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.171455Z"}, {"osd_id": 111, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:56:33.636189Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.748019Z"}, {"osd_id": 115, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:56:34.674664Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.213363Z”}] Regards Dev

2 3