- ceph-users - lists.ceph.io

Confusing documentation about ceph osd pool set pg_num
by Mark Schouten 03 Feb '25

03 Feb '25

Hi, We need to increase the number of PG’s for a pool on a Pacific cluster. I know we can easily do that using: ceph osd pool set $name pg_num $amount However, the documentation [1] states (both for Pacific and for Squid) that you need to upgrade pgp_num as well. But, I seem to remember that in newer versions, pgp_num will be upgraded automatically. And this message claims that as well [2] So which one is true? :) [1]: https://docs.ceph.com/en/squid/rados/operations/placement-groups/#setting-t… [2]: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/F6PLHO7BGR… — Mark Schouten CTO, Tuxis B.V. +31 318 200208 / mark(a)tuxis.nl

3 5

CephFS: EC pool with "leftover" objects
by Robert Sander 03 Feb '25

03 Feb '25

Hi, there is an old cluster (9 years) that gets constantly upgraded and is currently running version 17.2.7. 3 years ago (when running version 16) a new EC pool was added to the existing CephFS to be used with the directory layout feature. Now it was decided to remove that pool again. On the filesystem level all files have been copied to the original replicated pool and then deleted. No files or directories have the EC pool in their extended attributes referenced. But still this EC pool has appr 242 million objects with a total of ca 600 TB data stored. This shows up in "ceph df" and "ceph pg dump". The objects can be listed with "rados ls" but a "rados stat" or "rados get" will yield an error: error stat-ing cephfs_data_ec/1001275b3fe.00000241: (2) No such file or directory How can this be? Are these artifacts from not properly removed snapshots? Is it really save to remove this pool from CephFS and delete it? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

4 6

cephadm: Move DB/WAL from HDD to SSD
by Alan Murrell 03 Feb '25

03 Feb '25

Hello, I posted this on the Reddit sub, but it was suggested I might get better responses here. I am running a 5-node Ceph cluster (v18.2.2) installed using "cephadm". I am trying to migrate the DB/WAL on our slower HDDs to NVME; I am following this article: https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-fo… I have a 1TB NVME in each node, and there are four HDDs. I have created the VG ("cephdbX", where "X" is the node number) and four equal-sized LVs ("cephdb1", "cephdb2", "cephdb3", "cephdb4"). On the node I am trying to move the DB/WAL first, I have stopped the systemd OSD service for the OSD I am doing this first to. I have switched into the cephadm shell so I can run the ceph-volume commands, but when I run: ceph-volume lvm new-db --osd-id 10 --osd-fsid 474264fe-b00e-11ee-b586-ac1f6b0ff21a --target cephdb03/cephdb1 I get the following error: --> Target path cephdb03/cephdb1 is not a Logical Volume Unable to attach new volume : cephdb03/cephdb1 If I run 'lvs' in the cephadm shell, I can see the LVs (sorry about the formatting; hopefully you can view it in a way that it isn't a big "jumble"): LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-f85a57a8-e2f5-4bda-bc3b-e99d8b70768b ceph-341561e6-da91-4678-b6c8-0f0281443945 -wi-ao---- <1.75t osd-block-f1fd3d53-4ed9-4492-82a0-4686231d57e1 ceph-65ebde73-28ac-4dac-b0cb-4cf8df18bd4b -wi-ao---- 16.37t osd-block-3571394c-3afa-4177-904a-17550f8e902c ceph-6c8de2ed-cae3-4dd9-9ea8-49c94b746878 -wi-a----- 16.37t osd-block-41d44327-3df7-4166-a675-d9630bde4867 ceph-703962c7-6f28-4d8b-b77f-a6eba39da6b2 -wi-ao---- <1.75t osd-block-438c7681-ee6b-4d29-91f5-d487377c3ac9 ceph-71cc35c4-436d-42b7-a704-b21c2d22b43b -wi-ao---- 16.37t osd-block-2ebf78e8-1de1-464e-9125-14a8b7e6796f ceph-7c1fe149-8500-4a41-9052-64f27b2cb70b -wi-ao---- <1.75t osd-block-ca347144-eb84-4e9f-bfb5-81d60659f417 ceph-92595dfe-dc70-47c7-bcab-65b26d84448c -wi-ao---- 16.37t osd-block-2d338a42-83ce-4281-9762-b268e74f83b3 ceph-e9b51fa2-2be1-40f3-b96d-fb0844740afa -wi-ao---- <1.75t cephdb1 cephdb03 -wi-a----- 232.00g cephdb2 cephdb03 -wi-a----- 232.00g cephdb3 cephdb03 -wi-a----- 232.00g cephdb4 cephdb03 -wi-a----- 232.00g lv_root cephnode03-20240110 -wi-ao---- <468.36g lv_swap cephnode03-20240110 -wi-ao---- <7.63g All the official docs I read about it seem to assume the Ceph components are installed directly on the host, rather than in containers (which is what 'cephadm' does) Any advice for migrating the DB/WAL to the SSDs when using 'cephadm'? (I could probably destroy the OSD and manually re-create it with the options for pointing the DB/WAL to the SSD, but I would rather do it without forcing a data migration, otherwise I would have to wait for that with each OSD I am migrating) Followup question: if I do seem to have to go the route of destroying the OSD and re-creating it with the flags for point the DB/WAL to the SSD partition, the syntax I have seen is this: ceph-volume lvm prepare --bluestore --block.db --block.wal --data VOLUME_GROUP/LOGICAL_VOLUME but the above command seems to only specify the location for the data? Should the syntax be more like: ceph-volume lvm prepare --bluestore --block.db VG2/LV1 --block.wal VG2/LV1 --data /dev/sda (assuming my HDD is /dev/sda)? Thanks! :-)

3 2

Suggestions
by Devender Singh 01 Feb '25

01 Feb '25

Hello all Few more suggestions.. if can be added to further releases. 1. We faced some issue, can we add more command to control clients using watcher, rbd status pool/image Watchers: watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088 Some commands to control watcher and kill client.id <http://client.id/>. something like rbd lock remove <pool-name>/<image-name> <client_id> Or rbd watchers <pool-name>/<image-name> Or something rbd check <pool-name>/<image-name> Or rbd list watchers pool-name or pool/image 2. Also, as we have multiple ceph clusters, so on dashboard every time by going to hosts we are able to see hosts names to identify the nodes and cluster type lets say, dev or prod. Can we have a variable on dashboard to see “Name: Location-Dev”?, I think we have enough space to list name in this are. 3. Seems dashboard/mgr is not cleaning itself. Most of time we need to fail to manager to clear such errors, But it seems a similar issue as in step1 above. I mounted this volume and unmounted and clean everything, even mount point. But for last three days this alert is active, I have tried failing back to different mgrs. CephNodeDiskspaceWarning Mountpoint /mnt/dst-volume on prod-host1 will be full in less than 5 days based on the 48 hour trailing fill rate. 4. We need more command to control pool repair. If we have started a pool repair command, how we can stop it? Regards Dev

2 1

Error ENOENT: Module not found
by Devender Singh 01 Feb '25

01 Feb '25

Hello Need some help.. Tried draining host but it got stuck and now orchestrator is not running. But cluster health is OK , also I adde host back. # ceph health detail HEALTH_OK Tried adding blank queue file,, ceph config-key set mgr/cephadm/osd_remove_queue -I osd_remove_queue_blank.json and mgr fail but did not work.. # ceph config-key get mgr/cephadm/osd_remove_queue [{"osd_id": 27, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.100700Z"}, {"osd_id": 31, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.790403Z"}, {"osd_id": 35, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.857252Z"}, {"osd_id": 38, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.192332Z"}, {"osd_id": 42, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.236171Z"}, {"osd_id": 44, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.000889Z"}, {"osd_id": 49, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.027052Z"}, {"osd_id": 54, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.974898Z"}, {"osd_id": 58, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.834649Z"}, {"osd_id": 62, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.812439Z"}, {"osd_id": 66, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.902356Z"}, {"osd_id": 70, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.726199Z"}, {"osd_id": 74, "started": true, "draining": false, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.881018Z"}, {"osd_id": 78, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:05:02.498519Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.078261Z"}, {"osd_id": 82, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:03:48.916299Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.948462Z"}, {"osd_id": 86, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:02:28.907365Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.148633Z"}, {"osd_id": 89, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-02-01T00:01:15.727422Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.117332Z"}, {"osd_id": 94, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:59:54.173088Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.768943Z"}, {"osd_id": 98, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:57:24.192595Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.053079Z"}, {"osd_id": 102, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:57:25.201520Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.927255Z"}, {"osd_id": 106, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", ""drain_started_at": "2025-01-31T23:57:09.422738Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.171455Z"}, {"osd_id": 111, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:56:33.636189Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:25.748019Z"}, {"osd_id": 115, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "zap": false, "hostname": "node02", "drain_started_at": "2025-01-31T23:56:34.674664Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2025-01-31T23:54:26.213363Z”}] Regards Dev

2 3

SMB Support in Squid
by Tarrago, Eli (RIS-BCT) 01 Feb '25

01 Feb '25

Hello Ceph-users, I may be in the process of upgrading to Squid to add SMB support to our ceph clusters. Has anyone used this feature in a production environment and what has been your experience? Thank you. ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

2 1

RGW Exporter for Storage Class Metrics
by Preisler, Patrick 31 Jan '25

31 Jan '25

Hi, we would like to have a detailed usage report for our S3 Buckets. I already installed this rgw exporter https://github.com/blemmenes/radosgw_usage_exporter and it does provide useful information about the buckets and the amount of objects etc. However we have configured different storage-classes in our placement-targets and we would like to know how much data inside each bucket is placed in the STANDARD or STANDARD_IA Storage Class. Is there an exporter that does provide this data? Thanks and kind regards Patrick Preisler

2 1

RGW S3 Compatibility
by Anthony Fecarotta 31 Jan '25

31 Jan '25

Hello, Our company has a five node Proxmox cluster that recently implemented Ceph. We utilize multiple open-source applications that prefer (not require) S3 storage. Now that we are running Ceph, are we able to simply create a Ceph Object Gateway with the S3 API? I have read that not all S3 buckets share the same compatibility. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

4 6

slow backfilling and recovering
by Jaemin Joo 31 Jan '25

31 Jan '25

Hi, all I am suffering from slow backfilling and recovering. I increased to "max_backfills" = 64, "recovery_max_actives" = 64. I know it's unnessary to increase more in my ceph cluster. When I checked pg query, I found the number of "backfills_in_flight"&"recovering" is 4. I set "osd_recovery_max_single_start = 4" so I think "osd_recovery_max_single_start" affects to the number of "backfills_in_flight"&"recovering". I increased to "osd_recovery_max_single_start = 16" but the number of "backfills_in_flight"&"recovering" in pg query was not changed. I know "osd_recovery_max_single_start" runtime is true. Should I restart the osds to increase the number of "backfills_in_flight"&"recovering"? or Is there the other way?

4 5

pgmap version increasing like every second ok or excessive?
by Andreas Elvers 31 Jan '25

31 Jan '25

Hello, I see this in my logs: 2025-01-22T09:14:43.063966+0000 mgr.node1.joznex (mgr.584732) 337151 : cluster [DBG] pgmap v300985: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:45.066685+0000 mgr.node1.joznex (mgr.584732) 337154 : cluster [DBG] pgmap v300986: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:47.070458+0000 mgr.node1.joznex (mgr.584732) 337155 : cluster [DBG] pgmap v300987: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:49.074664+0000 mgr.node1.joznex (mgr.584732) 337158 : cluster [DBG] pgmap v300988: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:51.079225+0000 mgr.node1.joznex (mgr.584732) 337159 : cluster [DBG] pgmap v300989: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:53.081633+0000 mgr.node1.joznex (mgr.584732) 337160 : cluster [DBG] pgmap v300990: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:55.084216+0000 mgr.node1.joznex (mgr.584732) 337163 : cluster [DBG] pgmap v300991: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:57.087873+0000 mgr.node1.joznex (mgr.584732) 337164 : cluster [DBG] pgmap v300992: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:14:59.092225+0000 mgr.node1.joznex (mgr.584732) 337175 : cluster [DBG] pgmap v300993: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail 2025-01-22T09:15:01.096803+0000 mgr.node1.joznex (mgr.584732) 337176 : cluster [DBG] pgmap v300994: 497 pgs: 497 active+clean; 9.5 TiB data, 29 TiB used, 48 TiB / 76 TiB avail Cluster is healthy. 3 nodes with 2 HDD OSDs each. pgmap version is increasing like every second. Is this excessive or is it deemed normal?

2 1