- ceph-users - lists.ceph.io

Modify or override ceph_default_alerts.yml
by Devin A. Bougie 14 Jan '25

14 Jan '25

Hi, All. We are using cephadm to manage a 19.2.0 cluster on fully-updated AlmaLinux 9 hosts, and would greatly appreciate help modifying or overriding the alert rules in ceph_default_alerts.yml. Is the best option to simply update the /var/lib/ceph/<cluster_id>/home/ceph_default_alerts.yml file? In particular, we’d like to either disable the CephPGImbalance alert or change it to calculate averages per-pool or per-crush_rule instead of globally as in [1]. We currently have PG autoscaling enabled, and have two separate crush_rules (one with large spinning disks, one with much smaller nvme drives). Although I don’t believe it causes any technical issues with our configuration, our dashboard is full of CephPGImbalance alerts that would be nice to clean up without having to create periodic silences. Any help or suggestions would be greatly appreciated. Many thanks, Devin [1] https://github.com/rook/rook/discussions/13126#discussioncomment-10043490

3 14

Understanding filesystem size
by Nicola Mori 14 Jan '25

14 Jan '25

Dear Ceph users, I'd need some help with CephFS. Originally, my cluster consisted of 12 hosts with different total raw disk size, ranging from 16 to 32 TB; there were two pools, one 3x replicated metadata and one 6+2 EC data with host failure domain, and which I access via CephFS. In total there were 512 PGs. Naively, I would expect that the available size of the data pool would be about 75% of the total available space given the EC parameters (6/(6+2) = 75%), but I noticed that it was actually lower, around 66% (as reported by df -h, while for the total raw capacity I used the one reported by the Ceph dashboard). I guessed this could be due to smaller hosts limiting the total amount of available space for the filesystem, so I started a disk upgrade campaign to make the host sizes more even. For example, for a 16 TB host with 8x2TB I replaced two disks with 8 TB ones, for a new raw size of 28 TB. I did this and similar upgrades for other hosts in steps, every time waiting for backfill to finish and checking the new filesystem size before proceeding. And I noticed that the filesystem size reported by df -h always remained at 66% of the total raw capacity, despite host sizes are now all between 24 and 32 TB. So my guess about the limiting factor for the relative fs size is evidently wrong. I thought that for some reason maybe 66% is the actual limit and also about some possible explanation (e.g. some space being reserved for the metadata pool), and even considered the possibility that the size reported by df -h being meaningless, but I'd need some advice from an expert to sort this out. Thanks in advance for any help. Nicola

5 27

Issue with CopyObject version 18.2.4: Copied objects are not deleted below the pool
by tranthithuan180693＠gmail.com 14 Jan '25

14 Jan '25

I tried uploading without copying, the obj still deleted normally When using copy, obj cannot be deleted under the pool for storage-class COLD Here's how I do it: I created storage-class COLD radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class COLD radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class COLD --data-pool default.rgw.cold.data and restart service rgw Create bucket name hn3 radosgw-admin bucket stats --bucket hn3 { "bucket": "hn3", "num_shards": 11, "tenant": "", "zonegroup": "39c9f30a-42dd-4037-8ef7-02942f6bfcc7", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3", "marker": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3", "index_type": "Normal", "versioned": false, "versioning_enabled": false, "object_lock_enabled": false, "mfa_enabled": false, "owner": "test", "ver": "0#1,1#1,2#1,3#1,4#4,5#1,6#1,7#1,8#1,9#1,10#1", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "2025-01-08T08:44:50.831879Z", "creation_time": "2025-01-08T08:44:50.804023Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 37836, "size_actual": 40960, "size_utilized": 37836, "size_kb": 37, "size_kb_actual": 40, "size_kb_utilized": 37, "num_objects": 1 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } I upload any file with storage-class COLD I use mc (MinIO Client): mc cp test.txt s3/hn3/ Then copy that file with a different name using storage-class cold mc cp s3/hn3/test.txt s3/hn3/test.txt2 Check obj in pool default.rgw.cold.data rados -p default.rgw.cold.data ls | grep ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3 ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.ksiViAsAWgM7uyfn68yJiDNjLEXRCVc_0 Now i will delete this obj mc rm s3/hn3/test.txt mc rm s3/hn3/test.txt2 Check garbage collection radosgw-admin gc list --include-all | grep ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3 "tag": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.wubPUKz-g1B8XdzfiNXWM_rrD_oQeqM_1_01863ccdb47a110b5af24277107e76a9\u0000", "oid": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.wubPUKz-g1B8XdzfiNXWM_rrD_oQeqM_0", "oid": "ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.RWuk6chwz7Pkd8jvkBqY_eUibNDHnIq_0", Run manual gc radosgw-admin gc process --include-all Check again obj in pool , obj is not deleted rados -p default.rgw.cold.data ls | grep ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3 ca4afca3-2cc7-4a83-9045-f8ecc4769d64.4630.3__shadow_.ksiViAsAWgM7uyfn68yJiDNjLEXRCVc_0

2 1

Any idea why misplace recovery wont finish?
by Ml Ml 14 Jan '25

14 Jan '25

Hello List, i have this 3 node Setup with 17 hdds (new - spinning rust). After putting it all together and after resetting the manual Rack-Placements back to default it seems to recover forever. I also changed the pg placements at some time. The disks where very busy until this morning when i disabled scrubbing to see if this would speed up the recovery. Here is my current status: ---------------------------------------- root@ceph12:~# ceph -s cluster: id: 2ed9f8fb-1316-4ef1-996d-a3223a3dd594 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum ceph10,ceph11,ceph12 (age 2h) mgr: ceph11(active, since 3h), standbys: ceph10, ceph12 osd: 17 osds: 17 up (since 3h), 17 in (since 11d); 18 remapped pgs flags noscrub,nodeep-scrub data: pools: 2 pools, 257 pgs objects: 9.62M objects, 37 TiB usage: 110 TiB used, 134 TiB / 244 TiB avail pgs: 1534565/28857846 objects misplaced (5.318%) 239 active+clean 16 active+remapped+backfill_wait 2 active+remapped+backfilling io: recovery: 25 MiB/s, 6 objects/s I set: ceph tell 'osd.*' injectargs --osd-max-backfills=6 --osd-recovery-max-active=9 My hourly recovery process seems not to really proceed (it decreeses and increases again): 5.410 5.036 5.269 5.008 5.373 5.769 5.555 5.103 5.067 5.135 5.409 5.417 5.373 5.197 5.090 5.458 5.204 5.339 5.164 5.425 5.692 5.383 5.726 5.492 6.694 6.576 6.362 6.243 6.011 5.880 5.589 5.433 5.846 5.378 5.184 5.647 5.374 5.513 root@ceph12:~# ceph osd perf (this is okay for spinning rust i guess) osd commit_latency(ms) apply_latency(ms) 17 0 0 13 0 0 11 57 57 9 4 4 0 0 0 1 79 79 14 43 43 2 0 0 3 0 0 16 43 43 4 33 33 5 0 0 6 0 0 7 0 0 10 0 0 12 0 0 8 48 48 iostat -xt 3 shows that they are busy but its not overloaded as i can tell. Any idea why it will only backfill 2? How could i speed this up? My other cluster with 7 nodes and very old hdds and 45 OSDs can make easily 76 objects/s Cheers, Michael

5 4

Snaptriming speed degrade with pg increase
by Szabo, Istvan (Agoda) 14 Jan '25

14 Jan '25

Hi, When we scale the placement group on a pool located in a full nvme cluster, the snaptriming speed degrades a lot. Currently we are running with these values to not degrade client op and have some progress on snaptrimmin, but it is terrible. (octopus 15.2.17 on ubuntu 20.04) -osd_max_trimming_pgs=2 --osd_snap_trim_sleep=0.1 --osd_pg_max_concurrent_snap_trims=2 We had a big pool which we used to have 128PG and that length of the snaptrimming took around 45-60 minutes. Due to impossible to do maintenance on the cluster with 600GB pg sizes because it can easily max out a cluster (which we did), we increased to 1024 and the snaptrimming duration increased to 3.5 hours. Is there any good solution that we are missing to fix this? On the hardware level I've changed server profile to tune some numa settings but seems like didn't help still. Thank you

4 9

Re: OSDs won't come back after upgrade
by Jorge Garcia 13 Jan '25

13 Jan '25

Actually, stupid mistake on my part. I had selinux mode as enforcing. Changed it to disabled, and everything works again. Thanks for the help!

2 2

Ceph orch commands failing with Error ENOENT: Module not found
by Frank Frampton 13 Jan '25

13 Jan '25

Recent upgrade from 18.2 to 19.2, upgrade went fine. Since the upgrade and a manager fail over, I can no longer run orchestrator commands. The only error I can find on an active manager daemon is the following, or it is the only one that stands out. 2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Failed to construct class in 'cephadm' 2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 667, in __init__ self.cert_key_store.load() File "/usr/share/ceph/mgr/cephadm/inventory.py", line 2073, in load self.known_certs[entity] = json.loads(v) File "/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2025-01-06T18:48:41.702+0000 7fcf42b99640 -1 mgr operator() Failed to run module in active mode ('cephadm') For any thing to really work in the dashboard I must have the balancer off. While the balance is off I can make changes in dashboard to the orchestrator, and it doesn't give me any trouble. When trying different commands from a ceph node say "ceph cephadm config-check status" it returns "Error ENOTSUP: Module 'cephadm' is not enabled/loaded (required by command 'cephadm config-check status'): use `ceph mgr module enable cephadm` to enable it". Running "ceph mgr module enable cephadm" returns "module 'cephadm' is already enabled". I really don't know where to look or what to try to resolve this. Any "ceph orch" command results in "Error ENOENT: Module not found" I don't know that my issue is related to https://tracker.ceph.com/issues/68657, but maybe it is. I have tried the following. Manually adding a new mgr daemon on different node, it starts runs the dashboard fine, but things are still not functional. Failed the mgr several times. Disabled/Enabled balancer. Disabled/Enabled mgr modules. Disabled/Enabled dashboard. All physical nodes are running Debian 12. Frank Frampton Senior Network Services Administrator Salt Lake City School District Desk: (801) 578-8223 Follow the district: Facebook<https://www.facebook.com/slcschools> | Instagram<https://instagram.com/slcschools> | Twitter<https://twitter.com/slcschools> | www.slcschools.org<http://www.slcschools.org/> Excellence and Equity: every student, every classroom, every day Scanned By Microsoft EOP

2 2

Enterprise SSD/NVME
by Martin Konold 13 Jan '25

13 Jan '25

3 2

Help in recreating a old ceph cluster
by Jayant Dang 13 Jan '25

13 Jan '25

Hi Can someone help me understand what happens in a scenario if my os disk of all the ceph nodes gets destroyed somehow and I am only left with the OSDs or physical storage devices, How can I recreate the same ceph cluster using those old OSDs without any data loss ? Is there something I should regularly backup to recreate a ceph cluster ?

2 1

Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience
by Anthony D'Atri 13 Jan '25

13 Jan '25

[ ed: snag during moderation (somehow a newline was interpolated in the Subject), so I’m sending this on behalf of kasper_steengaard(a)hotmail.com <mailto:kasper_steengaard@hotmail.com> , to whom replies should be sent] I'm managing a ceph cluster with +1K OSDs distributed accross 56 host. Until now the crush rule used is the default replicated rule, but I want to change that in order to implement failure domain on rack level. Current plan is to - Disable rebalancing by executing - ceph osd set norebalance; - Add Rack to crushmap and distribute the hosts accordingly (8 in each.) by using the built in commands - ceph osd crush add-bucket rack1 rack root=default - ceph osd crush move osd-host1 rack=rack1 - Create the new rack split rule with command - ceph osd crush rule create-replicated rack_split default rack - Set the rule across all my pools - for p in $(ceph osd lspools | cut -d' ' -f 2) ; do echo $p $(ceph osd pool set $p crush_rule rack_split) ; done - Finally enable rebalancing - ceph osd unset norebalance; However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who have been in similar situations. —— [ ed: You only include commands for one CRUSH `rack` — would you create multiple `rack` CRUSH buckets, at least three of them? Are all of your pools replicated? No EC pools for RGW buckets, CephFS data, etc? What OSD media and networking does this cluster have? HDDs will be much slower and much more impacted during the process than SSDs. Is your client workload 24x7? Which Ceph release? These factors inform how impactful the grand shuffle will be. Are your mon DBs on SSDs? A popular strategy is to use upmap-remapped.py to freeze all of the PG mappings before unsetting the norebalance flag, then the balancer will gradually undo the mappings as it moves data to where it now belongs. This process has built-in throttling. ]

3 2