- ceph-users - lists.ceph.io

Interruption of rebalancing
by Jeffrey Turmelle 02 Mar '23

02 Mar '23

I have a Nautilus cluster with 7 nodes, 210 HDDs. I recently added the 7th node with 30 OSDs which are currently rebalancing very slowly. I just noticed that the ethernet interface only negotiated a 1Gb connection, even though it has a 10Gb interface. I’m not sure why, but would like to reboot the node to get the interface back to 10Gb. Is it ok to do this? What should I do to prep the cluster for the reboot? Jeffrey Turmelle International Research Institute for Climate & Society <https://iri.columbia.edu/> The Climate School <https://climate.columbia.edu/> at Columbia University <https://columbia.edu/> 845-652-3461

4 4

Very slow backfilling
by Joffrey 02 Mar '23

02 Mar '23

Hi, I have many 'not {deep-}scrubbed in time' and a1 PG remapped+backfilling and I don't understand why this backfilling is taking so long. root@hbgt-ceph1-mon3:/# ceph -s cluster: id: c300532c-51fa-11ec-9a41-0050569c3b55 health: HEALTH_WARN 15 pgs not deep-scrubbed in time 13 pgs not scrubbed in time services: mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3 (age 36h) mgr: hbgt-ceph1-mon2.nteihj(active, since 2d), standbys: hbgt-ceph1-mon1.thrnnu, hbgt-ceph1-mon3.gmfzqm osd: 60 osds: 60 up (since 13h), 60 in (since 13h); 1 remapped pgs rgw: 3 daemons active (3 hosts, 2 zones) data: pools: 13 pools, 289 pgs objects: 67.74M objects, 127 TiB usage: 272 TiB used, 769 TiB / 1.0 PiB avail pgs: 288 active+clean 1 active+remapped+backfilling io: client: 3.3 KiB/s rd, 1.5 MiB/s wr, 3 op/s rd, 8 op/s wr recovery: 790 KiB/s, 0 objects/s What can I do to understand this slow recovery (is it the backfill action ?) Thanks you 'Jof

1 0

ceph 16.2.10 - misplaced object after changing crush map only setting hdd class
by xadhoom76＠gmail.com 02 Mar '23

02 Mar '23

Hi to all and thanks for sharing your experience on ceph ! We have an easy setup with 9 osd all hdd and 3 nodes, 3 osd for each node. We started the cluster to test how it works with hdd with default and easy bootstrap . Then we decide to add ssd and create a pool to use only ssd. In order to have pools on hdd and pools on ssd only we edited the crushmap to add class hdd We do not enter anything about ssd till now, nor disk or rules only add the class map to the default rule. So i show you the rules before introducing class hdd # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure2_1 { id 2 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure-pool.meta { id 3 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure-pool.data { id 4 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } And here is the after # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure2_1 { id 2 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure-pool.meta { id 3 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure-pool.data { id 4 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } Just doing this triggered the misplaced of all pgs bind to EC pool. Is that correct ? and why ? Best regards Alessandro Bolgia

2 2

How do I troubleshoot radosgw errors STS?
by mat＠hazmat.dev 01 Mar '23

01 Mar '23

I've setup RadosGW with STS ontop of my ceph cluster. It works great and fine but I'm also trying to setup authentication with an OpenIDConnect provider. I'm have a hard time troubleshooting issues because the radosgw log file doesn't have much information in it. For example when I try to use the `sts:AssumeRoleWithWebIdentity` API it fails with `{'Code': 'AccessDenied', ...}` and all I see is the beat log showing an HTTP 403. Is there a way to enable more verbose logging so I can see what is failing and why I'm getting certain errors with STS, S3, or IAM apis? My ceph.conf looks like this for each node (mildly redacted): ``` [client.radosgw.pve4] host = pve4 keyring = /etc/pve/priv/ceph.client.radosgw.keyring log file = /var/log/ceph/client.radosgw.$host.log rgw_dns_name = s3.lab rgw_frontends = beast endpoint=0.0.0.0:7480 ssl_endpoint=0.0.0.0:443 ssl_certificate=/etc/pve/priv/ceph/s3.lab.crt ssl_private_key=/etc/pve/priv/ceph/s3.lab.key rgw_sts_key = 1111111111111111 rgw_s3_auth_use_sts = true rgw_enable_apis = s3, s3website, admin, sts, iam ```

2 3

PG Sizing Question
by Deep Dish 01 Mar '23

01 Mar '23

Hello Looking to get some official guidance on PG and PGP sizing. Is the goal to maintain approximately 100 PGs per OSD per pool or for the cluster general? Assume the following scenario: Cluster with 80 OSD across 8 nodes; 3 Pools: - Pool1 = Replicated 3x - Pool2 = Replicated 3x - Pool3 = Erasure Coded 6-4 Assuming the well published formula: Let (Target PGs / OSD) = 100 [ (Target PGs / OSD) * (# of OSDs) ] / (Replica Size) - Pool1 = (100*80)/3 = 2666.67 => 4096 - Pool2 = (100*80)/3 = 2666.67 => 4096 - Pool3 = (100*80)/10 = 800 => 1024 Total cluster would have 9216 PGs and PGPs. Are there any implications (performance / monitor / MDS / RGW sizing) with how many PGs are created on the cluster? Looking for validation and / or clarification of the above. Thank you.

2 3

RadosGW multipart fragments not being cleaned up by lifecycle policy on Quincy
by Sean Houghton 01 Mar '23

01 Mar '23

The latest version of quincy seems to be having problems cleaning up multipart fragments from canceled uploads. The bucket is empty: % s3cmd -c .s3cfg ls s3://warp-benchmark % However, it's got 11TB of data and 700k objects. # radosgw-admin bucket stats --bucket=warp-benchmark { "bucket": "warp-benchmark", "num_shards": 10, "tenant": "", "zonegroup": "6be863e8-a9f2-42c9-b114-c8651b1f1afa", "placement_rule": "ssd.ec63", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "aa099b5e-01d5-4394-b287-df99a4d63298.18924.1", "marker": "aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "index_type": "Normal", "owner": "warp_benchmark", "ver": "0#5580404,1#5593184,2#5586262,3#5591427,4#5591937,5#5588120,6#5589760,7#5582923,8#5579062,9#5578699", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0", "mtime": "0.000000", "creation_time": "2023-02-10T21:45:12.721604Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#", "usage": { "rgw.main": { "size": 12047620866048, "size_actual": 12047620866048, "size_utilized": 12047620866048, "size_kb": 11765254752, "size_kb_actual": 11765254752, "size_kb_utilized": 11765254752, "num_objects": 736113 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } A bucket list shows that they are all multipart fragments # radosgw-admin bucket list --bucket=warp-benchmark [ ... (LOTS OF THESE) { "name": "_multipart_(2F3(gCS/1.GagoUCrCRqawswb6.rnd.tg1efLm7-es41Xg3i-Nm6bYjS-c-No79.12", "instance": "", "ver": { "pool": 20, "epoch": 30984 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 16777216, "mtime": "2023-02-16T00:03:01.586472Z", "etag": "e7475bca6a58de35648ca5f25d6653bf", "storage_class": "", "owner": "warp_benchmark", "owner_display_name": "Warp Benchmark", "content_type": "", "accounted_size": 16777216, "user_data": "", "appendable": "false" }, "tag": "_YdopX7yxnVrvg2h35MIQGN3vsPyZx5W", "flags": 0, "pending_map": [], "versioned_epoch": 0 } ] Note that the timestamp is from 2 weeks ago so a lifecycle policy of "cleanup after 1 day" should delete them. cat cleanup-multipart.xml <LifecycleConfiguration> <Rule> <ID>abort-multipart-rule</ID> <Filter> <Prefix></Prefix> </Filter> <Status>Enabled</Status> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> % s3cmd dellifecycle s3://warp-benchmark s3://warp-benchmark/: Lifecycle Policy deleted % s3cmd setlifecycle cleanup-multipart.xml s3://warp-benchmark s3://warp-benchmark/: Lifecycle Policy updated A secondary problem is that the lifecycle policy never runs automatically and is stuck in the UNINITIAL state. This problem is for another day of debugging. # radosgw-admin lc list [ { "bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] However, it can be started manually # radosgw-admin lc process # radosgw-admin lc list [ { "bucket": ":warp-benchmark:aa099b5e-01d5-4394-b287-df99a4d63298.37403.1", "started": "Wed, 01 Mar 2023 17:35:27 GMT", "status": "COMPLETE" } ] This has no effect on the bucket and the bucket stats show the exact same size and object count (output omitted for brevity). Running a gc pass also has no effect # radosgw-admin gc list [] # radosgw-admin gc process # radosgw-admin gc list [] Any ideas?

1 0

Re: How do I troubleshoot radosgw errors STS?
by hazmat 01 Mar '23

01 Mar '23

Also, here is the ceph version: ceph version 17.2.5 (e04241aa9b639588fa6c864845287d2824cb6b55) quincy (stable)

1 0

Next quincy release (17.2.6)
by Yuri Weinstein 01 Mar '23

01 Mar '23

Hello We are planning to start QE validation release next week. If you have PRs that are to be part of it, please let us know by adding "needs-qa" for 'quincy' milestone ASAP. Thx YuriW

3 3

Ceph OSD imbalance and performance
by Dave Ingram 28 Feb '23

28 Feb '23

Hello, Our ceph cluster performance has become horrifically slow over the past few months. Nobody here is terribly familiar with ceph and we're inheriting this cluster without much direction. Architecture: 40Gbps QDR IB fabric between all ceph nodes and our ovirt VM hosts. 11 OSD nodes with a total of 163 OSDs. 14 pools, 3616 PGs, 1.19PB total capacity. Ceph versions: { "mon": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 3 }, "mgr": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 3 }, "osd": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 118, "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 22, "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 19 }, "mds": {}, "overall": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 124, "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 22, "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 19 } } The majority of disks are spindles but there are also NVMe SSDs. There is a lot of variability in drive sizes - two different sets of admins added disks sized between 6TB and 16TB and I suspect this and imbalanced weighting is to blame. Performance on the ovirt VMs can dip as low as several *kilobytes* per-second (!) on reads and a few MB/sec on writes. There are also several scrub errors. In short, it's a complete wreck. STATUS: [root@ceph-admin davei]# ceph -s cluster: id: 1b8d958c-e50b-40ef-a681-16cfeb9390b8 health: HEALTH_ERR 3 scrub errors Possible data damage: 3 pgs inconsistent services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 mgr: ceph3(active), standbys: ceph2, ceph1 osd: 163 osds: 159 up, 158 in data: pools: 14 pools, 3616 pgs objects: 46.28M objects, 174TiB usage: 527TiB used, 694TiB / 1.19PiB avail pgs: 3609 active+clean 4 active+clean+scrubbing+deep 3 active+clean+inconsistent io: client: 74.3MiB/s rd, 96.0MiB/s wr, 3.85kop/s rd, 3.68kop/s wr --- HEALTH: [root@ceph-admin davei]# ceph health detail HEALTH_ERR 3 scrub errors; Possible data damage: 3 pgs inconsistent OSD_SCRUB_ERRORS 3 scrub errors PG_DAMAGED Possible data damage: 3 pgs inconsistent pg 2.8a is active+clean+inconsistent, acting [13,152,127] pg 2.ce is active+clean+inconsistent, acting [145,13,152] pg 2.e8 is active+clean+inconsistent, acting [150,162,42] --- CEPH OSD DF: (not going to paste that all in here): https://pastebin.com/CNW5RKWx What else am I missing in terms of what to share with you all? Any advice on how we should 'reweight' these to get the performance to improve? Thanks all, -Dave

4 6

Do not use SSDs with (small) SLC cache
by Michael Wodniok 28 Feb '23

28 Feb '23

Hi all, digging around debugging, why our (small: 10 Hosts/~60 OSDs) cluster is so slow even while recovering I found out one of our key issues are some SSDs with SLC cache (in our case Samsung SSD 870 EVO) - which we just recycled from other use cases in the hope to speed up our mainly hdd based cluster. We know it's a little bit random which objects get accelerated when not used as cache. However the opposite was the case. These type's ssds are only fast when operating in their SLC cache, which is only several Gigabytes in a multi-TB ssd [1]. When doing a big write or a backfill onto these SSDs we got really low IO-rates (around 10 MB/s even with 4M-objects). But it got even worse. Disclaimer: This is my view as a user, maybe a more technically involved person is able to correct me. Cause seems to be the mclock-scheduler which measures the iops an osd is able to do. As in the blog measured [2], this is usually a good thing as there is done some profiling and queing is done different. But in our case the osd_mclock_max_capacity_iops_ssd for most of the corresponding osds was very low. But not for everyone. I assume that it depends when mclock-scheduler measured the iops capacity. That led to a broken scheduling where backfills were at low speed and the ssd itself had nearly no disk usage because it was operating in it's cache again and could work faster. That issue could be solved by switching back to wpq scheduler for the affected SSDs. This scheduler seems to just queue up ios without throttling because of maximum iops reached. Now we see a still bad IO situation because of the slow SSDs but at least they are operating at their maximum (having typical settings like osd_recovery_max_active and osd_recovery_sleep* tuned). We are going to replace the SSDs to hopefully more consistent performing ones (even if their peak performance would be not as good). I hope this may help somebody in the future when being stuck in low performance recoverys. Refs: [1] https://www.tomshardware.com/reviews/samsung-870-evo-sata-ssd-review-the-be… [2] https://ceph.io/en/news/blog/2022/mclock-vs-wpq-testing-with-background-ops… Happy Storing! Michael Wodniok -- Michael Wodniok M.Sc. WorNet AG Bürgermeister-Graf-Ring 28 82538 Geretsried Simply42 und SecuMail sind Marken der WorNet AG. http://www.wor.net/ Handelsregister Amtsgericht München (HRB 129882) Vorstand: Christian Eich Aufsichtsratsvorsitzender: Dirk Steinkopf

7 8