- ceph-users - lists.ceph.io

Unable to delete bucket - endless multipart uploads?
by David Monschein 08 Mar '21

08 Mar '21

Hi All, We've been dealing with what seems to be a pretty annoying bug for a while now. We are unable to delete a customer's bucket that seems to have an extremely large number of aborted multipart uploads. I've had $(radosgw-admin bucket rm --bucket=pusulax --purge-objects) running in a screen session for almost 3 weeks now and it's still not finished; it's most likely stuck in a loop or something. The screen session with debug-rgw=10 spams billions of these messages: 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/d3/04d33e18-3f13-433c-b924-56602d702d60-31.msg.2~0DTalUjTHsnIiKraN1klwIFO88Vc2E3.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/d7/04d7ad26-c8ec-4a39-9938-329acd6d9da7-102.msg.2~K_gAeTpfEongNvaOMNa0IFwSGPpQ1iA.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/da/04da4147-c949-4c3a-aca6-e63298f5ff62-102.msg.2~-hXBSFcjQKbMkiyEqSgLaXMm75qFzEp.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/db/04dbb0e6-dfb0-42fb-9d0f-49cceb18457f-102.msg.2~B5EhGgBU5U_U7EA5r8IhVpO3Aj2OvKg.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/df/04df39be-06ab-4c72-bc63-3fac1d2700a9-11.msg.2~_8h5fWlkNrIMqcrZgNbAoJfc8BN1Xx-.meta[] This is probably the 2nd or 3rd time I've been unable to delete this bucket. I also tried running $(radosgw-admin bucket check --fix --check-objects --bucket=pusulax) before kicking off the delete job, but that didn't work either. Here is the bucket in question, the num_objects counter never decreases after trying to delete the bucket: [root@os5 ~]# radosgw-admin bucket stats --bucket=pusulax { "bucket": "pusulax", "num_shards": 144, "tenant": "", "zonegroup": "dbb69c5b-b33f-4af2-950c-173d695a4d2c", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "c0d0b8a5-c63c-4c24-9dab-8deee88dbf0b.3209338.4", "marker": "c0d0b8a5-c63c-4c24-9dab-8deee88dbf0b.3292800.7", "index_type": "Normal", "owner": "REDACTED", "ver": "0#115613,1#115196,2#115884,3#115497,4#114649,5#114150,6#116127,7#114269,8#115220,9#115092,10#114003,11#114538,12#115235,13#113463,14#114928,15#115135,16#115535,17#114867,18#116010,19#115766,20#115274,21#114818,22#114805,23#114853,24#114099,25#114359,26#114966,27#115790,28#114572,29#114826,30#114767,31#115614,32#113995,33#115305,34#114227,35#114342,36#114144,37#114704,38#114088,39#114738,40#114133,41#114520,42#114420,43#114168,44#113820,45#115093,46#114788,47#115522,48#114713,49#115315,50#115055,51#114513,52#114086,53#114401,54#114079,55#113649,56#114089,57#114157,58#114064,59#115224,60#114753,61#114686,62#115169,63#114321,64#114949,65#115075,66#115003,67#114993,68#115320,69#114392,70#114893,71#114219,72#114190,73#114868,74#113432,75#114882,76#115300,77#114755,78#114598,79#114221,80#114895,81#114031,82#114566,83#113849,84#115155,85#113790,86#113334,87#113800,88#114856,89#114841,90#115073,91#113849,92#114554,93#114820,94#114256,95#113840,96#114838,97#113784,98#114876,99#115524,100#115686,101#112969,102#112156,103#112635,104#112732,105#112933,106#112412,107#113090,108#112239,109#112697,110#113444,111#111730,112#112446,113#114479,114#113318,115#113032,116#112048,117#112404,118#114545,119#112563,120#112341,121#112518,122#111719,123#112273,124#112014,125#112979,126#112209,127#112830,128#113186,129#112944,130#111991,131#112865,132#112688,133#113819,134#112586,135#113275,136#112172,137#113019,138#112872,139#113130,140#112716,141#112091,142#111859,143#112773", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0", "mtime": "2020-06-17 20:27:16.685833Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#", "usage": { "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 97009704, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 94737, "num_objects": 3592952 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } We're running 14.2.16 on our RGWs and OSD nodes. Anyone have any ideas? Is it possible to target this bucket via rados directly to try and delete it? I'm weary of doing stuff like that though. Thanks in advance. - Dave Monschein

3 2

Re: Ceph Object Gateway setup/tutorial
by Juan Miguel Olmo Martinez 08 Mar '21

08 Mar '21

Hi Rok Using cephadm you do not need to worry about changes in the config files, restarts and this kind of things: Understand the RGW functionality, the link you provided is ok. ( https://docs.ceph.com/en/latest/radosgw/ ) but for installation, and management of the RGW daemons use: https://docs.ceph.com/en/latest/cephadm/rgw/ -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

1 0

Questions RE: Ceph/CentOS/IBM
by Drew Weaver 08 Mar '21

08 Mar '21

Howdy, After the IBM acquisition of RedHat the landscape for CentOS quickly changed. As I understand it right now Ceph 14 is the last version that will run on CentOS/EL7 but CentOS8 was "killed off". So given that, if you were going to build a Ceph cluster today would you even bother doing it using a non-commercial distribution or would you just use RHEL 8 (or even their commercial Ceph product). Secondly, are we expecting IBM to "kill off" Ceph as well? Thanks, -Drew

15 20

Can't clear UPGRADE_REDEPLOY_DAEMON after fix
by Samy Ascha 08 Mar '21

08 Mar '21

Hi! I was upgrading from 15.2.8 to 15.2.9 via `ceph orch upgrade` (Ubuntu Bionic). One OSD seemed to have failed to upgrade, so I just redeployed it. The OSD is up/in, but this warning is not clearing: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.3 on host ceph-osd4 failed. It seems the warning is not listed in the docs, so I'm wondering how to go about getting the cluster back into HEALTH_OK. Hope you can help, Samy

2 2

balance OSD usage.
by ricardo.re.azevedo＠gmail.com 08 Mar '21

08 Mar '21

Hi All, Does anyone know how I can rebalance my cluster to balance out the OSD usage? I just added 12 more 14Tb HDDs to my cluster (cluster of made up of 12Tb and 14Tb disks) bringing my total to 48 OSDs. Ceph df reports my pool as 83% full (see below). I am aware this only reports the fullest drive. However, after backfilling, data distribution is still heavily skewed to a few OSDs. The oldest are at ~80% and the newest are at ~50. My issue is that data is currently being written to all drives equally and at this rate the cluster will start disabling writes despite still having significant raw capacity. PGs do not seem to be evenly distributed either. Weights are 12.73 for all 14Tb OSD and 10.9 for all 12Tb OSDs. I tried using the balancer to create a new plan but it had no real effect. Haven't seen much useful info online. --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 582 TiB 203 TiB 379 TiB 380 TiB 65.20 TOTAL 582 TiB 203 TiB 379 TiB 380 TiB 65.20 --- POOLS --- POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 2 261 TiB 68.62M 379 TiB 83.64 56 TiB cephfs_metadata 3 536 MiB 12.74k 1.6 GiB 0 25 TiB `osd df`: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 10.91409 1.00000 11 TiB 8.4 TiB 8.4 TiB 1.6 MiB 22 GiB 2.5 TiB 77.00 1.18 92 up 4 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 21 MiB 18 GiB 4.0 TiB 63.03 0.97 80 up 5 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.3 MiB 17 GiB 4.4 TiB 59.50 0.91 71 up 11 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.4 TiB 36 MiB 24 GiB 3.3 TiB 74.13 1.14 105 up 15 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.2 TiB 20 MiB 21 GiB 4.5 TiB 64.75 0.99 99 up 16 hdd 10.91409 1.00000 11 TiB 6.4 TiB 6.4 TiB 1.7 MiB 16 GiB 4.5 TiB 58.81 0.90 74 up 20 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 33 MiB 23 GiB 4.3 TiB 66.09 1.01 102 up 22 hdd 12.73340 1.00000 13 TiB 8.6 TiB 8.6 TiB 3.3 MiB 22 GiB 4.1 TiB 67.48 1.04 100 up 23 hdd 12.73340 1.00000 13 TiB 7.6 TiB 7.6 TiB 406 KiB 20 GiB 5.1 TiB 59.93 0.92 89 up 36 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.2 TiB 171 KiB 16 GiB 5.5 TiB 57.00 0.87 85 up 41 hdd 12.73340 1.00000 13 TiB 7.0 TiB 7.0 TiB 2.1 MiB 16 GiB 5.7 TiB 54.85 0.84 85 up 46 hdd 12.73340 1.00000 13 TiB 8.1 TiB 8.0 TiB 67 MiB 19 GiB 4.7 TiB 63.34 0.97 99 up 3 hdd 10.91409 1.00000 11 TiB 7.6 TiB 7.6 TiB 1.0 MiB 19 GiB 3.3 TiB 69.85 1.07 87 up 6 hdd 10.91409 1.00000 11 TiB 8.3 TiB 8.3 TiB 39 MiB 22 GiB 2.6 TiB 76.28 1.17 93 up 10 hdd 12.73340 1.00000 13 TiB 9.6 TiB 9.6 TiB 5.1 MiB 24 GiB 3.1 TiB 75.52 1.16 109 up 14 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.8 TiB 34 MiB 22 GiB 3.9 TiB 69.57 1.07 99 up 24 hdd 12.73340 1.00000 13 TiB 11 TiB 11 TiB 22 MiB 34 GiB 2.1 TiB 83.28 1.28 117 up 25 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 24 MiB 21 GiB 4.4 TiB 65.84 1.01 97 up 26 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.9 TiB 205 KiB 22 GiB 3.8 TiB 69.84 1.07 102 up 30 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 33 MiB 16 GiB 4.0 TiB 63.31 0.97 92 up 39 hdd 12.73340 1.00000 13 TiB 7.2 TiB 7.2 TiB 36 MiB 16 GiB 5.6 TiB 56.39 0.86 93 up 43 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 3.6 MiB 19 GiB 4.4 TiB 65.54 1.01 98 up 47 hdd 12.73340 1.00000 13 TiB 7.1 TiB 7.1 TiB 51 MiB 17 GiB 5.6 TiB 55.70 0.85 88 up 2 hdd 10.91409 1.00000 11 TiB 7.4 TiB 7.3 TiB 22 KiB 19 GiB 3.6 TiB 67.36 1.03 82 up 7 hdd 10.91409 1.00000 11 TiB 8.5 TiB 8.5 TiB 530 KiB 21 GiB 2.4 TiB 77.89 1.19 97 up 8 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 21 MiB 22 GiB 4.3 TiB 66.53 1.02 99 up 12 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 20 MiB 22 GiB 4.2 TiB 67.13 1.03 99 up 18 hdd 12.73340 1.00000 13 TiB 9.9 TiB 9.9 TiB 24 MiB 31 GiB 2.8 TiB 77.81 1.19 112 up 19 hdd 12.73340 1.00000 13 TiB 8.8 TiB 8.8 TiB 34 MiB 22 GiB 3.9 TiB 69.17 1.06 104 up 21 hdd 12.73340 1.00000 13 TiB 9.5 TiB 9.4 TiB 15 MiB 28 GiB 3.3 TiB 74.37 1.14 111 up 31 hdd 10.91409 1.00000 11 TiB 6.3 TiB 6.3 TiB 56 MiB 14 GiB 4.6 TiB 57.53 0.88 77 up 32 hdd 10.91409 1.00000 11 TiB 8.0 TiB 8.0 TiB 44 MiB 18 GiB 2.9 TiB 73.15 1.12 99 up 37 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.1 TiB 4.2 MiB 18 GiB 4.6 TiB 64.09 0.98 94 up 40 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.3 TiB 1.4 MiB 16 GiB 5.5 TiB 57.10 0.88 86 up 44 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 2.6 MiB 17 GiB 5.0 TiB 60.55 0.93 92 up 1 hdd 10.91409 1.00000 11 TiB 5.9 TiB 5.9 TiB 55 MiB 15 GiB 5.0 TiB 54.14 0.83 72 up 9 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.3 TiB 705 KiB 24 GiB 3.4 TiB 73.57 1.13 109 up 13 hdd 12.73340 1.00000 13 TiB 8.0 TiB 8.0 TiB 22 MiB 21 GiB 4.7 TiB 62.74 0.96 91 up 17 hdd 10.91409 1.00000 11 TiB 7.5 TiB 7.5 TiB 24 MiB 19 GiB 3.4 TiB 68.88 1.06 88 up 27 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.8 TiB 4.8 MiB 17 GiB 5.9 TiB 53.81 0.83 78 up 28 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 19 MiB 20 GiB 5.0 TiB 60.46 0.93 91 up 29 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 17 MiB 21 GiB 4.4 TiB 65.56 1.01 100 up 33 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.6 MiB 15 GiB 4.4 TiB 59.23 0.91 77 up 34 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.6 TiB 44 MiB 15 GiB 4.3 TiB 60.87 0.93 81 up 35 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.5 TiB 18 MiB 15 GiB 4.4 TiB 60.08 0.92 81 up 38 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 18 MiB 19 GiB 4.3 TiB 66.25 1.02 105 up 42 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.9 TiB 21 MiB 16 GiB 5.8 TiB 54.26 0.83 84 up 45 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 38 MiB 18 GiB 5.0 TiB 60.61 0.93 94 up TOTAL 582 TiB 380 TiB 379 TiB 942 MiB 950 GiB 203 TiB 65.20 Thank you, Ricardo

4 3

Re: Resolving LARGE_OMAP_OBJECTS
by Benoît Knecht 08 Mar '21

08 Mar '21

Hi Drew, On Friday, March 5th, 2021 at 20:22, Drew Weaver <drew.weaver(a)thenap.com> wrote: > Sorry for multi-reply, I got that command to run: > > for obj in $(rados -p default.rgw.buckets.index ls | grep 2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637); do printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys $obj | wc -l); done; > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.0 10423 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.15 10445 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.3 10542 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.6 10511 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.13 10414 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.12 10479 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.2 10486 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.5 10448 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.4 10470 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.8 10474 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.1 10470 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.10 10411 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.7 10445 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.14 10413 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.9 10356 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.11 10410 So this looks fine, it means that this bucket is sharded properly and the index keys are evenly distributed. However, the problematic object is `.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.213.0`; you said there's no bucket with that ID, which means it's probably a stale instance. Does it still exist in `default.rgw.buckets.index`? If so, you can try running `radosgw-admin reshard stale-instances rm`, it should get rid of it (you can verify afterwards with `rados -p default.rgw.buckets.index ls`). -- Ben

1 0

grafana-api-url not only for one host
by E Taka 05 Mar '21

05 Mar '21

Hi, if the host fails, to which the grafana-api-url points (in the example below ceph01.hostxyz.tld:3000), Ceph Dashboard can't Display Grafana Data: # ceph dashboard get-grafana-api-url https://ceph01.hostxyz.tld:3000 Is it possible to automagically switch to an other host? Thanks, Erich

3 2

what is quickest way to generate a new key for a user?
by Marc 05 Mar '21

05 Mar '21

what is quickest way to generate a new key for a user?

1 0

Best practices for OSD on bcache
by Norman.Kern 04 Mar '21

04 Mar '21

Hi, guys I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.

5 11

Metadata for LibRADOS
by Cary FitzHugh 04 Mar '21

04 Mar '21

Hello - We're trying to use native libRADOS and the only challenge we're running into is searching metadata. Using the rgw metadata sync seems to require all data to be pushed through the rgw, which is not something we're interested in setting up at the moment. Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)? Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things? Are there any features like this in libRADOS? Thank you

5 7