August 2020 - ceph-users

What The Benefit Of Choosing Epson Customer Service?
by mary smith 10 Aug '20

10 Aug '20

In order get benefitted by choosing the correct and dependable Epson Customer Service service providers, it is strongly recommended to have a healthy research before getting involved with any of such service providers as there are several fake companies are also there claiming to be the best but actually not. https://www.epsonprintersupportpro.net/

1 0

Get To Be Acquainted With HP Support Assistant In A Proper Manner.
by mary smith 10 Aug '20

10 Aug '20

In order to be acquainted with the HP Support Assistant in a proper manner, you should not leave any stone unturned in approaching the certified printer experts who will professionally assist you the exact troubleshooting solution to your problems in an effective manner. You can approach them by just using different convenient means at anytime. https://www.amiytech.com/hp-support-assistant/

1 0

OSDs flapping since upgrade to 14.2.10
by Ingo Reimann 10 Aug '20

10 Aug '20

Hi list, since our upgrade 14.2.9 -> 14.2.10 we observe flapping OSDs: * The mons claim every few minutes: 2020-08-07 09:49:09.783648 osd.243 (osd.243) 246 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:07:21.187945 osd.253 (osd.253) 469 : cluster [WRN] Monitor daemon marked osd.253 down, but it is still running 2020-08-07 10:04:35.440547 mon.cephmon01 (mon.0) 390132 : cluster [DBG] osd.243 reported failed by osd.33 2020-08-07 10:04:35.508412 mon.cephmon01 (mon.0) 390133 : cluster [DBG] osd.243 reported failed by osd.187 2020-08-07 10:04:35.508529 mon.cephmon01 (mon.0) 390134 : cluster [INF] osd.243 failed (root=default,datacenter=of,row=row-of-02,host=cephosd16) (2 reporters from different host after 44.000150 >= grace 25.935545) 2020-08-07 10:04:35.695171 mon.cephmon01 (mon.0) 390135 : cluster [DBG] osd.243 reported failed by osd.203 2020-08-07 10:04:35.771704 mon.cephmon01 (mon.0) 390136 : cluster [DBG] osd.243 reported failed by osd.163 2020-08-07 10:04:41.588530 mon.cephmon01 (mon.0) 390148 : cluster [INF] osd.243 [v2:10.198.10.16:6882/6611,v1:10.198.10.16:6885/6611] boot 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753712 osd.243 (osd.243) 249 : cluster [DBG] map e2683535 wrongly marked me down at e2683534 osd.33 says: 2020-08-07 10:04:35.437 7fcaaa4f3700 -1 osd.33 2683533 heartbeat_check: no reply from 10.198.10.16:6802 osd.243 since back 2020-08-07 10:03:51.223911 front 2020-08-07 10:03:51.224322 (oldest deadline 2020-08-07 10:04:35.322704) osd.243 says: 2020-08-07 10:03:55.065 7f0d33911700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:03:55.065 7f0d34112700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 [.. ~3000(!) Lines ..] 2020-08-07 10:04:33.644 7f0d33110700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:33.688 7f0d13acb700 0 bluestore(/var/lib/ceph/osd/ceph-243) log_latency_fn slow operation observed for upper_bound, latency = 20.9013s, after = omap_iterator(cid = 19.58a_head, oid = #19:51a21a27::: .dir.default.223091333.1.3:head#) 2020-08-07 10:04:33.688 7f0d13acb700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [DBG] : map e2683535 wrongly marked me down at e2683534 * as a consequence, old deep-scrubs did not finish, because they would be interrupted -> ' pgs not deep-scrubbed in time' for the latter, I increased the op-thread-timeout back to the pre 12(!).2.11 value of 30 i`m am not sure, if we really have a problem, but it does not look healthy. Any ideas, thoughts? regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

4 6

i would love more this content
by chiaranjun＠gmail.com 09 Aug '20

09 Aug '20

Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.Here are some ineteresting content i have to share as well; <a href="https://pomeranianhaustierhaus.com/product/pomeranian-for-sale-near-me/"rel="dofollow">pomeranian for sale near me</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-chihuahua-mix/"rel="dofollow">pomeranian chihuahua mix</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-for-sale/"rel="dofollow">pomeranian for sale</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-on-sale/"rel="dofollow">pomeranian on sale</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-black-and-white/"rel="dofollow">pomeranian black and white</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-miniature/"rel="dofollow">pomeranian miniature</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-black/"rel="dofollow">pomeranian black</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-adoption/"rel="dofollow">pomeranian adoption</a> <a href="https://pomeranianhaustierhaus.com/pomeranian-puppy/"rel="dofollow">pomeranian puppy</a>

4 5

Block Network IP Of Wireless HP Printer Using HP Support Assistant.
by mary smith 08 Aug '20

08 Aug '20

Yes, you should! If you are one of those who are have a word with the professionals who are capable of resolving the whole host of network problems. You can make use of HP Support Assistant who will solve the whole host of wireless printer problems. https://www.amiytech.com/hp-support-assistant/

1 0

Can I Unlock Yahoo Account without a telephone number?
by mary smith 08 Aug '20

08 Aug '20

While you are making a record on Yahoo messaging stage, at that point it will request an elective email address just as a portable number to send a confirmation code. Along these lines, you have compulsory to give all subtleties accurately and furthermore set a secret key. In the event that you feel that, would I be able to Unlock Yahoo Account without a telephone number? Thus, indeed, you can, yet you need to choose an elective alternative to get a confirmation code. https://www.contact-customer-service.net/blog/unlock-yahoo-account/

1 0

How to Get Refund for Cash App Payment Pending
by itisghana＠gmail.com 07 Aug '20

07 Aug '20

Welcome to Cash App Support Center +1-888-526-0829 Welcome to Cash App Support Center +1-888-526-0829 Welcome to Cash App Support Center +1-888-526-0829 https://cashapphelp.support/

1 0

RGW Garbage Collection (GC) does not make progress
by Wido den Hollander 07 Aug '20

07 Aug '20

Hi, On a Nautilus 14.2.8 cluster I'm seeing a large amount of GC data and the GC on the RGW does not seem to make progress. The .rgw.gc pool contains 39GB of data spread out over 32 objects. In the logs we do see references of the RGW GC doing work and it says it is removing objects. Those objects however still exist and only their 'refcount' attribute is updated. 2020-08-07 10:28:01.946 7fbd79f9a7c0 5 garbage collection: RGWGC::process removing .rgw.buckets.ec-v2:default.1834866551.1__multipart_fedora-data/datastreamStore/XX-YYY-5/5c/f3/info%3Afedora%2FCH-001514-5%3A36%2FORIGINAL%2FORIGINAL.0.2~yKGz1-SLXINhZvm3cQMBWgx9BJVoH5j.1 2020-08-07 10:28:01.946 7fbd79f9a7c0 5 garbage collection: RGWGC::process removing .rgw.buckets.ec-v2:default.1834866551.1__shadow_fedora-data/datastreamStore/XXX-YYY-5/5c/f3/info%3Afedora%2FCH-001514-5%3A36%2FORIGINAL%2FORIGINAL.0.2~yKGz1-SLXINhZvm3cQMBWgx9BJVoH5j.1_1 This objects however still exist, 'rados stat' shows me: mtime 2020-08-07 12:28:44.000000, size 4194304 Has anybody seen this before and has clues on what this could be? Wido

1 0

Get an Error Free Printing with the Help of HP Support
by pa6724453＠gmail.com 07 Aug '20

07 Aug '20

Printing of a document or a picture is not a big task but sometimes errors make this one very difficult. A minor problem may stop the whole printing task. If you are using HP printer then support.hp.com is the best option for you to get rid of all printing problems. You will get complete solution for every single query regarding HP printers. https://www.hpprintersupportpro.us/

1 0

How i can use bucket policy with subuser
by hoannv46＠gmail.com 07 Aug '20

07 Aug '20

Hi all. I have a cluster 14.2.10 versions. First, I create user hoannv radosgw-admin user create --uid=hoannv --display-name=hoannv Then create subuser hoannv:subuser1 by command : radosgw-admin subuser create --uid=hoannv --subuser=subuser1 --key-type=swift --gen-secret --access=full hoannv user create bucket : test1 hoannv:subuser1 subuser also full access permission to test1 bucket I want to subuser has no permission then i can use bucket policy to subuser. How i can do this? Thanks.

1 0

Problem with Cash App activity tab? Contact support to talk to a Cash App representative.
by jofraarcher1997＠gmail.com 07 Aug '20

07 Aug '20

The activity tab of the app is one of the features that help you with the refund issues. But if you can’t use the app’s feature, then you can get in touch with the tech support and talk to a Cash App representative or you can proceed to watch some tech videos and get troubleshooting solutions to fix the problem.https://bitcoin-customerservice.com/cash-app-customer-service-numbe…

11 11

Approach Epson Customer Service If Unable To Print Any Document.
by mary smith 07 Aug '20

07 Aug '20

Are you facing any kind of problems and hurdles during the course of epson printer? Consider taking help directly from the Epson Customer Service who will help you to resolve all your problems within a few seconds. So, you need instant technical help from the professionals who will surely assist you out in no time. https://www.epsonprintersupportpro.net/

1 0

Are Yahoo Mail Down deleted emails forever?
by mary smith 07 Aug '20

07 Aug '20

If you delete a message from Inbox, then it moves into the Trash folder, but if you delete a message from the Trash folder then emails deleted forever from your Yahoo mail account. But, there is a backup created, from here you can restore it from any 60 days. If Yahoo Mail Down, then it only stops to access this site for a few times, not more. https://www.contact-customer-service.net/blog/yahoo-mail-down/

1 0

A high-class Hyderabad escort, who teams up with another escort for your ultimate fun
by sehgalshreya553＠gmail.com 07 Aug '20

07 Aug '20

Hi guys, thanks for visiting my profile page. I am Sherya a young high-class Hyderabad escort, whose ultimate objective is to offer the desired fun and entertainment. You can pick up two Hyderabad escorts from Hyderabad Escorts Service and believe me, when you visit this website, the options are really endless. I have no problem in teaming-up with another high-profile independent Hyderabad escort in a bid to satisfy you. Surely, two cuties are going to double your pleasure. You’ll definitely enjoy being sand witched between two cute and curvaceous bodies. Why be entertained by one Hyderabad call girl, when you can have two such beauties. I am among the few Hyderabad gals who mix business with pleasure. One of the main advantages of hiring me is my great skill to blend into any social situation. So, if there is a party you don’t want to go alone, I can be your true companion. For sure, I’ll be the center of attraction for the guests and most of the eyes will be focused on me. For the perfect companionship inside and outside of the bedroom, no option is better than me. Call right away. www.shreyasehgal.com/

1 0

Mix-up while sending money through Cash App? Talk to a Cash App representative.
by david william 07 Aug '20

07 Aug '20

If you've fail to send money on account of some tech issues, by then you can use the help from the specialized help districts or you can watch help chronicles on Youtube. Despite that, you can call the assistance gathering and select to talk to a Cash App representative for getting the issue settled. https://www.cash-app-customer-service.com/blog/talk-to-a-cash-app-represent…

1 0

How does the Cash app refund go into account immediately?
by david william 07 Aug '20

07 Aug '20

On the other hand, Square revived this application to keep cash into the Cash app balance, until you move it genuinely into the monetary equalization. To move cash into your budgetary parity which you get for any portion or Cash app refund, you need to press the Cash Out catch so to speak. https://www.contact-customer-service.net/blog/cash-app-refund/

1 0

Is it possible to rebuild a bucket instance?
by Bryan Stillwell 06 Aug '20

06 Aug '20

I have a cluster running Nautilus where the bucket instance (backups.190) has gone missing: # radosgw-admin metadata list bucket | grep 'backups.19[0-1]' | sort "backups.190", "backups.191", # radosgw-admin metadata list bucket.instance | grep 'backups.19[0-1]' | sort "backups.191:00f195a6-c5f9-440f-9deb-62a35bd2b695.5684060.2808", Is there a way to recreate the bucket instance? Perhaps by using the bucket instance from backups.191 as a basis? From what I can tell the bucket index is fine, but since the bucket instance is gone I get this error: # radosgw-admin bucket list --bucket=backups.190 could not get bucket info for bucket=backups.190 ERROR: could not init bucket: (2) No such file or directory 2020-08-06 16:29:23.457 7f9a625076c0 -1 ERROR: get_bucket_instance_from_oid failed: -2 Thanks, Bryan

1 0

Re: Ceph does not recover from OSD restart
by Frank Schilder 06 Aug '20

06 Aug '20

Hi Eric, yes, I had network restarts as well along the way. However, these should also not lead to the redundancy degradation I observed, it doesn't really explain why ceph lost track of so many objects. A temporary network outage on a server is an event that the cluster ought to survive without such damage. What does "transitioning to Stray" mean/indicate here? I did another test today and collected logs for a tracker issue. The problem can be reproduced and occurs if an "old" OSD is restarted, it does not happen when a "new" OSD restarts. Ceph seems to loose track of any placement information computed according to the original crush map from before adding OSDs. It looks like PG remappings are deleted when an OSD shuts down and can only be recovered for placements according to the new crush map, hence, permanent loss of information on restart of an "old" OSD. If one restores the original crush map for long enough, for example, by moving OSDs out of the sub-tree, the cluster can restore all PG remappings and restore full redundancy. Correct behaviour would be either to maintain the remappings until an OSD is explicitly purged from the cluster, or to check for object locations with respect to all relevant crush maps in the history. Another option would be, that every OSD checks on boot if it holds a copy of a certain version of an object that the cluster is looking for (reports as missing) and says "hey, I have it here" if found. This is, in fact, what I expected was implemented. The current behaviour is a real danger. Rebalancing after storage extension is *not* a degraded state, but a normal and usually lengthy maintenance operation. A network fail or host reboot during such operations should not cause havoc, in particular, not this kind of "virtual" data loss while all data is physically present and all hrdware is healthy. Objects that are present on some disk should be found automatically under any circumstances. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 05 August 2020 13:35:44 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart You have a LOT of state transitions during your maintenance and I'm not really sure why (There are a lot of complaints about the network). There's are also a lot of "transitioning to Stray" after initial startup of an OSD. I'd say let your cluster heal first before you start doing a ton a maintenance so old PG maps can be trimmed. That's the best I can ascertain from the logs for now. -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 8:35 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart If with monitor log you mean the cluster log /var/log/ceph/ceph.log, I should have all of it. Please find a tgz-file here: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2ffiles.dtu.dk%2fu%2ftFCE… (valid 100 days). Contents: logs/ceph-2020-08-03.log - cluster log for the day of restart logs/ceph-osd.145.2020-08-03.log - log of "old" OSD trimmed to day of restart logs/ceph-osd.288.log - entire log of "new" OSD Hope this helps. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 14:15:11 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Do you have any monitor / OSD logs from the maintenance when the issues occurred? -------- Original message -------- From: Frank Schilder <frans(a)dtu.dk> Date: 8/4/20 8:07 AM (GMT-05:00) To: Eric Smith <Eric.Smith(a)vecima.com>, ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for the clarification, I did misunderstand you. > You should not have to move OSDs in and out of the CRUSH tree however > in order to solve any data placement problems (This is the baffling part). Exactly. Should I create a tracker issue? I think this is not hard to reproduce with a standard crush map where host-bucket=physical host and I would, in fact, expect that this scenario is part of the integration test. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 13:58:47 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart All seems in order in terms of your CRUSH layout. You can speed up the rebalancing / scale-out operations by increasing the osd_max_backfills on each OSD (Especially during off hours). The unnecessary degradation is not expected behavior with a cluster in HEALTH_OK status, but with backfill / rebalancing ongoing it's not unexpected. You should not have to move OSDs in and out of the CRUSH tree however in order to solve any data placement problems (This is the baffling part). -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 7:45 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Erik, I added the disks and started the rebalancing. When I run into the issue, ca. 3 days after start of rebalancing, it was about 25% done. The cluster does not go to HEALTH_OK before the rebalancing is finished, it shows the "xxx objects misplaced" warning. The OSD crush locations for the logical hosts are in ceph.conf, the OSDs come up in the proper crush bucket. > All seems in order then In what sense? The rebalancing is still ongoing and usually a very long operation. This time I added only 9 disks, but we will almost triple the number of disks of a larger pool soon, which has 150 OSDs at the moment. I expect the rebalancing for this expansion to take months. Due to a memory leak, I need to restart OSDs regularly. Also, a host may restart or we might have a power outage during this window. In these situations, it will be a real pain if I have to play the crush move game with 300+ OSDs. This unnecessary redundancy degradation on OSD restart cannot possibly be expected behaviour, or do I misunderstand something here? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 13:19:41 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart All seems in order then - when you ran into your maintenance issue, how long was if after you added the new OSDs and did Ceph ever get to HEALTH_OK so it could trim PG history? Also did the OSDs just start back up in the wrong place in the CRUSH tree? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 7:10 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, > Have you adjusted the min_size for pool sr-rbd-data-one-hdd Yes. For all EC pools located in datacenter ServerRoom, we currently set min_size=k=6, because we lack physical servers. Hosts ceph-21 and ceph-22 are logical but not physical, disks in these buckets are co-located such that no more than 2 host buckets share the same physical host. With failure domain = host, we can ensure that no more than 2 EC shards are on the same physical host. With m=2 and min_size=k we have continued service with any 1 physical host down for maintenance and also recovery will happen if a physical host fails. Some objects will have no redundancy for a while then. We will increase min_size to k+1 as soon as we have 2 additional hosts and simply move the OSDs from buckets ceph-21/22 to these without rebalancing. The distribution of disks and buckets is listed below as well (longer listing). Thanks and best regards, Frank # ceph osd erasure-code-profile ls con-ec-8-2-hdd con-ec-8-2-ssd default sr-ec-6-2-hdd This is the relevant one: # ceph osd erasure-code-profile get sr-ec-6-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ServerRoom jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 Note that the pool sr-rbd-data-one (id 2) was created with this profile and later moved to SSD. Therefore, the crush rule does not match the profile's device class any more. These two are under different roots: # ceph osd erasure-code-profile get con-ec-8-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get con-ec-8-2-ssd crush-device-class=ssd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 Full physical placement information for OSDs under tree "datacenter ServerRoom": ---------------- ceph-04 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 243 ceph-04 1.8T SSD osd-phy1 247 ceph-21 1.8T SSD osd-phy2 254 ceph-04 1.8T SSD osd-phy3 256 ceph-04 1.8T SSD osd-phy4 286 ceph-04 1.8T SSD osd-phy5 287 ceph-04 1.8T SSD osd-phy6 288 ceph-04 10.7T HDD osd-phy7 48 ceph-04 372.6G SSD osd-phy8 264 ceph-21 1.8T SSD osd-phy9 84 ceph-04 8.9T HDD osd-phy10 72 ceph-21 8.9T HDD osd-phy11 145 ceph-04 8.9T HDD osd-phy14 156 ceph-04 8.9T HDD osd-phy15 168 ceph-04 8.9T HDD osd-phy16 181 ceph-04 8.9T HDD osd-phy17 0 ceph-21 8.9T HDD ---------------- ceph-05 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 240 ceph-05 1.8T SSD osd-phy1 249 ceph-22 1.8T SSD osd-phy2 251 ceph-05 1.8T SSD osd-phy3 255 ceph-05 1.8T SSD osd-phy4 284 ceph-05 1.8T SSD osd-phy5 285 ceph-05 1.8T SSD osd-phy6 289 ceph-05 10.7T HDD osd-phy7 49 ceph-05 372.6G SSD osd-phy8 265 ceph-22 1.8T SSD osd-phy9 74 ceph-05 8.9T HDD osd-phy10 85 ceph-22 8.9T HDD osd-phy11 144 ceph-05 8.9T HDD osd-phy14 157 ceph-05 8.9T HDD osd-phy15 169 ceph-05 8.9T HDD osd-phy16 180 ceph-05 8.9T HDD osd-phy17 1 ceph-22 8.9T HDD ---------------- ceph-06 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 244 ceph-06 1.8T SSD osd-phy1 246 ceph-21 1.8T SSD osd-phy2 253 ceph-06 1.8T SSD osd-phy3 257 ceph-06 1.8T SSD osd-phy4 282 ceph-06 1.8T SSD osd-phy5 283 ceph-06 1.8T SSD osd-phy6 40 ceph-06 372.6G SSD osd-phy7 50 ceph-06 372.6G SSD osd-phy8 60 ceph-06 8.9T HDD osd-phy9 290 ceph-06 10.7T HDD osd-phy10 291 ceph-21 10.7T HDD osd-phy11 146 ceph-06 8.9T HDD osd-phy14 158 ceph-06 8.9T HDD osd-phy15 170 ceph-06 8.9T HDD osd-phy16 182 ceph-06 8.9T HDD osd-phy17 2 ceph-21 8.9T HDD ---------------- ceph-07 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 242 ceph-07 1.8T SSD osd-phy1 250 ceph-22 1.8T SSD osd-phy2 252 ceph-07 1.8T SSD osd-phy3 258 ceph-07 1.8T SSD osd-phy4 279 ceph-07 1.8T SSD osd-phy5 280 ceph-07 1.8T SSD osd-phy6 292 ceph-07 10.7T HDD osd-phy7 52 ceph-07 372.6G SSD osd-phy8 63 ceph-07 8.9T HDD osd-phy9 281 ceph-22 1.8T SSD osd-phy10 87 ceph-22 8.9T HDD osd-phy11 148 ceph-07 8.9T HDD osd-phy14 159 ceph-07 8.9T HDD osd-phy15 172 ceph-07 8.9T HDD osd-phy16 183 ceph-07 8.9T HDD osd-phy17 3 ceph-22 8.9T HDD ---------------- ceph-18 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 241 ceph-18 1.8T SSD osd-phy1 248 ceph-18 1.8T SSD osd-phy2 41 ceph-18 372.6G SSD osd-phy3 31 ceph-18 372.6G SSD osd-phy4 277 ceph-18 1.8T SSD osd-phy5 278 ceph-21 1.8T SSD osd-phy6 53 ceph-21 372.6G SSD osd-phy7 267 ceph-18 1.8T SSD osd-phy8 266 ceph-18 1.8T SSD osd-phy9 293 ceph-18 10.7T HDD osd-phy10 86 ceph-21 8.9T HDD osd-phy11 259 ceph-18 10.9T HDD osd-phy14 229 ceph-18 8.9T HDD osd-phy15 232 ceph-18 8.9T HDD osd-phy16 235 ceph-18 8.9T HDD osd-phy17 238 ceph-18 8.9T HDD ---------------- ceph-19 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 261 ceph-19 1.8T SSD osd-phy1 262 ceph-19 1.8T SSD osd-phy2 295 ceph-19 10.7T HDD osd-phy3 43 ceph-19 372.6G SSD osd-phy4 275 ceph-19 1.8T SSD osd-phy5 294 ceph-22 10.7T HDD osd-phy6 51 ceph-22 372.6G SSD osd-phy7 269 ceph-19 1.8T SSD osd-phy8 268 ceph-19 1.8T SSD osd-phy9 276 ceph-22 1.8T SSD osd-phy10 73 ceph-22 8.9T HDD osd-phy11 263 ceph-19 10.9T HDD osd-phy14 231 ceph-19 8.9T HDD osd-phy15 233 ceph-19 8.9T HDD osd-phy16 236 ceph-19 8.9T HDD osd-phy17 239 ceph-19 8.9T HDD ---------------- ceph-20 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 245 ceph-20 1.8T SSD osd-phy1 28 ceph-20 372.6G SSD osd-phy2 44 ceph-20 372.6G SSD osd-phy3 271 ceph-20 1.8T SSD osd-phy4 272 ceph-20 1.8T SSD osd-phy5 273 ceph-20 1.8T SSD osd-phy6 274 ceph-21 1.8T SSD osd-phy7 296 ceph-20 10.7T HDD osd-phy8 76 ceph-21 8.9T HDD osd-phy9 39 ceph-21 372.6G SSD osd-phy10 270 ceph-20 1.8T SSD osd-phy11 260 ceph-20 10.9T HDD osd-phy14 228 ceph-20 8.9T HDD osd-phy15 230 ceph-20 8.9T HDD osd-phy16 234 ceph-20 8.9T HDD osd-phy17 237 ceph-20 8.9T HDD CONT is the container name and encodes the physical slot on the host where the OSD is located. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 12:47:12 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Have you adjusted the min_size for pool sr-rbd-data-one-hdd at all? Also can you send the output of "ceph osd erasure-code-profile ls" and for each EC profile, "ceph osd erasure-code-profile get <profile>"? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 11:05 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Sorry for the many small e-mails: requested IDs in the commands, 288-296. One new OSD per host. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 16:59:04 To: Eric Smith; ceph-users Subject: [ceph-users] Re: Ceph does not recover from OSD restart Hi Eric, the procedure for re-discovering all objects is: # Flag: norebalance ceph osd crush move osd.288 host=bb-04 ceph osd crush move osd.289 host=bb-05 ceph osd crush move osd.290 host=bb-06 ceph osd crush move osd.291 host=bb-21 ceph osd crush move osd.292 host=bb-07 ceph osd crush move osd.293 host=bb-18 ceph osd crush move osd.295 host=bb-19 ceph osd crush move osd.294 host=bb-22 ceph osd crush move osd.296 host=bb-20 # Wait until all PGs are peered and recovery is done. In my case, there was only little I/O, # no more than 50-100 objects had writes missing and recovery was a few seconds. # # The bb-hosts are under a separate crush root that I use solely as parking space # and for draining OSDs. ceph osd crush move osd.288 host=ceph-04 ceph osd crush move osd.289 host=ceph-05 ceph osd crush move osd.290 host=ceph-06 ceph osd crush move osd.291 host=ceph-21 ceph osd crush move osd.292 host=ceph-07 ceph osd crush move osd.293 host=ceph-18 ceph osd crush move osd.295 host=ceph-19 ceph osd crush move osd.294 host=ceph-22 ceph osd crush move osd.296 host=ceph-20 After peering, no degraded PGs/objects any more, just the misplaced ones as expected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 16:45:28 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart You said you had to move some OSDs out and back in for Ceph to go back to normal (The OSDs you added). Which OSDs were added? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:55 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

OSD Shard processing operations slowly
by João Victor Mafra 06 Aug '20

06 Aug '20

Hi, I'm running FIO benchmark to test my simple cluster (3 OSD's, 128 pg's - using Nautilus - v14.2.10) and after certain load of clients performing random read operations, the OSDs show very different performances in terms of op latency. In extreme cases there is an OSD that performs much worse than the others, despite receiving a similar number of operations. Getting more information on the distribution of operations, I can see that the operations are well distributed among the OSD's and the PG's, but in the OSD with poor performance, there is an internal queue (OSD Shard) that is dispatching requests very slowly. In my use case, for example, there is a OSD shard whose average wait time for operations was 120 ms and a OSD Shard that served a few more requests with an average wait time of 1.5 sec. The behavior of this queue ends up affecting the performance of ceph as a whole. The osd op queue implementation used is wpq, and during the execution I get a specific attribute of this queue (probably total_priority) that remains unchanged for a long time. The strange behavior is also repeated in other implementations (prio, m_clock). I've used the mimic version, another pg's distribution and the behavior is always the same, but it can happen in a different OSD or in a different shard. By default, the OSD has 5 shards. Increasing the number of shards considerably improves the performance of this OSD, but I would like to understand what is happening with this specific queue in the default configuration. Does anyone have any idea what might be happening? Thanks, Mafra.

1 0

Quick interruptions in the Ceph cluster
by Gesiel Galvão Bernardes 06 Aug '20

06 Aug '20

Hi, I have been experiencing rapid outage events in the Ceph cluster. During these events I receive messages from slow ops, OSD downs, but at the same time it is operating. Magically everything is back to normal. These events usually last about 2 minutes. I couldn't find anything that could direct me to what causes these events, can you help me? I'm using Mimic (13.2.6) and CentOS7 on all nodes. Below is "ceph -s" and log of when event occurs: # ceph -s cluster: id: 4ea72929-6f9e-453a-8cd5-bb0712f6b874 health: HEALTH_OK services: mon: 2 daemons, quorum cmonitor,cmonitor2 mgr: cmonitor(active), standbys: cmonitor2 osd: 74 osds: 74 up, 74 in tcmu-runner: 10 daemons active data: pools: 7 pools, 3072 pgs objects: 22.17 M objects, 83 TiB usage: 225 TiB used, 203 TiB / 428 TiB avail pgs: 3063 active+clean 9 active+clean+scrubbing+deep ====================================== Log of event: 2020-08-05 18:00:00.000179 [INF] overall HEALTH_OK 2020-08-05 17:55:28.905024 [INF] Cluster is now healthy 2020-08-05 17:55:28.904975 [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/60350974 objects degraded (0.000%), 1 pg degraded) 2020-08-05 17:55:27.746606 [WRN] Health check update: Degraded data redundancy: 1/60350974 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2020-08-05 17:55:22.745820 [WRN] Health check update: Degraded data redundancy: 55/60350897 objects degraded (0.000%), 26 pgs degraded, 1 pg undersized (PG_DEGRADED) 2020-08-05 17:55:17.744218 [WRN] Health check update: Degraded data redundancy: 123/60350666 objects degraded (0.000%), 63 pgs degraded (PG_DEGRADED) 2020-08-05 17:55:12.743568 [WRN] Health check update: Degraded data redundancy: 192/60350660 objects degraded (0.000%), 88 pgs degraded (PG_DEGRADED) 2020-08-05 17:55:07.741759 [WRN] Health check update: Degraded data redundancy: 290/60350737 objects degraded (0.000%), 117 pgs degraded (PG_DEGRADED) 2020-08-05 17:55:02.737913 [WRN] Health check update: Degraded data redundancy: 299/60350764 objects degraded (0.000%), 119 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:57.736694 [WRN] Health check update: Degraded data redundancy: 299/60350746 objects degraded (0.000%), 119 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:52.736132 [WRN] Health check update: Degraded data redundancy: 299/60350731 objects degraded (0.000%), 119 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:47.735612 [WRN] Health check update: Degraded data redundancy: 299/60350689 objects degraded (0.000%), 119 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:42.734877 [WRN] Health check update: Degraded data redundancy: 301/60350677 objects degraded (0.000%), 120 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:38.210906 [INF] Health check cleared: SLOW_OPS (was: 35 slow ops, oldest one blocked for 1954017 sec, daemons [mon.cmonitor,mon.cmonitor2] have slow ops.) 2020-08-05 17:54:37.734218 [WRN] Health check update: 35 slow ops, oldest one blocked for 1954017 sec, daemons [mon.cmonitor,mon.cmonitor2] have slow ops. (SLOW_OPS) 2020-08-05 17:54:37.734132 [WRN] Health check update: Degraded data redundancy: 380/60350611 objects degraded (0.001%), 154 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:34.171483 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2020-08-05 17:54:32.733499 [WRN] Health check update: Degraded data redundancy: 52121/60350544 objects degraded (0.086%), 211 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:27.080529 [WRN] Monitor daemon marked osd.72 down, but it is still running 2020-08-05 17:54:32.102889 [WRN] Health check failed: 60 slow ops, oldest one blocked for 1954017 sec, daemons [osd.16,osd.22,osd.23,osd.27,osd.28,osd.29,osd.30,osd.35,osd.48,osd.5]... have slow ops. (SLOW_OPS) 2020-08-05 17:54:32.102699 [WRN] Health check update: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2020-08-05 17:54:27.951343 [INF] osd.72 192.168.200.25:6844/64565 boot 2020-08-05 17:54:27.935996 [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2020-08-05 17:54:27.732679 [WRN] Health check update: Degraded data redundancy: 1781748/60350443 objects degraded (2.952%), 381 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:26.916269 [WRN] Health check update: Reduced data availability: 5 pgs inactive, 10 pgs peering, 18 pgs incomplete (PG_AVAILABILITY) 2020-08-05 17:54:19.712714 [WRN] Monitor daemon marked osd.71 down, but it is still running 2020-08-05 17:54:22.716043 [WRN] Health check update: Degraded data redundancy: 4057630/60350485 objects degraded (6.723%), 566 pgs degraded (PG_DEGRADED) 2020-08-05 17:54:22.715939 [WRN] Health check update: 1 osds down (OSD_DOWN) 2020-08-05 17:54:14.042363 [WRN] Monitor daemon marked osd.62 down, but it is still running 2020-08-05 17:54:20.858582 [WRN] Health check update: Reduced data availability: 8 pgs inactive, 15 pgs peering, 37 pgs incomplete (PG_AVAILABILITY)

2 1

snaptrim blocks IO on ceph nautilus
by Manuel Lausch 06 Aug '20

06 Aug '20

Hi, since some days I try to debug a problem with snaptrimming under nautilus. I have a cluster with Nautilus (v14.2.10) , 44 Nodes á 24 OSDs á 14 TB I create every day a snapshot for 7 days. Every time the old snapshot is deleting I have bad IO performcance and blocked requests for several seconds until the snaptrim is done. Settings like snaptrim_sleep and osd_pg_max_concurrent_snap_trims don't affect this behavior. In the debug_osd 10/10 log I see the following: 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edda20 prio 196 cost 0 latency 0.019545 osd_repop_reply(client.22731418.0:615257 3.636 e22457/22372) v2 pg pg[3.636( v 22457'100855 (21737'97756,22457'100855] local-lis/les=22372/22374 n=27762 ec=2842/2839 lis/c 22372/22372 les/c/f 22374/22374/0 22372/22372/22343) [411,36,956,763] r=0 lpr=22372 luod=22457'100854 crt=22457'100855 lcod 22457'100853 mlcod 22457'100853 active+clean+snaptrim_wait trimq=[1d~1]] 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edda20 finish 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edc2c0 prio 127 cost 0 latency 0.043165 MOSDScrubReserve(2.2645 RELEASE e22457) v1 pg pg[2.2645( empty local-lis/les=22359/22364 n=0 ec=2403/2403 lis/c 22359/22359 les/c/f 22364/22367/0 22359/22359/22359) [379,411,884,975] r=1 lpr=22359 crt=0'0 active mbc={}] 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edc2c0 finish 2020-07-27 11:45:50.039 7fd8b8404700 10 osd.411 pg_epoch: 22457 pg[3.278e( v 22457'99491 (21594'96426,22457'99491] local-lis/les=22359/22362 n=27669 ec=2859/2839 lis/c 22359/22359 les/c/f 22362/22365/0 22359/22359/22343) [411,379,848,924] r=0 lpr=22359 crt=22457'99491 lcod 22457'99489 mlcod 22457'99489 active+clean+snaptrim trimq=[1d~1]] snap_trimmer posting 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 pg_epoch: 22457 pg[3.278e( v 22457'99493 (21594'96426,22457'99493] local-lis/les=22359/22362 n=27669 ec=2859/2839 lis/c 22359/22359 les/c/f 22362/22365/0 22359/22359/22343) [411,379,848,924] r=0 lpr=22359 luod=22457'99491 crt=22457'99493 lcod 22457'99489 mlcod 22457'99489 active+clean+snaptrim trimq=[1d~1]] snap_trimmer complete 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557880ac3760 prio 127 cost 663 latency 7.761823 osd_repop(osd.217.0:3025 3.1ca5 e22457/22378) v2 pg pg[3.1ca5( v 22457'100370 (21716'97357,22457'100370] local-lis/les=22378/22379 n=27532 ec=2855/2839 lis/c 22378/22378 les/c/f 22379/22379/0 22378/22378/22378) [217,411,551,1055] r=1 lpr=22378 luod=0'0 lua=22294'100006 crt=22457'100370 lcod 22457'100369 active mbc={}] 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557880ac3760 finish 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x5578813e1e40 prio 127 cost 0 latency 7.494296 MOSDScrubReserve(2.37e2 REQUEST e22457) v1 pg pg[2.37e2( empty local-lis/les=22355/22356 n=0 ec=2412/2412 lis/c 22355/22355 les/c/f 22356/22356/0 22355/22355/22355) [245,411,834,768] r=1 lpr=22355 crt=0'0 active mbc={}] 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x5578813e1e40 finish the dequeueing of ops works without pauses until the „snap_trimmer posting“ and „snap_trimmer complete“ loglines. This task takes in this example about 7 Seconds. The following operations which are dequeued have now a latency of about this time. I tried to drill down this in the code. (Developers are asked here) It seems, that the PG will be locked for every operation. The snap_trimmer posting and complete message comes from „osd/PrimaryLogPG.cc“ on line 4700. This indicates me, that the process of deleting a snapshot object will sometimes take some time. After further poking around. I see in „osd/SnapMapper.cc“ the method „SnapMapper::get_next_objects_to_trim“ which takes several seconds to get finished. I followed this further to the „common/map_cacher.hpp“ to the line 94: „int r = driver->get_next(key, &store);“ From there I lost the path. The slowness is not on all OSDs at the same time. Somteime, this few OSDs are affected, sometimes some others. Restart of an OSD does not help. With luminous and filestore, snapshot deletion was not an issue at all. With nautilus and bluestore this is not acceptable for my usecase. I don‘t know so far, if this is a bluestore specific problem or some general issue. I wonder a bit why there are no other who have this problem. Regards Manuel

1 2

Error in Cash App tabs? Contact help team and talk to a Cash App representative.
by jofraarcher1997＠gmail.com 06 Aug '20

06 Aug '20

The tabs are the features in the app that assist you in performing variety of tasks. But if you can’t use the tabs, then you must get in touch with the customer care and talk to a Cash App representative to get the issue resolved. In addition to that, you can also use tech support to get the matter resolved.https://www.yahoohome-page.com/cash-app-customer-service/

1 0

Cash App icon error? Talk to a Cash App representative for assistance.
by jofraarcher1997＠gmail.com 06 Aug '20

06 Aug '20

To perform any operation you first need to get inside the app and that can only be done by tapping on the icon. But if the icon is unresponsive, then you can get assistance by tech support sites or you can also try rebooting your device. You can talk to a Cash App representative to get the issue resolved.https://www.experts-support.com/cash-app-customer-service/

1 0

Fix Cash App touch Id feature? Talk to a Cash App representative for help.
by jofraarcher1997＠gmail.com 06 Aug '20

06 Aug '20

The touch Id feature of the app lets you approve the transaction by recognizing your fingerprints. But if the Id isn’t working, then you can use the assistance that is provided by various tech support sites or you can dial the tech support number to talk to a Cash App representative in order to get the error fixed.https://gettosupport.net/cash-app-customer-service/

1 0

Scanner not working in Cash App? Find support by picking to talk to a Cash App representative.
by jofraarcher1997＠gmail.com 06 Aug '20

06 Aug '20

One of the features of the application to help you with filling the nuances of the recipient is the scanner. Notwithstanding, if you can't use it as a result of some goof, by then you can get the fundamental assistance from the customer care site by picking to talk to a Cash App representative. Another choice is to look at to the help arrange for outlines.https://gettosupport.net/cash-app-payment-failed/

1 0

Issue with attestation in Cash App? For help talk to a Cash App representative.
by jofraarcher1997＠gmail.com 06 Aug '20

06 Aug '20

Affirmation is a key bit of the application. Before long, if you can't confirm a trade, by then you can use a few plans from the customer care by choosing to talk to a Cash App representative and get the tech issue settled. You can in like manner don't extra a second to inspect to the help site.https://www.pcmonks.net/blog/how-to-get-cash-app-refund/

1 0

Contact Id not obliging in Cash App? Talk to a Cash App representative.
by jofraarcher1997＠gmail.com 06 Aug '20

06 Aug '20

The touch Id feature of the application is one of the most colossal features as it gives you the adaptability to embrace a trade. If you can't, by then you can use the help and help that is found in the customer brain and pick to talk to a Cash App representative or you can research to the particular help site for additional assistance.https://www.pcmonks.net/blog/how-to-cancel-cash-app-payment/

1 0

Re: I can just add 4Kn drives, not?
by Martin Verges 06 Aug '20

06 Aug '20

Yes, no problem -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Do., 6. Aug. 2020 um 12:13 Uhr schrieb Marc Roos < M.Roos(a)f1-outsourcing.eu>: > > > I can just add 4Kn drives to my existing setup not? Since this > technology is only specific to how the osd daemon is talking to the > disk? > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

I can just add 4Kn drives, not?
by Marc Roos 06 Aug '20

06 Aug '20

I can just add 4Kn drives to my existing setup not? Since this technology is only specific to how the osd daemon is talking to the disk?

1 0

Bluestore cache size, bluestore cache settings with nvme
by Szabo, Istvan (Agoda) 06 Aug '20

06 Aug '20

Hi, Any idea what is going on if the "bluestore_cache_autotune" true, "bluestore_cache_size" 0 but the server has only nvme? Because if the bluestore_cache_size 0 it means it will pick ssd or hdd, but if no ssd and hdd, then what will be going on if autotune true? How I should size this number? Any help? Also is it good to use numa or not? On the internet different information is going around this topic. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Re: help me enable ceph iscsi gatewaty in ceph octopus
by Sebastian Wagner 06 Aug '20

06 Aug '20

hi David, hi Ricardo, I think we first have to clarify, if that was actually a cephadm deployment (and not ceph-ansible). If you install Ceph using ceph-ansible, then please refer to the ceph-ansible docs. If we're actually talking about cephadm here (which is not clear to me): iSCSI for cephadm will land in the next octopus release and at that point we can add a proper documentation. Hope that helps, Sebastian Am 05.08.20 um 11:11 schrieb Ricardo Marques: > Hi David, > > I was able to configure iSCSI gateways on my local test environment using the following spec: > > ``` > # tail -14 service_spec_gw.yml > --- > service_type: iscsi > service_id: iscsi_service > placement: > hosts: > - 'node1' > - 'node2' > spec: > pool: rbd > trusted_ip_list: 10.20.94.201,10.20.94.202,10.20.94.203 > api_port: 5000 > api_user: admin1 > api_password: admin2 > api_secure: False > > # ceph orch apply -i service_spec_gw.yml > ``` > > You can use this spec as a starting point, but note that the pool must exist (in this case `rbd` pool), and you will need to adapt `hosts`, `trusted_ip_list`, etc... > > You may also want to change `api_secure` to `True` and set `ssl_cert` and `ssh_key` accordingly. > > Unfortunately, iSCSI deployment is not included in the documentation yet (coming soon). > > [1] https://docs.ceph.com/docs/octopus/cephadm/install/ > > > Ricardo Marques > > ________________________________ > From: David Thuong <davidthuong2424(a)gmail.com> > Sent: Wednesday, August 5, 2020 5:16 AM > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] help me enable ceph iscsi gatewaty in ceph octopus > > Please help me enable ceph iscsi gatewaty in ceph octopus . when i install ceph complete . i see iscsi gateway not enable. please help me config it > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer

4 5

Many scrub errors after update to 14.2.10
by Andreas Haupt 06 Aug '20

06 Aug '20

Hi *, after updating our CEPH cluster from 14.2.9 to 14.2.10 it accumulates scrub errors on multiple osds: [cephmon1] /root # ceph health detail HEALTH_ERR 6 scrub errors; Possible data damage: 6 pgs inconsistent OSD_SCRUB_ERRORS 6 scrub errors PG_DAMAGED Possible data damage: 6 pgs inconsistent pg 3.69 is active+clean+inconsistent, acting [59,65,61] pg 3.73 is active+clean+inconsistent, acting [73,88,25] pg 12.29 is active+clean+inconsistent, acting [55,92,42] pg 12.38 is active+clean+inconsistent, acting [150,42,13] pg 12.46 is active+clean+inconsistent, acting [55,18,84] pg 12.75 is active+clean+inconsistent, acting [55,155,49] They all can easily get repaired (ceph pg repair $pg) - but I wonder what could be the source of the problem. The cluster started with Luminous some years ago, was updated to Mimic, then Nautilus. Never seen this before! OSDs are a mixture of HDD/SSD, both are affected. All on Bluestore. Any idea? Was there maybe a code change between 14.2.9 & 14.2.10 that could explain this? Errors in syslog look like this: Aug 5 19:21:21 krake08 ceph-osd: 2020-08-05 19:21:21.831 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 scrub : stat mismatch, got 74/74 objects, 20/20 clones, 74/74 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 182904850/172877842 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. Aug 5 19:21:21 krake08 ceph-osd: 2020-08-05 19:21:21.831 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 scrub 1 errors Aug 6 08:28:44 krake08 ceph-osd: 2020-08-06 08:28:44.477 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 repair : stat mismatch, got 76/76 objects, 22/22 clones, 76/76 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 183166994/173139986 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. Aug 6 08:28:44 krake08 ceph-osd: 2020-08-06 08:28:44.477 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 repair 1 errors, 1 fixed Thanks in advance, Andreas -- | Andreas Haupt | E-Mail: andreas.haupt(a)desy.de | DESY Zeuthen | WWW: http://www-zeuthen.desy.de/~ahaupt | Platanenallee 6 | Phone: +49/33762/7-7359 | D-15738 Zeuthen | Fax: +49/33762/7-7216

1 0

Get Cash App Customer Service to ask to for what reason is my cash not appearing?
by mary smith 06 Aug '20

06 Aug '20

In the event that you are not seeing your cash into your Cash app balance, at that point you have to guarantee that, is there any telephone number or email address that is related to you. Now and again, it happens cash goes to another telephone number or email address which is lined up with you. In the event that you get any issue here, at that point go to converse with Cash App Customer Service. https://www.customercare-email.com/customer-service/cash-app.html

1 0

Does Cash App Customer Service request for SSN?
by mary smith 06 Aug '20

06 Aug '20

Indeed, Cash App Customer Service requests your SSN to check your identity. It is fundamentally requested any government hostile to illegal tax avoidance and against tax avoidance. Here is a law, that makes it important to give your SSN. You don't have a need to enter your full SSN, enter just last four digits of it. Having such kind of warning permits them to affirm your identity, and by which they can easily catch dubious action. https://www.cash-app-customer-service.com/

1 0

Ceph influxDB support versus Telegraf Ceph plugin?
by victorhooi＠yahoo.com 06 Aug '20

06 Aug '20

Hi, I've read that Ceph has some InfluxDB reporting capabilities inbuilt (https://docs.ceph.com/docs/master/mgr/influx/) However, Telegraf, which is the system reporting daemon for InfluxDB, also has a Ceph plugin (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph) Just curious what people's thoughts on the two are, or what they are using in production? Which is easier to deploy/maintain, have you found? Or more useful for alerting, or tracking performance gremlins? Thanks, Victor

2 2

made a huge mistake, seeking recovery advice (osd zapped)
by Peter Sarossy 06 Aug '20

06 Aug '20

hey folks, I was deploying a new set of NVMe cards into my cluster, and while getting the new devices ready, it seems the device names got mixed up, and I managed to to run "sgdisk --zap-all" and "dd if=/dev/zero of="/dev/sd" bs=1M count=100" on some of the active devices. I was adding new cards so I could migrate off the 2k+2m erasure coded setup to a more redundant config, but in my mess up I ran the commands above on 3 of the 4 devices before the ceph status changed and I noticed the mistake. I managed to restore the LVM partition table from backup but it seems to not be enough to restart the OSD... I just need to recover one of the 3 drives to save all of my VM+Docker backing filesystem. I'm running on Kubernetes with Rook, after restoring the partition table it seems to be starting up ok, but then I get a stack trace and the container goes into Error state: https://pastebin.com/5wk1bKy9 Any ideas how to fix this? Or somehow extract the data and put it back together? -- Cheers, Peter Sarossy Technical Program Manager Data Center Data Security - Google LLC.

1 0

Authorized UV Printer Supplier in India - PH UV Printer
by ela1789brown＠gmail.com 06 Aug '20

06 Aug '20

If you are really searching for the best and most reputed UV printer supplier in India then there is only one name i.e. PH UV Printer https://phuvprinter.com/. it is the only UV printer supplier having the wide range of best quality UV printers allows you to print custom design over your products. With the help of this best quality UV printing you can easily print custom design over your products to make them more unique. If you want to know more then visit the official website of PH UV Printer. For more entertaining content visit: https://shayarikapitara.com/sorry-shayari/

1 0

librbd Image Watcher Errors
by Ml Ml 05 Aug '20

05 Aug '20

Hello, with cron i run backups with backurne (https://github.com/JackSlateur/backurne) which is rbd based. Sometimes i get those messages: 2020-08-05T18:42:18.915+0200 7fcdbd7fa700 -1 librbd::ImageWatcher: 0x7fcda400a6a0 image watch failed: 140521330717776, (107) Transport endpoint is not connected 2020-08-05T18:42:18.915+0200 7fcdbd7fa700 -1 librbd::Watcher: 0x7fcda400a6a0 handle_error: handle=140521330717776: (107) Transport endpoint is not connected 2020-08-05T18:45:13.496+0200 7fcdbd7fa700 -1 librbd::ImageWatcher: 0x7fcda400a6a0 image watch failed: 140521330717776, (110) Connection timed out 2020-08-05T18:45:13.496+0200 7fcdbd7fa700 -1 librbd::Watcher: 0x7fcda400a6a0 handle_error: handle=140521330717776: (110) Connection timed out 2020-08-05T18:58:01.408+0200 7fc66ffff700 -1 librbd::ImageWatcher: 0x7fc65c00a770 image watch failed: 140490057987152, (107) Transport endpoint is not connected 2020-08-05T18:58:01.408+0200 7fc66ffff700 -1 librbd::Watcher: 0x7fc65c00a770 handle_error: handle=140490057987152: (107) Transport endpoint is not connected Any idea what this indicates and how it could be solved? Thanks, Michael

1 0

RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade
by Chris Palmer 05 Aug '20

05 Aug '20

I have an RGW bucket (backups) that is versioned. A nightly job creates a new version of a few objects. There is a lifecycle policy (see below) that keeps 18 days of versions. This has been working perfectly and has not been changed. Until I upgraded Octopus... The nightly job creates separate log files, including a listing of the object versions. From these I can see that: 13/7 02:14 versions from 13/7 01:13 back to 24/6 01:17 (correct) 14/7 02:14 versions from 14/7 01:13 back to 25/6 01:14 (correct) 14/7 10:00 upgrade Octopus 15.2.3 -> 15.2.4 15/7 02:14 versions from 15/7 01:13 back to 25/6 01:14 (would have expected 25/6 to have expired) 16/7 02:14 versions from 16/7 01:13 back to 15/7 01:13 (now all pre-upgrade versions have wrongly disappeared) It's not a big deal for me as they are only backups, providing it continues to work correctly from now on. However it may affect some other people much more. Any ideas on the root cause? And if it is likely to be stable again now? Thanks, Chris { "Rules": [ { "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "Expiration & incomplete uploads", "Prefix": "", "Status": "Enabled", "NoncurrentVersionExpiration": { "NoncurrentDays": 18 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } } ] }

4 9

module cephadm has failed
by Daniel Davidson 05 Aug '20

05 Aug '20

I am having trouble getting rid of an error after creating a new ceph cluster. The error is: Module 'cephadm' has failed: auth get failed: failed to find client.crash.ceph-0 in keyring retval: -2 Checking the keyrings and disguising the keys: # ceph auth ls ... client.crash.ceph-0.data.igb.illinois.edu key: keyA caps: [mgr] profile crash caps: [mon] profile crash ... mgr.ceph-0.uymvya key: keyB caps: [mds] allow * caps: [mon] profile mgr caps: [osd] allow * ... Then on the node: # cat mgr.ceph-0.uymvya/keyring [mgr.ceph-0.uymvya] key = keyB caps mds = "allow *" caps mon = "profile mgr" caps osd = "allow *" So the keys match where they should, but I am thinking that the domain missing off the mgr might be the issue. This is keeping me from getting rados working since the cluster is showing a health issue. It looks like this is an issue in my version of ceph (15.2.4)? Can someone point me to the fix or at minimum tell me how to remove the mgr on ceph-0 from the config so that I can get the system healthy. thanks, Dan

1 0

Abysmal performance in Ceph cluster
by Loschwitz,Martin Gerhard 05 Aug '20

05 Aug '20

Folks, we’re building a Ceph cluster based on HDDs with SSDs for WAL/DB files. We have four nodes with 8TB disks and two SSDs and four nodes with many small HDDs (1.4-2.7TB) and four SSDs for the journals. HDDs are configured as RAID 0 on the controllers with writethrough enabled. I am writing this e-mail as we see absolutely catastrophic performance on the cluster (I am talking about anything between literally no throughput for seconds to 200mb/s, wildly varying). We’ve checked very single layer: network is far from being saturated (we have 25gbit/s uplinks and we can confirm that they deliver 25gbit/s). Using iostat, we can prove that during a „rados bench“ call neither the SSDs nor the actual hard disks are anywhere near 100% disk utilization. Usually usage does not exceed 55%. Servers are Dell RX740d with PERC onboard controllers. We also have four SSD-only nodes. When benchmarking against a pool on these, I reliably get 400-500 mb/s when doing the same 64k size test that I ran against the HDD pool. We’ve tried a number of things such as enabling the BBWC on the RAID controllers to no major success. „ceph tell osd.X bench“ will show 55-100 iops for HDDs even though their journals are on SSDs. We have also tried disabling the SSD’s write cache (the ones with the journals on them) to no success. Any pointer to what we may haver overseen would be greatly appreciated. Best regards Martin

1 0

Re: Ceph does not recover from OSD restart
by Frank Schilder 05 Aug '20

05 Aug '20

If with monitor log you mean the cluster log /var/log/ceph/ceph.log, I should have all of it. Please find a tgz-file here: https://files.dtu.dk/u/tFCEZJzQhH2mUIRk/logs.tgz?l (valid 100 days). Contents: logs/ceph-2020-08-03.log - cluster log for the day of restart logs/ceph-osd.145.2020-08-03.log - log of "old" OSD trimmed to day of restart logs/ceph-osd.288.log - entire log of "new" OSD Hope this helps. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 14:15:11 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Do you have any monitor / OSD logs from the maintenance when the issues occurred? -------- Original message -------- From: Frank Schilder <frans(a)dtu.dk> Date: 8/4/20 8:07 AM (GMT-05:00) To: Eric Smith <Eric.Smith(a)vecima.com>, ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for the clarification, I did misunderstand you. > You should not have to move OSDs in and out of the CRUSH tree however > in order to solve any data placement problems (This is the baffling part). Exactly. Should I create a tracker issue? I think this is not hard to reproduce with a standard crush map where host-bucket=physical host and I would, in fact, expect that this scenario is part of the integration test. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 13:58:47 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart All seems in order in terms of your CRUSH layout. You can speed up the rebalancing / scale-out operations by increasing the osd_max_backfills on each OSD (Especially during off hours). The unnecessary degradation is not expected behavior with a cluster in HEALTH_OK status, but with backfill / rebalancing ongoing it's not unexpected. You should not have to move OSDs in and out of the CRUSH tree however in order to solve any data placement problems (This is the baffling part). -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 7:45 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Erik, I added the disks and started the rebalancing. When I run into the issue, ca. 3 days after start of rebalancing, it was about 25% done. The cluster does not go to HEALTH_OK before the rebalancing is finished, it shows the "xxx objects misplaced" warning. The OSD crush locations for the logical hosts are in ceph.conf, the OSDs come up in the proper crush bucket. > All seems in order then In what sense? The rebalancing is still ongoing and usually a very long operation. This time I added only 9 disks, but we will almost triple the number of disks of a larger pool soon, which has 150 OSDs at the moment. I expect the rebalancing for this expansion to take months. Due to a memory leak, I need to restart OSDs regularly. Also, a host may restart or we might have a power outage during this window. In these situations, it will be a real pain if I have to play the crush move game with 300+ OSDs. This unnecessary redundancy degradation on OSD restart cannot possibly be expected behaviour, or do I misunderstand something here? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 13:19:41 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart All seems in order then - when you ran into your maintenance issue, how long was if after you added the new OSDs and did Ceph ever get to HEALTH_OK so it could trim PG history? Also did the OSDs just start back up in the wrong place in the CRUSH tree? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 7:10 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, > Have you adjusted the min_size for pool sr-rbd-data-one-hdd Yes. For all EC pools located in datacenter ServerRoom, we currently set min_size=k=6, because we lack physical servers. Hosts ceph-21 and ceph-22 are logical but not physical, disks in these buckets are co-located such that no more than 2 host buckets share the same physical host. With failure domain = host, we can ensure that no more than 2 EC shards are on the same physical host. With m=2 and min_size=k we have continued service with any 1 physical host down for maintenance and also recovery will happen if a physical host fails. Some objects will have no redundancy for a while then. We will increase min_size to k+1 as soon as we have 2 additional hosts and simply move the OSDs from buckets ceph-21/22 to these without rebalancing. The distribution of disks and buckets is listed below as well (longer listing). Thanks and best regards, Frank # ceph osd erasure-code-profile ls con-ec-8-2-hdd con-ec-8-2-ssd default sr-ec-6-2-hdd This is the relevant one: # ceph osd erasure-code-profile get sr-ec-6-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ServerRoom jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 Note that the pool sr-rbd-data-one (id 2) was created with this profile and later moved to SSD. Therefore, the crush rule does not match the profile's device class any more. These two are under different roots: # ceph osd erasure-code-profile get con-ec-8-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get con-ec-8-2-ssd crush-device-class=ssd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 Full physical placement information for OSDs under tree "datacenter ServerRoom": ---------------- ceph-04 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 243 ceph-04 1.8T SSD osd-phy1 247 ceph-21 1.8T SSD osd-phy2 254 ceph-04 1.8T SSD osd-phy3 256 ceph-04 1.8T SSD osd-phy4 286 ceph-04 1.8T SSD osd-phy5 287 ceph-04 1.8T SSD osd-phy6 288 ceph-04 10.7T HDD osd-phy7 48 ceph-04 372.6G SSD osd-phy8 264 ceph-21 1.8T SSD osd-phy9 84 ceph-04 8.9T HDD osd-phy10 72 ceph-21 8.9T HDD osd-phy11 145 ceph-04 8.9T HDD osd-phy14 156 ceph-04 8.9T HDD osd-phy15 168 ceph-04 8.9T HDD osd-phy16 181 ceph-04 8.9T HDD osd-phy17 0 ceph-21 8.9T HDD ---------------- ceph-05 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 240 ceph-05 1.8T SSD osd-phy1 249 ceph-22 1.8T SSD osd-phy2 251 ceph-05 1.8T SSD osd-phy3 255 ceph-05 1.8T SSD osd-phy4 284 ceph-05 1.8T SSD osd-phy5 285 ceph-05 1.8T SSD osd-phy6 289 ceph-05 10.7T HDD osd-phy7 49 ceph-05 372.6G SSD osd-phy8 265 ceph-22 1.8T SSD osd-phy9 74 ceph-05 8.9T HDD osd-phy10 85 ceph-22 8.9T HDD osd-phy11 144 ceph-05 8.9T HDD osd-phy14 157 ceph-05 8.9T HDD osd-phy15 169 ceph-05 8.9T HDD osd-phy16 180 ceph-05 8.9T HDD osd-phy17 1 ceph-22 8.9T HDD ---------------- ceph-06 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 244 ceph-06 1.8T SSD osd-phy1 246 ceph-21 1.8T SSD osd-phy2 253 ceph-06 1.8T SSD osd-phy3 257 ceph-06 1.8T SSD osd-phy4 282 ceph-06 1.8T SSD osd-phy5 283 ceph-06 1.8T SSD osd-phy6 40 ceph-06 372.6G SSD osd-phy7 50 ceph-06 372.6G SSD osd-phy8 60 ceph-06 8.9T HDD osd-phy9 290 ceph-06 10.7T HDD osd-phy10 291 ceph-21 10.7T HDD osd-phy11 146 ceph-06 8.9T HDD osd-phy14 158 ceph-06 8.9T HDD osd-phy15 170 ceph-06 8.9T HDD osd-phy16 182 ceph-06 8.9T HDD osd-phy17 2 ceph-21 8.9T HDD ---------------- ceph-07 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 242 ceph-07 1.8T SSD osd-phy1 250 ceph-22 1.8T SSD osd-phy2 252 ceph-07 1.8T SSD osd-phy3 258 ceph-07 1.8T SSD osd-phy4 279 ceph-07 1.8T SSD osd-phy5 280 ceph-07 1.8T SSD osd-phy6 292 ceph-07 10.7T HDD osd-phy7 52 ceph-07 372.6G SSD osd-phy8 63 ceph-07 8.9T HDD osd-phy9 281 ceph-22 1.8T SSD osd-phy10 87 ceph-22 8.9T HDD osd-phy11 148 ceph-07 8.9T HDD osd-phy14 159 ceph-07 8.9T HDD osd-phy15 172 ceph-07 8.9T HDD osd-phy16 183 ceph-07 8.9T HDD osd-phy17 3 ceph-22 8.9T HDD ---------------- ceph-18 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 241 ceph-18 1.8T SSD osd-phy1 248 ceph-18 1.8T SSD osd-phy2 41 ceph-18 372.6G SSD osd-phy3 31 ceph-18 372.6G SSD osd-phy4 277 ceph-18 1.8T SSD osd-phy5 278 ceph-21 1.8T SSD osd-phy6 53 ceph-21 372.6G SSD osd-phy7 267 ceph-18 1.8T SSD osd-phy8 266 ceph-18 1.8T SSD osd-phy9 293 ceph-18 10.7T HDD osd-phy10 86 ceph-21 8.9T HDD osd-phy11 259 ceph-18 10.9T HDD osd-phy14 229 ceph-18 8.9T HDD osd-phy15 232 ceph-18 8.9T HDD osd-phy16 235 ceph-18 8.9T HDD osd-phy17 238 ceph-18 8.9T HDD ---------------- ceph-19 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 261 ceph-19 1.8T SSD osd-phy1 262 ceph-19 1.8T SSD osd-phy2 295 ceph-19 10.7T HDD osd-phy3 43 ceph-19 372.6G SSD osd-phy4 275 ceph-19 1.8T SSD osd-phy5 294 ceph-22 10.7T HDD osd-phy6 51 ceph-22 372.6G SSD osd-phy7 269 ceph-19 1.8T SSD osd-phy8 268 ceph-19 1.8T SSD osd-phy9 276 ceph-22 1.8T SSD osd-phy10 73 ceph-22 8.9T HDD osd-phy11 263 ceph-19 10.9T HDD osd-phy14 231 ceph-19 8.9T HDD osd-phy15 233 ceph-19 8.9T HDD osd-phy16 236 ceph-19 8.9T HDD osd-phy17 239 ceph-19 8.9T HDD ---------------- ceph-20 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 245 ceph-20 1.8T SSD osd-phy1 28 ceph-20 372.6G SSD osd-phy2 44 ceph-20 372.6G SSD osd-phy3 271 ceph-20 1.8T SSD osd-phy4 272 ceph-20 1.8T SSD osd-phy5 273 ceph-20 1.8T SSD osd-phy6 274 ceph-21 1.8T SSD osd-phy7 296 ceph-20 10.7T HDD osd-phy8 76 ceph-21 8.9T HDD osd-phy9 39 ceph-21 372.6G SSD osd-phy10 270 ceph-20 1.8T SSD osd-phy11 260 ceph-20 10.9T HDD osd-phy14 228 ceph-20 8.9T HDD osd-phy15 230 ceph-20 8.9T HDD osd-phy16 234 ceph-20 8.9T HDD osd-phy17 237 ceph-20 8.9T HDD CONT is the container name and encodes the physical slot on the host where the OSD is located. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 12:47:12 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Have you adjusted the min_size for pool sr-rbd-data-one-hdd at all? Also can you send the output of "ceph osd erasure-code-profile ls" and for each EC profile, "ceph osd erasure-code-profile get <profile>"? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 11:05 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Sorry for the many small e-mails: requested IDs in the commands, 288-296. One new OSD per host. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 16:59:04 To: Eric Smith; ceph-users Subject: [ceph-users] Re: Ceph does not recover from OSD restart Hi Eric, the procedure for re-discovering all objects is: # Flag: norebalance ceph osd crush move osd.288 host=bb-04 ceph osd crush move osd.289 host=bb-05 ceph osd crush move osd.290 host=bb-06 ceph osd crush move osd.291 host=bb-21 ceph osd crush move osd.292 host=bb-07 ceph osd crush move osd.293 host=bb-18 ceph osd crush move osd.295 host=bb-19 ceph osd crush move osd.294 host=bb-22 ceph osd crush move osd.296 host=bb-20 # Wait until all PGs are peered and recovery is done. In my case, there was only little I/O, # no more than 50-100 objects had writes missing and recovery was a few seconds. # # The bb-hosts are under a separate crush root that I use solely as parking space # and for draining OSDs. ceph osd crush move osd.288 host=ceph-04 ceph osd crush move osd.289 host=ceph-05 ceph osd crush move osd.290 host=ceph-06 ceph osd crush move osd.291 host=ceph-21 ceph osd crush move osd.292 host=ceph-07 ceph osd crush move osd.293 host=ceph-18 ceph osd crush move osd.295 host=ceph-19 ceph osd crush move osd.294 host=ceph-22 ceph osd crush move osd.296 host=ceph-20 After peering, no degraded PGs/objects any more, just the misplaced ones as expected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 16:45:28 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart You said you had to move some OSDs out and back in for Ceph to go back to normal (The OSDs you added). Which OSDs were added? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:55 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

How to get rid of HP printer driver is unavailable issue?
by sofiacarter120＠gmail.com 05 Aug '20

05 Aug '20

I’m working on the latest printing machine for the excellent printing performance. I am observing that my printing machine is performing poorly. My printing device is not printing the documents correctly. My printer is showing an error message such as [url=https://www.hpprintersupportpro.com/blog/hp-printer-in-error-state/http… in error state[/url] problem. I don’t have the permanent solutions for solving this error. I guess that the low ink can be the main reason of this technical error. I am trying to find out the cause of this technical glitch, but I am not sure the reasons. It can be very difficult to find out the exact causes. So anyone can recommend the easy troubleshooting resolutions to fix this error as soon as possible.

1 0

help me enable ceph iscsi gatewaty in ceph octopus
by David Thuong 05 Aug '20

05 Aug '20

Please help me enable ceph iscsi gatewaty in ceph octopus . when i install ceph complete . i see iscsi gateway not enable. please help me config it

2 1

Will Cash app customer service eradicate my account forever?
by PEMLA landy 05 Aug '20

05 Aug '20

Generally no, until you didn't require, Cash app customer service can't eradicate your account for eternity in any scenario. Regardless, every so often happened that, the assistance bunch finds your account connected with any out of the ordinary activity. Thusly, in this circumstance, they can close your account on a passing reason. In case you have to get to it, then give all the docs to check your identity to use your account without any issue. https://www.pcmonks.net/blog/how-to-access-an-old-cash-app-account/

1 0

Can Cash app customer service explain beyond what many would consider possible?
by PEMLA landy 05 Aug '20

05 Aug '20

You can see beyond what many would consider possible which is depicted more or less. There is two spending limit, one is for the verified customer and other is for non-verified customers. If you are a non-verified customer, then, you can consume $250 dollar in any 7-days and get up to $1000 in any 30-days term. And, if you are a verified user, then you can send $7500 in a week and receive the unlimited amount. If any doubts, contact our Cash app customer service. https://www.pcmonks.net/blog/how-to-increase-cash-app-limit/

1 0

How Cash app customer service help us to report for cash?
by PEMLA landy 05 Aug '20

05 Aug '20

Here, it is obligatory to ensure for all wages, even it is gotten or paid in cash. All the tolerant cash portions for any work are submitted here to record the pay and go to promise it on their administration charge archives. It is at least a point for the customers to follow trades similarly to avoid duped cases. In case of any issue, go to use our [url=https://www.pcmonks.net/blog/transfer-money-cash-app-to-bank-account/]C… app customer service[/url] phone number.

1 0

How do I Talk to a Cash app representative to ask what amount I have to pay for $100?
by awalker97520＠gmail.com 05 Aug '20

05 Aug '20

In the event that you are going to make a transaction from your Cash app account that connected a credit card, at that point you have to pay a 3% charge expense. Along these lines, in the event that you have need to make a transaction of $100, at that point you have to add $103, to pay its charge expense as well. In the event that you have still any inquiries behind this idea, at that point proceed to How do I talk to a Cash app representative. https://www.techjaadu.com/cash-app-support-number/

1 0

Running fstrim (discard) inside KVM machine with RBD as disk device corrupts ext4 filesystem
by Georg Schönberger 05 Aug '20

05 Aug '20

Hey Ceph users, we are currently facing some serious problems on our Ceph Cluster with libvirt (KVM), RBD devices and FSTRIM running inside VMs. The problem is right after running the fstrim command inside the VM the ext4 filesystem is corrupted and read-only with the following error message: EXT4-fs error (device sda1): ext4_mb_generate_buddy:756: group 136, block bitmap and bg descriptor inconsistent: 32200 vs 32768 free clusters Aborting journal on device sda1-8 EXT4-fs (sda1): Remounting filesystem read-only EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal EXT4-fs (sda1): Remounting filesystem read-only This behavior is reproducible across several VMs with different OS (Ubuntu 14.04, 16.04 and 18.04) so we guess it is a bug or a configuration problem regarding RBD devices. Our setup on the hosts running the VMs looks like: # lsb_release -d Description: Ubuntu 20.04 LTS # uname -a Linux XXX 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux # ceph --version ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) -> I know there's the update to Ceph 15.2.4 but I haven't seen any fstrim/discard related changes in the changelog. If we could fix the problem with 15.2.4 I would be happy... The libvirt config for the RBD device with supporting fstrim (discard) is the following: <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='directsync' io='native' discard='unmap'/> <auth username='libvirt'> <secret type='ceph' usage='client.libvirt'/> </auth> <source protocol='rbd' name='cephstorage/testtrim_system'> <host name='XXX' port='6789'/> <host name='XXX' port='6789'/> <host name='XXX' port='6789'/> <host name='XXX' port='6789'/> <host name='XXX' port='6789'/> </source> <target dev='sda' bus='scsi'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> The ceph docs (https://docs.ceph.com/docs/octopus/rbd/qemu-rbd/) gave me some hints about enabling trim/discard and I tested using 4M as discard granularity, but I got the same error resulting in a corrupted ext4 file system. Changes made to the libvirt config: <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.scsi0-0-0-0.discard_granularity=4194304'/> </qemu:commandline> As the RBD devices are thin-provisioned we really need calling fstrim inside the VM regularly to free up unused blocks, otherwise our Ceph pool will run out of space. Any ideas what could be wrong with our RBD setup or can somebody else reproduce the problem? Any hints on how to debug this problem? Any related/open Ceph issues? (I could not fined one) Thanks a lot for your help, Georg

2 4

rados_connect timeout
by Daniel Mezentsev 05 Aug '20

05 Aug '20

Hi all, Can somebody point me to the timeout parameter for rados_connect function. When monitors are not available it hangs undefinetely. Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

1 0

help with deleting errant iscsi gateway
by Sharad Mehrotra 05 Aug '20

05 Aug '20

Hi: I am using ceph nautilus with CentOS 7.6 and working on adding a pair of iscsi gateways in our cluster, following the documentation here: https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/ I was in the "Configuring" section, step #3, "Create the iSCSI gateways" and ran into problems. When I tried adding the first gateway using the example shown, I ran into an error with the message "*The first gateway defined must be the local machine*". In googling around to remediate this error, I saw on this web page ( https://tracker.ceph.com/issues/24425) that I should try adding the first gateway with my localhost and issued this command instead /iscsi-target...-igw/gateways> create sv46 10.0.201.66 This led to the following configuration with the errant gateway "sv46" which I then tried to delete and failed. /iscsi-target...-gw:iscsi-igw> ls o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ........................................................ [Auth: None, Gateways: 1] o- disks ......................................................................................................... [Disks: 0] o- gateways ........................................................................................... [Up: 0/1, Portals: 1] | o- sv46 ...................................................................................... [10.0.201.66 (UNAUTHORIZED)] o- host-groups ................................................................................................. [Groups : 0] o- hosts ...................................................................................... [Auth: ACL_ENABLED, Hosts: 0] This is what happened when I tried to delete sv46, I got the message shown in *bold *but the gateway was not deleted. /iscsi-target...-igw/gateways> delete sv46 Deleting gateway, sv46 *Target is not mapped to any gateways.* /iscsi-target...-igw/gateways> Question #1: How do I delete this errant iscsi gateway please? Question #2: What is the proper way to handle the issue that got me into this jam to begin with, i.e. the error message "The first gateway defined must be the local machine" ? Thanks, Sharad

1 0

Apparent bucket corruption error: get_bucket_instance_from_oid failed
by Wesley Dillingham 04 Aug '20

04 Aug '20

Long running cluster, currently running 14.2.6 I have a certain user whose buckets have become corrupted in that the following commands: radosgw-admin bucket check --bucket <bucket> radosgw-admin bucket list --bucket= <bucket> return with the following: ERROR: could not init bucket: (2) No such file or directory 2020-08-04 13:47:03.417 7f94dfea86c0 -1 ERROR: get_bucket_instance_from_oid failed: -2 radosgw-admin metadata get bucket:<bucket> is successful. radosgw-admin metadata get bucket.instance:<bucket>:<bucket_id> yields: ERROR: can't get key: (2) No such file or directory radosgw-admin metadata list bucket.instance | grep -i <bucket> yields no results. When I drop to rados and look in the index pool I can see 128 objects matching the bucket_id as derived from the "metadata get" and this seems to match other functioning buckets. Unfortunately this issue was sleepy and happened many months ago unnoticed. We have not retained many of the ceph logs from this time. We do have the civetweb access logs and have found that error codes began on the same day that we lowered the pg_num on many of the rgw pools (all of them but the index_pool and the data_pool). OSDs were filestore at that time and have since been converted to bluestore. Other than the dates lining up we have no direct evidence these are related, and did not encounter any inconsistent PGs. We also used this process on other clusters with no ill effects. Ideally I would like to repair and restore the functionality of these buckets given that it appears the objects in the index pool still exist. Is there any way to repair these? Do these errors correlate to any known issues? Thanks in advance for any leads. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

1 0

Ldap Integration
by Jared Jacob 04 Aug '20

04 Aug '20

Hello, I am looking into connecting my rados gateway to LDAP and found the following documentation. https://docs.ceph.com/docs/master/radosgw/ldap-auth/ I would like to allow an LDAP group to have access to create and manage buckets. The questions I still have are the following: -Do the LDAP users need to log in to some sort of portal before their corresponding ceph user is created? If so, where do they go to do so? Or does the creation of ceph users and keys happen automatically? -How can you access the ldap users key and secret after they are integrated? Thanks in advance for any information you can provide. Regards, Jared

3 4

HEALTH_WARN crush map has legacy tunables (require firefly, min is hammer)
by Mike Garza 04 Aug '20

04 Aug '20

Hi, I've been tasked with moving Jewel clusters to Nautilus. After the final upgrade Ceph Health warns about legacy tunables. On clusters running SSD's I enabled the optimal flag. Which took weeks to chug through remappings. My remaining clusters run HDD's. Does anyone have experience with using the legacy flag? I'd like to clear up the health warning without outright silencing it. But, I also do not want to kick off any remapping either. Does anyone have experience with pushing this change down the road? Mike

1 0

Re: Ceph does not recover from OSD restart
by Frank Schilder 04 Aug '20

04 Aug '20

Hi Eric, thanks for the clarification, I did misunderstand you. > You should not have to move OSDs in and out of the CRUSH tree however > in order to solve any data placement problems (This is the baffling part). Exactly. Should I create a tracker issue? I think this is not hard to reproduce with a standard crush map where host-bucket=physical host and I would, in fact, expect that this scenario is part of the integration test. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 13:58:47 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart All seems in order in terms of your CRUSH layout. You can speed up the rebalancing / scale-out operations by increasing the osd_max_backfills on each OSD (Especially during off hours). The unnecessary degradation is not expected behavior with a cluster in HEALTH_OK status, but with backfill / rebalancing ongoing it's not unexpected. You should not have to move OSDs in and out of the CRUSH tree however in order to solve any data placement problems (This is the baffling part). -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 7:45 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Erik, I added the disks and started the rebalancing. When I run into the issue, ca. 3 days after start of rebalancing, it was about 25% done. The cluster does not go to HEALTH_OK before the rebalancing is finished, it shows the "xxx objects misplaced" warning. The OSD crush locations for the logical hosts are in ceph.conf, the OSDs come up in the proper crush bucket. > All seems in order then In what sense? The rebalancing is still ongoing and usually a very long operation. This time I added only 9 disks, but we will almost triple the number of disks of a larger pool soon, which has 150 OSDs at the moment. I expect the rebalancing for this expansion to take months. Due to a memory leak, I need to restart OSDs regularly. Also, a host may restart or we might have a power outage during this window. In these situations, it will be a real pain if I have to play the crush move game with 300+ OSDs. This unnecessary redundancy degradation on OSD restart cannot possibly be expected behaviour, or do I misunderstand something here? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 13:19:41 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart All seems in order then - when you ran into your maintenance issue, how long was if after you added the new OSDs and did Ceph ever get to HEALTH_OK so it could trim PG history? Also did the OSDs just start back up in the wrong place in the CRUSH tree? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 7:10 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, > Have you adjusted the min_size for pool sr-rbd-data-one-hdd Yes. For all EC pools located in datacenter ServerRoom, we currently set min_size=k=6, because we lack physical servers. Hosts ceph-21 and ceph-22 are logical but not physical, disks in these buckets are co-located such that no more than 2 host buckets share the same physical host. With failure domain = host, we can ensure that no more than 2 EC shards are on the same physical host. With m=2 and min_size=k we have continued service with any 1 physical host down for maintenance and also recovery will happen if a physical host fails. Some objects will have no redundancy for a while then. We will increase min_size to k+1 as soon as we have 2 additional hosts and simply move the OSDs from buckets ceph-21/22 to these without rebalancing. The distribution of disks and buckets is listed below as well (longer listing). Thanks and best regards, Frank # ceph osd erasure-code-profile ls con-ec-8-2-hdd con-ec-8-2-ssd default sr-ec-6-2-hdd This is the relevant one: # ceph osd erasure-code-profile get sr-ec-6-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ServerRoom jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 Note that the pool sr-rbd-data-one (id 2) was created with this profile and later moved to SSD. Therefore, the crush rule does not match the profile's device class any more. These two are under different roots: # ceph osd erasure-code-profile get con-ec-8-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get con-ec-8-2-ssd crush-device-class=ssd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 Full physical placement information for OSDs under tree "datacenter ServerRoom": ---------------- ceph-04 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 243 ceph-04 1.8T SSD osd-phy1 247 ceph-21 1.8T SSD osd-phy2 254 ceph-04 1.8T SSD osd-phy3 256 ceph-04 1.8T SSD osd-phy4 286 ceph-04 1.8T SSD osd-phy5 287 ceph-04 1.8T SSD osd-phy6 288 ceph-04 10.7T HDD osd-phy7 48 ceph-04 372.6G SSD osd-phy8 264 ceph-21 1.8T SSD osd-phy9 84 ceph-04 8.9T HDD osd-phy10 72 ceph-21 8.9T HDD osd-phy11 145 ceph-04 8.9T HDD osd-phy14 156 ceph-04 8.9T HDD osd-phy15 168 ceph-04 8.9T HDD osd-phy16 181 ceph-04 8.9T HDD osd-phy17 0 ceph-21 8.9T HDD ---------------- ceph-05 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 240 ceph-05 1.8T SSD osd-phy1 249 ceph-22 1.8T SSD osd-phy2 251 ceph-05 1.8T SSD osd-phy3 255 ceph-05 1.8T SSD osd-phy4 284 ceph-05 1.8T SSD osd-phy5 285 ceph-05 1.8T SSD osd-phy6 289 ceph-05 10.7T HDD osd-phy7 49 ceph-05 372.6G SSD osd-phy8 265 ceph-22 1.8T SSD osd-phy9 74 ceph-05 8.9T HDD osd-phy10 85 ceph-22 8.9T HDD osd-phy11 144 ceph-05 8.9T HDD osd-phy14 157 ceph-05 8.9T HDD osd-phy15 169 ceph-05 8.9T HDD osd-phy16 180 ceph-05 8.9T HDD osd-phy17 1 ceph-22 8.9T HDD ---------------- ceph-06 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 244 ceph-06 1.8T SSD osd-phy1 246 ceph-21 1.8T SSD osd-phy2 253 ceph-06 1.8T SSD osd-phy3 257 ceph-06 1.8T SSD osd-phy4 282 ceph-06 1.8T SSD osd-phy5 283 ceph-06 1.8T SSD osd-phy6 40 ceph-06 372.6G SSD osd-phy7 50 ceph-06 372.6G SSD osd-phy8 60 ceph-06 8.9T HDD osd-phy9 290 ceph-06 10.7T HDD osd-phy10 291 ceph-21 10.7T HDD osd-phy11 146 ceph-06 8.9T HDD osd-phy14 158 ceph-06 8.9T HDD osd-phy15 170 ceph-06 8.9T HDD osd-phy16 182 ceph-06 8.9T HDD osd-phy17 2 ceph-21 8.9T HDD ---------------- ceph-07 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 242 ceph-07 1.8T SSD osd-phy1 250 ceph-22 1.8T SSD osd-phy2 252 ceph-07 1.8T SSD osd-phy3 258 ceph-07 1.8T SSD osd-phy4 279 ceph-07 1.8T SSD osd-phy5 280 ceph-07 1.8T SSD osd-phy6 292 ceph-07 10.7T HDD osd-phy7 52 ceph-07 372.6G SSD osd-phy8 63 ceph-07 8.9T HDD osd-phy9 281 ceph-22 1.8T SSD osd-phy10 87 ceph-22 8.9T HDD osd-phy11 148 ceph-07 8.9T HDD osd-phy14 159 ceph-07 8.9T HDD osd-phy15 172 ceph-07 8.9T HDD osd-phy16 183 ceph-07 8.9T HDD osd-phy17 3 ceph-22 8.9T HDD ---------------- ceph-18 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 241 ceph-18 1.8T SSD osd-phy1 248 ceph-18 1.8T SSD osd-phy2 41 ceph-18 372.6G SSD osd-phy3 31 ceph-18 372.6G SSD osd-phy4 277 ceph-18 1.8T SSD osd-phy5 278 ceph-21 1.8T SSD osd-phy6 53 ceph-21 372.6G SSD osd-phy7 267 ceph-18 1.8T SSD osd-phy8 266 ceph-18 1.8T SSD osd-phy9 293 ceph-18 10.7T HDD osd-phy10 86 ceph-21 8.9T HDD osd-phy11 259 ceph-18 10.9T HDD osd-phy14 229 ceph-18 8.9T HDD osd-phy15 232 ceph-18 8.9T HDD osd-phy16 235 ceph-18 8.9T HDD osd-phy17 238 ceph-18 8.9T HDD ---------------- ceph-19 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 261 ceph-19 1.8T SSD osd-phy1 262 ceph-19 1.8T SSD osd-phy2 295 ceph-19 10.7T HDD osd-phy3 43 ceph-19 372.6G SSD osd-phy4 275 ceph-19 1.8T SSD osd-phy5 294 ceph-22 10.7T HDD osd-phy6 51 ceph-22 372.6G SSD osd-phy7 269 ceph-19 1.8T SSD osd-phy8 268 ceph-19 1.8T SSD osd-phy9 276 ceph-22 1.8T SSD osd-phy10 73 ceph-22 8.9T HDD osd-phy11 263 ceph-19 10.9T HDD osd-phy14 231 ceph-19 8.9T HDD osd-phy15 233 ceph-19 8.9T HDD osd-phy16 236 ceph-19 8.9T HDD osd-phy17 239 ceph-19 8.9T HDD ---------------- ceph-20 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 245 ceph-20 1.8T SSD osd-phy1 28 ceph-20 372.6G SSD osd-phy2 44 ceph-20 372.6G SSD osd-phy3 271 ceph-20 1.8T SSD osd-phy4 272 ceph-20 1.8T SSD osd-phy5 273 ceph-20 1.8T SSD osd-phy6 274 ceph-21 1.8T SSD osd-phy7 296 ceph-20 10.7T HDD osd-phy8 76 ceph-21 8.9T HDD osd-phy9 39 ceph-21 372.6G SSD osd-phy10 270 ceph-20 1.8T SSD osd-phy11 260 ceph-20 10.9T HDD osd-phy14 228 ceph-20 8.9T HDD osd-phy15 230 ceph-20 8.9T HDD osd-phy16 234 ceph-20 8.9T HDD osd-phy17 237 ceph-20 8.9T HDD CONT is the container name and encodes the physical slot on the host where the OSD is located. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 12:47:12 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Have you adjusted the min_size for pool sr-rbd-data-one-hdd at all? Also can you send the output of "ceph osd erasure-code-profile ls" and for each EC profile, "ceph osd erasure-code-profile get <profile>"? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 11:05 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Sorry for the many small e-mails: requested IDs in the commands, 288-296. One new OSD per host. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 16:59:04 To: Eric Smith; ceph-users Subject: [ceph-users] Re: Ceph does not recover from OSD restart Hi Eric, the procedure for re-discovering all objects is: # Flag: norebalance ceph osd crush move osd.288 host=bb-04 ceph osd crush move osd.289 host=bb-05 ceph osd crush move osd.290 host=bb-06 ceph osd crush move osd.291 host=bb-21 ceph osd crush move osd.292 host=bb-07 ceph osd crush move osd.293 host=bb-18 ceph osd crush move osd.295 host=bb-19 ceph osd crush move osd.294 host=bb-22 ceph osd crush move osd.296 host=bb-20 # Wait until all PGs are peered and recovery is done. In my case, there was only little I/O, # no more than 50-100 objects had writes missing and recovery was a few seconds. # # The bb-hosts are under a separate crush root that I use solely as parking space # and for draining OSDs. ceph osd crush move osd.288 host=ceph-04 ceph osd crush move osd.289 host=ceph-05 ceph osd crush move osd.290 host=ceph-06 ceph osd crush move osd.291 host=ceph-21 ceph osd crush move osd.292 host=ceph-07 ceph osd crush move osd.293 host=ceph-18 ceph osd crush move osd.295 host=ceph-19 ceph osd crush move osd.294 host=ceph-22 ceph osd crush move osd.296 host=ceph-20 After peering, no degraded PGs/objects any more, just the misplaced ones as expected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 16:45:28 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart You said you had to move some OSDs out and back in for Ceph to go back to normal (The OSDs you added). Which OSDs were added? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:55 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Ceph does not recover from OSD restart
by Frank Schilder 04 Aug '20

04 Aug '20

Hi Erik, I added the disks and started the rebalancing. When I run into the issue, ca. 3 days after start of rebalancing, it was about 25% done. The cluster does not go to HEALTH_OK before the rebalancing is finished, it shows the "xxx objects misplaced" warning. The OSD crush locations for the logical hosts are in ceph.conf, the OSDs come up in the proper crush bucket. > All seems in order then In what sense? The rebalancing is still ongoing and usually a very long operation. This time I added only 9 disks, but we will almost triple the number of disks of a larger pool soon, which has 150 OSDs at the moment. I expect the rebalancing for this expansion to take months. Due to a memory leak, I need to restart OSDs regularly. Also, a host may restart or we might have a power outage during this window. In these situations, it will be a real pain if I have to play the crush move game with 300+ OSDs. This unnecessary redundancy degradation on OSD restart cannot possibly be expected behaviour, or do I misunderstand something here? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 13:19:41 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart All seems in order then - when you ran into your maintenance issue, how long was if after you added the new OSDs and did Ceph ever get to HEALTH_OK so it could trim PG history? Also did the OSDs just start back up in the wrong place in the CRUSH tree? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Tuesday, August 4, 2020 7:10 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, > Have you adjusted the min_size for pool sr-rbd-data-one-hdd Yes. For all EC pools located in datacenter ServerRoom, we currently set min_size=k=6, because we lack physical servers. Hosts ceph-21 and ceph-22 are logical but not physical, disks in these buckets are co-located such that no more than 2 host buckets share the same physical host. With failure domain = host, we can ensure that no more than 2 EC shards are on the same physical host. With m=2 and min_size=k we have continued service with any 1 physical host down for maintenance and also recovery will happen if a physical host fails. Some objects will have no redundancy for a while then. We will increase min_size to k+1 as soon as we have 2 additional hosts and simply move the OSDs from buckets ceph-21/22 to these without rebalancing. The distribution of disks and buckets is listed below as well (longer listing). Thanks and best regards, Frank # ceph osd erasure-code-profile ls con-ec-8-2-hdd con-ec-8-2-ssd default sr-ec-6-2-hdd This is the relevant one: # ceph osd erasure-code-profile get sr-ec-6-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ServerRoom jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 Note that the pool sr-rbd-data-one (id 2) was created with this profile and later moved to SSD. Therefore, the crush rule does not match the profile's device class any more. These two are under different roots: # ceph osd erasure-code-profile get con-ec-8-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get con-ec-8-2-ssd crush-device-class=ssd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 Full physical placement information for OSDs under tree "datacenter ServerRoom": ---------------- ceph-04 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 243 ceph-04 1.8T SSD osd-phy1 247 ceph-21 1.8T SSD osd-phy2 254 ceph-04 1.8T SSD osd-phy3 256 ceph-04 1.8T SSD osd-phy4 286 ceph-04 1.8T SSD osd-phy5 287 ceph-04 1.8T SSD osd-phy6 288 ceph-04 10.7T HDD osd-phy7 48 ceph-04 372.6G SSD osd-phy8 264 ceph-21 1.8T SSD osd-phy9 84 ceph-04 8.9T HDD osd-phy10 72 ceph-21 8.9T HDD osd-phy11 145 ceph-04 8.9T HDD osd-phy14 156 ceph-04 8.9T HDD osd-phy15 168 ceph-04 8.9T HDD osd-phy16 181 ceph-04 8.9T HDD osd-phy17 0 ceph-21 8.9T HDD ---------------- ceph-05 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 240 ceph-05 1.8T SSD osd-phy1 249 ceph-22 1.8T SSD osd-phy2 251 ceph-05 1.8T SSD osd-phy3 255 ceph-05 1.8T SSD osd-phy4 284 ceph-05 1.8T SSD osd-phy5 285 ceph-05 1.8T SSD osd-phy6 289 ceph-05 10.7T HDD osd-phy7 49 ceph-05 372.6G SSD osd-phy8 265 ceph-22 1.8T SSD osd-phy9 74 ceph-05 8.9T HDD osd-phy10 85 ceph-22 8.9T HDD osd-phy11 144 ceph-05 8.9T HDD osd-phy14 157 ceph-05 8.9T HDD osd-phy15 169 ceph-05 8.9T HDD osd-phy16 180 ceph-05 8.9T HDD osd-phy17 1 ceph-22 8.9T HDD ---------------- ceph-06 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 244 ceph-06 1.8T SSD osd-phy1 246 ceph-21 1.8T SSD osd-phy2 253 ceph-06 1.8T SSD osd-phy3 257 ceph-06 1.8T SSD osd-phy4 282 ceph-06 1.8T SSD osd-phy5 283 ceph-06 1.8T SSD osd-phy6 40 ceph-06 372.6G SSD osd-phy7 50 ceph-06 372.6G SSD osd-phy8 60 ceph-06 8.9T HDD osd-phy9 290 ceph-06 10.7T HDD osd-phy10 291 ceph-21 10.7T HDD osd-phy11 146 ceph-06 8.9T HDD osd-phy14 158 ceph-06 8.9T HDD osd-phy15 170 ceph-06 8.9T HDD osd-phy16 182 ceph-06 8.9T HDD osd-phy17 2 ceph-21 8.9T HDD ---------------- ceph-07 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 242 ceph-07 1.8T SSD osd-phy1 250 ceph-22 1.8T SSD osd-phy2 252 ceph-07 1.8T SSD osd-phy3 258 ceph-07 1.8T SSD osd-phy4 279 ceph-07 1.8T SSD osd-phy5 280 ceph-07 1.8T SSD osd-phy6 292 ceph-07 10.7T HDD osd-phy7 52 ceph-07 372.6G SSD osd-phy8 63 ceph-07 8.9T HDD osd-phy9 281 ceph-22 1.8T SSD osd-phy10 87 ceph-22 8.9T HDD osd-phy11 148 ceph-07 8.9T HDD osd-phy14 159 ceph-07 8.9T HDD osd-phy15 172 ceph-07 8.9T HDD osd-phy16 183 ceph-07 8.9T HDD osd-phy17 3 ceph-22 8.9T HDD ---------------- ceph-18 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 241 ceph-18 1.8T SSD osd-phy1 248 ceph-18 1.8T SSD osd-phy2 41 ceph-18 372.6G SSD osd-phy3 31 ceph-18 372.6G SSD osd-phy4 277 ceph-18 1.8T SSD osd-phy5 278 ceph-21 1.8T SSD osd-phy6 53 ceph-21 372.6G SSD osd-phy7 267 ceph-18 1.8T SSD osd-phy8 266 ceph-18 1.8T SSD osd-phy9 293 ceph-18 10.7T HDD osd-phy10 86 ceph-21 8.9T HDD osd-phy11 259 ceph-18 10.9T HDD osd-phy14 229 ceph-18 8.9T HDD osd-phy15 232 ceph-18 8.9T HDD osd-phy16 235 ceph-18 8.9T HDD osd-phy17 238 ceph-18 8.9T HDD ---------------- ceph-19 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 261 ceph-19 1.8T SSD osd-phy1 262 ceph-19 1.8T SSD osd-phy2 295 ceph-19 10.7T HDD osd-phy3 43 ceph-19 372.6G SSD osd-phy4 275 ceph-19 1.8T SSD osd-phy5 294 ceph-22 10.7T HDD osd-phy6 51 ceph-22 372.6G SSD osd-phy7 269 ceph-19 1.8T SSD osd-phy8 268 ceph-19 1.8T SSD osd-phy9 276 ceph-22 1.8T SSD osd-phy10 73 ceph-22 8.9T HDD osd-phy11 263 ceph-19 10.9T HDD osd-phy14 231 ceph-19 8.9T HDD osd-phy15 233 ceph-19 8.9T HDD osd-phy16 236 ceph-19 8.9T HDD osd-phy17 239 ceph-19 8.9T HDD ---------------- ceph-20 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 245 ceph-20 1.8T SSD osd-phy1 28 ceph-20 372.6G SSD osd-phy2 44 ceph-20 372.6G SSD osd-phy3 271 ceph-20 1.8T SSD osd-phy4 272 ceph-20 1.8T SSD osd-phy5 273 ceph-20 1.8T SSD osd-phy6 274 ceph-21 1.8T SSD osd-phy7 296 ceph-20 10.7T HDD osd-phy8 76 ceph-21 8.9T HDD osd-phy9 39 ceph-21 372.6G SSD osd-phy10 270 ceph-20 1.8T SSD osd-phy11 260 ceph-20 10.9T HDD osd-phy14 228 ceph-20 8.9T HDD osd-phy15 230 ceph-20 8.9T HDD osd-phy16 234 ceph-20 8.9T HDD osd-phy17 237 ceph-20 8.9T HDD CONT is the container name and encodes the physical slot on the host where the OSD is located. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 12:47:12 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Have you adjusted the min_size for pool sr-rbd-data-one-hdd at all? Also can you send the output of "ceph osd erasure-code-profile ls" and for each EC profile, "ceph osd erasure-code-profile get <profile>"? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 11:05 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Sorry for the many small e-mails: requested IDs in the commands, 288-296. One new OSD per host. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 16:59:04 To: Eric Smith; ceph-users Subject: [ceph-users] Re: Ceph does not recover from OSD restart Hi Eric, the procedure for re-discovering all objects is: # Flag: norebalance ceph osd crush move osd.288 host=bb-04 ceph osd crush move osd.289 host=bb-05 ceph osd crush move osd.290 host=bb-06 ceph osd crush move osd.291 host=bb-21 ceph osd crush move osd.292 host=bb-07 ceph osd crush move osd.293 host=bb-18 ceph osd crush move osd.295 host=bb-19 ceph osd crush move osd.294 host=bb-22 ceph osd crush move osd.296 host=bb-20 # Wait until all PGs are peered and recovery is done. In my case, there was only little I/O, # no more than 50-100 objects had writes missing and recovery was a few seconds. # # The bb-hosts are under a separate crush root that I use solely as parking space # and for draining OSDs. ceph osd crush move osd.288 host=ceph-04 ceph osd crush move osd.289 host=ceph-05 ceph osd crush move osd.290 host=ceph-06 ceph osd crush move osd.291 host=ceph-21 ceph osd crush move osd.292 host=ceph-07 ceph osd crush move osd.293 host=ceph-18 ceph osd crush move osd.295 host=ceph-19 ceph osd crush move osd.294 host=ceph-22 ceph osd crush move osd.296 host=ceph-20 After peering, no degraded PGs/objects any more, just the misplaced ones as expected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 16:45:28 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart You said you had to move some OSDs out and back in for Ceph to go back to normal (The OSDs you added). Which OSDs were added? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:55 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Ceph does not recover from OSD restart
by Frank Schilder 04 Aug '20

04 Aug '20

Hi Eric, > Have you adjusted the min_size for pool sr-rbd-data-one-hdd Yes. For all EC pools located in datacenter ServerRoom, we currently set min_size=k=6, because we lack physical servers. Hosts ceph-21 and ceph-22 are logical but not physical, disks in these buckets are co-located such that no more than 2 host buckets share the same physical host. With failure domain = host, we can ensure that no more than 2 EC shards are on the same physical host. With m=2 and min_size=k we have continued service with any 1 physical host down for maintenance and also recovery will happen if a physical host fails. Some objects will have no redundancy for a while then. We will increase min_size to k+1 as soon as we have 2 additional hosts and simply move the OSDs from buckets ceph-21/22 to these without rebalancing. The distribution of disks and buckets is listed below as well (longer listing). Thanks and best regards, Frank # ceph osd erasure-code-profile ls con-ec-8-2-hdd con-ec-8-2-ssd default sr-ec-6-2-hdd This is the relevant one: # ceph osd erasure-code-profile get sr-ec-6-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ServerRoom jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 Note that the pool sr-rbd-data-one (id 2) was created with this profile and later moved to SSD. Therefore, the crush rule does not match the profile's device class any more. These two are under different roots: # ceph osd erasure-code-profile get con-ec-8-2-hdd crush-device-class=hdd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get con-ec-8-2-ssd crush-device-class=ssd crush-failure-domain=host crush-root=ContainerSquare jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 Full physical placement information for OSDs under tree "datacenter ServerRoom": ---------------- ceph-04 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 243 ceph-04 1.8T SSD osd-phy1 247 ceph-21 1.8T SSD osd-phy2 254 ceph-04 1.8T SSD osd-phy3 256 ceph-04 1.8T SSD osd-phy4 286 ceph-04 1.8T SSD osd-phy5 287 ceph-04 1.8T SSD osd-phy6 288 ceph-04 10.7T HDD osd-phy7 48 ceph-04 372.6G SSD osd-phy8 264 ceph-21 1.8T SSD osd-phy9 84 ceph-04 8.9T HDD osd-phy10 72 ceph-21 8.9T HDD osd-phy11 145 ceph-04 8.9T HDD osd-phy14 156 ceph-04 8.9T HDD osd-phy15 168 ceph-04 8.9T HDD osd-phy16 181 ceph-04 8.9T HDD osd-phy17 0 ceph-21 8.9T HDD ---------------- ceph-05 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 240 ceph-05 1.8T SSD osd-phy1 249 ceph-22 1.8T SSD osd-phy2 251 ceph-05 1.8T SSD osd-phy3 255 ceph-05 1.8T SSD osd-phy4 284 ceph-05 1.8T SSD osd-phy5 285 ceph-05 1.8T SSD osd-phy6 289 ceph-05 10.7T HDD osd-phy7 49 ceph-05 372.6G SSD osd-phy8 265 ceph-22 1.8T SSD osd-phy9 74 ceph-05 8.9T HDD osd-phy10 85 ceph-22 8.9T HDD osd-phy11 144 ceph-05 8.9T HDD osd-phy14 157 ceph-05 8.9T HDD osd-phy15 169 ceph-05 8.9T HDD osd-phy16 180 ceph-05 8.9T HDD osd-phy17 1 ceph-22 8.9T HDD ---------------- ceph-06 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 244 ceph-06 1.8T SSD osd-phy1 246 ceph-21 1.8T SSD osd-phy2 253 ceph-06 1.8T SSD osd-phy3 257 ceph-06 1.8T SSD osd-phy4 282 ceph-06 1.8T SSD osd-phy5 283 ceph-06 1.8T SSD osd-phy6 40 ceph-06 372.6G SSD osd-phy7 50 ceph-06 372.6G SSD osd-phy8 60 ceph-06 8.9T HDD osd-phy9 290 ceph-06 10.7T HDD osd-phy10 291 ceph-21 10.7T HDD osd-phy11 146 ceph-06 8.9T HDD osd-phy14 158 ceph-06 8.9T HDD osd-phy15 170 ceph-06 8.9T HDD osd-phy16 182 ceph-06 8.9T HDD osd-phy17 2 ceph-21 8.9T HDD ---------------- ceph-07 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 242 ceph-07 1.8T SSD osd-phy1 250 ceph-22 1.8T SSD osd-phy2 252 ceph-07 1.8T SSD osd-phy3 258 ceph-07 1.8T SSD osd-phy4 279 ceph-07 1.8T SSD osd-phy5 280 ceph-07 1.8T SSD osd-phy6 292 ceph-07 10.7T HDD osd-phy7 52 ceph-07 372.6G SSD osd-phy8 63 ceph-07 8.9T HDD osd-phy9 281 ceph-22 1.8T SSD osd-phy10 87 ceph-22 8.9T HDD osd-phy11 148 ceph-07 8.9T HDD osd-phy14 159 ceph-07 8.9T HDD osd-phy15 172 ceph-07 8.9T HDD osd-phy16 183 ceph-07 8.9T HDD osd-phy17 3 ceph-22 8.9T HDD ---------------- ceph-18 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 241 ceph-18 1.8T SSD osd-phy1 248 ceph-18 1.8T SSD osd-phy2 41 ceph-18 372.6G SSD osd-phy3 31 ceph-18 372.6G SSD osd-phy4 277 ceph-18 1.8T SSD osd-phy5 278 ceph-21 1.8T SSD osd-phy6 53 ceph-21 372.6G SSD osd-phy7 267 ceph-18 1.8T SSD osd-phy8 266 ceph-18 1.8T SSD osd-phy9 293 ceph-18 10.7T HDD osd-phy10 86 ceph-21 8.9T HDD osd-phy11 259 ceph-18 10.9T HDD osd-phy14 229 ceph-18 8.9T HDD osd-phy15 232 ceph-18 8.9T HDD osd-phy16 235 ceph-18 8.9T HDD osd-phy17 238 ceph-18 8.9T HDD ---------------- ceph-19 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 261 ceph-19 1.8T SSD osd-phy1 262 ceph-19 1.8T SSD osd-phy2 295 ceph-19 10.7T HDD osd-phy3 43 ceph-19 372.6G SSD osd-phy4 275 ceph-19 1.8T SSD osd-phy5 294 ceph-22 10.7T HDD osd-phy6 51 ceph-22 372.6G SSD osd-phy7 269 ceph-19 1.8T SSD osd-phy8 268 ceph-19 1.8T SSD osd-phy9 276 ceph-22 1.8T SSD osd-phy10 73 ceph-22 8.9T HDD osd-phy11 263 ceph-19 10.9T HDD osd-phy14 231 ceph-19 8.9T HDD osd-phy15 233 ceph-19 8.9T HDD osd-phy16 236 ceph-19 8.9T HDD osd-phy17 239 ceph-19 8.9T HDD ---------------- ceph-20 ---------------- CONT ID BUCKET SIZE TYP osd-phy0 245 ceph-20 1.8T SSD osd-phy1 28 ceph-20 372.6G SSD osd-phy2 44 ceph-20 372.6G SSD osd-phy3 271 ceph-20 1.8T SSD osd-phy4 272 ceph-20 1.8T SSD osd-phy5 273 ceph-20 1.8T SSD osd-phy6 274 ceph-21 1.8T SSD osd-phy7 296 ceph-20 10.7T HDD osd-phy8 76 ceph-21 8.9T HDD osd-phy9 39 ceph-21 372.6G SSD osd-phy10 270 ceph-20 1.8T SSD osd-phy11 260 ceph-20 10.9T HDD osd-phy14 228 ceph-20 8.9T HDD osd-phy15 230 ceph-20 8.9T HDD osd-phy16 234 ceph-20 8.9T HDD osd-phy17 237 ceph-20 8.9T HDD CONT is the container name and encodes the physical slot on the host where the OSD is located. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 04 August 2020 12:47:12 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Have you adjusted the min_size for pool sr-rbd-data-one-hdd at all? Also can you send the output of "ceph osd erasure-code-profile ls" and for each EC profile, "ceph osd erasure-code-profile get <profile>"? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 11:05 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Sorry for the many small e-mails: requested IDs in the commands, 288-296. One new OSD per host. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 16:59:04 To: Eric Smith; ceph-users Subject: [ceph-users] Re: Ceph does not recover from OSD restart Hi Eric, the procedure for re-discovering all objects is: # Flag: norebalance ceph osd crush move osd.288 host=bb-04 ceph osd crush move osd.289 host=bb-05 ceph osd crush move osd.290 host=bb-06 ceph osd crush move osd.291 host=bb-21 ceph osd crush move osd.292 host=bb-07 ceph osd crush move osd.293 host=bb-18 ceph osd crush move osd.295 host=bb-19 ceph osd crush move osd.294 host=bb-22 ceph osd crush move osd.296 host=bb-20 # Wait until all PGs are peered and recovery is done. In my case, there was only little I/O, # no more than 50-100 objects had writes missing and recovery was a few seconds. # # The bb-hosts are under a separate crush root that I use solely as parking space # and for draining OSDs. ceph osd crush move osd.288 host=ceph-04 ceph osd crush move osd.289 host=ceph-05 ceph osd crush move osd.290 host=ceph-06 ceph osd crush move osd.291 host=ceph-21 ceph osd crush move osd.292 host=ceph-07 ceph osd crush move osd.293 host=ceph-18 ceph osd crush move osd.295 host=ceph-19 ceph osd crush move osd.294 host=ceph-22 ceph osd crush move osd.296 host=ceph-20 After peering, no degraded PGs/objects any more, just the misplaced ones as expected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 16:45:28 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart You said you had to move some OSDs out and back in for Ceph to go back to normal (The OSDs you added). Which OSDs were added? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:55 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Ceph does not recover from OSD restart
by Frank Schilder 04 Aug '20

04 Aug '20

Hi Eric, the procedure for re-discovering all objects is: # Flag: norebalance ceph osd crush move osd.288 host=bb-04 ceph osd crush move osd.289 host=bb-05 ceph osd crush move osd.290 host=bb-06 ceph osd crush move osd.291 host=bb-21 ceph osd crush move osd.292 host=bb-07 ceph osd crush move osd.293 host=bb-18 ceph osd crush move osd.295 host=bb-19 ceph osd crush move osd.294 host=bb-22 ceph osd crush move osd.296 host=bb-20 # Wait until all PGs are peered and recovery is done. In my case, there was only little I/O, # no more than 50-100 objects had writes missing and recovery was a few seconds. # # The bb-hosts are under a separate crush root that I use solely as parking space # and for draining OSDs. ceph osd crush move osd.288 host=ceph-04 ceph osd crush move osd.289 host=ceph-05 ceph osd crush move osd.290 host=ceph-06 ceph osd crush move osd.291 host=ceph-21 ceph osd crush move osd.292 host=ceph-07 ceph osd crush move osd.293 host=ceph-18 ceph osd crush move osd.295 host=ceph-19 ceph osd crush move osd.294 host=ceph-22 ceph osd crush move osd.296 host=ceph-20 After peering, no degraded PGs/objects any more, just the misplaced ones as expected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 16:45:28 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart You said you had to move some OSDs out and back in for Ceph to go back to normal (The OSDs you added). Which OSDs were added? -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:55 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users <ceph-users(a)ceph.io> Subject: Re: Ceph does not recover from OSD restart Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

save some descriptions with rbd snapshots possible?
by Marc Roos 04 Aug '20

04 Aug '20

Is it already possible to save some descriptions when creating an rbd snapshot?

1 0

Module crash has failed (Octopus)
by Andrei Mikhailovsky 04 Aug '20

04 Aug '20

Hello everyone, I am running my Octopus 15.2.4 version and a couple of days ago noticed an ERROR state on the cluster with the following message: Module 'crash' has failed: dictionary changed size during iteration I couldn't find much info on this error. I've tried restarting the mon servers, which made no effect. How do I fix the error? Many thanks Andrei

2 2

Avast Battery Saver
by Avaknights14＠gmail.com 04 Aug '20

04 Aug '20

If your system runs out of battery very rapidly then avast is the best solution for you. It is one of the most trusted battery savers that stops applications that you are not using, will speed up your operating device and will extend your device’s battery life. It also tells how much time you are left with, on your device’s battery. To know more about how to extend your Android’s battery life with Avast Battery Saver either visit our site or contact our customer support. https://www.avast-support.net/avast-battery-saver/

1 0

Talk to a Cash App Representative to revive a shut account?
by mary smith 04 Aug '20

04 Aug '20

Here, you will get a bit of uplifting news, that weather this application deactivated your Cash app account because of any reason, so you can easily revive your shut account as well as you can also reactivate your Cash Card (Visa charge card). In this way, here are many features that make it popular overall. On the off chance that you any inquiries, please go to Talk to a Cash App Representative. https://www.customercare-email.com/customer-service/cash-app.html

1 0

Epson printer making weird sound? Reach tech support by contacting Epson Customer Service.
by mary smith 04 Aug '20

04 Aug '20

Sometimes an issue can arise when you might hear weird sounds coming from the printer due to some tech glitch. If that happens, then you can get the help by going to the tech help sites or you can call the Epson Customer Service to get the problem resolved. https://www.epsonprintersupportpro.net/

1 0

RadosGW/Keystone intergration issues
by Vladimir Sigunov 04 Aug '20

04 Aug '20

Hello community, Im trying to integrate ceph RadosGW with OpenStack Keystone. Everything is working as expected, but when I try to reach public buckets with public link generated in Horizon, I get a permanent error ‘NoSuchBucket’. However, this bucket and all it’s content does exists: I can access it as authenticated user in Horizon, I can access it as authenticated user via S3 browser/aws cli, I can see it with radosgw- admin bucket list --bucket <bucket>. We are running OpenStack Rocky and this issue appears to be with Ceph Octopus 15.2.4 (there was no issues with RGW on Nautilus and Luminous). Here is my configuration file: <...> [client.rgw.ceph-hdd-9.rgw0] host = ceph-hdd-9 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-hdd-9.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-hdd-9.rgw0.log rgw frontends = beast endpoint=10.10.200.179:8080 rgw thread pool size = 512 rgw zone = default rgw keystone api version = 3 rgw keystone url = https://<keystone url>:13000 rgw keystone accepted roles = admin, _member_, Member, member, creator, swiftoperator rgw keystone accepted admin roles = admin, _member_ #rgw keystone token cache size = 0 #rgw keystone revocation interval = 0 rgw keystone implicit tenants = true rgw keystone admin domain = default rgw keystone admin project = service rgw keystone admin user = swift rgw keystone admin password = swift_osp_password rgw s3 auth use keystone = true rgw s3 auth order = local, external rgw user default quota max size = -1 rgw swift account in url = true rgw dynamic resharding = false rgw bucket resharding = false rgw enable usage log = true rgw usage log tick interval = 30 rgw usage log flush threshold = 1024 rgw usage max shards = 32 rgw usage max user shards = 1 rgw verify ssl = false Please advise. Thank you in advance for your help, Vladimir

2 2

LDAP integration
by jhamster＠chem.byu.edu 03 Aug '20

03 Aug '20

Hello, I am looking into connecting my rados gateway to LDAP and found the following documentation. https://docs.ceph.com/docs/master/radosgw/ldap-auth/ I would like to allow an LDAP group to have access to create and manage buckets. The questions I still have are the following: -Do the LDAP users need to log in to some sort of portal before their corresponding ceph user is created? If so, where do they go to do so? Or does the creation of ceph users and keys happen automatically? -How can you access the ldap users key and secret after they are integrated? Thanks in advance for any information you can provide. Regards, Jared

1 0

HEALTH_WARN crush map has legacy tunables (require firefly, min is hammer)
by Mike Garza 03 Aug '20

03 Aug '20

Hi, I've been tasked with moving Jewel clusters to Nautilus. After the final upgrade Ceph Health warns about legacy tunables. On clusters running SSD's I enabled the optimal flag. Which took weeks to chug through remappings. My remaining clusters run HDD's. Does anyone have experience with using the legacy flag? I'd like to clear up the health warning without outright silencing it. But, I also do not want to kick off any remapping either. Does anyone have experience with pushing this change down the road? Mike

1 0

Re: Ceph does not recover from OSD restart
by Frank Schilder 03 Aug '20

03 Aug '20

Hi Eric, thanks for your fast response. Below the output, shortened a bit as indicated. Disks have been added to pool 11 'sr-rbd-data-one-hdd' only, this is the only pool with remapped PGs and is also the only pool experiencing the "loss of track" to objects. Every other pool recovers from restart by itself. Best regards, Frank # ceph osd pool stats pool sr-rbd-meta-one id 1 client io 5.3 KiB/s rd, 3.2 KiB/s wr, 4 op/s rd, 1 op/s wr pool sr-rbd-data-one id 2 client io 24 MiB/s rd, 32 MiB/s wr, 380 op/s rd, 594 op/s wr pool sr-rbd-one-stretch id 3 nothing is going on pool con-rbd-meta-hpc-one id 7 nothing is going on pool con-rbd-data-hpc-one id 8 client io 0 B/s rd, 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr pool sr-rbd-data-one-hdd id 11 53241814/346903376 objects misplaced (15.348%) client io 73 MiB/s rd, 3.4 MiB/s wr, 236 op/s rd, 69 op/s wr pool con-fs2-meta1 id 12 client io 106 KiB/s rd, 112 KiB/s wr, 3 op/s rd, 11 op/s wr pool con-fs2-meta2 id 13 client io 0 B/s wr, 0 op/s rd, 0 op/s wr pool con-fs2-data id 14 client io 5.5 MiB/s rd, 201 KiB/s wr, 34 op/s rd, 8 op/s wr pool con-fs2-data-ec-ssd id 17 nothing is going on pool ms-rbd-one id 18 client io 5.6 MiB/s wr, 0 op/s rd, 179 op/s wr # ceph osd pool ls detail pool 1 'sr-rbd-meta-one' replicated size 3 min_size 2 crush_rule 11 object_hash rjenkins pg_num 80 pgp_num 80 last_change 122597 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd removed_snaps [1~45] pool 2 'sr-rbd-data-one' erasure size 8 min_size 6 crush_rule 5 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186437 lfor 0/126858 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 43980465111040 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~3,5~2, ... huge list ... ,11f9d~1,11fa0~2] pool 3 'sr-rbd-one-stretch' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 160 pgp_num 160 last_change 143202 lfor 0/79983 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~7,b~2,11~2,14~2,17~9e,b8~1e] pool 7 'con-rbd-meta-hpc-one' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96357 lfor 0/90462 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 10737418240 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'con-rbd-data-hpc-one' erasure size 10 min_size 9 crush_rule 7 object_hash rjenkins pg_num 150 pgp_num 150 last_change 96358 lfor 0/90996 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 5497558138880 stripe_width 32768 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~7,9~2] pool 11 'sr-rbd-data-one-hdd' erasure size 8 min_size 6 crush_rule 9 object_hash rjenkins pg_num 560 pgp_num 560 last_change 186331 lfor 0/127768 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps max_bytes 219902325555200 stripe_width 24576 fast_read 1 compression_mode aggressive application rbd removed_snaps [1~59f,5a2~fe, ... less huge list ... ,2559~1,255b~1] removed_snaps_queue [1a64~5,1a6a~1,1a6c~1, ... long list ... ,220a~1,220c~1] pool 12 'con-fs2-meta1' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 57096 flags hashpspool,nodelete max_bytes 268435456000 stripe_width 0 application cephfs pool 13 'con-fs2-meta2' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 50 pgp_num 50 last_change 96359 flags hashpspool,nodelete max_bytes 107374182400 stripe_width 0 application cephfs pool 14 'con-fs2-data' erasure size 10 min_size 9 crush_rule 8 object_hash rjenkins pg_num 1350 pgp_num 1350 last_change 96360 lfor 0/91144 flags hashpspool,ec_overwrites,nodelete max_bytes 879609302220800 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 17 'con-fs2-data-ec-ssd' erasure size 10 min_size 9 crush_rule 10 object_hash rjenkins pg_num 55 pgp_num 55 last_change 96361 lfor 0/90473 flags hashpspool,ec_overwrites,nodelete max_bytes 1099511627776 stripe_width 32768 fast_read 1 compression_mode aggressive application cephfs pool 18 'ms-rbd-one' replicated size 3 min_size 2 crush_rule 12 object_hash rjenkins pg_num 150 pgp_num 150 last_change 143206 flags hashpspool,nodelete,selfmanaged_snaps max_bytes 1099511627776 stripe_width 0 compression_mode aggressive application rbd removed_snaps [1~3] # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -40 2384.09058 root DTU -42 0 region Lyngby -41 2384.09058 region Risoe 2 sub-trees on level datacenter removed for brevity -49 586.49347 datacenter ServerRoom -55 586.49347 room SR-113 -65 64.33617 host ceph-04 84 hdd 8.90999 osd.84 up 1.00000 1.00000 145 hdd 8.90999 osd.145 up 1.00000 1.00000 156 hdd 8.90999 osd.156 up 1.00000 1.00000 168 hdd 8.90999 osd.168 up 1.00000 1.00000 181 hdd 8.90999 osd.181 up 0.95000 1.00000 288 hdd 10.69229 osd.288 up 1.00000 1.00000 243 rbd_data 1.74599 osd.243 up 1.00000 1.00000 254 rbd_data 1.74599 osd.254 up 1.00000 1.00000 256 rbd_data 1.74599 osd.256 up 1.00000 1.00000 286 rbd_data 1.74599 osd.286 up 1.00000 1.00000 287 rbd_data 1.74599 osd.287 up 1.00000 1.00000 48 rbd_meta 0.36400 osd.48 up 1.00000 1.00000 -67 64.33617 host ceph-05 74 hdd 8.90999 osd.74 up 1.00000 1.00000 144 hdd 8.90999 osd.144 up 1.00000 1.00000 157 hdd 8.90999 osd.157 up 0.84999 1.00000 169 hdd 8.90999 osd.169 up 0.95000 1.00000 180 hdd 8.90999 osd.180 up 0.89999 1.00000 289 hdd 10.69229 osd.289 up 1.00000 1.00000 240 rbd_data 1.74599 osd.240 up 1.00000 1.00000 251 rbd_data 1.74599 osd.251 up 1.00000 1.00000 255 rbd_data 1.74599 osd.255 up 1.00000 1.00000 284 rbd_data 1.74599 osd.284 up 1.00000 1.00000 285 rbd_data 1.74599 osd.285 up 1.00000 1.00000 49 rbd_meta 0.36400 osd.49 up 1.00000 1.00000 -69 64.70016 host ceph-06 60 hdd 8.90999 osd.60 up 1.00000 1.00000 146 hdd 8.90999 osd.146 up 1.00000 1.00000 158 hdd 8.90999 osd.158 up 0.95000 1.00000 170 hdd 8.90999 osd.170 up 0.89999 1.00000 182 hdd 8.90999 osd.182 up 1.00000 1.00000 290 hdd 10.69229 osd.290 up 1.00000 1.00000 244 rbd_data 1.74599 osd.244 up 1.00000 1.00000 253 rbd_data 1.74599 osd.253 up 1.00000 1.00000 257 rbd_data 1.74599 osd.257 up 1.00000 1.00000 282 rbd_data 1.74599 osd.282 up 1.00000 1.00000 283 rbd_data 1.74599 osd.283 up 1.00000 1.00000 40 rbd_meta 0.36400 osd.40 up 1.00000 1.00000 50 rbd_meta 0.36400 osd.50 up 1.00000 1.00000 -71 64.33617 host ceph-07 63 hdd 8.90999 osd.63 up 1.00000 1.00000 148 hdd 8.90999 osd.148 up 0.95000 1.00000 159 hdd 8.90999 osd.159 up 1.00000 1.00000 172 hdd 8.90999 osd.172 up 0.95000 1.00000 183 hdd 8.90999 osd.183 up 0.84999 1.00000 292 hdd 10.69229 osd.292 up 1.00000 1.00000 242 rbd_data 1.74599 osd.242 up 1.00000 1.00000 252 rbd_data 1.74599 osd.252 up 1.00000 1.00000 258 rbd_data 1.74599 osd.258 up 1.00000 1.00000 279 rbd_data 1.74599 osd.279 up 1.00000 1.00000 280 rbd_data 1.74599 osd.280 up 1.00000 1.00000 52 rbd_meta 0.36400 osd.52 up 1.00000 1.00000 -81 66.70416 host ceph-18 229 hdd 8.90999 osd.229 up 1.00000 1.00000 232 hdd 8.90999 osd.232 up 1.00000 1.00000 235 hdd 8.90999 osd.235 up 1.00000 1.00000 238 hdd 8.90999 osd.238 up 0.95000 1.00000 259 hdd 10.91399 osd.259 up 1.00000 1.00000 293 hdd 10.69229 osd.293 up 1.00000 1.00000 241 rbd_data 1.74599 osd.241 up 1.00000 1.00000 248 rbd_data 1.74599 osd.248 up 1.00000 1.00000 266 rbd_data 1.74599 osd.266 up 1.00000 1.00000 267 rbd_data 1.74599 osd.267 up 1.00000 1.00000 277 rbd_data 1.74599 osd.277 up 1.00000 1.00000 31 rbd_meta 0.36400 osd.31 up 1.00000 1.00000 41 rbd_meta 0.36400 osd.41 up 1.00000 1.00000 -94 66.34016 host ceph-19 231 hdd 8.90999 osd.231 up 1.00000 1.00000 233 hdd 8.90999 osd.233 up 0.95000 1.00000 236 hdd 8.90999 osd.236 up 1.00000 1.00000 239 hdd 8.90999 osd.239 up 1.00000 1.00000 263 hdd 10.91399 osd.263 up 1.00000 1.00000 295 hdd 10.69229 osd.295 up 1.00000 1.00000 261 rbd_data 1.74599 osd.261 up 1.00000 1.00000 262 rbd_data 1.74599 osd.262 up 1.00000 1.00000 268 rbd_data 1.74599 osd.268 up 1.00000 1.00000 269 rbd_data 1.74599 osd.269 up 1.00000 1.00000 275 rbd_data 1.74599 osd.275 up 1.00000 1.00000 43 rbd_meta 0.36400 osd.43 up 1.00000 1.00000 -4 66.70416 host ceph-20 228 hdd 8.90999 osd.228 up 1.00000 1.00000 230 hdd 8.90999 osd.230 up 1.00000 1.00000 234 hdd 8.90999 osd.234 up 0.95000 1.00000 237 hdd 8.90999 osd.237 up 1.00000 1.00000 260 hdd 10.91399 osd.260 up 1.00000 1.00000 296 hdd 10.69229 osd.296 up 1.00000 1.00000 245 rbd_data 1.74599 osd.245 up 1.00000 1.00000 270 rbd_data 1.74599 osd.270 up 1.00000 1.00000 271 rbd_data 1.74599 osd.271 up 1.00000 1.00000 272 rbd_data 1.74599 osd.272 up 1.00000 1.00000 273 rbd_data 1.74599 osd.273 up 1.00000 1.00000 28 rbd_meta 0.36400 osd.28 up 1.00000 1.00000 44 rbd_meta 0.36400 osd.44 up 1.00000 1.00000 -64 64.70016 host ceph-21 0 hdd 8.90999 osd.0 up 1.00000 1.00000 2 hdd 8.90999 osd.2 up 0.95000 1.00000 72 hdd 8.90999 osd.72 up 1.00000 1.00000 76 hdd 8.90999 osd.76 up 1.00000 1.00000 86 hdd 8.90999 osd.86 up 1.00000 1.00000 291 hdd 10.69229 osd.291 up 1.00000 1.00000 246 rbd_data 1.74599 osd.246 up 1.00000 1.00000 247 rbd_data 1.74599 osd.247 up 1.00000 1.00000 264 rbd_data 1.74599 osd.264 up 1.00000 1.00000 274 rbd_data 1.74599 osd.274 up 1.00000 1.00000 278 rbd_data 1.74599 osd.278 up 1.00000 1.00000 39 rbd_meta 0.36400 osd.39 up 1.00000 1.00000 53 rbd_meta 0.36400 osd.53 up 1.00000 1.00000 -66 64.33617 host ceph-22 1 hdd 8.90999 osd.1 up 1.00000 1.00000 3 hdd 8.90999 osd.3 up 1.00000 1.00000 73 hdd 8.90999 osd.73 up 1.00000 1.00000 85 hdd 8.90999 osd.85 up 0.95000 1.00000 87 hdd 8.90999 osd.87 up 1.00000 1.00000 294 hdd 10.69229 osd.294 up 1.00000 1.00000 249 rbd_data 1.74599 osd.249 up 1.00000 1.00000 250 rbd_data 1.74599 osd.250 up 1.00000 1.00000 265 rbd_data 1.74599 osd.265 up 1.00000 1.00000 276 rbd_data 1.74599 osd.276 up 1.00000 1.00000 281 rbd_data 1.74599 osd.281 up 1.00000 1.00000 51 rbd_meta 0.36400 osd.51 up 1.00000 1.00000 # ceph osd crush rule dump # crush rules outside tree under "datacenter ServerRoom" removed for brevity [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 5, "rule_name": "sr-rbd-data-one", "ruleset": 5, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 50 }, { "op": "set_choose_tries", "num": 1000 }, { "op": "take", "item": -185, "item_name": "ServerRoom~rbd_data" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 9, "rule_name": "sr-rbd-data-one-hdd", "ruleset": 9, "type": 3, "min_size": 3, "max_size": 8, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -53, "item_name": "ServerRoom~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eric Smith <Eric.Smith(a)vecima.com> Sent: 03 August 2020 15:40 To: Frank Schilder; ceph-users Subject: RE: Ceph does not recover from OSD restart Can you post the output of these commands: ceph osd pool ls detail ceph osd tree ceph osd crush rule dump -----Original Message----- From: Frank Schilder <frans(a)dtu.dk> Sent: Monday, August 3, 2020 9:19 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Ceph does not recover from OSD restart After moving the newly added OSDs out of the crush tree and back in again, I get to exactly what I want to see: cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_WARN norebalance,norecover flag(s) set 53030026/1492404361 objects misplaced (3.553%) 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs flags norebalance,norecover data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 53030026/1492404361 objects misplaced (3.553%) 2902 active+clean 299 active+remapped+backfill_wait 8 active+remapped+backfilling 5 active+clean+scrubbing+deep 1 active+clean+snaptrim io: client: 69 MiB/s rd, 117 MiB/s wr, 399 op/s rd, 856 op/s wr Why does a cluster with remapped PGs not survive OSD restarts without loosing track of objects? Why is it not finding the objects by itself? A power outage of 3 hosts will halt everything for no reason until manual intervention. How can I avoid this problem? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 03 August 2020 15:03:05 To: ceph-users Subject: [ceph-users] Ceph does not recover from OSD restart Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Ceph does not recover from OSD restart
by Frank Schilder 03 Aug '20

03 Aug '20

Dear cephers, I have a serious issue with degraded objects after an OSD restart. The cluster was in a state of re-balancing after adding disks to each host. Before restart I had "X/Y objects misplaced". Apart from that, health was OK. I now restarted all OSDs of one host and the cluster does not recover from that: cluster: id: xxx health: HEALTH_ERR 45813194/1492348700 objects misplaced (3.070%) Degraded data redundancy: 6798138/1492348700 objects degraded (0.456%), 85 pgs degraded, 86 pgs undersized Degraded data redundancy (low space): 17 pgs backfill_toofull 1 pools nearfull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 297 osds: 272 up, 272 in; 307 remapped pgs data: pools: 11 pools, 3215 pgs objects: 177.3 M objects, 489 TiB usage: 696 TiB used, 1.2 PiB / 1.9 PiB avail pgs: 6798138/1492348700 objects degraded (0.456%) 45813194/1492348700 objects misplaced (3.070%) 2903 active+clean 209 active+remapped+backfill_wait 73 active+undersized+degraded+remapped+backfill_wait 9 active+remapped+backfill_wait+backfill_toofull 8 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 4 active+undersized+degraded+remapped+backfilling 3 active+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+clean+scrubbing 1 active+undersized+remapped+backfilling 1 active+clean+snaptrim io: client: 47 MiB/s rd, 61 MiB/s wr, 732 op/s rd, 792 op/s wr recovery: 195 MiB/s, 48 objects/s After restarting there should only be a small number of degraded objects, the ones that received writes during OSD restart. What I see, however, is that the cluster seems to have lost track of a huge amount of objects, the 0.456% degraded are 1-2 days worth of I/O. I did reboots before and saw only a few thousand objects degraded at most. The output of ceph health detail shows a lot of lines like these: [root@gnosis ~]# ceph health detail HEALTH_ERR 45804316/1492356704 objects misplaced (3.069%); Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized; Degraded data redundancy (low space): 17 pgs backfill_toofull; 1 pools nearfull OBJECT_MISPLACED 45804316/1492356704 objects misplaced (3.069%) PG_DEGRADED Degraded data redundancy: 6792562/1492356704 objects degraded (0.455%), 85 pgs degraded, 86 pgs undersized pg 11.9 is stuck undersized for 815.188981, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,148,2147483647,263,76,230,87,169] 8...9 pg 11.48 is active+undersized+degraded+remapped+backfill_wait, acting [159,60,180,263,237,3,2147483647,72] pg 11.4a is stuck undersized for 851.162862, current state active+undersized+degraded+remapped+backfill_wait, last acting [182,233,87,228,2,180,63,2147483647] [...] pg 11.22e is stuck undersized for 851.162402, current state active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last acting [234,183,239,2147483647,170,229,1,86] PG_DEGRADED_FULL Degraded data redundancy (low space): 17 pgs backfill_toofull pg 11.24 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [230,259,2147483647,1,144,159,233,146] [...] pg 11.1d9 is active+remapped+backfill_wait+backfill_toofull, acting [84,259,183,170,85,234,233,2] pg 11.225 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [236,183,1,2147483647,2147483647,169,229,230] pg 11.22e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [234,183,239,2147483647,170,229,1,86] POOL_NEAR_FULL 1 pools nearfull pool 'sr-rbd-data-one-hdd' has 164 TiB (max 200 TiB) It looks like a lot of PGs are not receiving theire complete crush map placement, as if the peering is incomplete. This is a serious issue, it looks like the cluster will see a total storage loss if just 2 more hosts reboot - without actually having lost any storage. The pool in question is a 6+2 EC pool. What is going on here? Why are the PG-maps not restored to their values from before the OSD reboot? The degraded PGs should receive the missing OSD IDs, everything is up exactly as it was before the reboot. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

mimic: much more raw used than reported
by Frank Schilder 03 Aug '20

03 Aug '20

Dear fellow cephers, I observe a wired problem on our mimic-13.2.8 cluster. We have an EC RBD pool backed by HDDs. These disks are not in any other pool. I noticed that the total capacity (=USED+MAX AVAIL) reported by "ceph df detail" has shrunk recently from 300TiB to 200TiB. Part but by no means all of this can be explained by imbalance of the data distribution. When I compare the output of "ceph df detail" and "ceph osd df tree", I find 69TiB raw capacity used but not accounted for; see calculations below. These 69TiB raw are equivalent to 20% usable capacity and I really need it back. Together with the imbalance, we loose about 30% capacity. What is using these extra 69TiB and how can I get it back? Some findings: These are the 5 largest images in the pool, accounting for a total of 97TiB out of 119TiB usage: # rbd du : NAME PROVISIONED USED one-133 25 TiB 14 TiB NAME PROVISIONED USED one-153@222 40 TiB 14 TiB one-153@228 40 TiB 357 GiB one-153@235 40 TiB 797 GiB one-153@241 40 TiB 509 GiB one-153@242 40 TiB 43 GiB one-153@243 40 TiB 16 MiB one-153@244 40 TiB 16 MiB one-153@245 40 TiB 324 MiB one-153@246 40 TiB 276 MiB one-153@247 40 TiB 96 MiB one-153@248 40 TiB 138 GiB one-153@249 40 TiB 1.8 GiB one-153@250 40 TiB 0 B one-153 40 TiB 204 MiB <TOTAL> 40 TiB 16 TiB NAME PROVISIONED USED one-391@3 40 TiB 432 MiB one-391@9 40 TiB 26 GiB one-391@15 40 TiB 90 GiB one-391@16 40 TiB 0 B one-391@17 40 TiB 0 B one-391@18 40 TiB 0 B one-391@19 40 TiB 0 B one-391@20 40 TiB 3.5 TiB one-391@21 40 TiB 5.4 TiB one-391@22 40 TiB 5.8 TiB one-391@23 40 TiB 8.4 TiB one-391@24 40 TiB 1.4 TiB one-391 40 TiB 2.2 TiB <TOTAL> 40 TiB 27 TiB NAME PROVISIONED USED one-394@3 70 TiB 1.4 TiB one-394@9 70 TiB 2.5 TiB one-394@15 70 TiB 20 GiB one-394@16 70 TiB 0 B one-394@17 70 TiB 0 B one-394@18 70 TiB 0 B one-394@19 70 TiB 383 GiB one-394@20 70 TiB 3.3 TiB one-394@21 70 TiB 5.0 TiB one-394@22 70 TiB 5.0 TiB one-394@23 70 TiB 9.0 TiB one-394@24 70 TiB 1.6 TiB one-394 70 TiB 2.5 TiB <TOTAL> 70 TiB 31 TiB NAME PROVISIONED USED one-434 25 TiB 9.1 TiB The large 70TiB images one-391 and one-394 are currently copied to with ca. 5TiB per day. Output of "ceph df detail" with some columns removed: NAME ID USED %USED MAX AVAIL OBJECTS RAW USED sr-rbd-data-one-hdd 11 119 TiB 58.45 84 TiB 31286554 158 TiB Pool is EC 6+2. USED is correct: 31286554*4MiB=119TiB. RAW USED is correct: 119*8/6=158TiB. Most of this data is freshly copied onto large RBD images. Compression is enabled on this pool (aggressive,snappy). However, when looking at "deph osd df tree", I get The combined raw capacity of OSDs backing this pool is 406.8TiB (sum over SIZE). Summing up column USE over all OSDs gives 227.5TiB. This gives a difference of 69TiB (=227-158) that is not accounted for. Here the output of "ceph osd df tree limited" to the drives backing the pool: ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME 84 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 180 MiB 16 GiB 3.9 TiB 56.43 1.72 103 osd.84 145 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.6 TiB 144 MiB 14 GiB 4.3 TiB 51.37 1.57 87 osd.145 156 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.1 TiB 173 MiB 16 GiB 3.8 TiB 57.91 1.77 100 osd.156 168 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 164 MiB 16 GiB 3.9 TiB 56.31 1.72 98 osd.168 181 hdd 8.90999 1.00000 8.9 TiB 5.5 TiB 5.4 TiB 121 MiB 17 GiB 3.5 TiB 61.26 1.87 105 osd.181 74 hdd 8.90999 1.00000 8.9 TiB 4.2 TiB 4.2 TiB 148 MiB 13 GiB 4.7 TiB 46.79 1.43 85 osd.74 144 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 106 MiB 15 GiB 4.2 TiB 53.17 1.62 94 osd.144 157 hdd 8.90999 1.00000 8.9 TiB 5.8 TiB 5.8 TiB 192 MiB 18 GiB 3.1 TiB 65.02 1.99 111 osd.157 169 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 172 MiB 16 GiB 3.8 TiB 56.99 1.74 102 osd.169 180 hdd 8.90999 1.00000 8.9 TiB 5.8 TiB 5.8 TiB 131 MiB 18 GiB 3.1 TiB 65.04 1.99 111 osd.180 60 hdd 8.90999 1.00000 8.9 TiB 4.5 TiB 4.5 TiB 155 MiB 14 GiB 4.4 TiB 50.40 1.54 93 osd.60 146 hdd 8.90999 1.00000 8.9 TiB 4.8 TiB 4.8 TiB 139 MiB 15 GiB 4.1 TiB 53.70 1.64 92 osd.146 158 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.5 TiB 183 MiB 17 GiB 3.4 TiB 62.30 1.90 109 osd.158 170 hdd 8.90999 1.00000 8.9 TiB 5.7 TiB 5.6 TiB 205 MiB 18 GiB 3.2 TiB 63.53 1.94 112 osd.170 182 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.6 TiB 105 MiB 14 GiB 4.3 TiB 52.27 1.60 92 osd.182 63 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 156 MiB 15 GiB 4.2 TiB 52.74 1.61 98 osd.63 148 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.1 TiB 119 MiB 16 GiB 3.8 TiB 57.82 1.77 100 osd.148 159 hdd 8.90999 1.00000 8.9 TiB 4.0 TiB 4.0 TiB 89 MiB 12 GiB 4.9 TiB 44.61 1.36 79 osd.159 172 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 173 MiB 16 GiB 3.8 TiB 57.22 1.75 98 osd.172 183 hdd 8.90999 1.00000 8.9 TiB 6.0 TiB 6.0 TiB 135 MiB 19 GiB 2.9 TiB 67.35 2.06 118 osd.183 229 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.6 TiB 127 MiB 15 GiB 4.3 TiB 52.05 1.59 93 osd.229 232 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.2 TiB 158 MiB 17 GiB 3.7 TiB 58.22 1.78 101 osd.232 235 hdd 8.90999 1.00000 8.9 TiB 4.1 TiB 4.1 TiB 103 MiB 13 GiB 4.8 TiB 45.96 1.40 79 osd.235 238 hdd 8.90999 1.00000 8.9 TiB 5.4 TiB 5.4 TiB 120 MiB 17 GiB 3.5 TiB 60.47 1.85 104 osd.238 259 hdd 10.91399 1.00000 11 TiB 6.2 TiB 6.2 TiB 140 MiB 19 GiB 4.7 TiB 56.54 1.73 120 osd.259 231 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 114 MiB 16 GiB 3.8 TiB 56.90 1.74 101 osd.231 233 hdd 8.90999 1.00000 8.9 TiB 5.5 TiB 5.5 TiB 123 MiB 17 GiB 3.4 TiB 61.78 1.89 106 osd.233 236 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 114 MiB 16 GiB 3.8 TiB 57.53 1.76 101 osd.236 239 hdd 8.90999 1.00000 8.9 TiB 4.2 TiB 4.2 TiB 95 MiB 13 GiB 4.7 TiB 47.41 1.45 86 osd.239 263 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.3 TiB 178 MiB 17 GiB 5.6 TiB 48.73 1.49 102 osd.263 228 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 113 MiB 16 GiB 3.8 TiB 57.10 1.74 96 osd.228 230 hdd 8.90999 1.00000 8.9 TiB 4.9 TiB 4.9 TiB 144 MiB 16 GiB 4.0 TiB 55.20 1.69 99 osd.230 234 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.6 TiB 164 MiB 18 GiB 3.3 TiB 63.29 1.93 109 osd.234 237 hdd 8.90999 1.00000 8.9 TiB 4.8 TiB 4.8 TiB 110 MiB 15 GiB 4.1 TiB 54.33 1.66 97 osd.237 260 hdd 10.91399 1.00000 11 TiB 5.4 TiB 5.4 TiB 152 MiB 17 GiB 5.5 TiB 49.35 1.51 104 osd.260 0 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.2 TiB 157 MiB 16 GiB 3.7 TiB 58.28 1.78 102 osd.0 2 hdd 8.90999 1.00000 8.9 TiB 5.3 TiB 5.2 TiB 122 MiB 16 GiB 3.6 TiB 59.05 1.80 106 osd.2 72 hdd 8.90999 1.00000 8.9 TiB 4.4 TiB 4.4 TiB 145 MiB 14 GiB 4.5 TiB 49.89 1.52 89 osd.72 76 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 178 MiB 16 GiB 3.8 TiB 56.89 1.74 102 osd.76 86 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.5 TiB 155 MiB 14 GiB 4.3 TiB 51.18 1.56 94 osd.86 1 hdd 8.90999 1.00000 8.9 TiB 4.9 TiB 4.9 TiB 141 MiB 15 GiB 4.0 TiB 54.73 1.67 95 osd.1 3 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 156 MiB 15 GiB 4.2 TiB 52.40 1.60 94 osd.3 73 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 4.9 TiB 146 MiB 16 GiB 3.9 TiB 55.68 1.70 102 osd.73 85 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.5 TiB 192 MiB 18 GiB 3.3 TiB 62.46 1.91 109 osd.85 87 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 189 MiB 16 GiB 3.9 TiB 55.91 1.71 102 osd.87 Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 12

Problems with long taking deep-scrubbing processes causing PG_NOT_DEEP_SCRUBBED
by Carsten Grommel - Profihost AG 03 Aug '20

03 Aug '20

Hi, we are having problems with really long taking deep-scrubb processes causing PG_NOT_DEEP_SCRUBBED and ceph HEALTH_WARN. One ph is waiting since 2020-05-18 for the deep-scrubb. Is there any way to speed up the deep-scrubbing? Ceph-Version: ceph version 14.2.8-3-gc6b8eedb77 (c6b8eedb771089fe3b0a95da93158ec4144758f3) nautilus (stable)

3 4

Re: EC profile datastore usage - question
by Mateusz Skała 03 Aug '20

03 Aug '20

Hello, I’m sorry for late response, here is output: ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.0 PiB 797 TiB 267 TiB 272 TiB 25.44 TOTAL 1.0 PiB 797 TiB 267 TiB 272 TiB 25.44 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 7.3 KiB 52 4.5 MiB 0 222 TiB default.rgw.control 2 0 B 25 0 B 0 222 TiB default.rgw.meta 3 15 KiB 152 10 MiB 0 222 TiB default.rgw.log 4 1.8 KiB 631 386 KiB 0 222 TiB default.rgw.buckets.index 6 71 MiB 37 71 MiB 0 222 TiB default.rgw.buckets.data 7 3.3 TiB 1.59M 9.9 TiB 1.46 222 TiB default.rgw.buckets.non-ec 9 962 B 10 385 KiB 0 222 TiB CephFS 10 0 B 0 0 B 0 445 TiB production-repo-old.rgw.buckets.non-ec 15 0 B 0 0 B 0 222 TiB production-repo-old.rgw.buckets.index 21 54 GiB 2 54 GiB 0 222 TiB production-repo-old.rgw.buckets.data 22 75 TiB 266.03M 257 TiB 27.82 400 TiB ceph osd pool ls detail pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 19535 lfor 0/19535/19533 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 19427 lfor 0/19427/19423 flags hashpspool stripe_width 0 pg_num_min 32 application rgw pool 3 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 19647 lfor 0/19647/19645 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15480 lfor 0/15480/15449 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 6 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15508 lfor 0/15508/15449 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 7 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 20549 lfor 0/19147/19152 flags hashpspool max_bytes 10995116277760 stripe_width 0 pg_num_min 256 application rgw pool 9 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 16099 lfor 0/16099/16097 flags hashpspool stripe_width 0 pg_num_min 8 application rgw pool 10 'CephFS' erasure size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 15718 lfor 0/15718/15716 flags hashpspool stripe_width 8192 pg_num_min 8 application cephfs,rbd,rgw pool 15 'production-repo-old.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 20404 flags hashpspool stripe_width 0 application rgw pool 21 'production-repo-old.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode on last_change 20547 flags hashpspool stripe_width 0 application rgw pool 22 'production-repo-old.rgw.buckets.data' erasure size 10 min_size 7 crush_rule 2 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 20546 lfor 0/20540/20542 flags hashpspool stripe_width 24576 pg_num_min 512 application rgw Best regards Mateusz Skała > Wiadomość napisana przez Eric Smith <Eric.Smith(a)vecima.com> w dniu 18.07.2020, o godz. 11:46: > > EXTERNAL EMAIL - Do not click any links or open any attachments unless you trust the sender and know the content is safe. > > Can you post the output of a couple of commands: > > ceph df > ceph osd pool ls detail > > Then we can probably explain the utilization you're seeing. > > -----Original Message----- > From: Mateusz Skała <mateusz.skala(a)gmail.com> > Sent: Saturday, July 18, 2020 1:35 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] EC profile datastore usage - question > > Hello Community, > I would like to ask about help in explanation situation. > There is Rados gateway with EC pool profile k=6 m=4. So it shoud take something about 1.4 - 2.0 data usage more from raw data if I’m correct. > Rados df shows me: > 116 TiB used and WR 26 TiB > Can You explain this? It is about 4.5*WR used data. Why? > Regards > Mateusz Skała > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [ANN] A framework for deploying Octopus using cephadm in the cloud
by Patrick Donnelly 02 Aug '20

02 Aug '20

On Sat, Aug 1, 2020 at 5:50 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > I can understand the benefits of having a CO, I am still testing with > mesos. However what is the benefit of having ceph daemons running in CO > environment? As I said in my original post, I use this for testing CephFS. There's no reason why you can't also use Ceph as a storage system in the cloud though. It just may not be as cost effective as the native cloud storage offering (like S3). >Except for your mds, mrg and radosgw, your osd daemons are > bound to the hardware / disks they are running on. It is not like if > osd.121 goes down, you can start it on some random node. Why not? -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Why newly added OSD need to get all historical OSDMAPs in pre-boot
by Xiaoxi Chen 02 Aug '20

02 Aug '20

Hi List, We see newly added OSD take long time to become active, it is fetching old osdmaps from monitors. Curious why it need that much historical ? The cluster is stable , no osd down/out except the ongoing cap add (one osd at a time). { "cluster_fsid": "e959f744-64be-4f4e-9606-103ccaa9d2d6", "osd_fsid": "0a2d4b9d-f6e5-4e40-be7f-16daee131a20", "whoami": 38, "state": "preboot", "oldest_map": 193222, "newest_map": 298982, "num_pgs": 0 } root@host:~# ceph osd getmap | head -5 got osdmap epoch 324825 -Xiaoxi

1 0

which exact decimal value is meant here for S64_MIN in CRUSH Mapper
by Bobby 01 Aug '20

01 Aug '20

Hi, In *mapper.c* file of Ceph CRUSH, I am trying to understand the definition of a linux macro ```*S64_MIN*``` used in the following ```*else*``` condition i.e. ```*draw = S64_MIN*```. Which exact decimal value is meant here for ```*S64_MIN*```? ``` if (weights[i]) { u = hash(bucket->h.hash, x, ids[i], r); u &= 0xffff; ln = crush_ln(u) - 0x1000000000000ll; __s64 draw = div64_s64(ln, weights[i]); } else { __s64 draw = S64_MIN; // #define S64_MAX ((s64)(U64_MAX >> 1)) // #define S64_MIN ((s64)(-S64_MAX -1)) } if (i == 0 || draw > high_draw) { high = i; high_draw = draw; } } return bucket->h.items[high]; } Thanks Bobby !

1 0

[ANN] A framework for deploying Octopus using cephadm in the cloud
by Patrick Donnelly 01 Aug '20

01 Aug '20

Hello all, There's a framework [0] I've been working on for a while to deploy Ceph in the cloud. For now, this is done through the Linode, LLC cloud provider [1]. Primarily I use this for testing CephFS performance/behavior. For example, it's fairly simple to create a decent sized test cluster with 150GB MDSs and 16x 16GB OSDs for a humble 1.7TB usable storage. Then provision 100+ client machine nodes for executing workflows against the Ceph cluster. This has been very useful for isolating aberrant behaviors that are only uncovered at scale. All of the code/playbooks I use for this purpose are also in the ceph-linode repository. Early versions of this project used ceph-ansible but I have updated the code to use the new cephadm deployment technology [2] in Ceph Octopus release. The ansible playbook to deploy the cluster [3] is delightfully simple. Anyway, I thought it may be useful to the broader community to have an option to try out cephadm on a throwaway cluster for pennies per hour through a VPS provider. Getting a small cluster started should take less than 10 minutes by following the README. I hope some folks out there find this useful for their own testing. Feedback is welcome. [0] https://github.com/batrick/ceph-linode [1] https://www.linode.com/ [2] https://docs.ceph.com/docs/master/cephadm/ [3] https://github.com/batrick/ceph-linode/blob/master/cephadm.yml Full disclosure: I have no relationship with Linode except as a customer. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 2

Boost your grades using assignment help online services
by kevin wick 01 Aug '20

01 Aug '20

Are you feeling irritated because of poor grades in assignment submission? Do you need assistance to boost your grades in assignment writing for Hong Kong universities? Then, divert your mind for assignment writing services and connect with a reliable service provider like us. We are the leading online assignment writing service provider in Hong Kong and help every scholar to enhance his or her marks. Our writers help students to expand their knowledge and boost their scores. You will receive the valuable Assignment Help services through online mode of communication in Hong Kong. By choosing our academic writing services, you can save your time and effort to perform great in your all academic tasks. Leave your irritation of getting lower marks using the assistance of subject matter experts in Hong Kong. https://www.greatassignmenthelp.com/hk/

1 0