October 2019 - ceph-users

Inconsistent PG with data_digest_mismatch_info on all OSDs
by Wido den Hollander 15 Oct '19

15 Oct '19

Hi, I have a Mimic 13.2.6 cluster which is throwing an error on a PG that it's inconsistent. PG_DAMAGED Possible data damage: 1 pg inconsistent pg 21.e6d is active+clean+inconsistent, acting [988,508,825] I checked 'list-inconsistent-obj' (See below) and it shows: selected_object_info: "data_digest": "0xf4342d4a all osds: "data_digest": "0x224d6ca6" This looks like issue 24994 [0], but this is Mimic 13.2.6 and not Luminous 12.2.7 I also tried to download the object: rados -p <pool> get rb.0.304abf.238e1f29.00000001cc48 rb.0.304abf.238e1f29.00000001cc48 That doesn't work. This blocks for ever and causes osd.988 to report a slow request. I don't want to repair the PG at the moment as this might be a bug. Right now I'm thinking of restarting all three OSDs of that PG as this might get things moving again. But how could this happen with all three OSDs reporting the same data digest? Ideas? Wido { "epoch": 1351206, "inconsistents": [ { "object": { "name": "rb.0.304abf.238e1f29.00000001cc48", "nspace": "", "locator": "", "snap": "head", "version": 28429940 }, "errors": [], "union_shard_errors": [ "data_digest_mismatch_info" ], "selected_object_info": { "oid": { "oid": "rb.0.304abf.238e1f29.00000001cc48", "key": "", "snapid": -2, "hash": 367865453, "max": 0, "pool": 21, "namespace": "" }, "version": "1246902'28520530", "prior_version": "1240840'28429940", "last_reqid": "osd.736.0:3448586", "user_version": 28429940, "size": 4194304, "mtime": "2019-08-12 05:24:02.672911", "local_mtime": "2019-08-12 05:24:02.673401", "lost": 0, "flags": [ "dirty", "data_digest", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xf4342d4a", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 508, "primary": false, "errors": [ "data_digest_mismatch_info" ], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x224d6ca6" }, { "osd": 825, "primary": false, "errors": [ "data_digest_mismatch_info" ], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x224d6ca6" }, { "osd": 988, "primary": true, "errors": [ "data_digest_mismatch_info" ], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x224d6ca6" } ] } ] } [0]: https://tracker.ceph.com/issues/24994

1 0

Re: ceph-users Digest, Vol 81, Issue 28
by renjianxinlover 15 Oct '19

15 Oct '19

hi, if needing to stat size of all root dirs in cephfs file system, is there any simple way to do that via ceph system tools? thanks | | renjianxinlover | | renjianxinlover(a)163.com | 签名由网易邮箱大师定制 On 10/15/2019 04:57，<ceph-users-request(a)ceph.io> wrote： Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Past_interval start interval mismatch (last_clean_epoch reported) (Huseyin Cotuk) 2. Re: Constant write load on 4 node ceph cluster (Ingo Schmidt) 3. Re: Constant write load on 4 node ceph cluster (Paul Emmerich) 4. RGW blocking on large objects (Robert LeBlanc) 5. Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects (Florian Haas) 6. Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects (Reed Dier) ---------------------------------------------------------------------- Date: Mon, 14 Oct 2019 18:41:42 +0300 From: Huseyin Cotuk <hcotuk(a)gmail.com> Subject: [ceph-users] Past_interval start interval mismatch (last_clean_epoch reported) To: ceph-users(a)ceph.io Message-ID: <0DB35170-05C5-4290-B4E2-9CB2C2BB3DA4(a)gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D" --Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi all, I also hit the bug #24866 in my test environment. According to the logs, = the last_clean_epoch in the specified OSD/PG is 17703, but the interval = starts with 17895. So the OSD fails to start. There are some other OSDs = in the same status.=20 2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: 18432 pg[18.51( = v 18388'4 lc 18386'3 (0'0,18388'4] local-lis/les=3D18430/18431 n=3D1 = ec=3D295/295 lis/c 18430/17702 les/c/f 18431/17703/0 18428/18430/18421) = [11,21]/[11,21,20] r=3D1 lpr=3D18431 pi=3D[17895,18430)/3 crt=3D18388'4 = lcod 0'0 unknown m=3D1 mbc=3D{}] 18.51 past_intervals [17895,18430) = start interval does not contain the required bound [17703,18430) start The cause is pg 18.51 went clean in 17703 but 17895 is reported to the = monitor.=20 I am using the last stable version of Mimic (13.2.6). Any idea how to fix it? Is there any way to bypass this check or fix the = reported epoch #? Thanks in advance.=20 Best regards, Huseyin Cotuk hcotuk(a)gmail.com --Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Hi = all,<div class=3D""><br class=3D""></div><div class=3D"">I also hit the = bug #24866 in my test environment. According to the logs, the = last_clean_epoch in the specified OSD/PG is 17703, but the interval = starts with 17895. So the OSD fails to start. There are some other OSDs = in the same status. </div><div class=3D""><br class=3D""></div><span = class=3D"">2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: = 18432 pg[18.51( v 18388'4 lc 18386'3 (0'0,18388'4] = local-lis/les=3D18430/18431 n=3D1 ec=3D295/295 lis/c 18430/17702 les/c/f = 18431/17703/0 18428/18430/18421) [11,21]/[11,21,20] r=3D1 lpr=3D18431= pi=3D[17895,18430)/3 crt=3D18388'4 lcod 0'0 unknown m=3D1 mbc=3D{}] = 18.51 past_intervals [17895,18430) start interval does not contain the = required bound [17703,18430) start<br class=3D""></span><span = class=3D""><br class=3D""></span><span class=3D"">The cause is pg 18.51 = went clean in 17703 but 17895 is reported to the = monitor. </span><div class=3D""><span class=3D""><br = class=3D""></span></div><div class=3D""><span class=3D"">I am using the = last stable version of Mimic (13.2.6).</span></div><div class=3D""><span = class=3D""><br class=3D""></span></div><div class=3D""><span = class=3D"">Any idea how to fix it? Is there any way to bypass this check = or fix the reported epoch #?</span></div><div class=3D""><br = class=3D""></div><div class=3D"">Thanks in advance. </div><div = class=3D""><br class=3D""><div class=3D""><div class=3D""> <div dir=3D"auto" style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; line-break: after-white-space;" class=3D""><div = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;">Best regards,</div><div style=3D"caret-color: rgb(0, 0, 0); = color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;">Huseyin = Cotuk</div><div style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><a href=3D"mailto:hcotuk@gmail.com" = class=3D"">hcotuk(a)gmail.com</a></div><div style=3D"caret-color: rgb(0, = 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><br = class=3D""></div></div></div></div></div></body></html>= --Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D-- ------------------------------ Date: Mon, 14 Oct 2019 18:34:17 +0200 (CEST) From: Ingo Schmidt <i.schmidt(a)langeoog.de> Subject: [ceph-users] Re: Constant write load on 4 node ceph cluster To: Ashley Merrick <singapore(a)amerrick.co.uk> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <1013769429.74266.1571070857785.JavaMail.zimbra(a)langeoog.de> Content-Type: text/plain; charset=utf-8 Great, this helped a lot. Although "ceph iostat" didn't give iostats of single images, but just general overview of IO, i remembered the new nautilus RDB performance monitoring. https://ceph.com/rbd/new-in-nautilus-rbd-performance-monitoring/ With a "simple" rbd perf image iotop i was able to see that the writes indeed are from the Log Server and the Zabbix Monitoring Server. I didn't expect that it would cause that much I/O... unbelieveable... ----- Ursprüngliche Mail ----- Von: "Ashley Merrick" <singapore(a)amerrick.co.uk> An: "i schmidt" <i.schmidt(a)langeoog.de> CC: "ceph-users" <ceph-users(a)ceph.io> Gesendet: Montag, 14. Oktober 2019 15:20:46 Betreff: Re: [ceph-users] Constant write load on 4 node ceph cluster Is the storage being used for the whole VM disk? If so have you checked none of your software is writing constant log's? Or something that could continuously write to disk. If your running a new version you can use : [ https://docs.ceph.com/docs/mimic/mgr/iostat/ | https://docs.ceph.com/docs/mimic/mgr/iostat/ ] to locate the exact RBD image. ------------------------------ Date: Mon, 14 Oct 2019 21:14:29 +0200 From: Paul Emmerich <paul.emmerich(a)croit.io> Subject: [ceph-users] Re: Constant write load on 4 node ceph cluster To: Ingo Schmidt <i.schmidt(a)langeoog.de> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <CAD9yTbE5UapmWXpj56yckWW2X7S6wyHz4PQSXLup6H_pzFm1vA(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" It's pretty common to see way more writes than reads if you got lots of idle VMs Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 6:34 PM Ingo Schmidt <i.schmidt(a)langeoog.de> wrote: Great, this helped a lot. Although "ceph iostat" didn't give iostats of single images, but just general overview of IO, i remembered the new nautilus RDB performance monitoring. https://ceph.com/rbd/new-in-nautilus-rbd-performance-monitoring/ With a "simple" rbd perf image iotop i was able to see that the writes indeed are from the Log Server and the Zabbix Monitoring Server. I didn't expect that it would cause that much I/O... unbelieveable... ----- Ursprüngliche Mail ----- Von: "Ashley Merrick" <singapore(a)amerrick.co.uk> An: "i schmidt" <i.schmidt(a)langeoog.de> CC: "ceph-users" <ceph-users(a)ceph.io> Gesendet: Montag, 14. Oktober 2019 15:20:46 Betreff: Re: [ceph-users] Constant write load on 4 node ceph cluster Is the storage being used for the whole VM disk? If so have you checked none of your software is writing constant log's? Or something that could continuously write to disk. If your running a new version you can use : [ https://docs.ceph.com/docs/mimic/mgr/iostat/ | https://docs.ceph.com/docs/mimic/mgr/iostat/ ] to locate the exact RBD image. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------ Date: Mon, 14 Oct 2019 12:54:05 -0700 From: Robert LeBlanc <robert(a)leblancnet.us> Subject: [ceph-users] RGW blocking on large objects To: ceph-users <ceph-users(a)ceph.io> Message-ID: <CAANLjFpWch8_mnHDeph4-=cdY1N1CXA2UJYTg7uE2w718HJ42w(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" We set up a new Nautilus cluster and only have RGW on it. While we had a job doing 200k IOPs of really small objects, I noticed that HAProxy was kicking out RGW backends because they were taking more than 2 seconds to return. We GET a large ~4GB file each minute and use that as a health check to determine if the system is taking too long to service requests. It seems that other IO is being blocked by this large transfer. This seems to be the case with both civetweb and beast. But I'm double checking beast at the moment because I'm not 100% sure we were using it at the start. Any ideas how to mitigate this? It seems that IOs are being scheduled on a thread and if they get unlucky enough to be scheduled behind a big IO, they are just stuck, in this case HAProxy could kick out the backend before the IO is returned and it has to re-request it. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 ------------------------------ Date: Mon, 14 Oct 2019 21:55:53 +0200 From: Florian Haas <florian(a)citynetwork.eu> Subject: [ceph-users] Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects To: Dan van der Ster <dan(a)vanderster.com> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <7b49695d-be1c-b36b-2ced-1b9cb0212530(a)citynetwork.eu> Content-Type: text/plain; charset=utf-8 On 14/10/2019 17:21, Dan van der Ster wrote: I'd appreciate a link to more information if you have one, but a PG autoscaling problem wouldn't really match with the issue already appearing in pre-Nautilus releases. :) https://github.com/ceph/ceph/pull/30479 Thanks! But no, this doesn't look like a likely culprit, for the reason that we also saw this in Luminous and hence, *definitely* without splits or merges in play. Has anyone else seen these scrub false positives — if that's what they are? Cheers, Florian ------------------------------ Date: Mon, 14 Oct 2019 15:57:05 -0500 From: Reed Dier <reed.dier(a)focusvq.com> Subject: [ceph-users] Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects To: Florian Haas <florian(a)citynetwork.eu> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <24D2F909-3015-4256-BAA3-0E56966BF778(a)focusvq.com> Content-Type: multipart/signed; boundary="Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB"; protocol="application/pkcs7-signature"; micalg=sha-256 --Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB Content-Type: multipart/alternative; boundary="Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC" --Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I had something slightly similar to you. However, my issue was specific/limited to the device_health_metrics pool = that is auto-created with 1 PG when you turn that mgr feature on. https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315.html = <https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315.html> I was never able to get a good resolution, other than finally running pg = repair on it, and it resolved itself. Reed On Oct 14, 2019, at 2:55 PM, Florian Haas <florian(a)citynetwork.eu> = wrote: =20 On 14/10/2019 17:21, Dan van der Ster wrote: I'd appreciate a link to more information if you have one, but a PG autoscaling problem wouldn't really match with the issue already appearing in pre-Nautilus releases. :) =20 https://github.com/ceph/ceph/pull/30479 =20 Thanks! But no, this doesn't look like a likely culprit, for the = reason that we also saw this in Luminous and hence, *definitely* without = splits or merges in play. =20 Has anyone else seen these scrub false positives =E2=80=94 if that's = what they are? =20 Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io --Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">I = had something slightly similar to you.<div class=3D""><br = class=3D""></div><div class=3D"">However, my issue was specific/limited = to the device_health_metrics pool that is auto-created with 1 PG = when you turn that mgr feature on.</div><div class=3D""><br = class=3D""></div><div class=3D""><a = href=3D"https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315.ht= ml" = class=3D"">https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315= .html</a></div><div class=3D""><br class=3D""></div><div class=3D"">I = was never able to get a good resolution, other than finally running pg = repair on it, and it resolved itself.</div><div class=3D""><br = class=3D""></div><div class=3D"">Reed<br class=3D""><div = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Oct 14, 2019, at 2:55 PM, Florian Haas <<a = href=3D"mailto:florian@citynetwork.eu" = class=3D"">florian(a)citynetwork.eu</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div class=3D"">On = 14/10/2019 17:21, Dan van der Ster wrote:<br class=3D""><blockquote = type=3D"cite" class=3D""><blockquote type=3D"cite" class=3D"">I'd = appreciate a link to more information if you have one, but a PG<br = class=3D"">autoscaling problem wouldn't really match with the issue = already<br class=3D"">appearing in pre-Nautilus releases. :)<br = class=3D""></blockquote><br class=3D""><a = href=3D"https://github.com/ceph/ceph/pull/30479" = class=3D"">https://github.com/ceph/ceph/pull/30479</a><br = class=3D""></blockquote><br class=3D"">Thanks! But no, this doesn't look = like a likely culprit, for the reason<br class=3D"">that we also saw = this in Luminous and hence, *definitely* without splits<br class=3D"">or = merges in play.<br class=3D""><br class=3D"">Has anyone else seen these = scrub false positives =E2=80=94 if that's what they are?<br class=3D""><br= class=3D"">Cheers,<br class=3D"">Florian<br = class=3D"">_______________________________________________<br = class=3D"">ceph-users mailing list -- <a = href=3D"mailto:ceph-users@ceph.io" class=3D"">ceph-users(a)ceph.io</a><br = class=3D"">To unsubscribe send an email to <a = href=3D"mailto:ceph-users-leave@ceph.io" = class=3D"">ceph-users-leave(a)ceph.io</a><br = class=3D""></div></div></blockquote></div><br = class=3D""></div></div></body></html>= --Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC-- --Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCDGow ggYQMIID+KADAgECAhBNlCwQ1DvglAnFgS06KwZPMA0GCSqGSIb3DQEBDAUAMIGIMQswCQYDVQQG EwJVUzETMBEGA1UECBMKTmV3IEplcnNleTEUMBIGA1UEBxMLSmVyc2V5IENpdHkxHjAcBgNVBAoT FVRoZSBVU0VSVFJVU1QgTmV0d29yazEuMCwGA1UEAxMlVVNFUlRydXN0IFJTQSBDZXJ0aWZpY2F0 aW9uIEF1dGhvcml0eTAeFw0xODExMDIwMDAwMDBaFw0zMDEyMzEyMzU5NTlaMIGWMQswCQYDVQQG EwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRgwFgYD VQQKEw9TZWN0aWdvIExpbWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNBIENsaWVudCBBdXRoZW50 aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEAyjztlApB/975Rrno1jvm2pK/KxBOqhq8gr2+JhwpKirSzZxQgT9tlC7zl6hn1fXjSo5MqXUf ItMltrMaXqcESJuK8dtK56NCSrq4iDKaKq9NxOXFmqXX2zN8HHGjQ2b2Xv0v1L5Nk1MQPKA19xeW QcpGEGFUUd0kN+oHox+L9aV1rjfNiCj3bJk6kJaOPabPi2503nn/ITX5e8WfPnGw4VuZ79Khj1YB rf24k5Ee1sLTHsLtpiK9OjG4iQRBdq6Z/TlVx/hGAez5h36bBJMxqdHLpdwIUkTqT8se3ed0PewD ch/8kHPo5fZl5u1B0ecpq/sDN/5sCG52Ds+QU5O5EwIDAQABo4IBZDCCAWAwHwYDVR0jBBgwFoAU U3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFAnA8vwL2pTbX/4r36iZQs/J4K0AMA4GA1Ud DwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF BQcDBDARBgNVHSAECjAIMAYGBFUdIAAwUAYDVR0fBEkwRzBFoEOgQYY/aHR0cDovL2NybC51c2Vy dHJ1c3QuY29tL1VTRVJUcnVzdFJTQUNlcnRpZmljYXRpb25BdXRob3JpdHkuY3JsMHYGCCsGAQUF BwEBBGowaDA/BggrBgEFBQcwAoYzaHR0cDovL2NydC51c2VydHJ1c3QuY29tL1VTRVJUcnVzdFJT QUFkZFRydXN0Q0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1c3QuY29tMA0G CSqGSIb3DQEBDAUAA4ICAQBBRHUAqznCFfXejpVtMnFojADdF9d6HBA4kMjjsb0XMZHztuOCtKF+ xswhh2GqkW5JQrM8zVlU+A2VP72Ky2nlRA1GwmIPgou74TZ/XTarHG8zdMSgaDrkVYzz1g3nIVO9 IHk96VwsacIvBF8JfqIs+8aWH2PfSUrNxP6Ys7U0sZYx4rXD6+cqFq/ZW5BUfClN/rhk2ddQXyn7 kkmka2RQb9d90nmNHdgKrwfQ49mQ2hWQNDkJJIXwKjYA6VUR/fZUFeCUisdDe/0ABLTI+jheXUV1 eoYV7lNwNBKpeHdNuO6Aacb533JlfeUHxvBz9OfYWUiXu09sMAviM11Q0DuMZ5760CdO2VnpsXP4 KxaYIhvqPqUMWqRdWyn7crItNkZeroXaecG03i3mM7dkiPaCkgocBg0EBYsbZDZ8bsG3a08LwEsL 1Ygz3SBsyECa0waq4hOf/Z85F2w2ZpXfP+w8q4ifwO90SGZZV+HR/Jh6rEaVPDRF/CEGVqR1hiuQ OZ1YL5ezMTX0ZSLwrymUE0pwi/KDaiYB15uswgeIAcA6JzPFf9pLkAFFWs1QNyN++niFhsM47qod x/PL+5jR87myx5uYdBEQkkDc+lKB1Wct6ucXqm2EmsaQ0M95QjTmy+rDWjkDYdw3Ms6mSWE3Bn7i 5ZgtwCLXgAIe5W8mybM2JzCCBlIwggU6oAMCAQICEC8JVM/NtNco+9fMvYhQtd8wDQYJKoZIhvcN AQELBQAwgZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNV BAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBS U0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTkwOTI2MDAw MDAwWhcNMjIwOTI1MjM1OTU5WjCCAV8xCzAJBgNVBAYTAlVTMQ4wDAYDVQQREwU3MDAwNTESMBAG A1UECBMJTG91aXNpYW5hMREwDwYDVQQHEwhNZXRhaXJpZTEsMCoGA1UECRMjU3RlIDI1MCBUaGUg SGVyaXRhZ2UgUGxhemEgQnVpbGRpbmcxIzAhBgNVBAkTGjExMSBWZXRlcmFucyBNZW1vcmlhbCBC bHZkMSYwJAYDVQQKEx1Gb2N1cyBBdXRvbWF0ZWQgRXF1aXRpZXMsIExMQzFDMEEGA1UECxM6SXNz dWVkIHRocm91Z2ggRm9jdXMgQXV0b21hdGVkIEVxdWl0aWVzLCBMTEMgRS1QS0kgTWFuYWdlcjEf MB0GA1UECxMWQ29ycG9yYXRlIFNlY3VyZSBFbWFpbDESMBAGA1UEAxMJUmVlZCBEaWVyMSQwIgYJ KoZIhvcNAQkBFhVyZWVkLmRpZXJAZm9jdXN2cS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQC8ELOuFrZlztYzJp/iLLqvEc1ugxFGOkqZJsfk9XmgYc+8y48P8SNzRvWQeVs84RmN HeLg3xnAtt7xdK0cjM2PdOqznYmb2wAV6deaodu4Kun+im+Rp4/7pjvkyaiMjEL0/EA+WM8bFgjQ OKy6/T+8W9PpaJkxtdgTSXTsYWKfkZYZcJfy/aOQgtAwBflehQ8pazCo4Qx8cOG7i58ZUoBEB46v EdrxOaFzrGeeO+rx2siq9dByT7rkR4YEnKrHfXNgAnDIjsz5Q8Qq9cqH7tbt20MUU+X4D8GQ0fT+ FR8nSQNoCpKSk1rTboOza1qKJyIaBGEwKfuq5cd+KzAIwEY1AgMBAAGjggHOMIIByjAfBgNVHSME GDAWgBQJwPL8C9qU21/+K9+omULPyeCtADAdBgNVHQ4EFgQUqHWIYbt2tNSGAjge5B0wOrOk5fsw DgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUF BwMCMEAGA1UdIAQ5MDcwNQYMKwYBBAGyMQECAQEBMCUwIwYIKwYBBQUHAgEWF2h0dHBzOi8vc2Vj dGlnby5jb20vQ1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuc2VjdGlnby5jb20vU2Vj dGlnb1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYoGCCsGAQUF BwEBBH4wfDBVBggrBgEFBQcwAoZJaHR0cDovL2NydC5zZWN0aWdvLmNvbS9TZWN0aWdvUlNBQ2xp ZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAjBggrBgEFBQcwAYYXaHR0cDov L29jc3Auc2VjdGlnby5jb20wIAYDVR0RBBkwF4EVcmVlZC5kaWVyQGZvY3VzdnEuY29tMA0GCSqG SIb3DQEBCwUAA4IBAQCmjIyvT1SaifV+zEYRcBBfkJppf3IV3Hu0C1CN29RNjdslsmk2YYHUGBya 03Ae5oSAKLOt1vZLaLLZeaYNdx9Kx7Oqux+S9ozg9Vg86sSCJroIICqWHzqjo4DoOIg1nFkVMMpM kWpjNxFdUSIZLYvKyWWGFTYsMmF6rp7RNO6pJ4AtNn88lNXhkT/40ebvJ85RBoy73W6LX21vtxsh e83r6jjzHNPe33kTwWzT39YoqHoCdbQwW3jfKFQUs/6a6HB1o1el7aazOMZIDjl2poHUEoKBPJWR pk8zb1JKD4WCNaDg+sQJSMO7ekcjpJeoBEzlfjI2dT4lHlhtizWgypVuMYIDxDCCA8ACAQEwgasw gZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1Nh bGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xp ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEC8JVM/NtNco+9fMvYhQtd8w DQYJYIZIAWUDBAIBBQCgggHpMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkF MQ8XDTE5MTAxNDIwNTcwNlowLwYJKoZIhvcNAQkEMSIEIJqv7OvtXJqA52V0p7Z1lyr0AxHrxFB3 ab4K0PWgfEF+MIG8BgkrBgEEAYI3EAQxga4wgaswgZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJH cmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGlt aXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1 cmUgRW1haWwgQ0ECEC8JVM/NtNco+9fMvYhQtd8wgb4GCyqGSIb3DQEJEAILMYGuoIGrMIGWMQsw CQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3Jk MRgwFgYDVQQKEw9TZWN0aWdvIExpbWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNBIENsaWVudCBB dXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhAvCVTPzbTXKPvXzL2IULXfMA0GCSqG SIb3DQEBAQUABIIBACWiX8zkYNxrRSXTsKV0iFrjJYo5xqGItFrbhyrEuJ1zDVCxHDYGoT6PV4OE pBPN6kpGbktyrGfH+Kt3YfcGng0oTOwRBiixlplAEvNSWJeAT/aHv0ta7Z6D+sQhlQh0/qsak9hk 1ycvd25X9wf7nqPb5Rlc9VDh7HUGlc/l2EOdlMbFjmdm0GvljtPzng/288bMtIGHEwltxLu654E1 9Q7l1bUE10RmUftILZTLIgDMGu0er6OuB18ZV4zIjsMiK1SXr/DIuk5lrbVJjydx5LnPKU/7PEUz g14k9x0/DZiepnzhMBiTJjARn3N+scKEK4QtlskpKoWh8cnvovTGpCsAAAAAAAA= --Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB-- ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 81, Issue 28 ******************************************

1 0

Ceph health status reports: Reduced data availability and this is resulting in slow requests are blocked
by Thomas 15 Oct '19

15 Oct '19

Hi, I want to use balancer mode "upmap" for all pools. This mode is currently enable for pool "hdb_backup" with ~600TB used space. root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B cephfs_data 1.1 TiB 89592 0 268776 0 0 0 0 0 B 43443 144 GiB 0 B 0 B cephfs_metadata 311 MiB 48 0 144 0 0 0 6 6 KiB 7465 106 MiB 0 B 0 B hdb_backup 585 TiB 51077985 0 153233955 0 0 0 12577024 4.3 TiB 281002173 523 TiB 0 B 0 B hdd 6.3 TiB 585051 0 1755153 0 0 0 4420255 69 GiB 8219453 1.2 TiB 0 B 0 B root@ld3955:~# ceph osd lspools 11 hdb_backup 51 hdd 52 backup 57 cephfs_data 58 cephfs_metadata I started with pool "cephfs_metadata" that allocated comparable low data. At the same moment when I executed ceph config set mgr mgr/balancer/pool_ids 11,52,58 Ceph status shows increasing number of Reduced data availability: <x> pg inactive, <y> pg peering And the number of <z> slow requests are blocked > 32 sec was increasing heavily up to 180. Observing Ceph log (ceph -w) it becomes clear that there's a correlation between pg inactive and slow requests are block in my cluster. How can I start analysis why the cluster reports slow requests? THX

1 0

Ceph health status reports: subtrees have overcommitted pool target_size_ratio + subtrees have overcommitted pool target_size_bytes
by Thomas 15 Oct '19

15 Oct '19

Hi, checking my Ceph health status I get this warning: 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio The details are as follows: Pools ['hdb_backup'] overcommit available storage by 1.288x due to target_size_bytes 0 on pools [] However, the relevant settings POOL_TARGET_SIZE_RATIO_OVERCOMMITTED according to health checks <https://docs.ceph.com/docs/master/rados/operations/health-checks/> are not enabled: root@ld3955:~# ceph osd pool get hdb_backup target_size_ratio Error ENOENT: option 'target_size_ratio' is not set on pool 'hdb_backup' root@ld3955:~# ceph osd pool get hdb_backup target_size_bytes Error ENOENT: option 'target_size_bytes' is not set on pool 'hdb_backup' Therefore I would consider this warning as a bug. Regards Thomas

1 0

Constant write load on 4 node ceph cluster
by Ingo Schmidt 14 Oct '19

14 Oct '19

Hi all We have a 4 node ceph cluster that runs generally fine. It is the storage backend for our virtualization cluster with Proxmox, that runs about 40 virtual machines (80% various Linux Servers). Now that we have implemented monitoring, i see that there is a quite constant write load of about 4-10MB/s throughout the whole day and even weekends, while read load clearly is bound to daily work hours. I am wondering whats up with this. Is this normal? Outside working hours, there is almost no read load, well under 1MB/s, while write Load stays relatively constant at around 5MB/s. During rebalancing of the Cluster, normal write load decreases significantly to under 1MB/s. Overall performance is not greatly affected by rebalancing as it is tuned to relatively slow backfills (osd_max_backfills = '3' | osd_recovery_max_active = '3') After rebalance has been finished, the average write load of about 5MB/s reappears. I have attached a little screenshot for reference. Can someone shed a little light on this? Thanks. kind regards Ingo Schmidt IT-Department Island Municipality Langeoog with in-house operations Tourismusservice and Schiffahrt Hauptstraße 28 26465 Langeoog Deutschland [ http://www.langeoog.de/ ]

3 3

object goes missing in bucket
by Benjamin.Zieglmeier 14 Oct '19

14 Oct '19

Hey all, Experiencing an odd issue over the last week or so with a single bucket in a Ceph Luminous (12.2.11) cluster. We occasionally get a complaint from the owner of one bucket (bucket1) that a single object they have written has gone missing. If we list the bucket, the object is indeed missing, and is not present if we look for it using `radosgw-admin bi list –bucket=bucket1`. If we dump the pool the object is placed using (rados -p <poolname> ls ) we can see the object is present in the pool. Looking in the logs we are having a hard time finding any errors, and if we look at the rgw logs we can see the PUT, followed by a couple GETs and HEADs (we believe this to be client app verification) but never a DELETE. This incident has happened a couple of times over the past week, but only to this bucket. The bucket in question does average anywhere from 100-200ops per second based on the rgw usage logs. Any assistance in knowing where to look to see why these objects might be disappearing from the index would be greatly appreciated! -Ben

1 0

CephFS and 32-bit inode numbers
by Dan van der Ster 14 Oct '19

14 Oct '19

Hi all, One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3. I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3. Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?) Thanks! Dan

1 0

RadosGW max worker threads
by Benjamin.Zieglmeier 12 Oct '19

12 Oct '19

Hello all, Looking for guidance on the recommended highest setting (or input on experiences from users who have a high setting) for rgw_thread_pool_size. We are running multiple Luminous 12.2.11 clusters with usually 3-4 RGW daemons in front of them. We set our rgw_thread_pool_size at 512 out of the gate, and run civetweb. We had occasional service outages in one of our clusters this week and determined the rgws were running out of available threads to handle requests. We doubled our thread pool size to 1024 on each rgw and everything has been ok so far. What, if any, would be the high-end limit to set for rgw_thread_pool_size? I’ve been unable to find anything in the documentation or the user list that depicts anything higher than the default 100 threads. Thanks, Ben

4 5

Nautilus: PGs stuck remapped+backfilling
by Eugen Block 11 Oct '19

11 Oct '19

Hi all, I have a strange issue with backfilling and I'm not sure what the cause is. It's a Nautilus cluster (upgraded) that has an SSD cache tier for OpenStack and CephFS metadata residing on the same SSDs, there were three SSDs in total. Today I added two new SSDs (NVMe) (osd.15, osd.16) to be able to shutoff one old server that has only one SSD-OSD left (osd.20). Setting the crush weight of osd.20 to 0 (and adjusting the weight of the remaining SSDs for an even distribution) leaves 3 PGs in active+remapped+backfilling state. I don't understand why the remaining PGs aren't backfilling, the crush rule is quite simple (all ssd pools are replicated with size 3). The backfilling PGs are all from the cephfs_metadata pool. Although there are 4 SSDs for 3 replicas the backfilling still should finish, right? Can anyone share their thoughts why 3 PGs can't be recovered? If more information about the cluster is required please let me know. Regards, Eugen ceph01:~ # ceph osd pool ls detail | grep meta pool 36 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 283362 flags hashpspool,nodelete,nodeep-scrub stripe_width 0 application cephfs ceph01:~ # ceph pg dump | grep remapp dumped all 36.b 28306 0 0 28910 0 8388608 101408323 219497 3078 3078 active+remapped+backfilling 2019-10-10 13:36:27.427527 284595'98565869 284595:254216941 [15,16,9] 15 [20,9,10] 20 284427'98489406 2019-10-10 00:16:02.682911 284089'98003598 2019-10-06 16:03:27.558267 0 36.d 28087 0 0 25327 0 26375382 106722204 231020 3041 3041 active+remapped+backfilling 2019-10-10 13:36:27.404739 284595'97933905 284595:252878816 [16,15,9] 16 [20,9,10] 20 284427'97887652 2019-10-10 04:13:29.371905 284259'97502135 2019-10-07 20:06:43.304593 0 36.4 28060 0 0 28406 0 8389242 104059103 225188 3061 3061 active+remapped+backfilling 2019-10-10 13:36:27.440390 284595'105299618 284595:312976619 [16,9,15] 16 [20,9,10] 20 284427'105218591 2019-10-10 00:18:07.924006 284089'104696098 2019-10-06 16:20:17.123149 0 rule ssd_ruleset { id 1 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 0 type host step emit } This is the relevant part of the osd tree: ceph01:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 34.21628 root default -31 11.25406 host ceph01 25 hdd 3.59999 osd.25 up 1.00000 1.00000 26 hdd 3.59999 osd.26 up 1.00000 1.00000 27 hdd 3.59999 osd.27 up 1.00000 1.00000 15 ssd 0.45409 osd.15 up 1.00000 1.00000 -34 11.25406 host ceph02 0 hdd 3.59999 osd.0 up 1.00000 1.00000 28 hdd 3.59999 osd.28 up 1.00000 1.00000 29 hdd 3.59999 osd.29 up 1.00000 1.00000 16 ssd 0.45409 osd.16 up 1.00000 1.00000 -37 10.79999 host ceph03 31 hdd 3.59999 osd.31 up 1.00000 1.00000 32 hdd 3.59999 osd.32 up 1.00000 1.00000 33 hdd 3.59999 osd.33 up 1.00000 1.00000 -24 0.45409 host san01-ssd 10 ssd 0.45409 osd.10 up 1.00000 1.00000 -23 0.45409 host san02-ssd 9 ssd 0.45409 osd.9 up 1.00000 1.00000 -22 0 host san03-ssd 20 ssd 0 osd.20 up 1.00000 1.00000 Don't be confused because of the '-ssd' suffix, we're using crush location hooks. This is the current PG distribution on the SSDs: ceph01:~ # ceph osd df | grep -E "^15 |^16 |^ 9|^10 |^20 " 15 ssd 0.45409 1.00000 465 GiB 34 GiB 32 GiB 1.2 GiB 857 MiB 431 GiB 7.29 0.22 27 up 16 ssd 0.45409 1.00000 465 GiB 37 GiB 34 GiB 1.5 GiB 964 MiB 428 GiB 7.87 0.23 31 up 10 ssd 0.45409 1.00000 745 GiB 27 GiB 25 GiB 1.7 GiB 950 MiB 718 GiB 3.65 0.11 29 up 9 ssd 0.45409 1.00000 745 GiB 34 GiB 32 GiB 1.3 GiB 902 MiB 711 GiB 4.60 0.14 30 up 20 ssd 0 1.00000 894 GiB 8.2 GiB 4.3 GiB 1.5 GiB 2.4 GiB 886 GiB 0.91 0.03 3 up Current ceph status: ceph01:~ # ceph -s cluster: id: 655cb05a-435a-41ba-83d9-8549f7c36167 health: HEALTH_OK services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 2d) mgr: ceph03(active, since 8d), standbys: ceph01, ceph02 mds: cephfs:1 {0=mds01=up:active} 1 up:standby-replay 1 up:standby osd: 26 osds: 26 up (since 66m), 26 in (since 66m); 3 remapped pgs data: pools: 8 pools, 264 pgs objects: 4.96M objects, 5.0 TiB usage: 16 TiB used, 31 TiB / 47 TiB avail pgs: 115745/14865558 objects misplaced (0.779%) 261 active+clean 3 active+remapped+backfilling io: client: 903 KiB/s rd, 8.8 MiB/s wr, 85 op/s rd, 266 op/s wr recovery: 0 B/s, 61 keys/s, 12 objects/s cache: 4.2 MiB/s flush, 15 MiB/s evict, 0 op/s promote

3 7

help
by Jörg Kastning 11 Oct '19

11 Oct '19

Am 11.10.2019 um 09:21 schrieb ceph-users-request(a)ceph.io: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: MDS rejects clients causing hanging mountpoint on linux kernel client > (Manuel Riel) > 2. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout") > (Janne Johansson) > 3. Re: Nautilus: PGs stuck remapped+backfilling (Eugen Block) > 4. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout") (潘东元) > 5. Nautilus power outage - 2/3 mons and mgrs dead and no cephfs > (Alex L) > > > ---------------------------------------------------------------------- > > Date: Thu, 10 Oct 2019 11:02:28 +0800 > From: Manuel Riel <manu(a)snapdragon.cc> > Subject: [ceph-users] Re: MDS rejects clients causing hanging > mountpoint on linux kernel client > To: ukernel(a)gmail.com > Cc: ceph-users(a)ceph.io > Message-ID: <3098DC39-AED4-44F0-B9CB-44B3468285BB(a)snapdragon.cc> > Content-Type: multipart/signed; > boundary="Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D"; > protocol="application/pkcs7-signature"; micalg=sha-256 > > > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D > Content-Transfer-Encoding: quoted-printable > Content-Type: text/plain; > charset=us-ascii > > I noticed a similar issue tonight. Still looking into the details, but = > here are the client logs I=20 > > Oct 9 19:27:59 mon5-cx kernel: libceph: mds0 ***:6800 socket closed = > (con state OPEN) > Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 connection reset > Oct 9 19:28:01 mon5-cx kernel: libceph: reset on mds0 > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 closed our session > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect start > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect denied > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109011c9980 1099517142146 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91096cc788d0 1099517142307 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9107da741f10 1099517142312 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109d5c40e60 1099517141612 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9108c9337da0 1099517142313 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109d5c70340 1099517141565 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff910955acf810 1099517141792 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91095ff56cf0 1099517142006 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91096cc7f280 1099517142309 > Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 socket closed = > (con state NEGOTIATING) > Oct 9 19:28:02 mon5-cx kernel: ceph: mds0 rejected session > Oct 9 19:28:02 mon5-cx monit: Lookup for '/srv/repos' filesystem failed = > -- not found in /proc/self/mounts > Oct 9 19:28:02 mon5-cx monit: Filesystem '/srv/repos' not mounted > Oct 9 19:28:02 mon5-cx monit: 'repos' unable to read filesystem = > '/srv/repos' state > ... > Oct 9 19:28:09 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 19:28:24 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 19:28:39 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > ... > Oct 9 21:27:09 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 21:27:24 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 21:27:27 mon5-cx monit: Lookup for '/srv/repos' filesystem failed = > -- not found in /proc/self/mounts > Oct 9 21:27:27 mon5-cx monit: Filesystem '/srv/repos' not mounted > Oct 9 21:27:27 mon5-cx monit: 'repos' unable to read filesystem = > '/srv/repos' state > Oct 9 21:27:27 mon5-cx monit: 'repos' trying to restart > > > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D > Content-Disposition: attachment; > filename=smime.p7s > Content-Type: application/pkcs7-signature; > name=smime.p7s > Content-Transfer-Encoding: base64 > > MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCCx4w > ggUwMIIEGKADAgECAhEAuxgm/OfWx7qPyHlON6sLpzANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE > BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG > A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl > bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgxMDMxMDAwMDAwWhcNMTkxMDMxMjM1 > OTU5WjAjMSEwHwYJKoZIhvcNAQkBFhJtYW51QHNuYXBkcmFnb24uY2MwggEiMA0GCSqGSIb3DQEB > AQUAA4IBDwAwggEKAoIBAQDSCVHH0d9wqo1reA0PKmK8x88NJQcffMt3EBYnqrIWt0rLypsDWYBE > uTUl+WAppxa2BI172rAexkkVX1J4bM8S5P5NRInp1JOlo+s/WdXM1f0IGD4qfCUtOBM+05rdvKMS > +PBAdBkAR4XnDe26fk7XNgBLbhVifw5TEn3UBaQWPSmr6OV7WwoysNTyR3X1tKzVHTCrJv9JWfSA > vlU2NIpo5XOa9X1Zs7uxlwO7LRgsC50AFrLHVYfsqIl9zr5iBlP2lBsZVciqhv0/tszThLBjS3di > fbSyJVzAhBQTf2KCvewJbkcZr7JE8vV4sXUqrvc1voyN3sf0G/qSTy0rCl8vAgMBAAGjggHoMIIB > 5DAfBgNVHSMEGDAWgBSCr2yM+MX+lmF86B89K3FIXsSLwDAdBgNVHQ4EFgQUMzEd3LCtdsH3VrT1 > WB6sUaWGU6UwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwIAYDVR0lBBkwFwYIKwYBBQUH > AwQGCysGAQQBsjEBAwUCMBEGCWCGSAGG+EIBAQQEAwIFIDBGBgNVHSAEPzA9MDsGDCsGAQQBsjEB > AgEBATArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBaBgNVHR8E > UzBRME+gTaBLhklodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9SU0FDbGllbnRBdXRoZW50 > aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3JsMIGLBggrBgEFBQcBAQR/MH0wVQYIKwYBBQUHMAKG > SWh0dHA6Ly9jcnQuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5k > U2VjdXJlRW1haWxDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAd > BgNVHREEFjAUgRJtYW51QHNuYXBkcmFnb24uY2MwDQYJKoZIhvcNAQELBQADggEBACmWfZZiKLiT > s7YxuZc2KvsmjlDqIUK2TF6b1LqA4CMFiDwwNDUvxoOClhZJFpDDfnw4w86fDheIM5fPinf+0wif > 8YAphESKyVIZPDl7t3D4zQmU5Ytsq44iCS+TTAQXcWbz7Qe6tBQWZvffKNb6H95blJj0t86Ga0wD > huUXMZbjFMOYcRLGsllUe5H2qJUs0CY/Y9MGGXjcy2HFbhStbuUyj8rwi9ynCvz3LFS30J9MXhYr > c+yu9tZiHZZdjl7gP7RCez+IDZme1TfVYu6KBXxrvkZMRU9+KILtcXmIvMwzelp8KC9Ha9AtXZNd > g9wSFvwRAOFtAEJ9+m0fhdzmq1cwggXmMIIDzqADAgECAhBqm+E4O/8ra58B1dm4p1JWMA0GCSqG > SIb3DQEBDAUAMIGFMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw > DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDErMCkGA1UEAxMiQ09N > T0RPIFJTQSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0xMzAxMTAwMDAwMDBaFw0yODAxMDky > MzU5NTlaMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD > VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09NT0RP > IFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZI > hvcNAQEBBQADggEPADCCAQoCggEBAL6znlesKHZ1QBbHOAOY08YYdiFQ8yV5C0y1oNF9Olg+nKcx > Lqf2NHbZhGra0D00SOTq9bus3/mxgUsg/Wh/eXQ0pnp8tZ8XZWAnlyKMpjL+qUByRjXCA6RQyDMq > VaVUkbIr5SU0RDX/kSsKwer3H1pT/HUrBN0X8sKtPTdGX8XAWt/VdMLBrZBlgvnkCos+KQWWCo63 > OTTqRvaq8aWccm+KOMjTcE6s2mj6RkalweyDI7X+7U5lNo6jzC8RTXtVV4/Vwdax720YpMPJQaDa > ElmOupyTf1Qib+cpukNJnQmwygjD8m046DQkLnpXNCAGjuJy1F5NATksUsbfJAr7FLUCAwEAAaOC > ATwwggE4MB8GA1UdIwQYMBaAFLuvfgI9+qbxPISOre44mOzZMjLUMB0GA1UdDgQWBBSCr2yM+MX+ > lmF86B89K3FIXsSLwDAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAE > CjAIMAYGBFUdIAAwTAYDVR0fBEUwQzBBoD+gPYY7aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09N > T0RPUlNBQ2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwcQYIKwYBBQUHAQEEZTBjMDsGCCsGAQUF > BzAChi9odHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FBZGRUcnVzdENBLmNydDAkBggr > BgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQB4XLKB > KDRPPO5fVs6fl1bsj6JrF/bz9kkIBtTYLzXN30D+03Hj6OxCDBEaIeNmsBhrJmuubvyE7HtoSmR8 > 09AgcYboW+rcTNZ/8u/Hv+GTrNI/AhqX2/kiQNxmgUPt/eJPs92Qclj0HnVyy9TnSvGkSDU7I5Px > +TbO+88G4zipA2psZaWeEykgzClZlPz1FjTCkk77ZXp5cQYYexE6zeeN4/0OqqoAloFrjAF4o50Y > JafX8mnahjp3I2Y2mkjhk0xQfhNqbzlLWPoT3m7j7U26u7zg6swjOq8hITYc3/np5tM5aVyu6t99 > p17bTbY7+1RTWBviN9YJzK8HxzObXYWBf/L+VGOYNsQDTxAk0Hbvb1j6KjUhg7fO294F29QIhhmi > NOr84JHoy+fNLpfvYc/Q9EtFOI5ISYgOxLk3nD/whbUe9rmEQXLp8MB933Ij474gwwCPUpwv9mj2 > PMnXoc7mbrS22XUSeTwxCTP9bcmUdp4jmIoWfhQm7X9w/Zgddg+JZ/YnIHOwsGsaTUgj7fIvxqit > h7DoJC91WJ8Lce3CVJqb1XWeKIJ84F7YLXZN0oa7TktYgDdmQVxYkZo1c5noaDKH9Oq9cbm/vOYR > UM1cWcef20Wkyk5S/GFyyPJwG0fR1nRas3DqAf4cXxMiEKcff7PNa4M3RGTqH0pWR8p6EjGCA8ow > ggPGAgEBMIGtMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw > DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09N > T0RPIFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIRALsYJvzn > 1se6j8h5TjerC6cwDQYJYIZIAWUDBAIBBQCgggHtMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw > HAYJKoZIhvcNAQkFMQ8XDTE5MTAxMDAzMDIyOVowLwYJKoZIhvcNAQkEMSIEIOo1zdaTl2T2mCM4 > ttyllcqgdZ0MagVCBglhDdSmEkM8MIG+BgkrBgEEAYI3EAQxgbAwga0wgZcxCzAJBgNVBAYTAkdC > MRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoT > EUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNh > dGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfWx7qPyHlON6sLpzCBwAYLKoZIhvcNAQkQ > AgsxgbCgga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO > BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01P > RE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfW > x7qPyHlON6sLpzANBgkqhkiG9w0BAQEFAASCAQCPNgVk/I2nxe1niT2avKeoKQiQfWhGIsFe6V3V > 8E8WDEGfQwQX0ApcF0sm465wPRuu+K3n3mMo7z2mNZtOyfpHeAYElsZsLOV/pMgMWPNYISwFPu2K > J7m9o4anPff7JT0BTMf6XbsFTezCUCIQotuGur/6KquXOCOddX8xYbhcSAuwlAPuhBzCnn42goU0 > tybdFZqmcjJl+5uRYYHmr68zhcA7XFF5tFTzIaq4krMDMfk0aKpKNTL5wIhfFwChe9FAUgsHT25h > p3bQkDlKZgfeWeqYtzkOoFCqpznWR65EqKaMmGaemWCQuOTMMP+8lVC7HIlxbNmbnEYH3sbdldZJ > AAAAAAAA > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D-- > > ------------------------------ > > Date: Thu, 10 Oct 2019 15:16:46 +0200 > From: Janne Johansson <icepic.dz(a)gmail.com> > Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide > timeout") > To: 潘东元 <dongyuanpan0(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CAA6-MF-SpecCptAd0q_GF0cHSUG4G4W8pZmiXEi1DP0kAwvrHw(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000004ebc3105948e3713" > > --0000000000004ebc3105948e3713 > Content-Type: text/plain; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > Den tors 10 okt. 2019 kl 15:12 skrev =E6=BD=98=E4=B8=9C=E5=85=83 <dongyuanp= > an0(a)gmail.com>: > >> hi all, >> my osd hit suicide timeout. >> >> common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeout") >> >> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) >> > > >> can you give some advice on troubleshooting? >> > > It is a very old release, chances are large whatever issue you get here > might have been fixed in the last 5 years. > > --=20 > May the most significant bit of your life be positive. > > --0000000000004ebc3105948e3713 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= > <div dir=3D"ltr" class=3D"gmail_attr">Den tors 10 okt. 2019 kl 15:12 skrev = > =E6=BD=98=E4=B8=9C=E5=85=83 <<a href=3D"mailto:dongyuanpan0@gmail.com">d= > ongyuanpan0(a)gmail.com</a>>:<br></div><blockquote class=3D"gmail_quote" s= > tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad= > ding-left:1ex">hi all,<br> > =C2=A0 =C2=A0 my osd hit suicide timeout.<br><br> > common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeou= > t")<br> > <br> > =C2=A0ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)<br> > </blockquote><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"ma= > rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:= > 1ex"> > can you give some advice on troubleshooting?<br></blockquote></div><div><br= >> </div><div>It is a very old release, chances are large whatever issue you = > get here might have been fixed in the last 5 years.</div><div><br></div>-- = > <br><div dir=3D"ltr" class=3D"gmail_signature">May the most significant bit= > of your life be positive.<br></div></div> > > --0000000000004ebc3105948e3713-- > > ------------------------------ > > Date: Thu, 10 Oct 2019 13:57:38 +0000 > From: Eugen Block <eblock(a)nde.ag> > Subject: [ceph-users] Re: Nautilus: PGs stuck remapped+backfilling > To: ceph-users(a)ceph.io > Message-ID: > <20191010135738.Horde.-gF9xNAEG9yFgxVbn83rdTU(a)webmail.nde.ag> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > Please ignore my email, the PGs have eventually recovered, it just > took way more time than expected or observed for the other PGs. I'll > try to be more patient next time. ;-) > > ------------------------------ > > Date: Fri, 11 Oct 2019 14:12:49 +0800 > From: 潘东元 <dongyuanpan0(a)gmail.com> > Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide > timeout") > To: Janne Johansson <icepic.dz(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CANkQ9Li-ay1xVUkKqGtM1yq_PbSHr57GXV35aRLvOQtaZwE-xQ(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > I‘m pretty sure, this issue here is that there is a communication > issue between the osds. > logged over and over again report initiating reconnect. > > I looked at my network,and have dropped packets,this is probably the > tcp queue full at osd daemon listen port. > My cluster had 21 nodes, with 5 osds on each node. > > [root@node-4 ~]# netstat -nat | grep -w "6801" | grep -w tcp | grep > ESTABLISHED | wc -l > 271 > > [root@node-4 ~]# sysctl -a | grep somax > net.core.somaxconn = 128 > > this will cause the tcp connection to reset. > > so,i will increase this parameter! > > ------------- > Regards! > > Janne Johansson <icepic.dz(a)gmail.com> 于2019年10月10日周四下午9:16写道： >> >> >> >> Den tors 10 okt. 2019 kl 15:12 skrev 潘东元 <dongyuanpan0(a)gmail.com>: >>> >>> hi all, >>> my osd hit suicide timeout. >>> >>> common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") >>> >>> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) >> >> >>> >>> can you give some advice on troubleshooting? >> >> >> It is a very old release, chances are large whatever issue you get here might have been fixed in the last 5 years. >> >> -- >> May the most significant bit of your life be positive. > > ------------------------------ > > Date: Fri, 11 Oct 2019 07:21:40 -0000 > From: "Alex L" <alexut.voicu(a)gmail.com> > Subject: [ceph-users] Nautilus power outage - 2/3 mons and mgrs dead > and no cephfs > To: ceph-users(a)ceph.io > Message-ID: <157077850022.1193.1225987085352609330@mailman-web> > Content-Type: text/plain; charset="utf-8" > > Hi list, > Had a power outage killing the whole cluster. Cephfs will not start at all, but RBD works just fine. > I did have 4 unfound objects that I eventually had to rollback or delete which I don't really understand as I should've had a copy of the those pbjects on the other drives? > > 2/3 mons and mgrs are damaged but without any errors. > > I have loads stored on cephfs so would very much like to get that running as a first priority. > > Thanks! > Alex > > Info about the home cluster: > I run 23 osds on 3 hosts. 6 of these are a SSD cache layer for the spinning rust, as well as the metadata portion for cephfs which in retrospect might have to be put back on the spinning rust. > > # ceph -v > ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable) > > # head ceph-mgr.pve21.log.7 > 2019-10-04 00:00:00.397 7fee56df3700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 > 2019-10-04 00:00:00.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855f6b7500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:00.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:01.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2680: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 4 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) > 2019-10-04 00:00:01.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e486380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:01.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:02.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e4b5500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:02.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:03.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2681: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 2 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) > 2019-10-04 00:00:03.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e3b0380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > > # head ceph-mon.pve21.log.7 > 2019-10-04 00:00:00.389 7f7c25b52700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 > 2019-10-04 00:00:00.397 7f7c25b52700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 > 2019-10-04 00:00:00.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:00.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:01.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:01.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:02.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:02.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:03.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:03.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > > > # head ceph-mds.pve21.log.7 > 2019-10-04 00:00:00.389 7f1b2f1b5700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 > 2019-10-04 00:00:00.397 7f1b2f1b5700 -1 received signal: Hangup from (PID: 193052) UID: 0 > 2019-10-04 00:00:04.881 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839bb200 0x556f838d4000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby > 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0) > 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=6 cs=4 l=0).handle_message_footer missed message? skipped from seq 0 to 2 > 2019-10-04 00:01:19.167 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:01:23.555 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.22:0/2875552603 conn(0x556f839bda80 0x556f837f9000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:02:08.768 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2427365808 conn(0x556f839bd180 0x556f83e9d800 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:02:20.140 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby > 2019-10-04 00:02:21.420 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839e0480 0x556f83d7f000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0) > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 81, Issue 19 > ****************************************** > -- Jörg Kastning Plattformen und Serverdienste BITS - Bielefelder IT-Servicezentrum Universität Bielefeld Universitätsstraße 25 33615 Bielefeld Telefon: +49 521 106-12520 E-Mail: joerg.kastning(a)uni-bielefeld.de Webseite: http://www.uni-bielefeld.de/bits/

1 0

HeartbeatMap FAILED assert(0 == "hit suicide timeout")
by 潘东元 11 Oct '19

11 Oct '19

hi all, my osd hit suicide timeout. some log: 2019-10-10 03:53:13.017760 7f1ab886e700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.25:6802/24020795 pipe(0x257eb80 sd=69 :47977 s=2 pgs=287284 cs=41 l=0 c=0x21431760).fault, initiating reconnect 2019-10-10 03:53:13.017799 7f1ab967c700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.25:6802/24020795 pipe(0x257eb80 sd=69 :47977 s=1 pgs=287284 cs=42 l=0 c=0x21431760).fault 2019-10-10 03:53:15.890773 7f1acdec3700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.19:6804/53020865 pipe(0x37537580 sd=59 :60121 s=2 pgs=423672 cs=85 l=0 c=0x21447900).fault, initiating reconnect 2019-10-10 03:53:15.890890 7f1aba288700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.19:6804/53020865 pipe(0x37537580 sd=59 :60121 s=1 pgs=423672 cs=86 l=0 c=0x21447900).fault 2019-10-10 03:53:16.209368 7f1addc3e700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f1ac29a3700' had timed out after 15 2019-10-10 03:53:16.209382 7f1addc3e700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f1ac29a3700' had suicide timed out after 150 2019-10-10 03:53:16.210765 7f1addc3e700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7f1addc3e700 time 2019-10-10 03:53:16.209415 common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x12b) [0xaf2b6b] 2: (ceph::HeartbeatMap::is_healthy()+0xa7) [0xaf3497] 3: (ceph::HeartbeatMap::check_touch_file()+0x28) [0xaf3988] 4: (CephContextServiceThread::entry()+0x13f) [0xb0353f] 5: (()+0x79d1) [0x7f1ae0b3c9d1] 6: (clone()+0x6d) [0x7f1adfaccb5d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. can you give some advice on troubleshooting?

3 4

Re: MDS rejects clients causing hanging mountpoint on linux kernel client
by Manuel Riel 10 Oct '19

10 Oct '19

I noticed a similar issue tonight. Still looking into the details, but here are the client logs I Oct 9 19:27:59 mon5-cx kernel: libceph: mds0 ***:6800 socket closed (con state OPEN) Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 connection reset Oct 9 19:28:01 mon5-cx kernel: libceph: reset on mds0 Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 closed our session Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect start Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect denied Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9109011c9980 1099517142146 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff91096cc788d0 1099517142307 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9107da741f10 1099517142312 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9109d5c40e60 1099517141612 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9108c9337da0 1099517142313 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9109d5c70340 1099517141565 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff910955acf810 1099517141792 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff91095ff56cf0 1099517142006 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff91096cc7f280 1099517142309 Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 socket closed (con state NEGOTIATING) Oct 9 19:28:02 mon5-cx kernel: ceph: mds0 rejected session Oct 9 19:28:02 mon5-cx monit: Lookup for '/srv/repos' filesystem failed -- not found in /proc/self/mounts Oct 9 19:28:02 mon5-cx monit: Filesystem '/srv/repos' not mounted Oct 9 19:28:02 mon5-cx monit: 'repos' unable to read filesystem '/srv/repos' state ... Oct 9 19:28:09 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 19:28:24 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 19:28:39 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm ... Oct 9 21:27:09 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 21:27:24 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 21:27:27 mon5-cx monit: Lookup for '/srv/repos' filesystem failed -- not found in /proc/self/mounts Oct 9 21:27:27 mon5-cx monit: Filesystem '/srv/repos' not mounted Oct 9 21:27:27 mon5-cx monit: 'repos' unable to read filesystem '/srv/repos' state Oct 9 21:27:27 mon5-cx monit: 'repos' trying to restart

1 0

Re: Unexpected increase in the memory usage of OSDs
by Anthony D'Atri 09 Oct '19

09 Oct '19

>>> Do you have statistics on the size of the OSDMaps or count of them >>> which were being maintained by the OSDs? >> No, I don't think so. How can I find this information? > > Hmm I don't know if we directly expose the size of maps. There are > perfcounters which expose the range of maps being kept around but I > don't know their names off-hand. FWIW I’ve been told that size of an OSDmap is roughly equivalent to `ceph pg dump |wc`, which if true would seem to mean that they’re trivially small for most purposes. Reality of course may be much different and/or nuanced.

1 0

14.2.4 Deduplication
by The Zombie Hunter 09 Oct '19

09 Oct '19

From my initial testing it looks like 14.2.4 fully supports the deduplication mentioned here: https://docs.ceph.com/docs/master/dev/deduplication/ However, I'm not sure where the struct object_manifest script goes in relation to foo and foo-chunk, and I'm not aware of what the offsets/caspool should be. If this still isn't fully implemented how does the dedup tool work? If I remove a file but it exists elsewhere on the volume, will it be purged or would the tool need to run again to clear the data?

3 2

Can't Modify Zone
by Mac Wynkoop 09 Oct '19

09 Oct '19

When trying to modify a zone in one of my clusters to promote it to the master zone, I get this error: ~ $ radosgw-admin zone modify --rgw-zone atl --master failed to update zonegroup: 2019-10-09 15:41:53.409 7f9ecae26840 0 ERROR: found existing zone name atl (94d26f94-d64c-40d1-9a33-56afa948d86a) in zonegroup seast (17) File exists ~ $ Anyone have any ideas what's going on here? Thanks all, Mac

1 0

Sick Nautilus cluster, OOM killing OSDs, lots of osdmaps
by Aaron Johnson 09 Oct '19

09 Oct '19

Hi all I have a smallish test cluster (14 servers, 84 OSDs) running 14.2.4. Monthly OS patching and reboots that go along with it have resulted in the cluster getting very unwell. Many of the servers in the cluster are OOM-killing the ceph-osd processes when they try to start. (6 OSDs per server running on filestore.). Strace shows the ceph-osd processes are spending hours reading through the 220k osdmap files after being started. This behavior started after we recently made it about 72% full to see how things behaved. We also upgraded it to Nautilus 14.2.2 at about the same time. I’ve tried starting just one OSD per server at a time in hopes of avoiding the OOM killer. Also tried setting noin, rebooting the whole cluster, waiting a day, then marking each of the OSDs in manually. The end result is the same either way. About 60% of PGs are still down, 30% are peering, and the rest are in worse shape. Anyone out there have suggestions about how I should go about getting this cluster healthy again? Any ideas appreciated. Thanks! - Aaron

2 1

Is it possible to have a 2nd cephfs_data volume? [Openstack]
by Jeremi Avenant 09 Oct '19

09 Oct '19

Good morning Q: Is it possible to have a 2nd cephfs_data volume and exposing it to the same openstack environment? Reason being: Our current profile is configured with erasure code value of k=3,m=1 (rack level) but we looking to buy another +- 6PB of storage w/ controllers and was thinking of moving to an erasure profile of k=2,m=1 since we're not so big on data redundancy but more on disk space + performance. For what I understand you can't change erasure profiles, therefor we need to essentially build a new ceph cluster but we're trying to understand if we can attach it to the existing openstack platform, then gradually move all the data over from the old cluster into the new cluster, destroy the old cluster and integrated it with the new one. If anyone has any recommendations to get more space out + performance at the cost of data redundancy with at least 1 rack please let me know as well. Regards -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600

2 1

CephFS no permissions for subdir
by Lars Täuber 09 Oct '19

09 Oct '19

Hi! Is it possible and if yes how to remove any permission to a subdir for a user. I'd tried to make this: ceph auth caps client.XYZ mon 'allow r' mds 'allow r, allow rws path=/XYZ, allow path=/ABC' osd 'allow rw pool=cephfs_data' but got: Error EINVAL: mds capability parse failed, stopped at ', allow path=/ABC' of 'allow r, allow rws path=/XYZ, allow path=/ABC' Thanks Lars

2 3

Re: ceph-mgr Module "zabbix" cannot send Data
by Ingo Schmidt 09 Oct '19

09 Oct '19

Thx for the hint. I fiddled around with the configuration and found this: > root@vm-2:~# ceph zabbix send > Failed to send data to Zabbix while > root@vm-2:~# zabbix_sender -vv -z 192.168.15.253 -p 10051 -s vm-2 -k ceph.num_osd -o 32 > zabbix_sender [1724513]: DEBUG: answer [{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000041"}] > info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.000041" > sent: 1; skipped: 0; total: 1 works just fine. I figured out that it could be a hostname mismatch betweend what "ceph zabbix send" transmits, and the hostname that is configured on the zabbix server. And well... it's almost embarassing that I missed this for about 3 months now but: The hostname the ceph zabbix module was submitting was in capital letters, while the hostname configured in zabbix was lowercase, even though, the hostname for that machine is in fact lowercase. I don't know why the ceph zabbix module makes it uppercase. I configured the host on zabbix with capital letters and now it works... kind regards Ingo Schmidt ---------------------------------------- IT-Department Island municipality Langeoog with in-house operations Tourismus Service and Schiffahrt

2 1

Large omap objects in radosgw .usage pool: is there a way to reshard the rgw usage log?
by Florian Haas 09 Oct '19

09 Oct '19

Hi, I am currently dealing with a cluster that's been in use for 5 years and during that time, has never had its radosgw usage log trimmed. Now that the cluster has been upgraded to Nautilus (and has completed a full deep-scrub), it is in a permanent state of HEALTH_WARN because of one large omap object: $ ceph health detail HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool '.usage' As far as I can tell, there are two thresholds that can trigger that warning: * The default omap object size warning threshold, osd_deep_scrub_large_omap_object_value_sum_threshold, is 1G. * The default omap object key count warning threshold, osd_deep_scrub_large_omap_object_key_threshold, is 200000. In this case, this was the original situation: osd.6 [WRN] : Large omap object found. Object: 15:169282cd:::usage.20:head Key count: 5834118 Size (bytes): 917351868 So that's 5.8M keys (way above threshold) and 875 MiB total object size (below threshold, but not by much). The usage log in this case was no longer needed that far back, so I trimmed it to keep only the entries from this year (radosgw-admin usage trim --end-date 2018-12-31), a process that took upward of an hour. After the trim (and a deep-scrub of the PG in question¹), my situation looks like this: osd.6 [WRN] Large omap object found. Object: 15:169282cd:::usage.20:head Key count: 1185694 Size (bytes): 187061564 So both the key count and the total object size have diminished by about 80%, which is about what you expect when you trim 5 years of usage log down to 1 year of usage log. However, my key count is still almost 6 times the threshold. I am aware that I can silence the warning by increasing osd_deep_scrub_large_omap_object_key_threshold by a factor of 10, but that's not my question. My question is what I can do to prevent the usage log from creating such large omap objects in the first place. Now, there's something else that you should know about this radosgw, which is that it is configured with the defaults for usage log sharding: rgw_usage_max_shards = 32 rgw_usage_max_user_shards = 1 ... and this cluster's radosgw is pretty much being used by a single application user. So the fact that it's happy to shard the usage log 32 ways is irrelevant as long as it puts the usage log for one user all into one shard. So, I am assuming that if I bump rgw_usage_max_user_shards up to, say, 16 or 32, all *new* usage log entries will be sharded. But I am not aware of any way to reshard the *existing* usage log. Is there such a thing? Otherwise, it seems like the only option in this situation would be to clear the usage log altogether, and tweak the sharding knobs, which should at least make the problem not reappear. Or, else, bump osd_deep_scrub_large_omap_object_key_threshold and just live with the large object. Also, is anyone aware of any adverse side effects of increasing these thresholds, and/or changing the usage log sharding settings, that I should keep in mind here? Thanks in advance for your thoughts. Cheers, Florian ¹For anyone reading this in the archives because they've run into the same problem, and wondering how you find out which PGs in a pool have too-large objects, here's a jq one-liner: ceph --format=json pg ls-by-pool <poolname> \ | jq '.pg_stats[]|select(.stat_sum.num_large_omap_objects>0)'

1 1

Nautilus: rgw hangs
by Mike Kelly 08 Oct '19

08 Oct '19

Hi, I have a fresh Nautilus Ceph cluster with radosgw as a front end. I've been testing with a slightly modified version of https://github.com/wasabi-tech/s3-benchmark/ I have 5 storage nodes with 4 osds each, for a total of 20 osds. I am testing locally on a single rgw node. First, I uploaded a bunch of 1GB objects. Now I'm attempting to download them in random order and measure the time it takes to fetch them. My problem is that during the download phase rgw will hang and the process will suck up 100% CPU on the civitwed-worker thread (according to top). The logs show that it downloads segments of the object but then stops part way though and never continues. I tried using beast instead of civitweb as a front-end, but it still hangs in the same way, leading me to believe that this is a back-end issue. This is the end of the logs, as you can see the first three lines show a successful read, and the last line show that it starts a read attempt but never completes: 2019-10-08 13:35:42.673 7fc6cec40700 20 rados->get_obj_iterate_cb oid=2217f6c8-5a9f-4cfc-a1a7-1ced740afb81.127425.2__shadow_.SCoV2VuKnMkiOqi2n3FcWgveOJYu4Io_18 obj-ofs=75497472 read_ofs=0 len=4194304 2019-10-08 13:35:42.673 7fc6cec40700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-10-08 13:35:42.673 7fc6cec40700 20 RGWObjManifest::operator++(): result: ofs=79691776 stripe_ofs=79691776 part_ofs=0 rule->part_size=0 2019-10-08 13:35:42.673 7fc6cec40700 20 rados->get_obj_iterate_cb oid=2217f6c8-5a9f-4cfc-a1a7-1ced740afb81.127425.2__shadow_.SCoV2VuKnMkiOqi2n3FcWgveOJYu4Io_19 obj-ofs=79691776 read_ofs=0 len=4194304 Can someone advise me if I've misconfigured something, or happened to find a bug? Thanks, Mike

1 0

Radosgw Usage Show Issue
by Mac Wynkoop 08 Oct '19

08 Oct '19

Hi Everyone, So it recently came to my attention that on one of our clusters, running the command "radosgw-admin usage show" returns a blank response. What is going on behind the scenes with this command, and why might it not be seeing any of the buckets properly? The data is still accessible over S3 via the rgw service, it's just not showing us either the Index or Metadata of the buckets. Greatly appreciate everyone's help in advance. Thanks, Mac

1 0

CephFS exposing public storage network
by Jaan Vaks 08 Oct '19

08 Oct '19

Hi all, I'm evaluation cephfs to serve our business as a file share that span across our 3 datacenters. One concern that I have is that when using cephfs and OpenStack Manila is that all guest vms needs access to the public storage net. This to me feels like a security concern. I've seen one suggestion is to put NFS gateways in between to prevent this, I would prefer not having to use NFS. Is there another way to solve this or is this a no concern to others, both the network and NFS? We are a small cloud provider and having different customers exposed to each other on the same storage net seems risky to me. Regards Jaan

4 4

Re: PG is stuck in repmapped and degraded
by 展荣臻（信泰） 08 Oct '19

08 Oct '19

>If the journal is no longer readable: the safe variant is to >completely re-create the OSDs after replacing the journal disk. (The >unsafe way to go is to just skip the --flush-journal part, not >recommended) Hello paul, Thank for your reply.we has replaced the journal disk. Last week we were on vacation,so this email is delayed. My confusion is that why pg was stuck? PG should to be repaired automatically,when osd is down isn't it?

2 1

Nautilus: BlueFS spillover
by Eugen Block 07 Oct '19

07 Oct '19

Hi, I'm following the discussion for a tracker issue [1] about spillover warnings that affects our upgraded Nautilus cluster. Just to clarify, would a resize of the rocksDB volume (and expanding with 'ceph-bluestore-tool bluefs-bdev-expand...') resolve that or do we have to recreate every OSD? Regards, Eugen [1] https://tracker.ceph.com/issues/38745

4 8

Re: cephx user performance impact
by iwesley＠mail.de 07 Oct '19

07 Oct '19

For performance stuff you’d better setup a running environment to benchmark by yourself. Regards, Wesley Peng > Am Oct 7, 2019 - 1:17 AM schrieb frankaritchie(a)gmail.com: > > > Would RBD performance be hurt by having thousands of cephx users defined? >

1 0

cephx user performance impact
by Frank R 06 Oct '19

06 Oct '19

Would RBD performance be hurt by having thousands of cephx users defined?

1 0

Comercial support - Brazil
by Gesiel Galvão Bernardes 04 Oct '19

04 Oct '19

Hi everyone, I searching to consulting and support of Ceph in Brazil. Does anyone on the list provide consulting in Brazil? Regards, Gesiel

2 1

Re: RAM recommendation with large OSDs?
by Anthony D'Atri 04 Oct '19

04 Oct '19

It’s not that the limit is *ignored*; sometimes the failure of the subtree isn’t *detected*. Eg., I’ve seen this happen when a node experienced kernel weirdness or OOM conditions such that the OSDs didn’t all get marked down at the same time, so the PGs all started recovering. Admitedly it’s been a while since I’ve seen this, my sense is that with Luminous the detection became a *lot* better. > On Oct 3, 2019, at 9:55 AM, Darrell Enns <darrelle(a)knowledge.ca> wrote: > > Thanks for the reply Anthony. > > Those are all considerations I am very much aware of. I'm very curious about this though: > >> mon_osd_down_out_subtree_limit. There are cases where it doesn’t kick in and a whole node will attempt to rebalance > > In what cases is the limit ignored? Do these exceptions also apply to mon_osd_min_in_ratio? Is this in the docs somewhere? > [ good Cephers trim their quoted text ]

1 0

Re: RAM recommendation with large OSDs?
by Anthony D'Atri 03 Oct '19

03 Oct '19

This is in part a question of *how many* of those dense OSD nodes you have. If you have a hundred of them, then most likely they’re spread across a decent number of racks and the loss of one or two is a tolerable *fraction* of the whole cluster. If you have a cluster of just, say, 3-4 of these dense nodes, component failure, network glitches, and even maintenance become problematic. You can *mostly* forestall whole-node rebalancing by careful alignment of fault domains with the value of mon_osd_down_out_subtree_limit. There are cases where it doesn’t kick in and a whole node will attempt to rebalance, which — assuming the CRUSH rules and topology are fault-tolerant — may cause surviving OSDs to reach full or backfillfull states, potentially resulting in an outage. If the limit does kick in, you’ll have reduced or no redundancy until you either bring the host/OSDs back up, or manually cause the recovery to proceed. As was already mentioned as well, having a small number of fault domains also limits the EC strategies you can safely use. > Thanks Paul. I was speaking more about total OSDs and RAM, rather than a single node. However, I am considering building a cluster with a large OSD/node count. This would be for archival use, with reduced performance and availability requirements. What issues would you anticipate with a large OSD/node count? Is the concern just the large rebalance if a node fails and takes out a large portion of the OSDs at once?

2 1

Objects degraded after adding disks
by Frank Schilder 03 Oct '19

03 Oct '19

I'm running a cepf fs with an 8+2 EC data pool. Disks are on 10 hosts and failure domain is host. Version is mimic 13.2.2. Today I added a few OSDs to one of the hosts and observed that a lot of PGs became inactive even though 9 out of 10 hosts were up all the time. After getting the 10th host and all disks up, I still ended up with a large amount of undersized PGs and degraded objects, which I don't understand as no OSD was removed. Here some details about the steps taken on the host with new disks, main questions at the end: - shut down OSDs (systemctl stop docker) - reboot host (this is necessary due to OS deployment via warewulf) Devices got renamed and not all disks came back up (4 OSDs remained down). This is expected, I need to re-deploy the containers to adjust for device name changes. Around this point PGs started peering and some failed waiting for 1 of the down OSDs. I don't understand why they didn't just remain active with 9 out of 10 disks. Until this moment of some OSDs coming up, all PGs were active. With min_size=9 I would expect all PGs to remain active with no changes to 9 out of the 10 hosts. - redeploy docker containers - all disks/OSDs come up, including the 4 OSDs from above - inactive PGs complete peering and become active - now I have a los of degraded Objects and undersized PGs even though not a single OSD was removed I don't understand why I have degraded objects. I should just have misplaced objects: HEALTH_ERR 22995992/145698909 objects misplaced (15.783%) Degraded data redundancy: 5213734/145698909 objects degraded (3.578%), 208 pgs degraded, 208 pgs undersized Degraded data redundancy (low space): 169 pgs backfill_toofull Note: The backfill_toofull with low utilization (usage: 38 TiB used, 1.5 PiB / 1.5 PiB avail) is a known issue in ceph (https://tracker.ceph.com/issues/39555) Also, I should be able to do whatever with 1 out of 10 hosts without loosing data access. What could be the problem here? Questions summary: Why does peering not succeed to keep all PGs active with 9 out of 10 OSDs up and in? Why do undersized PGs arise even though all OSDs are up? Why do degraded objects arise even though no OSD was removed? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

OSD down when deleting CephFS files/leveldb compaction
by Robert LeBlanc 03 Oct '19

03 Oct '19

I have one or two more stability issues I'm trying to solve in a cluster that I inherited that I just can't seem to figure out. One issue may be the cause for the other. This is a Jewel 10.2.11 cluster with ~760 - 10TB HDDs and 5GB journals on SSD. When a large number of files are deleted from CephFS (and possibly when leveldb compacts), the OSD will stop responding to heartbeats and get marked down, then come back and start recovery and then other OSDs will have the same issue until client load on the cluster eases up then it settles down. Is there a way to have leveldb compact more frequently or cause it to come up for air more frequently and respond to heartbeats and process some IO? I thought splitting PGs would help, but we are still seeing the problem (previously ~20 PGs per OSD to now ~150). I still have some space on the SSDs that I can double, almost triple the journal, but not sure if that will help in this situation. The other issue I'm seeing is that some IO just gets stuck when the OSDs are getting marked down and coming back through the cluster. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 2

Re: one read/write, many read only
by iwesley＠mail.de 02 Oct '19

02 Oct '19

There is a lock for object exists. If the file was not writing close, other ones can read only. Regards > Am Oct 2, 2019 - 12:09 AM schrieb khaled.atteya(a)gmail.com: > > > Hi, > > Is it possible to do this scenario : > If one open a file first , he will get read/write permissions and other will get read-only permission if they open the file after the first one. > > Thanks >

1 0

Re: RAM recommendation with large OSDs?
by Paul Emmerich 01 Oct '19

01 Oct '19

The problem with lots of OSDs per node is that this usually means you have too few nodes. It's perfectly fine to run 60 OSDs per node if you got a total of 1000 OSDs or so. But I've seen too many setups with 3-5 nodes where each node runs 60 OSDs which makes no sense (and usually isn't even cheaper than more nodes, especially once you consider the lost opportunity for running erasure coding). The usual backup cluster we are seeing is in the single-digit petabyte range with about 12 to 24 disks per server running ~8+3 erasure coding. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Oct 2, 2019 at 12:53 AM Darrell Enns <darrelle(a)knowledge.ca> wrote: > > Thanks Paul. I was speaking more about total OSDs and RAM, rather than a single node. However, I am considering building a cluster with a large OSD/node count. This would be for archival use, with reduced performance and availability requirements. What issues would you anticipate with a large OSD/node count? Is the concern just the large rebalance if a node fails and takes out a large portion of the OSDs at once? > > -----Original Message----- > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Tuesday, October 01, 2019 3:00 PM > To: Darrell Enns <darrelle(a)knowledge.ca> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] RAM recommendation with large OSDs? > > On Tue, Oct 1, 2019 at 6:12 PM Darrell Enns <darrelle(a)knowledge.ca> wrote: > > > > The standard advice is “1GB RAM per 1TB of OSD”. Does this actually still hold with large OSDs on bluestore? > > No > > > Can it be reasonably reduced with tuning? > > Yes > > > > From the docs, it looks like bluestore should target the “osd_memory_target” value by default. This is a fixed value (4GB by default), which does not depend on OSD size. So shouldn’t the advice really by “4GB per OSD”, rather than “1GB per TB”? Would it also be reasonable to reduce osd_memory_target for further RAM savings? > > Yes > > > For example, suppose we have 90 12TB OSD drives: > > Please don't put 90 drives in one node, that's not a good idea in 99.9% of the use cases. > > > > > “1GB per TB” rule: 1080GB RAM > > “4GB per OSD” rule: 360GB RAM > > “2GB per OSD” (osd_memory_target reduced to 2GB): 180GB RAM > > > > > > > > Those are some massively different RAM values. Perhaps the old advice was for filestore? Or there is something to consider beyond the bluestore memory target? What about when using very dense nodes (for example, 60 12TB OSDs on a single node)? > > Keep in mind that it's only a target value, it will use more during recovery if you set a low value. > We usually set a target of 3 GB per OSD and recommend 4 GB of RAM per OSD. > > RAM saving trick: use fewer PGs than recommended. > > > Paul > > > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io

1 0

Re: RAM recommendation with large OSDs?
by Paul Emmerich 01 Oct '19

01 Oct '19

On Tue, Oct 1, 2019 at 6:12 PM Darrell Enns <darrelle(a)knowledge.ca> wrote: > > The standard advice is “1GB RAM per 1TB of OSD”. Does this actually still hold with large OSDs on bluestore? No > Can it be reasonably reduced with tuning? Yes > From the docs, it looks like bluestore should target the “osd_memory_target” value by default. This is a fixed value (4GB by default), which does not depend on OSD size. So shouldn’t the advice really by “4GB per OSD”, rather than “1GB per TB”? Would it also be reasonable to reduce osd_memory_target for further RAM savings? Yes > For example, suppose we have 90 12TB OSD drives: Please don't put 90 drives in one node, that's not a good idea in 99.9% of the use cases. > > “1GB per TB” rule: 1080GB RAM > “4GB per OSD” rule: 360GB RAM > “2GB per OSD” (osd_memory_target reduced to 2GB): 180GB RAM > > > > Those are some massively different RAM values. Perhaps the old advice was for filestore? Or there is something to consider beyond the bluestore memory target? What about when using very dense nodes (for example, 60 12TB OSDs on a single node)? Keep in mind that it's only a target value, it will use more during recovery if you set a low value. We usually set a target of 3 GB per OSD and recommend 4 GB of RAM per OSD. RAM saving trick: use fewer PGs than recommended. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

one read/write, many read only
by khaled atteya 01 Oct '19

01 Oct '19

Hi, Is it possible to do this scenario : If one open a file first , he will get read/write permissions and other will get read-only permission if they open the file after the first one. Thanks

2 1

Doubt about ceph-iscsi and Vmware
by Gesiel Galvão Bernardes 01 Oct '19

01 Oct '19

Hi, I'm testing Ceph with Vmware, using Ceph-iscsi gateway. I reading documentation* and have doubts some points: - If I understanded, in general terms, for each VMFS datastore in VMware will match the an RBD image. (consequently in an RBD image I will possible have many VMWare disks). Its correct? - In documentation is this: "gwcli requires a pool with the name rbd, so it can store metadata like the iSCSI configuration". In part 4 of "Configuration", have: "Add a RBD image with the name disk_1 in the pool rbd". In this part, the use of "rbd" pool is a example and I could use any pool for storage of image, or the pool should be "rbd"? Resuming: gwcli require "rbd" pool for metadata and I could use any pool for image, or i will use just "rbd pool" for storage image and metadata? - How much memory ceph-iscsi use? Which is a good number of RAM? Regards Gesiel * https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/

4 5

Nautilus pg autoscale, data lost?
by Raymond Berg Hansen 01 Oct '19

01 Oct '19

Hi. I am new to ceph but have set it up on my homelab and started using it. It seemed very good intil I desided to try pg autoscale. After enabling autoscale to 3 of my pools, autoscale tried(?) to reduce the number of PGs and the pools are now unaccessible. I have tried to turn it off again, but no luck! Please help. ceph status: https://pastebin.com/88qNivJi (do not know why it lists 4 pools, I have 3. Maybe one of the pools I created after and deleted are in limbo?) ceph osd pool ls detail: https://pastebin.com/HZLz6yHL ceph health detail: https://pastebin.com/Kqd2YMtm

5 9

ceph-osd@n crash dumps
by Del Monaco, Andrea 01 Oct '19

01 Oct '19

Hi list, After the nodes ran OOM and after reboot, we are not able to restart the ceph-osd@x services anymore. (Details about the setup at the end). I am trying to do this manually, so we can see the error but all i see is several crash dumps - this is just one of the OSDs which is not starting. Any idea how to get past this?? [root@ceph001 ~]# /usr/bin/ceph-osd --debug_osd 10 -f --cluster ceph --id 83 --setuser ceph --setgroup ceph > /tmp/dump 2>&1 starting osd.83 at - osd_data /var/lib/ceph/osd/ceph-83 /var/lib/ceph/osd/ceph-83/journal /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] 2019-10-01 14:19:49.500 2aaaaaaf5540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] 2019-10-01 14:19:49.509 2aaaaaaf5540 -1 *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -693> 2019-10-01 14:19:49.500 2aaaaaaf5540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/ x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] -693> 2019-10-01 14:19:49.509 2aaaaaaf5540 -1 *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -693> 2019-10-01 14:19:49.500 2aaaaaaf5540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 2aaaaaaf5540 time 2019-10-01 14:19:49.494368 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h: 34: FAILED assert(stripe_width % stripe_size == 0) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x2aaaaaf3d36b] 2: (()+0x26e4f7) [0x2aaaaaf3d4f7] 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 6: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 7: (OSD::load_pgs()+0x4a9) [0x555555917e39] 8: (OSD::init()+0xc99) [0x5555559238e9] 9: (main()+0x23a3) [0x5555558017a3] 10: (__libc_start_main()+0xf5) [0x2aaab77de495] 11: (()+0x385900) [0x5555558d9900] -693> 2019-10-01 14:19:49.509 2aaaaaaf5540 -1 *** Caught signal (Aborted) ** in thread 2aaaaaaf5540 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0xf5d0) [0x2aaab69765d0] 2: (gsignal()+0x37) [0x2aaab77f22c7] 3: (abort()+0x148) [0x2aaab77f39b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x2aaaaaf3d468] 5: (()+0x26e4f7) [0x2aaaaaf3d4f7] 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*, std::shared_ptr<ceph::ErasureCodeInterface>, unsigned long)+0x46d) [0x555555c0bd3d] 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, CephContext*)+0x30a) [0x555555b0ba8a] 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, spg_t)+0x140) [0x555555abd100] 9: (OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t)+0x10cb) [0x555555914ecb] 10: (OSD::load_pgs()+0x4a9) [0x555555917e39] 11: (OSD::init()+0xc99) [0x5555559238e9] 12: (main()+0x23a3) [0x5555558017a3] 13: (__libc_start_main()+0xf5) [0x2aaab77de495] 14: (()+0x385900) [0x5555558d9900] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Environment: [root@ceph001 ~]# uname -r 3.10.0-957.27.2.el7.x86_64 [root@ceph001 ~]# cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) [root@ceph001 ~]# rpm -qa | grep -i ceph cm-config-ceph-release-mimic-8.2-73_cm8.2.noarch ceph-13.2.6-0.el7.x86_64 ceph-selinux-13.2.6-0.el7.x86_64 ceph-base-13.2.6-0.el7.x86_64 ceph-osd-13.2.6-0.el7.x86_64 cm-config-ceph-radosgw-systemd-8.2-6_cm8.2.noarch libcephfs2-13.2.6-0.el7.x86_64 ceph-common-13.2.6-0.el7.x86_64 ceph-mgr-13.2.6-0.el7.x86_64 cm-config-ceph-systemd-8.2-12_cm8.2.noarch ceph-mon-13.2.6-0.el7.x86_64 python-cephfs-13.2.6-0.el7.x86_64 ceph-mds-13.2.6-0.el7.x86_64 ceph osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 785.95801 root default -5 261.98599 host ceph001 1 hdd 7.27699 osd.1 up 1.00000 1.00000 3 hdd 7.27699 osd.3 down 1.00000 1.00000 6 hdd 7.27699 osd.6 down 1.00000 1.00000 9 hdd 7.27699 osd.9 down 0 1.00000 12 hdd 7.27699 osd.12 down 1.00000 1.00000 15 hdd 7.27699 osd.15 up 1.00000 1.00000 18 hdd 7.27699 osd.18 down 1.00000 1.00000 21 hdd 7.27699 osd.21 down 1.00000 1.00000 24 hdd 7.27699 osd.24 up 1.00000 1.00000 27 hdd 7.27699 osd.27 down 1.00000 1.00000 30 hdd 7.27699 osd.30 down 1.00000 1.00000 35 hdd 7.27699 osd.35 down 1.00000 1.00000 37 hdd 7.27699 osd.37 down 1.00000 1.00000 40 hdd 7.27699 osd.40 down 1.00000 1.00000 44 hdd 7.27699 osd.44 down 1.00000 1.00000 47 hdd 7.27699 osd.47 up 1.00000 1.00000 50 hdd 7.27699 osd.50 up 1.00000 1.00000 53 hdd 7.27699 osd.53 down 1.00000 1.00000 56 hdd 7.27699 osd.56 down 1.00000 1.00000 59 hdd 7.27699 osd.59 up 1.00000 1.00000 62 hdd 7.27699 osd.62 down 0 1.00000 65 hdd 7.27699 osd.65 down 1.00000 1.00000 68 hdd 7.27699 osd.68 down 1.00000 1.00000 71 hdd 7.27699 osd.71 down 1.00000 1.00000 74 hdd 7.27699 osd.74 down 1.00000 1.00000 77 hdd 7.27699 osd.77 up 1.00000 1.00000 80 hdd 7.27699 osd.80 down 1.00000 1.00000 83 hdd 7.27699 osd.83 up 1.00000 1.00000 86 hdd 7.27699 osd.86 down 1.00000 1.00000 88 hdd 7.27699 osd.88 down 1.00000 1.00000 91 hdd 7.27699 osd.91 down 1.00000 1.00000 94 hdd 7.27699 osd.94 down 1.00000 1.00000 97 hdd 7.27699 osd.97 down 1.00000 1.00000 100 hdd 7.27699 osd.100 down 0 1.00000 103 hdd 7.27699 osd.103 down 1.00000 1.00000 106 hdd 7.27699 osd.106 up 1.00000 1.00000 -3 261.98599 host ceph002 0 hdd 7.27699 osd.0 down 0 1.00000 4 hdd 7.27699 osd.4 up 1.00000 1.00000 7 hdd 7.27699 osd.7 up 1.00000 1.00000 11 hdd 7.27699 osd.11 down 1.00000 1.00000 13 hdd 7.27699 osd.13 up 1.00000 1.00000 16 hdd 7.27699 osd.16 down 1.00000 1.00000 19 hdd 7.27699 osd.19 down 0 1.00000 23 hdd 7.27699 osd.23 up 1.00000 1.00000 26 hdd 7.27699 osd.26 down 0 1.00000 29 hdd 7.27699 osd.29 down 0 1.00000 32 hdd 7.27699 osd.32 down 0 1.00000 33 hdd 7.27699 osd.33 down 0 1.00000 36 hdd 7.27699 osd.36 down 0 1.00000 39 hdd 7.27699 osd.39 down 1.00000 1.00000 43 hdd 7.27699 osd.43 up 1.00000 1.00000 46 hdd 7.27699 osd.46 up 1.00000 1.00000 49 hdd 7.27699 osd.49 down 1.00000 1.00000 52 hdd 7.27699 osd.52 down 1.00000 1.00000 55 hdd 7.27699 osd.55 down 0 1.00000 58 hdd 7.27699 osd.58 up 1.00000 1.00000 61 hdd 7.27699 osd.61 down 1.00000 1.00000 64 hdd 7.27699 osd.64 down 1.00000 1.00000 67 hdd 7.27699 osd.67 up 1.00000 1.00000 70 hdd 7.27699 osd.70 down 1.00000 1.00000 73 hdd 7.27699 osd.73 down 1.00000 1.00000 76 hdd 7.27699 osd.76 up 1.00000 1.00000 78 hdd 7.27699 osd.78 down 1.00000 1.00000 81 hdd 7.27699 osd.81 down 1.00000 1.00000 84 hdd 7.27699 osd.84 down 0 1.00000 87 hdd 7.27699 osd.87 down 1.00000 1.00000 90 hdd 7.27699 osd.90 down 0 1.00000 93 hdd 7.27699 osd.93 down 1.00000 1.00000 96 hdd 7.27699 osd.96 down 0 1.00000 99 hdd 7.27699 osd.99 down 0 1.00000 102 hdd 7.27699 osd.102 down 0 1.00000 105 hdd 7.27699 osd.105 up 1.00000 1.00000 -7 261.98599 host ceph003 2 hdd 7.27699 osd.2 up 1.00000 1.00000 5 hdd 7.27699 osd.5 down 1.00000 1.00000 8 hdd 7.27699 osd.8 up 1.00000 1.00000 10 hdd 7.27699 osd.10 down 0 1.00000 14 hdd 7.27699 osd.14 down 0 1.00000 17 hdd 7.27699 osd.17 up 1.00000 1.00000 20 hdd 7.27699 osd.20 down 0 1.00000 22 hdd 7.27699 osd.22 down 0 1.00000 25 hdd 7.27699 osd.25 up 1.00000 1.00000 28 hdd 7.27699 osd.28 up 1.00000 1.00000 31 hdd 7.27699 osd.31 down 0 1.00000 34 hdd 7.27699 osd.34 down 0 1.00000 38 hdd 7.27699 osd.38 down 0 1.00000 41 hdd 7.27699 osd.41 down 1.00000 1.00000 42 hdd 7.27699 osd.42 down 0 1.00000 45 hdd 7.27699 osd.45 up 1.00000 1.00000 48 hdd 7.27699 osd.48 up 1.00000 1.00000 51 hdd 7.27699 osd.51 down 1.00000 1.00000 54 hdd 7.27699 osd.54 up 1.00000 1.00000 57 hdd 7.27699 osd.57 down 1.00000 1.00000 60 hdd 7.27699 osd.60 down 1.00000 1.00000 63 hdd 7.27699 osd.63 up 1.00000 1.00000 66 hdd 7.27699 osd.66 down 1.00000 1.00000 69 hdd 7.27699 osd.69 up 1.00000 1.00000 72 hdd 7.27699 osd.72 up 1.00000 1.00000 75 hdd 7.27699 osd.75 down 1.00000 1.00000 79 hdd 7.27699 osd.79 up 1.00000 1.00000 82 hdd 7.27699 osd.82 down 1.00000 1.00000 85 hdd 7.27699 osd.85 down 1.00000 1.00000 89 hdd 7.27699 osd.89 down 0 1.00000 92 hdd 7.27699 osd.92 down 1.00000 1.00000 95 hdd 7.27699 osd.95 down 0 1.00000 98 hdd 7.27699 osd.98 down 0 1.00000 101 hdd 7.27699 osd.101 down 1.00000 1.00000 104 hdd 7.27699 osd.104 down 0 1.00000 107 hdd 7.27699 osd.107 up 1.00000 1.00000 Ceph status; [root@ceph001 ~]# ceph status cluster: id: 54052e72-6835-410e-88a9-af4ac17a8113 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 48 osds down Reduced data availability: 2053 pgs inactive, 2043 pgs down, 7 pgs peering, 3 pgs incomplete, 126 pgs stale Degraded data redundancy: 18473/27200783 objects degraded (0.068%), 106 pgs degraded, 103 pgs undersized too many PGs per OSD (258 > max 250) services: mon: 3 daemons, quorum filler001,filler002,bezavrdat-master01 mgr: bezavrdat-master01(active), standbys: filler002, filler001 mds: cephfs-1/1/1 up {0=filler002=up:replay}, 1 up:standby osd: 108 osds: 32 up, 80 in; 16 remapped pgs data: pools: 2 pools, 2176 pgs objects: 2.73 M objects, 1.7 TiB usage: 2.3 TiB used, 580 TiB / 582 TiB avail pgs: 94.347% pgs not active 18473/27200783 objects degraded (0.068%) 1951 down 79 active+undersized+degraded 76 stale+down 23 stale+active+undersized+degraded 14 down+remapped 14 stale+active+clean 6 stale+peering 3 active+clean 3 stale+active+recovery_wait+degraded 2 incomplete 2 stale+down+remapped 1 stale+incomplete 1 stale+remapped+peering 1 active+recovering+undersized+degraded+remapped Thank you in advance! Regards, [Atos logo] Andrea Del Monaco HPC Consultant – Big Data & Security M: +31 612031174 Burgemeester Rijnderslaan 30 – 1185 MC Amstelveen – The Netherlands atos.net<https://atos.net/> [LinkedIn icon]<https://www.linkedin.com/company/1259/> [Twitter icon] <https://twitter.com/atos> [Facebook icon] <https://www.facebook.com/Atos/> [Youtube icon] <https://www.youtube.com/user/Atos> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request.

1 1

moving EC pool from HDD to SSD without downtime
by Frank Schilder 01 Oct '19

01 Oct '19

I need to move a 6+2 EC pool from HDDs to SSDs while storage must remain accessible. All SSDs and HDDs are within the same failure domains. The crush rule in question is rule sr-rbd-data-one { id 5 type erasure min_size 3 max_size 8 step set_chooseleaf_tries 50 step set_choose_tries 1000 step take ServerRoom class hdd step chooseleaf indep 0 type host step emit } and I would be inclined just to change the entry "step take ServerRoom class hdd" to "step take ServerRoom class ssd" and wait for the dust to settle. However, this will almost certainly lead to all PGs being undersized and inaccessible as all objects are in the wrong place. I noticed that this is not an issue with PGs created by replicated rules as they can contain more OSDs than the replication factor while objects are moved. The same does not apply to EC rules. I suspect this is due to the setting "max_size 8", which does not allow for more than 6+2=8 OSDs being a member of a PG. What is the correct way to do what I need to do? Can I just set "max_size 16" and go? Will this work with EC rules? If not, what are my options? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

CephFS metadata: Large omap object found
by Eugen Block 01 Oct '19

01 Oct '19

Hi all, we have a new issue in our Nautilus cluster. The large omap warning seems to be more common for RGW usage, but we currently only use CephFS and RBD. I found one thread [1] regarding metadata pool, but it doesn't really help in our case. The deep-scrub of PG 36.6 brought up this message (deep-scrub finished with "ok"): 2019-09-30 20:18:22.548401 osd.9 (osd.9) 275 : cluster [WRN] Large omap object found. Object: 36:654134d2:::mds0_openfiles.0:head Key count: 238621 Size (bytes): 9994510 I checked xattr (none) and omapheader: ceph01:~ # rados -p cephfs-metadata listxattr mds0_openfiles.0 ceph01:~ # rados -p cephfs-metadata getomapheader mds0_openfiles.0 header (42 bytes) : 00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu| 00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 74 c3 12 |me v011......t..| 00000020 00 00 00 00 00 01 00 00 00 00 |..........| 0000002a ceph01:~ # ceph fs volume ls [ { "name": "cephfs" } ] The respective OSD has default thresholds regarding large_omap: ceph02:~ # ceph daemon osd.9 config show | grep large_omap "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", Can anyone point me to a solution for this? Best regards, Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033813.html

2 2

Ceph pool capacity question...
by Ilmir Mulyukov 01 Oct '19

01 Oct '19

Hello! We have a small proxmox farm with ceph consisting of three nodes. Each node has 6 disks each with a capacity of 4 TB. A only one pool has been created on these disks. Size 2/1. In theory, this pool should have a capacity: 32.74 TB But the ceph df command returns only: 22.4 TB (USED + MAX AVAIL)(16.7 + 5.7) How to explain this difference? *ceph version is:* 12.2.12-pve1 *ceph df command out:* POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED ala01vf01p01 7 N/A N/A 16.7TiB 74.53 5.70TiB 4411119 4.41M 2.62GiB 887MiB 33.4TiB *crush map:* host n01vf01 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily id -18 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.0 weight 3.669 item osd.13 weight 3.669 item osd.14 weight 3.669 item osd.15 weight 3.669 item osd.16 weight 3.669 item osd.17 weight 3.669 } host n02vf01 { id -5 # do not change unnecessarily id -6 class hdd # do not change unnecessarily id -19 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.1 weight 3.669 item osd.8 weight 3.669 item osd.9 weight 3.669 item osd.10 weight 3.669 item osd.11 weight 3.669 item osd.12 weight 3.669 } host n04vf01 { id -34 # do not change unnecessarily id -35 class hdd # do not change unnecessarily id -36 class nvme # do not change unnecessarily # weight 22.014 alg straw2 hash 0 # rjenkins1 item osd.7 weight 3.669 item osd.27 weight 3.669 item osd.24 weight 3.669 item osd.25 weight 3.669 item osd.26 weight 3.669 item osd.28 weight 3.669 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily id -21 class nvme # do not change unnecessarily # weight 66.042 alg straw2 hash 0 # rjenkins1 item n01vf01 weight 22.014 item n02vf01 weight 22.014 item n04vf01 weight 22.014 } rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit }

2 2