October 2019 - ceph-users

Inconsistent PG with data_digest_mismatch_info on all OSDs

by Wido den Hollander

Hi, I have a Mimic 13.2.6 cluster which is throwing an error on a PG that it's inconsistent. PG_DAMAGED Possible data damage: 1 pg inconsistent pg 21.e6d is active+clean+inconsistent, acting [988,508,825] I checked 'list-inconsistent-obj' (See below) and it shows: selected_object_info: "data_digest": "0xf4342d4a all osds: "data_digest": "0x224d6ca6" This looks like issue 24994 [0], but this is Mimic 13.2.6 and not Luminous 12.2.7 I also tried to download the object: rados -p <pool> get rb.0.304abf.238e1f29.00000001cc48 rb.0.304abf.238e1f29.00000001cc48 That doesn't work. This blocks for ever and causes osd.988 to report a slow request. I don't want to repair the PG at the moment as this might be a bug. Right now I'm thinking of restarting all three OSDs of that PG as this might get things moving again. But how could this happen with all three OSDs reporting the same data digest? Ideas? Wido { "epoch": 1351206, "inconsistents": [ { "object": { "name": "rb.0.304abf.238e1f29.00000001cc48", "nspace": "", "locator": "", "snap": "head", "version": 28429940 }, "errors": [], "union_shard_errors": [ "data_digest_mismatch_info" ], "selected_object_info": { "oid": { "oid": "rb.0.304abf.238e1f29.00000001cc48", "key": "", "snapid": -2, "hash": 367865453, "max": 0, "pool": 21, "namespace": "" }, "version": "1246902'28520530", "prior_version": "1240840'28429940", "last_reqid": "osd.736.0:3448586", "user_version": 28429940, "size": 4194304, "mtime": "2019-08-12 05:24:02.672911", "local_mtime": "2019-08-12 05:24:02.673401", "lost": 0, "flags": [ "dirty", "data_digest", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xf4342d4a", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 508, "primary": false, "errors": [ "data_digest_mismatch_info" ], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x224d6ca6" }, { "osd": 825, "primary": false, "errors": [ "data_digest_mismatch_info" ], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x224d6ca6" }, { "osd": 988, "primary": true, "errors": [ "data_digest_mismatch_info" ], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x224d6ca6" } ] } ] } [0]: https://tracker.ceph.com/issues/24994

4 years, 7 months

1
0
0 0

Re: ceph-users Digest, Vol 81, Issue 28

by renjianxinlover

hi, if needing to stat size of all root dirs in cephfs file system, is there any simple way to do that via ceph system tools? thanks | | renjianxinlover | | renjianxinlover(a)163.com | 签名由网易邮箱大师定制 On 10/15/2019 04:57，<ceph-users-request(a)ceph.io> wrote： Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Past_interval start interval mismatch (last_clean_epoch reported) (Huseyin Cotuk) 2. Re: Constant write load on 4 node ceph cluster (Ingo Schmidt) 3. Re: Constant write load on 4 node ceph cluster (Paul Emmerich) 4. RGW blocking on large objects (Robert LeBlanc) 5. Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects (Florian Haas) 6. Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects (Reed Dier) ---------------------------------------------------------------------- Date: Mon, 14 Oct 2019 18:41:42 +0300 From: Huseyin Cotuk <hcotuk(a)gmail.com> Subject: [ceph-users] Past_interval start interval mismatch (last_clean_epoch reported) To: ceph-users(a)ceph.io Message-ID: <0DB35170-05C5-4290-B4E2-9CB2C2BB3DA4(a)gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D" --Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi all, I also hit the bug #24866 in my test environment. According to the logs, = the last_clean_epoch in the specified OSD/PG is 17703, but the interval = starts with 17895. So the OSD fails to start. There are some other OSDs = in the same status.=20 2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: 18432 pg[18.51( = v 18388'4 lc 18386'3 (0'0,18388'4] local-lis/les=3D18430/18431 n=3D1 = ec=3D295/295 lis/c 18430/17702 les/c/f 18431/17703/0 18428/18430/18421) = [11,21]/[11,21,20] r=3D1 lpr=3D18431 pi=3D[17895,18430)/3 crt=3D18388'4 = lcod 0'0 unknown m=3D1 mbc=3D{}] 18.51 past_intervals [17895,18430) = start interval does not contain the required bound [17703,18430) start The cause is pg 18.51 went clean in 17703 but 17895 is reported to the = monitor.=20 I am using the last stable version of Mimic (13.2.6). Any idea how to fix it? Is there any way to bypass this check or fix the = reported epoch #? Thanks in advance.=20 Best regards, Huseyin Cotuk hcotuk(a)gmail.com --Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Hi = all,<div class=3D""><br class=3D""></div><div class=3D"">I also hit the = bug #24866 in my test environment. According to the logs, the = last_clean_epoch in the specified OSD/PG is 17703, but the interval = starts with 17895. So the OSD fails to start. There are some other OSDs = in the same status. </div><div class=3D""><br class=3D""></div><span = class=3D"">2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: = 18432 pg[18.51( v 18388'4 lc 18386'3 (0'0,18388'4] = local-lis/les=3D18430/18431 n=3D1 ec=3D295/295 lis/c 18430/17702 les/c/f = 18431/17703/0 18428/18430/18421) [11,21]/[11,21,20] r=3D1 lpr=3D18431= pi=3D[17895,18430)/3 crt=3D18388'4 lcod 0'0 unknown m=3D1 mbc=3D{}] = 18.51 past_intervals [17895,18430) start interval does not contain the = required bound [17703,18430) start<br class=3D""></span><span = class=3D""><br class=3D""></span><span class=3D"">The cause is pg 18.51 = went clean in 17703 but 17895 is reported to the = monitor. </span><div class=3D""><span class=3D""><br = class=3D""></span></div><div class=3D""><span class=3D"">I am using the = last stable version of Mimic (13.2.6).</span></div><div class=3D""><span = class=3D""><br class=3D""></span></div><div class=3D""><span = class=3D"">Any idea how to fix it? Is there any way to bypass this check = or fix the reported epoch #?</span></div><div class=3D""><br = class=3D""></div><div class=3D"">Thanks in advance. </div><div = class=3D""><br class=3D""><div class=3D""><div class=3D""> <div dir=3D"auto" style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; line-break: after-white-space;" class=3D""><div = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;">Best regards,</div><div style=3D"caret-color: rgb(0, 0, 0); = color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;">Huseyin = Cotuk</div><div style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><a href=3D"mailto:hcotuk@gmail.com" = class=3D"">hcotuk(a)gmail.com</a></div><div style=3D"caret-color: rgb(0, = 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><br = class=3D""></div></div></div></div></div></body></html>= --Apple-Mail=_B3AF029E-ABDE-4E3D-A975-9C670354218D-- ------------------------------ Date: Mon, 14 Oct 2019 18:34:17 +0200 (CEST) From: Ingo Schmidt <i.schmidt(a)langeoog.de> Subject: [ceph-users] Re: Constant write load on 4 node ceph cluster To: Ashley Merrick <singapore(a)amerrick.co.uk> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <1013769429.74266.1571070857785.JavaMail.zimbra(a)langeoog.de> Content-Type: text/plain; charset=utf-8 Great, this helped a lot. Although "ceph iostat" didn't give iostats of single images, but just general overview of IO, i remembered the new nautilus RDB performance monitoring. https://ceph.com/rbd/new-in-nautilus-rbd-performance-monitoring/ With a "simple" rbd perf image iotop i was able to see that the writes indeed are from the Log Server and the Zabbix Monitoring Server. I didn't expect that it would cause that much I/O... unbelieveable... ----- Ursprüngliche Mail ----- Von: "Ashley Merrick" <singapore(a)amerrick.co.uk> An: "i schmidt" <i.schmidt(a)langeoog.de> CC: "ceph-users" <ceph-users(a)ceph.io> Gesendet: Montag, 14. Oktober 2019 15:20:46 Betreff: Re: [ceph-users] Constant write load on 4 node ceph cluster Is the storage being used for the whole VM disk? If so have you checked none of your software is writing constant log's? Or something that could continuously write to disk. If your running a new version you can use : [ https://docs.ceph.com/docs/mimic/mgr/iostat/ | https://docs.ceph.com/docs/mimic/mgr/iostat/ ] to locate the exact RBD image. ------------------------------ Date: Mon, 14 Oct 2019 21:14:29 +0200 From: Paul Emmerich <paul.emmerich(a)croit.io> Subject: [ceph-users] Re: Constant write load on 4 node ceph cluster To: Ingo Schmidt <i.schmidt(a)langeoog.de> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <CAD9yTbE5UapmWXpj56yckWW2X7S6wyHz4PQSXLup6H_pzFm1vA(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" It's pretty common to see way more writes than reads if you got lots of idle VMs Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 6:34 PM Ingo Schmidt <i.schmidt(a)langeoog.de> wrote: Great, this helped a lot. Although "ceph iostat" didn't give iostats of single images, but just general overview of IO, i remembered the new nautilus RDB performance monitoring. https://ceph.com/rbd/new-in-nautilus-rbd-performance-monitoring/ With a "simple" rbd perf image iotop i was able to see that the writes indeed are from the Log Server and the Zabbix Monitoring Server. I didn't expect that it would cause that much I/O... unbelieveable... ----- Ursprüngliche Mail ----- Von: "Ashley Merrick" <singapore(a)amerrick.co.uk> An: "i schmidt" <i.schmidt(a)langeoog.de> CC: "ceph-users" <ceph-users(a)ceph.io> Gesendet: Montag, 14. Oktober 2019 15:20:46 Betreff: Re: [ceph-users] Constant write load on 4 node ceph cluster Is the storage being used for the whole VM disk? If so have you checked none of your software is writing constant log's? Or something that could continuously write to disk. If your running a new version you can use : [ https://docs.ceph.com/docs/mimic/mgr/iostat/ | https://docs.ceph.com/docs/mimic/mgr/iostat/ ] to locate the exact RBD image. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------ Date: Mon, 14 Oct 2019 12:54:05 -0700 From: Robert LeBlanc <robert(a)leblancnet.us> Subject: [ceph-users] RGW blocking on large objects To: ceph-users <ceph-users(a)ceph.io> Message-ID: <CAANLjFpWch8_mnHDeph4-=cdY1N1CXA2UJYTg7uE2w718HJ42w(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" We set up a new Nautilus cluster and only have RGW on it. While we had a job doing 200k IOPs of really small objects, I noticed that HAProxy was kicking out RGW backends because they were taking more than 2 seconds to return. We GET a large ~4GB file each minute and use that as a health check to determine if the system is taking too long to service requests. It seems that other IO is being blocked by this large transfer. This seems to be the case with both civetweb and beast. But I'm double checking beast at the moment because I'm not 100% sure we were using it at the start. Any ideas how to mitigate this? It seems that IOs are being scheduled on a thread and if they get unlucky enough to be scheduled behind a big IO, they are just stuck, in this case HAProxy could kick out the backend before the IO is returned and it has to re-request it. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 ------------------------------ Date: Mon, 14 Oct 2019 21:55:53 +0200 From: Florian Haas <florian(a)citynetwork.eu> Subject: [ceph-users] Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects To: Dan van der Ster <dan(a)vanderster.com> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <7b49695d-be1c-b36b-2ced-1b9cb0212530(a)citynetwork.eu> Content-Type: text/plain; charset=utf-8 On 14/10/2019 17:21, Dan van der Ster wrote: I'd appreciate a link to more information if you have one, but a PG autoscaling problem wouldn't really match with the issue already appearing in pre-Nautilus releases. :) https://github.com/ceph/ceph/pull/30479 Thanks! But no, this doesn't look like a likely culprit, for the reason that we also saw this in Luminous and hence, *definitely* without splits or merges in play. Has anyone else seen these scrub false positives — if that's what they are? Cheers, Florian ------------------------------ Date: Mon, 14 Oct 2019 15:57:05 -0500 From: Reed Dier <reed.dier(a)focusvq.com> Subject: [ceph-users] Re: Recurring issue: PG is inconsistent, but lists no inconsistent objects To: Florian Haas <florian(a)citynetwork.eu> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <24D2F909-3015-4256-BAA3-0E56966BF778(a)focusvq.com> Content-Type: multipart/signed; boundary="Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB"; protocol="application/pkcs7-signature"; micalg=sha-256 --Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB Content-Type: multipart/alternative; boundary="Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC" --Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I had something slightly similar to you. However, my issue was specific/limited to the device_health_metrics pool = that is auto-created with 1 PG when you turn that mgr feature on. https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315.html = <https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315.html> I was never able to get a good resolution, other than finally running pg = repair on it, and it resolved itself. Reed On Oct 14, 2019, at 2:55 PM, Florian Haas <florian(a)citynetwork.eu> = wrote: =20 On 14/10/2019 17:21, Dan van der Ster wrote: I'd appreciate a link to more information if you have one, but a PG autoscaling problem wouldn't really match with the issue already appearing in pre-Nautilus releases. :) =20 https://github.com/ceph/ceph/pull/30479 =20 Thanks! But no, this doesn't look like a likely culprit, for the = reason that we also saw this in Luminous and hence, *definitely* without = splits or merges in play. =20 Has anyone else seen these scrub false positives =E2=80=94 if that's = what they are? =20 Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io --Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">I = had something slightly similar to you.<div class=3D""><br = class=3D""></div><div class=3D"">However, my issue was specific/limited = to the device_health_metrics pool that is auto-created with 1 PG = when you turn that mgr feature on.</div><div class=3D""><br = class=3D""></div><div class=3D""><a = href=3D"https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315.ht= ml" = class=3D"">https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56315= .html</a></div><div class=3D""><br class=3D""></div><div class=3D"">I = was never able to get a good resolution, other than finally running pg = repair on it, and it resolved itself.</div><div class=3D""><br = class=3D""></div><div class=3D"">Reed<br class=3D""><div = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Oct 14, 2019, at 2:55 PM, Florian Haas <<a = href=3D"mailto:florian@citynetwork.eu" = class=3D"">florian(a)citynetwork.eu</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div class=3D"">On = 14/10/2019 17:21, Dan van der Ster wrote:<br class=3D""><blockquote = type=3D"cite" class=3D""><blockquote type=3D"cite" class=3D"">I'd = appreciate a link to more information if you have one, but a PG<br = class=3D"">autoscaling problem wouldn't really match with the issue = already<br class=3D"">appearing in pre-Nautilus releases. :)<br = class=3D""></blockquote><br class=3D""><a = href=3D"https://github.com/ceph/ceph/pull/30479" = class=3D"">https://github.com/ceph/ceph/pull/30479</a><br = class=3D""></blockquote><br class=3D"">Thanks! But no, this doesn't look = like a likely culprit, for the reason<br class=3D"">that we also saw = this in Luminous and hence, *definitely* without splits<br class=3D"">or = merges in play.<br class=3D""><br class=3D"">Has anyone else seen these = scrub false positives =E2=80=94 if that's what they are?<br class=3D""><br= class=3D"">Cheers,<br class=3D"">Florian<br = class=3D"">_______________________________________________<br = class=3D"">ceph-users mailing list -- <a = href=3D"mailto:ceph-users@ceph.io" class=3D"">ceph-users(a)ceph.io</a><br = class=3D"">To unsubscribe send an email to <a = href=3D"mailto:ceph-users-leave@ceph.io" = class=3D"">ceph-users-leave(a)ceph.io</a><br = class=3D""></div></div></blockquote></div><br = class=3D""></div></div></body></html>= --Apple-Mail=_0E365995-82AE-44C2-BC46-D7A98266CECC-- --Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCDGow ggYQMIID+KADAgECAhBNlCwQ1DvglAnFgS06KwZPMA0GCSqGSIb3DQEBDAUAMIGIMQswCQYDVQQG EwJVUzETMBEGA1UECBMKTmV3IEplcnNleTEUMBIGA1UEBxMLSmVyc2V5IENpdHkxHjAcBgNVBAoT FVRoZSBVU0VSVFJVU1QgTmV0d29yazEuMCwGA1UEAxMlVVNFUlRydXN0IFJTQSBDZXJ0aWZpY2F0 aW9uIEF1dGhvcml0eTAeFw0xODExMDIwMDAwMDBaFw0zMDEyMzEyMzU5NTlaMIGWMQswCQYDVQQG EwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRgwFgYD VQQKEw9TZWN0aWdvIExpbWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNBIENsaWVudCBBdXRoZW50 aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEAyjztlApB/975Rrno1jvm2pK/KxBOqhq8gr2+JhwpKirSzZxQgT9tlC7zl6hn1fXjSo5MqXUf ItMltrMaXqcESJuK8dtK56NCSrq4iDKaKq9NxOXFmqXX2zN8HHGjQ2b2Xv0v1L5Nk1MQPKA19xeW QcpGEGFUUd0kN+oHox+L9aV1rjfNiCj3bJk6kJaOPabPi2503nn/ITX5e8WfPnGw4VuZ79Khj1YB rf24k5Ee1sLTHsLtpiK9OjG4iQRBdq6Z/TlVx/hGAez5h36bBJMxqdHLpdwIUkTqT8se3ed0PewD ch/8kHPo5fZl5u1B0ecpq/sDN/5sCG52Ds+QU5O5EwIDAQABo4IBZDCCAWAwHwYDVR0jBBgwFoAU U3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFAnA8vwL2pTbX/4r36iZQs/J4K0AMA4GA1Ud DwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF BQcDBDARBgNVHSAECjAIMAYGBFUdIAAwUAYDVR0fBEkwRzBFoEOgQYY/aHR0cDovL2NybC51c2Vy dHJ1c3QuY29tL1VTRVJUcnVzdFJTQUNlcnRpZmljYXRpb25BdXRob3JpdHkuY3JsMHYGCCsGAQUF BwEBBGowaDA/BggrBgEFBQcwAoYzaHR0cDovL2NydC51c2VydHJ1c3QuY29tL1VTRVJUcnVzdFJT QUFkZFRydXN0Q0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1c3QuY29tMA0G CSqGSIb3DQEBDAUAA4ICAQBBRHUAqznCFfXejpVtMnFojADdF9d6HBA4kMjjsb0XMZHztuOCtKF+ xswhh2GqkW5JQrM8zVlU+A2VP72Ky2nlRA1GwmIPgou74TZ/XTarHG8zdMSgaDrkVYzz1g3nIVO9 IHk96VwsacIvBF8JfqIs+8aWH2PfSUrNxP6Ys7U0sZYx4rXD6+cqFq/ZW5BUfClN/rhk2ddQXyn7 kkmka2RQb9d90nmNHdgKrwfQ49mQ2hWQNDkJJIXwKjYA6VUR/fZUFeCUisdDe/0ABLTI+jheXUV1 eoYV7lNwNBKpeHdNuO6Aacb533JlfeUHxvBz9OfYWUiXu09sMAviM11Q0DuMZ5760CdO2VnpsXP4 KxaYIhvqPqUMWqRdWyn7crItNkZeroXaecG03i3mM7dkiPaCkgocBg0EBYsbZDZ8bsG3a08LwEsL 1Ygz3SBsyECa0waq4hOf/Z85F2w2ZpXfP+w8q4ifwO90SGZZV+HR/Jh6rEaVPDRF/CEGVqR1hiuQ OZ1YL5ezMTX0ZSLwrymUE0pwi/KDaiYB15uswgeIAcA6JzPFf9pLkAFFWs1QNyN++niFhsM47qod x/PL+5jR87myx5uYdBEQkkDc+lKB1Wct6ucXqm2EmsaQ0M95QjTmy+rDWjkDYdw3Ms6mSWE3Bn7i 5ZgtwCLXgAIe5W8mybM2JzCCBlIwggU6oAMCAQICEC8JVM/NtNco+9fMvYhQtd8wDQYJKoZIhvcN AQELBQAwgZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNV BAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBS U0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTkwOTI2MDAw MDAwWhcNMjIwOTI1MjM1OTU5WjCCAV8xCzAJBgNVBAYTAlVTMQ4wDAYDVQQREwU3MDAwNTESMBAG A1UECBMJTG91aXNpYW5hMREwDwYDVQQHEwhNZXRhaXJpZTEsMCoGA1UECRMjU3RlIDI1MCBUaGUg SGVyaXRhZ2UgUGxhemEgQnVpbGRpbmcxIzAhBgNVBAkTGjExMSBWZXRlcmFucyBNZW1vcmlhbCBC bHZkMSYwJAYDVQQKEx1Gb2N1cyBBdXRvbWF0ZWQgRXF1aXRpZXMsIExMQzFDMEEGA1UECxM6SXNz dWVkIHRocm91Z2ggRm9jdXMgQXV0b21hdGVkIEVxdWl0aWVzLCBMTEMgRS1QS0kgTWFuYWdlcjEf MB0GA1UECxMWQ29ycG9yYXRlIFNlY3VyZSBFbWFpbDESMBAGA1UEAxMJUmVlZCBEaWVyMSQwIgYJ KoZIhvcNAQkBFhVyZWVkLmRpZXJAZm9jdXN2cS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQC8ELOuFrZlztYzJp/iLLqvEc1ugxFGOkqZJsfk9XmgYc+8y48P8SNzRvWQeVs84RmN HeLg3xnAtt7xdK0cjM2PdOqznYmb2wAV6deaodu4Kun+im+Rp4/7pjvkyaiMjEL0/EA+WM8bFgjQ OKy6/T+8W9PpaJkxtdgTSXTsYWKfkZYZcJfy/aOQgtAwBflehQ8pazCo4Qx8cOG7i58ZUoBEB46v EdrxOaFzrGeeO+rx2siq9dByT7rkR4YEnKrHfXNgAnDIjsz5Q8Qq9cqH7tbt20MUU+X4D8GQ0fT+ FR8nSQNoCpKSk1rTboOza1qKJyIaBGEwKfuq5cd+KzAIwEY1AgMBAAGjggHOMIIByjAfBgNVHSME GDAWgBQJwPL8C9qU21/+K9+omULPyeCtADAdBgNVHQ4EFgQUqHWIYbt2tNSGAjge5B0wOrOk5fsw DgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUF BwMCMEAGA1UdIAQ5MDcwNQYMKwYBBAGyMQECAQEBMCUwIwYIKwYBBQUHAgEWF2h0dHBzOi8vc2Vj dGlnby5jb20vQ1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuc2VjdGlnby5jb20vU2Vj dGlnb1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYoGCCsGAQUF BwEBBH4wfDBVBggrBgEFBQcwAoZJaHR0cDovL2NydC5zZWN0aWdvLmNvbS9TZWN0aWdvUlNBQ2xp ZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAjBggrBgEFBQcwAYYXaHR0cDov L29jc3Auc2VjdGlnby5jb20wIAYDVR0RBBkwF4EVcmVlZC5kaWVyQGZvY3VzdnEuY29tMA0GCSqG SIb3DQEBCwUAA4IBAQCmjIyvT1SaifV+zEYRcBBfkJppf3IV3Hu0C1CN29RNjdslsmk2YYHUGBya 03Ae5oSAKLOt1vZLaLLZeaYNdx9Kx7Oqux+S9ozg9Vg86sSCJroIICqWHzqjo4DoOIg1nFkVMMpM kWpjNxFdUSIZLYvKyWWGFTYsMmF6rp7RNO6pJ4AtNn88lNXhkT/40ebvJ85RBoy73W6LX21vtxsh e83r6jjzHNPe33kTwWzT39YoqHoCdbQwW3jfKFQUs/6a6HB1o1el7aazOMZIDjl2poHUEoKBPJWR pk8zb1JKD4WCNaDg+sQJSMO7ekcjpJeoBEzlfjI2dT4lHlhtizWgypVuMYIDxDCCA8ACAQEwgasw gZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1Nh bGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xp ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEC8JVM/NtNco+9fMvYhQtd8w DQYJYIZIAWUDBAIBBQCgggHpMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkF MQ8XDTE5MTAxNDIwNTcwNlowLwYJKoZIhvcNAQkEMSIEIJqv7OvtXJqA52V0p7Z1lyr0AxHrxFB3 ab4K0PWgfEF+MIG8BgkrBgEEAYI3EAQxga4wgaswgZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJH cmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGlt aXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1 cmUgRW1haWwgQ0ECEC8JVM/NtNco+9fMvYhQtd8wgb4GCyqGSIb3DQEJEAILMYGuoIGrMIGWMQsw CQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3Jk MRgwFgYDVQQKEw9TZWN0aWdvIExpbWl0ZWQxPjA8BgNVBAMTNVNlY3RpZ28gUlNBIENsaWVudCBB dXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhAvCVTPzbTXKPvXzL2IULXfMA0GCSqG SIb3DQEBAQUABIIBACWiX8zkYNxrRSXTsKV0iFrjJYo5xqGItFrbhyrEuJ1zDVCxHDYGoT6PV4OE pBPN6kpGbktyrGfH+Kt3YfcGng0oTOwRBiixlplAEvNSWJeAT/aHv0ta7Z6D+sQhlQh0/qsak9hk 1ycvd25X9wf7nqPb5Rlc9VDh7HUGlc/l2EOdlMbFjmdm0GvljtPzng/288bMtIGHEwltxLu654E1 9Q7l1bUE10RmUftILZTLIgDMGu0er6OuB18ZV4zIjsMiK1SXr/DIuk5lrbVJjydx5LnPKU/7PEUz g14k9x0/DZiepnzhMBiTJjARn3N+scKEK4QtlskpKoWh8cnvovTGpCsAAAAAAAA= --Apple-Mail=_51C0D174-4907-403A-ADC3-D9E9E92FC1EB-- ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 81, Issue 28 ******************************************

4 years, 7 months

1
0
0 0

Ceph health status reports: Reduced data availability and this is resulting in slow requests are blocked

by Thomas

Hi, I want to use balancer mode "upmap" for all pools. This mode is currently enable for pool "hdb_backup" with ~600TB used space. root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B cephfs_data 1.1 TiB 89592 0 268776 0 0 0 0 0 B 43443 144 GiB 0 B 0 B cephfs_metadata 311 MiB 48 0 144 0 0 0 6 6 KiB 7465 106 MiB 0 B 0 B hdb_backup 585 TiB 51077985 0 153233955 0 0 0 12577024 4.3 TiB 281002173 523 TiB 0 B 0 B hdd 6.3 TiB 585051 0 1755153 0 0 0 4420255 69 GiB 8219453 1.2 TiB 0 B 0 B root@ld3955:~# ceph osd lspools 11 hdb_backup 51 hdd 52 backup 57 cephfs_data 58 cephfs_metadata I started with pool "cephfs_metadata" that allocated comparable low data. At the same moment when I executed ceph config set mgr mgr/balancer/pool_ids 11,52,58 Ceph status shows increasing number of Reduced data availability: <x> pg inactive, <y> pg peering And the number of <z> slow requests are blocked > 32 sec was increasing heavily up to 180. Observing Ceph log (ceph -w) it becomes clear that there's a correlation between pg inactive and slow requests are block in my cluster. How can I start analysis why the cluster reports slow requests? THX

4 years, 7 months

1
0
0 0

Ceph health status reports: subtrees have overcommitted pool target_size_ratio + subtrees have overcommitted pool target_size_bytes

by Thomas

Hi, checking my Ceph health status I get this warning: 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio The details are as follows: Pools ['hdb_backup'] overcommit available storage by 1.288x due to target_size_bytes 0 on pools [] However, the relevant settings POOL_TARGET_SIZE_RATIO_OVERCOMMITTED according to health checks <https://docs.ceph.com/docs/master/rados/operations/health-checks/> are not enabled: root@ld3955:~# ceph osd pool get hdb_backup target_size_ratio Error ENOENT: option 'target_size_ratio' is not set on pool 'hdb_backup' root@ld3955:~# ceph osd pool get hdb_backup target_size_bytes Error ENOENT: option 'target_size_bytes' is not set on pool 'hdb_backup' Therefore I would consider this warning as a bug. Regards Thomas

4 years, 7 months

1
0
0 0

Constant write load on 4 node ceph cluster

by Ingo Schmidt

Hi all We have a 4 node ceph cluster that runs generally fine. It is the storage backend for our virtualization cluster with Proxmox, that runs about 40 virtual machines (80% various Linux Servers). Now that we have implemented monitoring, i see that there is a quite constant write load of about 4-10MB/s throughout the whole day and even weekends, while read load clearly is bound to daily work hours. I am wondering whats up with this. Is this normal? Outside working hours, there is almost no read load, well under 1MB/s, while write Load stays relatively constant at around 5MB/s. During rebalancing of the Cluster, normal write load decreases significantly to under 1MB/s. Overall performance is not greatly affected by rebalancing as it is tuned to relatively slow backfills (osd_max_backfills = '3' | osd_recovery_max_active = '3') After rebalance has been finished, the average write load of about 5MB/s reappears. I have attached a little screenshot for reference. Can someone shed a little light on this? Thanks. kind regards Ingo Schmidt IT-Department Island Municipality Langeoog with in-house operations Tourismusservice and Schiffahrt Hauptstraße 28 26465 Langeoog Deutschland [ http://www.langeoog.de/ ]

4 years, 7 months

3
3
0 0

object goes missing in bucket

by Benjamin.Zieglmeier

Hey all, Experiencing an odd issue over the last week or so with a single bucket in a Ceph Luminous (12.2.11) cluster. We occasionally get a complaint from the owner of one bucket (bucket1) that a single object they have written has gone missing. If we list the bucket, the object is indeed missing, and is not present if we look for it using `radosgw-admin bi list –bucket=bucket1`. If we dump the pool the object is placed using (rados -p <poolname> ls ) we can see the object is present in the pool. Looking in the logs we are having a hard time finding any errors, and if we look at the rgw logs we can see the PUT, followed by a couple GETs and HEADs (we believe this to be client app verification) but never a DELETE. This incident has happened a couple of times over the past week, but only to this bucket. The bucket in question does average anywhere from 100-200ops per second based on the rgw usage logs. Any assistance in knowing where to look to see why these objects might be disappearing from the index would be greatly appreciated! -Ben

4 years, 7 months

1
0
0 0

CephFS and 32-bit inode numbers

by Dan van der Ster

Hi all, One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3. I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3. Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?) Thanks! Dan

4 years, 7 months

1
0
0 0

RadosGW max worker threads

by Benjamin.Zieglmeier

Hello all, Looking for guidance on the recommended highest setting (or input on experiences from users who have a high setting) for rgw_thread_pool_size. We are running multiple Luminous 12.2.11 clusters with usually 3-4 RGW daemons in front of them. We set our rgw_thread_pool_size at 512 out of the gate, and run civetweb. We had occasional service outages in one of our clusters this week and determined the rgws were running out of available threads to handle requests. We doubled our thread pool size to 1024 on each rgw and everything has been ok so far. What, if any, would be the high-end limit to set for rgw_thread_pool_size? I’ve been unable to find anything in the documentation or the user list that depicts anything higher than the default 100 threads. Thanks, Ben

4 years, 7 months

4
5
0 0

Nautilus: PGs stuck remapped+backfilling

by Eugen Block

Hi all, I have a strange issue with backfilling and I'm not sure what the cause is. It's a Nautilus cluster (upgraded) that has an SSD cache tier for OpenStack and CephFS metadata residing on the same SSDs, there were three SSDs in total. Today I added two new SSDs (NVMe) (osd.15, osd.16) to be able to shutoff one old server that has only one SSD-OSD left (osd.20). Setting the crush weight of osd.20 to 0 (and adjusting the weight of the remaining SSDs for an even distribution) leaves 3 PGs in active+remapped+backfilling state. I don't understand why the remaining PGs aren't backfilling, the crush rule is quite simple (all ssd pools are replicated with size 3). The backfilling PGs are all from the cephfs_metadata pool. Although there are 4 SSDs for 3 replicas the backfilling still should finish, right? Can anyone share their thoughts why 3 PGs can't be recovered? If more information about the cluster is required please let me know. Regards, Eugen ceph01:~ # ceph osd pool ls detail | grep meta pool 36 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 283362 flags hashpspool,nodelete,nodeep-scrub stripe_width 0 application cephfs ceph01:~ # ceph pg dump | grep remapp dumped all 36.b 28306 0 0 28910 0 8388608 101408323 219497 3078 3078 active+remapped+backfilling 2019-10-10 13:36:27.427527 284595'98565869 284595:254216941 [15,16,9] 15 [20,9,10] 20 284427'98489406 2019-10-10 00:16:02.682911 284089'98003598 2019-10-06 16:03:27.558267 0 36.d 28087 0 0 25327 0 26375382 106722204 231020 3041 3041 active+remapped+backfilling 2019-10-10 13:36:27.404739 284595'97933905 284595:252878816 [16,15,9] 16 [20,9,10] 20 284427'97887652 2019-10-10 04:13:29.371905 284259'97502135 2019-10-07 20:06:43.304593 0 36.4 28060 0 0 28406 0 8389242 104059103 225188 3061 3061 active+remapped+backfilling 2019-10-10 13:36:27.440390 284595'105299618 284595:312976619 [16,9,15] 16 [20,9,10] 20 284427'105218591 2019-10-10 00:18:07.924006 284089'104696098 2019-10-06 16:20:17.123149 0 rule ssd_ruleset { id 1 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 0 type host step emit } This is the relevant part of the osd tree: ceph01:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 34.21628 root default -31 11.25406 host ceph01 25 hdd 3.59999 osd.25 up 1.00000 1.00000 26 hdd 3.59999 osd.26 up 1.00000 1.00000 27 hdd 3.59999 osd.27 up 1.00000 1.00000 15 ssd 0.45409 osd.15 up 1.00000 1.00000 -34 11.25406 host ceph02 0 hdd 3.59999 osd.0 up 1.00000 1.00000 28 hdd 3.59999 osd.28 up 1.00000 1.00000 29 hdd 3.59999 osd.29 up 1.00000 1.00000 16 ssd 0.45409 osd.16 up 1.00000 1.00000 -37 10.79999 host ceph03 31 hdd 3.59999 osd.31 up 1.00000 1.00000 32 hdd 3.59999 osd.32 up 1.00000 1.00000 33 hdd 3.59999 osd.33 up 1.00000 1.00000 -24 0.45409 host san01-ssd 10 ssd 0.45409 osd.10 up 1.00000 1.00000 -23 0.45409 host san02-ssd 9 ssd 0.45409 osd.9 up 1.00000 1.00000 -22 0 host san03-ssd 20 ssd 0 osd.20 up 1.00000 1.00000 Don't be confused because of the '-ssd' suffix, we're using crush location hooks. This is the current PG distribution on the SSDs: ceph01:~ # ceph osd df | grep -E "^15 |^16 |^ 9|^10 |^20 " 15 ssd 0.45409 1.00000 465 GiB 34 GiB 32 GiB 1.2 GiB 857 MiB 431 GiB 7.29 0.22 27 up 16 ssd 0.45409 1.00000 465 GiB 37 GiB 34 GiB 1.5 GiB 964 MiB 428 GiB 7.87 0.23 31 up 10 ssd 0.45409 1.00000 745 GiB 27 GiB 25 GiB 1.7 GiB 950 MiB 718 GiB 3.65 0.11 29 up 9 ssd 0.45409 1.00000 745 GiB 34 GiB 32 GiB 1.3 GiB 902 MiB 711 GiB 4.60 0.14 30 up 20 ssd 0 1.00000 894 GiB 8.2 GiB 4.3 GiB 1.5 GiB 2.4 GiB 886 GiB 0.91 0.03 3 up Current ceph status: ceph01:~ # ceph -s cluster: id: 655cb05a-435a-41ba-83d9-8549f7c36167 health: HEALTH_OK services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 2d) mgr: ceph03(active, since 8d), standbys: ceph01, ceph02 mds: cephfs:1 {0=mds01=up:active} 1 up:standby-replay 1 up:standby osd: 26 osds: 26 up (since 66m), 26 in (since 66m); 3 remapped pgs data: pools: 8 pools, 264 pgs objects: 4.96M objects, 5.0 TiB usage: 16 TiB used, 31 TiB / 47 TiB avail pgs: 115745/14865558 objects misplaced (0.779%) 261 active+clean 3 active+remapped+backfilling io: client: 903 KiB/s rd, 8.8 MiB/s wr, 85 op/s rd, 266 op/s wr recovery: 0 B/s, 61 keys/s, 12 objects/s cache: 4.2 MiB/s flush, 15 MiB/s evict, 0 op/s promote

4 years, 7 months

3
7
0 0

help

by Jörg Kastning

Am 11.10.2019 um 09:21 schrieb ceph-users-request(a)ceph.io: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: MDS rejects clients causing hanging mountpoint on linux kernel client > (Manuel Riel) > 2. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout") > (Janne Johansson) > 3. Re: Nautilus: PGs stuck remapped+backfilling (Eugen Block) > 4. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout") (潘东元) > 5. Nautilus power outage - 2/3 mons and mgrs dead and no cephfs > (Alex L) > > > ---------------------------------------------------------------------- > > Date: Thu, 10 Oct 2019 11:02:28 +0800 > From: Manuel Riel <manu(a)snapdragon.cc> > Subject: [ceph-users] Re: MDS rejects clients causing hanging > mountpoint on linux kernel client > To: ukernel(a)gmail.com > Cc: ceph-users(a)ceph.io > Message-ID: <3098DC39-AED4-44F0-B9CB-44B3468285BB(a)snapdragon.cc> > Content-Type: multipart/signed; > boundary="Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D"; > protocol="application/pkcs7-signature"; micalg=sha-256 > > > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D > Content-Transfer-Encoding: quoted-printable > Content-Type: text/plain; > charset=us-ascii > > I noticed a similar issue tonight. Still looking into the details, but = > here are the client logs I=20 > > Oct 9 19:27:59 mon5-cx kernel: libceph: mds0 ***:6800 socket closed = > (con state OPEN) > Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 connection reset > Oct 9 19:28:01 mon5-cx kernel: libceph: reset on mds0 > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 closed our session > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect start > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect denied > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109011c9980 1099517142146 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91096cc788d0 1099517142307 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9107da741f10 1099517142312 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109d5c40e60 1099517141612 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9108c9337da0 1099517142313 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109d5c70340 1099517141565 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff910955acf810 1099517141792 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91095ff56cf0 1099517142006 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91096cc7f280 1099517142309 > Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 socket closed = > (con state NEGOTIATING) > Oct 9 19:28:02 mon5-cx kernel: ceph: mds0 rejected session > Oct 9 19:28:02 mon5-cx monit: Lookup for '/srv/repos' filesystem failed = > -- not found in /proc/self/mounts > Oct 9 19:28:02 mon5-cx monit: Filesystem '/srv/repos' not mounted > Oct 9 19:28:02 mon5-cx monit: 'repos' unable to read filesystem = > '/srv/repos' state > ... > Oct 9 19:28:09 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 19:28:24 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 19:28:39 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > ... > Oct 9 21:27:09 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 21:27:24 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 21:27:27 mon5-cx monit: Lookup for '/srv/repos' filesystem failed = > -- not found in /proc/self/mounts > Oct 9 21:27:27 mon5-cx monit: Filesystem '/srv/repos' not mounted > Oct 9 21:27:27 mon5-cx monit: 'repos' unable to read filesystem = > '/srv/repos' state > Oct 9 21:27:27 mon5-cx monit: 'repos' trying to restart > > > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D > Content-Disposition: attachment; > filename=smime.p7s > Content-Type: application/pkcs7-signature; > name=smime.p7s > Content-Transfer-Encoding: base64 > > MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCCx4w > ggUwMIIEGKADAgECAhEAuxgm/OfWx7qPyHlON6sLpzANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE > BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG > A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl > bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgxMDMxMDAwMDAwWhcNMTkxMDMxMjM1 > OTU5WjAjMSEwHwYJKoZIhvcNAQkBFhJtYW51QHNuYXBkcmFnb24uY2MwggEiMA0GCSqGSIb3DQEB > AQUAA4IBDwAwggEKAoIBAQDSCVHH0d9wqo1reA0PKmK8x88NJQcffMt3EBYnqrIWt0rLypsDWYBE > uTUl+WAppxa2BI172rAexkkVX1J4bM8S5P5NRInp1JOlo+s/WdXM1f0IGD4qfCUtOBM+05rdvKMS > +PBAdBkAR4XnDe26fk7XNgBLbhVifw5TEn3UBaQWPSmr6OV7WwoysNTyR3X1tKzVHTCrJv9JWfSA > vlU2NIpo5XOa9X1Zs7uxlwO7LRgsC50AFrLHVYfsqIl9zr5iBlP2lBsZVciqhv0/tszThLBjS3di > fbSyJVzAhBQTf2KCvewJbkcZr7JE8vV4sXUqrvc1voyN3sf0G/qSTy0rCl8vAgMBAAGjggHoMIIB > 5DAfBgNVHSMEGDAWgBSCr2yM+MX+lmF86B89K3FIXsSLwDAdBgNVHQ4EFgQUMzEd3LCtdsH3VrT1 > WB6sUaWGU6UwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwIAYDVR0lBBkwFwYIKwYBBQUH > AwQGCysGAQQBsjEBAwUCMBEGCWCGSAGG+EIBAQQEAwIFIDBGBgNVHSAEPzA9MDsGDCsGAQQBsjEB > AgEBATArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBaBgNVHR8E > UzBRME+gTaBLhklodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9SU0FDbGllbnRBdXRoZW50 > aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3JsMIGLBggrBgEFBQcBAQR/MH0wVQYIKwYBBQUHMAKG > SWh0dHA6Ly9jcnQuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5k > U2VjdXJlRW1haWxDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAd > BgNVHREEFjAUgRJtYW51QHNuYXBkcmFnb24uY2MwDQYJKoZIhvcNAQELBQADggEBACmWfZZiKLiT > s7YxuZc2KvsmjlDqIUK2TF6b1LqA4CMFiDwwNDUvxoOClhZJFpDDfnw4w86fDheIM5fPinf+0wif > 8YAphESKyVIZPDl7t3D4zQmU5Ytsq44iCS+TTAQXcWbz7Qe6tBQWZvffKNb6H95blJj0t86Ga0wD > huUXMZbjFMOYcRLGsllUe5H2qJUs0CY/Y9MGGXjcy2HFbhStbuUyj8rwi9ynCvz3LFS30J9MXhYr > c+yu9tZiHZZdjl7gP7RCez+IDZme1TfVYu6KBXxrvkZMRU9+KILtcXmIvMwzelp8KC9Ha9AtXZNd > g9wSFvwRAOFtAEJ9+m0fhdzmq1cwggXmMIIDzqADAgECAhBqm+E4O/8ra58B1dm4p1JWMA0GCSqG > SIb3DQEBDAUAMIGFMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw > DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDErMCkGA1UEAxMiQ09N > T0RPIFJTQSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0xMzAxMTAwMDAwMDBaFw0yODAxMDky > MzU5NTlaMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD > VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09NT0RP > IFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZI > hvcNAQEBBQADggEPADCCAQoCggEBAL6znlesKHZ1QBbHOAOY08YYdiFQ8yV5C0y1oNF9Olg+nKcx > Lqf2NHbZhGra0D00SOTq9bus3/mxgUsg/Wh/eXQ0pnp8tZ8XZWAnlyKMpjL+qUByRjXCA6RQyDMq > VaVUkbIr5SU0RDX/kSsKwer3H1pT/HUrBN0X8sKtPTdGX8XAWt/VdMLBrZBlgvnkCos+KQWWCo63 > OTTqRvaq8aWccm+KOMjTcE6s2mj6RkalweyDI7X+7U5lNo6jzC8RTXtVV4/Vwdax720YpMPJQaDa > ElmOupyTf1Qib+cpukNJnQmwygjD8m046DQkLnpXNCAGjuJy1F5NATksUsbfJAr7FLUCAwEAAaOC > ATwwggE4MB8GA1UdIwQYMBaAFLuvfgI9+qbxPISOre44mOzZMjLUMB0GA1UdDgQWBBSCr2yM+MX+ > lmF86B89K3FIXsSLwDAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAE > CjAIMAYGBFUdIAAwTAYDVR0fBEUwQzBBoD+gPYY7aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09N > T0RPUlNBQ2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwcQYIKwYBBQUHAQEEZTBjMDsGCCsGAQUF > BzAChi9odHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FBZGRUcnVzdENBLmNydDAkBggr > BgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQB4XLKB > KDRPPO5fVs6fl1bsj6JrF/bz9kkIBtTYLzXN30D+03Hj6OxCDBEaIeNmsBhrJmuubvyE7HtoSmR8 > 09AgcYboW+rcTNZ/8u/Hv+GTrNI/AhqX2/kiQNxmgUPt/eJPs92Qclj0HnVyy9TnSvGkSDU7I5Px > +TbO+88G4zipA2psZaWeEykgzClZlPz1FjTCkk77ZXp5cQYYexE6zeeN4/0OqqoAloFrjAF4o50Y > JafX8mnahjp3I2Y2mkjhk0xQfhNqbzlLWPoT3m7j7U26u7zg6swjOq8hITYc3/np5tM5aVyu6t99 > p17bTbY7+1RTWBviN9YJzK8HxzObXYWBf/L+VGOYNsQDTxAk0Hbvb1j6KjUhg7fO294F29QIhhmi > NOr84JHoy+fNLpfvYc/Q9EtFOI5ISYgOxLk3nD/whbUe9rmEQXLp8MB933Ij474gwwCPUpwv9mj2 > PMnXoc7mbrS22XUSeTwxCTP9bcmUdp4jmIoWfhQm7X9w/Zgddg+JZ/YnIHOwsGsaTUgj7fIvxqit > h7DoJC91WJ8Lce3CVJqb1XWeKIJ84F7YLXZN0oa7TktYgDdmQVxYkZo1c5noaDKH9Oq9cbm/vOYR > UM1cWcef20Wkyk5S/GFyyPJwG0fR1nRas3DqAf4cXxMiEKcff7PNa4M3RGTqH0pWR8p6EjGCA8ow > ggPGAgEBMIGtMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw > DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09N > T0RPIFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIRALsYJvzn > 1se6j8h5TjerC6cwDQYJYIZIAWUDBAIBBQCgggHtMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw > HAYJKoZIhvcNAQkFMQ8XDTE5MTAxMDAzMDIyOVowLwYJKoZIhvcNAQkEMSIEIOo1zdaTl2T2mCM4 > ttyllcqgdZ0MagVCBglhDdSmEkM8MIG+BgkrBgEEAYI3EAQxgbAwga0wgZcxCzAJBgNVBAYTAkdC > MRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoT > EUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNh > dGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfWx7qPyHlON6sLpzCBwAYLKoZIhvcNAQkQ > AgsxgbCgga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO > BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01P > RE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfW > x7qPyHlON6sLpzANBgkqhkiG9w0BAQEFAASCAQCPNgVk/I2nxe1niT2avKeoKQiQfWhGIsFe6V3V > 8E8WDEGfQwQX0ApcF0sm465wPRuu+K3n3mMo7z2mNZtOyfpHeAYElsZsLOV/pMgMWPNYISwFPu2K > J7m9o4anPff7JT0BTMf6XbsFTezCUCIQotuGur/6KquXOCOddX8xYbhcSAuwlAPuhBzCnn42goU0 > tybdFZqmcjJl+5uRYYHmr68zhcA7XFF5tFTzIaq4krMDMfk0aKpKNTL5wIhfFwChe9FAUgsHT25h > p3bQkDlKZgfeWeqYtzkOoFCqpznWR65EqKaMmGaemWCQuOTMMP+8lVC7HIlxbNmbnEYH3sbdldZJ > AAAAAAAA > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D-- > > ------------------------------ > > Date: Thu, 10 Oct 2019 15:16:46 +0200 > From: Janne Johansson <icepic.dz(a)gmail.com> > Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide > timeout") > To: 潘东元 <dongyuanpan0(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CAA6-MF-SpecCptAd0q_GF0cHSUG4G4W8pZmiXEi1DP0kAwvrHw(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000004ebc3105948e3713" > > --0000000000004ebc3105948e3713 > Content-Type: text/plain; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > Den tors 10 okt. 2019 kl 15:12 skrev =E6=BD=98=E4=B8=9C=E5=85=83 <dongyuanp= > an0(a)gmail.com>: > >> hi all, >> my osd hit suicide timeout. >> >> common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeout") >> >> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) >> > > >> can you give some advice on troubleshooting? >> > > It is a very old release, chances are large whatever issue you get here > might have been fixed in the last 5 years. > > --=20 > May the most significant bit of your life be positive. > > --0000000000004ebc3105948e3713 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= > <div dir=3D"ltr" class=3D"gmail_attr">Den tors 10 okt. 2019 kl 15:12 skrev = > =E6=BD=98=E4=B8=9C=E5=85=83 <<a href=3D"mailto:dongyuanpan0@gmail.com">d= > ongyuanpan0(a)gmail.com</a>>:<br></div><blockquote class=3D"gmail_quote" s= > tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad= > ding-left:1ex">hi all,<br> > =C2=A0 =C2=A0 my osd hit suicide timeout.<br><br> > common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeou= > t")<br> > <br> > =C2=A0ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)<br> > </blockquote><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"ma= > rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:= > 1ex"> > can you give some advice on troubleshooting?<br></blockquote></div><div><br= >> </div><div>It is a very old release, chances are large whatever issue you = > get here might have been fixed in the last 5 years.</div><div><br></div>-- = > <br><div dir=3D"ltr" class=3D"gmail_signature">May the most significant bit= > of your life be positive.<br></div></div> > > --0000000000004ebc3105948e3713-- > > ------------------------------ > > Date: Thu, 10 Oct 2019 13:57:38 +0000 > From: Eugen Block <eblock(a)nde.ag> > Subject: [ceph-users] Re: Nautilus: PGs stuck remapped+backfilling > To: ceph-users(a)ceph.io > Message-ID: > <20191010135738.Horde.-gF9xNAEG9yFgxVbn83rdTU(a)webmail.nde.ag> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > Please ignore my email, the PGs have eventually recovered, it just > took way more time than expected or observed for the other PGs. I'll > try to be more patient next time. ;-) > > ------------------------------ > > Date: Fri, 11 Oct 2019 14:12:49 +0800 > From: 潘东元 <dongyuanpan0(a)gmail.com> > Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide > timeout") > To: Janne Johansson <icepic.dz(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CANkQ9Li-ay1xVUkKqGtM1yq_PbSHr57GXV35aRLvOQtaZwE-xQ(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > I‘m pretty sure, this issue here is that there is a communication > issue between the osds. > logged over and over again report initiating reconnect. > > I looked at my network,and have dropped packets,this is probably the > tcp queue full at osd daemon listen port. > My cluster had 21 nodes, with 5 osds on each node. > > [root@node-4 ~]# netstat -nat | grep -w "6801" | grep -w tcp | grep > ESTABLISHED | wc -l > 271 > > [root@node-4 ~]# sysctl -a | grep somax > net.core.somaxconn = 128 > > this will cause the tcp connection to reset. > > so,i will increase this parameter! > > ------------- > Regards! > > Janne Johansson <icepic.dz(a)gmail.com> 于2019年10月10日周四下午9:16写道： >> >> >> >> Den tors 10 okt. 2019 kl 15:12 skrev 潘东元 <dongyuanpan0(a)gmail.com>: >>> >>> hi all, >>> my osd hit suicide timeout. >>> >>> common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") >>> >>> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) >> >> >>> >>> can you give some advice on troubleshooting? >> >> >> It is a very old release, chances are large whatever issue you get here might have been fixed in the last 5 years. >> >> -- >> May the most significant bit of your life be positive. > > ------------------------------ > > Date: Fri, 11 Oct 2019 07:21:40 -0000 > From: "Alex L" <alexut.voicu(a)gmail.com> > Subject: [ceph-users] Nautilus power outage - 2/3 mons and mgrs dead > and no cephfs > To: ceph-users(a)ceph.io > Message-ID: <157077850022.1193.1225987085352609330@mailman-web> > Content-Type: text/plain; charset="utf-8" > > Hi list, > Had a power outage killing the whole cluster. Cephfs will not start at all, but RBD works just fine. > I did have 4 unfound objects that I eventually had to rollback or delete which I don't really understand as I should've had a copy of the those pbjects on the other drives? > > 2/3 mons and mgrs are damaged but without any errors. > > I have loads stored on cephfs so would very much like to get that running as a first priority. > > Thanks! > Alex > > Info about the home cluster: > I run 23 osds on 3 hosts. 6 of these are a SSD cache layer for the spinning rust, as well as the metadata portion for cephfs which in retrospect might have to be put back on the spinning rust. > > # ceph -v > ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable) > > # head ceph-mgr.pve21.log.7 > 2019-10-04 00:00:00.397 7fee56df3700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 > 2019-10-04 00:00:00.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855f6b7500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:00.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:01.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2680: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 4 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) > 2019-10-04 00:00:01.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e486380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:01.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:02.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e4b5500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:02.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:03.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2681: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 2 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) > 2019-10-04 00:00:03.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e3b0380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > > # head ceph-mon.pve21.log.7 > 2019-10-04 00:00:00.389 7f7c25b52700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 > 2019-10-04 00:00:00.397 7f7c25b52700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 > 2019-10-04 00:00:00.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:00.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:01.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:01.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:02.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:02.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:03.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:03.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > > > # head ceph-mds.pve21.log.7 > 2019-10-04 00:00:00.389 7f1b2f1b5700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 > 2019-10-04 00:00:00.397 7f1b2f1b5700 -1 received signal: Hangup from (PID: 193052) UID: 0 > 2019-10-04 00:00:04.881 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839bb200 0x556f838d4000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby > 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0) > 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=6 cs=4 l=0).handle_message_footer missed message? skipped from seq 0 to 2 > 2019-10-04 00:01:19.167 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:01:23.555 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.22:0/2875552603 conn(0x556f839bda80 0x556f837f9000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:02:08.768 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2427365808 conn(0x556f839bd180 0x556f83e9d800 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:02:20.140 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby > 2019-10-04 00:02:21.420 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839e0480 0x556f83d7f000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0) > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 81, Issue 19 > ****************************************** > -- Jörg Kastning Plattformen und Serverdienste BITS - Bielefelder IT-Servicezentrum Universität Bielefeld Universitätsstraße 25 33615 Bielefeld Telefon: +49 521 106-12520 E-Mail: joerg.kastning(a)uni-bielefeld.de Webseite: http://www.uni-bielefeld.de/bits/

4 years, 7 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users October 2019