[ceph-users] help

11 Oct 2019


      Am 11.10.2019 um 09:21 schrieb ceph-users-request@ceph.io:
...
Send ceph-users mailing list submissions to
  ceph-users@ceph.io
To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
  ceph-users-request@ceph.io
You can reach the person managing the list at
  ceph-users-owner@ceph.io
When replying, please edit your Subject line so it is more specific
than "Re: Contents of ceph-users digest..."
Today's Topics:
1. Re: MDS rejects clients causing hanging mountpoint on linux kernel client
       (Manuel Riel)
    2. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout")
       (Janne Johansson)
    3. Re: Nautilus: PGs stuck remapped+backfilling (Eugen Block)
    4. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout") (潘东元)
    5. Nautilus power outage - 2/3 mons and mgrs dead and no cephfs
       (Alex L)
----------------------------------------------------------------------
Date: Thu, 10 Oct 2019 11:02:28 +0800
From: Manuel Riel <manu@snapdragon.cc>
Subject: [ceph-users] Re: MDS rejects clients causing hanging
  mountpoint on linux kernel client
To: ukernel@gmail.com
Cc: ceph-users@ceph.io
Message-ID: <3098DC39-AED4-44F0-B9CB-44B3468285BB@snapdragon.cc>
Content-Type: multipart/signed;
  boundary="Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D";
  protocol="application/pkcs7-signature"; micalg=sha-256
--Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
  charset=us-ascii
I noticed a similar issue tonight. Still looking into the details, but =
here are the client logs I=20
Oct  9 19:27:59 mon5-cx kernel: libceph: mds0 ***:6800 socket closed =
(con state OPEN)
Oct  9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 connection reset
Oct  9 19:28:01 mon5-cx kernel: libceph: reset on mds0
Oct  9 19:28:01 mon5-cx kernel: ceph: mds0 closed our session
Oct  9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect start
Oct  9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect denied
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff9109011c9980 1099517142146
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff91096cc788d0 1099517142307
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff9107da741f10 1099517142312
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff9109d5c40e60 1099517141612
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff9108c9337da0 1099517142313
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff9109d5c70340 1099517141565
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff910955acf810 1099517141792
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff91095ff56cf0 1099517142006
Oct  9 19:28:01 mon5-cx kernel: ceph:  dropping dirty+flushing Fw state =
for ffff91096cc7f280 1099517142309
Oct  9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 socket closed =
(con state NEGOTIATING)
Oct  9 19:28:02 mon5-cx kernel: ceph: mds0 rejected session
Oct  9 19:28:02 mon5-cx monit: Lookup for '/srv/repos' filesystem failed =
  -- not found in /proc/self/mounts
Oct  9 19:28:02 mon5-cx monit: Filesystem '/srv/repos' not mounted
Oct  9 19:28:02 mon5-cx monit: 'repos' unable to read filesystem =
'/srv/repos' state
...
Oct  9 19:28:09 mon5-cx kernel: ceph: get_quota_realm: ino =
(1.fffffffffffffffe) null i_snap_realm
Oct  9 19:28:24 mon5-cx kernel: ceph: get_quota_realm: ino =
(1.fffffffffffffffe) null i_snap_realm
Oct  9 19:28:39 mon5-cx kernel: ceph: get_quota_realm: ino =
(1.fffffffffffffffe) null i_snap_realm
...
Oct  9 21:27:09 mon5-cx kernel: ceph: get_quota_realm: ino =
(1.fffffffffffffffe) null i_snap_realm
Oct  9 21:27:24 mon5-cx kernel: ceph: get_quota_realm: ino =
(1.fffffffffffffffe) null i_snap_realm
Oct  9 21:27:27 mon5-cx monit: Lookup for '/srv/repos' filesystem failed =
  -- not found in /proc/self/mounts
Oct  9 21:27:27 mon5-cx monit: Filesystem '/srv/repos' not mounted
Oct  9 21:27:27 mon5-cx monit: 'repos' unable to read filesystem =
'/srv/repos' state
Oct  9 21:27:27 mon5-cx monit: 'repos' trying to restart
--Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D
Content-Disposition: attachment;
  filename=smime.p7s
Content-Type: application/pkcs7-signature;
  name=smime.p7s
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCCx4w
ggUwMIIEGKADAgECAhEAuxgm/OfWx7qPyHlON6sLpzANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE
BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG
A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl
bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgxMDMxMDAwMDAwWhcNMTkxMDMxMjM1
OTU5WjAjMSEwHwYJKoZIhvcNAQkBFhJtYW51QHNuYXBkcmFnb24uY2MwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDSCVHH0d9wqo1reA0PKmK8x88NJQcffMt3EBYnqrIWt0rLypsDWYBE
uTUl+WAppxa2BI172rAexkkVX1J4bM8S5P5NRInp1JOlo+s/WdXM1f0IGD4qfCUtOBM+05rdvKMS
+PBAdBkAR4XnDe26fk7XNgBLbhVifw5TEn3UBaQWPSmr6OV7WwoysNTyR3X1tKzVHTCrJv9JWfSA
vlU2NIpo5XOa9X1Zs7uxlwO7LRgsC50AFrLHVYfsqIl9zr5iBlP2lBsZVciqhv0/tszThLBjS3di
fbSyJVzAhBQTf2KCvewJbkcZr7JE8vV4sXUqrvc1voyN3sf0G/qSTy0rCl8vAgMBAAGjggHoMIIB
5DAfBgNVHSMEGDAWgBSCr2yM+MX+lmF86B89K3FIXsSLwDAdBgNVHQ4EFgQUMzEd3LCtdsH3VrT1
WB6sUaWGU6UwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwIAYDVR0lBBkwFwYIKwYBBQUH
AwQGCysGAQQBsjEBAwUCMBEGCWCGSAGG+EIBAQQEAwIFIDBGBgNVHSAEPzA9MDsGDCsGAQQBsjEB
AgEBATArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBaBgNVHR8E
UzBRME+gTaBLhklodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9SU0FDbGllbnRBdXRoZW50
aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3JsMIGLBggrBgEFBQcBAQR/MH0wVQYIKwYBBQUHMAKG
SWh0dHA6Ly9jcnQuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5k
U2VjdXJlRW1haWxDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAd
BgNVHREEFjAUgRJtYW51QHNuYXBkcmFnb24uY2MwDQYJKoZIhvcNAQELBQADggEBACmWfZZiKLiT
s7YxuZc2KvsmjlDqIUK2TF6b1LqA4CMFiDwwNDUvxoOClhZJFpDDfnw4w86fDheIM5fPinf+0wif
8YAphESKyVIZPDl7t3D4zQmU5Ytsq44iCS+TTAQXcWbz7Qe6tBQWZvffKNb6H95blJj0t86Ga0wD
huUXMZbjFMOYcRLGsllUe5H2qJUs0CY/Y9MGGXjcy2HFbhStbuUyj8rwi9ynCvz3LFS30J9MXhYr
c+yu9tZiHZZdjl7gP7RCez+IDZme1TfVYu6KBXxrvkZMRU9+KILtcXmIvMwzelp8KC9Ha9AtXZNd
g9wSFvwRAOFtAEJ9+m0fhdzmq1cwggXmMIIDzqADAgECAhBqm+E4O/8ra58B1dm4p1JWMA0GCSqG
SIb3DQEBDAUAMIGFMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw
DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDErMCkGA1UEAxMiQ09N
T0RPIFJTQSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0xMzAxMTAwMDAwMDBaFw0yODAxMDky
MzU5NTlaMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD
VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09NT0RP
IFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAL6znlesKHZ1QBbHOAOY08YYdiFQ8yV5C0y1oNF9Olg+nKcx
Lqf2NHbZhGra0D00SOTq9bus3/mxgUsg/Wh/eXQ0pnp8tZ8XZWAnlyKMpjL+qUByRjXCA6RQyDMq
VaVUkbIr5SU0RDX/kSsKwer3H1pT/HUrBN0X8sKtPTdGX8XAWt/VdMLBrZBlgvnkCos+KQWWCo63
OTTqRvaq8aWccm+KOMjTcE6s2mj6RkalweyDI7X+7U5lNo6jzC8RTXtVV4/Vwdax720YpMPJQaDa
ElmOupyTf1Qib+cpukNJnQmwygjD8m046DQkLnpXNCAGjuJy1F5NATksUsbfJAr7FLUCAwEAAaOC
ATwwggE4MB8GA1UdIwQYMBaAFLuvfgI9+qbxPISOre44mOzZMjLUMB0GA1UdDgQWBBSCr2yM+MX+
lmF86B89K3FIXsSLwDAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAE
CjAIMAYGBFUdIAAwTAYDVR0fBEUwQzBBoD+gPYY7aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09N
T0RPUlNBQ2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwcQYIKwYBBQUHAQEEZTBjMDsGCCsGAQUF
BzAChi9odHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FBZGRUcnVzdENBLmNydDAkBggr
BgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQB4XLKB
KDRPPO5fVs6fl1bsj6JrF/bz9kkIBtTYLzXN30D+03Hj6OxCDBEaIeNmsBhrJmuubvyE7HtoSmR8
09AgcYboW+rcTNZ/8u/Hv+GTrNI/AhqX2/kiQNxmgUPt/eJPs92Qclj0HnVyy9TnSvGkSDU7I5Px
+TbO+88G4zipA2psZaWeEykgzClZlPz1FjTCkk77ZXp5cQYYexE6zeeN4/0OqqoAloFrjAF4o50Y
JafX8mnahjp3I2Y2mkjhk0xQfhNqbzlLWPoT3m7j7U26u7zg6swjOq8hITYc3/np5tM5aVyu6t99
p17bTbY7+1RTWBviN9YJzK8HxzObXYWBf/L+VGOYNsQDTxAk0Hbvb1j6KjUhg7fO294F29QIhhmi
NOr84JHoy+fNLpfvYc/Q9EtFOI5ISYgOxLk3nD/whbUe9rmEQXLp8MB933Ij474gwwCPUpwv9mj2
PMnXoc7mbrS22XUSeTwxCTP9bcmUdp4jmIoWfhQm7X9w/Zgddg+JZ/YnIHOwsGsaTUgj7fIvxqit
h7DoJC91WJ8Lce3CVJqb1XWeKIJ84F7YLXZN0oa7TktYgDdmQVxYkZo1c5noaDKH9Oq9cbm/vOYR
UM1cWcef20Wkyk5S/GFyyPJwG0fR1nRas3DqAf4cXxMiEKcff7PNa4M3RGTqH0pWR8p6EjGCA8ow
ggPGAgEBMIGtMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw
DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09N
T0RPIFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIRALsYJvzn
1se6j8h5TjerC6cwDQYJYIZIAWUDBAIBBQCgggHtMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw
HAYJKoZIhvcNAQkFMQ8XDTE5MTAxMDAzMDIyOVowLwYJKoZIhvcNAQkEMSIEIOo1zdaTl2T2mCM4
ttyllcqgdZ0MagVCBglhDdSmEkM8MIG+BgkrBgEEAYI3EAQxgbAwga0wgZcxCzAJBgNVBAYTAkdC
MRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoT
EUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNh
dGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfWx7qPyHlON6sLpzCBwAYLKoZIhvcNAQkQ
AgsxgbCgga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01P
RE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfW
x7qPyHlON6sLpzANBgkqhkiG9w0BAQEFAASCAQCPNgVk/I2nxe1niT2avKeoKQiQfWhGIsFe6V3V
8E8WDEGfQwQX0ApcF0sm465wPRuu+K3n3mMo7z2mNZtOyfpHeAYElsZsLOV/pMgMWPNYISwFPu2K
J7m9o4anPff7JT0BTMf6XbsFTezCUCIQotuGur/6KquXOCOddX8xYbhcSAuwlAPuhBzCnn42goU0
tybdFZqmcjJl+5uRYYHmr68zhcA7XFF5tFTzIaq4krMDMfk0aKpKNTL5wIhfFwChe9FAUgsHT25h
p3bQkDlKZgfeWeqYtzkOoFCqpznWR65EqKaMmGaemWCQuOTMMP+8lVC7HIlxbNmbnEYH3sbdldZJ
AAAAAAAA
--Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D--
------------------------------
Date: Thu, 10 Oct 2019 15:16:46 +0200
From: Janne Johansson <icepic.dz@gmail.com>
Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide
  timeout")
To: 潘东元 <dongyuanpan0@gmail.com>
Cc: ceph-users <ceph-users@ceph.io>
Message-ID:
  <CAA6-MF-SpecCptAd0q_GF0cHSUG4G4W8pZmiXEi1DP0kAwvrHw@mail.gmail.com>
Content-Type: multipart/alternative;
  boundary="0000000000004ebc3105948e3713"
--0000000000004ebc3105948e3713
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Den tors 10 okt. 2019 kl 15:12 skrev =E6=BD=98=E4=B8=9C=E5=85=83 <dongyuanp=
an0@gmail.com>:
...
hi all,
     my osd hit suicide timeout.
common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeout")
ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
...
can you give some advice on troubleshooting?
It is a very old release, chances are large whatever issue you get here
might have been fixed in the last 5 years.
--=20
May the most significant bit of your life be positive.
--0000000000004ebc3105948e3713
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">Den tors 10 okt. 2019 kl 15:12 skrev =
=E6=BD=98=E4=B8=9C=E5=85=83 <<a href=3D"mailto:dongyuanpan0@gmail.com">d=
ongyuanpan0@gmail.com</a>>:<br></div><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad=
ding-left:1ex">hi all,<br>
=C2=A0 =C2=A0 my osd hit suicide timeout.<br><br>
common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeou=
t")<br>
<br>
=C2=A0ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)<br>
</blockquote><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:=
1ex">
can you give some advice on troubleshooting?<br></blockquote></div><div><br=
...
</div><div>It is a very old release, chances are large whatever issue you =
get here might have been fixed in the last 5 years.</div><div><br></div>-- =
<br><div dir=3D"ltr" class=3D"gmail_signature">May the most significant bit=
  of your life be positive.<br></div></div>
--0000000000004ebc3105948e3713--
------------------------------
Date: Thu, 10 Oct 2019 13:57:38 +0000
From: Eugen Block <eblock@nde.ag>
Subject: [ceph-users] Re: Nautilus: PGs stuck remapped+backfilling
To: ceph-users@ceph.io
Message-ID:
  <20191010135738.Horde.-gF9xNAEG9yFgxVbn83rdTU@webmail.nde.ag>
Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes
Please ignore my email, the PGs have eventually recovered, it just
took way more time than expected or observed for the other PGs. I'll
try to be more patient next time. ;-)
------------------------------
Date: Fri, 11 Oct 2019 14:12:49 +0800
From: 潘东元 <dongyuanpan0@gmail.com>
Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide
  timeout")
To: Janne Johansson <icepic.dz@gmail.com>
Cc: ceph-users <ceph-users@ceph.io>
Message-ID:
  <CANkQ9Li-ay1xVUkKqGtM1yq_PbSHr57GXV35aRLvOQtaZwE-xQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
I‘m pretty sure, this issue here is that there is a communication
issue between the osds.
logged over and over again report initiating reconnect.
I looked at my network,and have dropped packets,this is probably the
tcp queue full at osd daemon listen port.
My cluster had 21 nodes, with 5 osds on each node.
[root@node-4 ~]# netstat -nat | grep -w "6801" | grep -w tcp | grep
ESTABLISHED | wc -l
271
[root@node-4 ~]# sysctl -a | grep somax
net.core.somaxconn = 128
this will cause the tcp connection to reset.
so,i will increase this parameter!
-------------
Regards!
Janne Johansson <icepic.dz@gmail.com> 于2019年10月10日周四 下午9:16写道：
...
Den tors 10 okt. 2019 kl 15:12 skrev 潘东元 <dongyuanpan0@gmail.com>:
...
hi all,
     my osd hit suicide timeout.
common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")
ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
...
can you give some advice on troubleshooting?
It is a very old release, chances are large whatever issue you get here might have been fixed in the last 5 years.
--
May the most significant bit of your life be positive.
------------------------------
Date: Fri, 11 Oct 2019 07:21:40 -0000
From: "Alex L" <alexut.voicu@gmail.com>
Subject: [ceph-users] Nautilus power outage - 2/3 mons and mgrs dead
  and no cephfs
To: ceph-users@ceph.io
Message-ID: <157077850022.1193.1225987085352609330@mailman-web>
Content-Type: text/plain; charset="utf-8"
Hi list,
Had a power outage killing the whole cluster. Cephfs will not start at all, but RBD works just fine.
I did have 4 unfound objects that I eventually had to rollback or delete which I don't really understand as I should've had a copy of the those pbjects on the other drives?
2/3 mons and mgrs are damaged but without any errors.
I have loads stored on cephfs so would very much like to get that running as a first priority.
Thanks!
Alex
Info about the home cluster:
I run 23 osds on 3 hosts. 6 of these are a SSD cache layer for the spinning rust, as well as the metadata portion for cephfs which in retrospect might have to be put back on the spinning rust.
# ceph -v
ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable)
# head ceph-mgr.pve21.log.7
2019-10-04 00:00:00.397 7fee56df3700 -1 received  signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw  (PID: 193052) UID: 0
2019-10-04 00:00:00.573 7fee44af1700  0 ms_deliver_dispatch: unhandled message 0x55855f6b7500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901
2019-10-04 00:00:00.573 7fee545ee700  1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory
2019-10-04 00:00:01.553 7fee43aef700  0 log_channel(cluster) log [DBG] : pgmap v2680: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 4 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%)
2019-10-04 00:00:01.573 7fee44af1700  0 ms_deliver_dispatch: unhandled message 0x55855e486380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901
2019-10-04 00:00:01.573 7fee545ee700  1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory
2019-10-04 00:00:02.573 7fee44af1700  0 ms_deliver_dispatch: unhandled message 0x55855e4b5500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901
2019-10-04 00:00:02.573 7fee545ee700  1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory
2019-10-04 00:00:03.553 7fee43aef700  0 log_channel(cluster) log [DBG] : pgmap v2681: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 2 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%)
2019-10-04 00:00:03.573 7fee44af1700  0 ms_deliver_dispatch: unhandled message 0x55855e3b0380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901
# head ceph-mon.pve21.log.7
2019-10-04 00:00:00.389 7f7c25b52700 -1 received  signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  (PID: 193051) UID: 0
2019-10-04 00:00:00.397 7f7c25b52700 -1 received  signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw  (PID: 193052) UID: 0
2019-10-04 00:00:00.573 7f7c1f345700  0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1
2019-10-04 00:00:00.573 7f7c1f345700  0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch
2019-10-04 00:00:01.573 7f7c1f345700  0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1
2019-10-04 00:00:01.573 7f7c1f345700  0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch
2019-10-04 00:00:02.573 7f7c1f345700  0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1
2019-10-04 00:00:02.573 7f7c1f345700  0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch
2019-10-04 00:00:03.573 7f7c1f345700  0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1
2019-10-04 00:00:03.573 7f7c1f345700  0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch
# head ceph-mds.pve21.log.7
2019-10-04 00:00:00.389 7f1b2f1b5700 -1 received  signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  (PID: 193051) UID: 0
2019-10-04 00:00:00.397 7f1b2f1b5700 -1 received  signal: Hangup from  (PID: 193052) UID: 0
2019-10-04 00:00:04.881 7f1b319ba700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839bb200 0x556f838d4000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby
2019-10-04 00:00:06.157 7f1b321bb700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0)
2019-10-04 00:00:06.157 7f1b321bb700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=6 cs=4 l=0).handle_message_footer missed message?  skipped from seq 0 to 2
2019-10-04 00:01:19.167 7f1b311b9700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby
2019-10-04 00:01:23.555 7f1b311b9700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.22:0/2875552603 conn(0x556f839bda80 0x556f837f9000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby
2019-10-04 00:02:08.768 7f1b311b9700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2427365808 conn(0x556f839bd180 0x556f83e9d800 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby
2019-10-04 00:02:20.140 7f1b311b9700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby
2019-10-04 00:02:21.420 7f1b319ba700  0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839e0480 0x556f83d7f000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0)
------------------------------
Subject: Digest Footer
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
------------------------------
End of ceph-users Digest, Vol 81, Issue 19
******************************************
-- 
Jörg Kastning
Plattformen und Serverdienste
BITS - Bielefelder IT-Servicezentrum

Universität Bielefeld
Universitätsstraße 25
33615 Bielefeld
Telefon: +49 521 106-12520
E-Mail: joerg.kastning@uni-bielefeld.de
Webseite: http://www.uni-bielefeld.de/bits/