- ceph-users - lists.ceph.io

Ceph health status reports: Reduced data availability and this is resulting in slow requests are blocked
by Thomas 15 Oct '19

15 Oct '19

Hi, I want to use balancer mode "upmap" for all pools. This mode is currently enable for pool "hdb_backup" with ~600TB used space. root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B cephfs_data 1.1 TiB 89592 0 268776 0 0 0 0 0 B 43443 144 GiB 0 B 0 B cephfs_metadata 311 MiB 48 0 144 0 0 0 6 6 KiB 7465 106 MiB 0 B 0 B hdb_backup 585 TiB 51077985 0 153233955 0 0 0 12577024 4.3 TiB 281002173 523 TiB 0 B 0 B hdd 6.3 TiB 585051 0 1755153 0 0 0 4420255 69 GiB 8219453 1.2 TiB 0 B 0 B root@ld3955:~# ceph osd lspools 11 hdb_backup 51 hdd 52 backup 57 cephfs_data 58 cephfs_metadata I started with pool "cephfs_metadata" that allocated comparable low data. At the same moment when I executed ceph config set mgr mgr/balancer/pool_ids 11,52,58 Ceph status shows increasing number of Reduced data availability: <x> pg inactive, <y> pg peering And the number of <z> slow requests are blocked > 32 sec was increasing heavily up to 180. Observing Ceph log (ceph -w) it becomes clear that there's a correlation between pg inactive and slow requests are block in my cluster. How can I start analysis why the cluster reports slow requests? THX

1 0

Ceph health status reports: subtrees have overcommitted pool target_size_ratio + subtrees have overcommitted pool target_size_bytes
by Thomas 15 Oct '19

15 Oct '19

Hi, checking my Ceph health status I get this warning: 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio The details are as follows: Pools ['hdb_backup'] overcommit available storage by 1.288x due to target_size_bytes 0 on pools [] However, the relevant settings POOL_TARGET_SIZE_RATIO_OVERCOMMITTED according to health checks <https://docs.ceph.com/docs/master/rados/operations/health-checks/> are not enabled: root@ld3955:~# ceph osd pool get hdb_backup target_size_ratio Error ENOENT: option 'target_size_ratio' is not set on pool 'hdb_backup' root@ld3955:~# ceph osd pool get hdb_backup target_size_bytes Error ENOENT: option 'target_size_bytes' is not set on pool 'hdb_backup' Therefore I would consider this warning as a bug. Regards Thomas

1 0

Constant write load on 4 node ceph cluster
by Ingo Schmidt 14 Oct '19

14 Oct '19

Hi all We have a 4 node ceph cluster that runs generally fine. It is the storage backend for our virtualization cluster with Proxmox, that runs about 40 virtual machines (80% various Linux Servers). Now that we have implemented monitoring, i see that there is a quite constant write load of about 4-10MB/s throughout the whole day and even weekends, while read load clearly is bound to daily work hours. I am wondering whats up with this. Is this normal? Outside working hours, there is almost no read load, well under 1MB/s, while write Load stays relatively constant at around 5MB/s. During rebalancing of the Cluster, normal write load decreases significantly to under 1MB/s. Overall performance is not greatly affected by rebalancing as it is tuned to relatively slow backfills (osd_max_backfills = '3' | osd_recovery_max_active = '3') After rebalance has been finished, the average write load of about 5MB/s reappears. I have attached a little screenshot for reference. Can someone shed a little light on this? Thanks. kind regards Ingo Schmidt IT-Department Island Municipality Langeoog with in-house operations Tourismusservice and Schiffahrt Hauptstraße 28 26465 Langeoog Deutschland [ http://www.langeoog.de/ ]

3 3

object goes missing in bucket
by Benjamin.Zieglmeier 14 Oct '19

14 Oct '19

Hey all, Experiencing an odd issue over the last week or so with a single bucket in a Ceph Luminous (12.2.11) cluster. We occasionally get a complaint from the owner of one bucket (bucket1) that a single object they have written has gone missing. If we list the bucket, the object is indeed missing, and is not present if we look for it using `radosgw-admin bi list –bucket=bucket1`. If we dump the pool the object is placed using (rados -p <poolname> ls ) we can see the object is present in the pool. Looking in the logs we are having a hard time finding any errors, and if we look at the rgw logs we can see the PUT, followed by a couple GETs and HEADs (we believe this to be client app verification) but never a DELETE. This incident has happened a couple of times over the past week, but only to this bucket. The bucket in question does average anywhere from 100-200ops per second based on the rgw usage logs. Any assistance in knowing where to look to see why these objects might be disappearing from the index would be greatly appreciated! -Ben

1 0

CephFS and 32-bit inode numbers
by Dan van der Ster 14 Oct '19

14 Oct '19

Hi all, One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3. I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3. Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?) Thanks! Dan

1 0

RadosGW max worker threads
by Benjamin.Zieglmeier 12 Oct '19

12 Oct '19

Hello all, Looking for guidance on the recommended highest setting (or input on experiences from users who have a high setting) for rgw_thread_pool_size. We are running multiple Luminous 12.2.11 clusters with usually 3-4 RGW daemons in front of them. We set our rgw_thread_pool_size at 512 out of the gate, and run civetweb. We had occasional service outages in one of our clusters this week and determined the rgws were running out of available threads to handle requests. We doubled our thread pool size to 1024 on each rgw and everything has been ok so far. What, if any, would be the high-end limit to set for rgw_thread_pool_size? I’ve been unable to find anything in the documentation or the user list that depicts anything higher than the default 100 threads. Thanks, Ben

4 5

Nautilus: PGs stuck remapped+backfilling
by Eugen Block 11 Oct '19

11 Oct '19

Hi all, I have a strange issue with backfilling and I'm not sure what the cause is. It's a Nautilus cluster (upgraded) that has an SSD cache tier for OpenStack and CephFS metadata residing on the same SSDs, there were three SSDs in total. Today I added two new SSDs (NVMe) (osd.15, osd.16) to be able to shutoff one old server that has only one SSD-OSD left (osd.20). Setting the crush weight of osd.20 to 0 (and adjusting the weight of the remaining SSDs for an even distribution) leaves 3 PGs in active+remapped+backfilling state. I don't understand why the remaining PGs aren't backfilling, the crush rule is quite simple (all ssd pools are replicated with size 3). The backfilling PGs are all from the cephfs_metadata pool. Although there are 4 SSDs for 3 replicas the backfilling still should finish, right? Can anyone share their thoughts why 3 PGs can't be recovered? If more information about the cluster is required please let me know. Regards, Eugen ceph01:~ # ceph osd pool ls detail | grep meta pool 36 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 283362 flags hashpspool,nodelete,nodeep-scrub stripe_width 0 application cephfs ceph01:~ # ceph pg dump | grep remapp dumped all 36.b 28306 0 0 28910 0 8388608 101408323 219497 3078 3078 active+remapped+backfilling 2019-10-10 13:36:27.427527 284595'98565869 284595:254216941 [15,16,9] 15 [20,9,10] 20 284427'98489406 2019-10-10 00:16:02.682911 284089'98003598 2019-10-06 16:03:27.558267 0 36.d 28087 0 0 25327 0 26375382 106722204 231020 3041 3041 active+remapped+backfilling 2019-10-10 13:36:27.404739 284595'97933905 284595:252878816 [16,15,9] 16 [20,9,10] 20 284427'97887652 2019-10-10 04:13:29.371905 284259'97502135 2019-10-07 20:06:43.304593 0 36.4 28060 0 0 28406 0 8389242 104059103 225188 3061 3061 active+remapped+backfilling 2019-10-10 13:36:27.440390 284595'105299618 284595:312976619 [16,9,15] 16 [20,9,10] 20 284427'105218591 2019-10-10 00:18:07.924006 284089'104696098 2019-10-06 16:20:17.123149 0 rule ssd_ruleset { id 1 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 0 type host step emit } This is the relevant part of the osd tree: ceph01:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 34.21628 root default -31 11.25406 host ceph01 25 hdd 3.59999 osd.25 up 1.00000 1.00000 26 hdd 3.59999 osd.26 up 1.00000 1.00000 27 hdd 3.59999 osd.27 up 1.00000 1.00000 15 ssd 0.45409 osd.15 up 1.00000 1.00000 -34 11.25406 host ceph02 0 hdd 3.59999 osd.0 up 1.00000 1.00000 28 hdd 3.59999 osd.28 up 1.00000 1.00000 29 hdd 3.59999 osd.29 up 1.00000 1.00000 16 ssd 0.45409 osd.16 up 1.00000 1.00000 -37 10.79999 host ceph03 31 hdd 3.59999 osd.31 up 1.00000 1.00000 32 hdd 3.59999 osd.32 up 1.00000 1.00000 33 hdd 3.59999 osd.33 up 1.00000 1.00000 -24 0.45409 host san01-ssd 10 ssd 0.45409 osd.10 up 1.00000 1.00000 -23 0.45409 host san02-ssd 9 ssd 0.45409 osd.9 up 1.00000 1.00000 -22 0 host san03-ssd 20 ssd 0 osd.20 up 1.00000 1.00000 Don't be confused because of the '-ssd' suffix, we're using crush location hooks. This is the current PG distribution on the SSDs: ceph01:~ # ceph osd df | grep -E "^15 |^16 |^ 9|^10 |^20 " 15 ssd 0.45409 1.00000 465 GiB 34 GiB 32 GiB 1.2 GiB 857 MiB 431 GiB 7.29 0.22 27 up 16 ssd 0.45409 1.00000 465 GiB 37 GiB 34 GiB 1.5 GiB 964 MiB 428 GiB 7.87 0.23 31 up 10 ssd 0.45409 1.00000 745 GiB 27 GiB 25 GiB 1.7 GiB 950 MiB 718 GiB 3.65 0.11 29 up 9 ssd 0.45409 1.00000 745 GiB 34 GiB 32 GiB 1.3 GiB 902 MiB 711 GiB 4.60 0.14 30 up 20 ssd 0 1.00000 894 GiB 8.2 GiB 4.3 GiB 1.5 GiB 2.4 GiB 886 GiB 0.91 0.03 3 up Current ceph status: ceph01:~ # ceph -s cluster: id: 655cb05a-435a-41ba-83d9-8549f7c36167 health: HEALTH_OK services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 2d) mgr: ceph03(active, since 8d), standbys: ceph01, ceph02 mds: cephfs:1 {0=mds01=up:active} 1 up:standby-replay 1 up:standby osd: 26 osds: 26 up (since 66m), 26 in (since 66m); 3 remapped pgs data: pools: 8 pools, 264 pgs objects: 4.96M objects, 5.0 TiB usage: 16 TiB used, 31 TiB / 47 TiB avail pgs: 115745/14865558 objects misplaced (0.779%) 261 active+clean 3 active+remapped+backfilling io: client: 903 KiB/s rd, 8.8 MiB/s wr, 85 op/s rd, 266 op/s wr recovery: 0 B/s, 61 keys/s, 12 objects/s cache: 4.2 MiB/s flush, 15 MiB/s evict, 0 op/s promote

3 7

help
by Jörg Kastning 11 Oct '19

11 Oct '19

Am 11.10.2019 um 09:21 schrieb ceph-users-request(a)ceph.io: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: MDS rejects clients causing hanging mountpoint on linux kernel client > (Manuel Riel) > 2. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout") > (Janne Johansson) > 3. Re: Nautilus: PGs stuck remapped+backfilling (Eugen Block) > 4. Re: HeartbeatMap FAILED assert(0 == "hit suicide timeout") (潘东元) > 5. Nautilus power outage - 2/3 mons and mgrs dead and no cephfs > (Alex L) > > > ---------------------------------------------------------------------- > > Date: Thu, 10 Oct 2019 11:02:28 +0800 > From: Manuel Riel <manu(a)snapdragon.cc> > Subject: [ceph-users] Re: MDS rejects clients causing hanging > mountpoint on linux kernel client > To: ukernel(a)gmail.com > Cc: ceph-users(a)ceph.io > Message-ID: <3098DC39-AED4-44F0-B9CB-44B3468285BB(a)snapdragon.cc> > Content-Type: multipart/signed; > boundary="Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D"; > protocol="application/pkcs7-signature"; micalg=sha-256 > > > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D > Content-Transfer-Encoding: quoted-printable > Content-Type: text/plain; > charset=us-ascii > > I noticed a similar issue tonight. Still looking into the details, but = > here are the client logs I=20 > > Oct 9 19:27:59 mon5-cx kernel: libceph: mds0 ***:6800 socket closed = > (con state OPEN) > Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 connection reset > Oct 9 19:28:01 mon5-cx kernel: libceph: reset on mds0 > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 closed our session > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect start > Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect denied > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109011c9980 1099517142146 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91096cc788d0 1099517142307 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9107da741f10 1099517142312 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109d5c40e60 1099517141612 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9108c9337da0 1099517142313 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff9109d5c70340 1099517141565 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff910955acf810 1099517141792 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91095ff56cf0 1099517142006 > Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state = > for ffff91096cc7f280 1099517142309 > Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 socket closed = > (con state NEGOTIATING) > Oct 9 19:28:02 mon5-cx kernel: ceph: mds0 rejected session > Oct 9 19:28:02 mon5-cx monit: Lookup for '/srv/repos' filesystem failed = > -- not found in /proc/self/mounts > Oct 9 19:28:02 mon5-cx monit: Filesystem '/srv/repos' not mounted > Oct 9 19:28:02 mon5-cx monit: 'repos' unable to read filesystem = > '/srv/repos' state > ... > Oct 9 19:28:09 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 19:28:24 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 19:28:39 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > ... > Oct 9 21:27:09 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 21:27:24 mon5-cx kernel: ceph: get_quota_realm: ino = > (1.fffffffffffffffe) null i_snap_realm > Oct 9 21:27:27 mon5-cx monit: Lookup for '/srv/repos' filesystem failed = > -- not found in /proc/self/mounts > Oct 9 21:27:27 mon5-cx monit: Filesystem '/srv/repos' not mounted > Oct 9 21:27:27 mon5-cx monit: 'repos' unable to read filesystem = > '/srv/repos' state > Oct 9 21:27:27 mon5-cx monit: 'repos' trying to restart > > > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D > Content-Disposition: attachment; > filename=smime.p7s > Content-Type: application/pkcs7-signature; > name=smime.p7s > Content-Transfer-Encoding: base64 > > MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCCx4w > ggUwMIIEGKADAgECAhEAuxgm/OfWx7qPyHlON6sLpzANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE > BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG > A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl > bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgxMDMxMDAwMDAwWhcNMTkxMDMxMjM1 > OTU5WjAjMSEwHwYJKoZIhvcNAQkBFhJtYW51QHNuYXBkcmFnb24uY2MwggEiMA0GCSqGSIb3DQEB > AQUAA4IBDwAwggEKAoIBAQDSCVHH0d9wqo1reA0PKmK8x88NJQcffMt3EBYnqrIWt0rLypsDWYBE > uTUl+WAppxa2BI172rAexkkVX1J4bM8S5P5NRInp1JOlo+s/WdXM1f0IGD4qfCUtOBM+05rdvKMS > +PBAdBkAR4XnDe26fk7XNgBLbhVifw5TEn3UBaQWPSmr6OV7WwoysNTyR3X1tKzVHTCrJv9JWfSA > vlU2NIpo5XOa9X1Zs7uxlwO7LRgsC50AFrLHVYfsqIl9zr5iBlP2lBsZVciqhv0/tszThLBjS3di > fbSyJVzAhBQTf2KCvewJbkcZr7JE8vV4sXUqrvc1voyN3sf0G/qSTy0rCl8vAgMBAAGjggHoMIIB > 5DAfBgNVHSMEGDAWgBSCr2yM+MX+lmF86B89K3FIXsSLwDAdBgNVHQ4EFgQUMzEd3LCtdsH3VrT1 > WB6sUaWGU6UwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwIAYDVR0lBBkwFwYIKwYBBQUH > AwQGCysGAQQBsjEBAwUCMBEGCWCGSAGG+EIBAQQEAwIFIDBGBgNVHSAEPzA9MDsGDCsGAQQBsjEB > AgEBATArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBaBgNVHR8E > UzBRME+gTaBLhklodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9SU0FDbGllbnRBdXRoZW50 > aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3JsMIGLBggrBgEFBQcBAQR/MH0wVQYIKwYBBQUHMAKG > SWh0dHA6Ly9jcnQuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5k > U2VjdXJlRW1haWxDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAd > BgNVHREEFjAUgRJtYW51QHNuYXBkcmFnb24uY2MwDQYJKoZIhvcNAQELBQADggEBACmWfZZiKLiT > s7YxuZc2KvsmjlDqIUK2TF6b1LqA4CMFiDwwNDUvxoOClhZJFpDDfnw4w86fDheIM5fPinf+0wif > 8YAphESKyVIZPDl7t3D4zQmU5Ytsq44iCS+TTAQXcWbz7Qe6tBQWZvffKNb6H95blJj0t86Ga0wD > huUXMZbjFMOYcRLGsllUe5H2qJUs0CY/Y9MGGXjcy2HFbhStbuUyj8rwi9ynCvz3LFS30J9MXhYr > c+yu9tZiHZZdjl7gP7RCez+IDZme1TfVYu6KBXxrvkZMRU9+KILtcXmIvMwzelp8KC9Ha9AtXZNd > g9wSFvwRAOFtAEJ9+m0fhdzmq1cwggXmMIIDzqADAgECAhBqm+E4O/8ra58B1dm4p1JWMA0GCSqG > SIb3DQEBDAUAMIGFMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw > DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDErMCkGA1UEAxMiQ09N > T0RPIFJTQSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0xMzAxMTAwMDAwMDBaFw0yODAxMDky > MzU5NTlaMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD > VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09NT0RP > IFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZI > hvcNAQEBBQADggEPADCCAQoCggEBAL6znlesKHZ1QBbHOAOY08YYdiFQ8yV5C0y1oNF9Olg+nKcx > Lqf2NHbZhGra0D00SOTq9bus3/mxgUsg/Wh/eXQ0pnp8tZ8XZWAnlyKMpjL+qUByRjXCA6RQyDMq > VaVUkbIr5SU0RDX/kSsKwer3H1pT/HUrBN0X8sKtPTdGX8XAWt/VdMLBrZBlgvnkCos+KQWWCo63 > OTTqRvaq8aWccm+KOMjTcE6s2mj6RkalweyDI7X+7U5lNo6jzC8RTXtVV4/Vwdax720YpMPJQaDa > ElmOupyTf1Qib+cpukNJnQmwygjD8m046DQkLnpXNCAGjuJy1F5NATksUsbfJAr7FLUCAwEAAaOC > ATwwggE4MB8GA1UdIwQYMBaAFLuvfgI9+qbxPISOre44mOzZMjLUMB0GA1UdDgQWBBSCr2yM+MX+ > lmF86B89K3FIXsSLwDAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAE > CjAIMAYGBFUdIAAwTAYDVR0fBEUwQzBBoD+gPYY7aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09N > T0RPUlNBQ2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwcQYIKwYBBQUHAQEEZTBjMDsGCCsGAQUF > BzAChi9odHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FBZGRUcnVzdENBLmNydDAkBggr > BgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQB4XLKB > KDRPPO5fVs6fl1bsj6JrF/bz9kkIBtTYLzXN30D+03Hj6OxCDBEaIeNmsBhrJmuubvyE7HtoSmR8 > 09AgcYboW+rcTNZ/8u/Hv+GTrNI/AhqX2/kiQNxmgUPt/eJPs92Qclj0HnVyy9TnSvGkSDU7I5Px > +TbO+88G4zipA2psZaWeEykgzClZlPz1FjTCkk77ZXp5cQYYexE6zeeN4/0OqqoAloFrjAF4o50Y > JafX8mnahjp3I2Y2mkjhk0xQfhNqbzlLWPoT3m7j7U26u7zg6swjOq8hITYc3/np5tM5aVyu6t99 > p17bTbY7+1RTWBviN9YJzK8HxzObXYWBf/L+VGOYNsQDTxAk0Hbvb1j6KjUhg7fO294F29QIhhmi > NOr84JHoy+fNLpfvYc/Q9EtFOI5ISYgOxLk3nD/whbUe9rmEQXLp8MB933Ij474gwwCPUpwv9mj2 > PMnXoc7mbrS22XUSeTwxCTP9bcmUdp4jmIoWfhQm7X9w/Zgddg+JZ/YnIHOwsGsaTUgj7fIvxqit > h7DoJC91WJ8Lce3CVJqb1XWeKIJ84F7YLXZN0oa7TktYgDdmQVxYkZo1c5noaDKH9Oq9cbm/vOYR > UM1cWcef20Wkyk5S/GFyyPJwG0fR1nRas3DqAf4cXxMiEKcff7PNa4M3RGTqH0pWR8p6EjGCA8ow > ggPGAgEBMIGtMIGXMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAw > DgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE9MDsGA1UEAxM0Q09N > T0RPIFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIRALsYJvzn > 1se6j8h5TjerC6cwDQYJYIZIAWUDBAIBBQCgggHtMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw > HAYJKoZIhvcNAQkFMQ8XDTE5MTAxMDAzMDIyOVowLwYJKoZIhvcNAQkEMSIEIOo1zdaTl2T2mCM4 > ttyllcqgdZ0MagVCBglhDdSmEkM8MIG+BgkrBgEEAYI3EAQxgbAwga0wgZcxCzAJBgNVBAYTAkdC > MRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoT > EUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNh > dGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfWx7qPyHlON6sLpzCBwAYLKoZIhvcNAQkQ > AgsxgbCgga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO > BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQDEzRDT01P > RE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEAuxgm/OfW > x7qPyHlON6sLpzANBgkqhkiG9w0BAQEFAASCAQCPNgVk/I2nxe1niT2avKeoKQiQfWhGIsFe6V3V > 8E8WDEGfQwQX0ApcF0sm465wPRuu+K3n3mMo7z2mNZtOyfpHeAYElsZsLOV/pMgMWPNYISwFPu2K > J7m9o4anPff7JT0BTMf6XbsFTezCUCIQotuGur/6KquXOCOddX8xYbhcSAuwlAPuhBzCnn42goU0 > tybdFZqmcjJl+5uRYYHmr68zhcA7XFF5tFTzIaq4krMDMfk0aKpKNTL5wIhfFwChe9FAUgsHT25h > p3bQkDlKZgfeWeqYtzkOoFCqpznWR65EqKaMmGaemWCQuOTMMP+8lVC7HIlxbNmbnEYH3sbdldZJ > AAAAAAAA > --Apple-Mail=_C46A3FE0-49DB-4302-8B1B-BEC9BD6D398D-- > > ------------------------------ > > Date: Thu, 10 Oct 2019 15:16:46 +0200 > From: Janne Johansson <icepic.dz(a)gmail.com> > Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide > timeout") > To: 潘东元 <dongyuanpan0(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CAA6-MF-SpecCptAd0q_GF0cHSUG4G4W8pZmiXEi1DP0kAwvrHw(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000004ebc3105948e3713" > > --0000000000004ebc3105948e3713 > Content-Type: text/plain; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > Den tors 10 okt. 2019 kl 15:12 skrev =E6=BD=98=E4=B8=9C=E5=85=83 <dongyuanp= > an0(a)gmail.com>: > >> hi all, >> my osd hit suicide timeout. >> >> common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeout") >> >> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) >> > > >> can you give some advice on troubleshooting? >> > > It is a very old release, chances are large whatever issue you get here > might have been fixed in the last 5 years. > > --=20 > May the most significant bit of your life be positive. > > --0000000000004ebc3105948e3713 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= > <div dir=3D"ltr" class=3D"gmail_attr">Den tors 10 okt. 2019 kl 15:12 skrev = > =E6=BD=98=E4=B8=9C=E5=85=83 <<a href=3D"mailto:dongyuanpan0@gmail.com">d= > ongyuanpan0(a)gmail.com</a>>:<br></div><blockquote class=3D"gmail_quote" s= > tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad= > ding-left:1ex">hi all,<br> > =C2=A0 =C2=A0 my osd hit suicide timeout.<br><br> > common/HeartbeatMap.cc: 79: FAILED assert(0 =3D=3D "hit suicide timeou= > t")<br> > <br> > =C2=A0ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)<br> > </blockquote><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"ma= > rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:= > 1ex"> > can you give some advice on troubleshooting?<br></blockquote></div><div><br= >> </div><div>It is a very old release, chances are large whatever issue you = > get here might have been fixed in the last 5 years.</div><div><br></div>-- = > <br><div dir=3D"ltr" class=3D"gmail_signature">May the most significant bit= > of your life be positive.<br></div></div> > > --0000000000004ebc3105948e3713-- > > ------------------------------ > > Date: Thu, 10 Oct 2019 13:57:38 +0000 > From: Eugen Block <eblock(a)nde.ag> > Subject: [ceph-users] Re: Nautilus: PGs stuck remapped+backfilling > To: ceph-users(a)ceph.io > Message-ID: > <20191010135738.Horde.-gF9xNAEG9yFgxVbn83rdTU(a)webmail.nde.ag> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > Please ignore my email, the PGs have eventually recovered, it just > took way more time than expected or observed for the other PGs. I'll > try to be more patient next time. ;-) > > ------------------------------ > > Date: Fri, 11 Oct 2019 14:12:49 +0800 > From: 潘东元 <dongyuanpan0(a)gmail.com> > Subject: [ceph-users] Re: HeartbeatMap FAILED assert(0 == "hit suicide > timeout") > To: Janne Johansson <icepic.dz(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CANkQ9Li-ay1xVUkKqGtM1yq_PbSHr57GXV35aRLvOQtaZwE-xQ(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > I‘m pretty sure, this issue here is that there is a communication > issue between the osds. > logged over and over again report initiating reconnect. > > I looked at my network,and have dropped packets,this is probably the > tcp queue full at osd daemon listen port. > My cluster had 21 nodes, with 5 osds on each node. > > [root@node-4 ~]# netstat -nat | grep -w "6801" | grep -w tcp | grep > ESTABLISHED | wc -l > 271 > > [root@node-4 ~]# sysctl -a | grep somax > net.core.somaxconn = 128 > > this will cause the tcp connection to reset. > > so,i will increase this parameter! > > ------------- > Regards! > > Janne Johansson <icepic.dz(a)gmail.com> 于2019年10月10日周四下午9:16写道： >> >> >> >> Den tors 10 okt. 2019 kl 15:12 skrev 潘东元 <dongyuanpan0(a)gmail.com>: >>> >>> hi all, >>> my osd hit suicide timeout. >>> >>> common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") >>> >>> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) >> >> >>> >>> can you give some advice on troubleshooting? >> >> >> It is a very old release, chances are large whatever issue you get here might have been fixed in the last 5 years. >> >> -- >> May the most significant bit of your life be positive. > > ------------------------------ > > Date: Fri, 11 Oct 2019 07:21:40 -0000 > From: "Alex L" <alexut.voicu(a)gmail.com> > Subject: [ceph-users] Nautilus power outage - 2/3 mons and mgrs dead > and no cephfs > To: ceph-users(a)ceph.io > Message-ID: <157077850022.1193.1225987085352609330@mailman-web> > Content-Type: text/plain; charset="utf-8" > > Hi list, > Had a power outage killing the whole cluster. Cephfs will not start at all, but RBD works just fine. > I did have 4 unfound objects that I eventually had to rollback or delete which I don't really understand as I should've had a copy of the those pbjects on the other drives? > > 2/3 mons and mgrs are damaged but without any errors. > > I have loads stored on cephfs so would very much like to get that running as a first priority. > > Thanks! > Alex > > Info about the home cluster: > I run 23 osds on 3 hosts. 6 of these are a SSD cache layer for the spinning rust, as well as the metadata portion for cephfs which in retrospect might have to be put back on the spinning rust. > > # ceph -v > ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable) > > # head ceph-mgr.pve21.log.7 > 2019-10-04 00:00:00.397 7fee56df3700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 > 2019-10-04 00:00:00.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855f6b7500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:00.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:01.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2680: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 4 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) > 2019-10-04 00:00:01.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e486380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:01.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:02.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e4b5500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > 2019-10-04 00:00:02.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory > 2019-10-04 00:00:03.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2681: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 2 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) > 2019-10-04 00:00:03.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e3b0380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 > > # head ceph-mon.pve21.log.7 > 2019-10-04 00:00:00.389 7f7c25b52700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 > 2019-10-04 00:00:00.397 7f7c25b52700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 > 2019-10-04 00:00:00.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:00.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:01.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:01.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:02.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:02.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > 2019-10-04 00:00:03.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 > 2019-10-04 00:00:03.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch > > > # head ceph-mds.pve21.log.7 > 2019-10-04 00:00:00.389 7f1b2f1b5700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 > 2019-10-04 00:00:00.397 7f1b2f1b5700 -1 received signal: Hangup from (PID: 193052) UID: 0 > 2019-10-04 00:00:04.881 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839bb200 0x556f838d4000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby > 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0) > 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=6 cs=4 l=0).handle_message_footer missed message? skipped from seq 0 to 2 > 2019-10-04 00:01:19.167 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:01:23.555 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.22:0/2875552603 conn(0x556f839bda80 0x556f837f9000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:02:08.768 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2427365808 conn(0x556f839bd180 0x556f83e9d800 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby > 2019-10-04 00:02:20.140 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby > 2019-10-04 00:02:21.420 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839e0480 0x556f83d7f000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0) > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 81, Issue 19 > ****************************************** > -- Jörg Kastning Plattformen und Serverdienste BITS - Bielefelder IT-Servicezentrum Universität Bielefeld Universitätsstraße 25 33615 Bielefeld Telefon: +49 521 106-12520 E-Mail: joerg.kastning(a)uni-bielefeld.de Webseite: http://www.uni-bielefeld.de/bits/

1 0

HeartbeatMap FAILED assert(0 == "hit suicide timeout")
by 潘东元 11 Oct '19

11 Oct '19

hi all, my osd hit suicide timeout. some log: 2019-10-10 03:53:13.017760 7f1ab886e700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.25:6802/24020795 pipe(0x257eb80 sd=69 :47977 s=2 pgs=287284 cs=41 l=0 c=0x21431760).fault, initiating reconnect 2019-10-10 03:53:13.017799 7f1ab967c700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.25:6802/24020795 pipe(0x257eb80 sd=69 :47977 s=1 pgs=287284 cs=42 l=0 c=0x21431760).fault 2019-10-10 03:53:15.890773 7f1acdec3700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.19:6804/53020865 pipe(0x37537580 sd=59 :60121 s=2 pgs=423672 cs=85 l=0 c=0x21447900).fault, initiating reconnect 2019-10-10 03:53:15.890890 7f1aba288700 0 -- 192.168.1.5:6810/1028846 >> 192.168.1.19:6804/53020865 pipe(0x37537580 sd=59 :60121 s=1 pgs=423672 cs=86 l=0 c=0x21447900).fault 2019-10-10 03:53:16.209368 7f1addc3e700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f1ac29a3700' had timed out after 15 2019-10-10 03:53:16.209382 7f1addc3e700 1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7f1ac29a3700' had suicide timed out after 150 2019-10-10 03:53:16.210765 7f1addc3e700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7f1addc3e700 time 2019-10-10 03:53:16.209415 common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x12b) [0xaf2b6b] 2: (ceph::HeartbeatMap::is_healthy()+0xa7) [0xaf3497] 3: (ceph::HeartbeatMap::check_touch_file()+0x28) [0xaf3988] 4: (CephContextServiceThread::entry()+0x13f) [0xb0353f] 5: (()+0x79d1) [0x7f1ae0b3c9d1] 6: (clone()+0x6d) [0x7f1adfaccb5d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. can you give some advice on troubleshooting?

3 4

Re: MDS rejects clients causing hanging mountpoint on linux kernel client
by Manuel Riel 10 Oct '19

10 Oct '19

I noticed a similar issue tonight. Still looking into the details, but here are the client logs I Oct 9 19:27:59 mon5-cx kernel: libceph: mds0 ***:6800 socket closed (con state OPEN) Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 connection reset Oct 9 19:28:01 mon5-cx kernel: libceph: reset on mds0 Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 closed our session Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect start Oct 9 19:28:01 mon5-cx kernel: ceph: mds0 reconnect denied Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9109011c9980 1099517142146 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff91096cc788d0 1099517142307 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9107da741f10 1099517142312 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9109d5c40e60 1099517141612 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9108c9337da0 1099517142313 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff9109d5c70340 1099517141565 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff910955acf810 1099517141792 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff91095ff56cf0 1099517142006 Oct 9 19:28:01 mon5-cx kernel: ceph: dropping dirty+flushing Fw state for ffff91096cc7f280 1099517142309 Oct 9 19:28:01 mon5-cx kernel: libceph: mds0 ***:6800 socket closed (con state NEGOTIATING) Oct 9 19:28:02 mon5-cx kernel: ceph: mds0 rejected session Oct 9 19:28:02 mon5-cx monit: Lookup for '/srv/repos' filesystem failed -- not found in /proc/self/mounts Oct 9 19:28:02 mon5-cx monit: Filesystem '/srv/repos' not mounted Oct 9 19:28:02 mon5-cx monit: 'repos' unable to read filesystem '/srv/repos' state ... Oct 9 19:28:09 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 19:28:24 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 19:28:39 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm ... Oct 9 21:27:09 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 21:27:24 mon5-cx kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Oct 9 21:27:27 mon5-cx monit: Lookup for '/srv/repos' filesystem failed -- not found in /proc/self/mounts Oct 9 21:27:27 mon5-cx monit: Filesystem '/srv/repos' not mounted Oct 9 21:27:27 mon5-cx monit: 'repos' unable to read filesystem '/srv/repos' state Oct 9 21:27:27 mon5-cx monit: 'repos' trying to restart

1 0