[ceph-users] Re: high latency after maintenance]

Hi Marcel, The peering process is the process used by Ceph OSDs, on a per placement group basis, to agree on the state of that placement on each of the involved OSDs. In your case, 2/3 of the placement group metadata that needs to be agreed upon/checked is on the nodes that did not undergo maintenance. You also need to consider that the acting primary OSD for everything is now hosted on the OSDs that did not undergo any maintenance. This all means that all 'heavy' lifting is done by these nodes until the recovery/backfilling process is completed is done by the nodes that stayed online. Also consider that Ceph will, most likely, execute peering twice per pg. Once when the OSDs start again, and once when the recovery and backfillling is finished. I really don't want to RTFM, but I don't think it is useful to copy it here: https://docs.ceph.com/en/latest/dev/peering/#description-of-the-peering-pro… Peering the process of bringing all of the OSDs that store a Placement Group (PG) into agreement about the state of all of the objects (and their metadata) in that PG. Note that agreeing on the state does not mean that they all have the latest contents. Kind regards, Wout 42on ________________________________________ From: Marcel Kuiper <ceph(a)mknet.nl> Sent: Friday, 6 November 2020 10:23 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: high latency after maintenance] Hi Anthony Thank you for your respons I am looking at the"OSDs highest latency of write operations" panel of the grafana dashboard found in the ceph source in ./monitoring/grafana/dashboards/osds-overview.json. It is a topk graph that uses ceph_osd_op_w_latency_sum / ceph_osd_op_w_latency_count. During normal operations we see sometime latency spikes of 4 seconds max but during the bringing back of the rack we saw a consistent increase in latency for a lot of osds into the 20 seconds range The cluster has 1139 osds total of which we had 5 x 9 - 45 in maintenance We did not throttle the backfilling proces because we succesfully did the same maintenance before on a few occasions for other racks without problems. I will throttle backfills next time we have the same sort of maintenance in the next rack Can you elaborate a bit more what happens exactly during the peering process? I understand that the osds need to catch up. I also see that the nr of scrubs increases a lot when osds are brought back online. Is that part of the peering proces? Thx, Marcel _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io