Hi, the IO should not be blocked for the entire pool but only for those repeering PGs. Warnings during that phase are expected. Gradually reweighting instead of setting an OSD "out" directly is a common practice among ceph users, there should be plenty of threads on this list about that. The downside is of course a lot more manual intervention and possibly (some) redundant data movement. One knob to turn could be to decrease osd_heartbeat_interval (number of seconds between peer pings) from 6 seconds to a lower value (1 second is the minimum, but I probably would start with 5 or 4 seconds). That could (or probably would) increase network traffic significantly, so it's a trade-off. If you have a test cluster with similar behavior, test it there first. Regards, Eugen Zitat von Elias Carter via ceph-users <ceph-users@ceph.io>:

We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and upwards).

Something that I have noticed is that CRUSH changes (marking an OSD as out, or a new OSD joining the cluster) can cause many hundreds of PGs to remap and peer. During this time we get PG_AVAILABILITY health check warnings as the OSDs need to peer with their new PGs before they can serve IOs. RADOS tail latency suffers because IOs sent to a peering PG are blocked until peering completes.

I believe that the more PGs we have, the more PGs need to peer, and the longer IOs are blocked for the pool.

Does anyone have suggestions on how to deal with PG peering blocking IOs?

I was thinking about gradually re-weighting OSDs instead of marking them out or in, but that can result in multiple redundant data moves.

Elias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io