deep-scrub / backfilling: large amount of SLOW_OPS after upgrade to 13.2.8
Hi, After the upgrade to 13.2.8 deep-scrub has a big impact on client IO: loads of SLOW_OPS and high latency. We hardly ever had SLOW_OPS, but since the upgrade the impact is so big that we even have OSDs marking each other out (OSD op thread timeout) multiple times during the scrub window. Plenty of CPU / RAM / IOPS left, hardly any load on these OSD servers. Has there anything changed in this release that can explain this behaviour? Besides this the impact of rebalance is very severe as well. With only the balancer remapping a couple of PGs at a time there are loads of (MDS_)SLOW_OPS. This morning the cephfs metadata pool got rebalanced ... and that triggered a lot of SLOW_OPS. One particular OSD was pegged at 1000% CPU for more than half an hour (not doing that much IO): that's 10 cores going full throttle! After a restart this issue was gone. Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
I encountered persistent SLOW_OPS just a few days ago on a recently upgraded 13.2.8 cluster, which has an SSD pool and an HDD pool. All OSDs are Bluestore, we're not using separate journal / DB volumes. The HDD pool is more or less used for cold storage, so performance is not critical. One OSD in particular (HDD) was reporting the SLOW_OPS. I suspected that the drive was on the way out, but SMART stats looked ok, and there were no IO errors reported in the kernel log. Restarting that OSD helped initially, but eventually the SLOW_OPS starting to pile up again. We have a fair number of VMs running from RBDs, most of them on the SSD pool, but a few on HDD. Most of the VMs are configured with a weekly fstrim cronjob, and we have QEMU configured to pass the DISCARD commands down to Ceph. One VM however, which has a bunch of 50 GB files as part of a Bareos setup (fork of Bacula), has the filesystem mounted with discard option, so it will trim immediately when files are deleted. I tracked the SLOW_OPS to a time period during which that VM was recycling (i.e., deleting & trimming) some of these large 50 GB files. In other words, it seems that there might be a performance regression in deleting large numbers of rados objects at once.
Hi, Quoting Stefan Kooman (stefan@bit.nl):
Hi,
After the upgrade to 13.2.8 deep-scrub has a big impact on client IO: loads of SLOW_OPS and high latency. We hardly ever had SLOW_OPS, but since the upgrade the impact is so big that we even have OSDs marking each other out (OSD op thread timeout) multiple times during the scrub window. Plenty of CPU / RAM / IOPS left, hardly any load on these OSD servers. Has there anything changed in this release that can explain this behaviour?
Besides this the impact of rebalance is very severe as well. With only the balancer remapping a couple of PGs at a time there are loads of (MDS_)SLOW_OPS. This morning the cephfs metadata pool got rebalanced ... and that triggered a lot of SLOW_OPS. One particular OSD was pegged at 1000% CPU for more than half an hour (not doing that much IO): that's 10 cores going full throttle! After a restart this issue was gone.
We can now also trigger SLOW_OPS on a bunch of OSDs when we do a "rbd du -p $POOL", something that has never been an issue. The images in the rbd pools have the following features enabled: layering, exclusive-lock, object-map, fast-diff, deep-flatten. Has there anything changed in 13.2.8 that affects these kind of operations? Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Quoting Stefan Kooman (stefan@bit.nl):
We can now also trigger SLOW_OPS on a bunch of OSDs when we do a "rbd du -p $POOL", something that has never been an issue. The images in the rbd pools have the following features enabled: layering, exclusive-lock, object-map, fast-diff, deep-flatten.
Has there anything changed in 13.2.8 that affects these kind of operations?
Besides the upgrade to 13.2.8 we also configged a different CRUSH rule for our cephfs metadata pool. It currently contains 512 PGs and 437M objects. We moved all those objects (by far the most objects of any pool in the cluster) to a subset of the OSD nodes with NVMe. This turned out not to be a good decision. Instead of faster responses we hit the issues mentioned in this thread, i.e. SLOW_OPS during "rbd du", deep-scrubs and osd worker thread timeouts. We likely introduced PG lock contention on the OSDs, and as those OSDs were also involved in other pools the whole cluster suffered from this. After reverting the rule the cluster behaves way more predictable again. Lessons learned: - spread the workload as much as possible. - Rocksdb compaction / house keeping can take *a lot* of CPU (up to 13 CPU cores per OSD!). After moving those PGs it will take hours (and a lot of load / slow ops) before the cluster normalizes again. Sorry for the FUD, Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
participants (2)
-
Daniel Swarbrick
-
Stefan Kooman