23 Dec
2019
23 Dec
'19
9:55 p.m.
Hi,
After the upgrade to 13.2.8 deep-scrub has a big impact on client IO:
loads of SLOW_OPS and high latency. We hardly ever had SLOW_OPS, but
since the upgrade the impact is so big that we even have OSDs marking
each other out (OSD op thread timeout) multiple times during the scrub
window. Plenty of CPU / RAM / IOPS left, hardly any load on these OSD
servers. Has there anything changed in this release that can explain
this behaviour?
Besides this the impact of rebalance is very severe as well. With only
the balancer remapping a couple of PGs at a time there are loads of
(MDS_)SLOW_OPS. This morning the cephfs metadata pool got rebalanced ...
and that triggered a lot of SLOW_OPS. One particular OSD was pegged at
1000% CPU for more than half an hour (not doing that much IO): that's 10
cores going full throttle! After a restart this issue was gone.
Thanks,
Stefan
--
| BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
25 Dec
25 Dec
9:12 p.m.
I encountered persistent SLOW_OPS just a few days ago on a recently upgraded 13.2.8
cluster, which has an SSD pool and an HDD pool. All OSDs are Bluestore, we're not
using separate journal / DB volumes. The HDD pool is more or less used for cold storage,
so performance is not critical.
One OSD in particular (HDD) was reporting the SLOW_OPS. I suspected that the drive was on
the way out, but SMART stats looked ok, and there were no IO errors reported in the kernel
log. Restarting that OSD helped initially, but eventually the SLOW_OPS starting to pile up
again.
We have a fair number of VMs running from RBDs, most of them on the SSD pool, but a few on
HDD. Most of the VMs are configured with a weekly fstrim cronjob, and we have QEMU
configured to pass the DISCARD commands down to Ceph. One VM however, which has a bunch of
50 GB files as part of a Bareos setup (fork of Bacula), has the filesystem mounted with
discard option, so it will trim immediately when files are deleted. I tracked the SLOW_OPS
to a time period during which that VM was recycling (i.e., deleting & trimming) some
of these large 50 GB files. In other words, it seems that there might be a performance
regression in deleting large numbers of rados objects at once.
9 Jan
9 Jan
7:26 p.m.
New subject: deep-scrub / backfilling: large amount of SLOW_OPS after upgrade to 13.2.8
Hi,
Quoting Stefan Kooman (stefan(a)bit.nl):
Hi,
After the upgrade to 13.2.8 deep-scrub has a big impact on client IO:
loads of SLOW_OPS and high latency. We hardly ever had SLOW_OPS, but
since the upgrade the impact is so big that we even have OSDs marking
each other out (OSD op thread timeout) multiple times during the scrub
window. Plenty of CPU / RAM / IOPS left, hardly any load on these OSD
servers. Has there anything changed in this release that can explain
this behaviour?
Besides this the impact of rebalance is very severe as well. With only
the balancer remapping a couple of PGs at a time there are loads of
(MDS_)SLOW_OPS. This morning the cephfs metadata pool got rebalanced ...
and that triggered a lot of SLOW_OPS. One particular OSD was pegged at
1000% CPU for more than half an hour (not doing that much IO): that's 10
cores going full throttle! After a restart this issue was gone.
We can now also trigger SLOW_OPS on a bunch of OSDs when we do a "rbd du
-p $POOL", something that has never been an issue. The images in
the rbd pools have the following features enabled: layering,
exclusive-lock, object-map, fast-diff, deep-flatten.
Has there anything changed in 13.2.8 that affects these kind of
operations?
Gr. Stefan
--
| BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
22 Jan
22 Jan
3:32 a.m.
New subject: deep-scrub / backfilling: large amount of SLOW_OPS after upgrade to 13.2.8
Quoting Stefan Kooman (stefan(a)bit.nl):
We can now also trigger SLOW_OPS on a bunch of OSDs
when we do a "rbd du
-p $POOL", something that has never been an issue. The images in
the rbd pools have the following features enabled: layering,
exclusive-lock, object-map, fast-diff, deep-flatten.
Has there anything changed in 13.2.8 that affects these kind of
operations?
Besides the upgrade to 13.2.8 we also configged a different CRUSH rule
for our cephfs metadata pool. It currently contains 512 PGs and 437M
objects. We moved all those objects (by far the most objects of any pool
in the cluster) to a subset of the OSD nodes with NVMe. This turned out
not to be a good decision. Instead of faster responses we hit the issues
mentioned in this thread, i.e. SLOW_OPS during "rbd du", deep-scrubs and
osd worker thread timeouts. We likely introduced PG lock contention on
the OSDs, and as those OSDs were also involved in other pools the whole
cluster suffered from this. After reverting the rule the cluster behaves
way more predictable again.
Lessons learned:
- spread the workload as much as possible.
- Rocksdb compaction / house keeping can take *a lot* of CPU (up to 13
CPU cores per OSD!). After moving those PGs it will take hours (and a
lot of load / slow ops) before the cluster normalizes again.
Sorry for the FUD,
Gr. Stefan
--
| BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
1564
days inactive
1593
days old
3 comments
2 participants
participants (2)
-
Daniel Swarbrick -
Stefan Kooman