Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
Hey, I've been running a ceph cluster of arm64 SOCs on Luminous for the past year or so, with no major problems. I recently upgraded to 14.2.7, and the stability of the cluster immediately suffered. Seemed like any mon activity was subject to long pauses, and the cluster would hang frequently. Looking at ceph -s, it appeared the cluster was electing new masters very frequently - masters didn't seem to last longer than about 1-2 minutes. Looking further at the mons, two out of three of which are running on relatively slow-performing SD card storage on these SoCs, I saw them absolutely maxing out the root device's IO with writes. Logs show rocksdb constantly running compactions. I temporarily moved these mons to devices with better performing IO (but shouldn't be mons as they're also cephfs clients) and saw a sustained write rate of ~50MB/s. This seems pretty excessive, and is at least an order of magnitude higher than anything I saw when running Luminous. Not to mention this isn't really nice for SSD lifespan. As downgrading is not an option here, is there anything I can look to to figure out what exactly the mons are doing and how to prevent such heavy load? I seem to remember some bug related to telemetry, but can't find it on this list..
Almost forgot, here's a graph of the change in write rate: https://shortbus.org/x/ceph-mon-io.png
Hi Peter, could be a totally different problem but did you run the command "ceph osd require-osd-release nautilus" after the upgrade? We had poor performance after upgrading to nautilus and running this command fixed it. The same was reported by others for previous updates. Here is my original message regarding this issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPVW... We did not observe the master election problem though. Regards, Georg
How do you check if you issued this command in the past? -----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7 Hi Peter, could be a totally different problem but did you run the command "ceph osd require-osd-release nautilus" after the upgrade? We had poor performance after upgrading to nautilus and running this command fixed it. The same was reported by others for previous updates. Here is my original message regarding this issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPVW... We did not observe the master election problem though.
participants (4)
-
Georg F
-
Marc Roos
-
peter woodman
-
Peter Woodman