Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
This means it has been applied: # ceph osd dump -f json | jq .require_osd_release "nautilus" -- dan On Mon, Feb 17, 2020 at 11:10 AM Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
How do you check if you issued this command in the past?
-----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
Hi Peter,
could be a totally different problem but did you run the command "ceph osd require-osd-release nautilus" after the upgrade? We had poor performance after upgrading to nautilus and running this command fixed it. The same was reported by others for previous updates. Here is my original message regarding this issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPVW...
We did not observe the master election problem though.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yeah, applied that command. For some reason, after 3 days of this, the behavior calmed down, and the size of the mon store shrank down to ~100MB, where previously it was growing to upwards of 6GB. On Mon, Feb 17, 2020 at 3:14 AM Dan van der Ster <dan@vanderster.com> wrote:
This means it has been applied:
# ceph osd dump -f json | jq .require_osd_release "nautilus"
-- dan
On Mon, Feb 17, 2020 at 11:10 AM Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
How do you check if you issued this command in the past?
-----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
Hi Peter,
could be a totally different problem but did you run the command "ceph osd require-osd-release nautilus" after the upgrade? We had poor performance after upgrading to nautilus and running this command fixed it. The same was reported by others for previous updates. Here is my original message regarding this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPVW...
We did not observe the master election problem though.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Circling back on this- this was caused by the ceph-insights mgr plugin. Disabled this and behavior stopped. This was seemingly bloating the mon database- for some reason each .sst file (about 200mb) contained only 5 paxos entries per file, so something in that plugin was stuffing a ton into those entries. Either way, that plugin seems pretty harmful. On Tue, Feb 18, 2020 at 5:38 PM Peter Woodman <peter@shortbus.org> wrote:
Yeah, applied that command.
For some reason, after 3 days of this, the behavior calmed down, and the size of the mon store shrank down to ~100MB, where previously it was growing to upwards of 6GB.
On Mon, Feb 17, 2020 at 3:14 AM Dan van der Ster <dan@vanderster.com> wrote:
This means it has been applied:
# ceph osd dump -f json | jq .require_osd_release "nautilus"
-- dan
On Mon, Feb 17, 2020 at 11:10 AM Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
How do you check if you issued this command in the past?
-----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
Hi Peter,
could be a totally different problem but did you run the command "ceph osd require-osd-release nautilus" after the upgrade? We had poor performance after upgrading to nautilus and running this command fixed it. The same was reported by others for previous updates. Here is my original message regarding this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPVW...
We did not observe the master election problem though.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Dan van der Ster
-
Peter Woodman