osd fast shutdown provokes slow requests
Hi, I investigated an other problem with my nautilus 14.2.11 (with 14.2.10 as well) cluster. If I stop the OSDs on one node (systemctl stop ceph-osd.target, or shutdown/reboot) it took mostly several seconds until the cluster detects the OSDs as down and I run in slow requests. I identified the option "osd_fast_shutdown". If I configure it to "false" I see in the ceph.log immediately on shutdown the logs like this: cluster [INF] osd.837 marked itself down If the parameter is true (default setting) The cluster needs some time until I get hundert of lines like this in the ceph.log cluster [DBG] osd.317 reported immediately failed by osd.202 until all detection and peering is done I got slow requests. On smaller clusters, with only 48 OSDs on 4 Nodes the down detection works a lot of faster. But on the big one I need to set this to false to work like expected. I wonder if someone else see this. I think it is OK to short the shutdown process, but it would be nice, if the OSDs tell the mon its shutdown itself Manuel
Hi Manuel, Just to clarify -- do you override any of the settings related to peer down detection? heartbeat periods or timeouts or min down reporters or anything like that? Cheers, Dan On Thu, Aug 13, 2020 at 3:46 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi,
I investigated an other problem with my nautilus 14.2.11 (with 14.2.10 as well) cluster.
If I stop the OSDs on one node (systemctl stop ceph-osd.target, or shutdown/reboot) it took mostly several seconds until the cluster detects the OSDs as down and I run in slow requests.
I identified the option "osd_fast_shutdown". If I configure it to "false" I see in the ceph.log immediately on shutdown the logs like this: cluster [INF] osd.837 marked itself down
If the parameter is true (default setting) The cluster needs some time until I get hundert of lines like this in the ceph.log cluster [DBG] osd.317 reported immediately failed by osd.202
until all detection and peering is done I got slow requests.
On smaller clusters, with only 48 OSDs on 4 Nodes the down detection works a lot of faster. But on the big one I need to set this to false to work like expected.
I wonder if someone else see this. I think it is OK to short the shutdown process, but it would be nice, if the OSDs tell the mon its shutdown itself
Manuel
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, The only settings in my ceph.conf related to down/out and peering are this. mon osd down out interval = 1800 mon osd down out subtree limit = host mon osd min down reporters = 3 mon osd reporter subtree level = host The Cluster has 44 Hosts á 24 OSDs Manuel On Thu, 13 Aug 2020 16:17:46 +0200 Dan van der Ster <dan@vanderster.com> wrote:
Hi Manuel,
Just to clarify -- do you override any of the settings related to peer down detection? heartbeat periods or timeouts or min down reporters or anything like that?
Cheers, Dan
OK I just wanted to confirm you hadn't extended the osd_heartbeat_grace or similar. On your large cluster, what is the time from stopping an osd (with fasst shutdown enabled) to: cluster [DBG] osd.317 reported immediately failed by osd.202 -- dan On Thu, Aug 13, 2020 at 4:38 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi Dan,
The only settings in my ceph.conf related to down/out and peering are this.
mon osd down out interval = 1800 mon osd down out subtree limit = host mon osd min down reporters = 3 mon osd reporter subtree level = host
The Cluster has 44 Hosts á 24 OSDs
Manuel
On Thu, 13 Aug 2020 16:17:46 +0200 Dan van der Ster <dan@vanderster.com> wrote:
Hi Manuel,
Just to clarify -- do you override any of the settings related to peer down detection? heartbeat periods or timeouts or min down reporters or anything like that?
Cheers, Dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, stopping a single OSD took mostly 1 to 2 seconds betwenn stop and the first reporting in ceph.log. Stopping a whole node, in this case 24 OSDs, in the most cases it took 5 to 7 seconds. After the reporting peering begins, but this is quite fast. Since I have the fast shutdown disabled. The "reporting down by itself" messages appear more or less immediately, the cluster peers and all works as expected and without trouble. Manuel On Thu, 13 Aug 2020 16:45:20 +0200 Dan van der Ster <dan@vanderster.com> wrote:
OK I just wanted to confirm you hadn't extended the osd_heartbeat_grace or similar.
On your large cluster, what is the time from stopping an osd (with fasst shutdown enabled) to: cluster [DBG] osd.317 reported immediately failed by osd.202
-- dan
There's a bit of discussion on this at the original PR: https://github.com/ceph/ceph/pull/31677 Sage claims the IO interruption should be smaller with osd_fast_shutdown than without. -- dan On Fri, Aug 14, 2020 at 10:08 AM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi Dan,
stopping a single OSD took mostly 1 to 2 seconds betwenn stop and the first reporting in ceph.log. Stopping a whole node, in this case 24 OSDs, in the most cases it took 5 to 7 seconds. After the reporting peering begins, but this is quite fast.
Since I have the fast shutdown disabled. The "reporting down by itself" messages appear more or less immediately, the cluster peers and all works as expected and without trouble.
Manuel
On Thu, 13 Aug 2020 16:45:20 +0200 Dan van der Ster <dan@vanderster.com> wrote:
OK I just wanted to confirm you hadn't extended the osd_heartbeat_grace or similar.
On your large cluster, what is the time from stopping an osd (with fasst shutdown enabled) to: cluster [DBG] osd.317 reported immediately failed by osd.202
-- dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, thank you for the link. I read it as well as the linked conversation in the rook project. I don't get it why the fast shutdown should be better than the "normal" shutdown in which the OSD annouces its shutdown directly. Are there cases where the shutdown of the OSD takes longer until its markdown message is sent? Travis on the rook project mentioned the shutodwn issued IO interruptions from 20 to 30 seconds. (https://github.com/rook/rook/pull/4328#issuecomment-554480275) This is something I would expect if a complet host break down (e.g. caused by a hw failure) and only the heartbeat timeouts detect this. But not on a regular shutdown. Manuel On Fri, 14 Aug 2020 11:03:37 +0200 Dan van der Ster <dan@vanderster.com> wrote:
There's a bit of discussion on this at the original PR: https://github.com/ceph/ceph/pull/31677 Sage claims the IO interruption should be smaller with osd_fast_shutdown than without.
-- dan
Hi, I suppose the idea is that it's quicker to fail via the connection refused setting than by waiting for an osdmap to be propagated across the cluster. It looks simple enough in OSD.cc to also send the down message to the mon even with fast shutdown enabled. But I don't have any clue if that would cause other issues. -- Dan On Fri, Aug 14, 2020 at 1:51 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi Dan,
thank you for the link. I read it as well as the linked conversation in the rook project.
I don't get it why the fast shutdown should be better than the "normal" shutdown in which the OSD annouces its shutdown directly.
Are there cases where the shutdown of the OSD takes longer until its markdown message is sent? Travis on the rook project mentioned the shutodwn issued IO interruptions from 20 to 30 seconds. (https://github.com/rook/rook/pull/4328#issuecomment-554480275) This is something I would expect if a complet host break down (e.g. caused by a hw failure) and only the heartbeat timeouts detect this. But not on a regular shutdown.
Manuel
On Fri, 14 Aug 2020 11:03:37 +0200 Dan van der Ster <dan@vanderster.com> wrote:
There's a bit of discussion on this at the original PR: https://github.com/ceph/ceph/pull/31677 Sage claims the IO interruption should be smaller with osd_fast_shutdown than without.
-- dan
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I thought the "fail" needs to propagated as well. Am I false? Who can have a look if a markdown message in "fast shutdown" mode is a possibilitiy? I do not have the expertice to say if this would breakt something else. But if this is possible I would vote for this. Thanks Manuel On Fri, 14 Aug 2020 14:02:30 +0200 Dan van der Ster <dan@vanderster.com> wrote:
Hi,
I suppose the idea is that it's quicker to fail via the connection refused setting than by waiting for an osdmap to be propagated across the cluster.
It looks simple enough in OSD.cc to also send the down message to the mon even with fast shutdown enabled. But I don't have any clue if that would cause other issues.
-- Dan
I think the best course of action would be to open a tracker ticket with details about your environment and your observations, then the devs could try to see if something was overlooked with this change. -- dan On Fri, Aug 14, 2020 at 5:48 PM Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi,
I thought the "fail" needs to propagated as well. Am I false?
Who can have a look if a markdown message in "fast shutdown" mode is a possibilitiy? I do not have the expertice to say if this would breakt something else. But if this is possible I would vote for this.
Thanks Manuel
On Fri, 14 Aug 2020 14:02:30 +0200 Dan van der Ster <dan@vanderster.com> wrote:
Hi,
I suppose the idea is that it's quicker to fail via the connection refused setting than by waiting for an osdmap to be propagated across the cluster.
It looks simple enough in OSD.cc to also send the down message to the mon even with fast shutdown enabled. But I don't have any clue if that would cause other issues.
-- Dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, I opened the ticket last friday: https://tracker.ceph.com/issues/46978 Manuel On Fri, 14 Aug 2020 17:49:55 +0200 Dan van der Ster <dan@vanderster.com> wrote:
I think the best course of action would be to open a tracker ticket with details about your environment and your observations, then the devs could try to see if something was overlooked with this change.
-- dan
participants (2)
-
Dan van der Ster
-
Manuel Lausch