We are on octopus and I have not seen this in any other situation. It looks like it tends to happen when a PG completed migration and re-peers. Kind of weird though that this gets worse the fewer PGs and objects are on the disk. PGs are first flagged as laggy, then slow ops show up and finally we have this up - down - up in fast succession. I don't want to play with heartbeat graces etc as the cluster should still respond normally to actual fails. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Thursday, October 17, 2024 3:28 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Procedure for temporary evacuation and replacement Interesting and yea does sound like a bug of sorts. I would consider increasing your osd_heartbeat_grace (at global scope) maybe by 2x (to 40 if currently at default) to see you through the drain. What version are you using? Respectfully, Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com> On Thu, Oct 17, 2024 at 9:20 AM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all, I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important. I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time). Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here. We are almost done with the evacuation. I will report back how the replacement+rebalancing is going. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement Hi Robert, thanks, that solves it then. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert Sander <r.sander@heinlein-support.de<mailto:r.sander@heinlein-support.de>> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes. Manually marked out OSDs do not mark themselves in again. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>