Procedure for temporary evacuation and replacement
Hi all, a hopefully simple question this time. I would like a second opinion on a procedure for replacing a larger number of disks. We need to replace about 40 disks distributed over all 12 hosts backing a large pool with EC 8+3. We can't do it host by host as it would take way too long (replace disks per host and let recovery rebuild the data). Therefore, we would like to evacuate all data from these disks simultaneously and with as little data movement as possible. This is the procedure that seems to do the trick: 1.) For all OSDs: ceph osd reweight ID 0 # Note: not "osd crush reweight" 2.) Wait for rebalance to finish 3.) Replace disks and deploy OSDs with the same IDs as before per host 4.) Start OSDs and let rebalance back I tested step 1 on Octopus with 1 disk and it seems to work. The reason I ask is that step 1 actually marks the OSDs as OUT. However, they are still UP and I see only misplaced objects, not degraded objects. It is a bit counter-intuitive, but it seems that UP+OUT OSDs still participate in IO. Because it is counter-intuitive, I would like to have a second opinion. I have read before that others reweight to something like 0.001 and hope that this flushes all PGs. I would prefer not to rely on hope and a reweight to 0 apparently is a valid choice here, leading to a somewhat weird state with UP+OUT OSDs. Problems that could arise are timeouts I'm overlooking that will make data chunks on UP+OUT OSDs unavailable after some time. I'm also wondering if UP+OUT OSDs participate in peering in case there is an OSD restart somewhere in the pool. Thanks for your input and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
We need to replace about 40 disks distributed over all 12 hosts backing a large pool with EC 8+3. We can't do it host by host as it would take way too long (replace disks per host and let recovery rebuild the data)
<soapbox>This is one of the false economies of HDDs ;) </soapbox>
Therefore, we would like to evacuate all data from these disks simultaneously and with as little data movement as possible. This is the procedure that seems to do the trick:
1.) For all OSDs: ceph osd reweight ID 0 # Note: not "osd crush reweight"
Note that this will run afoul of the balancer module. I *think* also that it will result in the data moving to OSDs on the same host.
2.) Wait for rebalance to finish 3.) Replace disks and deploy OSDs with the same IDs as before per host 4.) Start OSDs and let rebalance back
I tested step 1 on Octopus with 1 disk and it seems to work. The reason I ask is that step 1 actually marks the OSDs as OUT. However, they are still UP and I see only misplaced objects, not degraded objects. It is a bit counter-intuitive, but it seems that UP+OUT OSDs still participate in IO.
Because it is counter-intuitive, I would like to have a second opinion. I have read before that others reweight to something like 0.001 and hope that this flushes all PGs. I would prefer not to rely on hope and a reweight to 0 apparently is a valid choice here, leading to a somewhat weird state with UP+OUT OSDs.
Problems that could arise are timeouts I'm overlooking that will make data chunks on UP+OUT OSDs unavailable after some time. I'm also wondering if UP+OUT OSDs participate in peering in case there is an OSD restart somewhere in the pool.
Thanks for your input and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I dont think your plan will work as expected. In step 3 you will introduce additional data movement with the manner in which you have tried to accomplish this. I suggest you do set the CRUSH weight to 0 for the OSD in which you intend to replace; do this for all OSDs you wish to replace whilst the "norebalance" flag is set. Now you can optionally run a tool like upmap-remapped.py to reduce your misplaced percent count ( i wont elaborate on this too much since its optional ) Once you get to misplaced PG count of 0 you want to "destroy" the OSD "ceph osd destroy". IMO "destroy" would have been better named "pending-replacement" At this point set the norebalance flag again and recreate the OSD specifying the re-use of the OSD ID and potentially if a multi-disk osd the block.db and block.wal LVs of the OSD that were previously in use. Once all OSDs are re-added, set the CRUSH weight of the replaced OSDs to their new value based upon the new disk size if they didn't get updated in the previous step. Optionally run the upmap-remapped.py script again. Unset the norebalance flag. Wait for all PGs to become active+clean again. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Thu, Oct 10, 2024 at 9:08 AM Anthony D'Atri <aad@dreamsnake.net> wrote:
We need to replace about 40 disks distributed over all 12 hosts backing
a large pool with EC 8+3. We can't do it host by host as it would take way too long (replace disks per host and let recovery rebuild the data)
<soapbox>This is one of the false economies of HDDs ;) </soapbox>
Therefore, we would like to evacuate all data from these disks simultaneously and with as little data movement as possible. This is the procedure that seems to do the trick:
1.) For all OSDs: ceph osd reweight ID 0 # Note: not "osd crush reweight"
Note that this will run afoul of the balancer module. I *think* also that it will result in the data moving to OSDs on the same host.
2.) Wait for rebalance to finish 3.) Replace disks and deploy OSDs with the same IDs as before per host 4.) Start OSDs and let rebalance back
I tested step 1 on Octopus with 1 disk and it seems to work. The reason I ask is that step 1 actually marks the OSDs as OUT. However, they are still UP and I see only misplaced objects, not degraded objects. It is a bit counter-intuitive, but it seems that UP+OUT OSDs still participate in IO.
Because it is counter-intuitive, I would like to have a second opinion. I have read before that others reweight to something like 0.001 and hope that this flushes all PGs. I would prefer not to rely on hope and a reweight to 0 apparently is a valid choice here, leading to a somewhat weird state with UP+OUT OSDs.
Problems that could arise are timeouts I'm overlooking that will make data chunks on UP+OUT OSDs unavailable after some time. I'm also wondering if UP+OUT OSDs participate in peering in case there is an OSD restart somewhere in the pool.
Thanks for your input and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Anthony and Wesley for your input. Let me explain in more detail why I'm interested in the somewhat obscure looking procedure in step 1. Whats the difference between "ceph osd reweight" and "ceph osd crush reweight"? the difference is that command 1 only remaps shards within the same failure domain (as Anthony noted), while command 2 implies global changes to the crush map with rediúndant data movement. In other words, using ceph osd reweight osd.X 0 will only move the shards from osd.X to other OSDs (in the same failue domain) while ceph osd crush reweight osd.X 0 has a global effect and will move a lot more around. This "a lot more" is what I want to avoid. There is necessary data movement, namely the data on the OSDs I want to evacuate, and there is redundant data movement, which is everything else. So, for evacuation, the first command is the command of choice if one wants to move exactly the shards that need to move. If one re-creates OSDs with exactly the same IDs and weights that the evacuated OSDs had, which is the default when using "ceph osd destroy" as it preserves the crush weights, then, after adding the new OSDs, it will be exactly the shards that were evacuated in step 1 that will move back. That's the minimum possible data movement: data moved = data that needs to move. I don't have balancer or anything enabled that could interfere with that procedure. Please don't bother commenting about things like that. My actual question is, how dangerous is it to use ceph osd reweight osd.X 0 instead of ceph osd reweight osd.X 0.001 The first command will mark the OSD OUT while the second won't. The second command might leave 1-2 PGs on the OSDs, while the first one won't. Does the OSD being formally UP+OUT make any difference compared with UP+IN for evacuation? My initial simplistic test says no, but I would like to be a bit more sure than that. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
If you are replacing the OSDs with the same size/weight device, I agree with your reweight approach. I've been doing some similar work myself that does require crush reweighting to 0 and have been in that headspace. I did a bit of testing around this: - Even with the lowest possible reweight an OSD would take, 1 PG was left on my up+in OSD "ceph osd reweight osd.1 0.00002" results in a reweight of 0.00002 however a "ceph osd reweight osd.1 0.00001" results in the reweight of 0 (out). - With my OSD in a state of UP + IN and a reweight of .00002 I used upmap to move that 1 PG off of the OSD to be left with 0 PGs there. - I attempted to destroy the osd in this state but it complained it was not down and so I marked it down and set the noup flag - With the PG in a down + in state the osd could be destroyed, and this surprised me, I assumed it would need to be marked (or transition to) out as well. So in summary, I think you will be left with 1 or more PGs on the OSD in your approach of reweighting to a very low value you will then either need to later mark it fully out / reweight it to 0 or use upmap approach to not degrade that subsequent PG when getting it marked down. I dont think there is any danger to reweighting to 0 (or marking it out) vs marking it to a very low value and as I have more clarity on what you want to do that is exactly the approach I would take (mark it out). Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Thu, Oct 10, 2024 at 9:58 AM Frank Schilder <frans@dtu.dk> wrote:
Thanks Anthony and Wesley for your input.
Let me explain in more detail why I'm interested in the somewhat obscure looking procedure in step 1.
Whats the difference between "ceph osd reweight" and "ceph osd crush reweight"? the difference is that command 1 only remaps shards within the same failure domain (as Anthony noted), while command 2 implies global changes to the crush map with rediúndant data movement. In other words, using
ceph osd reweight osd.X 0
will only move the shards from osd.X to other OSDs (in the same failue domain) while
ceph osd crush reweight osd.X 0
has a global effect and will move a lot more around. This "a lot more" is what I want to avoid. There is necessary data movement, namely the data on the OSDs I want to evacuate, and there is redundant data movement, which is everything else.
So, for evacuation, the first command is the command of choice if one wants to move exactly the shards that need to move.
If one re-creates OSDs with exactly the same IDs and weights that the evacuated OSDs had, which is the default when using "ceph osd destroy" as it preserves the crush weights, then, after adding the new OSDs, it will be exactly the shards that were evacuated in step 1 that will move back. That's the minimum possible data movement: data moved = data that needs to move.
I don't have balancer or anything enabled that could interfere with that procedure. Please don't bother commenting about things like that.
My actual question is, how dangerous is it to use
ceph osd reweight osd.X 0
instead of
ceph osd reweight osd.X 0.001
The first command will mark the OSD OUT while the second won't. The second command might leave 1-2 PGs on the OSDs, while the first one won't.
Does the OSD being formally UP+OUT make any difference compared with UP+IN for evacuation? My initial simplistic test says no, but I would like to be a bit more sure than that.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Dear Wesley, yes, the left-over PGs are something I need to avoid. Unfortunately, we have old clients connected and I cannot enable upmap. Therefore, I'm looking at the out-OSD approach. What you describe matches my interpretations of this piece of documentation: https://docs.ceph.com/en/reef/rados/operations/monitoring-osd-pg/?highlight=.... OUT OSDs continue participating in IO, but PGs are migrated away. Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again. I believe there was a config option that can prevent that (or a different osd out command that uses an ID). If you remember something like that, please let me know. If I find it, I will post it here. Thanks for your review! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Thursday, October 10, 2024 8:04 PM To: Frank Schilder Cc: Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Procedure for temporary evacuation and replacement If you are replacing the OSDs with the same size/weight device, I agree with your reweight approach. I've been doing some similar work myself that does require crush reweighting to 0 and have been in that headspace. I did a bit of testing around this: - Even with the lowest possible reweight an OSD would take, 1 PG was left on my up+in OSD "ceph osd reweight osd.1 0.00002" results in a reweight of 0.00002 however a "ceph osd reweight osd.1 0.00001" results in the reweight of 0 (out). - With my OSD in a state of UP + IN and a reweight of .00002 I used upmap to move that 1 PG off of the OSD to be left with 0 PGs there. - I attempted to destroy the osd in this state but it complained it was not down and so I marked it down and set the noup flag - With the PG in a down + in state the osd could be destroyed, and this surprised me, I assumed it would need to be marked (or transition to) out as well. So in summary, I think you will be left with 1 or more PGs on the OSD in your approach of reweighting to a very low value you will then either need to later mark it fully out / reweight it to 0 or use upmap approach to not degrade that subsequent PG when getting it marked down. I dont think there is any danger to reweighting to 0 (or marking it out) vs marking it to a very low value and as I have more clarity on what you want to do that is exactly the approach I would take (mark it out). Respectfully, Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com> On Thu, Oct 10, 2024 at 9:58 AM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Thanks Anthony and Wesley for your input. Let me explain in more detail why I'm interested in the somewhat obscure looking procedure in step 1. Whats the difference between "ceph osd reweight" and "ceph osd crush reweight"? the difference is that command 1 only remaps shards within the same failure domain (as Anthony noted), while command 2 implies global changes to the crush map with rediúndant data movement. In other words, using ceph osd reweight osd.X 0 will only move the shards from osd.X to other OSDs (in the same failue domain) while ceph osd crush reweight osd.X 0 has a global effect and will move a lot more around. This "a lot more" is what I want to avoid. There is necessary data movement, namely the data on the OSDs I want to evacuate, and there is redundant data movement, which is everything else. So, for evacuation, the first command is the command of choice if one wants to move exactly the shards that need to move. If one re-creates OSDs with exactly the same IDs and weights that the evacuated OSDs had, which is the default when using "ceph osd destroy" as it preserves the crush weights, then, after adding the new OSDs, it will be exactly the shards that were evacuated in step 1 that will move back. That's the minimum possible data movement: data moved = data that needs to move. I don't have balancer or anything enabled that could interfere with that procedure. Please don't bother commenting about things like that. My actual question is, how dangerous is it to use ceph osd reweight osd.X 0 instead of ceph osd reweight osd.X 0.001 The first command will mark the OSD OUT while the second won't. The second command might leave 1-2 PGs on the OSDs, while the first one won't. Does the OSD being formally UP+OUT make any difference compared with UP+IN for evacuation? My initial simplistic test says no, but I would like to be a bit more sure than that. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes. Manually marked out OSDs do not mark themselves in again. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert, thanks, that solves it then. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes. Manually marked out OSDs do not mark themselves in again. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important. I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time). Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here. We are almost done with the evacuation. I will report back how the replacement+rebalancing is going. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement Hi Robert, thanks, that solves it then. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes. Manually marked out OSDs do not mark themselves in again. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Interesting and yea does sound like a bug of sorts. I would consider increasing your osd_heartbeat_grace (at global scope) maybe by 2x (to 40 if currently at default) to see you through the drain. What version are you using? Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Thu, Oct 17, 2024 at 9:20 AM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important.
I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time).
Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here.
We are almost done with the evacuation. I will report back how the replacement+rebalancing is going.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
Hi Robert,
thanks, that solves it then.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes.
Manually marked out OSDs do not mark themselves in again.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We are on octopus and I have not seen this in any other situation. It looks like it tends to happen when a PG completed migration and re-peers. Kind of weird though that this gets worse the fewer PGs and objects are on the disk. PGs are first flagged as laggy, then slow ops show up and finally we have this up - down - up in fast succession. I don't want to play with heartbeat graces etc as the cluster should still respond normally to actual fails. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Thursday, October 17, 2024 3:28 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Procedure for temporary evacuation and replacement Interesting and yea does sound like a bug of sorts. I would consider increasing your osd_heartbeat_grace (at global scope) maybe by 2x (to 40 if currently at default) to see you through the drain. What version are you using? Respectfully, Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com> On Thu, Oct 17, 2024 at 9:20 AM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi all, I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important. I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time). Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here. We are almost done with the evacuation. I will report back how the replacement+rebalancing is going. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement Hi Robert, thanks, that solves it then. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Robert Sander <r.sander@heinlein-support.de<mailto:r.sander@heinlein-support.de>> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes. Manually marked out OSDs do not mark themselves in again. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Is this a high-object-count application (S3 or small files in cephfs)? My guess is that they're going down at the end of PG deletions, where a rocksdb scan needs to happen. This scan can be really slow and can exceed heartbeat timeouts, among other things. Some improvements have been made over major releases, so I'd be curious to know which release you're using (but we've seen this in at least as recent as Pacific). Given that you're completely draining these OSDs, a workaround that we've used in the past is to set "osd_delete_sleep" to something ridiculously high (say, 3600) for those OSDs, effectively disabling PG removal, avoiding this issue. The other possibility, assuming that this is an EC system, is that you're seeing backfill source overload; there's no reservation in Ceph today for backfill sources on EC and so there's no limits to the number of active backfills a given OSD could be participating in as a read source. This was one of the reasons why we built pgremapper for our own OSD drain usecases. Or maybe it's a combination of the two! HTH, Josh On Thu, Oct 17, 2024 at 7:45 AM Wesley Dillingham <wes@wesdillingham.com> wrote:
Interesting and yea does sound like a bug of sorts. I would consider increasing your osd_heartbeat_grace (at global scope) maybe by 2x (to 40 if currently at default) to see you through the drain. What version are you using?
Respectfully,
*Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com
On Thu, Oct 17, 2024 at 9:20 AM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important.
I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time).
Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here.
We are almost done with the evacuation. I will report back how the replacement+rebalancing is going.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
Hi Robert,
thanks, that solves it then.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes.
Manually marked out OSDs do not mark themselves in again.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Joshua, thanks for this reply. It is ceph fs with comparably large spinners and a significant percentage of small files. Thanks for pointing out the config option. There were still a few PGs left on the disks and I had time to try a few settings. Not sure if the results are really representative though. osd_delete_sleep = 10 : no real change osd_delete_sleep = 60 : maybe better? osd_delete_sleep = 300 : doesn't prevent OSDs from being marked down every now and then but seems to reduce both, frequency and impact. Does this setting affect PG removal only or is it affecting other operations as well? Essentially: can I leave it at its current value or should I reset it to default? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Joshua Baergen <jbaergen@digitalocean.com> Sent: Thursday, October 17, 2024 3:56 PM To: Wesley Dillingham Cc: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Procedure for temporary evacuation and replacement Is this a high-object-count application (S3 or small files in cephfs)? My guess is that they're going down at the end of PG deletions, where a rocksdb scan needs to happen. This scan can be really slow and can exceed heartbeat timeouts, among other things. Some improvements have been made over major releases, so I'd be curious to know which release you're using (but we've seen this in at least as recent as Pacific). Given that you're completely draining these OSDs, a workaround that we've used in the past is to set "osd_delete_sleep" to something ridiculously high (say, 3600) for those OSDs, effectively disabling PG removal, avoiding this issue. The other possibility, assuming that this is an EC system, is that you're seeing backfill source overload; there's no reservation in Ceph today for backfill sources on EC and so there's no limits to the number of active backfills a given OSD could be participating in as a read source. This was one of the reasons why we built pgremapper for our own OSD drain usecases. Or maybe it's a combination of the two! HTH, Josh On Thu, Oct 17, 2024 at 7:45 AM Wesley Dillingham <wes@wesdillingham.com> wrote:
Interesting and yea does sound like a bug of sorts. I would consider increasing your osd_heartbeat_grace (at global scope) maybe by 2x (to 40 if currently at default) to see you through the drain. What version are you using?
Respectfully,
*Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com
On Thu, Oct 17, 2024 at 9:20 AM Frank Schilder <frans@dtu.dk> wrote:
Hi all,
I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important.
I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time).
Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here.
We are almost done with the evacuation. I will report back how the replacement+rebalancing is going.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
Hi Robert,
thanks, that solves it then.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes.
Manually marked out OSDs do not mark themselves in again.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, how high is the disk utilization? We see this only from time to time on HDD OSDs during regular cluster operation (no recovery). If that really happens a lot during recovery, I would consider decreasing osd_max_backfills and osd_recovery_max_active_hdd in case they are not set to default. Or if you need high recovery rates, you could temporarily set 'ceph osd set nodown' to prevent this. Fortunately, I didn't have to use that flag for a long time, I used that more often in older releases. Regards, Eugen Zitat von Frank Schilder <frans@dtu.dk>:
Hi all,
I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important.
I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time).
Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here.
We are almost done with the evacuation. I will report back how the replacement+rebalancing is going.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
Hi Robert,
thanks, that solves it then.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes.
Manually marked out OSDs do not mark themselves in again.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ah yes, if you see disk read IOPS going up and up on those draining OSDs then you might be having issues with older PG deletion logic interacting poorly with rocksdb tombstones. Josh On Thu, Oct 17, 2024 at 8:13 AM Eugen Block <eblock@nde.ag> wrote:
Hi Frank,
how high is the disk utilization? We see this only from time to time on HDD OSDs during regular cluster operation (no recovery). If that really happens a lot during recovery, I would consider decreasing osd_max_backfills and osd_recovery_max_active_hdd in case they are not set to default. Or if you need high recovery rates, you could temporarily set 'ceph osd set nodown' to prevent this. Fortunately, I didn't have to use that flag for a long time, I used that more often in older releases.
Regards, Eugen
Zitat von Frank Schilder <frans@dtu.dk>:
Hi all,
I would like to share some preliminary experience. Just setting OSDs "out" manually (ceph osd out ID) does work as intended. the OSDs are drained and their data is placed on other OSDs on the same host. This also survives reboots of OSDs and peering and this turns out to be important.
I make the very strange observation that the OSDs that are drained are getting marked down quite often. This actually gets worse over time, the fewer PGs are left the more frequent are these "OSD marked down - OSD still running wrongly marked down by mon - OSD boot" events and I'm a bit at a loss what the cause might be. This is exclusively limited to OSDs that are marked up+out, none of the up+in OSDs shows that behavior. There seems no correlation with anything else present, its all of the OSDs going down->up (one at a time).
Some of these restarts might have to do with disk errors, but I doubt all do. There seems to be something else here at play. I don't think this is expected and maybe someone has additional information here.
We are almost done with the evacuation. I will report back how the replacement+rebalancing is going.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Friday, October 11, 2024 12:18 PM To: Robert Sander; ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
Hi Robert,
thanks, that solves it then.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Friday, October 11, 2024 10:20 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Procedure for temporary evacuation and replacement
On 10/11/24 10:07, Frank Schilder wrote:
Only problem is that setting an OSD OUT might not be sticky. If the OSD reboots for some reason it might mark itself IN again.
The Ceph cluster distinguishes between manually marked out ("ceph osd out N") and automatically marked out, when an OSD is down for more than 10 minutes.
Manually marked out OSDs do not mark themselves in again.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
Eugen Block
-
Frank Schilder
-
Joshua Baergen
-
Marc
-
Robert Sander
-
Wesley Dillingham