Slow initial boot of OSDs in large cluster with unclean state
Hi all, On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today... Looking at the logs of a new OSD when it's starting: 2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs. I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling. The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me. Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers? (There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.) So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation. Thanks, Tom
On our 6000+ HDD OSD cluster (pacific)
That’s the bleeding edge in a number of respects. Updating to at least Reef would bring various improvements, and I have some suggestions I'd like to*love* to run by you wrt upgrade speed in such a cluster, if you’re using cephadm / ceph orch. Would also love offline to get a copy of any non-default values you’ve found valuable.
we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases!
That’s pretty extreme.
We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long.
Patient: It turns when I do this Doctor: Don’t do that.
Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly.
Are you adding OSDs while the cluster is converging / backfilling / recovering?
This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
I might speculate that putting off OSD addition until the cluster is converged might help, maybe then a rolling mon compaction in advance of adding OSDs.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point).
From the mons I think.
All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon),
Are your mon DBs on spinners?
so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Please see my replies inline. I also just wanted to say I really enjoyed your talk about QLC flash at Cephalocon, there was a lot of useful info in there.
On our 6000+ HDD OSD cluster (pacific)
That’s the bleeding edge in a number of respects. Updating to at least Reef would bring various improvements, and I have some suggestions I'd like to*love* to run by you wrt upgrade speed in such a cluster, if you’re using cephadm / ceph orch. Would also love offline to get a copy of any non-default values you’ve found valuable.
Upgrades are planned, but we're currently on rocky8 with rpms, so there's a few steps before we can get to Reef unfortunately. Currently the plan is to migrate to cephadm post-quincy, so I'll be sure to get in touch once we're there. One of the big concerns about moving this cluster to cephadm has been the orchestrator performance at scale, so I'm happy to hear that you're interested! For more general Ceph tuning, the OSDs are fairly happy with the defaults, I assume because they are still seeing similar numbers of peers, irrespective of total OSD count. The main things are the monitors and managers (not unexpectedly) start having harder times getting everything done in the default intervals, so increased mon_lease and mgr_stats_periods help prevent election loops on map creation, and monitoring artifacts respectively. We've also had to rescale the crush weights somewhere around 80PB. Other than that it largely just works as intended, which is very cool.
we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases!
That’s pretty extreme.
We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long.
Patient: It turns when I do this Doctor: Don’t do that.
:D
Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly.
Are you adding OSDs while the cluster is converging / backfilling / recovering?
Yes - in general the cluster is always doing something, we have a fairly tight schedule of 'cluster time' (as we call it) for hardware addition/removal, rolling reboots and patching, letting the balancer catch up, etc. Although we try and schedule quiet periods before larger interventions, hardware problems often don't cooperate with this!
This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
I might speculate that putting off OSD addition until the cluster is converged might help, maybe then a rolling mon compaction in advance of adding OSDs.
This makes sense, we treat OSD re-additions after drive replacements as an ongoing process that we don't schedule around the larger operations, and in fact have been a lot more proactive about getting drives back in recently, perhaps this is actually hindering us here, and a batched approach to OSD re-addition would suit us better.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point).
From the mons I think.
All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon),
Are your mon DBs on spinners?
They're on dedicated SSDs. If the maps are being pulled from the mons for the initial boot, I'm a little less keen to try and fiddle with this, dosing the mons when you add a few hosts worth of OSDs sounds like something to avoid.
so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI < tom.byrne@stfc.ac.uk> wrote:
Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Wes, It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn. This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us. Thanks, Tom ________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period. Respectfully, Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com> On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all, On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today... Looking at the logs of a new OSD when it's starting: 2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs. I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling. The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me. Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers? (There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.) So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation. Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi Tom, Could you describe this cluster from a hardware perspective? Network speed and MTU size, HDD type and capacity, whether OSDs have their WAL/DB on SSD/NVMe or if they're collocated, whether MONs are using HDDs or SSDs/NVMe, what workloads this cluster is handling? You mentioned OSD flapping. This phenomenon should no longer occur today on any cluster, or very rarely, only in cases of actual hardware failure or when hardware is undersized relative to the workloads. All your OSDs are using Bluestore, correct? Regards, Frédéric. ----- Le 8 Jan 25, à 12:29, Thomas Byrne - STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Wes,
It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn.
This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us.
Thanks, Tom
________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period.
Respectfully,
Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frédéric, All of our recent OSD crashes can be attributed to genuine hardware issues (i.e. failed IO due to unreadable sectors). For reference I've had a look and it looks like we've had a handful of drive failures on this cluster in the past month, with no other significant flapping. I was trying to say doesn't take many drive failures, combined with the balancer running to result in a persistent level of OSDMap churn. Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores The workload is radosstriper access to EC pools, so very limited metadata requirements (hence the lack of flash for OSDs). More info on the workload details can be seen on an very old talk of mine from a Ceph day [1]. [1] https://indico.cern.ch/event/765214/contributions/3517140/ ________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, January 8, 2025 12:59 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state Hi Tom, Could you describe this cluster from a hardware perspective? Network speed and MTU size, HDD type and capacity, whether OSDs have their WAL/DB on SSD/NVMe or if they're collocated, whether MONs are using HDDs or SSDs/NVMe, what workloads this cluster is handling? You mentioned OSD flapping. This phenomenon should no longer occur today on any cluster, or very rarely, only in cases of actual hardware failure or when hardware is undersized relative to the workloads. All your OSDs are using Bluestore, correct? Regards, Frédéric. ----- Le 8 Jan 25, à 12:29, Thomas Byrne - STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Wes,
It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn.
This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us.
Thanks, Tom
________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period.
Respectfully,
Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tom, Great talk there! Since your cluster must be one of the largest in the world, it would be nice to share your experience with the community as a case study [1]. The Ceph project is looking for contributors right now. If interested, let me know and we'll see how we can organize that. I couldn't find how many MONs you're running in that big cluster. Hopefully 5 MONs. You said OSDs have collocated WAL/DBs on HDDs. Have you tried running OSDs with WAL/DBs on NVMes? I'm wondering about the influence of WAL/DBs collocated on HDDs on OSD creation time, OSD startup time, peering and osdmap updates, and the role it might play regarding flapping, when DB IOs compete with client IOs, even with 100% active+clean PGs. Cheers, Frédéric. [1] https://ceph.io/en/discover/case-studies/ ----- Le 8 Jan 25, à 16:10, Thomas Byrne, STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Frédéric,
All of our recent OSD crashes can be attributed to genuine hardware issues (i.e. failed IO due to unreadable sectors). For reference I've had a look and it looks like we've had a handful of drive failures on this cluster in the past month, with no other significant flapping. I was trying to say doesn't take many drive failures, combined with the balancer running to result in a persistent level of OSDMap churn.
Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores
The workload is radosstriper access to EC pools, so very limited metadata requirements (hence the lack of flash for OSDs). More info on the workload details can be seen on an very old talk of mine from a Ceph day [1].
[1] https://indico.cern.ch/event/765214/contributions/3517140/
________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, January 8, 2025 12:59 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state
Hi Tom,
Could you describe this cluster from a hardware perspective? Network speed and MTU size, HDD type and capacity, whether OSDs have their WAL/DB on SSD/NVMe or if they're collocated, whether MONs are using HDDs or SSDs/NVMe, what workloads this cluster is handling?
You mentioned OSD flapping. This phenomenon should no longer occur today on any cluster, or very rarely, only in cases of actual hardware failure or when hardware is undersized relative to the workloads. All your OSDs are using Bluestore, correct?
Regards, Frédéric.
----- Le 8 Jan 25, à 12:29, Thomas Byrne - STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Wes,
It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn.
This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us.
Thanks, Tom
________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period.
Respectfully,
Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm wondering about the influence of WAL/DBs collocated on HDDs on OSD creation time, OSD startup time, peering and osdmap updates, and the role it might play regarding flapping, when DB IOs compete with client IOs, even with 100% active+clean PGs.
FWIW, having encountered these long-startup issues many times in the past on both HDD and QLC OSDs, I can pretty confidently say that throwing flash at the problem doesn't make it go away. Fewer issues with DB IOs contending with client IOs, but flapping can still occur during PG cleaning on QLC, at least on Pacific. Josh On Thu, Jan 9, 2025 at 6:33 AM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Tom,
Great talk there!
Since your cluster must be one of the largest in the world, it would be nice to share your experience with the community as a case study [1]. The Ceph project is looking for contributors right now. If interested, let me know and we'll see how we can organize that.
I couldn't find how many MONs you're running in that big cluster. Hopefully 5 MONs.
You said OSDs have collocated WAL/DBs on HDDs. Have you tried running OSDs with WAL/DBs on NVMes?
I'm wondering about the influence of WAL/DBs collocated on HDDs on OSD creation time, OSD startup time, peering and osdmap updates, and the role it might play regarding flapping, when DB IOs compete with client IOs, even with 100% active+clean PGs.
Cheers, Frédéric.
[1] https://ceph.io/en/discover/case-studies/
----- Le 8 Jan 25, à 16:10, Thomas Byrne, STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Frédéric,
All of our recent OSD crashes can be attributed to genuine hardware issues (i.e. failed IO due to unreadable sectors). For reference I've had a look and it looks like we've had a handful of drive failures on this cluster in the past month, with no other significant flapping. I was trying to say doesn't take many drive failures, combined with the balancer running to result in a persistent level of OSDMap churn.
Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores
The workload is radosstriper access to EC pools, so very limited metadata requirements (hence the lack of flash for OSDs). More info on the workload details can be seen on an very old talk of mine from a Ceph day [1].
[1] https://indico.cern.ch/event/765214/contributions/3517140/
________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, January 8, 2025 12:59 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state
Hi Tom,
Could you describe this cluster from a hardware perspective? Network speed and MTU size, HDD type and capacity, whether OSDs have their WAL/DB on SSD/NVMe or if they're collocated, whether MONs are using HDDs or SSDs/NVMe, what workloads this cluster is handling?
You mentioned OSD flapping. This phenomenon should no longer occur today on any cluster, or very rarely, only in cases of actual hardware failure or when hardware is undersized relative to the workloads. All your OSDs are using Bluestore, correct?
Regards, Frédéric.
----- Le 8 Jan 25, à 12:29, Thomas Byrne - STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Wes,
It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn.
This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us.
Thanks, Tom
________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period.
Respectfully,
Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
----- Le 9 Jan 25, à 17:33, Joshua Baergen jbaergen@digitalocean.com a écrit :
I'm wondering about the influence of WAL/DBs collocated on HDDs on OSD creation time, OSD startup time, peering and osdmap updates, and the role it might play regarding flapping, when DB IOs compete with client IOs, even with 100% active+clean PGs.
FWIW, having encountered these long-startup issues many times in the past on both HDD and QLC OSDs, I can pretty confidently say that throwing flash at the problem doesn't make it go away. Fewer issues with DB IOs contending with client IOs, but flapping can still occur during PG cleaning on QLC, at least on Pacific.
Hi Josh, Interesting. Were these QLC OSDs collocated too? Or were they running DB on QLCs and data on HDDs? Cheers, Frédéric.
Josh
On Thu, Jan 9, 2025 at 6:33 AM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Tom,
Great talk there!
Since your cluster must be one of the largest in the world, it would be nice to share your experience with the community as a case study [1]. The Ceph project is looking for contributors right now. If interested, let me know and we'll see how we can organize that.
I couldn't find how many MONs you're running in that big cluster. Hopefully 5 MONs.
You said OSDs have collocated WAL/DBs on HDDs. Have you tried running OSDs with WAL/DBs on NVMes?
I'm wondering about the influence of WAL/DBs collocated on HDDs on OSD creation time, OSD startup time, peering and osdmap updates, and the role it might play regarding flapping, when DB IOs compete with client IOs, even with 100% active+clean PGs.
Cheers, Frédéric.
[1] https://ceph.io/en/discover/case-studies/
----- Le 8 Jan 25, à 16:10, Thomas Byrne, STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Frédéric,
All of our recent OSD crashes can be attributed to genuine hardware issues (i.e. failed IO due to unreadable sectors). For reference I've had a look and it looks like we've had a handful of drive failures on this cluster in the past month, with no other significant flapping. I was trying to say doesn't take many drive failures, combined with the balancer running to result in a persistent level of OSDMap churn.
Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores
The workload is radosstriper access to EC pools, so very limited metadata requirements (hence the lack of flash for OSDs). More info on the workload details can be seen on an very old talk of mine from a Ceph day [1].
[1] https://indico.cern.ch/event/765214/contributions/3517140/
________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, January 8, 2025 12:59 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state
Hi Tom,
Could you describe this cluster from a hardware perspective? Network speed and MTU size, HDD type and capacity, whether OSDs have their WAL/DB on SSD/NVMe or if they're collocated, whether MONs are using HDDs or SSDs/NVMe, what workloads this cluster is handling?
You mentioned OSD flapping. This phenomenon should no longer occur today on any cluster, or very rarely, only in cases of actual hardware failure or when hardware is undersized relative to the workloads. All your OSDs are using Bluestore, correct?
Regards, Frédéric.
----- Le 8 Jan 25, à 12:29, Thomas Byrne - STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Wes,
It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn.
This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us.
Thanks, Tom
________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period.
Respectfully,
Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
FWIW, having encountered these long-startup issues many times in the past on both HDD and QLC OSDs, I can pretty confidently say that throwing flash at the problem doesn't make it go away. Fewer issues with DB IOs contending with client IOs, but flapping can still occur during PG cleaning on QLC, at least on Pacific.
Interesting. Were these QLC OSDs collocated too? Or were they running DB on QLCs and data on HDDs?
Collocated in each case (i.e. wal/db was alongside data) Josh
Thanks for the input Josh. I actually started looking into this was because we're adding some SSD OSDs to this cluster, and they were basically as slow on their initial boot as HDD OSDs when the cluster hasn't trimmed OSDmaps in a while. I'd be interested to know if other people seeing this slow start also see a relatively slow (~50MB/s) rate of OSDmap download during that initial boot, or if this is something specific to our cluster. I agree with the general guidance of 'don't let clusters get into these states', but it would be interesting to know if there is something to do to speed this up. Thanks, Tom ________________________________________ From: Joshua Baergen <jbaergen@digitalocean.com> Sent: Friday, January 10, 2025 16:56 To: Frédéric Nass <frederic.nass@univ-lorraine.fr> Cc: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk>; Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state
FWIW, having encountered these long-startup issues many times in the past on both HDD and QLC OSDs, I can pretty confidently say that throwing flash at the problem doesn't make it go away. Fewer issues with DB IOs contending with client IOs, but flapping can still occur during PG cleaning on QLC, at least on Pacific.
Interesting. Were these QLC OSDs collocated too? Or were they running DB on QLCs and data on HDDs?
Collocated in each case (i.e. wal/db was alongside data) Josh
Hi Frédéric, We've had an internal discussion, and we would love to share our experience as a case study. If you still think this would be of interest, please let us know what we need to do. We have had 5 monitors on this cluster from about 2018 I think. I actually did a quick investigation in November '24 into OSD performance with our production cluster and workload with and without co-located RocksDB. The short answer was it made surprisingly little difference in average IOPS hitting the HDD under normal running, but as you say, I'm sure there are times where the HDD IOPS are a limiting factor. I'm happy to share my (fairly rough) report on the work with people if there is interest. As I said elsewhere, I actually noted the slow OSDmap download/initial boot when adding fully SSD OSDs to this cluster, which I expected to be a lot faster, hence why I started looking into it. Thanks, Tom ________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Thursday, January 9, 2025 13:32 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state Hi Tom, Great talk there! Since your cluster must be one of the largest in the world, it would be nice to share your experience with the community as a case study [1]. The Ceph project is looking for contributors right now. If interested, let me know and we'll see how we can organize that. I couldn't find how many MONs you're running in that big cluster. Hopefully 5 MONs. You said OSDs have collocated WAL/DBs on HDDs. Have you tried running OSDs with WAL/DBs on NVMes? I'm wondering about the influence of WAL/DBs collocated on HDDs on OSD creation time, OSD startup time, peering and osdmap updates, and the role it might play regarding flapping, when DB IOs compete with client IOs, even with 100% active+clean PGs. Cheers, Frédéric. [1] https://ceph.io/en/discover/case-studies/ ----- Le 8 Jan 25, à 16:10, Thomas Byrne, STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Frédéric,
All of our recent OSD crashes can be attributed to genuine hardware issues (i.e. failed IO due to unreadable sectors). For reference I've had a look and it looks like we've had a handful of drive failures on this cluster in the past month, with no other significant flapping. I was trying to say doesn't take many drive failures, combined with the balancer running to result in a persistent level of OSDMap churn.
Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores
The workload is radosstriper access to EC pools, so very limited metadata requirements (hence the lack of flash for OSDs). More info on the workload details can be seen on an very old talk of mine from a Ceph day [1].
[1] https://indico.cern.ch/event/765214/contributions/3517140/
________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, January 8, 2025 12:59 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state
Hi Tom,
Could you describe this cluster from a hardware perspective? Network speed and MTU size, HDD type and capacity, whether OSDs have their WAL/DB on SSD/NVMe or if they're collocated, whether MONs are using HDDs or SSDs/NVMe, what workloads this cluster is handling?
You mentioned OSD flapping. This phenomenon should no longer occur today on any cluster, or very rarely, only in cases of actual hardware failure or when hardware is undersized relative to the workloads. All your OSDs are using Bluestore, correct?
Regards, Frédéric.
----- Le 8 Jan 25, à 12:29, Thomas Byrne - STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Wes,
It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn.
This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us.
Thanks, Tom
________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period.
Respectfully,
Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tom, That's great news! The community will definitely benefit from hearing about your experience. During last week's user+dev monthly meeting (everyone can join, btw), we previewed and discussed the upcoming 'Ceph User Stories' survey, which will help feature use case studies on Ceph.io, if the user expresses the wish. I suggest waiting for this survey, completing it when it arrives, and then we'll go from there. Does that work for you? Cheers, Frédéric. PS: I can see highlighting the success of your Ceph implementation without fast disks and in line with your workload as a particularly interesting fact to share in the case study. ----- Le 16 Jan 25, à 13:19, Thomas Byrne, STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Frédéric,
We've had an internal discussion, and we would love to share our experience as a case study. If you still think this would be of interest, please let us know what we need to do.
We have had 5 monitors on this cluster from about 2018 I think. I actually did a quick investigation in November '24 into OSD performance with our production cluster and workload with and without co-located RocksDB. The short answer was it made surprisingly little difference in average IOPS hitting the HDD under normal running, but as you say, I'm sure there are times where the HDD IOPS are a limiting factor. I'm happy to share my (fairly rough) report on the work with people if there is interest.
As I said elsewhere, I actually noted the slow OSDmap download/initial boot when adding fully SSD OSDs to this cluster, which I expected to be a lot faster, hence why I started looking into it.
Thanks, Tom
________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Thursday, January 9, 2025 13:32 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state Hi Tom,
Great talk there!
Since your cluster must be one of the largest in the world, it would be nice to share your experience with the community as a case study [1]. The Ceph project is looking for contributors right now. If interested, let me know and we'll see how we can organize that.
I couldn't find how many MONs you're running in that big cluster. Hopefully 5 MONs.
You said OSDs have collocated WAL/DBs on HDDs. Have you tried running OSDs with WAL/DBs on NVMes?
I'm wondering about the influence of WAL/DBs collocated on HDDs on OSD creation time, OSD startup time, peering and osdmap updates, and the role it might play regarding flapping, when DB IOs compete with client IOs, even with 100% active+clean PGs.
Cheers, Frédéric.
[1] https://ceph.io/en/discover/case-studies/
----- Le 8 Jan 25, à 16:10, Thomas Byrne, STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Frédéric,
All of our recent OSD crashes can be attributed to genuine hardware issues (i.e. failed IO due to unreadable sectors). For reference I've had a look and it looks like we've had a handful of drive failures on this cluster in the past month, with no other significant flapping. I was trying to say doesn't take many drive failures, combined with the balancer running to result in a persistent level of OSDMap churn.
Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores
The workload is radosstriper access to EC pools, so very limited metadata requirements (hence the lack of flash for OSDs). More info on the workload details can be seen on an very old talk of mine from a Ceph day [1].
[1] https://indico.cern.ch/event/765214/contributions/3517140/
________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, January 8, 2025 12:59 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: Wesley Dillingham <wes@wesdillingham.com>; ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state
Hi Tom,
Could you describe this cluster from a hardware perspective? Network speed and MTU size, HDD type and capacity, whether OSDs have their WAL/DB on SSD/NVMe or if they're collocated, whether MONs are using HDDs or SSDs/NVMe, what workloads this cluster is handling?
You mentioned OSD flapping. This phenomenon should no longer occur today on any cluster, or very rarely, only in cases of actual hardware failure or when hardware is undersized relative to the workloads. All your OSDs are using Bluestore, correct?
Regards, Frédéric.
----- Le 8 Jan 25, à 12:29, Thomas Byrne - STFC UKRI tom.byrne@stfc.ac.uk a écrit :
Hi Wes,
It works out at about five new osdmaps a minute, which is about normal for this cluster's state changes as far as I can tell. It'll drop down to 2-3 maps/minute during quiet periods, but the combination of the upmap balancer making changes and occasional OSD flaps or crashes due to hardware issues is enough to cause a fairly reliable rate of osdmap churn.
This churn is something that we are working on understanding, and reducing where possible, now that we know becoming a pain point for us.
Thanks, Tom
________________________________ From: Wesley Dillingham <wes@wesdillingham.com> Sent: Tuesday, January 7, 2025 18:41 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
It went from normal osdmap range 500-1000 maps to 30,000 maps in 5 days? That seems like excessive accumulation to me in a 5 day period.
Respectfully,
Wes Dillingham LinkedIn<http://www.linkedin.com/in/wesleydillingham> wes@wesdillingham.com<mailto:wes@wesdillingham.com>
On Tue, Jan 7, 2025 at 1:18 PM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk<mailto:tom.byrne@stfc.ac.uk>> wrote: Hi all,
On our 6000+ HDD OSD cluster (pacific), we've been noticing takes significantly longer for brand new OSDs to go from booting to active when the cluster has been in a state of flux for some time. It can take over an hour for a newly created OSD to be marked up in some cases! We've just put up with it for some time, but I finally got annoyed enough with it to look into it today...
Looking at the logs of a new OSD when it's starting:
2025-01-07T13:44:05.534+0000 7f0b8b830700 3 osd.2016 5165598 handle_osd_map epochs [5165599,5165638], i have 5165598, src has [5146718,5175990] 2025-01-07T13:44:08.988+0000 7f0b8d6ed700 10 osd.2016 5165638 msg say newest map is 5175990, requesting more 2025-01-07T13:44:08.990+0000 7f0b8b830700 3 osd.2016 5165638 handle_osd_map epochs [5165639,5165678], i have 5165638, src has [5146718,5175990] 2025-01-07T13:44:12.391+0000 7f0b8d6ed700 10 osd.2016 5165678 msg say newest map is 5175990, requesting more 2025-01-07T13:44:12.394+0000 7f0b8b830700 3 osd.2016 5165678 handle_osd_map epochs [5165679,5165718], i have 5165678, src has [5146718,5175990] 2025-01-07T13:44:16.047+0000 7f0b8d6ed700 10 osd.2016 5165718 msg say newest map is 5175990, requesting more
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
The first thing I noted was the OSD block devices aren't busy during the OSDmap fetching process - they're barely doing 50MB/s and 50 wr/s. I started looking into raising 'osd_map_share_max_epochs' to hopefully increase the number of maps shared with the new OSD per request and improve the rate, but I balked a bit after realising I would have to do this across the whole cluster (I think, anyway, not actually sure where the maps are being pulled from at this point). All tuning of this value I could see talked about reducing this value which further scared me.
Additionally, there's clearly some interplay between 'osd_map_cache_size' and 'osd_map_message_max' to consider. These historic maps must be being pulled from disk in general (be it osd or mon), so it shouldn't make a difference if osd_map_share_max_epochs > osd_map_cache_size, but in general I suppose you don't want OSDs having to grab maps off disk for requests from peers?
(There may also be a completely different dominating factor of the time to download and store the maps that I'm not considering here.)
So, any advice on improving the speed of the OSDmap download for fresh OSDs would be appreciated, or any other thoughts about this situation.
Thanks, Tom _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm watching this thread with interest, for a few reasons. We have benefited a lot(!) from advice from various people in it over the years, there are some similarities between our setup and STFC's, and we haven't had been bothered by this issue so far. So, I'm intrigued. On 8/1/25 23:10, Thomas Byrne - STFC UKRI wrote:
Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores
Our nodes are 740XD2 with 24x 16TB HDDs plus SSD for RocksDB, and 100Gb with jumbo frames (9000 MTU). Quincy deb pkgs on Ubuntu 20.04, so we're planning for some necessary upgrade work. We have target_max_misplaced_ratio 0.3% since the last time we had a lot of backfilling (aided by upmap tools) that value left us with plenty of performance capacity for users. We're at 75-80% full with some capacity for growth, and it is operating fine. Sometimes starting an OSD can take up to 20 minutes, so there may be some shared experience there. However, apart from a harrowing period last year[1] we live in HEALTH_OK most of the time. We also don't schedule the balancer to ever be off, because it is often pretty quiet. Typical output from our two clusters: pgs: 32591 active+clean 588 active+clean+scrubbing+deep 21 active+clean+scrubbing The big one has just shy of 3000 OSDs, which is half of Thomas' cluster. Perhaps that is a key difference. Our hardware failure though is markedly less than half the rate. I think we had one in December, and none this year so far. [1] https://ceph2024.sched.com/event/1ktWK/get-that-cluster-back-online-but-hurr...
----- Le 23 Jan 25, à 9:51, Gregory Orange gregory.orange@pawsey.org.au a écrit :
I'm watching this thread with interest, for a few reasons. We have benefited a lot(!) from advice from various people in it over the years, there are some similarities between our setup and STFC's, and we haven't had been bothered by this issue so far. So, I'm intrigued.
On 8/1/25 23:10, Thomas Byrne - STFC UKRI wrote:
Storage nodes are all some derivative of a 24 bay, 2U chassis (e.g 760XD2) Single 25Gig connection, no jumboframes HDDs range from 12-20TB SAS HDDs depending on year purchased, with collocated WAL/DBs on the HDDs. All BlueStore OSDs Mons have dedicated flash devices for their stores
Our nodes are 740XD2 with 24x 16TB HDDs plus SSD for RocksDB, and 100Gb with jumbo frames (9000 MTU). Quincy deb pkgs on Ubuntu 20.04, so we're planning for some necessary upgrade work. We have target_max_misplaced_ratio 0.3% since the last time we had a lot of backfilling (aided by upmap tools) that value left us with plenty of performance capacity for users. We're at 75-80% full with some capacity for growth, and it is operating fine.
Sometimes starting an OSD can take up to 20 minutes,
Hey Gregory, Were these OSDs down for a long period of time? Since Bluestore and even in filestore, I've never seen an OSD taking that long to start. And we use the same hardware as you (730xd, 740xd 760xd2) with RocksDBs on SSD and/or NVMes and HDDs of 4TB,8TB and 16TB. I wonder if/how this is influenced by the number of OSDs in the cluster (we 'only' have 600 OSDs). I mean, it probably is but, but to that point... I'm surprised. Cheers, Frédéric.
so there may be some shared experience there. However, apart from a harrowing period last year[1] we live in HEALTH_OK most of the time.
We also don't schedule the balancer to ever be off, because it is often pretty quiet. Typical output from our two clusters:
pgs: 32591 active+clean 588 active+clean+scrubbing+deep 21 active+clean+scrubbing
The big one has just shy of 3000 OSDs, which is half of Thomas' cluster. Perhaps that is a key difference. Our hardware failure though is markedly less than half the rate. I think we had one in December, and none this year so far.
[1] https://ceph2024.sched.com/event/1ktWK/get-that-cluster-back-online-but-hurr... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I recently had a similar issue on one of my clusters that might be related. I found that when new OSDs were added to the cluster they were taking a long time to start. This ended up being caused by the new OSDs needing to pull down a couple hundred thousand osdmaps from the mon nodes. To see if you're also affected by this, try running the following command: ceph report 2>/dev/null | jq '(.osdmap_last_committed - .osdmap_first_committed)' This number should be between 500-1000 on a healthy cluster. I've seen this as high as 4.8 million before (roughly 50% of the data stored on the cluster ended up being osdmaps!) If you're curious how large a single osdmap is, you can run this command to save the current osdmap to a file: ceph osd getmap -o [filename] This appears to be a bug that should be fixed in the latest releases of Ceph (Quincy 17.2.8 & Reef 18.2.4) based on this report: https://tracker.ceph.com/issues/63883 In the meantime, if you are seeing a large difference between the first and last committed osdmaps you can usually clear that up by restarting each of the mon daemons sequentially, starting with the primary and waiting for it to finish peering before moving on to the next one. Doing this on the first cluster I mentioned above reduced the startup time of the new OSDs to seconds from 10-20 minutes each! Bryan From: Gregory Orange <gregory.orange@pawsey.org.au> Date: Thursday, January 23, 2025 at 01:52 To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Slow initial boot of OSDs in large cluster with unclean state Sometimes starting an OSD can take up to 20 minutes, so there may be some shared experience there. However, apart from a harrowing period last year[1] we live in HEALTH_OK most of the time.
On 24/1/25 06:45, Stillwell, Bryan wrote:
ceph report 2>/dev/null | jq '(.osdmap_last_committed - .osdmap_first_committed)'
This number should be between 500-1000 on a healthy cluster. I've seen this as high as 4.8 million before (roughly 50% of the data stored on the cluster ended up being osdmaps!)
Yes, ours is and has been healthy for a while... but we didn't start monitoring it until a few months ago, so it may relate to those slower startups.
This appears to be a bug that should be fixed in the latest releases of Ceph (Quincy 17.2.8 & Reef 18.2.4) based on this report:
Thanks, good to know! We'll get to 17.2.8 in the next couple of weeks, then 18.x later this year.
Hi Tom, On Tue, Jan 7, 2025 at 10:15 AM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk> wrote:
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
I find it's always best to aim to have all PGs clean at least once a week -- that way the osdmaps can be trimmed at least weekly, preventing all sorts of nastiness, one of which you mentioned here. Here's my recommended mgr balancer tuning: # Balance PGs Sunday to Friday, letting the backfilling finish on Saturdays. (adjust the exact days if needed -- the goal here is that at some point in the week, there needs to be 0 misplaced and 0 degraded objects.) ceph config set mgr mgr/balancer/begin_weekday 0 ceph config set mgr mgr/balancer/end_weekday 5 # [Alternatively] Balance PGs during working hours, letting the backfilling finish over night: ceph config set mgr mgr/balancer/begin_time 0830 ceph config set mgr mgr/balancer/end_time 1800 # Decrease the max misplaced from the default 5% to 0.5%, to minimize the impact of backfilling and ensure the tail of backfilling PGs can finish over the weekend or over night -- increase this percentage if your cluster can tolerate it. (IMHO 5% is way too many misplaced objects on large clusters, but this is very use-case-specific). ceph config set mgr target_max_misplaced_ratio 0.005 # Configure the balancer to aim for +/- 1 PG per pool per OSD -- this is the best uniformity we can hope for with the mgr balancer ceph config set mgr mgr/balancer/upmap_max_deviation 1 Then whenever you add/remove hardware, here's my recommended procedure: 1. Set some flags to prevent data from moving immediately when we add new OSDs: ceph osd set norebalance ceph balancer off 2. Add the new OSDs. (Or start draining -- but note that if you are draining OSDs, set the crush weights to 0.1 instead of 0.0 -- upmap magic tools don't work with OSDs having crush weight = 0). 3. Run ./upmap-remapped.py [1] until the number of misplaced objects is as close as possible to zero. 4. Then unset the flags so data starts rebalancing again. I.e. the mgr balancer will move data in a controlled manner to those new empty OSDs: ceph osd unset norebalance ceph balancer on I have a couple talks about this for more on this topic: - https://www.youtube.com/watch?v=6PQYHlerJ8k - https://www.youtube.com/watch?v=A4xG975UWts We also have a plan to get this logic directly into ceph: https://tracker.ceph.com/issues/67418 As to what you can do right now -- it's actually a great time to test out the above approach. Here's exactly what I'd do: 1. Stop those new OSDs (the ones that are not "in" yet) -- no point having them pull in 30000 osdmaps. Nothing should be degraded at this point -- if so, you either stopped too many OSDs, or there was some OSD flap that you need to recover from. 2. Since you have several remapped PGs right now, that's a perfect time to use upmap-remapped.py [1] -- it'll make the remapped PGs clean again. So try running it: ceph balancer off # disabled the mgr balancer, otherwise it would "undo" what we do next ./upmap-remapped.py # this just outputs commands directly to stdout. ./upmap-remapped.py | sh -x # this will run those commands. ./upmap-remapped.py | sh -x # run it again -- normally we need to just run it twice to get to a minimal number of misplaced PGs. 3. When you run it, you should see the % misplaced objects decreasing. Ideally it will go to 0, meaning all PGs are active+clean. At that point the OSDmaps should trim. 4. Confirm that osdmaps have trimmed by looking at the `ceph report`: ceph report | jq '(.osdmap_last_committed - .osdmap_first_committed)' ^^ the number above should be less than 750. If not -- then the osdmaps are not trimmed, and you need to investigate further. 5. Now start those new OSDs, they should pull in the ~750 osdmaps quickly, and then do the upmap-remapped procedure after configuring the balancer as I described. Hope this all helps, Happy New Year Tom. Cheers, Dan [1] https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remap... -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
I find it's always best to aim to have all PGs clean at least once a week -- that way the osdmaps can be trimmed at least weekly,
Hi Dan, Happy new year! preventing all sorts of nastiness, one of which you mentioned here. Just to check, are you recommending that at some point each week all PGs are clean *at the same time*, or that no PGs should be unclean for more than a week? The latter absolutely makes sense, but the former can be quite hard to manage sometimes this cluster, with about one drive failure a week we're somewhat at the mercy of probability. We do always try and aim for 'clean-ish' every so often though :) Also, just to double check my understanding here, the cluster needs to keep hold of osdmaps going back to the point at which the currently unclean PGs were last clean? So if a cluster has a bunch of backfill being queued continuously for a month, but individual PGs get remapped and then backfilled quickly (e.g. ~1day), the cluster only needs to hold onto maps for the day, rather than the entire month period? Or am I missing something? The above is how I would imagine an even larger cluster would operate, with the expectation is that there will always be at least one non-clean PG at any time. As long as PGs that are not clean will 'quickly' become clean, the range of maps needing to be kept around will be fairly small and the cluster could carry on in this state indefinitely. Thanks for your various recommendations, there are definitely a few things we don't do that we should (e.g. a balancer schedule). We don't make use upmap-remapped for normal operations currently, but I think what you're proposing here makes a lot of sense, especially combined with a balancer schedule. One of the issues I noted with this approach on this cluster is the inevitability of degraded PGs due to an unrelated failed drive/host stopping[1] the movement of data onto new disk/hosts/generations. This causes us issues in planning big data moves, although is something we could easily tweak. Finally, thanks for the hint about how to identify how many maps are being kept. Being able to track this is really handy, and takes a lot of the guesswork out of understanding the need to take breaks in cluster operations. I think we also need to pay more attention to 'unclean durations' of individual PGs, which is something we can do. Cheers, Tom [1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/balancer/module.py#L10... ________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Tuesday, January 7, 2025 21:15 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state Hi Tom, On Tue, Jan 7, 2025 at 10:15 AM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk> wrote:
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
I find it's always best to aim to have all PGs clean at least once a week -- that way the osdmaps can be trimmed at least weekly, preventing all sorts of nastiness, one of which you mentioned here. Here's my recommended mgr balancer tuning: # Balance PGs Sunday to Friday, letting the backfilling finish on Saturdays. (adjust the exact days if needed -- the goal here is that at some point in the week, there needs to be 0 misplaced and 0 degraded objects.) ceph config set mgr mgr/balancer/begin_weekday 0 ceph config set mgr mgr/balancer/end_weekday 5 # [Alternatively] Balance PGs during working hours, letting the backfilling finish over night: ceph config set mgr mgr/balancer/begin_time 0830 ceph config set mgr mgr/balancer/end_time 1800 # Decrease the max misplaced from the default 5% to 0.5%, to minimize the impact of backfilling and ensure the tail of backfilling PGs can finish over the weekend or over night -- increase this percentage if your cluster can tolerate it. (IMHO 5% is way too many misplaced objects on large clusters, but this is very use-case-specific). ceph config set mgr target_max_misplaced_ratio 0.005 # Configure the balancer to aim for +/- 1 PG per pool per OSD -- this is the best uniformity we can hope for with the mgr balancer ceph config set mgr mgr/balancer/upmap_max_deviation 1 Then whenever you add/remove hardware, here's my recommended procedure: 1. Set some flags to prevent data from moving immediately when we add new OSDs: ceph osd set norebalance ceph balancer off 2. Add the new OSDs. (Or start draining -- but note that if you are draining OSDs, set the crush weights to 0.1 instead of 0.0 -- upmap magic tools don't work with OSDs having crush weight = 0). 3. Run ./upmap-remapped.py [1] until the number of misplaced objects is as close as possible to zero. 4. Then unset the flags so data starts rebalancing again. I.e. the mgr balancer will move data in a controlled manner to those new empty OSDs: ceph osd unset norebalance ceph balancer on I have a couple talks about this for more on this topic: - https://www.youtube.com/watch?v=6PQYHlerJ8k - https://www.youtube.com/watch?v=A4xG975UWts We also have a plan to get this logic directly into ceph: https://tracker.ceph.com/issues/67418 As to what you can do right now -- it's actually a great time to test out the above approach. Here's exactly what I'd do: 1. Stop those new OSDs (the ones that are not "in" yet) -- no point having them pull in 30000 osdmaps. Nothing should be degraded at this point -- if so, you either stopped too many OSDs, or there was some OSD flap that you need to recover from. 2. Since you have several remapped PGs right now, that's a perfect time to use upmap-remapped.py [1] -- it'll make the remapped PGs clean again. So try running it: ceph balancer off # disabled the mgr balancer, otherwise it would "undo" what we do next ./upmap-remapped.py # this just outputs commands directly to stdout. ./upmap-remapped.py | sh -x # this will run those commands. ./upmap-remapped.py | sh -x # run it again -- normally we need to just run it twice to get to a minimal number of misplaced PGs. 3. When you run it, you should see the % misplaced objects decreasing. Ideally it will go to 0, meaning all PGs are active+clean. At that point the OSDmaps should trim. 4. Confirm that osdmaps have trimmed by looking at the `ceph report`: ceph report | jq '(.osdmap_last_committed - .osdmap_first_committed)' ^^ the number above should be less than 750. If not -- then the osdmaps are not trimmed, and you need to investigate further. 5. Now start those new OSDs, they should pull in the ~750 osdmaps quickly, and then do the upmap-remapped procedure after configuring the balancer as I described. Hope this all helps, Happy New Year Tom. Cheers, Dan [1] https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remap... -- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
Just to check, are you recommending that at some point each week all PGs are clean *at the same time*, or that no PGs should be unclean for more than a week?
The former I think, so that the cluster is converged, which in turn enables the mons to cull old maps and compact their DBs.
The latter absolutely makes sense, but the former can be quite hard to manage sometimes this cluster, with about one drive failure a week we're somewhat at the mercy of probability. We do always try and aim for 'clean-ish' every so often though :)
I think Dan’s suggestion of upmap-remapped is intended to address that, it lets one (temporarily) convince the cluster that there are no PGs / RADOS objects remapped / backfilling / recovering, so that the old maps can be released and everything compacted.
Also, just to double check my understanding here, the cluster needs to keep hold of osdmaps going back to the point at which the currently unclean PGs were last clean?
At PG granularity, I think so.
So if a cluster has a bunch of backfill being queued continuously for a month, but individual PGs get remapped and then backfilled quickly (e.g. ~1day), the cluster only needs to hold onto maps for the day, rather than the entire month period? Or am I missing something?
I suspect that the actual experience may be different, that PG convergence isn’t strictly a FIFO.
The above is how I would imagine an even larger cluster would operate, with the expectation is that there will always be at least one non-clean PG at any time
Dan made that important observation like ten years ago at an OpenStack Summit ;)
As long as PGs that are not clean will 'quickly' become clean, the range of maps needing to be kept around will be fairly small and the cluster could carry on in this state indefinitely.
That’s the thing, I wouldn’t assume that the range is a small, sliding window. Successive changes to topology and state may kick a given PG’s convergence back to the end of the line so to speak. Speculation on my part.
Thanks for your various recommendations, there are definitely a few things we don't do that we should (e.g. a balancer schedule).
We don't make use upmap-remapped for normal operations currently, but I think what you're proposing here makes a lot of sense, especially combined with a balancer schedule. One of the issues I noted with this approach on this cluster is the inevitability of degraded PGs due to an unrelated failed drive/host stopping[1] the movement of data onto new disk/hosts/generations. This causes us issues in planning big data moves, although is something we could easily tweak.
This is one of the nuances that drives me to counsel against HDDs — time to recover and thus increased risk of overlapping failures. R4 or EC with a relatively high value of m can guard against overlapping failures, but will themselves increase MTTR.
Finally, thanks for the hint about how to identify how many maps are being kept. Being able to track this is really handy, and takes a lot of the guesswork out of understanding the need to take breaks in cluster operations. I think we also need to pay more attention to 'unclean durations' of individual PGs, which is something we can do.
Dan is a treasure.
Cheers, Tom
[1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/balancer/module.py#L10... ________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Tuesday, January 7, 2025 21:15 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
Hi Tom,
On Tue, Jan 7, 2025 at 10:15 AM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk> wrote:
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
I find it's always best to aim to have all PGs clean at least once a week -- that way the osdmaps can be trimmed at least weekly, preventing all sorts of nastiness, one of which you mentioned here.
Here's my recommended mgr balancer tuning:
# Balance PGs Sunday to Friday, letting the backfilling finish on Saturdays. (adjust the exact days if needed -- the goal here is that at some point in the week, there needs to be 0 misplaced and 0 degraded objects.) ceph config set mgr mgr/balancer/begin_weekday 0 ceph config set mgr mgr/balancer/end_weekday 5
# [Alternatively] Balance PGs during working hours, letting the backfilling finish over night: ceph config set mgr mgr/balancer/begin_time 0830 ceph config set mgr mgr/balancer/end_time 1800
# Decrease the max misplaced from the default 5% to 0.5%, to minimize the impact of backfilling and ensure the tail of backfilling PGs can finish over the weekend or over night -- increase this percentage if your cluster can tolerate it. (IMHO 5% is way too many misplaced objects on large clusters, but this is very use-case-specific). ceph config set mgr target_max_misplaced_ratio 0.005
# Configure the balancer to aim for +/- 1 PG per pool per OSD -- this is the best uniformity we can hope for with the mgr balancer ceph config set mgr mgr/balancer/upmap_max_deviation 1
Then whenever you add/remove hardware, here's my recommended procedure:
1. Set some flags to prevent data from moving immediately when we add new OSDs: ceph osd set norebalance ceph balancer off
2. Add the new OSDs. (Or start draining -- but note that if you are draining OSDs, set the crush weights to 0.1 instead of 0.0 -- upmap magic tools don't work with OSDs having crush weight = 0).
3. Run ./upmap-remapped.py [1] until the number of misplaced objects is as close as possible to zero.
4. Then unset the flags so data starts rebalancing again. I.e. the mgr balancer will move data in a controlled manner to those new empty OSDs:
ceph osd unset norebalance ceph balancer on
I have a couple talks about this for more on this topic: - https://www.youtube.com/watch?v=6PQYHlerJ8k - https://www.youtube.com/watch?v=A4xG975UWts
We also have a plan to get this logic directly into ceph: https://tracker.ceph.com/issues/67418
As to what you can do right now -- it's actually a great time to test out the above approach. Here's exactly what I'd do:
1. Stop those new OSDs (the ones that are not "in" yet) -- no point having them pull in 30000 osdmaps. Nothing should be degraded at this point -- if so, you either stopped too many OSDs, or there was some OSD flap that you need to recover from.
2. Since you have several remapped PGs right now, that's a perfect time to use upmap-remapped.py [1] -- it'll make the remapped PGs clean again. So try running it:
ceph balancer off # disabled the mgr balancer, otherwise it would "undo" what we do next ./upmap-remapped.py # this just outputs commands directly to stdout. ./upmap-remapped.py | sh -x # this will run those commands. ./upmap-remapped.py | sh -x # run it again -- normally we need to just run it twice to get to a minimal number of misplaced PGs.
3. When you run it, you should see the % misplaced objects decreasing. Ideally it will go to 0, meaning all PGs are active+clean. At that point the OSDmaps should trim.
4. Confirm that osdmaps have trimmed by looking at the `ceph report`:
ceph report | jq '(.osdmap_last_committed - .osdmap_first_committed)'
^^ the number above should be less than 750. If not -- then the osdmaps are not trimmed, and you need to investigate further.
5. Now start those new OSDs, they should pull in the ~750 osdmaps quickly, and then do the upmap-remapped procedure after configuring the balancer as I described.
Hope this all helps, Happy New Year Tom.
Cheers, Dan
[1] https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remap...
-- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tom,
Just to check, are you recommending that at some point each week all PGs are clean *at the same time*, or that no PGs should be unclean for more than a week? The latter absolutely makes sense, but the former can be quite hard to manage sometimes this cluster, with about one drive failure a week we're somewhat at the mercy of probability. We do always try and aim for 'clean-ish' every so often though :) Also, just to double check my understanding here, the cluster needs to keep hold of osdmaps going back to the point at which the currently unclean PGs were last clean? So if a cluster has a bunch of backfill being queued continuously for a month, but individual PGs get remapped and then backfilled quickly (e.g. ~1day), the cluster only needs to hold onto maps for the day, rather than the entire month period? Or am I missing something?
The mon keeps track of the "last epoch clean" which is indeed the lowest osd epoch number out of *all* PGs (and also all *pools*, but that's usually not relevant). Clean means active, not degraded, and not remapped. So if you have any single PG which is not active, or it is degraded, undersized, or remapped -- then that's an unclean state for the cluster. The mons trims the osdmaps up to that "last epoch clean". (And all OSDs similarly keep the same osdmaps that the mon keeps). So my "rule of thumb" is that you should aim to have a fully clean cluster about once per week, because one week's worth of OSDmaps is a relatively manageable number of maps for all the daemons in the cluster to store and track.
It's pulling down OSD maps, 40 at a time, taking about 4 seconds each time. With the ~30,000(!) OSD maps it pulls down, it takes approximately an hour. At ~4MB a map, this then matches up with the ~115GB storage consumed by the resulting OSD with no PGs.
One small correction here: the OSDs *normally* get a delta-encoded version of each map -- a so-called "incremental map". Those incremental maps are much smaller than the full sized 4MB. And by default, the OSD stores each map in memory in a "deduplicated" manner, to minimize the memory consumption. You can confirm that (at least the memory part) by looking at the osdmap section in dump_mempools -- it's storring lots of maps, but not 115GB of memory ;-) But indeed those large size maps, and large number of maps, can create some pretty big issues: 1. Bootstrapping new OSDs takes a long time, as you're seeing. 2. In some disaster cases, the network bandwidth needed to serve up those osdmaps is too much for the mons to deliver. This is why we normally suggest more mons for larger clusters -- just to have the bandwidth to serve up osdmaps in that rare case it's needed. 3. The mon db needs to store all the maps, and that consumes a lot of space. You must be getting mon db size warnings, right? Anything more than ~10GB is getting too hefty, imho. 4...
One of the issues I noted with this approach on this cluster is the inevitability of degraded PGs due to an unrelated failed drive
You can still use upmap-remaped.py in that case -- if an osd fails, the "remapped" pgs will get mapped back to the current OSDs to avoid useless data movement. But the degraded replicas/shards will still be recovered as usual, which is good. But you're right -- the mgr balancer pauses if the cluster is degraded. IMHO that's usually fine -- we don't want a bunch of extra data movement created while we're trying to recover from a degraded state. But there is at least one case where the balancer would help *a lot* with degraded PGs, namely: https://tracker.ceph.com/issues/66755 I'd be happy to keep chatting to help you all get onto this mode of operating. Frankly, I don't know how anyone can operate a large Ceph cluster without upmap-remapped or pgremapper, which is why we're trying to get this approach upstreamed. Cheers, Dan On Wed, Jan 8, 2025 at 6:20 AM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk> wrote:
Hi Dan,
Happy new year!
I find it's always best to aim to have all PGs clean at least once a week -- that way the osdmaps can be trimmed at least weekly, preventing all sorts of nastiness, one of which you mentioned here.
Just to check, are you recommending that at some point each week all PGs are clean *at the same time*, or that no PGs should be unclean for more than a week?
The latter absolutely makes sense, but the former can be quite hard to manage sometimes this cluster, with about one drive failure a week we're somewhat at the mercy of probability. We do always try and aim for 'clean-ish' every so often though :)
Also, just to double check my understanding here, the cluster needs to keep hold of osdmaps going back to the point at which the currently unclean PGs were last clean? So if a cluster has a bunch of backfill being queued continuously for a month, but individual PGs get remapped and then backfilled quickly (e.g. ~1day), the cluster only needs to hold onto maps for the day, rather than the entire month period? Or am I missing something?
The above is how I would imagine an even larger cluster would operate, with the expectation is that there will always be at least one non-clean PG at any time. As long as PGs that are not clean will 'quickly' become clean, the range of maps needing to be kept around will be fairly small and the cluster could carry on in this state indefinitely.
Thanks for your various recommendations, there are definitely a few things we don't do that we should (e.g. a balancer schedule).
We don't make use upmap-remapped for normal operations currently, but I think what you're proposing here makes a lot of sense, especially combined with a balancer schedule. One of the issues I noted with this approach on this cluster is the inevitability of degraded PGs due to an unrelated failed drive/host stopping[1] the movement of data onto new disk/hosts/generations. This causes us issues in planning big data moves, although is something we could easily tweak.
Finally, thanks for the hint about how to identify how many maps are being kept. Being able to track this is really handy, and takes a lot of the guesswork out of understanding the need to take breaks in cluster operations. I think we also need to pay more attention to 'unclean durations' of individual PGs, which is something we can do.
Cheers, Tom
[1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/balancer/module.py#L10... ________________________________________ From: Dan van der Ster <dan.vanderster@clyso.com> Sent: Tuesday, January 7, 2025 21:15 To: Byrne, Thomas (STFC,RAL,SC) <tom.byrne@stfc.ac.uk> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Slow initial boot of OSDs in large cluster with unclean state
Hi Tom,
On Tue, Jan 7, 2025 at 10:15 AM Thomas Byrne - STFC UKRI <tom.byrne@stfc.ac.uk> wrote:
I realise the obvious answer here is don't leave big cluster in an unclean state for this long. Currently we've got PGs that have been remapped for 5 days, which matches the 30,000 OSDMap epoch range perfectly. This is something we're always looking at from a procedure point of view e.g. keeping max_backfills as high as possible by default, ensuring balancer max_misplaced is appropriate, re-evaluating disk and node addition/removal processes. But the reality on this cluster is that sometimes these 'logjams' happen, and it would be good to understand if we can improve the OSD addition experience so we can continue to be flexible with our operation scheduling.
I find it's always best to aim to have all PGs clean at least once a week -- that way the osdmaps can be trimmed at least weekly, preventing all sorts of nastiness, one of which you mentioned here.
Here's my recommended mgr balancer tuning:
# Balance PGs Sunday to Friday, letting the backfilling finish on Saturdays. (adjust the exact days if needed -- the goal here is that at some point in the week, there needs to be 0 misplaced and 0 degraded objects.) ceph config set mgr mgr/balancer/begin_weekday 0 ceph config set mgr mgr/balancer/end_weekday 5
# [Alternatively] Balance PGs during working hours, letting the backfilling finish over night: ceph config set mgr mgr/balancer/begin_time 0830 ceph config set mgr mgr/balancer/end_time 1800
# Decrease the max misplaced from the default 5% to 0.5%, to minimize the impact of backfilling and ensure the tail of backfilling PGs can finish over the weekend or over night -- increase this percentage if your cluster can tolerate it. (IMHO 5% is way too many misplaced objects on large clusters, but this is very use-case-specific). ceph config set mgr target_max_misplaced_ratio 0.005
# Configure the balancer to aim for +/- 1 PG per pool per OSD -- this is the best uniformity we can hope for with the mgr balancer ceph config set mgr mgr/balancer/upmap_max_deviation 1
Then whenever you add/remove hardware, here's my recommended procedure:
1. Set some flags to prevent data from moving immediately when we add new OSDs: ceph osd set norebalance ceph balancer off
2. Add the new OSDs. (Or start draining -- but note that if you are draining OSDs, set the crush weights to 0.1 instead of 0.0 -- upmap magic tools don't work with OSDs having crush weight = 0).
3. Run ./upmap-remapped.py [1] until the number of misplaced objects is as close as possible to zero.
4. Then unset the flags so data starts rebalancing again. I.e. the mgr balancer will move data in a controlled manner to those new empty OSDs:
ceph osd unset norebalance ceph balancer on
I have a couple talks about this for more on this topic: - https://www.youtube.com/watch?v=6PQYHlerJ8k - https://www.youtube.com/watch?v=A4xG975UWts
We also have a plan to get this logic directly into ceph: https://tracker.ceph.com/issues/67418
As to what you can do right now -- it's actually a great time to test out the above approach. Here's exactly what I'd do:
1. Stop those new OSDs (the ones that are not "in" yet) -- no point having them pull in 30000 osdmaps. Nothing should be degraded at this point -- if so, you either stopped too many OSDs, or there was some OSD flap that you need to recover from.
2. Since you have several remapped PGs right now, that's a perfect time to use upmap-remapped.py [1] -- it'll make the remapped PGs clean again. So try running it:
ceph balancer off # disabled the mgr balancer, otherwise it would "undo" what we do next ./upmap-remapped.py # this just outputs commands directly to stdout. ./upmap-remapped.py | sh -x # this will run those commands. ./upmap-remapped.py | sh -x # run it again -- normally we need to just run it twice to get to a minimal number of misplaced PGs.
3. When you run it, you should see the % misplaced objects decreasing. Ideally it will go to 0, meaning all PGs are active+clean. At that point the OSDmaps should trim.
4. Confirm that osdmaps have trimmed by looking at the `ceph report`:
ceph report | jq '(.osdmap_last_committed - .osdmap_first_committed)'
^^ the number above should be less than 750. If not -- then the osdmaps are not trimmed, and you need to investigate further.
5. Now start those new OSDs, they should pull in the ~750 osdmaps quickly, and then do the upmap-remapped procedure after configuring the balancer as I described.
Hope this all helps, Happy New Year Tom.
Cheers, Dan
[1] https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remap...
-- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
-- Dan van der Ster CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
participants (8)
-
Anthony D'Atri
-
Dan van der Ster
-
Frédéric Nass
-
Gregory Orange
-
Joshua Baergen
-
Stillwell, Bryan
-
Thomas Byrne - STFC UKRI
-
Wesley Dillingham