OSD takes more almost two hours to boot from Luminous -> Nautilus
Hi, Last night I upgraded a Luminous cluster to Nautilus. All went well, but there was one sleep depriving issue I would like to prevent from happening next week while upgrading another cluster. Maybe you people can help me figure out what actually happened. So I upgraded the packages and restarted mons and mgrs. Then I started restarting the OSD's on one of the nodes. Below are the start and 'start_boot' times, during which the disks read at full speed, I think the whole disk. 2020-08-19 02:08:10.568 7fd742b09c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:09:33.591 7fd742b09c80 1 osd.8 2188 start_boot 2020-08-19 02:08:10.592 7fb453887c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:17:40.878 7fb453887c80 1 osd.5 2188 start_boot 2020-08-19 02:08:10.836 7f907bc0cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:19:58.462 7f907bc0cc80 1 osd.3 2188 start_boot 2020-08-19 02:08:10.584 7f1ca892cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 03:13:24.179 7f1ca892cc80 1 osd.11 2188 start_boot 2020-08-19 02:08:10.568 7f059f80dc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 04:06:55.342 7f059f80dc80 1 osd.14 2188 start_boot So, while this is not an issue which breaks anything technical, I would like to know how I can arrange for the OSD to do this 'maintenance' beforehand so I don't have to wait too long. :) I do see a warning in the logging, is that related: "store not yet converted to per-pool stats" ? Thanks! -- Mark Schouten <mark@tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208
Hi Mark, most probably you're suffering from a per-pool omap statistics update. Which is performed during the first start of an upgraded OSD. One can disable this behavior via 'bluestore_fsck_quick_fix_on_mount' flag. But please expect incomplete OMAP usage report if stats are not updated. Thanks, Igor On 8/19/2020 1:16 PM, Mark Schouten wrote:
Hi,
Last night I upgraded a Luminous cluster to Nautilus. All went well, but there was one sleep depriving issue I would like to prevent from happening next week while upgrading another cluster. Maybe you people can help me figure out what actually happened.
So I upgraded the packages and restarted mons and mgrs. Then I started restarting the OSD's on one of the nodes. Below are the start and 'start_boot' times, during which the disks read at full speed, I think the whole disk.
2020-08-19 02:08:10.568 7fd742b09c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:09:33.591 7fd742b09c80 1 osd.8 2188 start_boot
2020-08-19 02:08:10.592 7fb453887c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:17:40.878 7fb453887c80 1 osd.5 2188 start_boot
2020-08-19 02:08:10.836 7f907bc0cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:19:58.462 7f907bc0cc80 1 osd.3 2188 start_boot
2020-08-19 02:08:10.584 7f1ca892cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 03:13:24.179 7f1ca892cc80 1 osd.11 2188 start_boot
2020-08-19 02:08:10.568 7f059f80dc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 04:06:55.342 7f059f80dc80 1 osd.14 2188 start_boot
So, while this is not an issue which breaks anything technical, I would like to know how I can arrange for the OSD to do this 'maintenance' beforehand so I don't have to wait too long. :)
I do see a warning in the logging, is that related: "store not yet converted to per-pool stats" ?
Thanks! -- Mark Schouten <mark@tuxis.nl>
Tuxis, Ede, https://www.tuxis.nl
T: +31 318 200208
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Aug 19, 2020 at 03:55:17PM +0300, Igor Fedotov wrote:
most probably you're suffering from a per-pool omap statistics update. Which is performed during the first start of an upgraded OSD. One can disable this behavior via 'bluestore_fsck_quick_fix_on_mount' flag. But please expect incomplete OMAP usage report if stats are not updated.
Thanks! If there is no technical issue with it, I think I could just configure bluestore_fsck_quick_fix_on_mount in ceph.conf, do the upgrade 'quick' and afterwards remove the configline and restart the osd's again at a more convenient time? -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info@tuxis.nl
yeah, this should work... On 8/19/2020 4:04 PM, Mark Schouten wrote:
On Wed, Aug 19, 2020 at 03:55:17PM +0300, Igor Fedotov wrote:
most probably you're suffering from a per-pool omap statistics update. Which is performed during the first start of an upgraded OSD. One can disable this behavior via 'bluestore_fsck_quick_fix_on_mount' flag. But please expect incomplete OMAP usage report if stats are not updated.
Thanks! If there is no technical issue with it, I think I could just configure bluestore_fsck_quick_fix_on_mount in ceph.conf, do the upgrade 'quick' and afterwards remove the configline and restart the osd's again at a more convenient time?
participants (2)
-
Igor Fedotov
-
Mark Schouten