Consequences of setting bluestore_fsck_quick_fix_on_mount to false?
Hi, Looking at the Octopus upgrade instructions, I see "the first time each OSD starts, it will do a format conversion to improve the accounting for “omap” data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data)." and that I can disable this by setting bluestore_fsck_quick_fix_on_mount to false. A couple of questions about this: i) what are the consequences of turning off this "quick fix"? Is it possible to have it run in the background or similar? ii) is there any way to narrow down the time estimate? Our production cluster has 3060 OSDs on hdd (with block.db on NVME), and obviously 3000 lots of "a few hours" is an awful lot of time... I'll be doing some testing on our test cluster (by putting 10M objects into an S3 bucket before trying the upgrade), but it'd be useful to have some idea of how this is likely to work at scale... Thanks, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Hi Matthew, Which version are you upgrading from? If recent nautilus, you may have already completed this conversion. When we did this fsck (not with octopus, but to a nautilus point release that had this conversion backported), we first upgraded one single osd just to see the typical downtime for our data. On our S3 cluster, the conversion completed within just a couple of minutes per OSD, so we decided to leave bluestore_fsck_quick_fix_on_mount at its default true, and did all the fsck's as we updated the osds. AFAIK, if you set bluestore_fsck_quick_fix_on_mount to false and postpone that conversion, you'll just have less-accurate `ceph df` for omap data. Cheers, Dan On Mon, Feb 15, 2021 at 6:00 PM Matthew Vernon <mv3@sanger.ac.uk> wrote:
Hi,
Looking at the Octopus upgrade instructions, I see "the first time each OSD starts, it will do a format conversion to improve the accounting for “omap” data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data)." and that I can disable this by setting bluestore_fsck_quick_fix_on_mount to false.
A couple of questions about this:
i) what are the consequences of turning off this "quick fix"? Is it possible to have it run in the background or similar?
ii) is there any way to narrow down the time estimate? Our production cluster has 3060 OSDs on hdd (with block.db on NVME), and obviously 3000 lots of "a few hours" is an awful lot of time...
I'll be doing some testing on our test cluster (by putting 10M objects into an S3 bucket before trying the upgrade), but it'd be useful to have some idea of how this is likely to work at scale...
Thanks,
Matthew
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 16/02/2021 08:06, Dan van der Ster wrote:
Which version are you upgrading from? If recent nautilus, you may have already completed this conversion.
Mimic (well, really Luminous with a pit-stop at Mimic).
When we did this fsck (not with octopus, but to a nautilus point release that had this conversion backported), we first upgraded one single osd just to see the typical downtime for our data. On our S3 cluster, the conversion completed within just a couple of minutes per OSD, so we decided to leave bluestore_fsck_quick_fix_on_mount at its default true, and did all the fsck's as we updated the osds.
Thanks; I'll see what it looks like on the test cluster. Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
participants (2)
-
Dan van der Ster
-
Matthew Vernon