Igor; Does this only impact CephFS then? Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 9:16 AM To: Dominic Hilsbos; ceph-users@ceph.io Subject: Re: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag The workaround would be to disable bluestore_fsck_quick_fix_on_mount, do an upgrade and then do a regular fsck. Depending on fsck results either proceed with a repair or not. Thanks, Igor On 4/12/2021 6:35 PM, DHilsbos@performair.com wrote:
Is there a way to check for these zombie blobs, and other issues needing repair, prior to the upgrade? That would allow us to know that issues might be coming, and perhaps address them before they result in corrupt OSDs.
I'm considering upgrading our clusters from 14 to 15, and would really like to avoid these kinds of issues.
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 7:55 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io