OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
Hi! After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag". The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs". Tracker issue with logs: https://tracker.ceph.com/issues/50017 Anyone else encountered this error? I've "suspended" the upgrade for now :) -- Jonas
Hi, Saw that, looks scary! I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try: "bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe. "bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198 -- Dan On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :) At least non-racecondition bugs are hopefully easier to spot... I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out. -- Jonas On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;) Have you excluded any possible hardware issues? 15.2.10 has a new option to check for all zero reads; maybe try it with true? Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"), The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you? Cheers, Dan On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry for being too late to the party... I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run. The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair. I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377 Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701) Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist... If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20... I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though.. Thanks, Igor On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is there a way to check for these zombie blobs, and other issues needing repair, prior to the upgrade? That would allow us to know that issues might be coming, and perhaps address them before they result in corrupt OSDs. I'm considering upgrading our clusters from 14 to 15, and would really like to avoid these kinds of issues. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 7:55 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag Sorry for being too late to the party... I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run. The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair. I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377 Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701) Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist... If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20... I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though.. Thanks, Igor On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The workaround would be to disable bluestore_fsck_quick_fix_on_mount, do an upgrade and then do a regular fsck. Depending on fsck results either proceed with a repair or not. Thanks, Igor On 4/12/2021 6:35 PM, DHilsbos@performair.com wrote:
Is there a way to check for these zombie blobs, and other issues needing repair, prior to the upgrade? That would allow us to know that issues might be coming, and perhaps address them before they result in corrupt OSDs.
I'm considering upgrading our clusters from 14 to 15, and would really like to avoid these kinds of issues.
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 7:55 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Igor; Does this only impact CephFS then? Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 9:16 AM To: Dominic Hilsbos; ceph-users@ceph.io Subject: Re: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag The workaround would be to disable bluestore_fsck_quick_fix_on_mount, do an upgrade and then do a regular fsck. Depending on fsck results either proceed with a repair or not. Thanks, Igor On 4/12/2021 6:35 PM, DHilsbos@performair.com wrote:
Is there a way to check for these zombie blobs, and other issues needing repair, prior to the upgrade? That would allow us to know that issues might be coming, and perhaps address them before they result in corrupt OSDs.
I'm considering upgrading our clusters from 14 to 15, and would really like to avoid these kinds of issues.
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 7:55 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Igor! I have plenty of OSDs to loose, as long as the recovery works well afterward, so I can go ahead with it :D What debug flags should I activate? osd=10, bluefs=20, bluestore=20, rocksdb=10, ...? I'm not sure it's really the transaction size, since the broken WriteBatch is dumped, and the command index is out of range (that's the WriteBatch tag). I don't see why the transaction size would result in such a corruption - my naive look at the rocksdb sources looks like 14851 repairs shouldn't overflow the 32-bit WriteBatch entry counter, but who knows. Are rocksdb keys like this normal? If yes, what's the construction logic? The pool is called 'dumpsite'. 0x80800000000000000a194027'Rdumpsite!rbd_data.6.28423ad8f48ca1.0000000001b366ff!='0xfffffffffffffffeffffffffffffffff'o' 0x80800000000000000a1940f69264756d'psite!rbd_data.6.28423ad8f48ca1.00000000011bdd0c!='0xfffffffffffffffeffffffffffffffff'o' -- Jonas On 12/04/2021 16.54, Igor Fedotov wrote:
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
No, it has absolutely no relation to CephFS. I presume it's a generic Bluestore/BlueFS issue. On 4/12/2021 9:07 PM, DHilsbos@performair.com wrote:
Igor;
Does this only impact CephFS then?
Thank you,
Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 9:16 AM To: Dominic Hilsbos; ceph-users@ceph.io Subject: Re: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
The workaround would be to disable bluestore_fsck_quick_fix_on_mount, do an upgrade and then do a regular fsck.
Depending on fsck results either proceed with a repair or not.
Thanks,
Igor
On 4/12/2021 6:35 PM, DHilsbos@performair.com wrote:
Is there a way to check for these zombie blobs, and other issues needing repair, prior to the upgrade? That would allow us to know that issues might be coming, and perhaps address them before they result in corrupt OSDs.
I'm considering upgrading our clusters from 14 to 15, and would really like to avoid these kinds of issues.
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 7:55 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, every osd in a SSD that I have upgraded from 15.2.9->15.210 logs errors like the ones below. The osd's in HD or NVME don't. But they restart ok and a deep-scrub of the entire pool finishes ok. Could be the same bug? 2021-04-14T00:29:27.740+0200 7f364750d700 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: truncated block read from db/044714.sst offset 18446744073709551615, expected 4 bytes, got 0 2021-04-14T00:29:51.852+0200 7f364750d700 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2326482265 in db/044743.sst offset 18446744073709551615 size 18446744073709551615 Br. El 12/4/21 a las 18:15, Igor Fedotov escribió:
The workaround would be to disable bluestore_fsck_quick_fix_on_mount, do an upgrade and then do a regular fsck.
Depending on fsck results either proceed with a repair or not.
Thanks,
Igor
On 4/12/2021 6:35 PM, DHilsbos@performair.com wrote:
Is there a way to check for these zombie blobs, and other issues needing repair, prior to the upgrade? That would allow us to know that issues might be coming, and perhaps address them before they result in corrupt OSDs.
I'm considering upgrading our clusters from 14 to 15, and would really like to avoid these kinds of issues.
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Igor Fedotov [mailto:ifedotov@suse.de] Sent: Monday, April 12, 2021 7:55 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you ... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL, Option::LEVEL_DEV) .set_default(false) .set_flag(Option::FLAG_RUNTIME) .set_description("Check data read for suspicious pages") .set_long_description("Looks into data read to check if there is a 4K block entirely filled with zeros. " "If this happens, we re-read data. If there is difference, we print error to log.") .add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does 15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@in.tum.de> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded another OSD. I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking that if you have already backfilled the degraded PGs, and can afford to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because https://github.com/facebook/rocksdb/issues/5068 showed a similar crash and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade for now :)
-- Jonas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Just pinging to check if this issue was understood yet? Cheers, Dan On Mon, Apr 12, 2021 at 9:12 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi Igor!
I have plenty of OSDs to loose, as long as the recovery works well afterward, so I can go ahead with it :D
What debug flags should I activate? osd=10, bluefs=20, bluestore=20, rocksdb=10, ...?
I'm not sure it's really the transaction size, since the broken WriteBatch is dumped, and the command index is out of range (that's the WriteBatch tag). I don't see why the transaction size would result in such a corruption - my naive look at the rocksdb sources looks like 14851 repairs shouldn't overflow the 32-bit WriteBatch entry counter, but who knows.
Are rocksdb keys like this normal? If yes, what's the construction logic? The pool is called 'dumpsite'.
0x80800000000000000a194027'Rdumpsite!rbd_data.6.28423ad8f48ca1.0000000001b366ff!='0xfffffffffffffffeffffffffffffffff'o' 0x80800000000000000a1940f69264756d'psite!rbd_data.6.28423ad8f48ca1.00000000011bdd0c!='0xfffffffffffffffeffffffffffffffff'o'
-- Jonas
On 12/04/2021 16.54, Igor Fedotov wrote:
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi! Unfortunately no, I've done some digging but didn't find a cause or solution yet. Igor, whats your suggestion how we should look for a solution? -- Jonas On 27/04/2021 09.47, Dan van der Ster wrote:
Hi,
Just pinging to check if this issue was understood yet?
Cheers, Dan
On Mon, Apr 12, 2021 at 9:12 PM Jonas Jelten <jelten@in.tum.de> wrote:
Hi Igor!
I have plenty of OSDs to loose, as long as the recovery works well afterward, so I can go ahead with it :D
What debug flags should I activate? osd=10, bluefs=20, bluestore=20, rocksdb=10, ...?
I'm not sure it's really the transaction size, since the broken WriteBatch is dumped, and the command index is out of range (that's the WriteBatch tag). I don't see why the transaction size would result in such a corruption - my naive look at the rocksdb sources looks like 14851 repairs shouldn't overflow the 32-bit WriteBatch entry counter, but who knows.
Are rocksdb keys like this normal? If yes, what's the construction logic? The pool is called 'dumpsite'.
0x80800000000000000a194027'Rdumpsite!rbd_data.6.28423ad8f48ca1.0000000001b366ff!='0xfffffffffffffffeffffffffffffffff'o' 0x80800000000000000a1940f69264756d'psite!rbd_data.6.28423ad8f48ca1.00000000011bdd0c!='0xfffffffffffffffeffffffffffffffff'o'
-- Jonas
On 12/04/2021 16.54, Igor Fedotov wrote:
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I presumt it's the one which causes BlueFS data corruption due to huge transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a rarely met bug by itself) but we had to some degree similar issue with upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the nearest future anyway though..
Thanks,
Igor
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Dan van der Ster
-
DHilsbos＠performair.com
-
Igor Fedotov
-
Jonas Jelten
-
Jorge Boncompte