Ceph 16.2.14: osd crash, bdev() _aio_thread got r=-1 ((1) Operation not permitted)
Hi, One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug. Usually an OSD, which crashed because of this bug, restarts within seconds and continues normal operation. This time it failed to restart and kept crashing: "assert_condition": "abort", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc", "assert_func": "void KernelDevice::_aio_thread()", "assert_line": 604, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f08520e2700 time 2023-12-03T04:00:36.689614+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg(\"Unexpected IO error. This may suggest HW issue. Please check your dmesg!\")\n", "assert_thread_name": "bstore_aio", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7f085e308cf0]", "gsignal()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x55f01d9494cb]", "(KernelDevice::_aio_thread()+0x1285) [0x55f01e4b5c15]", "(KernelDevice::AioCompletionThread::entry()+0x11) [0x55f01e4c0ee1]", "/lib64/libpthread.so.0(+0x81ca) [0x7f085e2fe1ca]", "clone()" ], There was nothing in dmesg though and the block device looked healthy. I took the OSD down, ran a long SMART test on its block drive, ran a read test on the drive and found no issues. I tried restarting the OSD again and found in its debug that it failed because of an "2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation not permitted)" error: https://pastebin.com/gDat6rfk I remember hitting this previously: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/GYL72G3F4PP..., and this time a host reboot completely resolved the issue. It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions. Best regards, Zakhar
On Sun, Dec 03, 2023 at 06:53:08AM +0200, Zakhar Kirpichenko wrote:
One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug.
<snip />
It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of https://tracker.ceph.com/issues/53907 In there you have the fix and a workaround until next minor is released. -- Kai Stian Olstad
Thanks! The bug I referenced is the reason for the 1st OSD crash, but not for the subsequent crashes. The reason for those is described where you <snip />. I'm asking for help with that one. /Z On Sun, 3 Dec 2023 at 15:31, Kai Stian Olstad <ceph+list@olstad.com> wrote:
On Sun, Dec 03, 2023 at 06:53:08AM +0200, Zakhar Kirpichenko wrote:
One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug.
<snip />
It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of https://tracker.ceph.com/issues/53907
In there you have the fix and a workaround until next minor is released.
-- Kai Stian Olstad
Hi, Just to reiterate, I'm referring to an OSD crash loop because of the following error: "2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation not permitted)". More relevant log entries: https://pastebin.com/gDat6rfk The crash log suggested that there could be a hardware issue but there was none, I was able to access the block device for testing purposes without any issues, and the problem went away after I rebooted the host, this OSD is currently operating without any issues under load. Any ideas? /Z On Sun, 3 Dec 2023 at 16:09, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks! The bug I referenced is the reason for the 1st OSD crash, but not for the subsequent crashes. The reason for those is described where you <snip />. I'm asking for help with that one.
/Z
On Sun, 3 Dec 2023 at 15:31, Kai Stian Olstad <ceph+list@olstad.com> wrote:
On Sun, Dec 03, 2023 at 06:53:08AM +0200, Zakhar Kirpichenko wrote:
One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug.
<snip />
It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of https://tracker.ceph.com/issues/53907
In there you have the fix and a workaround until next minor is released.
-- Kai Stian Olstad
Any input from anyone? /Z On Mon, 4 Dec 2023 at 12:52, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
Just to reiterate, I'm referring to an OSD crash loop because of the following error:
"2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation not permitted)". More relevant log entries: https://pastebin.com/gDat6rfk
The crash log suggested that there could be a hardware issue but there was none, I was able to access the block device for testing purposes without any issues, and the problem went away after I rebooted the host, this OSD is currently operating without any issues under load.
Any ideas?
/Z
On Sun, 3 Dec 2023 at 16:09, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks! The bug I referenced is the reason for the 1st OSD crash, but not for the subsequent crashes. The reason for those is described where you <snip />. I'm asking for help with that one.
/Z
On Sun, 3 Dec 2023 at 15:31, Kai Stian Olstad <ceph+list@olstad.com> wrote:
On Sun, Dec 03, 2023 at 06:53:08AM +0200, Zakhar Kirpichenko wrote:
One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug.
<snip />
It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of https://tracker.ceph.com/issues/53907
In there you have the fix and a workaround until next minor is released.
-- Kai Stian Olstad
On Tue, Dec 5, 2023 at 10:13 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Any input from anyone?
/Z
IIt's not clear whether or not these issues are related. I see three things in this e-mail chain: 1) bdev() _aio_thread with EPERM, as in the subject of this e-mail chain 2) bdev() _aio_thread with the I/O error condition (see [1], which is a *slightly* different if/else switch than the EPERM) 3) The tracker, which seems related to BlueFS (?): https://tracker.ceph.com/issues/53906 Shooting in a dark a bit... but for the EPERM issue you mention, maybe try raising the number of file descriptors available to the process/container/system? Not sure why else would you get EPERM in this context. On 2) I have definitely seen that happen before and it's always matched with an I/O error reported by the kernel in `dmesg` output. Sometimes the drive keeps going along fine after a restart of the OSD/reboot of the system, sometimes not. Cheers, Tyler
On Mon, 4 Dec 2023 at 12:52, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
Just to reiterate, I'm referring to an OSD crash loop because of the following error:
"2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation not permitted)". More relevant log entries: https://pastebin.com/gDat6rfk
The crash log suggested that there could be a hardware issue but there was none, I was able to access the block device for testing purposes without any issues, and the problem went away after I rebooted the host, this OSD is currently operating without any issues under load.
Any ideas?
/Z
On Sun, 3 Dec 2023 at 16:09, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks! The bug I referenced is the reason for the 1st OSD crash, but not for the subsequent crashes. The reason for those is described where you <snip />. I'm asking for help with that one.
/Z
On Sun, 3 Dec 2023 at 15:31, Kai Stian Olstad <ceph+list@olstad.com> wrote:
On Sun, Dec 03, 2023 at 06:53:08AM +0200, Zakhar Kirpichenko wrote:
One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug.
<snip />
It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of https://tracker.ceph.com/issues/53907
In there you have the fix and a workaround until next minor is released.
-- Kai Stian Olstad
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you, Tyler. Unfortunately (or fortunately?) the drive is fine in this case: there were no errors reported by the kernel at the time, and I successfully managed to run a bunch of tests on the drive for many hours before rebooting the host. The drive has worked without any issues for 3 days now. I've already checked the file descriptor numbers, the defaults already are very high and the usage is rather low. /Z On Wed, 6 Dec 2023 at 03:24, Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Tue, Dec 5, 2023 at 10:13 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Any input from anyone?
/Z
IIt's not clear whether or not these issues are related. I see three things in this e-mail chain: 1) bdev() _aio_thread with EPERM, as in the subject of this e-mail chain 2) bdev() _aio_thread with the I/O error condition (see [1], which is a *slightly* different if/else switch than the EPERM) 3) The tracker, which seems related to BlueFS (?): https://tracker.ceph.com/issues/53906
Shooting in a dark a bit... but for the EPERM issue you mention, maybe try raising the number of file descriptors available to the process/container/system? Not sure why else would you get EPERM in this context.
On 2) I have definitely seen that happen before and it's always matched with an I/O error reported by the kernel in `dmesg` output. Sometimes the drive keeps going along fine after a restart of the OSD/reboot of the system, sometimes not.
Cheers, Tyler
On Mon, 4 Dec 2023 at 12:52, Zakhar Kirpichenko <zakhar@gmail.com>
wrote:
Hi,
Just to reiterate, I'm referring to an OSD crash loop because of the following error:
"2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation
not
permitted)". More relevant log entries: https://pastebin.com/gDat6rfk
The crash log suggested that there could be a hardware issue but there was none, I was able to access the block device for testing purposes without any issues, and the problem went away after I rebooted the host, this OSD is currently operating without any issues under load.
Any ideas?
/Z
On Sun, 3 Dec 2023 at 16:09, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks! The bug I referenced is the reason for the 1st OSD crash, but not for the subsequent crashes. The reason for those is described where you <snip />. I'm asking for help with that one.
/Z
On Sun, 3 Dec 2023 at 15:31, Kai Stian Olstad <ceph+list@olstad.com> wrote:
On Sun, Dec 03, 2023 at 06:53:08AM +0200, Zakhar Kirpichenko wrote:
One of our 16.2.14 cluster OSDs crashed again because of the dreaded https://tracker.ceph.com/issues/53906 bug.
<snip />
It would be good to understand what has triggered this condition and how it can be resolved without rebooting the whole host. I would very much appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of https://tracker.ceph.com/issues/53907
In there you have the fix and a workaround until next minor is released.
-- Kai Stian Olstad
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Kai Stian Olstad
-
Tyler Stachecki
-
Zakhar Kirpichenko