Hello, I have two main questions here. 1. What can I do when `ceph-bluestore-tool` outputs a stack trace for `fsck`? 2. How does one recover from lost PGs / data corruption in an RGW Multi-site setup? --- I have a Luminous 12.2.12 cluster built on ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 for all daemons, no ceph packages are installed on the systems. The OSD nodes have 128GB RAM, 6 SATA SSDs (Micron 5200, 2TB) and 1 NVMe SSD split into 4 OSDs. osd_memory_target is set to 10GB and the OSD nodes have 128GB of RAM. That should put me at 100/128GB used. There are 3 PGs down, 3 of the OSDs that had those PGs won't stay online, and they crash fairly quickly after starting. These are running on SATA SSDs which are being replaced with NVMe SSDs. Crush re-weighting the SATA drives down causes some SATA OSDs to crash and some NVMe drives have slow or blocked ops (related to the down PGs). I installed the ceph-osd package on one OSD host. When I ran `ceph-bluestore-tool`, I got a bunch of tcmalloc and unexpected aio errors. Exact output below. I also tried `ceph-objectstore-tool` but received similar results. I cloned the other OSD that has the affected PGs to have a copy I can work on, but I got the exact same results as before. --- From what I can see, this is likely due to bad drives and automation trying to restart down OSDs several times. With 3 down PGs, I am assuming my next step would be to mark those PGs lost. From there, I am unsure what the recovery procedure is to sync "clean" data from other zones into the cluster that was impacted. Is RGW able to handle this? Do I need to use `rclone`? --- $ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-11 fsck tcmalloc: large alloc 1283989504 bytes == 0x557fdbe46000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 2567970816 bytes == 0x5580286c8000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 5135933440 bytes == 0x5580c17ca000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 3025510400 bytes == 0x557f8f6e6000 @ 0x7fc87e4126d0 0x7fc873354ae9 0x7fc87335582b 0x557f89d75d19 0x557f89d2edda 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) tcmalloc: large alloc 2269913088 bytes == 0x55832469e000 @ 0x7fc87e3f2e50 0x7fc87e4121b9 0x7fc8756ca4f7 0x7fc8756cd304 0x557f89cc4661 0x557f89ad0858 0x557f89ad2224 0x557f89cb7b1d 0x557f89de584c 0x557f89de6a7e 0x557f89e05e7b 0x557f89d2cf48 0x557f89d2efd2 0x557f89d30524 0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil) 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240 /var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but returned: 2147479552/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2023-07-30 08:27:27.544215 7fc86f689700 -1 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1> 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240 /var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but returned: 2147479552 0> 2023-07-30 08:27:27.544215 7fc86f689700 -1 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30 08:27:27.532004 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc8757242c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 4: (()+0x74a4) [0x7fc8740104a4] 5: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 0> 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) ** in thread 7fc86f689700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x424fc4) [0x557f89d25fc4] 2: (()+0x110e0) [0x7fc87401a0e0] 3: (gsignal()+0xcf) [0x7fc8729affff] 4: (abort()+0x16a) [0x7fc8729b142a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc87572444e] 6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d] 8: (()+0x74a4) [0x7fc8740104a4] 9: (clone()+0x3f) [0x7fc872a65d0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted $ ceph-objectstore-tool --data-path=/var/lib/ceph/osd/ceph-11 --op list-pgs tcmalloc: large alloc 1283989504 bytes == 0x5649b1bdc000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 2567970816 bytes == 0x5649fe45e000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 5135933440 bytes == 0x564a97560000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 3025510400 bytes == 0x56496547c000 @ 0x7f3af5e756d0 0x7f3aeafbbae9 0x7f3aeafbc82b 0x56495df34079 0x56495deed22a 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) tcmalloc: large alloc 2269913088 bytes == 0x564cfa402000 @ 0x7f3af5e55e50 0x7f3af5e751b9 0x7f3aed12d4f7 0x7f3aed130304 0x56495de9fbc1 0x56495de7a5f8 0x56495de7bfc4 0x56495de9307d 0x56495dfa32dc 0x56495dfa450e 0x56495dfc34db 0x56495deeb398 0x56495deed422 0x56495deee974 0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil) /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f3ae72f0700 time 2023-07-30 08:37:16.531432 /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 == "unexpected aio error") ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3aed1872c2] 2: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27] 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd] 4: (()+0x74a4) [0x7f3aeba734a4] 5: (clone()+0x3f) [0x7f3aea6ccd0f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** Caught signal (Aborted) ** in thread 7f3ae72f0700 thread_name:bstore_aio ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x94a0f4) [0x56495debe0f4] 2: (()+0x110e0) [0x7f3aeba7d0e0] 3: (gsignal()+0xcf) [0x7f3aea616fff] 4: (abort()+0x16a) [0x7f3aea61842a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7f3aed18744e] 6: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27] 7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd] 8: (()+0x74a4) [0x7f3aeba734a4] 9: (clone()+0x3f) [0x7f3aea6ccd0f] Aborted -- Gregory O’Neill