31 Jul
2023
31 Jul
'23
4:38 a.m.
Hello,
I have two main questions here.
1. What can I do when `ceph-bluestore-tool` outputs a stack trace for
`fsck`?
2. How does one recover from lost PGs / data corruption in an RGW
Multi-site setup?
---
I have a Luminous 12.2.12 cluster built on
ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 for all daemons, no
ceph packages are installed on the systems. The OSD nodes have 128GB RAM, 6
SATA SSDs (Micron 5200, 2TB) and 1 NVMe SSD split into 4 OSDs.
osd_memory_target is set to 10GB and the OSD nodes have 128GB of RAM. That
should put me at 100/128GB used.
There are 3 PGs down, 3 of the OSDs that had those PGs won't stay online,
and they crash fairly quickly after starting. These are running on SATA
SSDs which are being replaced with NVMe SSDs. Crush re-weighting the SATA
drives down causes some SATA OSDs to crash and some NVMe drives have slow
or blocked ops (related to the down PGs).
I installed the ceph-osd package on one OSD host. When I ran
`ceph-bluestore-tool`, I got a bunch of tcmalloc and unexpected aio errors.
Exact output below. I also tried `ceph-objectstore-tool` but received
similar results. I cloned the other OSD that has the affected PGs to have a
copy I can work on, but I got the exact same results as before.
---
From what I can see, this is likely due to bad drives and automation trying
to restart down OSDs several times. With 3 down PGs, I am assuming my next
step would be to mark those PGs lost. From there, I am unsure what the
recovery procedure is to sync "clean" data from other zones into the
cluster that was impacted. Is RGW able to handle this? Do I need to use
`rclone`?
---
$ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-11 fsck
tcmalloc: large alloc 1283989504 bytes == 0x557fdbe46000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 2567970816 bytes == 0x5580286c8000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 5135933440 bytes == 0x5580c17ca000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 3025510400 bytes == 0x557f8f6e6000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc87335582b 0x557f89d75d19 0x557f89d2edda 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 2269913088 bytes == 0x55832469e000 @ 0x7fc87e3f2e50
0x7fc87e4121b9 0x7fc8756ca4f7 0x7fc8756cd304 0x557f89cc4661 0x557f89ad0858
0x557f89ad2224 0x557f89cb7b1d 0x557f89de584c 0x557f89de6a7e 0x557f89e05e7b
0x557f89d2cf48 0x557f89d2efd2 0x557f89d30524 0x557f89d318ef 0x557f89d33147
0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1
0x557f89ab03fa (nil)
2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240
/var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but
returned: 2147479552/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc:
In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time
2023-07-30 08:27:27.532004
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7fc8757242c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
4: (()+0x74a4) [0x7fc8740104a4]
5: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
2023-07-30 08:27:27.544215 7fc86f689700 -1
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void
KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30
08:27:27.532004
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7fc8757242c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
4: (()+0x74a4) [0x7fc8740104a4]
5: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
-1> 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240
/var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but
returned: 2147479552
0> 2023-07-30 08:27:27.544215 7fc86f689700 -1
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void
KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30
08:27:27.532004
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7fc8757242c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
4: (()+0x74a4) [0x7fc8740104a4]
5: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
*** Caught signal (Aborted) **
in thread 7fc86f689700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x424fc4) [0x557f89d25fc4]
2: (()+0x110e0) [0x7fc87401a0e0]
3: (gsignal()+0xcf) [0x7fc8729affff]
4: (abort()+0x16a) [0x7fc8729b142a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7fc87572444e]
6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
8: (()+0x74a4) [0x7fc8740104a4]
9: (clone()+0x3f) [0x7fc872a65d0f]
2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) **
in thread 7fc86f689700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x424fc4) [0x557f89d25fc4]
2: (()+0x110e0) [0x7fc87401a0e0]
3: (gsignal()+0xcf) [0x7fc8729affff]
4: (abort()+0x16a) [0x7fc8729b142a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7fc87572444e]
6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
8: (()+0x74a4) [0x7fc8740104a4]
9: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
0> 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal
(Aborted) **
in thread 7fc86f689700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x424fc4) [0x557f89d25fc4]
2: (()+0x110e0) [0x7fc87401a0e0]
3: (gsignal()+0xcf) [0x7fc8729affff]
4: (abort()+0x16a) [0x7fc8729b142a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7fc87572444e]
6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
8: (()+0x74a4) [0x7fc8740104a4]
9: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
Aborted
$ ceph-objectstore-tool --data-path=/var/lib/ceph/osd/ceph-11 --op list-pgs
tcmalloc: large alloc 1283989504 bytes == 0x5649b1bdc000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 2567970816 bytes == 0x5649fe45e000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 5135933440 bytes == 0x564a97560000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 3025510400 bytes == 0x56496547c000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbc82b 0x56495df34079 0x56495deed22a 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 2269913088 bytes == 0x564cfa402000 @ 0x7f3af5e55e50
0x7f3af5e751b9 0x7f3aed12d4f7 0x7f3aed130304 0x56495de9fbc1 0x56495de7a5f8
0x56495de7bfc4 0x56495de9307d 0x56495dfa32dc 0x56495dfa450e 0x56495dfc34db
0x56495deeb398 0x56495deed422 0x56495deee974 0x56495deefd3f 0x56495def1597
0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1
0x56495d86853a (nil)
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void
KernelDevice::_aio_thread()' thread 7f3ae72f0700 time 2023-07-30
08:37:16.531432
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7f3aed1872c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd]
4: (()+0x74a4) [0x7f3aeba734a4]
5: (clone()+0x3f) [0x7f3aea6ccd0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
*** Caught signal (Aborted) **
in thread 7f3ae72f0700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x94a0f4) [0x56495debe0f4]
2: (()+0x110e0) [0x7f3aeba7d0e0]
3: (gsignal()+0xcf) [0x7f3aea616fff]
4: (abort()+0x16a) [0x7f3aea61842a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7f3aed18744e]
6: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd]
8: (()+0x74a4) [0x7f3aeba734a4]
9: (clone()+0x3f) [0x7f3aea6ccd0f]
Aborted
--
Gregory O’Neill
31 Jul
31 Jul
11:13 a.m.
Hi
On 31 Jul 2023, at 03:12, Gregory O'Neill <oneill.gs(a)gmail.com> wrote:
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
The Kernel was returned error actually. Try look to dmesg, may be your SATA SSD in read
only mode
k
Sent from my iPhone
8:41 p.m.
Hi Konstantin,
I dropped the SATA SSDs from /sys/block and rescanned it. Re-running `ceph-bluestore-tool
fsck` resulted in the same output (unexpected aio error). Syslog says the drive is not in
write-protect mode, however smart says life remaining is at 1%. Drives have approximately
42k power-on hours.
$ lsscsi -g
[0:0:0:0] disk ATA ST1000NX0423 TN04 /dev/sda /dev/sg0
[1:0:0:0] disk ATA ST1000NX0423 TN04 /dev/sdb /dev/sg1
[2:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdc /dev/sg2
[3:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdd /dev/sg3
[4:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sde /dev/sg4
[5:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdf /dev/sg5
[6:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdg /dev/sg6
[7:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdh /dev/sg7
$ echo 1 > /sys/class/scsi_generic/sg3/device/delete
$ tail -n 10 /var/log/messages
Jul 31 07:08:03 ceph-osd4 rsyslogd: -- MARK --
Jul 31 07:28:03 ceph-osd4 rsyslogd: -- MARK --
Jul 31 07:48:03 ceph-osd4 rsyslogd: -- MARK --
Jul 31 08:08:03 ceph-osd4 rsyslogd: -- MARK --
Jul 31 08:28:03 ceph-osd4 rsyslogd: -- MARK --
Jul 31 08:48:03 ceph-osd4 rsyslogd: -- MARK --
Jul 31 08:51:41 ceph-osd4 kernel: [144240.695000] sd 3:0:0:0: [sdd] Synchronizing SCSI
cache
Jul 31 08:51:41 ceph-osd4 kernel: [144240.695063] sd 3:0:0:0: [sdd] Stopping disk
Jul 31 08:51:41 ceph-osd4 kernel: [144241.011641] ata4.00: disabled
$ echo "- - -" > /sys/class/scsi_host/host3/scan
$ tail -n 10 /var/log/messages
Jul 31 08:52:48 ceph-osd4 kernel: [144308.293457] ata4.00: Enabling discard_zeroes_data
Jul 31 08:52:48 ceph-osd4 kernel: [144308.293480] sd 3:0:0:0: [sdi] 3750748848 512-byte
logical blocks: (1.92 TB/1.75 TiB)
Jul 31 08:52:48 ceph-osd4 kernel: [144308.293481] sd 3:0:0:0: [sdi] 4096-byte physical
blocks
Jul 31 08:52:48 ceph-osd4 kernel: [144308.293486] sd 3:0:0:0: [sdi] Write Protect is off
Jul 31 08:52:48 ceph-osd4 kernel: [144308.293493] sd 3:0:0:0: [sdi] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Jul 31 08:52:48 ceph-osd4 kernel: [144308.293517] sd 3:0:0:0: Attached scsi generic sg3
type 0
Jul 31 08:52:48 ceph-osd4 kernel: [144308.293670] ata4.00: Enabling discard_zeroes_data
Jul 31 08:52:48 ceph-osd4 kernel: [144308.297332] sdi: sdi1
Jul 31 08:52:48 ceph-osd4 kernel: [144308.297898] ata4.00: Enabling discard_zeroes_data
Jul 31 08:52:48 ceph-osd4 kernel: [144308.297928] sd 3:0:0:0: [sdi] Attached SCSI
removable disk
$ lsscsi -g
[0:0:0:0] disk ATA ST1000NX0423 TN04 /dev/sda /dev/sg0
[1:0:0:0] disk ATA ST1000NX0423 TN04 /dev/sdb /dev/sg1
[2:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdc /dev/sg2
[3:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdi /dev/sg3
[4:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sde /dev/sg4
[5:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdf /dev/sg5
[6:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdg /dev/sg6
[7:0:0:0] disk ATA Micron_5200_MTFD U004 /dev/sdh /dev/sg7
3 Aug
3 Aug
7:20 p.m.
Hi,
Can you show `smartctl -a` for this device?
This drives show input/output errors in dmesg when you try to run ceph-osd?
k
Sent from my iPhone
On 2 Aug 2023, at 21:44, Greg O'Neill
<oneill.gs(a)gmail.com> wrote:
Syslog says the drive is not in write-protect mode, however smart says life remaining is
at 1%.
291
days inactive
294
days old
3 comments
3 participants
participants (3)
-
Greg O'Neill -
Gregory O'Neill
-
Konstantin Shalygin