Hello,
I have two main questions here.
1. What can I do when `ceph-bluestore-tool` outputs a stack trace for
`fsck`?
2. How does one recover from lost PGs / data corruption in an RGW
Multi-site setup?
---
I have a Luminous 12.2.12 cluster built on
ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64 for all daemons, no
ceph packages are installed on the systems. The OSD nodes have 128GB RAM, 6
SATA SSDs (Micron 5200, 2TB) and 1 NVMe SSD split into 4 OSDs.
osd_memory_target is set to 10GB and the OSD nodes have 128GB of RAM. That
should put me at 100/128GB used.
There are 3 PGs down, 3 of the OSDs that had those PGs won't stay online,
and they crash fairly quickly after starting. These are running on SATA
SSDs which are being replaced with NVMe SSDs. Crush re-weighting the SATA
drives down causes some SATA OSDs to crash and some NVMe drives have slow
or blocked ops (related to the down PGs).
I installed the ceph-osd package on one OSD host. When I ran
`ceph-bluestore-tool`, I got a bunch of tcmalloc and unexpected aio errors.
Exact output below. I also tried `ceph-objectstore-tool` but received
similar results. I cloned the other OSD that has the affected PGs to have a
copy I can work on, but I got the exact same results as before.
---
From what I can see, this is likely due to bad drives and automation trying
to restart down OSDs several times. With 3 down PGs, I am assuming my next
step would be to mark those PGs lost. From there, I am unsure what the
recovery procedure is to sync "clean" data from other zones into the
cluster that was impacted. Is RGW able to handle this? Do I need to use
`rclone`?
---
$ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-11 fsck
tcmalloc: large alloc 1283989504 bytes == 0x557fdbe46000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 2567970816 bytes == 0x5580286c8000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 5135933440 bytes == 0x5580c17ca000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc873356073 0x557f89d3d680 0x557f89d2ebcd 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 3025510400 bytes == 0x557f8f6e6000 @ 0x7fc87e4126d0
0x7fc873354ae9 0x7fc87335582b 0x557f89d75d19 0x557f89d2edda 0x557f89d30524
0x557f89d318ef 0x557f89d33147 0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a
0x557f89a2c5e1 0x7fc87299d2e1 0x557f89ab03fa (nil)
tcmalloc: large alloc 2269913088 bytes == 0x55832469e000 @ 0x7fc87e3f2e50
0x7fc87e4121b9 0x7fc8756ca4f7 0x7fc8756cd304 0x557f89cc4661 0x557f89ad0858
0x557f89ad2224 0x557f89cb7b1d 0x557f89de584c 0x557f89de6a7e 0x557f89e05e7b
0x557f89d2cf48 0x557f89d2efd2 0x557f89d30524 0x557f89d318ef 0x557f89d33147
0x557f89bb0d6f 0x557f89b3c91b 0x557f89b6df8a 0x557f89a2c5e1 0x7fc87299d2e1
0x557f89ab03fa (nil)
2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240
/var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but
returned: 2147479552/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc:
In function 'void KernelDevice::_aio_thread()' thread 7fc86f689700 time
2023-07-30 08:27:27.532004
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7fc8757242c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
4: (()+0x74a4) [0x7fc8740104a4]
5: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
2023-07-30 08:27:27.544215 7fc86f689700 -1
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void
KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30
08:27:27.532004
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7fc8757242c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
4: (()+0x74a4) [0x7fc8740104a4]
5: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
-1> 2023-07-30 08:27:27.531919 7fc86f689700 -1 bdev(0x557f8add4240
/var/lib/ceph/osd/ceph-11/block) aio to 929504952320~2269908992 but
returned: 2147479552
0> 2023-07-30 08:27:27.544215 7fc86f689700 -1
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void
KernelDevice::_aio_thread()' thread 7fc86f689700 time 2023-07-30
08:27:27.532004
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7fc8757242c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
4: (()+0x74a4) [0x7fc8740104a4]
5: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
*** Caught signal (Aborted) **
in thread 7fc86f689700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x424fc4) [0x557f89d25fc4]
2: (()+0x110e0) [0x7fc87401a0e0]
3: (gsignal()+0xcf) [0x7fc8729affff]
4: (abort()+0x16a) [0x7fc8729b142a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7fc87572444e]
6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
8: (()+0x74a4) [0x7fc8740104a4]
9: (clone()+0x3f) [0x7fc872a65d0f]
2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal (Aborted) **
in thread 7fc86f689700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x424fc4) [0x557f89d25fc4]
2: (()+0x110e0) [0x7fc87401a0e0]
3: (gsignal()+0xcf) [0x7fc8729affff]
4: (abort()+0x16a) [0x7fc8729b142a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7fc87572444e]
6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
8: (()+0x74a4) [0x7fc8740104a4]
9: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
0> 2023-07-30 08:27:27.549175 7fc86f689700 -1 *** Caught signal
(Aborted) **
in thread 7fc86f689700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x424fc4) [0x557f89d25fc4]
2: (()+0x110e0) [0x7fc87401a0e0]
3: (gsignal()+0xcf) [0x7fc8729affff]
4: (abort()+0x16a) [0x7fc8729b142a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7fc87572444e]
6: (KernelDevice::_aio_thread()+0x1377) [0x557f89cc14c7]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x557f89cc725d]
8: (()+0x74a4) [0x7fc8740104a4]
9: (clone()+0x3f) [0x7fc872a65d0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
Aborted
$ ceph-objectstore-tool --data-path=/var/lib/ceph/osd/ceph-11 --op list-pgs
tcmalloc: large alloc 1283989504 bytes == 0x5649b1bdc000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 2567970816 bytes == 0x5649fe45e000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 5135933440 bytes == 0x564a97560000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbd073 0x56495defb9e0 0x56495deed01d 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 3025510400 bytes == 0x56496547c000 @ 0x7f3af5e756d0
0x7f3aeafbbae9 0x7f3aeafbc82b 0x56495df34079 0x56495deed22a 0x56495deee974
0x56495deefd3f 0x56495def1597 0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4
0x56495d7de4db 0x7f3aea6042e1 0x56495d86853a (nil)
tcmalloc: large alloc 2269913088 bytes == 0x564cfa402000 @ 0x7f3af5e55e50
0x7f3af5e751b9 0x7f3aed12d4f7 0x7f3aed130304 0x56495de9fbc1 0x56495de7a5f8
0x56495de7bfc4 0x56495de9307d 0x56495dfa32dc 0x56495dfa450e 0x56495dfc34db
0x56495deeb398 0x56495deed422 0x56495deee974 0x56495deefd3f 0x56495def1597
0x56495de0e47f 0x56495dd95dab 0x56495ddcf9e4 0x56495d7de4db 0x7f3aea6042e1
0x56495d86853a (nil)
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: In function 'void
KernelDevice::_aio_thread()' thread 7f3ae72f0700 time 2023-07-30
08:37:16.531432
/build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0
== "unexpected aio error")
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7f3aed1872c2]
2: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27]
3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd]
4: (()+0x74a4) [0x7f3aeba734a4]
5: (clone()+0x3f) [0x7f3aea6ccd0f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
*** Caught signal (Aborted) **
in thread 7f3ae72f0700 thread_name:bstore_aio
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
(stable)
1: (()+0x94a0f4) [0x56495debe0f4]
2: (()+0x110e0) [0x7f3aeba7d0e0]
3: (gsignal()+0xcf) [0x7f3aea616fff]
4: (abort()+0x16a) [0x7f3aea61842a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x7f3aed18744e]
6: (KernelDevice::_aio_thread()+0x1377) [0x56495de9ca27]
7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56495dea27bd]
8: (()+0x74a4) [0x7f3aeba734a4]
9: (clone()+0x3f) [0x7f3aea6ccd0f]
Aborted
--
Gregory O’Neill
On 2023/08/02 13:29, Roland Giesler wrote:
>
> On 2023/08/02 12:53, Igor Fedotov wrote:
>> Roland,
>>
>> First of all there are no block.db/block.wal symlinks in OSD folder.
>> Which means there are no standalone DB/WAL any more.
>
> That is surprising. So ceph-volume is not able to extract the DB/WAL
> from an OSD to migrate it it seems?
I figured out the if one doesn't specify and separate LV for the DB/WAL,
it is integrated into the data drive.
However, one can create a new DB/WAL for and OSD as follows:
# systemctl stop ceph-osd@14
# ceph-bluestore-tool bluefs-bdev-new-db --path
/var/lib/ceph/osd/ceph-14 --dev-target
/dev/NodeC-nvme1/NodeC-nvme-LV-RocksDB1 --bluestore-block-db-size 45G
inferring bluefs devices from bluestore path
DB device added /dev/dm-20
# systemctl start ceph-osd@14
And, viola!, it did it.
# ls -la /var/lib/ceph/osd/ceph-14/block*
lrwxrwxrwx 1 ceph ceph 50 Dec 25 2022 /var/lib/ceph/osd/ceph-14/block
-> /dev/mapper/0GVWr9-dQ65-LHcx-y6fD-z7fI-10A9-gVWZkY
lrwxrwxrwx 1 root root 10 Aug 2 21:17
/var/lib/ceph/osd/ceph-14/block.db -> /dev/dm-20
I'm just check it out now, so see if there are no errors and that it
actually does what I think it does.
Details of this release are summarized here:
https://tracker.ceph.com/issues/62231#note-1
Seeking approvals/reviews for:
smoke - Laura, Radek
rados - Neha, Radek, Travis, Ernesto, Adam King
rgw - Casey
fs - Venky
orch - Adam King
rbd - Ilya
krbd - Ilya
upgrade-clients:client-upgrade* - in progress
powercycle - Brad
Please reply to this email with approval and/or trackers of known
issues/PRs to address them.
bookworm distro support is an outstanding issue.
TIA
YuriW
Hi,
it seems that after reboot / OS update my disk labels / device paths may have changed. Since then i get an error like this:
CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.osd-12-22_hdd-2
###
RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9025b6db2e8be9173186e068a79a0da5a54ada -e NODE_NAME=ceph-osd07.intern -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd-12-22_hdd-2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/run/ceph:z -v /var/log/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/log/ceph:z -v /var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2cvmr5lf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpb38cuw7q:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9025b6db2e8be9173186e068a79a0da5a54ada lvm batch --no-auto /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq --db-devices /dev/sdg --yes --no-systemd
/bin/docker: stderr Traceback (most recent call last):
/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
/bin/docker: stderr self.main(self.argv)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/bin/docker: stderr return f(*a, **kw)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
/bin/docker: stderr instance.main()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 192, in dispatch
/bin/docker: stderr instance = mapper.get(arg)(argv[count:])
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 348, in __init__
/bin/docker: stderr self.args = parser.parse_args(argv)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1734, in parse_args
/bin/docker: stderr args, argv = self.parse_known_args(args, namespace)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1766, in parse_known_args
/bin/docker: stderr namespace, args = self._parse_known_args(args, namespace)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1954, in _parse_known_args
/bin/docker: stderr positionals_end_index = consume_positionals(start_index)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1931, in consume_positionals
/bin/docker: stderr take_action(action, args)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 1824, in take_action
/bin/docker: stderr argument_values = self._get_values(action, argument_strings)
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2279, in _get_values
/bin/docker: stderr value = [self._get_value(action, v) for v in arg_strings]
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2279, in <listcomp>
/bin/docker: stderr value = [self._get_value(action, v) for v in arg_strings]
/bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line 2294, in _get_value
/bin/docker: stderr result = type_func(arg_string)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 116, in __call__
/bin/docker: stderr return self._format_device(self._is_valid_device())
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 127, in _is_valid_device
/bin/docker: stderr super()._is_valid_device(raise_sys_exit=False)
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 104, in _is_valid_device
/bin/docker: stderr super()._is_valid_device()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 69, in _is_valid_device
/bin/docker: stderr super()._is_valid_device()
/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line 47, in _is_valid_device
/bin/docker: stderr raise RuntimeError("Device {} has partitions.".format(self.dev_path))
/bin/docker: stderr RuntimeError: Device /dev/sdq has partitions.
Traceback (most recent call last):
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9309, in <module>
main()
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9297, in main
r = ctx.func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1941, in _infer_config
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1872, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1969, in _infer_image
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1859, in _validate_fsid
return func(ctx)
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 5366, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd())
File "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1661, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
###
/dev/sdg is my boot device at the moment wich was formerly /dev/sda.
Is it safe to edit the ceph orch yaml file and change the device pathes to the new format? Like this:
ceph orch ls --service_name=<service-name> --export > myservice.yaml
vi (change device pathes in spec -> data_devices -> path | db_devices -> path)
ceph orch apply -i myservice.yaml [--dry-run]
Is that ok / expected behaviour? Or is there a better way?
However can see that mgr detects new devices in ceph orch log:
mgr.ceph-mon03.lrfomu [INF] Detected new or changed devices on ceph-osd07
Regards,
Kilian
Hi! No matter what I try, using the latest cephfs on an all
ceph-pacific setup, I've not been able to avoid this error message,
always similar to this on RHEL family clients:
SELinux: inode=1099954719159 on dev=ceph was found to have an invalid
context=system_u:object_r:unlabeled_t:s0. This indicates you may need
to relabel the inode or the filesystem in question.
What's the answer?
Thanks
Harry Coin
I need some help with this please. The command below gives and error
which is not helpful to me.
ceph-volume lvm migrate --osd-id 14 --osd-fsid
4de2a617-4452-420d-a99b-9e0cd6b2a99b --from db wal --target
NodeC-nvme1/NodeC-nvme-LV-RocksDB1
--> Source device list is empty
Unable to migrate to : NodeC-nvme1/NodeC-nvme-LV-RocksDB1
Alternatively I have tried to only specify --from db instead of
including wal, but it makes no difference.
Here is the OSD in question.
# ls -la /dev/ceph-025b887e-4f06-468f-845c-0ddf9ad04990/
lrwxrwxrwx 1 root root 7 Dec 25 2022
osd-block-4de2a617-4452-420d-a99b-9e0cd6b2a99b -> ../dm-4
What is happening here? I want to move the DB/WAL to NVMe storage
without trashing the data OSD and having to go through rebalancing for
each drive I do this for.
thanks
Roland
Hi,
I have trouble with large OMAP files in a cluster in the RGW index pool. Some
background information about the cluster: There is CephFS and RBD usage on the
main cluster but for this issue I think only S3 is interesting.
There is one realm, one zonegroup with two zones which have a bidirectional sync
set up. Since this does not allow for autoresharding we have to do it by hand in
this cluster – looking forward to Reef!
From the logs:
cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 :
cluster [WRN] Large omap object found. Object:
34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head
PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182
The offending bucket looks like this:
# radosgw-admin bucket stats \
| jq '.[] | select(.marker
=="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9")
|"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r
131 9463833
Last week the number of objects was about 12 million. Which is why I reshareded
the offending bucket twice, I think. Once to 129 and the second time to 131
because I wanted some leeway (or lieway? scnr, Sage).
Unfortunately, even after a week the objects were still to big (the log line
above is quite recent), so I looked into it again.
# rados -p raum.rgw.buckets.index ls \
|grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \
|sort -V
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9
.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10
# rados -p raum.rgw.buckets.index ls \
|grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \
|sort -V \
|xargs -IOMAP sh -c \
'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l'
1013854
1011007
1012287
1011232
1013565
998262
1012777
1012713
1012230
1010690
997111
Apparently, only 11 shards are in use. This would explain why the "Key usage"
(from the log line) is about ten times higher than I would expect.
How can I deal with this issue?
One thing I could try to fix this would be to reshard to a lower number, but I
am not sure if there are any risks associated with "downsharding". After that I
could reshard to something like 97. Or I could directly "downshard" to 97.
Also, the second zone has a similar problem, but as the error messsage lets me
know, this would be a bad idea. Will it just take more time until the sharding
is transferred to the seconds zone?
Best,
Christian Kugler
