Re: ceph-volume lvm migrate error
On 2023/08/02 13:29, Roland Giesler wrote:
On 2023/08/02 12:53, Igor Fedotov wrote:
Roland,
First of all there are no block.db/block.wal symlinks in OSD folder. Which means there are no standalone DB/WAL any more.
That is surprising. So ceph-volume is not able to extract the DB/WAL from an OSD to migrate it it seems?
I figured out the if one doesn't specify and separate LV for the DB/WAL, it is integrated into the data drive. However, one can create a new DB/WAL for and OSD as follows: # systemctl stop ceph-osd@14 # ceph-bluestore-tool bluefs-bdev-new-db --path /var/lib/ceph/osd/ceph-14 --dev-target /dev/NodeC-nvme1/NodeC-nvme-LV-RocksDB1 --bluestore-block-db-size 45G inferring bluefs devices from bluestore path DB device added /dev/dm-20 # systemctl start ceph-osd@14 And, viola!, it did it. # ls -la /var/lib/ceph/osd/ceph-14/block* lrwxrwxrwx 1 ceph ceph 50 Dec 25 2022 /var/lib/ceph/osd/ceph-14/block -> /dev/mapper/0GVWr9-dQ65-LHcx-y6fD-z7fI-10A9-gVWZkY lrwxrwxrwx 1 root root 10 Aug 2 21:17 /var/lib/ceph/osd/ceph-14/block.db -> /dev/dm-20 I'm just check it out now, so see if there are no errors and that it actually does what I think it does.
Ouch, I got exited too quickly! On 2023/08/02 21:27, Roland Giesler wrote:
# systemctl start ceph-osd@14
And, viola!, it did it.
# ls -la /var/lib/ceph/osd/ceph-14/block* lrwxrwxrwx 1 ceph ceph 50 Dec 25 2022 /var/lib/ceph/osd/ceph-14/block -> /dev/mapper/0GVWr9-dQ65-LHcx-y6fD-z7fI-10A9-gVWZkY lrwxrwxrwx 1 root root 10 Aug 2 21:17 /var/lib/ceph/osd/ceph-14/block.db -> /dev/dm-20
It crashed! # systemctl status ceph-osd@14 ● ceph-osd@14.service - Ceph object storage daemon osd.14 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled) Drop-In: /usr/lib/systemd/system/ceph-osd@.service.d └─ceph-after-pve-cluster.conf Active: failed (Result: exit-code) since Wed 2023-08-02 21:18:54 SAST; 10min ago Process: 520652 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 14 (code=exited, status=0/SUCCESS) Process: 520660 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 14 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 520660 (code=exited, status=1/FAILURE) CPU: 90ms Aug 02 21:18:54 FT1-NodeC systemd[1]: ceph-osd@14.service: Scheduled restart job, restart counter is at 3. Aug 02 21:18:54 FT1-NodeC systemd[1]: Stopped Ceph object storage daemon osd.14. Aug 02 21:18:54 FT1-NodeC systemd[1]: ceph-osd@14.service: Start request repeated too quickly. Aug 02 21:18:54 FT1-NodeC systemd[1]: ceph-osd@14.service: Failed with result 'exit-code'. Aug 02 21:18:54 FT1-NodeC systemd[1]: Failed to start Ceph object storage daemon osd.14. Aug 02 21:28:49 FT1-NodeC systemd[1]: ceph-osd@14.service: Start request repeated too quickly. Aug 02 21:28:49 FT1-NodeC systemd[1]: ceph-osd@14.service: Failed with result 'exit-code'. Aug 02 21:28:49 FT1-NodeC systemd[1]: Failed to start Ceph object storage daemon osd.14.
Check out the ownership of the newly created DB device, according to your output it belongs to the root user. In the osd.log you probably should see something related to "permission denied". If you change it to ceph:ceph the OSD might start properly. Zitat von Roland Giesler <roland@giesler.za.net>:
Ouch, I got exited too quickly!
On 2023/08/02 21:27, Roland Giesler wrote:
# systemctl start ceph-osd@14
And, viola!, it did it.
# ls -la /var/lib/ceph/osd/ceph-14/block* lrwxrwxrwx 1 ceph ceph 50 Dec 25 2022 /var/lib/ceph/osd/ceph-14/block -> /dev/mapper/0GVWr9-dQ65-LHcx-y6fD-z7fI-10A9-gVWZkY lrwxrwxrwx 1 root root 10 Aug 2 21:17 /var/lib/ceph/osd/ceph-14/block.db -> /dev/dm-20
It crashed!
# systemctl status ceph-osd@14 ● ceph-osd@14.service - Ceph object storage daemon osd.14 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled) Drop-In: /usr/lib/systemd/system/ceph-osd@.service.d └─ceph-after-pve-cluster.conf Active: failed (Result: exit-code) since Wed 2023-08-02 21:18:54 SAST; 10min ago Process: 520652 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 14 (code=exited, status=0/SUCCESS) Process: 520660 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 14 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 520660 (code=exited, status=1/FAILURE) CPU: 90ms
Aug 02 21:18:54 FT1-NodeC systemd[1]: ceph-osd@14.service: Scheduled restart job, restart counter is at 3. Aug 02 21:18:54 FT1-NodeC systemd[1]: Stopped Ceph object storage daemon osd.14. Aug 02 21:18:54 FT1-NodeC systemd[1]: ceph-osd@14.service: Start request repeated too quickly. Aug 02 21:18:54 FT1-NodeC systemd[1]: ceph-osd@14.service: Failed with result 'exit-code'. Aug 02 21:18:54 FT1-NodeC systemd[1]: Failed to start Ceph object storage daemon osd.14. Aug 02 21:28:49 FT1-NodeC systemd[1]: ceph-osd@14.service: Start request repeated too quickly. Aug 02 21:28:49 FT1-NodeC systemd[1]: ceph-osd@14.service: Failed with result 'exit-code'. Aug 02 21:28:49 FT1-NodeC systemd[1]: Failed to start Ceph object storage daemon osd.14. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Roland Giesler