reboot breaks OSDs converted from ceph-disk to ceph-volume simple
Dear all, ceph version: mimic 13.2.10 I'm facing a serious bug with devices converted from "ceph-disk" to "ceph-volume simple". I "converted" all ceph-disk devices using "ceph-volume simple scan ..." And everything worked fine at the beginning. Today I needed to reboot an OSD host and since then most ceph-disk OSDs are screwed up. Apparently, "ceph-volume simple scan ..." creates symlinks to the block partition /dev/sd?2 using the "/dev/sd?2" name for the link target. These names are not stable and are expected to change after every reboot. Now I have a bunch of OSDs with new /dev/sd?2" names that won't boot any more, because this link points to the wrong block partition. Doing another "ceph-volume simple scan ..." doesn't help, it just "rediscovers" the wrong location. Here is what a broken OSD looks like (fresh "ceph-volume simple scan --stdout ..." output): { "active": "ok", "block": { "path": "/dev/sda2", "uuid": "b5ac1462-510a-4483-8f42-604e6adc5c9d" }, "block_uuid": "1d9d89a2-18c7-4610-9dcd-167d44ce1879", "bluefs": 1, "ceph_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "cluster_name": "ceph", "data": { "path": "/dev/sdb1", "uuid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb" }, "fsid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb", "keyring": "AQAZJ6ddedALDxAAJI7NLJ2CRFoQWK5STRpHuw==", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "none": "", "ready": "ready", "require_osd_release": "", "type": "bluestore", "whoami": 241 } OSD 241's data partition looks like this (after mount /dev/sdb1 /var/lib/ceph/osd/ceph-241): [root@ceph-adm:ceph-18 ceph-241]# ls -l /var/lib/ceph/osd/ceph-241 total 56 -rw-r--r--. 1 root root 411 Oct 16 2019 activate.monmap -rw-r--r--. 1 ceph ceph 3 Oct 16 2019 active lrwxrwxrwx. 1 root root 9 Mar 2 14:19 block -> /dev/sda2 -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 block_uuid -rw-r--r--. 1 ceph disk 2 Oct 16 2019 bluefs -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 ceph_fsid -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 fsid -rw-------. 1 ceph ceph 58 Oct 16 2019 keyring -rw-r--r--. 1 ceph disk 8 Oct 16 2019 kv_backend -rw-r--r--. 1 ceph ceph 21 Oct 16 2019 magic -rw-r--r--. 1 ceph disk 4 Oct 16 2019 mkfs_done -rw-r--r--. 1 ceph ceph 0 Nov 23 14:58 none -rw-r--r--. 1 ceph disk 6 Oct 16 2019 ready -rw-r--r--. 1 ceph disk 2 Jan 31 2020 require_osd_release -rw-r--r--. 1 ceph ceph 10 Oct 16 2019 type -rw-r--r--. 1 ceph ceph 4 Oct 16 2019 whoami The symlink "block -> /dev/sda2" goes to the wrong disk. How can I fix that in a stable way? Also, why are not stable "/dev/disk/by-uuid/..." link targets created instead? Can I change that myself? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
For comparison, the output of device discovery from ceph-disk and ceph-volume. ceph-disk does it correctly, ceph-volume is screwed up: [root@ceph-adm:ceph-18 ceph-241]# ceph-disk list /dev/sdb /usr/lib/python2.7/site-packages/ceph_disk/main.py:5689: UserWarning: ******************************************************************************* This tool is now deprecated in favor of ceph-volume. It is recommended to use ceph-volume for OSD deployments. For details see: http://docs.ceph.com/docs/master/ceph-volume/#migrating ******************************************************************************* warnings.warn(DEPRECATION_WARNING) /dev/sdb : /dev/sdb1 ceph data, active, cluster ceph, osd.241, block /dev/sdb2 /dev/sdb2 ceph block, for /dev/sdb1 /usr/lib/python2.7/site-packages/ceph_disk/main.py:5721: UserWarning: ******************************************************************************* This tool is now deprecated in favor of ceph-volume. It is recommended to use ceph-volume for OSD deployments. For details see: http://docs.ceph.com/docs/master/ceph-volume/#migrating ******************************************************************************* warnings.warn(DEPRECATION_WARNING) [root@ceph-adm:ceph-18 ceph-241]# ceph-volume simple scan --stdout /dev/sdb1 Running command: /usr/sbin/cryptsetup status /dev/sdb1 { "active": "ok", "block": { "path": "/dev/sda2", "uuid": "b5ac1462-510a-4483-8f42-604e6adc5c9d" }, "block_uuid": "1d9d89a2-18c7-4610-9dcd-167d44ce1879", "bluefs": 1, "ceph_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "cluster_name": "ceph", "data": { "path": "/dev/sdb1", "uuid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb" }, "fsid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb", "keyring": "AQAZJ6ddedALDxAAJI7NLJ2CRFoQWK5STRpHuw==", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "none": "", "ready": "ready", "require_osd_release": "", "type": "bluestore", "whoami": 241 } ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 02 March 2021 14:35:25 To: ceph-users@ceph.io Subject: [ceph-users] reboot breaks OSDs converted from ceph-disk to ceph-volume simple Dear all, ceph version: mimic 13.2.10 I'm facing a serious bug with devices converted from "ceph-disk" to "ceph-volume simple". I "converted" all ceph-disk devices using "ceph-volume simple scan ..." And everything worked fine at the beginning. Today I needed to reboot an OSD host and since then most ceph-disk OSDs are screwed up. Apparently, "ceph-volume simple scan ..." creates symlinks to the block partition /dev/sd?2 using the "/dev/sd?2" name for the link target. These names are not stable and are expected to change after every reboot. Now I have a bunch of OSDs with new /dev/sd?2" names that won't boot any more, because this link points to the wrong block partition. Doing another "ceph-volume simple scan ..." doesn't help, it just "rediscovers" the wrong location. Here is what a broken OSD looks like (fresh "ceph-volume simple scan --stdout ..." output): { "active": "ok", "block": { "path": "/dev/sda2", "uuid": "b5ac1462-510a-4483-8f42-604e6adc5c9d" }, "block_uuid": "1d9d89a2-18c7-4610-9dcd-167d44ce1879", "bluefs": 1, "ceph_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "cluster_name": "ceph", "data": { "path": "/dev/sdb1", "uuid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb" }, "fsid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb", "keyring": "AQAZJ6ddedALDxAAJI7NLJ2CRFoQWK5STRpHuw==", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "none": "", "ready": "ready", "require_osd_release": "", "type": "bluestore", "whoami": 241 } OSD 241's data partition looks like this (after mount /dev/sdb1 /var/lib/ceph/osd/ceph-241): [root@ceph-adm:ceph-18 ceph-241]# ls -l /var/lib/ceph/osd/ceph-241 total 56 -rw-r--r--. 1 root root 411 Oct 16 2019 activate.monmap -rw-r--r--. 1 ceph ceph 3 Oct 16 2019 active lrwxrwxrwx. 1 root root 9 Mar 2 14:19 block -> /dev/sda2 -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 block_uuid -rw-r--r--. 1 ceph disk 2 Oct 16 2019 bluefs -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 ceph_fsid -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 fsid -rw-------. 1 ceph ceph 58 Oct 16 2019 keyring -rw-r--r--. 1 ceph disk 8 Oct 16 2019 kv_backend -rw-r--r--. 1 ceph ceph 21 Oct 16 2019 magic -rw-r--r--. 1 ceph disk 4 Oct 16 2019 mkfs_done -rw-r--r--. 1 ceph ceph 0 Nov 23 14:58 none -rw-r--r--. 1 ceph disk 6 Oct 16 2019 ready -rw-r--r--. 1 ceph disk 2 Jan 31 2020 require_osd_release -rw-r--r--. 1 ceph ceph 10 Oct 16 2019 type -rw-r--r--. 1 ceph ceph 4 Oct 16 2019 whoami The symlink "block -> /dev/sda2" goes to the wrong disk. How can I fix that in a stable way? Also, why are not stable "/dev/disk/by-uuid/..." link targets created instead? Can I change that myself? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (1)
-
Frank Schilder