2 Mar
2021
2 Mar
'21
1:35 p.m.
Dear all,
ceph version: mimic 13.2.10
I'm facing a serious bug with devices converted from "ceph-disk" to
"ceph-volume simple". I "converted" all ceph-disk devices using
"ceph-volume simple scan ..." And everything worked fine at the beginning. Today
I needed to reboot an OSD host and since then most ceph-disk OSDs are screwed up.
Apparently, "ceph-volume simple scan ..." creates symlinks to the block
partition /dev/sd?2 using the "/dev/sd?2" name for the link target. These names
are not stable and are expected to change after every reboot. Now I have a bunch of OSDs
with new /dev/sd?2" names that won't boot any more, because this link points to
the wrong block partition. Doing another "ceph-volume simple scan ..."
doesn't help, it just "rediscovers" the wrong location. Here is what a
broken OSD looks like (fresh "ceph-volume simple scan --stdout ..." output):
{
"active": "ok",
"block": {
"path": "/dev/sda2",
"uuid": "b5ac1462-510a-4483-8f42-604e6adc5c9d"
},
"block_uuid": "1d9d89a2-18c7-4610-9dcd-167d44ce1879",
"bluefs": 1,
"ceph_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
"cluster_name": "ceph",
"data": {
"path": "/dev/sdb1",
"uuid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb"
},
"fsid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb",
"keyring": "AQAZJ6ddedALDxAAJI7NLJ2CRFoQWK5STRpHuw==",
"kv_backend": "rocksdb",
"magic": "ceph osd volume v026",
"mkfs_done": "yes",
"none": "",
"ready": "ready",
"require_osd_release": "",
"type": "bluestore",
"whoami": 241
}
OSD 241's data partition looks like this (after mount /dev/sdb1
/var/lib/ceph/osd/ceph-241):
[root@ceph-adm:ceph-18 ceph-241]# ls -l /var/lib/ceph/osd/ceph-241
total 56
-rw-r--r--. 1 root root 411 Oct 16 2019 activate.monmap
-rw-r--r--. 1 ceph ceph 3 Oct 16 2019 active
lrwxrwxrwx. 1 root root 9 Mar 2 14:19 block -> /dev/sda2
-rw-r--r--. 1 ceph ceph 37 Oct 16 2019 block_uuid
-rw-r--r--. 1 ceph disk 2 Oct 16 2019 bluefs
-rw-r--r--. 1 ceph ceph 37 Oct 16 2019 ceph_fsid
-rw-r--r--. 1 ceph ceph 37 Oct 16 2019 fsid
-rw-------. 1 ceph ceph 58 Oct 16 2019 keyring
-rw-r--r--. 1 ceph disk 8 Oct 16 2019 kv_backend
-rw-r--r--. 1 ceph ceph 21 Oct 16 2019 magic
-rw-r--r--. 1 ceph disk 4 Oct 16 2019 mkfs_done
-rw-r--r--. 1 ceph ceph 0 Nov 23 14:58 none
-rw-r--r--. 1 ceph disk 6 Oct 16 2019 ready
-rw-r--r--. 1 ceph disk 2 Jan 31 2020 require_osd_release
-rw-r--r--. 1 ceph ceph 10 Oct 16 2019 type
-rw-r--r--. 1 ceph ceph 4 Oct 16 2019 whoami
The symlink "block -> /dev/sda2" goes to the wrong disk. How can I fix that
in a stable way? Also, why are not stable "/dev/disk/by-uuid/..." link targets
created instead? Can I change that myself?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14