Can't add OSD id in manual deploy
Hey all, I'm trying to deploy Ceph 15.2.4 on Ubuntu 20.04 and I am going through the manual deploy process [1]. I was able to successfully bootstrap the monitor and manager and am now trying to add the OSD's, but the `ceph-volume` command is hanging when running `ceph osd new`. It appears the command hangs for about 30 minutes and then times out with this error: user@node1:~$ sudo ceph-volume lvm create --data /dev/sdc Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9a305232-6b0a-4fde-9685-68fe6150da0e stderr: [errno 110] RADOS timed out (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id I have also tried to manually run the `ceph osd new` command, but it results in the same error. My cluster shows as up (at least the monitor and manager): user@node1:~$ sudo ceph -s cluster: id: 6aacdfe1-42e4-4106-9503-3e16e9812195 health: HEALTH_WARN 1 monitors have not enabled msgr2 Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum bllcloudceph01 (age 23h) mgr: bllcloudceph01(active, since 23h) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown And ceph appears to be listening: user@node1:~$ sudo ss -tupanl | grep ceph tcp LISTEN 0 512 10.2.22.1:6789 0.0.0.0:* users:(("ceph-mon",pid=136244,fd=25)) tcp LISTEN 0 512 10.2.22.1:6800 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=25)) tcp LISTEN 0 512 10.2.22.1:6801 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=26)) It seems like I'm missing something simple, but I can't spot it. Can anybody point me in the right direction? [1] https://docs.ceph.com/docs/master/install/manual-deployment/ -- Thanks, Joshua Schaeffer
The OSD node also needs the ceph.conf, it seems that is not the case in your setup. Zitat von Joshua Schaeffer <jschaeffer@harmonywave.com>:
Hey all,
I'm trying to deploy Ceph 15.2.4 on Ubuntu 20.04 and I am going through the manual deploy process [1]. I was able to successfully bootstrap the monitor and manager and am now trying to add the OSD's, but the `ceph-volume` command is hanging when running `ceph osd new`. It appears the command hangs for about 30 minutes and then times out with this error:
user@node1:~$ sudo ceph-volume lvm create --data /dev/sdc Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9a305232-6b0a-4fde-9685-68fe6150da0e stderr: [errno 110] RADOS timed out (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id
I have also tried to manually run the `ceph osd new` command, but it results in the same error. My cluster shows as up (at least the monitor and manager):
user@node1:~$ sudo ceph -s cluster: id: 6aacdfe1-42e4-4106-9503-3e16e9812195 health: HEALTH_WARN 1 monitors have not enabled msgr2 Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum bllcloudceph01 (age 23h) mgr: bllcloudceph01(active, since 23h) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown
And ceph appears to be listening:
user@node1:~$ sudo ss -tupanl | grep ceph tcp LISTEN 0 512 10.2.22.1:6789 0.0.0.0:* users:(("ceph-mon",pid=136244,fd=25)) tcp LISTEN 0 512 10.2.22.1:6800 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=25)) tcp LISTEN 0 512 10.2.22.1:6801 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=26))
It seems like I'm missing something simple, but I can't spot it. Can anybody point me in the right direction?
[1] https://docs.ceph.com/docs/master/install/manual-deployment/
-- Thanks, Joshua Schaeffer
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The OSD node is the same as the monitor and manager node at the moment and has the ceph.conf file: user@node1:~$ ls -l /etc/ceph/ total 15 -rw------- 1 root root 151 Aug 13 15:50 ceph.client.admin.keyring -rw-r--r-- 1 root root 432 Aug 13 16:09 ceph.conf -rw-r--r-- 1 root root 92 Jun 30 16:44 rbdmap On 8/14/20 10:19 AM, Eugen Block wrote:
The OSD node also needs the ceph.conf, it seems that is not the case in your setup.
Zitat von Joshua Schaeffer <jschaeffer@harmonywave.com>:
Hey all,
I'm trying to deploy Ceph 15.2.4 on Ubuntu 20.04 and I am going through the manual deploy process [1]. I was able to successfully bootstrap the monitor and manager and am now trying to add the OSD's, but the `ceph-volume` command is hanging when running `ceph osd new`. It appears the command hangs for about 30 minutes and then times out with this error:
user@node1:~$ sudo ceph-volume lvm create --data /dev/sdc Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9a305232-6b0a-4fde-9685-68fe6150da0e stderr: [errno 110] RADOS timed out (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id
I have also tried to manually run the `ceph osd new` command, but it results in the same error. My cluster shows as up (at least the monitor and manager):
user@node1:~$ sudo ceph -s cluster: id: 6aacdfe1-42e4-4106-9503-3e16e9812195 health: HEALTH_WARN 1 monitors have not enabled msgr2 Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum bllcloudceph01 (age 23h) mgr: bllcloudceph01(active, since 23h) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown
And ceph appears to be listening:
user@node1:~$ sudo ss -tupanl | grep ceph tcp LISTEN 0 512 10.2.22.1:6789 0.0.0.0:* users:(("ceph-mon",pid=136244,fd=25)) tcp LISTEN 0 512 10.2.22.1:6800 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=25)) tcp LISTEN 0 512 10.2.22.1:6801 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=26))
It seems like I'm missing something simple, but I can't spot it. Can anybody point me in the right direction?
[1] https://docs.ceph.com/docs/master/install/manual-deployment/
-- Thanks, Joshua Schaeffer
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Thanks, Joshua Schaeffer
I didn't notice that. Have you tried this multiple times with the same disk? Do you see any other error messages in syslog? Zitat von Joshua Schaeffer <jschaeffer@harmonywave.com>:
The OSD node is the same as the monitor and manager node at the moment and has the ceph.conf file:
user@node1:~$ ls -l /etc/ceph/ total 15 -rw------- 1 root root 151 Aug 13 15:50 ceph.client.admin.keyring -rw-r--r-- 1 root root 432 Aug 13 16:09 ceph.conf -rw-r--r-- 1 root root 92 Jun 30 16:44 rbdmap
On 8/14/20 10:19 AM, Eugen Block wrote:
The OSD node also needs the ceph.conf, it seems that is not the case in your setup.
Zitat von Joshua Schaeffer <jschaeffer@harmonywave.com>:
Hey all,
I'm trying to deploy Ceph 15.2.4 on Ubuntu 20.04 and I am going through the manual deploy process [1]. I was able to successfully bootstrap the monitor and manager and am now trying to add the OSD's, but the `ceph-volume` command is hanging when running `ceph osd new`. It appears the command hangs for about 30 minutes and then times out with this error:
user@node1:~$ sudo ceph-volume lvm create --data /dev/sdc Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9a305232-6b0a-4fde-9685-68fe6150da0e stderr: [errno 110] RADOS timed out (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id
I have also tried to manually run the `ceph osd new` command, but it results in the same error. My cluster shows as up (at least the monitor and manager):
user@node1:~$ sudo ceph -s cluster: id: 6aacdfe1-42e4-4106-9503-3e16e9812195 health: HEALTH_WARN 1 monitors have not enabled msgr2 Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum bllcloudceph01 (age 23h) mgr: bllcloudceph01(active, since 23h) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown
And ceph appears to be listening:
user@node1:~$ sudo ss -tupanl | grep ceph tcp LISTEN 0 512 10.2.22.1:6789 0.0.0.0:* users:(("ceph-mon",pid=136244,fd=25)) tcp LISTEN 0 512 10.2.22.1:6800 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=25)) tcp LISTEN 0 512 10.2.22.1:6801 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=26))
It seems like I'm missing something simple, but I can't spot it. Can anybody point me in the right direction?
[1] https://docs.ceph.com/docs/master/install/manual-deployment/
-- Thanks, Joshua Schaeffer
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Thanks, Joshua Schaeffer
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/14/20 10:57 AM, Eugen Block wrote:
I didn't notice that. Have you tried this multiple times with the same disk? Do you see any other error messages in syslog? Thanks Eugen for your fast response. Yes, I have tried it multiple times, but I'm trying again right now just to be sure the outcome is the same.
- ceph.log and ceph-mgr.node1.log don't have much. - syslog itself doesn't show anything Ceph related. - ceph-volume logs show the following: [2020-08-14 17:15:02,479][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size [2020-08-14 17:15:02,661][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/sdc [2020-08-14 17:15:02,674][ceph_volume.process][INFO ] stdout NAME="sdc" KNAME="sdc" MAJ:MIN="8:32" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="LOGICAL_VOLUME" SIZE="68.3G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="mq-deadline" TYPE="disk" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" PKNAME="" PARTLABEL="" [2020-08-14 17:15:02,675][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -p /dev/sdc [2020-08-14 17:15:02,696][ceph_volume.process][INFO ] Running command: /usr/sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size /dev/sdc [2020-08-14 17:15:02,824][ceph_volume.process][INFO ] stderr Failed to find physical volume "/dev/sdc". [2020-08-14 17:15:02,826][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sdc [2020-08-14 17:15:02,905][ceph_volume.process][INFO ] stderr unable to read label for /dev/sdc: (2) No such file or directory [2020-08-14 17:15:02,907][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sdc [2020-08-14 17:15:02,985][ceph_volume.process][INFO ] stderr unable to read label for /dev/sdc: (2) No such file or directory [2020-08-14 17:15:02,987][ceph_volume.process][INFO ] Running command: /usr/bin/udevadm info --query=property /dev/sdc [2020-08-14 17:15:02,997][ceph_volume.process][INFO ] stdout DEVPATH=/devices/pci0000:40/0000:40:11.0/0000:48:00.0/host4/target4:1:0/4:1:0:2/block/sdc [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout DEVNAME=/dev/sdc [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout DEVTYPE=disk [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout MAJOR=8 [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout MINOR=32 [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout SUBSYSTEM=block [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=2653962 [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_SCSI=1 [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_VENDOR=HP [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_VENDOR_ENC=HP\x20\x20\x20\x20\x20\x20 [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_MODEL=LOGICAL_VOLUME [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_MODEL_ENC=LOGICAL\x20VOLUME\x20\x20 [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_REVISION=3.00 [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_TYPE=disk [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_SERIAL=3600508b1001039565a35315242571100 [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_SERIAL_SHORT=600508b1001039565a35315242571100 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_WWN=0x600508b100103956 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_WWN_VENDOR_EXTENSION=0x5a35315242571100 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_WWN_WITH_EXTENSION=0x600508b1001039565a35315242571100 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_SCSI_SERIAL=PACCR0M9VZ51RBW [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_BUS=scsi [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout ID_PATH=pci-0000:48:00.0-scsi-0:1:0:2 [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout ID_PATH_TAG=pci-0000_48_00_0-scsi-0_1_0_2 [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/disk/by-path/pci-0000:48:00.0-scsi-0:1:0:2 /dev/disk/by-id/wwn-0x600508b1001039565a35315242571100 /dev/disk/by-id/scsi-3600508b1001039565a35315242571100 [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout TAGS=:systemd: [2020-08-14 17:15:03,004][ceph_volume.api.lvm][WARNING] device is not part of ceph: None [2020-08-14 17:15:03,005][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key [2020-08-14 17:15:03,059][ceph_volume.process][INFO ] stdout AQCXxjZf5glOAxAAWBFST6v49TlLUGlENabhUw== [2020-08-14 17:15:03,061][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 65550aaa-e137-42ea-b16f-dcec5fb15406 It looks like it is probing the disk for LVM information. Do I need to manually prepare this disk first with LVM before I can use it? Currently the disk should be clean. Any prior filesystems should be wiped completely, no RAID, GRUB, or boot partitions on it either. -- Thanks, Joshua Schaeffer
Usually it should also accept the device path (although I haven't tried that in Octopus yet), you could try `ceph-volume lvm prepare --data /path/to/device` first and then activate it. If that doesn't work, try to create a vg and lv and try it with LVM syntax (ceph-volume lvm prepare --data {vg}/{lv}). I don't have a cluster at hand right now so I can't double check. But I find it strange that it doesn't accept the device path, maybe someone with more Octopus experience can chime in. Zitat von Joshua Schaeffer <jschaeffer@harmonywave.com>:
I didn't notice that. Have you tried this multiple times with the same disk? Do you see any other error messages in syslog? Thanks Eugen for your fast response. Yes, I have tried it multiple times, but I'm trying again right now just to be sure the outcome is
On 8/14/20 10:57 AM, Eugen Block wrote: the same.
- ceph.log and ceph-mgr.node1.log don't have much. - syslog itself doesn't show anything Ceph related. - ceph-volume logs show the following:
[2020-08-14 17:15:02,479][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size [2020-08-14 17:15:02,661][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/sdc [2020-08-14 17:15:02,674][ceph_volume.process][INFO ] stdout NAME="sdc" KNAME="sdc" MAJ:MIN="8:32" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="LOGICAL_VOLUME" SIZE="68.3G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="mq-deadline" TYPE="disk" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" PKNAME="" PARTLABEL="" [2020-08-14 17:15:02,675][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -p /dev/sdc [2020-08-14 17:15:02,696][ceph_volume.process][INFO ] Running command: /usr/sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size /dev/sdc [2020-08-14 17:15:02,824][ceph_volume.process][INFO ] stderr Failed to find physical volume "/dev/sdc". [2020-08-14 17:15:02,826][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sdc [2020-08-14 17:15:02,905][ceph_volume.process][INFO ] stderr unable to read label for /dev/sdc: (2) No such file or directory [2020-08-14 17:15:02,907][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sdc [2020-08-14 17:15:02,985][ceph_volume.process][INFO ] stderr unable to read label for /dev/sdc: (2) No such file or directory [2020-08-14 17:15:02,987][ceph_volume.process][INFO ] Running command: /usr/bin/udevadm info --query=property /dev/sdc [2020-08-14 17:15:02,997][ceph_volume.process][INFO ] stdout DEVPATH=/devices/pci0000:40/0000:40:11.0/0000:48:00.0/host4/target4:1:0/4:1:0:2/block/sdc [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout DEVNAME=/dev/sdc [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout DEVTYPE=disk [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout MAJOR=8 [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout MINOR=32 [2020-08-14 17:15:02,998][ceph_volume.process][INFO ] stdout SUBSYSTEM=block [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=2653962 [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_SCSI=1 [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_VENDOR=HP [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_VENDOR_ENC=HP\x20\x20\x20\x20\x20\x20 [2020-08-14 17:15:02,999][ceph_volume.process][INFO ] stdout ID_MODEL=LOGICAL_VOLUME [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_MODEL_ENC=LOGICAL\x20VOLUME\x20\x20 [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_REVISION=3.00 [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_TYPE=disk [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_SERIAL=3600508b1001039565a35315242571100 [2020-08-14 17:15:03,000][ceph_volume.process][INFO ] stdout ID_SERIAL_SHORT=600508b1001039565a35315242571100 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_WWN=0x600508b100103956 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_WWN_VENDOR_EXTENSION=0x5a35315242571100 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_WWN_WITH_EXTENSION=0x600508b1001039565a35315242571100 [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_SCSI_SERIAL=PACCR0M9VZ51RBW [2020-08-14 17:15:03,001][ceph_volume.process][INFO ] stdout ID_BUS=scsi [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout ID_PATH=pci-0000:48:00.0-scsi-0:1:0:2 [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout ID_PATH_TAG=pci-0000_48_00_0-scsi-0_1_0_2 [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/disk/by-path/pci-0000:48:00.0-scsi-0:1:0:2 /dev/disk/by-id/wwn-0x600508b1001039565a35315242571100 /dev/disk/by-id/scsi-3600508b1001039565a35315242571100 [2020-08-14 17:15:03,002][ceph_volume.process][INFO ] stdout TAGS=:systemd: [2020-08-14 17:15:03,004][ceph_volume.api.lvm][WARNING] device is not part of ceph: None [2020-08-14 17:15:03,005][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key [2020-08-14 17:15:03,059][ceph_volume.process][INFO ] stdout AQCXxjZf5glOAxAAWBFST6v49TlLUGlENabhUw== [2020-08-14 17:15:03,061][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 65550aaa-e137-42ea-b16f-dcec5fb15406
It looks like it is probing the disk for LVM information. Do I need to manually prepare this disk first with LVM before I can use it? Currently the disk should be clean. Any prior filesystems should be wiped completely, no RAID, GRUB, or boot partitions on it either.
-- Thanks, Joshua Schaeffer
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 8/14/20 11:52 AM, Eugen Block wrote:
Usually it should also accept the device path (although I haven't tried that in Octopus yet), you could try `ceph-volume lvm prepare --data /path/to/device` first and then activate it. If that doesn't work, try to create a vg and lv and try it with LVM syntax (ceph-volume lvm prepare --data {vg}/{lv}). I don't have a cluster at hand right now so I can't double check. But I find it strange that it doesn't accept the device path, maybe someone with more Octopus experience can chime in. So I still wasn't able to add the device with the above methods, however I did figure out which command (and specifically which options) were causing the problem. When I run the `ceph osd new` command without the -n and -k options it immediately returns with a new OSD ID:
user@node1:~$ UUID=$(uuidgen) user@node1:~$ OSD_SECRET=$(ceph-authtool --gen-print-key) user@node1:~$ echo "{\"cephx_secret\": \"$OSD_SECRET\"}" | sudo ceph osd new $UUID -i - 7 If I run it with the -n and -k options it hangs sames as when I call `ceph-volume` and eventually times out with the same error: user@node1:~$ UUID2=$(uuidgen) user@node1:~$ OSD_SECRET2=$(ceph-authtool --gen-print-key) user@node1:~$ echo "{\"cephx_secret\": \"$OSD_SECRET2\"}" | sudo ceph osd new $UUID2 -i - -n client.bootstrap-osd -k /var/lib/ceph/bootstrap-osd/ceph.keyring [errno 110] RADOS timed out (error connecting to the cluster) If I run it with just the -n option is immediately comes back with an error: user@node1:~$ UUID3=$(uuidgen) user@node1:~$ OSD_SECRET3=$(ceph-authtool --gen-print-key) user@node1:~$ echo "{\"cephx_secret\": \"$OSD_SECRET3\"}" | sudo ceph osd new $UUID3 -i - -n client.bootstrap-osd [errno 2] RADOS object not found (error connecting to the cluster) -- Thanks, Joshua Schaeffer
participants (2)
-
Eugen Block
-
Joshua Schaeffer