28 Mar
2023
28 Mar
'23
8:12 a.m.
Hi,
I am trying to add a new server to an existing cluster, but cannot get the OSDs to create
correctly
When I try
Cephadm ceph-volume lvm create, it returns nothing but the container info.
[root@hiho ~]# cephadm ceph-volume lvm create --bluestore --data /dev/sdd --block.db
/dev/nvme0n1p3
Inferring fsid fe3a7cb0-69ca-11eb-8d45-c86000d08867
Using ceph image with id 'cc65afd6173a' and tag '<none>' created on
2022-10-17 23:41:41 +0000 UTC
quay.io/ceph/ceph@sha256:2b73ccc9816e0a1ee1dfbe21ba9a8cc085210f1220f597b5050ebfcac4bdd346<mailto:quay.io/ceph/ceph@sha256:2b73ccc9816e0a1ee1dfbe21ba9a8cc085210f1220f597b5050ebfcac4bdd346>
so I tried cephadm shell,
and
ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
41dafd4d-0579-4119-acca-6db31586a10f
stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 auth: unable to find a keyring on
/var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d0060d70) no
keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 auth: unable to find a keyring on
/var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d0063da0) no
keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
stderr: 2023-03-28T03:32:27.437+0000 7fa5d6253700 -1 auth: unable to find a keyring on
/var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
stderr: 2023-03-28T03:32:27.437+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d6251ea0) no
keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
stderr: 2023-03-28T03:32:27.451+0000 7fa5ceffd700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
stderr: 2023-03-28T03:32:27.453+0000 7fa5cf7fe700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
stderr: 2023-03-28T03:32:27.473+0000 7fa5cffff700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
stderr: 2023-03-28T03:32:27.474+0000 7fa5d6253700 -1 monclient: authenticate NOTE: no
keyring found; disabled cephx authentication
stderr: [errno 13] RADOS permission denied (error connecting to the cluster)
--> RuntimeError: Unable to create a new OSD id
I then copy the key ring file into the container using scp, but by that time the
orchestrator created OSDs on the drives, so I have to delete the OSDs and start over.
Then if I get the timing just right, I get this (from within cephadm shell):
[ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd
--block.db /dev/nvme0n1p3
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
e6e316d4-670d-4a9b-a50c-bc14d57394a3
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes
ceph-4d95584a-df28-4e21-9480-09a13f1fb804 /dev/sdd
stdout: Physical volume "/dev/sdd" successfully created.
stdout: Volume group "ceph-4d95584a-df28-4e21-9480-09a13f1fb804" successfully
created
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 953861 -n
osd-block-e6e316d4-670d-4a9b-a50c-bc14d57394a3 ceph-4d95584a-df28-4e21-9480-09a13f1fb804
stdout: Logical volume "osd-block-e6e316d4-670d-4a9b-a50c-bc14d57394a3"
created.
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 119209 -n
osd-db-9fc4f199-2c95-4ca7-a35c-ef4b08c86804 ceph-948a633c-420e-4f55-8515-b33e1c0ef18c
stderr: Volume group "ceph-948a633c-420e-4f55-8515-b33e1c0ef18c" has
insufficient free space (0 extents): 119209 required.
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.12 --yes-i-really-mean-it
stderr: purged osd.12
--> RuntimeError: Unable to find any LV for zapping OSD: 12
[ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd
--block.db /dev/nvme0n1p3
--> RuntimeError: Device /dev/sdd has a filesystem.
[ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd
--block.db /dev/nvme0n1p3
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
b93e1a8a-af88-431c-b705-f49d717b050f
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes
ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 /dev/sdd
stdout: Physical volume "/dev/sdd" successfully created.
stdout: Volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" successfully
created
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 953861 -n
osd-block-b93e1a8a-af88-431c-b705-f49d717b050f ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83
stdout: Logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f"
created.
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes
ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 /dev/nvme0n1p3
stdout: Physical volume "/dev/nvme0n1p3" successfully created.
stdout: Volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" successfully
created
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 119209 -n
osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 ceph-4abdb6f8-c891-43f8-8135-dd8470f80130
stdout: Wiping ceph_bluestore signature on
/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13.
stdout: Logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13" created.
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-12
Running command: /usr/bin/chown -h ceph:ceph
/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5
Running command: /usr/bin/ln -s
/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f
/var/lib/ceph/osd/ceph-12/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-12/activate.monmap
stderr: got monmap epoch 33
--> Creating keyring file for osd.12
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12/
Running command: /usr/bin/chown -h ceph:ceph
/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 12
--monmap /var/lib/ceph/osd/ceph-12/activate.monmap --keyfile - --bluestore-block-db-path
/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13
--osd-data /var/lib/ceph/osd/ceph-12/ --osd-uuid b93e1a8a-af88-431c-b705-f49d717b050f
--setuser ceph --setgroup ceph
stderr: 2023-03-28T03:23:21.180+0000 7f0a40fd63c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/)
_read_fsid unparsable uuid
--> ceph-volume lvm prepare successful for: /dev/sdd
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f
--path /var/lib/ceph/osd/ceph-12 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f
/var/lib/ceph/osd/ceph-12/block
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12
Running command: /usr/bin/ln -snf
/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13
/var/lib/ceph/osd/ceph-12/block.db
Running command: /usr/bin/chown -h ceph:ceph
/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block.db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f
stderr: Created symlink
/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f.service<mailto:/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f.service>
->
/usr/lib/systemd/system/ceph-volume@.service<mailto:/usr/lib/systemd/system/ceph-volume@.service>.
Running command: /usr/bin/systemctl enable --runtime ceph-osd@12
stderr: Created symlink
/run/systemd/system/ceph-osd.target.wants/ceph-osd@12.service<mailto:/run/systemd/system/ceph-osd.target.wants/ceph-osd@12.service>
->
/usr/lib/systemd/system/ceph-osd@.service<mailto:/usr/lib/systemd/system/ceph-osd@.service>.
Running command: /usr/bin/systemctl start ceph-osd@12
stderr: Failed to connect to bus: No such file or directory
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.12 --yes-i-really-mean-it
stderr: purged osd.12
--> Zapping:
/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f
--> Unmounting /var/lib/ceph/osd/ceph-12
Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-12
stderr: umount: /var/lib/ceph/osd/ceph-12 unmounted
Running command: /usr/bin/dd if=/dev/zero
of=/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f
bs=1M count=10 conv=fsync
stderr: 10+0 records in
10+0 records out
stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0771422 s, 136 MB/s
--> Only 1 LV left in VG, will proceed to destroy volume group
ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f
ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83
stderr: Removing
ceph--e91c86b4--613f--45b0--b9d1--3bb76ed10f83-osd--block--b93e1a8a--af88--431c--b705--f49d717b050f
(253:5)
stderr: Releasing logical volume
"osd-block-b93e1a8a-af88-431c-b705-f49d717b050f"
Archiving volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" metadata
(seqno 5).
stdout: Logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f"
successfully removed.
stderr: Removing physical volume "/dev/sdd" from volume group
"ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83"
stdout: Volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" successfully
removed
stderr: Creating volume group backup
"/etc/lvm/backup/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" (seqno 6).
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdd
stdout: Labels on physical volume "/dev/sdd" successfully wiped.
--> Zapping:
/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13
Running command: /usr/bin/dd if=/dev/zero
of=/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13
bs=1M count=10 conv=fsync
stderr: 10+0 records in
10+0 records out
stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131332 s, 798 MB/s
--> Only 1 LV left in VG, will proceed to destroy volume group
ceph-4abdb6f8-c891-43f8-8135-dd8470f80130
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f
ceph-4abdb6f8-c891-43f8-8135-dd8470f80130
stderr: Removing
ceph--4abdb6f8--c891--43f8--8135--dd8470f80130-osd--db--5bab4d99--e2e5--48e7--aee6--d1a2103b9d13
(253:8)
stderr: Releasing logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13"
Archiving volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" metadata
(seqno 5).
stdout: Logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13"
successfully removed.
stderr: Removing physical volume "/dev/nvme0n1p3" from volume group
"ceph-4abdb6f8-c891-43f8-8135-dd8470f80130"
stdout: Volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" successfully
removed
stderr: Creating volume group backup
"/etc/lvm/backup/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" (seqno 6).
Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc
--net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f
/dev/nvme0n1p3
stdout: Labels on physical volume "/dev/nvme0n1p3" successfully wiped.
--> Zapping successful for OSD: 12
--> RuntimeError: command returned non-zero exit status: 1
I have 3 other servers with the split data and block db that I was able to install just
using cephadm, so I am not sure what is off on this one.
Is there any way to either add the block.db to the already created OSDs or get around the
missing bootstrap.osd/ceph or manually configure the disk and block db?
I have also tried
ceph orch apply osd --all-available-devices --unmanaged=true
to stop ceph from trying to take the OSD, but it still does
Thanks,
Rob
28 Mar
28 Mar
12:19 p.m.
Hi,
On 28.03.23 05:42, Robert W. Eckert wrote:
I am trying to add a new server to an existing cluster, but cannot get the OSDs to create
correctly
When I try
Cephadm ceph-volume lvm create, it returns nothing but the container info.
You are running a containerized cluster with the cephadm orchestrator?
Which version?
Have you tried
ceph orch daemon add osd host1:data_devices=/dev/sda,/dev/sdb,db_devices=/dev/nvme0
as shown on https://docs.ceph.com/en/quincy/cephadm/services/osd/ ?
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
29 Mar
29 Mar
3:39 a.m.
Thanks -
I am on 17.2.5 - I was able to get to it by cephadm shell, a few zap and deletes of the
/dev/sdx and ceph-volume lvm-prepare.
I did miss seeing the db_devices part. For ceph orch apply - that would have saved a lot
of effort. Does the osds_per_device create the partitions on the db device?
Also is there any way to disable --all-available-devices if it was turned on.
The
ceph orch apply osd --all-available-devices --unmanaged=true
command doesn't seem to disable the behavior of adding new drives.
-----Original Message-----
From: Robert Sander <r.sander(a)heinlein-support.de>
Sent: Tuesday, March 28, 2023 3:50 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Adding new server to existing ceph cluster - with separate
block.db on NVME
Hi,
On 28.03.23 05:42, Robert W. Eckert wrote:
I am trying to add a new server to an existing cluster, but cannot get
the OSDs to create correctly When I try Cephadm ceph-volume lvm
create, it returns nothing but the container info.
You are running a containerized cluster with the cephadm orchestrator?
Which version?
Have you tried
ceph orch daemon add osd host1:data_devices=/dev/sda,/dev/sdb,db_devices=/dev/nvme0
as shown on https://docs.ceph.com/en/quincy/cephadm/services/osd/ ?
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
11:39 a.m.
On 29.03.23 01:09, Robert W. Eckert wrote:
I did miss seeing the db_devices part. For ceph orch
apply - that would have saved a lot of effort. Does the osds_per_device create the
partitions on the db device?
No, osds_per_device creates multiple OSDs on one data device, could be
useful for NVMe, do no use on HDD.
The command automatically creates the number of db slots on the
db_device based on how many data_devices you pass it.
If you want more slots for the RocksDB then pass it the db_slots parameter.
Also is there any way to disable
--all-available-devices if it was turned on.
The
ceph orch apply osd --all-available-devices --unmanaged=true
command doesn't seem to disable the behavior of adding new drives.
You can set the service to unmanaged when exporting the specification.
ceph orch ls osd --export > osd.yml
Edit osd.yml and add "unmanaged: true" to the specification. After that
ceph orch apply -i osd.yml
Or you could just remove the specification with "ceph orch rm NAME".
The OSD service will be removed but the OSD will remain.
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
414
days inactive
415
days old
3 comments
2 participants
participants (2)
-
Robert Sander -
Robert W. Eckert