19.2.1: filestore created when bluestore requested
Hi, I am working on a small 3 node ceph cluster which used to work as expected. When creating a new ceph osd the ceph-volume command throws some errors and filestore instead of bluestore is created. (The drive was clean using blkdiscard before and no traces left in /etc from previous attempts. Any hints? Regards --martin # ceph-volume lvm create --bluestore --data vgsdb/sdb1 --> Incompatible flags were found, some values may get ignored --> Cannot use None (None) with --bluestore (bluestore) Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 19959f26-f2ed-451d-acb7-a808d15510cf Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/ln -s /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 AuthRegistry(0x772d200660d0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx stderr: got monmap epoch 20 --> Creating keyring file for osd.0 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 19959f26-f2ed-451d-acb7-a808d15510cf --setuser ceph --setgroup ceph stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found stderr: 2025-05-16T14:47:09.190+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vgsdb/sdb1 --path /var/lib/ceph/osd/ceph-0 --no-mon-config Running command: /usr/bin/ln -snf /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf.service → /lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@0 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service → /lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@0 --> ceph-volume lvm activate successful for osd ID: 0 --> ceph-volume lvm create successful for: vgsdb/sdb1 -- Martin Konold - Prokurist, CTO KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany
Hi, which Ceph version is this? It's apparently not managed by cephadm. Zitat von "Konold, Martin" <martin.konold@konsec.com>:
Hi,
I am working on a small 3 node ceph cluster which used to work as expected.
When creating a new ceph osd the ceph-volume command throws some errors and filestore instead of bluestore is created. (The drive was clean using blkdiscard before and no traces left in /etc from previous attempts.
Any hints?
Regards --martin
# ceph-volume lvm create --bluestore --data vgsdb/sdb1 --> Incompatible flags were found, some values may get ignored --> Cannot use None (None) with --bluestore (bluestore) Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 19959f26-f2ed-451d-acb7-a808d15510cf Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/ln -s /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 AuthRegistry(0x772d200660d0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx stderr: got monmap epoch 20 --> Creating keyring file for osd.0 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 19959f26-f2ed-451d-acb7-a808d15510cf --setuser ceph --setgroup ceph stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found stderr: 2025-05-16T14:47:09.190+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vgsdb/sdb1 --path /var/lib/ceph/osd/ceph-0 --no-mon-config Running command: /usr/bin/ln -snf /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf.service → /lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@0 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service → /lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@0 --> ceph-volume lvm activate successful for osd ID: 0 --> ceph-volume lvm create successful for: vgsdb/sdb1
-- Martin Konold - Prokurist, CTO KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 2025-05-16 18:46, schrieb Eugen Block:
Hi,
which Ceph version is this? It's apparently not managed by cephadm.
It is Ceph 19.2.1 from Proxmox running on Debian 12 (bookworm). Regards -- --martin konold ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany
Hello Martin, In proxmox I create OSD using this: ceph-volume lvm create --objectstore bluestore --dmcrypt --osd-id 28 --data /dev/disk/by-id/wwn-0x5000a720300aef63 Friday, May 16, 2025, 5:54:56 PM, you wrote:
Hi,
I am working on a small 3 node ceph cluster which used to work as expected.
When creating a new ceph osd the ceph-volume command throws some errors and filestore instead of bluestore is created. (The drive was clean using blkdiscard before and no traces left in /etc from previous attempts.
Any hints?
Regards --martin
# ceph-volume lvm create --bluestore --data vgsdb/sdb1
Incompatible flags were found, some values may get ignored Cannot use None (None) with --bluestore (bluestore) Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 19959f26-f2ed-451d-acb7-a808d15510cf Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/ln -s /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 AuthRegistry(0x772d200660d0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx stderr: got monmap epoch 20
Creating keyring file for osd.0 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 19959f26-f2ed-451d-acb7-a808d15510cf --setuser ceph --setgroup ceph stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found stderr: 2025-05-16T14:47:09.190+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
ceph-volume lvm prepare successful for: vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vgsdb/sdb1 --path /var/lib/ceph/osd/ceph-0 --no-mon-config Running command: /usr/bin/ln -snf /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf.service → /lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@0 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service → /lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@0
ceph-volume lvm activate successful for osd ID: 0 ceph-volume lvm create successful for: vgsdb/sdb1
-- Martin Konold - Prokurist, CTO KONSEC GmbH - make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best regards, Nmz
participants (3)
-
Eugen Block
-
Konold, Martin
-
Nmz