Issue Replacing OSD with cephadm: Partition Path Not Accepted
I am on a journey, so far successful, to update our clusters to supported versions. I started with Luminous and Ubuntu 16.04, and now we are on Reef with Ubuntu 20.04. We still have more updates to do, but at the moment, I encountered an issue with an OSD, and it was necessary to replace a disk. Since the cluster was adopted, I'm not entirely sure what the best way to replace this OSD is, as with cephadm, it doesn't like when the path for the device is a partition. I could recreate the OSD using traditional methods and then adopt the OSD, but that doesn't seem like the best approach. Does anyone know how I should proceed to recreate this OSD? I had the same problem in my lab, where I am already on Quincy. What I am trying to do is: # ceph orch osd rm 6 --replace --zap # ceph orch daemon add osd osd-nodeXXX:data_devices=/dev/sdb1,db_devices=/dev/sda3 The error it gives is: /usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices
I would try it with a spec file that contains a path to the partition (limit the placement to that host only). Or have you tried it already? I don’t use partitions for ceph, but there have been threads from other users who use partitions and with spec files it seemed to work. You can generate a preview with ‚ceph orch apply -i osd-spec.yaml --dry-run‘. Zitat von Herbert Faleiros <faleiros@gmail.com>:
I am on a journey, so far successful, to update our clusters to supported versions. I started with Luminous and Ubuntu 16.04, and now we are on Reef with Ubuntu 20.04. We still have more updates to do, but at the moment, I encountered an issue with an OSD, and it was necessary to replace a disk. Since the cluster was adopted, I'm not entirely sure what the best way to replace this OSD is, as with cephadm, it doesn't like when the path for the device is a partition. I could recreate the OSD using traditional methods and then adopt the OSD, but that doesn't seem like the best approach. Does anyone know how I should proceed to recreate this OSD? I had the same problem in my lab, where I am already on Quincy.
What I am trying to do is:
# ceph orch osd rm 6 --replace --zap # ceph orch daemon add osd osd-nodeXXX:data_devices=/dev/sdb1,db_devices=/dev/sda3
The error it gives is:
/usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 9/2/24 20:24, Herbert Faleiros wrote:
/usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices
A Ceph OSD nowadays needs a logical volume because it stores crucial metadata in the LV tags. This helps to activate the OSD. IMHO you will have to redeploy the OSD to use LVM on the disk. It does not need to be the whole disk if there is other data on it. It should be sufficient to make /dev/sdb1 a PV of a new VG for the LV of the OSD. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
I thought bluestore stored that stuff in non lvm mode? ________________________________________ From: Robert Sander <r.sander@heinlein-support.de> Sent: Monday, September 2, 2024 11:35 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Issue Replacing OSD with cephadm: Partition Path Not Accepted Check twice before you click! This email originated from outside PNNL. Hi, On 9/2/24 20:24, Herbert Faleiros wrote:
/usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices
A Ceph OSD nowadays needs a logical volume because it stores crucial metadata in the LV tags. This helps to activate the OSD. IMHO you will have to redeploy the OSD to use LVM on the disk. It does not need to be the whole disk if there is other data on it. It should be sufficient to make /dev/sdb1 a PV of a new VG for the LV of the OSD. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de/ Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 03/09/2024 03:35, Robert Sander wrote:
Hi,
Hello,
On 9/2/24 20:24, Herbert Faleiros wrote:
/usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices
A Ceph OSD nowadays needs a logical volume because it stores crucial metadata in the LV tags. This helps to activate the OSD. IMHO you will have to redeploy the OSD to use LVM on the disk. It does not need to be the whole disk if there is other data on it. It should be sufficient to make /dev/sdb1 a PV of a new VG for the LV of the OSD.
thank you for the suggestion. I understand the need for Ceph OSDs to use LVM due to the metadata stored in LV tags. However, I’m facing a challenge with the disk replacement process. Since I’ve already migrated the OSDs to use |ceph-volume|, I was hoping that |cephadm| would handle the creation of LVM structures automatically. Unfortunately, it doesn’t seem to recreate these structures on its own when replacing a disk, and manually creating them isn’t ideal because |ceph-volume| uses its own specific naming conventions. Do you have any recommendations on how to proceed with |cephadm| in a way that it can handle the LVM setup automatically, or perhaps another method that aligns with the conventions used by |ceph-volume|? -- Herbert
Regards
Hi, apparently, I was wrong about specifying a partition in the path option of the spec file. In my quick test it doesn't work either. Creating a PV, VG, LV on that partition makes it work: ceph orch daemon add osd soc9-ceph:data_devices=ceph-manual-vg/ceph-osd Created osd(s) 3 on host 'soc9-ceph' But if you want an easy cluster management, especially OSDs, I'd recommend to use the entire (raw) devices. It really makes (almost) everything easier. Just recently a customer was quite pleased with the process compared to prior ceph versions. They removed a failed disk and only had to run 'ceph orch osd rm <OSD> --replace' and ceph did the rest (given the existing service specs cover it). They literally said:"wow, this changes our whole impression of ceph". They didn't have many disk replacements in the past 5 years, this was the first one since they adopted the cluster with cephadm. Fiddling with partitions seems quite unnecessary, especially on larger deployments. Regards, Eugen Zitat von Herbert Faleiros <faleiros@gmail.com>:
On 03/09/2024 03:35, Robert Sander wrote:
Hi,
Hello,
On 9/2/24 20:24, Herbert Faleiros wrote:
/usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices
A Ceph OSD nowadays needs a logical volume because it stores crucial metadata in the LV tags. This helps to activate the OSD. IMHO you will have to redeploy the OSD to use LVM on the disk. It does not need to be the whole disk if there is other data on it. It should be sufficient to make /dev/sdb1 a PV of a new VG for the LV of the OSD.
thank you for the suggestion. I understand the need for Ceph OSDs to use LVM due to the metadata stored in LV tags. However, I’m facing a challenge with the disk replacement process. Since I’ve already migrated the OSDs to use |ceph-volume|, I was hoping that |cephadm| would handle the creation of LVM structures automatically. Unfortunately, it doesn’t seem to recreate these structures on its own when replacing a disk, and manually creating them isn’t ideal because |ceph-volume| uses its own specific naming conventions.
Do you have any recommendations on how to proceed with |cephadm| in a way that it can handle the LVM setup automatically, or perhaps another method that aligns with the conventions used by |ceph-volume|?
--
Herbert
Regards
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
One of my major regrets is that there isn't a "Ceph Lite" for setups where you want a cluster with "only" a few terabytes and a half-dozen servers. Ceph excels at really, really big storage and the tuning parameters reflect that. I, too ran into the issue where I couldn't allocate a disk partition to Ceph. I didn't really want the extra potential overhead of LVM, but I had a 4TB SSD and only needed about a quarter of that for an OSD with the rest for local private use, so a partition seemed like a good compromise. No such luck. It was either the entire 4TB or LVM. So it ended up as LVM. Tim On Wed, 2024-09-04 at 10:47 +0000, Eugen Block wrote:
Hi,
apparently, I was wrong about specifying a partition in the path option of the spec file. In my quick test it doesn't work either. Creating a PV, VG, LV on that partition makes it work:
ceph orch daemon add osd soc9-ceph:data_devices=ceph-manual-vg/ceph- osd Created osd(s) 3 on host 'soc9-ceph'
But if you want an easy cluster management, especially OSDs, I'd recommend to use the entire (raw) devices. It really makes (almost) everything easier. Just recently a customer was quite pleased with the process compared to prior ceph versions. They removed a failed disk and only had to run 'ceph orch osd rm <OSD> --replace' and ceph did the rest (given the existing service specs cover it). They literally said:"wow, this changes our whole impression of ceph". They didn't have many disk replacements in the past 5 years, this was the first one since they adopted the cluster with cephadm.
Fiddling with partitions seems quite unnecessary, especially on larger deployments.
Regards, Eugen
Zitat von Herbert Faleiros <faleiros@gmail.com>:
On 03/09/2024 03:35, Robert Sander wrote:
Hi,
Hello,
On 9/2/24 20:24, Herbert Faleiros wrote:
/usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices
A Ceph OSD nowadays needs a logical volume because it stores crucial metadata in the LV tags. This helps to activate the OSD. IMHO you will have to redeploy the OSD to use LVM on the disk. It does not need to be the whole disk if there is other data on it. It should be sufficient to make /dev/sdb1 a PV of a new VG for the LV of the OSD.
thank you for the suggestion. I understand the need for Ceph OSDs to use LVM due to the metadata stored in LV tags. However, I’m facing a challenge with the disk replacement process. Since I’ve already migrated the OSDs to use |ceph-volume|, I was hoping that |cephadm| would handle the creation of LVM structures automatically. Unfortunately, it doesn’t seem to recreate these structures on its own when replacing a disk, and manually creating them isn’t ideal because |ceph-volume| uses its own specific naming conventions.
Do you have any recommendations on how to proceed with |cephadm| in a way that it can handle the LVM setup automatically, or perhaps another method that aligns with the conventions used by |ceph- volume|?
--
Herbert
Regards
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Eugen Block
-
Fox, Kevin M
-
Herbert Faleiros
-
Robert Sander
-
Tim Holloway