Problem with cephadm and deploying 4 ODSs on nvme Storage
Hi Community, currently i’m installing an nvme only storage cluster with cephadm from scratch (v.17.2.5). Everything works fine. Each of my nodes (6) has 3 enterprise nvme’s with 7TB capacity. At the beginning I only installed one OSD per nvme, now I want to use four instead of one but I’m struggling with that. Frist of all I set the following option in my cluster: ceph orch apply osd --all-available-devices --unmanaged=true As I understand this option should prevent cephadm to automatically fetch new, available disks and deploying OSD daemons. But that seems not to work. If I delete and purge my OSD and zapping the disk with ceph orch device zap ceph-nvme01 /dev/nvme2n1 –force the disk came available for the cluster and seconds later it deploys the same OSD ID than it has before. I checked that the old OSD was completely removed and the docker container was not started. My next try was to set: ceph orch host label add ceph-nvme01 _no_schedule purge the OSD zapping the disk and following: ceph orch daemon add osd ceph-nvme01:/dev/nvme2n1,osds_per_device=4 removing the _no_schedule flag and again: the old OSD was recreated and not 4. So where is my mistake? Thank you!
Hi, can you paste this output: ceph orch ls osd —export Zitat von claas.goltz@contact-software.com:
Hi Community, currently i’m installing an nvme only storage cluster with cephadm from scratch (v.17.2.5). Everything works fine. Each of my nodes (6) has 3 enterprise nvme’s with 7TB capacity.
At the beginning I only installed one OSD per nvme, now I want to use four instead of one but I’m struggling with that.
Frist of all I set the following option in my cluster: ceph orch apply osd --all-available-devices --unmanaged=true
As I understand this option should prevent cephadm to automatically fetch new, available disks and deploying OSD daemons. But that seems not to work.
If I delete and purge my OSD and zapping the disk with ceph orch device zap ceph-nvme01 /dev/nvme2n1 –force
the disk came available for the cluster and seconds later it deploys the same OSD ID than it has before. I checked that the old OSD was completely removed and the docker container was not started.
My next try was to set: ceph orch host label add ceph-nvme01 _no_schedule purge the OSD zapping the disk and following: ceph orch daemon add osd ceph-nvme01:/dev/nvme2n1,osds_per_device=4 removing the _no_schedule flag
and again: the old OSD was recreated and not 4.
So where is my mistake? Thank you! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This post took a while to be checked from a moderator and meanwhile I found a Service rule, that fetched all my available diskes. I deleted it and after that, all commands works as foreseen. Thanks to all for reading.
Hi Claas, which type of SSD are you using? If these are enterprise-grade NVMe SSDs, there is a good chance they support multiple namespaces. In that case, i would suggest to create 4 namespaces per SSD (you might consider more, depending on your load, available CPU cores and type of SSD) and deploy 1 OSD per namespace. Namespaces save you from all the additional complexity around multiple OSD per SSD, as each namespace will show up as a separate SSD from ceph point of view. E.g. /dev/nvme0n1, /dev/nvme0n2, /dev/nvme0n3… Cheers Gregor
On 6. Mar 2023, at 11:59, claas.goltz@contact-software.com wrote:
This post took a while to be checked from a moderator and meanwhile I found a Service rule, that fetched all my available diskes. I deleted it and after that, all commands works as foreseen.
Thanks to all for reading. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
claas.goltz＠contact-software.com
-
Eugen Block
-
Gregor Radtke