How to specify id on newly created OSD with Ceph Orchestrator
Hi, Yesterday I had to replace a couple of failed/failing OSDs using Ceph Orchestrator (on Ceph 18.2.1), but I've made a mistake and I removed them without the "--replace" option. I wasn't able to find the correct option (if any) to specify an id for newly created OSD. The closest one was "osd_id_claims", but if I understood correctly is used to claim an OSD id which is still in the crush map (as destroyed). Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation? Cheers Iztok P.S. At the end I (re)added the OSD in the correct order, starting with the lowest id device. -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu
On 7/23/24 08:24, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation?
Why would you want to do that? A new OSD always gets the lowest available ID. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
On 23/07/24 08:41, Robert Sander wrote:
On 7/23/24 08:24, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation?
Why would you want to do that?
For me there wasn't a "real need", I could imagine a scenario in which you want to have a specific osd id range allocated to a specific host, but in my case it is just a curiosity. With "ceph-volume" is possible to specify an id during the OSD creation, but with ceph-orch I didn't find the equivalent option and I'm curious if I've missed something. Cheers Iztok
Why would you want to do that?
You would want to do that to have minimal data movement, that is, limit the wear on disks to the absolutely necessary minimum. If you replace a disk and re-deploy the OSD with the same ID on the same host with the same device class, only the PGs that land on this OSD will move. If you assign the free IDs randomly, a lot more data movement will occur as the entire crush map will be affected and many more PGs become remapped. Try it on a test cluster. Down an OSD and destroy it. See how many PGs are remapped. When recovery finished, purge the OSD ID. Another huge set of data movement will happen. This second part will happen a second time when adding the disk back and is unnecessary if you plan to replace the disk with an (from ceph point of view) identical disk model. That's why the "destroyed" state for OSDs is there, it keeps the OSD IDs place in the crush map, prevents large data movement on OSD ID removal, can be re-used and then again prevents large data movement on adding (two times the second movement I mentioned above). There was also a bug if OSD IDs are not sequential (there are holed in the ID list). This is fixed, but better safe than sorry. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Iztok Gregori <iztok.gregori@elettra.eu> Sent: Tuesday, July 23, 2024 9:10 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: How to specify id on newly created OSD with Ceph Orchestrator On 23/07/24 08:41, Robert Sander wrote:
On 7/23/24 08:24, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation?
Why would you want to do that?
For me there wasn't a "real need", I could imagine a scenario in which you want to have a specific osd id range allocated to a specific host, but in my case it is just a curiosity. With "ceph-volume" is possible to specify an id during the OSD creation, but with ceph-orch I didn't find the equivalent option and I'm curious if I've missed something. Cheers Iztok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Jul 23, 2024 at 08:24:21AM +0200, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation?
You can use osd_id_claims. This command is for replacing a HDD in hybrid osd.344 and reuse the block.db device on the SSD. ceph orch daemon add osd <ceph_osd_host>:data_devices=/dev/sdX,db_devices=/dev/ceph-<uuid1>/osd-block-<uuid2>,osd_id_claims=344 -- Kai Stian
On 26/07/24 12:35, Kai Stian Olstad wrote:
On Tue, Jul 23, 2024 at 08:24:21AM +0200, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation?
You can use osd_id_claims.
I tried the osd_id_claims in a yaml file like this:
service_type: osd placement: hosts: - <hostname> data_devices: paths: - /dev/<device> osd_id_claims: <hostname>: ['<osd_id>']
An then applied it, but the created OSD didn't have the id I specified. It could be that the syntax of my yaml is wrong, but I gave me no errors when I applied it. I didn't try to directly specify the osd_id_claims on the command line. The command should be something like this: # ceph orch daemon add osd <hosname>:<device>,osd_id_claims=<osd_id> ? I don't know if it matters, but I've deleted/removed (not replaced) the OSD (the OSD id wasn't present in the crush map anymore, not even as "destroyed"). Cheers Iztok
On Fri, Jul 26, 2024 at 04:18:05PM +0200, Iztok Gregori wrote:
On 26/07/24 12:35, Kai Stian Olstad wrote:
On Tue, Jul 23, 2024 at 08:24:21AM +0200, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation?
You can use osd_id_claims.
I tried the osd_id_claims in a yaml file like this:
service_type: osd placement: hosts: - <hostname> data_devices: paths: - /dev/<device> osd_id_claims: <hostname>: ['<osd_id>']
An then applied it, but the created OSD didn't have the id I specified. It could be that the syntax of my yaml is wrong, but I gave me no errors when I applied it. I didn't try to directly specify the osd_id_claims on the command line. The command should be something like this:
# ceph orch daemon add osd <hosname>:<device>,osd_id_claims=<osd_id>
According to the documentation[1] you can use osd_id_claim. I use: ceph orch daemon add osd <hosname>:data_devices=<device>,osd_id_claims=<osd_id> The difference is "data_devices=", if you need it or not I don't know.
I don't know if it matters, but I've deleted/removed (not replaced) the OSD (the OSD id wasn't present in the crush map anymore, not even as "destroyed").
It might, I don't think I have tried without --replace since I use a script to replace devices in Ceph so I never forget to add the --replace. [1] https://docs.ceph.com/en/reef/cephadm/services/osd/?highlight=osd_id_claims#... -- Kai Stian
participants (4)
-
Frank Schilder
-
Iztok Gregori
-
Kai Stian Olstad
-
Robert Sander