Influencing the osd.id when creating or replacing an osd
Hello, I am still using ceph-deploy to add osd’s to my cluster. From what I have read ceph-deploy does not allow you to specify the osd.id when creating new osds, however I am wondering if there is a way to influence the number that ceph will assign for the next osd that is created. I know that it really shouldn’t matter what osd number gets assigned to the disk but as the number of osd increases it is much easier to keep track of where things are if you can control the id when replacing failed disks or adding new nodes. Thank you, Shain
What release are you running where ceph-deploy still works? I get what you're saying, but really you should get used to OSD IDs being arbitrary. - ``ceph osd ls-tree <name>`` will output a list of OSD ids under the given CRUSH name (like a host or rack name). This is useful for applying changes to entire subtrees. For example, ``ceph osd down `ceph osd ls-tree rack1```. This is useful for one-off scripts, where you can e.g. use it to get a list of OSDs on a given host. Normally the OSD ID selected is the lowest-numbered unused one. Which can either be an ID that has never been used before, or one that has been deleted. So if you delete an OSD entirely and redeploy, you may or may not get the same ID depending on the cluster’s history. - ``ceph osd destroy`` will mark an OSD destroyed and remove its cephx and lockbox keys. However, the OSD id and CRUSH map entry will remain in place, allowing the id to be reused by a replacement device with minimal data rebalancing. Destroying OSDs and redeploying them can help with what you’re after.
On Oct 17, 2024, at 9:14 PM, Shain Miley <SMiley@npr.org> wrote:
Hello, I am still using ceph-deploy to add osd’s to my cluster. From what I have read ceph-deploy does not allow you to specify the osd.id when creating new osds, however I am wondering if there is a way to influence the number that ceph will assign for the next osd that is created.
I know that it really shouldn’t matter what osd number gets assigned to the disk but as the number of osd increases it is much easier to keep track of where things are if you can control the id when replacing failed disks or adding new nodes.
Thank you, Shain
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We are running octopus but will be upgrading to reef or squid in the next few weeks. As part of that upgrade I am planning on switching over to using cephadm as well. Part of what I am doing right now is going through and replacing old drives and removing some of our oldest nodes and replacing them with new ones…then I will convert the rest of the filestore osd over to bluestore so that I can upgrade. One other question based on your suggestion below…my typical process of removing or replacing an osd involves the following: ceph osd crush reweight osd.id 0.0 ceph osd out osd.id service ceph stop osd.id ceph osd crush remove osd.id ceph auth del osd.id ceph osd rm id Does `ceph osd destroy` do something other than the last 3 commands above or am I just doing the same thing using multiple commands? If I need to start issuing the destroy command as well I can. Thank you. Shain From: Anthony D'Atri <aad@dreamsnake.net> Date: Friday, October 18, 2024 at 9:01 AM To: Shain Miley <SMiley@npr.org> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Influencing the osd.id when creating or replacing an osd !-------------------------------------------------------------------| External Email - Use Caution |-------------------------------------------------------------------! What release are you running where ceph-deploy still works? I get what you're saying, but really you should get used to OSD IDs being arbitrary. - ``ceph osd ls-tree <name>`` will output a list of OSD ids under the given CRUSH name (like a host or rack name). This is useful for applying changes to entire subtrees. For example, ``ceph osd down `ceph osd ls-tree rack1```. This is useful for one-off scripts, where you can e.g. use it to get a list of OSDs on a given host. Normally the OSD ID selected is the lowest-numbered unused one. Which can either be an ID that has never been used before, or one that has been deleted. So if you delete an OSD entirely and redeploy, you may or may not get the same ID depending on the cluster’s history. - ``ceph osd destroy`` will mark an OSD destroyed and remove its cephx and lockbox keys. However, the OSD id and CRUSH map entry will remain in place, allowing the id to be reused by a replacement device with minimal data rebalancing. Destroying OSDs and redeploying them can help with what you’re after.
On Oct 17, 2024, at 9:14 PM, Shain Miley <SMiley@npr.org> wrote:
Hello, I am still using ceph-deploy to add osd’s to my cluster. From what I have read ceph-deploy does not allow you to specify the osd.id when creating new osds, however I am wondering if there is a way to influence the number that ceph will assign for the next osd that is created.
I know that it really shouldn’t matter what osd number gets assigned to the disk but as the number of osd increases it is much easier to keep track of where things are if you can control the id when replacing failed disks or adding new nodes.
Thank you, Shain
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Oct 19, 2024, at 2:47 PM, Shain Miley <SMiley@npr.org> wrote:
We are running octopus but will be upgrading to reef or squid in the next few weeks. As part of that upgrade I am planning on switching over to using cephadm as well.
Part of what I am doing right now is going through and replacing old drives and removing some of our oldest nodes and replacing them with new ones…then I will convert the rest of the filestore osd over to bluestore so that I can upgrade.
One other question based on your suggestion below…my typical process of removing or replacing an osd involves the following:
ceph osd crush reweight osd.id <http://osd.id/> 0.0 ceph osd out osd.id <http://osd.id/> service ceph stop osd.id <http://osd.id/> ceph osd crush remove osd.id <http://osd.id/> ceph auth del osd.id <http://osd.id/> ceph osd rm id
Does `ceph osd destroy` do something other than the last 3 commands above or am I just doing the same thing using multiple commands? If I need to start issuing the destroy command as well I can.
I don’t recall if it will stop the service if running, but it does leave the OSD in the CRUSH map marked as ‘destroyed’. I *think* it leaves the auth but I’m not sure.
Hi, after you have reqeighted the osd to 0 and waited for the rebalancing to finish, you can just stop the osd process and "purge" the osd instead of marking it out (since data reshuffling already happened). The "purge" command does a couple of things at once, like removing it from the crush tree and deleting auth caps. This if from the command help output (ceph osd purge -h): osd purge <id|osd.id> [--force] [--yes-i-really-mean-it] --> purge all osd data from the monitors including the OSD id and CRUSH position So your procedure could be reduced to three steps if you don't need to retain the osd id: 1. ceph osd crush reweight osd.<ID> 0 2. stop osd 3. ceph osd purge <ID> [--force] [--yes-i-really-mean-it] You can also add one more safety switch before purging it: ceph osd safe-to-destroy <ID> The "destroy" commmand leaves the ID, auth caps and crush weight intact in case you want to replace the OSD with a drive of the same size. In that case you would not do the "crush reweight" but just stop the osd and set it "out", wait for the recovery to finish, then mark it as "destroyed" and then recreate the osd with a new drive. Regards, Eugen Zitat von Anthony D'Atri <aad@dreamsnake.net>:
On Oct 19, 2024, at 2:47 PM, Shain Miley <SMiley@npr.org> wrote:
We are running octopus but will be upgrading to reef or squid in the next few weeks. As part of that upgrade I am planning on switching over to using cephadm as well.
Part of what I am doing right now is going through and replacing old drives and removing some of our oldest nodes and replacing them with new ones…then I will convert the rest of the filestore osd over to bluestore so that I can upgrade.
One other question based on your suggestion below…my typical process of removing or replacing an osd involves the following:
ceph osd crush reweight osd.id <http://osd.id/> 0.0 ceph osd out osd.id <http://osd.id/> service ceph stop osd.id <http://osd.id/> ceph osd crush remove osd.id <http://osd.id/> ceph auth del osd.id <http://osd.id/> ceph osd rm id
Does `ceph osd destroy` do something other than the last 3 commands above or am I just doing the same thing using multiple commands? If I need to start issuing the destroy command as well I can.
I don’t recall if it will stop the service if running, but it does leave the OSD in the CRUSH map marked as ‘destroyed’. I *think* it leaves the auth but I’m not sure. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 10/18/24 03:14, Shain Miley wrote:
I know that it really shouldn’t matter what osd number gets assigned to the disk but as the number of osd increases it is much easier to keep track of where things are if you can control the id when replacing failed disks or adding new nodes.
My advice: Do not try to manually number your OSDs. Use commands like "ceph osd tree-from HOST" to list the OSDs of a host or "ceph osd ok-to-stop ID" to see if an OSD may be stopped or "ceph osd metadata ID" to see where the OSD is running. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Yes, but it's irritating. Ideally, I'd like my OSD IDs and hostnames to track so that if a server going pong I can find it and fix it ASAP. But it doesn't take much maintenance to break that scheme and the only thing more painful than renaming a Ceph host is re-numbering an OSD. On 10/28/24 06:29, Robert Sander wrote:
On 10/18/24 03:14, Shain Miley wrote:
I know that it really shouldn’t matter what osd number gets assigned to the disk but as the number of osd increases it is much easier to keep track of where things are if you can control the id when replacing failed disks or adding new nodes.
My advice: Do not try to manually number your OSDs.
Use commands like "ceph osd tree-from HOST" to list the OSDs of a host or "ceph osd ok-to-stop ID" to see if an OSD may be stopped or "ceph osd metadata ID" to see where the OSD is running.
Regards
Yes, but it's irritating. Ideally, I'd like my OSD IDs and hostnames to track so that if a server going pong I can find it and fix it ASAP
`ceph osd tree down` etc. (including alertmanager rules and Grafana panels) arguably make that faster and easier than everyone having to memorize OSD numbers, especially as the clusters grow.
But it doesn't take much maintenance to break that scheme and the only thing more painful than renaming a Ceph host is re-numbering an OSD.
Yep!
My advice: Do not try to manually number your OSDs.
This. I’ve been there myself, but it’s truly a sisyphean goal.
On Mon, Oct 28, 2024 at 9:22 AM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Yes, but it's irritating. Ideally, I'd like my OSD IDs and hostnames to track so that if a server going pong I can find it and fix it ASAP
`ceph osd tree down` etc. (including alertmanager rules and Grafana panels) arguably make that faster and easier than everyone having to memorize OSD numbers, especially as the clusters grow.
But it doesn't take much maintenance to break that scheme and the only thing more painful than renaming a Ceph host is re-numbering an OSD.
Yep!
My advice: Do not try to manually number your OSDs.
This. I’ve been there myself, but it’s truly a sisyphean goal.
As compared to Nautilus, at least the Reef Dashboard has a 'Physical
Devices' page and a Devices tab on the Hosts page that should make it easier to know which OSD is on which host. That plus 'ipmitool chassis identify' and other such tools should make it easy to track down the correct box and drive bay. However, it would be nice to have something like 'ceph osd location {id}' from the command line. If such exists, I haven't seen it.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 10/28/24 17:41, Dave Hall wrote:
However, it would be nice to have something like 'ceph osd location {id}' from the command line. If such exists, I haven't seen it.
"ceph osd metadata {id} | jq -r .hostname" will give you the hostname Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Or: ceph osd find {ID} :-) Zitat von Robert Sander <r.sander@heinlein-support.de>:
On 10/28/24 17:41, Dave Hall wrote:
However, it would be nice to have something like 'ceph osd location {id}' from the command line. If such exists, I haven't seen it.
"ceph osd metadata {id} | jq -r .hostname" will give you the hostname
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Well sure, if you want to do it the EASY way :rolleyes:
On Oct 28, 2024, at 1:02 PM, Eugen Block <eblock@nde.ag> wrote:
Or:
ceph osd find {ID}
:-)
Zitat von Robert Sander <r.sander@heinlein-support.de>:
On 10/28/24 17:41, Dave Hall wrote:
However, it would be nice to have something like 'ceph osd location {id}' from the command line. If such exists, I haven't seen it.
"ceph osd metadata {id} | jq -r .hostname" will give you the hostname
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Anthony D'Atri
-
Anthony D'Atri
-
Dave Hall
-
Eugen Block
-
Robert Sander
-
Shain Miley
-
Tim Holloway