Hello, I have been wondering for quite some time whether or not it is possible to influence the osd.id numbers that are assigned during an install. I have made an attempt to keep our osds in order over the last few years, but it is a losing battle without having some control over the osd assignment. I am currently using ceph-deploy to handle adding nodes to the cluster. Thanks in advance, Shain Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley@npr.org
On 11/09/2020 16:11, Shain Miley wrote:
Hello, I have been wondering for quite some time whether or not it is possible to influence the osd.id numbers that are assigned during an install.
I have made an attempt to keep our osds in order over the last few years, but it is a losing battle without having some control over the osd assignment.
I am currently using ceph-deploy to handle adding nodes to the cluster.
You can reuse osd numbers, but I strongly advice you not to focus on precise IDs. The reason is that you can have such combination of server faults, which will swap IDs no matter what. It's a false sense of beauty to have 'ID of OSD match ID in the name of the server'. How to reuse osd nums? OSD number is used (and should be cleaned if OSD dies) in three places in Ceph: 1) Crush map: ceph osd crush rm osd.x 2) osd list: ceph osd rm osd.x 3) auth: ceph auth rm osd.x The last one is often forgoten and is a usual reason for ceph-ansible to fail on new disk in the server.
Thank you for your answer below. I'm not looking to reuse them as much as I am trying to control what unused number is actually used. For example if I have 20 osds and 2 have failed...when I replace a disk in one server I don't want it to automatically use the next lowest number for the osd assignment. I understand what you mean about not focusing on the osd ids...but my ocd is making me ask the question. Thanks, Shain On 9/11/20, 9:45 AM, "George Shuklin" <george.shuklin@gmail.com> wrote: On 11/09/2020 16:11, Shain Miley wrote: > Hello, > I have been wondering for quite some time whether or not it is possible to influence the osd.id numbers that are assigned during an install. > > I have made an attempt to keep our osds in order over the last few years, but it is a losing battle without having some control over the osd assignment. > > I am currently using ceph-deploy to handle adding nodes to the cluster. > You can reuse osd numbers, but I strongly advice you not to focus on precise IDs. The reason is that you can have such combination of server faults, which will swap IDs no matter what. It's a false sense of beauty to have 'ID of OSD match ID in the name of the server'. How to reuse osd nums? OSD number is used (and should be cleaned if OSD dies) in three places in Ceph: 1) Crush map: ceph osd crush rm osd.x 2) osd list: ceph osd rm osd.x 3) auth: ceph auth rm osd.x The last one is often forgoten and is a usual reason for ceph-ansible to fail on new disk in the server. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Now that’s a *very* different question from numbers assigned during an install. With recent releases instead of going down the full removal litany listed below, you can instead down/out the OSD and `destroy` it. That preserves the CRUSH bucket and OSD ID, then when you use ceph-disk, ceph-volume, what-have-you to deploy a replacement, you can specify the same and desired OSD ID on the commandline. Note that as of 12.2.2, you’ll want to record and re-set any override reweight (manual or reweight-by-utilization) as that usually or never survives. Also note that again as of that release, if the replacement drive is a different size, the CRUSH weight is not adjusted, so you may (or may not) want to adjust the CRUSH weight. Slight differences aren’t usually a huge deal; big differences can mean you have unused capacity, or overloaded drives. — Anthony
Thank you for your answer below.
I'm not looking to reuse them as much as I am trying to control what unused number is actually used.
For example if I have 20 osds and 2 have failed...when I replace a disk in one server I don't want it to automatically use the next lowest number for the osd assignment.
I understand what you mean about not focusing on the osd ids...but my ocd is making me ask the question.
Thanks, Shain
On 9/11/20, 9:45 AM, "George Shuklin" <george.shuklin@gmail.com> wrote:
On 11/09/2020 16:11, Shain Miley wrote:
Hello, I have been wondering for quite some time whether or not it is possible to influence the osd.id numbers that are assigned during an install.
I have made an attempt to keep our osds in order over the last few years, but it is a losing battle without having some control over the osd assignment.
I am currently using ceph-deploy to handle adding nodes to the cluster.
You can reuse osd numbers, but I strongly advice you not to focus on precise IDs. The reason is that you can have such combination of server faults, which will swap IDs no matter what.
It's a false sense of beauty to have 'ID of OSD match ID in the name of the server'.
How to reuse osd nums?
OSD number is used (and should be cleaned if OSD dies) in three places in Ceph:
1) Crush map: ceph osd crush rm osd.x
2) osd list: ceph osd rm osd.x
3) auth: ceph auth rm osd.x
The last one is often forgoten and is a usual reason for ceph-ansible to fail on new disk in the server. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 11/09/2020 22:43, Shain Miley wrote:
Thank you for your answer below.
I'm not looking to reuse them as much as I am trying to control what unused number is actually used.
For example if I have 20 osds and 2 have failed...when I replace a disk in one server I don't want it to automatically use the next lowest number for the osd assignment.
I understand what you mean about not focusing on the osd ids...but my ocd is making me ask the question.
Well, technically, you can create fake OSD to hold numbers and release those 'fake OSD' if you need to use their numbers, but you are really complicate everything. I suggest you to stop worrying about numbers. If you are ok that every OSD on every sever is using /dev/sdb (OCD requires that server1 uses /dev/sda, server2 uses /dev/sdb, server 3 /dev/sdc, etc), so you should be fine with random OSD numbers. Moreover, you should be fine with discrepancy on sorting order of OSD uuid and their numbers, misalignment of IP adddress and OSD тumber (192.168.0.4 for OSD.1 ????). While it's may be fun to play with numbers in a lab, if you are using Ceph in production, you should avoid doing unnecessary changes, as they will surprise other people (and you!) trying to keep this thing running.
participants (3)
-
Anthony D'Atri
-
George Shuklin
-
Shain Miley