11 Sep
2020
11 Sep
'20
5:41 p.m.
Hello,
I have been wondering for quite some time whether or not it is possible to influence the
osd.id numbers that are assigned during an install.
I have made an attempt to keep our osds in order over the last few years, but it is a
losing battle without having some control over the osd assignment.
I am currently using ceph-deploy to handle adding nodes to the cluster.
Thanks in advance,
Shain
Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley(a)npr.org
11 Sep
11 Sep
6:12 p.m.
On 11/09/2020 16:11, Shain Miley wrote:
Hello,
I have been wondering for quite some time whether or not it is possible to influence the
osd.id numbers that are assigned during an install.
I have made an attempt to keep our osds in order over the last few years, but it is a
losing battle without having some control over the osd assignment.
I am currently using ceph-deploy to handle adding nodes to the cluster.
You can reuse osd numbers, but I strongly advice you not to focus on
precise IDs. The reason is that you can have such combination of server
faults, which will swap IDs no matter what.
It's a false sense of beauty to have 'ID of OSD match ID in the name of
the server'.
How to reuse osd nums?
OSD number is used (and should be cleaned if OSD dies) in three places
in Ceph:
1) Crush map: ceph osd crush rm osd.x
2) osd list: ceph osd rm osd.x
3) auth: ceph auth rm osd.x
The last one is often forgoten and is a usual reason for ceph-ansible to
fail on new disk in the server.
12 Sep
12 Sep
12:13 a.m.
Thank you for your answer below.
I'm not looking to reuse them as much as I am trying to control what unused number is
actually used.
For example if I have 20 osds and 2 have failed...when I replace a disk in one server I
don't want it to automatically use the next lowest number for the osd assignment.
I understand what you mean about not focusing on the osd ids...but my ocd is making me ask
the question.
Thanks,
Shain
On 9/11/20, 9:45 AM, "George Shuklin" <george.shuklin(a)gmail.com> wrote:
On 11/09/2020 16:11, Shain Miley wrote:
Hello,
I have been wondering for quite some time whether or not it is possible to influence the
osd.id numbers that are assigned during an install.
I have made an attempt to keep our osds in order over the last few years, but it is a
losing battle without having some control over the osd assignment.
I am currently using ceph-deploy to handle adding nodes to the cluster.
You can reuse osd numbers, but I strongly advice you not to focus on
precise IDs. The reason is that you can have such combination of server
faults, which will swap IDs no matter what.
It's a false sense of beauty to have 'ID of OSD match ID in the name of
the server'.
How to reuse osd nums?
OSD number is used (and should be cleaned if OSD dies) in three places
in Ceph:
1) Crush map: ceph osd crush rm osd.x
2) osd list: ceph osd rm osd.x
3) auth: ceph auth rm osd.x
The last one is often forgoten and is a usual reason for ceph-ansible to
fail on new disk in the server.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:42 a.m.
Now that’s a *very* different question from numbers assigned during an install.
With recent releases instead of going down the full removal litany listed below, you can
instead down/out the OSD and `destroy` it. That preserves the CRUSH bucket and OSD ID,
then when you use ceph-disk, ceph-volume, what-have-you to deploy a replacement, you can
specify the same and desired OSD ID on the commandline.
Note that as of 12.2.2, you’ll want to record and re-set any override reweight (manual or
reweight-by-utilization) as that usually or never survives. Also note that again as of
that release, if the replacement drive is a different size, the CRUSH weight is not
adjusted, so you may (or may not) want to adjust the CRUSH weight. Slight differences
aren’t usually a huge deal; big differences can mean you have unused capacity, or
overloaded drives.
— Anthony
Thank you for your answer below.
I'm not looking to reuse them as much as I am trying to control what unused number is
actually used.
For example if I have 20 osds and 2 have failed...when I replace a disk in one server I
don't want it to automatically use the next lowest number for the osd assignment.
I understand what you mean about not focusing on the osd ids...but my ocd is making me
ask the question.
Thanks,
Shain
On 9/11/20, 9:45 AM, "George Shuklin" <george.shuklin(a)gmail.com> wrote:
On 11/09/2020 16:11, Shain Miley wrote:
Hello,
I have been wondering for quite some time whether or not it is possible to influence the
osd.id numbers that are assigned during an install.
I have made an attempt to keep our osds in order over the last few years, but it is a
losing battle without having some control over the osd assignment.
I am currently using ceph-deploy to handle adding nodes to the cluster.
You can reuse osd numbers, but I strongly advice you not to focus on
precise IDs. The reason is that you can have such combination of server
faults, which will swap IDs no matter what.
It's a false sense of beauty to have 'ID of OSD match ID in the name of
the server'.
How to reuse osd nums?
OSD number is used (and should be cleaned if OSD dies) in three places
in Ceph:
1) Crush map: ceph osd crush rm osd.x
2) osd list: ceph osd rm osd.x
3) auth: ceph auth rm osd.x
The last one is often forgoten and is a usual reason for ceph-ansible to
fail on new disk in the server.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
14 Sep
14 Sep
2 p.m.
On 11/09/2020 22:43, Shain Miley wrote:
Thank you for your answer below.
I'm not looking to reuse them as much as I am trying to control what unused number is
actually used.
For example if I have 20 osds and 2 have failed...when I replace a disk in one server I
don't want it to automatically use the next lowest number for the osd assignment.
I understand what you mean about not focusing on the osd ids...but my ocd is making me
ask the question.
Well, technically, you can create fake OSD to hold numbers and release
those 'fake OSD' if you need to use their numbers, but you are really
complicate everything. I suggest you to stop worrying about numbers. If
you are ok that every OSD on every sever is using /dev/sdb (OCD requires
that server1 uses /dev/sda, server2 uses /dev/sdb, server 3 /dev/sdc,
etc), so you should be fine with random OSD numbers. Moreover, you
should be fine with discrepancy on sorting order of OSD uuid and their
numbers, misalignment of IP adddress and OSD тumber (192.168.0.4 for
OSD.1 ????).
While it's may be fun to play with numbers in a lab, if you are using
Ceph in production, you should avoid doing unnecessary changes, as they
will surprise other people (and you!) trying to keep this thing running.
1344
days inactive
1347
days old
4 comments
3 participants
participants (3)
-
Anthony D'Atri -
George Shuklin -
Shain Miley