How to deal with increasing HDD sizes ? 1 OSD for 2 LVM-packed HDDs ?
Hi, Use case: * Ceph cluster with old nodes having 6TB HDDs * Add new node with new 12TB HDDs Is it supported/recommended to pack 2 6TB HDDs handled by 2 old OSDs into 1 12TB LVM disk handled by 1 new OSD ? Regards, Renaud Miel
On 10/18/23 09:25, Renaud Jean Christophe Miel wrote:
Hi,
Use case: * Ceph cluster with old nodes having 6TB HDDs * Add new node with new 12TB HDDs
Is it supported/recommended to pack 2 6TB HDDs handled by 2 old OSDs into 1 12TB LVM disk handled by 1 new OSD ?
The 12 TB HDD will get double the IO than one of the 6 TB HDDs. But it will still only be able to handle about 120 IOPs. This makes the newer larger HDDs a bottleneck when run in the same pool. If you are not planning to decommission the smaller HDDs it is recommended to use the larger ones in a separate pool for performance reasons. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
This is one of many reasons for not using HDDs ;) One nuance that is easy overlooked is the CRUSH weight of failure domains. If, say, you have a failure domain of "rack" with size=3 replicated pools and 3x CRUSH racks, if you add the new, larger OSDs to only one rack, you will not increase the cluster's capacity. If in this scenario you add them as a fourth rack, this is mostly obviated. Another strategy is to add them uniformly to the existing racks. The larger OSDs will get more PGs as Herr Sander touches upon. * The higher IO can be somewhat ameliorated by adjusting primary-affinity values to favor the smaller OSDs being primaries for given PGs. * The larger OSDs will have an increased risk of running into the mon_max_pg_per_osd limit, especially when an OSD or host fails. Ensure that this setting is high enough to avoid this, suggest 500 as a value.
On Oct 18, 2023, at 04:05, Robert Sander <r.sander@heinlein-support.de> wrote:
On 10/18/23 09:25, Renaud Jean Christophe Miel wrote:
Hi, Use case: * Ceph cluster with old nodes having 6TB HDDs * Add new node with new 12TB HDDs Is it supported/recommended to pack 2 6TB HDDs handled by 2 old OSDs into 1 12TB LVM disk handled by 1 new OSD ?
The 12 TB HDD will get double the IO than one of the 6 TB HDDs. But it will still only be able to handle about 120 IOPs. This makes the newer larger HDDs a bottleneck when run in the same pool.
If you are not planning to decommission the smaller HDDs it is recommended to use the larger ones in a separate pool for performance reasons.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you for your feedback. We have a failure domain of "node". The question here is a rather simple one: when you add to an existing Ceph cluster a new node having disks twice (12TB) the size of the existing disks (6TB), how do you let Ceph evenly distribute the data across all disks ? You mentioned CRUSH: would the creation of a new "12TB-virtual-disk" CRUSH hierarchy level do the trick ? In this level you would either pick 1 12TB HDD on the new node, or a pair of 2 6TB HDDs on an old node. Has someone already experimented with such kind of CRUSH hierarchy ? Regards, Renaud Miel NAOJ ________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Thursday, October 19, 2023 00:59 To: Robert Sander <r.sander@heinlein-support.de> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: How to deal with increasing HDD sizes ? 1 OSD for 2 LVM-packed HDDs ? This is one of many reasons for not using HDDs ;) One nuance that is easy overlooked is the CRUSH weight of failure domains. If, say, you have a failure domain of "rack" with size=3 replicated pools and 3x CRUSH racks, if you add the new, larger OSDs to only one rack, you will not increase the cluster's capacity. If in this scenario you add them as a fourth rack, this is mostly obviated. Another strategy is to add them uniformly to the existing racks. The larger OSDs will get more PGs as Herr Sander touches upon. * The higher IO can be somewhat ameliorated by adjusting primary-affinity values to favor the smaller OSDs being primaries for given PGs. * The larger OSDs will have an increased risk of running into the mon_max_pg_per_osd limit, especially when an OSD or host fails. Ensure that this setting is high enough to avoid this, suggest 500 as a value.
On Oct 18, 2023, at 04:05, Robert Sander <r.sander@heinlein-support.de> wrote:
On 10/18/23 09:25, Renaud Jean Christophe Miel wrote:
Hi, Use case: * Ceph cluster with old nodes having 6TB HDDs * Add new node with new 12TB HDDs Is it supported/recommended to pack 2 6TB HDDs handled by 2 old OSDs into 1 12TB LVM disk handled by 1 new OSD ?
The 12 TB HDD will get double the IO than one of the 6 TB HDDs. But it will still only be able to handle about 120 IOPs. This makes the newer larger HDDs a bottleneck when run in the same pool.
If you are not planning to decommission the smaller HDDs it is recommended to use the larger ones in a separate pool for performance reasons.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The question here is a rather simple one: when you add to an existing Ceph cluster a new node having disks twice (12TB) the size of the existing disks (6TB), how do you let Ceph evenly distribute the data across all disks ?
Ceph already does this. If you have 6TB+12TB in one node you will see that the 12TB will get more data (at least on Nautilus)
* Ceph cluster with old nodes having 6TB HDDs * Add new node with new 12TB HDDs
Halving IOPS-per-TB? https://www.sabi.co.uk/blog/17-one.html?170610#170610 https://www.sabi.co.uk/blog/15-one.html?150329#150329
Is it supported/recommended to pack 2 6TB HDDs handled by 2 old OSDs into 1 12TB LVM disk handled by 1 new OSD ?
The OSDs are just random daemons, what matters to chunk distribution in Ceph is buckets, and in this case leaf buckets. So it all depends on the CRUSH map but I suspect that manipulating it so that two existing leaf buckets become one is not possible or too tricky to attempt. One option would be to divide the 12TB disk in 2 partitions/LVs of 6TB and run 2 OSDs against it. It is not recommended, but I don't see a big issue in this case other than IOPS-per-TB.
participants (5)
-
Anthony D'Atri
-
Marc
-
pg＠ceph.list.sabi.co.UK
-
Renaud Jean Christophe Miel
-
Robert Sander