Does the number of PGs affect the total usable size of a pool?
Hello guys, Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single pool that consumes all OSDs of all nodes. After adding another host, I noticed that no extra space was added. Can this be a result of the number of PGs I am using? I mean, when adding more hosts/OSDs, should I always consider increasing the number of PGs from a pool?
Hello guys, Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single pool that consumes all OSDs of all nodes. After adding another host, I noticed that no extra space was added. Can this be a result of the number of PGs I am using?
I mean, when adding more hosts/OSDs, should I always consider increasing the number of PGs from a pool?
ceph osd tree shows all up and with correct weight?
Thanks for the prompt reply. Yes, it does. All of them are up, with the correct class that is used by the crush algorithm. On Thu, Feb 13, 2025 at 7:47 AM Marc <Marc@f1-outsourcing.eu> wrote:
Hello guys, Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single pool that consumes all OSDs of all nodes. After adding another host, I noticed that no extra space was added. Can this be a result of the number of PGs I am using?
I mean, when adding more hosts/OSDs, should I always consider increasing the number of PGs from a pool?
ceph osd tree
shows all up and with correct weight?
I still think they are not part of the cluster somehow. "ceph osd status" shows most likely they are not used. When you add just 1 osd you should see something in your cluster capacity and some rebalancing. Status of ceph is HEALTH_OK?
Thanks for the prompt reply.
Yes, it does. All of them are up, with the correct class that is used by the crush algorithm.
On Thu, Feb 13, 2025 at 7:47 AM Marc <Marc@f1-outsourcing.eu <mailto:Marc@f1-outsourcing.eu> > wrote:
Hello guys, Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single pool that consumes all OSDs of all nodes. After adding another host, I noticed that no extra space was added. Can this be a result of the number of PGs I am using?
I mean, when adding more hosts/OSDs, should I always consider increasing the number of PGs from a pool?
ceph osd tree
shows all up and with correct weight?
Thanks for the feedback! Yes, the Heath_ok is there.] The OSD status show all of them as "exists,up". The interesting part is that "ceph df" shows the correct values in the "RAW STORAGE" section. However, for the SSD pool I have, it shows only the previous value as the max usable value. I had 384 TiB as the RAW space before. The SSD pool is a replicated pool with replica as 3. Therefore, I had about 128TiB as possible usable space for the pool before. Now that I added a new node, I would expect 480 RAW space, which is what I have in the RAW STORAGE section, but the usable space to be used in the pool has not changed. I would expect the usable space to grow at about 160TiB. I know that these limits will never be reached as we have locks in 85%-90% for each OSD. On Thu, Feb 13, 2025 at 8:41 AM Marc <Marc@f1-outsourcing.eu> wrote:
I still think they are not part of the cluster somehow. "ceph osd status" shows most likely they are not used. When you add just 1 osd you should see something in your cluster capacity and some rebalancing. Status of ceph is HEALTH_OK?
Thanks for the prompt reply.
Yes, it does. All of them are up, with the correct class that is used by the crush algorithm.
On Thu, Feb 13, 2025 at 7:47 AM Marc <Marc@f1-outsourcing.eu <mailto:Marc@f1-outsourcing.eu> > wrote:
> Hello guys, > Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single > pool that consumes all OSDs of all nodes. After adding another host, I > noticed that no extra space was added. Can this be a result of the > number > of PGs I am using? > > I mean, when adding more hosts/OSDs, should I always consider increasing > the number of PGs from a pool? >
ceph osd tree
shows all up and with correct weight?
Does the new host show up under the proper CRUSH bucket? Do its OSDs? Send `ceph osd tree` please.
Hello guys, Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single pool that consumes all OSDs of all nodes. After adding another host, I noticed that no extra space was added. Can this be a result of the number of PGs I am using?
I mean, when adding more hosts/OSDs, should I always consider increasing the number of PGs from a pool?
ceph osd tree
shows all up and with correct weight?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den tors 13 feb. 2025 kl 12:54 skrev Work Ceph <work.ceph.user.mailing@gmail.com>:
Thanks for the feedback! Yes, the Heath_ok is there.] The OSD status show all of them as "exists,up".
The interesting part is that "ceph df" shows the correct values in the "RAW STORAGE" section. However, for the SSD pool I have, it shows only the previous value as the max usable value. I had 384 TiB as the RAW space before. The SSD pool is a replicated pool with replica as 3. Therefore, I had about 128TiB as possible usable space for the pool before. Now that I added a new node, I would expect 480 RAW space, which is what I have in the RAW STORAGE section, but the usable space to be used in the pool has not changed. I would expect the usable space to grow at about 160TiB. I know that these limits will never be reached as we have locks in 85%-90% for each OSD.
Has all PGs moved yet? If not, then you have to wait until the old OSDs have moved PGs over the the newly added ones. -- May the most significant bit of your life be positive.
Yes, everything has finished converging already. On Thu, Feb 13, 2025 at 12:33 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den tors 13 feb. 2025 kl 12:54 skrev Work Ceph <work.ceph.user.mailing@gmail.com>:
Thanks for the feedback! Yes, the Heath_ok is there.] The OSD status show all of them as "exists,up".
The interesting part is that "ceph df" shows the correct values in the "RAW STORAGE" section. However, for the SSD pool I have, it shows only the previous value as the max usable value. I had 384 TiB as the RAW space before. The SSD pool is a replicated pool with replica as 3. Therefore, I had about 128TiB as possible usable space for the pool before. Now that I added a new node, I would expect 480 RAW space, which is what I have in the RAW STORAGE section, but the usable space to be used in the pool has not changed. I would expect the usable space to grow at about 160TiB. I know that these limits will never be reached as we have locks in 85%-90% for each OSD.
Has all PGs moved yet? If not, then you have to wait until the old OSDs have moved PGs over the the newly added ones.
-- May the most significant bit of your life be positive.
Yes, the bucket that represents the new host is under the ROOT bucket as the others. Also, the OSDs are in the right/expected bucket. I am guessing that the problem is the number of PGs. I have 120 OSDs across all hosts, and I guess that 512 PGS, which is what the pool is using, is not enough. I did not change it yet, because I wanted to understand the effect on PG number in Ceph pool usable volume. On Thu, Feb 13, 2025 at 12:03 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Does the new host show up under the proper CRUSH bucket? Do its OSDs? Send `ceph osd tree` please.
> Hello guys, > Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single > pool that consumes all OSDs of all nodes. After adding another host, I > noticed that no extra space was added. Can this be a result of the > number > of PGs I am using? > > I mean, when adding more hosts/OSDs, should I always consider increasing > the number of PGs from a pool? >
ceph osd tree
shows all up and with correct weight?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Assuming that the pool is replicated, 512 PGs is pretty low if this is the only substantial pool on the cluster. When you do `ceph osd df`, if this is the only substantial pool, the PGS column at right would average around 12 or 13 which is suuuuper low.
On Feb 13, 2025, at 11:40 AM, Work Ceph <work.ceph.user.mailing@gmail.com> wrote:
Yes, the bucket that represents the new host is under the ROOT bucket as the others. Also, the OSDs are in the right/expected bucket.
I am guessing that the problem is the number of PGs. I have 120 OSDs across all hosts, and I guess that 512 PGS, which is what the pool is using, is not enough. I did not change it yet, because I wanted to understand the effect on PG number in Ceph pool usable volume.
On Thu, Feb 13, 2025 at 12:03 PM Anthony D'Atri <anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> wrote:
Does the new host show up under the proper CRUSH bucket? Do its OSDs? Send `ceph osd tree` please.
> Hello guys, > Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single > pool that consumes all OSDs of all nodes. After adding another host, I > noticed that no extra space was added. Can this be a result of the > number > of PGs I am using? > > I mean, when adding more hosts/OSDs, should I always consider increasing > the number of PGs from a pool? >
ceph osd tree
shows all up and with correct weight?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Exactly, that is what I am assuming. However, my question is. Can I assume that the PG number will affect the Max available space that a pool will be able to use? On Thu, Feb 13, 2025 at 3:09 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Assuming that the pool is replicated, 512 PGs is pretty low if this is the only substantial pool on the cluster. When you do `ceph osd df`, if this is the only substantial pool, the PGS column at right would average around 12 or 13 which is suuuuper low.
On Feb 13, 2025, at 11:40 AM, Work Ceph <work.ceph.user.mailing@gmail.com> wrote:
Yes, the bucket that represents the new host is under the ROOT bucket as the others. Also, the OSDs are in the right/expected bucket.
I am guessing that the problem is the number of PGs. I have 120 OSDs across all hosts, and I guess that 512 PGS, which is what the pool is using, is not enough. I did not change it yet, because I wanted to understand the effect on PG number in Ceph pool usable volume.
On Thu, Feb 13, 2025 at 12:03 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Does the new host show up under the proper CRUSH bucket? Do its OSDs? Send `ceph osd tree` please.
> Hello guys, > Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single > pool that consumes all OSDs of all nodes. After adding another host, I > noticed that no extra space was added. Can this be a result of the > number > of PGs I am using? > > I mean, when adding more hosts/OSDs, should I always consider increasing > the number of PGs from a pool? >
ceph osd tree
shows all up and with correct weight?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think that would only happen if pg_num for a 3R pool were less than like ~ 1/3 the number of OSDs. Assuming aligned device classes, proper CRUSH rules and topology etc. Mind you if pg_num is low, the balancer won’t be able to do a great job of distributing data uniformly. If you set pg_num to a non power of two that complicates things as well since you’d have PGs of very different sizes, but this is something rarely seen. Sharing `ceph osd tree` and `ceph osd dump | grep pool` and `ceph osd df` would help.
On Feb 13, 2025, at 1:11 PM, Work Ceph <work.ceph.user.mailing@gmail.com> wrote:
Exactly, that is what I am assuming. However, my question is. Can I assume that the PG number will affect the Max available space that a pool will be able to use?
On Thu, Feb 13, 2025 at 3:09 PM Anthony D'Atri <anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> wrote:
Assuming that the pool is replicated, 512 PGs is pretty low if this is the only substantial pool on the cluster. When you do `ceph osd df`, if this is the only substantial pool, the PGS column at right would average around 12 or 13 which is suuuuper low.
On Feb 13, 2025, at 11:40 AM, Work Ceph <work.ceph.user.mailing@gmail.com <mailto:work.ceph.user.mailing@gmail.com>> wrote:
Yes, the bucket that represents the new host is under the ROOT bucket as the others. Also, the OSDs are in the right/expected bucket.
I am guessing that the problem is the number of PGs. I have 120 OSDs across all hosts, and I guess that 512 PGS, which is what the pool is using, is not enough. I did not change it yet, because I wanted to understand the effect on PG number in Ceph pool usable volume.
On Thu, Feb 13, 2025 at 12:03 PM Anthony D'Atri <anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> wrote:
Does the new host show up under the proper CRUSH bucket? Do its OSDs? Send `ceph osd tree` please.
> Hello guys, > Let's say I have a cluster with 4 nodes with 24 SSDs each, and a single > pool that consumes all OSDs of all nodes. After adding another host, I > noticed that no extra space was added. Can this be a result of the > number > of PGs I am using? > > I mean, when adding more hosts/OSDs, should I always consider increasing > the number of PGs from a pool? >
ceph osd tree
shows all up and with correct weight?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
participants (4)
-
Anthony D'Atri
-
Janne Johansson
-
Marc
-
Work Ceph