What is the problem with many PGs per OSD
Hi all, during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing. As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing. Yet, I'm holding back on an increase of PG count due to these rumors. My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450. I heard in related rumors that some users have 1000+ PGs per OSD without problems. I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do. I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object" or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors. An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500? Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense: HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400 These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario: An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, half a year ago I asked a related question (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/I3TQC42KN2FC...), when we needed to split huge PGs on a customer cluster. I wasn't sure either how far we could go with the ratio PGs per OSD. We increased the pg_num to the target value (4096) before new hardware arrived, temporarily the old OSDs (240 * 8 TB) had around 300 PGs/OSD, it wasn't well balanced yet. The new OSDs are larger drives (12 TB), but having the same capacity per node, and after all remapping finished and the balancer did its job, they're now at around 250 PGs/OSD for the smaller drives, 350 PGs/OSD on the larger drives. All OSDs are spinners with rocksDB on SSDs. It seems to work well, no complaints yet, but note that it's an archive cluster, so the performance requirements aren't very high. It's more about resiliency and availibilty in this case. This is all I can contribute to your question. Zitat von Anthony D'Atri <aad@dreamsnake.net>:
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense:
HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400
These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario:
An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, thanks for looking at this. I followed the thread you refer to and it doesn't answer my question. Unfortunately, the statement
... It seems to work well, no complaints yet, but note that it's an archive cluster, so the performance requirements aren't very high. ...
is reproducing the rumor that many PGs somehow impact performance in a negative way. What is this based on? As I wrote, since the number of PGs per OSD times the objects per PG = objects per OSD is a constant, I don't see an immediate justification for the assumption that more PGs imply less performance? What do you base that on? I don't see algorithms at work here for which splitting PGs could impact performance noticeably in a bad way. On the contrary, my experience with pools with the highest PG/OSD count rather says that reducing the number of objects per PG by splitting PGs speeds everything up. Yet, the programmers set a quite low limit without really explaining why. The docs just state a rumor without any solid information a sysadmin/user could use to decide whether or not its worth going high. This is of really high interest, because there is probably a critical value for which any drawbacks (if they actually exist) might outweigh the benefits and without some solid information based on what algorithms do the main work and what complexity class they have its impossible to make an informed decision or diagnose if this happened. Do you have performance metrics before/after? Did you actually observe any performance degradation? Was there an increased memory consumption? Anything that justifies making a statement alluding to (potential) negative performance impact? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 9, 2024 9:24 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: What is the problem with many PGs per OSD Hi, half a year ago I asked a related question (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/I3TQC42KN2FC...), when we needed to split huge PGs on a customer cluster. I wasn't sure either how far we could go with the ratio PGs per OSD. We increased the pg_num to the target value (4096) before new hardware arrived, temporarily the old OSDs (240 * 8 TB) had around 300 PGs/OSD, it wasn't well balanced yet. The new OSDs are larger drives (12 TB), but having the same capacity per node, and after all remapping finished and the balancer did its job, they're now at around 250 PGs/OSD for the smaller drives, 350 PGs/OSD on the larger drives. All OSDs are spinners with rocksDB on SSDs. It seems to work well, no complaints yet, but note that it's an archive cluster, so the performance requirements aren't very high. It's more about resiliency and availibilty in this case. This is all I can contribute to your question. Zitat von Anthony D'Atri <aad@dreamsnake.net>:
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense:
HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400
These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario:
An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I know it doesn't answer your question, I just wanted to point out that I'd be interested to know as well which impacts such configurations can have. :-) More comments inline. Zitat von Frank Schilder <frans@dtu.dk>:
Hi Eugen,
thanks for looking at this. I followed the thread you refer to and it doesn't answer my question. Unfortunately, the statement
... It seems to work well, no complaints yet, but note that it's an archive cluster, so the performance requirements aren't very high. ...
is reproducing the rumor that many PGs somehow impact performance in a negative way. What is this based on? As I wrote, since the number of PGs per OSD times the objects per PG = objects per OSD is a constant, I don't see an immediate justification for the assumption that more PGs imply less performance? What do you base that on? I don't see algorithms at work here for which splitting PGs could impact performance noticeably in a bad way.
I just assume that if the PG count reaches a certain number, the increased amount of parallel requests could overload an OSD. But I have no real proof for that assumption. I tend to be quite hesitant on customer clusters to "play around" and rather stick to the defaults as close as possible.
On the contrary, my experience with pools with the highest PG/OSD count rather says that reducing the number of objects per PG by splitting PGs speeds everything up. Yet, the programmers set a quite low limit without really explaining why. The docs just state a rumor without any solid information a sysadmin/user could use to decide whether or not its worth going high. This is of really high interest, because there is probably a critical value for which any drawbacks (if they actually exist) might outweigh the benefits and without some solid information based on what algorithms do the main work and what complexity class they have its impossible to make an informed decision or diagnose if this happened.
I second that, we also have usually benefitted from PG splits on each cluster we maintain. But at the same time we tried to avoid getting above the recommendations, as already stated. Many default values don't match real-world deployments, I've learned that a lot in the recent years both in Ceph and OpenStack. Maybe those recommendations are a bit outdated, but I'd like to learn as well how far one could go and which impacts are expected. Unfortunately, I only have a couple of virtual test clusters, I'd love to have a hardware test cluster to play with. :-D
Do you have performance metrics before/after? Did you actually observe any performance degradation? Was there an increased memory consumption? Anything that justifies making a statement alluding to (potential) negative performance impact?
Unfortunately, I don't have access to the cluster or metrics. And the retention time of their Prometheus instance is not very long, so no, I don't have anything to show. I can ask them if they did monitor that by any chance, but I'm not very confident that they did. :-/
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 9, 2024 9:24 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: What is the problem with many PGs per OSD
Hi,
half a year ago I asked a related question (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/I3TQC42KN2FC...), when we needed to split huge PGs on a customer cluster. I wasn't sure either how far we could go with the ratio PGs per OSD. We increased the pg_num to the target value (4096) before new hardware arrived, temporarily the old OSDs (240 * 8 TB) had around 300 PGs/OSD, it wasn't well balanced yet. The new OSDs are larger drives (12 TB), but having the same capacity per node, and after all remapping finished and the balancer did its job, they're now at around 250 PGs/OSD for the smaller drives, 350 PGs/OSD on the larger drives. All OSDs are spinners with rocksDB on SSDs. It seems to work well, no complaints yet, but note that it's an archive cluster, so the performance requirements aren't very high. It's more about resiliency and availibilty in this case.
This is all I can contribute to your question.
Zitat von Anthony D'Atri <aad@dreamsnake.net>:
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense:
HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400
These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario:
An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Thanks for chiming in. Unfortunately, it doesn't really help answering my questions either. Concurrency: A system like ceph that hashes data into PGs translates any IO into random IO anyways. So it's irrelevant for spinners, they have to seek anyways and the degree of parallelism doesn't matter on systems with sufficient load. In addition, for OSDs at least up to pacific the kv_sync_thread serializes everything (writes only?) anyways, so whatever concurrency more PGs add, this thread puts it back in sequence. That's why deploying multiple OSDs per SSD is such a great way to improve performance on devices where 4K random IO throughput scales with iodepth. You get one kv_sync_thread per OSD instance (and the extra PG count on top of that) and benefit from the iodepth scaling. Memory: I have never used file store, so can't relate to that. For our bluestore OSDs I have never observed any change in memory consumption after PG splitting (and not even during). I have never seen memory ballooning during peering either. That's why I would like to know if there is indeed a fixed allocation per PG and how much that would be. Without actual numbers its rumors and anecdotes. 9000 PGs/OSD was too much for what kind of system? What CPU? How much RAM? How many OSDs per host? Did it even work with 200PGs with the same data (recovery after power loss)? Was it maybe the death spiral (https://ceph-users.ceph.narkive.com/KAzvjjPc/explanation-for-ceph-osd-set-no...) that prevented the cluster from coming up and not so much the PG count? Rumors: Yes, 1000 PGs/OSD on spinners without issues. I guess we are not talking about barely working home systems with lack of all sorts of resources here. The goal: Let's say I want to go 500-1000PGs/OSD on 16T spinners to trim PGs to about 10-20G each. What are the resources that count will require compared with, say, 200 PGs/OSD? That's the interesting question and if I can make the resources available I would consider doing that. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <aad@dreamsnake.net> Sent: Wednesday, October 9, 2024 2:40 AM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense: HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400 These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario: An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for chiming in. Unfortunately, it doesn't really help answering my questions either.
Concurrency: A system like ceph that hashes data into PGs translates any IO into random IO anyways. So it's irrelevant for spinners, they have to seek anyways and the degree of parallelism doesn't matter on systems with sufficient load. In addition, for OSDs at least up to pacific the kv_sync_thread serializes everything (writes only?) anyways, so whatever concurrency more PGs add, this thread puts it back in sequence.
I could imagine that each PG had a certain amount of meta-operations, like logging, database vacuuming or reindexing and so on that happens at some intervals regardless of if you access objects or not. In that case, the PG meta ops would scale with the number of PGs in the OSD but as you state above not with the number of objects, which of course stays more or less the same. If this was true, then going from 100 to 1000 PGs would make these ops upto 10x more while object IO would stay the same. -- May the most significant bit of your life be positive.
Hi Janne, thanks for looking at this. I'm afraid I have to flag this as rumor as well, you are basically stating it yourself:
I could imagine that each PG had a certain amount of meta-operations, ...
So, yes, maybe. But for sure? Why would this not be proportional to IO operations? To object count? My own imagination is not good enough to see that PGs just do stuff for fun. A lot of code in ceph triggers book keeping and cleanup together with client ops, so its a constant overhead per client OP. Also operations like data base vacuuming are usually at least N log N with N the size of the DB. Reducing the PG size should also reduce the PG data base size, which would result in M*(N/M log N/M) after splitting by a factor of M, and we have an actual improvement since M*(N/M log N/M) = N log N/M < N log N It is a good idea to collect such hypotheses, assuming that a dev drops by and can comment on that with background from the implementation. I just won't be satisfied with speculation this time around and will keep bugging. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: Wednesday, October 9, 2024 11:20 AM To: Frank Schilder Cc: Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: What is the problem with many PGs per OSD
Thanks for chiming in. Unfortunately, it doesn't really help answering my questions either.
Concurrency: A system like ceph that hashes data into PGs translates any IO into random IO anyways. So it's irrelevant for spinners, they have to seek anyways and the degree of parallelism doesn't matter on systems with sufficient load. In addition, for OSDs at least up to pacific the kv_sync_thread serializes everything (writes only?) anyways, so whatever concurrency more PGs add, this thread puts it back in sequence.
I could imagine that each PG had a certain amount of meta-operations, like logging, database vacuuming or reindexing and so on that happens at some intervals regardless of if you access objects or not. In that case, the PG meta ops would scale with the number of PGs in the OSD but as you state above not with the number of objects, which of course stays more or less the same. If this was true, then going from 100 to 1000 PGs would make these ops upto 10x more while object IO would stay the same. -- May the most significant bit of your life be positive.
Den ons 9 okt. 2024 kl 11:34 skrev Frank Schilder <frans@dtu.dk>:
Hi Janne, thanks for looking at this. I'm afraid I have to flag this as rumor as well, you are basically stating it yourself: It is a good idea to collect such hypotheses, assuming that a dev drops by and can comment on that with background from the implementation. I just won't be satisfied with speculation this time around and will keep bugging.
Yes, I was deliberately stating it so that someone who really knows can say if it was totally true or totally false. -- May the most significant bit of your life be positive.
Unfortunately, it doesn't really help answering my questions either.
Sometimes the best we can do is grunt and shrug :-/. Before Nautilus we couldn’t merge PGs, so we could raise pg_num for a pool but not decrease it, so a certain fear of overshooting was established. Mark is the go-to here.
That's why deploying multiple OSDs per SSD is such a great way to improve performance on devices where 4K random IO throughput scales with iodepth.
Mark’s testing have shown this to not be so much the case with recent releases — do you still see this? Until recently I was expecting 30TB TLC SSDs for RBD, and in the next year perhaps as large as 122T for object so I was thinking of splitting just because of the size - and the systems in question were overequipped with CPU.
Memory: I have never used file store, so can't relate to that.
XFS - I experienced a lot of ballooning, to the point of OOMkilling. In mixed clusters under duress the BlueStore OSDs consistently behaved better.
9000 PGs/OSD was too much for what kind of system? What CPU? How much RAM? How many OSDs per host?
Those were Cisco UCS… C240m3. Dual 16c Sandy Bridge IIRC, 10x SATA HDD OSDs @ 3TB, 64GB I think.
Did it even work with 200PGs with the same data (recovery after power loss)?
I didn’t have remote power control, and being a shared lab it was difficult to take a cluster down for such testing. We did have a larger integration cluster (450 OSDs) with a PG ratio of ~~ 200 where we tested a rack power drop. Ceph was fine (this was …. Firefly I think) but the LSI RoC HBAs lost data like crazy due to hardware, firmware, and utility bugs.
Was it maybe the death spiral (https://ceph-users.ceph.narkive.com/KAzvjjPc/explanation-for-ceph-osd-set-no...) that prevented the cluster from coming up and not so much the PG count?
Not in this case, though I’ve seen a similar cascading issue in another context.
Rumors: Yes, 1000 PGs/OSD on spinners without issues. I guess we are not talking about barely working home systems with lack of all sorts of resources here.
I’d be curious how such systems behave under duress. I’ve seen a cluster that had grown - the mons ended up with enough RAM to run but not to boot, so I did urgent RAM upgrades on the mons. That was the mixed Filestore / BlueStore cluster (Luminous 12.2.2) where the Filestore OSDs were much more affected by a cascading event than the [mostly larger] BlueStore OSDs. I suspect that had the whole cluster been BlueStore it might not have cascaded.
The goal: Let's say I want to go 500-1000PGs/OSD on 16T spinners to trim PGs to about 10-20G each. What are the resources that count will require compared with, say, 200 PGs/OSD? That's the interesting question and if I can make the resources available I would consider doing that.
The proof is in the proverbial pudding. Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher.
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <aad@dreamsnake.net> Sent: Wednesday, October 9, 2024 2:40 AM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense:
HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400
These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario:
An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony.
... Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher.
That's not an option. What would help is a-priori information based on the implementation. I'm looking at a pool with 5PB of data and 8192 PGs. If I increase that by, say a factor 4, its in one step, not gradual to avoid excessive redundant data movement. I don't want to spend hardware life for nothing and also don't want to wait for months or more for this to complete or get stuck along the way due to something catastrophic. What I would like to know is is there a fundamental scaling limit in the PG implementation that someone who was staring at the code for a long time knows about. This is usually something that grows much worse than N log N in time- or memory complexity. The answer to this is in the code and boils down to "why the recommendation of 100 PGs per OSD" and not 200 or 1000 or 100 per TB - the latter would make a looooot more sense). There ought to be a reason other than "we didn't know what else to write". I would like to know the scaling in (worst-case) complexity as a function of the number of PGs. Making a fixed recommendation of a specific number independent of anything else is something really weird. It indicates that there is something catastrophic in the code that will blow up once an (unknown/undocumented!!) threshold is crossed. For example, a tiny but important function that is exponential in the number of PGs. If there is nothing catastrophic in the code, then why is the recommendation not floating, specifying what increase in resource consumption one should expect. None of the discussions I have seen so far address this extreme weirdness of the recommendation. If there is an unsolved scaling problem, please anyone state what it is, why its there and what the critical threshold is. What part of the code will explode? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Wednesday, October 9, 2024 3:52 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD
Unfortunately, it doesn't really help answering my questions either.
Sometimes the best we can do is grunt and shrug :-/. Before Nautilus we couldn’t merge PGs, so we could raise pg_num for a pool but not decrease it, so a certain fear of overshooting was established. Mark is the go-to here.
That's why deploying multiple OSDs per SSD is such a great way to improve performance on devices where 4K random IO throughput scales with iodepth.
Mark’s testing have shown this to not be so much the case with recent releases — do you still see this? Until recently I was expecting 30TB TLC SSDs for RBD, and in the next year perhaps as large as 122T for object so I was thinking of splitting just because of the size - and the systems in question were overequipped with CPU.
Memory: I have never used file store, so can't relate to that.
XFS - I experienced a lot of ballooning, to the point of OOMkilling. In mixed clusters under duress the BlueStore OSDs consistently behaved better.
9000 PGs/OSD was too much for what kind of system? What CPU? How much RAM? How many OSDs per host?
Those were Cisco UCS… C240m3. Dual 16c Sandy Bridge IIRC, 10x SATA HDD OSDs @ 3TB, 64GB I think.
Did it even work with 200PGs with the same data (recovery after power loss)?
I didn’t have remote power control, and being a shared lab it was difficult to take a cluster down for such testing. We did have a larger integration cluster (450 OSDs) with a PG ratio of ~~ 200 where we tested a rack power drop. Ceph was fine (this was …. Firefly I think) but the LSI RoC HBAs lost data like crazy due to hardware, firmware, and utility bugs.
Was it maybe the death spiral (https://ceph-users.ceph.narkive.com/KAzvjjPc/explanation-for-ceph-osd-set-no...) that prevented the cluster from coming up and not so much the PG count?
Not in this case, though I’ve seen a similar cascading issue in another context.
Rumors: Yes, 1000 PGs/OSD on spinners without issues. I guess we are not talking about barely working home systems with lack of all sorts of resources here.
I’d be curious how such systems behave under duress. I’ve seen a cluster that had grown - the mons ended up with enough RAM to run but not to boot, so I did urgent RAM upgrades on the mons. That was the mixed Filestore / BlueStore cluster (Luminous 12.2.2) where the Filestore OSDs were much more affected by a cascading event than the [mostly larger] BlueStore OSDs. I suspect that had the whole cluster been BlueStore it might not have cascaded.
The goal: Let's say I want to go 500-1000PGs/OSD on 16T spinners to trim PGs to about 10-20G each. What are the resources that count will require compared with, say, 200 PGs/OSD? That's the interesting question and if I can make the resources available I would consider doing that.
The proof is in the proverbial pudding. Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher.
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <aad@dreamsnake.net> Sent: Wednesday, October 9, 2024 2:40 AM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense:
HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400
These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario:
An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony.
... Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher.
That's not an option. What would help is a-priori information based on the implementation
I think with so many variables in play that would be tough to quantify.
I'm looking at a pool with 5PB of data and 8192 PGs. If I increase that by, say a factor 4, its in one step, not gradual to avoid excessive redundant data movement. I don't want to spend hardware life for nothing and also don't want to wait for months or more for this to complete or get stuck along the way due to something catastrophic.
Used to be that Ceph wouldn’t let you more than double pg_num in one step. You might consider going to just, say, 9216 and see what happens. Non power of 2 pg_num isn’t THAT big a deal these days, you’ll end up some some PGs larger than others, but it’s not horrible for a short term. My sense re hardware life is that writes due to rebalancing are trivial.
What I would like to know is is there a fundamental scaling limit in the PG implementation that someone who was staring at the code for a long time knows about. This is usually something that grows much worse than N log N in time- or memory complexity. The answer to this is in the code and boils down to "why the recommendation of 100 PGs per OSD" and not 200 or 1000 or 100 per TB - the latter would make a looooot more sense). There ought to be a reason other than "we didn't know what else to write".
I would like to know the scaling in (worst-case) complexity as a function of the number of PGs. Making a fixed recommendation of a specific number independent of anything else is something really weird. It indicates that there is something catastrophic in the code that will blow up once an (unknown/undocumented!!) threshold is crossed. For example, a tiny but important function that is exponential in the number of PGs. If there is nothing catastrophic in the code, then why is the recommendation not floating, specifying what increase in resource consumption one should expect.
None of the discussions I have seen so far address this extreme weirdness of the recommendation. If there is an unsolved scaling problem, please anyone state what it is, why its there and what the critical threshold is. What part of the code will explode?
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Wednesday, October 9, 2024 3:52 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD
Unfortunately, it doesn't really help answering my questions either.
Sometimes the best we can do is grunt and shrug :-/. Before Nautilus we couldn’t merge PGs, so we could raise pg_num for a pool but not decrease it, so a certain fear of overshooting was established. Mark is the go-to here.
That's why deploying multiple OSDs per SSD is such a great way to improve performance on devices where 4K random IO throughput scales with iodepth.
Mark’s testing have shown this to not be so much the case with recent releases — do you still see this? Until recently I was expecting 30TB TLC SSDs for RBD, and in the next year perhaps as large as 122T for object so I was thinking of splitting just because of the size - and the systems in question were overequipped with CPU.
Memory: I have never used file store, so can't relate to that.
XFS - I experienced a lot of ballooning, to the point of OOMkilling. In mixed clusters under duress the BlueStore OSDs consistently behaved better.
9000 PGs/OSD was too much for what kind of system? What CPU? How much RAM? How many OSDs per host?
Those were Cisco UCS… C240m3. Dual 16c Sandy Bridge IIRC, 10x SATA HDD OSDs @ 3TB, 64GB I think.
Did it even work with 200PGs with the same data (recovery after power loss)?
I didn’t have remote power control, and being a shared lab it was difficult to take a cluster down for such testing. We did have a larger integration cluster (450 OSDs) with a PG ratio of ~~ 200 where we tested a rack power drop. Ceph was fine (this was …. Firefly I think) but the LSI RoC HBAs lost data like crazy due to hardware, firmware, and utility bugs.
Was it maybe the death spiral (https://ceph-users.ceph.narkive.com/KAzvjjPc/explanation-for-ceph-osd-set-no...) that prevented the cluster from coming up and not so much the PG count?
Not in this case, though I’ve seen a similar cascading issue in another context.
Rumors: Yes, 1000 PGs/OSD on spinners without issues. I guess we are not talking about barely working home systems with lack of all sorts of resources here.
I’d be curious how such systems behave under duress. I’ve seen a cluster that had grown - the mons ended up with enough RAM to run but not to boot, so I did urgent RAM upgrades on the mons. That was the mixed Filestore / BlueStore cluster (Luminous 12.2.2) where the Filestore OSDs were much more affected by a cascading event than the [mostly larger] BlueStore OSDs. I suspect that had the whole cluster been BlueStore it might not have cascaded.
The goal: Let's say I want to go 500-1000PGs/OSD on 16T spinners to trim PGs to about 10-20G each. What are the resources that count will require compared with, say, 200 PGs/OSD? That's the interesting question and if I can make the resources available I would consider doing that.
The proof is in the proverbial pudding. Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher.
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <aad@dreamsnake.net> Sent: Wednesday, October 9, 2024 2:40 AM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense:
HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400
These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario:
An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, replying here to points that were somewhat outside the scope of my original question:
That's why deploying multiple OSDs per SSD is such a great way to improve performance on devices where 4K random IO throughput scales with iodepth.
Mark’s testing have shown this to not be so much the case with recent releases
I use Octopus and there this is very prominent. The kv_sync_thread is a bottleneck at least until pacific. I'm not sure if this is really resolved. As far as I understood the devs were looking into splitting this thread up and decided no to, because it is easier to deploy multiple OSDs per disk. With recent rocksdb format changes (new sharding) this thread might use less resources speeding things up. Its still a synchronisation point for concurrent operations though.
9000 PGs/OSD was too much for what kind of system? What CPU? How much RAM? How many OSDs per host?
Those were Cisco UCS… C240m3. Dual 16c Sandy Bridge IIRC, 10x SATA HDD OSDs @ 3TB, 64GB I think.
And you say the OSDs were going OOM on restart? It might be possible that the PG count played a role. More likely something like the pglog-dup bug though, which had exactly this as its hallmark, insane memory ballooning on OSD startup. The question is, was it really the resource requirements due to PG count or was it something else. That's indeed really a question I would like to have an answer from the devs to: Is there a bug in the code/rocksdb that is more likely triggered for high PG counts and that's why the weird recommendation is made? I consider this recommendation of PG count per OSD weird, because its made independent of anything else that's available: network, CPU, RAM, disk size, disk performance, etc. etc. For literally any other parameter there are tuning guides that explain how to ramp up things, when requirements will peak and how much to have spare to survive peak loads. The PG count per OSD is a striking exception. Its just a number (well a range with 100 recommended and 200 as a max: https://docs.ceph.com/en/latest/rados/operations/pgcalc/#keyDL). It just is. And this doesn't make any sense unless there is something really evil lurking in the dark. For comparison, a guidance that does make sense is something like 100PGs per TB. That I would vaguely understand: to keep the average PG size constant at a max of about 10G. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Wednesday, October 9, 2024 3:52 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD
Unfortunately, it doesn't really help answering my questions either.
Sometimes the best we can do is grunt and shrug :-/. Before Nautilus we couldn’t merge PGs, so we could raise pg_num for a pool but not decrease it, so a certain fear of overshooting was established. Mark is the go-to here.
That's why deploying multiple OSDs per SSD is such a great way to improve performance on devices where 4K random IO throughput scales with iodepth.
Mark’s testing have shown this to not be so much the case with recent releases — do you still see this? Until recently I was expecting 30TB TLC SSDs for RBD, and in the next year perhaps as large as 122T for object so I was thinking of splitting just because of the size - and the systems in question were overequipped with CPU.
Memory: I have never used file store, so can't relate to that.
XFS - I experienced a lot of ballooning, to the point of OOMkilling. In mixed clusters under duress the BlueStore OSDs consistently behaved better.
9000 PGs/OSD was too much for what kind of system? What CPU? How much RAM? How many OSDs per host?
Those were Cisco UCS… C240m3. Dual 16c Sandy Bridge IIRC, 10x SATA HDD OSDs @ 3TB, 64GB I think.
Did it even work with 200PGs with the same data (recovery after power loss)?
I didn’t have remote power control, and being a shared lab it was difficult to take a cluster down for such testing. We did have a larger integration cluster (450 OSDs) with a PG ratio of ~~ 200 where we tested a rack power drop. Ceph was fine (this was …. Firefly I think) but the LSI RoC HBAs lost data like crazy due to hardware, firmware, and utility bugs.
Was it maybe the death spiral (https://ceph-users.ceph.narkive.com/KAzvjjPc/explanation-for-ceph-osd-set-no...) that prevented the cluster from coming up and not so much the PG count?
Not in this case, though I’ve seen a similar cascading issue in another context.
Rumors: Yes, 1000 PGs/OSD on spinners without issues. I guess we are not talking about barely working home systems with lack of all sorts of resources here.
I’d be curious how such systems behave under duress. I’ve seen a cluster that had grown - the mons ended up with enough RAM to run but not to boot, so I did urgent RAM upgrades on the mons. That was the mixed Filestore / BlueStore cluster (Luminous 12.2.2) where the Filestore OSDs were much more affected by a cascading event than the [mostly larger] BlueStore OSDs. I suspect that had the whole cluster been BlueStore it might not have cascaded.
The goal: Let's say I want to go 500-1000PGs/OSD on 16T spinners to trim PGs to about 10-20G each. What are the resources that count will require compared with, say, 200 PGs/OSD? That's the interesting question and if I can make the resources available I would consider doing that.
The proof is in the proverbial pudding. Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher.
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <aad@dreamsnake.net> Sent: Wednesday, October 9, 2024 2:40 AM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] What is the problem with many PGs per OSD
I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball.
during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing.
A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue.
As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing.
My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA.
Yet, I'm holding back on an increase of PG count due to these rumors.
My personal sense:
HDD OSD: PG ratio 100-200 SATA/SAS SSD OSD: 200-300 NVMe SSD OSD: 300-400
These are not empirical figures. ymmv.
My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450.
The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be.
I heard in related rumors that some users have 1000+ PGs per OSD without problems.
On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000.
I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do.
It WILL use more RAM.
I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object”
Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead.
or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors.
An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500?
Consider this scenario:
An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up.
Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den ons 9 okt. 2024 kl 20:48 skrev Frank Schilder <frans@dtu.dk>:
The PG count per OSD is a striking exception. Its just a number (well a range with 100 recommended and 200 as a max: https://docs.ceph.com/en/latest/rados/operations/pgcalc/#keyDL). It just is. And this doesn't make any sense unless there is something really evil lurking in the dark. For comparison, a guidance that does make sense is something like 100PGs per TB. That I would vaguely understand: to keep the average PG size constant at a max of about 10G.
To be fair, this number could just be something vaguely related to "spin drives have 100-200 iops", and while cent/rhel linux kernels 10 years ago did have some issues in getting io done in parallel as much as possible towards a single device, doing multiple OSDs on flash devices would have been both a way to get around this limitation in the IO middle layer, and a way to "tell" ceph it can send more IO to the device since it has multiple OSDs on it. -- May the most significant bit of your life be positive.
Hi Janne.
To be fair, this number could just be something vaguely related to "spin drives have 100-200 iops" ...
It could be, but is it? Or is it just another rumor? I simply don't see how the PG count could possibly impact Io load on a disk. How about this guess: It could be dragged along from a time when HDDs were <=1T and it simply means to have PGs not larger than 10G. Sounds reasonable, but is it? I think we should really stop second-guessing here. This discussion was not meant to be a long thread where we all just guess but never know. I would appreciate if someone who actually knows something about why this recommendation is really there would ship in here. As far as I can tell, it could be anything or nothing. I actually tend toward its nothing, it was just never updated along with new developments and nowadays nobody knows any more. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: Thursday, October 10, 2024 8:51 AM To: Frank Schilder Cc: Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: What is the problem with many PGs per OSD Den ons 9 okt. 2024 kl 20:48 skrev Frank Schilder <frans@dtu.dk>:
The PG count per OSD is a striking exception. Its just a number (well a range with 100 recommended and 200 as a max: https://docs.ceph.com/en/latest/rados/operations/pgcalc/#keyDL). It just is. And this doesn't make any sense unless there is something really evil lurking in the dark. For comparison, a guidance that does make sense is something like 100PGs per TB. That I would vaguely understand: to keep the average PG size constant at a max of about 10G.
To be fair, this number could just be something vaguely related to "spin drives have 100-200 iops", and while cent/rhel linux kernels 10 years ago did have some issues in getting io done in parallel as much as possible towards a single device, doing multiple OSDs on flash devices would have been both a way to get around this limitation in the IO middle layer, and a way to "tell" ceph it can send more IO to the device since it has multiple OSDs on it. -- May the most significant bit of your life be positive.
Yes, this was an old lesson and AFAIK nobody has intentionally pushed the bounds in a long time because it was a very painful lesson for anybody who ran into it. The main problem was the increase in ram use scaling with PGs, which in normal operation is often fine but as we all know balloons in failure conditions. There are many developments that may have made things behave better, but early on some clusters just couldn’t be recovered until they received double their starting ram and were babysat through careful manually-orchestrated startup. (Or maybe worse — I forget.) Nobody’s run experiments, presumably because the current sizing guidelines are generally good enough to be getting on with, for anybody who has the resources to try and engage in the measurement work it would take to re-validate them. I will be surprised if anybody has information of the sort you seem to be searching for. -Greg On Thu, Oct 10, 2024 at 12:13 AM Frank Schilder <frans@dtu.dk> wrote:
Hi Janne.
To be fair, this number could just be something vaguely related to "spin drives have 100-200 iops" ...
It could be, but is it? Or is it just another rumor? I simply don't see how the PG count could possibly impact Io load on a disk.
How about this guess: It could be dragged along from a time when HDDs were <=1T and it simply means to have PGs not larger than 10G. Sounds reasonable, but is it?
I think we should really stop second-guessing here. This discussion was not meant to be a long thread where we all just guess but never know. I would appreciate if someone who actually knows something about why this recommendation is really there would ship in here. As far as I can tell, it could be anything or nothing. I actually tend toward its nothing, it was just never updated along with new developments and nowadays nobody knows any more.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Janne Johansson <icepic.dz@gmail.com> Sent: Thursday, October 10, 2024 8:51 AM To: Frank Schilder Cc: Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: What is the problem with many PGs per OSD
Den ons 9 okt. 2024 kl 20:48 skrev Frank Schilder <frans@dtu.dk>:
The PG count per OSD is a striking exception. Its just a number (well a range with 100 recommended and 200 as a max: https://docs.ceph.com/en/latest/rados/operations/pgcalc/#keyDL). It just is. And this doesn't make any sense unless there is something really evil lurking in the dark. For comparison, a guidance that does make sense is something like 100PGs per TB. That I would vaguely understand: to keep the average PG size constant at a max of about 10G.
To be fair, this number could just be something vaguely related to "spin drives have 100-200 iops", and while cent/rhel linux kernels 10 years ago did have some issues in getting io done in parallel as much as possible towards a single device, doing multiple OSDs on flash devices would have been both a way to get around this limitation in the IO middle layer, and a way to "tell" ceph it can send more IO to the device since it has multiple OSDs on it.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Greg, thanks for chiming in here.
... presumably because the current sizing guidelines are generally good enough to be getting on with ...
That's exactly why I'm bringing this up with such insistence. The guidelines are *not* good enough for EC pools on large HDDs that store a high percentage of small objects, in our case, files. In fact, they are really bad in that case and there were a number of recent ceph-user threads where a significant increase in PG count would probably help a lot. Including problems caused by very high object count per PG I'm dancing around on our cluster. For the guessing where the recommendation comes from, I'm actually leaning towards the "PGs should be limited in size" explanation. The recommendation of 100 PGs per OSD was good enough for a very long time together with bugs or observations you mention where it was never really assessed what the actual cause was or what resources are actually needed per PG compared with the total object count per OSD. PGs were originally invented to chunk up large disks for distributed RAID. To keep all-to-all rebuild time constant independent of the scale of the cluster. That's how you get scale-out capability. A fixed PG count counteracts that with the insane increase of capacity per disk we have lately. That's why I actually lean towards that the recommendation was intended to keep PGs below 5-10G each (and or <N objects) and was never updated with hardware developments. I have serious problems seeing how the PG count could be a single number screwing a cluster up. Peering, recovery, rocksdb size, everything is tied to the object count of an OSD. PGs just split this up into smaller units that are easier to manage. As a principle, for *any* problem with non-linear complexity (greater than linear complexity), solving M problems of size N/M is easier than solving 1 problem of size N. So, increasing the PG count should *improve* things just out of this principle. Unless there is a serious implementation problem I really don't understand why anyone would claim the opposite. If there is such an implementation problem, please anyone come forward. So I question here the anecdotal reports about the PG count being to blame alone. There have been a number of bugs discovered that were triggered by PG splitting. That these bugs are more likely hit when using high PG counts is kind of obvious. So its per se not the PG count that's the problem. Testing and experiments could be useful to update the guidelines. However, a good look at the code of a PG code maintainer would probably be faster and if there is something problematic it would be better to refer to the code than to experiments that might have missed the critical section. So the question really is, is there a piece of code that is more than quadratic in the PG count in any resource? Worse yet, is there something exponential? If there is something like that, there is no point making experiments. If there is nothing like that in the code, its worth conducting experiments and provide a table with resource usage depending on PG count. That would be very much appreciated. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Gregory Farnum <gfarnum@redhat.com> Sent: Thursday, October 10, 2024 10:19 AM To: Frank Schilder Cc: Janne Johansson; Anthony D'Atri; ceph-users@ceph.io Subject: Re: [ceph-users] Re: What is the problem with many PGs per OSD Yes, this was an old lesson and AFAIK nobody has intentionally pushed the bounds in a long time because it was a very painful lesson for anybody who ran into it. The main problem was the increase in ram use scaling with PGs, which in normal operation is often fine but as we all know balloons in failure conditions. There are many developments that may have made things behave better, but early on some clusters just couldn’t be recovered until they received double their starting ram and were babysat through careful manually-orchestrated startup. (Or maybe worse — I forget.) Nobody’s run experiments, presumably because the current sizing guidelines are generally good enough to be getting on with, for anybody who has the resources to try and engage in the measurement work it would take to re-validate them. I will be surprised if anybody has information of the sort you seem to be searching for. -Greg On Thu, Oct 10, 2024 at 12:13 AM Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi Janne.
To be fair, this number could just be something vaguely related to "spin drives have 100-200 iops" ...
It could be, but is it? Or is it just another rumor? I simply don't see how the PG count could possibly impact Io load on a disk. How about this guess: It could be dragged along from a time when HDDs were <=1T and it simply means to have PGs not larger than 10G. Sounds reasonable, but is it? I think we should really stop second-guessing here. This discussion was not meant to be a long thread where we all just guess but never know. I would appreciate if someone who actually knows something about why this recommendation is really there would ship in here. As far as I can tell, it could be anything or nothing. I actually tend toward its nothing, it was just never updated along with new developments and nowadays nobody knows any more. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Janne Johansson <icepic.dz@gmail.com<mailto:icepic.dz@gmail.com>> Sent: Thursday, October 10, 2024 8:51 AM To: Frank Schilder Cc: Anthony D'Atri; ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: Re: [ceph-users] Re: What is the problem with many PGs per OSD Den ons 9 okt. 2024 kl 20:48 skrev Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>:
The PG count per OSD is a striking exception. Its just a number (well a range with 100 recommended and 200 as a max: https://docs.ceph.com/en/latest/rados/operations/pgcalc/#keyDL). It just is. And this doesn't make any sense unless there is something really evil lurking in the dark. For comparison, a guidance that does make sense is something like 100PGs per TB. That I would vaguely understand: to keep the average PG size constant at a max of about 10G.
To be fair, this number could just be something vaguely related to "spin drives have 100-200 iops", and while cent/rhel linux kernels 10 years ago did have some issues in getting io done in parallel as much as possible towards a single device, doing multiple OSDs on flash devices would have been both a way to get around this limitation in the IO middle layer, and a way to "tell" ceph it can send more IO to the device since it has multiple OSDs on it. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
On Thu, 10 Oct 2024 08:53:08 +0000, Frank Schilder <frans@dtu.dk> said:
The guidelines are *not* good enough for EC pools on large HDDs that store a high percentage of small objects, in our case, files.
Arguably *nothing* is good enough for that, because it is the worst possible case scenario (A Ceph instance I inherited was like that). Ceph was designed to have large swarms of small OSSes with a 1 (or at most a few) small devices each. Having lots of small OSDs on many servers is a critical assumption of many aspects of Ceph (even if the opposite can be made to "workish"). This is very right though:
In fact, they are really bad in that case and there were a number of recent ceph-user threads where a significant increase in PG count would probably help a lot.
Including problems caused by very high object count per PG I'm dancing around on our cluster.
The main problem is not so much high object count per PG, it is that large PGs (in the legacy case above around 200GB) have to be "rebalanced" *in their entirety*, or go "damaged" again as a whole, impacting enormous numbers of objects. This happens whether their pool/profile is EC or replicas. Which point is part of my general observation that all systems have both a user and a background admin workload (which for storage tends to be whole-instance scans), and many are only sized to sustain the user workload, and Ceph as self-healing and self-balancing has a particularly high admin workload (resync, rebalance, scrub, "backup", ...).
For the guessing where the recommendation comes from, I'm actually leaning towards the "PGs should be limited in size" explanation. The recommendation of 100 PGs per OSD was good enough for a very long time
It was good for the intended use case, lots of small (by today's standards, around 1TB) OSDs on many servers working in parallel. Note: HDDs larger than 1TB are not really suitable for significant parallel user workloads and most admin workloads: http://www.sabi.co.uk/blog/17-one.html?170610#170610
PGs were originally invented to chunk up large disks for distributed RAID. To keep all-to-all rebuild time constant independent of the scale of the cluster.
That I guess was a secondary goal: the main goal I think was to reduce the size of metadata keeping track of redundancy (whether replicas or EC) from every-object-shard to every-PG, from a one level table to a two-level (list of PGs, list of object shards in a PG) table. The choice is then to have a larger numbers of PGs or to have a larger numbers of objects per PG as the number of total objects increases, and I too prefer the larger number of PGs.
So I question here the anecdotal reports about the PG count being to blame alone. There have been a number of bugs discovered that were triggered by PG splitting.
I remember the legacy (Pacific) Ceph instance that I had inherited did have lots of problems because of that.
Hi Peter, thanks for your comment. So it is mainly related to PG size. Unfortunately, we need to have a reality check here:
It was good for the intended use case, lots of small (by today's standards, around 1TB) OSDs on many servers working in parallel.
Note: HDDs larger than 1TB are not really suitable for significant parallel user workloads and most admin workloads: http://www.sabi.co.uk/blog/17-one.html?170610#170610
I'm afraid nobody will build a 100PB cluster with 1T drives. That's just absurd. So, the sharp increase of per-device capacity has to be taken into account. Specifically as the same development is happening with SSDs. There is no way around 100TB drives in the near future and a system like ceph is either able to handle that or will die. I would like to point out that there are scale-out storage systems that have adopted their architecture for this scenario and use large HDDs very well. For comparison, our University operates an all-HDD qumolo cluster that handles the administrative and user/student storage for the entire University and has about a factor 10 higher aggregated sustained IOP/s performance compared with a similarly sized ceph cluster (FS performance). Using the same EC redundancy, 16TB HDDs, having hundreds of snapshots on the file system and mirroring to a secondary site - all that basically on the same server configs we use for ceph. My best guess is that there is a tiering between replicated front-end- and EC back-end storage going on internally. But who knows, its closed source. However, it shows that doing something "good enough" with large drives is very well possible. The main problem with the high admin workload on ceph is that user data is directly mapped onto RADOS level objects. That was maybe a good idea back in the days, however, it has become a heavy legacy now, because all admin operations happen on the small user objects instead of aggregated objects that are much easier to operate on. For now I understand your comment in summary as: - Yes, 200G PGs are insane, they should be a lot smaller. - For large PGs the meta-data workload is actually higher than for small PGs (kind of what I also argue about) and its more efficient to rebuild and keep track of redundancy. - You also don't know why the recommendation today is 100-200 per OSD fixed except that it was suitable for 1TB drives. - You also can't answer what will happen if one goes for 100-200 PGs per TB, meaning 1600-3200 PGs per 16TB drive. So my main question, the last one, is still looking for an answer. Thanks for your comment and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Peter Grandi <pg@ceph.list.sabi.co.UK> Sent: Thursday, October 10, 2024 1:01 PM To: list Linux fs Ceph Subject: [ceph-users] Re: What is the problem with many PGs per OSD
On Thu, 10 Oct 2024 08:53:08 +0000, Frank Schilder <frans@dtu.dk> said:
The guidelines are *not* good enough for EC pools on large HDDs that store a high percentage of small objects, in our case, files.
Arguably *nothing* is good enough for that, because it is the worst possible case scenario (A Ceph instance I inherited was like that). Ceph was designed to have large swarms of small OSSes with a 1 (or at most a few) small devices each. Having lots of small OSDs on many servers is a critical assumption of many aspects of Ceph (even if the opposite can be made to "workish"). This is very right though:
In fact, they are really bad in that case and there were a number of recent ceph-user threads where a significant increase in PG count would probably help a lot.
Including problems caused by very high object count per PG I'm dancing around on our cluster.
The main problem is not so much high object count per PG, it is that large PGs (in the legacy case above around 200GB) have to be "rebalanced" *in their entirety*, or go "damaged" again as a whole, impacting enormous numbers of objects. This happens whether their pool/profile is EC or replicas. Which point is part of my general observation that all systems have both a user and a background admin workload (which for storage tends to be whole-instance scans), and many are only sized to sustain the user workload, and Ceph as self-healing and self-balancing has a particularly high admin workload (resync, rebalance, scrub, "backup", ...).
For the guessing where the recommendation comes from, I'm actually leaning towards the "PGs should be limited in size" explanation. The recommendation of 100 PGs per OSD was good enough for a very long time
It was good for the intended use case, lots of small (by today's standards, around 1TB) OSDs on many servers working in parallel. Note: HDDs larger than 1TB are not really suitable for significant parallel user workloads and most admin workloads: http://www.sabi.co.uk/blog/17-one.html?170610#170610
PGs were originally invented to chunk up large disks for distributed RAID. To keep all-to-all rebuild time constant independent of the scale of the cluster.
That I guess was a secondary goal: the main goal I think was to reduce the size of metadata keeping track of redundancy (whether replicas or EC) from every-object-shard to every-PG, from a one level table to a two-level (list of PGs, list of object shards in a PG) table. The choice is then to have a larger numbers of PGs or to have a larger numbers of objects per PG as the number of total objects increases, and I too prefer the larger number of PGs.
So I question here the anecdotal reports about the PG count being to blame alone. There have been a number of bugs discovered that were triggered by PG splitting.
I remember the legacy (Pacific) Ceph instance that I had inherited did have lots of problems because of that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm afraid nobody will build a 100PB cluster with 1T drives. That's just absurd
Check the archives for the panoply of absurdity that I’ve encountered ;)
So, the sharp increase of per-device capacity has to be taken into account. Specifically as the same development is happening with SSDs. There is no way around 100TB drives in the near future and a system like ceph is either able to handle that or will die
Agreed. I expect 122TB QLC in 1H2025. With NVMe and PCI-e Gen 5 one might experiment with slicing each into two OSDs. But for archival and object workloads latency usually isn’t so big a deal, so we may increasingly see a strategy adapted to the workloads.
10 higher aggregated sustained IOP/s performance compared with a similarly sized ceph cluster
But not, I suspect, nearly as many tentacles.
But not, I suspect, nearly as many tentacles.
No, that's the really annoying part. It just works. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: Thursday, October 10, 2024 2:13 PM To: Frank Schilder Cc: Peter Grandi; list Linux fs Ceph Subject: Re: [ceph-users] What is the problem with many PGs per OSD I'm afraid nobody will build a 100PB cluster with 1T drives. That's just absurd Check the archives for the panoply of absurdity that I’ve encountered ;) So, the sharp increase of per-device capacity has to be taken into account. Specifically as the same development is happening with SSDs. There is no way around 100TB drives in the near future and a system like ceph is either able to handle that or will die Agreed. I expect 122TB QLC in 1H2025. With NVMe and PCI-e Gen 5 one might experiment with slicing each into two OSDs. But for archival and object workloads latency usually isn’t so big a deal, so we may increasingly see a strategy adapted to the workloads. 10 higher aggregated sustained IOP/s performance compared with a similarly sized ceph cluster But not, I suspect, nearly as many tentacles.
[... number of PGs per OSD ...]
So it is mainly related to PG size.
Indeed and secondarily number of objects: many objects per PG mean lower metadata overhead, but bigger PGs mean higher admin workload latency.
Note: HDDs larger than 1TB are not really suitable for significant parallel user workloads and most admin workloads: http://www.sabi.co.uk/blog/17-one.html?170610#170610 "How challenging is a goal of 18MB/s per TB of storage, and latency"
I'm afraid nobody will build a 100PB cluster with 1T drives.
Well if they want it to be able to do a non-negligible user workload plus the necessary admin workload too bad for them. My post above does not depend at all on filesystem, but simply on the physical parameter of IOPS-per-TB (a "figure of merit" that is widely underestimate or ignored) of HDDs, and having enough IOPS-per-TB to sustain both user and admin workload. A couple of legacy Ceph instances I saw in the past had 8TB and 18TB drives and as they got full the instances basically congealed (latencies in the several seconds or even dozens of second range) even under modest user workloads, and anyhow expensive admin workloads like scrubbing (never mind deep scrubbing) got behind by a year or two, and rebalancing was nearly impossible. Again not because of Ceph.
That's just absurd. So, the sharp increase of per-device capacity has to be taken into account.
Indeed by taking into account that HDDs 4TB and above behave as slow-random-access tapes.
Specifically as the same development is happening with SSDs.
But that is completely different: SSDs have *much* higher IOPS, even SATA ones, so even large SSDs have enormously better IOPS-per-TB.
I would like to point out that there are scale-out storage systems that have adopted their architecture for this scenario and use large HDDs very well.
That is *physically impossible* as they just do not have enough IOPS-per-TB for many "live" workloads. The illusion that they might work well happens in one of two cases: * Either because they have not filled up yet, or because they have filled up but only a minuscule subset of the data is in active use, the IOPS-per-*active*-TB of the user workload is still good enough. The problem with that is that for many admin workloads a lot of the data or even or *all* the data becomes active. * If the *active data* is mostly read-only and gets cached on a SSD tier of sufficient size, and admin workload does not matter.
For comparison, our University operates an all-HDD qumolo cluster that handles the administrative and user/student storage for the entire University and has about a factor 10 higher aggregated sustained IOP/s performance compared with a similarly sized ceph cluster (FS performance).
I have some idea of how Qumulo does things and that is very unlikely, Ceph is not fundamentally inferior to their design. Perhaps the workload's anisotropy matches particularly well that of that particular Qumulo instance: https://www.sabi.co.uk/blog/16-one.html?160322#160322 "Examples of anisotropy in Isilon and Qumulo"
doing something "good enough" with large drives is very well possible.
It is possible to use large drives for "Glacier" style storage with an SSD front tier. But "Glacier" style storage is not trivial to design or manage. https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ https://storagemojo.com/2014/04/30/glacier-redux/ https://blog.dshr.org/2014/09/more-on-facebooks-cold-storage.html https://www.theregister.com/2015/05/07/facebook_maid_gets_cold/ https://blog.dshr.org/2021/05/storage-update.html
- You also don't know why the recommendation today is 100-200 per OSD fixed except that it was suitable for 1TB drives. - You also can't answer what will happen if one goes for 100-200 PGs per TB, meaning 1600-3200 PGs per 16TB drive.
So my main question, the last one, is still looking for an answer.
The famous joke goes: a guy goes to the doctor and says "Doctor when I stab my hand hard with a fork it really hurts a lots, how can you fix that?" the doctor says "Do not do that". If you do not like that answer, keep looking :-). Put another way if the combined user+admin workload requires N*100 IOPS-per-TB, using N IOPS-per-TB storage media is not goint to give a happy experience. But lots of people know better.
but simply on the physical parameter of IOPS-per-TB (a "figure of merit" that is widely underestimate or ignored)
hear hear!
of HDDs, and having enough IOPS-per-TB to sustain both user and admin workload.
Even with SATA SSDs I twice had to expand a cluster to meet SLO long before it was nearly full. The SNIA TCO calculator includes a multiplier for number of drives one has to provision for semi-acceptable IOPs.
A couple of legacy Ceph instances I saw in the past had 8TB and 18TB drives and as they got full the instances basically congealed (latencies in the several seconds or even dozens of second range) even under modest user workloads, and anyhow expensive admin workloads like scrubbing (never mind deep scrubbing) got behind by a year or two, and rebalancing was nearly impossible. Again not because of Ceph.
Been there, ITSY’d. Fragmentation matters with rotational media, even with op re-ordering within the drive or the driver.
But that is completely different: SSDs have *much* higher IOPS, even SATA ones, so even large SSDs have enormously better IOPS-per-TB.
And IOPS-per-yourlocalcurrency. Coarse-IU QLC is a bit of a wrinkle depending on workload...
I would like to point out that there are scale-out storage systems that have adopted their architecture for this scenario and use large HDDs very well.
That is *physically impossible* as they just do not have enough IOPS-per-TB for many "live" workloads. The illusion that they might work well happens in one of two cases:
* Either because they have not filled up yet,
I saw this with RGW on ultradense HDD toploaders.
or because they have filled up but only a minuscule subset of the data is in active use, the IOPS-per-*active*-TB of the user workload is still good enough.
Archival workloads - sure. Sometimes even backups. Even then, prudently-sourced QLC often has superior TCO compared to spinners.
* If the *active data* is mostly read-only and gets cached on a SSD tier of sufficient size, and admin workload does not matter.
And sometimes when that data active because of full backups, that process effectively flushes the cache to boot.
I have some idea of how Qumulo does things and that is very unlikely, Ceph is not fundamentally inferior to their design. Perhaps the workload's anisotropy matches particularly well that of that particular Qumulo instance:
Like a DB that’s column-oriented vs row-oriented?
The main problem was the increase in ram use scaling with PGs, which in normal operation is often fine but as we all know balloons in failure conditions.
Less so with BlueStore in my experience. I think in part this surfaces a bit of Filestore legacy that we might re-examine with Filestore being deprecated.
There are many developments that may have made things behave better
Very, very much so.
but early on some clusters just couldn’t be recovered until they received double their starting ram and were babysat through careful manually-orchestrated startup. (Or maybe worse — I forget.)
I helped a colleague through just such a 40 hour outage, trust me the only way it coulda been worse was if it were unrecoverable, as was the lab setup I described with a 9000 ratio. Ask Michael Kidd about the SVL disaster, he probably remembers ;) That outage was in part precipitated (or at least exacerbated) by a user issuing a rather large number of snap trims at once. I subsequently jacked up the snap trim cost and delay values. We did emergency RAM upgrades followed by babysitting. My colleague wrote a Python script that watched MemAvailable and gracefully restarted the OSDs on the given system as it reached a low water mark. This way recovery could at least make incremental progress. During which I increased the markdown count, and I think adjusted the reporters value. The one and only time I’ve ever run “ceph osd pause”. Luminous with mixed Filestore and BlueStore, and OSDs ranging from 1.6T to 3.84T. That cluster had initially been deployed in only two racks, so the CRUSH rules weren’t ideal. I subsequently refactored it and siblings to improve the failure domain situation and spread capacity. In the end one larger and one smaller cluster became four clusters, each with nearly uniform OSD sizes. And all the Filestore OSDs got redeployed in the process. Two weeks previously I’d found that the mons in this very cluster had enough RAM to run but not enough to boot — a function of growth and dedicated mon nodes. I’d arranged a Z0MG RAM upgrade on them. If I hadn’t, that outage indeed would have been much, much worse.
Nobody’s run experiments, presumably because the current sizing guidelines are generally good enough to be getting on with, for anybody who has the resources to try and engage in the measurement work it would take to re-validate them. I will be surprised if anybody has information of the sort you seem to be searching for.
The inestimable Mr. Farnum here describes an opportunity for community contribution (nudge nudge wink wink ;)
On 10/10/24 08:01, Anthony D'Atri wrote:
The main problem was the increase in ram use scaling with PGs, which in normal operation is often fine but as we all know balloons in failure conditions. Less so with BlueStore in my experience. I think in part this surfaces a bit of Filestore legacy that we might re-examine with Filestore being deprecated.
We have somewhat better observability around this now, but at Clyso we've still encountered situations where even having incredibly short PG log lengths can balloon into huge amounts of RAM usage with EC due to xattr rollback when there is gigantic cumulative xattrs. We've gotten around this by literally dropping the PG log length to 1 (10 wasn't good enough!) but with more PGs this could potentially be trickier. Having said that, I'm a huge fan of increasing PG counts (at least for small clusters) while decreasing PG log lengths. What I would like to see us do is make pglog length a per-pool attribute and favor automatically adjusting it via the prioritycache system instead of defaulting to PG autoscaling. I'd like to see us have the flexibility to support more pools at higher default PG counts. Enough so that autoscaling is more of a last resort sort of thing than the first tool we reach for. We'll also probably need to migrate to a stochastic sampling method for things like pg stat updates in the mgr (or at least lowering the update frequency), but that's a different topic.
There are many developments that may have made things behave better Very, very much so.
Agreed.
but early on some clusters just couldn’t be recovered until they received double their starting ram and were babysat through careful manually-orchestrated startup. (Or maybe worse — I forget.) I helped a colleague through just such a 40 hour outage, trust me the only way it coulda been worse was if it were unrecoverable, as was the lab setup I described with a 9000 ratio. Ask Michael Kidd about the SVL disaster, he probably remembers ;)
That outage was in part precipitated (or at least exacerbated) by a user issuing a rather large number of snap trims at once. I subsequently jacked up the snap trim cost and delay values.
We did emergency RAM upgrades followed by babysitting. My colleague wrote a Python script that watched MemAvailable and gracefully restarted the OSDs on the given system as it reached a low water mark. This way recovery could at least make incremental progress. During which I increased the markdown count, and I think adjusted the reporters value. The one and only time I’ve ever run “ceph osd pause”. Luminous with mixed Filestore and BlueStore, and OSDs ranging from 1.6T to 3.84T. That cluster had initially been deployed in only two racks, so the CRUSH rules weren’t ideal. I subsequently refactored it and siblings to improve the failure domain situation and spread capacity. In the end one larger and one smaller cluster became four clusters, each with nearly uniform OSD sizes. And all the Filestore OSDs got redeployed in the process.
Two weeks previously I’d found that the mons in this very cluster had enough RAM to run but not enough to boot — a function of growth and dedicated mon nodes. I’d arranged a Z0MG RAM upgrade on them. If I hadn’t, that outage indeed would have been much, much worse.
Nobody’s run experiments, presumably because the current sizing guidelines are generally good enough to be getting on with, for anybody who has the resources to try and engage in the measurement work it would take to re-validate them. I will be surprised if anybody has information of the sort you seem to be searching for. The inestimable Mr. Farnum here describes an opportunity for community contribution (nudge nudge wink wink ;)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
participants (8)
-
Anthony D'Atri
-
Anthony D'Atri
-
Eugen Block
-
Frank Schilder
-
Gregory Farnum
-
Janne Johansson
-
Mark Nelson
-
pg＠ceph.list.sabi.co.UK