crushmap rules :: host selection
Hi! I'm new with ceph and i struggle to make a mapping between my current storage knowledge and ceph... So, i will state my understanding of the context and the question so please correct me with anything that i got wrong :) So, files (or pieces of files) are put in PGs that are given sections of OSDs. The crushmap gives a physical OSDs map to be chosen for placement or access Pools are a logical name for a storage space but how can i specify what osds or host are part of a pool? For replication, how can i specify: if a replica is missing (for a given time) start rebuilding on some available OSD? Is there a notion of "spare" so if an osd is missing on action, the rebuild to start on another host and when the old OSD is back (the hdd is replaced, or the machine was repaired) to be automatically cleaned up and used? I'm thinking about a 3 node cluster with the replica=2 failure domain = host, in such a way if one node is down, the data from there to be replicated on the remaining nodes (with some drives kept as spares..) I am almost certain that from the point of view of ceph, what i'm thinking is wrong so i would love to receive some advice :) Thanks a lot! Adrian
-------- Original Message -------- Subject: [ceph-users] crushmap rules :: host selection From: Anthony D'Atri To: Adrian Sevcenco Date: 1/28/2024, 3:56:21 AM > First a all, thanks a lot for for info and taking time to help a beginner :) >> Pools are a logical name for a storage space but how can i specify >> what osds or host are part of a pool? > > Every pool specifies a CRUSH rule, which does this. By default all OSDs will be used. You can specify all, SSDs only, > HDDs only, etc. With a custom OSD device class you can select arbitrary hosts or OSDs. Oh! so the device class is more like an arbitrary label not a immutable defined property! looking at https://docs.ceph.com/en/reef/rados/operations/crush-map/#device-classes this is not specified ... So i can create arbitrary sets of OSDs on which a crush rule will be set and when that crush rule will be applied on a pool, this will actually tie the tagged OSDs (with the arbitrary class name) to pool.. did i get it right? >> For replication, how can i specify: if a replica is missing (for a given time) >> start rebuilding on some available OSD? > > Automatic. > >> >> Is there a notion of "spare" so if an osd is missing on action, the rebuild to >> start on another host and when the old OSD is back (the hdd is replaced, or the >> machine was repaired) to be automatically cleaned up and used? > > An available OSD will be selected to heal each PG according to the constraints in the pool’s CRUSH rule. By default no > PG will use more than one OSD on the same host. It is common to use racks instead for this failure domain in the > cluster is large enough and spread accordingly. so it depends on failure domain .. but with host failure domain, if there is space on some other OSDs will the missing OSDs be "healed" on the available space on some other OSDs? also, what will happen with the old ones? i ask for 2 main scenarios: 1. 1 machine breaks : the drives are ok, let's say it's just and power distributor problem and then the machine is put online after repair .. what will happen with the data on the OSDs? 2. a drive breaks : it is replaced, the drive is prepared and added with the same OSD number (as it is replaced) presumably the data was already replicated/healed : what will happen with the OSD that now is empty? will it be detected as replaced and just used? >> I'm thinking about a 3 node cluster with the replica=2 >> failure domain = host, in such a way if one node is down, the data >> from there to be replicated on the remaining nodes > > If one node is down the PGs will remain undersized because OSDs must be on disjoint hosts. Oh wait, you wrote size=2. and will the healing (rebuilding/resilvering) process start immediately ? or after some time? > Don’t do that. You will be likely to eventually lose data. Use size=3 min_size=2. If one node is down the PGs will > be undersized but active. hmm .. that means that there is a mecanism that i do not understand :) with RAID1 with 2 devices, if one is down when the replacement is added it will take ~seq speed to rebuild the mirror so for 22 TB usually is ~40 hours what is different, and what are the potential problems that can explode to data loss? >> (with some drives kept as spares..) > > This isn’t crummy RAID ;). You generally deploy OSDs on all drives and let Ceph grow new replicas to heal if it needs to. ok.. but is there some kind of space reservation mechanism that would allow that spare space to be used only when pool needs healing? >> I am almost certain that from the point of view of ceph, what i'm thinking is wrong >> so i would love to receive some advice :) > > Learning Ceph - Second Edition: Unifed, scalable, and reliable open source storage solution > https://a.co/d/9AwlerS <https://a.co/d/9AwlerS> ooh!! great, thanks a lot for info! :) > ;) > > Some nuances have changed since publication but the fundamentals are still fundamental. > > Welcome to Ceph — Ceph Documentation <https://docs.ceph.com/en/reef/> > docs.ceph.com <https://docs.ceph.com/en/reef/> > favicon.ico <https://docs.ceph.com/en/reef/> > > <https://docs.ceph.com/en/reef/> > > There is work underway to add a beginner’s guide. Until then, I suggest search engines, this list, and the first four > chapters of the above. Yup, i will do that :) Thanks a lot for help! Adrian > >> >> Thanks a lot! >> Adrian >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io
First a all, thanks a lot for for info and taking time to help a beginner :)
Nichts zu denken. This is a community, it’s what we do. Next year you’ll help someone else.
Oh! so the device class is more like an arbitrary label not a immutable defined property! looking at https://docs.ceph.com/en/reef/rados/operations/crush-map/#device-classes this is not specified ...
Yes. I verified this in the sources a couple of years ago. I thought I added something to docs about that, I’ll take a look at the above. For whatever reason I think Ceph does not automatically distinguish between SATA, SAS, and PCI (NVMe) SSDs, notably. So operators sometimes change the class of NVMe SSDs to “nvme”. We see classes like “fast-ssd” and “slow-ssd”; in our RGW work a couple years ago with Lua we set “qlc” since one would not want to mix those with TLC SSDs. People with lots of money might have set “optane”.
So i can create arbitrary sets of OSDs on which a crush rule will be set and when that crush rule will be applied on a pool, this will actually tie the tagged OSDs (with the arbitrary class name) to pool.. did i get it right?
Yes. But this is not something most people ever need or want to do. If you feel compelled to do so, there’s probably a better approach. Note that multiple pools can share OSDs and often do.
so it depends on failure domain .. but with host failure domain, if there is space on some other OSDs will the missing OSDs be "healed" on the available space on some other OSDs?
Yes, if you have enough hosts. When using 3x replication it is thus advantageous to have at least 4 hosts. Remember that the placement granularity is the PG, not OSD. Each placement group is placed independently. Think of CRUSH as a hash function. When an OSD or host is down, the topology changes, which is an input to the function, so the resulting output - the placement mappings- change. So Ceph converges the running state to match what’s expected. I’m kinda surprised at how much other software still needs to be managed manually. Ceph spoils us. For maintenance one can temporarily set flags that tell Ceph to not do this healing. So when rebooting to activate a new kernel, replace a DIMM, etc, the host will be right back, so you don’t want to bother rebalancing / healing.
also, what will happen with the old ones? i ask for 2 main scenarios: 1. 1 machine breaks : the drives are ok, let's say it's just and power distributor problem and then the machine is put online after repair .. what will happen with the data on the OSDs?
The OSDs will come “up” and “in” (hopefully), which is a topology change, so the cluster rebalances. Each PG peers, and only the data that changed while the OSDs were down is updated, which we call “recovery”. If a host is down for a very long time, it can be faster to wipe the OSDs and repopulate them in toto, but that’s an optimization you don’t need to worry about anytime soon.
2. a drive breaks : it is replaced, the drive is prepared and added with the same OSD number
The OSD ID may or may not be the same, depending on various factors. But don’t worry about that at this stage.
(as it is replaced) presumably the data was already replicated/healed : what will happen with the OSD that now is empty? will it be detected as replaced and just used?
See above. Remember that this isn’t dumb RAID. Ceph will more or less balance usage across available OSDs.
I'm thinking about a 3 node cluster with the replica=2 failure domain = host, in such a way if one node is down, the data from there to be replicated on the remaining nodes If one node is down the PGs will remain undersized because OSDs must be on disjoint hosts. Oh wait, you wrote size=2. and will the healing (rebuilding/resilvering) process start immediately ? or after some time?
Pretty much immediately. Ceph is fanatical about strong consistency.
Don’t do that. You will be likely to eventually lose data. Use size=3 min_size=2. If one node is down the PGs will be undersized but active. hmm .. that means that there is a mecanism that i do not understand :) with RAID1 with 2 devices
That’s a bad idea for some of the same reasons. I was doing RAID1 across three devices years ago. I think I even got HP to add that ability to their RoC HBA firmware. When you only keep two copies of data, sooner or later you’ll experiment overlapping failures and you’ll lose data. Additionally in a scale-out distributed system like Ceph there are certain sequences of events that result in not having even one replica of data that is known to be complete and up to date. I have personally experienced all of the above. The danger is real.
, if one is down when the replacement is added it will take ~seq speed to rebuild the mirror so for 22 TB
Ugh, HDDs for the lose.
usually is ~40 hours what is different, and what are the potential problems that can explode to data loss?
HDD dirty little secret # 437: slow healing means an extended period of risky degraded redundancy. Especially if you get stuck with SMR drives. I would expect much more than 40 hours for a 22TB spinner - Ceph tries to limit recovery speed so that clients aren’t DoSed. Erasure coding exacerbates this situation. I’ve seen an 8TB OSD take 4 weeks to backfill when throttled enough to allow client traffic. HDDs have slow, narrow interfaces with rotational / seek latency and thus are a false economy.
(with some drives kept as spares..) This isn’t crummy RAID ;). You generally deploy OSDs on all drives and let Ceph grow new replicas to heal if it needs to. ok.. but is there some kind of space reservation mechanism that would allow that spare space to be used only when pool needs healing?
Spare space is unused space. There is no distinction. Unlike many RAID implementations Ceph does not resilver blindly at drive granularity. Ceph has (adjustable) ratios for OSD fullness. You generally want to maintain enough unused space to allow healing when drives fail. If one is used to embedded or even software RAID this idea can take some time to get. It makes Ceph WAY more flexible, e.g. you don’t have to maintain exact-size spare drives. Though there are advantages to having not having huge variation in size.
I am almost certain that from the point of view of ceph, what i'm thinking is wrong so i would love to receive some advice :) Learning Ceph - Second Edition: Unifed, scalable, and reliable open source storage solution https://a.co/d/9AwlerS <https://a.co/d/9AwlerS> ooh!! great, thanks a lot for info! :)
Best book about Ceph ever written ;)
;) Some nuances have changed since publication but the fundamentals are still fundamental. Welcome to Ceph — Ceph Documentation <https://docs.ceph.com/en/reef/> docs.ceph.com <https://docs.ceph.com/en/reef/> favicon.ico <https://docs.ceph.com/en/reef/> <https://docs.ceph.com/en/reef/> There is work underway to add a beginner’s guide. Until then, I suggest search engines, this list, and the first four chapters of the above. Yup, i will do that :)
Thanks a lot for help! Adrian
Thanks a lot! Adrian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Oh! so the device class is more like an arbitrary label not a immutable defined property! looking at https://docs.ceph.com/en/reef/rados/operations/crush-map/#device-classes this is not specified …
"By default, OSDs automatically set their class at startup to hdd, ssd, or nvme in accordance with the type of device they are backed by.” Maybe we started setting “nvme” recently? I did indeed document a while ago that deviceclass names are arbitrary: https://docs.ceph.com/en/reef/rados/operations/crush-map/#creating-a-rule-for-a-replicated-pool
-------- Original Message -------- Subject: [ceph-users] Re: crushmap rules :: host selection From: Anthony D'Atri To: Adrian Sevcenco Date: 1/28/2024, 6:03:21 PM > >> >> First a all, thanks a lot for for info and taking time to help >> a beginner :) > > Nichts zu denken. This is a community, it’s what we do. Next year you’ll help someone else. :) >>>> >> Oh! so the device class is more like an arbitrary label not a immutable defined property! >> looking at https://docs.ceph.com/en/reef/rados/operations/crush-map/#device-classes >> this is not specified ... > > Yes. I verified this in the sources a couple of years ago. I thought I added something to docs about that, I’ll take a look at the above. i see, thanks! > For whatever reason I think Ceph does not automatically distinguish between SATA, SAS, and PCI (NVMe) SSDs, notably. So operators sometimes change the class of NVMe SSDs to “nvme”. We see classes like “fast-ssd” and “slow-ssd”; in our RGW work a couple years ago with Lua we set “qlc” since one would not want to mix those with TLC SSDs. People with lots of money might have set “optane”. > > >> So i can create arbitrary sets of OSDs on which a crush rule will be set >> and when that crush rule will be applied on a pool, this will actually tie >> the tagged OSDs (with the arbitrary class name) to pool.. did i get it right? > > Yes. But this is not something most people ever need or want to do. If you feel compelled to do so, there’s probably a better approach. > > Note that multiple pools can share OSDs and often do. well, in cases where users are teams with "this is my toy" and one try to unify different "toys" (and funding resources) under a common administrative umbrella (but keeping a clear distinction of hardware that is used only by owners) i might have to do this .. but this is only a gathering for information to make my case for a funding request for a pilot ceph installation .. >> so it depends on failure domain .. but with host failure domain, if there is space on some other OSDs >> will the missing OSDs be "healed" on the available space on some other OSDs? > > Yes, if you have enough hosts. When using 3x replication it is thus advantageous to have at least 4 hosts. Remember that the yeah, this is the reason that i thought to 2x replication, as i was thinking to request 3 hosts > placement granularity is the PG, not OSD. Each placement group is placed independently. Think of CRUSH as a hash function. yeah, i keep remembering this and then i forget :) is it wrong to think of PGs like a kind of object bucket (S3 like)? if so, is the size of PG the limit of a file size? (for replicated case)? if i have 20 PGs of 10 Gb size, uniformly occupied 50%, will i get an "no space" error if i try to write an 6 GB file? > When an OSD or host is down, the topology changes, which is an input to the function, so the resulting output - the placement mappings- change. So Ceph converges the running state to match what’s expected. I’m kinda surprised at how much other software still needs to be managed manually. Ceph spoils us. i see > For maintenance one can temporarily set flags that tell Ceph to not do this healing. So when rebooting to activate a new kernel, replace a DIMM, etc, the host will be right back, so you don’t want to bother rebalancing / healing. i will have to keep this in mind .. when i will have the actual hardware to experiment on i will get back here with questions :) >> also, what will happen with the old ones? i ask for 2 main scenarios: >> 1. 1 machine breaks : the drives are ok, let's say it's just and power distributor problem and then >> the machine is put online after repair .. what will happen with the data on the OSDs? > > The OSDs will come “up” and “in” (hopefully), which is a topology change, so the cluster rebalances. Each PG peers, and only the data that changed while the OSDs were down is updated, which we call “recovery”. cool! > If a host is down for a very long time, it can be faster to wipe the OSDs and repopulate them in toto, but that’s an optimization you don’t need to worry about anytime soon. ok, i will keep this in mind and ask about the exact procedure when the questions will not be hypothetical :) >> 2. a drive breaks : it is replaced, the drive is prepared and added with the same OSD number > > The OSD ID may or may not be the same, depending on various factors. But don’t worry about that at this stage. got it >> (as it is replaced) >> presumably the data was already replicated/healed : what will happen with the OSD that now is empty? >> will it be detected as replaced and just used? > > See above. Remember that this isn’t dumb RAID. Ceph will more or less balance usage across available OSDs. yeah.. it will be a tough change of habits :) >>>> I'm thinking about a 3 node cluster with the replica=2 >>>> failure domain = host, in such a way if one node is down, the data >>>> from there to be replicated on the remaining nodes >>> If one node is down the PGs will remain undersized because OSDs must be on disjoint hosts. Oh wait, you wrote size=2. >> and will the healing (rebuilding/resilvering) process start immediately ? or after some time? > > Pretty much immediately. Ceph is fanatical about strong consistency. isn't there any time delay? maybe configurable? for example if some technician pulls the wrong network cable for a few seconds i wouldn't like the healing to start on the microsecond that something went wrong >>> Don’t do that. You will be likely to eventually lose data. Use size=3 min_size=2. If one node is down the PGs will be undersized but active. >> hmm .. that means that there is a mecanism that i do not understand :) >> with RAID1 with 2 devices > > That’s a bad idea for some of the same reasons. I was doing RAID1 across three devices years ago. I think I even got HP to add that ability to their RoC HBA firmware. > > When you only keep two copies of data, sooner or later you’ll experiment overlapping failures and you’ll lose data. Additionally in a scale-out distributed system like Ceph there are certain sequences of events that result in not having even one replica of data that is known to be complete and up to date. > > I have personally experienced all of the above. The danger is real. hmm .. got it, 4 host, 3 replica min_size=2 it is then :) >> , if one is down when the replacement is added it will take ~seq speed to rebuild the mirror >> so for 22 TB > > Ugh, HDDs for the lose. well, i have to goals : 1. replacing the multiple VM images that i have replicated on different hosts (multiple for the same VM!! .. it's a mess) and have a RBD pool for these 2. use CEPHFS to replace a few NFS servers that lately hit performance ceil and brought me some hard headaches with stability so, while for (1) i plan to use some nvme ssds for (2) there will be hdds (with the plan that after i make the pilot running to somehow use the drives from the nfs servers to ceph in a N+2 ECC way, similarly to the RAID6 that i have now and with OSD level failure domain) but first i have to get the funding and hardware for the pilot installation :) >> usually is ~40 hours >> what is different, and what are the potential problems that can explode to data loss? > > HDD dirty little secret # 437: slow healing means an extended period of risky degraded redundancy. Especially if you get stuck with SMR drives. i do not use SMR, i have only the normal CMR, and this 30-40 hours is the 10+2 RAID6 rebuilding time and yes, there is always possibility to lose 2 drives in partition, so i have to take out the RAID from production and really hope that nothing will happen until i get in data-center to change the drives :D > I would expect much more than 40 hours for a 22TB spinner - Ceph tries to limit recovery speed so that clients aren’t DoSed. Erasure coding exacerbates this situation. I’ve seen an 8TB OSD take 4 weeks to backfill when throttled enough to allow client traffic. HDDs have slow, narrow interfaces with rotational / seek latency and thus are a false economy. well, then this is worrisome for me .. as i said above, in a 12 disks RAID6 array, the rebuild time for 1 disk is under 40 hours (but it's true that i make the partition RO so no additional writes hammers the drives, only reads) >>>> (with some drives kept as spares..) >>> This isn’t crummy RAID ;). You generally deploy OSDs on all drives and let Ceph grow new replicas to heal if it needs to. >> ok.. but is there some kind of space reservation mechanism that would allow that spare space to be used only >> when pool needs healing? > > Spare space is unused space. There is no distinction. Unlike many RAID implementations Ceph does not resilver blindly at drive granularity. i see, got it > Ceph has (adjustable) ratios for OSD fullness. You generally want to maintain enough unused space to allow healing when drives fail. If one is used to embedded or even software RAID this idea can take some time to get. It makes Ceph WAY more flexible, e.g. you don’t have to maintain exact-size spare drives. Though there are advantages to having not having huge variation in size. oh, that means that you can do something like "use at most 90% of space for normal opperations, but if there is a healing balancing use the rest of 10% until new OSD is in place and balancing will free that 10%" ? >>>> I am almost certain that from the point of view of ceph, what i'm thinking is wrong >>>> so i would love to receive some advice :) >>> Learning Ceph - Second Edition: Unifed, scalable, and reliable open source storage solution >>> https://a.co/d/9AwlerS <https://a.co/d/9AwlerS> >> ooh!! great, thanks a lot for info! :) > > Best book about Ceph ever written ;) Ha! I will make sure to verify the claim :))) Thanks a lot!! Adrian
> >>> so it depends on failure domain .. but with host failure domain, if there is space on some other OSDs >>> will the missing OSDs be "healed" on the available space on some other OSDs? >> Yes, if you have enough hosts. When using 3x replication it is thus advantageous to have at least 4 hosts. Remember that the > yeah, this is the reason that i thought to 2x replication, as i was thinking to request 3 hosts You can do RF=3 with 3 hosts. >> placement granularity is the PG, not OSD. Each placement group is placed independently. Think of CRUSH as a hash function. > yeah, i keep remembering this and then i forget :) > is it wrong to think of PGs like a kind of object bucket (S3 like)? Mostly, yes. > if so, is the size of PG the limit of a file size? (for replicated case)? Modern Ceph clusters use the BlueStore back-end for OSDs. There is no limit as such on PG size. A PG will on average store 1/pg_num of the data in a pool > if i have 20 PGs of 10 Gb size, uniformly occupied 50%, will i get an "no space" error > if i try to write an 6 GB file? PGs aren’t sized, so no. > > >>>>> I'm thinking about a 3 node cluster with the replica=2 >>>>> failure domain = host, in such a way if one node is down, the data >>>>> from there to be replicated on the remaining nodes >>>> If one node is down the PGs will remain undersized because OSDs must be on disjoint hosts. Oh wait, you wrote size=2. >>> and will the healing (rebuilding/resilvering) process start immediately ? or after some time? >> Pretty much immediately. Ceph is fanatical about strong consistency. > isn't there any time delay? maybe configurable? > for example if some technician pulls the wrong network cable for a few seconds > i wouldn't like the healing to start on the microsecond that something went wrong There are lots of nuances haven’t mentioned so as to not confuse you with stuff you don’t need to know, at least not yet. There is a configurable grace period before an OSD is marked down, and if you use `rack` failure domains you can prevent automatic rebalancing if an entire host goes down. “Immediately” is relative. I think the default grace is …. 5 minutes or so. Ceph has like 2000 “options” that can be set. Most of them you don’t need to know about and should never touch. > >>> , if one is down when the replacement is added it will take ~seq speed to rebuild the mirror >>> so for 22 TB >> Ugh, HDDs for the lose. > well, i have to goals : > 1. replacing the multiple VM images that i have replicated on different hosts > (multiple for the same VM!! .. it's a mess) and have a RBD pool for these Ceph is great at block storage with RBD. Most OpenStack installations use it for Glance and Cinder. Note that if by image you mean guest OS images, then you may get away with HDDs. If you mean images as in collections of data that get attached as a block storage volume to a VM that will use it for a database or as a filesystem, those huge spinners will likely not make you glad. > > >>> usually is ~40 hours >>> what is different, and what are the potential problems that can explode to data loss? >> HDD dirty little secret # 437: slow healing means an extended period of risky degraded redundancy. Especially if you get stuck with SMR drives. > i do not use SMR, i have only the normal CMR, and this 30-40 hours is the 10+2 RAID6 rebuilding time 10+2? Yeah. Striping 2x 5+2 parity groups would give you double the write performance. With conventional parity RAID the space amplification benefit to large parity groups rapidly sees diminishing returns, but the write performance hit is extreme. > >> I would expect much more than 40 hours for a 22TB spinner - Ceph tries to limit recovery speed so that clients aren’t DoSed. Erasure coding exacerbates this situation. I’ve seen an 8TB OSD take 4 weeks to backfill when throttled enough to allow client traffic. HDDs have slow, narrow interfaces with rotational / seek latency and thus are a false economy. > well, then this is worrisome for me .. as i said above, in a 12 disks RAID6 array, the rebuild time for 1 disk is under > 40 hours (but it's true that i make the partition RO so no additional writes hammers the drives, only reads) Ceph recovery times likely will be less, because you aren’t necessarily healing the full range of LBAs on a drive. > >> Ceph has (adjustable) ratios for OSD fullness. You generally want to maintain enough unused space to allow healing when drives fail. If one is used to embedded or even software RAID this idea can take some time to get. It makes Ceph WAY more flexible, e.g. you don’t have to maintain exact-size spare drives. Though there are advantages to having not having huge variation in size. > oh, that means that you can do something like "use at most 90% of space for normal opperations, but if > there is a healing balancing use the rest of 10% until new OSD is in place and balancing will free that 10%" ? Something like that. ceph/src/common/options/global.yaml.in : # writes will fail if an OSD exceeds this fullness - name: mon_osd_full_ratio type: float level: advanced desc: full ratio of OSDs to be set during initial creation of the cluster default: 0.95 flags: - no_mon_update - cluster_create with_legacy: true # an OSD will refuse taking backfill if it exceeds this fullness - name: mon_osd_backfillfull_ratio type: float level: advanced default: 0.9 flags: - no_mon_update - cluster_create with_legacy: true # The cluster’s health state will go WARN if any OSD exceeds this fullness - name: mon_osd_nearfull_ratio type: float level: advanced desc: nearfull ratio for OSDs to be set during initial creation of cluster default: 0.85 flags: - no_mon_update - cluster_create Recent Ceph releases enforce that the values of these options must be set in this order. Having the nearfull ratio be 99% when the full ratio is 95% for example would make no sense.
-------- Original Message -------- Subject: [ceph-users] crushmap rules :: host selection From: Anthony D'Atri To: Adrian Sevcenco Date: 1/28/2024, 11:34:00 PM >> >>>> so it depends on failure domain .. but with host failure domain, if there is space on some other OSDs >>>> will the missing OSDs be "healed" on the available space on some other OSDs? >>> Yes, if you have enough hosts. When using 3x replication it is thus advantageous to have at least 4 hosts. Remember that the >> yeah, this is the reason that i thought to 2x replication, as i was thinking to request 3 hosts > > You can do RF=3 with 3 hosts. yeah, i will see what i can get approved >>> placement granularity is the PG, not OSD. Each placement group is placed independently. Think of CRUSH as a hash function. >> yeah, i keep remembering this and then i forget :) >> is it wrong to think of PGs like a kind of object bucket (S3 like)? > > Mostly, yes. so .. in a PG there are no "file data" but pieces of "file data"? so 100 GB file with 2x replication will be placed in more than 2 PGs? Is there some kind of strips or chunks that a given data conglomerate is split into and then _those_ are put in multiple PGs with the rule that they have to be in a Replicated=X ? >> if so, is the size of PG the limit of a file size? (for replicated case)? > > Modern Ceph clusters use the BlueStore back-end for OSDs. There is no limit as such on PG size. A PG will on average store 1/pg_num of the data in a pool > >> if i have 20 PGs of 10 Gb size, uniformly occupied 50%, will i get an "no space" error >> if i try to write an 6 GB file? > > PGs aren’t sized, so no. >> >> >>>>>> I'm thinking about a 3 node cluster with the replica=2 >>>>>> failure domain = host, in such a way if one node is down, the data >>>>>> from there to be replicated on the remaining nodes >>>>> If one node is down the PGs will remain undersized because OSDs must be on disjoint hosts. Oh wait, you wrote size=2. >>>> and will the healing (rebuilding/resilvering) process start immediately ? or after some time? >>> Pretty much immediately. Ceph is fanatical about strong consistency. >> isn't there any time delay? maybe configurable? >> for example if some technician pulls the wrong network cable for a few seconds >> i wouldn't like the healing to start on the microsecond that something went wrong > > There are lots of nuances haven’t mentioned so as to not confuse you with stuff you don’t need to know, at least not yet. There is a configurable grace period before an OSD is marked down, and if you use `rack` failure domains you can prevent automatic rebalancing if an entire host goes down. “Immediately” is relative. I think the default grace is …. 5 minutes or so. Oh, got it! well, at least is there, if i need it i will ask about :) > Ceph has like 2000 “options” that can be set. Most of them you don’t need to know about and should never touch. great! better to have many tunables and then just ask for usage then to have things hardcoded >>>> , if one is down when the replacement is added it will take ~seq speed to rebuild the mirror >>>> so for 22 TB >>> Ugh, HDDs for the lose. >> well, i have to goals : >> 1. replacing the multiple VM images that i have replicated on different hosts >> (multiple for the same VM!! .. it's a mess) and have a RBD pool for these > > Ceph is great at block storage with RBD. Most OpenStack installations use it for Glance and Cinder. > > Note that if by image you mean guest OS images, then you may get away with HDDs. If you mean images as in collections of data that get attached as a block storage volume to a VM that will use it for a database or as a filesystem, those huge spinners will likely not make you glad. The 2nd case, and yes for RBD i plan to use nvme (i have the OS images for now but i know that i can convert and import them into rbd) >>>> usually is ~40 hours >>>> what is different, and what are the potential problems that can explode to data loss? >>> HDD dirty little secret # 437: slow healing means an extended period of risky degraded redundancy. Especially if you get stuck with SMR drives. >> i do not use SMR, i have only the normal CMR, and this 30-40 hours is the 10+2 RAID6 rebuilding time > > 10+2? Yeah. Striping 2x 5+2 parity groups would give you double the write performance. With conventional parity RAID the space amplification benefit to large parity groups rapidly sees diminishing returns, but the write performance hit is extreme. on these cases (NFS like usage) i seldom have random access but space is the most important, but this is to be seen after i have the pilot working already >>> I would expect much more than 40 hours for a 22TB spinner - Ceph tries to limit recovery speed so that clients aren’t DoSed. Erasure coding exacerbates this situation. I’ve seen an 8TB OSD take 4 weeks to backfill when throttled enough to allow client traffic. HDDs have slow, narrow interfaces with rotational / seek latency and thus are a false economy. >> well, then this is worrisome for me .. as i said above, in a 12 disks RAID6 array, the rebuild time for 1 disk is under >> 40 hours (but it's true that i make the partition RO so no additional writes hammers the drives, only reads) > > Ceph recovery times likely will be less, because you aren’t necessarily healing the full range of LBAs on a drive. hmmm... right!! this will be a nice thing! >>> Ceph has (adjustable) ratios for OSD fullness. You generally want to maintain enough unused space to allow healing when drives fail. If one is used to embedded or even software RAID this idea can take some time to get. It makes Ceph WAY more flexible, e.g. you don’t have to maintain exact-size spare drives. Though there are advantages to having not having huge variation in size. >> oh, that means that you can do something like "use at most 90% of space for normal opperations, but if >> there is a healing balancing use the rest of 10% until new OSD is in place and balancing will free that 10%" ? > > Something like that. > > ceph/src/common/options/global.yaml.in : > > > # writes will fail if an OSD exceeds this fullness > - name: mon_osd_full_ratio > type: float > level: advanced > desc: full ratio of OSDs to be set during initial creation of the cluster > default: 0.95 > flags: > - no_mon_update > - cluster_create > with_legacy: true > > > # an OSD will refuse taking backfill if it exceeds this fullness > - name: mon_osd_backfillfull_ratio > type: float > level: advanced > default: 0.9 > flags: > - no_mon_update > - cluster_create > with_legacy: true > > # The cluster’s health state will go WARN if any OSD exceeds this fullness > - name: mon_osd_nearfull_ratio > type: float > level: advanced > desc: nearfull ratio for OSDs to be set during initial creation of cluster > default: 0.85 > flags: > - no_mon_update > - cluster_create > > Recent Ceph releases enforce that the values of these options must be set in this order. Having the nearfull ratio be 99% when the full ratio is 95% for example would make no sense. > Thanks a lot for info!!! This is very useful to me and overall all this information is enough help me to make a case for the funding request to management for a pilot installation :) Adrian
so .. in a PG there are no "file data" but pieces of "file data"?
Yes. Chapter 8 may help here, but be warned, it’s pretty dense and may confuse more than help. The foundation layer of Ceph is RADOS — services including block (RBD), file (CephFS), and object (RGW) storage are built on top of it. RADOS handles placing data and ensuring that configured replication is maintained. RADOS stores chunks of data in RADOS objects. Which are not to be confused with S3 (or Swift) objects. Yes, the term is overloaded and confusing. I disambiguate in the docs whenever I come across an unspecified reference.
so 100 GB file with 2x replication will be placed in more than 2 PGs?
For sure. RADOS objects are if I recall correctly at most 4MiB in size. I once found a 16MiB RADOS object in a cluster — one of my colleagues must have done something rather outré to create it. So that 100GB file (block volume, S3 object …) will be split into at least 25 RADOS objects spread across some number of PGs within the pool in question. Each PG is independently placed on — in your case — 2 OSDs, which by default will enforce anti-affinity: those 2 OSDs won’t be on the same host. When you have a larger number of OSDs, PGs will live on different pairs of OSDs. In your extremely small cluster, if you have only 3 hosts and do 3-way replication, every PG will live on every host — but distributed among the OSDs on each host.
Is there some kind of strips or chunks that a given data conglomerate is split into and then _those_ are put in multiple PGs with the rule that they have to be in a Replicated=X ?
That sounds like a RADOS object. But be clear that a PG generally has replicas (or shards) on multiple OSDs, it’s a one-to-many mapping. A PG with the ID 1.ff might have replicas on, say, host1 (osd.3) and host2 (osd.11). PG 1.11 might have replicas on host1 (osd.0) and host3 (osd.13).
Ceph has like 2000 “options” that can be set. Most of them you don’t need to know about and should never touch. great! better to have many tunables and then just ask for usage then to have things hardcoded
Indeed. And most or all of them are documented these days. That wasn’t always the case ;) There are a few things hardcoded, but nothing you need to worry about.
The 2nd case, and yes for RBD i plan to use nvme (i have the OS images for now but i know that i can convert and import them into rbd)
Be sure to use enterprise-grade drives. Client (consumer, desktop) drives are a false economy. They often have limited durability and are prone to cliffing, where rather than presenting sustained performance, at some point performance will drop substantially. They also may lack power loss protection, so if a server/rack/DC loses power suddenly, data in flight may be lost.
Den sön 28 jan. 2024 kl 23:02 skrev Adrian Sevcenco <Adrian.Sevcenco@cern.ch>:
is it wrong to think of PGs like a kind of object bucket (S3 like)?
Mostly, yes. so .. in a PG there are no "file data" but pieces of "file data"? so 100 GB file with 2x replication will be placed in more than 2 PGs? Is there some kind of strips or chunks that a given data conglomerate is split into and then _those_ are put in multiple PGs with the rule that they have to be in a Replicated=X ?
PGs have no fixed size, they accept objects until the OSD(s) they lie on cannot accept more data. For repl=3, it means three full copies of each object on three separate PGs which are chose to be from three different OSDs, by default also on three different OSD hosts. That said, if you use RadosGW or RBD (for VM disk images for example), those two will split your large "files" into many smaller objects, at sizes like 2M or 4M each. So if you upload a 10G S3 object to RadosGW or have your virtualization platform assign a 10G "disk" to a VM, that 10G ceph object will not be in one PG, it will be made into 2-4M pieces and those will have different names like 10G_image.1001, 10G_image.1002 which means the pieces will spread out on many different PGs. This has the benefit of making it possible to store really huge images on many small drives, and also spread load since the VM might itself ask for IO to be done on several places on its disk, and these would be served by different drives in your ceph cluster. If the VM image ended up as a single object in a single PG, then it would mean all IO for it would always hit this one drive and you would get really bad performance which would be worse than if the hypervisor had this drive locally attached. So for most usage as a normal user, the libraries underneath will do the object splitting for you and it will be totally transparent to you but depending on how you use the cluster and at which level you are looking, those final objects in ceph do not get split again by the cluster itself, unless you are using Erasure Coding. With rados calls one can make 100G objects, but most usage goes via librbd or radosgw or something on top of rados and hence it does the splitting for you.
if so, is the size of PG the limit of a file size? (for replicated case)?
Mostly no. The corner case here is if you have OSDs with very different amounts of free space. This is mostly visible in reports like "ceph df" because it will say a pool (made up of PGs) has a max-free based on the OSD with least free space, multiplied by number-of-OSDs in the pool. This is because the PG placement is semi-random and depending on what you write, it could happen that all data goes into the OSD with the least amount of free space every time, and then the pool goes full long before the largest OSD is filled. The fewer OSDs you have and the larger the difference in sizes of those OSDs you get, the worse the situation becomes. As an example, lets say you have a pool that is told by crush to end up on these three OSD hosts with one drive each: HostA: 10TB free HostB: 10TB free HostC: 10G free In this case "ceph df" will say "you should consider this pool to have (raw) 30G free at most" and if you have repl=3 on this pool then it can only take 10G data before either primary data or one of the two copies for the data written will make HostC go full. This situation makes "ceph df" tell you weird numbers, but also the most truthful ones, since they tell you what your clients will experience. If you end up in a situation like the above, you can also notice that if any OSD breaks, hostC has no chance of hosting extra copies during repairs, since it will go full almost immediately, when it is unbalanced like this, your redundancy also suffers. If your cluster is holding important data, you would make sure there are most osd hosts than the replication factor so it can repair copies onto another host when one host dirs, and since there is soft limits at 85, 90 and 95% OSD capacity that stops certain fileops from driving the OSDs totally full, you would want to keep the cluster at 50-60% at most for small clusters so that if one host dies, all of its data can fit onto the others without passing the first 85% limit while still allowing all PGs to hold three copies in total. If you have many OSD hosts, each one represents a small percentage of the total, and it will be easier for a cluster to handle a host failure using space from all the others to rebuild, but then at some point, having 100+ OSD hosts or more would make chances larger that any one host is experiencing planned or unplanned maintenance. So while PGs don't have fixed size limits, they do have a current size, and when moving or balancing data, ceph moves PGs around, not files or objects. That is why it is preferred to have something like 100+ PGs per OSD, so that ceph can balance a part of data at a time that would be around 1% of the disk capacity and not just the whole OSD back and forth. -- May the most significant bit of your life be positive.
participants (3)
-
Adrian Sevcenco
-
Anthony D'Atri
-
Janne Johansson