Hey all! I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask. So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph. I'd like to at least be able to take one node at a time out of rotation for maintenance purposes. I'd be using cephadm for the setup, which in my testing works really well. As for the questions: 1. 3x replication is a tough sell IMO. For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant. But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.) 2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool. What about changing k, m, or the plugin type? My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed? 3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups, and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS? So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway. 4. In v20 there seems to be a new ec_optimizations feature for ec pools used in CephFS or RBD. Are those a good idea with this kind of (large) qcow2 image workload on top of CephFS? 5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that. Should I be waiting for it / start a new cluster with v20 now already? Thanks for any insights you can give! Kind regards Matthias Riße -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------
I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask.
So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph. I'd like to at least be able to take one node at a time out of rotation for maintenance purposes. I'd be using cephadm for the setup, which in my testing works really well.
As for the questions:
1. 3x replication is a tough sell IMO. For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant. But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.)
Those crazy K+Ms are going to make any small io involve A LOT of drives. There is a good feeling about having host failure domain (and if repl=3) also knowing all pieces always are on different hosts.
2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool. What about changing k, m, or the plugin type? My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed?
Don't know what you say happened, but unless something changed very recently, it should not alter the pool. You can change the k+m policy you once created such a pool from, but it will not change the pool itself. The pool copies the policy at creation time and then never re-reads it. Changing repl=3,4,5 values just create or delete copies while retaining the other replicas, but changing K+M would require the cluster to recalculate all pieces of all data in all PGs. That would be technically be possible but a very long process for a large cluster.
3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups, and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS? So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway.
RBDs are lots simpler, and MDSs eat/require a fair bit of ram to continue performing. Could be that tests while idle show both are good enough, but that may not be true later on.
4. In v20 there seems to be a new ec_optimizations feature for ec pools used in CephFS or RBD. Are those a good idea with this kind of (large) qcow2 image workload on top of CephFS?
Normally EC pools are not the best fit for neither CephFS nor RBD, even if it can be made to work. Same as above, on idle clusters, the overhead might not be visible. Then again, you might never grow your usage to where it really starts being noticed. For our use cases, we mostly let the guests do their own caching, so we give the ram to the instances so they have a margin to use for caches and buffers. If you go with CephFS the MDSs are taking over that part in some sense, so you could possibly lower guests ram, but chances are you will be caching same/similar data in multiple places. If budgets are tight, it might be worth considering.
5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that. Should I be waiting for it / start a new cluster with v20 now already?
I think I would start with v20 if the release (20.2.0) comes out fairly quickly, otherwise go with latest 19 and upgrade when 20 comes out so you get to practice and document that too. -- May the most significant bit of your life be positive.
Thanks for the fast reply!
I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask.
So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph. I'd like to at least be able to take one node at a time out of rotation for maintenance purposes. I'd be using cephadm for the setup, which in my testing works really well.
As for the questions:
1. 3x replication is a tough sell IMO. For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant. But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.)
Those crazy K+Ms are going to make any small io involve A LOT of drives. There is a good feeling about having host failure domain (and if repl=3) also knowing all pieces always are on different hosts.
I take from that that it might be useful if at some point we need bulk storage for large sequential data, but definitely not as VM storage. I actually just learned that we should have another ten 18 TiB drives laying around that we could just slot in, so with that my estimation is that we are more limited on compute than on raw storage with what we have. Which would mean 3x replication after all...
2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool. What about changing k, m, or the plugin type? My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed?
Don't know what you say happened, but unless something changed very recently, it should not alter the pool. You can change the k+m policy you once created such a pool from, but it will not change the pool itself. The pool copies the policy at creation time and then never re-reads it. Changing repl=3,4,5 values just create or delete copies while retaining the other replicas, but changing K+M would require the cluster to recalculate all pieces of all data in all PGs. That would be technically be possible but a very long process for a large cluster.
OK, then I understood that correctly. Recalculating would indeed be very expensive. I guess the planned "pool migration" feature would be the only way to do this then, at the cost of explicitly doing this recalculation. Just to make sure though, failure domains and device classes /can/ be changed after the fact, right? Might become relevant whenever we add SSDs into the mix.
3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups, and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS? So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway.
RBDs are lots simpler, and MDSs eat/require a fair bit of ram to continue performing. Could be that tests while idle show both are good enough, but that may not be true later on.
Ack. But RBD does not yet have a way to export at least crash-consistent snapshots of a group of images, right? I think I saw that the export functionality for group snapshots is still in development... This is really a limitation of the tape backup system we have: I can't push more than about 8 TiB at a time to it, otherwise the connection drops and the backup fails. And this only works for RBD exports with an unofficial and unsupported client, otherwise it must be files on a filesystem. So what we have right now is multiple 5 TiB qcow2 images for the VMs that need a lot of storage, and we LVM them together in the VM. I could do the same with RBD, but would need a way to make a consistent backup of them, which I don't see yet. But maybe I am just missing something? Our requirements to performance should overall be relatively modest, so I am inclined to just try it with CephFS and see how far it goes. We can still migrate to RBD in the future, if it makes sense.
5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that. Should I be waiting for it / start a new cluster with v20 now already?
I think I would start with v20 if the release (20.2.0) comes out fairly quickly, otherwise go with latest 19 and upgrade when 20 comes out so you get to practice and document that too.
Makes sense. I'll be on vacation for a week, so I guess I'll see what the world looks like then :) -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------
Just to make sure though, failure domains and device classes /can/ be changed after the fact, right? Might become relevant whenever we add SSDs into the mix.
Yes, those two factors are basically only telling a pool on which OSDs to move to, and can be set and reset as you want. We have given some OSDs special device-class names (like ssd-rack-1) so we later can point a pool to only choose those OSDs to place itself on when migrating from older ssds to newer ones. In this case, the type of replication/EC doesn't matter since it is not changing. Also, if you screw this up (like asking a 7+3 pool to move to a list of only 8 OSDs so it doesn't "fit") it will just make the PGs "misplaced" and refuse to move, while still serving data, so you can undo this choice. -- May the most significant bit of your life be positive.
Le 13/11/2025 à 15:58, Matthias Riße a écrit :
Hey all!
I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask.
So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph. I'd like to at least be able to take one node at a time out of rotation for maintenance purposes. I'd be using cephadm for the setup, which in my testing works really well.
As for the questions:
1. 3x replication is a tough sell IMO. For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant. But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.)
Hi Matthias Just a partial answer with my small experience. I've built a small ceph cluster with 5 nodes for capacitive storage (at a research team level 😉) so with *HDDs*. I'm using EC profiles and *CephFS* kernel mount on 3 computational clusters (approx 1000 cores aggregated) for data storage of the simulations. Only 4 OSDs per node in this first round, it's low. The network is ethernet 25Gb/s on servers and on clients (same switch, same VLAN) I'm using a 5+3 profile with minsize set to 6 and 2 chunks per node (on 2 different OSD) and it work fine. I can loose one node in production. Of course it is a minimal setup and my goal is to add a node and some OSDs. I've run some comparison with a 2+2 EC profile (*distributing only one chunk per node*) but the I/O performances are significantly lower when writing. Effiiciency with 24 MPI processes writing/reading 4GB each (k=2+m=2 set as the reference) *k=5+m=3* *k=2+m=2* Write 126% 100 % Read 95% 100 % Average performance on 10 runs (10 runs writing with different files names and then 10 runs reading) , unit is MB/s 24x4Go avg k=5+m=3 write 747,72 k=5+m=3 read 1344,00 k=2+m=2 write 595,46 k=2+m=2 read 1418,00 Effiiciency with 8 MPI processes writing/reading 8GB each (k=2+m=2 set as the reference) *k=6+m=2* *k=5+m=3* *k=2+m=2* Write 137% 118% 100 % Read 98% 117% 100 % Average performance on 10 runs (10 runs writing with differents files names and then 10 runs reading), unit is MB/s 8x8Go avg k=6+m=2 write 908,88 k=6+m=2 read 1227,00 k=5+m=3 write 783,10 k=5+m=3 read 1385,00 k=2+m=2 write 665,01 k=2+m=2 read 1252,00 The k=6+m=2 is not secure in production with 5 nodes and 2 chunks per node So I finaly stay on k=5+m=3 which also allows more available storage. Patrick
2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool. What about changing k, m, or the plugin type? My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed?
3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups, and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS? So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway.
4. In v20 there seems to be a new ec_optimizations feature for ec pools used in CephFS or RBD. Are those a good idea with this kind of (large) qcow2 image workload on top of CephFS?
5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that. Should I be waiting for it / start a new cluster with v20 now already?
Thanks for any insights you can give!
Kind regards Matthias Riße
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hey Patrick! Thanks, that's very insightful. We also have HDDs only and 25Gb/s ethernet between the machines, so similar to your setup, except we have >=8 disks per machine. Your workload seems more sequential than what I am expecting from our VMs, so maybe that favors EC for you. Out of curiosity, did you try the same benchmarks with 3x replication too? The raw numbers look promising, what I am seeing currently is about 350MB/s max sequential writes with the local RAID6's we have, which Ceph was able to saturate with a single OSD on a LV for testing. You already get about double of that, and we have twice as many disks, so should get even more. Have you tried 8+4 or 7+5 with 3 OSDs per host too? -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------
On Nov 13, 2025, at 12:25 PM, Matthias Riße <m.risse@fz-juelich.de> wrote:
Hey Patrick! Thanks, that's very insightful.
We also have HDDs only and 25Gb/s ethernet between the machines, so similar to your setup, except we have >=8 disks per machine.
Your workload seems more sequential than what I am expecting from our VMs, so maybe that favors EC for you.
OSDs usually see a random workload regardless, the IO blender effect.
Out of curiosity, did you try the same benchmarks with 3x replication too?
The raw numbers look promising, what I am seeing currently is about 350MB/s max sequential writes with the local RAID6's we have, which Ceph was able to saturate with a single OSD on a LV for testing. You already get about double of that, and we have twice as many disks, so should get even more.
Have you tried 8+4 or 7+5 with 3 OSDs per host too?
-- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Patrick! Thanks, that's very insightful.
We also have HDDs only and 25Gb/s ethernet between the machines, so similar to your setup, except we have >=8 disks per machine.
Your workload seems more sequential than what I am expecting from our VMs, so maybe that favors EC for you. Out of curiosity, did you try the same benchmarks with 3x replication too? Typical use of this storage is infrequent I/O operations, but
Le 13/11/2025 à 18:25, Matthias Riße a écrit : potentially by several hundred processes on several dozen HDF5 files. My nodes can be upgraded up to 12 OSDs/node. Replication is not a strategy for this storage as it increase significantly the strorage cost/TB. I have an other cluster, 3 nodes, running Proxmox and hosting VMs on a replicated Ceph storage. This allows very fast VM migration/restart when a node crashes. But this cluster is quite old (>10 years) and benchmarking will not give any pertinent informations. It will be replaced these next years.
The raw numbers look promising, what I am seeing currently is about 350MB/s max sequential writes with the local RAID6's we have, which Ceph was able to saturate with a single OSD on a LV for testing. You already get about double of that, and we have twice as many disks, so should get even more.
Have you tried 8+4 or 7+5 with 3 OSDs per host too?
No I do not. With this setup each PG will rely on more OSDs and PGs osd will overlap more frequently with few HDDs ? Patrick
On Nov 13, 2025, at 9:58 AM, Matthias Riße <m.risse@fz-juelich.de> wrote:
Hey all!
I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask.
Guten Tag, Herr Riße .
So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph.
Depending on your virtualization strategy, you could do that with Ceph. Either with a standalone cluster external to the virtualization solution, or via Rook / K8s. Labeling etc. could be used to prevent compute pods from being scheduled on the more modest systems.
I'd like to at least be able to take one node at a time out of rotation for maintenance purposes.
Quite prudent, and that’s one reason to use all five nodes for Ceph.
I'd be using cephadm for the setup, which in my testing works really well.
As for the questions:
1. 3x replication is a tough sell IMO.
That depends on your use-case. If you’re using HDDs, then your performance is going to be limited anyway, though.
For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant.
Depending on your use case, I would agree. There is significant value in having at least one more node than the replication factor. You *could* do 3+2, but there are a number of drawbacks on 5 hosts and I would not recommend.
But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.)
It’s not a stupid idea at all, but it’s not the choice I would make. With 2+2, your space amp factor is 2.0: (k+m) / k With 17+7, you have 1.41, but you will find your write speed distinctly impacted, and since HDDs have very limited IOPs, your reads as well. Your scrubs won’t be able to keep up and recovery/backfill will be very slow. 16+8 wouldn’t be much different in many regards. MSR rules are very clever, props to Sam there. There was an analysis a few. years back that concluded that with EC there is a CPU benefit to values of k and m with small prime factors. 17+7 thus would be disadvantageous, though the dynamics may be different with the Fast EC changes coming in Tentacle. Here’s a table showing the space amp factor of various EC profiles. My sense is that the diminishing returns past, say, 8+3 aren’t worth the impact to performance and especially recovery.
2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool.
A delicate operation, but yes.
What about changing k, m, or the plugin type?
That you can’t do. We’ve tried to make this clear in the docs, please let me know if it isn’t.
My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed?
Crossing the streams. All bets are off. But note that you can change the EC profile setting on an existing pool, but that doesn’t actually DO anything.
3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead
libvirt can use librbd quite effectively. Zillions of OpenStack instances and certain well-known VPS providers do just this.
, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups
You might use “rbd export” to feed into such a system. Vultr does just that.
and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS?
I don’t know enough to say for sure, but this idea makes me twitch. I suspect that FUSE mounts wouldn’t be as performant.
So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway.
4. In v20 there seems to be a new ec_optimizations feature for ec pools used in CephFS or RBD. Are those a good idea with this kind of (large) qcow2 image workload on top of CephFS?
The Fast EC code is beneficial to any EC workload, to extents that vary on the EC profile, the access modality, and the workload, but I am not aware of a situation in which they would be deleterious. For production deployment of course you might let 20.2.0 soak for a bit, consider waiting for 20.2.1, the usual dynamics.
5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that.
RSN. Really! It’s sooooo close.
Should I be waiting for it / start a new cluster with v20 now already?
Depends on your time demands. For RBD the Fast EC space amp improvements may not be as dramatic as for, say, RGW with mixed large and small objects. You can always retrofit after an update to Tentacle. EC *speed* improvements will be signifiant, though, and you can retrofit an existing pool.
I take from that that it might be useful if at some point we need bulk storage for large sequential data, but definitely not as VM storage.
That workload typically does a lot of small-block random IO and does not do well with classic EC (or HDDs at all). Fast EC in Tentacle should measurably improve the performance of EC for such workloads.
I actually just learned that we should have another ten 18 TiB drives laying around that we could just slot in, so with that my estimation is that we are more limited on compute than on raw storage with what we have. Which would mean 3x replication after all…
Even with Tentacle, R3 will be faster than 2,2 EC. But note that more OSDs per node means more RAM and CPU consumption. If your existing OSDs are 10TB drives, I strongly suggest taking steps before deploying: mon_target_pg_per_osd = 300 mon_max_pg_per_osd = 600 mgr/balancer/upmap_max_deviation = 1
Just to make sure though, failure domains and device classes /can/ be changed after the fact, right? Might become relevant whenever we add SSDs into the mix.
Yes, absolutely. You can use upmap-remapped.py to moderate the thundering heard of data movement that can result.
Ack. But RBD does not yet have a way to export at least crash-consistent snapshots of a group of images, right? I think I saw that the export functionality for group snapshots is still in development…
In Tentacle I think, or perhaps a Tentacle dot release.
This is really a limitation of the tape backup system we have: I can't push more than about 8 TiB at a time to it, otherwise the connection drops and the backup fails
AMANDA addressed that years ago with its staging filesystem ;)
Thanks for any insights you can give!
Kind regards Matthias Riße
-- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Janne Johansson
-
Matthias Riße
-
Patrick Begou