Question about speeding hdd based cluster
Hello ceph-users, I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools. What would increase the speed of the hdd-based cephfs more, and in what usage scenarios: 1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links. It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck? Thank you! George
Hi Kyriazis, depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd. Joachim Kraftmayer www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi George, the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you all. The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though. What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees. Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks? How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power. Thank you! George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Oct 2, 2024, at 2:19 PM, quaglio@bol.com.br wrote:
Hi Kyriazis, I work with a cluster similar to yours : 142 HDDs and 18 SSDs. I had a lot of performance gains when I made the following settings:
1-) For the pool that is configured on the HDDs (here, home directories are on HDDs), reduce the following replica settings (I don't know what your resilience requirement is): *size=2 * min_size=1
I do this for at least 4 years with no problems (even when there is a need to change discs or reboot a server, this config never got me in trouble).
It is nonetheless risky. The wrong sequence of cascading events, of overlapping failures and you may lose data.
2-) Move the filesystem metadata pools to use at least SSD only.
Absolutely. The CephFS docs suggest using size=4 for the MD pool.
3-) Increase server and client cache. Here I left it like this: osd_memory_target_autotune=true (each OSD always has more than 12G).
For clients: client_cache_size=163840 client_oc_max_dirty=1048576000 client_oc_max_dirty_age=50 client_oc_max_objects=10000 client_oc_size=2097152000 client_oc_target_dirty=838860800
Evaluate, following the documentation, which of these variables makes sense for your cluster.
For the backup scenario, I imagine that decreasing the size and min_size values will change the impact. However, you must evaluate your needs for these settings.
Rafael.
De: "Kyriazis, George" <george.kyriazis@intel.com> Enviada: 2024/10/02 13:06:09 Para: eblock@nde.ag, ceph-users@ceph.io Assunto: [ceph-users] Re: Question about speeding hdd based cluster
Thank you all.
The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though.
What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees.
Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks? How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power.
Thank you!
George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com <http://www.clyso.com/>
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Oct 2, 2024, at 2:27 PM, Anthony D'Atri <aad@dreamsnake.net> wrote: On Oct 2, 2024, at 2:19 PM, quaglio@bol.com.br wrote: Hi Kyriazis, I work with a cluster similar to yours : 142 HDDs and 18 SSDs. I had a lot of performance gains when I made the following settings: 1-) For the pool that is configured on the HDDs (here, home directories are on HDDs), reduce the following replica settings (I don't know what your resilience requirement is): *size=2 * min_size=1 I do this for at least 4 years with no problems (even when there is a need to change discs or reboot a server, this config never got me in trouble). It is nonetheless risky. The wrong sequence of cascading events, of overlapping failures and you may lose data. Our setup is with 3/2. size=3 seems much safer than 2. 2-) Move the filesystem metadata pools to use at least SSD only. Absolutely. The CephFS docs suggest using size=4 for the MD pool. Hmm.. I don’t remember reading that anywhere, but it makes sense. Thanks! George 3-) Increase server and client cache. Here I left it like this: osd_memory_target_autotune=true (each OSD always has more than 12G). For clients: client_cache_size=163840 client_oc_max_dirty=1048576000 client_oc_max_dirty_age=50 client_oc_max_objects=10000 client_oc_size=2097152000 client_oc_target_dirty=838860800 Evaluate, following the documentation, which of these variables makes sense for your cluster. For the backup scenario, I imagine that decreasing the size and min_size values will change the impact. However, you must evaluate your needs for these settings. Rafael. ________________________________ De: "Kyriazis, George" <george.kyriazis@intel.com> Enviada: 2024/10/02 13:06:09 Para: eblock@nde.ag, ceph-users@ceph.io Assunto: [ceph-users] Re: Question about speeding hdd based cluster Thank you all. The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though. What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees. Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks? How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power. Thank you! George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com<http://www.clyso.com/>
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
It is nonetheless risky. The wrong sequence of cascading events, of overlapping failures and you may lose data.
Our setup is with 3/2. size=3 seems much safer than 2.
Indeed that is the default for replicated pools. Additional replicas exhibit diminishing returns in most cases at high cost.
2-) Move the filesystem metadata pools to use at least SSD only.
Absolutely. The CephFS docs suggest using size=4 for the MD pool.
Hmm.. I don’t remember reading that anywhere, but it makes sense.
https://docs.ceph.com/en/quincy/cephfs/createfs/#creating-pools We recommend configuring at least 3 replicas for the metadata pool, as data loss in this pool can render the entire file system inaccessible. Configuring 4 would not be extreme, especially since the metadata pool’s capacity requirements are quite modest.
Thanks!
George
3-) Increase server and client cache. Here I left it like this: osd_memory_target_autotune=true (each OSD always has more than 12G).
For clients: client_cache_size=163840 client_oc_max_dirty=1048576000 client_oc_max_dirty_age=50 client_oc_max_objects=10000 client_oc_size=2097152000 client_oc_target_dirty=838860800
Evaluate, following the documentation, which of these variables makes sense for your cluster.
For the backup scenario, I imagine that decreasing the size and min_size values will change the impact. However, you must evaluate your needs for these settings.
Rafael.
De: "Kyriazis, George" <george.kyriazis@intel.com> Enviada: 2024/10/02 13:06:09 Para: eblock@nde.ag, ceph-users@ceph.io Assunto: [ceph-users] Re: Question about speeding hdd based cluster
Thank you all.
The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though.
What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees.
Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks? How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power.
Thank you!
George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com <http://www.clyso.com/>
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Thank you for that email, and the rest of the chain. I have a sort of similar setup and the suggestions seem to have improved performance already. My first instinct was to let Ceph just use the aggregated IO of all of my motley mix of drives. However, I now have a bunch of found SSDs in the mix moving the index and meta data onto an SSD-only pool gains me more than what I lost moving most of the other random IO off of them. I also created a fast-HDD and slow-HDD pools since I have mostly NetApp 600GB 10Ks and a few random 15Ks from old Dells but also a lot of 4TB 7.2Ks with a few random 5.4s off the recycle pile. The NAS-like data goes on fast disks, the dev/test/backup images on slow. Maybe I’ll merge them back at some point, but the suggestions seems to be helping for now. Thanks! Chris From: quaglio@bol.com.br <quaglio@bol.com.br> Sent: Wednesday, October 2, 2024 1:19 PM To: ceph-users@ceph.io; george.kyriazis@intel.com Subject: [ceph-users] Re: Question about speeding hdd based cluster Hi Kyriazis, I work with a cluster similar to yours : 142 HDDs and 18 SSDs. I had a lot of performance gains when I made the following settings: 1-) For the pool that is configured on the HDDs (here, home directories are on HDDs), reduce the following replica settings (I don't know what your resilience requirement is): *size=2 * min_size=1 I do this for at least 4 years with no problems (even when there is a need to change discs or reboot a server, this config never got me in trouble). 2-) Move the filesystem metadata pools to use at least SSD only. 3-) Increase server and client cache. Here I left it like this: osd_memory_target_autotune=true (each OSD always has more than 12G). For clients: client_cache_size=163840 client_oc_max_dirty=1048576000 client_oc_max_dirty_age=50 client_oc_max_objects=10000 client_oc_size=2097152000 client_oc_target_dirty=838860800 Evaluate, following the documentation, which of these variables makes sense for your cluster. For the backup scenario, I imagine that decreasing the size and min_size values will change the impact. However, you must evaluate your needs for these settings. Rafael. _____ De: "Kyriazis, George" <george.kyriazis@intel.com <mailto:george.kyriazis@intel.com> > Enviada: 2024/10/02 13:06:09 Para: eblock@nde.ag <mailto:eblock@nde.ag> , ceph-users@ceph.io <mailto:ceph-users@ceph.io> Assunto: [ceph-users] Re: Question about speeding hdd based cluster Thank you all. The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though. What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees. Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks? How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power. Thank you! George
-----Original Message----- From: Eugen Block <eblock@nde.ag <mailto:eblock@nde.ag> > Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io <mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com <mailto:joachim.kraftmayer@clyso.com> >:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com <http://www.clyso.com/>
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com <mailto:george.kyriazis@intel.com> > schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Short comment on Replikation size 2: is not the question if you will loose data only when. Joachim Kraftmayer joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 <christopher.colvin@dotdashmdp.com> schrieb am Sa., 16. Nov. 2024, 05:09:
Thank you for that email, and the rest of the chain. I have a sort of similar setup and the suggestions seem to have improved performance already. My first instinct was to let Ceph just use the aggregated IO of all of my motley mix of drives. However, I now have a bunch of found SSDs in the mix moving the index and meta data onto an SSD-only pool gains me more than what I lost moving most of the other random IO off of them. I also created a fast-HDD and slow-HDD pools since I have mostly NetApp 600GB 10Ks and a few random 15Ks from old Dells but also a lot of 4TB 7.2Ks with a few random 5.4s off the recycle pile. The NAS-like data goes on fast disks, the dev/test/backup images on slow. Maybe I’ll merge them back at some point, but the suggestions seems to be helping for now.
Thanks!
Chris
From: quaglio@bol.com.br <quaglio@bol.com.br> Sent: Wednesday, October 2, 2024 1:19 PM To: ceph-users@ceph.io; george.kyriazis@intel.com Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi Kyriazis, I work with a cluster similar to yours : 142 HDDs and 18 SSDs. I had a lot of performance gains when I made the following settings:
1-) For the pool that is configured on the HDDs (here, home directories are on HDDs), reduce the following replica settings (I don't know what your resilience requirement is): *size=2 * min_size=1
I do this for at least 4 years with no problems (even when there is a need to change discs or reboot a server, this config never got me in trouble).
2-) Move the filesystem metadata pools to use at least SSD only.
3-) Increase server and client cache. Here I left it like this: osd_memory_target_autotune=true (each OSD always has more than 12G).
For clients: client_cache_size=163840
client_oc_max_dirty=1048576000
client_oc_max_dirty_age=50 client_oc_max_objects=10000
client_oc_size=2097152000
client_oc_target_dirty=838860800
Evaluate, following the documentation, which of these variables makes sense for your cluster.
For the backup scenario, I imagine that decreasing the size and min_size values will change the impact. However, you must evaluate your needs for these settings.
Rafael.
_____
De: "Kyriazis, George" <george.kyriazis@intel.com <mailto: george.kyriazis@intel.com> > Enviada: 2024/10/02 13:06:09 Para: eblock@nde.ag <mailto:eblock@nde.ag> , ceph-users@ceph.io <mailto: ceph-users@ceph.io> Assunto: [ceph-users] Re: Question about speeding hdd based cluster
Thank you all.
The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though.
What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees.
Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks? How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power.
Thank you!
George
-----Original Message----- From: Eugen Block <eblock@nde.ag <mailto:eblock@nde.ag> > Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io <mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com <mailto: joachim.kraftmayer@clyso.com> >:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com <http://www.clyso.com/>
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com <mailto: george.kyriazis@intel.com> > schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto: ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Absolutely. I’ve seen it happen.
On Nov 16, 2024, at 5:00 AM, Joachim Kraftmayer <joachim.kraftmayer@clyso.com> wrote:
Short comment on Replikation size 2: is not the question if you will loose data only when.
Joachim Kraftmayer
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
<christopher.colvin@dotdashmdp.com> schrieb am Sa., 16. Nov. 2024, 05:09:
Hi, response inline. Zitat von "Kyriazis, George" <george.kyriazis@intel.com>:
Thank you all.
The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though.
What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees.
Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks?
Moving the metadata to SSDs first is absolutely reasonable and relatively cheap since it usually doesn't contain huge amounts of data.
How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power.
The answer is as always "it depends". If you rebuild the OSDs entirely (host-wise) instead of migrating the DB off to SSDs, you might encounter slow requests as you already noticed yourself. But the whole process would be faster than migrating each DB individually. If you take the migration approach, it would be less invasive, each OSD would just have to catch up after restart, reducing the load drastically compared to a rebuild. But then again, it would take way more time to complete. How large are the OSDs and how much are they utilized? Do you have some history how long a host rebuild would usually take?
Thank you!
George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Oct 7, 2024, at 2:16 AM, Eugen Block <eblock@nde.ag> wrote:
Hi, response inline.
Zitat von "Kyriazis, George" <george.kyriazis@intel.com>:
Thank you all.
The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though.
What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees.
Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks?
Moving the metadata to SSDs first is absolutely reasonable and relatively cheap since it usually doesn't contain huge amounts of data.
How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power.
The answer is as always "it depends". If you rebuild the OSDs entirely (host-wise) instead of migrating the DB off to SSDs, you might encounter slow requests as you already noticed yourself. But the whole process would be faster than migrating each DB individually. If you take the migration approach, it would be less invasive, each OSD would just have to catch up after restart, reducing the load drastically compared to a rebuild. But then again, it would take way more time to complete. How large are the OSDs and how much are they utilized? Do you have some history how long a host rebuild would usually take?
I have no problem destroying and re-creating the OSDs (in place) if that’s what it takes. It will take time to do them all, but if “eventually” it works better, then so be it. Do you happen to have a documentation pointer no how to migrate DB to SSDs? On a similar topic: Does it make sense to use compression on a metadata pool? Would it matter if the metadata pool is on hdd vs ssd? Thank you! George
Thank you!
George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sure: https://docs.ceph.com/en/latest/ceph-volume/lvm/newdb/ In this case you'll have to prepare the db LV beforehand. I haven't done that in a while, here's an example from Clyso: https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for... Note that in a cephadm deployment you'll need to execute that in a shell, for example: cephadm shell --name osd.6 --env CEPH_ARGS='--bluestore_block_db_size=1341967564' -- ceph-bluestore-tool bluefs-bdev-new-db --dev-target /dev/data_vg1/lv4 --path /var/lib/ceph/osd/ceph-6 Note that these are two different approaches to achieve the same goal. One is via 'ceph-volume lvm new-db', the other one with 'ceph-bluestore-tool bluefs-bdev-new-db'. I would assume they both work, so I can't tell which one to prefer. I feel like the docs could use some clarification on this topic.
On a similar topic: Does it make sense to use compression on a metadata pool? Would it matter if the metadata pool is on hdd vs ssd?
As already stated, metadata should be on fast devices, independent of compression. The metadata pool doesn't consume a lot of data, so I'd say there's not too much of a benefit compressing that. Zitat von "Kyriazis, George" <george.kyriazis@intel.com>:
On Oct 7, 2024, at 2:16 AM, Eugen Block <eblock@nde.ag> wrote:
Hi, response inline.
Zitat von "Kyriazis, George" <george.kyriazis@intel.com>:
Thank you all.
The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though.
What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees.
Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks?
Moving the metadata to SSDs first is absolutely reasonable and relatively cheap since it usually doesn't contain huge amounts of data.
How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power.
The answer is as always "it depends". If you rebuild the OSDs entirely (host-wise) instead of migrating the DB off to SSDs, you might encounter slow requests as you already noticed yourself. But the whole process would be faster than migrating each DB individually. If you take the migration approach, it would be less invasive, each OSD would just have to catch up after restart, reducing the load drastically compared to a rebuild. But then again, it would take way more time to complete. How large are the OSDs and how much are they utilized? Do you have some history how long a host rebuild would usually take?
I have no problem destroying and re-creating the OSDs (in place) if that’s what it takes. It will take time to do them all, but if “eventually” it works better, then so be it. Do you happen to have a documentation pointer no how to migrate DB to SSDs?
On a similar topic: Does it make sense to use compression on a metadata pool? Would it matter if the metadata pool is on hdd vs ssd?
Thank you!
George
Thank you!
George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
(Resending, apparently my mail didn't get to the ML) Sure: https://docs.ceph.com/en/latest/ceph-volume/lvm/newdb/ In this case you'll have to prepare the db LV beforehand. I haven't done that in a while, here's an example from Clyso: https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for... Note that in a cephadm deployment you'll need to execute that in a shell, for example: cephadm shell --name osd.6 --env CEPH_ARGS='--bluestore_block_db_size=1341967564' -- ceph-bluestore-tool bluefs-bdev-new-db --dev-target /dev/data_vg1/lv4 --path /var/lib/ceph/osd/ceph-6 and don't forget to migrate the data, otherwise you might encounter spillover: cephadm shell --name osd.6 -- ceph-bluestore-tool bluefs-bdev-migrate --dev-target /var/lib/ceph/osd/ceph-6/block.db --path /var/lib/ceph/osd/ceph-6 --devs-source /var/lib/ceph/osd/ceph-6/block Note that these are two different approaches to achieve the same goal. One is via 'ceph-volume lvm new-db', the other one with 'ceph-bluestore-tool bluefs-bdev-new-db'. I would assume they both work, so I can't tell which one to prefer. I feel like the docs could use some clarification on this topic.
On a similar topic: Does it make sense to use compression on a metadata pool? Would it matter if the metadata pool is on hdd vs ssd?
As already stated, metadata should be on fast devices, independent of compression. The metadata pool doesn't consume a lot of data, so I'd say there's not too much of a benefit compressing that. Zitat von "Kyriazis, George" <george.kyriazis@intel.com>:
On Oct 7, 2024, at 2:16 AM, Eugen Block <eblock@nde.ag> wrote:
Hi, response inline.
Zitat von "Kyriazis, George" <george.kyriazis@intel.com>:
Thank you all.
The cluster is used mostly for backup of large files currently, but we are hoping to use it for home directories (compiles, etc.) soon. Most usage would be for large files, though.
What I've observed with its current usage is that ceph rebalances, and proxmox-initiated VM backups bring the storage to its knees.
Would a safe approach be to move the metadata pool to ssd first, see how it goes (since it would be cheaper), and then add DB/WAL disks?
Moving the metadata to SSDs first is absolutely reasonable and relatively cheap since it usually doesn't contain huge amounts of data.
How would ceph behave if we are adding DB/WAL disks "slowly" (ie one node at a time)? We have about 100 OSDs (mix hdd/ssd) spread across about 25 hosts. Hosts are server-grade with plenty of memory and processing power.
The answer is as always "it depends". If you rebuild the OSDs entirely (host-wise) instead of migrating the DB off to SSDs, you might encounter slow requests as you already noticed yourself. But the whole process would be faster than migrating each DB individually. If you take the migration approach, it would be less invasive, each OSD would just have to catch up after restart, reducing the load drastically compared to a rebuild. But then again, it would take way more time to complete. How large are the OSDs and how much are they utilized? Do you have some history how long a host rebuild would usually take?
I have no problem destroying and re-creating the OSDs (in place) if that’s what it takes. It will take time to do them all, but if “eventually” it works better, then so be it. Do you happen to have a documentation pointer no how to migrate DB to SSDs?
On a similar topic: Does it make sense to use compression on a metadata pool? Would it matter if the metadata pool is on hdd vs ssd?
Thank you!
George
Thank you!
George
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, October 2, 2024 2:18 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Question about speeding hdd based cluster
Hi George,
the docs [0] strongly recommend to have dedicated SSD or NVMe OSDs for the metadata pool. You'll also benefit from dedicated DB/WAL devices. But as Joachim already stated, it depends on a couple of factors like the number of clients, the load they produce, file sizes etc. There's no easy answer.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/cephfs/createfs/#creating-pools
Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi Kyriazis,
depends on the workload. I would recommend to add ssd/nvme DB/WAL to each osd.
Joachim Kraftmayer
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Kyriazis, George <george.kyriazis@intel.com> schrieb am Mi., 2. Okt. 2024, 07:37:
Hello ceph-users,
I’ve been wondering…. I have a proxmox hdd-based cephfs pool with no DB/WAL drives. I also have ssd drives in this setup used for other pools.
What would increase the speed of the hdd-based cephfs more, and in what usage scenarios:
1. Adding ssd/nvme DB/WAL drives for each node 2. Moving the metadata pool for my cephfs to ssd 3. Increasing the performance of the network. I currently have 10gbe links.
It doesn’t look like the network is currently saturated, so I’m thinking (3) is not a solution. However, if I choose any of the other options, would I need to also upgrade the network so that the network does not become a bottleneck?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Anthony D'Atri
-
christopher.colvin＠dotdashmdp.com
-
Eugen Block
-
Joachim Kraftmayer
-
Kyriazis, George
-
quaglio＠bol.com.br