We are planning our ceph architecture and I have a question: How should NVMe drives be used when our spinning storage devices use bluestore: 1. block WAL and DB partitions (https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref...) 2. Cache tier (https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/) 3. Something else? Hardware- Each node has: 3x 8 TB HDD 1x 450 GB NVMe drive 192 GB RAM 2x Xeon CPUs (24 cores total) I plan to have three OSD daemons running on the node. There are 95 nodes total with the same hardware. Use Case: The plan is create cephfs and use this filesystem to store people's home directories and data. I anticipate more read operations than writes. Regarding cache tiering: The online documentation says cache tiering will often degrade performance. But when I read various threads on this ML there do seem to be people using cache tiering with success. I do see that it is heavily dependent upon one's use-case. In 2019 is there any updated recommendations as to whether to use cache tiering? If there is a third suggestion that people have I would be interested in hearing it. Thanks in advance. Sincerely, Shawn Kwang
On Wed, Sep 18, 2019 at 11:47 AM Shawn A Kwang <kwangs@uwm.edu> wrote:
We are planning our ceph architecture and I have a question:
How should NVMe drives be used when our spinning storage devices use bluestore:
1. block WAL and DB partitions (https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref...) 2. Cache tier (https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/) 3. Something else?
Hardware- Each node has: 3x 8 TB HDD 1x 450 GB NVMe drive 192 GB RAM 2x Xeon CPUs (24 cores total)
I plan to have three OSD daemons running on the node. There are 95 nodes total with the same hardware.
Use Case:
The plan is create cephfs and use this filesystem to store people's home directories and data. I anticipate more read operations than writes.
Regarding cache tiering: The online documentation says cache tiering will often degrade performance. But when I read various threads on this ML there do seem to be people using cache tiering with success. I do see that it is heavily dependent upon one's use-case. In 2019 is there any updated recommendations as to whether to use cache tiering?
If there is a third suggestion that people have I would be interested in hearing it. Thanks in advance.
I've had good success when I've been able to hold all the 'hot' data for 24 hours in a cache tier. That reduces the amount of data being evicted from the tier and being added to the tier such that you reduce the penalty from those operations. You can adjust the config (hit rate, etc) to help reduce promotions for rarely accessed objects. The size of the NVMe drive may best be suited for WAL (I highly recommend that for any HDD install) for each OSD, then carve out the rest as an SSD pool that you can put the CephFS metadata pool on. I don't think you would have a good experience with cache tier at that size. However, you know your access patterns far better than I do and it may be a good fit. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On 9/23/19 9:38 AM, Robert LeBlanc wrote:
On Wed, Sep 18, 2019 at 11:47 AM Shawn A Kwang <kwangs@uwm.edu> wrote:
We are planning our ceph architecture and I have a question:
How should NVMe drives be used when our spinning storage devices use bluestore:
1. block WAL and DB partitions (https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref...) 2. Cache tier (https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/) 3. Something else?
Hardware- Each node has: 3x 8 TB HDD 1x 450 GB NVMe drive 192 GB RAM 2x Xeon CPUs (24 cores total)
I plan to have three OSD daemons running on the node. There are 95 nodes total with the same hardware.
Use Case:
The plan is create cephfs and use this filesystem to store people's home directories and data. I anticipate more read operations than writes.
Regarding cache tiering: The online documentation says cache tiering will often degrade performance. But when I read various threads on this ML there do seem to be people using cache tiering with success. I do see that it is heavily dependent upon one's use-case. In 2019 is there any updated recommendations as to whether to use cache tiering?
If there is a third suggestion that people have I would be interested in hearing it. Thanks in advance.
I've had good success when I've been able to hold all the 'hot' data for 24 hours in a cache tier. That reduces the amount of data being evicted from the tier and being added to the tier such that you reduce the penalty from those operations. You can adjust the config (hit rate, etc) to help reduce promotions for rarely accessed objects. The size of the NVMe drive may best be suited for WAL (I highly recommend that for any HDD install) for each OSD, then carve out the rest as an SSD pool that you can put the CephFS metadata pool on. I don't think you would have a good experience with cache tier at that size. However, you know your access patterns far better than I do and it may be a good fit.
Robert, I like your idea of partitioning each SSD for bluestore's DB [1], and then extra space for the cephFS metadata pool. [1] Question: You wrote 'WAL', but did you mean block.wal or block.db? Or both? Sincerely, Shawn -- Associate Scientist Center for Gravitation, Cosmology, and Astrophysics University of Wisconsin-Milwaukee office: +1 414 229 4960 kwangs@uwm.edu
On Mon, Sep 23, 2019 at 10:51 AM Shawn A Kwang <kwangs@uwm.edu> wrote:
On 9/23/19 9:38 AM, Robert LeBlanc wrote:
On Wed, Sep 18, 2019 at 11:47 AM Shawn A Kwang <kwangs@uwm.edu> wrote:
We are planning our ceph architecture and I have a question:
How should NVMe drives be used when our spinning storage devices use bluestore:
1. block WAL and DB partitions (https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref...) 2. Cache tier (https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/) 3. Something else?
Hardware- Each node has: 3x 8 TB HDD 1x 450 GB NVMe drive 192 GB RAM 2x Xeon CPUs (24 cores total)
I plan to have three OSD daemons running on the node. There are 95 nodes total with the same hardware.
Use Case:
The plan is create cephfs and use this filesystem to store people's home directories and data. I anticipate more read operations than writes.
Regarding cache tiering: The online documentation says cache tiering will often degrade performance. But when I read various threads on this ML there do seem to be people using cache tiering with success. I do see that it is heavily dependent upon one's use-case. In 2019 is there any updated recommendations as to whether to use cache tiering?
If there is a third suggestion that people have I would be interested in hearing it. Thanks in advance.
I've had good success when I've been able to hold all the 'hot' data for 24 hours in a cache tier. That reduces the amount of data being evicted from the tier and being added to the tier such that you reduce the penalty from those operations. You can adjust the config (hit rate, etc) to help reduce promotions for rarely accessed objects. The size of the NVMe drive may best be suited for WAL (I highly recommend that for any HDD install) for each OSD, then carve out the rest as an SSD pool that you can put the CephFS metadata pool on. I don't think you would have a good experience with cache tier at that size. However, you know your access patterns far better than I do and it may be a good fit.
Robert,
I like your idea of partitioning each SSD for bluestore's DB [1], and then extra space for the cephFS metadata pool.
[1] Question: You wrote 'WAL', but did you mean block.wal or block.db? Or both?
Yes, DB + WAL is what I meant, sorry. If you specify DB, then WAL will go with it automatically. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
participants (2)
-
Robert LeBlanc
-
Shawn A Kwang