Activate Cache Tier on Running Pools
Hi, I'm using Ceph in combination with Openstack. For the "VMs" Pool I'd like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ . Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything. Best Robert
Have you checked that the user/keys that your VMs are connecting to have access rights to the cache pool? ---- On Mon, 16 Sep 2019 17:36:38 +0800 Eikermann, Robert <eikermann@se-rwth.de> wrote ---- Hi, I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ . Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything. Best Robert _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello on 2019/9/16 17:36, Eikermann, Robert wrote:
Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything.
After enable cache tier, did you test to reboot a VM to make it become effective? regards.
On 9/16/19 11:36 AM, Eikermann, Robert wrote:
Hi,
I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/.
Can you explain why? The cache tiering has some serious flaws and can even decrease performance instead of improve it. What are you trying to solve? Wido
Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything.
Best
Robert
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We have terrible IO performance when multiple VMs do some file IO. Mainly do some java compilation on that servers. If we have 2 parallel jobs everything is fine, but having 10 jobs we see the warning "HEALTH_WARN X requests are blocked > 32 sec; Y osds have slow requests". I have two enterprise SSDs which gained good results in ceph tested with fio. They are too small to have a separate "ssd_vms" pool and advertise it in openstack as a separate storage backend -- ----------------------------------------------------------------- Robert Eikermann M.Sc.RWTH | Software Engineering Lehrstuhl für Software Engineering | RWTH Aachen University Ahornstr. 55, 52074 Aachen, Germany | http://www.se-rwth.de<http://www.se-rwth.de/> Phone ++49 241 80-21306 / Fax -22218 | Von: Wido den Hollander [mailto:wido@42on.com] Gesendet: Montag, 16. September 2019 11:52 An: Eikermann, Robert <eikermann@se-rwth.de>; ceph-users@ceph.io Betreff: Re: [ceph-users] Activate Cache Tier on Running Pools On 9/16/19 11:36 AM, Eikermann, Robert wrote: Hi, I'm using Ceph in combination with Openstack. For the "VMs" Pool I'd like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ . Can you explain why? The cache tiering has some serious flaws and can even decrease performance instead of improve it. What are you trying to solve? Wido Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything. Best Robert _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
I hope the data your running the CEPH server isn't important if your looking to run a Cache tier with just 2 SSDS / Replication of 2. If your cache tier fails, you basically corrupt most data on the pool below. Also as Wido said, as much as you may get it to work, I don't think it will give you the performance you expected, can you not create a separate small SSD pool as a scratch disk for the VM's for when they are doing heavy temp I/O? Leaving your current data and setup untouched. ---- On Mon, 16 Sep 2019 18:21:47 +0800 Eikermann, Robert <mailto:eikermann@se-rwth.de> wrote ---- We have terrible IO performance when multiple VMs do some file IO. Mainly do some java compilation on that servers. If we have 2 parallel jobs everything is fine, but having 10 jobs we see the warning “HEALTH_WARN X requests are blocked > 32 sec; Y osds have slow requests”. I have two enterprise SSDs which gained good results in ceph tested with fio. They are too small to have a separate “ssd_vms” pool and advertise it in openstack as a separate storage backend -- ----------------------------------------------------------------- Robert Eikermann M.Sc.RWTH | Software Engineering Lehrstuhl für Software Engineering | RWTH Aachen University Ahornstr. 55, 52074 Aachen, Germany | http://www.se-rwth.de/ Phone ++49 241 80-21306 / Fax -22218 | Von: Wido den Hollander [mailto:mailto:wido@42on.com] Gesendet: Montag, 16. September 2019 11:52 An: Eikermann, Robert <mailto:eikermann@se-rwth.de>; mailto:ceph-users@ceph.io Betreff: Re: [ceph-users] Activate Cache Tier on Running Pools On 9/16/19 11:36 AM, Eikermann, Robert wrote: Hi, I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to enable writeback caching tier, like described here: https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ . Can you explain why? The cache tiering has some serious flaws and can even decrease performance instead of improve it. What are you trying to solve? Wido Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything. Best Robert _______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io
Hi! Cache tiering is a great solution if the cache size is larger than the hot data. Even better if the data can cool quietly in the cache. Otherwise, it’s really better not to do this. ----- Original Message -----
From: "Wido den Hollander" <wido@42on.com> To: "Eikermann, Robert" <eikermann@se-rwth.de>, ceph-users@ceph.io Sent: Monday, 16 September, 2019 12:51:45 Subject: [ceph-users] Re: Activate Cache Tier on Running Pools
On 9/16/19 11:36 AM, Eikermann, Robert wrote:
Hi,
I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to enable writeback caching tier, like described here: [ https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ | https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ ] .
Can you explain why? The cache tiering has some serious flaws and can even decrease performance instead of improve it.
What are you trying to solve?
Wido
Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything.
Best
Robert
_______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That would be the case for me. I think all data during one day would fit into the cache, and we could slowly flush back over night (or even over the weekend). But my impression is, I would have to test it. So my initial question: Do I have to stop all VMs before activating the cache? And restart one by one with the cache enabled? -- ----------------------------------------------------------------- Robert Eikermann M.Sc.RWTH | Software Engineering Lehrstuhl für Software Engineering | RWTH Aachen University Ahornstr. 55, 52074 Aachen, Germany | http://www.se-rwth.de Phone ++49 241 80-21306 / Fax -22218 | -----Ursprüngliche Nachricht----- Von: Fyodor Ustinov [mailto:ufm@ufm.su] Gesendet: Montag, 16. September 2019 13:00 An: Wido den Hollander <wido@42on.com> Cc: Eikermann, Robert <eikermann@se-rwth.de>; ceph-users@ceph.io Betreff: Re: [ceph-users] Re: Activate Cache Tier on Running Pools Hi! Cache tiering is a great solution if the cache size is larger than the hot data. Even better if the data can cool quietly in the cache. Otherwise, it’s really better not to do this. ----- Original Message -----
From: "Wido den Hollander" <wido@42on.com> To: "Eikermann, Robert" <eikermann@se-rwth.de>, ceph-users@ceph.io Sent: Monday, 16 September, 2019 12:51:45 Subject: [ceph-users] Re: Activate Cache Tier on Running Pools
On 9/16/19 11:36 AM, Eikermann, Robert wrote:
Hi,
I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to enable writeback caching tier, like described here: [ https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ | https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ ] .
Can you explain why? The cache tiering has some serious flaws and can even decrease performance instead of improve it.
What are you trying to solve?
Wido
Should it be possible to do that on a running pool? I tried to do so and immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly filesystems. No errors were shown in ceph (but also no traffic arrived after enabling the cache tier). Removing the cache tier , rebooting the VMs and doing a filesystemcheck repaired everything.
Best
Robert
_______________________________________________ ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users@ceph.io ] To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io | ceph-users-leave@ceph.io ]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Robert, As long as you triple-check permissions on the cache tier (should be the same as your actual storage pool) you should be fine. In our setup I applied this a few times. The first time I made the assumption permissions would be inherited or not applicable but IOPs get redirected towards the tier and therefore permissions should be sane. It stopped all IOPs until I removed the overlay again. But alas; some VMs died already. Later on I extra checked the permissions and all was well. K. On 16-09-2019 13:12, Eikermann, Robert wrote:
That would be the case for me. I think all data during one day would fit into the cache, and we could slowly flush back over night (or even over the weekend). But my impression is, I would have to test it. So my initial question: Do I have to stop all VMs before activating the cache? And restart one by one with the cache enabled?
Hi, this solved my problem! I forgot the permissions on the cache pool Best Robert -----Ursprüngliche Nachricht----- Von: Kees Meijs <kees@nefos.nl> Gesendet: Montag, 16. September 2019 13:21 An: Eikermann, Robert <eikermann@se-rwth.de>; Fyodor Ustinov <ufm@ufm.su>; Wido den Hollander <wido@42on.com> Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Re: Activate Cache Tier on Running Pools Hi Robert, As long as you triple-check permissions on the cache tier (should be the same as your actual storage pool) you should be fine. In our setup I applied this a few times. The first time I made the assumption permissions would be inherited or not applicable but IOPs get redirected towards the tier and therefore permissions should be sane. It stopped all IOPs until I removed the overlay again. But alas; some VMs died already. Later on I extra checked the permissions and all was well. K. On 16-09-2019 13:12, Eikermann, Robert wrote:
That would be the case for me. I think all data during one day would fit into the cache, and we could slowly flush back over night (or even over the weekend). But my impression is, I would have to test it. So my initial question: Do I have to stop all VMs before activating the cache? And restart one by one with the cache enabled?
participants (6)
-
Ashley Merrick
-
Eikermann, Robert
-
Fyodor Ustinov
-
Kees Meijs
-
Wesley Peng
-
Wido den Hollander