Constant write load on 4 node ceph cluster
Hi all We have a 4 node ceph cluster that runs generally fine. It is the storage backend for our virtualization cluster with Proxmox, that runs about 40 virtual machines (80% various Linux Servers). Now that we have implemented monitoring, i see that there is a quite constant write load of about 4-10MB/s throughout the whole day and even weekends, while read load clearly is bound to daily work hours. I am wondering whats up with this. Is this normal? Outside working hours, there is almost no read load, well under 1MB/s, while write Load stays relatively constant at around 5MB/s. During rebalancing of the Cluster, normal write load decreases significantly to under 1MB/s. Overall performance is not greatly affected by rebalancing as it is tuned to relatively slow backfills (osd_max_backfills = '3' | osd_recovery_max_active = '3') After rebalance has been finished, the average write load of about 5MB/s reappears. I have attached a little screenshot for reference. Can someone shed a little light on this? Thanks. kind regards Ingo Schmidt IT-Department Island Municipality Langeoog with in-house operations Tourismusservice and Schiffahrt Hauptstraße 28 26465 Langeoog Deutschland [ http://www.langeoog.de/ ]
Is the storage being used for the whole VM disk? If so have you checked none of your software is writing constant log's? Or something that could continuously write to disk. If your running a new version you can use : https://docs.ceph.com/docs/mimic/mgr/iostat/ to locate the exact RBD image. ---- On Mon, 14 Oct 2019 20:32:53 +0800 Ingo Schmidt <i.schmidt@langeoog.de> wrote ---- Hi all We have a 4 node ceph cluster that runs generally fine. It is the storage backend for our virtualization cluster with Proxmox, that runs about 40 virtual machines (80% various Linux Servers). Now that we have implemented monitoring, i see that there is a quite constant write load of about 4-10MB/s throughout the whole day and even weekends, while read load clearly is bound to daily work hours. I am wondering whats up with this. Is this normal? Outside working hours, there is almost no read load, well under 1MB/s, while write Load stays relatively constant at around 5MB/s. During rebalancing of the Cluster, normal write load decreases significantly to under 1MB/s. Overall performance is not greatly affected by rebalancing as it is tuned to relatively slow backfills (osd_max_backfills = '3' | osd_recovery_max_active = '3') After rebalance has been finished, the average write load of about 5MB/s reappears. I have attached a little screenshot for reference. Can someone shed a little light on this? Thanks. kind regards Ingo Schmidt IT-Department Island Municipality Langeoog with in-house operations Tourismusservice and Schiffahrt Hauptstraße 28 26465 Langeoog Deutschland [ http://www.langeoog.de/ ] _______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io
Great, this helped a lot. Although "ceph iostat" didn't give iostats of single images, but just general overview of IO, i remembered the new nautilus RDB performance monitoring. https://ceph.com/rbd/new-in-nautilus-rbd-performance-monitoring/ With a "simple"
rbd perf image iotop i was able to see that the writes indeed are from the Log Server and the Zabbix Monitoring Server. I didn't expect that it would cause that much I/O... unbelieveable...
----- Ursprüngliche Mail ----- Von: "Ashley Merrick" <singapore@amerrick.co.uk> An: "i schmidt" <i.schmidt@langeoog.de> CC: "ceph-users" <ceph-users@ceph.io> Gesendet: Montag, 14. Oktober 2019 15:20:46 Betreff: Re: [ceph-users] Constant write load on 4 node ceph cluster Is the storage being used for the whole VM disk? If so have you checked none of your software is writing constant log's? Or something that could continuously write to disk. If your running a new version you can use : [ https://docs.ceph.com/docs/mimic/mgr/iostat/ | https://docs.ceph.com/docs/mimic/mgr/iostat/ ] to locate the exact RBD image.
It's pretty common to see way more writes than reads if you got lots of idle VMs Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 6:34 PM Ingo Schmidt <i.schmidt@langeoog.de> wrote:
Great, this helped a lot. Although "ceph iostat" didn't give iostats of single images, but just general overview of IO, i remembered the new nautilus RDB performance monitoring.
https://ceph.com/rbd/new-in-nautilus-rbd-performance-monitoring/
With a "simple"
rbd perf image iotop i was able to see that the writes indeed are from the Log Server and the Zabbix Monitoring Server. I didn't expect that it would cause that much I/O... unbelieveable...
----- Ursprüngliche Mail ----- Von: "Ashley Merrick" <singapore@amerrick.co.uk> An: "i schmidt" <i.schmidt@langeoog.de> CC: "ceph-users" <ceph-users@ceph.io> Gesendet: Montag, 14. Oktober 2019 15:20:46 Betreff: Re: [ceph-users] Constant write load on 4 node ceph cluster
Is the storage being used for the whole VM disk?
If so have you checked none of your software is writing constant log's? Or something that could continuously write to disk.
If your running a new version you can use : [ https://docs.ceph.com/docs/mimic/mgr/iostat/ | https://docs.ceph.com/docs/mimic/mgr/iostat/ ] to locate the exact RBD image.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Ashley Merrick
-
Ingo Schmidt
-
Paul Emmerich