Write i/o in CephFS metadata pool
Hi! I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool. The filesystem is otherwise stable and I'm seeing no usage issues. I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool. For example: # ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live. I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment. Some of it may be expected in when considering details of my environment and CephFS implementation specifics, so please ask away if more details are needed. With my experience using NFS, I would start by looking at client io stats, like `nfsstat` and tuning e.g. mount options, but I haven't been able to find such statistics for CephFS clients. Is there anything of the sort for CephFS? Are similar stats obtainable in some other way? This might be a somewhat broad question and shallow description, so yeah, let me know if there's anything you would like more details on. Thanks a lot, Samy
On 29/01/2020 10:24, Samy Ascha wrote:
I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool.
The filesystem is otherwise stable and I'm seeing no usage issues.
I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool.
[...]
This might be a somewhat broad question and shallow description, so yeah, let me know if there's anything you would like more details on.
No explanation, but chiming in, as I've seen something similar happen on my single node "cluster" at home, where I'm exposing a cephfs through Samba using vfs_ceph, mostly for time machine backups. Running ceph 14.2.6 on debian buster. I can easily perform debugging operations there, no SLA in place :) Jasper
Sammy; I had a thought; since you say the FS has high read activity, but you're seeing large write I/O... is it possible that this is related to atime (Linux last access time)? If I remember my Linux FS basics, atime is stored in the file entry for the file in the directory, and I believe directory information is stored in the metadata pool (dentries?). As a test; you might try mounting the CephFS with the noatime flag. Then see if the write I/O is reduced. I honestly don't know if CephFS supports atime, but I would expect it would. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Samy Ascha [mailto:samy@xel.nl] Sent: Wednesday, January 29, 2020 2:25 AM To: ceph-users@ceph.io Subject: [ceph-users] Write i/o in CephFS metadata pool Hi! I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool. The filesystem is otherwise stable and I'm seeing no usage issues. I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool. For example: # ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live. I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment. Some of it may be expected in when considering details of my environment and CephFS implementation specifics, so please ask away if more details are needed. With my experience using NFS, I would start by looking at client io stats, like `nfsstat` and tuning e.g. mount options, but I haven't been able to find such statistics for CephFS clients. Is there anything of the sort for CephFS? Are similar stats obtainable in some other way? This might be a somewhat broad question and shallow description, so yeah, let me know if there's anything you would like more details on. Thanks a lot, Samy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dominic, I should have mentioned that I've set noatime already. I have not found any obvious other mount options that would contribute to 'write on read' behaviour.. Thx Samy
On 29 Jan 2020, at 15:43, DHilsbos@performair.com wrote:
Sammy;
I had a thought; since you say the FS has high read activity, but you're seeing large write I/O... is it possible that this is related to atime (Linux last access time)? If I remember my Linux FS basics, atime is stored in the file entry for the file in the directory, and I believe directory information is stored in the metadata pool (dentries?).
As a test; you might try mounting the CephFS with the noatime flag. Then see if the write I/O is reduced.
I honestly don't know if CephFS supports atime, but I would expect it would.
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Samy Ascha [mailto:samy@xel.nl] Sent: Wednesday, January 29, 2020 2:25 AM To: ceph-users@ceph.io Subject: [ceph-users] Write i/o in CephFS metadata pool
Hi!
I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool.
The filesystem is otherwise stable and I'm seeing no usage issues.
I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool.
For example:
# ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr
pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr
I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live.
I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment.
Some of it may be expected in when considering details of my environment and CephFS implementation specifics, so please ask away if more details are needed.
With my experience using NFS, I would start by looking at client io stats, like `nfsstat` and tuning e.g. mount options, but I haven't been able to find such statistics for CephFS clients.
Is there anything of the sort for CephFS? Are similar stats obtainable in some other way?
This might be a somewhat broad question and shallow description, so yeah, let me know if there's anything you would like more details on.
Thanks a lot, Samy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Jan 29, 2020 at 1:25 AM Samy Ascha <samy@xel.nl> wrote:
Hi!
I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool.
The filesystem is otherwise stable and I'm seeing no usage issues.
I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool.
For example:
# ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr
pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr
I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live.
I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment.
The MDS is still writing its journal and updating the "open file table". The MDS needs to record certain information about the state of its cache and the state issued to clients. Even if the clients aren't changing anything. (This is workload dependent but will be most obvious when clients are opening files _not_ in cache already.) -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 2 Feb 2020, at 12:45, Patrick Donnelly <pdonnell@redhat.com> wrote:
On Wed, Jan 29, 2020 at 1:25 AM Samy Ascha <samy@xel.nl> wrote:
Hi!
I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool.
The filesystem is otherwise stable and I'm seeing no usage issues.
I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool.
For example:
# ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr
pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr
I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live.
I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment.
The MDS is still writing its journal and updating the "open file table". The MDS needs to record certain information about the state of its cache and the state issued to clients. Even if the clients aren't changing anything. (This is workload dependent but will be most obvious when clients are opening files _not_ in cache already.)
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi Patrick, Thanks for this extra information. I should be able to confirm this by checking network traffic flowing from the MDSes to the OSDs, and compare it to what's coming in from the CephFS clients. I'll report back when I have more information on that. I'm a little caught up in other stuff right now, but I wanted to just acknowledge your message. Samy
On 4 Feb 2020, at 16:14, Samy Ascha <samy@xel.nl> wrote:
On 2 Feb 2020, at 12:45, Patrick Donnelly <pdonnell@redhat.com> wrote:
On Wed, Jan 29, 2020 at 1:25 AM Samy Ascha <samy@xel.nl> wrote:
Hi!
I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool.
The filesystem is otherwise stable and I'm seeing no usage issues.
I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool.
For example:
# ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr
pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr
I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live.
I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment.
The MDS is still writing its journal and updating the "open file table". The MDS needs to record certain information about the state of its cache and the state issued to clients. Even if the clients aren't changing anything. (This is workload dependent but will be most obvious when clients are opening files _not_ in cache already.)
-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Hi Patrick,
Thanks for this extra information.
I should be able to confirm this by checking network traffic flowing from the MDSes to the OSDs, and compare it to what's coming in from the CephFS clients.
I'll report back when I have more information on that. I'm a little caught up in other stuff right now, but I wanted to just acknowledge your message.
Samy
Hi! I've confirmed that the write IO to the metadata pool is coming form active MDSes. I'm experiencing very poor write performance on clients and I would like to see if there's anything I can do to optimise the performance. Right now, I'm specifically focussing on speeding up this use case: In CephFS mounted dir: $ time unzip -q wordpress-seo.12.9.1.zip real 0m47.596s user 0m0.218s sys 0m0.157s On RBD mount: $ time unzip -q wordpress-seo.12.9.1.zip real 0m0.176s user 0m0.131s sys 0m0.045s The difference is just too big. I'm having real trouble finding a good reference to check my setup for bad configuration etc. I have network bandwidth, RAM and CPU to spare, but I'm unsure on how to put it to work to help my case. Thanks a lot, Samy
Hi!
I've confirmed that the write IO to the metadata pool is coming form active MDSes.
I'm experiencing very poor write performance on clients and I would like to see if there's anything I can do to optimise the performance.
Right now, I'm specifically focussing on speeding up this use case:
In CephFS mounted dir:
$ time unzip -q wordpress-seo.12.9.1.zip
real 0m47.596s user 0m0.218s sys 0m0.157s
On RBD mount:
$ time unzip -q wordpress-seo.12.9.1.zip
real 0m0.176s user 0m0.131s sys 0m0.045s
The difference is just too big. I'm having real trouble finding a good reference to check my setup for bad configuration etc.
I have network bandwidth, RAM and CPU to spare, but I'm unsure on how to put it to work to help my case.
Are there a lot of directories to be created from that zip file? I think it boils down to the directory operations that need to be performed synchrously. See https://fosdem.org/2020/schedule/event/sds_ceph_async_directory_ops/ https://fosdem.org/2020/schedule/event/sds_ceph_async_directory_ops/attachme... https://video.fosdem.org/2020/H.1308/sds_ceph_async_directory_ops.webm Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
On 6 Feb 2020, at 11:23, Stefan Kooman <stefan@bit.nl> wrote:
Hi!
I've confirmed that the write IO to the metadata pool is coming form active MDSes.
I'm experiencing very poor write performance on clients and I would like to see if there's anything I can do to optimise the performance.
Right now, I'm specifically focussing on speeding up this use case:
In CephFS mounted dir:
$ time unzip -q wordpress-seo.12.9.1.zip
real 0m47.596s user 0m0.218s sys 0m0.157s
On RBD mount:
$ time unzip -q wordpress-seo.12.9.1.zip
real 0m0.176s user 0m0.131s sys 0m0.045s
The difference is just too big. I'm having real trouble finding a good reference to check my setup for bad configuration etc.
I have network bandwidth, RAM and CPU to spare, but I'm unsure on how to put it to work to help my case.
Are there a lot of directories to be created from that zip file? I think it boils down to the directory operations that need to be performed synchrously. See https://fosdem.org/2020/schedule/event/sds_ceph_async_directory_ops/ https://fosdem.org/2020/schedule/event/sds_ceph_async_directory_ops/attachme... https://video.fosdem.org/2020/H.1308/sds_ceph_async_directory_ops.webm
Hi! Last Friday, I did a round of updates that were pending and planned for installation. After the updates, all server and client components were running their latest version and systems were rebooted to latest kernel versions. Ceph: Mimic Kernel: 5.3 (Ubuntu HWE) The write IO to the metadata pool is down by a factor of 10 and performance seems much improved. Though this does not give me a lot of intel on what the problem was, I'm glad that it is now pretty much resolved ;) Before the updates, I was running different (minor) versions of Ceph and kernel clients. This may have not been ideal, but I'm not sure on details of possible issues with that. Rebooting everything may have also eliminated some issues. I did not have the opportunity to do much analysis on that, since I was working in a production environment. Well, maybe some of you have extra insights. I'm happy to close this issue and will be monitoring and recording related info in case this happens again. Thanks much for your inputs, and have a good week, Samy
participants (5)
-
DHilsbos＠performair.com
-
Jasper Spaans
-
Patrick Donnelly
-
Samy Ascha
-
Stefan Kooman