MDSs report slow metadata IOs
Hi, see many of this in cluster log channel. many are blocked with long period of seconds. It should hurt client access performance. Any ideas to get rid of them? Thanks, Ben --------------------------------- 7/7/23 4:48:50 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:48:09 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:48:09 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 559 secs 7/7/23 4:47:47 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:47:11 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:47:10 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 377 secs 7/7/23 4:46:22 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:46:12 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:45:40 PM [WRN] Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:45:40 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 199 secs 7/7/23 4:45:12 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:45:07 PM [WRN] Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 7/7/23 4:44:58 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 565 secs 7/7/23 4:44:58 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
https://docs.ceph.com/en/quincy/cephfs/createfs/#creating-pools As an additional note: it's recommended to put the metadata pool on a dedicated set of SSDs to prevent client load from disrupting MDS performance. On Fri, Jul 7, 2023 at 4:56 AM Ben <ruidong.gao@gmail.com> wrote:
Hi,
see many of this in cluster log channel. many are blocked with long period of seconds. It should hurt client access performance. Any ideas to get rid of them?
Thanks, Ben --------------------------------- 7/7/23 4:48:50 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:48:09 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:48:09 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 559 secs
7/7/23 4:47:47 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:47:11 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:47:10 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 377 secs
7/7/23 4:46:22 PM [WRN] Health check update: 8 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:46:12 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:45:40 PM [WRN] Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:45:40 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 199 secs
7/7/23 4:45:12 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:45:07 PM [WRN] Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
7/7/23 4:44:58 PM [INF] MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 565 secs
7/7/23 4:44:58 PM [WRN] Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
participants (2)
-
Ben
-
Patrick Donnelly