have fix it according http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034946.html. please ignore this thread. On Thu, Jul 30, 2020 at 7:52 PM Frank Yu <flyxiaoyu@gmail.com> wrote:
No, I think I should get the reason why it fall into read only, and found the correct method to fix it.... Just restart the mds is dangerous, I think.
On Thu, Jul 30, 2020 at 7:35 PM sathvik vutukuri <7vik.sathvik@gmail.com> wrote:
Have you tried restart of the MDS.
On Thu, 30 Jul 2020, 16:40 Frank Yu, <flyxiaoyu@gmail.com> wrote:
I got some error from mds.log as below:
2020-07-30 18:14:38.574 7f6473346700 0 log_channel(cluster) log [WRN] : 94 slow requests, 0 included below; oldest blocked for > 619910.524984 secs 2020-07-30 18:14:43.574 7f6473346700 0 log_channel(cluster) log [WRN] : 94 slow requests, 0 included below; oldest blocked for > 619915.525079 secs 2020-07-30 18:14:44.835 7f646f33e700 -1 mds.0.159432 unhandled write error (90) Message too long, force readonly... 2020-07-30 18:14:44.835 7f646f33e700 1 mds.0.cache force file system read-only 2020-07-30 18:14:44.835 7f646f33e700 0 log_channel(cluster) log [WRN] : force file system read-only 2020-07-30 18:15:18.000 7f6473346700 0 log_channel(cluster) log [WRN] : 114 slow requests, 5 included below; oldest blocked for > 619949.950199 secs
On Thu, Jul 30, 2020 at 6:55 PM Frank Yu <flyxiaoyu@gmail.com> wrote:
Hi guys,
I have a ceph cluster with three MDS servers, two of them in active status, while the left one is in standby-replay mode. Today I found the message '1 MDSs are read only' show up when check the cluster status with 'ceph -s', details as below:
# ceph -s cluster: id: 3d43e9a5-50dc-4f84-9493-656bf4f06f8c health: HEALTH_WARN 5 clients failing to advance oldest client/flush tid 1 MDSs are read only 2 MDSs report slow requests 2 MDSs behind on trimming BlueFS spillover detected on 33 OSD(s)
services: mon: 3 daemons, quorum bjcpu-001,bjcpu-002,bjcpu-003 (age 3M) mgr: bjcpu-001.xxxx.io(active, since 3M), standbys: bjcpu-003.xxxx.io, bjcpu-002.xxxx.io mds: cephfs:2 {0=bjcpu-003.xxxx.io=up:active,1=bjcpu-001.xxxx.io =up:active} 1 up:standby-replay osd: 48 osds: 48 up (since 7w), 48 in (since 7M)
data: pools: 3 pools, 2304 pgs objects: 301.35M objects, 70 TiB usage: 246 TiB used, 280 TiB / 527 TiB avail pgs: 2295 active+clean 9 active+clean+scrubbing+deep
io: client: 254 B/s rd, 44 MiB/s wr, 0 op/s rd, 15 op/s wr
What should I do to fix the error message? it seems the cluster still works fine(can read and write).
Many thanks
-- Regards Frank Yu
-- Regards Frank Yu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Frank Yu
-- Regards Frank Yu