Aditional information: - We already tried to restart services and hole machine - Part of jounalctl: jan 13 02:40:18 s1.ceph.infra.ufscar.br ceph-bab39b74-c93a-4e34-aae9-a44a5569d52c-mon-s1[6343]: debug 2023-01-13T05:40:18.653+0000 7fc370b64700 0 log_channel(cluster) log [WRN] : Replacing daemon mds.cephfs.s1.nvopyf as rank 1 with standby daemon mds.cephfs.s2.qikxmw jan 13 02:40:18 s1.ceph.infra.ufscar.br ceph-bab39b74-c93a-4e34-aae9-a44a5569d52c-mon-s1[6343]: debug 2023-01-13T05:40:18.653+0000 7fc370b64700 1 mon.s1@0(leader).mds e653196 fail_mds_gid 107853765 mds.cephfs.s1.nvopyf role 1 jan 13 02:40:18 s1.ceph.infra.ufscar.br ceph-bab39b74-c93a-4e34-aae9-a44a5569d52c-mon-s1[6343]: debug 2023-01-13T05:40:18.653+0000 7fc370b64700 0 log_channel(cluster) log [INF] : MDS daemon mds.cephfs.s1.nvopyf is removed because it is dead or otherwise unavailable. jan 13 02:40:18 s1.ceph.infra.ufscar.br ceph-bab39b74-c93a-4e34-aae9-a44a5569d52c-mon-s1[6343]: debug 2023-01-13T05:40:18.677+0000 7fc370b64700 0 log_channel(cluster) log [WRN] : Health check failed: 1 filesystem is degraded (FS_DEGRADED) jan 13 02:40:18 s1.ceph.infra.ufscar.br ceph-bab39b74-c93a-4e34-aae9-a44a5569d52c-mon-s1[6343]: debug 2023-01-13T05:40:18.677+0000 7fc370b64700 0 log_channel(cluster) log [WRN] : Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)

Hi André, On Sat, Jan 14, 2023 at 12:14 AM André de Freitas Smaira <afsmaira@ufscar.br> wrote:

Hello!

Yesterday we found some errors in our cephadm disks, which is making it impossible to access our HPC Cluster:

# ceph health detail HEALTH_WARN 3 failed cephadm daemon(s); insufficient standby MDS daemons available [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) daemon mds.cephfs.s1.nvopyf on s1.ceph.infra.ufscar.br is in error state daemon mds.cephfs.s2.qikxmw on s2.ceph.infra.ufscar.br is in error state daemon mds.cftv.s2.anybzk on s2.ceph.infra.ufscar.br is in error state [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more

What's in the MDS logs?

Googling we found out that we should remove the failed MDS, but the data in these disks is relatively important. We would like to know if we need to remove it or if it can be fixed, and if we have to remove it if the data will be lost. Please tell me if you need more information.

Thanks in advance, André de Freitas Smaira Federal University of São Carlos - UFSCar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Cheers, Venky