Starting a new thread, forgot subject in the previous. So our FS down. Got the following error, what can I do? # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs atlassian/prod is degraded [ERR] MDS_DAMAGE: 1 mds daemon damaged fs atlassian-prod mds.1 is damaged # ceph fs get atlassian-prod Filesystem 'atlassian-prod' (2) fs_name atlassian-prod epoch 43440 flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay created 2023-05-10T08:45:46.911064+0000 modified 2023-12-21T06:47:19.291154+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 29480 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 3 in 0,1,2 up {0=1073573,2=1073583} failed damaged 1 stopped data_pools [5] metadata_pool 4 inline_data disabled balancer standby_count_wanted 1 [mds.atlassian-prod.pwsoel13142.egsdfl{0:1073573} state up:resolve seq 573 join_fscid=2 addr [v2:10.233.127.22:6800/61692284,v1:10.233.127.22:6801/61692284] compat {c=[1],r=[1],i=[7ff]}] [mds.atlassian-prod.pwsoel13143.qlvypn{2:1073583} state up:resolve seq 571 join_fscid=2 addr [v2:10.233.127.18:6800/3627858294,v1:10.233.127.18:6801/3627858294] compat {c=[1],r=[1],i=[7ff]}] Best regards, Sake
Hi Sake, I would start by decrementing max_mds by 1: ceph fs set atlassian-prod max_mds 2 The mds.1 no longer restarts? logs? Le jeu. 21 déc. 2023 à 08:11, Sake Ceph <ceph@paulusma.eu> a écrit :
Starting a new thread, forgot subject in the previous. So our FS down. Got the following error, what can I do?
# ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs atlassian/prod is degraded [ERR] MDS_DAMAGE: 1 mds daemon damaged fs atlassian-prod mds.1 is damaged
# ceph fs get atlassian-prod Filesystem 'atlassian-prod' (2) fs_name atlassian-prod epoch 43440 flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay created 2023-05-10T08:45:46.911064+0000 modified 2023-12-21T06:47:19.291154+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 29480 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 3 in 0,1,2 up {0=1073573,2=1073583} failed damaged 1 stopped data_pools [5] metadata_pool 4 inline_data disabled balancer standby_count_wanted 1 [mds.atlassian-prod.pwsoel13142.egsdfl{0:1073573} state up:resolve seq 573 join_fscid=2 addr [v2: 10.233.127.22:6800/61692284,v1:10.233.127.22:6801/61692284] compat {c=[1],r=[1],i=[7ff]}] [mds.atlassian-prod.pwsoel13143.qlvypn{2:1073583} state up:resolve seq 571 join_fscid=2 addr [v2: 10.233.127.18:6800/3627858294,v1:10.233.127.18:6801/3627858294] compat {c=[1],r=[1],i=[7ff]}]
Best regards, Sake _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi David Reducing max_mds didn't work. So I executed a fs reset: ceph fs set atlassian-prod allow_standby_replay false ceph fs set atlassian-prod cluster_down true ceph mds fail atlassian-prod.pwsoel13142.egsdfl ceph mds fail atlassian-prod.pwsoel13143.qlvypn ceph fs reset atlassian-prod ceph fs reset atlassian-prod --yes-i-really-mean-it This brought the fs back online and the servers/applications are working again. Question: can I increase the max_mds and active standby_replay? Will collect logs, maybe we can pinpoint the cause. Best regards, Sake
On Thu, Dec 21, 2023 at 3:05 AM Sake Ceph <ceph@paulusma.eu> wrote:
Hi David
Reducing max_mds didn't work. So I executed a fs reset: ceph fs set atlassian-prod allow_standby_replay false ceph fs set atlassian-prod cluster_down true ceph mds fail atlassian-prod.pwsoel13142.egsdfl ceph mds fail atlassian-prod.pwsoel13143.qlvypn ceph fs reset atlassian-prod ceph fs reset atlassian-prod --yes-i-really-mean-it
This brought the fs back online and the servers/applications are working again.
This was not the right thing to do. You can mark the rank repaired. See end of: https://docs.ceph.com/en/latest/cephfs/administration/#daemons (ceph mds repaired <role>) I admit that is not easy to find. I will add a ticket to improve the documentation: https://tracker.ceph.com/issues/63885 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
That wasn't really clear in the docs :(
Op 21-12-2023 17:26 CET schreef Patrick Donnelly <pdonnell@redhat.com>:
On Thu, Dec 21, 2023 at 3:05 AM Sake Ceph <ceph@paulusma.eu> wrote:
Hi David
Reducing max_mds didn't work. So I executed a fs reset: ceph fs set atlassian-prod allow_standby_replay false ceph fs set atlassian-prod cluster_down true ceph mds fail atlassian-prod.pwsoel13142.egsdfl ceph mds fail atlassian-prod.pwsoel13143.qlvypn ceph fs reset atlassian-prod ceph fs reset atlassian-prod --yes-i-really-mean-it
This brought the fs back online and the servers/applications are working again.
This was not the right thing to do. You can mark the rank repaired. See end of:
https://docs.ceph.com/en/latest/cephfs/administration/#daemons
(ceph mds repaired <role>)
I admit that is not easy to find. I will add a ticket to improve the documentation:
https://tracker.ceph.com/issues/63885
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Thu, Dec 21, 2023 at 2:49 AM David C. <david.casier@aevoo.fr> wrote:
I would start by decrementing max_mds by 1: ceph fs set atlassian-prod max_mds 2
This will have no positive effect. The monitors will not alter the number of ranks (i.e. stop a rank) if the cluster is degraded. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On Thu, Dec 21, 2023 at 2:11 AM Sake Ceph <ceph@paulusma.eu> wrote:
Starting a new thread, forgot subject in the previous. So our FS down. Got the following error, what can I do?
# ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs atlassian/prod is degraded [ERR] MDS_DAMAGE: 1 mds daemon damaged fs atlassian-prod mds.1 is damaged
Identify what is damaged by reviewing the MDS logs. Increase mds debugging and mark the rank repaired if there is insufficient information (which assumes that whatever caused the MDS to become damaged will reoccur when it restarts). -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
HI Mr Patrick, We are in same situation with Sake, now my MDS is crashed , NFS service is down with CEPHFS not responding. with my "ceph -s" result health: HEALTH_WARN 3 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available data: volumes: 0/1 healthy, 1 recovering pools: 15 pools, 1457 pgs pgs: 15664126/110662485 objects misplaced (14.155%) 1110 active+clean 305 active+remapped+backfill_wait 17 active+remapped+backfilling 13 active+remapped+backfill_toofull Could you help me explain the status of volume "recovering" ? what is it ? how can we track the progress of this?
participants (4)
-
David C.
-
nguyenvandiep＠baoviet.com.vn
-
Patrick Donnelly
-
Sake Ceph