ceph mon out of quorom and cephfs degraded
Hi, I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm. First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command. 2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum) For now this is not appearing again. Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w) data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running. Below is a log snip from one of the mds service. -25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online.
I think either your mdlog or the purge_queue journal is corrupted: 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Can you paste the output of this command? cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue journal inspect cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog journal inspect I expect one or more damaged entries. Check this thread for more details: https://www.spinics.net/lists/ceph-users/msg80124.html You should try to backup the journal, but in my case that wasn't possible, so I had no other choice than resetting it. Zitat von Amudhan P <amudhan83@gmail.com>:
Hi,
I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm.
First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command.
2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum)
For now this is not appearing again.
Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command
cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean
Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running.
Below is a log snip from one of the mds service.
-25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eugen, This is the output for the command cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue journal inspect Overall journal integrity: OK cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal inspect Overall journal integrity: OK On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <eblock@nde.ag> wrote:
I think either your mdlog or the purge_queue journal is corrupted:
2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay
Can you paste the output of this command?
cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue journal inspect cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog journal inspect
I expect one or more damaged entries. Check this thread for more details:
https://www.spinics.net/lists/ceph-users/msg80124.html
You should try to backup the journal, but in my case that wasn't possible, so I had no other choice than resetting it.
Zitat von Amudhan P <amudhan83@gmail.com>:
Hi,
I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm.
First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command.
2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum)
For now this is not appearing again.
Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command
cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean
Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running.
Below is a log snip from one of the mds service.
-25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There are few more logs in MDS. I have highlighted few log lines which I am not sure what it is. Apr 16 20:13:59 strg-node3 bash[59114]: debug -79> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command dump inode hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -78> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command exit hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -77> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command respawn hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -76> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command heap hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -75> 2025-04-16T14:43:59.170+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127517 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -74> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command cpu_profiler hook 0x560a2c35458 0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -73> 2025-04-16T14:43:59.170+0000 7f74b302c700 5 mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot seq 1 Apr 16 20:13:59 strg-node3 bash[59114]: debug -72> 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/ 0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -71> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127518 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -70> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -69> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/ 0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -68> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -67> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 asok(0x560a2c44e000) register_command objecter_requests hook 0x560a2c 3544c0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -66> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -65> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/ 0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -64> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Apr 16 20:13:59 strg-node3 bash[59114]: debug -63> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -62> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0 handle_osd_map epoch 0, 0 new blocklist entries Apr 16 20:13:59 strg-node3 bash[59114]: debug -61> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map i am now mds.0.127518
Apr 16 20:13:59 strg-node3 bash[59114]: debug -60> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map state change up:boot --> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -59> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state: up:boot -> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -58> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 replay_start *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 waiting for osdmap 45749 (which blocklists prior instance)* Apr 16 20:13:59 strg-node3 bash[59114]: debug -56> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/0 *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap*
On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <amudhan83@gmail.com> wrote:
Eugen,
This is the output for the command cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue journal inspect Overall journal integrity: OK cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal inspect Overall journal integrity: OK
On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <eblock@nde.ag> wrote:
I think either your mdlog or the purge_queue journal is corrupted:
2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay
Can you paste the output of this command?
cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue journal inspect cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog journal inspect
I expect one or more damaged entries. Check this thread for more details:
https://www.spinics.net/lists/ceph-users/msg80124.html
You should try to backup the journal, but in my case that wasn't possible, so I had no other choice than resetting it.
Zitat von Amudhan P <amudhan83@gmail.com>:
Hi,
I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm.
First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command.
2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum)
For now this is not appearing again.
Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command
cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean
Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running.
Below is a log snip from one of the mds service.
-25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What’s your overall Ceph status? It says data pool 3 not found. Zitat von Amudhan P <amudhan83@gmail.com>:
There are few more logs in MDS. I have highlighted few log lines which I am not sure what it is.
Apr 16 20:13:59 strg-node3 bash[59114]: debug -79> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command dump inode hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -78> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command exit hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -77> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command respawn hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -76> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command heap hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -75> 2025-04-16T14:43:59.170+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127517 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -74> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command cpu_profiler hook 0x560a2c35458
Apr 16 20:13:59 strg-node3 bash[59114]: debug -73> 2025-04-16T14:43:59.170+0000 7f74b302c700 5 mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot seq 1 Apr 16 20:13:59 strg-node3 bash[59114]: debug -72> 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -71> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127518 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -70> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -69> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -68> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -67> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 asok(0x560a2c44e000) register_command objecter_requests hook 0x560a2c 3544c0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -66> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -65> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -64> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Apr 16 20:13:59 strg-node3 bash[59114]: debug -63> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -62> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0 handle_osd_map epoch 0, 0 new blocklist entries Apr 16 20:13:59 strg-node3 bash[59114]: debug -61> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map i am now mds.0.127518
Apr 16 20:13:59 strg-node3 bash[59114]: debug -60> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map state change up:boot --> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -59> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state: up:boot -> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -58> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 replay_start *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 waiting for osdmap 45749 (which blocklists prior instance)* Apr 16 20:13:59 strg-node3 bash[59114]: debug -56> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/0 *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap*
On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <amudhan83@gmail.com> wrote:
Eugen,
This is the output for the command cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue journal inspect Overall journal integrity: OK cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal inspect Overall journal integrity: OK
On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <eblock@nde.ag> wrote:
I think either your mdlog or the purge_queue journal is corrupted:
2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay
Can you paste the output of this command?
cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue journal inspect cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog journal inspect
I expect one or more damaged entries. Check this thread for more details:
https://www.spinics.net/lists/ceph-users/msg80124.html
You should try to backup the journal, but in my case that wasn't possible, so I had no other choice than resetting it.
Zitat von Amudhan P <amudhan83@gmail.com>:
Hi,
I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm.
First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command.
2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum)
For now this is not appearing again.
Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command
cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean
Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running.
Below is a log snip from one of the mds service.
-25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have run the status and stat command below is the output. ceph -s cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available 7 daemons have recently crashed services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 20h) mgr: strg-node2.unyimy(active, since 20h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 20h), 32 in (since 10w) data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 317 active+clean 4 active+clean+scrubbing+deep ceph mds stat mumstrg:1/1 {0=mumstrg.strg-node1.gchapr=up:replay(laggy or crashed)} ceph osd lspools 1 device_health_metrics 2 cephfs.mumstrg.meta 3 cephfs.mumstrg.data On Thu, Apr 17, 2025 at 10:33 AM Eugen Block <eblock@nde.ag> wrote:
What’s your overall Ceph status? It says data pool 3 not found.
Zitat von Amudhan P <amudhan83@gmail.com>:
There are few more logs in MDS. I have highlighted few log lines which I am not sure what it is.
Apr 16 20:13:59 strg-node3 bash[59114]: debug -79> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command dump inode hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -78> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command exit hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -77> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command respawn hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -76> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command heap hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -75> 2025-04-16T14:43:59.170+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127517 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -74> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command cpu_profiler hook 0x560a2c35458
Apr 16 20:13:59 strg-node3 bash[59114]: debug -73> 2025-04-16T14:43:59.170+0000 7f74b302c700 5 mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot seq 1 Apr 16 20:13:59 strg-node3 bash[59114]: debug -72> 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -71> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127518 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -70> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -69> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -68> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -67> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 asok(0x560a2c44e000) register_command objecter_requests hook 0x560a2c 3544c0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -66> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -65> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -64> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Apr 16 20:13:59 strg-node3 bash[59114]: debug -63> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -62> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0 handle_osd_map epoch 0, 0 new blocklist entries Apr 16 20:13:59 strg-node3 bash[59114]: debug -61> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map i am now mds.0.127518
Apr 16 20:13:59 strg-node3 bash[59114]: debug -60> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map state change up:boot --> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -59> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state: up:boot -> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -58> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 replay_start *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 waiting for osdmap 45749 (which blocklists prior instance)* Apr 16 20:13:59 strg-node3 bash[59114]: debug -56> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/0 *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap*
On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <amudhan83@gmail.com> wrote:
Eugen,
This is the output for the command cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue journal inspect Overall journal integrity: OK cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal inspect Overall journal integrity: OK
On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <eblock@nde.ag> wrote:
I think either your mdlog or the purge_queue journal is corrupted:
2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay
Can you paste the output of this command?
cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue journal inspect cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog journal inspect
I expect one or more damaged entries. Check this thread for more details:
https://www.spinics.net/lists/ceph-users/msg80124.html
You should try to backup the journal, but in my case that wasn't possible, so I had no other choice than resetting it.
Zitat von Amudhan P <amudhan83@gmail.com>:
Hi,
I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm.
First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command.
2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum)
For now this is not appearing again.
Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command
cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean
Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running.
Below is a log snip from one of the mds service.
-25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Oh right, I just noticed you had the status in your first message. Is there any chance the MDS is out of memory? Can you paste a complete (fresh) startup log on some pastebin or so? Also, a complete 'ceph crash info <crash>' could be useful. Zitat von Amudhan P <amudhan83@gmail.com>:
I have run the status and stat command below is the output.
ceph -s
cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available 7 daemons have recently crashed
services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 20h) mgr: strg-node2.unyimy(active, since 20h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 20h), 32 in (since 10w)
data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 317 active+clean 4 active+clean+scrubbing+deep
ceph mds stat mumstrg:1/1 {0=mumstrg.strg-node1.gchapr=up:replay(laggy or crashed)}
ceph osd lspools 1 device_health_metrics 2 cephfs.mumstrg.meta 3 cephfs.mumstrg.data
On Thu, Apr 17, 2025 at 10:33 AM Eugen Block <eblock@nde.ag> wrote:
What’s your overall Ceph status? It says data pool 3 not found.
Zitat von Amudhan P <amudhan83@gmail.com>:
There are few more logs in MDS. I have highlighted few log lines which I am not sure what it is.
Apr 16 20:13:59 strg-node3 bash[59114]: debug -79> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command dump inode hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -78> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command exit hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -77> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command respawn hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -76> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command heap hook 0x560a2c354580 Apr 16 20:13:59 strg-node3 bash[59114]: debug -75> 2025-04-16T14:43:59.170+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127517 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -74> 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) register_command cpu_profiler hook 0x560a2c35458
Apr 16 20:13:59 strg-node3 bash[59114]: debug -73> 2025-04-16T14:43:59.170+0000 7f74b302c700 5 mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot seq 1 Apr 16 20:13:59 strg-node3 bash[59114]: debug -72> 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -71> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.mumstrg.strg-node3.xhxbwx Updating MDS map to version 127518 f rom mon.2 Apr 16 20:13:59 strg-node3 bash[59114]: debug -70> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -69> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -68> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -67> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 asok(0x560a2c44e000) register_command objecter_requests hook 0x560a2c 3544c0 Apr 16 20:13:59 strg-node3 bash[59114]: debug -66> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs Apr 16 20:13:59 strg-node3 bash[59114]: debug -65> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/
Apr 16 20:13:59 strg-node3 bash[59114]: debug -64> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) Apr 16 20:13:59 strg-node3 bash[59114]: debug -63> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap Apr 16 20:13:59 strg-node3 bash[59114]: debug -62> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0 handle_osd_map epoch 0, 0 new blocklist entries Apr 16 20:13:59 strg-node3 bash[59114]: debug -61> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map i am now mds.0.127518
Apr 16 20:13:59 strg-node3 bash[59114]: debug -60> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 handle_mds_map state change up:boot --> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -59> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state: up:boot -> up:replay Apr 16 20:13:59 strg-node3 bash[59114]: debug -58> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 replay_start *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 waiting for osdmap 45749 (which blocklists prior instance)* Apr 16 20:13:59 strg-node3 bash[59114]: debug -56> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _send_mon_message to mon.strg-node3 at v2:10.0.103.3:3300/0 *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap*
On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <amudhan83@gmail.com> wrote:
Eugen,
This is the output for the command cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue journal inspect Overall journal integrity: OK cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal inspect Overall journal integrity: OK
On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <eblock@nde.ag> wrote:
I think either your mdlog or the purge_queue journal is corrupted:
2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay
Can you paste the output of this command?
cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue journal inspect cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog journal inspect
I expect one or more damaged entries. Check this thread for more details:
https://www.spinics.net/lists/ceph-users/msg80124.html
You should try to backup the journal, but in my case that wasn't possible, so I had no other choice than resetting it.
Zitat von Amudhan P <amudhan83@gmail.com>:
Hi,
I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm.
First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command.
2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum)
For now this is not appearing again.
Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command
cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean
Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running.
Below is a log snip from one of the mds service.
-25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Amudhan P
-
Eugen Block