Hi Patrick, I'm not sure that its exactly the same issue. I observed that "ceph tell mds.xyz session ls" had all counters 0. On Friday before we had a power loss on a rack that took out a JBOD with a few meta-data disks and I suspect that the reporting of zeroes started after this crash. No hard evidence though. I uploaded all logs with a bit of explanation to tag 1c022c43-04a7-419d-bdb0-e33c97ef06b8. I don't have any more detail than that recorded. It was 3 other MDSes that restarted on fail of rank 0, not 5 as I wrote before. The readme.txt contains pointers to find the info in the logs. We didn't have any user complaints. Therefore, I'm reasonably confident that the file system was actually accessible the whole time (from Friday afternoon until Sunday night when I restarted everything). I hope you can find something useful. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Patrick Donnelly <pdonnell@redhat.com> Sent: Monday, September 11, 2023 7:51 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS daemons don't report any more Hello Frank, On Mon, Sep 11, 2023 at 8:39 AM Frank Schilder <frans@dtu.dk> wrote:
Update: I did a systematic fail of all MDSes that didn't report, starting with the stand-by daemons and continuing from high to low ranks. One by one they started showing up again with version and stats and the fail went as usual with one exception: rank 0.
It might be https://tracker.ceph.com/issues/24403
The moment I failed rank 0 it took 5 other MDSes down with it.
Can you be more precise about what happened? Can you share logs?
This is, in fact, the second time I have seen such an event, failing an MDS crashes others. Given the weird observation in my previous e-mail together with what I saw when restarting everything, does this indicate a problem with data integrity or is this an annoying yet harmless bug?
It sounds like an annoyance but certainly one we'd like to track down. Keep in mind that the "fix" for https://tracker.ceph.com/issues/24403 is not going to Octopus. You need to upgrade and Pacific will soon be EOL.
Thanks for any help! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Sunday, September 10, 2023 12:39 AM To: ceph-users@ceph.io Subject: [ceph-users] MDS daemons don't report any more
Hi all, I make a weird observation. 8 out of 12 MDS daemons seem not to report to the cluster any more:
# ceph fs status con-fs2 - 1625 clients ======= RANK STATE MDS ACTIVITY DNS INOS 0 active ceph-16 Reqs: 0 /s 0 0 1 active ceph-09 Reqs: 128 /s 4251k 4250k 2 active ceph-17 Reqs: 0 /s 0 0 3 active ceph-15 Reqs: 0 /s 0 0 4 active ceph-24 Reqs: 269 /s 3567k 3567k 5 active ceph-11 Reqs: 0 /s 0 0 6 active ceph-14 Reqs: 0 /s 0 0 7 active ceph-23 Reqs: 0 /s 0 0 POOL TYPE USED AVAIL con-fs2-meta1 metadata 2169G 7081G con-fs2-meta2 data 0 7081G con-fs2-data data 1248T 4441T con-fs2-data-ec-ssd data 705G 22.1T con-fs2-data2 data 3172T 4037T STANDBY MDS ceph-08 ceph-10 ceph-12 ceph-13 VERSION DAEMONS None ceph-16, ceph-17, ceph-15, ceph-11, ceph-14, ceph-23, ceph-10, ceph-12 ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) ceph-09, ceph-24, ceph-08, ceph-13
Version is "none" for these and there are no stats. Ceph versions reports only 4 MDSes out of the 12. 8 are not shown at all:
[root@gnosis ~]# ceph versions { "mon": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 5 }, "mgr": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 5 }, "osd": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 1282 }, "mds": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 4 }, "overall": { "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)": 1296 } }
Ceph status reports everything as up and OK:
[root@gnosis ~]# ceph status cluster: id: e4ece518-f2cb-4708-b00f-b6bf511e91d9 health: HEALTH_OK
services: mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 2w) mgr: ceph-03(active, since 61s), standbys: ceph-25, ceph-01, ceph-02, ceph-26 mds: con-fs2:8 4 up:standby 8 up:active osd: 1284 osds: 1282 up (since 31h), 1282 in (since 33h); 567 remapped pgs
data: pools: 14 pools, 25065 pgs objects: 2.14G objects, 3.7 PiB usage: 4.7 PiB used, 8.4 PiB / 13 PiB avail pgs: 79908208/18438361040 objects misplaced (0.433%) 23063 active+clean 1225 active+clean+snaptrim_wait 317 active+remapped+backfill_wait 250 active+remapped+backfilling 208 active+clean+snaptrim 2 active+clean+scrubbing+deep
io: client: 596 MiB/s rd, 717 MiB/s wr, 4.16k op/s rd, 3.04k op/s wr recovery: 8.7 GiB/s, 3.41k objects/s
My first thought is that the status module failed. However, I don't manage to restart it (always on). An MGR fail-over did not help.
Any ideas what is going on here?
Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D