Hello everyone, We're having an issue with our backup filesystem running Reef (18.2.4) on an Alma9.4 (version 5.14 kernel) cluster where are our filesystem "ceph_spare" is in a constant degraded state: root@pebbles-n4 11:49 [~]: ceph fs status ceph_spare - 360 clients ========== RANK  STATE      MDS      ACTIVITY   DNS    INOS   DIRS   CAPS  0    replay  pebbles-s2            14.1M  6858k   240k     0            POOL               TYPE     USED  AVAIL        mds_spare_fs         metadata  1451G  3216G ec82_spare_primary_fs_data    data       0   3216G       ec82pool_spare          data    8820T  4626T STANDBY MDS  pebbles-s1  pebbles-s2 MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) root@pebbles-n4 11:49 [~]: ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 large omap objects; 1 MDSs report oversized cache; 1 MDSs behind on trimming [WRN] FS_DEGRADED: 1 filesystem is degraded     fs ceph_spare is degraded [WRN] LARGE_OMAP_OBJECTS: 1 large omap objects     1 large objects found in pool 'mds_spare_fs'     Search the cluster log for 'Large omap object found' for more details. [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache     mds.pebbles-s2(mds.0): MDS cache is too large (24GB/4GB); 0 inodes in use by clients, 0 stray files [WRN] MDS_TRIM: 1 MDSs behind on trimming     mds.pebbles-s2(mds.0): Behind on trimming (69293/128) max_segments: 128, num_segments: 69293 Our MDS is stuck in replay constantly and crashing before finishing despite a restart, reboot, and wiping of the client sessions ("ceph config set mds mds_wipe_sessions true"). When following the replay, we've noticed that the MDS always crashes just before the "journal_read_pos" reaches the "journal_write_pos" where the MDS logs with a 10/20 debug report the following: ...    -14> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay dir 0x1002a7c9f3a.010001*    -13> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x1002a7c9f3a.010001* /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_Sma llHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/ [2,head] auth v=283167 cv=0/0 state=1073741824 f(v0 m2024-09-30T02:20:11.052706+0100 4262=4262+0) n(v37 rc2024-09-30T02:20:11.052706+0100  b4168080204 4262=4262+0) hs=877+853,ss=3+0 dirty=1733 | child=1 0x5642e8d5d680]    -12> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.cache.dir(0x1002a7c9f3a.010001*) lookup_exact_snap (head, 'FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks _plot_0012.ef980630.partial')    -11> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.cache.dir(0x1002a7c9f3a.010001*) lookup_exact_snap (head, 'FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks _plot_0012.ef980630.partial')    -10> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 12 mds.0.cache.dir(0x1002a7c9f3a.010001*) add_null_dentry [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallH ole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lo ck) pv=0 v=283167 ino=(nil) state=1073741824 0x564f219fca00]     -9> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_P u_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lock) v=28316 6 ino=(nil) state=1610612736 | dirty=1 0x564f219fca00]     -8> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 12 mds.0.cache.dir(0x1002a7c9f3a.010001*) link_primary_inode [dentry #0x1/cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_Sma llHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_46164 89_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial [126,head] auth NULL (dversion lock) v=283166 ino=(nil) state=1610612736 | dirty=1 0x564f219fca00] [inode 0x1002c5f1d10 [126,head] #1002c5f1d10 auth v283166 s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} 0x564f21a00000]     -7> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay added [inode 0x1002c5f1d10 [126,head] /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial auth v283166 s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} 0x564f21a00000]     -6> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.cache.ino(0x1002c5f1d10) mark_dirty_parent     -5> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay noting opened inode [inode 0x1002c5f1d10 [126,head] /cephfs2-users/crusso/cjrLabArchives/knayde/Data2020/2020March04_LHC2C7_SmallHole_F4MSM_Pu_Process/Polish/job030/MotionCorrectedMovies/FoilHole_4616489_Data_3448109_3448111_20200306_014931_Fractions_cor_stack_tracks_plot_0012.ef980630.partial auth v283166 DIRTYPARENT s=0 n(v0 1=1+0) (iversion lock) cr={159821641=0-4194304@125} | dirtyparent=1 dirty=1 0x564f21a00000]     -4> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay inotable tablev 1481899 <= table 1481899     -3> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 10 mds.0.journal EMetaBlob.replay sessionmap v 746010300, table 746010299 prealloc [] used 0x1002c5f1d10     -2> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 20 mds.0.journal  (session prealloc [0x1002bdf9f74~0x22,0x1002bdfb51d~0x7c,0x1002c59def8~0xfb,0x1002c5eec0c~0x86,0x1002c5eee87~0x1f5])     -1> 2024-10-02T15:20:52.494+0100 7fa8b6d95640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7fa8b6d95640 time 2024-10-02T15:20:52.495403+0100 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start)  ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa8c416b04d]  2: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fa8c416b20b]  3: /usr/bin/ceph-mds(+0x1ef0fe) [0x5642daf490fe]  4: /usr/bin/ceph-mds(+0x1ef145) [0x5642daf49145]  5: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bad) [0x5642db17b67d]  6: (EUpdate::replay(MDSRank*)+0x5d) [0x5642db18477d]  7: (MDLog::_replay_thread()+0x75e) [0x5642db12d00e]  8: /usr/bin/ceph-mds(+0x13c561) [0x5642dae96561]  9: /lib64/libc.so.6(+0x89c02) [0x7fa8c3889c02]  10: /lib64/libc.so.6(+0x10ec40) [0x7fa8c390ec40]      0> 2024-10-02T15:20:52.495+0100 7fa8b6d95640 -1 *** Caught signal (Aborted) **  in thread 7fa8b6d95640 thread_name:md_log_replay  ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)  1: /lib64/libc.so.6(+0x3e6f0) [0x7fa8c383e6f0]  2: /lib64/libc.so.6(+0x8b94c) [0x7fa8c388b94c]  3: raise()  4: abort()  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa8c416b0a7]  6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fa8c416b20b]  7: /usr/bin/ceph-mds(+0x1ef0fe) [0x5642daf490fe]  8: /usr/bin/ceph-mds(+0x1ef145) [0x5642daf49145]  9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bad) [0x5642db17b67d]  10: (EUpdate::replay(MDSRank*)+0x5d) [0x5642db18477d]  11: (MDLog::_replay_thread()+0x75e) [0x5642db12d00e]  12: /usr/bin/ceph-mds(+0x13c561) [0x5642dae96561]  13: /lib64/libc.so.6(+0x89c02) [0x7fa8c3889c02]  14: /lib64/libc.so.6(+0x10ec40) [0x7fa8c390ec40]  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels ---    0/ 5 none    0/ 1 lockdep    0/ 1 context    1/ 1 crush   10/20 mds    1/ 5 mds_balancer    1/ 5 mds_locker    1/ 5 mds_log    1/ 5 mds_log_expire    1/ 5 mds_migrator    0/ 1 buffer    0/ 1 timer    0/ 1 filer    0/ 1 striper    0/ 1 objecter    0/ 5 rados    0/ 5 rbd    0/ 5 rbd_mirror    0/ 5 rbd_replay    0/ 5 rbd_pwl    0/ 5 journaler    0/ 5 objectcacher    0/ 5 immutable_obj_cache    0/ 5 client    1/ 5 osd    0/ 5 optracker    0/ 5 objclass    1/ 3 filestore    1/ 3 journal    0/ 0 ms    1/ 5 mon    0/10 monc    1/ 5 paxos    0/ 5 tp    1/ 5 auth    1/ 5 crypto    1/ 1 finisher    1/ 1 reserver    1/ 5 heartbeatmap    1/ 5 perfcounter    1/ 5 rgw    1/ 5 rgw_sync    1/ 5 rgw_datacache    1/ 5 rgw_access    1/ 5 rgw_dbstore    1/ 5 rgw_flight    1/ 5 javaclient    1/ 5 asok    1/ 1 throttle    0/ 0 refs    1/ 5 compressor    1/ 5 bluestore    1/ 5 bluefs    1/ 3 bdev    1/ 5 kstore    4/ 5 rocksdb    4/ 5 leveldb    1/ 5 fuse    2/ 5 mgr    1/ 5 mgrc    1/ 5 dpdk    1/ 5 eventtrace    1/ 5 prioritycache    0/ 5 test    0/ 5 cephfs_mirror    0/ 5 cephsqlite    0/ 5 seastore    0/ 5 seastore_onode    0/ 5 seastore_odata    0/ 5 seastore_omap    0/ 5 seastore_tm    0/ 5 seastore_t    0/ 5 seastore_cleaner    0/ 5 seastore_epm    0/ 5 seastore_lba    0/ 5 seastore_fixedkv_tree    0/ 5 seastore_cache    0/ 5 seastore_journal    0/ 5 seastore_device    0/ 5 seastore_backref    0/ 5 alienstore    1/ 5 mclock    0/ 5 cyanstore    1/ 5 ceph_exporter    1/ 5 memstore   -2/-2 (syslog threshold)   -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads ---   7fa8b6d95640 / md_log_replay   max_recent     10000   max_new         1000   log_file /var/log/ceph/ceph-mds.pebbles-s3.log --- end dump of recent events --- Our MDS then starts at the beginning of the replay process and continually re-replays the journal until it crashes again at the same point. It looks like our journal has gotten corrupted at this file from what I understand and our journal (worryingly) is exceptionally large where we've had to use a 2 TiB machine just to try and export it. What is causing this issue? Can we do small modifications to the journal or similar to rectify this issue or move the faulty object in the journal out of the bulk object-store to fail (and thus skip) the transaction? We really do not want to go through disaster recovery again (https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#disaster-rec...) as this is the 2nd time this has happened to this cluster in the last 4 months and it took over a month to recover the data last time Kindest regards, Ivan -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH