CephFS: no MDS does join the filesystem
Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
I'm pretty sure the reason is due to the damaged MDS daemon. If you are able to clear that up it should allow the filesystem to come back up. I seen something like this a few months ago. We were just able to mark the mds as "repaired" and haven't seen any issue since, however I would discourage doing that without further investigation into the source of the damaged daemon. Regards, Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 2025-06-30 10:28, Robert Sander wrote:
Hi,
we are having an issue at a customer site where a 3PB CephFS is in failed state.
The cluster itself is unhealthy and awaits replacements disks:
# ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED)
services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub
data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull
io: recovery: 183 MiB/s, 312 objects/s
The CephFS metadata pool is not affected by the inconsistent PGs.
The MDSs have this line in their logfile:
"Monitors have assigned me to become a standby."
The filesystem is joinable:
# ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session
But no MDS joins:
# ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
Why?
Regards
Hi, Am 6/30/25 um 15:34 schrieb Bailey Allison:
I'm pretty sure the reason is due to the damaged MDS daemon. If you are able to clear that up it should allow the filesystem to come back up. I seen something like this a few months ago. We were just able to mark the mds as "repaired" and haven't seen any issue since, however I would discourage doing that without further investigation into the source of the damaged daemon.
The command "ceph tell mds.storage_cluster:0 damage ls" does not work as there is currently no active MDS in rank 0 for this filesystem. So I do not know why the MDS is damaged. The cluster also does not report damaged metadata but only the damaged MDS. Restarting the daemon does not change the situation. After a restart the MDS gets told to become a standby by the MONs. AFAIK an MDS does not have any persistent data in its installation directory. With marking the MDS as repaired you mean the command "ceph mds repaired storage_cluster:0", right? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, Am 6/30/25 um 16:50 schrieb Robert Sander:
With marking the MDS as repaired you mean the command "ceph mds repaired storage_cluster:0", right?
It looks like we hit this bug: https://tracker.ceph.com/issues/65094 From the MDS log: No subtrees found for root MDS rank! Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 handle_mds_map i am now mds.0.1753398 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 handle_mds_map state change up:reconnect --> up:rejoin Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_start Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_joint_start Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_done Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_channel(cluster) log [ERR] : No subtrees found for root MDS rank! Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu set_want_state: up:rejoin -> down:damaged Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_client will send 2025-07-01T06:50:56.151556+0000 mds.storage_cluster.sn04.cbvzzu (mds.0) 1 : cluster [ERR] No subtrees found for root MDS rank! Jul 01 08:50:56 sn04 ceph-mds[1563881]: monclient: _send_mon_message to mon.sn03 at v2:192.168.91.53:3300/0 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu Sending beacon down:damaged seq 11440 Jul 01 08:50:56 sn04 ceph-mds[1563881]: monclient: _send_mon_message to mon.sn03 at v2:192.168.91.53:3300/0 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu received beacon reply up:rejoin seq 11439 rtt 1.011 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu received beacon reply down:damaged seq 11440 rtt 0.0880002 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu respawn! Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: -9999> 2025-07-01T06:50:56.150+0000 7f5daca13640 -1 log_channel(cluster) log [ERR] : No subtrees found for root MDS rank! Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu e: '/usr/bin/ceph-mds' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 0: '/usr/bin/ceph-mds' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 1: '-n' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 2: 'mds.storage_cluster.sn04.cbvzzu' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 3: '-f' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 4: '--setuser' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 5: 'ceph' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 6: '--setgroup' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 7: 'ceph' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 8: '--default-log-to-file=false' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 9: '--default-log-to-journald=true' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 10: '--default-log-to-stderr=false' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu respawning with exe /usr/bin/ceph-mds Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu exe_path /proc/self/exe Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: ignoring --setuser ceph since I am not root Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: ignoring --setgroup ceph since I am not root Jul 01 08:50:56 sn04 ceph-mds[1563881]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process ceph-mds, pid 2 Jul 01 08:50:56 sn04 ceph-mds[1563881]: main not setting numa affinity Jul 01 08:50:56 sn04 ceph-mds[1563881]: pidfile_write: ignore empty --pid-file Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: starting mds.storage_cluster.sn04.cbvzzu at Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu Updating MDS map to version 1753401 from mon.2 Jul 01 08:50:57 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu Updating MDS map to version 1753402 from mon.2 Jul 01 08:50:57 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu Monitors have assigned me to become a standby. And that's it. The MDS journal integrity seems to be OK: # /usr/bin/cephfs-journal-tool --rank=storage_cluster:all journal inspect Overall journal integrity: OK How do we get this filesystem online again? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert, were you able to resolve this issue? I haven't faced that error myself yet, so I can't really comment. But it would be interesting to know if and how you got out of it. Thanks, Eugen Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
Am 6/30/25 um 16:50 schrieb Robert Sander:
With marking the MDS as repaired you mean the command "ceph mds repaired storage_cluster:0", right?
It looks like we hit this bug: https://tracker.ceph.com/issues/65094
From the MDS log: No subtrees found for root MDS rank!
Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 handle_mds_map i am now mds.0.1753398 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 handle_mds_map state change up:reconnect --> up:rejoin Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_start Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_joint_start Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_done Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_channel(cluster) log [ERR] : No subtrees found for root MDS rank! Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu set_want_state: up:rejoin -> down:damaged Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_client will send 2025-07-01T06:50:56.151556+0000 mds.storage_cluster.sn04.cbvzzu (mds.0) 1 : cluster [ERR] No subtrees found for root MDS rank! Jul 01 08:50:56 sn04 ceph-mds[1563881]: monclient: _send_mon_message to mon.sn03 at v2:192.168.91.53:3300/0 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu Sending beacon down:damaged seq 11440 Jul 01 08:50:56 sn04 ceph-mds[1563881]: monclient: _send_mon_message to mon.sn03 at v2:192.168.91.53:3300/0 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu received beacon reply up:rejoin seq 11439 rtt 1.011 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.beacon.storage_cluster.sn04.cbvzzu received beacon reply down:damaged seq 11440 rtt 0.0880002 Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu respawn! Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: -9999> 2025-07-01T06:50:56.150+0000 7f5daca13640 -1 log_channel(cluster) log [ERR] : No subtrees found for root MDS rank! Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu e: '/usr/bin/ceph-mds' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 0: '/usr/bin/ceph-mds' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 1: '-n' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 2: 'mds.storage_cluster.sn04.cbvzzu' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 3: '-f' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 4: '--setuser' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 5: 'ceph' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 6: '--setgroup' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 7: 'ceph' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 8: '--default-log-to-file=false' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 9: '--default-log-to-journald=true' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu 10: '--default-log-to-stderr=false' Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu respawning with exe /usr/bin/ceph-mds Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu exe_path /proc/self/exe Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: ignoring --setuser ceph since I am not root Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: ignoring --setgroup ceph since I am not root Jul 01 08:50:56 sn04 ceph-mds[1563881]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process ceph-mds, pid 2 Jul 01 08:50:56 sn04 ceph-mds[1563881]: main not setting numa affinity Jul 01 08:50:56 sn04 ceph-mds[1563881]: pidfile_write: ignore empty --pid-file Jul 01 08:50:56 sn04 ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: starting mds.storage_cluster.sn04.cbvzzu at Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu Updating MDS map to version 1753401 from mon.2 Jul 01 08:50:57 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu Updating MDS map to version 1753402 from mon.2 Jul 01 08:50:57 sn04 ceph-mds[1563881]: mds.storage_cluster.sn04.cbvzzu Monitors have assigned me to become a standby.
And that's it. The MDS journal integrity seems to be OK:
# /usr/bin/cephfs-journal-tool --rank=storage_cluster:all journal inspect Overall journal integrity: OK
How do we get this filesystem online again?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 7/10/25 um 09:14 schrieb Eugen Block:
were you able to resolve this issue? I haven't faced that error myself yet, so I can't really comment. But it would be interesting to know if and how you got out of it.
The issue is still ongoing as the PGs were still not able to repair themselves. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
My first idea is that you have to increase the timeout, before the ceph mon will mark the mds as standby. joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Do., 10. Juli 2025 um 14:33 Uhr schrieb Robert Sander < r.sander@heinlein-support.de>:
Am 7/10/25 um 09:14 schrieb Eugen Block:
were you able to resolve this issue? I haven't faced that error myself yet, so I can't really comment. But it would be interesting to know if and how you got out of it.
The issue is still ongoing as the PGs were still not able to repair themselves.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 7/10/25 um 14:59 schrieb Joachim Kraftmayer:
My first idea is that you have to increase the timeout, before the ceph mon will mark the mds as standby.
The error happens immediately after the start of the MDS: Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_channel(cluster) log [ERR] : No subtrees found for root MDS rank! The filesystem has been offline now for over two weeks… Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hello Robert, did you find a solution finally? One of our clusters has exactly the same behavior. It's Ceph 19.2.2. I am going to run all the cephf-journal-tool commands trying to fix that. There is nothing else I can find to solve this. Best, Malte On 7/14/25 14:38, Robert Sander wrote:
Am 7/10/25 um 14:59 schrieb Joachim Kraftmayer:
My first idea is that you have to increase the timeout, before the ceph mon will mark the mds as standby.
The error happens immediately after the start of the MDS:
Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_channel(cluster) log [ERR] : No subtrees found for root MDS rank!
The filesystem has been offline now for over two weeks…
Regards
Hi, how do I use the commands listed on https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#recovery-fro... when the cluster has multiple filesystems? Neither cephfs-table-tool nor cephfs-data-scan has an option to provide the filesystem name. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi,
https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#recovery-fro... when the cluster has multiple filesystems? Neither cephfs-table-tool nor cephfs-data-scan has an option to provide the
"... --rank=<cephfs-name>:0 ..." should work Regards Uwe On Mon, 14 Jul 2025, Robert Sander wrote:
Date: Mon, 14 Jul 2025 14:53:02 +0200 From: Robert Sander <r.sander@heinlein-support.de> To: ceph-users@ceph.io Subject: [ceph-users] Re: CephFS: no MDS does join the filesystem
Hi,
how do I use the commands listed on https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#recovery-fro... when the cluster has multiple filesystems?
Neither cephfs-table-tool nor cephfs-data-scan has an option to provide the filesystem name.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Uwe Richter Friedrich-Schiller-Universität Jena, Fakultät für Mathematik und Informatik, KSZ Ernst-Abbe-Platz 2, Zi. 3418, D-07740 Jena mailto:uwe.richter@uni-jena.de
Or even without --rank: # cephfs-table-tool secondfs:0 show session { "0": { "data": { "sessions": [] }, "result": 0 } } Zitat von Uwe Richter <uwe.richter@uni-jena.de>:
Hi,
https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#recovery-fro... when the cluster has multiple filesystems? Neither cephfs-table-tool nor cephfs-data-scan has an option to provide the
"... --rank=<cephfs-name>:0 ..." should work
Regards Uwe
On Mon, 14 Jul 2025, Robert Sander wrote:
Date: Mon, 14 Jul 2025 14:53:02 +0200 From: Robert Sander <r.sander@heinlein-support.de> To: ceph-users@ceph.io Subject: [ceph-users] Re: CephFS: no MDS does join the filesystem
Hi,
how do I use the commands listed on https://docs.ceph.com/en/reef/cephfs/disaster-recovery-experts/#recovery-fro... when the cluster has multiple filesystems?
Neither cephfs-table-tool nor cephfs-data-scan has an option to provide the filesystem name.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Uwe Richter Friedrich-Schiller-Universität Jena, Fakultät für Mathematik und Informatik, KSZ Ernst-Abbe-Platz 2, Zi. 3418, D-07740 Jena mailto:uwe.richter@uni-jena.de _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Am 7/14/25 um 15:06 schrieb Eugen Block:
# cephfs-table-tool secondfs:0 show session
That works. But what about cephfs-data-scan? # cephfs-data-scan init Specify a filesystem with --filesystem Error ((22) Invalid argument) # cephfs-data-scan --filesystem test init Unknown argument 'test' Error ((22) Invalid argument) # cephfs-data-scan --filesystem=test init Unknown argument 'init' Error ((22) Invalid argument) # cephfs-data-scan init --filesystem=test Unknown argument '--filesystem=test' Error ((22) Invalid argument) # cephfs-data-scan --help does not even list "--filesystem" as a valid argument. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Yeah, seems like the help output is incomplete. But this seems to work (use "init" before "--filesystem"): # cephfs-data-scan init --filesystem secondfs Inode 0x0x1 already exists, skipping create. Use --force-init to overwrite the existing object. Inode 0x0x100 already exists, skipping create. Use --force-init to overwrite the existing object. Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
Am 7/14/25 um 15:06 schrieb Eugen Block:
# cephfs-table-tool secondfs:0 show session
That works. But what about cephfs-data-scan?
# cephfs-data-scan init Specify a filesystem with --filesystem Error ((22) Invalid argument) # cephfs-data-scan --filesystem test init Unknown argument 'test' Error ((22) Invalid argument) # cephfs-data-scan --filesystem=test init Unknown argument 'init' Error ((22) Invalid argument) # cephfs-data-scan init --filesystem=test Unknown argument '--filesystem=test' Error ((22) Invalid argument)
# cephfs-data-scan --help does not even list "--filesystem" as a valid argument.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Bailey Allison
-
Eugen Block
-
Joachim Kraftmayer
-
Malte Stroem
-
Robert Sander
-
Uwe Richter