Ceph 14.2.22 MONs keep crashing in MDSMonitor::maybe_resize_cluster (out of range)
Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster. OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones. After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active. If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls): ``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ``` Anyone got any clues about how to diagnose or better-yet repair this? Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight. Thanks for at least reading this far, M0les.
Hi, correct, SUSE's Ceph product was Salt-based, in this case 14.2.22 was shipped with SES 6. ;-) Do you also have some mon logs from right before the crash, maybe with a higher debug level? It could make sense to stop client traffic and OSDs as well to be able to recover. But unfortunately, I can't really comment on the stack trace. Maybe someone has a different idea, but if you get one MON up, I would probably reduce the monmap to 1 MON to bring the cluster back up. Back up all the MON stores, just in case you have to start over. Then extract the monmap, remove all but one, and inject the modified monmap into the MON you want to revive. The procedure is described here [0]. Just don't change the address but only reduce the monmap. ;-) Regards, Eugen [0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-... Zitat von Miles Goodhew <ceph@m0les.com>:
Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster.
OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones.
After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active.
If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls):
``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ```
Anyone got any clues about how to diagnose or better-yet repair this?
Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight.
Thanks for at least reading this far,
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, Thanks for your response. Out of interest things that I've done overnight are stopping all the daemons (OSDs and RGWs were the ones still running) - so I'm just dealing with the 3 MONs now. Trying different start-sequences, I can determine: * mon3 was the last one working * Starting mon1 will kill mon3 (and prevent it starting) with that crash mentioned in the original email * Similarly starting mon2 will kill both mon1 and mon3 in the same way * Only mon3 gets the fast spamming of "e6 handle_auth_request failed to assign global_id" log messages when it's running. * Dumping the monmap results in the same file on all 3 mons. As for your suggestion of reducing the monmap to 1 node and rebuilding, we were also thinking of heading down that path. I'm hoping that deploying a temporary 4th mon on a new node might be able to get two nodes running (without killing the "old" one). Probably using mon3 because it's likely the most up-to-date. If that works, we could try clobbering and redeploying the other two "old" mon daemons and removing the temporary one to get back to the original 3 mons. As you say: using their original IP addresses (one of the clients is Openstack/RBD, which can be sentimental about mon IPs). I'm just in a bit of decision paralysis about which mon to take as the survivor. All can run _individually_, but only mon2 will survive a group start. mon3 was the last one working, but it has the mysterious "failed to assign global ID" errors. I'm leaning toward using mon3.. or mon2. Thanks for listening, M0les. On Wed, 18 Jun 2025, at 17:04, Eugen Block wrote:
Hi,
correct, SUSE's Ceph product was Salt-based, in this case 14.2.22 was shipped with SES 6. ;-)
Do you also have some mon logs from right before the crash, maybe with a higher debug level? It could make sense to stop client traffic and OSDs as well to be able to recover. But unfortunately, I can't really comment on the stack trace.
Maybe someone has a different idea, but if you get one MON up, I would probably reduce the monmap to 1 MON to bring the cluster back up. Back up all the MON stores, just in case you have to start over. Then extract the monmap, remove all but one, and inject the modified monmap into the MON you want to revive. The procedure is described here [0]. Just don't change the address but only reduce the monmap. ;-)
Regards, Eugen
[0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...
Zitat von Miles Goodhew <ceph@m0les.com>:
Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster.
OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones.
After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active.
If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls):
``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ```
Anyone got any clues about how to diagnose or better-yet repair this?
Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight.
Thanks for at least reading this far,
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Eugen, Sorry, I forgot to add that this is what the monmap looks like now (IPs/names sanitised): ``` min_mon_release 14 (nautilus) 0: [v2:IP_MON3:3300/0,v1:IP_MON3:6789/0] mon3 1: v1:IP_MON1:6789/0 mon1 2: v1:IP_MON2:6789/0 mon2 ``` Not sure why mon3 has the v2 + v1 setup and mon1/2 don't Thanks again, M0les. On Wed, 18 Jun 2025, at 17:42, Miles Goodhew wrote:
Hi Eugen, Thanks for your response.
Out of interest things that I've done overnight are stopping all the daemons (OSDs and RGWs were the ones still running) - so I'm just dealing with the 3 MONs now. Trying different start-sequences, I can determine:
* mon3 was the last one working * Starting mon1 will kill mon3 (and prevent it starting) with that crash mentioned in the original email * Similarly starting mon2 will kill both mon1 and mon3 in the same way * Only mon3 gets the fast spamming of "e6 handle_auth_request failed to assign global_id" log messages when it's running. * Dumping the monmap results in the same file on all 3 mons.
As for your suggestion of reducing the monmap to 1 node and rebuilding, we were also thinking of heading down that path. I'm hoping that deploying a temporary 4th mon on a new node might be able to get two nodes running (without killing the "old" one). Probably using mon3 because it's likely the most up-to-date. If that works, we could try clobbering and redeploying the other two "old" mon daemons and removing the temporary one to get back to the original 3 mons. As you say: using their original IP addresses (one of the clients is Openstack/RBD, which can be sentimental about mon IPs).
I'm just in a bit of decision paralysis about which mon to take as the survivor. All can run _individually_, but only mon2 will survive a group start. mon3 was the last one working, but it has the mysterious "failed to assign global ID" errors. I'm leaning toward using mon3.. or mon2.
Thanks for listening,
M0les.
On Wed, 18 Jun 2025, at 17:04, Eugen Block wrote:
Hi,
correct, SUSE's Ceph product was Salt-based, in this case 14.2.22 was shipped with SES 6. ;-)
Do you also have some mon logs from right before the crash, maybe with a higher debug level? It could make sense to stop client traffic and OSDs as well to be able to recover. But unfortunately, I can't really comment on the stack trace.
Maybe someone has a different idea, but if you get one MON up, I would probably reduce the monmap to 1 MON to bring the cluster back up. Back up all the MON stores, just in case you have to start over. Then extract the monmap, remove all but one, and inject the modified monmap into the MON you want to revive. The procedure is described here [0]. Just don't change the address but only reduce the monmap. ;-)
Regards, Eugen
[0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...
Zitat von Miles Goodhew <ceph@m0les.com>:
Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster.
OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones.
After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active.
If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls):
``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ```
Anyone got any clues about how to diagnose or better-yet repair this?
Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight.
Thanks for at least reading this far,
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That does look strange indeed, either an upgrade went wrong or someone already fiddled with the monmap, I'd say. But anyway, I wouldn't try to deploy a 4th mon since it would want to sync the store, but we don't know in which state the store actually is. And besides from that, 2 out of 4 MONs still isn't a quorum, so there's no real benefit. So my best bet would be on the mon with the most recent store. And if the cluster comes back up with one mon, you'll need to wipe the traces of the previous mons so DeepSea can redeploy additional mons cleanly. Or is the cluster not managed by DeepSea anymore? Zitat von Miles Goodhew <ceph@m0les.com>:
Eugen, Sorry, I forgot to add that this is what the monmap looks like now (IPs/names sanitised):
``` min_mon_release 14 (nautilus) 0: [v2:IP_MON3:3300/0,v1:IP_MON3:6789/0] mon3 1: v1:IP_MON1:6789/0 mon1 2: v1:IP_MON2:6789/0 mon2 ```
Not sure why mon3 has the v2 + v1 setup and mon1/2 don't
Thanks again,
M0les.
On Wed, 18 Jun 2025, at 17:42, Miles Goodhew wrote:
Hi Eugen, Thanks for your response.
Out of interest things that I've done overnight are stopping all the daemons (OSDs and RGWs were the ones still running) - so I'm just dealing with the 3 MONs now. Trying different start-sequences, I can determine:
* mon3 was the last one working * Starting mon1 will kill mon3 (and prevent it starting) with that crash mentioned in the original email * Similarly starting mon2 will kill both mon1 and mon3 in the same way * Only mon3 gets the fast spamming of "e6 handle_auth_request failed to assign global_id" log messages when it's running. * Dumping the monmap results in the same file on all 3 mons.
As for your suggestion of reducing the monmap to 1 node and rebuilding, we were also thinking of heading down that path. I'm hoping that deploying a temporary 4th mon on a new node might be able to get two nodes running (without killing the "old" one). Probably using mon3 because it's likely the most up-to-date. If that works, we could try clobbering and redeploying the other two "old" mon daemons and removing the temporary one to get back to the original 3 mons. As you say: using their original IP addresses (one of the clients is Openstack/RBD, which can be sentimental about mon IPs).
I'm just in a bit of decision paralysis about which mon to take as the survivor. All can run _individually_, but only mon2 will survive a group start. mon3 was the last one working, but it has the mysterious "failed to assign global ID" errors. I'm leaning toward using mon3.. or mon2.
Thanks for listening,
M0les.
On Wed, 18 Jun 2025, at 17:04, Eugen Block wrote:
Hi,
correct, SUSE's Ceph product was Salt-based, in this case 14.2.22 was shipped with SES 6. ;-)
Do you also have some mon logs from right before the crash, maybe with a higher debug level? It could make sense to stop client traffic and OSDs as well to be able to recover. But unfortunately, I can't really comment on the stack trace.
Maybe someone has a different idea, but if you get one MON up, I would probably reduce the monmap to 1 MON to bring the cluster back up. Back up all the MON stores, just in case you have to start over. Then extract the monmap, remove all but one, and inject the modified monmap into the MON you want to revive. The procedure is described here [0]. Just don't change the address but only reduce the monmap. ;-)
Regards, Eugen
[0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...
Zitat von Miles Goodhew <ceph@m0les.com>:
Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster.
OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones.
After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active.
If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls):
``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ```
Anyone got any clues about how to diagnose or better-yet repair this?
Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight.
Thanks for at least reading this far,
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Although I'm not a dev, I looked into the code [0] anyway. The comments before the maybe_resize_cluster function say: * If a cluster is undersized (with respect to max_mds), then * attempt to find daemons to grow it. If the cluster is oversized * (with respect to max_mds) then shrink it by stopping its highest rank. Is it possible that an operator/admin tried to resize (shrink or grow the number of MDS daemons) the MDS culster? Or was a DeepSea stage executed in order to deploy additional daemons? Maybe some history could help understand what might have happened. [0] https://github.com/ceph/ceph/blob/v14.2.22/src/mon/MDSMonitor.cc#L1801 Zitat von Eugen Block <eblock@nde.ag>:
That does look strange indeed, either an upgrade went wrong or someone already fiddled with the monmap, I'd say. But anyway, I wouldn't try to deploy a 4th mon since it would want to sync the store, but we don't know in which state the store actually is. And besides from that, 2 out of 4 MONs still isn't a quorum, so there's no real benefit. So my best bet would be on the mon with the most recent store. And if the cluster comes back up with one mon, you'll need to wipe the traces of the previous mons so DeepSea can redeploy additional mons cleanly. Or is the cluster not managed by DeepSea anymore?
Zitat von Miles Goodhew <ceph@m0les.com>:
Eugen, Sorry, I forgot to add that this is what the monmap looks like now (IPs/names sanitised):
``` min_mon_release 14 (nautilus) 0: [v2:IP_MON3:3300/0,v1:IP_MON3:6789/0] mon3 1: v1:IP_MON1:6789/0 mon1 2: v1:IP_MON2:6789/0 mon2 ```
Not sure why mon3 has the v2 + v1 setup and mon1/2 don't
Thanks again,
M0les.
On Wed, 18 Jun 2025, at 17:42, Miles Goodhew wrote:
Hi Eugen, Thanks for your response.
Out of interest things that I've done overnight are stopping all the daemons (OSDs and RGWs were the ones still running) - so I'm just dealing with the 3 MONs now. Trying different start-sequences, I can determine:
* mon3 was the last one working * Starting mon1 will kill mon3 (and prevent it starting) with that crash mentioned in the original email * Similarly starting mon2 will kill both mon1 and mon3 in the same way * Only mon3 gets the fast spamming of "e6 handle_auth_request failed to assign global_id" log messages when it's running. * Dumping the monmap results in the same file on all 3 mons.
As for your suggestion of reducing the monmap to 1 node and rebuilding, we were also thinking of heading down that path. I'm hoping that deploying a temporary 4th mon on a new node might be able to get two nodes running (without killing the "old" one). Probably using mon3 because it's likely the most up-to-date. If that works, we could try clobbering and redeploying the other two "old" mon daemons and removing the temporary one to get back to the original 3 mons. As you say: using their original IP addresses (one of the clients is Openstack/RBD, which can be sentimental about mon IPs).
I'm just in a bit of decision paralysis about which mon to take as the survivor. All can run _individually_, but only mon2 will survive a group start. mon3 was the last one working, but it has the mysterious "failed to assign global ID" errors. I'm leaning toward using mon3.. or mon2.
Thanks for listening,
M0les.
On Wed, 18 Jun 2025, at 17:04, Eugen Block wrote:
Hi,
correct, SUSE's Ceph product was Salt-based, in this case 14.2.22 was shipped with SES 6. ;-)
Do you also have some mon logs from right before the crash, maybe with a higher debug level? It could make sense to stop client traffic and OSDs as well to be able to recover. But unfortunately, I can't really comment on the stack trace.
Maybe someone has a different idea, but if you get one MON up, I would probably reduce the monmap to 1 MON to bring the cluster back up. Back up all the MON stores, just in case you have to start over. Then extract the monmap, remove all but one, and inject the modified monmap into the MON you want to revive. The procedure is described here [0]. Just don't change the address but only reduce the monmap. ;-)
Regards, Eugen
[0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...
Zitat von Miles Goodhew <ceph@m0les.com>:
Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster.
OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones.
After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active.
If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls):
``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ```
Anyone got any clues about how to diagnose or better-yet repair this?
Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight.
Thanks for at least reading this far,
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, 18 Jun 2025, at 18:09, Eugen Block wrote:
That does look strange indeed, either an upgrade went wrong or someone already fiddled with the monmap, I'd say. But anyway, I wouldn't try to deploy a 4th mon since it would want to sync the store, but we don't know in which state the store actually is. And besides from that, 2 out of 4 MONs still isn't a quorum, so there's no real benefit. So my best bet would be on the mon with the most recent store. And if the cluster comes back up with one mon, you'll need to wipe the traces of the previous mons so DeepSea can redeploy additional mons cleanly. Or is the cluster not managed by DeepSea anymore?
Replies to fragments from above are below:
either an upgrade went wrong or someone already fiddled with the monmap
That's entirely possible. I'm playing the role of a "guy who knows a bit about Ceph" to try and un-explode an old cluster on unsupported OS and hardware. The original deployers are long-since gone and the day-to-day admins were never given much handover. There are legends of several phases of upgrades and deployment system replacements, but concrete documentation is thin on the ground. Certainly I recently found evidence of failed OS upgrades that broke part of the RGW services years ago. I had previously documented a plan to migrate the cluster to a new/supported hardware, OS and Ceph version, but the client was still thinking about it when this happened.
wouldn't try to deploy a 4th mon The idea for the 4th MON was to just see if I can deploy a new MON without breaking the cluster much more. However given I can't get >1 MONs to start, it's pretty broken right now. If that deployment worked, I intended then to remove/redeploy each of the other two MONs before retiring the 4th MON again. A side-benefit of this is that it lets me test some of my cluster upgrade plan. One of the cluster clients is Openstack, which in my experience is pretty "sentimental" about its set of MON IPs.
mon with the most recent store How would I find out which MON that is? I'm told mon3 was the last one operating (but it gives the wall of "e6 handle_auth_request failed to assign global_id" logs when running). mon2 is the one that survives if you try to start all of them. I've tried inspecting the (SQLite?) DBs, but can't get much comprehensible info out of them yet (I don't have any experience tinkering with SQLite, but I'm OK with an "actual" SQL repl). I can't get quorum, so I can't run "ceph ..." command lines, but I can calk to each of the MONs on their Unix sockets when they're running.
Or is the cluster not managed by DeepSea anymore? I don't think it is. None of the admins (nor I) have very deep experience in Salt stuff (I'm more Ansible). The aforementioned "legends" of the system's lifetime also say there were multiple different management systems over the years. I've mainly used the existing Salt config to break-into managed nodes I didn't yet have an account for and to do fleet-wide "shell command" operations. Given the probability of historic broken OS upgrades and possibly abandoned Salt management, I'd be wary of trying to use this for deployment automation.
(Now in a later email)
Although I'm not a dev, I looked into the code [0] anyway.
The comments before the maybe_resize_cluster function say:
* If a cluster is undersized (with respect to max_mds), then * attempt to find daemons to grow it. If the cluster is oversized * (with respect to max_mds) then shrink it by stopping its highest rank.
Is it possible that an operator/admin tried to resize (shrink or grow the number of MDS daemons) the MDS culster? Or was a DeepSea stage executed in order to deploy additional daemons? Maybe some history could help understand what might have happened."
Yes, I saw all that too. I was told that this all started because one of the admins noticed that the CephFS service was slow and was reporting laggy MDSes. This may well be a latent issue from possible historical failed upgrade (pure guesses here). The admin tried restarting some daemons and eventually only mon2 would run (I'm a bit vague on the detail). I don't *think* they tried removing the MDS daemons, but it's possible (I'll check tomorrow). One of my possible plans of attack was to see if that "maybe resize..." method might be skipped with some "No"-flag or other config. Hopefully then to try and get quorum established before re-enabling it and possibly coming back to health. This is probably too wishful a prospect, though. Thanks again for all your feedback. Even if this just turns out to be a massive "rubber ducking" session, you've given me some new ideas and threads to pull. My main question now is "which is the 'latest' MON?" M0les.
My main question now is "which is the 'latest' MON?"
Timestamp of the files within the mon db store. ;-) no need to dig through the db itself. If you don't feel confident to manage the cluster with DeepSea (I know of people who were literally afraid of DeepSea stages :-D ), then don't. :-) Without cephadm you can try to deploy daemons relatively easy. Two years ago I wrote an article [1] how to migrate from SES 6 (Nautilus) to Upstream (Pacific) after testing this procedure for a potential customer. You can reach out to me if you should get to that point. I'm still in favor of the single mon approach, it has worked many times for us (actually it always worked), and it's relatively quick and easy to test. If that shouldn't work, there's still the procedure to collect the maps from the OSDs to create a new mon store. But let's see how far you get before exploring this option. [1] https://heiterbiswolkig.blogs.nde.ag/2023/08/14/how-to-migrate-from-suse-ent... Zitat von Miles Goodhew <ceph@m0les.com>:
On Wed, 18 Jun 2025, at 18:09, Eugen Block wrote:
That does look strange indeed, either an upgrade went wrong or someone already fiddled with the monmap, I'd say. But anyway, I wouldn't try to deploy a 4th mon since it would want to sync the store, but we don't know in which state the store actually is. And besides from that, 2 out of 4 MONs still isn't a quorum, so there's no real benefit. So my best bet would be on the mon with the most recent store. And if the cluster comes back up with one mon, you'll need to wipe the traces of the previous mons so DeepSea can redeploy additional mons cleanly. Or is the cluster not managed by DeepSea anymore?
Replies to fragments from above are below:
either an upgrade went wrong or someone already fiddled with the monmap
That's entirely possible. I'm playing the role of a "guy who knows a bit about Ceph" to try and un-explode an old cluster on unsupported OS and hardware. The original deployers are long-since gone and the day-to-day admins were never given much handover. There are legends of several phases of upgrades and deployment system replacements, but concrete documentation is thin on the ground. Certainly I recently found evidence of failed OS upgrades that broke part of the RGW services years ago.
I had previously documented a plan to migrate the cluster to a new/supported hardware, OS and Ceph version, but the client was still thinking about it when this happened.
wouldn't try to deploy a 4th mon The idea for the 4th MON was to just see if I can deploy a new MON without breaking the cluster much more. However given I can't get >1 MONs to start, it's pretty broken right now. If that deployment worked, I intended then to remove/redeploy each of the other two MONs before retiring the 4th MON again. A side-benefit of this is that it lets me test some of my cluster upgrade plan. One of the cluster clients is Openstack, which in my experience is pretty "sentimental" about its set of MON IPs.
mon with the most recent store How would I find out which MON that is? I'm told mon3 was the last one operating (but it gives the wall of "e6 handle_auth_request failed to assign global_id" logs when running). mon2 is the one that survives if you try to start all of them. I've tried inspecting the (SQLite?) DBs, but can't get much comprehensible info out of them yet (I don't have any experience tinkering with SQLite, but I'm OK with an "actual" SQL repl). I can't get quorum, so I can't run "ceph ..." command lines, but I can calk to each of the MONs on their Unix sockets when they're running.
Or is the cluster not managed by DeepSea anymore? I don't think it is. None of the admins (nor I) have very deep experience in Salt stuff (I'm more Ansible). The aforementioned "legends" of the system's lifetime also say there were multiple different management systems over the years. I've mainly used the existing Salt config to break-into managed nodes I didn't yet have an account for and to do fleet-wide "shell command" operations. Given the probability of historic broken OS upgrades and possibly abandoned Salt management, I'd be wary of trying to use this for deployment automation.
(Now in a later email)
Although I'm not a dev, I looked into the code [0] anyway.
The comments before the maybe_resize_cluster function say:
* If a cluster is undersized (with respect to max_mds), then * attempt to find daemons to grow it. If the cluster is oversized * (with respect to max_mds) then shrink it by stopping its highest rank.
Is it possible that an operator/admin tried to resize (shrink or grow the number of MDS daemons) the MDS culster? Or was a DeepSea stage executed in order to deploy additional daemons? Maybe some history could help understand what might have happened."
Yes, I saw all that too. I was told that this all started because one of the admins noticed that the CephFS service was slow and was reporting laggy MDSes. This may well be a latent issue from possible historical failed upgrade (pure guesses here). The admin tried restarting some daemons and eventually only mon2 would run (I'm a bit vague on the detail). I don't *think* they tried removing the MDS daemons, but it's possible (I'll check tomorrow).
One of my possible plans of attack was to see if that "maybe resize..." method might be skipped with some "No"-flag or other config. Hopefully then to try and get quorum established before re-enabling it and possibly coming back to health. This is probably too wishful a prospect, though.
Thanks again for all your feedback. Even if this just turns out to be a massive "rubber ducking" session, you've given me some new ideas and threads to pull. My main question now is "which is the 'latest' MON?"
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, 19 Jun 2025, at 00:04, Eugen Block wrote:
My main question now is "which is the 'latest' MON?"
Timestamp of the files within the mon db store. ;-) no need to dig through the db itself.
That's what I thought too. Sadly all the DBs get touched when the MONs start, so they're all dated "yesterday". Despite me saying "It's SQLite" - now I see this is all levelDB (don't know where I got confused). I wrote some janky Python code using plyvel to dump-out the keys and kv pairs as b-strings. Seems mon1 and mon2 have an identical key set. There are 2 keys that differ between mon1/2 and mon3 as well as 3 keys that differ on all 3 hosts: mon1/2: b'monitor\x00election_epoch' => b'\x16\x0c\x00\x00\x00\x00\x00\x00' # 3094 if it's a 64-bit little-endian integer mon3: b'monitor\x00election_epoch' => b'\x12\x0c\x00\x00\x00\x00\x00\x00' # 3090 mon1/2: b'monmap\x00last_committed' => b'\x04\x00\x00\x00\x00\x00\x00\x00' # 4 mon3: b'monmap\x00last_committed' => b'\x06\x00\x00\x00\x00\x00\x00\x00' # 6 mon1; b'monitor\x00election_writeable_test' => b'\xee\xeaI\\\x00\x00\x00\x00' mon2: b'monitor\x00election_writeable_test' => b'\xb2\x8a\xd6\x06\x00\x00\x00\x00' mon3: b'monitor\x00election_writeable_test' => b'i\x98<d\x00\x00\x00\x00' mon1: b'paxos\x00accepted_pn' => b'tS\x01\x00\x00\x00\x00\x00' # 86900 mon2: b'paxos\x00accepted_pn' => b'1V\x01\x00\x00\x00\x00\x00' # 87601 mon3: b'paxos\x00accepted_pn' => b'\xc0>\x01\x00\x00\x00\x00\x00' # 81600 mon1: b'paxos\x00last_pn' => b'1V\x01\x00\x00\x00\x00\x00' # 87601 mon2: b'paxos\x00last_pn' => b'\xf9=\x01\x00\x00\x00\x00\x00' # 81401 mon3: b'paxos\x00last_pn' => b'\xccU\x01\x00\x00\x00\x00\x00' # 87500 (Note all those integer decodes were done by hand and are subject to errors) Also there is one key in mon1/2 that's not in mon3 and 3 keys in mon3 that aren't in mon1/2: mon1/2: b'mon_sync\x00temp_newer_monmap' mon3: b'monmap\x005' mon3: b'monmap\x006' mon3: b'monmap\x00latest' So: Given that mon1/2 mention "mon_sync.tmp_newer_monmap" and mon3 has "monmap.latest", I feel like mon3 is the most complete and up-to-date. But really, this is all just "digital phrenology".
If you don't feel confident to manage the cluster with DeepSea (I know of people who were literally afraid of DeepSea stages :-D ), then don't. :-) Without cephadm you can try to deploy daemons relatively easy. Two years ago I wrote an article [1] how to migrate from SES 6 (Nautilus) to Upstream (Pacific) after testing this procedure for a potential customer. You can reach out to me if you should get to that point.
That looks totally like what I'm planning. Thanks!
I'm still in favor of the single mon approach, it has worked many times for us (actually it always worked), and it's relatively quick and easy to test. If that shouldn't work, there's still the procedure to collect the maps from the OSDs to create a new mon store. But let's see how far you get before exploring this option.
Yes, thanks again for all your great help. M0les.
All, An update: I tried editing the monmap on mon3 to be just [mon3] and starting the mon daemon there. It died immediately with the "maybe_resize_cluster" method exception mentioned before. I then tried the same trick on mon2 (that's the daemon that would stay up if all 3 are started) and got the same behaviour. I'm guessing that mon2 is only "running" because it's waiting for a response from the paxos election (and none of the others is responding by now). If we remove that block, it goes straight to the "maybe_resize_cluster" call and gets the same "out of range" exception. I suspect the issue is with whatever "max_mds" is and what is "out of range". Also, I discovered "ceph-kvstore-tool", which makes looking in the leveldb a lot easier. Thanks, M0les.
Too bad. :-/ Could you increase the debug log level to 20? Maybe it gets a bit clearer where exactly it fails. Just to understand the current situation, you did reduce the monmap to 1 (mon3), then you tried the same with mon2. Because when you write:
I'm guessing that mon2 is only "running" because it's waiting for a response from the paxos election (and none of the others is responding by now).
it reads like you did try to start all of them again. If you only have mon in the monmap, there's no waiting for an election. Zitat von Miles Goodhew <ceph@m0les.com>:
All, An update: I tried editing the monmap on mon3 to be just [mon3] and starting the mon daemon there. It died immediately with the "maybe_resize_cluster" method exception mentioned before. I then tried the same trick on mon2 (that's the daemon that would stay up if all 3 are started) and got the same behaviour. I'm guessing that mon2 is only "running" because it's waiting for a response from the paxos election (and none of the others is responding by now). If we remove that block, it goes straight to the "maybe_resize_cluster" call and gets the same "out of range" exception.
I suspect the issue is with whatever "max_mds" is and what is "out of range".
Also, I discovered "ceph-kvstore-tool", which makes looking in the leveldb a lot easier.
Thanks,
M0les.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, 19 Jun 2025, at 17:48, Eugen Block wrote:
Too bad. :-/ Could you increase the debug log level to 20? Maybe it gets a bit clearer where exactly it fails.
I guess that's in `ceph.conf` with: [mon] debug_mon = 20 ? Good thinking: I'll try that tomorrow.
Just to understand the current situation, you did reduce the monmap to 1 (mon3), then you tried the same with mon2. Because when you write:
I'm guessing that mon2 is only "running" because it's waiting for a response from the paxos election (and none of the others is responding by now).
it reads like you did try to start all of them again. If you only have mon in the monmap, there's no waiting for an election.
Yes, sorry - I was mixing time contexts there. What I tried today: 1: Reduce monmap on mon3 to just [mon3] and try to start mon3 (immediate crash of mon3) 2: reduce monmap on mon2 to just [mon2] and try to start mon2 (immediate crash of mon2 - used to look like it was "running" before) That talk of "mon2 is only 'running'..." was concerning the original state where starting all daemons results in mon1 and mon3 dieing and mon2 apparently "running" (but I think it was just waiting for an election or something). So I think the problem lies in the MDS config, but it's a bit hazy. Thanks again, M0les.
Zitat von Miles Goodhew <ceph@m0les.com>:
On Thu, 19 Jun 2025, at 17:48, Eugen Block wrote:
Too bad. :-/ Could you increase the debug log level to 20? Maybe it gets a bit clearer where exactly it fails.
I guess that's in `ceph.conf` with:
[mon] debug_mon = 20 ?
Correct.
Good thinking: I'll try that tomorrow.
Just to understand the current situation, you did reduce the monmap to 1 (mon3), then you tried the same with mon2. Because when you write:
I'm guessing that mon2 is only "running" because it's waiting for a response from the paxos election (and none of the others is responding by now).
it reads like you did try to start all of them again. If you only have mon in the monmap, there's no waiting for an election.
Yes, sorry - I was mixing time contexts there. What I tried today: 1: Reduce monmap on mon3 to just [mon3] and try to start mon3 (immediate crash of mon3) 2: reduce monmap on mon2 to just [mon2] and try to start mon2 (immediate crash of mon2 - used to look like it was "running" before)
That talk of "mon2 is only 'running'..." was concerning the original state where starting all daemons results in mon1 and mon3 dieing and mon2 apparently "running" (but I think it was just waiting for an election or something).
So I think the problem lies in the MDS config, but it's a bit hazy.
Thanks again,
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thu, 19 Jun 2025, at 18:39, Eugen Block wrote:
Zitat von Miles Goodhew <ceph@m0les.com>:
On Thu, 19 Jun 2025, at 17:48, Eugen Block wrote:
Too bad. :-/ Could you increase the debug log level to 20? Maybe it gets a bit clearer where exactly it fails.
I guess that's in `ceph.conf` with:
[mon] debug_mon = 20 ?
Correct.
Some progress has been made! The mon.mds print_map() output shows an "in" set of [0,1,2,3] (i.e. size 4, but only 2 are actually perceived as "up") and a max_mds value of 2. With the log level increased to 20, the last dout log we see is on line 1810 ("in 4 max 2"). So none of the other 4 dout logs are seen from lines 1818, 1834, 1847 or 1855. So that must mean that one of the calls at lines 1816 (mds_map.isresizable x 2), 1845 (mds_map.get_info) or 1846 (mds_map.is_active) would have to cause the crash. We were toying with methods to see if we could set values to drop-out of this code earlier and it was decided that the MDS service was not the most important part of the cluster (The Openstack cluster on top of it was more important). So for a test, we used the `ceph-kvstore-tool` to just trim-off the "mds*" prefixes from the DB: ``` ceph-kvstore-tool leveldb ${DB_PATH} rm-prefix mdsmap ceph-kvstore-tool leveldb ${DB_PATH} rm-prefix mds_health ceph-kvstore-tool leveldb ${DB_PATH} rm-prefix mds_metadata ceph-kvstore-tool leveldb ${DB_PATH} rm health mdsmap ``` (I suspect the "mdsmap" part was the most important, but we're mostly just going by feel at this level). To our surprised delight, the MONs all started-up and formed a quorum. We then started-up all the MGRs without issue. We progressively started all the OSDs (minor rebalancing from an actually unhealthy disk). The cluster got back to a "nominally operational except for CephFS" state and the Openstack cluster was verified and repaired. All green at COB. The RGW services restarted and verified operational by their clients. So we're leaving it like this for now and conducting a review on Monday. There are a few immediate bits of maintenance that can be done, but this whole incident puts a fire under the "let's get this updated and moved to supported OS/Hardware" plan. Thanks again for all your help, Eugen - much appreciated!!! M0les.
Cool, that's fantastic news! And a great analysis, too! I'm glad you got it back up and client operations could resume. Happy to help! Zitat von Miles Goodhew <ceph@m0les.com>:
On Thu, 19 Jun 2025, at 18:39, Eugen Block wrote:
Zitat von Miles Goodhew <ceph@m0les.com>:
On Thu, 19 Jun 2025, at 17:48, Eugen Block wrote:
Too bad. :-/ Could you increase the debug log level to 20? Maybe it gets a bit clearer where exactly it fails.
I guess that's in `ceph.conf` with:
[mon] debug_mon = 20 ?
Correct.
Some progress has been made! The mon.mds print_map() output shows an "in" set of [0,1,2,3] (i.e. size 4, but only 2 are actually perceived as "up") and a max_mds value of 2. With the log level increased to 20, the last dout log we see is on line 1810 ("in 4 max 2"). So none of the other 4 dout logs are seen from lines 1818, 1834, 1847 or 1855. So that must mean that one of the calls at lines 1816 (mds_map.isresizable x 2), 1845 (mds_map.get_info) or 1846 (mds_map.is_active) would have to cause the crash.
We were toying with methods to see if we could set values to drop-out of this code earlier and it was decided that the MDS service was not the most important part of the cluster (The Openstack cluster on top of it was more important).
So for a test, we used the `ceph-kvstore-tool` to just trim-off the "mds*" prefixes from the DB:
``` ceph-kvstore-tool leveldb ${DB_PATH} rm-prefix mdsmap ceph-kvstore-tool leveldb ${DB_PATH} rm-prefix mds_health ceph-kvstore-tool leveldb ${DB_PATH} rm-prefix mds_metadata ceph-kvstore-tool leveldb ${DB_PATH} rm health mdsmap ```
(I suspect the "mdsmap" part was the most important, but we're mostly just going by feel at this level).
To our surprised delight, the MONs all started-up and formed a quorum. We then started-up all the MGRs without issue. We progressively started all the OSDs (minor rebalancing from an actually unhealthy disk).
The cluster got back to a "nominally operational except for CephFS" state and the Openstack cluster was verified and repaired. All green at COB. The RGW services restarted and verified operational by their clients.
So we're leaving it like this for now and conducting a review on Monday. There are a few immediate bits of maintenance that can be done, but this whole incident puts a fire under the "let's get this updated and moved to supported OS/Hardware" plan.
Thanks again for all your help, Eugen - much appreciated!!!
M0les. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Miles Goodhew