Best way to remove a node from cluster temp to replace faulty network cable
Hi Guys, We are getting an increasing number of RX errors on the ports on some of the nodes in the cluster including mon , mds and osd nodes In an attempt to fix it , we plan to replace SFPs / cables on these nodes. I think I will need to temp remove those nodes one by one from the cluster and replace the network cables / SFPs and add them back to cluster So, plz let me know the best / safest way to proceed. Thanks, Gagan
Hi, Am 21.11.25 um 5:11 AM schrieb gagan tiwari:
We are getting an increasing number of RX errors on the ports on some of the nodes in the cluster including mon , mds and osd nodes
In an attempt to fix it , we plan to replace SFPs / cables on these nodes.
I think I will need to temp remove those nodes one by one from the cluster and replace the network cables / SFPs and add them back to cluster
So, plz let me know the best / safest way to proceed.
With the cephadm orchestrator you can use ceph orch host maintenance enter HOSTNAME do your stuff ceph orch host maintenance exit HOSTNAME for each host in a row. Without the orchestrator you can use ceph osd set-group noout HOSTNAME do your stuff ceph osd unset-group noout HOSTNAME for each host. But never two hosts at the same time. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Thanks Robert for your advice. We are using cephadm container based set-up. So, ceph orch will do. Thanks, Gagan On Fri, Nov 21, 2025 at 1:08 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 21.11.25 um 5:11 AM schrieb gagan tiwari:
We are getting an increasing number of RX errors on
the
ports on some of the nodes in the cluster including mon , mds and osd nodes
In an attempt to fix it , we plan to replace SFPs / cables on these nodes.
I think I will need to temp remove those nodes one by one from the cluster and replace the network cables / SFPs and add them back to cluster
So, plz let me know the best / safest way to proceed.
With the cephadm orchestrator you can use
ceph orch host maintenance enter HOSTNAME
do your stuff
ceph orch host maintenance exit HOSTNAME
for each host in a row.
Without the orchestrator you can use
ceph osd set-group noout HOSTNAME
do your stuff
ceph osd unset-group noout HOSTNAME
for each host.
But never two hosts at the same time.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Gagan, There's a little bug with ceph orch host maintenance. https://github.com/ceph/ceph.io/issues/875 In my case (19.2.2) if you exit a host the command ends up in an uncought exception every single time. Just re-execute the exact same command again and the host exits maintenance mode as expected. Pretty scary the first time I saw this ;) . Wannes ________________________________ From: gagan tiwari <gagan.tiwari@mathisys-india.com> Sent: Friday, November 21, 2025 10:12 To: Robert Sander <r.sander@heinlein-support.de> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] Re: Best way to remove a node from cluster temp to replace faulty network cable Thanks Robert for your advice. We are using cephadm container based set-up. So, ceph orch will do. Thanks, Gagan On Fri, Nov 21, 2025 at 1: 08 PM Robert Sander <r. sander@ heinlein-support. de> wrote: > Hi, > > Am 21. 11. 25 um 5: 11 ZjQcmQRYFpfptBannerStart This Message Is From an Untrusted Sender You have not previously corresponded with this sender. ZjQcmQRYFpfptBannerEnd Thanks Robert for your advice. We are using cephadm container based set-up. So, ceph orch will do. Thanks, Gagan On Fri, Nov 21, 2025 at 1:08 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 21.11.25 um 5:11 AM schrieb gagan tiwari:
We are getting an increasing number of RX errors on
the
ports on some of the nodes in the cluster including mon , mds and osd nodes
In an attempt to fix it , we plan to replace SFPs / cables on these nodes.
I think I will need to temp remove those nodes one by one from the cluster and replace the network cables / SFPs and add them back to cluster
So, plz let me know the best / safest way to proceed.
With the cephadm orchestrator you can use
ceph orch host maintenance enter HOSTNAME
do your stuff
ceph orch host maintenance exit HOSTNAME
for each host in a row.
Without the orchestrator you can use
ceph osd set-group noout HOSTNAME
do your stuff
ceph osd unset-group noout HOSTNAME
for each host.
But never two hosts at the same time.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Honestly I think taking all the daemons down will be more disruptive than just staging the optics / cables and doing a hot-swap, which should take like 5 seconds. ymmv.
On Nov 21, 2025, at 2:38 AM, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 21.11.25 um 5:11 AM schrieb gagan tiwari:
We are getting an increasing number of RX errors on the ports on some of the nodes in the cluster including mon , mds and osd nodes In an attempt to fix it , we plan to replace SFPs / cables on these nodes. I think I will need to temp remove those nodes one by one from the cluster and replace the network cables / SFPs and add them back to cluster So, plz let me know the best / safest way to proceed.
With the cephadm orchestrator you can use
ceph orch host maintenance enter HOSTNAME
do your stuff
ceph orch host maintenance exit HOSTNAME
for each host in a row.
Without the orchestrator you can use
ceph osd set-group noout HOSTNAME
do your stuff
ceph osd unset-group noout HOSTNAME
for each host.
But never two hosts at the same time.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
gagan tiwari
-
Robert Sander
-
Wannes Smet