HI Eugen, We have 80 osds in the cluster. So, to stop them I will need to run the command *ceph orch stop osd.<ID>* for all 80 osds one by one. Is there any way to stop all of them in one command ? Also, once all nodes will be back after power maintenance activity , will I need to start all daemons mon , mds , osd,etc ? or will they be up automatically once servers will be up ? Thanks, Gagan On Wed, Aug 6, 2025 at 4:45 PM Eugen Block <eblock@nde.ag> wrote:

Although SUSE discontinued their product, the procedure is still correct [0]:

1. Tell the Ceph cluster not to mark OSDs as out:

ceph osd set noout

2. Stop daemons and nodes in the following order:

Storage clients

Gateways, for example NFS Ganesha or Object Gateway

Metadata Server

Ceph OSD

Ceph Manager

Ceph Monitor

3. If required, perform maintenance tasks.

4. Start the nodes and servers in the reverse order of the shutdown process:

Ceph Monitor

Ceph Manager

Ceph OSD

Metadata Server

Gateways, for example NFS Ganesha or Object Gateway

Storage clients

5. Remove the noout flag:

ceph osd unset noout

[0]

https://documentation.suse.com/en-us/ses/7.1/html/ses-all/storage-salt-clust...

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

Hi Guys, I have recently set-up a production ceph cluster which consists of 3 monitor nodes and 7 Osd nodes.

There is power maintenance activity scheduled at the data centre coming weekend and due to that I need to power off all the devices.

Can you please advise me on the safest way to power off all servers?

Should I power off all 7 OSD servers one by one followed by all 3 monitor nodes or vice versa?

Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Read this blog post on how (not) to shut down a Ceph cluster https://www.croit.io/blog/how-not-to-shut-down-a-ceph-cluster On Wed, 6 Aug 2025 at 21:24, Eugen Block <eblock@nde.ag> wrote:

You don't need to stop all the OSDs (or other daemons) manually, just shut down the servers (most likely you have services colocated). When they boot again, the Ceph daemons will also start automatically (they're handled by systemd). I can check tomorrow which steps exactly our shutdown procedure consists of when we have planned power outages etc.

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

HI Eugen, We have 80 osds in the cluster.

So, to stop them I will need to run the command *ceph orch stop osd.<ID>* for all 80 osds one by one. Is there any way to stop all of them in one command ?

Also, once all nodes will be back after power maintenance activity , will I need to start all daemons mon , mds , osd,etc ? or will they be up automatically once servers will be up ?

Thanks, Gagan

On Wed, Aug 6, 2025 at 4:45 PM Eugen Block <eblock@nde.ag> wrote:

...

Although SUSE discontinued their product, the procedure is still correct [0]:

1. Tell the Ceph cluster not to mark OSDs as out:

ceph osd set noout

2. Stop daemons and nodes in the following order:

Storage clients

Gateways, for example NFS Ganesha or Object Gateway

Metadata Server

Ceph OSD

Ceph Manager

Ceph Monitor

3. If required, perform maintenance tasks.

4. Start the nodes and servers in the reverse order of the shutdown process:

Ceph Monitor

Ceph Manager

Ceph OSD

Metadata Server

Gateways, for example NFS Ganesha or Object Gateway

Storage clients

5. Remove the noout flag:

ceph osd unset noout

[0]

https://documentation.suse.com/en-us/ses/7.1/html/ses-all/storage-salt-clust...

...

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

Hi Guys, I have recently set-up a production ceph cluster

which

...

...

consists of 3 monitor nodes and 7 Osd nodes.

There is power maintenance activity scheduled at the data centre coming weekend and due to that I need to power off all the devices.

Can you please advise me on the safest way to power off all servers?

Should I power off all 7 OSD servers one by one followed by all 3 monitor nodes or vice versa?

Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, I had deployed the cluster via cephadm. so, it's container based. Just fyi.. So, you meant I don't need to run any commands like "ceph osd set noout" , "ceph orch stop osd.<ID>" ,etc. I just need to power off all nodes starting clients nodes , osd nodes , mon , mds nodes and power them on after maintenance activity is over. Thanks, Gagan On Wed, Aug 6, 2025 at 11:53 PM Eugen Block <eblock@nde.ag> wrote:

You don't need to stop all the OSDs (or other daemons) manually, just shut down the servers (most likely you have services colocated). When they boot again, the Ceph daemons will also start automatically (they're handled by systemd). I can check tomorrow which steps exactly our shutdown procedure consists of when we have planned power outages etc.

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

HI Eugen, We have 80 osds in the cluster.

So, to stop them I will need to run the command *ceph orch stop osd.<ID>* for all 80 osds one by one. Is there any way to stop all of them in one command ?

Also, once all nodes will be back after power maintenance activity , will I need to start all daemons mon , mds , osd,etc ? or will they be up automatically once servers will be up ?

Thanks, Gagan

On Wed, Aug 6, 2025 at 4:45 PM Eugen Block <eblock@nde.ag> wrote:

...

Although SUSE discontinued their product, the procedure is still correct [0]:

1. Tell the Ceph cluster not to mark OSDs as out:

ceph osd set noout

2. Stop daemons and nodes in the following order:

Storage clients

Gateways, for example NFS Ganesha or Object Gateway

Metadata Server

Ceph OSD

Ceph Manager

Ceph Monitor

3. If required, perform maintenance tasks.

4. Start the nodes and servers in the reverse order of the shutdown process:

Ceph Monitor

Ceph Manager

Ceph OSD

Metadata Server

Gateways, for example NFS Ganesha or Object Gateway

Storage clients

5. Remove the noout flag:

ceph osd unset noout

[0]

https://documentation.suse.com/en-us/ses/7.1/html/ses-all/storage-salt-clust...

...

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

Hi Guys, I have recently set-up a production ceph cluster

which

...

...

consists of 3 monitor nodes and 7 Osd nodes.

There is power maintenance activity scheduled at the data centre coming weekend and due to that I need to power off all the devices.

Can you please advise me on the safest way to power off all servers?

Should I power off all 7 OSD servers one by one followed by all 3 monitor nodes or vice versa?

Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, Thanks for clarifying! I will just set the noout flag and power off all servers. Hopefully, it will work fine. Thanks, Gagan On Thu, Aug 7, 2025 at 5:59 PM Eugen Block <eblock@nde.ag> wrote:

No, that's not what I meant. Of course you need the noout flag, that's the first point on the list I sent. Other flags are not required as pointed out by croit. I have never disabled cephfs before shutting it down as mentioned by Gerard, so I can't comment on that.

The procedure I quoted from SUSE is still working fine, that's how we power off our (cephadm) based clusters as well.

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

Hi Eugen, I had deployed the cluster via cephadm. so, it's container based. Just fyi..

So, you meant I don't need to run any commands like "ceph osd set noout" , "ceph orch stop osd.<ID>" ,etc.

I just need to power off all nodes starting clients nodes , osd nodes , mon , mds nodes and power them on after maintenance activity is over.

Thanks, Gagan

On Wed, Aug 6, 2025 at 11:53 PM Eugen Block <eblock@nde.ag> wrote:

...

You don't need to stop all the OSDs (or other daemons) manually, just shut down the servers (most likely you have services colocated). When they boot again, the Ceph daemons will also start automatically (they're handled by systemd). I can check tomorrow which steps exactly our shutdown procedure consists of when we have planned power outages etc.

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

HI Eugen, We have 80 osds in the cluster.

So, to stop them I will need to run the command *ceph orch stop osd.<ID>* for all 80 osds one by one. Is there any way to stop all of them in one command ?

Also, once all nodes will be back after power maintenance activity , will I need to start all daemons mon , mds , osd,etc ? or will they be up automatically once servers will be up ?

Thanks, Gagan

On Wed, Aug 6, 2025 at 4:45 PM Eugen Block <eblock@nde.ag> wrote:

...

Although SUSE discontinued their product, the procedure is still correct [0]:

1. Tell the Ceph cluster not to mark OSDs as out:

ceph osd set noout

2. Stop daemons and nodes in the following order:

Storage clients

Gateways, for example NFS Ganesha or Object Gateway

Metadata Server

Ceph OSD

Ceph Manager

Ceph Monitor

3. If required, perform maintenance tasks.

4. Start the nodes and servers in the reverse order of the shutdown process:

Ceph Monitor

Ceph Manager

Ceph OSD

Metadata Server

Gateways, for example NFS Ganesha or Object Gateway

Storage clients

5. Remove the noout flag:

ceph osd unset noout

[0]

https://documentation.suse.com/en-us/ses/7.1/html/ses-all/storage-salt-clust...

...

...

...

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

Hi Guys, I have recently set-up a production ceph cluster

which

...

...

consists of 3 monitor nodes and 7 Osd nodes.

There is power maintenance activity scheduled at the data centre coming weekend and due to that I need to power off all the devices.

Can you please advise me on the safest way to power off all servers?

Should I power off all 7 OSD servers one by one followed by all 3 monitor nodes or vice versa?

Thanks, Gagan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, I'm glad it worked for you as well. :-) Thanks, Eugen Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

Hi Eugen, Thanks for your help! Everything went well and the cluster came back without any problems.

Thanks, Gagan

On Thu, Aug 7, 2025 at 8:03 PM gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:

...

Hi Eugen, Thanks for clarifying! I will just set the noout flag and power off all servers. Hopefully, it will work fine.

Thanks, Gagan

On Thu, Aug 7, 2025 at 5:59 PM Eugen Block <eblock@nde.ag> wrote:

...

No, that's not what I meant. Of course you need the noout flag, that's the first point on the list I sent. Other flags are not required as pointed out by croit. I have never disabled cephfs before shutting it down as mentioned by Gerard, so I can't comment on that.

The procedure I quoted from SUSE is still working fine, that's how we power off our (cephadm) based clusters as well.

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

Hi Eugen, I had deployed the cluster via cephadm. so, it's container based. Just fyi..

So, you meant I don't need to run any commands like "ceph osd set noout" , "ceph orch stop osd.<ID>" ,etc.

I just need to power off all nodes starting clients nodes , osd nodes , mon , mds nodes and power them on after maintenance activity is over.

Thanks, Gagan

On Wed, Aug 6, 2025 at 11:53 PM Eugen Block <eblock@nde.ag> wrote:

...

You don't need to stop all the OSDs (or other daemons) manually, just shut down the servers (most likely you have services colocated). When they boot again, the Ceph daemons will also start automatically (they're handled by systemd). I can check tomorrow which steps exactly our shutdown procedure consists of when we have planned power outages etc.

Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>:

...

HI Eugen, We have 80 osds in the cluster.

So, to stop them I will need to run the command *ceph orch stop osd.<ID>* for all 80 osds one by one. Is there any way to stop all of them in one command ?

Also, once all nodes will be back after power maintenance activity , will I need to start all daemons mon , mds , osd,etc ? or will they be up automatically once servers will be up ?

Thanks, Gagan

On Wed, Aug 6, 2025 at 4:45 PM Eugen Block <eblock@nde.ag> wrote:

> Although SUSE discontinued their product, the procedure is still correct > [0]: > > 1. Tell the Ceph cluster not to mark OSDs as out: > > ceph osd set noout > > 2. Stop daemons and nodes in the following order: > > Storage clients > > Gateways, for example NFS Ganesha or Object Gateway > > Metadata Server > > Ceph OSD > > Ceph Manager > > Ceph Monitor > > 3. If required, perform maintenance tasks. > > 4. Start the nodes and servers in the reverse order of the shutdown > process: > > Ceph Monitor > > Ceph Manager > > Ceph OSD > > Metadata Server > > Gateways, for example NFS Ganesha or Object Gateway > > Storage clients > > 5. Remove the noout flag: > > ceph osd unset noout > > [0] > >

https://documentation.suse.com/en-us/ses/7.1/html/ses-all/storage-salt-clust...

...

...

> > Zitat von gagan tiwari <gagan.tiwari@mathisys-india.com>: > > > Hi Guys, > > I have recently set-up a production ceph cluster which > > consists of 3 monitor nodes and 7 Osd nodes. > > > > There is power maintenance activity scheduled at the data centre coming > > weekend and due to that I need to power off all the devices. > > > > Can you please advise me on the safest way to power off all servers? > > > > Should I power off all 7 OSD servers one by one followed by all 3 monitor > > nodes or vice versa? > > > > Thanks, > > Gagan > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >

Thanks Gerard / Lukasz / Eugen, I will follow these instructions white shutting nodes. Thanks, Gagan On Wed, Aug 6, 2025 at 7:19 PM Hand, Gerard <g.hand@lancaster.ac.uk> wrote:

If you are using CephFS you will need to disable that first:

ceph fs set <file system> max_mds 1 ceph fs fail <file system> ceph fs set <file system> joinable false

Regards Gerard _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Isn't noout enough to shutdown OSD ? Heard that some of those flags can cause some problems when you restart the cluster https://www.croit.io/blog/how-not-to-shut-down-a-ceph-cluster ________________________________ De : Joshua Blanch <joshua.blanch@clyso.com> Envoyé : mercredi 6 août 2025 15:47:21 À : gagan tiwari Cc : Hand, Gerard; lukasz@borek.org.pl; ceph-users Objet : [ceph-users] Re: [External] Best/Safest way to power off cluster for osds I would run it in this exact order to shut it down. ceph osd set noout ceph osd set norecover ceph osd set norebalance ceph osd set nobackfill ceph osd set nodown ceph osd set noscrub ceph osd set nodeep-scrub ceph osd set pause then to boot back up, run it in reverse order. On Wed, 6 Aug 2025 at 07:36, gagan tiwari <gagan.tiwari@mathisys-india.com> wrote:

Thanks Gerard / Lukasz / Eugen, I will follow these instructions white shutting nodes.

Thanks, Gagan

On Wed, Aug 6, 2025 at 7:19 PM Hand, Gerard <g.hand@lancaster.ac.uk> wrote:

...

If you are using CephFS you will need to disable that first:

ceph fs set <file system> max_mds 1 ceph fs fail <file system> ceph fs set <file system> joinable false

Regards Gerard _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io