Seeking feedback on Improving cephadm bootstrap process
Dear ceph community, As you are aware, cephadm has become the default tool for installing Ceph on bare-metal systems. Currently, during the bootstrap process of a new cluster, if the user interrupts the process manually or if there are any issues causing the bootstrap process to fail, cephadm leaves behind the failed cluster files and processes on the current host. While this can be beneficial for debugging and resolving issues related to the cephadm bootstrap process, it can create difficulties for inexperienced users who need to delete the faulty cluster and proceed with the Ceph installation. The problem described in the tracker https://tracker.ceph.com/issues/57016 is a good example of this issue.In the cephadm development team, we are considering ways to enhance the user experience during the bootstrap of a new cluster. We have discussed the following options:1) Retain the cluster files without deleting them, but provide the user with a clear command to remove the broken/faulty cluster. 2) Automatically delete the broken/failed ceph installation and offer an option for the user to disable this behavior if desired.Both options have their advantages and disadvantages, which is why we are seeking your feedback. We would like to know which option you prefer and the reasoning behind your choice. Please provide reasonable arguments to justify your preference.Your feedback will be taken into careful consideration when we work on improving the ceph bootstrap process.Thank you, Redouane, On behalf of cephadm dev team.
Hello Redouane, much appreciated kick-off for improving cephadm. I was wondering why cephadm does not use a similar approach to rook in the sense of "repeat until it is fixed?" For the background, rook uses a controller that checks the state of the cluster, the state of monitors, whether there are disks to be added, etc. It periodically restarts the checks and when needed shifts monitors, creates OSDs, etc. My question is, why not have a daemon or checker subcommand of cephadm that a) checks what the current cluster status is (i.e. cephadm verify-cluster) and b) fixes the situation (i.e. cephadm verify-and-fix-cluster)? I think that option would be much more beneficial than the other two suggested ones. Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch
Hi, I would like to second Nico's comment. What happened to the idea that a deployment tool should be idempotent? The most natural option would be: 1) start install -> something fails 2) fix problem 3) repeat exact same deploy command -> deployment picks up at current state (including cleaning up failed state markers) and tries to continue until next issue (go to 2) I'm not sure (meaning: its a terrible idea) if its a good idea to provide a single command to wipe a cluster. Just for the fat finger syndrome. This seems safe only if it would be possible to mark a cluster as production somehow (must be sticky, that is, cannot be unset), which prevents a cluster destroy command (or any too dangerous command) from executing. I understand the test case in the tracker, but having such test-case utils that can run on a production cluster and destroy everything seems a bit dangerous. I think destroying a cluster should be a manual and tedious process and figuring out how to do it should be part of the learning experience. So my answer to "how do I start over" would be "go figure it out, its an important lesson". Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Friday, May 26, 2023 10:40 PM To: Redouane Kachach Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process Hello Redouane, much appreciated kick-off for improving cephadm. I was wondering why cephadm does not use a similar approach to rook in the sense of "repeat until it is fixed?" For the background, rook uses a controller that checks the state of the cluster, the state of monitors, whether there are disks to be added, etc. It periodically restarts the checks and when needed shifts monitors, creates OSDs, etc. My question is, why not have a daemon or checker subcommand of cephadm that a) checks what the current cluster status is (i.e. cephadm verify-cluster) and b) fixes the situation (i.e. cephadm verify-and-fix-cluster)? I think that option would be much more beneficial than the other two suggested ones. Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Frank, in regards to destroying a cluster, I'd suggest to reuse the old --yes-i-really-mean-it parameter, as it is already in use by ceph osd destroy [0]. Then it doesn't matter whether it's prod or not, if you really mean it ... ;-) Best regards, Nico [0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/ Frank Schilder <frans@dtu.dk> writes:
Hi, I would like to second Nico's comment. What happened to the idea that a deployment tool should be idempotent? The most natural option would be:
1) start install -> something fails 2) fix problem 3) repeat exact same deploy command -> deployment picks up at current state (including cleaning up failed state markers) and tries to continue until next issue (go to 2)
I'm not sure (meaning: its a terrible idea) if its a good idea to provide a single command to wipe a cluster. Just for the fat finger syndrome. This seems safe only if it would be possible to mark a cluster as production somehow (must be sticky, that is, cannot be unset), which prevents a cluster destroy command (or any too dangerous command) from executing. I understand the test case in the tracker, but having such test-case utils that can run on a production cluster and destroy everything seems a bit dangerous.
I think destroying a cluster should be a manual and tedious process and figuring out how to do it should be part of the learning experience. So my answer to "how do I start over" would be "go figure it out, its an important lesson".
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Friday, May 26, 2023 10:40 PM To: Redouane Kachach Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process
Hello Redouane,
much appreciated kick-off for improving cephadm. I was wondering why cephadm does not use a similar approach to rook in the sense of "repeat until it is fixed?"
For the background, rook uses a controller that checks the state of the cluster, the state of monitors, whether there are disks to be added, etc. It periodically restarts the checks and when needed shifts monitors, creates OSDs, etc.
My question is, why not have a daemon or checker subcommand of cephadm that a) checks what the current cluster status is (i.e. cephadm verify-cluster) and b) fixes the situation (i.e. cephadm verify-and-fix-cluster)?
I think that option would be much more beneficial than the other two suggested ones.
Best regards,
Nico
-- Sustainable and modern Infrastructures by ungleich.ch
What I'm having in mind is if the command is already in history. A wrong history reference can execute a command with "--yes-i-really-mean-it" even though you really don't mean it. Been there. For an OSD this is maybe tolerable, but for an entire cluster ... not really. Some things need to be hard to limit the blast radius of a typo (or attacker). For example, when issuing such a command the first time, the cluster could print a nonce that needs to be included in such a command to make it happen and which is only valid once for this exact command, so one actually needs to type something new every time to destroy stuff. An exception could be if a "safe-to-destroy" query for any daemon (pool etc.) returns true. I would still not allow an entire cluster to be wiped with a single command. In a single step, only allow to destroy what could be recovered in some way (there has to be some form of undo). And there should be notifications to all admins about what is going on to be able to catch malicious execution of destructive commands. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Tuesday, May 30, 2023 10:51 AM To: Frank Schilder Cc: Nico Schottelius; Redouane Kachach; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process Hey Frank, in regards to destroying a cluster, I'd suggest to reuse the old --yes-i-really-mean-it parameter, as it is already in use by ceph osd destroy [0]. Then it doesn't matter whether it's prod or not, if you really mean it ... ;-) Best regards, Nico [0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/ Frank Schilder <frans@dtu.dk> writes:
Hi, I would like to second Nico's comment. What happened to the idea that a deployment tool should be idempotent? The most natural option would be:
1) start install -> something fails 2) fix problem 3) repeat exact same deploy command -> deployment picks up at current state (including cleaning up failed state markers) and tries to continue until next issue (go to 2)
I'm not sure (meaning: its a terrible idea) if its a good idea to provide a single command to wipe a cluster. Just for the fat finger syndrome. This seems safe only if it would be possible to mark a cluster as production somehow (must be sticky, that is, cannot be unset), which prevents a cluster destroy command (or any too dangerous command) from executing. I understand the test case in the tracker, but having such test-case utils that can run on a production cluster and destroy everything seems a bit dangerous.
I think destroying a cluster should be a manual and tedious process and figuring out how to do it should be part of the learning experience. So my answer to "how do I start over" would be "go figure it out, its an important lesson".
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Friday, May 26, 2023 10:40 PM To: Redouane Kachach Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process
Hello Redouane,
much appreciated kick-off for improving cephadm. I was wondering why cephadm does not use a similar approach to rook in the sense of "repeat until it is fixed?"
For the background, rook uses a controller that checks the state of the cluster, the state of monitors, whether there are disks to be added, etc. It periodically restarts the checks and when needed shifts monitors, creates OSDs, etc.
My question is, why not have a daemon or checker subcommand of cephadm that a) checks what the current cluster status is (i.e. cephadm verify-cluster) and b) fixes the situation (i.e. cephadm verify-and-fix-cluster)?
I think that option would be much more beneficial than the other two suggested ones.
Best regards,
Nico
-- Sustainable and modern Infrastructures by ungleich.ch
+1 Michel Le 30/05/2023 à 11:23, Frank Schilder a écrit :
What I'm having in mind is if the command is already in history. A wrong history reference can execute a command with "--yes-i-really-mean-it" even though you really don't mean it. Been there. For an OSD this is maybe tolerable, but for an entire cluster ... not really. Some things need to be hard to limit the blast radius of a typo (or attacker).
For example, when issuing such a command the first time, the cluster could print a nonce that needs to be included in such a command to make it happen and which is only valid once for this exact command, so one actually needs to type something new every time to destroy stuff. An exception could be if a "safe-to-destroy" query for any daemon (pool etc.) returns true.
I would still not allow an entire cluster to be wiped with a single command. In a single step, only allow to destroy what could be recovered in some way (there has to be some form of undo). And there should be notifications to all admins about what is going on to be able to catch malicious execution of destructive commands.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Tuesday, May 30, 2023 10:51 AM To: Frank Schilder Cc: Nico Schottelius; Redouane Kachach; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process
Hey Frank,
in regards to destroying a cluster, I'd suggest to reuse the old --yes-i-really-mean-it parameter, as it is already in use by ceph osd destroy [0]. Then it doesn't matter whether it's prod or not, if you really mean it ... ;-)
Best regards,
Nico
[0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/
Frank Schilder <frans@dtu.dk> writes:
Hi, I would like to second Nico's comment. What happened to the idea that a deployment tool should be idempotent? The most natural option would be:
1) start install -> something fails 2) fix problem 3) repeat exact same deploy command -> deployment picks up at current state (including cleaning up failed state markers) and tries to continue until next issue (go to 2)
I'm not sure (meaning: its a terrible idea) if its a good idea to provide a single command to wipe a cluster. Just for the fat finger syndrome. This seems safe only if it would be possible to mark a cluster as production somehow (must be sticky, that is, cannot be unset), which prevents a cluster destroy command (or any too dangerous command) from executing. I understand the test case in the tracker, but having such test-case utils that can run on a production cluster and destroy everything seems a bit dangerous.
I think destroying a cluster should be a manual and tedious process and figuring out how to do it should be part of the learning experience. So my answer to "how do I start over" would be "go figure it out, its an important lesson".
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Friday, May 26, 2023 10:40 PM To: Redouane Kachach Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process
Hello Redouane,
much appreciated kick-off for improving cephadm. I was wondering why cephadm does not use a similar approach to rook in the sense of "repeat until it is fixed?"
For the background, rook uses a controller that checks the state of the cluster, the state of monitors, whether there are disks to be added, etc. It periodically restarts the checks and when needed shifts monitors, creates OSDs, etc.
My question is, why not have a daemon or checker subcommand of cephadm that a) checks what the current cluster status is (i.e. cephadm verify-cluster) and b) fixes the situation (i.e. cephadm verify-and-fix-cluster)?
I think that option would be much more beneficial than the other two suggested ones.
Best regards,
Nico
-- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm a new ceph user and I have some trouble with boostraping with cephadm: using Pacific or Quincy no hard drive are detected by Ceph. Using Octopus all the hard drives are detected. As I do not know how to really clean, even a successful install but not functional, each test requires me a full reinstall of the node (it is a test node, no problem except needed time). A detailed (and working) cleaning (or uninstalling) methode (or command) of a ceph deployment for a Ceph newbie will be very helpfull. About how to do this, I'm using proxmox for vitualization and removing a VM via the web interface requires typing again the ID of the VM. May be Ceph could require the user providing the cluster ID when running the command ? In the command arguments if building a new cluster create always a different id or when command is running as a double check. Best regards, Patrick Le 30/05/2023 à 11:23, Frank Schilder a écrit :
What I'm having in mind is if the command is already in history. A wrong history reference can execute a command with "--yes-i-really-mean-it" even though you really don't mean it. Been there. For an OSD this is maybe tolerable, but for an entire cluster ... not really. Some things need to be hard to limit the blast radius of a typo (or attacker).
For example, when issuing such a command the first time, the cluster could print a nonce that needs to be included in such a command to make it happen and which is only valid once for this exact command, so one actually needs to type something new every time to destroy stuff. An exception could be if a "safe-to-destroy" query for any daemon (pool etc.) returns true.
I would still not allow an entire cluster to be wiped with a single command. In a single step, only allow to destroy what could be recovered in some way (there has to be some form of undo). And there should be notifications to all admins about what is going on to be able to catch malicious execution of destructive commands.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Tuesday, May 30, 2023 10:51 AM To: Frank Schilder Cc: Nico Schottelius; Redouane Kachach; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process
Hey Frank,
in regards to destroying a cluster, I'd suggest to reuse the old --yes-i-really-mean-it parameter, as it is already in use by ceph osd destroy [0]. Then it doesn't matter whether it's prod or not, if you really mean it ... ;-)
Best regards,
Nico
[0] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/
Frank Schilder <frans@dtu.dk> writes:
Hi, I would like to second Nico's comment. What happened to the idea that a deployment tool should be idempotent? The most natural option would be:
1) start install -> something fails 2) fix problem 3) repeat exact same deploy command -> deployment picks up at current state (including cleaning up failed state markers) and tries to continue until next issue (go to 2)
I'm not sure (meaning: its a terrible idea) if its a good idea to provide a single command to wipe a cluster. Just for the fat finger syndrome. This seems safe only if it would be possible to mark a cluster as production somehow (must be sticky, that is, cannot be unset), which prevents a cluster destroy command (or any too dangerous command) from executing. I understand the test case in the tracker, but having such test-case utils that can run on a production cluster and destroy everything seems a bit dangerous.
I think destroying a cluster should be a manual and tedious process and figuring out how to do it should be part of the learning experience. So my answer to "how do I start over" would be "go figure it out, its an important lesson".
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Friday, May 26, 2023 10:40 PM To: Redouane Kachach Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process
Hello Redouane,
much appreciated kick-off for improving cephadm. I was wondering why cephadm does not use a similar approach to rook in the sense of "repeat until it is fixed?"
For the background, rook uses a controller that checks the state of the cluster, the state of monitors, whether there are disks to be added, etc. It periodically restarts the checks and when needed shifts monitors, creates OSDs, etc.
My question is, why not have a daemon or checker subcommand of cephadm that a) checks what the current cluster status is (i.e. cephadm verify-cluster) and b) fixes the situation (i.e. cephadm verify-and-fix-cluster)?
I think that option would be much more beneficial than the other two suggested ones.
Best regards,
Nico
-- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello all, Thank you very much for your valuable feedback. I'd like to provide some context and clarify certain points as there seems to be some confusion regarding the objective of this discussion and how a cephadm initial bootstrap works. As you know, Ceph has the capability to run multiple clusters on the same nodes, with certain limitations that I won't delve into in this discussion. Each cluster has its own unique identifier (called fsid), which in fact is a UUID generated by cephadm during cluster bootstrap or provided by the user. Almost all cluster-related files, including cluster and daemon configurations, systemd units, logs, etc., are specific to each cluster and are stored in dedicated directories based on the fsid, such as /var/lib/ceph/<fsid>, /var/log/ceph/<fsid>, /run/ceph/<fsid>, and so on. These directories ensure isolation between cluster files and daemons, preventing any file or configuration sharing between clusters. Typically, as a user, you need not concern yourself with the exact location of the cluster files when deleting a cluster. For this purpose, cephadm provides a dedicated command, "cephadm rm-cluster," ( https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster) which handles the deletion of cluster files, removal of daemons, and so forth. Importantly, this command uses the fsid to ensure the command's safety in environments where multiple clusters coexist. That being clarified, I want to emphasize that this discussion does not revolve around the workings or options provided by the "cephadm rm-cluster" command. This command is the official method for deleting a cluster and is employed in both upstream and production clusters. In case you have suggestions for improving the user experience with this command we can start a separate thread for that purpose. Back to the original subject: During the process of bootstrapping a new cluster with cephadm, in addition to installing files in their respective locations, core ceph daemons such as mgr and mon are started. If the bootstrap process succeeds we end up with a minimal ceph cluster consisting only of the necessary files and daemons. In case of bootstrap failure, a minimal, broken, non-functional ceph cluster is created with no actual data (no OSDs), and potentially with some daemons (mgr/mon) running on the current node. Retaining these files and daemons provides no real benefit to the user apart from facilitating the investigation of bugs or issues that may prevent the bootstrap process. Even in such cases, once the investigation is complete and the issue is resolved, the user must delete this cluster since it is useless and may have active daemons listening on mon/mgr sockets, thereby obstructing the creation of future clusters on the same node due to occupied mon/mgr ports. The purpose of this email thread is to discuss how to address this situation. Given that we have full control over the bootstrap process, we can automatically clean up this broken cluster (or at least assist the user in doing so). The proposed rollback options are: either an automatic cleanup (option 2) or a manual cleanup (option 1) as mentioned in the original email. The goal of this thread is to get some feedback about your preference as a user and gather input on the additional information you would like to receive regarding each option. Side Note: As a response to the question why we don't use some mechanism like Rook does. The answer is cephadm is a "binary" meant for bare-metal deployments. Unlike Rook, which operates within the framework of a higher-level orchestration system like k8s or Openshift, in the case of cephadm we have no daemon nor any other high level controller that can watch and fix a broken installation. cephadm is the only binary needed (+ some minimal dependencies) to bootstrap a new ceph cluster. Best Regards, Redouane. On Tue, May 30, 2023 at 10:30 AM Frank Schilder <frans@dtu.dk> wrote:
Hi, I would like to second Nico's comment. What happened to the idea that a deployment tool should be idempotent? The most natural option would be:
1) start install -> something fails 2) fix problem 3) repeat exact same deploy command -> deployment picks up at current state (including cleaning up failed state markers) and tries to continue until next issue (go to 2)
I'm not sure (meaning: its a terrible idea) if its a good idea to provide a single command to wipe a cluster. Just for the fat finger syndrome. This seems safe only if it would be possible to mark a cluster as production somehow (must be sticky, that is, cannot be unset), which prevents a cluster destroy command (or any too dangerous command) from executing. I understand the test case in the tracker, but having such test-case utils that can run on a production cluster and destroy everything seems a bit dangerous.
I think destroying a cluster should be a manual and tedious process and figuring out how to do it should be part of the learning experience. So my answer to "how do I start over" would be "go figure it out, its an important lesson".
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: Friday, May 26, 2023 10:40 PM To: Redouane Kachach Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process
Hello Redouane,
much appreciated kick-off for improving cephadm. I was wondering why cephadm does not use a similar approach to rook in the sense of "repeat until it is fixed?"
For the background, rook uses a controller that checks the state of the cluster, the state of monitors, whether there are disks to be added, etc. It periodically restarts the checks and when needed shifts monitors, creates OSDs, etc.
My question is, why not have a daemon or checker subcommand of cephadm that a) checks what the current cluster status is (i.e. cephadm verify-cluster) and b) fixes the situation (i.e. cephadm verify-and-fix-cluster)?
I think that option would be much more beneficial than the other two suggested ones.
Best regards,
Nico
-- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just a user opinion, maybe add the following additions to the options? For option 1: * Clear instructions how to remove all traces to the failed installation (if you can automate it, you can write a manual) or provide instructions to start a cleanup script. * Don't allow another deployment of Cephadm if there's a failed deployment, only if everything is cleaned up. For option 2: * If an installation failed and gotten completely removed, don't allow another run unless the user sets an override (or removes the thing which triggers the check for failed installations). This to prevent a user in an endless loop to try and deploy Cephadm. Inform the user about the last failed deployment, show the available options for a retry and the option to keep the deployment files to troubleshoot the issue. * If the deployment failed (or got interrupted) and the user wanted to keep a failed deployment, provide just like Option 1 clear instructions how to clean up the failed deployment. With the above additions, I would prefer Option 1. Because there's almost always a reason a deployment fails and I would like to investigate directly why it happened. Best regards, Sake
participants (6)
-
Frank Schilder
-
Michel Jouvin
-
Nico Schottelius
-
Patrick Begou
-
Redouane Kachach
-
Sake Paulusma