How to know is ceph is ready?
I have a proxmox cluster using an external CEPH cluster. Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot. I want to add a dependency in proxmox to wait for ceph to be ready. I can work with a HEALTH_WARN as long the RBD pool is usable. ceph status exit status doesn´t helps Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright? -- Alfrenovsky
My prefered way: check if all pgs are active+clean Joachim Joachim Kraftmayer CEO joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Alfredo Rezinovsky <alfrenovsky@gmail.com> schrieb am Do., 29. Aug. 2024, 12:53:
I have a proxmox cluster using an external CEPH cluster.
Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot.
I want to add a dependency in proxmox to wait for ceph to be ready.
I can work with a HEALTH_WARN as long the RBD pool is usable.
ceph status exit status doesn´t helps
Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright?
-- Alfrenovsky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, It's a hacky idea but create a script that checks if the Ceph RBD pool is fully "active+clean" to ensure it's ready before starting Proxmox VMs. Something like this... 1. Bash Script: - Write a script that checks if the Ceph RBD pool is in the "active+clean" state using ceph pg stat or ceph -s. - The script should run until the pool is ready before exiting. A loop or something... 2. Systemd Service for Ceph Readiness: - Create a systemd service unit file (something like: ceph-ready.service) that runs the script at startup. - Ensure the service waits until the Ceph cluster is ready before proceeding. 3. Systemd Service for Proxmox VMs: - Modify an existing Proxmox service or create a new one to depend on the ceph-ready.service. - Use After=ceph-ready.service in the unit file to ensure Proxmox VMs start only after Ceph is ready. 4. Enable and Start the Services: - Enable both systemd services to start at boot with systemctl enable. - Reboot to ensure that Proxmox waits for Ceph before starting VMs. Just an idea... Thank you, Bogdan Velica croit.io On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky <alfrenovsky@gmail.com> wrote:
I have a proxmox cluster using an external CEPH cluster.
Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot.
I want to add a dependency in proxmox to wait for ceph to be ready.
I can work with a HEALTH_WARN as long the RBD pool is usable.
ceph status exit status doesn´t helps
Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright?
-- Alfrenovsky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Short note, if you do it like this it will fail. You have to use a regex like active+clean*. Because you always habe active+clean+scrub or deep-scrub Joachim Joachim Kraftmayer CEO joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Bogdan Adrian Velica <vbogdan@gmail.com> schrieb am Do., 29. Aug. 2024, 13:04:
Hi,
It's a hacky idea but create a script that checks if the Ceph RBD pool is fully "active+clean" to ensure it's ready before starting Proxmox VMs. Something like this...
1. Bash Script: - Write a script that checks if the Ceph RBD pool is in the "active+clean" state using ceph pg stat or ceph -s. - The script should run until the pool is ready before exiting. A loop or something...
2. Systemd Service for Ceph Readiness: - Create a systemd service unit file (something like: ceph-ready.service) that runs the script at startup. - Ensure the service waits until the Ceph cluster is ready before proceeding.
3. Systemd Service for Proxmox VMs: - Modify an existing Proxmox service or create a new one to depend on the ceph-ready.service. - Use After=ceph-ready.service in the unit file to ensure Proxmox VMs start only after Ceph is ready.
4. Enable and Start the Services: - Enable both systemd services to start at boot with systemctl enable. - Reboot to ensure that Proxmox waits for Ceph before starting VMs.
Just an idea...
Thank you, Bogdan Velica croit.io
On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky <alfrenovsky@gmail.com> wrote:
I have a proxmox cluster using an external CEPH cluster.
Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot.
I want to add a dependency in proxmox to wait for ceph to be ready.
I can work with a HEALTH_WARN as long the RBD pool is usable.
ceph status exit status doesn´t helps
Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright?
-- Alfrenovsky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Short note, if you do it like this it will fail. You have to use a regex like active+clean*. Because you always habe active+clean+scrub or deep-scrub
This. A few releases back backfill was changed to no longer itself trigger HEALTH_WARN, so code that waited for HEALTH_OK needed to be updated. Mind you sometimes you care about backfill and sometimes you don't. In this case the PG states should suffice, no reason to make Proxmox wait for backfill.
Joachim
Joachim Kraftmayer
CEO
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Bogdan Adrian Velica <vbogdan@gmail.com> schrieb am Do., 29. Aug. 2024, 13:04:
Hi,
It's a hacky idea but create a script that checks if the Ceph RBD pool is fully "active+clean" to ensure it's ready before starting Proxmox VMs. Something like this...
1. Bash Script: - Write a script that checks if the Ceph RBD pool is in the "active+clean" state using ceph pg stat or ceph -s. - The script should run until the pool is ready before exiting. A loop or something...
2. Systemd Service for Ceph Readiness: - Create a systemd service unit file (something like: ceph-ready.service) that runs the script at startup. - Ensure the service waits until the Ceph cluster is ready before proceeding.
3. Systemd Service for Proxmox VMs: - Modify an existing Proxmox service or create a new one to depend on the ceph-ready.service. - Use After=ceph-ready.service in the unit file to ensure Proxmox VMs start only after Ceph is ready.
4. Enable and Start the Services: - Enable both systemd services to start at boot with systemctl enable. - Reboot to ensure that Proxmox waits for Ceph before starting VMs.
Just an idea...
Thank you, Bogdan Velica croit.io
On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky <alfrenovsky@gmail.com> wrote:
I have a proxmox cluster using an external CEPH cluster.
Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot.
I want to add a dependency in proxmox to wait for ceph to be ready.
I can work with a HEALTH_WARN as long the RBD pool is usable.
ceph status exit status doesn´t helps
Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright?
-- Alfrenovsky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
steps 2 to 4 are exactly my idea. In step 1 All I will check is all active (o non inactive or unknown). Clean is not necessary, I can boot VMs with degraded.as long they are active. El jue, 29 ago 2024 a las 8:03, Bogdan Adrian Velica (<vbogdan@gmail.com>) escribió:
Hi,
It's a hacky idea but create a script that checks if the Ceph RBD pool is fully "active+clean" to ensure it's ready before starting Proxmox VMs. Something like this...
1. Bash Script: - Write a script that checks if the Ceph RBD pool is in the "active+clean" state using ceph pg stat or ceph -s. - The script should run until the pool is ready before exiting. A loop or something...
2. Systemd Service for Ceph Readiness: - Create a systemd service unit file (something like: ceph-ready.service) that runs the script at startup. - Ensure the service waits until the Ceph cluster is ready before proceeding.
3. Systemd Service for Proxmox VMs: - Modify an existing Proxmox service or create a new one to depend on the ceph-ready.service. - Use After=ceph-ready.service in the unit file to ensure Proxmox VMs start only after Ceph is ready.
4. Enable and Start the Services: - Enable both systemd services to start at boot with systemctl enable. - Reboot to ensure that Proxmox waits for Ceph before starting VMs.
Just an idea...
Thank you, Bogdan Velica croit.io
On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky <alfrenovsky@gmail.com> wrote:
I have a proxmox cluster using an external CEPH cluster.
Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot.
I want to add a dependency in proxmox to wait for ceph to be ready.
I can work with a HEALTH_WARN as long the RBD pool is usable.
ceph status exit status doesn´t helps
Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright?
-- Alfrenovsky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alfrenovsky
"Ceph is ready" covers a lot of territory. It's more like, "How can I delay util Ceph is available for the particular service I need? I've been taking a systemd-bsaed approach. Since I don't actually care bout Ceph in the abstract, but I'm actually looking for the Ceph or Ceph NFS shares, I create dependencies in my systemd setup such that the dependent VM or container (quadlet) requires its ceph mounts to be online before that service can start. I'm still evaluating how well that works. I'll let you know after the next hurricane hits. But so far, so good. Tim On Fri, 2024-08-30 at 11:02 -0300, Alfredo Rezinovsky wrote:
steps 2 to 4 are exactly my idea.
In step 1 All I will check is all active (o non inactive or unknown). Clean is not necessary, I can boot VMs with degraded.as long they are active.
El jue, 29 ago 2024 a las 8:03, Bogdan Adrian Velica (<vbogdan@gmail.com>) escribió:
Hi,
It's a hacky idea but create a script that checks if the Ceph RBD pool is fully "active+clean" to ensure it's ready before starting Proxmox VMs. Something like this...
1. Bash Script: - Write a script that checks if the Ceph RBD pool is in the "active+clean" state using ceph pg stat or ceph -s. - The script should run until the pool is ready before exiting. A loop or something...
2. Systemd Service for Ceph Readiness: - Create a systemd service unit file (something like: ceph- ready.service) that runs the script at startup. - Ensure the service waits until the Ceph cluster is ready before proceeding.
3. Systemd Service for Proxmox VMs: - Modify an existing Proxmox service or create a new one to depend on the ceph-ready.service. - Use After=ceph-ready.service in the unit file to ensure Proxmox VMs start only after Ceph is ready.
4. Enable and Start the Services: - Enable both systemd services to start at boot with systemctl enable. - Reboot to ensure that Proxmox waits for Ceph before starting VMs.
Just an idea...
Thank you, Bogdan Velica croit.io
On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky <alfrenovsky@gmail.com> wrote:
I have a proxmox cluster using an external CEPH cluster.
Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot.
I want to add a dependency in proxmox to wait for ceph to be ready.
I can work with a HEALTH_WARN as long the RBD pool is usable.
ceph status exit status doesn´t helps
Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright?
-- Alfrenovsky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alfrenovsky _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Alfredo Rezinovsky
-
Anthony D'Atri
-
Bogdan Adrian Velica
-
Joachim Kraftmayer
-
Tim Holloway