How to speed up OSD deployment process
Hi, I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices). I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete. However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next. Is there a way to speed up the OSD deployment process? Thanks in advance for your help! Best regards, Yufan Chen
I've worked with systems much smaller than that where I would have LOVED to get everything up in only an hour. Kids these days. 1. Have you tried using a spec file? Might help, might not. 2. You could always do the old "&" Unix shell operator for asynchronous commands. I think you could get Ansible to do that also, although by default Ansible runs sequentially for a given host and parallel for multiple hosts. Note that spawning multiple tasks that fight for the same resource may offer little to no speed improvement. Regards, Tim On 11/8/24 10:05, YuFan Chen wrote:
Hi,
I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices).
I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete.
However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next.
Is there a way to speed up the OSD deployment process? Thanks in advance for your help!
Best regards, Yufan Chen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tim, Yes, I use Ansible and osd spec yaml to deploy ceph custer. And I use lvm to manage these HDDs and nvme SSDs the osd spec is like that: --- service_type: osd service_id: osd.hybrid placement: label: 'osd' data_devices: paths: - /dev/hybrid/bdev01 ... - /dev/hybrid/bdev32 db_devices: paths: # 'ceph-volume' maps data_devices and db_devices in reverse order - /dev/hybrid/db32 .. - /dev/hybrid/db01 --- When Ansible task applies the osd spec: ceph orch apply -i osd.yml cephadm does not perform the deployment in a more parallel way: step 1: create 192 OSDs sequentially. step 2: on each node, it starts a single OSD and waits for it to become ready before moving on to the next. Regards, Yufan Tim Holloway <timh@mousetech.com> 於 2024年11月9日 週六 上午1:58寫道：
I've worked with systems much smaller than that where I would have LOVED to get everything up in only an hour. Kids these days.
1. Have you tried using a spec file? Might help, might not.
2. You could always do the old "&" Unix shell operator for asynchronous commands. I think you could get Ansible to do that also, although by default Ansible runs sequentially for a given host and parallel for multiple hosts.
Note that spawning multiple tasks that fight for the same resource may offer little to no speed improvement.
Regards,
Tim
On 11/8/24 10:05, YuFan Chen wrote:
Hi,
I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices).
I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete.
However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next.
Is there a way to speed up the OSD deployment process? Thanks in advance for your help!
Best regards, Yufan Chen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We're not optimistic. It's possible that Ceph will run this stuff in serial no matter how you submit it. But here's some useful stuff on getting Ansible to run things in parallel: https://toptechtips.github.io/2023-06-26-ansible-parallel/ https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_strategies.... On Sat, 2024-11-09 at 07:09 +0800, YuFan Chen wrote:
Hi Tim,
Yes, I use Ansible and osd spec yaml to deploy ceph custer.
And I use lvm to manage these HDDs and nvme SSDs the osd spec is like that: --- service_type: osd service_id: osd.hybrid placement: label: 'osd' data_devices: paths: - /dev/hybrid/bdev01 ... - /dev/hybrid/bdev32 db_devices: paths: # 'ceph-volume' maps data_devices and db_devices in reverse order - /dev/hybrid/db32 .. - /dev/hybrid/db01 ---
When Ansible task applies the osd spec: ceph orch apply -i osd.yml
cephadm does not perform the deployment in a more parallel way: step 1: create 192 OSDs sequentially. step 2: on each node, it starts a single OSD and waits for it to become ready before moving on to the next.
Regards, Yufan
Tim Holloway <timh@mousetech.com> 於 2024年11月9日 週六 上午1:58寫道：
I've worked with systems much smaller than that where I would have LOVED to get everything up in only an hour. Kids these days.
1. Have you tried using a spec file? Might help, might not.
2. You could always do the old "&" Unix shell operator for asynchronous commands. I think you could get Ansible to do that also, although by default Ansible runs sequentially for a given host and parallel for multiple hosts.
Note that spawning multiple tasks that fight for the same resource may offer little to no speed improvement.
Regards,
Tim
On 11/8/24 10:05, YuFan Chen wrote:
Hi,
I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices).
I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete.
However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next.
Is there a way to speed up the OSD deployment process? Thanks in advance for your help!
Best regards, Yufan Chen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I don't see how it would be currently possible. The OSD creation is handled by ceph-volume, which activates each OSD separately: [2024-11-22 14:03:08,415][ceph_volume.main][INFO ] Running command: ceph-volume activate --osd-id 0 --osd-uuid aacabeca-9adb-465c-88ee-935f06fa45f7 --no-systemd --no-tmpfs [2024-11-22 14:03:09,343][ceph_volume.devices.raw.activate][INFO ] Activating osd.0 uuid aacabeca-9adb-465c-88ee-935f06fa45f7 cluster e57f7b6a-a8d9-11ef-af3c-fa163e2ad8c5 The ceph-volume lvm activate description [0] states:
It is possible to activate all existing OSDs at once by using the --all flag. For example:
ceph-volume lvm activate --all
This call will inspect all the OSDs created by ceph-volume that are inactive and will activate them one by one.
I assume that even if the OSD creation process could be tweaked in a way that all OSDs are created first without separate activation, and then cephadm would issue "ceph-volume lvm activate --all", the OSDs would still be activated one by one. But as Tim already stated, an hour for almost 200 OSDs is not that bad. ;-) I guess you could create a tracker issue for an enhancement, maybe some of the devs can clarify why the OSDs need to be activated one by one. Regards, Eugen [0] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/ Zitat von YuFan Chen <wiz.chen@gmail.com>:
Hi,
I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices).
I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete.
However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next.
Is there a way to speed up the OSD deployment process? Thanks in advance for your help!
Best regards, Yufan Chen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I can remember that it took with the tools before cephadm somehow 8 hours to deploy a ceph cluster with more than 2000 osds. But I also know that CBT has a much faster approach to installing a Ceph cluster. Just an idea: maybe you can look at the approach at CBT to make cephadm faster. Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Fr., 22. Nov. 2024 um 15:46 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi,
I don't see how it would be currently possible. The OSD creation is handled by ceph-volume, which activates each OSD separately:
[2024-11-22 14:03:08,415][ceph_volume.main][INFO ] Running command: ceph-volume activate --osd-id 0 --osd-uuid aacabeca-9adb-465c-88ee-935f06fa45f7 --no-systemd --no-tmpfs
[2024-11-22 14:03:09,343][ceph_volume.devices.raw.activate][INFO ] Activating osd.0 uuid aacabeca-9adb-465c-88ee-935f06fa45f7 cluster e57f7b6a-a8d9-11ef-af3c-fa163e2ad8c5
The ceph-volume lvm activate description [0] states:
It is possible to activate all existing OSDs at once by using the --all flag. For example:
ceph-volume lvm activate --all
This call will inspect all the OSDs created by ceph-volume that are inactive and will activate them one by one.
I assume that even if the OSD creation process could be tweaked in a way that all OSDs are created first without separate activation, and then cephadm would issue "ceph-volume lvm activate --all", the OSDs would still be activated one by one.
But as Tim already stated, an hour for almost 200 OSDs is not that bad. ;-) I guess you could create a tracker issue for an enhancement, maybe some of the devs can clarify why the OSDs need to be activated one by one.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/
Zitat von YuFan Chen <wiz.chen@gmail.com>:
Hi,
I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices).
I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete.
However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next.
Is there a way to speed up the OSD deployment process? Thanks in advance for your help!
Best regards, Yufan Chen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Thank you. The Ceph cluster is running smoothly so far. However, during our testing, we re-installed it multiple times and observed that the ceph-volume command took over a minute to activate the OSD. In the activation stage, ceph-volume called "ceph-bluestore-tool show-label". It appears that the command scans all disks to identify which disk is being activated. Best regards, Yufan Chen Joachim Kraftmayer <joachim.kraftmayer@clyso.com> 於 2024年11月23日 週六 00:50 寫道：
Hi, I can remember that it took with the tools before cephadm somehow 8 hours to deploy a ceph cluster with more than 2000 osds.
But I also know that CBT has a much faster approach to installing a Ceph cluster. Just an idea: maybe you can look at the approach at CBT to make cephadm faster.
Regards, Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich <https://www.google.com/maps/search/Hohenzollernstr.+27,+80801+Munich?entry=gmail&source=g>
Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Fr., 22. Nov. 2024 um 15:46 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi,
I don't see how it would be currently possible. The OSD creation is handled by ceph-volume, which activates each OSD separately:
[2024-11-22 14:03:08,415][ceph_volume.main][INFO ] Running command: ceph-volume activate --osd-id 0 --osd-uuid aacabeca-9adb-465c-88ee-935f06fa45f7 --no-systemd --no-tmpfs
[2024-11-22 14:03:09,343][ceph_volume.devices.raw.activate][INFO ] Activating osd.0 uuid aacabeca-9adb-465c-88ee-935f06fa45f7 cluster e57f7b6a-a8d9-11ef-af3c-fa163e2ad8c5
The ceph-volume lvm activate description [0] states:
It is possible to activate all existing OSDs at once by using the --all flag. For example:
ceph-volume lvm activate --all
This call will inspect all the OSDs created by ceph-volume that are inactive and will activate them one by one.
I assume that even if the OSD creation process could be tweaked in a way that all OSDs are created first without separate activation, and then cephadm would issue "ceph-volume lvm activate --all", the OSDs would still be activated one by one.
But as Tim already stated, an hour for almost 200 OSDs is not that bad. ;-) I guess you could create a tracker issue for an enhancement, maybe some of the devs can clarify why the OSDs need to be activated one by one.
Regards, Eugen
[0] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/
Zitat von YuFan Chen <wiz.chen@gmail.com>:
Hi,
I’m setting up a 6-node Ceph cluster using Ceph Squid. Each node is configured with 32 OSDs (32 HDDs and 8 NVMe SSDs for db_devices).
I’ve created an OSD service specification and am using cephadm to apply the configuration. The deployment of all 192 OSDs takes about an hour to complete.
However, I’ve noticed that cephadm creates the OSDs sequentially. Then, on each node, it starts a single OSD and waits for it to become ready before moving on to the next.
Is there a way to speed up the OSD deployment process? Thanks in advance for your help!
Best regards, Yufan Chen _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Joachim Kraftmayer
-
Tim Holloway
-
YuFan Chen