[v19.2.3] All OSDs are not created with a managed spec
On Hi Cephers, I'm building a new Squid cluster with cephadm on Ubuntu 24.04. After expanding my cluster in the Dashboard (adding my 7 hosts), I choose throughput_optimized proflie wich create a generic spec for hybrid HDD/SSD : service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore The cluster is for a LAB environment, on each of the 7 nodes, I have 17 HDD SAS 1.2TB drives and 1 SSD SAS Enterprise 400GB drive. On my first try, only 28 OSD where created (out of the 119), the others appeared as down, but they won't start, I didn't find systemd units created on the hosts. But, the VGs and LVs where created, there are 17 LVs on the SSD for WAL/DB of the 17 HDD (yes, small : 29GB). On my second try, it creates 72 OSDs, still it stops, and never tries to continue, or re-create the down OSDs. I didn't manage to find them, but it seems I saw some OSD creation timeout in the logs... What can I do to have my missing OSD created ? I tried restarting, redeploying the orch OSD service, but it only restart/redeploy le OSDs it has already created...
Le 2025-08-18 11:30, Gilles Mocellin a écrit :
On Hi Cephers,
I'm building a new Squid cluster with cephadm on Ubuntu 24.04. After expanding my cluster in the Dashboard (adding my 7 hosts), I choose throughput_optimized proflie wich create a generic spec for hybrid HDD/SSD :
service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore
The cluster is for a LAB environment, on each of the 7 nodes, I have 17 HDD SAS 1.2TB drives and 1 SSD SAS Enterprise 400GB drive. On my first try, only 28 OSD where created (out of the 119), the others appeared as down, but they won't start, I didn't find systemd units created on the hosts. But, the VGs and LVs where created, there are 17 LVs on the SSD for WAL/DB of the 17 HDD (yes, small : 29GB).
On my second try, it creates 72 OSDs, still it stops, and never tries to continue, or re-create the down OSDs.
I didn't manage to find them, but it seems I saw some OSD creation timeout in the logs...
What can I do to have my missing OSD created ?
Some additional information : ceph -s cluster: id: 3ebf83bf-7927-11f0-9f3a-246e96bd90a4 health: HEALTH_OK services: mon: 5 daemons, quorum fidcl-lyo1-sto-sds-lab-01,fidcl-lyo1-sto-sds-lab-02,fidcl-lyo1-sto-sds-lab-03,fidcl-lyo1-sto-sds-lab-04,fidcl-lyo1-sto-sds-lab-05 (age 3d) mgr: fidcl-lyo1-sto-sds-lab-01.ovbjpb(active, since 3d), standbys: fidcl-lyo1-sto-sds-lab-02.nqdhpl, fidcl-lyo1-sto-sds-lab-03.cizytz osd: 119 osds: 72 up (since 3d), 89 in (since 41m) data: pools: 1 pools, 1 pgs objects: 2 objects, 769 KiB usage: 1.5 TiB used, 79 TiB / 80 TiB avail pgs: 1 active+clean One of the "missing" (not fully created) OSD, is present, but not found : ceph osd find 6 { "osd": 6, "addrs": { "addrvec": [] }, "osd_fsid": "f8745284-8026-4713-8329-cd57cb6842f7", "crush_location": {} } ceph osd info 6 osd.6 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) autoout,exists,new f8745284-8026-4713-8329-cd57cb6842f7 The missing OSDs are not chown with the device list command : ceph device ls | grep osd.6 is empty...
Le 2025-08-18 11:47, Gilles Mocellin a écrit :
Le 2025-08-18 11:30, Gilles Mocellin a écrit :
On Hi Cephers,
I'm building a new Squid cluster with cephadm on Ubuntu 24.04. After expanding my cluster in the Dashboard (adding my 7 hosts), I choose throughput_optimized proflie wich create a generic spec for hybrid HDD/SSD :
service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore
The cluster is for a LAB environment, on each of the 7 nodes, I have 17 HDD SAS 1.2TB drives and 1 SSD SAS Enterprise 400GB drive. On my first try, only 28 OSD where created (out of the 119), the others appeared as down, but they won't start, I didn't find systemd units created on the hosts. But, the VGs and LVs where created, there are 17 LVs on the SSD for WAL/DB of the 17 HDD (yes, small : 29GB).
On my second try, it creates 72 OSDs, still it stops, and never tries to continue, or re-create the down OSDs.
I didn't manage to find them, but it seems I saw some OSD creation timeout in the logs...
What can I do to have my missing OSD created ?
Some additional information :
ceph -s cluster: id: 3ebf83bf-7927-11f0-9f3a-246e96bd90a4 health: HEALTH_OK
services: mon: 5 daemons, quorum fidcl-lyo1-sto-sds-lab-01,fidcl-lyo1-sto-sds-lab-02,fidcl-lyo1-sto-sds-lab-03,fidcl-lyo1-sto-sds-lab-04,fidcl-lyo1-sto-sds-lab-05 (age 3d) mgr: fidcl-lyo1-sto-sds-lab-01.ovbjpb(active, since 3d), standbys: fidcl-lyo1-sto-sds-lab-02.nqdhpl, fidcl-lyo1-sto-sds-lab-03.cizytz osd: 119 osds: 72 up (since 3d), 89 in (since 41m)
data: pools: 1 pools, 1 pgs objects: 2 objects, 769 KiB usage: 1.5 TiB used, 79 TiB / 80 TiB avail pgs: 1 active+clean
One of the "missing" (not fully created) OSD, is present, but not found :
ceph osd find 6 { "osd": 6, "addrs": { "addrvec": [] }, "osd_fsid": "f8745284-8026-4713-8329-cd57cb6842f7", "crush_location": {} }
ceph osd info 6 osd.6 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) autoout,exists,new f8745284-8026-4713-8329-cd57cb6842f7
The missing OSDs are not chown with the device list command : ceph device ls | grep osd.6 is empty...
Also, the MGR reports that on all not fully created OSD : Aug 18 10:09:41 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[4186078]: mgr get_metadata_python Requested missing service osd.99
Hi, the last message you sent is normal for an OSD that hasn't reported back its status yet. I would check the ceph-volume.log and the cephadm.log, maybe an OSD log as well if it tried to boot. If this is a test cluster, did you properly wipe all disks before trying to deploy OSDs? For me 'cephadm ceph-volume lvm zap --destroy /dev/sdx /dev/sdy /dev/sdz ...' (locally on the host) has been working great for years now. You can zap them with the orchestrator as well, but only one disk a time, so a for loop would be useful. There have been reports every now and then from users who tried to deploy many disks per host, I don't have a link available right now. And I haven't had the chance yet to deploy multiple hosts/many OSDs with 19.2.3, so there might be a regression in ceph-volume. Regards, Eugen Zitat von Gilles Mocellin <gilles.mocellin@nuagelibre.org>:
Le 2025-08-18 11:47, Gilles Mocellin a écrit :
Le 2025-08-18 11:30, Gilles Mocellin a écrit :
On Hi Cephers,
I'm building a new Squid cluster with cephadm on Ubuntu 24.04. After expanding my cluster in the Dashboard (adding my 7 hosts), I choose throughput_optimized proflie wich create a generic spec for hybrid HDD/SSD :
service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore
The cluster is for a LAB environment, on each of the 7 nodes, I have 17 HDD SAS 1.2TB drives and 1 SSD SAS Enterprise 400GB drive. On my first try, only 28 OSD where created (out of the 119), the others appeared as down, but they won't start, I didn't find systemd units created on the hosts. But, the VGs and LVs where created, there are 17 LVs on the SSD for WAL/DB of the 17 HDD (yes, small : 29GB).
On my second try, it creates 72 OSDs, still it stops, and never tries to continue, or re-create the down OSDs.
I didn't manage to find them, but it seems I saw some OSD creation timeout in the logs...
What can I do to have my missing OSD created ?
Some additional information :
ceph -s cluster: id: 3ebf83bf-7927-11f0-9f3a-246e96bd90a4 health: HEALTH_OK
services: mon: 5 daemons, quorum fidcl-lyo1-sto-sds-lab-01,fidcl-lyo1-sto-sds-lab-02,fidcl-lyo1-sto-sds-lab-03,fidcl-lyo1-sto-sds-lab-04,fidcl-lyo1-sto-sds-lab-05 (age 3d) mgr: fidcl-lyo1-sto-sds-lab-01.ovbjpb(active, since 3d), standbys: fidcl-lyo1-sto-sds-lab-02.nqdhpl, fidcl-lyo1-sto-sds-lab-03.cizytz osd: 119 osds: 72 up (since 3d), 89 in (since 41m)
data: pools: 1 pools, 1 pgs objects: 2 objects, 769 KiB usage: 1.5 TiB used, 79 TiB / 80 TiB avail pgs: 1 active+clean
One of the "missing" (not fully created) OSD, is present, but not found :
ceph osd find 6 { "osd": 6, "addrs": { "addrvec": [] }, "osd_fsid": "f8745284-8026-4713-8329-cd57cb6842f7", "crush_location": {} }
ceph osd info 6 osd.6 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) autoout,exists,new f8745284-8026-4713-8329-cd57cb6842f7
The missing OSDs are not chown with the device list command : ceph device ls | grep osd.6 is empty...
Also, the MGR reports that on all not fully created OSD : Aug 18 10:09:41 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[4186078]: mgr get_metadata_python Requested missing service osd.99 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 2025-08-18 13:43, Eugen Block a écrit :
Hi,
the last message you sent is normal for an OSD that hasn't reported back its status yet.
I would check the ceph-volume.log and the cephadm.log, maybe an OSD log as well if it tried to boot. If this is a test cluster, did you properly wipe all disks before trying to deploy OSDs? For me 'cephadm ceph-volume lvm zap --destroy /dev/sdx /dev/sdy /dev/sdz ...' (locally on the host) has been working great for years now. You can zap them with the orchestrator as well, but only one disk a time, so a for loop would be useful.
There have been reports every now and then from users who tried to deploy many disks per host, I don't have a link available right now. And I haven't had the chance yet to deploy multiple hosts/many OSDs with 19.2.3, so there might be a regression in ceph-volume.
Regards, Eugen
Yes I have zapped all drives before each try... I'm looking in ceph-volume logs, but I don't have much history. I think I'd have to try again, looking at the logs...
Le 2025-08-18 16:21, Anthony D'Atri a écrit :
Yes I have zapped all drives before each try...
Did you subsequently check for success with `ceph device ls` and `lsblk`?
I've found that sometimes the orch zap doesn't succeed fully and one must manually stop and remove LVMs before the drive can be truly zapped.
I think yes, but I will retry. Another thing I've never done before, is using encryption. Perhaps it adds delays with my config, leading to timeouts... Does someone know which exact ceph-volume command is launched by such a spec file, if I want to launch it manually ? service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: unmanaged: false data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore
Yes I have zapped all drives before each try...
Did you subsequently check for success with `ceph device ls` and `lsblk`?
I've found that sometimes the orch zap doesn't succeed fully and one must manually stop and remove LVMs before the drive can be truly zapped.
I think yes, but I will retry. Another thing I've never done before, is using encryption. Perhaps it adds delays with my config, leading to timeouts...
I don't think it adds relevant delays, but it does make a new volume inside the crypto container which in itself is inside the LVM so both of those needs to be closed/removed before zap is going to be useful, otherwise the kernel retains info about the old devices and thats bad.
Hi again Gilles, I don't have the exact command , but you can use this ansible collection => https://galaxy.ansible.com/ui/repo/published/ceph/automation/ It provide the same command cephadm will use ( you can use here the ceph-volume module )and it's fully compatible with cephadm . I used it to built new ceph cluster . About , the ceph orch device ls command I told you earlier , I just want to check that you can see all your devices with their status (Available or not / Reason of rejection). Best Regards On 8/18/25 17:00, Gilles Mocellin wrote:
Le 2025-08-18 16:21, Anthony D'Atri a écrit :
Yes I have zapped all drives before each try...
Did you subsequently check for success with `ceph device ls` and `lsblk`?
I've found that sometimes the orch zap doesn't succeed fully and one must manually stop and remove LVMs before the drive can be truly zapped.
I think yes, but I will retry. Another thing I've never done before, is using encryption. Perhaps it adds delays with my config, leading to timeouts...
Does someone know which exact ceph-volume command is launched by such a spec file, if I want to launch it manually ?
service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: unmanaged: false data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 2025-08-18 17:00, Gilles Mocellin a écrit :
Le 2025-08-18 16:21, Anthony D'Atri a écrit :
Yes I have zapped all drives before each try...
Did you subsequently check for success with `ceph device ls` and `lsblk`?
I've found that sometimes the orch zap doesn't succeed fully and one must manually stop and remove LVMs before the drive can be truly zapped.
I think yes, but I will retry. Another thing I've never done before, is using encryption. Perhaps it adds delays with my config, leading to timeouts...
Does someone know which exact ceph-volume command is launched by such a spec file, if I want to launch it manually ?
service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: unmanaged: false data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
H, New try, from scratch every device clean (no Physical Volume). I've found that issue and recomandations : https://access.redhat.com/solutions/6545511 https://www.ibm.com/docs/en/storage-ceph/8.0.0?topic=80-bug-fixes So I set a higher timeout for cephadm commands : ceph config set global mgr/cephadm/default_cephadm_command_timeout 1800 Default was 900. I see le orchestrator launching ceph-volume commands with a timeout of 1795 (it was 895 before, don't know why it's 5s less than the config...). But, I've done that change after creating the OSD spec, so some daemon have failed/timed out still : root@fidcl-lyo1-sto-sds-lab-01:~# ceph health detail HEALTH_WARN Failed to apply 1 service(s): osd.throughput_optimized; 7 failed cephadm daemon(s); noout flag(s) set [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.throughput_optimized osd.throughput_optimized: Command timed out on host cephadm deploy (osd daemon) (default 1800 second timeout) [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) daemon osd.115 on fidcl-lyo1-sto-sds-lab-01 is in unknown state daemon osd.24 on fidcl-lyo1-sto-sds-lab-02 is in unknown state daemon osd.116 on fidcl-lyo1-sto-sds-lab-03 is in unknown state daemon osd.118 on fidcl-lyo1-sto-sds-lab-04 is in unknown state daemon osd.23 on fidcl-lyo1-sto-sds-lab-05 is in unknown state daemon osd.14 on fidcl-lyo1-sto-sds-lab-06 is in unknown state daemon osd.117 on fidcl-lyo1-sto-sds-lab-07 is in unknown state As in the RedHat issue, I've launched on every host : systemctl daemon-reload systemctl reset-failed But nothing. Until the end of cephadm OSD spec creation launch. Then, every thing is normal on ceph health, but I only have 82 OSD up out of 119 : root@fidcl-lyo1-sto-sds-lab-01:~# ceph -s cluster: id: 46030d0e-7d08-11f0-a50b-246e96bd90a4 health: HEALTH_WARN noout flag(s) set services: mon: 5 daemons, quorum fidcl-lyo1-sto-sds-lab-01,fidcl-lyo1-sto-sds-lab-02,fidcl-lyo1-sto-sds-lab-03,fidcl-lyo1-sto-sds-lab-05,fidcl-lyo1-sto-sds-lab-04 (age 78m) mgr: fidcl-lyo1-sto-sds-lab-01.ymlinv(active, since 89m), standbys: fidcl-lyo1-sto-sds-lab-02.otnpcx, fidcl-lyo1-sto-sds-lab-03.zasagv osd: 119 osds: 82 up (since 38m), 119 in (since 62m) flags noout data: pools: 1 pools, 1 pgs objects: 2 objects, 577 KiB usage: 1.8 TiB used, 90 TiB / 91 TiB avail pgs: 1 active+clean The OSDs are only visible in ceph osd ls command and in the dashboard. Daemons are not started, but PV/VG/LV are created. I will forget dmcrypt, I think with LVs tags, it's really easier to find the link between an OSD and it's LVs...
Hello Gilles, You can try to debug 'live' your failed OSDs by connecting to each concerned host , then finding the "unit.run" file ( of each osd ) which systemd unit use it to start the osd . Generally the file is under the following path => `/var/lib/ceph/<fsid>/osd.<id>/unit.run` Then , from that file run directly the two podman command (/usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM ...). Regards On 8/19/25 18:19, Gilles Mocellin wrote:
Le 2025-08-18 17:00, Gilles Mocellin a écrit :
Le 2025-08-18 16:21, Anthony D'Atri a écrit :
Yes I have zapped all drives before each try...
Did you subsequently check for success with `ceph device ls` and `lsblk`?
I've found that sometimes the orch zap doesn't succeed fully and one must manually stop and remove LVMs before the drive can be truly zapped.
I think yes, but I will retry. Another thing I've never done before, is using encryption. Perhaps it adds delays with my config, leading to timeouts...
Does someone know which exact ceph-volume command is launched by such a spec file, if I want to launch it manually ?
service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: unmanaged: false data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
H,
New try, from scratch every device clean (no Physical Volume).
I've found that issue and recomandations : https://access.redhat.com/solutions/6545511 https://www.ibm.com/docs/en/storage-ceph/8.0.0?topic=80-bug-fixes
So I set a higher timeout for cephadm commands : ceph config set global mgr/cephadm/default_cephadm_command_timeout 1800 Default was 900. I see le orchestrator launching ceph-volume commands with a timeout of 1795 (it was 895 before, don't know why it's 5s less than the config...).
But, I've done that change after creating the OSD spec, so some daemon have failed/timed out still :
root@fidcl-lyo1-sto-sds-lab-01:~# ceph health detail HEALTH_WARN Failed to apply 1 service(s): osd.throughput_optimized; 7 failed cephadm daemon(s); noout flag(s) set [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.throughput_optimized osd.throughput_optimized: Command timed out on host cephadm deploy (osd daemon) (default 1800 second timeout) [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) daemon osd.115 on fidcl-lyo1-sto-sds-lab-01 is in unknown state daemon osd.24 on fidcl-lyo1-sto-sds-lab-02 is in unknown state daemon osd.116 on fidcl-lyo1-sto-sds-lab-03 is in unknown state daemon osd.118 on fidcl-lyo1-sto-sds-lab-04 is in unknown state daemon osd.23 on fidcl-lyo1-sto-sds-lab-05 is in unknown state daemon osd.14 on fidcl-lyo1-sto-sds-lab-06 is in unknown state daemon osd.117 on fidcl-lyo1-sto-sds-lab-07 is in unknown state
As in the RedHat issue, I've launched on every host : systemctl daemon-reload systemctl reset-failed
But nothing. Until the end of cephadm OSD spec creation launch.
Then, every thing is normal on ceph health, but I only have 82 OSD up out of 119 :
root@fidcl-lyo1-sto-sds-lab-01:~# ceph -s cluster: id: 46030d0e-7d08-11f0-a50b-246e96bd90a4 health: HEALTH_WARN noout flag(s) set
services: mon: 5 daemons, quorum fidcl-lyo1-sto-sds-lab-01,fidcl-lyo1-sto-sds-lab-02,fidcl-lyo1-sto-sds-lab-03,fidcl-lyo1-sto-sds-lab-05,fidcl-lyo1-sto-sds-lab-04 (age 78m) mgr: fidcl-lyo1-sto-sds-lab-01.ymlinv(active, since 89m), standbys: fidcl-lyo1-sto-sds-lab-02.otnpcx, fidcl-lyo1-sto-sds-lab-03.zasagv osd: 119 osds: 82 up (since 38m), 119 in (since 62m) flags noout
data: pools: 1 pools, 1 pgs objects: 2 objects, 577 KiB usage: 1.8 TiB used, 90 TiB / 91 TiB avail pgs: 1 active+clean
The OSDs are only visible in ceph osd ls command and in the dashboard. Daemons are not started, but PV/VG/LV are created.
I will forget dmcrypt, I think with LVs tags, it's really easier to find the link between an OSD and it's LVs... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 2025-08-19 18:19, Gilles Mocellin a écrit :
Hi,
New try, from scratch every device clean (no Physical Volume).
I've found that issue and recomandations : https://access.redhat.com/solutions/6545511 https://www.ibm.com/docs/en/storage-ceph/8.0.0?topic=80-bug-fixes
So I set a higher timeout for cephadm commands : ceph config set global mgr/cephadm/default_cephadm_command_timeout 1800 Default was 900. I see le orchestrator launching ceph-volume commands with a timeout of 1795 (it was 895 before, don't know why it's 5s less than the config...).
But, I've done that change after creating the OSD spec, so some daemon have failed/timed out still :
root@fidcl-lyo1-sto-sds-lab-01:~# ceph health detail HEALTH_WARN Failed to apply 1 service(s): osd.throughput_optimized; 7 failed cephadm daemon(s); noout flag(s) set [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.throughput_optimized osd.throughput_optimized: Command timed out on host cephadm deploy (osd daemon) (default 1800 second timeout) [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) daemon osd.115 on fidcl-lyo1-sto-sds-lab-01 is in unknown state daemon osd.24 on fidcl-lyo1-sto-sds-lab-02 is in unknown state daemon osd.116 on fidcl-lyo1-sto-sds-lab-03 is in unknown state daemon osd.118 on fidcl-lyo1-sto-sds-lab-04 is in unknown state daemon osd.23 on fidcl-lyo1-sto-sds-lab-05 is in unknown state daemon osd.14 on fidcl-lyo1-sto-sds-lab-06 is in unknown state daemon osd.117 on fidcl-lyo1-sto-sds-lab-07 is in unknown state
As in the RedHat issue, I've launched on every host : systemctl daemon-reload systemctl reset-failed
[...]
The OSDs are only visible in ceph osd ls command and in the dashboard. Daemons are not started, but PV/VG/LV are created.
I will forget dmcrypt, I think with LVs tags, it's really easier to find the link between an OSD and it's LVs... _______________________________________________
After several rebuild, I can say now that : - dmcrypt does not play a role - It's a timeout issue, my OSD creations stop ~15min afeter OSD spec is applied (default cephadm timeout is 900s, 895 seen in the logs). If I create an empty cluster, set a bigger timeout with : ceph config set global mgr/cephadm/default_cephadm_command_timeout 1800 Then apply my OSD spec, it works. When I tried that last time, I change the timeout during cephadm execution, so not all cephadm commands used it, and the error reporting the timeout certainly show the new timeout (1800)... What I've seen that certainly could be improved is the non-atomic, even non serialized operations that leads to : 1- all OSD are created (OSD as in `ceph osd ls` 2- devices are created (PV/VG/LV/dmcrypt) 3- then one OSD by one, the folder /var/lib/ceph/FSID/osd.ID with block and block.db links 4- and the systemd unit is created and started The timeout happens during the step 3. As all the devices are now in use and ceph OSD ID already created, there is no means to recover I think and restart...
Yeah, it rang a bell when you mentioned the timeout. At least one other user reported this here on this list not long ago, but I forgot about that already. More responses inline... Zitat von Gilles Mocellin <gilles.mocellin@nuagelibre.org>:
Le 2025-08-19 18:19, Gilles Mocellin a écrit :
Hi,
New try, from scratch every device clean (no Physical Volume).
I've found that issue and recomandations : https://access.redhat.com/solutions/6545511 https://www.ibm.com/docs/en/storage-ceph/8.0.0?topic=80-bug-fixes
So I set a higher timeout for cephadm commands : ceph config set global mgr/cephadm/default_cephadm_command_timeout 1800 Default was 900. I see le orchestrator launching ceph-volume commands with a timeout of 1795 (it was 895 before, don't know why it's 5s less than the config...).
But, I've done that change after creating the OSD spec, so some daemon have failed/timed out still :
root@fidcl-lyo1-sto-sds-lab-01:~# ceph health detail HEALTH_WARN Failed to apply 1 service(s): osd.throughput_optimized; 7 failed cephadm daemon(s); noout flag(s) set [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.throughput_optimized osd.throughput_optimized: Command timed out on host cephadm deploy (osd daemon) (default 1800 second timeout) [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) daemon osd.115 on fidcl-lyo1-sto-sds-lab-01 is in unknown state daemon osd.24 on fidcl-lyo1-sto-sds-lab-02 is in unknown state daemon osd.116 on fidcl-lyo1-sto-sds-lab-03 is in unknown state daemon osd.118 on fidcl-lyo1-sto-sds-lab-04 is in unknown state daemon osd.23 on fidcl-lyo1-sto-sds-lab-05 is in unknown state daemon osd.14 on fidcl-lyo1-sto-sds-lab-06 is in unknown state daemon osd.117 on fidcl-lyo1-sto-sds-lab-07 is in unknown state
As in the RedHat issue, I've launched on every host : systemctl daemon-reload systemctl reset-failed
[...]
The OSDs are only visible in ceph osd ls command and in the dashboard. Daemons are not started, but PV/VG/LV are created.
I will forget dmcrypt, I think with LVs tags, it's really easier to find the link between an OSD and it's LVs... _______________________________________________
After several rebuild, I can say now that : - dmcrypt does not play a role - It's a timeout issue, my OSD creations stop ~15min afeter OSD spec is applied (default cephadm timeout is 900s, 895 seen in the logs).
If I create an empty cluster, set a bigger timeout with : ceph config set global mgr/cephadm/default_cephadm_command_timeout 1800
Then apply my OSD spec, it works.
When I tried that last time, I change the timeout during cephadm execution, so not all cephadm commands used it, and the error reporting the timeout certainly show the new timeout (1800)...
What I've seen that certainly could be improved is the non-atomic, even non serialized operations that leads to : 1- all OSD are created (OSD as in `ceph osd ls` 2- devices are created (PV/VG/LV/dmcrypt) 3- then one OSD by one, the folder /var/lib/ceph/FSID/osd.ID with block and block.db links 4- and the systemd unit is created and started
The timeout happens during the step 3.
This has also already been brought up, not sure if here or on Slack though. It seems like one of the suboptimal default settings that work for most use cases, but not all. Maybe a note in the docs could suffice to increase the timeout when the operator intends to deploy many OSDs per node at once. Not sure if adding one more option during bootstrap is worth the hastle.
As all the devices are now in use and ceph OSD ID already created, there is no means to recover I think and restart...
Not sure I understand what you mean.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If I create an empty cluster, set a bigger timeout with : ceph config set global mgr/cephadm/default_cephadm_command_timeout 1800
Then apply my OSD spec, it works.
When I tried that last time, I change the timeout during cephadm execution, so not all cephadm commands used it, and the error reporting the timeout certainly show the new timeout (1800)...
What I've seen that certainly could be improved is the non-atomic, even non serialized operations that leads to : 1- all OSD are created (OSD as in `ceph osd ls` 2- devices are created (PV/VG/LV/dmcrypt) 3- then one OSD by one, the folder /var/lib/ceph/FSID/osd.ID with block and block.db links 4- and the systemd unit is created and started
The timeout happens during the step 3.
This has also already been brought up, not sure if here or on Slack though. It seems like one of the suboptimal default settings that work for most use cases, but not all. Maybe a note in the docs could suffice to increase the timeout when the operator intends to deploy many OSDs per node at once. Not sure if adding one more option during bootstrap is worth the hastle.
That timeout seems a bit misplaced. If it runs the above loop and is making progress, ie, new OSDs coming online, then it should reset the timer or something, since the value 900 is just taken out of thin air on some design or some hw that is fast enough to not take 900s and not some other metric. Timeouts should reflect how much patience we have when it is NOT making progress. -- May the most significant bit of your life be positive.
Interesting idea. I suspect there may be nuances, but I that is terrific fodder for an RFE tracker issue.
On Aug 20, 2025, at 10:50 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
. If it runs the above loop and is making progress, ie, new OSDs coming online, then it should reset the timer or something, since the value 900 is just taken out of thin air on some design or some hw that is fast enough to not take 900s and not some other metric. Timeouts should reflect how much patience we have when it is NOT making progress.
Le 2025-08-20 17:15, Anthony D'Atri a écrit :
Interesting idea. I suspect there may be nuances, but I that is terrific fodder for an RFE tracker issue.
On Aug 20, 2025, at 10:50 AM, Janne Johansson <icepic.dz@gmail.com> wrote:
. If it runs the above loop and is making progress, ie, new OSDs coming online, then it should reset the timer or something, since the value 900 is just taken out of thin air on some design or some hw that is fast enough to not take 900s and not some other metric. Timeouts should reflect how much patience we have when it is NOT making progress.
An Issue on the tracker seems to match the timeout problem : https://tracker.ceph.com/issues/72603
Salut Gilles Did you try to check if ceph-volume regonized all of your HDD and SSD devices ? https://docs.ceph.com/en/squid/cephadm/services/osd/#list-devices Simply check the output of the followings command :
ceph orch device ls.
Regards On 8/18/25 12:11, Gilles Mocellin wrote:
Le 2025-08-18 11:47, Gilles Mocellin a écrit :
Le 2025-08-18 11:30, Gilles Mocellin a écrit :
On Hi Cephers,
I'm building a new Squid cluster with cephadm on Ubuntu 24.04. After expanding my cluster in the Dashboard (adding my 7 hosts), I choose throughput_optimized proflie wich create a generic spec for hybrid HDD/SSD :
service_type: osd service_id: throughput_optimized service_name: osd.throughput_optimized placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore
The cluster is for a LAB environment, on each of the 7 nodes, I have 17 HDD SAS 1.2TB drives and 1 SSD SAS Enterprise 400GB drive. On my first try, only 28 OSD where created (out of the 119), the others appeared as down, but they won't start, I didn't find systemd units created on the hosts. But, the VGs and LVs where created, there are 17 LVs on the SSD for WAL/DB of the 17 HDD (yes, small : 29GB).
On my second try, it creates 72 OSDs, still it stops, and never tries to continue, or re-create the down OSDs.
I didn't manage to find them, but it seems I saw some OSD creation timeout in the logs...
What can I do to have my missing OSD created ?
Some additional information :
ceph -s cluster: id: 3ebf83bf-7927-11f0-9f3a-246e96bd90a4 health: HEALTH_OK
services: mon: 5 daemons, quorum fidcl-lyo1-sto-sds-lab-01,fidcl-lyo1-sto-sds-lab-02,fidcl-lyo1-sto-sds-lab-03,fidcl-lyo1-sto-sds-lab-04,fidcl-lyo1-sto-sds-lab-05 (age 3d) mgr: fidcl-lyo1-sto-sds-lab-01.ovbjpb(active, since 3d), standbys: fidcl-lyo1-sto-sds-lab-02.nqdhpl, fidcl-lyo1-sto-sds-lab-03.cizytz osd: 119 osds: 72 up (since 3d), 89 in (since 41m)
data: pools: 1 pools, 1 pgs objects: 2 objects, 769 KiB usage: 1.5 TiB used, 79 TiB / 80 TiB avail pgs: 1 active+clean
One of the "missing" (not fully created) OSD, is present, but not found :
ceph osd find 6 { "osd": 6, "addrs": { "addrvec": [] }, "osd_fsid": "f8745284-8026-4713-8329-cd57cb6842f7", "crush_location": {} }
ceph osd info 6 osd.6 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) autoout,exists,new f8745284-8026-4713-8329-cd57cb6842f7
The missing OSDs are not chown with the device list command : ceph device ls | grep osd.6 is empty...
Also, the MGR reports that on all not fully created OSD : Aug 18 10:09:41 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[4186078]: mgr get_metadata_python Requested missing service osd.99 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 2025-08-18 14:28, Wissem MIMOUNA - Ceph Users a écrit :
Salut Gilles
Did you try to check if ceph-volume regonized all of your HDD and SSD devices ?
https://docs.ceph.com/en/squid/cephadm/services/osd/#list-devices
Simply check the output of the followings command :
ceph orch device ls.
Regards
Hi Wissem ! As I showed in one of my commands in previous mail, cephadm device list does show disks, but only with a created and operational OSDs. But, in the Dashboard, on the Disk pane and on the Host pane, I can see the other disks... I will retry, keeping an eye on ceph-volume logs.
participants (6)
-
Anthony D'Atri
-
Anthony D'Atri
-
Eugen Block
-
Gilles Mocellin
-
Janne Johansson
-
Wissem MIMOUNA - Ceph Users