Re: Advice on managing adopted OSDs
Ok, that was the conclusion that I was starting to come to. Thanks so much for the notes - I'll have a careful look through them. I found this note in Service Management\Declarative State that reinforces that... /The “osd” service used to track OSDs that are not tied to any specific service spec is special and will always be marked unmanaged. Attempting to modify it with |ceph orch set-unmanaged| or |ceph orch set-managed| will result in a message |No service of name osd found. Check "ceph orch ls" for all known services|/ I also found a note in OSD Service\Creating New OSDs... /When deploying new OSDs with |cephadm|, ensure that the |ceph-osd| package is not already installed on the target host. If it is installed, conflicts may arise in the management and control of the OSD that may lead to errors or unexpected behavior./ ...so will remove all the packages first (except cephadm). It would be useful to have some definitive statements about managing OSDs in the Converting an existing cluster to cephadm doc, other than just the initial OSD adoption itself. Regards, Chris On 13/01/2026 17:45, Julien Laurenceau via ceph-users wrote:
Hello,
To my knowledge there is no “adopt” or in-place migration procedure in Squid.
You must remove and redeploy OSDs to bring them under management.
Migration Steps: a. Identify unmanaged OSDs ``` ceph orch ls --service-type osd ceph orch ps --daemon-type osd ``` or ``` ceph osd tree # easy way to retrieve the device used by a specific osd : ceph device ls # ceph device ls |grep -v nvme | awk -F '.' '{print $NF}' | sort -n ceph osd metadata ${osdid} |grep -e '"hostname"' -e '"devices"'
listosd=$(ceph device ls |grep $h |grep osd | awk -F '.' '{print $NF}' | awk '{print $1}'| sort -n) osd_devices=$(ceph device ls |grep $h |grep osd | awk -F "${h}:" '{print $NF}' | awk '{print $1}'| sort -n) for osdid in ${listosd}; do ceph osd metadata ${osdid} |grep -e '"hostname"' -e '"devices"'; done ``` b. Mark OSDs out and wait for healthy ``` ceph osd out ${osdid} for osdid in ${listosd}; do ceph osd out ${osdid}; done # Wait for until no object degraded and no Reduced data availability ``` Wait until there are no degraded objects. c. Mark the OSD as “down” ``` ceph osd down ${osdid} for osdid in ${listosd}; do ceph osd down ${osdid}; done ``` d. Stop the OSD daemon on the host On the host where the OSD runs, stop the OSD service: ``` systemctl list-units --no-legend --no-pager "ceph-*@osd.*" systemctl stop ceph-${clusterid}@osd.${osdid} for osdid in ${listosd}; do systemctl stop ceph-${clusterid}@osd.${osdid}; done ``` e. Remove the OSD ``` ceph osd purge ${osdid} --yes-i-really-mean-it ceph orch daemon rm osd.$osdid --force for osdid in ${listosd}; do ceph osd purge ${osdid} --yes-i-really-mean-it; done for osdid in ${listosd}; do ceph orch daemon rm osd.$osdid --force; done ``` f. Clean the device on the host ``` # osd_devices="nvme0n1 nvme1n1 nvme2n1 nvme3n1 nvme4n1 nvme5n1 nvme6n1 nvme7n1" # osd_devices="sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp sdq sdr sds sdt sdu sdv sdw sdx" ceph-volume lvm zap /dev/sdX --destroy # or sgdisk --zap-all /dev/sdX for d in ${osd_devices}; do sgdisk --zap-all /dev/${d}; done
if zap it is not sufficient NUKE THE DRIVE : for dm in $(dmsetup ls | grep '^ceph--' | awk '{print $1}'); do dmsetup remove ${dm}; done for vg in $(vgdisplay -s | grep '"ceph-' |awk '{print $1}' | tr -d '"') ; do vgremove --force $vg; done for d in ${osd_devices}; do echo "nuke $d"; pvremove /dev/${d}; wipefs -af /dev/${d}; sfdisk --delete /dev/${d}; partprobe /dev/${d}; done ``` g. Create and apply a DriveGroup YAML for the host/device
(see section above) ``` cat << EOF >ceph-osd-$h.yaml service_type: osd service_id: osd-${h} placement: hosts: - ${h} data_devices: paths: EOF for dev in $(echo ${osd_devices} | tr -s ',' ' '); do echo " - /dev/${dev}" >> ceph-osd-$h.yaml done ceph orch apply -i ceph-osd-${h}.yaml ceph orch ls --service-type osd # ceph orch rm osd.osd-$h
# apply device-class if needed : mydevclass=nvme-all listosd=$(ceph device ls |grep $h |grep osd | awk -F '.' '{print $NF}' | awk '{print $1}'| sort -n) for osdid in ${listosd}; do ceph osd crush set-device-class ${mydevclass} ${osdid}; done ```
h. Verify ``` ceph orch ps --daemon-type osd ceph osd tree ceph -s ```
On 1/13/26 17:00, Kirby Haze via ceph-users wrote:
There should be a `orch set-managed` and `orch set-unmanaged` so doing `ceph orch set-managed osd` should work, then it should reflect in the yaml shortly after.
- Kirby
On Tue, Jan 13, 2026 at 1:24 AM Chris Palmer via ceph-users < ceph-users@ceph.io> wrote:
I would appreciate advice on the "best" way to manage adopted OSDs.
This is a test squid 19.2.3 cluster, originally installed using centos 9 packages. I am testing the procedure for adopting, then will use cephadm to update to tentacle. Once I am happy with the whole process I will repeat on a small production cluster.
I have successfully adopted mon, mgr, osd, rgw & mds as described in the docs. That left rgw & mds managed, but mon, mgr & osds unmanaged. Setting the mon & mgr services managed was easy by tweaking the yml and applying. But I am unclear how to best handle osds. That leaves me in this state:
[root@c4 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mds.cephfs 2/2 51s ago 21h c1;c2 mgr 3/3 52s ago 21h c1;c2;c4 mon 3/3 51s ago 21h c1;c2;c3 osd 12 51s ago - <unmanaged> rgw.rgw ?:80 2/2 51s ago 21h c1;c2
The exported yml had the following for osds, which had to be commented out as it would not parse (missing placement, missing data_devices etc):
service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore
My questions are:
* With the OSDs unmanaged, what functionality am I missing? In particular, things like using cephadm to replace a failed disk... * These are only small clusters, so I am happy to have the OSDs enumerated in the yml, and not have ceph try to find and automatically use available disks. Is it possible to have the full functionality except from automatically grabbing available disks? * How can I generate yml that reflects the current state of the OSDs? We currently have pools (per application) for NVMe, SSD, and HDD with shared NVMe DB/WAL. * I can't find documentation for transitioning OSDs from unmanaged to managed. Have I just overlooked it? * Is any part of this easier on tentacle than squid? If so I can upgrade the package squid to tentacle before adopting. (But unless there is a good reason, I would rather have familiarised myself with the cephadm upgrade process before starting any of this on the production cluster...)
Many thanks for any insights.
Chris _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (1)
-
Chris Palmer