Hi there, I have ceph cluster created by ceph-volume - bluestore, in every node is 12 HDD and 1 NVMe, which is divided to 24 LVM partition for DB and WAL. I've turned this cluster to 'ceph orch' management, then I've moved to quincy release (now I'm using a 17.2.5 version). I had to move whole cluster to another addreses and another hostnames. MON, MGR and MDS goes without problem, but OSD was really pain process :-( Now I have cluster with this problem: # ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.mon1.ulytsa mon1 running (11w) 3m ago 11w 5609M - 17.2.5 cc65afd6173a db1aa336263a mds.cephfs.mon2.zxhxqk mon2 running (11w) 3m ago 11w 33.1M - 17.2.5 cc65afd6173a 5b9ced4a4b71 mds.cephfs.mon3.rpkvlt mon3 running (11w) 3m ago 11w 32.4M - 17.2.5 cc65afd6173a 045e23f124aa mgr.mon1.buqyga mon1 *:8080 running (11w) 3m ago 11w 2300M - 17.2.5 cc65afd6173a 9577239527b5 mgr.mon2.goghws mon2 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 4fb1ae26765e mgr.mon3.slpgay mon3 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 06e491084a5e mon.mon1 mon1 running (11w) 3m ago 11w 1576M 2048M 17.2.5 cc65afd6173a 2f18c737faa9 mon.mon2 mon2 running (11w) 3m ago 11w 1598M 2048M 17.2.5 cc65afd6173a 31091cbbfb8e mon.mon3 mon3 running (11w) 3m ago 11w 1463M 2048M 17.2.5 cc65afd6173a 4d0b094c9ca1 osd.0 osd1 running (9w) 3m ago 10w 5133M 3745M 17.2.5 cc65afd6173a 3b28e48d3630 osd.1 osd1 running (7w) 3m ago 10w 5425M 3745M 17.2.5 cc65afd6173a 3336ccdfd232 osd.2 osd1 running (9w) 3m ago 10w 5223M 3745M 17.2.5 cc65afd6173a e8fc077aef59 osd.3 osd1 running (9w) 3m ago 10w 5050M 3745M 17.2.5 cc65afd6173a 4fbf34450237 osd.4 osd1 running (9w) 3m ago 10w 7526M 3745M 17.2.5 cc65afd6173a a4875c354540 osd.5 osd1 running (9w) 3m ago 10w 4854M 3745M 17.2.5 cc65afd6173a b006526228ae osd.6 osd1 running (9w) 3m ago 10w 6498M 3745M 17.2.5 cc65afd6173a 4c326271e188 osd.7 osd1 running (9w) 3m ago 10w 4410M 3745M 17.2.5 cc65afd6173a ca0f3ce31031 osd.8 osd1 running (9w) 3m ago 10w 7337M 3745M 17.2.5 cc65afd6173a 99269a832819 osd.9 osd1 running (9w) 3m ago 10w 4717M 3745M 17.2.5 cc65afd6173a f39ce0bb5316 osd.10 osd1 running (9w) 3m ago 10w 4295M 3745M 17.2.5 cc65afd6173a 0871793fa261 osd.11 osd1 running (9w) 3m ago 10w 5552M 3745M 17.2.5 cc65afd6173a 32a8b589b3bd osd.24 osd3 running (109m) 3m ago 6M 3306M 3745M 17.2.5 cc65afd6173a 466d80a55d96 osd.25 osd3 running (109m) 3m ago 6M 3145M 3745M 17.2.5 cc65afd6173a b1705621116a osd.26 osd3 running (109m) 3m ago 6M 3063M 3745M 17.2.5 cc65afd6173a c30253a1a83f osd.27 osd3 running (109m) 3m ago 6M 3257M 3745M 17.2.5 cc65afd6173a aa0a647d93f1 osd.28 osd3 running (109m) 3m ago 6M 2244M 3745M 17.2.5 cc65afd6173a d3c68ed6572b osd.29 osd3 running (109m) 3m ago 6M 3509M 3745M 17.2.5 cc65afd6173a 2c425b17abf7 osd.30 osd3 running (109m) 3m ago 6M 3814M 3745M 17.2.5 cc65afd6173a 44747256b34a osd.31 osd3 running (109m) 3m ago 6M 2958M 3745M 17.2.5 cc65afd6173a b7b7946fa24e osd.32 osd3 running (109m) 3m ago 6M 3016M 3745M 17.2.5 cc65afd6173a fc9c024fed4f osd.33 osd3 running (109m) 3m ago 6M 5366M 3745M 17.2.5 cc65afd6173a edc2dbd9c556 osd.34 osd3 running (109m) 3m ago 6M 4577M 3745M 17.2.5 cc65afd6173a 46d7668742cf osd.35 osd3 running (109m) 3m ago 6M 2538M 3745M 17.2.5 cc65afd6173a 96a15a9ad3d7 osd.36 osd4 running (103m) 3m ago 8w 2707M 3745M 17.2.5 cc65afd6173a adf884af609b osd.37 osd4 running (103m) 3m ago 6M 3347M 3745M 17.2.5 cc65afd6173a 8f824026c6ae osd.38 osd4 running (103m) 3m ago 6M 3377M 3745M 17.2.5 cc65afd6173a 2a70c0b860ff osd.39 osd4 running (103m) 3m ago 6M 2814M 3745M 17.2.5 cc65afd6173a 4d5833f1faaf osd.40 osd4 running (103m) 3m ago 6M 2633M 3745M 17.2.5 cc65afd6173a e2e492c4f4a6 osd.41 osd4 running (103m) 3m ago 6M 2141M 3745M 17.2.5 cc65afd6173a 80f67faf1238 osd.42 osd4 running (103m) 3m ago 6M 1527M 3745M 17.2.5 cc65afd6173a c1aef5891ad5 osd.43 osd4 running (103m) 3m ago 6M 1927M 3745M 17.2.5 cc65afd6173a b092b03f211d osd.44 osd4 running (103m) 3m ago 6M 3332M 3745M 17.2.5 cc65afd6173a 9309d93c80da osd.45 osd4 running (103m) 3m ago 6M 3172M 3745M 17.2.5 cc65afd6173a 8a47d7c155cd osd.46 osd4 running (103m) 3m ago 6M 2545M 3745M 17.2.5 cc65afd6173a 0550b66a837f osd.47 osd4 running (103m) 3m ago 6M 3010M 3745M 17.2.5 cc65afd6173a 52328564e503 osd.48 osd5 running (8w) 8m ago 8w 6054M 3745M 17.2.5 cc65afd6173a c5494c95566e osd.49 osd5 running (8w) 8m ago 8w 5648M 3745M 17.2.5 cc65afd6173a cebc3405950e osd.50 osd5 running (8w) 8m ago 8w 4898M 3745M 17.2.5 cc65afd6173a 44bd2f018476 osd.51 osd5 running (8w) 8m ago 8w 5066M 3745M 17.2.5 cc65afd6173a 008cb9d89e68 osd.52 osd5 running (8w) 8m ago 8w 6325M 3745M 17.2.5 cc65afd6173a db3d01bbfce7 osd.53 osd5 running (8w) 8m ago 8w 6040M 3745M 17.2.5 cc65afd6173a 215c70cf7ca3 osd.54 osd5 running (8w) 8m ago 8w 5147M 3745M 17.2.5 cc65afd6173a 92a4bc3eae20 osd.55 osd5 running (8w) 8m ago 8w 6590M 3745M 17.2.5 cc65afd6173a ce07fe507b47 osd.56 osd5 running (8w) 8m ago 8w 3838M 3745M 17.2.5 cc65afd6173a 540cf969419d osd.57 osd5 running (8w) 8m ago 8w 4820M 3745M 17.2.5 cc65afd6173a 89653e57e85c osd.58 osd5 running (8w) 8m ago 8w 5016M 3745M 17.2.5 cc65afd6173a c19d06b09533 osd.59 osd5 running (8w) 8m ago 8w 6547M 3745M 17.2.5 cc65afd6173a 8c387d3d06a6 osd.60 osd6 running (99m) 3m ago 6M 2050M 3745M 17.2.5 cc65afd6173a 90c1d1745480 osd.61 osd6 running (99m) 3m ago 6M 2694M 3745M 17.2.5 cc65afd6173a 32da4b4cec45 osd.62 osd6 running (99m) 3m ago 6M 2608M 3745M 17.2.5 cc65afd6173a 43788b932105 osd.63 osd6 running (99m) 3m ago 6M 4048M 3745M 17.2.5 cc65afd6173a 4a6863748bc5 osd.64 osd6 running (99m) 3m ago 6M 2542M 3745M 17.2.5 cc65afd6173a c49dd095aa64 osd.65 osd6 running (99m) 3m ago 6M 2796M 3745M 17.2.5 cc65afd6173a cdb2bed58cf3 osd.66 osd6 running (99m) 3m ago 6M 2129M 3745M 17.2.5 cc65afd6173a c17db58e047f osd.67 osd6 running (99m) 3m ago 6M 3559M 3745M 17.2.5 cc65afd6173a 46f2527c6112 osd.68 osd6 running (99m) 3m ago 6M 2552M 3745M 17.2.5 cc65afd6173a c4491c1b8de9 osd.69 osd6 running (99m) 3m ago 6M 2750M 3745M 17.2.5 cc65afd6173a 9e03c1d5351e osd.70 osd6 running (99m) 3m ago 6M 1849M 3745M 17.2.5 cc65afd6173a 3454e09aab70 osd.72 osd6 running (99m) 3m ago 8w 5368M 3745M 17.2.5 cc65afd6173a 0c1ca9a8677d # ceph orch host ls HOST ADDR LABELS STATUS iscsi1 192.168.6.166 iscsi2 192.168.6.167 mon1 192.168.7.208 mon2 192.168.7.209 mon3 192.168.7.210 osd1 192.168.7.214 osd2 192.168.7.215 osd3 192.168.7.216 osd4 192.168.7.217 osd5 192.168.7.218 osd6 192.168.7.219 11 hosts in cluster # ceph health detail HEALTH_WARN 12 stray daemon(s) not managed by cephadm; 3 stray host(s) with 35 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 12 stray daemon(s) not managed by cephadm stray daemon osd.12 on host osd2 not managed by cephadm stray daemon osd.13 on host osd2 not managed by cephadm stray daemon osd.14 on host osd2 not managed by cephadm stray daemon osd.15 on host osd2 not managed by cephadm stray daemon osd.16 on host osd2 not managed by cephadm stray daemon osd.17 on host osd2 not managed by cephadm stray daemon osd.18 on host osd2 not managed by cephadm stray daemon osd.19 on host osd2 not managed by cephadm stray daemon osd.20 on host osd2 not managed by cephadm stray daemon osd.21 on host osd2 not managed by cephadm stray daemon osd.22 on host osd2 not managed by cephadm stray daemon osd.23 on host osd2 not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 3 stray host(s) with 35 daemon(s) not managed by cephadm stray host c-osd3 has 12 stray daemons: ['osd.24', 'osd.25', 'osd.26', 'osd.27', 'osd.28', 'osd.29', 'osd.30', 'osd.31', 'osd.32', 'osd.33', 'osd.34', 'osd.35'] stray host c-osd4 has 12 stray daemons: ['osd.36', 'osd.37', 'osd.38', 'osd.39', 'osd.40', 'osd.41', 'osd.42', 'osd.43', 'osd.44', 'osd.45', 'osd.46', 'osd.47'] stray host c-osd6 has 11 stray daemons: ['osd.60', 'osd.61', 'osd.62', 'osd.63', 'osd.64', 'osd.65', 'osd.66', 'osd.67', 'osd.68', 'osd.69', 'osd.70'] Comments: 1) c-osd3, c-osd4, c-osd6 are old names (with old addresses) which are not in cluster now, as you can see from 'ceph orch ps' listing. 2) On the osd1 machine I've tried create osd daemons bych 'ceph orch', but I cannot create with one NVMe for DB and WAL together, thus I have OSD without cached DB and WAL. 3) On the osd2, there is a OSD daemons created by ceph-volume 4) On the other hosts I've renamed host, moved to another IP address, 'ceph orch rm c-osdx', 'ceph orch add osdx', in LVM changed all c-osdx tags to osdx tags, now I've changed in /var/lib/ceph/uuid/osd.x/unit.run and unit.poststop every c-osdx to osdx. And now questions: Is there a way to definitelly remove c-osd3, c-osd4 and c-osd6 from 'ceph health detail' and manage this host via 'ceph orch'? On the osd2, there is a problem to adopt OSD created by ceph-volume to ceph orchestrator: cephadm adopt --style legacy -n osd.12 Pulling container image quay.io/ceph/ceph:v17... Found online OSD at //var/lib/ceph/osd/ceph-12/fsid objectstore_type is bluestore Stopping old systemd unit ceph-osd@12... Disabling old systemd unit ceph-osd@12... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9468, in <module> main() File "/usr/sbin/cephadm", line 9456, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2135, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6582, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6755, in command_adopt_ceph os.rmdir(data_dir_src) OSError: [Errno 39] Directory not empty: '//var/lib/ceph/osd/ceph-12' Thanks a lot. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html