I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland