I did a lot of tests with a virtual test cluster in the past two weeks with encrypted OSDs and I haven't encountered that yet. I would check for ceph-osd and ceph-volume package on the affected hosts. Or for remainders from previous cephadm installations, anything not properly cleaned up maybe? What do the OSDs log when they try to start up? It sounds like some stale systemd units or something, hard to tell. Have you rebooted the hosts in question? On one of my lab clusters I noticed issues with systemd when I re-bootstrap a cluster without cephadm multiple times. I have a bash script that works just fine on a freshly rebooted system, but after wiping everything and starting over, at some point the script doesn't work anymore, especially when it comes to create OSDs. But this might be unrelated, just mentioning it. Zitat von Eugeniusz Pokora <ceph-users@ceph.io>:
When upgrading Ceph cluster from version 19.2.4 to 19.2.6 using cephadm, an issue occurred with encrypted OSDs.
2026-09-02 10:53:28,494 7f24e08141c0 DEBUG Determined image: 'quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089' 2026-09-02 10:53:28,501 7f24e08141c0 INFO Reconfig daemon osd.36 ... 2026-09-02 10:53:29,885 7f24e08141c0 INFO Stopping osd.36 to update osd_key bluestore label 2026-09-02 10:53:31,670 7f24e08141c0 INFO Opening osd.36 with crypsetup 2026-09-02 10:53:33,955 7f24e08141c0 INFO Non-zero exit code 5 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /tmp/cryptsetup_action.sh --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 -e NODE_NAME=k08r06cph03 -v /dev:/dev -v /tmp/cephadm-osd-36-rotate-helper.sh:/tmp/cryptsetup_action.sh quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 2026-09-02 10:53:33,956 7f24e08141c0 INFO /usr/bin/podman: stderr Cannot use device /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or mounted). 2026-09-02 10:53:33,956 7f24e08141c0 ERROR ERROR: Got error rotating osd keyring while using cryptsetup tool Out: Err:Cannot use device /dev/ceph-d1270182-40c9-49c8-9a52-f29d0475aeba/osd-block-80ae3d20-07df-4db5-96b6-2ad575a592ad which is in use (already mapped or mounted).
The issue did not occur on all hosts in the cluster (fortunately, it was a test env), but it affected all OSDs on affected host. After error occurred OSDs remained in a down state and had to be up manually. After several attempts, the update was successfully completed, but we had to rotate the OSD keyrings ourselves. A few hours latter when monmap was changed and cephadm scheduled a reconfig this problem happened again, on 2 out of 4 nodes of the cluster all osds goes down due to cephadm reconfig and this same error: "Cannot use device... which is in use (already mapped or mounted)".
Has anyone encountered similar problems?
Regards Eugeniusz P