Hi, is it really 19.2.1 and not 19.2.0? Sounds like the balancer issue to me, but it was supposed to be fixed in 19.2.1. Would you mind sharing ‚ceph versions‘ output just to make sure? If it’s not the balancer bug, could you please share the mgr log? Zitat von Moritz Baumann <mo@mo.homeip.net>:

I was able to set the orchestrator back to "cephadm" via the web-dashboard.

So ceph02[0]:~# ceph config get mgr mgr/orchestrator/orchestrator

yields

cephadm

But it is still not working.

Am 09.03.2025 um 09:39 schrieb Moritz Baumann:

...

Hi, we are running a Ceph cluster (on almalinux-9 installed with cephadm) and after I failed the active ceph mgr with "ceph mgr fail "ceph01.xxxxx" to do a "ceph orch host maintenance enter ceph01"

I got a:

Error ENOENT: Module not found

I then did a:

ceph01[0]:~# ceph orch set backend "" ceph01[0]:~# ceph orch set backend "cephadm" Error ENOENT: Module not found ceph01[2]:~# ceph orch host ls Error ENOENT: No orchestrator configured (try `ceph orch set backend`)

I do I get my orchestrator back to up and running?

Best Moritz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, I am trying to add OSDs to freshly installed 3 Node Ceph Cluster but don't know how to solve my problem. It looks like all nodes share the same problem. Kernel Options "cgroup_memory=1 cgroup_enable=memory" are set. ceph orch apply osd --all-available-devices --unmanaged=false ceph health detail HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices; OSD count 0 < osd_pool_default_size 3 [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices     osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/mon.pi5n2/config Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 -e NODE_NAME=pi5n2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/run/ceph:z -v /var/log/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/log/ceph:z -v /var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk8ddqklv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp467idmf9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 lvm batch --no-auto /dev/nvme0n1 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 1 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a7ca71ee-9a57-4fda-a73d-4ba3373df344 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /dev/nvme0n1 /usr/bin/podman: stderr  stdout: Physical volume "/dev/nvme0n1" successfully created. /usr/bin/podman: stderr  stdout: Volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" successfully created /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 244190 -n osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /usr/bin/podman: stderr  stdout: Logical volume "osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344" created. /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2 /usr/bin/podman: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 /usr/bin/podman: stderr Running command: /usr/bin/ln -s /dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 /var/lib/ceph/osd/ceph-2/block /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap /usr/bin/podman: stderr  stderr: got monmap epoch 3 /usr/bin/podman: stderr --> Creating keyring file for osd.2 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/ /usr/bin/podman: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid a7ca71ee-9a57-4fda-a73d-4ba3373df344 --setuser ceph --setgroup ceph /usr/bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.2 --yes-i-really-mean-it /usr/bin/podman: stderr  stderr: purged osd.2 /usr/bin/podman: stderr --> Zapping: /dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 /usr/bin/podman: stderr --> Unmounting /var/lib/ceph/osd/ceph-2 /usr/bin/podman: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2 /usr/bin/podman: stderr  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted /usr/bin/podman: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 bs=1M count=10 conv=fsync /usr/bin/podman: stderr  stderr: 10+0 records in /usr/bin/podman: stderr 10+0 records out /usr/bin/podman: stderr  stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0413744 s, 253 MB/s /usr/bin/podman: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /usr/bin/podman: stderr  stderr: Removing ceph--722ced72--3e8c--41bb--946b--94d10f6c9734-osd--block--a7ca71ee--9a57--4fda--a73d--4ba3373df344 (254:0) /usr/bin/podman: stderr  stderr: Releasing logical volume "osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344" /usr/bin/podman: stderr   Archiving volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" metadata (seqno 5). /usr/bin/podman: stderr  stdout: Logical volume "osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344" successfully removed. /usr/bin/podman: stderr  stderr: Removing physical volume "/dev/nvme0n1" from volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" /usr/bin/podman: stderr  stdout: Volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" successfully removed /usr/bin/podman: stderr  stderr: Creating volume group backup "/etc/lvm/backup/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" (seqno 6). /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme0n1 /usr/bin/podman: stderr  stdout: Labels on physical volume "/dev/nvme0n1" successfully wiped. /usr/bin/podman: stderr --> Zapping successful for OSD: 2 /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 33, in <module> /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/podman: stderr     self.main(self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr     return f(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main /usr/bin/podman: stderr     self._execute(plan) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute /usr/bin/podman: stderr     c.create(argparse.Namespace(**args)) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create /usr/bin/podman: stderr     prepare_step.safe_prepare(args) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare /usr/bin/podman: stderr     self.prepare() /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/prepare.py", line 278, in prepare /usr/bin/podman: stderr     prepare_bluestore( /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/prepare.py", line 59, in prepare_bluestore /usr/bin/podman: stderr     prepare_utils.osd_mkfs_bluestore( /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore /usr/bin/podman: stderr     raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command))) /usr/bin/podman: stderr RuntimeError: Command failed with exit code -11: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid a7ca71ee-9a57-4fda-a73d-4ba3373df344 --setuser ceph --setgroup ceph Traceback (most recent call last):   File "<frozen runpy>", line 198, in _run_module_as_main   File "<frozen runpy>", line 88, in _run_code   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 10889, in <module>     main()     ~~~~^^   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 10877, in main     r = ctx.func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2576, in _infer_config     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2492, in _infer_fsid     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2604, in _infer_image     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2479, in _validate_fsid     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 7145, in command_ceph_volume     out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2267, in call_throws     raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 -e NODE_NAME=pi5n2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/run/ceph:z -v /var/log/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/log/ceph:z -v /var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk8ddqklv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp467idmf9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 lvm batch --no-auto /dev/nvme0n1 --yes --no-systemd [WRN] TOO_FEW_OSDS: OSD count 0 < osd_pool_default_size 3

Or do you have multiple OSD specs defined? Can you show the output of ‚ceph orch ls osd --export‘? Zitat von filip Mutterer <filip@3dds.io>:

Hi,

I am trying to add OSDs to freshly installed 3 Node Ceph Cluster but don't know how to solve my problem.

It looks like all nodes share the same problem.

Kernel Options "cgroup_memory=1 cgroup_enable=memory" are set.

ceph orch apply osd --all-available-devices --unmanaged=false

ceph health detail HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices; OSD count 0 < osd_pool_default_size 3 [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices     osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/mon.pi5n2/config Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 -e NODE_NAME=pi5n2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/run/ceph:z -v /var/log/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/log/ceph:z -v /var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk8ddqklv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp467idmf9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 lvm batch --no-auto /dev/nvme0n1 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 1 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a7ca71ee-9a57-4fda-a73d-4ba3373df344 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /dev/nvme0n1 /usr/bin/podman: stderr  stdout: Physical volume "/dev/nvme0n1" successfully created. /usr/bin/podman: stderr  stdout: Volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" successfully created /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 244190 -n osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /usr/bin/podman: stderr  stdout: Logical volume "osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344" created. /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2 /usr/bin/podman: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 /usr/bin/podman: stderr Running command: /usr/bin/ln -s /dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 /var/lib/ceph/osd/ceph-2/block /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap /usr/bin/podman: stderr  stderr: got monmap epoch 3 /usr/bin/podman: stderr --> Creating keyring file for osd.2 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/ /usr/bin/podman: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid a7ca71ee-9a57-4fda-a73d-4ba3373df344 --setuser ceph --setgroup ceph /usr/bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.2 --yes-i-really-mean-it /usr/bin/podman: stderr  stderr: purged osd.2 /usr/bin/podman: stderr --> Zapping: /dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 /usr/bin/podman: stderr --> Unmounting /var/lib/ceph/osd/ceph-2 /usr/bin/podman: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2 /usr/bin/podman: stderr  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted /usr/bin/podman: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734/osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344 bs=1M count=10 conv=fsync /usr/bin/podman: stderr  stderr: 10+0 records in /usr/bin/podman: stderr 10+0 records out /usr/bin/podman: stderr  stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0413744 s, 253 MB/s /usr/bin/podman: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-722ced72-3e8c-41bb-946b-94d10f6c9734 /usr/bin/podman: stderr  stderr: Removing ceph--722ced72--3e8c--41bb--946b--94d10f6c9734-osd--block--a7ca71ee--9a57--4fda--a73d--4ba3373df344 (254:0) /usr/bin/podman: stderr  stderr: Releasing logical volume "osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344" /usr/bin/podman: stderr   Archiving volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" metadata (seqno 5). /usr/bin/podman: stderr  stdout: Logical volume "osd-block-a7ca71ee-9a57-4fda-a73d-4ba3373df344" successfully removed. /usr/bin/podman: stderr  stderr: Removing physical volume "/dev/nvme0n1" from volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" /usr/bin/podman: stderr  stdout: Volume group "ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" successfully removed /usr/bin/podman: stderr  stderr: Creating volume group backup "/etc/lvm/backup/ceph-722ced72-3e8c-41bb-946b-94d10f6c9734" (seqno 6). /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme0n1 /usr/bin/podman: stderr  stdout: Labels on physical volume "/dev/nvme0n1" successfully wiped. /usr/bin/podman: stderr --> Zapping successful for OSD: 2 /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 33, in <module> /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/podman: stderr     self.main(self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr     return f(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main /usr/bin/podman: stderr     self._execute(plan) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute /usr/bin/podman: stderr     c.create(argparse.Namespace(**args)) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create /usr/bin/podman: stderr     prepare_step.safe_prepare(args) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare /usr/bin/podman: stderr     self.prepare() /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/prepare.py", line 278, in prepare /usr/bin/podman: stderr     prepare_bluestore( /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/prepare.py", line 59, in prepare_bluestore /usr/bin/podman: stderr     prepare_utils.osd_mkfs_bluestore( /usr/bin/podman: stderr   File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore /usr/bin/podman: stderr     raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command))) /usr/bin/podman: stderr RuntimeError: Command failed with exit code -11: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osdspec-affinity all-available-devices --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid a7ca71ee-9a57-4fda-a73d-4ba3373df344 --setuser ceph --setgroup ceph Traceback (most recent call last):   File "<frozen runpy>", line 198, in _run_module_as_main   File "<frozen runpy>", line 88, in _run_code   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 10889, in <module>     main()     ~~~~^^   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 10877, in main     r = ctx.func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2576, in _infer_config     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2492, in _infer_fsid     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2604, in _infer_image     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2479, in _validate_fsid     return func(ctx)   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 7145, in command_ceph_volume     out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/cephadm.91b52e446d8f1d91339889933063a5070027dc00f54d563f523727c6dd22b172/__main__.py", line 2267, in call_throws     raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 -e NODE_NAME=pi5n2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/run/ceph:z -v /var/log/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba:/var/log/ceph:z -v /var/lib/ceph/9df6f788-ff59-11ef-849b-d83adde0aaba/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk8ddqklv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp467idmf9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:259cbcf17122953765b8febf2e01682caf11c883bd41ffcea12d2afb88f5b9b5 lvm batch --no-auto /dev/nvme0n1 --yes --no-systemd [WRN] TOO_FEW_OSDS: OSD count 0 < osd_pool_default_size 3

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io