OSD creation from service spec fails to check all db_devices for available space
Hi I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL. The orchaetrator now fails to create any more OSDs due to: " /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled " It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed. Suggestions? " 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd " " [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd--block--76e673e9--d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block--b7330837--b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd--block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd--block--8a0c4a74--fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block--c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd--block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd--block--618f7582--fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd--block--8e9b21b6--cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd--block--14f68253--b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd--block--2b475461--a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block--f08c7e71--ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd--block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block--c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block--c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd--block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block--a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd--block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block--ff12d109--dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block--b3e38c3d--e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# " Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
On 19/03/2025 08:33, Torkil Svensgaard wrote:
Hi
I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL.
The orchaetrator now fails to create any more OSDs due to:
" /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled "
So I kept adding more HDDs since that has to be done anyway and noticed that the error changed: /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Added 4 drives /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 212.92 GB can be fulfilled Added 4 drives /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 186.30 GB can be fulfilled So it looks like ceph-volume is doing some sort of total calculation with the drives and devices passed and thinks there is too little room for some reason. Workaround seems to be adding the drives one at a time. Mvh. Torkil
It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed.
Suggestions?
" 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config / var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd "
" [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd--block--76e673e9-- d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block--b7330837-- b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd-- block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd--block--8a0c4a74-- fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block-- c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd-- block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd--block--618f7582-- fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd--block--8e9b21b6-- cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd--block--14f68253-- b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd--block--2b475461-- a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block--f08c7e71-- ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd-- block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block-- c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block-- c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd-- block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block-- a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd-- block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block--ff12d109-- dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block--b3e38c3d-- e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd-- db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e-- a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348-- fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd-- db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1-- e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5-- ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827-- ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498-- d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# "
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
On 19/03/2025 11:31, Torkil Svensgaard wrote:
On 19/03/2025 08:33, Torkil Svensgaard wrote:
Hi
I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL.
The orchaetrator now fails to create any more OSDs due to:
" /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled "
So I kept adding more HDDs since that has to be done anyway and noticed that the error changed:
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 212.92 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 186.30 GB can be fulfilled
So it looks like ceph-volume is doing some sort of total calculation with the drives and devices passed and thinks there is too little room for some reason.
Workaround seems to be adding the drives one at a time.
Or not, that only worked for a few drives, until every NVMe had 5 db partitions. We can't get the math to fit exactly, but we have this: " nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 │ 253:13 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--d34e427d--749f--460f--bc15--db5ab3900a8e │ 253:39 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--2342ed1b--8eec--4fc6--9176--bf5a149d3c30 │ 253:41 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--e305cdcf--4a1a--435e--9d61--65c672b3ca5b 253:43 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm " The two 1.5T NVMe do not have room for any more 270G db partitions, but the two 2.9T ones have plenty of room. The error is similar to what I had originally, so I think ceph-volume is simply trying to use one of the small NVMes and not the bigger ones with free space. " 2025-03-19T11:19:00.031007+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd " Mvh. Torkil
Mvh.
Torkil
It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed.
Suggestions?
" 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config / var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd "
" [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd-- block--76e673e9-- d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block-- b7330837-- b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd-- block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd-- block--8a0c4a74-- fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block-- c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd-- block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd-- block--618f7582-- fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd-- block--8e9b21b6-- cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd-- block--14f68253-- b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd-- block--2b475461-- a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block-- f08c7e71-- ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd-- block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block-- c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block-- c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd-- block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block-- a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd-- block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block-- ff12d109-- dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block-- b3e38c3d-- e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd-- db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e-- a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348-- fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd-- db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1-- e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5-- ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827-- ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498-- d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# "
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
Hi Torkil, I feel like this is some kind of corner case with DB devices of different sizes. I'm not really surprised that ceph-volume can't handle that as you would expect. Maybe one of the devs can chime in here. Did you eventually manage to deploy all the OSDs? Zitat von Torkil Svensgaard <torkil@drcmr.dk>:
On 19/03/2025 11:31, Torkil Svensgaard wrote:
On 19/03/2025 08:33, Torkil Svensgaard wrote:
Hi
I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL.
The orchaetrator now fails to create any more OSDs due to:
" /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled "
So I kept adding more HDDs since that has to be done anyway and noticed that the error changed:
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 212.92 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 186.30 GB can be fulfilled
So it looks like ceph-volume is doing some sort of total calculation with the drives and devices passed and thinks there is too little room for some reason.
Workaround seems to be adding the drives one at a time.
Or not, that only worked for a few drives, until every NVMe had 5 db partitions. We can't get the math to fit exactly, but we have this:
" nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 │ 253:13 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--d34e427d--749f--460f--bc15--db5ab3900a8e │ 253:39 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--2342ed1b--8eec--4fc6--9176--bf5a149d3c30 │ 253:41 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--e305cdcf--4a1a--435e--9d61--65c672b3ca5b 253:43 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm "
The two 1.5T NVMe do not have room for any more 270G db partitions, but the two 2.9T ones have plenty of room.
The error is similar to what I had originally, so I think ceph-volume is simply trying to use one of the small NVMes and not the bigger ones with free space.
" 2025-03-19T11:19:00.031007+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd
"
Mvh.
Torkil
Mvh.
Torkil
It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed.
Suggestions?
" 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config / var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd "
" [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd-- block--76e673e9-- d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block-- b7330837-- b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd-- block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd-- block--8a0c4a74-- fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block-- c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd-- block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd-- block--618f7582-- fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd-- block--8e9b21b6-- cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd-- block--14f68253-- b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd-- block--2b475461-- a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block-- f08c7e71-- ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd-- block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block-- c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block-- c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd-- block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block-- a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd-- block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block-- ff12d109-- dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block-- b3e38c3d-- e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd-- db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e-- a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348-- fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd-- db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1-- e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5-- ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827-- ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498-- d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# "
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 24-03-2025 08:54, Eugen Block wrote:
Hi Torkil,
Hi Eugen
I feel like this is some kind of corner case with DB devices of different sizes. I'm not really surprised that ceph-volume can't handle that as you would expect. Maybe one of the devs can chime in here. Did you eventually manage to deploy all the OSDs?
We are now unable to deploy any more OSDs. We've tried a number of things but it fails even with just one HDD and any single one of the NVMe's in the service spec, citing that none of the passed fast devices are available. Every LVM command we can think of states there's plenty of room. It looks to me like 2 issues, one being ceph-volume choking on our multiple different NVMe configuration and something else which blocks the NVMes. I've filed a ticket for this: https://tracker.ceph.com/issues/70652 " ... nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:14 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:17 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:18 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:22 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:23 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:24 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 │ 253:26 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--70571884--5e52--45d5--9517--43a4329dec98 │ 253:28 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--66e50d61--9bc1--46f5--8a60--83272c98a875 │ 253:29 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--bed0e59b--66d7--4e81--ad7c--dff430690f8e 253:31 0 270.1G 0 lvm nvme1n1 259:6 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:37 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:39 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:41 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:42 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:43 0 270.1G 0 lvm nvme3n1 259:7 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:3 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 │ 253:4 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--d34e427d--749f--460f--bc15--db5ab3900a8e │ 253:5 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--2342ed1b--8eec--4fc6--9176--bf5a149d3c30 │ 253:6 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--e305cdcf--4a1a--435e--9d61--65c672b3ca5b 253:8 0 270.1G 0 lvm [root@franky ~]# vgs VG #PV #LV #SN Attr VSize VFree ceph-20c0fa86-7668-4139-bf39-f7bb8b2e1623 1 5 0 wz--n- <1.46t 140.00g ceph-9fc0a64d-9ab8-4b12-9a8f-6a48e6c95211 1 4 0 wz--n- 2.91t <1.86t ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 1 8 0 wz--n- 2.91t 820.16g ceph-f1df0c25-920d-4d13-ba48-ad56302a4099 1 5 0 wz--n- <1.46t 140.00g ... [root@franky ~]# pvs PV VG Fmt Attr PSize PFree /dev/nvme0n1 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 lvm2 a-- 2.91t 820.16g /dev/nvme1n1 ceph-f1df0c25-920d-4d13-ba48-ad56302a4099 lvm2 a-- <1.46t 140.00g /dev/nvme2n1 ceph-9fc0a64d-9ab8-4b12-9a8f-6a48e6c95211 lvm2 a-- 2.91t <1.86t /dev/nvme3n1 ceph-20c0fa86-7668-4139-bf39-f7bb8b2e1623 lvm2 a-- <1.46t 140.00g ... [root@franky ~]# lvs | grep db | grep -v block osd-db-2342ed1b-8eec-4fc6-9176-bf5a149d3c30 ceph-20c0fa86-7668-4139-bf39-f7bb8b2e1623 -wi-ao---- 270.08g osd-db-3f4c63c1-e23c-4390-a034-54d4a224b2a2 ceph-20c0fa86-7668-4139-bf39-f7bb8b2e1623 -wi-ao---- 270.08g osd-db-a45b02f5-ad6d-453d-91bc-8a52f1bfa533 ceph-20c0fa86-7668-4139-bf39-f7bb8b2e1623 -wi-ao---- 270.08g osd-db-d34e427d-749f-460f-bc15-db5ab3900a8e ceph-20c0fa86-7668-4139-bf39-f7bb8b2e1623 -wi-ao---- 270.08g osd-db-e305cdcf-4a1a-435e-9d61-65c672b3ca5b ceph-20c0fa86-7668-4139-bf39-f7bb8b2e1623 -wi-ao---- 270.08g osd-db-1ddcc845-799a-4d0d-96f1-90078e2cf0cf ceph-9fc0a64d-9ab8-4b12-9a8f-6a48e6c95211 -wi-ao---- 270.08g osd-db-d28b3fa5-5ace-49a1-ad3d-fc4d14f1b8db ceph-9fc0a64d-9ab8-4b12-9a8f-6a48e6c95211 -wi-ao---- 270.08g osd-db-d4f3c31f-2327-4662-9517-f86dbe35c510 ceph-9fc0a64d-9ab8-4b12-9a8f-6a48e6c95211 -wi-ao---- 270.08g osd-db-eebae904-4ae7-4550-8c3a-0af3ef1fec1c ceph-9fc0a64d-9ab8-4b12-9a8f-6a48e6c95211 -wi-ao---- 270.08g osd-db-12f47f4e-a197-4ecf-a021-94b135039661 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-66e50d61-9bc1-46f5-8a60-83272c98a875 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-70571884-5e52-45d5-9517-43a4329dec98 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-79ce0245-338a-432c-92dd-1437dcaf3917 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-bed0e59b-66d7-4e81-ad7c-dff430690f8e ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-da108348-fc04-4e89-823d-5ebdf26e0408 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-e4e2b277-03c3-47aa-bf32-12f63faee4e5 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-ee274757-51d8-48e3-a41e-b2b321da7170 ceph-c2aeb797-fc4a-4054-9d12-7c5550ac1641 -wi-ao---- 270.08g osd-db-36f82df5-953b-450a-b9d7-5e2ba334a0e7 ceph-f1df0c25-920d-4d13-ba48-ad56302a4099 -wi-ao---- 270.08g osd-db-5468cc56-54aa-4965-8a32-cf4d6b29fb3a ceph-f1df0c25-920d-4d13-ba48-ad56302a4099 -wi-ao---- 270.08g osd-db-6d4941f5-96d6-4e67-81f5-77780e1a3ab0 ceph-f1df0c25-920d-4d13-ba48-ad56302a4099 -wi-ao---- 270.08g osd-db-857ec827-ff63-43d4-a4e8-43681ad8229b ceph-f1df0c25-920d-4d13-ba48-ad56302a4099 -wi-ao---- 270.08g osd-db-b370a498-d3b9-4ddd-b752-ab95e86bc027 ceph-f1df0c25-920d-4d13-ba48-ad56302a4099 -wi-ao---- 270.08g ... " Mvh. Torkil
Zitat von Torkil Svensgaard <torkil@drcmr.dk>:
On 19/03/2025 11:31, Torkil Svensgaard wrote:
On 19/03/2025 08:33, Torkil Svensgaard wrote:
Hi
I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL.
The orchaetrator now fails to create any more OSDs due to:
" /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled "
So I kept adding more HDDs since that has to be done anyway and noticed that the error changed:
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 212.92 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 186.30 GB can be fulfilled
So it looks like ceph-volume is doing some sort of total calculation with the drives and devices passed and thinks there is too little room for some reason.
Workaround seems to be adding the drives one at a time.
Or not, that only worked for a few drives, until every NVMe had 5 db partitions. We can't get the math to fit exactly, but we have this:
" nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 │ 253:13 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--d34e427d--749f--460f--bc15--db5ab3900a8e │ 253:39 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--2342ed1b--8eec--4fc6--9176--bf5a149d3c30 │ 253:41 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--e305cdcf--4a1a--435e--9d61--65c672b3ca5b 253:43 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm "
The two 1.5T NVMe do not have room for any more 270G db partitions, but the two 2.9T ones have plenty of room.
The error is similar to what I had originally, so I think ceph-volume is simply trying to use one of the small NVMes and not the bigger ones with free space.
" 2025-03-19T11:19:00.031007+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd
"
Mvh.
Torkil
Mvh.
Torkil
It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed.
Suggestions?
" 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config / var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd "
" [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd-- block--76e673e9-- d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block-- b7330837-- b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd-- block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd-- block--8a0c4a74-- fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block-- c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd-- block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd-- block--618f7582-- fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd-- block--8e9b21b6-- cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd-- block--14f68253-- b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd-- block--2b475461-- a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block-- f08c7e71-- ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd-- block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block-- c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block-- c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd-- block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block-- a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd-- block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block-- ff12d109-- dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block-- b3e38c3d-- e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd-- db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e-- a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348-- fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd-- db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1-- e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5-- ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827-- ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498-- d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# "
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
participants (2)
-
Eugen Block
-
Torkil Svensgaard