Hi Torkil, I feel like this is some kind of corner case with DB devices of different sizes. I'm not really surprised that ceph-volume can't handle that as you would expect. Maybe one of the devs can chime in here. Did you eventually manage to deploy all the OSDs? Zitat von Torkil Svensgaard <torkil@drcmr.dk>:
On 19/03/2025 11:31, Torkil Svensgaard wrote:
On 19/03/2025 08:33, Torkil Svensgaard wrote:
Hi
I am adding HDDs to a replacement server which will fit 34 HDDs and 2 SATA SSDs, and has 4 NVMe devices for DB/WAL.
The orchaetrator now fails to create any more OSDs due to:
" /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled "
So I kept adding more HDDs since that has to be done anyway and noticed that the error changed:
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 212.92 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 186.30 GB can be fulfilled
So it looks like ceph-volume is doing some sort of total calculation with the drives and devices passed and thinks there is too little room for some reason.
Workaround seems to be adding the drives one at a time.
Or not, that only worked for a few drives, until every NVMe had 5 db partitions. We can't get the math to fit exactly, but we have this:
" nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533 │ 253:13 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--d34e427d--749f--460f--bc15--db5ab3900a8e │ 253:39 0 270.1G 0 lvm ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--2342ed1b--8eec--4fc6--9176--bf5a149d3c30 │ 253:41 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--e305cdcf--4a1a--435e--9d61--65c672b3ca5b 253:43 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm "
The two 1.5T NVMe do not have room for any more 270G db partitions, but the two 2.9T ones have plenty of room.
The error is similar to what I had originally, so I think ceph-volume is simply trying to use one of the small NVMes and not the bigger ones with free space.
" 2025-03-19T11:19:00.031007+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd
"
Mvh.
Torkil
Mvh.
Torkil
It looks to me like 1 of the 4 NVMe devices is indeed too full to fit another DB/WAL partition but ceph-volume is being passed all 4 devices so it should just pick another? Also, it has been creating DB/WAL partitions across all 4 devices up til now, so it's not like it only looks at the first device passed.
Suggestions?
" 2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected new or changed devices on franky 2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to apply osd.slow spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 1000G:7000G filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config / var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop- signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd /usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM /usr/bin/podman: stderr --> relative data size: 1.0 /usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM /usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size, but only 248.40 GB can be fulfilled Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module> File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/ cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host -- stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host -- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk -- init -e CONTAINER_IMAGE=quay.io/ceph/ ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd "
" [root@franky ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2.7T 0 disk └─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd-- block--76e673e9-- d0c0--46bb--8c3c--e666ac5cef3b 253:2 0 2.7T 0 lvm sdb 8:16 0 223.6G 0 disk ├─sdb1 8:17 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdb2 8:18 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdb3 8:19 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 1000M 0 part │ └─md0 9:0 0 999M 0 raid1 /boot ├─sdc2 8:34 0 512M 0 part │ └─md126 9:126 0 511.9M 0 raid1 /boot/efi └─sdc3 8:35 0 222.1G 0 part └─md127 9:127 0 222G 0 raid1 ├─rhel-root 253:0 0 221G 0 lvm /var/lib/containers/storage/overlay │ / └─rhel-swap 253:1 0 1G 0 lvm sdd 8:48 0 1.8T 0 disk └─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block-- b7330837-- b986--46d7--9e28--57db65945098 253:4 0 1.8T 0 lvm sde 8:64 0 2.7T 0 disk └─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd-- block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3 253:6 0 2.7T 0 lvm sdf 8:80 0 1.8T 0 disk └─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd-- block--8a0c4a74-- fedc--46b3--b2a8--d60fd18a37c1 253:8 0 1.8T 0 lvm sdg 8:96 0 2.7T 0 disk └─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block-- c96afbde--9c71--408f--a961--68c6d14a701f 253:12 0 2.7T 0 lvm sdh 8:112 0 16.4T 0 disk └─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd-- block--29747e0e--9c71--44e6--b750--93a7878977ee 253:14 0 16.4T 0 lvm sdi 8:128 0 16.4T 0 disk └─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd-- block--618f7582-- fce0--41f6--aad8--6d0231ef303a 253:16 0 16.4T 0 lvm sdj 8:144 0 1.8T 0 disk └─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd-- block--8e9b21b6-- cc39--4c7b--b5f8--9e83e33fa146 253:18 0 1.8T 0 lvm sdk 8:160 0 447.1G 0 disk └─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd-- block--14f68253-- b370--4300--a319--0c39311a34e1 253:10 0 447.1G 0 lvm sdl 8:176 0 186.3G 0 disk └─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd-- block--2b475461-- a85d--4e7b--a7e2--8ab1c9d14c6e 253:11 0 186.3G 0 lvm sdm 8:192 0 3.6T 0 disk └─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block-- f08c7e71-- ddf5--4939--8f0c--42396de2210b 253:26 0 3.6T 0 lvm sdn 8:208 0 2.7T 0 disk └─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd-- block--40cd9045--587a--4831--95f7--607c019ef862 253:20 0 2.7T 0 lvm sdo 8:224 0 2.7T 0 disk └─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block-- c2087805--441c--422a--b12f--de10a75b7e0b 253:22 0 2.7T 0 lvm sdp 8:240 0 3.6T 0 disk └─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block-- c0ef5f16--4bcd--4390--abf8--260c5913cb14 253:24 0 3.6T 0 lvm sdq 65:0 0 3.6T 0 disk └─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd-- block--8f783ade--6da6--4967--b87c--fb6d1827460f 253:28 0 3.6T 0 lvm sdr 65:16 0 16.4T 0 disk └─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block-- a3dbca80--9cf5--4c98--ad9b--b38803230b1f 253:30 0 16.4T 0 lvm sds 65:32 0 3.6T 0 disk └─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd-- block--032fc777--69d9--4f39--8fe6--485e0959ce66 253:32 0 3.6T 0 lvm sdt 65:48 0 16.4T 0 disk └─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block-- ff12d109-- dfaa--4608--a5e2--36ec39623f36 253:34 0 16.4T 0 lvm sdu 65:64 0 3.6T 0 disk └─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block-- b3e38c3d-- e3c0--49c2--9e96--93f9e4c909d4 253:36 0 3.6T 0 lvm sdv 65:80 0 3.6T 0 disk sdw 65:96 0 5.5T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 16.4T 0 disk nvme2n1 259:2 0 2.9T 0 disk ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db │ 253:7 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d │ 253:15 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd-- db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf │ 253:21 0 270.1G 0 lvm ├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- d4f3c31f--2327--4662--9517--f86dbe35c510 │ 253:27 0 270.1G 0 lvm └─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db-- eebae904--4ae7--4550--8c3a--0af3ef1fec1c 253:29 0 270.1G 0 lvm nvme0n1 259:3 0 2.9T 0 disk ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e-- a197--4ecf--a021--94b135039661 │ 253:3 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348-- fc04--4e89--823d--5ebdf26e0408 │ 253:19 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- ee274757--51d8--48e3--a41e--b2b321da7170 │ 253:25 0 270.1G 0 lvm ├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd-- db--79ce0245--338a--432c--92dd--1437dcaf3917 │ 253:35 0 270.1G 0 lvm └─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db-- e4e2b277--03c3--47aa--bf32--12f63faee4e5 253:37 0 270.1G 0 lvm nvme3n1 259:6 0 1.5T 0 disk ├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1-- e23c--4390--a034--54d4a224b2a2 │ 253:5 0 270.1G 0 lvm └─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5-- ad6d--453d--91bc--8a52f1bfa533 253:13 0 270.1G 0 lvm nvme1n1 259:7 0 1.5T 0 disk ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--36f82df5--953b--450a--b9d7--5e2ba334a0e7 │ 253:9 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827-- ff63--43d4--a4e8--43681ad8229b │ 253:17 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498-- d3b9--4ddd--b752--ab95e86bc027 │ 253:23 0 270.1G 0 lvm ├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a │ 253:31 0 270.1G 0 lvm └─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd-- db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0 253:33 0 270.1G 0 lvm [root@franky ~]# "
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io