- ceph-users - lists.ceph.io

Ceph Network Design
by Alexander Leutz 14 Nov '25

14 Nov '25

Hi Ceph Users, I need to understand the Ceph network design and the way it communicates with different partners. Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24 After this I only see this as paramter in the GUI screen, under host / ceph / configuration in the section Configuration Database Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host). Is this correct? I thought, all storage traffic needs to be isolated in my storage network 10.10.x.x - no IPs in the mgmt network for ceph. Or do I have the wrong understanding of the ceph network design? Best regards, Enjoy your weekend, Beste Grüße, Schönes Wochenende, Alexander Concat AG IT Solutions | Berliner Ring 127 - 129 | 64625 Bensheim | www.concat.de alexander.leutz(a)concat.de | Tel.: +49 6251 7026-143 | Mobil: +49 151 18856009 | Fax: +49 6251 7026-356 Hauptsitz Bensheim a. d. Bergstraße | Vorstand Olaf von Heyer Aufsichtsratsvorsitzender Tim Patronik | Amtsgericht Darmstadt (HRB 25326)

4 5

New user with some questions...
by Matthias Riße 14 Nov '25

14 Nov '25

Hey all! I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask. So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph. I'd like to at least be able to take one node at a time out of rotation for maintenance purposes. I'd be using cephadm for the setup, which in my testing works really well. As for the questions: 1. 3x replication is a tough sell IMO. For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant. But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.) 2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool. What about changing k, m, or the plugin type? My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed? 3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups, and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS? So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway. 4. In v20 there seems to be a new ec_optimizations feature for ec pools used in CephFS or RBD. Are those a good idea with this kind of (large) qcow2 image workload on top of CephFS? 5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that. Should I be waiting for it / start a new cluster with v20 now already? Thanks for any insights you can give! Kind regards Matthias Riße -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

4 8

Re: Large OMAP & Bucket Not found
by Joo Jaemin 14 Nov '25

14 Nov '25

I integrated ceph with keystone. bucket name "aaa/bbb" means "<keystone user-id>/<bucket-name>" I can't change the bucket name '/'

2 1

How to increase the performance of rgw
by Szabo, Istvan (Agoda) 14 Nov '25

14 Nov '25

Hi, We have multiple gateways behind haproxy but seems like some user can max out some value in rgw which makes the request super slow like 54 sec for a 7kb object: 2025-11-13T01:16:07.367+0700 7f9bb27fc640 1 beast: 0x7f9d24402710: 1.1.1.1 - user [13/Nov/2025:01:15:12.683 +0700] "PUT /002b92966f0b HTTP/1.1" 200 7754 - "aws-sdk-java/1.12.201 Linux/5.15.0-143-generic OpenJDK_64-Bit_Server_VM/21.0.9+10-LTS java/21.0.9 scala/2.12.18 kotlin/1.3.50 vendor/Eclipse_Adoptium cfg/retry-mode/legacy" - latency=54.683818817s The server itself on the network card/CPU/memory didn't spike, on the backend osds I don't see high latency either so would be cusrious which values should be tuned to increase performance for individual gateway. Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2

Large OMAP & Bucket Not found
by Jaemin Joo 14 Nov '25

14 Nov '25

Hi all, I saw several "Large OMAP Objects" in "ceph -s" And I found bucket "Not found" error about I'm using the bucket, when I executed "rados-admin bucket stats --bucket="755d/abc" e.g.) radosgw-admin bucket stats --bucket="755d/abc" error getting bucket stats bucket=abc ret=-2 Failure (2) no such file or directory I guess that there is the index mapping problem after "reshard" fails. I tried "radosgw-admin bucket check --bucket=755d/abc" --fix". and "radosgw-admin bilog trim --bucket="bucket_name" --bucket-id="bucket_id" but it doesn't work. Is there another solution or diagnostic way?

2 1

ceph-volume /dev/mapper not found even if it exists
by Francesco Di Nucci 13 Nov '25

13 Nov '25

Hi, I have provisioned a Ceph cluster with cephadm and I'm having issues with "ceph-volume inventory" on OSD nodes. It fails with errors like # ceph-volume inventory --> RuntimeError: /dev/mapper/ceph--009df43a--5864--43ca--9507--59a9c1207e08-osd--block--410f9c5c--c83b--4208--9060--fffd4829ae96 not found. Even though that device exists under /dev/mapper I've tried rebooting and node creation with "vgscan --mknodes" but the result is still the same, but nothing. Has anyone encountered the same errors? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

3 4

Are osd-secondary and osd-tertiary copys parallel writes?
by jsterr＠proxforge.de 12 Nov '25

12 Nov '25

Hello Ceph-Users, I could not find a information about this in the docs: Are osd-secondary and osd-tertiary parallel writes? Thanks for the help. Greetings -- Jonas Sterr

3 2

Ceph cluster design advice
by Gustavo Garcia Rondina 12 Nov '25

12 Nov '25

Hello Ceph community, We are provisioning a new Ceph cluster and would appreciate design advice. Below is our current hardware: - 13 data servers each with 18 x 24 TB HDD and 2 x 3.5 TB SSD (6 servers in rack A, 7 in rack B) - 3 control servers each with 2 x 960 GB SSD and 4 x 1.9 TB SSD (2 in rack A, 1 in rack B) The planned topology is: - A single CephFS backed by a data pool - The 3 control servers will run MON/MGR/MDS and other daemons. The 13 data servers will host OSDs and some extra MONs as needed. Current idea (constraints: we cannot add hardware) is on each data server we will create a RAID1 from the two 3.5 TB SSDs and carve it into a 500 GB virtual disk for the OS and a 3 TB virtual disk for DB/WAL (Bluestore). Each HDD will be an OSD, giving 18 x 24 TB = 432 TB raw HDD per data node. 1) Is it acceptable to host WAL/DB on a RAID1 virtual disk made from the two SSDs, or is that a bad idea for performance/reliability? 2) Is 3 TB for DB/WAL per data node (~167 GB per OSD) likely to be sufficient for 18 x 24 TB OSDs, or should we expect to need more DB space given CephFS metadata/object counts? (We've seen a 2% rule cited, that would be ~8.6 TB per node which is much larger) 3) Alternative: keep WAL/DB on the HDDs (i.e., do not separate), and use the 3 TB RAID1 SSD area plus the control servers' SSDs (4 x 1.9 TB each) as SSD OSDs for the CephFS metadata pool. Would that be a reasonable approach? 4) If we do SSD-backed metadata OSDs with 3x replication for the metadata pool, we were thinking of CRUSH domains: domain1 = 6 data servers in rack A (6 x 3 TB SSD OSDs), domain2 = 7 data servers in rack B (7 x 3 TB), domain3 = 3 control servers (12 x 1.9 TB SSD OSDs). Any pitfalls with that CRUSH layout? Any suggestions would be highly appreciated. Thank you, Gustavo

2 5

pools nearfull (because cephfs is lazy in releasing space for deleted files ?)
by Massimo Sgaravatto 12 Nov '25

12 Nov '25

Dear all We have a portion of a Cephfs file system that maps to a ceph pool called cephfs_data_ssd. If I perform a "du -sh" on this portion of the file system, I see that the value matches the "STORED" field of the "ceph df" output for the cephfs_data_ssd pool. So far so good. I set a quota of 4.5 TB for this file system area. During the weekend, this pool (and other pools of the same device class) became nearfull. A "ceph df" showed that the problem was indeed in the the cephfs_data_ssd pool, with a reported usage of 7 TiB of data (21 TiB in replica 3): cephfs_data_ssd 62 32 7.1 TiB 2.02M 21 TiB 89.06 898 GiB This sounds strange to me because I set a quota of 4.5 TB in that area, and because a "du -sh" of the relevant directory showed a usage of 600 GB. When I lowered the disk quota from 4.5 TB to 600 GB, the jobs writing in that area failed (because of disk quota exceeded) and after a while the space was released. The only explanation I can think of is that, as far as I understand, cephfs can take a while to release the space for deleted files (https://docs.ceph.com/en/reef/dev/delayed-delete/) This would also be consistent with the fact that it looks like some jobs were performing a lot of writes and deletions (they kept writing a ~ 5GB checkpoint file, and deleting the previous one after each iteration). How can I understand from the log files if this was indeed the problem ? Or do you have some other possible explanations for this problem ? And, most important, how can I prevent scenarios such as this one ? Thanks, Massimo

3 8

OSD idname already in use
by Francesco Di Nucci 10 Nov '25

10 Nov '25

Hi all, I'm deploying a Ceph cluster with cephadm and I'm having some issues with OSD deploy - they fail with errors like "/bin/podman: stderr stderr: WARNING: adding device /dev/sde with idname naa.5000c500940453db which is already used for /dev/sds", how should I prevent this? (Might it be related to usage of storage arrays?) A sample log is below Related to this - When monitoring logs with "ceph -W cephadm", the operations are retried continously, is "ceph mgr module disable cephadm" enough to disable them while debugging? How can I list the operations orch is trying to apply and in case stop/delete them? SAMPLE LOG 2025-11-04T15:11:13.632450+0000 mgr.my-monmgr01.dospbf [ERR] Failed to apply osd.osd02 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: osd02 service_name: osd.osd02 placement: hosts: - my-osd02.example.com spec: data_devices: size: '4TB:' filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd /bin/podman: stderr --> passed data devices: 20 physical, 0 LVM /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 6065211e-ba14-4c47-b786-0c4ab3dd6f12 /bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365 /dev/sde /bin/podman: stderr stderr: WARNING: adding device /dev/sde with idname naa.5000c500866fdcdb which is already used for /dev/sdq. /bin/podman: stderr stdout: Physical volume "/dev/sde" successfully created. /bin/podman: stderr stdout: Volume group "ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365" successfully created /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.17 --yes-i-really-mean-it /bin/podman: stderr stderr: purged osd.17 /bin/podman: stderr --> No OSD identified by "17" was found among LVM-based OSDs. /bin/podman: stderr --> Proceeding to check RAW-based OSDs. /bin/podman: stderr No OSD were found. Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in <module> File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 409, in _infer_config File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 437, in _infer_image File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in create_from_spec_one ret_msg = await self.create_single_host( File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd /bin/podman: stderr --> passed data devices: 20 physical, 0 LVM /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 6065211e-ba14-4c47-b786-0c4ab3dd6f12 /bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365 /dev/sde /bin/podman: stderr stderr: WARNING: adding device /dev/sde with idname naa.5000c500866fdcdb which is already used for /dev/sdq. /bin/podman: stderr stdout: Physical volume "/dev/sde" successfully created. /bin/podman: stderr stdout: Volume group "ceph-e90f3c54-857f-41ef-9a2e-5635c64c4365" successfully created /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.17 --yes-i-really-mean-it /bin/podman: stderr stderr: purged osd.17 /bin/podman: stderr --> No OSD identified by "17" was found among LVM-based OSDs. /bin/podman: stderr --> Proceeding to check RAW-based OSDs. /bin/podman: stderr No OSD were found. Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in <module> File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 409, in _infer_config File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 437, in _infer_image File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/call_wrappers.py", line 310, in call_throws RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec -e NODE_NAME=my-osd02.example.com -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd02 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/run/ceph:z -v /var/log/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142:/var/log/ceph:z -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d3209bec-b987-11f0-aaa9-ac1f6b0de142/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5hr75pjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpt9txcizn:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:7c69e59beaeea61ca714e71cb84ff6d5e533db7f1fd84143dd9ba6649a5fd2ec lvm batch --no-auto /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdn /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd 2025-11-04T15:11:30.613025+0000 mgr.my-monmgr01.dospbf [INF] Detected new or changed devices on my-osd02.example.com Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

2 2