- ceph-users - lists.ceph.io

Cephadm using node exporter container from previous installation
by gagan tiwari 05 Aug '25

05 Aug '25

HI Guys, Good Morning! I have re-deployed the ceph cluster using cephadm because of some issues with osds. Everything is working fine now with the new cluster except that Cephadm is still trying to deploy node_exporter daemon on one of the monitor hosts using the old container which no longer exists. node_exporter is running fine on that host in the new container. But ceph is still trying to deploy it on that host using an old container and since it couldn't find that podman container it throws the warning. I did try to remove that node_exporter service on that node and added it again but no luck. When I reboot that host after its coming ceph status shows "Health OK" but after sometime it starts showing same waring ceph health detail HEALTH_WARN Failed to place 1 daemon(s) [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s) Failed while placing node-exporter.ceph-mon2 on ceph-mon2: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter-ceph-mon2 /usr/bin/podman: stderr Error: no such container ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter-ceph-mon2 Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter.ceph-mon2 /usr/bin/podman: stderr Error: no such container ceph-1aa23d18-6eb0-11f0-823c-cc28aa448c14-node-exporter.ceph-mon2 Deploy daemon node-exporter.ceph-mon2 ... Verifying port 0.0.0.0:9100 ... Cannot bind to IP 0.0.0.0 port 9100: [Errno 98] Address already in use ERROR: TCP Port(s) '0.0.0.0:9100' required for node-exporter already in use Please let me know how to fix it. Thanks, Gagan

2 4

v19.2.3 Squid released
by Yuri Weinstein 04 Aug '25

04 Aug '25

We're happy to announce the third backport release in the Squid series. https://ceph.io/en/news/blog/2025/v19-2-3-squid-released/ Notable Changes --------------- * RGW: PutObjectLockConfiguration can now be used to enable S3 Object Lock on an existing versioning-enabled bucket that was not created with Object Lock enabled. * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired. Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867 * RBD: Moving an image that is a member of a group to trash is no longer allowed. `rbd trash mv` command now behaves the same way as `rbd rm` in this scenario. * MGR: MGR's always-on modules/plugins can now be force-disabled. This can be necessary in cases where MGR(s) needs to be prevented from being flooded by the module commands when corresponding Ceph service is down/degraded. * RGW: An authentication bypass vulnerability in STS [CVE-2023-43040] has been fixed. * RGW: S3 policy now enforces ARN-based conditionals. * RGW: Copying an object to itself no longer causes data loss. Potential corruption on ETIMEDOUT (not enabled by default), was also fixed. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: c92aebb279828e9c3c1f5d24613efca272649e62

11 22

Changing the failure domain of an EC cluster still shows old profile
by Niklas Hambüchen 04 Aug '25

04 Aug '25

I made an EC 4+2 cluster with `crush-failure-domain=host`. Later after adding more machines, I changed it from `host` to `datacenter`: ceph osd erasure-code-profile set my_ec_profile_datacenter k=4 m=2 crush-failure-domain=datacenter crush-device-class=hdd ceph osd crush rule create-erasure rule_my_data_ec_datacenter my_ec_profile_datacenter ceph osd pool set my_data_ec crush_rule rule_my_data_ec_datacenter This seems to have worked, and `ceph osd pool get my_data_ec crush_rule` outputs: crush_rule: rule_my_data_ec_datacenter But `ceph osd pool ls detail` still shows pool 3 'my_data_ec' erasure profile my_ec_profile ... with `my_ec_profile` instead of `my_ec_profile_datacenter`. Is this a problem? Who wins, the profile or the crush-rule? If it's not a problem, it is at least confusing; can I fix it somehow? Thanks! Niklas

3 3

to the maintainer/owners of the list - dmar dkim.... Yahoo?
by lejeczek 03 Aug '25

03 Aug '25

Hi guys. If my memory serves me well - being a consumer of quite a few, rhel/fedora related, mailing lists - relatively old issue, namely.. Some policies (dkim, dmarc, more?) Yahoo did change (perhaps other popular mail providers) which resulted is some tweaks to a mailing list are necessary, so mailing list would send senders their own emails. Could it be that this list "suffers" from this problem - I've only subscribed now, I seem not to get my own emails. many thanks, L.

1 0

Performance issues
by Ron Gage 02 Aug '25

02 Aug '25

Hello from Detroit MI: I have been doing some limited benchmarking of a Squid cluster. The arrangement of the cluster: Server Function c01 MGR, MON c02 MGR, MON o01 OSD o02 OSD o03 OSD o04 OSD Each OSD has 2 x NVME disks for Ceph, each at 370 Gig The backing network is as follows: ens18 Gigabit, mon-ip (192.168.0.0/23) regular MTU (1500) ens19 2.5 Gigabit, Cluster Network (10.0.0.0/24) Jumbo MTU (9000) Behind all this is a small ProxMox cluster. All Ceph machines are running on a single node. All Ceph drives are exposed and an NFS mounted NVME drive. All Ceph OSD drives are mounted with no cache and single controller per drive. Networking bridges are all set to either MTU 9000 or MTU 1500 as appropriate. iPerf3 is showing 2.46 Gbit/sec between servers c01 and o01 on the ens19 network. Firewall is off all the way around. OS is CentOS 10. SELinux of disabled. No network enhancements have been performed (increasing send/rcv buffer size, queue length, etc). The concern given all this: rados bench can't exceed 110 MB/s in all tests. In fact if I didn't know better I would swear that the traffic is being either throttled or is somehow routing through a 1Gbit network. The numbers that are returning from rados bench are acting like saturation at Gigabit and not exhibiting any evidence of being on a 2.5 Gbit network. Monitoring at both Ceph and ProxMox consoles confirm the same. Cluster traffic is confirmed to be going out ens19 - tested via tcpdump. Typical command line used for rados bench: rados bench -p s3block 20 write What the heck am I doing wrong here? Ron Gage

4 8

Re: How to improve write latency on R3 pool Images
by Devender Singh 02 Aug '25

02 Aug '25

Hello Anthony I am directly measuring on all rbd images from ceph node using rbd iostat. Any parameter to improve images ? Regards Dev On Fri, 1 Aug 2025 at 8:05 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > How are you measuring this? > > VMs do add an emulation cost on devices. > > > On Aug 1, 2025, at 11:03 PM, Devender Singh <devender(a)netskrt.io> wrote: > > No all’s are ssd and none > > Regards > Dev > > > On Fri, 1 Aug 2025 at 7:58 PM, Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > >> What is this the output of? Is this reading from HDDs? >> >> > On Aug 1, 2025, at 9:01 PM, Devender Singh <devender(a)netskrt.io> wrote: >> > >> > Hello All >> > >> > Using R3 pool with images running ubuntu vms. >> > >> > Having three nodes… >> > >> > I have default settings for ms_asyc threads to 3 and rbd_op_thread to 1 >> > How to improve below latency…mainly write.... >> > >> > # sed 's/^.\{60\}//' a >> > WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT >> > 50/s 0/s 275 KiB/s 0 B/s 17.36 ms 0.00 ns >> > 45/s 4/s 230 KiB/s 56 KiB/s 66.02 ms 6.90 ms >> > 25/s 0/s 146 KiB/s 0 B/s 68.52 ms 0.00 ns >> > 24/s 0/s 548 KiB/s 0 B/s 22.49 ms 0.00 ns >> > 22/s 0/s 114 KiB/s 0 B/s 20.85 ms 0.00 ns >> > 18/s 0/s 204 KiB/s 0 B/s 48.31 ms 0.00 ns >> > 17/s 0/s 182 KiB/s 0 B/s 10.00 ms 0.00 ns >> > 15/s 4/s 25 MiB/s 17 KiB/s 254.61 ms 17.42 ms >> > 12/s 0/s 268 KiB/s 0 B/s 29.67 ms 0.00 ns >> > 12/s 2/s 13 MiB/s 8.8 KiB/s 147.05 ms 2.30 ms >> > 11/s 0/s 77 KiB/s 0 B/s 50.42 ms 0.00 ns >> > 11/s 0/s 74 KiB/s 0 B/s 35.54 ms 0.00 ns >> > 10/s 0/s 101 KiB/s 0 B/s 80.83 ms 0.00 ns >> > 9/s 0/s 65 KiB/s 0 B/s 24.81 ms 0.00 ns >> > 7/s 0/s 37 KiB/s 0 B/s 59.11 ms 0.00 ns >> > 7/s 0/s 33 KiB/s 0 B/s 43.95 ms 0.00 ns >> > 6/s 8/s 31 KiB/s 34 KiB/s 131.78 ms 127.46 us >> > 4/s 0/s 18 KiB/s 0 B/s 56.90 ms 0.00 ns >> > 4/s 0/s 23 KiB/s 0 B/s 74.13 ms 0.00 ns >> > 3/s 0/s 18 KiB/s 0 B/s 38.25 ms 0.00 ns >> > 2/s 0/s 10 KiB/s 0 B/s 26.28 ms 0.00 ns >> > 2/s 0/s 11 KiB/s 0 B/s 49.80 ms 0.00 ns >> > 1/s 0/s 90 KiB/s 0 B/s 43.02 ms 0.00 ns >> > 1/s 0/s 8.8 KiB/s 0 B/s 24.99 ms 0.00 ns >> > 1/s 0/s 10 KiB/s 0 B/s 102.38 ms 0.00 ns >> > 1/s 0/s 15 KiB/s 0 B/s 57.23 ms 0.00 ns >> > 0/s 0/s 7.2 KiB/s 0 B/s 149.90 ms 0.00 ns >> > 0/s 0/s 4 KiB/s 0 B/s 1.58 ms 0.00 ns >> > 0/s 0/s 3.2 KiB/s 0 B/s 1.02 ms 0.00 ns >> > 0/s 0/s 819 B/s 0 B/s 40.45 ms 0.00 ns >> > >> > Regards >> > Dev >> > >> > >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

1 0

How to improve write latency on R3 pool Images
by Devender Singh 02 Aug '25

02 Aug '25

Hello All Using R3 pool with images running ubuntu vms. Having three nodes… I have default settings for ms_asyc threads to 3 and rbd_op_thread to 1 How to improve below latency…mainly write.... # sed 's/^.\{60\}//' a WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT 50/s 0/s 275 KiB/s 0 B/s 17.36 ms 0.00 ns 45/s 4/s 230 KiB/s 56 KiB/s 66.02 ms 6.90 ms 25/s 0/s 146 KiB/s 0 B/s 68.52 ms 0.00 ns 24/s 0/s 548 KiB/s 0 B/s 22.49 ms 0.00 ns 22/s 0/s 114 KiB/s 0 B/s 20.85 ms 0.00 ns 18/s 0/s 204 KiB/s 0 B/s 48.31 ms 0.00 ns 17/s 0/s 182 KiB/s 0 B/s 10.00 ms 0.00 ns 15/s 4/s 25 MiB/s 17 KiB/s 254.61 ms 17.42 ms 12/s 0/s 268 KiB/s 0 B/s 29.67 ms 0.00 ns 12/s 2/s 13 MiB/s 8.8 KiB/s 147.05 ms 2.30 ms 11/s 0/s 77 KiB/s 0 B/s 50.42 ms 0.00 ns 11/s 0/s 74 KiB/s 0 B/s 35.54 ms 0.00 ns 10/s 0/s 101 KiB/s 0 B/s 80.83 ms 0.00 ns 9/s 0/s 65 KiB/s 0 B/s 24.81 ms 0.00 ns 7/s 0/s 37 KiB/s 0 B/s 59.11 ms 0.00 ns 7/s 0/s 33 KiB/s 0 B/s 43.95 ms 0.00 ns 6/s 8/s 31 KiB/s 34 KiB/s 131.78 ms 127.46 us 4/s 0/s 18 KiB/s 0 B/s 56.90 ms 0.00 ns 4/s 0/s 23 KiB/s 0 B/s 74.13 ms 0.00 ns 3/s 0/s 18 KiB/s 0 B/s 38.25 ms 0.00 ns 2/s 0/s 10 KiB/s 0 B/s 26.28 ms 0.00 ns 2/s 0/s 11 KiB/s 0 B/s 49.80 ms 0.00 ns 1/s 0/s 90 KiB/s 0 B/s 43.02 ms 0.00 ns 1/s 0/s 8.8 KiB/s 0 B/s 24.99 ms 0.00 ns 1/s 0/s 10 KiB/s 0 B/s 102.38 ms 0.00 ns 1/s 0/s 15 KiB/s 0 B/s 57.23 ms 0.00 ns 0/s 0/s 7.2 KiB/s 0 B/s 149.90 ms 0.00 ns 0/s 0/s 4 KiB/s 0 B/s 1.58 ms 0.00 ns 0/s 0/s 3.2 KiB/s 0 B/s 1.02 ms 0.00 ns 0/s 0/s 819 B/s 0 B/s 40.45 ms 0.00 ns Regards Dev

1 0

squid 19.2.2 - osd_memory_target_autotune - best practices when host has lots of RAM
by Steven Vacaroaia 01 Aug '25

01 Aug '25

Hi What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ? My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD Should I disable autotune and allocate more RAM? I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD Many thanks Steven

3 5

Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected on all newly added osd disks
by gagan tiwari 01 Aug '25

01 Aug '25

Hi Guys, I am setting up a new ceph cluster. After adding osd devices when I did run *ceph orch device ls* It shows * Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected for all osd devices.* All the OSD devices are new nvme disks. I tried to rm , zap , destroy disks and re-added them but again same messages for all disks. Please let me know how to fix this. Thanks, Gagan

3 5

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8