- ceph-users - lists.ceph.io

Re: dashboard dial tcp connection refused
by Nizamudeen A 08 Dec '25

08 Dec '25

Hi, Was that cluster ever configured with ssl before? Maybe the initial alertmanager webhook config had the ssl enabled url added to it and its now failing to detect it. Have you tried to redeploy the alertmanager to probably refresh its config? It might be worth checking the alertmanager configuration to see what urls its pointing to. Also it might help more if you can share some info about your cluster like the version it is in right now. I remember there were some bugs around the standby server and alertmanager configuration but they seem to have been fixed in some version of ceph. Regards, Nizam On Mon, Dec 8, 2025 at 5:22 PM lejeczek via ceph-users <ceph-users(a)ceph.io> wrote: > Hi guys. > > I've no _ssl_ used by _dashboard_ > > -> $ ceph mgr services > { > "dashboard": "http://10.1.1.63:8080/", > "prometheus": "http://10.1.1.63:9283/" > } > -> $ ceph config get mgr mgr/dashboard/ssl > false > > and the 'standby' servers logs: > ... > Dec 08 11:33:39 > ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: > ts=2025-12-08T11:33:39.781Z caller=dispatch.go:353 > level=error component=dispatcher msg="Notify for alerts > failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify > retry canceled after 7 attempts: Post \"<redacted>\": dial > tcp 10.1.1.63:8443: connect: connection refused; > ceph-dashboard/webhook[1]: notify retry canceled after 7 > attempts: Post \"<redacted>\": dial tcp 10.1.1.61:8443: > connect: connection refused" > Dec 08 11:33:39 > ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: > ts=2025-12-08T11:33:39.781Z caller=notify.go:848 level=warn > component=dispatcher receiver=ceph-dashboard > integration=webhook[0] > aggrGroup="{}/{}:{alertname=\"CephHealthWarning\"}" > msg="Notify attempt failed, will retry later" attempts=1 > err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: > connection refused" > ... > with the snippet of the logs above _ceph_ says only: > -> $ ceph health detail > HEALTH_WARN mon podster1 is low on available space > [WRN] MON_DISK_LOW: mon podster1 is low on available space > mon.podster1 has 12% avail > > nothing listens on 8443 anywhere, which is as it should be - > why standby server wants that - would anybody know? > 10.1.1.61 is standby's own IP. > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Size or IOPS for DB+WAL?
by Andrew Klaassen 08 Dec '25

08 Dec '25

Hi, I'm looking at converting an old Nutanix+VMware cluster to Ceph+Proxmox. It'll host a mix of a couple of hundred VMs - web servers, Active Directory, Elasticsearch clusters, postgres and mysql databases, software deployment servers, log forwarders, some stuff that I don't even know what it does - basically a grab bag of workloads. It'll be a 9 node cluster. CPU+RAM is a mix from 12 core Intel Haswell with 512GB to 18 core Intel Ice Lake with 1TB. NICs will be two dual-port 10GbE (total four ports) per node. Disks per node will be a mix of: HDD: 2x12TB + 1x8TB SSD: 2x3.84TB HDD: 1x12TB + 2x6TB + 1x8TB SSD: 1x7.68TB This will leave me with one slot in each node for a DB+WAL drive. Is either of these a clear winner? Smaller and faster, or bigger and slower? Micron 7450 MTFDKCB960TFR - 960GB - 85K random write IOPS - 2% of raw storage KIOXIA KCD6XLUL3T84 - 3.84TB - 60K random write IOPS - 8% of raw storage If there is a clear winner, are there any setup or tuning options I should know about to take full advantage of them? Thanks. Andrew

1 0

smartctl issues on squid (ubuntu 24.04)
by Nathan Butcher 08 Dec '25

08 Dec '25

Hi, We've been running squid 19.2.3 on Ubuntu 24.04.3 LTS We encountered some issues where the Ceph dashboard was unable to get SMART data from OSDs under the Device Health tab. At first we were getting -22 errors from that tab suggesting that smartctl was under version 7. That wasn't the case at all. System logging determined that there was an apparmor profile called unix-chkpwd which was producing errors as ceph tried to authenticate into the pod to run smartctl. We had to disable that profile in order to get that to work. We had to set the profile to complain mode using aa-complain (and even then Ubuntu had some bugs getting that command to work) Our OSD node has a HPE smart array card on it which ensures the kernel sees all drives on it as scsi devices, whether in HBA or RAID mode. From within the OSD pod, it seems that smartctl isn't able to autodetect the type of drives it is trying to access, possibly because attributed from udev and getting blocked somehow. We needed the "-d scsi" command to work, but the ceph code doesn't allow any customisation or flexibility in how smartctl is called. In the end I had to write a wrapper script to inject the -d scsi command around smartctl in order to get it to work within the confines of the OSD pod. Then the wrapper had to be distributed across every OSD pod. It's a real messy hack. So a couple of things: How can we control how the OSD pods supposed to present allowing smartctl to work but without compromising on security? Is this just an ubuntu apparmor thing that ubuntu should deal with? I can see that there is a separate apparmor profile applied to pods. I'm sure we're not the only ones who have hit this roadblock. In any case, I hope someone else having the same issues find these workarounds useful Nathan Butcher Senior Systems Engineer 1300 852 110 Chat on Teams nbutcher(a)ct.com.au We are passionate about service excellence. If you would like to contact my Team Leader for any reason, their name is Tim Sandy, email address is tsandy(a)ct.com.au and phone number is (03) 9998 6377. We’re 100% Carbon Neutral and ISO 14001 certified. Check out our Environmental Sustainability pledge here. Adelaide | Brisbane | Canberra | Darwin | Hobart | Melbourne | Perth | Sydney Disclaimer This email message is confidential and may be privileged. Unauthorised use, copying or distribution of any part of this email including attachments is prohibited. If you are not the intended recipient, please forward the email to infosec(a)ct.com.au and delete the original. Any advice contained in this email is of a general nature only and has been prepared without considering your relevant personal circumstances. To the extent permitted by law, we exclude (and where the law does not permit exclusion, limit to the extent permitted by law) all liability for any direct, indirect and consequential losses, damages and expenses incurred in any way (including but not limited to that arising from negligence), connected with any use or access to or any reliance on information contained in this email or any attachments. Centorrino Technologies Pty Ltd (ABN: 83 606 931 524). All Rights Reserved.

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 08 Dec '25

08 Dec '25

This test branch was built loner then 2.5 weeks ago I have to rebuild it first and then will schedule krbd On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > > > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > > > Hi Yuri, > > > > > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > > > > > Note: this is an attempt to test and publish post EOL reef point > release > > > > > > > > Details of this release are summarized here: > > > > > > > > https://tracker.ceph.com/issues/73906#note-1 > > > > Release Notes - TBD > > > > LRC upgrade - N/A > > > > > > > > distro => jammy focal centos9 windows bookworm > > > > > > > > Seeking approvals/reviews for: > > > > > > > > smoke - Laura approved? > > > > > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > approved? > > > > (rados is also running on build 2 == build 1 + > > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > > > > > rgw - Adam E approved? > > > > > > > > fs - Venky approved? > > > > > > There are lot many failures in the fs suite resembling > > > > > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > > > type, bad option ... > > > > > > which is an issue with the testing kernel. Adding Alex who was looking > > > into this issue. > > > > > > As far as the run validation is concerned, it's risky to ignore the > > > large number of failures. We have two options: > > > > > > - fix the testing kernel issue and rerun > > > - rerun with `for-linus` branch (with `-k for-linus` in the > > > teuthology-suite schedule command) > > > > > > The latter would be using an older kernel branch, but I think it > > > should suffice for this case. > > > > Hi Venky, > > > > With your report of the fs suite being affected as well, I carved out > > some time to take a look. > > > > I think it's happening because XFS maintainers disabled support for V4 > > filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's > > no longer compiled in by default. The proper fix would be to create an > > image with/move to V5 on-disk format; I have opened [1] to try to > > achieve that. We can also re-enable V4 on-disk format in our kernel > > builds as a workaround. > > > > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > This should be fixed now. Yuri, please schedule a rerun for krbd. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 08 Dec '25

08 Dec '25

On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Yuri, > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > Note: this is an attempt to test and publish post EOL reef point release > > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/73906#note-1 > > Release Notes - TBD > > LRC upgrade - N/A > > > > distro => jammy focal centos9 windows bookworm > > > > Seeking approvals/reviews for: > > > > smoke - Laura approved? > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? > > (rados is also running on build 2 == build 1 + > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > rgw - Adam E approved? > > > > fs - Venky approved? > > There are lot many failures in the fs suite resembling > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > type, bad option ... > > which is an issue with the testing kernel. Adding Alex who was looking > into this issue. > > As far as the run validation is concerned, it's risky to ignore the > large number of failures. We have two options: > > - fix the testing kernel issue and rerun > - rerun with `for-linus` branch (with `-k for-linus` in the > teuthology-suite schedule command) > > The latter would be using an older kernel branch, but I think it > should suffice for this case. Hi Venky, With your report of the fs suite being affected as well, I carved out some time to take a look. I think it's happening because XFS maintainers disabled support for V4 filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's no longer compiled in by default. The proper fix would be to create an image with/move to V5 on-disk format; I have opened [1] to try to achieve that. We can also re-enable V4 on-disk format in our kernel builds as a workaround. [1] https://github.com/ceph/ceph-cm-ansible/pull/799 Thanks, Ilya

1 1

Re: SMB on ARM
by John Mulligan 08 Dec '25

08 Dec '25

On Monday, December 8, 2025 10:02:56 AM Eastern Standard Time Daniel Brown via ceph-users wrote: > Upgraded to tentacle over the weekend - mostly a smooth process. Had one > node that was a bit pokey and had to manually download the ceph container > image, but other than that, all good. > > I did try to setup SMB — unfortunately, my cluster is running on Arm64 > machines, and the SMB setup appears to be trying to start an Amd64 > container. Didn’t see a lot of logging, but after some digging around under > /var/lib/ceph/[my cluster id here]/… > > > $ cat unit.image > > quay.io/samba.org/samba-server:ceph20-centos-amd64 > > > Don’t suppose there’s a way to update that for the correct architecture? And > or to document it if it isn’t going to be possible? Thanks for trying the SMB support out. Unfortunately, the samba container images [1] do not support ARM64 at this time. We worked on it a while ago but got pulled off of that onto other tasks and never really resumed working on arm support. I have filed an issue to revisit arm64 support for samba container images on that project. [1] https://github.com/samba-in-kubernetes/samba-container

1 0

Re: SMB on ARM
by Robert Sander 08 Dec '25

08 Dec '25

Hi, Am 08.12.25 um 4:02 PM schrieb Daniel Brown via ceph-users: > > I did try to setup SMB The SMB service and the SMB MGR module are far from being production ready. I only get errors when trying to use them on AMD64. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Anthony D'Atri 08 Dec '25

08 Dec '25

> On Dec 8, 2025, at 4:07 AM, Olivier Delcourt via ceph-users <ceph-users(a)ceph.io> wrote: > > > >> Le 8 déc. 2025 à 09:36, Robert Sander via ceph-users <ceph-users(a)ceph.io> a écrit : >> >> Hi, >> >> Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: >> >>> since migrating to Squid (19.2.3), I get the following health warning: >>> [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) >>> osd.25 0.806301 How large is this OSD? >>> However while this is a warning, there seems to be no documented way of >>> fixing it. >>> What do I do with this warning / how to defrag the bluestore filesystem? >> >> Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… >> >> My best bet is to just recreate the OSD. > > Before that, what about trying to empty the OSD ( ceph osd reweight XXX 0) and when empty, put it back to business ( ceph osd reweight XXX 1) ? I suspect that would do the trick, but would suggest instead "crush reweigh" so that the balancer doesn't get confused.

1 0

SMB on ARM
by Daniel Brown 08 Dec '25

08 Dec '25

Upgraded to tentacle over the weekend - mostly a smooth process. Had one node that was a bit pokey and had to manually download the ceph container image, but other than that, all good. I did try to setup SMB — unfortunately, my cluster is running on Arm64 machines, and the SMB setup appears to be trying to start an Amd64 container. Didn’t see a lot of logging, but after some digging around under /var/lib/ceph/[my cluster id here]/… $ cat unit.image quay.io/samba.org/samba-server:ceph20-centos-amd64 Don’t suppose there’s a way to update that for the correct architecture? And or to document it if it isn’t going to be possible?

1 0

dashboard dial tcp connection refused
by lejeczek 08 Dec '25

08 Dec '25

Hi guys. I've no _ssl_ used by _dashboard_ -> $ ceph mgr services { "dashboard": "http://10.1.1.63:8080/", "prometheus": "http://10.1.1.63:9283/" } -> $ ceph config get mgr mgr/dashboard/ssl false and the 'standby' servers logs: ... Dec 08 11:33:39 ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: ts=2025-12-08T11:33:39.781Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"<redacted>\": dial tcp 10.1.1.61:8443: connect: connection refused" Dec 08 11:33:39 ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: ts=2025-12-08T11:33:39.781Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephHealthWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" ... with the snippet of the logs above _ceph_ says only: -> $ ceph health detail HEALTH_WARN mon podster1 is low on available space [WRN] MON_DISK_LOW: mon podster1 is low on available space mon.podster1 has 12% avail nothing listens on 8443 anywhere, which is as it should be - why standby server wants that - would anybody know? 10.1.1.61 is standby's own IP. many thanks, L.

1 0