- ceph-users - lists.ceph.io

ceph squid 19.2.2 - modifyng HDD OSD to use NVME DB
by Steven Vacaroaia 19 Feb '26

19 Feb '26

Hi, I have a 7 nodes cluster deployed using cephadm with a mixt of HDD, SSD and NVME I would like to reconfigure some of the HDD OSDs and put their DB on NVME There are only 6 x18 TB OSDs and there is a dedicate , "empty" 1.6 TB NVME installed for this purpose The cluster is configured as EC 4+2 and is not under heavy use I would appreciate if you could advise what is the best way of achieving that As far as I was able to find, there are these 2 options 1. remove and redeploy OSDs 2. use ceph-volume Option 1 is pretty straight forward but more time consuming For Option2 though I was not able to find any clear instructions The first thing I stumbled one was how to use ceph-volume in a cephadm managed cluster on a host that does not have it ( "cephadm shell --host HOSTNAME -- ceph-volume ls" is complaining about "unrecognized arguments --host") Any help,links,suggestions will be greatly appreciated Steven

1 0

Re: 8443: connect: connection refused
by Eugen Block 18 Feb '26

18 Feb '26

Hi, port 8443 is the default dashboard port on the active MGR. If none of your hosts listens on that port, then your dashboard most likely isn't working. Is the cluster healthy? What does 'ceph mgr services' report (mask sensitive data)? The message you quote is the alertmanager, trying to connect to the dashboard so you would see the alerts in the dashboard (I assume). Zitat von listy via ceph-users <ceph-users(a)ceph.io>: > Hi guys. > > One node logs: > ... > 5.744Z caller=notify.go:848 level=warn component=dispatcher > receiver=ceph-dashboard integration=webhook[0] > aggrGroup="{}/{}:{alertname=\"CephPGBackfillAtRisk\"}" msg="Notify > attempt failed, will retry later" attempts=1 err="Post > \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" > ... > but no node listen on this port and I have: > > -> $ ceph config get mgr mgr/dashboard/ssl > false > > What am I missing or having mis-configured? > > all thoughts are much appreciated, thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Debian version and ceph.
by Eugen Block 18 Feb '26

18 Feb '26

Right, if you want to bootstrap a cephadm cluster, you don't need to worry too much about the package versions on the host. Although we do recommend to keep them somewhat aligned. But there's also an option for the cephadm bootstrap command: --allow-mismatched-release allow bootstrap of ceph that doesn't match this version of cephadm I used that many times on openSUSE hosts where the latest available version was Pacific (there are packages available again with Reef and Squid from openSUSE Leap 16, Tumbleweed probably too, haven't checked). Last week I also bootstrapped a Squid cluster like that on a Debian Trixie where the host package versions are 18.2.7. Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Am 18.02.26 um 5:53 PM schrieb Albert Shih via ceph-users: > >> 2/ Install a Debian 13 and use the Debian ceph pkg. > > This is the way. > > The client code does not need to match the cluster version. > I usually install ceph-common from the distribution. > > Just make sure you have a cephadm binary matching the desired Ceph > version before bootstrapping the cluster. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

8443: connect: connection refused
by listy 18 Feb '26

18 Feb '26

Hi guys. One node logs: ... 5.744Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephPGBackfillAtRisk\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" ... but no node listen on this port and I have: -> $ ceph config get mgr mgr/dashboard/ssl false What am I missing or having mis-configured? all thoughts are much appreciated, thanks, L.

1 0

Re: Debian version and ceph.
by Robert Sander 18 Feb '26

18 Feb '26

Am 18.02.26 um 5:53 PM schrieb Albert Shih via ceph-users: > 2/ Install a Debian 13 and use the Debian ceph pkg. This is the way. The client code does not need to match the cluster version. I usually install ceph-common from the distribution. Just make sure you have a cephadm binary matching the desired Ceph version before bootstrapping the cluster. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Debian version and ceph.
by Albert Shih 18 Feb '26

18 Feb '26

Hi everyone. I need to install a new cluster ceph. New hardware mean of couse new fresh install. Currently I still don't see any package for Debian trixie. In any case the ceph would be install with cephadm with podman, so no huge dependencies with the OS. But still ceph-base ceph-common libcephfs2 python3-ceph python3-ceph-argparse python3-ceph-common python3-cephfs In that case what would be advise to do ? (I want to stay with Debian). I see 1/ Install a Debian 12 and use the official ceph.com pkg. 2/ Install a Debian 13 and use the Debian ceph pkg. What would you do ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 18 févr. 2026 17:44:51 CET

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Konold, Martin 18 Feb '26

18 Feb '26

Am 2026-02-17 20:24, schrieb Robert Lukan via ceph-users: Hi Robert, Are there known risks to "stepping down" PGs in increments of, say, 10-20% at a time while the cluster is under load? 2. Reducing the number is best done via halveing not some aribtrary percentage in order to keep the number a power of 2. Given the 16k PG count, is there a specific "recovery/backfill" throttle we should prioritize to ensure VM performance isn't impacted during the merge process? The performance impact of reducing the number of PGs should be fine. Regards -- --martin konold ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Anthony D'Atri 17 Feb '26

17 Feb '26

> >> *Hardware:* Intel NVMe drives, Intel, or Solidigm? I highly recommend using SST (or if you got them from Dell, DSU) to update firmware. >> *Fails:* Cloning via QEMU (librbd). Check that your qemu / ProxMox defaults to enabling the layering, object-map, and fast-diff feature flags on RBD volumes. >> , the autoscaler increased the total PG >> count to 16k. We suspect this excessive PG count might be contributing to >> the cloning failures, though we aren't certain. I’m skeptical. >> We would like to safely decrease the number of PGs back to a standard range What do you consider a standard range? If you have one OSD deployed on each SSD, the PGS column from `ceph osd df ` should be in the 200-300 range. The autoscaler pgs_per_osd_target defaults to 100 which is rather low for most clusters, plus it’s actually a ceiling not a target. I suggest setting to 300 and mon_max_pgs_per_osd to 600. 16k PGs, assuming no other large pools and replica size=3, would thus be just smurfy across ….. ~~ 240 OSDs, plus or minus. How many do you have? What PGS values do you have? >> Decrease the PG count in small increments. The Manager does this for you since Nautilus I think it was. >> Monitor for I/O latency on the Intel NVMes. Remember that iostat %util isn’t great for that. And update your firmware! >> >> *Questions for the list:* >> >> 1. >> >> Are there known risks to "stepping down" PGs in increments of, say, >> 10-20% at a time while the cluster is under load? No, the Manager steps pg_num and pgp_num for you. Remember to only set the pg_num target to a power of two. >> 2. >> >> Given the 16k PG count, is there a specific "recovery/backfill" throttle >> we should prioritize to ensure VM performance isn't impacted during the >> merge process? osd_max_backfills <= 4 >> >> We are trying to decide if this approach is feasible or if we should create >> a new cluster and migrate VMs I doubt that would be necessary. >> . Any insights would be greatly appreciated. >> >> Thank you, >> >> Robert >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Eugen Block 17 Feb '26

17 Feb '26

IIRC, pg merge is only done one PG at a time, so you have kind of a „natural“ throttling, but it will take quite some time to complete. What’s the output of ‚ceph osd df‘? Do you see ceph warnings? Do you have any more info on the qemu errors, either on client side or in ceph logs? I can’t comment on qemu clones, we only do that with rbd (manually) or let nova/glance/cinder handle it. Zitat von Robert Lukan via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph Community, > > We are seeking advice regarding a strange behavior with cloning > functionality and a strategy for PG reduction on our current cluster. > > *Environment:* > > - > > *Ceph Version:* Squid 19.2.3 (via Proxmox) > - > > *Hardware:* Intel NVMe drives, with very good margin of resources (CPU, > RAM and bandwidth) > - > > *Workload:* Active VM storage - RBD only (a couple 100 VMs, but no > extreme load) > > *The Issue:* We are experiencing a failure with cloning functionality, but > it appears to be specific to which way we do it: > > - > > *Fails:* Cloning via QEMU (librbd). > - > > *Succeeds:* Cloning via rbd clone or deep-copy. > - > > *Succeeds:* Cloning when using krbd enabled on a pool > > This cluster was created about a year ago for testing(PoC) and has evolved > into production. Initially cloning worked fine. Due to disk replacements, > expansions and misconfiguration, the autoscaler increased the total PG > count to 16k. We suspect this excessive PG count might be contributing to > the cloning failures, though we aren't certain. > > We would like to safely decrease the number of PGs back to a standard range > without destabilizing the running VMs. Our plan is to: > > 1. > > Decrease the PG count in small increments. > 2. > > Monitor for I/O latency on the Intel NVMes. > > *Questions for the list:* > > 1. > > Are there known risks to "stepping down" PGs in increments of, say, > 10-20% at a time while the cluster is under load? > 2. > > Given the 16k PG count, is there a specific "recovery/backfill" throttle > we should prioritize to ensure VM performance isn't impacted during the > merge process? > > We are trying to decide if this approach is feasible or if we should create > a new cluster and migrate VMs. Any insights would be greatly appreciated. > > Thank you, > > Robert > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Robert Lukan 17 Feb '26

17 Feb '26

Hello Ceph Community, We are seeking advice regarding a strange behavior with cloning functionality and a strategy for PG reduction on our current cluster. *Environment:* - *Ceph Version:* Squid 19.2.3 (via Proxmox) - *Hardware:* Intel NVMe drives, with very good margin of resources (CPU, RAM and bandwidth) - *Workload:* Active VM storage - RBD only (a couple 100 VMs, but no extreme load) *The Issue:* We are experiencing a failure with cloning functionality, but it appears to be specific to which way we do it: - *Fails:* Cloning via QEMU (librbd). - *Succeeds:* Cloning via rbd clone or deep-copy. - *Succeeds:* Cloning when using krbd enabled on a pool This cluster was created about a year ago for testing(PoC) and has evolved into production. Initially cloning worked fine. Due to disk replacements, expansions and misconfiguration, the autoscaler increased the total PG count to 16k. We suspect this excessive PG count might be contributing to the cloning failures, though we aren't certain. We would like to safely decrease the number of PGs back to a standard range without destabilizing the running VMs. Our plan is to: 1. Decrease the PG count in small increments. 2. Monitor for I/O latency on the Intel NVMes. *Questions for the list:* 1. Are there known risks to "stepping down" PGs in increments of, say, 10-20% at a time while the cluster is under load? 2. Given the 16k PG count, is there a specific "recovery/backfill" throttle we should prioritize to ensure VM performance isn't impacted during the merge process? We are trying to decide if this approach is feasible or if we should create a new cluster and migrate VMs. Any insights would be greatly appreciated. Thank you, Robert

1 0