- ceph-users - lists.ceph.io

Re: Debian version and ceph.
by Eugen Block 18 Feb '26

18 Feb '26

Right, if you want to bootstrap a cephadm cluster, you don't need to worry too much about the package versions on the host. Although we do recommend to keep them somewhat aligned. But there's also an option for the cephadm bootstrap command: --allow-mismatched-release allow bootstrap of ceph that doesn't match this version of cephadm I used that many times on openSUSE hosts where the latest available version was Pacific (there are packages available again with Reef and Squid from openSUSE Leap 16, Tumbleweed probably too, haven't checked). Last week I also bootstrapped a Squid cluster like that on a Debian Trixie where the host package versions are 18.2.7. Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Am 18.02.26 um 5:53 PM schrieb Albert Shih via ceph-users: > >> 2/ Install a Debian 13 and use the Debian ceph pkg. > > This is the way. > > The client code does not need to match the cluster version. > I usually install ceph-common from the distribution. > > Just make sure you have a cephadm binary matching the desired Ceph > version before bootstrapping the cluster. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

8443: connect: connection refused
by listy 18 Feb '26

18 Feb '26

Hi guys. One node logs: ... 5.744Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephPGBackfillAtRisk\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" ... but no node listen on this port and I have: -> $ ceph config get mgr mgr/dashboard/ssl false What am I missing or having mis-configured? all thoughts are much appreciated, thanks, L.

1 0

Re: Debian version and ceph.
by Robert Sander 18 Feb '26

18 Feb '26

Am 18.02.26 um 5:53 PM schrieb Albert Shih via ceph-users: > 2/ Install a Debian 13 and use the Debian ceph pkg. This is the way. The client code does not need to match the cluster version. I usually install ceph-common from the distribution. Just make sure you have a cephadm binary matching the desired Ceph version before bootstrapping the cluster. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Debian version and ceph.
by Albert Shih 18 Feb '26

18 Feb '26

Hi everyone. I need to install a new cluster ceph. New hardware mean of couse new fresh install. Currently I still don't see any package for Debian trixie. In any case the ceph would be install with cephadm with podman, so no huge dependencies with the OS. But still ceph-base ceph-common libcephfs2 python3-ceph python3-ceph-argparse python3-ceph-common python3-cephfs In that case what would be advise to do ? (I want to stay with Debian). I see 1/ Install a Debian 12 and use the official ceph.com pkg. 2/ Install a Debian 13 and use the Debian ceph pkg. What would you do ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 18 févr. 2026 17:44:51 CET

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Konold, Martin 18 Feb '26

18 Feb '26

Am 2026-02-17 20:24, schrieb Robert Lukan via ceph-users: Hi Robert, Are there known risks to "stepping down" PGs in increments of, say, 10-20% at a time while the cluster is under load? 2. Reducing the number is best done via halveing not some aribtrary percentage in order to keep the number a power of 2. Given the 16k PG count, is there a specific "recovery/backfill" throttle we should prioritize to ensure VM performance isn't impacted during the merge process? The performance impact of reducing the number of PGs should be fine. Regards -- --martin konold ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Anthony D'Atri 18 Feb '26

18 Feb '26

> >> *Hardware:* Intel NVMe drives, Intel, or Solidigm? I highly recommend using SST (or if you got them from Dell, DSU) to update firmware. >> *Fails:* Cloning via QEMU (librbd). Check that your qemu / ProxMox defaults to enabling the layering, object-map, and fast-diff feature flags on RBD volumes. >> , the autoscaler increased the total PG >> count to 16k. We suspect this excessive PG count might be contributing to >> the cloning failures, though we aren't certain. I’m skeptical. >> We would like to safely decrease the number of PGs back to a standard range What do you consider a standard range? If you have one OSD deployed on each SSD, the PGS column from `ceph osd df ` should be in the 200-300 range. The autoscaler pgs_per_osd_target defaults to 100 which is rather low for most clusters, plus it’s actually a ceiling not a target. I suggest setting to 300 and mon_max_pgs_per_osd to 600. 16k PGs, assuming no other large pools and replica size=3, would thus be just smurfy across ….. ~~ 240 OSDs, plus or minus. How many do you have? What PGS values do you have? >> Decrease the PG count in small increments. The Manager does this for you since Nautilus I think it was. >> Monitor for I/O latency on the Intel NVMes. Remember that iostat %util isn’t great for that. And update your firmware! >> >> *Questions for the list:* >> >> 1. >> >> Are there known risks to "stepping down" PGs in increments of, say, >> 10-20% at a time while the cluster is under load? No, the Manager steps pg_num and pgp_num for you. Remember to only set the pg_num target to a power of two. >> 2. >> >> Given the 16k PG count, is there a specific "recovery/backfill" throttle >> we should prioritize to ensure VM performance isn't impacted during the >> merge process? osd_max_backfills <= 4 >> >> We are trying to decide if this approach is feasible or if we should create >> a new cluster and migrate VMs I doubt that would be necessary. >> . Any insights would be greatly appreciated. >> >> Thank you, >> >> Robert >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Eugen Block 17 Feb '26

17 Feb '26

IIRC, pg merge is only done one PG at a time, so you have kind of a „natural“ throttling, but it will take quite some time to complete. What’s the output of ‚ceph osd df‘? Do you see ceph warnings? Do you have any more info on the qemu errors, either on client side or in ceph logs? I can’t comment on qemu clones, we only do that with rbd (manually) or let nova/glance/cinder handle it. Zitat von Robert Lukan via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph Community, > > We are seeking advice regarding a strange behavior with cloning > functionality and a strategy for PG reduction on our current cluster. > > *Environment:* > > - > > *Ceph Version:* Squid 19.2.3 (via Proxmox) > - > > *Hardware:* Intel NVMe drives, with very good margin of resources (CPU, > RAM and bandwidth) > - > > *Workload:* Active VM storage - RBD only (a couple 100 VMs, but no > extreme load) > > *The Issue:* We are experiencing a failure with cloning functionality, but > it appears to be specific to which way we do it: > > - > > *Fails:* Cloning via QEMU (librbd). > - > > *Succeeds:* Cloning via rbd clone or deep-copy. > - > > *Succeeds:* Cloning when using krbd enabled on a pool > > This cluster was created about a year ago for testing(PoC) and has evolved > into production. Initially cloning worked fine. Due to disk replacements, > expansions and misconfiguration, the autoscaler increased the total PG > count to 16k. We suspect this excessive PG count might be contributing to > the cloning failures, though we aren't certain. > > We would like to safely decrease the number of PGs back to a standard range > without destabilizing the running VMs. Our plan is to: > > 1. > > Decrease the PG count in small increments. > 2. > > Monitor for I/O latency on the Intel NVMes. > > *Questions for the list:* > > 1. > > Are there known risks to "stepping down" PGs in increments of, say, > 10-20% at a time while the cluster is under load? > 2. > > Given the 16k PG count, is there a specific "recovery/backfill" throttle > we should prioritize to ensure VM performance isn't impacted during the > merge process? > > We are trying to decide if this approach is feasible or if we should create > a new cluster and migrate VMs. Any insights would be greatly appreciated. > > Thank you, > > Robert > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Robert Lukan 17 Feb '26

17 Feb '26

Hello Ceph Community, We are seeking advice regarding a strange behavior with cloning functionality and a strategy for PG reduction on our current cluster. *Environment:* - *Ceph Version:* Squid 19.2.3 (via Proxmox) - *Hardware:* Intel NVMe drives, with very good margin of resources (CPU, RAM and bandwidth) - *Workload:* Active VM storage - RBD only (a couple 100 VMs, but no extreme load) *The Issue:* We are experiencing a failure with cloning functionality, but it appears to be specific to which way we do it: - *Fails:* Cloning via QEMU (librbd). - *Succeeds:* Cloning via rbd clone or deep-copy. - *Succeeds:* Cloning when using krbd enabled on a pool This cluster was created about a year ago for testing(PoC) and has evolved into production. Initially cloning worked fine. Due to disk replacements, expansions and misconfiguration, the autoscaler increased the total PG count to 16k. We suspect this excessive PG count might be contributing to the cloning failures, though we aren't certain. We would like to safely decrease the number of PGs back to a standard range without destabilizing the running VMs. Our plan is to: 1. Decrease the PG count in small increments. 2. Monitor for I/O latency on the Intel NVMes. *Questions for the list:* 1. Are there known risks to "stepping down" PGs in increments of, say, 10-20% at a time while the cluster is under load? 2. Given the 16k PG count, is there a specific "recovery/backfill" throttle we should prioritize to ensure VM performance isn't impacted during the merge process? We are trying to decide if this approach is feasible or if we should create a new cluster and migrate VMs. Any insights would be greatly appreciated. Thank you, Robert

1 0

Re: discard options for bdevs (OSDs)
by Igor Fedotov 17 Feb '26

17 Feb '26

Hi Robert, here is a brief overview on these parameters: bdev_enable_discard - flag to enable/disable trimming. Once enabled and depending on bdev_async_discard_threads parameter discard requests could be issued either synchronously or asynchronously. Each requests originates when OSD releases specific disk extent, e.g. when overwriting or removing an object (or part of it). In async mode requests are put into a queue and are processed by one of relevant threads. bdev_async_discard_threads - amount of threads to process discard requests asynchronously, if set to 0 and bdev_enable_discart is set to true - all the discard request are issued synchronously bdev_async_discard - this is a deprecated parameter, please don't use it I have no idea how this supposed to work with qcow2 files as the primary targets are SSD drives but in recent Ceph releases (e.g. v19.2.3 and upcoming v18.2.8) one can learn the amount of issued discard requests from OSD perf counters: ceph tell osd.0 perf dump | grep discard -B 1 "blk-kernel-device-bluestore": { "discard_op": 0, "discard_threads": 0 -- "blk-kernel-device-db": { "discard_op": 0, "discard_threads": 0 -- "blk-kernel-device-slow": { "discard_op": 0, "discard_threads": 0 -- "blk-kernel-device-wal": { "discard_op": 0, "discard_threads": 0 Hope this is helpful. Best regards, Igor On 17/02/2026 12:15, Robert Sander via ceph-users wrote: > Hi, > > what is the purpose and effect of these configuration options? > > bdev_async_discard > bdev_async_discard_threads > bdev_enable_discard > > They are not mentioned on docs.ceph.com. > > They read as they would enable a regular trim job on OSDs where the > hardware supports trimming/discarding. How often would this operation > run? > > On a virtual test cluster where the OSDs are backed with qcow2 files > and are presented to the VM as SSDs with discard they seem to have no > effect on the size of the qcow2 file. > > With a normal filesystem an fstrim operation will reduce the size of > the qcow2 file. > > Regards

1 0

Re: CephFS Metadata loss, data present, inodes in pool, scan_links finishes instantly - Squid
by Eugen Block 17 Feb '26

17 Feb '26

Hi, it's not really clear to me if you had the NVMe drives for the CephFS metadata pool only or if you had DB/WAL for spinners as well on those drives. I assume it was only the pool on them, regarding your statement that there are active+clean PGs. More comments inline... Zitat von Enzo Van Thorre via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph users and readers, > > I am facing a really unknown situation for me (being kinda new) with > a CephFS cluster (Squid 19.2.2, Cephadm) where the data pool is > intact (7.2 TB used, 4.9TB stored), inode objects exist in the > recovered metadata pool, but when i try to mount the filesystem, its > empty. > > What initially happened is that I had two groups of drives: nvme’s > and spinning. I had separate pools for both, and I eventually > deleted the nvme’s pool and then removed the nvmes from the machine. > I never thought that they had included metadata for spinning rust’s > pool, but here we are. For the future: 'ceph osd ok-to-stop <ID>' can be helpful if you're not sure if it's okay to stop an OSD. That could have prevented it from happening. > > Version: 19.2.2 Squid > Metadata pool ID 1 (cephfs.cephfs.meta), Size=1 (No Replicas :(, yes > I know I messed up). yeah... > Data Pool ID 6 (ec_data), erasure coded with k=2 m=1. that's not great either (k2, m1) > All OSDs are up. 256 PGs are active+clean. there’s then quite a bit > that are inconsistent, degraded, undersized, remapped, and other > stuff. what is "other stuff"? A 'ceph health detail' could be helpful. > Here’s what I’ve done since losing the metadata pool due to OSD > purges. I tried to rebuild metadata from the data pool by doing: > 1. cephfs-data-scan init --force-init > 2. cephfs-data-scan scan_extents > 3. cephfs-data-scan scan_inodes > > > > ceph df shows 7.2 TiB stored in pool ec_data (which is the spinning > rust pool) > Inodes do exist. running rados -p cephfs.cephfs.meta ls | grep > "100." returns tons of objects (ex. 1000005f1f3.00000000). > If i try to mount via ceph-fuse, it shows an empty root. there is > no lost+found folder either. > > A big problem is that when I try to link the orphaned inodes to > lost+found, the command finishes instantly with no output, as if it > believes the cluster is clean. > > I run: cephfs-data-scan scan_links --filesystem cephfs > # and it returns code 0, no output, and runs in <1 second > > So my final question is since scan_inodes has populated the metadata > pool with objects (100.xxxx), but scan_links ignores them, how do I > force a re-scan or just do SOMETHING so scan_links will actually > link these files into lost+found? Unfortunately, I don't have a comment on that right now. But maybe increasing debug level for MDS can help here? > > All The Best, > Enzo > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1