Hello Ceph Community, We are seeking advice regarding a strange behavior with cloning functionality and a strategy for PG reduction on our current cluster. *Environment:* - *Ceph Version:* Squid 19.2.3 (via Proxmox) - *Hardware:* Intel NVMe drives, with very good margin of resources (CPU, RAM and bandwidth) - *Workload:* Active VM storage - RBD only (a couple 100 VMs, but no extreme load) *The Issue:* We are experiencing a failure with cloning functionality, but it appears to be specific to which way we do it: - *Fails:* Cloning via QEMU (librbd). - *Succeeds:* Cloning via rbd clone or deep-copy. - *Succeeds:* Cloning when using krbd enabled on a pool This cluster was created about a year ago for testing(PoC) and has evolved into production. Initially cloning worked fine. Due to disk replacements, expansions and misconfiguration, the autoscaler increased the total PG count to 16k. We suspect this excessive PG count might be contributing to the cloning failures, though we aren't certain. We would like to safely decrease the number of PGs back to a standard range without destabilizing the running VMs. Our plan is to: 1. Decrease the PG count in small increments. 2. Monitor for I/O latency on the Intel NVMes. *Questions for the list:* 1. Are there known risks to "stepping down" PGs in increments of, say, 10-20% at a time while the cluster is under load? 2. Given the 16k PG count, is there a specific "recovery/backfill" throttle we should prioritize to ensure VM performance isn't impacted during the merge process? We are trying to decide if this approach is feasible or if we should create a new cluster and migrate VMs. Any insights would be greatly appreciated. Thank you, Robert