Hi Mike, On 10/12/26 15:54, Michael Thomas wrote:
The background:
We are preparing to migrate our Octopus cluster to Quincy (as a first step on the way to Tentacle). The primary use of the cluster is cephfs, with separate data pools on HDD and SSD.
During the migration we accidentally reformatted 6 OSDs in a SSD 6+2 EC pool. Fortunately, all PGs recovered, except for one. Three of the lost OSDs contributed to this PG, so we are accepting that it is gone and any files that were placed on this PG are lost. We identified the lost files with cephfs-data-scan and found that all can be recovered from our non-ceph backup.
The problem:
Now that we've determined that all lost data can be recovered from backup, we'd like to get the PG active again so we can start this recovery. Unfortunately, the PG is stuck in the incomplete state. We've tried a few things to get it going again, some of which may have made matters worse:
ceph osd reweight $osd 0.0 ceph osd lost $osd
This left the PG blocked from peering due to 'history_les_bound'. After about an hour we tried:
ceph tell $osd config set osd_find_best_info_ignore_history_les 1
This didn't get it peering again, so a couple of hours later we tried forcing a repeer:
ceph pg repeer 8.7b
That didn't get it going either. Over the next couple of days we then tried a combination of repeer and restarting OSDs. During this we tried doing a force-create on the pg:
ceph osd force-create-pg 8.7b
However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions?
your attachment was stripped from the ML, so I don't know the exact state of the PG. Are there any shards of the PG 8.7b still alive? If yes, I assume this prevents force-create-pg from recreating the PG, because some OSDs still reporting they have shards of this PG. You have to stop all OSDs that still hosting PG 8.7b and remove the PG with ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-X --op export-remove --pgid 8.7bsY (replace Y with the shard number, use --op list-pgs to list all PGs). Then bring back the OSDs online. The PG should go into stale state and force-create-pg should work now. I don't have an old octopus setup, but I was able to verify this with a dummy pool on a umbrella test cluster. As long as at least one OSD reports the PG existence, force-create-pg has no effect in my tests. Don't forget to set noout. -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher