Octopus PG not going active
The background: We are preparing to migrate our Octopus cluster to Quincy (as a first step on the way to Tentacle). The primary use of the cluster is cephfs, with separate data pools on HDD and SSD. During the migration we accidentally reformatted 6 OSDs in a SSD 6+2 EC pool. Fortunately, all PGs recovered, except for one. Three of the lost OSDs contributed to this PG, so we are accepting that it is gone and any files that were placed on this PG are lost. We identified the lost files with cephfs-data-scan and found that all can be recovered from our non-ceph backup. The problem: Now that we've determined that all lost data can be recovered from backup, we'd like to get the PG active again so we can start this recovery. Unfortunately, the PG is stuck in the incomplete state. We've tried a few things to get it going again, some of which may have made matters worse: ceph osd reweight $osd 0.0 ceph osd lost $osd This left the PG blocked from peering due to 'history_les_bound'. After about an hour we tried: ceph tell $osd config set osd_find_best_info_ignore_history_les 1 This didn't get it peering again, so a couple of hours later we tried forcing a repeer: ceph pg repeer 8.7b That didn't get it going either. Over the next couple of days we then tried a combination of repeer and restarting OSDs. During this we tried doing a force-create on the pg: ceph osd force-create-pg 8.7b However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions? --Mike
Hi, I don't have any suggestions yet, but your attachment wasn't uploaded, you'll have to use a pastebin or add the query output in this thread. Zitat von Michael Thomas <ceph-users@ceph.io>:
The background:
We are preparing to migrate our Octopus cluster to Quincy (as a first step on the way to Tentacle). The primary use of the cluster is cephfs, with separate data pools on HDD and SSD.
During the migration we accidentally reformatted 6 OSDs in a SSD 6+2 EC pool. Fortunately, all PGs recovered, except for one. Three of the lost OSDs contributed to this PG, so we are accepting that it is gone and any files that were placed on this PG are lost. We identified the lost files with cephfs-data-scan and found that all can be recovered from our non-ceph backup.
The problem:
Now that we've determined that all lost data can be recovered from backup, we'd like to get the PG active again so we can start this recovery. Unfortunately, the PG is stuck in the incomplete state. We've tried a few things to get it going again, some of which may have made matters worse:
ceph osd reweight $osd 0.0 ceph osd lost $osd
This left the PG blocked from peering due to 'history_les_bound'. After about an hour we tried:
ceph tell $osd config set osd_find_best_info_ignore_history_les 1
This didn't get it peering again, so a couple of hours later we tried forcing a repeer:
ceph pg repeer 8.7b
That didn't get it going either. Over the next couple of days we then tried a combination of repeer and restarting OSDs. During this we tried doing a force-create on the pg:
ceph osd force-create-pg 8.7b
However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions?
--Mike _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Mike, On 10/12/26 15:54, Michael Thomas wrote:
The background:
We are preparing to migrate our Octopus cluster to Quincy (as a first step on the way to Tentacle). The primary use of the cluster is cephfs, with separate data pools on HDD and SSD.
During the migration we accidentally reformatted 6 OSDs in a SSD 6+2 EC pool. Fortunately, all PGs recovered, except for one. Three of the lost OSDs contributed to this PG, so we are accepting that it is gone and any files that were placed on this PG are lost. We identified the lost files with cephfs-data-scan and found that all can be recovered from our non-ceph backup.
The problem:
Now that we've determined that all lost data can be recovered from backup, we'd like to get the PG active again so we can start this recovery. Unfortunately, the PG is stuck in the incomplete state. We've tried a few things to get it going again, some of which may have made matters worse:
ceph osd reweight $osd 0.0 ceph osd lost $osd
This left the PG blocked from peering due to 'history_les_bound'. After about an hour we tried:
ceph tell $osd config set osd_find_best_info_ignore_history_les 1
This didn't get it peering again, so a couple of hours later we tried forcing a repeer:
ceph pg repeer 8.7b
That didn't get it going either. Over the next couple of days we then tried a combination of repeer and restarting OSDs. During this we tried doing a force-create on the pg:
ceph osd force-create-pg 8.7b
However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions?
your attachment was stripped from the ML, so I don't know the exact state of the PG. Are there any shards of the PG 8.7b still alive? If yes, I assume this prevents force-create-pg from recreating the PG, because some OSDs still reporting they have shards of this PG. You have to stop all OSDs that still hosting PG 8.7b and remove the PG with ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-X --op export-remove --pgid 8.7bsY (replace Y with the shard number, use --op list-pgs to list all PGs). Then bring back the OSDs online. The PG should go into stale state and force-create-pg should work now. I don't have an old octopus setup, but I was able to verify this with a dummy pool on a umbrella test cluster. As long as at least one OSD reports the PG existence, force-create-pg has no effect in my tests. Don't forget to set noout. -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher
On 10/12/26 8:54 AM, Michael Thomas wrote: [...]
However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions? [...] My apologies for the missing attachment. The output from `ceph pg query` is available here:
On 10/14/26 21:20, Michael Thomas wrote:
On 10/12/26 8:54 AM, Michael Thomas wrote: [...]
However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions? [...] My apologies for the missing attachment. The output from `ceph pg query` is available here:
The dump list shards for PG 8.7b: osd.4 8.7bs0 osd.516 8.7bs1 osd.446 8.7bs3 osd.192 8.7bs5 osd.126 8.7bs6 osd.351 8.7bs1 osd.2 8.7bs1 Once these are removed with ceph-objectstore-tool manually, I bet force-create-pg finally recreates the PG. -- Best regards / Mit freundlichen Grüßen Daniel Vogelbacher
On 10/14/26 11:56 PM, Daniel Vogelbacher wrote:
On 10/14/26 21:20, Michael Thomas wrote:
On 10/12/26 8:54 AM, Michael Thomas wrote: [...]
However none of this helped. Attached is the latest output from 'ceph pg 8.7b query'. I'm at a loss at this point on what we can do to get the pg active again so that we can start restoring the lost files. Any suggestions? [...] My apologies for the missing attachment. The output from `ceph pg query` is available here:
The dump list shards for PG 8.7b:
osd.4 8.7bs0 osd.516 8.7bs1 osd.446 8.7bs3 osd.192 8.7bs5 osd.126 8.7bs6 osd.351 8.7bs1 osd.2 8.7bs1
Once these are removed with ceph-objectstore-tool manually, I bet force- create-pg finally recreates the PG. This was very helpful. I've removed the shards and the pg is now up and active. Now the process of restoring the files from backup can start.
Thanks for the help, --Mike
participants (3)
-
Daniel Vogelbacher
-
Eugen Block
-
Michael Thomas