Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?
Hey everyone. Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication. Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point) # ceph pg ls incomplete --> is attached for reference. Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc. I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal. My questions: -- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking? Thank you for your time, Joshua West
Hi Joshua, I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered. Here's how I recovered from the situation. Note that this will permanently remove the affected files from ceph. Restoring them from backup is an excercise left to the reader. * Make a list of the affected PGs: ceph pg dump_stuck | grep recovery_unfound > pg.txt * Make a list of the affected objects (OIDs): cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' > oid.txt * Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and put the results in a file called 'inum.txt' * On a ceph client, find the files that correspond to the affected inodes: cat inum.txt | while read inum ; do echo -n "${inum} " ; find /ceph/frames/O3/raw -inum ${inum} ; done > files.txt * It may be helpful to put this table of PG, OID, inum, and files into a spreadsheet to keep track of what's been done. * On the ceph client, use 'unlink' to remove the files from the filesystem. Do not use 'rm', as it will hang while calling 'stat()' on each file. Even unlink may hang when you first try it. If it does hang, do the following to get it unstuck: - Reboot the client - Restart each mon and the mgr. I rebooted each mon/mgr, but it may be sufficient to restart the services without a reboot. - Try using 'unlink' again * After all of the affected files have been removed, go through the list of PGs and remove the unfound OIDs: ceph pg $pgid mark_unfound_lost delete ...or if you're feeling brave, delete them all at once: cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg mark_unfound_lost delete ; done * Watch the output of 'ceph -s' to see the health of the pools/pgs recover. * Restore the deleted files from backup, or decide that you don't care about them and don't do anything This procedure lets you fix the problem without deleting the affected pool. To be honest, the first time it happened, my solution was to first copy all of the data off of the affected pool and onto a new pool. I later found this to be unnecessary. But if you want to pursue this, here's what I suggest: * Follow the steps above to get rid of the affected files. I feel this should still be done even though you don't care about saving the data, to prevent corruption in the cephfs metadata. * Go through the entire filesystem and look for: - files that are located on the pool (ceph.file.layout.pool = $pool_name) - directories that are set to write files to the pool (ceph.dir.layout.pool = $pool_name) * After you confirm that no files or directories are pointing at the pool anymore, run 'ceph df' and look at the number of objects in the pool. Ideally, it would be zero. But more than likely it isn't. This could be a simple mismatch in the object count in cephfs (harmless), or there could be clients with open filehandles on files that have been removed. such objects will still appear in the rados listing of the pool[1]: rados -p $pool_name ls for obj in $(rados -p $pool_name ls); do echo $obj; rados -p $pool_name getxattr parent | strings; done * To check for clients with access to these stray objects, dump the mds cache: ceph daemon mds.ceph1 dump cache /tmp/cache.txt * Look for lines that refer to the stray objects, like this: [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537 s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 | caps=1 authpin=0 0x563a7e52a000] * The 'caps' field in the output above contains the client session id (eg 9541437). Search the MDS for sessions that match to identify the client: ceph daemon mds.ceph1 session ls > session.txt Search through 'session.txt' for matching entries. This will give you the IP address of the client: "id": 9541437, "entity": { "name": { "type": "client", "num": 9541437 }, "addr": { "type": "v1", "addr": "10.13.5.48:0", "nonce": 2011077845 } }, * Restart the client's connection to ceph to get it to drop the cap. I did this by rebooting the client, but there may be gentler ways to do it. * Once you've done this clean up, it should be safe to remove the pool from cephfs: ceph fs rm_data_pool $fs_name $pool_name * Once the pool has been detached from cephfs, you can remove it from ceph altogether: ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it Hope this helps, --Mike [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html On 4/8/21 5:41 PM, Joshua West wrote:
Hey everyone.
Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication.
Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point)
# ceph pg ls incomplete --> is attached for reference.
Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of
I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc.
I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal.
My questions:
-- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking?
Thank you for your time,
Joshua West _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you Mike! This is honestly a way more detailed reply than I was expecting. You've equipped me with new tools to work with. Thank you! I don't actually have any unfound pgs... only "incomplete" ones, which limits the usefulness of: `grep recovery_unfound` `ceph pg $pg list_unfound` `ceph pg $pg mark_unfound_lost delete` I don't seem to see equivalent commands for incomplete pgs, save for grep of course. This does make me slightly more hopeful that recovery might be possible if the pgs are incomplete and stuck, but not unfound..? Not going to get my hopes too high. Going to attach a few items just to keep from bugging me, if anyone can take a glance, it would be appreciated. In the meantime, in the absence of the above commands, what's the best way to clean this up under the assumption that the data is lost? ~Joshua Joshua West President 403-456-0072 CAYK.ca On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered.
Here's how I recovered from the situation. Note that this will permanently remove the affected files from ceph. Restoring them from backup is an excercise left to the reader.
* Make a list of the affected PGs: ceph pg dump_stuck | grep recovery_unfound > pg.txt
* Make a list of the affected objects (OIDs): cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' > oid.txt
* Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and put the results in a file called 'inum.txt'
* On a ceph client, find the files that correspond to the affected inodes: cat inum.txt | while read inum ; do echo -n "${inum} " ; find /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
* It may be helpful to put this table of PG, OID, inum, and files into a spreadsheet to keep track of what's been done.
* On the ceph client, use 'unlink' to remove the files from the filesystem. Do not use 'rm', as it will hang while calling 'stat()' on each file. Even unlink may hang when you first try it. If it does hang, do the following to get it unstuck: - Reboot the client - Restart each mon and the mgr. I rebooted each mon/mgr, but it may be sufficient to restart the services without a reboot. - Try using 'unlink' again
* After all of the affected files have been removed, go through the list of PGs and remove the unfound OIDs: ceph pg $pgid mark_unfound_lost delete
...or if you're feeling brave, delete them all at once: cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg mark_unfound_lost delete ; done
* Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
* Restore the deleted files from backup, or decide that you don't care about them and don't do anything This procedure lets you fix the problem without deleting the affected pool. To be honest, the first time it happened, my solution was to first copy all of the data off of the affected pool and onto a new pool. I later found this to be unnecessary. But if you want to pursue this, here's what I suggest:
* Follow the steps above to get rid of the affected files. I feel this should still be done even though you don't care about saving the data, to prevent corruption in the cephfs metadata.
* Go through the entire filesystem and look for: - files that are located on the pool (ceph.file.layout.pool = $pool_name) - directories that are set to write files to the pool (ceph.dir.layout.pool = $pool_name)
* After you confirm that no files or directories are pointing at the pool anymore, run 'ceph df' and look at the number of objects in the pool. Ideally, it would be zero. But more than likely it isn't. This could be a simple mismatch in the object count in cephfs (harmless), or there could be clients with open filehandles on files that have been removed. such objects will still appear in the rados listing of the pool[1]: rados -p $pool_name ls for obj in $(rados -p $pool_name ls); do echo $obj; rados -p $pool_name getxattr parent | strings; done
* To check for clients with access to these stray objects, dump the mds cache: ceph daemon mds.ceph1 dump cache /tmp/cache.txt
* Look for lines that refer to the stray objects, like this: [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537 s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 | caps=1 authpin=0 0x563a7e52a000]
* The 'caps' field in the output above contains the client session id (eg 9541437). Search the MDS for sessions that match to identify the client: ceph daemon mds.ceph1 session ls > session.txt Search through 'session.txt' for matching entries. This will give you the IP address of the client: "id": 9541437, "entity": { "name": { "type": "client", "num": 9541437 }, "addr": { "type": "v1", "addr": "10.13.5.48:0", "nonce": 2011077845 } },
* Restart the client's connection to ceph to get it to drop the cap. I did this by rebooting the client, but there may be gentler ways to do it.
* Once you've done this clean up, it should be safe to remove the pool from cephfs: ceph fs rm_data_pool $fs_name $pool_name
* Once the pool has been detached from cephfs, you can remove it from ceph altogether: ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
Hope this helps,
--Mike [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
On 4/8/21 5:41 PM, Joshua West wrote:
Hey everyone.
Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication.
Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point)
# ceph pg ls incomplete --> is attached for reference.
Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of
I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc.
I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal.
My questions:
-- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking?
Thank you for your time,
Joshua West _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Joshua, I'll dig into this output a bit more later, but here are my thoughts right now. I'll preface this by saying that I've never had to clean up from unrecoverable incomplete PGs, so some of what I suggest may not work/apply or be the ideal fix in your case. Correct me if I'm wrong, but you are willing to throw away all of the data on this pool? This should make it easier because we don't have to worry about recovering any lost data. If this is the case, then I think the general strategy would be: 1) Identify and remove any files/directories in cephfs that are located on this pool (based on ceph.file.layout.pool=claypool and ceph.dir.layout.pool=claypool). Use 'unlink' instead of 'rm' to remove the files; it should be less prone to hanging. 2) Wait a bit for ceph to clean up any unreferenced objects. Watch the output of 'ceph df' to see how many objects are listed for the pool. 3) Use 'rados -p claypool ls' to identify the remaining objects. Use the OID identifier to calculate the inode number of each file, then search cephfs to identify which files these belong to. I would expect it would be none, as you already deleted the files in step 1. 4) With nothing in the cephfs metadata referring to the objects anymore, it should be safe to remove them with 'rados -p rm'. 5) Remove the now-empty pool from cephfs 6) Remove the now-empty pool from ceph Can you also include the output of 'ceph df'? --Mike On 4/9/21 7:31 AM, Joshua West wrote:
Thank you Mike!
This is honestly a way more detailed reply than I was expecting. You've equipped me with new tools to work with. Thank you!
I don't actually have any unfound pgs... only "incomplete" ones, which limits the usefulness of: `grep recovery_unfound` `ceph pg $pg list_unfound` `ceph pg $pg mark_unfound_lost delete`
I don't seem to see equivalent commands for incomplete pgs, save for grep of course.
This does make me slightly more hopeful that recovery might be possible if the pgs are incomplete and stuck, but not unfound..? Not going to get my hopes too high.
Going to attach a few items just to keep from bugging me, if anyone can take a glance, it would be appreciated.
In the meantime, in the absence of the above commands, what's the best way to clean this up under the assumption that the data is lost?
~Joshua
Joshua West President 403-456-0072 CAYK.ca
On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered.
Here's how I recovered from the situation. Note that this will permanently remove the affected files from ceph. Restoring them from backup is an excercise left to the reader.
* Make a list of the affected PGs: ceph pg dump_stuck | grep recovery_unfound > pg.txt
* Make a list of the affected objects (OIDs): cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' > oid.txt
* Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and put the results in a file called 'inum.txt'
* On a ceph client, find the files that correspond to the affected inodes: cat inum.txt | while read inum ; do echo -n "${inum} " ; find /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
* It may be helpful to put this table of PG, OID, inum, and files into a spreadsheet to keep track of what's been done.
* On the ceph client, use 'unlink' to remove the files from the filesystem. Do not use 'rm', as it will hang while calling 'stat()' on each file. Even unlink may hang when you first try it. If it does hang, do the following to get it unstuck: - Reboot the client - Restart each mon and the mgr. I rebooted each mon/mgr, but it may be sufficient to restart the services without a reboot. - Try using 'unlink' again
* After all of the affected files have been removed, go through the list of PGs and remove the unfound OIDs: ceph pg $pgid mark_unfound_lost delete
...or if you're feeling brave, delete them all at once: cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg mark_unfound_lost delete ; done
* Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
* Restore the deleted files from backup, or decide that you don't care about them and don't do anything This procedure lets you fix the problem without deleting the affected pool. To be honest, the first time it happened, my solution was to first copy all of the data off of the affected pool and onto a new pool. I later found this to be unnecessary. But if you want to pursue this, here's what I suggest:
* Follow the steps above to get rid of the affected files. I feel this should still be done even though you don't care about saving the data, to prevent corruption in the cephfs metadata.
* Go through the entire filesystem and look for: - files that are located on the pool (ceph.file.layout.pool = $pool_name) - directories that are set to write files to the pool (ceph.dir.layout.pool = $pool_name)
* After you confirm that no files or directories are pointing at the pool anymore, run 'ceph df' and look at the number of objects in the pool. Ideally, it would be zero. But more than likely it isn't. This could be a simple mismatch in the object count in cephfs (harmless), or there could be clients with open filehandles on files that have been removed. such objects will still appear in the rados listing of the pool[1]: rados -p $pool_name ls for obj in $(rados -p $pool_name ls); do echo $obj; rados -p $pool_name getxattr parent | strings; done
* To check for clients with access to these stray objects, dump the mds cache: ceph daemon mds.ceph1 dump cache /tmp/cache.txt
* Look for lines that refer to the stray objects, like this: [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537 s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 | caps=1 authpin=0 0x563a7e52a000]
* The 'caps' field in the output above contains the client session id (eg 9541437). Search the MDS for sessions that match to identify the client: ceph daemon mds.ceph1 session ls > session.txt Search through 'session.txt' for matching entries. This will give you the IP address of the client: "id": 9541437, "entity": { "name": { "type": "client", "num": 9541437 }, "addr": { "type": "v1", "addr": "10.13.5.48:0", "nonce": 2011077845 } },
* Restart the client's connection to ceph to get it to drop the cap. I did this by rebooting the client, but there may be gentler ways to do it.
* Once you've done this clean up, it should be safe to remove the pool from cephfs: ceph fs rm_data_pool $fs_name $pool_name
* Once the pool has been detached from cephfs, you can remove it from ceph altogether: ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
Hope this helps,
--Mike [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
On 4/8/21 5:41 PM, Joshua West wrote:
Hey everyone.
Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication.
Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point)
# ceph pg ls incomplete --> is attached for reference.
Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of
I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc.
I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal.
My questions:
-- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking?
Thank you for your time,
Joshua West _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Absolutely! Attached the files, they're not duplicate, but revised (as I tidied up what I could to make things easier)
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool?
Correct, if push comes to shove, I accept that data-loss is probable. If I can manage to save the data, I would definitely be okay with that too though. Still learning to program, but know python quite well. I am going to push off on a script to clean up per your previously noted steps in the language I know! But will hold off on unlinking everything for the moment. Thank you again for your time, your help has already been invaluable to me. Joshua Joshua West President 403-456-0072 CAYK.ca On Fri, Apr 9, 2021 at 7:03 AM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I'll dig into this output a bit more later, but here are my thoughts right now. I'll preface this by saying that I've never had to clean up from unrecoverable incomplete PGs, so some of what I suggest may not work/apply or be the ideal fix in your case.
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool? This should make it easier because we don't have to worry about recovering any lost data.
If this is the case, then I think the general strategy would be:
1) Identify and remove any files/directories in cephfs that are located on this pool (based on ceph.file.layout.pool=claypool and ceph.dir.layout.pool=claypool). Use 'unlink' instead of 'rm' to remove the files; it should be less prone to hanging.
2) Wait a bit for ceph to clean up any unreferenced objects. Watch the output of 'ceph df' to see how many objects are listed for the pool.
3) Use 'rados -p claypool ls' to identify the remaining objects. Use the OID identifier to calculate the inode number of each file, then search cephfs to identify which files these belong to. I would expect it would be none, as you already deleted the files in step 1.
4) With nothing in the cephfs metadata referring to the objects anymore, it should be safe to remove them with 'rados -p rm'.
5) Remove the now-empty pool from cephfs
6) Remove the now-empty pool from ceph
Can you also include the output of 'ceph df'?
--Mike
On 4/9/21 7:31 AM, Joshua West wrote:
Thank you Mike!
This is honestly a way more detailed reply than I was expecting. You've equipped me with new tools to work with. Thank you!
I don't actually have any unfound pgs... only "incomplete" ones, which limits the usefulness of: `grep recovery_unfound` `ceph pg $pg list_unfound` `ceph pg $pg mark_unfound_lost delete`
I don't seem to see equivalent commands for incomplete pgs, save for grep of course.
This does make me slightly more hopeful that recovery might be possible if the pgs are incomplete and stuck, but not unfound..? Not going to get my hopes too high.
Going to attach a few items just to keep from bugging me, if anyone can take a glance, it would be appreciated.
In the meantime, in the absence of the above commands, what's the best way to clean this up under the assumption that the data is lost?
~Joshua
Joshua West President 403-456-0072 CAYK.ca
On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered.
Here's how I recovered from the situation. Note that this will permanently remove the affected files from ceph. Restoring them from backup is an excercise left to the reader.
* Make a list of the affected PGs: ceph pg dump_stuck | grep recovery_unfound > pg.txt
* Make a list of the affected objects (OIDs): cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' > oid.txt
* Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and put the results in a file called 'inum.txt'
* On a ceph client, find the files that correspond to the affected inodes: cat inum.txt | while read inum ; do echo -n "${inum} " ; find /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
* It may be helpful to put this table of PG, OID, inum, and files into a spreadsheet to keep track of what's been done.
* On the ceph client, use 'unlink' to remove the files from the filesystem. Do not use 'rm', as it will hang while calling 'stat()' on each file. Even unlink may hang when you first try it. If it does hang, do the following to get it unstuck: - Reboot the client - Restart each mon and the mgr. I rebooted each mon/mgr, but it may be sufficient to restart the services without a reboot. - Try using 'unlink' again
* After all of the affected files have been removed, go through the list of PGs and remove the unfound OIDs: ceph pg $pgid mark_unfound_lost delete
...or if you're feeling brave, delete them all at once: cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg mark_unfound_lost delete ; done
* Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
* Restore the deleted files from backup, or decide that you don't care about them and don't do anything This procedure lets you fix the problem without deleting the affected pool. To be honest, the first time it happened, my solution was to first copy all of the data off of the affected pool and onto a new pool. I later found this to be unnecessary. But if you want to pursue this, here's what I suggest:
* Follow the steps above to get rid of the affected files. I feel this should still be done even though you don't care about saving the data, to prevent corruption in the cephfs metadata.
* Go through the entire filesystem and look for: - files that are located on the pool (ceph.file.layout.pool = $pool_name) - directories that are set to write files to the pool (ceph.dir.layout.pool = $pool_name)
* After you confirm that no files or directories are pointing at the pool anymore, run 'ceph df' and look at the number of objects in the pool. Ideally, it would be zero. But more than likely it isn't. This could be a simple mismatch in the object count in cephfs (harmless), or there could be clients with open filehandles on files that have been removed. such objects will still appear in the rados listing of the pool[1]: rados -p $pool_name ls for obj in $(rados -p $pool_name ls); do echo $obj; rados -p $pool_name getxattr parent | strings; done
* To check for clients with access to these stray objects, dump the mds cache: ceph daemon mds.ceph1 dump cache /tmp/cache.txt
* Look for lines that refer to the stray objects, like this: [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537 s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 | caps=1 authpin=0 0x563a7e52a000]
* The 'caps' field in the output above contains the client session id (eg 9541437). Search the MDS for sessions that match to identify the client: ceph daemon mds.ceph1 session ls > session.txt Search through 'session.txt' for matching entries. This will give you the IP address of the client: "id": 9541437, "entity": { "name": { "type": "client", "num": 9541437 }, "addr": { "type": "v1", "addr": "10.13.5.48:0", "nonce": 2011077845 } },
* Restart the client's connection to ceph to get it to drop the cap. I did this by rebooting the client, but there may be gentler ways to do it.
* Once you've done this clean up, it should be safe to remove the pool from cephfs: ceph fs rm_data_pool $fs_name $pool_name
* Once the pool has been detached from cephfs, you can remove it from ceph altogether: ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
Hope this helps,
--Mike [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
On 4/8/21 5:41 PM, Joshua West wrote:
Hey everyone.
Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication.
Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point)
# ceph pg ls incomplete --> is attached for reference.
Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of
I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc.
I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal.
My questions:
-- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking?
Thank you for your time,
Joshua West _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just working this through, how does one identify the OIDs within a PG, without list_unfound? I've been poking around, but can't seem to find a command that outputs the necessary OIDs. I tried a handful of cephfs commands, but they of course become stuck, and ceph pg commands haven't revealed the OID yet. Joshua Joshua West President 403-456-0072 CAYK.ca On Fri, Apr 9, 2021 at 12:15 PM Joshua West <josh@cayk.ca> wrote:
Absolutely!
Attached the files, they're not duplicate, but revised (as I tidied up what I could to make things easier)
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool?
Correct, if push comes to shove, I accept that data-loss is probable. If I can manage to save the data, I would definitely be okay with that too though.
Still learning to program, but know python quite well. I am going to push off on a script to clean up per your previously noted steps in the language I know! But will hold off on unlinking everything for the moment.
Thank you again for your time, your help has already been invaluable to me.
Joshua
Joshua West President 403-456-0072 CAYK.ca
On Fri, Apr 9, 2021 at 7:03 AM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I'll dig into this output a bit more later, but here are my thoughts right now. I'll preface this by saying that I've never had to clean up from unrecoverable incomplete PGs, so some of what I suggest may not work/apply or be the ideal fix in your case.
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool? This should make it easier because we don't have to worry about recovering any lost data.
If this is the case, then I think the general strategy would be:
1) Identify and remove any files/directories in cephfs that are located on this pool (based on ceph.file.layout.pool=claypool and ceph.dir.layout.pool=claypool). Use 'unlink' instead of 'rm' to remove the files; it should be less prone to hanging.
2) Wait a bit for ceph to clean up any unreferenced objects. Watch the output of 'ceph df' to see how many objects are listed for the pool.
3) Use 'rados -p claypool ls' to identify the remaining objects. Use the OID identifier to calculate the inode number of each file, then search cephfs to identify which files these belong to. I would expect it would be none, as you already deleted the files in step 1.
4) With nothing in the cephfs metadata referring to the objects anymore, it should be safe to remove them with 'rados -p rm'.
5) Remove the now-empty pool from cephfs
6) Remove the now-empty pool from ceph
Can you also include the output of 'ceph df'?
--Mike
On 4/9/21 7:31 AM, Joshua West wrote:
Thank you Mike!
This is honestly a way more detailed reply than I was expecting. You've equipped me with new tools to work with. Thank you!
I don't actually have any unfound pgs... only "incomplete" ones, which limits the usefulness of: `grep recovery_unfound` `ceph pg $pg list_unfound` `ceph pg $pg mark_unfound_lost delete`
I don't seem to see equivalent commands for incomplete pgs, save for grep of course.
This does make me slightly more hopeful that recovery might be possible if the pgs are incomplete and stuck, but not unfound..? Not going to get my hopes too high.
Going to attach a few items just to keep from bugging me, if anyone can take a glance, it would be appreciated.
In the meantime, in the absence of the above commands, what's the best way to clean this up under the assumption that the data is lost?
~Joshua
Joshua West President 403-456-0072 CAYK.ca
On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered.
Here's how I recovered from the situation. Note that this will permanently remove the affected files from ceph. Restoring them from backup is an excercise left to the reader.
* Make a list of the affected PGs: ceph pg dump_stuck | grep recovery_unfound > pg.txt
* Make a list of the affected objects (OIDs): cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' > oid.txt
* Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and put the results in a file called 'inum.txt'
* On a ceph client, find the files that correspond to the affected inodes: cat inum.txt | while read inum ; do echo -n "${inum} " ; find /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
* It may be helpful to put this table of PG, OID, inum, and files into a spreadsheet to keep track of what's been done.
* On the ceph client, use 'unlink' to remove the files from the filesystem. Do not use 'rm', as it will hang while calling 'stat()' on each file. Even unlink may hang when you first try it. If it does hang, do the following to get it unstuck: - Reboot the client - Restart each mon and the mgr. I rebooted each mon/mgr, but it may be sufficient to restart the services without a reboot. - Try using 'unlink' again
* After all of the affected files have been removed, go through the list of PGs and remove the unfound OIDs: ceph pg $pgid mark_unfound_lost delete
...or if you're feeling brave, delete them all at once: cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg mark_unfound_lost delete ; done
* Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
* Restore the deleted files from backup, or decide that you don't care about them and don't do anything This procedure lets you fix the problem without deleting the affected pool. To be honest, the first time it happened, my solution was to first copy all of the data off of the affected pool and onto a new pool. I later found this to be unnecessary. But if you want to pursue this, here's what I suggest:
* Follow the steps above to get rid of the affected files. I feel this should still be done even though you don't care about saving the data, to prevent corruption in the cephfs metadata.
* Go through the entire filesystem and look for: - files that are located on the pool (ceph.file.layout.pool = $pool_name) - directories that are set to write files to the pool (ceph.dir.layout.pool = $pool_name)
* After you confirm that no files or directories are pointing at the pool anymore, run 'ceph df' and look at the number of objects in the pool. Ideally, it would be zero. But more than likely it isn't. This could be a simple mismatch in the object count in cephfs (harmless), or there could be clients with open filehandles on files that have been removed. such objects will still appear in the rados listing of the pool[1]: rados -p $pool_name ls for obj in $(rados -p $pool_name ls); do echo $obj; rados -p $pool_name getxattr parent | strings; done
* To check for clients with access to these stray objects, dump the mds cache: ceph daemon mds.ceph1 dump cache /tmp/cache.txt
* Look for lines that refer to the stray objects, like this: [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537 s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 | caps=1 authpin=0 0x563a7e52a000]
* The 'caps' field in the output above contains the client session id (eg 9541437). Search the MDS for sessions that match to identify the client: ceph daemon mds.ceph1 session ls > session.txt Search through 'session.txt' for matching entries. This will give you the IP address of the client: "id": 9541437, "entity": { "name": { "type": "client", "num": 9541437 }, "addr": { "type": "v1", "addr": "10.13.5.48:0", "nonce": 2011077845 } },
* Restart the client's connection to ceph to get it to drop the cap. I did this by rebooting the client, but there may be gentler ways to do it.
* Once you've done this clean up, it should be safe to remove the pool from cephfs: ceph fs rm_data_pool $fs_name $pool_name
* Once the pool has been detached from cephfs, you can remove it from ceph altogether: ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
Hope this helps,
--Mike [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
On 4/8/21 5:41 PM, Joshua West wrote:
Hey everyone.
Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication.
Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point)
# ceph pg ls incomplete --> is attached for reference.
Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of
I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc.
I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal.
My questions:
-- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking?
Thank you for your time,
Joshua West _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Additional to my last note, I should have mentioned, I am exploring options to delete the damaged data, but in hopes to preserve what I can, prior to moving to simply deleting all data on that pool. When trying to simply empty pgs, it seems like the pgs don't exist. In attempting to follow: https://medium.com/opsops/recovering-ceph-from-reduced-data-availability-3-p... with regard to deleting pgs with zero objects/data, I receive: #ceph pg ls incomplete .... 47.3ff 0 0 0 0 0 0 0 0 incomplete 3m 0'0 527856:6054 [7,9,2]p7 [7,9,2]p7... #ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --op info --pgid 47.3ff PG '47.3ff' not found #ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --op remove --pgid 47.3ff --force PG '47.3ff' not found #ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --op mark-complete --pgid 47.3ff PG '47.3ff' not found # ceph osd force-create-pg 47.3ff --yes-i-really-mean-it (worked, but I don't have the output handy) --> No change. Since I am having troubles with this process, can't delete pgs, can't get OIDs for incomplete pgs, I had the idea to start from the other end: Is there a method to determine which files are not stuck, to copy them out, prior to deleting the whole pool? As we know, `ls` becomes stuck, so how is it best to get a list of filepaths+filenames for cephfs? My current plan is to get that list, and simply brute force attempting to copy all files, but each copy in it's own thread + timeout. Does this make sense? Joshua On Wed, Apr 14, 2021 at 6:03 AM Joshua West <josh@cayk.ca> wrote:
Just working this through, how does one identify the OIDs within a PG, without list_unfound?
I've been poking around, but can't seem to find a command that outputs the necessary OIDs. I tried a handful of cephfs commands, but they of course become stuck, and ceph pg commands haven't revealed the OID yet.
Joshua
Joshua West President 403-456-0072 CAYK.ca
On Fri, Apr 9, 2021 at 12:15 PM Joshua West <josh@cayk.ca> wrote:
Absolutely!
Attached the files, they're not duplicate, but revised (as I tidied up what I could to make things easier)
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool?
Correct, if push comes to shove, I accept that data-loss is probable. If I can manage to save the data, I would definitely be okay with that too though.
Still learning to program, but know python quite well. I am going to push off on a script to clean up per your previously noted steps in the language I know! But will hold off on unlinking everything for the moment.
Thank you again for your time, your help has already been invaluable to me.
Joshua
Joshua West President 403-456-0072 CAYK.ca
On Fri, Apr 9, 2021 at 7:03 AM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I'll dig into this output a bit more later, but here are my thoughts right now. I'll preface this by saying that I've never had to clean up from unrecoverable incomplete PGs, so some of what I suggest may not work/apply or be the ideal fix in your case.
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool? This should make it easier because we don't have to worry about recovering any lost data.
If this is the case, then I think the general strategy would be:
1) Identify and remove any files/directories in cephfs that are located on this pool (based on ceph.file.layout.pool=claypool and ceph.dir.layout.pool=claypool). Use 'unlink' instead of 'rm' to remove the files; it should be less prone to hanging.
2) Wait a bit for ceph to clean up any unreferenced objects. Watch the output of 'ceph df' to see how many objects are listed for the pool.
3) Use 'rados -p claypool ls' to identify the remaining objects. Use the OID identifier to calculate the inode number of each file, then search cephfs to identify which files these belong to. I would expect it would be none, as you already deleted the files in step 1.
4) With nothing in the cephfs metadata referring to the objects anymore, it should be safe to remove them with 'rados -p rm'.
5) Remove the now-empty pool from cephfs
6) Remove the now-empty pool from ceph
Can you also include the output of 'ceph df'?
--Mike
On 4/9/21 7:31 AM, Joshua West wrote:
Thank you Mike!
This is honestly a way more detailed reply than I was expecting. You've equipped me with new tools to work with. Thank you!
I don't actually have any unfound pgs... only "incomplete" ones, which limits the usefulness of: `grep recovery_unfound` `ceph pg $pg list_unfound` `ceph pg $pg mark_unfound_lost delete`
I don't seem to see equivalent commands for incomplete pgs, save for grep of course.
This does make me slightly more hopeful that recovery might be possible if the pgs are incomplete and stuck, but not unfound..? Not going to get my hopes too high.
Going to attach a few items just to keep from bugging me, if anyone can take a glance, it would be appreciated.
In the meantime, in the absence of the above commands, what's the best way to clean this up under the assumption that the data is lost?
~Joshua
Joshua West President 403-456-0072 CAYK.ca
On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered.
Here's how I recovered from the situation. Note that this will permanently remove the affected files from ceph. Restoring them from backup is an excercise left to the reader.
* Make a list of the affected PGs: ceph pg dump_stuck | grep recovery_unfound > pg.txt
* Make a list of the affected objects (OIDs): cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' > oid.txt
* Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and put the results in a file called 'inum.txt'
* On a ceph client, find the files that correspond to the affected inodes: cat inum.txt | while read inum ; do echo -n "${inum} " ; find /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
* It may be helpful to put this table of PG, OID, inum, and files into a spreadsheet to keep track of what's been done.
* On the ceph client, use 'unlink' to remove the files from the filesystem. Do not use 'rm', as it will hang while calling 'stat()' on each file. Even unlink may hang when you first try it. If it does hang, do the following to get it unstuck: - Reboot the client - Restart each mon and the mgr. I rebooted each mon/mgr, but it may be sufficient to restart the services without a reboot. - Try using 'unlink' again
* After all of the affected files have been removed, go through the list of PGs and remove the unfound OIDs: ceph pg $pgid mark_unfound_lost delete
...or if you're feeling brave, delete them all at once: cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg mark_unfound_lost delete ; done
* Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
* Restore the deleted files from backup, or decide that you don't care about them and don't do anything This procedure lets you fix the problem without deleting the affected pool. To be honest, the first time it happened, my solution was to first copy all of the data off of the affected pool and onto a new pool. I later found this to be unnecessary. But if you want to pursue this, here's what I suggest:
* Follow the steps above to get rid of the affected files. I feel this should still be done even though you don't care about saving the data, to prevent corruption in the cephfs metadata.
* Go through the entire filesystem and look for: - files that are located on the pool (ceph.file.layout.pool = $pool_name) - directories that are set to write files to the pool (ceph.dir.layout.pool = $pool_name)
* After you confirm that no files or directories are pointing at the pool anymore, run 'ceph df' and look at the number of objects in the pool. Ideally, it would be zero. But more than likely it isn't. This could be a simple mismatch in the object count in cephfs (harmless), or there could be clients with open filehandles on files that have been removed. such objects will still appear in the rados listing of the pool[1]: rados -p $pool_name ls for obj in $(rados -p $pool_name ls); do echo $obj; rados -p $pool_name getxattr parent | strings; done
* To check for clients with access to these stray objects, dump the mds cache: ceph daemon mds.ceph1 dump cache /tmp/cache.txt
* Look for lines that refer to the stray objects, like this: [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537 s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 | caps=1 authpin=0 0x563a7e52a000]
* The 'caps' field in the output above contains the client session id (eg 9541437). Search the MDS for sessions that match to identify the client: ceph daemon mds.ceph1 session ls > session.txt Search through 'session.txt' for matching entries. This will give you the IP address of the client: "id": 9541437, "entity": { "name": { "type": "client", "num": 9541437 }, "addr": { "type": "v1", "addr": "10.13.5.48:0", "nonce": 2011077845 } },
* Restart the client's connection to ceph to get it to drop the cap. I did this by rebooting the client, but there may be gentler ways to do it.
* Once you've done this clean up, it should be safe to remove the pool from cephfs: ceph fs rm_data_pool $fs_name $pool_name
* Once the pool has been detached from cephfs, you can remove it from ceph altogether: ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
Hope this helps,
--Mike [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
On 4/8/21 5:41 PM, Joshua West wrote:
Hey everyone.
Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication.
Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point)
# ceph pg ls incomplete --> is attached for reference.
Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of
I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc.
I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal.
My questions:
-- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking?
Thank you for your time,
Joshua West _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Okay, I gave up, as I should have long ago. Unlinking was a massive pain in the rear, even when scripted, as I never found a good method to catch the hanging without hanging the script, or the filesystem. After several attempts, because I only have a couple of primary directories which were using the damaged pool, I simply removed the pool from cephfs, and deleted the pool outright, with the plan to move the damaged directories into a .dead hidden dir to psuedo fix the issue for myself. After the pool deletion, I noticed that the directories were browsable, all damaged files removed. However, I am unsure if this was because all files were unlinked by the script successfully (script didn't actually hang, just hit a task queue bug and never finished/closed) or if removing the pool also removed the files. Either way, I am down 20T of (off-site recoverable) data, so off to downloading I go! haha Michael, Thank you for your help earlier. Hopefully this little saga is useful to someone in future too! Joshua On Wed, Apr 14, 2021 at 7:08 AM Joshua West <josh@cayk.ca> wrote:
Additional to my last note, I should have mentioned, I am exploring options to delete the damaged data, but in hopes to preserve what I can, prior to moving to simply deleting all data on that pool.
When trying to simply empty pgs, it seems like the pgs don't exist.
In attempting to follow: https://medium.com/opsops/recovering-ceph-from-reduced-data-availability-3-p... with regard to deleting pgs with zero objects/data, I receive:
#ceph pg ls incomplete .... 47.3ff 0 0 0 0 0 0 0 0 incomplete 3m 0'0 527856:6054 [7,9,2]p7 [7,9,2]p7...
#ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --op info --pgid 47.3ff PG '47.3ff' not found
#ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --op remove --pgid 47.3ff --force PG '47.3ff' not found
#ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --op mark-complete --pgid 47.3ff PG '47.3ff' not found
# ceph osd force-create-pg 47.3ff --yes-i-really-mean-it (worked, but I don't have the output handy) --> No change.
Since I am having troubles with this process, can't delete pgs, can't get OIDs for incomplete pgs, I had the idea to start from the other end: Is there a method to determine which files are not stuck, to copy them out, prior to deleting the whole pool?
As we know, `ls` becomes stuck, so how is it best to get a list of filepaths+filenames for cephfs? My current plan is to get that list, and simply brute force attempting to copy all files, but each copy in it's own thread + timeout. Does this make sense?
Joshua
On Wed, Apr 14, 2021 at 6:03 AM Joshua West <josh@cayk.ca> wrote:
Just working this through, how does one identify the OIDs within a PG, without list_unfound?
I've been poking around, but can't seem to find a command that outputs the necessary OIDs. I tried a handful of cephfs commands, but they of course become stuck, and ceph pg commands haven't revealed the OID yet.
Joshua
Joshua West President 403-456-0072 CAYK.ca
On Fri, Apr 9, 2021 at 12:15 PM Joshua West <josh@cayk.ca> wrote:
Absolutely!
Attached the files, they're not duplicate, but revised (as I tidied up what I could to make things easier)
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool?
Correct, if push comes to shove, I accept that data-loss is probable. If I can manage to save the data, I would definitely be okay with that too though.
Still learning to program, but know python quite well. I am going to push off on a script to clean up per your previously noted steps in the language I know! But will hold off on unlinking everything for the moment.
Thank you again for your time, your help has already been invaluable to me.
Joshua
Joshua West President 403-456-0072 CAYK.ca
On Fri, Apr 9, 2021 at 7:03 AM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I'll dig into this output a bit more later, but here are my thoughts right now. I'll preface this by saying that I've never had to clean up from unrecoverable incomplete PGs, so some of what I suggest may not work/apply or be the ideal fix in your case.
Correct me if I'm wrong, but you are willing to throw away all of the data on this pool? This should make it easier because we don't have to worry about recovering any lost data.
If this is the case, then I think the general strategy would be:
1) Identify and remove any files/directories in cephfs that are located on this pool (based on ceph.file.layout.pool=claypool and ceph.dir.layout.pool=claypool). Use 'unlink' instead of 'rm' to remove the files; it should be less prone to hanging.
2) Wait a bit for ceph to clean up any unreferenced objects. Watch the output of 'ceph df' to see how many objects are listed for the pool.
3) Use 'rados -p claypool ls' to identify the remaining objects. Use the OID identifier to calculate the inode number of each file, then search cephfs to identify which files these belong to. I would expect it would be none, as you already deleted the files in step 1.
4) With nothing in the cephfs metadata referring to the objects anymore, it should be safe to remove them with 'rados -p rm'.
5) Remove the now-empty pool from cephfs
6) Remove the now-empty pool from ceph
Can you also include the output of 'ceph df'?
--Mike
On 4/9/21 7:31 AM, Joshua West wrote:
Thank you Mike!
This is honestly a way more detailed reply than I was expecting. You've equipped me with new tools to work with. Thank you!
I don't actually have any unfound pgs... only "incomplete" ones, which limits the usefulness of: `grep recovery_unfound` `ceph pg $pg list_unfound` `ceph pg $pg mark_unfound_lost delete`
I don't seem to see equivalent commands for incomplete pgs, save for grep of course.
This does make me slightly more hopeful that recovery might be possible if the pgs are incomplete and stuck, but not unfound..? Not going to get my hopes too high.
Going to attach a few items just to keep from bugging me, if anyone can take a glance, it would be appreciated.
In the meantime, in the absence of the above commands, what's the best way to clean this up under the assumption that the data is lost?
~Joshua
Joshua West President 403-456-0072 CAYK.ca
On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas <wart@caltech.edu> wrote:
Hi Joshua,
I have had a similar issue three different times on one of my cephfs pools (15.2.10). The first time this happened I had lost some OSDs. In all cases I ended up with degraded PGs with unfound objects that could not be recovered.
Here's how I recovered from the situation. Note that this will permanently remove the affected files from ceph. Restoring them from backup is an excercise left to the reader.
* Make a list of the affected PGs: ceph pg dump_stuck | grep recovery_unfound > pg.txt
* Make a list of the affected objects (OIDs): cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' > oid.txt
* Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and put the results in a file called 'inum.txt'
* On a ceph client, find the files that correspond to the affected inodes: cat inum.txt | while read inum ; do echo -n "${inum} " ; find /ceph/frames/O3/raw -inum ${inum} ; done > files.txt
* It may be helpful to put this table of PG, OID, inum, and files into a spreadsheet to keep track of what's been done.
* On the ceph client, use 'unlink' to remove the files from the filesystem. Do not use 'rm', as it will hang while calling 'stat()' on each file. Even unlink may hang when you first try it. If it does hang, do the following to get it unstuck: - Reboot the client - Restart each mon and the mgr. I rebooted each mon/mgr, but it may be sufficient to restart the services without a reboot. - Try using 'unlink' again
* After all of the affected files have been removed, go through the list of PGs and remove the unfound OIDs: ceph pg $pgid mark_unfound_lost delete
...or if you're feeling brave, delete them all at once: cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg $pg mark_unfound_lost delete ; done
* Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
* Restore the deleted files from backup, or decide that you don't care about them and don't do anything This procedure lets you fix the problem without deleting the affected pool. To be honest, the first time it happened, my solution was to first copy all of the data off of the affected pool and onto a new pool. I later found this to be unnecessary. But if you want to pursue this, here's what I suggest:
* Follow the steps above to get rid of the affected files. I feel this should still be done even though you don't care about saving the data, to prevent corruption in the cephfs metadata.
* Go through the entire filesystem and look for: - files that are located on the pool (ceph.file.layout.pool = $pool_name) - directories that are set to write files to the pool (ceph.dir.layout.pool = $pool_name)
* After you confirm that no files or directories are pointing at the pool anymore, run 'ceph df' and look at the number of objects in the pool. Ideally, it would be zero. But more than likely it isn't. This could be a simple mismatch in the object count in cephfs (harmless), or there could be clients with open filehandles on files that have been removed. such objects will still appear in the rados listing of the pool[1]: rados -p $pool_name ls for obj in $(rados -p $pool_name ls); do echo $obj; rados -p $pool_name getxattr parent | strings; done
* To check for clients with access to these stray objects, dump the mds cache: ceph daemon mds.ceph1 dump cache /tmp/cache.txt
* Look for lines that refer to the stray objects, like this: [inode 0x10000020fbc [2,head] ~mds0/stray6/10000020fbc auth v7440537 s=252778863 nl=0 n(v0 rc2020-12-11T21:17:59.454863-0600 b252778863 1=1+0) (iversion lock) caps={9541437=pAsLsXsFscr/pFscr@2},l=9541437 | caps=1 authpin=0 0x563a7e52a000]
* The 'caps' field in the output above contains the client session id (eg 9541437). Search the MDS for sessions that match to identify the client: ceph daemon mds.ceph1 session ls > session.txt Search through 'session.txt' for matching entries. This will give you the IP address of the client: "id": 9541437, "entity": { "name": { "type": "client", "num": 9541437 }, "addr": { "type": "v1", "addr": "10.13.5.48:0", "nonce": 2011077845 } },
* Restart the client's connection to ceph to get it to drop the cap. I did this by rebooting the client, but there may be gentler ways to do it.
* Once you've done this clean up, it should be safe to remove the pool from cephfs: ceph fs rm_data_pool $fs_name $pool_name
* Once the pool has been detached from cephfs, you can remove it from ceph altogether: ceph osd pool rm $pool_name $pool_name --yes-i-really-really-mean-it
Hope this helps,
--Mike [1]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005234.html
On 4/8/21 5:41 PM, Joshua West wrote: > Hey everyone. > > Inside of cephfs, I have a directory which I setup a directory layout > field to use an erasure coded (CLAY) pool, specific to the task. The > rest of my cephfs is using normal replication. > > Fast forward some time, and the EC directory has been used pretty > extensively, and through some bad luck and poor timing, ~200pgs are in > an incomplete state, and the OSDs are completely gone and > unrecoverable. (Specifically OSD 31 and 34, not that it matters at > this point) > > # ceph pg ls incomplete --> is attached for reference. > > Fortunately, it's primarily (only) my on-site backups, and other > replaceable data inside of > > I tried for a few days to recover the PGs: > - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) > - Deep Scrub > - osd_find_best_info_ignore_history_les = true (`pg query` was > showing related error) > etc. > > I've finally just accepted this pool to be a lesson learned, and want > to get the rest of my cephfs back to normal. > > My questions: > > -- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs > with 0 objects > -- Deleting the pool seems like an appropriate step, but as I am > using an xattr within cephfs, which is otherwise on another pool, I am > not confident that this approach is safe? > -- cephfs currently blocks when attemping to impact every third file > in the EC directory. Once I delete the pool, how will I remove the > files if even `rm` is blocking? > > Thank you for your time, > > Joshua West > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
Hi, So finally how did you solve it? Which method out of the three? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com --------------------------------------------------- -----Original Message----- From: Joshua West <josh@cayk.ca> Sent: Friday, April 9, 2021 5:41 AM To: ceph-users@ceph.io Subject: [ceph-users] Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs? Hey everyone. Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication. Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point) # ceph pg ls incomplete --> is attached for reference. Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc. I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal. My questions: -- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking? Thank you for your time, Joshua West _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
participants (3)
-
Joshua West
-
Michael Thomas
-
Szabo, Istvan (Agoda)