Hi Frank, Yes, it does sounds similar to your ticket. I've tried a few things to restore the failed files: * Locate a missing object with 'ceph pg $pgid list_unfound' * Convert the hex oid to a decimal inode number * Identify the affected file with 'find /ceph -inum $inode' At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available. Next I tried to remove the affected object with: * ceph pg $pgid mark_unfound_lost delete Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang. Finally, I ran a scrub over the part of the filesystem containing the affected file: ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive Nothing seemed to come up during the scrub: 2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads: 2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched Does anyone have any suggestions on how else to clean up from a permanently lost object? --Mike On 9/16/20 2:03 AM, Frank Schilder wrote:

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/.... I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

task status: scrub status: mds.ceph4: idle

data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded

progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Frank, On 9/18/20 2:50 AM, Frank Schilder wrote:

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system. This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed. Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/), I ran: ceph pg 7.1fb mark_unfound_lost delete So I presume that the entire object has been deleted.

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object. *Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours.

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times. I've used three different methods to restart the OSD: * systemctl restart ceph-osd@120 * init 6 * ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120 I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain. Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects. --Mike

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote:

...

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/.... I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

task status: scrub status: mds.ceph4: idle

data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded

progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Dear Michael, maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing. Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects Dear Michael,

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.

I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself. The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls. An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave? Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet. The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s). I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation. Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects Hi Frank, On 9/18/20 2:50 AM, Frank Schilder wrote:

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system. This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed. Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/), I ran: ceph pg 7.1fb mark_unfound_lost delete So I presume that the entire object has been deleted.

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object. *Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours.

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times. I've used three different methods to restart the OSD: * systemctl restart ceph-osd@120 * init 6 * ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120 I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain. Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects. --Mike

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote:

...

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/.... I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

task status: scrub status: mds.ceph4: idle

data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded

progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Dear Michael,

I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage.

I would probably not attempt any such maintenance before there was a period of at least 1 day with HEALTH_OK. The reason is that certain historical information is not trimmed unless the cluster is in HEALTH_OK. The more such information is accumulated, the more risk one runs that a cluster becomes unstable. Can you post the output of ceph status, ceph health detail, ceph osd pool stats and ceph osd df tree (on pastebin.com)? If I remember correctly, you removed OSDs/PGs following a trouble-shooting guide? I suspect that the removal has left something in an inconsistent state that requires manual clean up for recovery to proceed. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 09 October 2020 22:33:46 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects Hi Frank, That was a good tip. I was able to move the broken files out of the way and restore them for users. However, after 2 weeks I'm still left with unfound objects. Even more annoying, I now have 82k objects degraded (up from 74), which hasn't changed in over a week. I'm ready to claim that the auto-repair capabilities of ceph are not able to fix my particular issues, and will have to continue to investigate alternate ways to clean this up, including a pg export/import (as you suggested) and perhaps a mds backward scrub (after testing in a junk pool first). I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage. One particular PG is in a curious state: 7.39d 82163 82165 246734 1 344060777807 0 0 2139 active+recovery_unfound+undersized+degraded+remapped 23m 50755'112549 50766:960500 [116,72,122,48,45,131,73,81]p116 [71,109,99,48,45,90,73,NONE]p71 2020-08-13T23:02:34.325887-0500 2020-08-07T11:01:45.657036-0500 Note the 'NONE' in the acting set. I do not know which OSD this may have been, nor how to find out. I suspect (without evidence) that this is part of the cause of no action on the degraded and misplaced objects. --Mike On 9/18/20 11:26 AM, Frank Schilder wrote:

Dear Michael,

maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing.

Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Dear Michael,

...

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.

I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself.

The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls.

An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave?

Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet.

The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s).

I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation.

Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

On 9/18/20 2:50 AM, Frank Schilder wrote:

...

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system.

This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed.

Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

...

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/), I ran:

ceph pg 7.1fb mark_unfound_lost delete

So I presume that the entire object has been deleted.

...

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object.

*Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours.

...

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times. I've used three different methods to restart the OSD:

* systemctl restart ceph-osd@120

* init 6

* ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120

I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain.

Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects.

--Mike

...

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote:

...

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/.... I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

task status: scrub status: mds.ceph4: idle

data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded

progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hello Mike, do your OSDs go down from time to time? I once has an issue with unrecoverable objects, because I had only n+1 (size 2) redundancy and ceph wasn't able to decide, what's the correct copy of the object. In my case there half-deleted snapshots  in one of the copies. I used ceph-objectstoretool to remove the "wrong" part. Did you check you OSD logs? Do the osd go down wirth an obscure stacktrace (and maybe they are restartet by systemd ...) rgds, j. On 09.10.20 22:33, Michael Thomas wrote:

Hi Frank,

That was a good tip.  I was able to move the broken files out of the way and restore them for users.  However, after 2 weeks I'm still left with unfound objects.  Even more annoying, I now have 82k objects degraded (up from 74), which hasn't changed in over a week.

I'm ready to claim that the auto-repair capabilities of ceph are not able to fix my particular issues, and will have to continue to investigate alternate ways to clean this up, including a pg export/import (as you suggested) and perhaps a mds backward scrub (after testing in a junk pool first).

I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage.

One particular PG is in a curious state:

7.39d    82163     82165     246734        1  344060777807            0   0   2139  active+recovery_unfound+undersized+degraded+remapped 23m  50755'112549   50766:960500       [116,72,122,48,45,131,73,81]p116       [71,109,99,48,45,90,73,NONE]p71  2020-08-13T23:02:34.325887-0500 2020-08-07T11:01:45.657036-0500

Note the 'NONE' in the acting set.  I do not know which OSD this may have been, nor how to find out.  I suspect (without evidence) that this is part of the cause of no action on the degraded and misplaced objects.

--Mike

On 9/18/20 11:26 AM, Frank Schilder wrote:

...

Dear Michael,

maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing.

Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Dear Michael,

...

I disagree with the statement that trying to recover health by deleting data is a contradiction.  In some cases (such as mine), the data in ceph is backed up in another location (eg tape library).  Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.

I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself.

The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls.

An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave?

Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet.

The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s).

I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation.

Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

On 9/18/20 2:50 AM, Frank Schilder wrote:

...

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction.  In some cases (such as mine), the data in ceph is backed up in another location (eg tape library).  Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.  For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system.

This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well.    I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed.

Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

...

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/),

I ran:

ceph pg 7.1fb mark_unfound_lost delete

So I presume that the entire object has been deleted.

...

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang.  Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang.  Even worse, attempts to 'ls' in the directory itself would hang.  This hasn't changed after removing the object.

*Update*: The stat() operations may not be hanging indefinitely.  It seems to hang for somewhere between 10 minutes and 8 hours.

...

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times.  I've used three different methods to restart the OSD:

* systemctl restart ceph-osd@120

* init 6

* ceph osd out 120     ...wait for repeering to finish...     systemctl restart ceph-osd@120     ceph osd in 120

I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'.  But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain.

Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster.  I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects.

--Mike

...

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object.  As expected, attempts to read the file simply hang.  Unexpectedly, attempts to 'ls' the file or its containing directory also hang.  I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700  1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700  1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700  0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700  0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700  0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700  1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700  1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700  0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700  0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700  0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700  0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang.  The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700  0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote:

...

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time.  The OSDs are part of an erasure coded pool.  At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster.  After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects.  There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool.  The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/....

     I found a number of OSDs that are still 'not queried'.  Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed.  I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations."  I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

      cluster:         id:     066f558c-6789-4a93-aaf1-5af1ba01a3ad         health: HEALTH_ERR                 1 clients failing to respond to capability release                 1 MDSs report slow requests                 25/78520351 objects unfound (0.000%)                 2 nearfull osd(s)                 Reduced data availability: 1 pg inactive                 Possible data damage: 9 pgs recovery_unfound                 Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded                 1013 pgs not deep-scrubbed in time                 1013 pgs not scrubbed in time                 2 pool(s) nearfull                 1 daemons have recently crashed                 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

      services:         mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d)         mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1         mds: archive:1 {0=ceph4=up:active} 3 up:standby         osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

      task status:         scrub status:             mds.ceph4: idle

      data:         pools:   9 pools, 2433 pgs         objects: 78.52M objects, 298 TiB         usage:   412 TiB used, 545 TiB / 956 TiB avail         pgs:     0.041% pgs unknown                  75/626645098 objects degraded (0.000%)                  135224/626645098 objects misplaced (0.022%)                  25/78520351 objects unfound (0.000%)                  2421 active+clean                  5    active+recovery_unfound+degraded                  3    active+recovery_unfound+degraded+remapped                  2    active+clean+scrubbing+deep                  1    unknown                  1    active+forced_recovery+recovery_unfound+degraded

      progress:         PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d)           [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet

Hi Michael, it doesn't look too bad. All degraded objects are due to the undersized PG. If this is an EC pool with m>=2, data is currently not in danger. I see a few loose ends to pick up, let's hope this is something simple. For any of the below, before attempting the next step, please wait until all induced recovery IO has completed before continuing. 1) Could you please paste the output of the following commands to pastebin (bash syntax): ceph osd pool get device_health_metrics all ceph osd pool get fs.data.archive.frames all ceph pg dump |& grep -i -e PG_STAT -e "^7.39d" ceph osd crush rule ls ceph osd erasure-code-profile ls ceph osd crush dump # this is a big one, please be careful with copy-paste (see point 3 below) 2) I don't see any IO reported (neither user nor recovery). Could you please confirm that the command outputs were taken during a zero-IO period? 3) Something is wrong with osd.41. Can you check its health status with smartctl? If it is reported healthy, give it one more clean restart. If the slow ops do not disappear, it could be a disk fail that is not detected by health monitoring. You could set it to "out" and see if the cluster recovers to a healthy state (modulo the currently degraded objects) with no slow ops. If so, I would replace the disk. 4) In the output of "df tree" node141 shows up twice. Could you confirm that this is a copy-paste error or is this node indeed twice in the output? This is easiest to see in the pastebin when switching to "raw" view. 5) The crush tree contains an empty host bucket (node308). Please delete this host bucket (ceph osd crush rm node308) for now and let me know if this caused any data movements (recovery IO). 6) The crush tree looks a bit exotic. Do the nodes with a single OSD correspond to a physical host with 1 OSD disk? If not, could you please state how the host buckets are mapped onto physical hosts? 7) In case there was a change to the health status, could you please include an updated "ceph health detail"? I don't expect to get the incomplete PG resolved with the above, but it will move some issues out of the way before proceeding. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 14 October 2020 20:52:10 To: Andreas John; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects Hello, The original cause of the OSD instability has already been fixed. It was due to user jobs (via condor) consuming too much memory and causing the machine to swap. The OSDs didn't actually crash, but weren't responding in time and were being flagged as down. In most cases, the problematic OSD servers were also not responding on the console and had to be physically power cycled to recover. Since adding additional memory limits to user jobs, we have only had 1 or 2 unstable OSDs that were fixed by killing the remaining rogue user jobs. Regards, --Mike On 10/10/20 9:22 AM, Andreas John wrote:

Hello Mike,

do your OSDs go down from time to time? I once has an issue with unrecoverable objects, because I had only n+1 (size 2) redundancy and ceph wasn't able to decide, what's the correct copy of the object. In my case there half-deleted snapshots in one of the copies. I used ceph-objectstoretool to remove the "wrong" part. Did you check you OSD logs? Do the osd go down wirth an obscure stacktrace (and maybe they are restartet by systemd ...)

rgds,

j.

On 09.10.20 22:33, Michael Thomas wrote:

...

Hi Frank,

That was a good tip. I was able to move the broken files out of the way and restore them for users. However, after 2 weeks I'm still left with unfound objects. Even more annoying, I now have 82k objects degraded (up from 74), which hasn't changed in over a week.

I'm ready to claim that the auto-repair capabilities of ceph are not able to fix my particular issues, and will have to continue to investigate alternate ways to clean this up, including a pg export/import (as you suggested) and perhaps a mds backward scrub (after testing in a junk pool first).

I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage.

One particular PG is in a curious state:

7.39d 82163 82165 246734 1 344060777807 0 0 2139 active+recovery_unfound+undersized+degraded+remapped 23m 50755'112549 50766:960500 [116,72,122,48,45,131,73,81]p116 [71,109,99,48,45,90,73,NONE]p71 2020-08-13T23:02:34.325887-0500 2020-08-07T11:01:45.657036-0500

Note the 'NONE' in the acting set. I do not know which OSD this may have been, nor how to find out. I suspect (without evidence) that this is part of the cause of no action on the degraded and misplaced objects.

--Mike

On 9/18/20 11:26 AM, Frank Schilder wrote:

...

Dear Michael,

maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing.

Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Dear Michael,

...

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.

I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself.

The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls.

An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave?

Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet.

The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s).

I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation.

Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

On 9/18/20 2:50 AM, Frank Schilder wrote:

...

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system.

This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed.

Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

...

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/),

I ran:

ceph pg 7.1fb mark_unfound_lost delete

So I presume that the entire object has been deleted.

...

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object.

*Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours.

...

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times. I've used three different methods to restart the OSD:

* systemctl restart ceph-osd@120

* init 6

* ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120

I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain.

Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects.

--Mike

...

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote:

...

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/....

I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations."  I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

task status: scrub status: mds.ceph4: idle

data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded

progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Dear Michael, thanks for this initial work. I will need to look through the files you posted in more detail. In the meantime: Please mark OSD 41 as "in" again and wait for some slow ops to show up. As far as I can see, marking it "out" might have cleared hanging slow ops (there were 1000 before), but they then started piling up again. From the OSD log it looks like an operation that is sent to/from PG 1.0, which doesn't respond because it is inactive. Hence, getting PG 1.0 active should resolve this issue (later). Its a bit strange that I see slow ops for OSD 41 in the latest health detail (https://pastebin.com/3G3ij9ui). Was the OSD still out when this health report was created? I think we might have misunderstood my question 6. My question was whether or not each host bucket corresponds to a physical host and vice versa, that is, each physical host has exactly 1 host bucket. I'm asking because it is possible to have multiple host buckets assigned to a single physical host and this has implications on how to manage things. Coming back to PG 1.0 (the only PG in pool device_health_metrics as far as I can see), the problem is that is has no OSDs assigned. I need to look a bit longer at the data you uploaded to find out why. I can't see anything obvious. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 October 2020 02:08:01 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects On 10/14/20 3:49 PM, Frank Schilder wrote:

Hi Michael,

it doesn't look too bad. All degraded objects are due to the undersized PG. If this is an EC pool with m>=2, data is currently not in danger.

I see a few loose ends to pick up, let's hope this is something simple. For any of the below, before attempting the next step, please wait until all induced recovery IO has completed before continuing.

1) Could you please paste the output of the following commands to pastebin (bash syntax):

ceph osd pool get device_health_metrics all

https://pastebin.com/6D83mjsV

ceph osd pool get fs.data.archive.frames all

https://pastebin.com/7XAaQcpC

ceph pg dump |& grep -i -e PG_STAT -e "^7.39d"

https://pastebin.com/tBLaq63Q

ceph osd crush rule ls

https://pastebin.com/6f5B778G

ceph osd erasure-code-profile ls

https://pastebin.com/uhAaMH1c

ceph osd crush dump # this is a big one, please be careful with copy-paste (see point 3 below)

https://pastebin.com/u92D23jV

2) I don't see any IO reported (neither user nor recovery). Could you please confirm that the command outputs were taken during a zero-IO period?

That's correct, there was no activity at this time. Access to the cephfs filesystem is very bursty, varying from completely idle to multiple GB/s (read).

3) Something is wrong with osd.41. Can you check its health status with smartctl? If it is reported healthy, give it one more clean restart. If the slow ops do not disappear, it could be a disk fail that is not detected by health monitoring. You could set it to "out" and see if the cluster recovers to a healthy state (modulo the currently degraded objects) with no slow ops. If so, I would replace the disk.

smartctl reports no problems. osd.41 (and osd.0) was one of the original OSDs used for the device_health_metrics pool. Early on, before I knew better, I had removed this OSD (and osd.0) from the cluster, and the OSD ids got recycled when new disks were later added. This is when the slow ops on osd.0 and osd.41 started getting reported. On advice from another user on ceph-users, I updated my crush map to remap the device_health_metrics pool to a different set of OSDs (and the slow ops persisted). osd.0 usually also shows slow ops. I was a little surprised that it didn't when I took this snapshot, but now it does. I have now run 'ceph osd out 41', and the recovery I/O has finished. With the exception of one less OSD marked in, the output of 'ceph status' looks the same. The last few lines of the osd.41 logfile are here: https://pastebin.com/k06aArW4 How long does it take for ceph to clear the slow ops status?

4) In the output of "df tree" node141 shows up twice. Could you confirm that this is a copy-paste error or is this node indeed twice in the output? This is easiest to see in the pastebin when switching to "raw" view.

This was a copy/paste error.

5) The crush tree contains an empty host bucket (node308). Please delete this host bucket (ceph osd crush rm node308) for now and let me know if this caused any data movements (recovery IO).

This did not cause any data movement, according to 'ceph status'.

6) The crush tree looks a bit exotic. Do the nodes with a single OSD correspond to a physical host with 1 OSD disk? If not, could you please state how the host buckets are mapped onto physical hosts?

Each OSD corresponds to a single physical disk. Hosts may have 1, 2 or 3 OSDs of varying types (HDD, SSD, or SSD+NVME). There are a few different crush types used in the cluster: 3 x replicated nvme - used for cephfs metadata 3 x replicated SSD - used for ovirt block storage EC HDD - used for the bulk of the experiment data EC SSD - used for frequently accessed experiment data

7) In case there was a change to the health status, could you please include an updated "ceph health detail"?

Looks like the only difference is a new slow MDS op, and one PG that hasn't been deep scrubbed in the last week: https://pastebin.com/3G3ij9ui --Mike

I don't expect to get the incomplete PG resolved with the above, but it will move some issues out of the way before proceeding.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 14 October 2020 20:52:10 To: Andreas John; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Hello,

The original cause of the OSD instability has already been fixed. It was due to user jobs (via condor) consuming too much memory and causing the machine to swap. The OSDs didn't actually crash, but weren't responding in time and were being flagged as down.

In most cases, the problematic OSD servers were also not responding on the console and had to be physically power cycled to recover.

Since adding additional memory limits to user jobs, we have only had 1 or 2 unstable OSDs that were fixed by killing the remaining rogue user jobs.

Regards,

--Mike

On 10/10/20 9:22 AM, Andreas John wrote:

...

Hello Mike,

do your OSDs go down from time to time? I once has an issue with unrecoverable objects, because I had only n+1 (size 2) redundancy and ceph wasn't able to decide, what's the correct copy of the object. In my case there half-deleted snapshots in one of the copies. I used ceph-objectstoretool to remove the "wrong" part. Did you check you OSD logs? Do the osd go down wirth an obscure stacktrace (and maybe they are restartet by systemd ...)

rgds,

j.

On 09.10.20 22:33, Michael Thomas wrote:

...

Hi Frank,

That was a good tip. I was able to move the broken files out of the way and restore them for users. However, after 2 weeks I'm still left with unfound objects. Even more annoying, I now have 82k objects degraded (up from 74), which hasn't changed in over a week.

I'm ready to claim that the auto-repair capabilities of ceph are not able to fix my particular issues, and will have to continue to investigate alternate ways to clean this up, including a pg export/import (as you suggested) and perhaps a mds backward scrub (after testing in a junk pool first).

I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage.

One particular PG is in a curious state:

7.39d 82163 82165 246734 1 344060777807 0 0 2139 active+recovery_unfound+undersized+degraded+remapped 23m 50755'112549 50766:960500 [116,72,122,48,45,131,73,81]p116 [71,109,99,48,45,90,73,NONE]p71 2020-08-13T23:02:34.325887-0500 2020-08-07T11:01:45.657036-0500

Note the 'NONE' in the acting set. I do not know which OSD this may have been, nor how to find out. I suspect (without evidence) that this is part of the cause of no action on the degraded and misplaced objects.

--Mike

On 9/18/20 11:26 AM, Frank Schilder wrote:

...

Dear Michael,

maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing.

Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Dear Michael,

...

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.

I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself.

The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls.

An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave?

Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet.

The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s).

I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation.

Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

On 9/18/20 2:50 AM, Frank Schilder wrote:

...

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system.

This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed.

Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

...

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/),

I ran:

ceph pg 7.1fb mark_unfound_lost delete

So I presume that the entire object has been deleted.

...

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object.

*Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours.

...

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times. I've used three different methods to restart the OSD:

* systemctl restart ceph-osd@120

* init 6

* ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120

I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain.

Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects.

--Mike

...

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote:

...

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/....

I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations."  I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

task status: scrub status: mds.ceph4: idle

data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded

progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Dear Michael, this is a bit of a nut. I can't see anything obvious. I have two hypotheses that you might consider testing. 1) Problem with 1 incomplete PG. In the shadow hierarchy for your cluster I can see quite a lot of nodes like { "id": -135, "name": "node229~hdd", "type_id": 1, "type_name": "host", "weight": 0, "alg": "straw2", "hash": "rjenkins1", "items": [] }, I would have expected that hosts without a device of a certain device class are *excluded* completely from a tree instead of having weight 0. I'm wondering if this could lead to the crush algorithm fail in the way described here: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/#cr... . This might be a long shot, but could you export your crush map and play with the tunables as described under this link to see if more tries lead to a valid mapping? Note that testing this is harmless and does not change anything on the cluster. The hypothesis here is that buckets with weight 0 are not excluded from drawing a-priori, but a-posteriori. If there are too many draws of an empty bucket, a mapping fails. Allowing more tries should then lead to success. We should at least rule out this possibility. 2) About the incomplete PG. I'm wondering if the problem is that the pool has exactly 1 PG. I don't have a test pool with Nautilus and cannot try this out. Can you create a test pool with pg_num=pgp_num=1 and see if the PG gets an OSD mapping? If not, can you then increase pg_num and pgp_num to, say, 10 and see if this has any effect? I'm wondering here if there needs to be a minimum number >1 of PGs in a pool. Again, this is more about ruling out a possibility than expecting success. As an extension to this test, you could increase pg_num and pgp_num of the pool device_health_metrics to see if this has any effect. The crush rules and crush tree look OK to me. I can't really see why the missing OSDs are not assigned to the two PGs 1.0 and 7.39d. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 16 October 2020 15:41:29 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects Dear Michael,

Please mark OSD 41 as "in" again and wait for some slow ops to show up.

I forgot. "wait for some slow ops to show up" ... and then what? Could you please go to the host of the affected OSD and look at the output of "ceph daemon osd.ID ops" or "ceph daemon osd.ID dump_historic_slow_ops" and check what type of operations get stuck? I'm wondering if its administrative, like peering attempts. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder Sent: 16 October 2020 15:09:20 To: Michael Thomas; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects Dear Michael, thanks for this initial work. I will need to look through the files you posted in more detail. In the meantime: Please mark OSD 41 as "in" again and wait for some slow ops to show up. As far as I can see, marking it "out" might have cleared hanging slow ops (there were 1000 before), but they then started piling up again. From the OSD log it looks like an operation that is sent to/from PG 1.0, which doesn't respond because it is inactive. Hence, getting PG 1.0 active should resolve this issue (later). Its a bit strange that I see slow ops for OSD 41 in the latest health detail (https://pastebin.com/3G3ij9ui). Was the OSD still out when this health report was created? I think we might have misunderstood my question 6. My question was whether or not each host bucket corresponds to a physical host and vice versa, that is, each physical host has exactly 1 host bucket. I'm asking because it is possible to have multiple host buckets assigned to a single physical host and this has implications on how to manage things. Coming back to PG 1.0 (the only PG in pool device_health_metrics as far as I can see), the problem is that is has no OSDs assigned. I need to look a bit longer at the data you uploaded to find out why. I can't see anything obvious. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 October 2020 02:08:01 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects On 10/14/20 3:49 PM, Frank Schilder wrote:

Hi Michael,

it doesn't look too bad. All degraded objects are due to the undersized PG. If this is an EC pool with m>=2, data is currently not in danger.

I see a few loose ends to pick up, let's hope this is something simple. For any of the below, before attempting the next step, please wait until all induced recovery IO has completed before continuing.

1) Could you please paste the output of the following commands to pastebin (bash syntax):

ceph osd pool get device_health_metrics all

https://pastebin.com/6D83mjsV

ceph osd pool get fs.data.archive.frames all

https://pastebin.com/7XAaQcpC

ceph pg dump |& grep -i -e PG_STAT -e "^7.39d"

https://pastebin.com/tBLaq63Q

ceph osd crush rule ls

https://pastebin.com/6f5B778G

ceph osd erasure-code-profile ls

https://pastebin.com/uhAaMH1c

ceph osd crush dump # this is a big one, please be careful with copy-paste (see point 3 below)

https://pastebin.com/u92D23jV

2) I don't see any IO reported (neither user nor recovery). Could you please confirm that the command outputs were taken during a zero-IO period?

That's correct, there was no activity at this time. Access to the cephfs filesystem is very bursty, varying from completely idle to multiple GB/s (read).

3) Something is wrong with osd.41. Can you check its health status with smartctl? If it is reported healthy, give it one more clean restart. If the slow ops do not disappear, it could be a disk fail that is not detected by health monitoring. You could set it to "out" and see if the cluster recovers to a healthy state (modulo the currently degraded objects) with no slow ops. If so, I would replace the disk.

smartctl reports no problems. osd.41 (and osd.0) was one of the original OSDs used for the device_health_metrics pool. Early on, before I knew better, I had removed this OSD (and osd.0) from the cluster, and the OSD ids got recycled when new disks were later added. This is when the slow ops on osd.0 and osd.41 started getting reported. On advice from another user on ceph-users, I updated my crush map to remap the device_health_metrics pool to a different set of OSDs (and the slow ops persisted). osd.0 usually also shows slow ops. I was a little surprised that it didn't when I took this snapshot, but now it does. I have now run 'ceph osd out 41', and the recovery I/O has finished. With the exception of one less OSD marked in, the output of 'ceph status' looks the same. The last few lines of the osd.41 logfile are here: https://pastebin.com/k06aArW4 How long does it take for ceph to clear the slow ops status?

4) In the output of "df tree" node141 shows up twice. Could you confirm that this is a copy-paste error or is this node indeed twice in the output? This is easiest to see in the pastebin when switching to "raw" view.

This was a copy/paste error.

5) The crush tree contains an empty host bucket (node308). Please delete this host bucket (ceph osd crush rm node308) for now and let me know if this caused any data movements (recovery IO).

This did not cause any data movement, according to 'ceph status'.

6) The crush tree looks a bit exotic. Do the nodes with a single OSD correspond to a physical host with 1 OSD disk? If not, could you please state how the host buckets are mapped onto physical hosts?

Each OSD corresponds to a single physical disk. Hosts may have 1, 2 or 3 OSDs of varying types (HDD, SSD, or SSD+NVME). There are a few different crush types used in the cluster: 3 x replicated nvme - used for cephfs metadata 3 x replicated SSD - used for ovirt block storage EC HDD - used for the bulk of the experiment data EC SSD - used for frequently accessed experiment data

7) In case there was a change to the health status, could you please include an updated "ceph health detail"?

Looks like the only difference is a new slow MDS op, and one PG that hasn't been deep scrubbed in the last week: https://pastebin.com/3G3ij9ui --Mike

I don't expect to get the incomplete PG resolved with the above, but it will move some issues out of the way before proceeding.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 14 October 2020 20:52:10 To: Andreas John; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Hello,

The original cause of the OSD instability has already been fixed. It was due to user jobs (via condor) consuming too much memory and causing the machine to swap. The OSDs didn't actually crash, but weren't responding in time and were being flagged as down.

In most cases, the problematic OSD servers were also not responding on the console and had to be physically power cycled to recover.

Since adding additional memory limits to user jobs, we have only had 1 or 2 unstable OSDs that were fixed by killing the remaining rogue user jobs.

Regards,

--Mike

On 10/10/20 9:22 AM, Andreas John wrote:

...

Hello Mike,

do your OSDs go down from time to time? I once has an issue with unrecoverable objects, because I had only n+1 (size 2) redundancy and ceph wasn't able to decide, what's the correct copy of the object. In my case there half-deleted snapshots in one of the copies. I used ceph-objectstoretool to remove the "wrong" part. Did you check you OSD logs? Do the osd go down wirth an obscure stacktrace (and maybe they are restartet by systemd ...)

rgds,

j.

On 09.10.20 22:33, Michael Thomas wrote:

...

Hi Frank,

That was a good tip. I was able to move the broken files out of the way and restore them for users. However, after 2 weeks I'm still left with unfound objects. Even more annoying, I now have 82k objects degraded (up from 74), which hasn't changed in over a week.

I'm ready to claim that the auto-repair capabilities of ceph are not able to fix my particular issues, and will have to continue to investigate alternate ways to clean this up, including a pg export/import (as you suggested) and perhaps a mds backward scrub (after testing in a junk pool first).

I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage.

One particular PG is in a curious state:

7.39d 82163 82165 246734 1 344060777807 0 0 2139 active+recovery_unfound+undersized+degraded+remapped 23m 50755'112549 50766:960500 [116,72,122,48,45,131,73,81]p116 [71,109,99,48,45,90,73,NONE]p71 2020-08-13T23:02:34.325887-0500 2020-08-07T11:01:45.657036-0500

Note the 'NONE' in the acting set. I do not know which OSD this may have been, nor how to find out. I suspect (without evidence) that this is part of the cause of no action on the degraded and misplaced objects.

--Mike

On 9/18/20 11:26 AM, Frank Schilder wrote:

...

Dear Michael,

maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing.

Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Dear Michael,

...

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.

I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself.

The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls.

An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave?

Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet.

The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s).

I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation.

Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

On 9/18/20 2:50 AM, Frank Schilder wrote:

...

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system.

This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed.

Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

...

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/),

I ran:

ceph pg 7.1fb mark_unfound_lost delete

So I presume that the entire object has been deleted.

...

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object.

*Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours.

...

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times. I've used three different methods to restart the OSD:

* systemctl restart ceph-osd@120

* init 6

* ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120

I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain.

Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects.

--Mike

...

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote:

...

Sounds similar to this one: https://tracker.ceph.com/issues/46847

If you have or can reconstruct the crush map from before adding the OSDs, you might be able to discover everything with the temporary reversal of the crush map method.

Not sure if there is another method, i never got a reply to my question in the tracker.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 September 2020 01:27:19 To: ceph-users@ceph.io Subject: [ceph-users] multiple OSD crash, unfound objects

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects.

The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem).

I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/....

I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects.

I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below).

The guide also says "Sometimes it simply takes some time for the cluster to query possible locations."  I'm not sure how long "some time" might take, but it hasn't changed after several hours.

My questions are:

* Is there a way to force the cluster to query the possible locations sooner?

* Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes?

--Mike

ceph -s:

cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops.

services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs

task status: scrub status: mds.ceph4: idle

data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded

progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................] _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Dear Michael,

...

Can you create a test pool with pg_num=pgp_num=1 and see if the PG gets an OSD mapping?

I meant here with crush rule replicated_host_nvme. Sorry, forgot.

Yes, the OSD was still out when the previous health report was created.

Hmm, this is odd. If this is correct, then it did report a slow op even though it was out of the cluster:

from https://pastebin.com/3G3ij9ui: [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 8133 sec, daemons [osd.0,osd.41] have slow ops.

Not sure what to make of that. It looks almost like you have a ghost osd.41. I think (some of) the slow ops you are seeing are directed to the health_metrics pool and can be ignored. If it is too annoying, you could try to find out who runs the client with IDs client.7524484 and disable it. Might be an MGR module. Looking at the data you provided and also some older threads of yours (https://www.mail-archive.com/ceph-users@ceph.io/msg05842.html), I start considering that we are looking at the fall-out of a past admin operation. A possibility is, that an upmap for PG 1.0 exists that conflicts with the crush rule replicated_host_nvme and, hence, prevents the assignment of OSDs to PG 1.0. For example, the upmap specifies HDDs, but the crush rule required NVMEs. This result is an empty set. I couldn't really find a simple command to list up-maps. The only non-destructive way seems to be to extract the osdmap and create a clean-up command file. The cleanup file should contain a command for every PG with an upmap. To check this, you can execute (see also https://docs.ceph.com/en/latest/man/8/osdmaptool/) # ceph osd getmap > osd.map # osdmaptool osd.map --upmap-cleanup cleanup.cmd If you do this, could you please post as usual the contents of cleanup.cmd? Also, with the OSD map of your cluster, you can simulate certain admin operations and check resulting PG mappings for pools and other things without having to touch the cluster; see https://docs.ceph.com/en/latest/man/8/osdmaptool/. To dig a little bit deeper, could you please post as usual the output of: - ceph pg 1.0 query - ceph pg 7.39d query It would also be helpful if you could post the decoded crush map. You can get the map as a txt-file as follows: # ceph osd getcrushmap -o crush-orig.bin # crushtool -d crush-orig.bin -o crush.txt and post the contents of file crush.txt. Did the slow MDS request complete by now? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 Contents of previous messages removed.

Hi Michael, some quick thoughts. That you can create a pool with 1 PG is a good sign, the crush rule is OK. That pg query says it doesn't have PG 1.0 points in the right direction. There is an inconsistency in the cluster. This is also indicated by the fact that no upmaps seem to exist (the clean-up script was empty). With the osd map you extracted, you could check what the osd map believes the mapping of the PGs of pool 1 are: # osdmaptool osd.map --test-map-pgs-dump --pool 1 or if it also claims the PG does not exist. It looks like something went wrong during pool creation and you are not the only one having problems with this particular pool: https://www.spinics.net/lists/ceph-users/msg52665.html . Sounds a lot like a bug in cephadm. In principle, it looks like the idea to delete and recreate the health metrics pool is a way forward. Please look at the procedure mentioned in the thread quoted above. Deletion of the pool there lead to some crashes and some surgery on some OSDs was necessary. However, in your case it might just work, because you redeployed the OSDs in question already - if I remember correctly. In order to do so cleanly, however, you will probably want to shut down all clients accessing this pool. Note that clients accessing the health metrics pool are not FS clients, so the mds cannot tell you anything about them. The only command that seems to list all clients is # ceph daemon mon.MON-ID sessions that needs to be executed on all mon hosts. On the other hand, you could also just go ahead and see if something crashes (an MGR module probably) or disable all MGR modules during this recovery attempt. I found some info that cephadm creates this pool and starts an MGR module. If you google "device_health_metric pool" you should find descriptions of similar cases. It looks solvable. I will look at the incomplete PG issue. I hope this is just some PG tuning. At least pg query didn't complain :) The stuck MDS request could be an attempt to access an unfound object. It should be possible to locate the fs client and find out what it was trying to do. I see this sometimes when people are too impatient. They manage to trigger a race condition and an MDS operation gets stuck (there are MDS bugs and in my case it was an ls command that got stuck). Usually, evicting the client temporarily solves the issue (but tell the user :). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 20 October 2020 23:48:36 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects On 10/20/20 1:18 PM, Frank Schilder wrote:

Dear Michael,

...

...

Can you create a test pool with pg_num=pgp_num=1 and see if the PG gets an OSD mapping?

I meant here with crush rule replicated_host_nvme. Sorry, forgot.

Seems to have worked fine: https://pastebin.com/PFgDE4J1

...

Yes, the OSD was still out when the previous health report was created.

Hmm, this is odd. If this is correct, then it did report a slow op even though it was out of the cluster:

...

from https://pastebin.com/3G3ij9ui: [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 8133 sec, daemons [osd.0,osd.41] have slow ops.

Not sure what to make of that. It looks almost like you have a ghost osd.41.

I think (some of) the slow ops you are seeing are directed to the health_metrics pool and can be ignored. If it is too annoying, you could try to find out who runs the client with IDs client.7524484 and disable it. Might be an MGR module.

I'm also pretty certain that the slow ops are related to the health metrics pool, which is why I've been ignoring them. What I'm not sure about is whether re-creating the device_health_metrics pool will cause any problems in the ceph cluster.

Looking at the data you provided and also some older threads of yours (https://www.mail-archive.com/ceph-users@ceph.io/msg05842.html), I start considering that we are looking at the fall-out of a past admin operation. A possibility is, that an upmap for PG 1.0 exists that conflicts with the crush rule replicated_host_nvme and, hence, prevents the assignment of OSDs to PG 1.0. For example, the upmap specifies HDDs, but the crush rule required NVMEs. This result is an empty set.

So var I've been unable to locate the client with the ID 7524484. It's not showing up in the manager dashboard -> Filesystems page, nor in the output of 'ceph tell mds.ceph1 client ls'. I'm digging through the compress logs for the past week to see if I can find the culprit.

I couldn't really find a simple command to list up-maps. The only non-destructive way seems to be to extract the osdmap and create a clean-up command file. The cleanup file should contain a command for every PG with an upmap. To check this, you can execute (see also https://docs.ceph.com/en/latest/man/8/osdmaptool/)

# ceph osd getmap > osd.map # osdmaptool osd.map --upmap-cleanup cleanup.cmd

If you do this, could you please post as usual the contents of cleanup.cmd?

It was empty: [root@ceph1 ~]# ceph osd getmap > osd.map got osdmap epoch 52833 [root@ceph1 ~]# osdmaptool osd.map --upmap-cleanup cleanup.cmd osdmaptool: osdmap file 'osd.map' writing upmap command output to: cleanup.cmd checking for upmap cleanups [root@ceph1 ~]# wc cleanup.cmd 0 0 0 cleanup.cmd

Also, with the OSD map of your cluster, you can simulate certain admin operations and check resulting PG mappings for pools and other things without having to touch the cluster; see https://docs.ceph.com/en/latest/man/8/osdmaptool/.

To dig a little bit deeper, could you please post as usual the output of:

- ceph pg 1.0 query - ceph pg 7.39d query

Oddly, it claims that it doesn't have pgid 1.0. https://pastebin.com/pHh33Dq7

It would also be helpful if you could post the decoded crush map. You can get the map as a txt-file as follows:

# ceph osd getcrushmap -o crush-orig.bin # crushtool -d crush-orig.bin -o crush.txt

and post the contents of file crush.txt.

https://pastebin.com/EtEGpWy3

Did the slow MDS request complete by now?

Nope. --Mike

Sounds good. Did you re-create the pool again? If not, please do to give the devicehealth manager module its storage. In case you can't see any IO, it might be necessary to restart the MGR to flush out a stale rados connection. I would probably give the pool 10 PGs instead of 1, but that's up to you. I hope I find time today to look at the incomplete PG. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 21 October 2020 22:58:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects On 10/21/20 6:47 AM, Frank Schilder wrote:

Hi Michael,

some quick thoughts.

That you can create a pool with 1 PG is a good sign, the crush rule is OK. That pg query says it doesn't have PG 1.0 points in the right direction. There is an inconsistency in the cluster. This is also indicated by the fact that no upmaps seem to exist (the clean-up script was empty). With the osd map you extracted, you could check what the osd map believes the mapping of the PGs of pool 1 are:

# osdmaptool osd.map --test-map-pgs-dump --pool 1

https://pastebin.com/seh6gb7R As I suspected, it thinks that OSDs 0, 41 are the acting set.

or if it also claims the PG does not exist. It looks like something went wrong during pool creation and you are not the only one having problems with this particular pool: https://www.spinics.net/lists/ceph-users/msg52665.html . Sounds a lot like a bug in cephadm.

In principle, it looks like the idea to delete and recreate the health metrics pool is a way forward. Please look at the procedure mentioned in the thread quoted above. Deletion of the pool there lead to some crashes and some surgery on some OSDs was necessary. However, in your case it might just work, because you redeployed the OSDs in question already - if I remember correctly.

That is correct. The original OSDs 0 and 41 were removed and redeployed on new disks.

In order to do so cleanly, however, you will probably want to shut down all clients accessing this pool. Note that clients accessing the health metrics pool are not FS clients, so the mds cannot tell you anything about them. The only command that seems to list all clients is

# ceph daemon mon.MON-ID sessions

that needs to be executed on all mon hosts. On the other hand, you could also just go ahead and see if something crashes (an MGR module probably) or disable all MGR modules during this recovery attempt. I found some info that cephadm creates this pool and starts an MGR module.

If you google "device_health_metric pool" you should find descriptions of similar cases. It looks solvable.

Unfortunately, in Octopus you can not disable the devicehealth manager module, and the manager is required for operation. So I just went ahead and removed the pool with everything still running. Fortunately, this did not appear to cause any problems, and the single unknown PG has disappeared from the ceph health output.

I will look at the incomplete PG issue. I hope this is just some PG tuning. At least pg query didn't complain :)

I have OSDs ready to add to the pool, in case you think we should try.

The stuck MDS request could be an attempt to access an unfound object. It should be possible to locate the fs client and find out what it was trying to do. I see this sometimes when people are too impatient. They manage to trigger a race condition and an MDS operation gets stuck (there are MDS bugs and in my case it was an ls command that got stuck). Usually, evicting the client temporarily solves the issue (but tell the user :).

I found the fs client and rebooted it. The MDS still reports the slow OPs, but according to the mds logs the offending ops were established before the client was rebooted, and the offending client session (now defunct) has been blacklisted. I'll check back later to see if the slow OPS get cleared from 'ceph status'. Regards, --Mike ________________________________________

From: Michael Thomas <wart@caltech.edu> Sent: 20 October 2020 23:48:36 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

On 10/20/20 1:18 PM, Frank Schilder wrote:

...

Dear Michael,

...

...

Can you create a test pool with pg_num=pgp_num=1 and see if the PG gets an OSD mapping?

I meant here with crush rule replicated_host_nvme. Sorry, forgot.

Seems to have worked fine:

https://pastebin.com/PFgDE4J1

...

...

Yes, the OSD was still out when the previous health report was created.

Hmm, this is odd. If this is correct, then it did report a slow op even though it was out of the cluster:

...

from https://pastebin.com/3G3ij9ui: [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 8133 sec, daemons [osd.0,osd.41] have slow ops.

Not sure what to make of that. It looks almost like you have a ghost osd.41.

I think (some of) the slow ops you are seeing are directed to the health_metrics pool and can be ignored. If it is too annoying, you could try to find out who runs the client with IDs client.7524484 and disable it. Might be an MGR module.

I'm also pretty certain that the slow ops are related to the health metrics pool, which is why I've been ignoring them.

What I'm not sure about is whether re-creating the device_health_metrics pool will cause any problems in the ceph cluster.

...

Looking at the data you provided and also some older threads of yours (https://www.mail-archive.com/ceph-users@ceph.io/msg05842.html), I start considering that we are looking at the fall-out of a past admin operation. A possibility is, that an upmap for PG 1.0 exists that conflicts with the crush rule replicated_host_nvme and, hence, prevents the assignment of OSDs to PG 1.0. For example, the upmap specifies HDDs, but the crush rule required NVMEs. This result is an empty set.

So var I've been unable to locate the client with the ID 7524484. It's not showing up in the manager dashboard -> Filesystems page, nor in the output of 'ceph tell mds.ceph1 client ls'.

I'm digging through the compress logs for the past week to see if I can find the culprit.

...

I couldn't really find a simple command to list up-maps. The only non-destructive way seems to be to extract the osdmap and create a clean-up command file. The cleanup file should contain a command for every PG with an upmap. To check this, you can execute (see also https://docs.ceph.com/en/latest/man/8/osdmaptool/)

# ceph osd getmap > osd.map # osdmaptool osd.map --upmap-cleanup cleanup.cmd

If you do this, could you please post as usual the contents of cleanup.cmd?

It was empty:

[root@ceph1 ~]# ceph osd getmap > osd.map got osdmap epoch 52833

[root@ceph1 ~]# osdmaptool osd.map --upmap-cleanup cleanup.cmd osdmaptool: osdmap file 'osd.map' writing upmap command output to: cleanup.cmd checking for upmap cleanups

[root@ceph1 ~]# wc cleanup.cmd 0 0 0 cleanup.cmd

...

Also, with the OSD map of your cluster, you can simulate certain admin operations and check resulting PG mappings for pools and other things without having to touch the cluster; see https://docs.ceph.com/en/latest/man/8/osdmaptool/.

To dig a little bit deeper, could you please post as usual the output of:

- ceph pg 1.0 query - ceph pg 7.39d query

Oddly, it claims that it doesn't have pgid 1.0.

https://pastebin.com/pHh33Dq7

...

It would also be helpful if you could post the decoded crush map. You can get the map as a txt-file as follows:

# ceph osd getcrushmap -o crush-orig.bin # crushtool -d crush-orig.bin -o crush.txt

and post the contents of file crush.txt.

https://pastebin.com/EtEGpWy3

...

Did the slow MDS request complete by now?

Nope.

--Mike

On 10/22/20 3:22 AM, Frank Schilder wrote:

Could you also execute (and post the output of)

# osdmaptool osd.map --test-map-pgs-dump --pool 7

osdmaptool dumped core. Here is stdout: https://pastebin.com/HPtSqcS1 The PG map for 7.39d matches the pg dump, with the expected difference of 2147483647 -> NONE. ...and here is stderr: https://pastebin.com/CrtwE54r Regards, --Mike

with the osd map you pulled out (pool 7 should be the fs data pool)? Please check what mapping is reported for PG 7.39d? Just checking if osd map and pg dump agree here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 22 October 2020 09:32:07 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Sounds good. Did you re-create the pool again? If not, please do to give the devicehealth manager module its storage. In case you can't see any IO, it might be necessary to restart the MGR to flush out a stale rados connection. I would probably give the pool 10 PGs instead of 1, but that's up to you.

I hope I find time today to look at the incomplete PG.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 21 October 2020 22:58:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

On 10/21/20 6:47 AM, Frank Schilder wrote:

...

Hi Michael,

some quick thoughts.

That you can create a pool with 1 PG is a good sign, the crush rule is OK. That pg query says it doesn't have PG 1.0 points in the right direction. There is an inconsistency in the cluster. This is also indicated by the fact that no upmaps seem to exist (the clean-up script was empty). With the osd map you extracted, you could check what the osd map believes the mapping of the PGs of pool 1 are:

# osdmaptool osd.map --test-map-pgs-dump --pool 1

https://pastebin.com/seh6gb7R

As I suspected, it thinks that OSDs 0, 41 are the acting set.

...

or if it also claims the PG does not exist. It looks like something went wrong during pool creation and you are not the only one having problems with this particular pool: https://www.spinics.net/lists/ceph-users/msg52665.html . Sounds a lot like a bug in cephadm.

In principle, it looks like the idea to delete and recreate the health metrics pool is a way forward. Please look at the procedure mentioned in the thread quoted above. Deletion of the pool there lead to some crashes and some surgery on some OSDs was necessary. However, in your case it might just work, because you redeployed the OSDs in question already - if I remember correctly.

That is correct. The original OSDs 0 and 41 were removed and redeployed on new disks.

...

In order to do so cleanly, however, you will probably want to shut down all clients accessing this pool. Note that clients accessing the health metrics pool are not FS clients, so the mds cannot tell you anything about them. The only command that seems to list all clients is

# ceph daemon mon.MON-ID sessions

that needs to be executed on all mon hosts. On the other hand, you could also just go ahead and see if something crashes (an MGR module probably) or disable all MGR modules during this recovery attempt. I found some info that cephadm creates this pool and starts an MGR module.

If you google "device_health_metric pool" you should find descriptions of similar cases. It looks solvable.

Unfortunately, in Octopus you can not disable the devicehealth manager module, and the manager is required for operation. So I just went ahead and removed the pool with everything still running. Fortunately, this did not appear to cause any problems, and the single unknown PG has disappeared from the ceph health output.

...

I will look at the incomplete PG issue. I hope this is just some PG tuning. At least pg query didn't complain :)

I have OSDs ready to add to the pool, in case you think we should try.

...

The stuck MDS request could be an attempt to access an unfound object. It should be possible to locate the fs client and find out what it was trying to do. I see this sometimes when people are too impatient. They manage to trigger a race condition and an MDS operation gets stuck (there are MDS bugs and in my case it was an ls command that got stuck). Usually, evicting the client temporarily solves the issue (but tell the user :).

I found the fs client and rebooted it. The MDS still reports the slow OPs, but according to the mds logs the offending ops were established before the client was rebooted, and the offending client session (now defunct) has been blacklisted. I'll check back later to see if the slow OPS get cleared from 'ceph status'.

Regards,

--Mike ________________________________________

...

From: Michael Thomas <wart@caltech.edu> Sent: 20 October 2020 23:48:36 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

On 10/20/20 1:18 PM, Frank Schilder wrote:

...

Dear Michael,

...

...

Can you create a test pool with pg_num=pgp_num=1 and see if the PG gets an OSD mapping?

I meant here with crush rule replicated_host_nvme. Sorry, forgot.

Seems to have worked fine:

https://pastebin.com/PFgDE4J1

...

...

Yes, the OSD was still out when the previous health report was created.

Hmm, this is odd. If this is correct, then it did report a slow op even though it was out of the cluster:

...

from https://pastebin.com/3G3ij9ui: [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 8133 sec, daemons [osd.0,osd.41] have slow ops.

Not sure what to make of that. It looks almost like you have a ghost osd.41.

I think (some of) the slow ops you are seeing are directed to the health_metrics pool and can be ignored. If it is too annoying, you could try to find out who runs the client with IDs client.7524484 and disable it. Might be an MGR module.

I'm also pretty certain that the slow ops are related to the health metrics pool, which is why I've been ignoring them.

What I'm not sure about is whether re-creating the device_health_metrics pool will cause any problems in the ceph cluster.

...

Looking at the data you provided and also some older threads of yours (https://www.mail-archive.com/ceph-users@ceph.io/msg05842.html), I start considering that we are looking at the fall-out of a past admin operation. A possibility is, that an upmap for PG 1.0 exists that conflicts with the crush rule replicated_host_nvme and, hence, prevents the assignment of OSDs to PG 1.0. For example, the upmap specifies HDDs, but the crush rule required NVMEs. This result is an empty set.

So var I've been unable to locate the client with the ID 7524484. It's not showing up in the manager dashboard -> Filesystems page, nor in the output of 'ceph tell mds.ceph1 client ls'.

I'm digging through the compress logs for the past week to see if I can find the culprit.

...

I couldn't really find a simple command to list up-maps. The only non-destructive way seems to be to extract the osdmap and create a clean-up command file. The cleanup file should contain a command for every PG with an upmap. To check this, you can execute (see also https://docs.ceph.com/en/latest/man/8/osdmaptool/)

# ceph osd getmap > osd.map # osdmaptool osd.map --upmap-cleanup cleanup.cmd

If you do this, could you please post as usual the contents of cleanup.cmd?

It was empty:

[root@ceph1 ~]# ceph osd getmap > osd.map got osdmap epoch 52833

[root@ceph1 ~]# osdmaptool osd.map --upmap-cleanup cleanup.cmd osdmaptool: osdmap file 'osd.map' writing upmap command output to: cleanup.cmd checking for upmap cleanups

[root@ceph1 ~]# wc cleanup.cmd 0 0 0 cleanup.cmd

...

Also, with the OSD map of your cluster, you can simulate certain admin operations and check resulting PG mappings for pools and other things without having to touch the cluster; see https://docs.ceph.com/en/latest/man/8/osdmaptool/.

To dig a little bit deeper, could you please post as usual the output of:

- ceph pg 1.0 query - ceph pg 7.39d query

Oddly, it claims that it doesn't have pgid 1.0.

https://pastebin.com/pHh33Dq7

...

It would also be helpful if you could post the decoded crush map. You can get the map as a txt-file as follows:

# ceph osd getcrushmap -o crush-orig.bin # crushtool -d crush-orig.bin -o crush.txt

and post the contents of file crush.txt.

https://pastebin.com/EtEGpWy3

...

Did the slow MDS request complete by now?

Nope.

--Mike

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Michael.

I still don't see any traffic to the pool, though I'm also unsure how much traffic is to be expected.

Probably not much. If ceph df shows that the pool contains some objects, I guess that's sorted. That osdmaptool crashes indicates that your cluster runs with corrupted internal data. I tested your crush map and you should get complete PGs for the fs data pool. That you don't and that osdmaptool crashes points at a corruption of internal data. I'm afraid this is the point where you need support from ceph developers and should file a tracker report (https://tracker.ceph.com/projects/ceph/issues). A short description of the origin of the situation with the osdmaptool output and a reference to this thread linked in should be sufficient. Please post a link to the ticket here. In parallel, you should probably open a new thread focussed on the osd map corruption. Maybe there are low-level commands to repair it. You should wait with trying to clean up the unfound objects until this is resolved. Not sure about adding further storage either. To me, this sounds quite serious. Best regards and good luck! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

Dear Michael, I was also wondering whether deleting the broken pool could clean up everything. The difficulty is, that while migrating a pool to new devices is easy via a crush rule change, migrating data between pools is not so easy. In particular, if you can't afford downtime. In case you can afford some downtime, it might be possible to migrate fast by creating a new pool and use the pool copy command to migrate the data (rados cppool ...). Its important that the FS is shutdown (no MDS active) during this copy process. After copy, one could either rename the pools to have the copy match the fs data pool name, or change the data pool at the top level directory. You might need to set some pool meta data by hand, notably, the fs tag. Having said that, I have no idea how a ceph fs reacts if presented with a replacement data pool. Although I don't believe that meta data contains the pool IDs, I cannot exclude that complication. The copy pool variant should be tested with an isolated FS first. The other option is what you describe, create a new data pool, make the fs root placed on this pool and copy every file onto itself. This should also do the trick. However, with this method you will not be able to get rid of the broken pool. After the copy, you could, however, reduce the number of PGs to below the unhealthy one and the broken PG(s) might get deleted cleanly. Then you still have a surplus pool, but at least all PGs are clean. I hope one of these will work. Please post your experience here. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 22 November 2020 18:29:16 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects On 10/23/20 3:07 AM, Frank Schilder wrote:

Hi Michael.

...

I still don't see any traffic to the pool, though I'm also unsure how much traffic is to be expected.

Probably not much. If ceph df shows that the pool contains some objects, I guess that's sorted.

That osdmaptool crashes indicates that your cluster runs with corrupted internal data. I tested your crush map and you should get complete PGs for the fs data pool. That you don't and that osdmaptool crashes points at a corruption of internal data. I'm afraid this is the point where you need support from ceph developers and should file a tracker report (https://tracker.ceph.com/projects/ceph/issues). A short description of the origin of the situation with the osdmaptool output and a reference to this thread linked in should be sufficient. Please post a link to the ticket here.

https://tracker.ceph.com/issues/48059

In parallel, you should probably open a new thread focussed on the osd map corruption. Maybe there are low-level commands to repair it.

Will do.

You should wait with trying to clean up the unfound objects until this is resolved. Not sure about adding further storage either. To me, this sounds quite serious.

Another approach that I'm considering is to create a new pool using the same set of OSDs, adding it to the set of cephfs data pools, and migrating the data from the "broken" pool to the new pool. I have some additional unused storage that I could add to this new pool, if I can figure out the right crush rules to make sure they don't get used for the "broken" pool too. --Mike

Dear Michael, yes, your plan will work if the temporary space requirement can be addressed. Good luck! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 22 November 2020 20:14:09 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects Hi Frank, From my understanding, with my current filesystem layout, I should be able to remove the "broken" pool once the data has been moved off of it. This is because the "broken" pool is not the default data pool. According to the documentation[1]: fs rm_data_pool <file system name> <pool name/id> "This command removes the specified pool from the list of data pools for the file system. If any files have layouts for the removed data pool, the file data will become unavailable. The default data pool (when creating the file system) cannot be removed." My default data pool (triply replicated on SSD) is still healthy. The "broken" pool is EC on HDD, and while it holds a majority of the filesystem data (~400TB), it is not the root of the filesystem. My plan would be: * Create a new data pool matching the "broken" pool * Create a parallel directory tree matching the directories that are mapped to the "broken" pool. eg Broken: /ceph/frames/..., New: /ceph/frames.new/... * Use 'setfattr -n ceph.dir.layout.pool' on this parallel directory tree to map the content to the new data pool * Use parallel+rsync to copy data from the broken pool to the new pool. * After each directory gets filled in the new pool, mv/rename the old and new directories so that users start accessing the data from the new pool. * Delete data from the renamed old pool directories as they are replaced, to keep the OSDs from filling up * After all data is moved off of the old pool (verified by checking ceph.dir.layout.pool and ceph.file.layout.pool on all files in the fs, as well as rados ls, ceph df), remove the pool from the fs. This is effectively the same strategy I did when moving frequently accessed directories from the EC pool to a replicated SSD pool, except that in the previous situation I didn't need to remove any pools at the end. It's time consuming, because every file on the "broken" pool needs to be copied, but it minimizes downtime. Being able to add some temporary new OSDs to the new pool (but not the "broken" pool) would reduce some pressure of filling up the OSDs. If the old and new pools use the same crush rule, would disabling backfilling+rebalancing keep the OSDs from being used in the old pool until the old pool is deleted (with the exception of the occasional new file)? --Mike [1]https://docs.ceph.com/en/latest/cephfs/administration/#file-systems On 11/22/20 12:19 PM, Frank Schilder wrote:

Dear Michael,

I was also wondering whether deleting the broken pool could clean up everything. The difficulty is, that while migrating a pool to new devices is easy via a crush rule change, migrating data between pools is not so easy. In particular, if you can't afford downtime.

In case you can afford some downtime, it might be possible to migrate fast by creating a new pool and use the pool copy command to migrate the data (rados cppool ...). Its important that the FS is shutdown (no MDS active) during this copy process. After copy, one could either rename the pools to have the copy match the fs data pool name, or change the data pool at the top level directory. You might need to set some pool meta data by hand, notably, the fs tag.

Having said that, I have no idea how a ceph fs reacts if presented with a replacement data pool. Although I don't believe that meta data contains the pool IDs, I cannot exclude that complication. The copy pool variant should be tested with an isolated FS first.

The other option is what you describe, create a new data pool, make the fs root placed on this pool and copy every file onto itself. This should also do the trick. However, with this method you will not be able to get rid of the broken pool. After the copy, you could, however, reduce the number of PGs to below the unhealthy one and the broken PG(s) might get deleted cleanly. Then you still have a surplus pool, but at least all PGs are clean.

I hope one of these will work. Please post your experience here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 22 November 2020 18:29:16 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

On 10/23/20 3:07 AM, Frank Schilder wrote:

...

Hi Michael.

...

I still don't see any traffic to the pool, though I'm also unsure how much traffic is to be expected.

Probably not much. If ceph df shows that the pool contains some objects, I guess that's sorted.

That osdmaptool crashes indicates that your cluster runs with corrupted internal data. I tested your crush map and you should get complete PGs for the fs data pool. That you don't and that osdmaptool crashes points at a corruption of internal data. I'm afraid this is the point where you need support from ceph developers and should file a tracker report (https://tracker.ceph.com/projects/ceph/issues). A short description of the origin of the situation with the osdmaptool output and a reference to this thread linked in should be sufficient. Please post a link to the ticket here.

https://tracker.ceph.com/issues/48059

...

In parallel, you should probably open a new thread focussed on the osd map corruption. Maybe there are low-level commands to repair it.

Will do.

...

You should wait with trying to clean up the unfound objects until this is resolved. Not sure about adding further storage either. To me, this sounds quite serious.

Another approach that I'm considering is to create a new pool using the same set of OSDs, adding it to the set of cephfs data pools, and migrating the data from the "broken" pool to the new pool.

I have some additional unused storage that I could add to this new pool, if I can figure out the right crush rules to make sure they don't get used for the "broken" pool too.

--Mike

Hi Michael, that sounds like a big step forward. I would probably remove the data pool from the ceph fs first before doing anything on it. Is the new pool set as data pool on the root of the entire ceph fs? If so, I see no reason for not detaching the pool from the ceph fs right away. Also to confirm that this goes without issues. Your choice though. This is a decisive moment and I would sweat as well. A shot of whiskey for the nerves maybe :) If you manage to "ceph fs rm_data_pool fs.data.archive.frames" the pool without problems, you are then safe to play with it. I think it might b a good idea to keep the broken pool for a while for debugging and not destroy any objects in it (or dump the objects/pool before changing). I'm a bit surprised that no developers seem to show interest in this case. The pool reduced to problematic objects only should hold interesting information about the original cause of the degradation. In a way, I really wonder how the pool delete will go. I guess there is still the problem with the OSD map that has broken PG information. The pool delete would be the last step that can lead to hiccups. I hope it goes away without taking anything with it. Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 15 December 2020 21:00:12 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects Hi Frank, I was able to migrate the data off of the "broken" pool (fs.data.archive.frames) and onto the new one (fs.data.archive.newframes). I verified that no useful data is left on the "broken" pool: * 'find + getfattr -n ceph.file.layout.pool' shows no files on the bad pool * 'find + getfattr -n ceph.dir.layout.pool' shows no future files will land on the bad pool * 'ceph -s' shows some misplaced/degraded/unfound objects on the bad pool: data: pools: 14 pools, 3492 pgs objects: 111.94M objects, 425 TiB usage: 587 TiB used, 525 TiB / 1.1 PiB avail pgs: 68/893408279 objects degraded (0.000%) 35/893408279 objects misplaced (0.000%) 24/111943463 objects unfound (0.000%) 3480 active+clean 5 active+recovery_unfound+degraded+remapped 4 active+clean+scrubbing+deep 2 active+recovery_unfound+undersized+degraded+remapped 1 active+recovery_unfound+degraded * 'rados ls --pool fs.data.archive.frames' shows these orphaned objects. I extracted the first component of the rados object names (eg 10000020fa1.00000030) and ran 'find /ceph -inum XXX' to verify that none of these objects maps back to a known file in the cephfs filesystem. Here are the next steps that I plan to perform: * 'rados rm --pool fs.data.archive.frames <obj_id>' on a couple of objects to see how ceph handles it. * 'rados purge fs.data.archive.frames' to purge all objects in the "broken" pool * ceph fs rm_data_pool fs.data.archive.frames Is there anything else you think I ought to check before finalizing the removal of this broken pool? --Mike On 11/22/20 1:59 PM, Frank Schilder wrote:

Dear Michael,

yes, your plan will work if the temporary space requirement can be addressed. Good luck!

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 22 November 2020 20:14:09 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

Hi Frank,

From my understanding, with my current filesystem layout, I should be able to remove the "broken" pool once the data has been moved off of it. This is because the "broken" pool is not the default data pool. According to the documentation[1]:

fs rm_data_pool <file system name> <pool name/id>

"This command removes the specified pool from the list of data pools for the file system. If any files have layouts for the removed data pool, the file data will become unavailable. The default data pool (when creating the file system) cannot be removed."

My default data pool (triply replicated on SSD) is still healthy. The "broken" pool is EC on HDD, and while it holds a majority of the filesystem data (~400TB), it is not the root of the filesystem.

My plan would be:

* Create a new data pool matching the "broken" pool * Create a parallel directory tree matching the directories that are mapped to the "broken" pool. eg Broken: /ceph/frames/..., New: /ceph/frames.new/... * Use 'setfattr -n ceph.dir.layout.pool' on this parallel directory tree to map the content to the new data pool * Use parallel+rsync to copy data from the broken pool to the new pool. * After each directory gets filled in the new pool, mv/rename the old and new directories so that users start accessing the data from the new pool. * Delete data from the renamed old pool directories as they are replaced, to keep the OSDs from filling up * After all data is moved off of the old pool (verified by checking ceph.dir.layout.pool and ceph.file.layout.pool on all files in the fs, as well as rados ls, ceph df), remove the pool from the fs.

This is effectively the same strategy I did when moving frequently accessed directories from the EC pool to a replicated SSD pool, except that in the previous situation I didn't need to remove any pools at the end. It's time consuming, because every file on the "broken" pool needs to be copied, but it minimizes downtime. Being able to add some temporary new OSDs to the new pool (but not the "broken" pool) would reduce some pressure of filling up the OSDs. If the old and new pools use the same crush rule, would disabling backfilling+rebalancing keep the OSDs from being used in the old pool until the old pool is deleted (with the exception of the occasional new file)?

--Mike [1]https://docs.ceph.com/en/latest/cephfs/administration/#file-systems

On 11/22/20 12:19 PM, Frank Schilder wrote:

...

Dear Michael,

I was also wondering whether deleting the broken pool could clean up everything. The difficulty is, that while migrating a pool to new devices is easy via a crush rule change, migrating data between pools is not so easy. In particular, if you can't afford downtime.

In case you can afford some downtime, it might be possible to migrate fast by creating a new pool and use the pool copy command to migrate the data (rados cppool ...). Its important that the FS is shutdown (no MDS active) during this copy process. After copy, one could either rename the pools to have the copy match the fs data pool name, or change the data pool at the top level directory. You might need to set some pool meta data by hand, notably, the fs tag.

Having said that, I have no idea how a ceph fs reacts if presented with a replacement data pool. Although I don't believe that meta data contains the pool IDs, I cannot exclude that complication. The copy pool variant should be tested with an isolated FS first.

The other option is what you describe, create a new data pool, make the fs root placed on this pool and copy every file onto itself. This should also do the trick. However, with this method you will not be able to get rid of the broken pool. After the copy, you could, however, reduce the number of PGs to below the unhealthy one and the broken PG(s) might get deleted cleanly. Then you still have a surplus pool, but at least all PGs are clean.

I hope one of these will work. Please post your experience here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 22 November 2020 18:29:16 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

On 10/23/20 3:07 AM, Frank Schilder wrote:

...

Hi Michael.

...

I still don't see any traffic to the pool, though I'm also unsure how much traffic is to be expected.

Probably not much. If ceph df shows that the pool contains some objects, I guess that's sorted.

That osdmaptool crashes indicates that your cluster runs with corrupted internal data. I tested your crush map and you should get complete PGs for the fs data pool. That you don't and that osdmaptool crashes points at a corruption of internal data. I'm afraid this is the point where you need support from ceph developers and should file a tracker report (https://tracker.ceph.com/projects/ceph/issues). A short description of the origin of the situation with the osdmaptool output and a reference to this thread linked in should be sufficient. Please post a link to the ticket here.

https://tracker.ceph.com/issues/48059

...

In parallel, you should probably open a new thread focussed on the osd map corruption. Maybe there are low-level commands to repair it.

Will do.

...

You should wait with trying to clean up the unfound objects until this is resolved. Not sure about adding further storage either. To me, this sounds quite serious.

Another approach that I'm considering is to create a new pool using the same set of OSDs, adding it to the set of cephfs data pools, and migrating the data from the "broken" pool to the new pool.

I have some additional unused storage that I could add to this new pool, if I can figure out the right crush rules to make sure they don't get used for the "broken" pool too.

--Mike

I left osd.41 out over the weekend, and put it back in this morning. After the recovery finished, here are the results of the ops queries: ceph daemon osd.41 ops: https://pastebin.com/keYBMVbH ceph daemon osd.41 dump_historic_slow_ops https://pastebin.com/axbZNh7M Yes, the OSD was still out when the previous health report was created. As for the host buckets, each host bucket corresponds to a physical host. --Mike On 10/16/20 8:41 AM, Frank Schilder wrote:

Dear Michael,

...

Please mark OSD 41 as "in" again and wait for some slow ops to show up.

I forgot. "wait for some slow ops to show up" ... and then what?

Could you please go to the host of the affected OSD and look at the output of "ceph daemon osd.ID ops" or "ceph daemon osd.ID dump_historic_slow_ops" and check what type of operations get stuck? I'm wondering if its administrative, like peering attempts.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder Sent: 16 October 2020 15:09:20 To: Michael Thomas; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

Dear Michael,

thanks for this initial work. I will need to look through the files you posted in more detail. In the meantime:

Please mark OSD 41 as "in" again and wait for some slow ops to show up. As far as I can see, marking it "out" might have cleared hanging slow ops (there were 1000 before), but they then started piling up again. From the OSD log it looks like an operation that is sent to/from PG 1.0, which doesn't respond because it is inactive. Hence, getting PG 1.0 active should resolve this issue (later).

Its a bit strange that I see slow ops for OSD 41 in the latest health detail (https://pastebin.com/3G3ij9ui). Was the OSD still out when this health report was created?

I think we might have misunderstood my question 6. My question was whether or not each host bucket corresponds to a physical host and vice versa, that is, each physical host has exactly 1 host bucket. I'm asking because it is possible to have multiple host buckets assigned to a single physical host and this has implications on how to manage things.

Coming back to PG 1.0 (the only PG in pool device_health_metrics as far as I can see), the problem is that is has no OSDs assigned. I need to look a bit longer at the data you uploaded to find out why. I can't see anything obvious.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 16 October 2020 02:08:01 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: multiple OSD crash, unfound objects

On 10/14/20 3:49 PM, Frank Schilder wrote:

...

Hi Michael,

it doesn't look too bad. All degraded objects are due to the undersized PG. If this is an EC pool with m>=2, data is currently not in danger.

I see a few loose ends to pick up, let's hope this is something simple. For any of the below, before attempting the next step, please wait until all induced recovery IO has completed before continuing.

1) Could you please paste the output of the following commands to pastebin (bash syntax):

ceph osd pool get device_health_metrics all

https://pastebin.com/6D83mjsV

...

ceph osd pool get fs.data.archive.frames all

https://pastebin.com/7XAaQcpC

...

ceph pg dump |& grep -i -e PG_STAT -e "^7.39d"

https://pastebin.com/tBLaq63Q

...

ceph osd crush rule ls

https://pastebin.com/6f5B778G

...

ceph osd erasure-code-profile ls

https://pastebin.com/uhAaMH1c

...

ceph osd crush dump # this is a big one, please be careful with copy-paste (see point 3 below)

https://pastebin.com/u92D23jV

...

2) I don't see any IO reported (neither user nor recovery). Could you please confirm that the command outputs were taken during a zero-IO period?

That's correct, there was no activity at this time. Access to the cephfs filesystem is very bursty, varying from completely idle to multiple GB/s (read).

...

3) Something is wrong with osd.41. Can you check its health status with smartctl? If it is reported healthy, give it one more clean restart. If the slow ops do not disappear, it could be a disk fail that is not detected by health monitoring. You could set it to "out" and see if the cluster recovers to a healthy state (modulo the currently degraded objects) with no slow ops. If so, I would replace the disk.

smartctl reports no problems.

osd.41 (and osd.0) was one of the original OSDs used for the device_health_metrics pool. Early on, before I knew better, I had removed this OSD (and osd.0) from the cluster, and the OSD ids got recycled when new disks were later added. This is when the slow ops on osd.0 and osd.41 started getting reported. On advice from another user on ceph-users, I updated my crush map to remap the device_health_metrics pool to a different set of OSDs (and the slow ops persisted).

osd.0 usually also shows slow ops. I was a little surprised that it didn't when I took this snapshot, but now it does.

I have now run 'ceph osd out 41', and the recovery I/O has finished. With the exception of one less OSD marked in, the output of 'ceph status' looks the same.

The last few lines of the osd.41 logfile are here:

https://pastebin.com/k06aArW4

How long does it take for ceph to clear the slow ops status?

...

4) In the output of "df tree" node141 shows up twice. Could you confirm that this is a copy-paste error or is this node indeed twice in the output? This is easiest to see in the pastebin when switching to "raw" view.

This was a copy/paste error.

...

5) The crush tree contains an empty host bucket (node308). Please delete this host bucket (ceph osd crush rm node308) for now and let me know if this caused any data movements (recovery IO).

This did not cause any data movement, according to 'ceph status'.

...

6) The crush tree looks a bit exotic. Do the nodes with a single OSD correspond to a physical host with 1 OSD disk? If not, could you please state how the host buckets are mapped onto physical hosts?

Each OSD corresponds to a single physical disk. Hosts may have 1, 2 or 3 OSDs of varying types (HDD, SSD, or SSD+NVME). There are a few different crush types used in the cluster:

3 x replicated nvme - used for cephfs metadata 3 x replicated SSD - used for ovirt block storage EC HDD - used for the bulk of the experiment data EC SSD - used for frequently accessed experiment data

...

7) In case there was a change to the health status, could you please include an updated "ceph health detail"?

Looks like the only difference is a new slow MDS op, and one PG that hasn't been deep scrubbed in the last week:

https://pastebin.com/3G3ij9ui

--Mike

...

I don't expect to get the incomplete PG resolved with the above, but it will move some issues out of the way before proceeding.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 14 October 2020 20:52:10 To: Andreas John; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Hello,

The original cause of the OSD instability has already been fixed. It was due to user jobs (via condor) consuming too much memory and causing the machine to swap. The OSDs didn't actually crash, but weren't responding in time and were being flagged as down.

In most cases, the problematic OSD servers were also not responding on the console and had to be physically power cycled to recover.

Since adding additional memory limits to user jobs, we have only had 1 or 2 unstable OSDs that were fixed by killing the remaining rogue user jobs.

Regards,

--Mike

On 10/10/20 9:22 AM, Andreas John wrote:

...

Hello Mike,

do your OSDs go down from time to time? I once has an issue with unrecoverable objects, because I had only n+1 (size 2) redundancy and ceph wasn't able to decide, what's the correct copy of the object. In my case there half-deleted snapshots in one of the copies. I used ceph-objectstoretool to remove the "wrong" part. Did you check you OSD logs? Do the osd go down wirth an obscure stacktrace (and maybe they are restartet by systemd ...)

rgds,

j.

On 09.10.20 22:33, Michael Thomas wrote:

...

Hi Frank,

That was a good tip. I was able to move the broken files out of the way and restore them for users. However, after 2 weeks I'm still left with unfound objects. Even more annoying, I now have 82k objects degraded (up from 74), which hasn't changed in over a week.

I'm ready to claim that the auto-repair capabilities of ceph are not able to fix my particular issues, and will have to continue to investigate alternate ways to clean this up, including a pg export/import (as you suggested) and perhaps a mds backward scrub (after testing in a junk pool first).

I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage.

One particular PG is in a curious state:

7.39d 82163 82165 246734 1 344060777807 0 0 2139 active+recovery_unfound+undersized+degraded+remapped 23m 50755'112549 50766:960500 [116,72,122,48,45,131,73,81]p116 [71,109,99,48,45,90,73,NONE]p71 2020-08-13T23:02:34.325887-0500 2020-08-07T11:01:45.657036-0500

Note the 'NONE' in the acting set. I do not know which OSD this may have been, nor how to find out. I suspect (without evidence) that this is part of the cause of no action on the degraded and misplaced objects.

--Mike

On 9/18/20 11:26 AM, Frank Schilder wrote:

...

Dear Michael,

maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing.

Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users@ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects

Dear Michael,

...

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most.

I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself.

The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls.

An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave?

Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet.

The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s).

I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation.

Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

On 9/18/20 2:50 AM, Frank Schilder wrote:

...

Dear Michael,

firstly, I'm a bit confused why you started deleting data. The objects were unfound, but still there. That's a small issue. Now the data might be gone and that's a real issue.

---------------------------- Interval:

Anyone reading this: I have seen many threads where ceph admins started deleting objects or PGs or even purging OSDs way too early from a cluster. Trying to recover health by deleting data is a contradiction. Ceph has bugs and sometimes it needs some help finding everything again. As far as I know, for most of these bugs there are workarounds that allow full recovery with a bit of work.

I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system.

This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed.

Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure.

...

First question is, did you delete the entire object or just a shard on one disk? Are there OSDs that might still have a copy?

Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/),

I ran:

ceph pg 7.1fb mark_unfound_lost delete

So I presume that the entire object has been deleted.

...

If the object is gone for good, the file references something that doesn't exist - its like a bad sector. You probably need to delete the file. Bit strange that the operation does not err out with a read error. Maybe it doesn't because it waits for the unfound objects state to be resolved?

Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object.

*Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours.

...

For all the other unfound objects, they are there somewhere - you didn't loose a disk or something. Try pushing ceph to scan the correct OSDs, for example, by restarting the newly added OSDs one by one or something similar. Sometimes exporting and importing a PG from one OSD to another forces a re-scan and subsequent discovery of unfound objects. It is also possible that ceph will find these objects along the way of recovery or when OSDs scrub or check for objects that can be deleted.

I have restarted the new OSDs countless times. I've used three different methods to restart the OSD:

* systemctl restart ceph-osd@120

* init 6

* ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120

I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain.

Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects.

--Mike

...

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Michael Thomas <wart@caltech.edu> Sent: 17 September 2020 22:27:47 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects

Hi Frank,

Yes, it does sounds similar to your ticket.

I've tried a few things to restore the failed files:

* Locate a missing object with 'ceph pg $pgid list_unfound'

* Convert the hex oid to a decimal inode number

* Identify the affected file with 'find /ceph -inum $inode'

At this point, I know which file is affected by the missing object. As expected, attempts to read the file simply hang. Unexpectedly, attempts to 'ls' the file or its containing directory also hang. I presume from this that the stat() system call needs some information that is contained in the missing object, and is waiting for the object to become available.

Next I tried to remove the affected object with:

* ceph pg $pgid mark_unfound_lost delete

Now 'ceph status' shows one fewer missing objects, but attempts to 'ls' or 'rm' the affected file continue to hang.

Finally, I ran a scrub over the part of the filesystem containing the affected file:

ceph tell mds.ceph4 scrub start /frames/postO3/hoft recursive

Nothing seemed to come up during the scrub:

2020-09-17T14:56:15.208-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T14:58:58.013-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub start {path=/frames/postO3/hoft,prefix=scrub start,scrubops=[recursive]} (starting...) 2020-09-17T14:58:58.013-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub queued for path: /frames/postO3/hoft 2020-09-17T14:58:58.014-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: active [paths:/frames/postO3/hoft] 2020-09-17T14:59:02.535-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:00:12.520-0500 7f39bca24700 1 mds.ceph4 asok_command: scrub status {prefix=scrub status} (starting...) 2020-09-17T15:02:32.944-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub complete with tag '1405e5c7-3ecf-4754-918e-129e9d101f7a' 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub completed for path: /frames/postO3/hoft 2020-09-17T15:02:32.945-0500 7f39b5215700 0 log_channel(cluster) log [INF] : scrub summary: idle

After the scrub completed, access to the file (ls or rm) continue to hang. The MDS reports slow reads:

2020-09-17T15:11:05.654-0500 7f39b9a1e700 0 log_channel(cluster) log [WRN] : slow request 481.867381 seconds old, received at 2020-09-17T15:03:03.788058-0500: client_request(client.451432:11309 getattr pAsLsXsFs #0x1000005b1c0 2020-09-17T15:03:03.787602-0500 caller_uid=0, caller_gid=0{}) currently dispatched

Does anyone have any suggestions on how else to clean up from a permanently lost object?

--Mike

On 9/16/20 2:03 AM, Frank Schilder wrote: > Sounds similar to this one: https://tracker.ceph.com/issues/46847 > > If you have or can reconstruct the crush map from before adding the > OSDs, you might be able to discover everything with the temporary > reversal of the crush map method. > > Not sure if there is another method, i never got a reply to my > question in the tracker. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Michael Thomas <wart@caltech.edu> > Sent: 16 September 2020 01:27:19 > To: ceph-users@ceph.io > Subject: [ceph-users] multiple OSD crash, unfound objects > > Over the weekend I had multiple OSD servers in my Octopus cluster > (15.2.4) crash and reboot at nearly the same time. The OSDs are > part of > an erasure coded pool. At the time the cluster had been busy with a > long-running (~week) remapping of a large number of PGs after I > incrementally added more OSDs to the cluster. After bringing all > of the > OSDs back up, I have 25 unfound objects and 75 degraded objects. > There > are other problems reported, but I'm primarily concerned with these > unfound/degraded objects. > > The pool with the missing objects is a cephfs pool. The files > stored in > the pool are backed up on tape, so I can easily restore individual > files > as needed (though I would not want to restore the entire filesystem). > > I tried following the guide at > https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg/.... > > I found a number of OSDs that are still 'not queried'. > Restarting a > sampling of these OSDs changed the state from 'not queried' to > 'already > probed', but that did not recover any of the unfound or degraded > objects. > > I have also tried 'ceph pg deep-scrub' on the affected PGs, but never > saw them get scrubbed. I also tried doing a 'ceph pg > force-recovery' on > the affected PGs, but only one seems to have been tagged accordingly > (see ceph -s output below). > > The guide also says "Sometimes it simply takes some time for the > cluster > to query possible locations."  I'm not sure how long "some time" might > take, but it hasn't changed after several hours. > > My questions are: > > * Is there a way to force the cluster to query the possible locations > sooner? > > * Is it possible to identify the files in cephfs that are affected, so > that I could delete only the affected files and restore them from > backup > tapes? > > --Mike > > ceph -s: > > cluster: > id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad > health: HEALTH_ERR > 1 clients failing to respond to capability release > 1 MDSs report slow requests > 25/78520351 objects unfound (0.000%) > 2 nearfull osd(s) > Reduced data availability: 1 pg inactive > Possible data damage: 9 pgs recovery_unfound > Degraded data redundancy: 75/626645098 objects > degraded > (0.000%), 9 pgs degraded > 1013 pgs not deep-scrubbed in time > 1013 pgs not scrubbed in time > 2 pool(s) nearfull > 1 daemons have recently crashed > 4 slow ops, oldest one blocked for 77939 sec, daemons > [osd.0,osd.41] have slow ops. > > services: > mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) > mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 > mds: archive:1 {0=ceph4=up:active} 3 up:standby > osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 > remapped pgs > > task status: > scrub status: > mds.ceph4: idle > > data: > pools: 9 pools, 2433 pgs > objects: 78.52M objects, 298 TiB > usage: 412 TiB used, 545 TiB / 956 TiB avail > pgs: 0.041% pgs unknown > 75/626645098 objects degraded (0.000%) > 135224/626645098 objects misplaced (0.022%) > 25/78520351 objects unfound (0.000%) > 2421 active+clean > 5 active+recovery_unfound+degraded > 3 active+recovery_unfound+degraded+remapped > 2 active+clean+scrubbing+deep > 1 unknown > 1 active+forced_recovery+recovery_unfound+degraded > > progress: > PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) > [............................] > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io