[ceph-users] Re: multiple OSD crash, unfound objects

I have other tasks I need to perform on the filesystem (removing OSDs, adding new OSDs, increasing PG count), but I feel like I need to address these degraded/lost objects before risking any more damage.

Dear Michael, maybe there is a way to restore access for users and solve the issues later. Someone else with a lost/unfound object was able to move the affected file (or directory containing the file) to a separate location and restore the now missing data from backup. This will "park" the problem of cluster health for later fixing. Best regads, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans(a)dtu.dk> Sent: 18 September 2020 15:38:51 To: Michael Thomas; ceph-users(a)ceph.io Subject: [ceph-users] Re: multiple OSD crash, unfound objects Dear Michael, I would agree with that if the data was deleted using the appropriate high-level operation. Deleting an unfound object is like marking a sector on a disk as bad with smartctl. How should the file system react to that? Purging an OSD is like removing a disk from a raid set. Such operations increase inconsistencies/degradation rather than resolving them. Cleaning this up also requires to execute other operations to remove all references to the object and, finally, the file inode itself. The ls on a dir with corrupted file(s) hangs if ls calls stat on every file. For example, when coloring is enabled, ls will stat every file in the dir to be able to choose the color according to permissions. If one then disables coloring, a plain "ls" will return all names while an "ls -l" will hang due to stat calls. An "rm" or "rm -f" should succeed if the folder permissions allow that. It should not stat the file itself, so it sounds a bit odd that its hanging. I guess in some situations it does, like "rm -i", which will ask before removing read-only files. How does "unlink FILE" behave? Most admin commands on ceph are asynchronous. A command like "pg repair" or "osd scrub" only schedules an operation. The command "ceph pg 7.1fb mark_unfound_lost delete" does probably just the same. Unfortunately, I don't know how to check that a scheduled operation has started/completed/succeeded/failed. I asked this in an earlier thread (about PG repair) and didn't get an answer. On our cluster, the actual repair happened ca. 6-12 hours after scheduling (on a healthy cluster!). I would conclude that (some of) these operations have very low priority and will not start at least as long as there is recovery going on. One might want to consider the possibility that some of the scheduled commands have not been executed yet. The output of "pg query" contains the IDs of the missing objects (in mimic) and each of these objects is on one of the peer OSDs of the PG (I think object here refers to shard or copy). It should be possible to find the corresponding OSD (or at least obtain confirmation that the object is really gone) and move the object to a place where it is expected to be found. This can probably be achieved with "PG export" and "PG import". I don't know of any other way(s). I guess, in the current situation, sitting it out a bit longer might be a good strategy. I don't know how many asynchronous commands you executed and giving the cluster time to complete these jobs might improve the situation. Sorry that I can't be of more help here. However, if you figure out a solution (ideally non-destructive), please post it here. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Michael Thomas <wart(a)caltech.edu> Sent: 18 September 2020 14:15:53 To: Frank Schilder; ceph-users(a)ceph.io Subject: Re: [ceph-users] multiple OSD crash, unfound objects Hi Frank, On 9/18/20 2:50 AM, Frank Schilder wrote: I disagree with the statement that trying to recover health by deleting data is a contradiction. In some cases (such as mine), the data in ceph is backed up in another location (eg tape library). Restoring a few files from tape is a simple and cheap operation that takes a minute, at most. For the sake of expediency, sometimes it's quicker and easier to simply delete the affected files and restore them from the backup system. This procedure has worked fine with our previous distributed filesystem (hdfs), so I (naively?) thought that it could be used with ceph as well. I was a bit surprised that cephs behavior was to indefinitely block the 'rm' operation so that the affected file could not even be removed. Since I have 25 unfound objects spread across 9 PGs, I used a PG with a single unfound object to test this alternate recovery procedure. Per the troubleshooting guide (https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/), I ran: ceph pg 7.1fb mark_unfound_lost delete So I presume that the entire object has been deleted. Even before the object was removed, all read operations on the file would hang. Even worse, attempts to stat() the file with commands such as 'ls' or 'rm' would hang. Even worse, attempts to 'ls' in the directory itself would hang. This hasn't changed after removing the object. *Update*: The stat() operations may not be hanging indefinitely. It seems to hang for somewhere between 10 minutes and 8 hours. I have restarted the new OSDs countless times. I've used three different methods to restart the OSD: * systemctl restart ceph-osd@120 * init 6 * ceph osd out 120 ...wait for repeering to finish... systemctl restart ceph-osd@120 ceph osd in 120 I've done this for all OSDs that a PG has listed in the 'not queried' state in 'ceph pg $pgid detail'. But even when all OSDs in the PG are back to the 'already probed' state, the missing objects remain. Over 90% of my PGs have not been deep scrubbed recently, due to the amount of backfilling and importing of data into the ceph cluster. I plan to leave the cluster mostly idle over the weekend so that hopefully the deep scrubs can catch up and possibly locate any missing objects. --Mike _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io