Hi Sake,
On Tue, Jan 2, 2024 at 4:02 AM Sake Ceph <ceph(a)paulusma.eu> wrote:
Hi again, hopefully for the last time with problems.
We had a MDS crash earlier with the MDS staying in failed state and used a command to
reset the filesystem (this was wrong, I know now, thanks Patrick Donnelly for pointing
this out). I did a full scrub on the filesystem and two files were damaged. One of those
got repaired, but the following file keeps giving errors and can't be removed.
What can I do now? Below some information.
# ceph tell mds.atlassian-prod:0 damage ls
[
{
"damage_type": "backtrace",
"id": 2244444901,
"ino": 1099534008829,
"path":
"/app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01"
}
]
Trying to repair the error (online research shows this should work for a backtrace damage
type)
----------
# ceph tell mds.atlassian-prod:0 scrub start /app1/shared/data/repositories/11271
recursive,repair,force
{
"return_code": 0,
"scrub_tag": "d10ead42-5280-4224-971e-4f3022e79278",
"mode": "asynchronous"
}
Cluster logs after this
----------
1/2/24 9:37:05 AM
[INF]
scrub summary: idle
1/2/24 9:37:02 AM
[INF]
scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271]
1/2/24 9:37:01 AM
[INF]
scrub summary: active paths [/app1/shared/data/repositories/11271]
1/2/24 9:37:01 AM
[INF]
scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271]
1/2/24 9:37:01 AM
[INF]
scrub queued for path: /app1/shared/data/repositories/11271
But the error doesn't disappear and still can't remove the file.
On the client trying to remove the file (we got a backup)
----------
$ rm -f
/mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01
rm: cannot remove
'/mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01':
Input/output error
Did you try `damage rm <id>` after scrubbing?
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D