1 Nov
2020
1 Nov
'20
7:16 p.m.
Hi all
I have a Ceph cluster (Nautilus 14.2.11) with 3 Ceph nodes.
A crash happened and all 3 Ceph nodes went down.
One (1) PG turned "active+clean+inconsistent", I tried to repair it. After the
repair, now shows "active+clean+inconsistent+failed_repair" for the PG in the
question and cannot bring the cluster to "active+clean".
How do I rescue the cluster? Is this a false positive?
Here are the detail:
All three Ceph nodes run ceph-mon, ceph-mgr, ceph-osd and ceph-mds.
1. ceph -s
health: HEALTH_ERR 3 scrub errors Possible data damage: 1 pg
inconsistent
pgs: 191 active+clean 1 active+clean+inconsistent
2. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg
inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg
inconsistent pg 3.b is active+clean+inconsistent, acting [0,1,2]
3. rados list-inconsistent-pg rbd[]
4. ceph pg deep-scrub 3.b
5. ceph pg repair 3.b
6. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg
inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg
inconsistent pg 3.b is active+clean+inconsistent+failed_repair, acting [0,1,2]
7. rados list-inconsistent-obj 3.b --format=json-pretty{ "epoch": 4769,
"inconsistents": []}
8. ceph pg 3.b list_unfound { "num_missing": 0, "num_unfound":
0, "objects": [], "more": false}
Appreciate your help.
ThanksSagara
1 Nov
1 Nov
7:36 p.m.
I think this happens when a PG has 3 different copies and cannot decide which one is
correct. You might have hit a very rare case. You should start with the scrub errors,
check which PGs and which copies (OSDs) are affected. It sounds almost like all 3 scrub
errors are on the same PG.
You might have had a combination of crash and OSD fail, your situation is probably not
covered by "single point of failure".
In case you have a PG with scrub errors on 2 copies, you should be able to reconstruct the
PG from the third with PG export/PG import commands.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 01 November 2020 13:16:08
To: ceph-users(a)ceph.io
Subject: [ceph-users] How to recover from active+clean+inconsistent+failed_repair?
Hi all
I have a Ceph cluster (Nautilus 14.2.11) with 3 Ceph nodes.
A crash happened and all 3 Ceph nodes went down.
One (1) PG turned "active+clean+inconsistent", I tried to repair it. After the
repair, now shows "active+clean+inconsistent+failed_repair" for the PG in the
question and cannot bring the cluster to "active+clean".
How do I rescue the cluster? Is this a false positive?
Here are the detail:
All three Ceph nodes run ceph-mon, ceph-mgr, ceph-osd and ceph-mds.
1. ceph -s
health: HEALTH_ERR 3 scrub errors Possible data damage: 1 pg
inconsistent
pgs: 191 active+clean 1 active+clean+inconsistent
2. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg
inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg
inconsistent pg 3.b is active+clean+inconsistent, acting [0,1,2]
3. rados list-inconsistent-pg rbd[]
4. ceph pg deep-scrub 3.b
5. ceph pg repair 3.b
6. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg
inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg
inconsistent pg 3.b is active+clean+inconsistent+failed_repair, acting [0,1,2]
7. rados list-inconsistent-obj 3.b --format=json-pretty{ "epoch": 4769,
"inconsistents": []}
8. ceph pg 3.b list_unfound { "num_missing": 0, "num_unfound":
0, "objects": [], "more": false}
Appreciate your help.
ThanksSagara
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:05 p.m.
Hi Frank
Thanks for the reply.
I think this happens when a PG has 3 different copies
and cannot decide which one is correct. You might have hit a very rare case. You should
start with the scrub errors, check which PGs and which copies (OSDs) are affected. It
sounds almost like all 3 scrub errors are on the same PG.
Yes, all 3 errors are for
the same PG and on the same OSD:
2020-11-01 18:25:09.333339 osd.0 [ERR] 3.b shard 2 soid
3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate
had a missing info key
2020-11-01 18:25:09.333342 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head :
failed to pick suitable object info
2020-11-01 18:26:33.496255 osd.0 [ERR] 3.b repair 3 errors, 0 fixed
You might have had a combination of crash and OSD
fail, your situation is probably not covered by "single point of failure".
Yes it was a complex crash, all went down.
In case you have a PG with scrub errors on 2 copies,
you should be able to reconstruct the PG from the third with PG export/PG import commands.
I have not done a PG export/import before. Mind if you could send the instructions
or a link for it.
Thanks
Sagara
8:35 p.m.
Hi Sagara,
looks like your situation is more complex. Before doing anything potentially destructive,
you need to investigate some more. A possible interpretation (numbering just for the
example):
OSD 0 PG at version 1
OSD 1 PG at version 2
OSD 2 PG has scrub error
Depending on the version of the PG on OSD 2, either OSD 0 needs to roll forward (OSD 2 PG
at version 2), or OSD 1 needs to roll back (OSD 2 PG at version 1). Part of the relevant
information on OSD 2 seems to be unreadable, therefore pg repair bails out.
You need to find out if you are in this situation or some other case. If you are, you need
to find out somehow if you need to roll back or forward. I'm afraid in your current
situation, even taking the OSD with the scrub errors down will not rebuild the PG.
I would probably try:
- find out with smartctl if the OSD with scrub errors is in a pre-fail state (has remapped
sectors)
- if it is:
* take it down and try to make a full copy with ddrescue
* if ddrescure manages to copy everything, copy back to a new disk and add to ceph
* if ddrescue fails to copy everything, you could try if badblocks manages to get the
disk back; ddrescue can force remappings of broken sectors (non-destructive read-write
check) and it can happen that data becomes readable again, exchange the disk as soon as
possible thereafter
- if the disk is healthy:
* try to find out if you can deduce the state of the copies on every OSD
The tool for low-level operations is bluestore-tool. I never used it, so you need to look
at the documentation.
If everything fails, I guess your last option is to decide for one of the copies, export
it from one OSD and inject it to another one (but not any of 0,1,2!). This will establish
2 identical copies and the third one will be changed to this one automatically. Note that
this may lead to data loss on objects that were in the undefined state. As far as I can
see, its only 1 object and probably possible to recover from (backup, snapshot).
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 01 November 2020 14:05:36
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?
Hi Frank
Thanks for the reply.
I think this happens when a PG has 3 different copies
and cannot decide which one is correct. You might have hit a very rare case. You should
start with the scrub errors, check which PGs and which copies (OSDs) are affected. It
sounds almost like all 3 scrub errors are on the same PG.
Yes, all 3 errors are for
the same PG and on the same OSD:
2020-11-01 18:25:09.333339 osd.0 [ERR] 3.b shard 2 soid
3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate
had a missing info key
2020-11-01 18:25:09.333342 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head :
failed to pick suitable object info
2020-11-01 18:26:33.496255 osd.0 [ERR] 3.b repair 3 errors, 0 fixed
You might have had a combination of crash and OSD
fail, your situation is probably not covered by "single point of failure".
Yes it was a complex crash, all went down.
In case you have a PG with scrub errors on 2 copies,
you should be able to reconstruct the PG from the third with PG export/PG import commands.
I have not done a PG export/import before. Mind if you could send the instructions
or a link for it.
Thanks
Sagara
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:37 p.m.
sorry: *badblocks* can force remappings of broken sectors (non-destructive read-write
check)
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 01 November 2020 14:35:35
To: Sagara Wijetunga; ceph-users(a)ceph.io
Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?
Hi Sagara,
looks like your situation is more complex. Before doing anything potentially destructive,
you need to investigate some more. A possible interpretation (numbering just for the
example):
OSD 0 PG at version 1
OSD 1 PG at version 2
OSD 2 PG has scrub error
Depending on the version of the PG on OSD 2, either OSD 0 needs to roll forward (OSD 2 PG
at version 2), or OSD 1 needs to roll back (OSD 2 PG at version 1). Part of the relevant
information on OSD 2 seems to be unreadable, therefore pg repair bails out.
You need to find out if you are in this situation or some other case. If you are, you need
to find out somehow if you need to roll back or forward. I'm afraid in your current
situation, even taking the OSD with the scrub errors down will not rebuild the PG.
I would probably try:
- find out with smartctl if the OSD with scrub errors is in a pre-fail state (has remapped
sectors)
- if it is:
* take it down and try to make a full copy with ddrescue
* if ddrescure manages to copy everything, copy back to a new disk and add to ceph
* if ddrescue fails to copy everything, you could try if badblocks manages to get the
disk back; ddrescue can force remappings of broken sectors (non-destructive read-write
check) and it can happen that data becomes readable again, exchange the disk as soon as
possible thereafter
- if the disk is healthy:
* try to find out if you can deduce the state of the copies on every OSD
The tool for low-level operations is bluestore-tool. I never used it, so you need to look
at the documentation.
If everything fails, I guess your last option is to decide for one of the copies, export
it from one OSD and inject it to another one (but not any of 0,1,2!). This will establish
2 identical copies and the third one will be changed to this one automatically. Note that
this may lead to data loss on objects that were in the undefined state. As far as I can
see, its only 1 object and probably possible to recover from (backup, snapshot).
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 01 November 2020 14:05:36
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?
Hi Frank
Thanks for the reply.
I think this happens when a PG has 3 different copies
and cannot decide which one is correct. You might have hit a very rare case. You should
start with the scrub errors, check which PGs and which copies (OSDs) are affected. It
sounds almost like all 3 scrub errors are on the same PG.
Yes, all 3 errors are for
the same PG and on the same OSD:
2020-11-01 18:25:09.333339 osd.0 [ERR] 3.b shard 2 soid
3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate
had a missing info key
2020-11-01 18:25:09.333342 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head :
failed to pick suitable object info
2020-11-01 18:26:33.496255 osd.0 [ERR] 3.b repair 3 errors, 0 fixed
You might have had a combination of crash and OSD
fail, your situation is probably not covered by "single point of failure".
Yes it was a complex crash, all went down.
In case you have a PG with scrub errors on 2 copies,
you should be able to reconstruct the PG from the third with PG export/PG import commands.
I have not done a PG export/import before. Mind if you could send the instructions
or a link for it.
Thanks
Sagara
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Nov
2 Nov
1:50 p.m.
Hi Sagara,
I'm not sure if my hypothesis can be correct. Ceph sends an acknowledge of a write
only after all copies are on disk. In other words, if PGs end up on different versions
after a power outage, one always needs to roll back. Since you have two healthy OSDs in
the PG and the PG is active (successfully peered), it might just be a broken disk and
read/write errors. I would focus on that.
Another question, do you have write caches enabled (disk cache and controller cache)? This
is know to cause problems on power outages and also degraded performance with ceph. You
should check and disable any caches if necessary.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 01 November 2020 14:37:41
To: Sagara Wijetunga; ceph-users(a)ceph.io
Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?
sorry: *badblocks* can force remappings of broken sectors (non-destructive read-write
check)
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 01 November 2020 14:35:35
To: Sagara Wijetunga; ceph-users(a)ceph.io
Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?
Hi Sagara,
looks like your situation is more complex. Before doing anything potentially destructive,
you need to investigate some more. A possible interpretation (numbering just for the
example):
OSD 0 PG at version 1
OSD 1 PG at version 2
OSD 2 PG has scrub error
Depending on the version of the PG on OSD 2, either OSD 0 needs to roll forward (OSD 2 PG
at version 2), or OSD 1 needs to roll back (OSD 2 PG at version 1). Part of the relevant
information on OSD 2 seems to be unreadable, therefore pg repair bails out.
You need to find out if you are in this situation or some other case. If you are, you need
to find out somehow if you need to roll back or forward. I'm afraid in your current
situation, even taking the OSD with the scrub errors down will not rebuild the PG.
I would probably try:
- find out with smartctl if the OSD with scrub errors is in a pre-fail state (has remapped
sectors)
- if it is:
* take it down and try to make a full copy with ddrescue
* if ddrescure manages to copy everything, copy back to a new disk and add to ceph
* if ddrescue fails to copy everything, you could try if badblocks manages to get the
disk back; ddrescue can force remappings of broken sectors (non-destructive read-write
check) and it can happen that data becomes readable again, exchange the disk as soon as
possible thereafter
- if the disk is healthy:
* try to find out if you can deduce the state of the copies on every OSD
The tool for low-level operations is bluestore-tool. I never used it, so you need to look
at the documentation.
If everything fails, I guess your last option is to decide for one of the copies, export
it from one OSD and inject it to another one (but not any of 0,1,2!). This will establish
2 identical copies and the third one will be changed to this one automatically. Note that
this may lead to data loss on objects that were in the undefined state. As far as I can
see, its only 1 object and probably possible to recover from (backup, snapshot).
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 01 November 2020 14:05:36
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?
Hi Frank
Thanks for the reply.
I think this happens when a PG has 3 different copies
and cannot decide which one is correct. You might have hit a very rare case. You should
start with the scrub errors, check which PGs and which copies (OSDs) are affected. It
sounds almost like all 3 scrub errors are on the same PG.
Yes, all 3 errors are for
the same PG and on the same OSD:
2020-11-01 18:25:09.333339 osd.0 [ERR] 3.b shard 2 soid
3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate
had a missing info key
2020-11-01 18:25:09.333342 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head :
failed to pick suitable object info
2020-11-01 18:26:33.496255 osd.0 [ERR] 3.b repair 3 errors, 0 fixed
You might have had a combination of crash and OSD
fail, your situation is probably not covered by "single point of failure".
Yes it was a complex crash, all went down.
In case you have a PG with scrub errors on 2 copies,
you should be able to reconstruct the PG from the third with PG export/PG import commands.
I have not done a PG export/import before. Mind if you could send the instructions
or a link for it.
Thanks
Sagara
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:10 p.m.
Hi Frank
I'm not sure if my hypothesis can be correct. Ceph
sends an acknowledge of a write only after all copies are on disk. In other words, if PGs
end up on different versions after a power outage, one always needs to roll back. Since
you have two healthy OSDs in the PG and the PG is active (successfully peered), it might
just be a broken disk and read/write errors. I would focus on that.
I tried to revert the PG as follows:
# ceph pg 3.b query | grep version "last_user_version": 2263481,
"version": "4825'2264303",
"last_user_version": 2263481, "version":
"4825'2264301",
"last_user_version": 2263481, "version":
"4825'2264301",
ceph pg 3.b list_unfound
{ "num_missing": 0, "num_unfound": 0, "objects":
[], "more": false}
# ceph pg 3.b mark_unfound_lost revertpg has no unfound objects
# ceph pg 3.b revertInvalid command: revert not in querypg <pgid> query : show
details of a specific pgError EINVAL: invalid command
How to revert/rollback a PG?
Another question, do you have write caches enabled
(disk cache and controller cache)? This is know to cause problems on power outages and
also degraded performance with ceph. You should check and disable any caches if necessary.
No. HDD is directly connected to motherboard.
Thank you
Sagara
4:31 p.m.
Hi Sagra,
looks like you have one on a new and 2 on an old version. Can you add the information
about which OSD each version resides?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 02 November 2020 10:10:02
To: ceph-users(a)ceph.io; Frank Schilder
Subject: Re: [ceph-users] Re: How to recover from
active+clean+inconsistent+failed_repair?
Hi Frank
I'm not sure if my hypothesis can be correct. Ceph
sends an acknowledge of a write only after all copies are on disk. In other words, if PGs
end up on different versions after a power outage, one always needs to roll back. Since
you have two healthy OSDs in the PG and the PG is active (successfully peered), it might
just be a broken disk and read/write errors. I would focus on that.
I tried to revert the PG as follows:
# ceph pg 3.b query | grep version
"last_user_version": 2263481,
"version": "4825'2264303",
"last_user_version": 2263481,
"version": "4825'2264301",
"last_user_version": 2263481,
"version": "4825'2264301",
ceph pg 3.b list_unfound
{
"num_missing": 0,
"num_unfound": 0,
"objects": [],
"more": false
}
# ceph pg 3.b mark_unfound_lost revert
pg has no unfound objects
# ceph pg 3.b revert
Invalid command: revert not in query
pg <pgid> query : show details of a specific pg
Error EINVAL: invalid command
How to revert/rollback a PG?
Another question, do you have write caches enabled
(disk cache and controller cache)? This is know to cause problems on power outages and
also degraded performance with ceph. You should check and disable any caches if necessary.
No. HDD is directly connected to motherboard.
Thank you
Sagara
5:02 p.m.
Hi Frank
looks like you have one on a new and 2 on an old
version. Can you add the information about which OSD each version resides?
The "ceph pg 3.b query" shows following:
"peer_info": [
{
"peer": "1",
"pgid": "3.b",
"last_update": "4825'2264303",
"last_complete": "4825'2264303",
"log_tail": "4759'2261298",
"last_user_version": 2263481,
:
:
"stats": {
"version": "4825'2264301",
}
},
{
"peer": "2",
"pgid": "3.b",
"last_update": "4825'2264303",
"last_complete": "4825'2264303",
"log_tail": "4759'2261298",
"last_user_version": 2263481,
:
:
"stats": {
"version": "4825'2264301",
}
}
Please note, there is no peer 0 in "ceph pg 3.b query". Also no word osd.
Is "peer": "1" means osd.1?
I have osd.0, osd.1 and osd2.
Note, version "4825'2264303" and "4825'2264301" appear in both
above peer 1 and 2.
Thanks.
Sagara
7:15 p.m.
Hi Sagara,
the primary OSD is probably not listed as a peer. Can you post the complete output of
- ceph pg 3.b query
- ceph pg dump
- ceph osd df tree
in a pastebin?
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 02 November 2020 11:53:58
To: ceph-users(a)ceph.io; Frank Schilder
Subject: Re: [ceph-users] Re: How to recover from
active+clean+inconsistent+failed_repair?
Hi Frank
Please note, there is no peer 0 in "ceph pg 3.b
query". Also no word osd.
I checked other PGs with "active+clean", there is a "peer":
"0".
But "ceph pg pgid query" always shows only two peers, sometime peer 0 and 1, or
1 and 2, 0 and 2, etc.
Regards
Sagara
8:25 p.m.
Hi Frank
the primary OSD is probably not listed as a peer. Can
you post the complete output of
- ceph pg 3.b query
- ceph pg dump
- ceph osd df tree
in a pastebin?
Yes, the Primary OSD is 0.
I have attached above as .txt files. Please let me know if you still cannot read them.
Regards
Sagara
9:11 p.m.
Hmm, I'm getting a bit confused. Could you also send the output of "ceph osd pool
ls detail".
Did you look at the disk/controller cache settings?
I think you should start a deep-scrub with "ceph pg deep-scrub 3.b" and record
the output of "ceph -w | grep '3\.b'" (note the single quotes).
The error messages you included in one of your first e-mails are only on 1 out of 3 scrub
errors (3 lines for 1 error). We need to find all 3 errors.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 02 November 2020 14:25:08
To: ceph-users(a)ceph.io; Frank Schilder
Subject: Re: [ceph-users] Re: How to recover from
active+clean+inconsistent+failed_repair?
Hi Frank
the primary OSD is probably not listed as a peer. Can
you post the complete output of
- ceph pg 3.b query
- ceph pg dump
- ceph osd df tree
in a pastebin?
Yes, the Primary OSD is 0.
I have attached above as .txt files. Please let me know if you still cannot read them.
Regards
Sagara
10:08 p.m.
Hmm, I'm getting a bit confused. Could you also
send the output of "ceph osd pool ls detail".
File ceph-osd-pool-ls-detail.txt attached.
Did you look at the disk/controller cache settings?
I don't have disk controllers on Ceph machines. The hard disk is directly
attached to the motherboard via SATA cable. But there can be a on chip disk controller on
the motherboard, I'm not sure.
If your worry is fsync persistence, I have thoroughly tested database fsync reliability on
Ceph RBD with hundreds of transactions per second and remove network cable and restart the
database machine, etc. while inserts going on. and I did not lose a single transaction. I
simulated this many times and persistence on my Ceph cluster was perfect (i.e not a single
loss).
I think you should start a deep-scrub with "ceph
pg deep-scrub 3.b" and record the output of "ceph -w | grep '3\.b'"
(note the single quotes).
The error messages you included in one of your first
e-mails are only on 1 out of 3 scrub errors (3 lines for 1 error). We need to find all 3
errors.
I ran again the "ceph pg deep-scrub 3.b", here is the whole output of ceph -w:
2020-11-02 22:33:48.224392 osd.0 [ERR] 3.b shard 2 soid
3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate
had a missing info key
2020-11-02 22:33:48.224396 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head :
failed to pick suitable object info
2020-11-02 22:35:30.087042 osd.0 [ERR] 3.b deep-scrub 3 errors
Btw, I'm very grateful for your perseverance on this.
Best regards
Sagara
10:29 p.m.
But there can be a on chip disk controller on the
motherboard, I'm not sure.
There is always some kind of controller. Could be on-board. Usually, the cache settings
are accessible when booting into the BIOS set-up.
If your worry is fsync persistence
No, what I worry about is volatile write cache, which is usually enabled by default. This
cache exists on disk as well as on controller. To avoid loosing writes on power fail, the
controller needs to be in write-through mode and the disk write cache disabled. The latter
can be done with smartctl, the former in the BIOS setup.
Did you test power failure? If so, how often? On how many hosts simultaneously? Pulling
network cables will not trigger cache related problems. The problem with write cache is,
that you rely on a lot of bells and whistles where some usually fail. With ceph, this will
lead to exactly the problem you are observing now.
Your pool configuration looks OK. You need to find out where exactly the scrub errors are
situated. It looks like meta-data damage and you might loose some data. Be careful to do
only read-only admin operations for now.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 02 November 2020 16:08:58
To: ceph-users(a)ceph.io; Frank Schilder
Subject: Re: [ceph-users] Re: How to recover from
active+clean+inconsistent+failed_repair?
Hmm, I'm getting a bit confused. Could you also
send the output of "ceph osd pool ls detail".
File ceph-osd-pool-ls-detail.txt attached.
Did you look at the disk/controller cache settings?
I don't have disk controllers on Ceph machines. The hard disk is directly attached to
the motherboard via SATA cable. But there can be a on chip disk controller on the
motherboard, I'm not sure.
If your worry is fsync persistence, I have thoroughly tested database fsync reliability on
Ceph RBD with hundreds of transactions per second and remove network cable and restart the
database machine, etc. while inserts going on. and I did not lose a single transaction. I
simulated this many times and persistence on my Ceph cluster was perfect (i.e not a single
loss).
I think you should start a deep-scrub with "ceph
pg deep-scrub 3.b" and record the output of "ceph -w | grep '3\.b'"
(note the single quotes).
The error messages you included in one of your first
e-mails are only on 1 out of 3 scrub errors (3 lines for 1 error). We need to find all 3
errors.
I ran again the "ceph pg deep-scrub 3.b", here is the whole output of ceph -w:
2020-11-02 22:33:48.224392 osd.0 [ERR] 3.b shard 2 soid
3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate
had a missing info key
2020-11-02 22:33:48.224396 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head :
failed to pick suitable object info
2020-11-02 22:35:30.087042 osd.0 [ERR] 3.b deep-scrub 3 errors
Btw, I'm very grateful for your perseverance on this.
Best regards
Sagara
3 Nov
3 Nov
4:39 p.m.
Hi Frank
1. We will disable the disk controller and disk-level caching to avoid future issues.
2. My pools are:
ceph osd lspools
2 cephfs_metadata
3 cephfs_data
4 rbd
The PG now inconsistent is 3.b, therefore, it belongs to cephfs_data pool.
Following also shows the PG 3.b belongs to cephfs_data:
ceph pg ls-by-pool cephfs_data | grep 3.b
3.b 6992 0 0 0 9649392528 0 0 3005
active+clean+inconsistent ...
3. Deep scrubs shows only one object having an issue: soid
3:d577e975:::1000023675e.00000000
This object seems lost.
rados -p cephfs_metadata ls | grep 1000023675e.00000000
rados -p cephfs_data ls | grep 1000023675e.00000000
rados -p rbd ls | grep 1000023675e.00000000
4. I tried to find what are the files effected by this issue, but I get "No such file
or directory" for the path. I have properly mounted ceph on home as before.
cephfs-data-scan -c /etc/ceph/ceph.conf pg_files /home/sagara
3.b2020-11-03T17:06:21.770+0800 7f3f213ab100 -1 pgeffects.hit_dir: Failed to open path:
(2) No such file or directory
How do I see what are the files effected by this issue?
5. What should be the course of the action now to bring the cluster to
"active+clean" to move forward? I don't mind roll back the PG having the
issue. I have a file-level backup. If roll back the PG is the way forward, how to do?
Thank you.
Best regards
Sagara
On Monday, November 2, 2020, 11:29:55 PM GMT+8, Frank Schilder <frans(a)dtu.dk>
wrote:
But there can be a on chip disk controller on the
motherboard, I'm not sure.
There is always some kind of controller. Could be on-board. Usually, the cache settings
are accessible when booting into the BIOS set-up.
If your worry is fsync persistence
No, what I worry about is volatile write cache, which is usually enabled by default. This
cache exists on disk as well as on controller. To avoid loosing writes on power fail, the
controller needs to be in write-through mode and the disk write cache disabled. The latter
can be done with smartctl, the former in the BIOS setup.
Did you test power failure? If so, how often? On how many hosts simultaneously? Pulling
network cables will not trigger cache related problems. The problem with write cache is,
that you rely on a lot of bells and whistles where some usually fail. With ceph, this will
lead to exactly the problem you are observing now.
Your pool configuration looks OK. You need to find out where exactly the scrub errors are
situated. It looks like meta-data damage and you might loose some data. Be careful to do
only read-only admin operations for now.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
4:59 p.m.
Hi Sagara,
did you collect the output of "ceph -w | grep '3\.b'" while the PG was
deep-scrubbed? Please open a new thread with the error message in the subject line. This
should draw the attention of someone with internal knowledge. Include a short description
of how this situation came about with the deep-scrub errors and log messages.
The problem is, that no objects are reported as lost or damaged. Therefore, I do not
believe that the usual actions will help, which would include things like marking unfound
objects as lost. This doesn't apply to your case.
If you know where data access fails on the file system, you can move the folder to a
"parking spot" and restore the data from backup. This would allow your users to
continue working while you buy time to fix the PG.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Sagara Wijetunga <sagarawmw(a)yahoo.com>
Sent: 03 November 2020 10:39:18
To: ceph-users(a)ceph.io; Frank Schilder
Subject: Re: [ceph-users] Re: How to recover from
active+clean+inconsistent+failed_repair?
Hi Frank
1. We will disable the disk controller and disk-level caching to avoid future issues.
2. My pools are:
ceph osd lspools
2 cephfs_metadata
3 cephfs_data
4 rbd
The PG now inconsistent is 3.b, therefore, it belongs to cephfs_data pool.
Following also shows the PG 3.b belongs to cephfs_data:
ceph pg ls-by-pool cephfs_data | grep 3.b
3.b 6992 0 0 0 9649392528 0 0 3005
active+clean+inconsistent ...
3. Deep scrubs shows only one object having an issue: soid
3:d577e975:::1000023675e.00000000
This object seems lost.
rados -p cephfs_metadata ls | grep 1000023675e.00000000
rados -p cephfs_data ls | grep 1000023675e.00000000
rados -p rbd ls | grep 1000023675e.00000000
4. I tried to find what are the files effected by this issue, but I get "No such file
or directory" for the path. I have properly mounted ceph on home as before.
cephfs-data-scan -c /etc/ceph/ceph.conf pg_files /home/sagara 3.b
2020-11-03T17:06:21.770+0800 7f3f213ab100 -1 pgeffects.hit_dir: Failed to open path: (2)
No such file or directory
How do I see what are the files effected by this issue?
5. What should be the course of the action now to bring the cluster to
"active+clean" to move forward? I don't mind roll back the PG having the
issue. I have a file-level backup. If roll back the PG is the way forward, how to do?
Thank you.
Best regards
Sagara
On Monday, November 2, 2020, 11:29:55 PM GMT+8, Frank Schilder <frans(a)dtu.dk>
wrote:
But there can be a on chip disk controller on the
motherboard, I'm not sure.
There is always some kind of controller. Could be on-board. Usually, the cache settings
are accessible when booting into the BIOS set-up.
If your worry is fsync persistence
No, what I worry about is volatile write cache, which is usually enabled by default. This
cache exists on disk as well as on controller. To avoid loosing writes on power fail, the
controller needs to be in write-through mode and the disk write cache disabled. The latter
can be done with smartctl, the former in the BIOS setup.
Did you test power failure? If so, how often? On how many hosts simultaneously? Pulling
network cables will not trigger cache related problems. The problem with write cache is,
that you rely on a lot of bells and whistles where some usually fail. With ceph, this will
lead to exactly the problem you are observing now.
Your pool configuration looks OK. You need to find out where exactly the scrub errors are
situated. It looks like meta-data damage and you might loose some data. Be careful to do
only read-only admin operations for now.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
1270
days inactive
1272
days old
17 comments
2 participants
participants (2)
-
Frank Schilder -
Sagara Wijetunga