Hi Sagara, looks like your situation is more complex. Before doing anything potentially destructive, you need to investigate some more. A possible interpretation (numbering just for the example): OSD 0 PG at version 1 OSD 1 PG at version 2 OSD 2 PG has scrub error Depending on the version of the PG on OSD 2, either OSD 0 needs to roll forward (OSD 2 PG at version 2), or OSD 1 needs to roll back (OSD 2 PG at version 1). Part of the relevant information on OSD 2 seems to be unreadable, therefore pg repair bails out. You need to find out if you are in this situation or some other case. If you are, you need to find out somehow if you need to roll back or forward. I'm afraid in your current situation, even taking the OSD with the scrub errors down will not rebuild the PG. I would probably try: - find out with smartctl if the OSD with scrub errors is in a pre-fail state (has remapped sectors) - if it is: * take it down and try to make a full copy with ddrescue * if ddrescure manages to copy everything, copy back to a new disk and add to ceph * if ddrescue fails to copy everything, you could try if badblocks manages to get the disk back; ddrescue can force remappings of broken sectors (non-destructive read-write check) and it can happen that data becomes readable again, exchange the disk as soon as possible thereafter - if the disk is healthy: * try to find out if you can deduce the state of the copies on every OSD The tool for low-level operations is bluestore-tool. I never used it, so you need to look at the documentation. If everything fails, I guess your last option is to decide for one of the copies, export it from one OSD and inject it to another one (but not any of 0,1,2!). This will establish 2 identical copies and the third one will be changed to this one automatically. Note that this may lead to data loss on objects that were in the undefined state. As far as I can see, its only 1 object and probably possible to recover from (backup, snapshot). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sagara Wijetunga <sagarawmw@yahoo.com> Sent: 01 November 2020 14:05:36 To: ceph-users@ceph.io Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? Hi Frank Thanks for the reply.

I think this happens when a PG has 3 different copies and cannot decide which one is correct. You might have hit a very rare case. You should start with the scrub errors, check which PGs and which copies (OSDs) are affected. It sounds almost like all 3 scrub errors are on the same PG. Yes, all 3 errors are for the same PG and on the same OSD: 2020-11-01 18:25:09.333339 osd.0 [ERR] 3.b shard 2 soid 3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate had a missing info key 2020-11-01 18:25:09.333342 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head : failed to pick suitable object info 2020-11-01 18:26:33.496255 osd.0 [ERR] 3.b repair 3 errors, 0 fixed

You might have had a combination of crash and OSD fail, your situation is probably not covered by "single point of failure". Yes it was a complex crash, all went down.

In case you have a PG with scrub errors on 2 copies, you should be able to reconstruct the PG from the third with PG export/PG import commands. I have not done a PG export/import before. Mind if you could send the instructions or a link for it.

Thanks Sagara _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Sagara, I'm not sure if my hypothesis can be correct. Ceph sends an acknowledge of a write only after all copies are on disk. In other words, if PGs end up on different versions after a power outage, one always needs to roll back. Since you have two healthy OSDs in the PG and the PG is active (successfully peered), it might just be a broken disk and read/write errors. I would focus on that. Another question, do you have write caches enabled (disk cache and controller cache)? This is know to cause problems on power outages and also degraded performance with ceph. You should check and disable any caches if necessary. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 01 November 2020 14:37:41 To: Sagara Wijetunga; ceph-users@ceph.io Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? sorry: *badblocks* can force remappings of broken sectors (non-destructive read-write check) ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 01 November 2020 14:35:35 To: Sagara Wijetunga; ceph-users@ceph.io Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? Hi Sagara, looks like your situation is more complex. Before doing anything potentially destructive, you need to investigate some more. A possible interpretation (numbering just for the example): OSD 0 PG at version 1 OSD 1 PG at version 2 OSD 2 PG has scrub error Depending on the version of the PG on OSD 2, either OSD 0 needs to roll forward (OSD 2 PG at version 2), or OSD 1 needs to roll back (OSD 2 PG at version 1). Part of the relevant information on OSD 2 seems to be unreadable, therefore pg repair bails out. You need to find out if you are in this situation or some other case. If you are, you need to find out somehow if you need to roll back or forward. I'm afraid in your current situation, even taking the OSD with the scrub errors down will not rebuild the PG. I would probably try: - find out with smartctl if the OSD with scrub errors is in a pre-fail state (has remapped sectors) - if it is: * take it down and try to make a full copy with ddrescue * if ddrescure manages to copy everything, copy back to a new disk and add to ceph * if ddrescue fails to copy everything, you could try if badblocks manages to get the disk back; ddrescue can force remappings of broken sectors (non-destructive read-write check) and it can happen that data becomes readable again, exchange the disk as soon as possible thereafter - if the disk is healthy: * try to find out if you can deduce the state of the copies on every OSD The tool for low-level operations is bluestore-tool. I never used it, so you need to look at the documentation. If everything fails, I guess your last option is to decide for one of the copies, export it from one OSD and inject it to another one (but not any of 0,1,2!). This will establish 2 identical copies and the third one will be changed to this one automatically. Note that this may lead to data loss on objects that were in the undefined state. As far as I can see, its only 1 object and probably possible to recover from (backup, snapshot). Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sagara Wijetunga <sagarawmw@yahoo.com> Sent: 01 November 2020 14:05:36 To: ceph-users@ceph.io Subject: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? Hi Frank Thanks for the reply.

I think this happens when a PG has 3 different copies and cannot decide which one is correct. You might have hit a very rare case. You should start with the scrub errors, check which PGs and which copies (OSDs) are affected. It sounds almost like all 3 scrub errors are on the same PG. Yes, all 3 errors are for the same PG and on the same OSD: 2020-11-01 18:25:09.333339 osd.0 [ERR] 3.b shard 2 soid 3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate had a missing info key 2020-11-01 18:25:09.333342 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head : failed to pick suitable object info 2020-11-01 18:26:33.496255 osd.0 [ERR] 3.b repair 3 errors, 0 fixed

You might have had a combination of crash and OSD fail, your situation is probably not covered by "single point of failure". Yes it was a complex crash, all went down.

In case you have a PG with scrub errors on 2 copies, you should be able to reconstruct the PG from the third with PG export/PG import commands. I have not done a PG export/import before. Mind if you could send the instructions or a link for it.

Thanks Sagara _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Sagra, looks like you have one on a new and 2 on an old version. Can you add the information about which OSD each version resides? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sagara Wijetunga <sagarawmw@yahoo.com> Sent: 02 November 2020 10:10:02 To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? Hi Frank

I'm not sure if my hypothesis can be correct. Ceph sends an acknowledge of a write only after all copies are on disk. In other words, if PGs end up on different versions after a power outage, one always needs to roll back. Since you have two healthy OSDs in the PG and the PG is active (successfully peered), it might just be a broken disk and read/write errors. I would focus on that.

I tried to revert the PG as follows: # ceph pg 3.b query | grep version "last_user_version": 2263481, "version": "4825'2264303", "last_user_version": 2263481, "version": "4825'2264301", "last_user_version": 2263481, "version": "4825'2264301", ceph pg 3.b list_unfound { "num_missing": 0, "num_unfound": 0, "objects": [], "more": false } # ceph pg 3.b mark_unfound_lost revert pg has no unfound objects # ceph pg 3.b revert Invalid command: revert not in query pg <pgid> query : show details of a specific pg Error EINVAL: invalid command How to revert/rollback a PG?

Another question, do you have write caches enabled (disk cache and controller cache)? This is know to cause problems on power outages and also degraded performance with ceph. You should check and disable any caches if necessary.

No. HDD is directly connected to motherboard. Thank you Sagara

Hi Frank

Please note, there is no peer 0 in "ceph pg 3.b query". Also no word osd.I checked other PGs with active+clean, there is a "peer": "0".  But "ceph pg pgid query" always shows only two peers, sometime peer 0 and 1, or 1 and 2, 0 and 2, etc.

Regards Sagara

Hi Sagara, the primary OSD is probably not listed as a peer. Can you post the complete output of - ceph pg 3.b query - ceph pg dump - ceph osd df tree in a pastebin? ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sagara Wijetunga <sagarawmw@yahoo.com> Sent: 02 November 2020 11:53:58 To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? Hi Frank

Please note, there is no peer 0 in "ceph pg 3.b query". Also no word osd.

I checked other PGs with "active+clean", there is a "peer": "0". But "ceph pg pgid query" always shows only two peers, sometime peer 0 and 1, or 1 and 2, 0 and 2, etc. Regards Sagara

Hmm, I'm getting a bit confused. Could you also send the output of "ceph osd pool ls detail". Did you look at the disk/controller cache settings? I think you should start a deep-scrub with "ceph pg deep-scrub 3.b" and record the output of "ceph -w | grep '3\.b'" (note the single quotes). The error messages you included in one of your first e-mails are only on 1 out of 3 scrub errors (3 lines for 1 error). We need to find all 3 errors. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sagara Wijetunga <sagarawmw@yahoo.com> Sent: 02 November 2020 14:25:08 To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? Hi Frank

the primary OSD is probably not listed as a peer. Can you post the complete output of

- ceph pg 3.b query - ceph pg dump - ceph osd df tree

in a pastebin?

Yes, the Primary OSD is 0. I have attached above as .txt files. Please let me know if you still cannot read them. Regards Sagara

But there can be a on chip disk controller on the motherboard, I'm not sure.

There is always some kind of controller. Could be on-board. Usually, the cache settings are accessible when booting into the BIOS set-up.

If your worry is fsync persistence

No, what I worry about is volatile write cache, which is usually enabled by default. This cache exists on disk as well as on controller. To avoid loosing writes on power fail, the controller needs to be in write-through mode and the disk write cache disabled. The latter can be done with smartctl, the former in the BIOS setup. Did you test power failure? If so, how often? On how many hosts simultaneously? Pulling network cables will not trigger cache related problems. The problem with write cache is, that you rely on a lot of bells and whistles where some usually fail. With ceph, this will lead to exactly the problem you are observing now. Your pool configuration looks OK. You need to find out where exactly the scrub errors are situated. It looks like meta-data damage and you might loose some data. Be careful to do only read-only admin operations for now. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sagara Wijetunga <sagarawmw@yahoo.com> Sent: 02 November 2020 16:08:58 To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?

Hmm, I'm getting a bit confused. Could you also send the output of "ceph osd pool ls detail".

File ceph-osd-pool-ls-detail.txt attached.

Did you look at the disk/controller cache settings?

I don't have disk controllers on Ceph machines. The hard disk is directly attached to the motherboard via SATA cable. But there can be a on chip disk controller on the motherboard, I'm not sure. If your worry is fsync persistence, I have thoroughly tested database fsync reliability on Ceph RBD with hundreds of transactions per second and remove network cable and restart the database machine, etc. while inserts going on. and I did not lose a single transaction. I simulated this many times and persistence on my Ceph cluster was perfect (i.e not a single loss).

I think you should start a deep-scrub with "ceph pg deep-scrub 3.b" and record the output of "ceph -w | grep '3\.b'" (note the single quotes).

The error messages you included in one of your first e-mails are only on 1 out of 3 scrub errors (3 lines for 1 error). We need to find all 3 errors.

I ran again the "ceph pg deep-scrub 3.b", here is the whole output of ceph -w: 2020-11-02 22:33:48.224392 osd.0 [ERR] 3.b shard 2 soid 3:d577e975:::1000023675e.00000000:head : candidate had a missing snapset key, candidate had a missing info key 2020-11-02 22:33:48.224396 osd.0 [ERR] 3.b soid 3:d577e975:::1000023675e.00000000:head : failed to pick suitable object info 2020-11-02 22:35:30.087042 osd.0 [ERR] 3.b deep-scrub 3 errors Btw, I'm very grateful for your perseverance on this. Best regards Sagara

Hi Sagara, did you collect the output of "ceph -w | grep '3\.b'" while the PG was deep-scrubbed? Please open a new thread with the error message in the subject line. This should draw the attention of someone with internal knowledge. Include a short description of how this situation came about with the deep-scrub errors and log messages. The problem is, that no objects are reported as lost or damaged. Therefore, I do not believe that the usual actions will help, which would include things like marking unfound objects as lost. This doesn't apply to your case. If you know where data access fails on the file system, you can move the folder to a "parking spot" and restore the data from backup. This would allow your users to continue working while you buy time to fix the PG. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sagara Wijetunga <sagarawmw@yahoo.com> Sent: 03 November 2020 10:39:18 To: ceph-users@ceph.io; Frank Schilder Subject: Re: [ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? Hi Frank 1. We will disable the disk controller and disk-level caching to avoid future issues. 2. My pools are: ceph osd lspools 2 cephfs_metadata 3 cephfs_data 4 rbd The PG now inconsistent is 3.b, therefore, it belongs to cephfs_data pool. Following also shows the PG 3.b belongs to cephfs_data: ceph pg ls-by-pool cephfs_data | grep 3.b 3.b 6992 0 0 0 9649392528 0 0 3005 active+clean+inconsistent ... 3. Deep scrubs shows only one object having an issue: soid 3:d577e975:::1000023675e.00000000 This object seems lost. rados -p cephfs_metadata ls | grep 1000023675e.00000000 rados -p cephfs_data ls | grep 1000023675e.00000000 rados -p rbd ls | grep 1000023675e.00000000 4. I tried to find what are the files effected by this issue, but I get "No such file or directory" for the path. I have properly mounted ceph on home as before. cephfs-data-scan -c /etc/ceph/ceph.conf pg_files /home/sagara 3.b 2020-11-03T17:06:21.770+0800 7f3f213ab100 -1 pgeffects.hit_dir: Failed to open path: (2) No such file or directory How do I see what are the files effected by this issue? 5. What should be the course of the action now to bring the cluster to "active+clean" to move forward? I don't mind roll back the PG having the issue. I have a file-level backup. If roll back the PG is the way forward, how to do? Thank you. Best regards Sagara On Monday, November 2, 2020, 11:29:55 PM GMT+8, Frank Schilder <frans@dtu.dk> wrote:

But there can be a on chip disk controller on the motherboard, I'm not sure.

There is always some kind of controller. Could be on-board. Usually, the cache settings are accessible when booting into the BIOS set-up.

If your worry is fsync persistence

No, what I worry about is volatile write cache, which is usually enabled by default. This cache exists on disk as well as on controller. To avoid loosing writes on power fail, the controller needs to be in write-through mode and the disk write cache disabled. The latter can be done with smartctl, the former in the BIOS setup. Did you test power failure? If so, how often? On how many hosts simultaneously? Pulling network cables will not trigger cache related problems. The problem with write cache is, that you rely on a lot of bells and whistles where some usually fail. With ceph, this will lead to exactly the problem you are observing now. Your pool configuration looks OK. You need to find out where exactly the scrub errors are situated. It looks like meta-data damage and you might loose some data. Be careful to do only read-only admin operations for now. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14