1 pg inconsistent and does not recover
Hi, I have a 3x-replicated pool with Ceph 12.2.7. One HDD broke, its OSD "2" was automatically marked as "out", the disk was physically replaced by a new one, and that added back in. Now `ceph health detail` continues to permanently show: [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.87 is active+clean+inconsistent, acting [33,2,20] What exactly is wrong here? Why can Ceph not fix the issue? With BlueStore I have checksums, on two unbroken disks, so what remaining inconsistency can there be? The suggested command in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#commands-for-di... does not work: # rados list-inconsistent-obj 2.87 No scrub information available for pg 2.87 error 2: (2) No such file or directory Further, I find the documentation in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#more-informatio... extremely unclear. It says
In the case of replicated pools, recovery is beyond the scope of pg repair.
while many people on the Internet suggest that `ceph pg repair` might fix the issue. Yet again others claim that Ceph will fix the issue itself. I am hesitant to run "ceph pg repair" without understanding what the problem is and what exactly this will do. I have already reported the "error 2" and the documentation in issue https://tracker.ceph.com/issues/61739 but not received a reply yet, and my cluster stays "inconsistent". How can this be fixed? I would appreciate any help!
Hi Niklas, You may have a hardware error, but who knows... Can you post the entire Ceph status output? (Pastebin) sometimes list-inconsistent-obj throws that error if a scrub job is still running. Also, please try to find mode information from the logs by doing. grep -Hn 'ERR' /var/log/ceph/ceph-osd.33.log Cheers. On Tue, Jun 27, 2023 at 4:47 PM Niklas Hambüchen <mail@nh2.me> wrote:
Hi,
I have a 3x-replicated pool with Ceph 12.2.7.
One HDD broke, its OSD "2" was automatically marked as "out", the disk was physically replaced by a new one, and that added back in.
Now `ceph health detail` continues to permanently show:
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.87 is active+clean+inconsistent, acting [33,2,20]
What exactly is wrong here?
Why can Ceph not fix the issue? With BlueStore I have checksums, on two unbroken disks, so what remaining inconsistency can there be?
The suggested command in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#commands-for-di... does not work:
# rados list-inconsistent-obj 2.87 No scrub information available for pg 2.87 error 2: (2) No such file or directory
Further, I find the documentation in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#more-informatio... extremely unclear. It says
In the case of replicated pools, recovery is beyond the scope of pg repair.
while many people on the Internet suggest that `ceph pg repair` might fix the issue. Yet again others claim that Ceph will fix the issue itself. I am hesitant to run "ceph pg repair" without understanding what the problem is and what exactly this will do.
I have already reported the "error 2" and the documentation in issue https://tracker.ceph.com/issues/61739 but not received a reply yet, and my cluster stays "inconsistent".
How can this be fixed?
I would appreciate any help! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
Hi Alvaro,
Can you post the entire Ceph status output?
Pasting here since it is short cluster: id: d9000ec0-93c2-479f-bd5d-94ae9673e347 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent services: mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m) mgr: node-5(active, since 7d), standbys: node-6, node-4 mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 5d), 36 in (since 6d) data: volumes: 1/1 healthy pools: 3 pools, 832 pgs objects: 506.83M objects, 67 TiB usage: 207 TiB used, 232 TiB / 439 TiB avail pgs: 826 active+clean 5 active+clean+scrubbing+deep 1 active+clean+inconsistent io: client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error if a scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD "2" already 7 days ago, and "list-inconsistent-obj" has not worked at any time since then.
grep -Hn 'ERR' /var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head : candidate had a read error /var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects /var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors The time "2023-06-16T16:29:57" above is the time at which the disk that carried OSD "2" broke, its logs around the time are: /var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1 bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output error) /var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1 log_channel(cluster) log [ERR] : 2.a6 missing primary copy of 2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28 -- note time jump by 3 days -- /var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000 7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000 7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1 ** ERROR: osd init failed: (5) Input/output error Does "candidate had a read error" on OSD "33" mean that a BlueStore checksum error was detected on OSD "33" at the same time as the OSD "2" disk failed? If yes, maybe that is the explanation: * pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD 33 detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20) whose checksums disagree. I am just guessing this, though. Also, if this is correct, the next question would be: What is with OSD 20? Since there is no error reported at all for OSD 20, I assume that its checksum agrees with its data. Now, can I find out whether OSD 20's checksum agrees with OSD 33's data? (Side note: The disk of OSD 33 looks fine in smartctl.) Thanks, Niklas
Hello Niklas, The explanation looks plausible. What you can do is try extracting the PG from the dead OSD disk (please make absolutely sure that the OSD daemon is stopped!!!) and reinjecting it into some other OSD (again, stop the daemon during this procedure). This extra copy should act as an arbiter. The relevant commands are: systemctl stop ceph-osd@2 systemctl stop ceph-osd@3 # or whatever other OSD exists on the same host systemctl mask ceph-osd@2 systemctl mask ceph-osd@3 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --pgid 2.87 --op export --file /some/local/storage/pg-2.87.exp ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3/ --type bluestore --pgid 2.87 --op import --file /some/local/storage/pg-2.87.exp systemctl unmask ceph-osd@3 systemctl start ceph-osd@3 systemctl unmask ceph-osd@2 On Wed, Jun 28, 2023 at 8:31 AM Niklas Hambüchen <mail@nh2.me> wrote:
Hi Alvaro,
Can you post the entire Ceph status output?
Pasting here since it is short
cluster: id: d9000ec0-93c2-479f-bd5d-94ae9673e347 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent
services: mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m) mgr: node-5(active, since 7d), standbys: node-6, node-4 mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 5d), 36 in (since 6d)
data: volumes: 1/1 healthy pools: 3 pools, 832 pgs objects: 506.83M objects, 67 TiB usage: 207 TiB used, 232 TiB / 439 TiB avail pgs: 826 active+clean 5 active+clean+scrubbing+deep 1 active+clean+inconsistent
io: client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error if a scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD "2" already 7 days ago, and "list-inconsistent-obj" has not worked at any time since then.
grep -Hn 'ERR' /var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head : candidate had a read error /var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects /var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors
The time "2023-06-16T16:29:57" above is the time at which the disk that carried OSD "2" broke, its logs around the time are:
/var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1 bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output error) /var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1 log_channel(cluster) log [ERR] : 2.a6 missing primary copy of 2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28 -- note time jump by 3 days -- /var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000 7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000 7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1 ** ERROR: osd init failed: (5) Input/output error
Does "candidate had a read error" on OSD "33" mean that a BlueStore checksum error was detected on OSD "33" at the same time as the OSD "2" disk failed? If yes, maybe that is the explanation:
* pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD 33 detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20) whose checksums disagree.
I am just guessing this, though. Also, if this is correct, the next question would be: What is with OSD 20? Since there is no error reported at all for OSD 20, I assume that its checksum agrees with its data. Now, can I find out whether OSD 20's checksum agrees with OSD 33's data?
(Side note: The disk of OSD 33 looks fine in smartctl.)
Thanks, Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov
Hi Niklas, please don't do any of the recovery steps yet! Your problem is almost certainly a non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error messeges you have: ceph status/df/pool stats/health detail at 00:00:06: cluster: health: HEALTH_ERR 3 scrub errors Possible data damage: 3 pgs inconsistent After rebuilding the data, it still looked like: cluster: health: HEALTH_ERR 2 scrub errors Possible data damage: 2 pgs inconsistent What's the issue here? The issue is that the OGs have not been deep-scrubbed after rebuild. The reply "no scrub data available" of the list-inconsistent is the clue. The response to that is not to try manual repair but to issue a deep-scrub. Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the deep scrub reservation almost always gets cancelled very quickly. I got a script to force repair/deep-scrub (I don't remember who sent it to me) and that gets the job done: ===================== #!/bin/bash [[ -r "/etc/profile.d/ceph.sh" ]] && source "/etc/profile.d/ceph.sh" for PG in $(ceph pg ls inconsistent -f json | jq -r .pg_stats[].pgid) do echo Checking inconsistent PG $PG if ceph pg ls repair | grep -wq ${PG} then echo PG $PG is already repairing, skipping continue fi # disable other scrubs ceph osd set nodeep-scrub ceph osd set noscrub # bump up osd_max_scrubs ACTING=$(ceph pg $PG query | jq -r .acting[]) for OSD in $ACTING do cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=3 --osd_scrub_during_recovery=true ) echo "executing: ${cmd[@]}" "${cmd[@]}" done ceph pg repair $PG sleep 10 for OSD in $ACTING do cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=1 --osd_scrub_during_recovery=false ) echo "executing: ${cmd[@]}" "${cmd[@]}" done # disable other scrubs ceph osd unset nodeep-scrub ceph osd unset noscrub done =================== You can also just wait for the regular deep-scrub to happen. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Alexander E. Patrakov <patrakov@gmail.com> Sent: Wednesday, June 28, 2023 5:24 AM To: Niklas Hambüchen Cc: ceph-users@ceph.io Subject: [ceph-users] Re: 1 pg inconsistent and does not recover Hello Niklas, The explanation looks plausible. What you can do is try extracting the PG from the dead OSD disk (please make absolutely sure that the OSD daemon is stopped!!!) and reinjecting it into some other OSD (again, stop the daemon during this procedure). This extra copy should act as an arbiter. The relevant commands are: systemctl stop ceph-osd@2 systemctl stop ceph-osd@3 # or whatever other OSD exists on the same host systemctl mask ceph-osd@2 systemctl mask ceph-osd@3 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --pgid 2.87 --op export --file /some/local/storage/pg-2.87.exp ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3/ --type bluestore --pgid 2.87 --op import --file /some/local/storage/pg-2.87.exp systemctl unmask ceph-osd@3 systemctl start ceph-osd@3 systemctl unmask ceph-osd@2 On Wed, Jun 28, 2023 at 8:31 AM Niklas Hambüchen <mail@nh2.me> wrote:
Hi Alvaro,
Can you post the entire Ceph status output?
Pasting here since it is short
cluster: id: d9000ec0-93c2-479f-bd5d-94ae9673e347 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent
services: mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m) mgr: node-5(active, since 7d), standbys: node-6, node-4 mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 5d), 36 in (since 6d)
data: volumes: 1/1 healthy pools: 3 pools, 832 pgs objects: 506.83M objects, 67 TiB usage: 207 TiB used, 232 TiB / 439 TiB avail pgs: 826 active+clean 5 active+clean+scrubbing+deep 1 active+clean+inconsistent
io: client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error if a scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD "2" already 7 days ago, and "list-inconsistent-obj" has not worked at any time since then.
grep -Hn 'ERR' /var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head : candidate had a read error /var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects /var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors
The time "2023-06-16T16:29:57" above is the time at which the disk that carried OSD "2" broke, its logs around the time are:
/var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1 bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output error) /var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1 log_channel(cluster) log [ERR] : 2.a6 missing primary copy of 2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28 -- note time jump by 3 days -- /var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000 7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000 7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1 ** ERROR: osd init failed: (5) Input/output error
Does "candidate had a read error" on OSD "33" mean that a BlueStore checksum error was detected on OSD "33" at the same time as the OSD "2" disk failed? If yes, maybe that is the explanation:
* pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD 33 detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20) whose checksums disagree.
I am just guessing this, though. Also, if this is correct, the next question would be: What is with OSD 20? Since there is no error reported at all for OSD 20, I assume that its checksum agrees with its data. Now, can I find out whether OSD 20's checksum agrees with OSD 33's data?
(Side note: The disk of OSD 33 looks fine in smartctl.)
Thanks, Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just for reference for everybody, the original source is https://github.com/cernceph/ceph-scripts/blob/master/tools/scrubbing/autorep... maintained by Dan van der Ster. The repo as such is a rich source of good tools in general and worth looking at before doing anything that requires more than 30 minutes searching and reading documentation. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Wednesday, June 28, 2023 9:41 AM To: Alexander E. Patrakov; Niklas Hambüchen Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover Hi Niklas, please don't do any of the recovery steps yet! Your problem is almost certainly a non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error messeges you have: ceph status/df/pool stats/health detail at 00:00:06: cluster: health: HEALTH_ERR 3 scrub errors Possible data damage: 3 pgs inconsistent After rebuilding the data, it still looked like: cluster: health: HEALTH_ERR 2 scrub errors Possible data damage: 2 pgs inconsistent What's the issue here? The issue is that the OGs have not been deep-scrubbed after rebuild. The reply "no scrub data available" of the list-inconsistent is the clue. The response to that is not to try manual repair but to issue a deep-scrub. Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the deep scrub reservation almost always gets cancelled very quickly. I got a script to force repair/deep-scrub (I don't remember who sent it to me) and that gets the job done: ===================== #!/bin/bash [[ -r "/etc/profile.d/ceph.sh" ]] && source "/etc/profile.d/ceph.sh" for PG in $(ceph pg ls inconsistent -f json | jq -r .pg_stats[].pgid) do echo Checking inconsistent PG $PG if ceph pg ls repair | grep -wq ${PG} then echo PG $PG is already repairing, skipping continue fi # disable other scrubs ceph osd set nodeep-scrub ceph osd set noscrub # bump up osd_max_scrubs ACTING=$(ceph pg $PG query | jq -r .acting[]) for OSD in $ACTING do cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=3 --osd_scrub_during_recovery=true ) echo "executing: ${cmd[@]}" "${cmd[@]}" done ceph pg repair $PG sleep 10 for OSD in $ACTING do cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=1 --osd_scrub_during_recovery=false ) echo "executing: ${cmd[@]}" "${cmd[@]}" done # disable other scrubs ceph osd unset nodeep-scrub ceph osd unset noscrub done =================== You can also just wait for the regular deep-scrub to happen. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Alexander E. Patrakov <patrakov@gmail.com> Sent: Wednesday, June 28, 2023 5:24 AM To: Niklas Hambüchen Cc: ceph-users@ceph.io Subject: [ceph-users] Re: 1 pg inconsistent and does not recover Hello Niklas, The explanation looks plausible. What you can do is try extracting the PG from the dead OSD disk (please make absolutely sure that the OSD daemon is stopped!!!) and reinjecting it into some other OSD (again, stop the daemon during this procedure). This extra copy should act as an arbiter. The relevant commands are: systemctl stop ceph-osd@2 systemctl stop ceph-osd@3 # or whatever other OSD exists on the same host systemctl mask ceph-osd@2 systemctl mask ceph-osd@3 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --pgid 2.87 --op export --file /some/local/storage/pg-2.87.exp ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3/ --type bluestore --pgid 2.87 --op import --file /some/local/storage/pg-2.87.exp systemctl unmask ceph-osd@3 systemctl start ceph-osd@3 systemctl unmask ceph-osd@2 On Wed, Jun 28, 2023 at 8:31 AM Niklas Hambüchen <mail@nh2.me> wrote:
Hi Alvaro,
Can you post the entire Ceph status output?
Pasting here since it is short
cluster: id: d9000ec0-93c2-479f-bd5d-94ae9673e347 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent
services: mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m) mgr: node-5(active, since 7d), standbys: node-6, node-4 mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 5d), 36 in (since 6d)
data: volumes: 1/1 healthy pools: 3 pools, 832 pgs objects: 506.83M objects, 67 TiB usage: 207 TiB used, 232 TiB / 439 TiB avail pgs: 826 active+clean 5 active+clean+scrubbing+deep 1 active+clean+inconsistent
io: client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error if a scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD "2" already 7 days ago, and "list-inconsistent-obj" has not worked at any time since then.
grep -Hn 'ERR' /var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head : candidate had a read error /var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects /var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors
The time "2023-06-16T16:29:57" above is the time at which the disk that carried OSD "2" broke, its logs around the time are:
/var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1 bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output error) /var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1 log_channel(cluster) log [ERR] : 2.a6 missing primary copy of 2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28 -- note time jump by 3 days -- /var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000 7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000 7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000 7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 : cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32 /var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1 ** ERROR: osd init failed: (5) Input/output error
Does "candidate had a read error" on OSD "33" mean that a BlueStore checksum error was detected on OSD "33" at the same time as the OSD "2" disk failed? If yes, maybe that is the explanation:
* pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD 33 detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20) whose checksums disagree.
I am just guessing this, though. Also, if this is correct, the next question would be: What is with OSD 20? Since there is no error reported at all for OSD 20, I assume that its checksum agrees with its data. Now, can I find out whether OSD 20's checksum agrees with OSD 33's data?
(Side note: The disk of OSD 33 looks fine in smartctl.)
Thanks, Niklas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/28/23 09:41, Frank Schilder wrote:
Hi Niklas,
please don't do any of the recovery steps yet! Your problem is almost certainly a non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error messeges you have:
ceph status/df/pool stats/health detail at 00:00:06: cluster: health: HEALTH_ERR 3 scrub errors Possible data damage: 3 pgs inconsistent
After rebuilding the data, it still looked like:
cluster: health: HEALTH_ERR 2 scrub errors Possible data damage: 2 pgs inconsistent
What's the issue here? The issue is that the OGs have not been deep-scrubbed after rebuild. The reply "no scrub data available" of the list-inconsistent is the clue. The response to that is not to try manual repair but to issue a deep-scrub.
Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the deep scrub reservation almost always gets cancelled very quickly.
On what Ceph version do you have this issue? We use this command everyday, hunderds of times, and it always works. Or is this an issue when you have a degraded cluster? Gr. Stefan
Hi Stefan, we run Octopus. The deep-scrub request is (immediately) cancelled if the PG/OSD is already part of another (deep-)scrub or if some peering happens. As far as I understood, the commands osd/pg deep-scrub and pg repair do not create persistent reservations. If you issue this command, when does the PG actually start scrubbing? As soon as another one finishes or when it is its natural turn? Do you monitor the scrub order to confirm it was the manual command that initiated a scrub? What I see is that the pg repair and the pg deep-scrub are almost immediately forgotten on our cluster. This is most prominent with the repair command, which can be really hard to get going and complete. Only an osd deep-scrub seems to have some effect. On the other hand, when I run the script, which stops all operations that conflict with manual reservations, the repair/deep-scrub actually start on request. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Wednesday, June 28, 2023 9:54 AM To: Frank Schilder; Alexander E. Patrakov; Niklas Hambüchen Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover On 6/28/23 09:41, Frank Schilder wrote:
Hi Niklas,
please don't do any of the recovery steps yet! Your problem is almost certainly a non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error messeges you have:
ceph status/df/pool stats/health detail at 00:00:06: cluster: health: HEALTH_ERR 3 scrub errors Possible data damage: 3 pgs inconsistent
After rebuilding the data, it still looked like:
cluster: health: HEALTH_ERR 2 scrub errors Possible data damage: 2 pgs inconsistent
What's the issue here? The issue is that the OGs have not been deep-scrubbed after rebuild. The reply "no scrub data available" of the list-inconsistent is the clue. The response to that is not to try manual repair but to issue a deep-scrub.
Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the deep scrub reservation almost always gets cancelled very quickly.
On what Ceph version do you have this issue? We use this command everyday, hunderds of times, and it always works. Or is this an issue when you have a degraded cluster? Gr. Stefan
On 6/28/23 10:45, Frank Schilder wrote:
Hi Stefan,
we run Octopus. The deep-scrub request is (immediately) cancelled if the PG/OSD is already part of another (deep-)scrub or if some peering happens. As far as I understood, the commands osd/pg deep-scrub and pg repair do not create persistent reservations. If you issue this command, when does the PG actually start scrubbing? As soon as another one finishes or when it is its natural turn? Do you monitor the scrub order to confirm it was the manual command that initiated a scrub?
We request a deep-scrub ... a few seconds later it starts deep-scrubbing. We do not verify in this process if the PG really did start, but they do. See example from a PG below: Jun 27 22:59:50 mon1 pg_scrub[2478540]: [27-06-2023 22:59:34] Scrub PG 5.48a (last deep-scrub: 2023-06-16T22:54:58.684038+0200) ^^ deep_scrub daemon requests a deep-scrub, based on latest deep-scrub timestamp. After a couple of minutes it's deep-scrubbed. See below the deep-scrub timestamp (info from a PG query of 5.48a): "last_deep_scrub_stamp": "2023-06-27T23:06:01.823894+0200" We have been using this in Octopus (actually since Luminous, but in a different way). Now we are on Pacific. Gr. Stefan
I, too have the problem that `ceph pg deep-scrub` does not start the scrub, with Ceph 16.2.7. # ceph pg deep-scrub 2.87 instructing pg 2.87 on osd.33 to deep-scrub However then, on the machine where that osd.33 is: # ceph daemon osd.33 dump_scrubs | jq . | head -n 13 [ { "pgid": "2.87", "sched_time": "1.000000", "deadline": "1.000000", "forced": true }, { "pgid": "1.4c", "sched_time": "2023-06-15T16:30:58.696634+0000", "deadline": "2023-07-09T09:01:50.420581+0000", "forced": false }, ... I am not sure what the "1.000000" is supposed to mean here for the problematic OSD. As shown, other OSDs have a normal string ISO time. If I request scrubbing a PG that's on a different OSD (but on the same machine ad the problematic one), then the deep-scrub starts immediately. Niklas
Hi Stefan, after you wrote that you issue hundreds of deep-scrub commands per day I was already suspecting something like
[...] deep_scrub daemon requests a deep-scrub [...]
Its not a Minion you hired that types these commends every so many seconds by hand and hopes for the best. My guess is rather that you are actually running a cluster specifically configured for manual scrub scheduling as was discussed in a thread some time ago to solve the problem of "not deep scrubbed in time" messages due to the built-in scrub scheduler not using the last-scrubbed timestamp for priority (among other things). On such a system I would not be surprised that these commands have their desired effect. To know why it works for you it would be helpful to disclose the whole story, for example what ceph config parameters are active within the context of your daemon executing the deep-scrub instructions and what other ceph-commands surround it in the same way that the pg repair is surrounded by injectargs instructions in the script I posted. It doesn't work like that on a ceph cluster with default config. For example, on our cluster there is a very high likelihood that at least one OSD of any PG is part of a scrub at any time already. In that case, if a PG is not eligible for scrubbing because one of its OSDs has already max-scrubs (default=1) scrubs running, the reservation has no observable effect. Some time ago I had a ceph-user thread discussing exactly that, I wanted to increase the concurrent scrubs running without increasing max-scrubs. The default scheduler seems to be very poor with ordering scrubs in such a way that a maximum number of pgs is scrubbed at any given time. One of the suggestions was to run manual scheduling, which seems exactly like what you are doing. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: Wednesday, June 28, 2023 2:17 PM To: Frank Schilder; Alexander E. Patrakov; Niklas Hambüchen Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover On 6/28/23 10:45, Frank Schilder wrote:
Hi Stefan,
we run Octopus. The deep-scrub request is (immediately) cancelled if the PG/OSD is already part of another (deep-)scrub or if some peering happens. As far as I understood, the commands osd/pg deep-scrub and pg repair do not create persistent reservations. If you issue this command, when does the PG actually start scrubbing? As soon as another one finishes or when it is its natural turn? Do you monitor the scrub order to confirm it was the manual command that initiated a scrub?
We request a deep-scrub ... a few seconds later it starts deep-scrubbing. We do not verify in this process if the PG really did start, but they do. See example from a PG below: Jun 27 22:59:50 mon1 pg_scrub[2478540]: [27-06-2023 22:59:34] Scrub PG 5.48a (last deep-scrub: 2023-06-16T22:54:58.684038+0200) ^^ deep_scrub daemon requests a deep-scrub, based on latest deep-scrub timestamp. After a couple of minutes it's deep-scrubbed. See below the deep-scrub timestamp (info from a PG query of 5.48a): "last_deep_scrub_stamp": "2023-06-27T23:06:01.823894+0200" We have been using this in Octopus (actually since Luminous, but in a different way). Now we are on Pacific. Gr. Stefan
Frank,
high likelihood that at least one OSD of any PG is part of a scrub at any time already. In that case, if a PG is not eligible for scrubbing because one of its OSDs has already max-scrubs (default=1) scrubs running, the reservation has no observable effect.
This is a great hint. I have increased the number of scrubs per OSD from 1 to 3 using `ceph config set osd osd_max_scrubs 3`. Now the problematic PG is scrubbing in `ceph pg ls`: active+clean+scrubbing+deep+inconsistent I will report what happens. If this works, I think we should file an issue to motivate that Ceph should prioritise scrubs that will result in fixing issues over normal routine scrubs.
On 28/06/2023 21:26, Niklas Hambüchen wrote:
I have increased the number of scrubs per OSD from 1 to 3 using `ceph config set osd osd_max_scrubs 3`. Now the problematic PG is scrubbing in `ceph pg ls`:
active+clean+scrubbing+deep+inconsistent
This succeeded! The deep-scrub fixed the PG and the cluster is healthy again. Thanks a lot! So indeed the issue was that the deep-scrub I had asked for was simply never scheduled because Ceph always picked some other scrub to do first on the relevant OSD. Increasing `osd_max_scrubs` beyond 1 made it possible to force the scrub to start. I conclude that most of the information online, including the Ceph docs, does not give the correct advice when recommending `ceph pg repair`. Instead, the docs should make clear that a scrub will fix such issues without involvement of `ceph pg repair`. I find lack of docs disturbing, because a disk failing and being replaced is an extremely common operation for storage cluster. Including some relevant logs of the scrub recovery: # grep '\b2\.87\b' /var/log/ceph/ceph-osd.33.log | grep deep 2023-05-16T16:33:58.398+0000 7f9a985e5640 0 log_channel(cluster) log [DBG] : 2.87 deep-scrub ok 2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects 2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors 2023-06-26T05:06:17.412+0000 7f9b15bfe640 0 log_channel(cluster) log [INF] : osd.33 pg 2.87 Deep scrub errors, upgrading scrub to deep-scrub 2023-06-29T10:14:07.791+0000 7f9a985e5640 0 log_channel(cluster) log [DBG] : 2.87 deep-scrub ok ceph.log: 2023-06-29T10:14:07.792432+0000 osd.33 (osd.33) 938 : cluster [DBG] 2.87 deep-scrub ok 2023-06-29T10:14:09.311257+0000 mgr.node-5 (mgr.2454216) 385434 : cluster [DBG] pgmap v385836: 832 pgs: 1 active+clean+scrubbing, 17 active+clean+scrubbing+deep, 814 active+clean; 68 TiB data, 210 TiB used, 229 TiB / 439 TiB avail; 80 MiB/s rd, 40 MiB/s wr, 45 op/s 2023-06-29T10:14:09.427733+0000 mon.node-4 (mon.0) 20923054 : cluster [INF] Health check cleared: OSD_SCRUB_ERRORS (was: 1 scrub errors) 2023-06-29T10:14:09.427758+0000 mon.node-4 (mon.0) 20923055 : cluster [INF] Health check cleared: PG_DAMAGED (was: Possible data damage: 1 pg inconsistent) 2023-06-29T10:14:09.427786+0000 mon.node-4 (mon.0) 20923056 : cluster [INF] Cluster is now healthy From this, it seems bad that Ceph did not manage to schedule the cluster-fixing scrub within 7 days of the faulty disk being replaced, nor managed to schedule a human-requested scrub within 2 days. What mechanism in Ceph decides the scheduling of scrubs? I see the config value `osd_requested_scrub_priority` which is for "the priority set for user requested scrub on the work queue", but I cannot tell if this also affects scrub start scheduling, or only the priority of IO operations vs e.g. client operations once a scrub has already been started.
A repair always comes with a deep scrub. You can replace it if you want. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Niklas Hambüchen <mail@nh2.me> Sent: Wednesday, June 28, 2023 3:05 PM To: Frank Schilder; Alexander E. Patrakov Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover Hi Frank,
The response to that is not to try manual repair but to issue a deep-scrub.
I am a bit confused, because in your script you do issue "ceph pg repair", not a scrub.
participants (5)
-
Alexander E. Patrakov
-
Alvaro Soto
-
Frank Schilder
-
Niklas Hambüchen
-
Stefan Kooman