28 Jun
2023
28 Jun
'23
7:47 a.m.
Hi,
I have a 3x-replicated pool with Ceph 12.2.7.
One HDD broke, its OSD "2" was automatically marked as "out", the disk
was physically replaced by a new one, and that added back in.
Now `ceph health detail` continues to permanently show:
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 2.87 is active+clean+inconsistent, acting [33,2,20]
What exactly is wrong here?
Why can Ceph not fix the issue?
With BlueStore I have checksums, on two unbroken disks, so what remaining inconsistency
can there be?
The suggested command in
https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#commands-for-d…
does not work:
# rados list-inconsistent-obj 2.87
No scrub information available for pg 2.87
error 2: (2) No such file or directory
Further, I find the documentation in
https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#more-informati…
extremely unclear.
It says
In the case of replicated pools, recovery is beyond
the scope of pg repair.
while many people on the Internet suggest that `ceph pg repair` might fix the issue.
Yet again others claim that Ceph will fix the issue itself.
I am hesitant to run "ceph pg repair" without understanding what the problem is
and what exactly this will do.
I have already reported the "error 2" and the documentation in issue
https://tracker.ceph.com/issues/61739 but not received a reply yet, and my cluster stays
"inconsistent".
How can this be fixed?
I would appreciate any help!
28 Jun
28 Jun
8:43 a.m.
Hi Niklas,
You may have a hardware error, but who knows...
Can you post the entire Ceph status output? (Pastebin) sometimes
list-inconsistent-obj throws that error if a scrub job is still running.
Also, please try to find mode information from the logs by doing.
grep -Hn 'ERR' /var/log/ceph/ceph-osd.33.log
Cheers.
On Tue, Jun 27, 2023 at 4:47 PM Niklas Hambüchen <mail(a)nh2.me> wrote:
Hi,
I have a 3x-replicated pool with Ceph 12.2.7.
One HDD broke, its OSD "2" was automatically marked as "out", the
disk was
physically replaced by a new one, and that added back in.
Now `ceph health detail` continues to permanently show:
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 2.87 is active+clean+inconsistent, acting [33,2,20]
What exactly is wrong here?
Why can Ceph not fix the issue?
With BlueStore I have checksums, on two unbroken disks, so what remaining
inconsistency can there be?
The suggested command in
https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#commands-for-d…
does not work:
# rados list-inconsistent-obj 2.87
No scrub information available for pg 2.87
error 2: (2) No such file or directory
Further, I find the documentation in
https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#more-informati…
extremely unclear.
It says
--
Alvaro Soto
*Note: My work hours may not be your work hours. Please do not feel the
need to respond during a time that is not convenient for you.*
----------------------------------------------------------
Great people talk about ideas,
ordinary people talk about things,
small people talk... about other people.
In the case of replicated pools, recovery is
beyond the scope of pg
repair.
while many people on the Internet suggest that `ceph pg repair` might fix
the issue.
Yet again others claim that Ceph will fix the issue itself.
I am hesitant to run "ceph pg repair" without understanding what the
problem is and what exactly this will do.
I have already reported the "error 2" and the documentation in issue
https://tracker.ceph.com/issues/61739 but not received a reply yet, and
my cluster stays "inconsistent".
How can this be fixed?
I would appreciate any help!
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:30 a.m.
Hi Alvaro,
Can you post the entire Ceph status output?
Pasting here since it is short
cluster:
id: d9000ec0-93c2-479f-bd5d-94ae9673e347
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m)
mgr: node-5(active, since 7d), standbys: node-6, node-4
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 5d), 36 in (since 6d)
data:
volumes: 1/1 healthy
pools: 3 pools, 832 pgs
objects: 506.83M objects, 67 TiB
usage: 207 TiB used, 232 TiB / 439 TiB avail
pgs: 826 active+clean
5 active+clean+scrubbing+deep
1 active+clean+inconsistent
io:
client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error if a
scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD
"2" already 7 days ago, and "list-inconsistent-obj" has not worked at
any time since then.
grep -Hn 'ERR' /var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head
: candidate had a read error
/var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects
/var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors
The time "2023-06-16T16:29:57" above is the time at which the disk that carried
OSD "2" broke, its logs around the time are:
/var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1
bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output
error)
/var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1
log_channel(cluster) log [ERR] : 2.a6 missing primary copy of
2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28
-- note time jump by 3 days --
/var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000
7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000
7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000
7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1 **
ERROR: osd init failed: (5) Input/output error
Does "candidate had a read error" on OSD "33" mean that a BlueStore
checksum error was detected on OSD "33" at the same time as the OSD
"2" disk failed?
If yes, maybe that is the explanation:
* pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD 33
detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20) whose
checksums disagree.
I am just guessing this, though.
Also, if this is correct, the next question would be: What is with OSD 20?
Since there is no error reported at all for OSD 20, I assume that its checksum agrees with
its data.
Now, can I find out whether OSD 20's checksum agrees with OSD 33's data?
(Side note: The disk of OSD 33 looks fine in smartctl.)
Thanks,
Niklas
12:24 p.m.
Hello Niklas,
The explanation looks plausible.
What you can do is try extracting the PG from the dead OSD disk
(please make absolutely sure that the OSD daemon is stopped!!!) and
reinjecting it into some other OSD (again, stop the daemon during this
procedure). This extra copy should act as an arbiter.
The relevant commands are:
systemctl stop ceph-osd@2
systemctl stop ceph-osd@3 # or whatever other OSD exists on the same host
systemctl mask ceph-osd@2
systemctl mask ceph-osd@3
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --pgid
2.87 --op export --file /some/local/storage/pg-2.87.exp
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3/ --type
bluestore --pgid 2.87 --op import --file
/some/local/storage/pg-2.87.exp
systemctl unmask ceph-osd@3
systemctl start ceph-osd@3
systemctl unmask ceph-osd@2
On Wed, Jun 28, 2023 at 8:31 AM Niklas Hambüchen <mail(a)nh2.me> wrote:
Hi Alvaro,
--
Alexander E. Patrakov
Can you post the entire Ceph status output?
Pasting here since it is short
cluster:
id: d9000ec0-93c2-479f-bd5d-94ae9673e347
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m)
mgr: node-5(active, since 7d), standbys: node-6, node-4
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 5d), 36 in (since 6d)
data:
volumes: 1/1 healthy
pools: 3 pools, 832 pgs
objects: 506.83M objects, 67 TiB
usage: 207 TiB used, 232 TiB / 439 TiB avail
pgs: 826 active+clean
5 active+clean+scrubbing+deep
1 active+clean+inconsistent
io:
client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error
if a scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD
"2" already 7 days ago, and "list-inconsistent-obj" has not worked at
any time since then.
grep -Hn 'ERR'
/var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head
: candidate had a read error
/var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects
/var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors
The time "2023-06-16T16:29:57" above is the time at which the disk that carried
OSD "2" broke, its logs around the time are:
/var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1
bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output
error)
/var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1
log_channel(cluster) log [ERR] : 2.a6 missing primary copy of
2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28
-- note time jump by 3 days --
/var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000
7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000
7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000
7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1
** ERROR: osd init failed: (5) Input/output error
Does "candidate had a read error" on OSD "33" mean that a BlueStore
checksum error was detected on OSD "33" at the same time as the OSD
"2" disk failed?
If yes, maybe that is the explanation:
* pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD
33 detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20)
whose checksums disagree.
I am just guessing this, though.
Also, if this is correct, the next question would be: What is with OSD 20?
Since there is no error reported at all for OSD 20, I assume that its checksum agrees
with its data.
Now, can I find out whether OSD 20's checksum agrees with OSD 33's data?
(Side note: The disk of OSD 33 looks fine in smartctl.)
Thanks,
Niklas
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
4:41 p.m.
Hi Niklas,
please don't do any of the recovery steps yet! Your problem is almost certainly a
non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error
messeges you have:
ceph status/df/pool stats/health detail at 00:00:06:
cluster:
health: HEALTH_ERR
3 scrub errors
Possible data damage: 3 pgs inconsistent
After rebuilding the data, it still looked like:
cluster:
health: HEALTH_ERR
2 scrub errors
Possible data damage: 2 pgs inconsistent
What's the issue here? The issue is that the OGs have not been deep-scrubbed after
rebuild. The reply "no scrub data available" of the list-inconsistent is the
clue. The response to that is not to try manual repair but to issue a deep-scrub.
Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the
deep scrub reservation almost always gets cancelled very quickly. I got a script to force
repair/deep-scrub (I don't remember who sent it to me) and that gets the job done:
=====================
#!/bin/bash
[[ -r "/etc/profile.d/ceph.sh" ]] && source
"/etc/profile.d/ceph.sh"
for PG in $(ceph pg ls inconsistent -f json | jq -r .pg_stats[].pgid)
do
echo Checking inconsistent PG $PG
if ceph pg ls repair | grep -wq ${PG}
then
echo PG $PG is already repairing, skipping
continue
fi
# disable other scrubs
ceph osd set nodeep-scrub
ceph osd set noscrub
# bump up osd_max_scrubs
ACTING=$(ceph pg $PG query | jq -r .acting[])
for OSD in $ACTING
do
cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=3
--osd_scrub_during_recovery=true )
echo "executing: ${cmd[@]}"
"${cmd[@]}"
done
ceph pg repair $PG
sleep 10
for OSD in $ACTING
do
cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=1
--osd_scrub_during_recovery=false )
echo "executing: ${cmd[@]}"
"${cmd[@]}"
done
# disable other scrubs
ceph osd unset nodeep-scrub
ceph osd unset noscrub
done
===================
You can also just wait for the regular deep-scrub to happen.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Alexander E. Patrakov <patrakov(a)gmail.com>
Sent: Wednesday, June 28, 2023 5:24 AM
To: Niklas Hambüchen
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: 1 pg inconsistent and does not recover
Hello Niklas,
The explanation looks plausible.
What you can do is try extracting the PG from the dead OSD disk
(please make absolutely sure that the OSD daemon is stopped!!!) and
reinjecting it into some other OSD (again, stop the daemon during this
procedure). This extra copy should act as an arbiter.
The relevant commands are:
systemctl stop ceph-osd@2
systemctl stop ceph-osd@3 # or whatever other OSD exists on the same host
systemctl mask ceph-osd@2
systemctl mask ceph-osd@3
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --pgid
2.87 --op export --file /some/local/storage/pg-2.87.exp
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3/ --type
bluestore --pgid 2.87 --op import --file
/some/local/storage/pg-2.87.exp
systemctl unmask ceph-osd@3
systemctl start ceph-osd@3
systemctl unmask ceph-osd@2
On Wed, Jun 28, 2023 at 8:31 AM Niklas Hambüchen <mail(a)nh2.me> wrote:
Hi Alvaro,
--
Alexander E. Patrakov
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Can you post the entire Ceph status output?
Pasting here since it is short
cluster:
id: d9000ec0-93c2-479f-bd5d-94ae9673e347
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m)
mgr: node-5(active, since 7d), standbys: node-6, node-4
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 5d), 36 in (since 6d)
data:
volumes: 1/1 healthy
pools: 3 pools, 832 pgs
objects: 506.83M objects, 67 TiB
usage: 207 TiB used, 232 TiB / 439 TiB avail
pgs: 826 active+clean
5 active+clean+scrubbing+deep
1 active+clean+inconsistent
io:
client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error
if a scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD
"2" already 7 days ago, and "list-inconsistent-obj" has not worked at
any time since then.
grep -Hn 'ERR'
/var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head
: candidate had a read error
/var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects
/var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors
The time "2023-06-16T16:29:57" above is the time at which the disk that carried
OSD "2" broke, its logs around the time are:
/var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1
bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output
error)
/var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1
log_channel(cluster) log [ERR] : 2.a6 missing primary copy of
2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28
-- note time jump by 3 days --
/var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000
7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000
7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000
7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1
** ERROR: osd init failed: (5) Input/output error
Does "candidate had a read error" on OSD "33" mean that a BlueStore
checksum error was detected on OSD "33" at the same time as the OSD
"2" disk failed?
If yes, maybe that is the explanation:
* pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD
33 detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20)
whose checksums disagree.
I am just guessing this, though.
Also, if this is correct, the next question would be: What is with OSD 20?
Since there is no error reported at all for OSD 20, I assume that its checksum agrees
with its data.
Now, can I find out whether OSD 20's checksum agrees with OSD 33's data?
(Side note: The disk of OSD 33 looks fine in smartctl.)
Thanks,
Niklas
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:50 p.m.
Just for reference for everybody, the original source is
https://github.com/cernceph/ceph-scripts/blob/master/tools/scrubbing/autore…
maintained by Dan van der Ster. The repo as such is a rich source of good tools in general
and worth looking at before doing anything that requires more than 30 minutes searching
and reading documentation.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Wednesday, June 28, 2023 9:41 AM
To: Alexander E. Patrakov; Niklas Hambüchen
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover
Hi Niklas,
please don't do any of the recovery steps yet! Your problem is almost certainly a
non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error
messeges you have:
ceph status/df/pool stats/health detail at 00:00:06:
cluster:
health: HEALTH_ERR
3 scrub errors
Possible data damage: 3 pgs inconsistent
After rebuilding the data, it still looked like:
cluster:
health: HEALTH_ERR
2 scrub errors
Possible data damage: 2 pgs inconsistent
What's the issue here? The issue is that the OGs have not been deep-scrubbed after
rebuild. The reply "no scrub data available" of the list-inconsistent is the
clue. The response to that is not to try manual repair but to issue a deep-scrub.
Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the
deep scrub reservation almost always gets cancelled very quickly. I got a script to force
repair/deep-scrub (I don't remember who sent it to me) and that gets the job done:
=====================
#!/bin/bash
[[ -r "/etc/profile.d/ceph.sh" ]] && source
"/etc/profile.d/ceph.sh"
for PG in $(ceph pg ls inconsistent -f json | jq -r .pg_stats[].pgid)
do
echo Checking inconsistent PG $PG
if ceph pg ls repair | grep -wq ${PG}
then
echo PG $PG is already repairing, skipping
continue
fi
# disable other scrubs
ceph osd set nodeep-scrub
ceph osd set noscrub
# bump up osd_max_scrubs
ACTING=$(ceph pg $PG query | jq -r .acting[])
for OSD in $ACTING
do
cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=3
--osd_scrub_during_recovery=true )
echo "executing: ${cmd[@]}"
"${cmd[@]}"
done
ceph pg repair $PG
sleep 10
for OSD in $ACTING
do
cmd=( ceph tell osd.${OSD} injectargs -- --osd_max_scrubs=1
--osd_scrub_during_recovery=false )
echo "executing: ${cmd[@]}"
"${cmd[@]}"
done
# disable other scrubs
ceph osd unset nodeep-scrub
ceph osd unset noscrub
done
===================
You can also just wait for the regular deep-scrub to happen.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Alexander E. Patrakov <patrakov(a)gmail.com>
Sent: Wednesday, June 28, 2023 5:24 AM
To: Niklas Hambüchen
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: 1 pg inconsistent and does not recover
Hello Niklas,
The explanation looks plausible.
What you can do is try extracting the PG from the dead OSD disk
(please make absolutely sure that the OSD daemon is stopped!!!) and
reinjecting it into some other OSD (again, stop the daemon during this
procedure). This extra copy should act as an arbiter.
The relevant commands are:
systemctl stop ceph-osd@2
systemctl stop ceph-osd@3 # or whatever other OSD exists on the same host
systemctl mask ceph-osd@2
systemctl mask ceph-osd@3
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --pgid
2.87 --op export --file /some/local/storage/pg-2.87.exp
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3/ --type
bluestore --pgid 2.87 --op import --file
/some/local/storage/pg-2.87.exp
systemctl unmask ceph-osd@3
systemctl start ceph-osd@3
systemctl unmask ceph-osd@2
On Wed, Jun 28, 2023 at 8:31 AM Niklas Hambüchen <mail(a)nh2.me> wrote:
Hi Alvaro,
--
Alexander E. Patrakov
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Can you post the entire Ceph status output?
Pasting here since it is short
cluster:
id: d9000ec0-93c2-479f-bd5d-94ae9673e347
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum node-4,node-5,node-6 (age 52m)
mgr: node-5(active, since 7d), standbys: node-6, node-4
mds: 1/1 daemons up, 2 standby
osd: 36 osds: 36 up (since 5d), 36 in (since 6d)
data:
volumes: 1/1 healthy
pools: 3 pools, 832 pgs
objects: 506.83M objects, 67 TiB
usage: 207 TiB used, 232 TiB / 439 TiB avail
pgs: 826 active+clean
5 active+clean+scrubbing+deep
1 active+clean+inconsistent
io:
client: 18 MiB/s wr, 0 op/s rd, 5 op/s wr
sometimes list-inconsistent-obj throws that error
if a scrub job is still running.
This would be surprising to me, because I did the disk replacement of the broken OSD
"2" already 7 days ago, and "list-inconsistent-obj" has not worked at
any time since then.
grep -Hn 'ERR'
/var/log/ceph/ceph-osd.33.log
/var/log/ceph/ceph-osd.33.log:8005229:2023-06-16T16:29:57.704+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 shard 2 soid 2:e18c2025:::1001c78d046.00000000:head
: candidate had a read error
/var/log/ceph/ceph-osd.33.log:8018716:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 0 missing, 1 inconsistent objects
/var/log/ceph/ceph-osd.33.log:8018717:2023-06-16T20:03:26.923+0000 7f9a985e5640 -1
log_channel(cluster) log [ERR] : 2.87 deep-scrub 1 errors
The time "2023-06-16T16:29:57" above is the time at which the disk that carried
OSD "2" broke, its logs around the time are:
/var/log/ceph/ceph-osd.2.log:7855741:2023-06-16T16:29:57.690+0000 7fbae3cf7640 -1
bdev(0x7fbaeef6c400 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-5 ((5) Input/output
error)
/var/log/ceph/ceph-osd.2.log:7855743:2023-06-16T16:29:57.690+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8df449f9:::10016e7a962.00000000:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:7855747:2023-06-16T16:29:57.691+0000 7fba63064640 -1
log_channel(cluster) log [ERR] : 2.a6 missing primary copy of
2:65bd8cda:::10016ea4e67.00000000:head, will try copies on 17,28
-- note time jump by 3 days --
/var/log/ceph/ceph-osd.2.log:8096330:2023-06-19T06:42:48.712+0000 7fba62863640 -1
log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108684: -1867> 2023-06-19T06:42:48.712+0000
7fba62863640 -1 log_channel(cluster) log [ERR] : 2.b1 missing primary copy of
2:8d51be04:::1001d7b8447.00000334:head, will try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108766: -1785> 2023-06-19T06:42:49.035+0000
7fba6d879640 10 log_client will send 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8108770: -1781> 2023-06-19T06:42:49.525+0000
7fba7787f640 10 log_client logged 2023-06-19T06:42:48.713712+0000 osd.2 (osd.2) 179 :
cluster [ERR] 2.b1 missing primary copy of 2:8d51be04:::1001d7b8447.00000334:head, will
try copies on 19,32
/var/log/ceph/ceph-osd.2.log:8111339:2023-06-19T06:51:13.940+0000 7fb1518126c0 -1
** ERROR: osd init failed: (5) Input/output error
Does "candidate had a read error" on OSD "33" mean that a BlueStore
checksum error was detected on OSD "33" at the same time as the OSD
"2" disk failed?
If yes, maybe that is the explanation:
* pg 2.87 is backed by OSDs [33,2,20]; OSD 2's hardware broke during the scrub, OSD
33 detected a checksum error during the scrub, and thus we have 2 OSDs left (33 and 20)
whose checksums disagree.
I am just guessing this, though.
Also, if this is correct, the next question would be: What is with OSD 20?
Since there is no error reported at all for OSD 20, I assume that its checksum agrees
with its data.
Now, can I find out whether OSD 20's checksum agrees with OSD 33's data?
(Side note: The disk of OSD 33 looks fine in smartctl.)
Thanks,
Niklas
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:54 p.m.
On 6/28/23 09:41, Frank Schilder wrote:
Hi Niklas,
please don't do any of the recovery steps yet! Your problem is almost certainly a
non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error
messeges you have:
ceph status/df/pool stats/health detail at 00:00:06:
cluster:
health: HEALTH_ERR
3 scrub errors
Possible data damage: 3 pgs inconsistent
After rebuilding the data, it still looked like:
cluster:
health: HEALTH_ERR
2 scrub errors
Possible data damage: 2 pgs inconsistent
What's the issue here? The issue is that the OGs have not been deep-scrubbed after
rebuild. The reply "no scrub data available" of the list-inconsistent is the
clue. The response to that is not to try manual repair but to issue a deep-scrub.
Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the
deep scrub reservation almost always gets cancelled very quickly.
On what Ceph version do you have this issue? We use this command
everyday, hunderds of times, and it always works.
Or is this an issue when you have a degraded cluster?
Gr. Stefan
5:45 p.m.
Hi Stefan,
we run Octopus. The deep-scrub request is (immediately) cancelled if the PG/OSD is already
part of another (deep-)scrub or if some peering happens. As far as I understood, the
commands osd/pg deep-scrub and pg repair do not create persistent reservations. If you
issue this command, when does the PG actually start scrubbing? As soon as another one
finishes or when it is its natural turn? Do you monitor the scrub order to confirm it was
the manual command that initiated a scrub?
What I see is that the pg repair and the pg deep-scrub are almost immediately forgotten on
our cluster. This is most prominent with the repair command, which can be really hard to
get going and complete. Only an osd deep-scrub seems to have some effect. On the other
hand, when I run the script, which stops all operations that conflict with manual
reservations, the repair/deep-scrub actually start on request.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Stefan Kooman <stefan(a)bit.nl>
Sent: Wednesday, June 28, 2023 9:54 AM
To: Frank Schilder; Alexander E. Patrakov; Niklas Hambüchen
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover
On 6/28/23 09:41, Frank Schilder wrote:
Hi Niklas,
please don't do any of the recovery steps yet! Your problem is almost certainly a
non-issue. I had a failed disk with 3 scrub-errors, leading to the candidate read error
messeges you have:
ceph status/df/pool stats/health detail at 00:00:06:
cluster:
health: HEALTH_ERR
3 scrub errors
Possible data damage: 3 pgs inconsistent
After rebuilding the data, it still looked like:
cluster:
health: HEALTH_ERR
2 scrub errors
Possible data damage: 2 pgs inconsistent
What's the issue here? The issue is that the OGs have not been deep-scrubbed after
rebuild. The reply "no scrub data available" of the list-inconsistent is the
clue. The response to that is not to try manual repair but to issue a deep-scrub.
Unfortunately, the command "ceph pg deep-scrub ..." does not really work, the
deep scrub reservation almost always gets cancelled very quickly.
On what Ceph version do you have this issue? We use this command
everyday, hunderds of times, and it always works.
Or is this an issue when you have a degraded cluster?
Gr. Stefan
9:17 p.m.
On 6/28/23 10:45, Frank Schilder wrote:
Hi Stefan,
we run Octopus. The deep-scrub request is (immediately) cancelled if the PG/OSD is
already part of another (deep-)scrub or if some peering happens. As far as I understood,
the commands osd/pg deep-scrub and pg repair do not create persistent reservations. If you
issue this command, when does the PG actually start scrubbing? As soon as another one
finishes or when it is its natural turn? Do you monitor the scrub order to confirm it was
the manual command that initiated a scrub?
We request a deep-scrub ... a few seconds later it starts
deep-scrubbing. We do not verify in this process if the PG really did
start, but they do. See example from a PG below:
Jun 27 22:59:50 mon1 pg_scrub[2478540]: [27-06-2023 22:59:34] Scrub PG
5.48a (last deep-scrub: 2023-06-16T22:54:58.684038+0200)
^^ deep_scrub daemon requests a deep-scrub, based on latest deep-scrub
timestamp. After a couple of minutes it's deep-scrubbed. See below the
deep-scrub timestamp (info from a PG query of 5.48a):
"last_deep_scrub_stamp": "2023-06-27T23:06:01.823894+0200"
We have been using this in Octopus (actually since Luminous, but in a
different way). Now we are on Pacific.
Gr. Stefan
10:02 p.m.
I, too have the problem that `ceph pg deep-scrub` does not start the scrub, with Ceph
16.2.7.
# ceph pg deep-scrub 2.87
instructing pg 2.87 on osd.33 to deep-scrub
However then, on the machine where that osd.33 is:
# ceph daemon osd.33 dump_scrubs | jq . | head -n 13
[
{
"pgid": "2.87",
"sched_time": "1.000000",
"deadline": "1.000000",
"forced": true
},
{
"pgid": "1.4c",
"sched_time": "2023-06-15T16:30:58.696634+0000",
"deadline": "2023-07-09T09:01:50.420581+0000",
"forced": false
},
...
I am not sure what the "1.000000" is supposed to mean here for the problematic
OSD.
As shown, other OSDs have a normal string ISO time.
If I request scrubbing a PG that's on a different OSD (but on the same machine ad the
problematic one), then the deep-scrub starts immediately.
Niklas
11:41 p.m.
Hi Stefan,
after you wrote that you issue hundreds of deep-scrub commands per day I was already
suspecting something like
[...] deep_scrub daemon requests a deep-scrub [...]
Its not a Minion you hired that types these commends every so many seconds by hand and
hopes for the best. My guess is rather that you are actually running a cluster
specifically configured for manual scrub scheduling as was discussed in a thread some time
ago to solve the problem of "not deep scrubbed in time" messages due to the
built-in scrub scheduler not using the last-scrubbed timestamp for priority (among other
things).
On such a system I would not be surprised that these commands have their desired effect.
To know why it works for you it would be helpful to disclose the whole story, for example
what ceph config parameters are active within the context of your daemon executing the
deep-scrub instructions and what other ceph-commands surround it in the same way that the
pg repair is surrounded by injectargs instructions in the script I posted.
It doesn't work like that on a ceph cluster with default config. For example, on our
cluster there is a very high likelihood that at least one OSD of any PG is part of a scrub
at any time already. In that case, if a PG is not eligible for scrubbing because one of
its OSDs has already max-scrubs (default=1) scrubs running, the reservation has no
observable effect.
Some time ago I had a ceph-user thread discussing exactly that, I wanted to increase the
concurrent scrubs running without increasing max-scrubs. The default scheduler seems to be
very poor with ordering scrubs in such a way that a maximum number of pgs is scrubbed at
any given time. One of the suggestions was to run manual scheduling, which seems exactly
like what you are doing.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Stefan Kooman <stefan(a)bit.nl>
Sent: Wednesday, June 28, 2023 2:17 PM
To: Frank Schilder; Alexander E. Patrakov; Niklas Hambüchen
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover
On 6/28/23 10:45, Frank Schilder wrote:
Hi Stefan,
we run Octopus. The deep-scrub request is (immediately) cancelled if the PG/OSD is
already part of another (deep-)scrub or if some peering happens. As far as I understood,
the commands osd/pg deep-scrub and pg repair do not create persistent reservations. If you
issue this command, when does the PG actually start scrubbing? As soon as another one
finishes or when it is its natural turn? Do you monitor the scrub order to confirm it was
the manual command that initiated a scrub?
We request a deep-scrub ... a few seconds later it starts
deep-scrubbing. We do not verify in this process if the PG really did
start, but they do. See example from a PG below:
Jun 27 22:59:50 mon1 pg_scrub[2478540]: [27-06-2023 22:59:34] Scrub PG
5.48a (last deep-scrub: 2023-06-16T22:54:58.684038+0200)
^^ deep_scrub daemon requests a deep-scrub, based on latest deep-scrub
timestamp. After a couple of minutes it's deep-scrubbed. See below the
deep-scrub timestamp (info from a PG query of 5.48a):
"last_deep_scrub_stamp": "2023-06-27T23:06:01.823894+0200"
We have been using this in Octopus (actually since Luminous, but in a
different way). Now we are on Pacific.
Gr. Stefan
29 Jun
29 Jun
4:26 a.m.
Frank,
high likelihood that at least one OSD of any PG is
part of a scrub at any time already. In that case, if a PG is not eligible for scrubbing
because one of its OSDs has already max-scrubs (default=1) scrubs running, the reservation
has no observable effect.
This is a great hint.
I have increased the number of scrubs per OSD from 1 to 3 using `ceph config set osd
osd_max_scrubs 3`.
Now the problematic PG is scrubbing in `ceph pg ls`:
active+clean+scrubbing+deep+inconsistent
I will report what happens.
If this works, I think we should file an issue to motivate that Ceph should prioritise
scrubs that will result in fixing issues over normal routine scrubs.
9:11 p.m.
On 28/06/2023 21:26, Niklas Hambüchen wrote:
I have increased the number of scrubs per OSD from 1
to 3 using `ceph config set osd osd_max_scrubs 3`.
Now the problematic PG is scrubbing in `ceph pg ls`:
active+clean+scrubbing+deep+inconsistent
This succeeded!
The deep-scrub fixed the PG and the cluster is healthy again.
Thanks a lot!
So indeed the issue was that the deep-scrub I had asked for was simply never scheduled
because Ceph always picked some other scrub to do first on the relevant OSD.
Increasing `osd_max_scrubs` beyond 1 made it possible to force the scrub to start.
I conclude that most of the information online, including the Ceph docs, does not give the
correct advice when recommending `ceph pg repair`.
Instead, the docs should make clear that a scrub will fix such issues without involvement
of `ceph pg repair`.
I find lack of docs disturbing, because a disk failing and being replaced is an extremely
common operation for storage cluster.
Including some relevant logs of the scrub recovery:
# grep '\b2\.87\b' /var/log/ceph/ceph-osd.33.log | grep deep
2023-05-16T16:33:58.398+0000 7f9a985e5640 0 log_channel(cluster) log [DBG] : 2.87
deep-scrub ok
2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87
deep-scrub 0 missing, 1 inconsistent objects
2023-06-16T20:03:26.923+0000 7f9a985e5640 -1 log_channel(cluster) log [ERR] : 2.87
deep-scrub 1 errors
2023-06-26T05:06:17.412+0000 7f9b15bfe640 0 log_channel(cluster) log [INF] : osd.33
pg 2.87 Deep scrub errors, upgrading scrub to deep-scrub
2023-06-29T10:14:07.791+0000 7f9a985e5640 0 log_channel(cluster) log [DBG] : 2.87
deep-scrub ok
ceph.log:
2023-06-29T10:14:07.792432+0000 osd.33 (osd.33) 938 : cluster [DBG] 2.87 deep-scrub
ok
2023-06-29T10:14:09.311257+0000 mgr.node-5 (mgr.2454216) 385434 : cluster [DBG] pgmap
v385836: 832 pgs: 1 active+clean+scrubbing, 17 active+clean+scrubbing+deep, 814
active+clean; 68 TiB data, 210 TiB used, 229 TiB / 439 TiB avail; 80 MiB/s rd, 40 MiB/s
wr, 45 op/s
2023-06-29T10:14:09.427733+0000 mon.node-4 (mon.0) 20923054 : cluster [INF] Health
check cleared: OSD_SCRUB_ERRORS (was: 1 scrub errors)
2023-06-29T10:14:09.427758+0000 mon.node-4 (mon.0) 20923055 : cluster [INF] Health
check cleared: PG_DAMAGED (was: Possible data damage: 1 pg inconsistent)
2023-06-29T10:14:09.427786+0000 mon.node-4 (mon.0) 20923056 : cluster [INF] Cluster
is now healthy
From this, it seems bad that Ceph did not manage to schedule the cluster-fixing scrub
within 7 days of the faulty disk being replaced, nor managed to schedule a human-requested
scrub within 2 days.
What mechanism in Ceph decides the scheduling of scrubs?
I see the config value `osd_requested_scrub_priority` which is for "the priority set
for user requested scrub on the work queue", but I cannot tell if this also affects
scrub start scheduling, or only the priority of IO operations vs e.g. client operations
once a scrub has already been started.
11:27 p.m.
A repair always comes with a deep scrub. You can replace it if you want.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Niklas Hambüchen <mail(a)nh2.me>
Sent: Wednesday, June 28, 2023 3:05 PM
To: Frank Schilder; Alexander E. Patrakov
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: 1 pg inconsistent and does not recover
Hi Frank,
The response to that is not to try manual repair but
to issue a deep-scrub.
I am a bit confused, because in your script you do issue "ceph pg repair", not a
scrub.
328
days inactive
330
days old
15 comments
5 participants
participants (5)
-
Alexander E. Patrakov -
Alvaro Soto -
Frank Schilder -
Niklas Hambüchen -
Stefan Kooman