1 Feb
2021
1 Feb
'21
9:48 p.m.
Hi,
With 3 replicas, a pg hs 3 osds. If all those 3 osds are down,
the pg becomes unknow. Is that right?
If those 3 osds are replaced and in and on, is that pg going to
be eventually back to active? Or anything else has to be done
to fix it?
Thanks!
Tony
2 Feb
2 Feb
7:42 a.m.
On 01/02/2021 22:48, Tony Liu wrote:
Hi,
With 3 replicas, a pg hs 3 osds. If all those 3 osds are down,
the pg becomes unknow. Is that right?
Yes. As no OSD can report the status to the MONs.
If those 3 osds are replaced and in and on, is that pg
going to
be eventually back to active? Or anything else has to be done
to fix it?
If you can bring back the OSDs without wiping them: Yes
As you mention the word 'replaced' I was wondering what you mean by
that. If you replace the disks without data recovery the PGs will be lost.
So you need to bring back the OSDs with their data in tact for the PG to
come back online.
Wido
>
> Thanks!
> Tony
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
4:57 p.m.
I'm in a similar but not identical situation.
I was in the middle of a rebalance on a small test cluster, without about
1% of pgs degraded, and shut the cluster entirely down for maintenance. On
startup, many pgs are entirely unknown, and most stale. In fact most pgs
can't be queried! No mon failures. No obvious signs of OSD failure (and the
problem is too widespread for that.) Is there a specific way to force OSDs
to rescan and re-advertise their pgs? Is there a specific startup order
that fixes this, i.e., start all OSDs first and then start mons?
I'm baffled,
Jeremy
On Mon, Feb 1, 2021 at 10:43 PM Wido den Hollander <wido(a)42on.com> wrote:
On 01/02/2021 22:48, Tony Liu wrote:
--
Jeremy Austin
jhaustin(a)gmail.com
Hi,
With 3 replicas, a pg hs 3 osds. If all those 3 osds are down,
the pg becomes unknow. Is that right?
Yes. As no OSD can report the status to the MONs.
If those 3 osds are replaced and in and on, is
that pg going to
be eventually back to active? Or anything else has to be done
to fix it?
If you can bring back the OSDs without wiping them: Yes
As you mention the word 'replaced' I was wondering what you mean by
that. If you replace the disks without data recovery the PGs will be lost.
So you need to bring back the OSDs with their data in tact for the PG to
come back online.
Wido
Thanks!
Tony
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:38 p.m.
Thank you all for kind response!
This problem didn't happen naturally. It was caused by operation
mistake. Anyways, 3 OSDs were replaced by zapped disk. That caused
two unknown PGs. Data on those 2 PGs are permanently lost unfortunately.
"pg dump" shows unknown. "pg map <id>" shows those 3 replaced
OSDs.
"pg query <id>" can't find it. I did "osd force-create-pg
<id>" to
recreate them. PG map remains on those 3 OSDs.
Now, they are active+clean.
Tony
-----Original Message-----
From: Jeremy Austin <jhaustin(a)gmail.com>
Sent: Tuesday, February 2, 2021 8:58 AM
To: Wido den Hollander <wido(a)42on.com>
Cc: Tony Liu <tonyliu0592(a)hotmail.com>om>; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: is unknown pg going to be active after
osds are fixed?
I'm in a similar but not identical situation.
I was in the middle of a rebalance on a small test cluster, without
about 1% of pgs degraded, and shut the cluster entirely down for
maintenance. On startup, many pgs are entirely unknown, and most stale.
In fact most pgs can't be queried! No mon failures. No obvious signs of
OSD failure (and the problem is too widespread for that.) Is there a
specific way to force OSDs to rescan and re-advertise their pgs? Is
there a specific startup order that fixes this, i.e., start all OSDs
first and then start mons?
I'm baffled,
Jeremy
On Mon, Feb 1, 2021 at 10:43 PM Wido den Hollander <wido(a)42on.com
<mailto:wido@42on.com> > wrote:
On 01/02/2021 22:48, Tony Liu wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-
users(a)ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
--
Jeremy Austin
jhaustin(a)gmail.com <mailto:jhaustin@gmail.com>
Hi,
With 3 replicas, a pg hs 3 osds. If all those 3 osds are down,
the pg becomes unknow. Is that right?
Yes. As no OSD can report the status to the MONs.
If those 3 osds are replaced and in and on, is
that pg going to
be eventually back to active? Or anything else has to be done
to fix it?
If you can bring back the OSDs without wiping them: Yes
As you mention the word 'replaced' I was wondering what you mean by
that. If you replace the disks without data recovery the PGs will
be lost.
So you need to bring back the OSDs with their data in tact for the
PG to
come back online.
Wido
Thanks!
Tony
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-
users(a)ceph.io>
To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
1186
days inactive
1187
days old
3 comments
3 participants
participants (3)
-
Jeremy Austin -
Tony Liu -
Wido den Hollander