25 Mar
2021
25 Mar
'21
7:33 p.m.
Hi
I'm in a bit of a panic :-(
Recently we started attempting to configure a radosgw to our ceph
cluster, which was until now only doing cephfs (and rbd wss working as
well). We were messing about with ceph-ansible, as this was how we
originally installed the cluster. Anyway, it installed nautilus 14.2.18
on the radosgw and I though it would be good to pull up the rest of the
cluster to that level as well using our tried and tested ceph upgrade
script (it basically does an update of all ceph nodes one by one and
checks whether ceph is ok again before doing the next)
After the 3rd mon/mgr was done, all pg's were unavailable :-(
obviously, the script is not continuing, but ceph is also broken now...
The message deceptively is: HEALTH_WARN Reduced data availability: 5568
pgs inactive
That's all PGs!
I tried as a desperate measure to upgrade one ceph OSD node, but that
broke as well, the osd service on that node gets an interrupt from the
kernel....
the versions are now like:
20:29 [root@cephmon1 ~]# ceph versions
{
"mon": {
"ceph version 14.2.18
(befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
},
"mgr": {
"ceph version 14.2.18
(befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
},
"osd": {
"ceph version 14.2.15
(afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156
},
"mds": {
"ceph version 14.2.15
(afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2
},
"overall": {
"ceph version 14.2.15
(afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158,
"ceph version 14.2.18
(befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6
}
}
12 OSDs are down
# ceph -s
cluster:
id: b489547c-ba50-4745-a914-23eb78e0e5dc
health: HEALTH_WARN
Reduced data availability: 5568 pgs inactive
services:
mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m)
mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2
mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby
osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722
remapped pgs
data:
pools: 12 pools, 5568 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 100.000% pgs unknown
5568 unknown
progress:
Rebalancing after osd.103 marked in
[..............................]
25 Mar
25 Mar
7:42 p.m.
netstat -anp | grep LISTEN | grep mgr
has it bound to 127.0.0.1 ?
(also check the other daemons).
If so this is another case of https://tracker.ceph.com/issues/49938
-- dan
On Thu, Mar 25, 2021 at 8:34 PM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote:
>
> Hi
>
> I'm in a bit of a panic :-(
>
> Recently we started attempting to configure a radosgw to our ceph
> cluster, which was until now only doing cephfs (and rbd wss working as
> well). We were messing about with ceph-ansible, as this was how we
> originally installed the cluster. Anyway, it installed nautilus 14.2.18
> on the radosgw and I though it would be good to pull up the rest of the
> cluster to that level as well using our tried and tested ceph upgrade
> script (it basically does an update of all ceph nodes one by one and
> checks whether ceph is ok again before doing the next)
>
> After the 3rd mon/mgr was done, all pg's were unavailable :-(
> obviously, the script is not continuing, but ceph is also broken now...
>
> The message deceptively is: HEALTH_WARN Reduced data availability: 5568
> pgs inactive
>
> That's all PGs!
>
> I tried as a desperate measure to upgrade one ceph OSD node, but that
> broke as well, the osd service on that node gets an interrupt from the
> kernel....
>
> the versions are now like:
> 20:29 [root@cephmon1 ~]# ceph versions
> {
> "mon": {
> "ceph version 14.2.18
> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
> },
> "mgr": {
> "ceph version 14.2.18
> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
> },
> "osd": {
> "ceph version 14.2.15
> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156
> },
> "mds": {
> "ceph version 14.2.15
> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2
> },
> "overall": {
> "ceph version 14.2.15
> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158,
> "ceph version 14.2.18
> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6
> }
> }
>
>
> 12 OSDs are down
>
> # ceph -s
> cluster:
> id: b489547c-ba50-4745-a914-23eb78e0e5dc
> health: HEALTH_WARN
> Reduced data availability: 5568 pgs inactive
>
> services:
> mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m)
> mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2
> mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby
> osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722
> remapped pgs
>
> data:
> pools: 12 pools, 5568 pgs
> objects: 0 objects, 0 B
> usage: 0 B used, 0 B / 0 B avail
> pgs: 100.000% pgs unknown
> 5568 unknown
>
> progress:
> Rebalancing after osd.103 marked in
> [..............................]
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:47 p.m.
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr
tcp 0 0 127.0.0.1:6801 0.0.0.0:*
LISTEN 1310/ceph-mgr
tcp 0 0 127.0.0.1:6800 0.0.0.0:*
LISTEN 1310/ceph-mgr
tcp6 0 0 :::8443 :::*
LISTEN 1310/ceph-mgr
tcp6 0 0 :::9283 :::*
LISTEN 1310/ceph-mgr
unix 2 [ ACC ] STREAM LISTENING 26205 1564/master
private/tlsmgr
unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr
/var/run/ceph/ceph-mgr.cephmon1.asok
Looks like :-(
/Simon
7:49 p.m.
So do this. Get the ip of the host running the mgr, and put it this in
the config file:
[global]
public addr = <mgr ip>
cluster addr = <mgr ip>
Then restart your mgr.
IMHO we really need a 14.2.19 with this fixed asap.
On Thu, Mar 25, 2021 at 8:47 PM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote:
>
> On 25/03/2021 20:42, Dan van der Ster wrote:
> > netstat -anp | grep LISTEN | grep mgr
>
> # netstat -anp | grep LISTEN | grep mgr
> tcp 0 0 127.0.0.1:6801 0.0.0.0:*
> LISTEN 1310/ceph-mgr
> tcp 0 0 127.0.0.1:6800 0.0.0.0:*
> LISTEN 1310/ceph-mgr
> tcp6 0 0 :::8443 :::*
> LISTEN 1310/ceph-mgr
> tcp6 0 0 :::9283 :::*
> LISTEN 1310/ceph-mgr
> unix 2 [ ACC ] STREAM LISTENING 26205 1564/master
> private/tlsmgr
> unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr
> /var/run/ceph/ceph-mgr.cephmon1.asok
>
>
> Looks like :-(
>
> /Simon
7:56 p.m.
On 3/25/21 8:47 PM, Simon Oosthoek wrote:
On 25/03/2021 20:42, Dan van der Ster wrote:
Ok, but that is easily fixable:
ceph config set osd.$id public_addr your_ip_here
Or you can put that in the ceph.conf for the OSDs on each storage server.
Do you have a cluster network as well? If so you should set that IP too.
Only when you run IPv6 only and have not yet set ms_bind_ipv4=false you
should not do this. In that case you first have to make sure you set
ms_bind_ipv4=false.
As soon as your OSDs are bound to their correct IP again they can peer
with each other and it will fix itself.
@Ceph devs: a 14.2.19 with a fix for this issue would avoid other people
running into this issue.
Gr. Stefan
netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr
tcp 0 0 127.0.0.1:6801 0.0.0.0:* LISTEN
1310/ceph-mgr
tcp 0 0 127.0.0.1:6800 0.0.0.0:* LISTEN
1310/ceph-mgr
tcp6 0 0 :::8443 :::* LISTEN 1310/ceph-mgr
tcp6 0 0 :::9283 :::* LISTEN 1310/ceph-mgr
unix 2 [ ACC ] STREAM LISTENING 26205 1564/master
private/tlsmgr
unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr
/var/run/ceph/ceph-mgr.cephmon1.asok
Looks like :-(
8:08 p.m.
On 25/03/2021 20:56, Stefan Kooman wrote:
On 3/25/21 8:47 PM, Simon Oosthoek wrote:
Hoi Stefan
tnx, I only have one network (25Gbit should be enough), after fixing the
mon/mgr nodes and the one OSD node that I upgraded, the cluster seems to
be recovering.
At first I understood Dan's fix to put the mgr's address in all nodes'
configs, but after watching the errors, I changed it to the node's own
address on each node...
I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19
will be available quickly.
Cheers
/Simon
On 25/03/2021 20:42, Dan van der Ster wrote:
Ok, but that is easily fixable:
ceph config set osd.$id public_addr your_ip_here
Or you can put that in the ceph.conf for the OSDs on each storage server.
Do you have a cluster network as well? If so you should set that IP too.
Only when you run IPv6 only and have not yet set ms_bind_ipv4=false you
should not do this. In that case you first have to make sure you set
ms_bind_ipv4=false.
As soon as your OSDs are bound to their correct IP again they can peer
with each other and it will fix itself.
@Ceph devs: a 14.2.19 with a fix for this issue would avoid other people
running into this issue.
Gr. Stefan netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr
tcp 0 0 127.0.0.1:6801 0.0.0.0:* LISTEN
1310/ceph-mgr
tcp 0 0 127.0.0.1:6800 0.0.0.0:* LISTEN
1310/ceph-mgr
tcp6 0 0 :::8443 :::* LISTEN
1310/ceph-mgr
tcp6 0 0 :::9283 :::* LISTEN
1310/ceph-mgr
unix 2 [ ACC ] STREAM LISTENING 26205 1564/master
private/tlsmgr
unix 2 [ ACC ] STREAM LISTENING 26410
1310/ceph-mgr /var/run/ceph/ceph-mgr.cephmon1.asok
Looks like :-( 
30 Mar
30 Mar
1:01 p.m.
Any time frame on 14.2.19?
On Fri, Mar 26, 2021, 1:43 AM Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
Finally master is merged now
k
Sent from my iPhone
On 25 Mar 2021, at 23:09, Simon Oosthoek
<s.oosthoek(a)science.ru.nl>
wrote:
I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19
will be
available quickly.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9 Apr
9 Apr
8:17 a.m.
On 25/03/2021 21:08, Simon Oosthoek wrote:
I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19
will be available quickly.
Hi Dan,
Just FYI, I upgraded the cluster this week to 14.2.19 and all systems
are good now. I've removed the workaround configuration in the
/etc/ceph/ceph.conf again.
Thanks for the quick help at the time!
Cheers
/Simon
25 Mar
25 Mar
7:50 p.m.
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
has it bound to 127.0.0.1 ?
(also check the other daemons).
If so this is another case of https://tracker.ceph.com/issues/49938
Do you have any idea for a workaround (or should I downgrade?). I'm
running ceph on ubuntu 18.04 LTS
this seems to be happening on the mons/mgrs and osds
Cheers
/Simon
-- dan
On Thu, Mar 25, 2021 at 8:34 PM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote:
>
> Hi
>
> I'm in a bit of a panic :-(
>
> Recently we started attempting to configure a radosgw to our ceph
> cluster, which was until now only doing cephfs (and rbd wss working as
> well). We were messing about with ceph-ansible, as this was how we
> originally installed the cluster. Anyway, it installed nautilus 14.2.18
> on the radosgw and I though it would be good to pull up the rest of the
> cluster to that level as well using our tried and tested ceph upgrade
> script (it basically does an update of all ceph nodes one by one and
> checks whether ceph is ok again before doing the next)
>
> After the 3rd mon/mgr was done, all pg's were unavailable :-(
> obviously, the script is not continuing, but ceph is also broken now...
>
> The message deceptively is: HEALTH_WARN Reduced data availability: 5568
> pgs inactive
>
> That's all PGs!
>
> I tried as a desperate measure to upgrade one ceph OSD node, but that
> broke as well, the osd service on that node gets an interrupt from the
> kernel....
>
> the versions are now like:
> 20:29 [root@cephmon1 ~]# ceph versions
> {
> "mon": {
> "ceph version 14.2.18
> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
> },
> "mgr": {
> "ceph version 14.2.18
> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
> },
> "osd": {
> "ceph version 14.2.15
> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156
> },
> "mds": {
> "ceph version 14.2.15
> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2
> },
> "overall": {
> "ceph version 14.2.15
> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158,
> "ceph version 14.2.18
> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6
> }
> }
>
>
> 12 OSDs are down
>
> # ceph -s
> cluster:
> id: b489547c-ba50-4745-a914-23eb78e0e5dc
> health: HEALTH_WARN
> Reduced data availability: 5568 pgs inactive
>
> services:
> mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m)
> mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2
> mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby
> osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722
> remapped pgs
>
> data:
> pools: 12 pools, 5568 pgs
> objects: 0 objects, 0 B
> usage: 0 B used, 0 B / 0 B avail
> pgs: 100.000% pgs unknown
> 5568 unknown
>
> progress:
> Rebalancing after osd.103 marked in
> [..............................]
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:53 p.m.
In each host's ceph.conf, put:
[global]
public addr = <host ip>
cluster addr = <host ip>
Or use the public network / cluster network options = a.b.c.0/24 or
however your network is defined.
-- dan
On Thu, Mar 25, 2021 at 8:50 PM Simon Oosthoek <s.oosthoek(a)science.ru.nl> wrote:
>
> On 25/03/2021 20:42, Dan van der Ster wrote:
> > netstat -anp | grep LISTEN | grep mgr
> >
> > has it bound to 127.0.0.1 ?
> >
> > (also check the other daemons).
> >
> > If so this is another case of https://tracker.ceph.com/issues/49938
>
> Do you have any idea for a workaround (or should I downgrade?). I'm
> running ceph on ubuntu 18.04 LTS
>
> this seems to be happening on the mons/mgrs and osds
>
> Cheers
>
> /Simon
> >
> > -- dan
> >
> > On Thu, Mar 25, 2021 at 8:34 PM Simon Oosthoek <s.oosthoek(a)science.ru.nl>
wrote:
> >>
> >> Hi
> >>
> >> I'm in a bit of a panic :-(
> >>
> >> Recently we started attempting to configure a radosgw to our ceph
> >> cluster, which was until now only doing cephfs (and rbd wss working as
> >> well). We were messing about with ceph-ansible, as this was how we
> >> originally installed the cluster. Anyway, it installed nautilus 14.2.18
> >> on the radosgw and I though it would be good to pull up the rest of the
> >> cluster to that level as well using our tried and tested ceph upgrade
> >> script (it basically does an update of all ceph nodes one by one and
> >> checks whether ceph is ok again before doing the next)
> >>
> >> After the 3rd mon/mgr was done, all pg's were unavailable :-(
> >> obviously, the script is not continuing, but ceph is also broken now...
> >>
> >> The message deceptively is: HEALTH_WARN Reduced data availability: 5568
> >> pgs inactive
> >>
> >> That's all PGs!
> >>
> >> I tried as a desperate measure to upgrade one ceph OSD node, but that
> >> broke as well, the osd service on that node gets an interrupt from the
> >> kernel....
> >>
> >> the versions are now like:
> >> 20:29 [root@cephmon1 ~]# ceph versions
> >> {
> >> "mon": {
> >> "ceph version 14.2.18
> >> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
> >> },
> >> "mgr": {
> >> "ceph version 14.2.18
> >> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
> >> },
> >> "osd": {
> >> "ceph version 14.2.15
> >> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156
> >> },
> >> "mds": {
> >> "ceph version 14.2.15
> >> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2
> >> },
> >> "overall": {
> >> "ceph version 14.2.15
> >> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158,
> >> "ceph version 14.2.18
> >> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6
> >> }
> >> }
> >>
> >>
> >> 12 OSDs are down
> >>
> >> # ceph -s
> >> cluster:
> >> id: b489547c-ba50-4745-a914-23eb78e0e5dc
> >> health: HEALTH_WARN
> >> Reduced data availability: 5568 pgs inactive
> >>
> >> services:
> >> mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m)
> >> mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2
> >> mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby
> >> osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722
> >> remapped pgs
> >>
> >> data:
> >> pools: 12 pools, 5568 pgs
> >> objects: 0 objects, 0 B
> >> usage: 0 B used, 0 B / 0 B avail
> >> pgs: 100.000% pgs unknown
> >> 5568 unknown
> >>
> >> progress:
> >> Rebalancing after osd.103 marked in
> >> [..............................]
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
1112
days inactive
1127
days old
11 comments
5 participants
participants (5)
-
Dan van der Ster -
Konstantin Shalygin -
Sasha Litvak -
Simon Oosthoek -
Stefan Kooman