upgrade problem nautilus 14.2.15 -> 14.2.18? (Broken ceph!)
Hi I'm in a bit of a panic :-( Recently we started attempting to configure a radosgw to our ceph cluster, which was until now only doing cephfs (and rbd wss working as well). We were messing about with ceph-ansible, as this was how we originally installed the cluster. Anyway, it installed nautilus 14.2.18 on the radosgw and I though it would be good to pull up the rest of the cluster to that level as well using our tried and tested ceph upgrade script (it basically does an update of all ceph nodes one by one and checks whether ceph is ok again before doing the next) After the 3rd mon/mgr was done, all pg's were unavailable :-( obviously, the script is not continuing, but ceph is also broken now... The message deceptively is: HEALTH_WARN Reduced data availability: 5568 pgs inactive That's all PGs! I tried as a desperate measure to upgrade one ceph OSD node, but that broke as well, the osd service on that node gets an interrupt from the kernel.... the versions are now like: 20:29 [root@cephmon1 ~]# ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156 }, "mds": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158, "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6 } } 12 OSDs are down # ceph -s cluster: id: b489547c-ba50-4745-a914-23eb78e0e5dc health: HEALTH_WARN Reduced data availability: 5568 pgs inactive services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m) mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2 mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722 remapped pgs data: pools: 12 pools, 5568 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 5568 unknown progress: Rebalancing after osd.103 marked in [..............................]
netstat -anp | grep LISTEN | grep mgr has it bound to 127.0.0.1 ? (also check the other daemons). If so this is another case of https://tracker.ceph.com/issues/49938 -- dan On Thu, Mar 25, 2021 at 8:34 PM Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
Hi
I'm in a bit of a panic :-(
Recently we started attempting to configure a radosgw to our ceph cluster, which was until now only doing cephfs (and rbd wss working as well). We were messing about with ceph-ansible, as this was how we originally installed the cluster. Anyway, it installed nautilus 14.2.18 on the radosgw and I though it would be good to pull up the rest of the cluster to that level as well using our tried and tested ceph upgrade script (it basically does an update of all ceph nodes one by one and checks whether ceph is ok again before doing the next)
After the 3rd mon/mgr was done, all pg's were unavailable :-( obviously, the script is not continuing, but ceph is also broken now...
The message deceptively is: HEALTH_WARN Reduced data availability: 5568 pgs inactive
That's all PGs!
I tried as a desperate measure to upgrade one ceph OSD node, but that broke as well, the osd service on that node gets an interrupt from the kernel....
the versions are now like: 20:29 [root@cephmon1 ~]# ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156 }, "mds": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158, "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6 } }
12 OSDs are down
# ceph -s cluster: id: b489547c-ba50-4745-a914-23eb78e0e5dc health: HEALTH_WARN Reduced data availability: 5568 pgs inactive
services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m) mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2 mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722 remapped pgs
data: pools: 12 pools, 5568 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 5568 unknown
progress: Rebalancing after osd.103 marked in [..............................]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr tcp 0 0 127.0.0.1:6801 0.0.0.0:* LISTEN 1310/ceph-mgr tcp 0 0 127.0.0.1:6800 0.0.0.0:* LISTEN 1310/ceph-mgr tcp6 0 0 :::8443 :::* LISTEN 1310/ceph-mgr tcp6 0 0 :::9283 :::* LISTEN 1310/ceph-mgr unix 2 [ ACC ] STREAM LISTENING 26205 1564/master private/tlsmgr unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr /var/run/ceph/ceph-mgr.cephmon1.asok Looks like :-( /Simon
So do this. Get the ip of the host running the mgr, and put it this in the config file: [global] public addr = <mgr ip> cluster addr = <mgr ip> Then restart your mgr. IMHO we really need a 14.2.19 with this fixed asap. On Thu, Mar 25, 2021 at 8:47 PM Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr tcp 0 0 127.0.0.1:6801 0.0.0.0:* LISTEN 1310/ceph-mgr tcp 0 0 127.0.0.1:6800 0.0.0.0:* LISTEN 1310/ceph-mgr tcp6 0 0 :::8443 :::* LISTEN 1310/ceph-mgr tcp6 0 0 :::9283 :::* LISTEN 1310/ceph-mgr unix 2 [ ACC ] STREAM LISTENING 26205 1564/master private/tlsmgr unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr /var/run/ceph/ceph-mgr.cephmon1.asok
Looks like :-(
/Simon
On 3/25/21 8:47 PM, Simon Oosthoek wrote:
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr tcp 0 0 127.0.0.1:6801 0.0.0.0:* LISTEN 1310/ceph-mgr tcp 0 0 127.0.0.1:6800 0.0.0.0:* LISTEN 1310/ceph-mgr tcp6 0 0 :::8443 :::* LISTEN 1310/ceph-mgr tcp6 0 0 :::9283 :::* LISTEN 1310/ceph-mgr unix 2 [ ACC ] STREAM LISTENING 26205 1564/master private/tlsmgr unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr /var/run/ceph/ceph-mgr.cephmon1.asok
Looks like :-(
Ok, but that is easily fixable: ceph config set osd.$id public_addr your_ip_here Or you can put that in the ceph.conf for the OSDs on each storage server. Do you have a cluster network as well? If so you should set that IP too. Only when you run IPv6 only and have not yet set ms_bind_ipv4=false you should not do this. In that case you first have to make sure you set ms_bind_ipv4=false. As soon as your OSDs are bound to their correct IP again they can peer with each other and it will fix itself. @Ceph devs: a 14.2.19 with a fix for this issue would avoid other people running into this issue. Gr. Stefan
On 25/03/2021 20:56, Stefan Kooman wrote:
On 3/25/21 8:47 PM, Simon Oosthoek wrote:
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
# netstat -anp | grep LISTEN | grep mgr tcp 0 0 127.0.0.1:6801 0.0.0.0:* LISTEN 1310/ceph-mgr tcp 0 0 127.0.0.1:6800 0.0.0.0:* LISTEN 1310/ceph-mgr tcp6 0 0 :::8443 :::* LISTEN 1310/ceph-mgr tcp6 0 0 :::9283 :::* LISTEN 1310/ceph-mgr unix 2 [ ACC ] STREAM LISTENING 26205 1564/master private/tlsmgr unix 2 [ ACC ] STREAM LISTENING 26410 1310/ceph-mgr /var/run/ceph/ceph-mgr.cephmon1.asok
Looks like :-(
Ok, but that is easily fixable:
ceph config set osd.$id public_addr your_ip_here
Or you can put that in the ceph.conf for the OSDs on each storage server.
Do you have a cluster network as well? If so you should set that IP too.
Only when you run IPv6 only and have not yet set ms_bind_ipv4=false you should not do this. In that case you first have to make sure you set ms_bind_ipv4=false.
As soon as your OSDs are bound to their correct IP again they can peer with each other and it will fix itself.
@Ceph devs: a 14.2.19 with a fix for this issue would avoid other people running into this issue.
Gr. Stefan
Hoi Stefan tnx, I only have one network (25Gbit should be enough), after fixing the mon/mgr nodes and the one OSD node that I upgraded, the cluster seems to be recovering. At first I understood Dan's fix to put the mgr's address in all nodes' configs, but after watching the errors, I changed it to the node's own address on each node... I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19 will be available quickly. Cheers /Simon
Any time frame on 14.2.19? On Fri, Mar 26, 2021, 1:43 AM Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Finally master is merged now
k
Sent from my iPhone
On 25 Mar 2021, at 23:09, Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19 will be available quickly.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 25/03/2021 21:08, Simon Oosthoek wrote:
I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19 will be available quickly.
Hi Dan, Just FYI, I upgraded the cluster this week to 14.2.19 and all systems are good now. I've removed the workaround configuration in the /etc/ceph/ceph.conf again. Thanks for the quick help at the time! Cheers /Simon
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
has it bound to 127.0.0.1 ?
(also check the other daemons).
If so this is another case of https://tracker.ceph.com/issues/49938
Do you have any idea for a workaround (or should I downgrade?). I'm running ceph on ubuntu 18.04 LTS this seems to be happening on the mons/mgrs and osds Cheers /Simon
-- dan
On Thu, Mar 25, 2021 at 8:34 PM Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
Hi
I'm in a bit of a panic :-(
Recently we started attempting to configure a radosgw to our ceph cluster, which was until now only doing cephfs (and rbd wss working as well). We were messing about with ceph-ansible, as this was how we originally installed the cluster. Anyway, it installed nautilus 14.2.18 on the radosgw and I though it would be good to pull up the rest of the cluster to that level as well using our tried and tested ceph upgrade script (it basically does an update of all ceph nodes one by one and checks whether ceph is ok again before doing the next)
After the 3rd mon/mgr was done, all pg's were unavailable :-( obviously, the script is not continuing, but ceph is also broken now...
The message deceptively is: HEALTH_WARN Reduced data availability: 5568 pgs inactive
That's all PGs!
I tried as a desperate measure to upgrade one ceph OSD node, but that broke as well, the osd service on that node gets an interrupt from the kernel....
the versions are now like: 20:29 [root@cephmon1 ~]# ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156 }, "mds": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158, "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6 } }
12 OSDs are down
# ceph -s cluster: id: b489547c-ba50-4745-a914-23eb78e0e5dc health: HEALTH_WARN Reduced data availability: 5568 pgs inactive
services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m) mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2 mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722 remapped pgs
data: pools: 12 pools, 5568 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 5568 unknown
progress: Rebalancing after osd.103 marked in [..............................]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
In each host's ceph.conf, put: [global] public addr = <host ip> cluster addr = <host ip> Or use the public network / cluster network options = a.b.c.0/24 or however your network is defined. -- dan On Thu, Mar 25, 2021 at 8:50 PM Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
On 25/03/2021 20:42, Dan van der Ster wrote:
netstat -anp | grep LISTEN | grep mgr
has it bound to 127.0.0.1 ?
(also check the other daemons).
If so this is another case of https://tracker.ceph.com/issues/49938
Do you have any idea for a workaround (or should I downgrade?). I'm running ceph on ubuntu 18.04 LTS
this seems to be happening on the mons/mgrs and osds
Cheers
/Simon
-- dan
On Thu, Mar 25, 2021 at 8:34 PM Simon Oosthoek <s.oosthoek@science.ru.nl> wrote:
Hi
I'm in a bit of a panic :-(
Recently we started attempting to configure a radosgw to our ceph cluster, which was until now only doing cephfs (and rbd wss working as well). We were messing about with ceph-ansible, as this was how we originally installed the cluster. Anyway, it installed nautilus 14.2.18 on the radosgw and I though it would be good to pull up the rest of the cluster to that level as well using our tried and tested ceph upgrade script (it basically does an update of all ceph nodes one by one and checks whether ceph is ok again before doing the next)
After the 3rd mon/mgr was done, all pg's were unavailable :-( obviously, the script is not continuing, but ceph is also broken now...
The message deceptively is: HEALTH_WARN Reduced data availability: 5568 pgs inactive
That's all PGs!
I tried as a desperate measure to upgrade one ceph OSD node, but that broke as well, the osd service on that node gets an interrupt from the kernel....
the versions are now like: 20:29 [root@cephmon1 ~]# ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156 }, "mds": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158, "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6 } }
12 OSDs are down
# ceph -s cluster: id: b489547c-ba50-4745-a914-23eb78e0e5dc health: HEALTH_WARN Reduced data availability: 5568 pgs inactive
services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m) mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2 mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722 remapped pgs
data: pools: 12 pools, 5568 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 5568 unknown
progress: Rebalancing after osd.103 marked in [..............................]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Dan van der Ster
-
Konstantin Shalygin
-
Sasha Litvak
-
Simon Oosthoek
-
Stefan Kooman