Hi, I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles. I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time. They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again. Here is some info about my cluster: ------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df) data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering ------------------------------------------- ------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 ------------------------------------------- At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing. What should I do to get my cluster back up? Just wait? Regards, Rodrigo Severo
Rodrigo; Are all your hosts using the same IP addresses as before the move? Is the new network structured the same? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Rodrigo Severo - Fábrica [mailto:rodrigo@fabricadeideias.com] Sent: Tuesday, February 04, 2020 8:40 AM To: ceph-users Subject: [ceph-users] All pgs peering indefinetely Hi, I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles. I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time. They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again. Here is some info about my cluster: ------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df) data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering ------------------------------------------- ------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 ------------------------------------------- At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing. What should I do to get my cluster back up? Just wait? Regards, Rodrigo Severo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Em ter., 4 de fev. de 2020 às 12:39, Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> escreveu:
Hi,
I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles.
I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time.
They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again.
Here is some info about my cluster:
------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops
services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df)
data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering -------------------------------------------
------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 -------------------------------------------
At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing.
What should I do to get my cluster back up? Just wait?
I just found out that I have a few pgs "stuck peering": ------------------------------------------- # ceph health detail | grep peering HEALTH_WARN 1 filesystem is degraded; 1 MDSs report slow metadata IOs; 2 osds down; 1 host (2 osds) down; Reduced data availability: 324 pgs inactive, 324 pgs peering; 7 daemons have recently crashed; 80 slow ops, oldest one blocked for 33 sec, daemons [osd.0,osd.1] have slow ops. PG_AVAILABILITY Reduced data availability: 324 pgs inactive, 324 pgs peering pg 1.39 is stuck peering for 14011.965915, current state peering, last acting [0,1] pg 1.3a is stuck peering for 14084.993947, current state peering, last acting [0,1] pg 1.3b is stuck peering for 14274.225311, current state peering, last acting [0,1] pg 1.3c is stuck peering for 15937.859532, current state peering, last acting [1,0] pg 1.3d is stuck peering for 15786.873447, current state peering, last acting [1,0] pg 1.3e is stuck peering for 15841.947891, current state peering, last acting [1,0] pg 1.3f is stuck peering for 15841.912853, current state peering, last acting [1,0] pg 1.40 is stuck peering for 14031.769901, current state peering, last acting [0,1] pg 1.41 is stuck peering for 14010.216124, current state peering, last acting [0,1] pg 1.42 is stuck peering for 15841.895446, current state peering, last acting [1,0] pg 1.43 is stuck peering for 15915.024413, current state peering, last acting [1,0] pg 1.44 is stuck peering for 13872.015272, current state peering, last acting [0,1] pg 1.45 is stuck peering for 15684.413850, current state peering, last acting [1,0] pg 1.46 is stuck peering for 15906.378461, current state peering, last acting [1,0] pg 1.47 is stuck peering for 14377.822032, current state peering, last acting [0,1] pg 1.48 is stuck peering for 14085.032316, current state peering, last acting [0,1] pg 1.49 is stuck peering for 14085.030366, current state peering, last acting [0,1] pg 1.4a is stuck peering for 14667.451862, current state peering, last acting [0,1] pg 1.4b is stuck peering for 14048.714764, current state peering, last acting [0,1] pg 1.4c is stuck peering for 13998.360919, current state peering, last acting [0,1] pg 1.4d is stuck peering for 15693.831021, current state peering, last acting [1,0] pg 2.38 is stuck peering for 15841.882464, current state peering, last acting [1,0] pg 2.39 is stuck peering for 15841.881968, current state peering, last acting [1,0] pg 2.3a is stuck peering for 14085.032520, current state peering, last acting [0,1] pg 2.3b is stuck inactive for 12717.975044, current state peering, last acting [0,1] pg 2.3c is stuck peering for 15841.947367, current state peering, last acting [1,0] pg 2.3d is stuck peering for 15732.221067, current state peering, last acting [1,0] pg 2.3e is stuck peering for 15938.007321, current state peering, last acting [0,1] pg 2.3f is stuck peering for 14084.992407, current state peering, last acting [0,1] pg 7.38 is stuck peering for 14080.942444, current state peering, last acting [3,4] pg 7.39 is stuck peering for 14048.869554, current state peering, last acting [3,4] pg 7.3a is stuck peering for 14048.869790, current state peering, last acting [3,4] pg 7.3b is stuck peering for 14080.943240, current state peering, last acting [3,4] pg 7.3c is stuck peering for 15842.114296, current state peering, last acting [4,3] pg 7.3d is stuck peering for 14048.870194, current state peering, last acting [3,4] pg 7.3e is stuck peering for 15842.105944, current state peering, last acting [4,3] pg 7.3f is stuck peering for 15842.111549, current state peering, last acting [4,3] pg 7.40 is stuck peering for 14048.869572, current state peering, last acting [3,4] pg 7.41 is stuck peering for 14048.868747, current state peering, last acting [3,4] pg 7.42 is stuck peering for 15845.175729, current state peering, last acting [4,3] pg 7.43 is stuck peering for 15842.105227, current state peering, last acting [4,3] pg 7.44 is stuck peering for 15845.196486, current state peering, last acting [4,3] pg 7.45 is stuck peering for 14048.869849, current state peering, last acting [3,4] pg 7.46 is stuck peering for 14080.942650, current state peering, last acting [3,4] pg 7.47 is stuck peering for 15845.197875, current state peering, last acting [4,3] pg 7.4a is stuck peering for 15842.113906, current state peering, last acting [4,3] pg 7.4b is stuck peering for 15845.197205, current state peering, last acting [4,3] pg 7.4c is stuck peering for 14048.869937, current state peering, last acting [3,4] pg 7.4d is stuck peering for 14048.869137, current state peering, last acting [3,4] pg 7.4e is stuck peering for 15842.111699, current state peering, last acting [4,3] pg 7.4f is stuck peering for 14080.943391, current state peering, last acting [3,4] ------------------------------------------- Why is that? How can I fix it? Rodrigo
I would guess that you have something preventing osd to osd communication on ports 6800-7300 or osd to mon communication on port 6789 and/or 3300. Respectfully, *Wes Dillingham* wes@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Feb 4, 2020 at 12:44 PM Rodrigo Severo - Fábrica < rodrigo@fabricadeideias.com> wrote:
Em ter., 4 de fev. de 2020 às 12:39, Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> escreveu:
Hi,
I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles.
I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time.
They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again.
Here is some info about my cluster:
------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has
slow ops
services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df)
data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering -------------------------------------------
------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 -------------------------------------------
At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing.
What should I do to get my cluster back up? Just wait?
I just found out that I have a few pgs "stuck peering":
------------------------------------------- # ceph health detail | grep peering HEALTH_WARN 1 filesystem is degraded; 1 MDSs report slow metadata IOs; 2 osds down; 1 host (2 osds) down; Reduced data availability: 324 pgs inactive, 324 pgs peering; 7 daemons have recently crashed; 80 slow ops, oldest one blocked for 33 sec, daemons [osd.0,osd.1] have slow ops. PG_AVAILABILITY Reduced data availability: 324 pgs inactive, 324 pgs peering pg 1.39 is stuck peering for 14011.965915, current state peering, last acting [0,1] pg 1.3a is stuck peering for 14084.993947, current state peering, last acting [0,1] pg 1.3b is stuck peering for 14274.225311, current state peering, last acting [0,1] pg 1.3c is stuck peering for 15937.859532, current state peering, last acting [1,0] pg 1.3d is stuck peering for 15786.873447, current state peering, last acting [1,0] pg 1.3e is stuck peering for 15841.947891, current state peering, last acting [1,0] pg 1.3f is stuck peering for 15841.912853, current state peering, last acting [1,0] pg 1.40 is stuck peering for 14031.769901, current state peering, last acting [0,1] pg 1.41 is stuck peering for 14010.216124, current state peering, last acting [0,1] pg 1.42 is stuck peering for 15841.895446, current state peering, last acting [1,0] pg 1.43 is stuck peering for 15915.024413, current state peering, last acting [1,0] pg 1.44 is stuck peering for 13872.015272, current state peering, last acting [0,1] pg 1.45 is stuck peering for 15684.413850, current state peering, last acting [1,0] pg 1.46 is stuck peering for 15906.378461, current state peering, last acting [1,0] pg 1.47 is stuck peering for 14377.822032, current state peering, last acting [0,1] pg 1.48 is stuck peering for 14085.032316, current state peering, last acting [0,1] pg 1.49 is stuck peering for 14085.030366, current state peering, last acting [0,1] pg 1.4a is stuck peering for 14667.451862, current state peering, last acting [0,1] pg 1.4b is stuck peering for 14048.714764, current state peering, last acting [0,1] pg 1.4c is stuck peering for 13998.360919, current state peering, last acting [0,1] pg 1.4d is stuck peering for 15693.831021, current state peering, last acting [1,0] pg 2.38 is stuck peering for 15841.882464, current state peering, last acting [1,0] pg 2.39 is stuck peering for 15841.881968, current state peering, last acting [1,0] pg 2.3a is stuck peering for 14085.032520, current state peering, last acting [0,1] pg 2.3b is stuck inactive for 12717.975044, current state peering, last acting [0,1] pg 2.3c is stuck peering for 15841.947367, current state peering, last acting [1,0] pg 2.3d is stuck peering for 15732.221067, current state peering, last acting [1,0] pg 2.3e is stuck peering for 15938.007321, current state peering, last acting [0,1] pg 2.3f is stuck peering for 14084.992407, current state peering, last acting [0,1] pg 7.38 is stuck peering for 14080.942444, current state peering, last acting [3,4] pg 7.39 is stuck peering for 14048.869554, current state peering, last acting [3,4] pg 7.3a is stuck peering for 14048.869790, current state peering, last acting [3,4] pg 7.3b is stuck peering for 14080.943240, current state peering, last acting [3,4] pg 7.3c is stuck peering for 15842.114296, current state peering, last acting [4,3] pg 7.3d is stuck peering for 14048.870194, current state peering, last acting [3,4] pg 7.3e is stuck peering for 15842.105944, current state peering, last acting [4,3] pg 7.3f is stuck peering for 15842.111549, current state peering, last acting [4,3] pg 7.40 is stuck peering for 14048.869572, current state peering, last acting [3,4] pg 7.41 is stuck peering for 14048.868747, current state peering, last acting [3,4] pg 7.42 is stuck peering for 15845.175729, current state peering, last acting [4,3] pg 7.43 is stuck peering for 15842.105227, current state peering, last acting [4,3] pg 7.44 is stuck peering for 15845.196486, current state peering, last acting [4,3] pg 7.45 is stuck peering for 14048.869849, current state peering, last acting [3,4] pg 7.46 is stuck peering for 14080.942650, current state peering, last acting [3,4] pg 7.47 is stuck peering for 15845.197875, current state peering, last acting [4,3] pg 7.4a is stuck peering for 15842.113906, current state peering, last acting [4,3] pg 7.4b is stuck peering for 15845.197205, current state peering, last acting [4,3] pg 7.4c is stuck peering for 14048.869937, current state peering, last acting [3,4] pg 7.4d is stuck peering for 14048.869137, current state peering, last acting [3,4] pg 7.4e is stuck peering for 15842.111699, current state peering, last acting [4,3] pg 7.4f is stuck peering for 14080.943391, current state peering, last acting [3,4] -------------------------------------------
Why is that? How can I fix it?
Rodrigo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Em ter., 4 de fev. de 2020 às 14:54, Wesley Dillingham <wes@wesdillingham.com> escreveu:
I would guess that you have something preventing osd to osd communication on ports 6800-7300 or osd to mon communication on port 6789 and/or 3300.
The 3 servers are on the same subnet. They are connect to a non-managed switch. And none have any firewall (iptables) rules blocking anything. They can ping one the other. Can you think about some other way that some traffic could be blocked? Or some other test I could do to check for connectivity? Regards, Rodrigo
Respectfully,
Wes Dillingham wes@wesdillingham.com LinkedIn
On Tue, Feb 4, 2020 at 12:44 PM Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> wrote:
Em ter., 4 de fev. de 2020 às 12:39, Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> escreveu:
Hi,
I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles.
I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time.
They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again.
Here is some info about my cluster:
------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops
services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df)
data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering -------------------------------------------
------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 -------------------------------------------
At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing.
What should I do to get my cluster back up? Just wait?
I just found out that I have a few pgs "stuck peering":
------------------------------------------- # ceph health detail | grep peering HEALTH_WARN 1 filesystem is degraded; 1 MDSs report slow metadata IOs; 2 osds down; 1 host (2 osds) down; Reduced data availability: 324 pgs inactive, 324 pgs peering; 7 daemons have recently crashed; 80 slow ops, oldest one blocked for 33 sec, daemons [osd.0,osd.1] have slow ops. PG_AVAILABILITY Reduced data availability: 324 pgs inactive, 324 pgs peering pg 1.39 is stuck peering for 14011.965915, current state peering, last acting [0,1] pg 1.3a is stuck peering for 14084.993947, current state peering, last acting [0,1] pg 1.3b is stuck peering for 14274.225311, current state peering, last acting [0,1] pg 1.3c is stuck peering for 15937.859532, current state peering, last acting [1,0] pg 1.3d is stuck peering for 15786.873447, current state peering, last acting [1,0] pg 1.3e is stuck peering for 15841.947891, current state peering, last acting [1,0] pg 1.3f is stuck peering for 15841.912853, current state peering, last acting [1,0] pg 1.40 is stuck peering for 14031.769901, current state peering, last acting [0,1] pg 1.41 is stuck peering for 14010.216124, current state peering, last acting [0,1] pg 1.42 is stuck peering for 15841.895446, current state peering, last acting [1,0] pg 1.43 is stuck peering for 15915.024413, current state peering, last acting [1,0] pg 1.44 is stuck peering for 13872.015272, current state peering, last acting [0,1] pg 1.45 is stuck peering for 15684.413850, current state peering, last acting [1,0] pg 1.46 is stuck peering for 15906.378461, current state peering, last acting [1,0] pg 1.47 is stuck peering for 14377.822032, current state peering, last acting [0,1] pg 1.48 is stuck peering for 14085.032316, current state peering, last acting [0,1] pg 1.49 is stuck peering for 14085.030366, current state peering, last acting [0,1] pg 1.4a is stuck peering for 14667.451862, current state peering, last acting [0,1] pg 1.4b is stuck peering for 14048.714764, current state peering, last acting [0,1] pg 1.4c is stuck peering for 13998.360919, current state peering, last acting [0,1] pg 1.4d is stuck peering for 15693.831021, current state peering, last acting [1,0] pg 2.38 is stuck peering for 15841.882464, current state peering, last acting [1,0] pg 2.39 is stuck peering for 15841.881968, current state peering, last acting [1,0] pg 2.3a is stuck peering for 14085.032520, current state peering, last acting [0,1] pg 2.3b is stuck inactive for 12717.975044, current state peering, last acting [0,1] pg 2.3c is stuck peering for 15841.947367, current state peering, last acting [1,0] pg 2.3d is stuck peering for 15732.221067, current state peering, last acting [1,0] pg 2.3e is stuck peering for 15938.007321, current state peering, last acting [0,1] pg 2.3f is stuck peering for 14084.992407, current state peering, last acting [0,1] pg 7.38 is stuck peering for 14080.942444, current state peering, last acting [3,4] pg 7.39 is stuck peering for 14048.869554, current state peering, last acting [3,4] pg 7.3a is stuck peering for 14048.869790, current state peering, last acting [3,4] pg 7.3b is stuck peering for 14080.943240, current state peering, last acting [3,4] pg 7.3c is stuck peering for 15842.114296, current state peering, last acting [4,3] pg 7.3d is stuck peering for 14048.870194, current state peering, last acting [3,4] pg 7.3e is stuck peering for 15842.105944, current state peering, last acting [4,3] pg 7.3f is stuck peering for 15842.111549, current state peering, last acting [4,3] pg 7.40 is stuck peering for 14048.869572, current state peering, last acting [3,4] pg 7.41 is stuck peering for 14048.868747, current state peering, last acting [3,4] pg 7.42 is stuck peering for 15845.175729, current state peering, last acting [4,3] pg 7.43 is stuck peering for 15842.105227, current state peering, last acting [4,3] pg 7.44 is stuck peering for 15845.196486, current state peering, last acting [4,3] pg 7.45 is stuck peering for 14048.869849, current state peering, last acting [3,4] pg 7.46 is stuck peering for 14080.942650, current state peering, last acting [3,4] pg 7.47 is stuck peering for 15845.197875, current state peering, last acting [4,3] pg 7.4a is stuck peering for 15842.113906, current state peering, last acting [4,3] pg 7.4b is stuck peering for 15845.197205, current state peering, last acting [4,3] pg 7.4c is stuck peering for 14048.869937, current state peering, last acting [3,4] pg 7.4d is stuck peering for 14048.869137, current state peering, last acting [3,4] pg 7.4e is stuck peering for 15842.111699, current state peering, last acting [4,3] pg 7.4f is stuck peering for 14080.943391, current state peering, last acting [3,4] -------------------------------------------
Why is that? How can I fix it?
Rodrigo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Rodrigo; Best bet would be to check logs. Check the OSD logs on the affected server. Check cluster logs on the MONs. Check OSD logs on other servers. Your Ceph version(s) and your OS distribution and version would also be useful to help you troubleshoot this OSD flapping issue. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Rodrigo Severo - Fábrica [mailto:rodrigo@fabricadeideias.com] Sent: Tuesday, February 04, 2020 11:05 AM To: Wesley Dillingham Cc: ceph-users Subject: [ceph-users] Re: All pgs peering indefinetely Em ter., 4 de fev. de 2020 às 14:54, Wesley Dillingham <wes@wesdillingham.com> escreveu:
I would guess that you have something preventing osd to osd communication on ports 6800-7300 or osd to mon communication on port 6789 and/or 3300.
The 3 servers are on the same subnet. They are connect to a non-managed switch. And none have any firewall (iptables) rules blocking anything. They can ping one the other. Can you think about some other way that some traffic could be blocked? Or some other test I could do to check for connectivity? Regards, Rodrigo
Respectfully,
Wes Dillingham wes@wesdillingham.com LinkedIn
On Tue, Feb 4, 2020 at 12:44 PM Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> wrote:
Em ter., 4 de fev. de 2020 às 12:39, Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> escreveu:
Hi,
I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles.
I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time.
They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again.
Here is some info about my cluster:
------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops
services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df)
data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering -------------------------------------------
------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 -------------------------------------------
At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing.
What should I do to get my cluster back up? Just wait?
I just found out that I have a few pgs "stuck peering":
------------------------------------------- # ceph health detail | grep peering HEALTH_WARN 1 filesystem is degraded; 1 MDSs report slow metadata IOs; 2 osds down; 1 host (2 osds) down; Reduced data availability: 324 pgs inactive, 324 pgs peering; 7 daemons have recently crashed; 80 slow ops, oldest one blocked for 33 sec, daemons [osd.0,osd.1] have slow ops. PG_AVAILABILITY Reduced data availability: 324 pgs inactive, 324 pgs peering pg 1.39 is stuck peering for 14011.965915, current state peering, last acting [0,1] pg 1.3a is stuck peering for 14084.993947, current state peering, last acting [0,1] pg 1.3b is stuck peering for 14274.225311, current state peering, last acting [0,1] pg 1.3c is stuck peering for 15937.859532, current state peering, last acting [1,0] pg 1.3d is stuck peering for 15786.873447, current state peering, last acting [1,0] pg 1.3e is stuck peering for 15841.947891, current state peering, last acting [1,0] pg 1.3f is stuck peering for 15841.912853, current state peering, last acting [1,0] pg 1.40 is stuck peering for 14031.769901, current state peering, last acting [0,1] pg 1.41 is stuck peering for 14010.216124, current state peering, last acting [0,1] pg 1.42 is stuck peering for 15841.895446, current state peering, last acting [1,0] pg 1.43 is stuck peering for 15915.024413, current state peering, last acting [1,0] pg 1.44 is stuck peering for 13872.015272, current state peering, last acting [0,1] pg 1.45 is stuck peering for 15684.413850, current state peering, last acting [1,0] pg 1.46 is stuck peering for 15906.378461, current state peering, last acting [1,0] pg 1.47 is stuck peering for 14377.822032, current state peering, last acting [0,1] pg 1.48 is stuck peering for 14085.032316, current state peering, last acting [0,1] pg 1.49 is stuck peering for 14085.030366, current state peering, last acting [0,1] pg 1.4a is stuck peering for 14667.451862, current state peering, last acting [0,1] pg 1.4b is stuck peering for 14048.714764, current state peering, last acting [0,1] pg 1.4c is stuck peering for 13998.360919, current state peering, last acting [0,1] pg 1.4d is stuck peering for 15693.831021, current state peering, last acting [1,0] pg 2.38 is stuck peering for 15841.882464, current state peering, last acting [1,0] pg 2.39 is stuck peering for 15841.881968, current state peering, last acting [1,0] pg 2.3a is stuck peering for 14085.032520, current state peering, last acting [0,1] pg 2.3b is stuck inactive for 12717.975044, current state peering, last acting [0,1] pg 2.3c is stuck peering for 15841.947367, current state peering, last acting [1,0] pg 2.3d is stuck peering for 15732.221067, current state peering, last acting [1,0] pg 2.3e is stuck peering for 15938.007321, current state peering, last acting [0,1] pg 2.3f is stuck peering for 14084.992407, current state peering, last acting [0,1] pg 7.38 is stuck peering for 14080.942444, current state peering, last acting [3,4] pg 7.39 is stuck peering for 14048.869554, current state peering, last acting [3,4] pg 7.3a is stuck peering for 14048.869790, current state peering, last acting [3,4] pg 7.3b is stuck peering for 14080.943240, current state peering, last acting [3,4] pg 7.3c is stuck peering for 15842.114296, current state peering, last acting [4,3] pg 7.3d is stuck peering for 14048.870194, current state peering, last acting [3,4] pg 7.3e is stuck peering for 15842.105944, current state peering, last acting [4,3] pg 7.3f is stuck peering for 15842.111549, current state peering, last acting [4,3] pg 7.40 is stuck peering for 14048.869572, current state peering, last acting [3,4] pg 7.41 is stuck peering for 14048.868747, current state peering, last acting [3,4] pg 7.42 is stuck peering for 15845.175729, current state peering, last acting [4,3] pg 7.43 is stuck peering for 15842.105227, current state peering, last acting [4,3] pg 7.44 is stuck peering for 15845.196486, current state peering, last acting [4,3] pg 7.45 is stuck peering for 14048.869849, current state peering, last acting [3,4] pg 7.46 is stuck peering for 14080.942650, current state peering, last acting [3,4] pg 7.47 is stuck peering for 15845.197875, current state peering, last acting [4,3] pg 7.4a is stuck peering for 15842.113906, current state peering, last acting [4,3] pg 7.4b is stuck peering for 15845.197205, current state peering, last acting [4,3] pg 7.4c is stuck peering for 14048.869937, current state peering, last acting [3,4] pg 7.4d is stuck peering for 14048.869137, current state peering, last acting [3,4] pg 7.4e is stuck peering for 15842.111699, current state peering, last acting [4,3] pg 7.4f is stuck peering for 14080.943391, current state peering, last acting [3,4] -------------------------------------------
Why is that? How can I fix it?
Rodrigo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Em ter., 4 de fev. de 2020 às 15:19, <DHilsbos@performair.com> escreveu:
Rodrigo;
Best bet would be to check logs. Check the OSD logs on the affected server. Check cluster logs on the MONs. Check OSD logs on other servers.
Your Ceph version(s) and your OS distribution and version would also be useful to help you troubleshoot this OSD flapping issue.
Looking at the logs I finally found the issue: when I said that there were no changes in network topology, I was mistaken. I removed an unused (or at least I thought so) network board from each server. These servers had 2 network boards that I installed and configured so I would have a "public network" and a "cluster network". That was when I was first installing the ceph cluster. After having some problems with this set up I was advised by members of this list to not use this dual network setup as it could make debugging much more difficult. I followed this advice, or at least tried to. To make a long story short, ceph was still trying to use the second network for some OSDs. With a "ceph config rm global cluster_network" and a general restart of the cluster, everything started working again. Thanks for the help and sorry for the confusion. Regards, Rodrigo
participants (3)
-
DHilsbos＠performair.com
-
Rodrigo Severo - Fábrica
-
Wesley Dillingham