Em ter., 4 de fev. de 2020 às 14:54, Wesley Dillingham <wes(a)wesdillingham.com> escreveu: The 3 servers are on the same subnet. They are connect to a non-managed switch. And none have any firewall (iptables) rules blocking anything. They can ping one the other. Can you think about some other way that some traffic could be blocked? Or some other test I could do to check for connectivity? Regards, Rodrigo > > > Respectfully, > > Wes Dillingham > wes(a)wesdillingham.com > LinkedIn > > > On Tue, Feb 4, 2020 at 12:44 PM Rodrigo Severo - Fábrica <rodrigo(a)fabricadeideias.com> wrote: >> >> Em ter., 4 de fev. de 2020 às 12:39, Rodrigo Severo - Fábrica >> <rodrigo(a)fabricadeideias.com> escreveu: >> > >> > Hi, >> > >> > >> > I have a rather small cephfs cluster with 3 machines right now: all of >> > them sharing MDS, MON, MGS and OSD roles. >> > >> > I had to move all machines to a new physical location and, >> > unfortunately, I had to move all of them at the same time. >> > >> > They are already on again but ceph won't be accessible as all pgs are >> > in peering state and OSD keep going down and up again. >> > >> > Here is some info about my cluster: >> > >> > ------------------------------------------- >> > # ceph -s >> > cluster: >> > id: e348b63c-d239-4a15-a2ce-32f29a00431c >> > health: HEALTH_WARN >> > 1 filesystem is degraded >> > 1 MDSs report slow metadata IOs >> > 2 osds down >> > 1 host (2 osds) down >> > Reduced data availability: 324 pgs inactive, 324 pgs peering >> > 7 daemons have recently crashed >> > 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops >> > >> > services: >> > mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) >> > mgr: a2-df(active, since 82m), standbys: a3-df, a1-df >> > mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby >> > osd: 6 osds: 4 up (since 5s), 6 in (since 47m) >> > rgw: 1 daemon active (a2-df) >> > >> > data: >> > pools: 7 pools, 324 pgs >> > objects: 850.25k objects, 744 GiB >> > usage: 2.3 TiB used, 14 TiB / 16 TiB avail >> > pgs: 100.000% pgs not active >> > 324 peering >> > ------------------------------------------- >> > >> > ------------------------------------------- >> > # ceph osd df tree >> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META >> > AVAIL %USE VAR PGS STATUS TYPE NAME >> > -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB >> > 14 TiB 13.83 1.00 - root default >> > -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB >> > 14 TiB 13.83 1.00 - datacenter df >> > -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB >> > 4.7 TiB 13.83 1.00 - host a1-df >> > 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB >> > 3.6 TiB 0.03 0.00 0 down osd.3 >> > 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB >> > 1.1 TiB 41.43 3.00 0 down osd.0 >> > -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB >> > 4.7 TiB 13.83 1.00 - host a2-df >> > 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB >> > 3.6 TiB 0.03 0.00 100 up osd.4 >> > 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB >> > 1.1 TiB 41.42 3.00 224 up osd.1 >> > -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB >> > 4.7 TiB 13.83 1.00 - host a3-df >> > 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB >> > 3.6 TiB 0.03 0.00 100 up osd.5 >> > 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB >> > 1.1 TiB 41.43 3.00 224 up osd.2 >> > TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB >> > 14 TiB 13.83 >> > MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 >> > ------------------------------------------- >> > >> > At this exact moment both OSDs from server a1-df were down but that's >> > changing. Sometimes I have only one OSD down, but most of the times I >> > have 2. And exactly which ones are actually down keeps changing. >> > >> > What should I do to get my cluster back up? Just wait? >> >> I just found out that I have a few pgs "stuck peering": >> >> ------------------------------------------- >> # ceph health detail | grep peering >> HEALTH_WARN 1 filesystem is degraded; 1 MDSs report slow metadata IOs; >> 2 osds down; 1 host (2 osds) down; Reduced data availability: 324 pgs >> inactive, 324 pgs peering; 7 daemons have recently crashed; 80 slow >> ops, oldest one blocked for 33 sec, daemons [osd.0,osd.1] have slow >> ops. >> PG_AVAILABILITY Reduced data availability: 324 pgs inactive, 324 pgs peering >> pg 1.39 is stuck peering for 14011.965915, current state peering, >> last acting [0,1] >> pg 1.3a is stuck peering for 14084.993947, current state peering, >> last acting [0,1] >> pg 1.3b is stuck peering for 14274.225311, current state peering, >> last acting [0,1] >> pg 1.3c is stuck peering for 15937.859532, current state peering, >> last acting [1,0] >> pg 1.3d is stuck peering for 15786.873447, current state peering, >> last acting [1,0] >> pg 1.3e is stuck peering for 15841.947891, current state peering, >> last acting [1,0] >> pg 1.3f is stuck peering for 15841.912853, current state peering, >> last acting [1,0] >> pg 1.40 is stuck peering for 14031.769901, current state peering, >> last acting [0,1] >> pg 1.41 is stuck peering for 14010.216124, current state peering, >> last acting [0,1] >> pg 1.42 is stuck peering for 15841.895446, current state peering, >> last acting [1,0] >> pg 1.43 is stuck peering for 15915.024413, current state peering, >> last acting [1,0] >> pg 1.44 is stuck peering for 13872.015272, current state peering, >> last acting [0,1] >> pg 1.45 is stuck peering for 15684.413850, current state peering, >> last acting [1,0] >> pg 1.46 is stuck peering for 15906.378461, current state peering, >> last acting [1,0] >> pg 1.47 is stuck peering for 14377.822032, current state peering, >> last acting [0,1] >> pg 1.48 is stuck peering for 14085.032316, current state peering, >> last acting [0,1] >> pg 1.49 is stuck peering for 14085.030366, current state peering, >> last acting [0,1] >> pg 1.4a is stuck peering for 14667.451862, current state peering, >> last acting [0,1] >> pg 1.4b is stuck peering for 14048.714764, current state peering, >> last acting [0,1] >> pg 1.4c is stuck peering for 13998.360919, current state peering, >> last acting [0,1] >> pg 1.4d is stuck peering for 15693.831021, current state peering, >> last acting [1,0] >> pg 2.38 is stuck peering for 15841.882464, current state peering, >> last acting [1,0] >> pg 2.39 is stuck peering for 15841.881968, current state peering, >> last acting [1,0] >> pg 2.3a is stuck peering for 14085.032520, current state peering, >> last acting [0,1] >> pg 2.3b is stuck inactive for 12717.975044, current state peering, >> last acting [0,1] >> pg 2.3c is stuck peering for 15841.947367, current state peering, >> last acting [1,0] >> pg 2.3d is stuck peering for 15732.221067, current state peering, >> last acting [1,0] >> pg 2.3e is stuck peering for 15938.007321, current state peering, >> last acting [0,1] >> pg 2.3f is stuck peering for 14084.992407, current state peering, >> last acting [0,1] >> pg 7.38 is stuck peering for 14080.942444, current state peering, >> last acting [3,4] >> pg 7.39 is stuck peering for 14048.869554, current state peering, >> last acting [3,4] >> pg 7.3a is stuck peering for 14048.869790, current state peering, >> last acting [3,4] >> pg 7.3b is stuck peering for 14080.943240, current state peering, >> last acting [3,4] >> pg 7.3c is stuck peering for 15842.114296, current state peering, >> last acting [4,3] >> pg 7.3d is stuck peering for 14048.870194, current state peering, >> last acting [3,4] >> pg 7.3e is stuck peering for 15842.105944, current state peering, >> last acting [4,3] >> pg 7.3f is stuck peering for 15842.111549, current state peering, >> last acting [4,3] >> pg 7.40 is stuck peering for 14048.869572, current state peering, >> last acting [3,4] >> pg 7.41 is stuck peering for 14048.868747, current state peering, >> last acting [3,4] >> pg 7.42 is stuck peering for 15845.175729, current state peering, >> last acting [4,3] >> pg 7.43 is stuck peering for 15842.105227, current state peering, >> last acting [4,3] >> pg 7.44 is stuck peering for 15845.196486, current state peering, >> last acting [4,3] >> pg 7.45 is stuck peering for 14048.869849, current state peering, >> last acting [3,4] >> pg 7.46 is stuck peering for 14080.942650, current state peering, >> last acting [3,4] >> pg 7.47 is stuck peering for 15845.197875, current state peering, >> last acting [4,3] >> pg 7.4a is stuck peering for 15842.113906, current state peering, >> last acting [4,3] >> pg 7.4b is stuck peering for 15845.197205, current state peering, >> last acting [4,3] >> pg 7.4c is stuck peering for 14048.869937, current state peering, >> last acting [3,4] >> pg 7.4d is stuck peering for 14048.869137, current state peering, >> last acting [3,4] >> pg 7.4e is stuck peering for 15842.111699, current state peering, >> last acting [4,3] >> pg 7.4f is stuck peering for 14080.943391, current state peering, >> last acting [3,4] >> ------------------------------------------- >> >> >> Why is that? How can I fix it? >> >> >> Rodrigo >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Em ter., 4 de fev. de 2020 às 15:19, <DHilsbos(a)performair.com> escreveu: Looking at the logs I finally found the issue: when I said that there were no changes in network topology, I was mistaken. I removed an unused (or at least I thought so) network board from each server. These servers had 2 network boards that I installed and configured so I would have a "public network" and a "cluster network". That was when I was first installing the ceph cluster. After having some problems with this set up I was advised by members of this list to not use this dual network setup as it could make debugging much more difficult. I followed this advice, or at least tried to. To make a long story short, ceph was still trying to use the second network for some OSDs. With a "ceph config rm global cluster_network" and a general restart of the cluster, everything started working again. Thanks for the help and sorry for the confusion. Regards, Rodrigo