Unknown PGs after osd move
Hello, after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs: cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_WARN noscrub flag(s) set 2 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 241 pgs inactive 1532 slow requests are blocked > 32 sec 789 slow ops, oldest one blocked for 1949 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops. services: mon: 3 daemons, quorum black1,black2,black3 (age 97m) mgr: black2(active, since 96m), standbys: black1, black3 osd: 85 osds: 85 up, 82 in; 118 remapped pgs flags noscrub rgw: 1 daemon active (admin) data: pools: 12 pools, 3000 pgs objects: 33.96M objects, 129 TiB usage: 388 TiB used, 159 TiB / 548 TiB avail pgs: 8.033% pgs unknown 409151/101874117 objects misplaced (0.402%) 2634 active+clean 241 unknown 107 active+remapped+backfill_wait 11 active+remapped+backfilling 7 active+clean+scrubbing+deep io: client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr recovery: 67 MiB/s, 17 objects/s This used to be around 700+ unknown, however these 241 are stuck in this state for more than 1h. Below is a sample of pgs from "ceph pg dump all | grep unknown" 2.7f7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c2 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7ab 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.78b 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.788 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.76e 0 Using ceph pg 2.7f7 query hangs. We checked and one server did have an incorrect MTU setting (9204 instead of the correct 9000), but that was fixed some hours ago. Does anyone have a hint on how to find those unknown osds? Version wise this is 14.2.9: [20:42:20] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } From ceph health detail: [20:42:58] black2.place6:~# ceph health detail HEALTH_WARN noscrub flag(s) set; 2 nearfull osd(s); 1 pool(s) nearfull; Reduced data availability: 241 pgs inactive; 1575 slow requests are blocked > 32 sec; 751 slow ops, oldest one blocked for 1986 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.31,osd.33,osd.35]... have slow ops. OSDMAP_FLAGS noscrub flag(s) set OSD_NEARFULL 2 nearfull osd(s) osd.36 is near full osd.54 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'ssd' is nearfull PG_AVAILABILITY Reduced data availability: 241 pgs inactive pg 2.82 is stuck inactive for 6027.042489, current state unknown, last acting [] pg 2.88 is stuck inactive for 6027.042489, current state unknown, last acting [] ... pg 19.6e is stuck inactive for 6027.042489, current state unknown, last acting [] pg 20.69 is stuck inactive for 6027.042489, current state unknown, last acting [] As can be seen, multiple pools are affected even though most missing pgs are from pool 2. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Hello, On 22.09.20 20:45, Nico Schottelius wrote:
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs:
You mean, that you re-seated the OSDs into another chassis/host? Is the crush map aware about that? I didn't ever try that, but don't you need to cursh move it?
Hey Andreas, Andreas John <aj@net-lab.net> writes:
Hello,
On 22.09.20 20:45, Nico Schottelius wrote:
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs:
You mean, that you re-seated the OSDs into another chassis/host?
That is correct.
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new server it is automatically put into the serve8 bucket. Cheers, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Hey Nico, maybe you "pinned" the IP of the OSDs in question in ceph.conf to the IP of the old chassis? Good Luck, derjohn P.S. < 100MB/sec is a terrible performance for recovery with 85 OSDs. Is it rotational on 1 GBit/sec network? You could set ceph osd set nodeep-scrub to prevent too much read from the plattners and get better recovery performance.
Hey Andreas, Andreas John <aj@net-lab.net> writes:
Hey Nico,
maybe you "pinned" the IP of the OSDs in question in ceph.conf to the IP of the old chassis?
That would be nice - unfortunately our ceph.conf is almost empty: [22:11:59] server15.place6:/sys/class/block/sdg# cat /etc/ceph/ceph.conf # cdist maintained - do not change [global] fsid = 1ccd84f6-e362-4c50-9ffe-59436745e445 public network = 2a0a:e5c0:2:1::/64 cluster network = 2a0a:e5c0:2:1::/64 mon initial members = ceph1.place6.ungleich.ch, ceph2.place6.ungleich.ch, ceph3.place6.ungleich.ch mon host = ceph1.place6.ungleich.ch, ceph2.place6.ungleich.ch, ceph3.place6.ungleich.ch auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 # Required since nautilus, otherwise ceph fails to bind to public IP # 2020-05-15, Nico! ms_bind_ipv4 = false ms_bind_ipv6 = true # Restrain recovery operations so that normal cluster is not affected [osd] osd max backfills = 1 osd recovery max active = 1 osd recovery op priority = 2
Good Luck,
derjohn
P.S. < 100MB/sec is a terrible performance for recovery with 85 OSDs. Is it rotational on 1 GBit/sec network? You could set ceph osd set nodeep-scrub to prevent too much read from the plattners and get better recovery performance.
All nodes are connected with 2x 10 Gbit/s bonded/LACP, so I'd expect at least a couple of hundred MB/s network bandwidth per OSD. On one server I just restarted the OSDs and now the read performance dropped down to 1-4 MB/s per OSD with being about 90% busy. Since nautilus we observed much longer starting times of OSDs and I wonder if the osd does some kind of fsck these days and delays the peering process because of that? The disks in question are 3.5"/10TB/6 Gbit/s SATA disks connected to an H800 controller - so generally speaking I do not see a reasonable bottleneck here. -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
On 22.09.20 22:09, Nico Schottelius wrote: [...]
All nodes are connected with 2x 10 Gbit/s bonded/LACP, so I'd expect at least a couple of hundred MB/s network bandwidth per OSD.
On one server I just restarted the OSDs and now the read performance dropped down to 1-4 MB/s per OSD with being about 90% busy.
Since nautilus we observed much longer starting times of OSDs and I wonder if the osd does some kind of fsck these days and delays the peering process because of that?
The disks in question are 3.5"/10TB/6 Gbit/s SATA disks connected to an H800 controller - so generally speaking I do not see a reasonable bottleneck here. Yes, I should! I saw in your mail:
1.) 1532 slow requests are blocked > 32 sec 789 slow ops, oldest one blocked for 1949 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops. An request that is blocked for > 32 sec is odd! Same goes for 1949 sec. I my experience, they will never finish. Sometimes they go away with osd restarts. Are those OSD the ones you relocated? 2.) client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr recovery: 67 MiB/s, 17 objects/s 67 MB/sec is slower than a single rotational disk can deliver. Even 67 + 91 MB/s is not much, especially not for an 85 OSD @ 10G cluster. The ~2500 IOPS client I/O will translate to 7500 "net" IOPS with pook size 3, maybe that is the limit. But I guess you already know that. But before tuning, you should probably listen to Frank's advice about the placements (See other post). ASAP the unknown OSDs come back, the speed will probably go up due to parallelism. rgds, j.
Hey Andreas, thanks for the insights. Maybe a bit more background: We are running a variety of pools, the majority of data is stored on the "hdd" and "ssd" pools, which make use of the "ssd" and "hdd-big" (as in 3.5") classes. Andreas John <aj@net-lab.net> writes:
On 22.09.20 22:09, Nico Schottelius wrote: [...]
All nodes are connected with 2x 10 Gbit/s bonded/LACP, so I'd expect at
The disks in question are 3.5"/10TB/6 Gbit/s SATA disks connected to an H800 controller - so generally speaking I do not see a reasonable bottleneck here. Yes, I should! I saw in your mail:
1.) 1532 slow requests are blocked > 32 sec 789 slow ops, oldest one blocked for 1949 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops.
An request that is blocked for > 32 sec is odd! Same goes for 1949 sec. I my experience, they will never finish. Sometimes they go away with osd restarts. Are those OSD the ones you relocated?
We tried restarting some of the osds, however the slow ops are coming back soon after restart. And this is the most puzzling part: The move of the osds only affected PGs that are related to the "ssd" pool. While data was rebalancing, one hdd osd crashed and was restarted, but what we at the moment is that there are slow ops on a lot of osds: REQUEST_SLOW 4560 slow requests are blocked > 32 sec 1262 ops are blocked > 2097.15 sec 1121 ops are blocked > 1048.58 sec 602 ops are blocked > 524.288 sec 849 ops are blocked > 262.144 sec 407 ops are blocked > 131.072 sec 175 ops are blocked > 65.536 sec 144 ops are blocked > 32.768 sec osd.82 has blocked requests > 131.072 sec osds 1,9,11,19,28,44,45,48,58,72,73,84 have blocked requests > 262.144 sec osds 2,4,21,22,27,29,31,34,61 have blocked requests > 524.288 sec osds 15,20,32,52,55,62,71,74,79,83 have blocked requests > 1048.58 sec osds 5,6,7,12,14,16,18,25,33,35,47,50,51,69 have blocked requests > 2097.15 sec REQUEST_STUCK 1228 stuck requests are blocked > 4096 sec 330 ops are blocked > 8388.61 sec 898 ops are blocked > 4194.3 sec osds 3,23,56,59,60 have stuck requests > 4194.3 sec osds 30,46,49,63,64,65,66,68,70,75,85 have stuck requests > 8388.61 sec SLOW_OPS 2360 slow ops, oldest one blocked for 6517 sec, daemons [osd.0,osd.1,osd.11,osd.12,osd.14,osd.15,osd.16,osd.18,osd.19,osd.2]... have slow ops. We have checked DNS, MTU, network congestion via prometheus and on the network side nothing seems to be wrong.
2.) client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr recovery: 67 MiB/s, 17 objects/s
67 MB/sec is slower than a single rotational disk can deliver. Even 67 + 91 MB/s is not much, especially not for an 85 OSD @ 10G cluster. The ~2500 IOPS client I/O will translate to 7500 "net" IOPS with pook size 3, maybe that is the limit.
But I guess you already know that. But before tuning, you should probably listen to Frank's advice about the placements (See other post). ASAP the unknown OSDs come back, the speed will probably go up due to parallelism.
I am not sure whether after the long rebalance progress over some hours this is a good idea at the moment. What really looks wrong is the extreme long peering and activation times: data: pools: 12 pools, 3000 pgs objects: 35.03M objects, 133 TiB usage: 394 TiB used, 163 TiB / 557 TiB avail pgs: 5.667% pgs unknown 24.967% pgs not active 1365063/105076392 objects degraded (1.299%) 252605/105076392 objects misplaced (0.240%) 1955 active+clean 608 peering 170 unknown 59 activating 57 active+remapped+backfill_wait 35 activating+undersized 32 active+undersized+degraded 20 stale+peering 17 activating+undersized+degraded 9 active+remapped+backfilling 6 stale+active+clean 5 active+recovery_wait 4 active+undersized 4 activating+degraded 4 active+clean+scrubbing+deep 4 stale+activating 3 active+recovery_wait+degraded 3 active+undersized+degraded+remapped+backfill_wait 2 remapped+peering 1 active+recovery_wait+undersized+degraded 1 active+undersized+degraded+remapped+backfilling 1 active+remapped+backfill_toofull io: client: 34 MiB/s rd, 3.6 MiB/s wr, 1.08k op/s rd, 324 op/s wr recovery: 82 MiB/s, 20 objects/s Still debugging. It's impressive how the very simple task of moving 4 SSDs caused/causes such problems. I wonder (and suspect) that something else must be wrong here. We recently (some months ago) upgraded from luminous via mimic to nautilus, I will triple check if there are any changes that can cause these effects. -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new server it is automatically put into the serve8 bucket.
It does not work like this, unfortunately. If you physically move disks to a new server without "informing ceph" in advance, hat is, crush move the OSD while they are up, ceph looses placement information. You can post-repair such a situation by temporarily "crush moving" (software move, not hardware move) the OSDs back to their previous host buckets, wait for peering to complete, and then "crush move" them to their new location again. Do not restart OSDs during this process or while rebalancing of misplaced objects is going on. There is a long-standing issue that causes placement information to be lost again and one would need to repeat the procedure. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: 22 September 2020 21:14:07 To: Andreas John Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Unknown PGs after osd move Hey Andreas, Andreas John <aj@net-lab.net> writes:
Hello,
On 22.09.20 20:45, Nico Schottelius wrote:
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs:
You mean, that you re-seated the OSDs into another chassis/host?
That is correct.
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new server it is automatically put into the serve8 bucket. Cheers, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Frank, Frank Schilder <frans@dtu.dk> writes:
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new server it is automatically put into the serve8 bucket.
It does not work like this, unfortunately. If you physically move disks to a new server without "informing ceph" in advance, hat is, crush move the OSD while they are up, ceph looses placement information. You can post-repair such a situation by temporarily "crush moving" (software move, not hardware move) the OSDs back to their previous host buckets, wait for peering to complete, and then "crush move" them to their new location again.
That is good to know. So in theory: - crush move osd to a different server bucket - shutdown osd - move physically to another server - no rebalancing needed Should do the job? It won't accept today's rebalance, but it would be good to have a sane way for the future. Cheers, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
No, the recipe I gave was for trying to recover healthy status of all PGs in the current situation. I would avoid moving OSDs at all cost, because it will always imply rebalancing. Any change to the crush map changes how PGs are hashed onto OSDs, which in turn triggers a rebalancing. If moving OSDs cannot be avoided, I usually do: - evacuate OSDs that need to move - move empty (!) OSDs to new location - let data move back onto OSDs There are other ways of doing it, with their own pro's and cons. For example, if your client load allows high-bandwidth rebuild operations, you can also - shut down OSDs that need to move (make sure you don't shut down too many from different failure domains at the same time) - let the remaining OSDs rebuild the missing data - after health is back to OK, move OSDs and start up The second way is usually faster, but has the drawback that new writes will go to less redundant storage for a while. The first method takes longer, but there is no redundancy degradation along the way. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: 22 September 2020 22:13:49 To: Frank Schilder Cc: Nico Schottelius; Andreas John; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Unknown PGs after osd move Hey Frank, Frank Schilder <frans@dtu.dk> writes:
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new server it is automatically put into the serve8 bucket.
It does not work like this, unfortunately. If you physically move disks to a new server without "informing ceph" in advance, hat is, crush move the OSD while they are up, ceph looses placement information. You can post-repair such a situation by temporarily "crush moving" (software move, not hardware move) the OSDs back to their previous host buckets, wait for peering to complete, and then "crush move" them to their new location again.
That is good to know. So in theory: - crush move osd to a different server bucket - shutdown osd - move physically to another server - no rebalancing needed Should do the job? It won't accept today's rebalance, but it would be good to have a sane way for the future. Cheers, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Update: Restarting other osds on the server that we took osds from seems to have reduced the amount of unknown pgs down to 170. However the peering and activating states seem to stay very long for these OSDs: cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_ERR noscrub flag(s) set 1 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 1043 pgs inactive, 745 pgs peering Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull Degraded data redundancy: 2384069/105076299 objects degraded (2.269%), 110 pgs degraded, 156 pgs undersized 4970 slow requests are blocked > 32 sec 41 stuck requests are blocked > 4096 sec 2909 slow ops, oldest one blocked for 2266 sec, daemons [osd.0,osd.1,osd.11,osd.12,osd.14,osd.15,osd.16,osd.18,osd.19,osd.2]... have slow ops. services: mon: 3 daemons, quorum black1,black2,black3 (age 2h) mgr: black2(active, since 2h), standbys: black1, black3 osd: 85 osds: 85 up, 82 in; 90 remapped pgs flags noscrub rgw: 1 daemon active (admin) data: pools: 12 pools, 3000 pgs objects: 35.03M objects, 133 TiB usage: 401 TiB used, 165 TiB / 566 TiB avail pgs: 5.667% pgs unknown 29.200% pgs not active 2384069/105076299 objects degraded (2.269%) 330786/105076299 objects misplaced (0.315%) 1776 active+clean 745 peering 170 unknown 79 active+remapped+backfill_wait 68 active+undersized+degraded 53 activating+undersized 43 activating 30 activating+undersized+degraded 10 active+remapped+backfilling 6 active+recovery_wait+degraded 5 activating+degraded 5 active+recovery_wait 4 active+undersized 4 active+clean+scrubbing+deep 1 active+recovery_wait+undersized+degraded 1 active+remapped+backfill_toofull io: client: 44 MiB/s rd, 4.2 MiB/s wr, 991 op/s rd, 389 op/s wr recovery: 71 MiB/s, 18 objects/s Nico Schottelius <nico.schottelius@ungleich.ch> writes:
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs:
cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_WARN noscrub flag(s) set 2 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 241 pgs inactive 1532 slow requests are blocked > 32 sec 789 slow ops, oldest one blocked for 1949 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops.
services: mon: 3 daemons, quorum black1,black2,black3 (age 97m) mgr: black2(active, since 96m), standbys: black1, black3 osd: 85 osds: 85 up, 82 in; 118 remapped pgs flags noscrub rgw: 1 daemon active (admin)
data: pools: 12 pools, 3000 pgs objects: 33.96M objects, 129 TiB usage: 388 TiB used, 159 TiB / 548 TiB avail pgs: 8.033% pgs unknown 409151/101874117 objects misplaced (0.402%) 2634 active+clean 241 unknown 107 active+remapped+backfill_wait 11 active+remapped+backfilling 7 active+clean+scrubbing+deep
io: client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr recovery: 67 MiB/s, 17 objects/s
This used to be around 700+ unknown, however these 241 are stuck in this state for more than 1h. Below is a sample of pgs from "ceph pg dump all | grep unknown"
2.7f7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c2 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7ab 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.78b 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.788 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.76e 0
Using ceph pg 2.7f7 query hangs.
We checked and one server did have an incorrect MTU setting (9204 instead of the correct 9000), but that was fixed some hours ago.
Does anyone have a hint on how to find those unknown osds?
Version wise this is 14.2.9:
[20:42:20] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } }
From ceph health detail:
[20:42:58] black2.place6:~# ceph health detail HEALTH_WARN noscrub flag(s) set; 2 nearfull osd(s); 1 pool(s) nearfull; Reduced data availability: 241 pgs inactive; 1575 slow requests are blocked > 32 sec; 751 slow ops, oldest one blocked for 1986 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.31,osd.33,osd.35]... have slow ops. OSDMAP_FLAGS noscrub flag(s) set OSD_NEARFULL 2 nearfull osd(s) osd.36 is near full osd.54 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'ssd' is nearfull PG_AVAILABILITY Reduced data availability: 241 pgs inactive pg 2.82 is stuck inactive for 6027.042489, current state unknown, last acting [] pg 2.88 is stuck inactive for 6027.042489, current state unknown, last acting [] ... pg 19.6e is stuck inactive for 6027.042489, current state unknown, last acting [] pg 20.69 is stuck inactive for 6027.042489, current state unknown, last acting []
As can be seen, multiple pools are affected even though most missing pgs are from pool 2.
Best regards,
Nico
-- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
participants (3)
-
Andreas John
-
Frank Schilder
-
Nico Schottelius