22 Sep
2020
22 Sep
'20
6:45 p.m.
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue
- see previous mail), we ran into 241 unknown pgs:
cluster:
id: 1ccd84f6-e362-4c50-9ffe-59436745e445
health: HEALTH_WARN
noscrub flag(s) set
2 nearfull osd(s)
1 pool(s) nearfull
Reduced data availability: 241 pgs inactive
1532 slow requests are blocked > 32 sec
789 slow ops, oldest one blocked for 1949 sec, daemons
[osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops.
services:
mon: 3 daemons, quorum black1,black2,black3 (age 97m)
mgr: black2(active, since 96m), standbys: black1, black3
osd: 85 osds: 85 up, 82 in; 118 remapped pgs
flags noscrub
rgw: 1 daemon active (admin)
data:
pools: 12 pools, 3000 pgs
objects: 33.96M objects, 129 TiB
usage: 388 TiB used, 159 TiB / 548 TiB avail
pgs: 8.033% pgs unknown
409151/101874117 objects misplaced (0.402%)
2634 active+clean
241 unknown
107 active+remapped+backfill_wait
11 active+remapped+backfilling
7 active+clean+scrubbing+deep
io:
client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr
recovery: 67 MiB/s, 17 objects/s
This used to be around 700+ unknown, however these 241 are stuck in this
state for more than 1h. Below is a sample of pgs from "ceph pg dump
all | grep unknown"
2.7f7 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.7c7 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.7c2 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.7ab 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.78b 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.788 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.76e 0
Using ceph pg 2.7f7 query hangs.
We checked and one server did have an incorrect MTU setting (9204
instead of the correct 9000), but that was fixed some hours ago.
Does anyone have a hint on how to find those unknown osds?
Version wise this is 14.2.9:
[20:42:20] black2.place6:~# ceph versions
{
"mon": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 3
},
"mgr": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 3
},
"osd": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 85
},
"mds": {},
"rgw": {
"ceph version 20200428-923-g4004f081ec
(4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1
},
"overall": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 91,
"ceph version 20200428-923-g4004f081ec
(4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1
}
}
From ceph health detail:
[20:42:58] black2.place6:~# ceph health detail
HEALTH_WARN noscrub flag(s) set; 2 nearfull osd(s); 1 pool(s) nearfull; Reduced data
availability: 241 pgs inactive; 1575 slow requests are blocked > 32 sec; 751 slow ops,
oldest one blocked for 1986 sec, daemons
[osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.31,osd.33,osd.35]... have slow ops.
OSDMAP_FLAGS noscrub flag(s) set
OSD_NEARFULL 2 nearfull osd(s)
osd.36 is near full
osd.54 is near full
POOL_NEARFULL 1 pool(s) nearfull
pool 'ssd' is nearfull
PG_AVAILABILITY Reduced data availability: 241 pgs inactive
pg 2.82 is stuck inactive for 6027.042489, current state unknown, last acting []
pg 2.88 is stuck inactive for 6027.042489, current state unknown, last acting []
...
pg 19.6e is stuck inactive for 6027.042489, current state unknown, last acting []
pg 20.69 is stuck inactive for 6027.042489, current state unknown, last acting []
As can be seen, multiple pools are affected even though most missing pgs
are from pool 2.
Best regards,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
22 Sep
22 Sep
7:01 p.m.
Hello,
On 22.09.20 20:45, Nico Schottelius wrote:
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue
- see previous mail), we ran into 241 unknown pgs:
You mean, that you re-seated the OSDs into another chassis/host? Is the
crush map aware about that?
I didn't ever try that, but don't you need to cursh move it?
7:14 p.m.
Hey Andreas,
Andreas John <aj(a)net-lab.net> writes:
Hello,
On 22.09.20 20:45, Nico Schottelius wrote:
That is correct.
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue
- see previous mail), we ran into 241 unknown pgs:
You mean, that you re-seated the OSDs into another chassis/host? Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to
cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new
server it is automatically put into the serve8 bucket.
Cheers,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
7:47 p.m.
Hey Nico,
maybe you "pinned" the IP of the OSDs in question in ceph.conf to the IP
of the old chassis?
Good Luck,
derjohn
P.S. < 100MB/sec is a terrible performance for recovery with 85 OSDs.
Is it rotational on 1 GBit/sec network? You could set ceph osd set
nodeep-scrub to prevent too much read from the plattners and get better
recovery performance.
8:09 p.m.
Hey Andreas,
Andreas John <aj(a)net-lab.net> writes:
Hey Nico,
maybe you "pinned" the IP of the OSDs in question in ceph.conf to the IP
of the old chassis?
That would be nice - unfortunately our ceph.conf is almost empty:
[22:11:59] server15.place6:/sys/class/block/sdg# cat /etc/ceph/ceph.conf
# cdist maintained - do not change
[global]
fsid = 1ccd84f6-e362-4c50-9ffe-59436745e445
public network = 2a0a:e5c0:2:1::/64
cluster network = 2a0a:e5c0:2:1::/64
mon initial members = ceph1.place6.ungleich.ch, ceph2.place6.ungleich.ch,
ceph3.place6.ungleich.ch
mon host = ceph1.place6.ungleich.ch, ceph2.place6.ungleich.ch,
ceph3.place6.ungleich.ch
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd pool default size = 3
# Required since nautilus, otherwise ceph fails to bind to public IP
# 2020-05-15, Nico!
ms_bind_ipv4 = false
ms_bind_ipv6 = true
# Restrain recovery operations so that normal cluster is not affected
[osd]
osd max backfills = 1
osd recovery max active = 1
osd recovery op priority = 2
Good Luck,
derjohn
P.S. < 100MB/sec is a terrible performance for recovery with 85 OSDs.
Is it rotational on 1 GBit/sec network? You could set ceph osd set
nodeep-scrub to prevent too much read from the plattners and get better
recovery performance.
All nodes are connected with 2x 10 Gbit/s bonded/LACP, so I'd expect at
least a couple of hundred MB/s network bandwidth per OSD.
On one server I just restarted the OSDs and now the read performance
dropped down to 1-4 MB/s per OSD with being about 90% busy.
Since nautilus we observed much longer starting times of OSDs and I
wonder if the osd does some kind of fsck these days and delays the
peering process because of that?
The disks in question are 3.5"/10TB/6 Gbit/s SATA disks connected to an
H800 controller - so generally speaking I do not see a reasonable
bottleneck here.
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
8:29 p.m.
On 22.09.20 22:09, Nico Schottelius wrote:
[...]
All nodes are connected with 2x 10 Gbit/s bonded/LACP,
so I'd expect at
least a couple of hundred MB/s network bandwidth per OSD.
On one server I just restarted the OSDs and now the read performance
dropped down to 1-4 MB/s per OSD with being about 90% busy.
Since nautilus we observed much longer starting times of OSDs and I
wonder if the osd does some kind of fsck these days and delays the
peering process because of that?
The disks in question are 3.5"/10TB/6 Gbit/s SATA disks connected to an
H800 controller - so generally speaking I do not see a reasonable
bottleneck here.
Yes, I should! I saw in your mail:
1.) 1532 slow requests are blocked > 32 sec
789 slow ops, oldest one blocked for 1949 sec, daemons
[osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]...
have slow ops.
An request that is blocked for > 32 sec is odd! Same goes for 1949 sec.
I my experience, they will never finish. Sometimes they go away with osd
restarts. Are those OSD the ones you relocated?
2.) client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr
recovery: 67 MiB/s, 17 objects/s
67 MB/sec is slower than a single rotational disk can deliver. Even 67
+ 91 MB/s is not much, especially not for an 85 OSD @ 10G cluster. The
~2500 IOPS client I/O will translate to 7500 "net" IOPS with pook size
3, maybe that is the limit.
But I guess you already know that. But before tuning, you should
probably listen to Frank's advice about the placements (See other post).
ASAP the unknown OSDs come back, the speed will probably go up due to
parallelism.
rgds,
j.
8:51 p.m.
Hey Andreas,
thanks for the insights. Maybe a bit more background:
We are running a variety of pools, the majority of data is stored on the
"hdd" and "ssd" pools, which make use of the "ssd" and
"hdd-big" (as in
3.5") classes.
Andreas John <aj(a)net-lab.net> writes:
On 22.09.20 22:09, Nico Schottelius wrote:
[...]
We tried restarting some of the osds, however the slow ops are coming
back soon after restart. And this is the most puzzling part:
The move of the osds only affected PGs that are related to the "ssd"
pool. While data was rebalancing, one hdd osd crashed and was restarted,
but what we at the moment is that there are slow ops on a lot of osds:
REQUEST_SLOW 4560 slow requests are blocked > 32 sec
1262 ops are blocked > 2097.15 sec
1121 ops are blocked > 1048.58 sec
602 ops are blocked > 524.288 sec
849 ops are blocked > 262.144 sec
407 ops are blocked > 131.072 sec
175 ops are blocked > 65.536 sec
144 ops are blocked > 32.768 sec
osd.82 has blocked requests > 131.072 sec
osds 1,9,11,19,28,44,45,48,58,72,73,84 have blocked requests > 262.144 sec
osds 2,4,21,22,27,29,31,34,61 have blocked requests > 524.288 sec
osds 15,20,32,52,55,62,71,74,79,83 have blocked requests > 1048.58 sec
osds 5,6,7,12,14,16,18,25,33,35,47,50,51,69 have blocked requests > 2097.15 sec
REQUEST_STUCK 1228 stuck requests are blocked > 4096 sec
330 ops are blocked > 8388.61 sec
898 ops are blocked > 4194.3 sec
osds 3,23,56,59,60 have stuck requests > 4194.3 sec
osds 30,46,49,63,64,65,66,68,70,75,85 have stuck requests > 8388.61 sec
SLOW_OPS 2360 slow ops, oldest one blocked for 6517 sec, daemons
[osd.0,osd.1,osd.11,osd.12,osd.14,osd.15,osd.16,osd.18,osd.19,osd.2]... have slow ops.
We have checked DNS, MTU, network congestion via prometheus and on the
network side nothing seems to be wrong.
All nodes are connected with 2x 10 Gbit/s
bonded/LACP, so I'd expect at
The disks in question are 3.5"/10TB/6 Gbit/s SATA disks connected to an
H800 controller - so generally speaking I do not see a reasonable
bottleneck here.
Yes, I should! I saw in your mail:
1.) 1532 slow requests are blocked > 32 sec
789 slow ops, oldest one blocked for 1949 sec, daemons
[osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]...
have slow ops.
An request that is blocked for > 32 sec is odd! Same goes for 1949 sec.
I my experience, they will never finish. Sometimes they go away with osd
restarts. Are those OSD the ones you relocated? 2.) client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd,
686 op/s wr
recovery: 67 MiB/s, 17 objects/s
67 MB/sec is slower than a single rotational disk can deliver. Even 67
+ 91 MB/s is not much, especially not for an 85 OSD @ 10G cluster. The
~2500 IOPS client I/O will translate to 7500 "net" IOPS with pook size
3, maybe that is the limit.
But I guess you already know that. But before tuning, you should
probably listen to Frank's advice about the placements (See other post).
ASAP the unknown OSDs come back, the speed will probably go up due to
parallelism.
I am not sure whether after the long rebalance progress over some hours
this is a good idea at the moment.
What really looks wrong is the extreme long peering and activation
times:
data:
pools: 12 pools, 3000 pgs
objects: 35.03M objects, 133 TiB
usage: 394 TiB used, 163 TiB / 557 TiB avail
pgs: 5.667% pgs unknown
24.967% pgs not active
1365063/105076392 objects degraded (1.299%)
252605/105076392 objects misplaced (0.240%)
1955 active+clean
608 peering
170 unknown
59 activating
57 active+remapped+backfill_wait
35 activating+undersized
32 active+undersized+degraded
20 stale+peering
17 activating+undersized+degraded
9 active+remapped+backfilling
6 stale+active+clean
5 active+recovery_wait
4 active+undersized
4 activating+degraded
4 active+clean+scrubbing+deep
4 stale+activating
3 active+recovery_wait+degraded
3 active+undersized+degraded+remapped+backfill_wait
2 remapped+peering
1 active+recovery_wait+undersized+degraded
1 active+undersized+degraded+remapped+backfilling
1 active+remapped+backfill_toofull
io:
client: 34 MiB/s rd, 3.6 MiB/s wr, 1.08k op/s rd, 324 op/s wr
recovery: 82 MiB/s, 20 objects/s
Still debugging. It's impressive how the very simple task of moving 4
SSDs caused/causes such problems. I wonder (and suspect) that something
else must be wrong here.
We recently (some months ago) upgraded from luminous via mimic to
nautilus, I will triple check if there are any changes that can cause
these effects.
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
8 p.m.
Is the crush
map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you
need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new
server it is automatically put into the serve8 bucket. Hello,
On 22.09.20 20:45, Nico Schottelius wrote:
That is correct.
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue
- see previous mail), we ran into 241 unknown pgs:
You mean, that you re-seated the OSDs into another chassis/host? Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to
cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new
server it is automatically put into the serve8 bucket.
Cheers,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:13 p.m.
Hey Frank,
Frank Schilder <frans(a)dtu.dk> writes:
It does not work like this, unfortunately. If you physically move
disks to a new server without "informing ceph" in advance, hat is,
crush move the OSD while they are up, ceph looses placement
information. You can post-repair such a situation by temporarily
"crush moving" (software move, not hardware move) the OSDs back to
their previous host buckets, wait for peering to complete, and then
"crush move" them to their new location again.
That is good to know. So in theory:
- crush move osd to a different server bucket
- shutdown osd
- move physically to another server
- no rebalancing needed
Should do the job?
It won't accept today's rebalance, but it would be good to have a sane
way for the future.
Cheers,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you
need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new
server it is automatically put into the serve8 bucket.
8:52 p.m.
No, the recipe I gave was for trying to recover healthy status of all PGs in the current
situation.
I would avoid moving OSDs at all cost, because it will always imply rebalancing. Any
change to the crush map changes how PGs are hashed onto OSDs, which in turn triggers a
rebalancing.
If moving OSDs cannot be avoided, I usually do:
- evacuate OSDs that need to move
- move empty (!) OSDs to new location
- let data move back onto OSDs
There are other ways of doing it, with their own pro's and cons. For example, if your
client load allows high-bandwidth rebuild operations, you can also
- shut down OSDs that need to move (make sure you don't shut down too many from
different failure domains at the same time)
- let the remaining OSDs rebuild the missing data
- after health is back to OK, move OSDs and start up
The second way is usually faster, but has the drawback that new writes will go to less
redundant storage for a while. The first method takes longer, but there is no redundancy
degradation along the way.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Nico Schottelius <nico.schottelius(a)ungleich.ch>
Sent: 22 September 2020 22:13:49
To: Frank Schilder
Cc: Nico Schottelius; Andreas John; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: Unknown PGs after osd move
Hey Frank,
Frank Schilder <frans(a)dtu.dk> writes:
It does not work like this, unfortunately. If you physically move
disks to a new server without "informing ceph" in advance, hat is,
crush move the OSD while they are up, ceph looses placement
information. You can post-repair such a situation by temporarily
"crush moving" (software move, not hardware move) the OSDs back to
their previous host buckets, wait for peering to complete, and then
"crush move" them to their new location again.
That is good to know. So in theory:
- crush move osd to a different server bucket
- shutdown osd
- move physically to another server
- no rebalancing needed
Should do the job?
It won't accept today's rebalance, but it would be good to have a sane
way for the future.
Cheers,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you
need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new
server it is automatically put into the serve8 bucket.
7:32 p.m.
Update:
Restarting other osds on the server that we took osds from seems to
have reduced the amount of unknown pgs down to 170. However the peering
and activating states seem to stay very long for these OSDs:
cluster:
id: 1ccd84f6-e362-4c50-9ffe-59436745e445
health: HEALTH_ERR
noscrub flag(s) set
1 nearfull osd(s)
1 pool(s) nearfull
Reduced data availability: 1043 pgs inactive, 745 pgs peering
Low space hindering backfill (add storage if this doesn't resolve itself):
1 pg backfill_toofull
Degraded data redundancy: 2384069/105076299 objects degraded (2.269%), 110 pgs
degraded, 156 pgs undersized
4970 slow requests are blocked > 32 sec
41 stuck requests are blocked > 4096 sec
2909 slow ops, oldest one blocked for 2266 sec, daemons
[osd.0,osd.1,osd.11,osd.12,osd.14,osd.15,osd.16,osd.18,osd.19,osd.2]... have slow ops.
services:
mon: 3 daemons, quorum black1,black2,black3 (age 2h)
mgr: black2(active, since 2h), standbys: black1, black3
osd: 85 osds: 85 up, 82 in; 90 remapped pgs
flags noscrub
rgw: 1 daemon active (admin)
data:
pools: 12 pools, 3000 pgs
objects: 35.03M objects, 133 TiB
usage: 401 TiB used, 165 TiB / 566 TiB avail
pgs: 5.667% pgs unknown
29.200% pgs not active
2384069/105076299 objects degraded (2.269%)
330786/105076299 objects misplaced (0.315%)
1776 active+clean
745 peering
170 unknown
79 active+remapped+backfill_wait
68 active+undersized+degraded
53 activating+undersized
43 activating
30 activating+undersized+degraded
10 active+remapped+backfilling
6 active+recovery_wait+degraded
5 activating+degraded
5 active+recovery_wait
4 active+undersized
4 active+clean+scrubbing+deep
1 active+recovery_wait+undersized+degraded
1 active+remapped+backfill_toofull
io:
client: 44 MiB/s rd, 4.2 MiB/s wr, 991 op/s rd, 389 op/s wr
recovery: 71 MiB/s, 18 objects/s
Nico Schottelius <nico.schottelius(a)ungleich.ch> writes:
Hello,
after having moved 4 ssds to another host (+ the ceph tell hanging issue
- see previous mail), we ran into 241 unknown pgs:
cluster:
id: 1ccd84f6-e362-4c50-9ffe-59436745e445
health: HEALTH_WARN
noscrub flag(s) set
2 nearfull osd(s)
1 pool(s) nearfull
Reduced data availability: 241 pgs inactive
1532 slow requests are blocked > 32 sec
789 slow ops, oldest one blocked for 1949 sec, daemons
[osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops.
services:
mon: 3 daemons, quorum black1,black2,black3 (age 97m)
mgr: black2(active, since 96m), standbys: black1, black3
osd: 85 osds: 85 up, 82 in; 118 remapped pgs
flags noscrub
rgw: 1 daemon active (admin)
data:
pools: 12 pools, 3000 pgs
objects: 33.96M objects, 129 TiB
usage: 388 TiB used, 159 TiB / 548 TiB avail
pgs: 8.033% pgs unknown
409151/101874117 objects misplaced (0.402%)
2634 active+clean
241 unknown
107 active+remapped+backfill_wait
11 active+remapped+backfilling
7 active+clean+scrubbing+deep
io:
client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr
recovery: 67 MiB/s, 17 objects/s
This used to be around 700+ unknown, however these 241 are stuck in this
state for more than 1h. Below is a sample of pgs from "ceph pg dump
all | grep unknown"
2.7f7 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.7c7 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.7c2 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.7ab 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.78b 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.788 0 0 0 0 0 0 0
0 0 0 unknown 2020-09-22 19:03:00.694873
0'0 0:0 [] -1 [] -1
0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873
0
2.76e 0
Using ceph pg 2.7f7 query hangs.
We checked and one server did have an incorrect MTU setting (9204
instead of the correct 9000), but that was fixed some hours ago.
Does anyone have a hint on how to find those unknown osds?
Version wise this is 14.2.9:
[20:42:20] black2.place6:~# ceph versions
{
"mon": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 3
},
"mgr": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 3
},
"osd": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 85
},
"mds": {},
"rgw": {
"ceph version 20200428-923-g4004f081ec
(4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1
},
"overall": {
"ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus
(stable)": 91,
"ceph version 20200428-923-g4004f081ec
(4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1
}
}
From ceph health detail:
[20:42:58] black2.place6:~# ceph health detail
HEALTH_WARN noscrub flag(s) set; 2 nearfull osd(s); 1 pool(s) nearfull; Reduced data
availability: 241 pgs inactive; 1575 slow requests are blocked > 32 sec; 751 slow ops,
oldest one blocked for 1986 sec, daemons
[osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.31,osd.33,osd.35]... have slow ops.
OSDMAP_FLAGS noscrub flag(s) set
OSD_NEARFULL 2 nearfull osd(s)
osd.36 is near full
osd.54 is near full
POOL_NEARFULL 1 pool(s) nearfull
pool 'ssd' is nearfull
PG_AVAILABILITY Reduced data availability: 241 pgs inactive
pg 2.82 is stuck inactive for 6027.042489, current state unknown, last acting []
pg 2.88 is stuck inactive for 6027.042489, current state unknown, last acting []
...
pg 19.6e is stuck inactive for 6027.042489, current state unknown, last acting []
pg 20.69 is stuck inactive for 6027.042489, current state unknown, last acting []
As can be seen, multiple pools are affected even though most missing pgs
are from pool 2.
Best regards,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
1331
days inactive
1331
days old
10 comments
3 participants
participants (3)
-
Andreas John -
Frank Schilder -
Nico Schottelius