17 Feb
2020
17 Feb
'20
2:58 p.m.
Hi,
the current output of ceph -s reports a warning:
2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
This time is increasing.
root@ld3955:~# ceph -s
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
9 daemons have recently crashed
2 slow ops, oldest one blocked for 347335 sec, mon.ld5505
has slow ops
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
mgr: ld5507(active, since 8m), standbys: ld5506, ld5505
mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
up:standby-replay 3 up:standby
osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
data:
pools: 7 pools, 19628 pgs
objects: 65.78M objects, 251 TiB
usage: 753 TiB used, 779 TiB / 1.5 PiB avail
pgs: 19628 active+clean
io:
client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
The details are as follows:
root@ld3955:~# ceph health detail
HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one
blocked for 347755 sec, mon.ld5505 has slow ops
RECENT_CRASH 9 daemons have recently crashed
mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z
mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z
mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z
mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z
mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z
mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z
mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z
mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z
mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z
SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has
slow ops
There's no error on services (mgr, mon, osd).
Can you please advise how to identify the root cause of this slow ops?
THX
17 Feb
17 Feb
6:31 p.m.
that's probably just https://tracker.ceph.com/issues/43893
(a harmless bug)
Restart the mons to get rid of the message
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Mon, Feb 17, 2020 at 2:59 PM Thomas Schneider <74cmonty(a)gmail.com> wrote:
>
> Hi,
>
> the current output of ceph -s reports a warning:
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
> This time is increasing.
>
> root@ld3955:~# ceph -s
> cluster:
> id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
> health: HEALTH_WARN
> 9 daemons have recently crashed
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505
> has slow ops
>
> services:
> mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
> mgr: ld5507(active, since 8m), standbys: ld5506, ld5505
> mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
> up:standby-replay 3 up:standby
> osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
>
> data:
> pools: 7 pools, 19628 pgs
> objects: 65.78M objects, 251 TiB
> usage: 753 TiB used, 779 TiB / 1.5 PiB avail
> pgs: 19628 active+clean
>
> io:
> client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
>
> The details are as follows:
> root@ld3955:~# ceph health detail
> HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one
> blocked for 347755 sec, mon.ld5505 has slow ops
> RECENT_CRASH 9 daemons have recently crashed
> mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z
> mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z
> mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z
> mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z
> mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z
> SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has
> slow ops
>
> There's no error on services (mgr, mon, osd).
>
> Can you please advise how to identify the root cause of this slow ops?
>
> THX
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5 Mar
5 Mar
9:26 a.m.
Hi,
I have stopped all 3 MON services sequentially.
After starting the 3 MON services again, the slow ops where gone.
However, just after 1 min. of MON service uptime, the slow ops are back
again, and the blocked time is increasing constantly.
root@ld3955:/home/ceph-scripts
# ceph -w
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
17 nearfull osd(s)
1 pool(s) nearfull
2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has
slow ops
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 67s)
mgr: ld5505(active, since 11d), standbys: ld5506, ld5507
mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
up:standby-replay 3 up:standby
osd: 442 osds: 442 up (since 4w), 442 in (since 4w); 10 remapped pgs
data:
pools: 7 pools, 19628 pgs
objects: 72.14M objects, 275 TiB
usage: 826 TiB used, 705 TiB / 1.5 PiB avail
pgs: 16920/216422157 objects misplaced (0.008%)
19618 active+clean
10 active+remapped+backfilling
io:
client: 454 KiB/s rd, 15 MiB/s wr, 905 op/s rd, 463 op/s wr
recovery: 125 MiB/s, 31 objects/s
2020-03-05 09:21:48.647440 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:21:53.648708 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 68 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:21:58.650186 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 73 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:03.651447 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 78 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:08.653066 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 83 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:13.654699 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 88 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:18.655912 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 93 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:23.657263 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 98 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:28.658514 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 103 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:33.659965 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 108 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:38.661360 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 113 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:43.662727 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 118 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:48.663940 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 123 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:53.685451 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 128 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:58.691603 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 133 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:03.692841 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 138 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:08.694502 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 143 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:13.695991 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 148 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:18.697689 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 153 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:23.698945 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 158 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:28.700331 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 163 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:33.701754 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 168 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:38.703021 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 173 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:43.704396 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 178 sec, mon.ld5505 has slow ops (SLOW_OPS)
I have the impression that this is not a harmless bug anymore.
Please advise how to proceed.
THX
Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just
https://tracker.ceph.com/issues/43893
(a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us
at https://croit.io croit GmbH Freseniusstr. 31h 81247 München
www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM
Thomas Schneider <74cmonty(a)gmail.com> wrote:
> Hi,
>
> the current output of ceph -s reports a warning:
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
> This time is increasing.
>
> root@ld3955:~# ceph -s
> cluster:
> id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
> health: HEALTH_WARN
> 9 daemons have recently crashed
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505
> has slow ops
>
> services:
> mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
> mgr: ld5507(active, since 8m), standbys: ld5506, ld5505
> mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
> up:standby-replay 3 up:standby
> osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
>
> data:
> pools: 7 pools, 19628 pgs
> objects: 65.78M objects, 251 TiB
> usage: 753 TiB used, 779 TiB / 1.5 PiB avail
> pgs: 19628 active+clean
>
> io:
> client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
>
> The details are as follows:
> root@ld3955:~# ceph health detail
> HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one
> blocked for 347755 sec, mon.ld5505 has slow ops
> RECENT_CRASH 9 daemons have recently crashed
> mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z
> mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z
> mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z
> mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z
> mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z
> SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has
> slow ops
>
> There's no error on services (mgr, mon, osd).
>
> Can you please advise how to identify the root cause of this slow ops?
>
> THX
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8 Mar
8 Mar
12:33 a.m.
Hi Thomas,
I would First try to get more space - as ceph will block io when your disks are full -
perhaps your PGs are unbalaced.
Does ceph osd df tree give any hint?
Or is this already resolved?
Hth
Mehmet
Am 5. März 2020 09:26:13 MEZ schrieb Thomas Schneider <74cmonty(a)gmail.com>om>:
Hi,
I have stopped all 3 MON services sequentially.
After starting the 3 MON services again, the slow ops where gone.
However, just after 1 min. of MON service uptime, the slow ops are back
again, and the blocked time is increasing constantly.
root@ld3955:/home/ceph-scripts
# ceph -w
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
17 nearfull osd(s)
1 pool(s) nearfull
2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has
slow ops
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 67s)
mgr: ld5505(active, since 11d), standbys: ld5506, ld5507
mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
up:standby-replay 3 up:standby
osd: 442 osds: 442 up (since 4w), 442 in (since 4w); 10 remapped
pgs
data:
pools: 7 pools, 19628 pgs
objects: 72.14M objects, 275 TiB
usage: 826 TiB used, 705 TiB / 1.5 PiB avail
pgs: 16920/216422157 objects misplaced (0.008%)
19618 active+clean
10 active+remapped+backfilling
io:
client: 454 KiB/s rd, 15 MiB/s wr, 905 op/s rd, 463 op/s wr
recovery: 125 MiB/s, 31 objects/s
2020-03-05 09:21:48.647440 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:21:53.648708 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 68 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:21:58.650186 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 73 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:03.651447 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 78 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:08.653066 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 83 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:13.654699 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 88 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:18.655912 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 93 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:23.657263 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 98 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:28.658514 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 103 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:33.659965 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 108 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:38.661360 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 113 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:43.662727 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 118 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:48.663940 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 123 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:53.685451 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 128 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:58.691603 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 133 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:03.692841 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 138 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:08.694502 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 143 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:13.695991 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 148 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:18.697689 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 153 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:23.698945 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 158 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:28.700331 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 163 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:33.701754 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 168 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:38.703021 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 173 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:43.704396 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 178 sec, mon.ld5505 has slow ops (SLOW_OPS)
I have the impression that this is not a harmless bug anymore.
Please advise how to proceed.
THX
Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just
https://tracker.ceph.com/issues/43893
(a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us
at https://croit.io croit GmbH Freseniusstr. 31h 81247 München
www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM
Thomas Schneider <74cmonty(a)gmail.com> wrote:
> Hi,
>
> the current output of ceph -s reports a warning:
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow
ops
> This time is increasing.
>
> root@ld3955:~# ceph -s
> cluster:
> id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
> health: HEALTH_WARN
> 9 daemons have recently crashed
> 2 slow ops, oldest one blocked for 347335 sec,
mon.ld5505
> has slow ops
>
> services:
> mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
> mgr: ld5507(active, since 8m), standbys: ld5506, ld5505
> mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
> up:standby-replay 3 up:standby
> osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
>
> data:
> pools: 7 pools, 19628 pgs
> objects: 65.78M objects, 251 TiB
> usage: 753 TiB used, 779 TiB / 1.5 PiB avail
> pgs: 19628 active+clean
>
> io:
> client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
>
> The details are as follows:
> root@ld3955:~# ceph health detail
> HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one
> blocked for 347755 sec, mon.ld5505 has slow ops
> RECENT_CRASH 9 daemons have recently crashed
> mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z
> mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z
> mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z
> mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z
> mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z
> SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505
has
> slow ops
>
> There's no error on services (mgr, mon, osd).
>
> Can you please advise how to identify the root cause of this slow
ops?
>
> THX
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9 Mar
9 Mar
9:12 a.m.
Hi,
there's no issue with full OSDs / pools after setting weight on specific
OSDs.
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
2 slow ops, oldest one blocked for 345057 sec, mon.ld5505
has slow ops
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
mgr: ld5505(active, since 2w), standbys: ld5506, ld5507
mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
up:standby-replay 3 up:standby
osd: 442 osds: 442 up (since 4w), 442 in (since 3d)
data:
pools: 7 pools, 19628 pgs
objects: 67.21M objects, 256 TiB
usage: 769 TiB used, 762 TiB / 1.5 PiB avail
pgs: 19628 active+clean
io:
client: 458 KiB/s rd, 21 MiB/s wr, 907 op/s rd, 666 op/s wr
Regards
Thomas
Am 08.03.2020 um 00:33 schrieb ceph(a)elchaka.de:
Hi Thomas,
I would First try to get more space - as ceph will block io when your disks are full -
perhaps your PGs are unbalaced.
Does ceph osd df tree give any hint?
Or is this already resolved?
Hth
Mehmet
Am 5. März 2020 09:26:13 MEZ schrieb Thomas Schneider <74cmonty(a)gmail.com>om>:
Hi,
I have stopped all 3 MON services sequentially.
After starting the 3 MON services again, the slow ops where gone.
However, just after 1 min. of MON service uptime, the slow ops are back
again, and the blocked time is increasing constantly.
root@ld3955:/home/ceph-scripts
# ceph -w
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
17 nearfull osd(s)
1 pool(s) nearfull
2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has
slow ops
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 67s)
mgr: ld5505(active, since 11d), standbys: ld5506, ld5507
mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
up:standby-replay 3 up:standby
osd: 442 osds: 442 up (since 4w), 442 in (since 4w); 10 remapped
pgs
data:
pools: 7 pools, 19628 pgs
objects: 72.14M objects, 275 TiB
usage: 826 TiB used, 705 TiB / 1.5 PiB avail
pgs: 16920/216422157 objects misplaced (0.008%)
19618 active+clean
10 active+remapped+backfilling
io:
client: 454 KiB/s rd, 15 MiB/s wr, 905 op/s rd, 463 op/s wr
recovery: 125 MiB/s, 31 objects/s
2020-03-05 09:21:48.647440 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:21:53.648708 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 68 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:21:58.650186 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 73 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:03.651447 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 78 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:08.653066 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 83 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:13.654699 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 88 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:18.655912 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 93 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:23.657263 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 98 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:28.658514 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 103 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:33.659965 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 108 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:38.661360 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 113 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:43.662727 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 118 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:48.663940 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 123 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:53.685451 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 128 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:22:58.691603 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 133 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:03.692841 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 138 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:08.694502 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 143 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:13.695991 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 148 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:18.697689 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 153 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:23.698945 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 158 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:28.700331 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 163 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:33.701754 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 168 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:38.703021 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 173 sec, mon.ld5505 has slow ops (SLOW_OPS)
2020-03-05 09:23:43.704396 mon.ld5505 [WRN] Health check update: 2 slow
ops, oldest one blocked for 178 sec, mon.ld5505 has slow ops (SLOW_OPS)
I have the impression that this is not a harmless bug anymore.
Please advise how to proceed.
THX
Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io that's probably just
https://tracker.ceph.com/issues/43893
(a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us
at https://croit.io croit GmbH Freseniusstr. 31h 81247 München
www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM
Thomas Schneider <74cmonty(a)gmail.com> wrote:
> Hi,
>
> the current output of ceph -s reports a warning:
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow
ops
> This time is increasing.
>
> root@ld3955:~# ceph -s
> cluster:
> id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
> health: HEALTH_WARN
> 9 daemons have recently crashed
> 2 slow ops, oldest one blocked for 347335 sec,
mon.ld5505
> has slow ops
>
> services:
> mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
> mgr: ld5507(active, since 8m), standbys: ld5506, ld5505
> mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
> up:standby-replay 3 up:standby
> osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
>
> data:
> pools: 7 pools, 19628 pgs
> objects: 65.78M objects, 251 TiB
> usage: 753 TiB used, 779 TiB / 1.5 PiB avail
> pgs: 19628 active+clean
>
> io:
> client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
>
> The details are as follows:
> root@ld3955:~# ceph health detail
> HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one
> blocked for 347755 sec, mon.ld5505 has slow ops
> RECENT_CRASH 9 daemons have recently crashed
> mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z
> mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z
> mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z
> mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z
> mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z
> SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505
has
> slow ops
>
> There's no error on services (mgr, mon, osd).
>
> Can you please advise how to identify the root cause of this slow
ops?
> THX
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Mar
23 Mar
9:40 a.m.
Hi,
I have upgraded to 14.2.8 and rebooted all nodes sequentially including
all 3 MON services.
However the slow ops are still displayed with increasing block time.
# ceph -s
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
17 daemons have recently crashed
2263 slow ops, oldest one blocked for 183885 sec, mon.ld5505
has slow ops
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 2d)
mgr: ld5505(active, since 2d), standbys: ld5506, ld5507
mds: cephfs:2 {0=ld4257=up:active,1=ld5508=up:active} 2
up:standby-replay 3 up:standby
osd: 442 osds: 441 up (since 38h), 441 in (since 38h)
data:
pools: 7 pools, 19628 pgs
objects: 68.65M objects, 262 TiB
usage: 786 TiB used, 744 TiB / 1.5 PiB avail
pgs: 19628 active+clean
io:
client: 3.3 KiB/s rd, 3.1 MiB/s wr, 7 op/s rd, 25 op/s wr
I have the impression that this is not a harmless bug anymore.
Please advise how to proceed.
THX
Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just
https://tracker.ceph.com/issues/43893
(a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us
at https://croit.io croit GmbH Freseniusstr. 31h 81247 München
www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM
Thomas Schneider <74cmonty(a)gmail.com> wrote:
> Hi,
>
> the current output of ceph -s reports a warning:
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
> This time is increasing.
>
> root@ld3955:~# ceph -s
> cluster:
> id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
> health: HEALTH_WARN
> 9 daemons have recently crashed
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505
> has slow ops
>
> services:
> mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
> mgr: ld5507(active, since 8m), standbys: ld5506, ld5505
> mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
> up:standby-replay 3 up:standby
> osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
>
> data:
> pools: 7 pools, 19628 pgs
> objects: 65.78M objects, 251 TiB
> usage: 753 TiB used, 779 TiB / 1.5 PiB avail
> pgs: 19628 active+clean
>
> io:
> client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
>
> The details are as follows:
> root@ld3955:~# ceph health detail
> HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one
> blocked for 347755 sec, mon.ld5505 has slow ops
> RECENT_CRASH 9 daemons have recently crashed
> mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z
> mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z
> mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z
> mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z
> mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z
> SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has
> slow ops
>
> There's no error on services (mgr, mon, osd).
>
> Can you please advise how to identify the root cause of this slow ops?
>
> THX
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
27 Mar
27 Mar
5:18 p.m.
Hi,
I have upgraded to 14.2.8 and rebooted all nodes sequentially including
all 3 MON services.
However the slow ops are still displayed with increasing block time.
If this is a (harmless) bug anymore, then I would like to understand
what's causing it.
But if this is not a bug, then I would like to know how to start analyse
the root cause, means which operations are affected, which OSDs
contribute to slow IO, etc.
Please advise how to proceed.
# ceph -s
cluster:
id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
health: HEALTH_WARN
17 daemons have recently crashed
2263 slow ops, oldest one blocked for 183885 sec, mon.ld5505
has slow ops
services:
mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 2d)
mgr: ld5505(active, since 2d), standbys: ld5506, ld5507
mds: cephfs:2 {0=ld4257=up:active,1=ld5508=up:active} 2
up:standby-replay 3 up:standby
osd: 442 osds: 441 up (since 38h), 441 in (since 38h)
data:
pools: 7 pools, 19628 pgs
objects: 68.65M objects, 262 TiB
usage: 786 TiB used, 744 TiB / 1.5 PiB avail
pgs: 19628 active+clean
io:
client: 3.3 KiB/s rd, 3.1 MiB/s wr, 7 op/s rd, 25 op/s wr
Please advise how to proceed.
THX
Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just
https://tracker.ceph.com/issues/43893
(a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us
at https://croit.io croit GmbH Freseniusstr. 31h 81247 München
www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM
Thomas Schneider <74cmonty(a)gmail.com> wrote:
> Hi,
>
> the current output of ceph -s reports a warning:
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
> This time is increasing.
>
> root@ld3955:~# ceph -s
> cluster:
> id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae
> health: HEALTH_WARN
> 9 daemons have recently crashed
> 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505
> has slow ops
>
> services:
> mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d)
> mgr: ld5507(active, since 8m), standbys: ld5506, ld5505
> mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2
> up:standby-replay 3 up:standby
> osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
>
> data:
> pools: 7 pools, 19628 pgs
> objects: 65.78M objects, 251 TiB
> usage: 753 TiB used, 779 TiB / 1.5 PiB avail
> pgs: 19628 active+clean
>
> io:
> client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
>
> The details are as follows:
> root@ld3955:~# ceph health detail
> HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one
> blocked for 347755 sec, mon.ld5505 has slow ops
> RECENT_CRASH 9 daemons have recently crashed
> mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z
> mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z
> mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z
> mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z
> mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z
> mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z
> mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z
> SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has
> slow ops
>
> There's no error on services (mgr, mon, osd).
>
> Can you please advise how to identify the root cause of this slow ops?
>
> THX
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1484
days inactive
1523
days old
6 comments
3 participants
participants (3)
-
ceph＠elchaka.de -
Paul Emmerich -
Thomas Schneider