Hi, the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing. root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d) data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops There's no error on services (mgr, mon, osd). Can you please advise how to identify the root cause of this slow ops? THX
that's probably just https://tracker.ceph.com/issues/43893 (a harmless bug) Restart the mons to get rid of the message Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing.
root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean
io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops
There's no error on services (mgr, mon, osd).
Can you please advise how to identify the root cause of this slow ops?
THX
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I have stopped all 3 MON services sequentially. After starting the 3 MON services again, the slow ops where gone. However, just after 1 min. of MON service uptime, the slow ops are back again, and the blocked time is increasing constantly. root@ld3955:/home/ceph-scripts # ceph -w cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 17 nearfull osd(s) 1 pool(s) nearfull 2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 67s) mgr: ld5505(active, since 11d), standbys: ld5506, ld5507 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 4w), 442 in (since 4w); 10 remapped pgs data: pools: 7 pools, 19628 pgs objects: 72.14M objects, 275 TiB usage: 826 TiB used, 705 TiB / 1.5 PiB avail pgs: 16920/216422157 objects misplaced (0.008%) 19618 active+clean 10 active+remapped+backfilling io: client: 454 KiB/s rd, 15 MiB/s wr, 905 op/s rd, 463 op/s wr recovery: 125 MiB/s, 31 objects/s 2020-03-05 09:21:48.647440 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:21:53.648708 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 68 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:21:58.650186 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 73 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:03.651447 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 78 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:08.653066 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 83 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:13.654699 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 88 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:18.655912 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 93 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:23.657263 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 98 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:28.658514 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 103 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:33.659965 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 108 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:38.661360 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 113 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:43.662727 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 118 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:48.663940 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 123 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:53.685451 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 128 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:58.691603 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 133 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:03.692841 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 138 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:08.694502 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 143 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:13.695991 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 148 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:18.697689 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 153 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:23.698945 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 158 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:28.700331 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 163 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:33.701754 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 168 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:38.703021 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 173 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:43.704396 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 178 sec, mon.ld5505 has slow ops (SLOW_OPS) I have the impression that this is not a harmless bug anymore. Please advise how to proceed. THX Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just https://tracker.ceph.com/issues/43893 (a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing.
root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean
io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops
There's no error on services (mgr, mon, osd).
Can you please advise how to identify the root cause of this slow ops?
THX
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Thomas, I would First try to get more space - as ceph will block io when your disks are full - perhaps your PGs are unbalaced. Does ceph osd df tree give any hint? Or is this already resolved? Hth Mehmet Am 5. März 2020 09:26:13 MEZ schrieb Thomas Schneider <74cmonty@gmail.com>:
Hi,
I have stopped all 3 MON services sequentially. After starting the 3 MON services again, the slow ops where gone. However, just after 1 min. of MON service uptime, the slow ops are back again, and the blocked time is increasing constantly.
root@ld3955:/home/ceph-scripts # ceph -w cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 17 nearfull osd(s) 1 pool(s) nearfull 2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 67s) mgr: ld5505(active, since 11d), standbys: ld5506, ld5507 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 4w), 442 in (since 4w); 10 remapped pgs
data: pools: 7 pools, 19628 pgs objects: 72.14M objects, 275 TiB usage: 826 TiB used, 705 TiB / 1.5 PiB avail pgs: 16920/216422157 objects misplaced (0.008%) 19618 active+clean 10 active+remapped+backfilling
io: client: 454 KiB/s rd, 15 MiB/s wr, 905 op/s rd, 463 op/s wr recovery: 125 MiB/s, 31 objects/s
2020-03-05 09:21:48.647440 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:21:53.648708 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 68 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:21:58.650186 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 73 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:03.651447 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 78 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:08.653066 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 83 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:13.654699 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 88 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:18.655912 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 93 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:23.657263 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 98 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:28.658514 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 103 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:33.659965 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 108 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:38.661360 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 113 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:43.662727 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 118 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:48.663940 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 123 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:53.685451 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 128 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:58.691603 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 133 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:03.692841 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 138 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:08.694502 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 143 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:13.695991 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 148 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:18.697689 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 153 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:23.698945 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 158 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:28.700331 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 163 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:33.701754 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 168 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:38.703021 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 173 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:43.704396 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 178 sec, mon.ld5505 has slow ops (SLOW_OPS)
I have the impression that this is not a harmless bug anymore.
Please advise how to proceed.
THX
Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just https://tracker.ceph.com/issues/43893 (a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing.
root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean
io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops
There's no error on services (mgr, mon, osd).
Can you please advise how to identify the root cause of this slow ops?
THX
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, there's no issue with full OSDs / pools after setting weight on specific OSDs. cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 2 slow ops, oldest one blocked for 345057 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5505(active, since 2w), standbys: ld5506, ld5507 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 4w), 442 in (since 3d) data: pools: 7 pools, 19628 pgs objects: 67.21M objects, 256 TiB usage: 769 TiB used, 762 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 458 KiB/s rd, 21 MiB/s wr, 907 op/s rd, 666 op/s wr Regards Thomas Am 08.03.2020 um 00:33 schrieb ceph@elchaka.de:
Hi Thomas,
I would First try to get more space - as ceph will block io when your disks are full - perhaps your PGs are unbalaced.
Does ceph osd df tree give any hint?
Or is this already resolved?
Hth Mehmet
Am 5. März 2020 09:26:13 MEZ schrieb Thomas Schneider <74cmonty@gmail.com>:
Hi,
I have stopped all 3 MON services sequentially. After starting the 3 MON services again, the slow ops where gone. However, just after 1 min. of MON service uptime, the slow ops are back again, and the blocked time is increasing constantly.
root@ld3955:/home/ceph-scripts # ceph -w cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 17 nearfull osd(s) 1 pool(s) nearfull 2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 67s) mgr: ld5505(active, since 11d), standbys: ld5506, ld5507 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 4w), 442 in (since 4w); 10 remapped pgs
data: pools: 7 pools, 19628 pgs objects: 72.14M objects, 275 TiB usage: 826 TiB used, 705 TiB / 1.5 PiB avail pgs: 16920/216422157 objects misplaced (0.008%) 19618 active+clean 10 active+remapped+backfilling
io: client: 454 KiB/s rd, 15 MiB/s wr, 905 op/s rd, 463 op/s wr recovery: 125 MiB/s, 31 objects/s
2020-03-05 09:21:48.647440 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 63 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:21:53.648708 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 68 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:21:58.650186 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 73 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:03.651447 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 78 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:08.653066 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 83 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:13.654699 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 88 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:18.655912 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 93 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:23.657263 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 98 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:28.658514 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 103 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:33.659965 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 108 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:38.661360 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 113 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:43.662727 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 118 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:48.663940 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 123 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:53.685451 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 128 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:22:58.691603 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 133 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:03.692841 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 138 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:08.694502 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 143 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:13.695991 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 148 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:18.697689 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 153 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:23.698945 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 158 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:28.700331 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 163 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:33.701754 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 168 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:38.703021 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 173 sec, mon.ld5505 has slow ops (SLOW_OPS) 2020-03-05 09:23:43.704396 mon.ld5505 [WRN] Health check update: 2 slow ops, oldest one blocked for 178 sec, mon.ld5505 has slow ops (SLOW_OPS)
I have the impression that this is not a harmless bug anymore.
Please advise how to proceed.
THX
Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just https://tracker.ceph.com/issues/43893 (a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing.
root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean
io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops
There's no error on services (mgr, mon, osd).
Can you please advise how to identify the root cause of this slow ops? THX
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I have upgraded to 14.2.8 and rebooted all nodes sequentially including all 3 MON services. However the slow ops are still displayed with increasing block time. # ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 17 daemons have recently crashed 2263 slow ops, oldest one blocked for 183885 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 2d) mgr: ld5505(active, since 2d), standbys: ld5506, ld5507 mds: cephfs:2 {0=ld4257=up:active,1=ld5508=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 441 up (since 38h), 441 in (since 38h) data: pools: 7 pools, 19628 pgs objects: 68.65M objects, 262 TiB usage: 786 TiB used, 744 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 3.3 KiB/s rd, 3.1 MiB/s wr, 7 op/s rd, 25 op/s wr I have the impression that this is not a harmless bug anymore. Please advise how to proceed. THX Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just https://tracker.ceph.com/issues/43893 (a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing.
root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean
io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops
There's no error on services (mgr, mon, osd).
Can you please advise how to identify the root cause of this slow ops?
THX
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I have upgraded to 14.2.8 and rebooted all nodes sequentially including all 3 MON services. However the slow ops are still displayed with increasing block time. If this is a (harmless) bug anymore, then I would like to understand what's causing it. But if this is not a bug, then I would like to know how to start analyse the root cause, means which operations are affected, which OSDs contribute to slow IO, etc. Please advise how to proceed. # ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 17 daemons have recently crashed 2263 slow ops, oldest one blocked for 183885 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 2d) mgr: ld5505(active, since 2d), standbys: ld5506, ld5507 mds: cephfs:2 {0=ld4257=up:active,1=ld5508=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 441 up (since 38h), 441 in (since 38h) data: pools: 7 pools, 19628 pgs objects: 68.65M objects, 262 TiB usage: 786 TiB used, 744 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 3.3 KiB/s rd, 3.1 MiB/s wr, 7 op/s rd, 25 op/s wr Please advise how to proceed. THX Am 17.02.2020 um 18:31 schrieb Paul Emmerich:
that's probably just https://tracker.ceph.com/issues/43893 (a harmless bug)
Restart the mons to get rid of the message
Paul
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 17, 2020 at 2:59 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing.
root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d)
data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean
io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr
The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops
There's no error on services (mgr, mon, osd).
Can you please advise how to identify the root cause of this slow ops?
THX
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
ceph＠elchaka.de
-
Paul Emmerich
-
Thomas Schneider