OSD disk is active in node but ceph show osd down and out
Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue? #ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000 root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>
Hello I am getting the following error for SSD osd.10. root@ceph6:~# tailf /var/log/ceph/ceph-osd.10.log 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 _get_class not permitted to load lua 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 warning: got an error loading one or more classes: (1) Operation not permitted 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376, adjusting msgr requires for clients 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 3314933000854323200, adjusting msgr requires for osds 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 check_osdmap_features require_osd_release unknown -> octopus 2023-11-12T20:47:22.942+0600 7f17e9213d80 0 osd.10 239394 load_pgs 2023-11-12T20:47:53.767+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 25923326091ns 2023-11-12T20:49:16.757+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 74835459564ns 2023-11-12T20:50:14.991+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 42822161884ns Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Sun, Nov 12, 2023 at 8:43 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue?
#ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000
root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com
Hello THis osd got up but getting following error and it getting up and down. 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c05fa700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c15fc700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c1dfd700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15 Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Sun, Nov 12, 2023 at 8:51 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello I am getting the following error for SSD osd.10.
root@ceph6:~# tailf /var/log/ceph/ceph-osd.10.log 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 _get_class not permitted to load lua 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 warning: got an error loading one or more classes: (1) Operation not permitted 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376, adjusting msgr requires for clients 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 3314933000854323200, adjusting msgr requires for osds 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 check_osdmap_features require_osd_release unknown -> octopus 2023-11-12T20:47:22.942+0600 7f17e9213d80 0 osd.10 239394 load_pgs 2023-11-12T20:47:53.767+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 25923326091ns 2023-11-12T20:49:16.757+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 74835459564ns 2023-11-12T20:50:14.991+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 42822161884ns
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com
On Sun, Nov 12, 2023 at 8:43 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue?
#ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000
root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com
Hello Now the OSD down again and show following log for the osd. Nov 12 20:51:34 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:51:34.009+0600 7f17dd634700 -1 osd.10 239591 set_numa_affinity unable to identify public interface '' n Nov 12 20:52:43 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:52:43.075+0600 7f17dd634700 -1 osd.10 239597 set_numa_affinity unable to identify public interface '' n Nov 12 20:53:53 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:53:53.066+0600 7f17dd634700 -1 osd.10 239603 set_numa_affinity unable to identify public interface '' n Nov 12 20:55:01 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:55:01.640+0600 7f17dd634700 -1 osd.10 239611 set_numa_affinity unable to identify public interface '' n Nov 12 20:56:09 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:56:09.274+0600 7f17dd634700 -1 osd.10 239617 set_numa_affinity unable to identify public interface '' n Nov 12 20:57:18 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:57:18.508+0600 7f17dd634700 -1 osd.10 239623 set_numa_affinity unable to identify public interface '' n Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.650+0600 7f17d341a700 -1 osd.10 239629 _committed_osd_maps marked down 6 > osd_max_markdown_count Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.654+0600 7f17e0e97700 -1 received signal: Interrupt from Kernel ( Could be generated by pthread_ Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.654+0600 7f17e0e97700 -1 osd.10 239629 *** Got signal Interrupt *** Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.654+0600 7f17e0e97700 -1 osd.10 239629 *** Immediate shutdown (osd_fast_shutdown=true) *** lines 1-15/15 (END) Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Sun, Nov 12, 2023 at 8:53 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello THis osd got up but getting following error and it getting up and down.
2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c05fa700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c15fc700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c1dfd700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com
On Sun, Nov 12, 2023 at 8:51 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello I am getting the following error for SSD osd.10.
root@ceph6:~# tailf /var/log/ceph/ceph-osd.10.log 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 _get_class not permitted to load lua 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 warning: got an error loading one or more classes: (1) Operation not permitted 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376, adjusting msgr requires for clients 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 3314933000854323200, adjusting msgr requires for osds 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 check_osdmap_features require_osd_release unknown -> octopus 2023-11-12T20:47:22.942+0600 7f17e9213d80 0 osd.10 239394 load_pgs 2023-11-12T20:47:53.767+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 25923326091ns 2023-11-12T20:49:16.757+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 74835459564ns 2023-11-12T20:50:14.991+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 42822161884ns
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com
On Sun, Nov 12, 2023 at 8:43 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue?
#ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000
root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com
Hi, can you share the following output: ceph -s ceph health detail ceph versions ceph osd df tree ceph osd dump I see this line in the logs:
check_osdmap_features require_osd_release unknown -> octopus
which makes me wonder if you really run a Nautilus cluster. Are your OSDs saturated? Check iostat for them. If the OSDs tend to "flap" you could temporarily try to set 'ceph osd set nodown' and see if the cluster stabilizes. If it does you'll need to figure out what leads to the flapping. Did you check syslog of your server and/or dmesg output? Zitat von Mosharaf Hossain <mosharaf.hossain@bol-online.com>:
Hello I am getting the following error for SSD osd.10.
root@ceph6:~# tailf /var/log/ceph/ceph-osd.10.log 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 _get_class not permitted to load lua 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 warning: got an error loading one or more classes: (1) Operation not permitted 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376, adjusting msgr requires for clients 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 3314933000854323200, adjusting msgr requires for osds 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 check_osdmap_features require_osd_release unknown -> octopus 2023-11-12T20:47:22.942+0600 7f17e9213d80 0 osd.10 239394 load_pgs 2023-11-12T20:47:53.767+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 25923326091ns 2023-11-12T20:49:16.757+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 74835459564ns 2023-11-12T20:50:14.991+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 42822161884ns
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>
On Sun, Nov 12, 2023 at 8:43 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue?
#ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000
root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Mosharaf Hossain