Hello I am getting the following error for SSD osd.10. root@ceph6:~# tailf /var/log/ceph/ceph-osd.10.log 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 _get_class not permitted to load lua 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 warning: got an error loading one or more classes: (1) Operation not permitted 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376, adjusting msgr requires for clients 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has features 3314933000854323200, adjusting msgr requires for osds 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 check_osdmap_features require_osd_release unknown -> octopus 2023-11-12T20:47:22.942+0600 7f17e9213d80 0 osd.10 239394 load_pgs 2023-11-12T20:47:53.767+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 25923326091ns 2023-11-12T20:49:16.757+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 74835459564ns 2023-11-12T20:50:14.991+0600 7f17e9213d80 0 bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed for next, latency = 42822161884ns Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Sun, Nov 12, 2023 at 8:43 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue?
#ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000
root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
Regards Mosharaf Hossain Manager, Product Development IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com