- ceph-users - lists.ceph.io

virtual machines crashes after upgrade to octopus
by Lomayani S. Laizer 24 Sep '20

24 Sep '20

Hello, Last week i got time to try debug crashes of these vms Below log includes rados debug which i left last time https://storage.habari.co.tz/index.php/s/AQEJ7tQS7epC4Zn I have observed the following with these settings in openstack and ceph disk_cachemodes="network=writeback" is set in openstack environment and in ceph rbd_cache_policy = writearound is set--- crashes occur disk_cachemodes="network=writeback" is set in openstack environment and in ceph rbd_cache_policy = writeback is set--- no crashes disk_cachemodes="none" is set in openstack environment and in ceph rbd_cache_policy = writearound is set---- no crashes disk_cachemodes="none" is set in openstack environment and in ceph rbd_cache_policy = writeback is set --- crashes occur Is disk_cachemodes="network=writeback" is no longer recommended in octopus because i see it is left out in new documentation for octopus? https://ceph.readthedocs.io/en/latest/rbd/rbd-openstack/ >* > Hello,* >* >* >* > Below is full debug log of 2 minutes before crash of virtual machine.* >* Download from below url* >* >* >* > https://storage.habari.co.tz/index.php/s/31eCwZbOoRTMpcU <https://storage.habari.co.tz/index.php/s/31eCwZbOoRTMpcU>* > >* This log has rbd debug output, but not rados :(* > >* I guess you'll need to try and capture a coredump if you can't get a* >* backtrace.* > >* I'd also suggest opening a tracker in case one of the rbd devs has any* >* ideas on this, or has seen something similar. Without a backtrace or* >* core it will be impossible to definitively identify the issue though.* > +1 to needing the backtrace. I don't see any indications of a problem in that log. >* >* >* >* >* > apport.log* >* >* >* > Wed May 13 09:35:30 2020: host pid 4440 crashed in a separate mount* >* namespace, ignoring* >* >* >* > kernel.log* >* > May 13 09:35:30 compute5 kernel: [123071.373217] fn-radosclient[4485]:* >* segfault at 0 ip 00007f4c8c85d7ed sp 00007f4c66ffc470 error 4 in* >* librbd.so.1.12.0[7f4c8c65a000+5cb000]* >* > May 13 09:35:30 compute5 kernel: [123071.373228] Code: 8d 44 24 08 48 81* >* c3 d8 3e 00 00 49 21 f9 48 c1 e8 30 83 c0 01 48 c1 e0 30 48 89 02 48 8b 03* >* 48 89 04 24 48 8b 34 24 48 21 fe <48> 8b 06 48 89 44 24 08 48 8b 44 24 08* >* 48 8b 0b 48 21 f8 48 39 0c* >* > May 13 09:35:33 compute5 kernel: [123074.832700] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* > May 13 09:35:33 compute5 kernel: [123074.838520] device tap33511c4d-2c* >* left promiscuous mode* >* > May 13 09:35:33 compute5 kernel: [123074.838527] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* >* >* > syslog* >* > compute5 kernel: [123071.373217] fn-radosclient[4485]: segfault at 0 ip* >* 00007f4c8c85d7ed sp 00007f4c66ffc470 error 4 i* >* > n librbd.so.1.12.0[7f4c8c65a000+5cb000]* >* > May 13 09:35:30 compute5 kernel: [123071.373228] Code: 8d 44 24 08 48 81* >* c3 d8 3e 00 00 49 21 f9 48 c1 e8 30 83 c0 01 48 c1 e0 30 48 8* >* > 9 02 48 8b 03 48 89 04 24 48 8b 34 24 48 21 fe <48> 8b 06 48 89 44 24 08* >* 48 8b 44 24 08 48 8b 0b 48 21 f8 48 39 0c* >* > May 13 09:35:30 compute5 libvirtd[1844]: internal error: End of file* >* from qemu monitor* >* > May 13 09:35:33 compute5 systemd-networkd[1326]: tap33511c4d-2c: Link* >* DOWN* >* > May 13 09:35:33 compute5 systemd-networkd[1326]: tap33511c4d-2c: Lost* >* carrier* >* > May 13 09:35:33 compute5 kernel: [123074.832700] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* > May 13 09:35:33 compute5 kernel: [123074.838520] device tap33511c4d-2c* >* left promiscuous mode* >* > May 13 09:35:33 compute5 kernel: [123074.838527] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* > May 13 09:35:33 compute5 networkd-dispatcher[1614]: Failed to request* >* link: No such device* >* >* >* > On Fri, May 8, 2020 at 5:40 AM Brad Hubbard <bhubbard@xxxxxxxxxx> wrote:* >* >>* >* >> On Fri, May 8, 2020 at 12:10 PM Lomayani S. Laizer <lomlaizer@xxxxxxxxx>* >* wrote:* >* >> >* >* >> > Hello,* >* >> > On my side at point of vm crash these are logs below. At the moment* >* my debug is at 10 value. I will rise to 20 for full debug. these crashes* >* are random and so far happens on very busy vms. Downgrading clients in host* >* to Nautilus these crashes disappear* >* >>* >* >> You could try adding debug_rados as well but you may get a very large* >* >> log so keep an eye on things.* >* >>* >* >> >* >* >> > Qemu is not shutting down in general because other vms on the same* >* host continues working* >* >>* >* >> A process can not reliably continue after encountering a segfault so* >* >> the qemu-kvm process must be ending and therefore it should be* >* >> possible to capture a coredump with the right configuration.* >* >>* >* >> In the following example, if you were to search for pid 6060 you would* >* >> find it is no longer running.* >* >> >> > [ 7682.233684] fn-radosclient[6060]: segfault at 2b19 ip* >* 00007f8165cc0a50 sp 00007f81397f6490 error 4 in* >* librbd.so.1.12.0[7f8165ab4000+537000]* >* >>* >* >> Without a backtrace at a minimum it may be very difficult to work out* >* >> what's going on with certainty. If you open a tracker for the issue* >* >> though maybe one of the devs specialising in rbd may have some* >* >> feedback.* >* >>* >* >> >* >* >> > 2020-05-07T13:02:12.121+0300 7f88d57fa700 10 librbd::io::ReadResult:* >* 0x7f88c80bfbf0 finish: got {} for [0,24576] bl 24576* >* >> > 2020-05-07T13:02:12.193+0300 7f88d57fa700 10 librbd::io::ReadResult:* >* 0x7f88c80f9330 finish: C_ObjectReadRequest: r=0* >* >> > 2020-05-07T13:02:12.193+0300 7f88d57fa700 10 librbd::io::ReadResult:* >* 0x7f88c80f9330 finish: got {} for [0,16384] bl 16384* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10 librbd::ImageState:* >* 0x5569b5da9bb0 0x5569b5da9bb0 send_close_unlock* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10 librbd::ImageState:* >* 0x5569b5da9bb0 0x5569b5da9bb0 send_close_unlock* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_block_image_watcher* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 block_notifies* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 5 librbd::Watcher:* >* 0x7f88c400dfe0 block_notifies: blocked_count=1* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_block_image_watcher: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_update_watchers* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_shut_down_update_watchers: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_io_queue* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 5* >* librbd::io::ImageRequestWQ: 0x7f88e8001570 shut_down: shut_down: in_flight=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_shut_down_io_queue: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_exclusive_lock* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ExclusiveLock:* >* 0x7f88c4011ba0 shut_down* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 shut_down:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 send_shutdown:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 send_shutdown_release:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ExclusiveLock:* >* 0x7f88c4011ba0 pre_release_lock_handler* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_cancel_op_requests:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_cancel_op_requests: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020 send_block_writes:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 5* >* librbd::io::ImageRequestWQ: 0x7f88e8001570 block_writes: 0x5569b5e1ffd0,* >* num=1* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_block_writes: r=0* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020 send_wait_for_ops:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_wait_for_ops:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_invalidate_cache:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 5* >* librbd::io::ObjectDispatcher: 0x5569b5dab700 invalidate_cache:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_invalidate_cache: r=0* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_flush_notifies:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_flush_notifies:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_close_object_map:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::object_map::UnlockRequest: 0x7f88c807a450 send_unlock:* >* oid=rbd_object_map.2f18f2a67fad72* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::object_map::UnlockRequest: 0x7f88c807a450 handle_unlock: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_close_object_map: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020 send_unlock:* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 handle_shutdown_pre_release: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10* >* librbd::managed_lock::ReleaseRequest: 0x7f88c80b68a0 send_unlock:* >* entity=client.58292796, cookie=auto 140225447738256* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::managed_lock::ReleaseRequest: 0x7f88c80b68a0 handle_unlock: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ExclusiveLock:* >* 0x7f88c4011ba0 post_release_lock_handler: r=0 shutting_down=1* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 5* >* librbd::io::ImageRequestWQ: 0x7f88e8001570 unblock_writes: 0x5569b5e1ffd0,* >* num=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 notify released lock* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 current lock owner: [0,0]* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 handle_shutdown_post_release: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 wait_for_tracked_ops: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 complete_shutdown: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_shut_down_exclusive_lock: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_unregister_image_watcher* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 unregistering image watcher* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::Watcher:* >* 0x7f88c400dfe0 unregister_watch:* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 5 librbd::Watcher:* >* 0x7f88c400dfe0 notifications_blocked: blocked=1* >* >> > 2020-05-07T13:02:28.706+0300 7f88ceffd700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_unregister_image_watcher: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88ceffd700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_flush_readahead* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_flush_readahead: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_object_dispatcher* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::io::ObjectDispatcher: 0x5569b5dab700 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::io::ObjectDispatch: 0x5569b5ee8360 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::io::SimpleSchedulerObjectDispatch: 0x7f88c4013ce0 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::cache::WriteAroundObjectDispatch: 0x7f88c8003780 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_shut_down_object_dispatcher: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_flush_op_work_queue* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_flush_op_work_queue: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_flush_image_watcher: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10 librbd::ImageState:* >* 0x5569b5da9bb0 0x5569b5da9bb0 handle_close: r=0* >* >> >* >* >> > On Fri, May 8, 2020 at 12:40 AM Brad Hubbard <bhubbard@xxxxxxxxxx>* >* wrote:* >* >> >>* >* >> >> On Fri, May 8, 2020 at 3:42 AM Erwin Lubbers <erwin@xxxxxxxxxxx>* >* wrote:* >* >> >> >* >* >> >> > Hi,* >* >> >> >* >* >> >> > Did anyone found a way to resolve the problem? I'm seeing the same* >* on a clean Octopus Ceph installation on Ubuntu 18 with an Octopus compiled* >* KVM server running on CentOS 7.8. The KVM machine shows:* >* >> >> >* >* >> >> > [ 7682.233684] fn-radosclient[6060]: segfault at 2b19 ip* >* 00007f8165cc0a50 sp 00007f81397f6490 error 4 in* >* librbd.so.1.12.0[7f8165ab4000+537000]* >* >> >>* >* >> >> Are you able to either capture a backtrace from a coredump or set up* >* >> >> logging and hopefully capture a backtrace that way?* >* >> >>* >* >> >> >* >* >> >> > Ceph is healthy and stable for a few weeks and I did not get these* >* messages while running on KVM compiled with Luminous libraries.* >* >> >> >* >* >> >> > Regards,* >* >> >> > Erwin* >* >> >> > _______________________________________________* >* >> >> > ceph-users mailing list -- ceph-users@xxxxxxx* >* >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx* >* >> >> >* >* >> >>* >* >> >>* >* >> >> --* >* >> >> Cheers,* >* >> >> Brad* >* >> >> _______________________________________________* >* >> >> ceph-users mailing list -- ceph-users@xxxxxxx* >* >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx* >* >>* >* >>* >* >>* >* >> --* >* >> Cheers,* >* >> Brad* >* >>* > > >* --* >* Cheers,* >* Brad*

4 4

ScholarOn
by glenda.davisn＠gmail.com 24 Sep '20

24 Sep '20

Students are welcome to utilize our <a href=" https://scholaron.com/subjects/human-resource-management ">Human Resource Management Answers</a> available on <a href=" https://scholaron.com/ ">Scholar On</a> with 100% quality and satisfaction guarantee. We aim at making all your educational endeavors fruitful with the help of multiple learning resources prepared with up-to-date academic needs of highered students.

1 0

Solve the issue of time scarcity via assignment help in Kuwait
by kevin wick 24 Sep '20

24 Sep '20

Are you facing a scarcity of timings or lack of time while composing assignments for Kuwait Universities? Are you not sure while taking the help of anybody to compose your assignments? For every student, assignments sound necessary but time-consuming tasks. You have to manage your time for your projects to score high marks as you can’t ignore your assignments during your study tenure. In this situation, if you can’t manage your time and require reliable assistance for your important task, place your order for assignment help even in Kuwait. Two important things that you need to keep in mind while working on your assignments are time management and quality content. Most students fail to submit their assignments on time because they could not manage their time and can’t collect relevant information for drafting their academic papers. However, don’t burst out and suffer your marks because of any reason. Instead of it, connect with assignment writing service provider and get your work done on time if you have less time to write your assignment. Many students achieve high marks on their projects due to the assistance of experts and professional writers. When you quote order for assignment writing help, you will provide enough time to engage yourself in some other academic tasks. If you don’t have time for writing your assignments and have no idea how to collect particulars for your work, transfer your project to professionals. Experts know how to arrange relevant information for composing the effective academic papers so you will not lose your marks. Expert’s knowledge and experience will allow you to connect with the right source of information and help you to score high marks. So, if you have issues in writing your academic papers, don’t forget to check out the services of online academic writing. https://www.greatassignmenthelp.com/kw/

3 2

ScholarOn
by glenda.davisn＠gmail.com 24 Sep '20

24 Sep '20

Students are welcome to utilize our <a href="https://scholaron.com/subjects/human-resource-management">Human Resource Management Answers</a> available on <a href="https://scholaron.com/">Scholar On</a> with 100% quality and satisfaction guarantee. We aim at making all your educational endeavors fruitful with the help of multiple learning resources prepared with up-to-date academic needs of highered students.

1 0

Module 'cephadm' has failed: auth get failed: failed to find osd.6 in keyring retval: -2
by bwolas＠gmail.com 24 Sep '20

24 Sep '20

Hello, I got this error after I tried upgrade from ceph:v15.2.4 to ceph:v15.2.5 with dead osd.6 cluster: id: 4e01640b-951b-4f75-8dca-0bad4faf1b11 health: HEALTH_ERR Module 'cephadm' has failed: auth get failed: failed to find osd.6 in keyring retval: -2 In short I added osd.6 on centos, but due network problems osd.6 failed to start, at a time I didint understand why and there was similar looking problem fixed in ceph:v15.2.5. So I tried to upgrade with cephadm, upgrade got stuck with this error, I canceled upgrade. Now osd.6 is fixed and working, but I still have this error. "ceph auth ls" shows osd.6 like every other osd it is working fine. I tried to upgrade again but this error keeps appearing in ceph -w and upgrade gets stuck again(everything is upgraded to ceph:v15.2.5 anyway) osd.6 is on separate host which has no other daemons.

1 1

OSD v15.2.5 daemon not starting on Centos7
by Christoph Ackermann 23 Sep '20

23 Sep '20

Hello, does anyone tried to update 15.2.4 on Centos7 to 15.2.5? I did a full: yum -y update on my first OSD node and after this no OSD on this want to start anymore. No log will be written, so osd process stops immediately i think. Starting osd daemon in forground shows that no tmpfs will be mountet under /var/lib/ceph/osd/ No kyering will be found! [root@ceph1n011 ~]# ceph-osd --id 10 2020-09-23T15:20:47.380+0200 7f1e0a018bc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2020-09-23T15:20:47.380+0200 7f1e0a018bc0 -1 AuthRegistry(0x560ee2ab8938) no keyring found at /var/lib/ceph/osd/ceph-10/keyring, disabling cephx 2020-09-23T15:20:47.382+0200 7f1e0a018bc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2020-09-23T15:20:47.382+0200 7f1e0a018bc0 -1 AuthRegistry(0x7ffe06f76400) no keyring found at /var/lib/ceph/osd/ceph-10/keyring, disabling cephx failed to fetch mon config (--no-mon-config to skip) I checked all systemd realated files and found no differences so far. This is quite strange and i look for some time to find the reason ..!? Other OSD on other (untouched) node shows this on startup.... [root@ceph1n012 ~]# ceph-osd --id 24 2020-09-23T15:15:23.514+0200 7f55ae95abc0 -1 Falling back to public interface 2020-09-23T15:15:43.143+0200 7f55ae95abc0 -1 osd.24 2154072 log_to_monitors {default=true} [root@ceph1n012 ~]# After some seconds osd process fork itself to background and works fine.... Does anybody have a clever hint for me? Best regards, Christoph -- Christoph Ackermann | System Engineer INFOSERVE GmbH | Am Felsbrunnen 15 | D-66119 Saarbrücken Fon +49 (0)681 88008-59 | Fax +49 (0)681 88008-33 | mailto:C.Ackermann@infoserve.de | https://www.infoserve.de INFOSERVE Datenschutzhinweise: https://infoserve.de/datenschutz Handelsregister: Amtsgericht Saarbrücken, HRB 11001 | Erfüllungsort: Saarbrücken Geschäftsführer: Dr. Stefan Leinenbach | Ust-IdNr.: DE168970599

1 1

Why seek help from an estate agent?
by devidpaul41＠gmail.com 23 Sep '20

23 Sep '20

Working with estate agents might not be the first choice for many of us. But if you are trying to find someone who can help you with your house for sale in Chichester or finding a renter, the estate agent can help you to meet potential investors and buyers. Estate agents in Chichester always ensure you get the best service. Opposed to the seemingly endless negotiations and contract signings for purchasing, estate agent make the process faster and do all authentication before bringing any offer in front of you. Estate agents in Chichester always act informative, helpful and responsive and look at tenants as a potential buyer. However, if you are looking for houses in Chichester to rent, ensure to check with the legal property owner and be permitted to let out the property. https://www.todanstee.com/cgi-bin/properties-for-sale-rent/rental-search.pl

1 0

Low level bluestore usage
by Ivan Kurnosov 23 Sep '20

23 Sep '20

Hi, this morning I woke up to a degraded test ceph cluster (managed by rook, but it does not really change anything for the question I'm about to ask). After checking logs I have found that bluestore on one of the OSDs run out of space. Some cluster details: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) it runs on 3 little OSDs 10Gb each `ceph osd df` returned RAW USE of about 4.5GB on every node, happily reporting about 5.5GB of AVAIL. Yet: debug -9> 2020-09-22T20:23:15.421+0000 7f29e9798f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1600806195424423, "job": 1, "event": "recovery_started", "log_files": [347, 350]} debug -8> 2020-09-22T20:23:15.421+0000 7f29e9798f40 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #347 mode 0 debug -7> 2020-09-22T20:23:16.465+0000 7f29e9798f40 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #350 mode 0 debug -6> 2020-09-22T20:23:18.689+0000 7f29e9798f40 1 bluefs _allocate failed to allocate 0x17a2360 on bdev 1, free 0x390000; fallback to bdev 2 debug -5> 2020-09-22T20:23:18.689+0000 7f29e9798f40 1 bluefs _allocate unable to allocate 0x17a2360 on bdev 2, free 0xffffffffffffffff; fallback to slow device expander debug -4> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluestore(/var/lib/ceph/osd/ceph-0) allocate_bluefs_freespace failed to allocate on 0x39a20000 min_size 0x17b0000 > allocated total 0x6250000 bluefs_shared_alloc_size 0x10000 allocated 0x0 available 0x 12ee32000 debug -3> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluefs _allocate failed to expand slow device to fit +0x17a2360 debug -2> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x17a2360 debug -1> 2020-09-22T20:23:18.693+0000 7f29e9798f40 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4/rpm/el8/BUILD/ceph-15.2.4/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f29e9798f40 time 2020-09-22T20:23:18.690014+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4/rpm/el8/BUILD/ceph-15.2.4/src/os/bluestore/BlueFS.cc: 2696: ceph_abort_msg("bluefs enospc") So, my question would be: how could I have prevented that? From monitoring I have (prometheus) - OSDs are healthy, have plenty of space, yet they are not. What command (and prometheus metric) would help me understand the actual real bluestore use? Or am I missing something? Oh, and I "fixed" the cluster by expanding the broken osd.0 with a larger 15GB volume. And 2 other OSDs still run on 10GB volumes. Thanks in advance for any thoughts. -- With best regards, Ivan Kurnosov

4 4

samba vfs_ceph: client_mds_namespace not working?
by Frank Schilder 23 Sep '20

23 Sep '20

Dear all, maybe someone has experienced this before. We are setting up a SAMBA gateway and would like to use the vfs_ceph module. In case of several file systems one needs to choose an mds namespace. There is an option in ceph.conf: client mds namespace = CEPH-FS-NAME Unfortunately, it seems not to work. I tried it in all possible versions, in [global] and [client], with and without "client" at the beginning, to no avail. I either get a time out or an error. I also found the libcephfs function ceph_select_filesystem(cmount, CEPH-FS-NAME) added it to vfs_ceph.c just before the ceph_mount with the same result, I get an error (operation not permitted). Does anyone know how to get this to work? And, yes, I tested an ordinary kernel fs mount with the credentials for the ceph client without problems. I can't access any documentation on the libcephfs api, I always get a page not found error. My last resort is now to ceph fs set-default CEPH-FS-NAME to the fs to be used and live with the implied restrictions and ugliness. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3

A disk move gone wrong & Luminous vs. Nautilus performance
by Nico Schottelius 23 Sep '20

23 Sep '20

Good morning, you might have seen my previous mails and I wanted to discuss some findings over the last day+night over what happened and why it happened here. As the system behaved inexplicitly for us, we are now looking for someone to analyse the root cause on consultancy basis - if you are interested & available, please drop me a PM. What happened ------------- Yesterday at about 1520 we moved 4 SSDs from server15 to server8. These SSDs are located in 3 SSD only pools. Soon after that the cluster begun to exhibit slow I/O on clients and saw many PGs in the peering state. Additionally we began to see slow ops on unrelated OSDs, i.e. osds that have a device class "hdd-big" set and are only selected by the "hdd" pool. Excerpt from ceph -s: 167 slow ops, oldest one blocked for 958 sec, daemons [osd.0,osd.11,osd.12,osd.14,osd.15,osd. 16,osd.19,osd.2,osd.22,osd.25]... have slow ops. All pgs that were hanging in peering state seemed to be related to pools from the moved SSDs. However the question remains why other osds began to be affected. Then we noticed that the newly started OSDs on server8 are consuming 100% cpu, the associated disks have a queue depth of 1 and the disks are 80-99% busy. This is a change that we noticed some weeks ago in the nautilus osd behaviour: while in luminous, osds would start right away, osds often take seconds to minutes to start in nautilus, accompanied with high cpu usage. Before yesterday however, the maximum startup time was in the range of minutes. When we noticed blocked/slow IOPS, we wanted to ensure that the clients are least possible affected. We tried to submit the two changes to all osds: ceph tell osd.* injectargs '--osd-recovery-max-active 1' ceph tell osd.* injectargs '--osd-max-backfills 1' However at this time, ceph tell would hang. We then switched to address each OSD individually using a for loop: for osd in $(ceph osd tree | grep osd. | awk '{ print $4 }'); do echo $osd; ( ceph tell $osd injectargs '--osd-max-backfills 1' &) ; done However this resulted in many hanging ceph tell processes and some of them began to report the following error: 2020-09-22 15:59:25.407 7fbc0359e700 0 --1- [2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/1280010984 >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6856/4078 conn(0x7fbbec02edc0 0x7fbbec02f5d0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER This message would be repeated many times a second per ceph tell process until we SIGINT killed it. After this failed we noticed that the remaining HDDs in server15 were at 100% i/o usage at about 0.6-2 MB/s reading rate. These low rates would very well explain the slow I/O on the clients - but it made us wonder why the rates were suddenly so low. We restarted one of the OSDs on this server, but the effect was that the osd would now use 100% cpu additionally to fully utilising the disk. We began to suspected a hardware problem and moved all HDDs to server8. Unfortunately, all newly HDD OSDs behave exactly the same: Very high cpu usage (60-100%), very high disk utilisation (about 90%+) and very low transfer rates (~1 MB/s on average). After a few hours (!) some of the OSDs normalised to 20-50MB/s read rates and after about 8h later all of them normalised. What is curious that during the slow phase of many hours, the average queue depth of the disk consistently stays at 1, while afterwards they usually average around 10-15. Given full utilisation, we suspect that the nautilus OSDs cause severly disk seeks. Our expectation (and so far behaviour with Luminous) was that if we have a single host failure, setting max backfills and max recovery to 1 and keeping the osd_recovery_op_priority low (usually at 2 in our clusters) has some, but more a minor impact on client I/O. This time, purely moving 4 osds with existing data made the cluster practically unusable, something we never experienced in a Luminous setup before. While I do not want to render out any potential mistakes or incorrect designs from our side, the effects seen are not expected from our side. While all of this happening we measured the network bandwidth on all nodes without congestion, monitored the monitors with some (spikes up to 50% from one core), but not high cpu usage. I was wondering whether anyone on the list has an insight on these questions: - Why were OSDs slow that are used in a different pool? - What changed in the OSDs from Luminous to Nautilus to make the startup phase slow? - What are nautilus OSDs doing at startup and why do they consume a lot of CPU? - What are the OSDs doing to severly impact the I/O? Is our seek theory correct? - Why does / can ceph tell hang in Nautilus? We never experienced this problem in Luminous. - Where does the BADAUTHORIZER message come from and what is the fix for it? - How can we debug / ensure that ceph tell does not hang? - Did code in the osds change that de-prioritizes command traffic? Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

1 0