[ceph-users] rbd: map failed: rbd: sysfs write failed -- (108) Cannot send after transport endpoint shutdown

Hi, mapping of rbd volumes fails clusterwide. The volumes that are mapped, are ok, but new volumes wont map. Receiving errors liks: (108) Cannot send after transport endpoint shutdown or executing: #rbd -p CEPH map vm-120-disk-0 will show: rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (108) Cannot send after transport endpoint shutdown I added some more debugging for the logs and got this: rbd: sysfs write failed 2021-07-01T10:28:44.258+0200 7f803c4a33c0 5 librbd::io::ImageRequestWQ: 0x7f802c0011f0 ImageRequestWQ: ictx=0x55846e936c40 2021-07-01T10:28:44.258+0200 7f803c4a33c0 20 librbd::ImageState: 0x7f802c001420 open 2021-07-01T10:28:44.258+0200 7f803c4a33c0 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 send_open_unlock 2021-07-01T10:28:44.258+0200 7f803c4a33c0 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_detect_header 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: handle_v2_detect_header: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_id 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: handle_v2_get_id: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_initial_metadata 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: handle_v2_get_initial_metadata: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_create_timestamp 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 handle_v2_get_create_timestamp: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_access_modify_timestamp 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 handle_v2_get_access_modify_timestamp: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_data_pool 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 handle_v2_get_data_pool: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::ImageCtx: init_layout stripe_unit 4194304 stripe_count 1 object_size 4194304 prefix rbd_data.21f6f3a7ec4663 format rbd_data.21f6f3a7ec4663.%016llx 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_refresh 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::ConfigWatcher: init: 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_mutable_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_mutable_metadata: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_parent: legacy=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_parent: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f80100126a0 metadata_list: start_key=conf_ 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f80100126a0 handle_metadata_list: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f80100126a0 finish: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_metadata: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_pool_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f8010012c10 metadata_list: start_key=conf_ 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f8010012c10 handle_metadata_list: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f8010012c10 finish: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_pool_metadata: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 20 librbd::ImageCtx: apply_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 5 librbd::ImageCtx: 0x55846e936c40: disabling zero-copy writes 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_group 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_group: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_apply 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_apply 2021-07-01T10:28:44.266+0200 7f8028ff9700 20 librbd::image::RefreshRequest: 0x7f8010011870 apply 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::OpenRequest: handle_refresh: r=0 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::OpenRequest: send_parent_cache: r=0 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::OpenRequest: 0x55846e979620 send_init_cache 2021-07-01T10:28:44.266+0200 7f8028ff9700 5 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 init: 2021-07-01T10:28:44.266+0200 7f8028ff9700 5 librbd::io::ObjectDispatcher: 0x55846e979670 register_object_dispatch: object_dispatch_layer=1 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 handle_open: r=0 In some cases useful info is found in syslog - try "dmesg | tail". 2021-07-01T10:28:44.266+0200 7f803c4a33c0 20 librbd::ImageState: 0x7f802c001420 close 2021-07-01T10:28:44.266+0200 7f803c4a33c0 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 send_close_unlock 2021-07-01T10:28:44.266+0200 7f803c4a33c0 10 librbd::ConfigWatcher: shut_down: 2021-07-01T10:28:44.266+0200 7f803c4a33c0 10 librbd::image::CloseRequest: 0x55846e97a510 send_block_image_watcher 2021-07-01T10:28:44.270+0200 7f803c4a33c0 10 librbd::ImageWatcher: 0x7f8010003e00 block_notifies 2021-07-01T10:28:44.270+0200 7f803c4a33c0 5 librbd::Watcher: 0x7f8010003e00 block_notifies: blocked_count=1 2021-07-01T10:28:44.270+0200 7f803c4a33c0 10 librbd::image::CloseRequest: 0x55846e97a510 handle_block_image_watcher: r=0 2021-07-01T10:28:44.270+0200 7f803c4a33c0 10 librbd::image::CloseRequest: 0x55846e97a510 send_shut_down_update_watchers 2021-07-01T10:28:44.270+0200 7f803c4a33c0 20 librbd::ImageState: 0x7f802c001420 shut_down_update_watchers 2021-07-01T10:28:44.270+0200 7f803c4a33c0 20 librbd::ImageState: 0x55846e80a230 ImageUpdateWatchers::shut_down 2021-07-01T10:28:44.270+0200 7f803c4a33c0 20 librbd::ImageState: 0x55846e80a230 ImageUpdateWatchers::shut_down: completing shut down 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_shut_down_update_watchers: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_shut_down_io_queue 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::io::ImageRequestWQ: 0x7f802c0011f0 shut_down: shut_down: in_flight=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f80180024c8 start_op 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ImageRequest: 0x7f8028ff3d30 send: aio_flush: ictx=0x55846e936c40, completion=0x7f80180023d0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f80180023d0 set_request_count: pending=1 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ObjectDispatcher: 0x55846e979670 send: object_dispatch_spec=0x7f8018002520 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 flush: 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f80180023d0 complete_request: cb=1, pending=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f80180023d0 finalize: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_shut_down_io_queue: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_flush 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f8018002718 start_op 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ImageRequest: 0x7f8028ff3c70 send: aio_flush: ictx=0x55846e936c40, completion=0x7f8018002620 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f8018002620 set_request_count: pending=1 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f80180024c8 finish_op 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: completed flush: 0x7f801000e6d0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ObjectDispatcher: 0x55846e979670 send: object_dispatch_spec=0x7f8018002520 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 flush: 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f8018002620 complete_request: cb=1, pending=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f8018002620 finalize: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_unregister_image_watcher 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::ImageWatcher: 0x7f8010003e00 unregistering image watcher 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::Watcher: 0x7f8010003e00 unregister_watch: 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f8018002718 finish_op 2021-07-01T10:28:44.270+0200 7f8022ffd700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_unregister_image_watcher: r=0 2021-07-01T10:28:44.270+0200 7f8022ffd700 10 librbd::image::CloseRequest: 0x55846e97a510 send_flush_readahead 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush_readahead: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_shut_down_object_dispatcher 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::io::ObjectDispatcher: 0x55846e979670 shut_down: 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::io::ObjectDispatch: 0x55846e97b430 shut_down: 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 shut_down: 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_shut_down_object_dispatcher: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_flush_op_work_queue 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush_op_work_queue: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush_image_watcher: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 handle_close: r=0 rbd: map failed: (108) Cannot send after transport endpoint shutdown ... which does actually not tell me too much honestly. For now, i am out of idea's, can someone lead a way to analyse / fix this ? New volumes can be created and removed without issues. But they wont map. Thank you! -- Mit freundlichen Gruessen / Best regards Oliver Dzombic Layer7 Networks mailto:info@layer7.net Anschrift: Layer7 Networks GmbH Zum Sonnenberg 1-3 63571 Gelnhausen HRB 96293 beim Amtsgericht Hanau Geschäftsführung: Oliver Dzombic UST ID: DE259845632