[ceph-users] Lock errors in iscsi gateway

Hi; I've build two iscsi gateway for our (small) ceph cluster.The cluster is a nautilus installation, 4 nodes with 9x4TB each, and it's working fine. We mainly use it via s3 object storage interface, but I've deployed also some rbd block devices and a cephfs filesystem. Now I'm trying to connect it to my xenserver installation. Xenserver doesn't speak rados, so I've build the iscsi gateways. Right now they are self-hosted on the xenserver, with plan to move them into physical boxes if/when needed. The gateways are build on centos8, tcmu-runner just cloned from git (I think it's 1.5.2). I've been able to connect them to our six nodes xenserver cluster, and now I'm trying to use it. When I attempt a migration of a VM disk, on the new iscsi volume, I've got these messages on the logfile that I find very worrying: Apr 27 17:32:21 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:22 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:22 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:25 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:25 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:26 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:26 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:27 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:27 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:28 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:28 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:29 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:29 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:30 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:30 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:31 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:31 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:32 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:32 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:33 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:33 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:34 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:34 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:36 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. After a while the migration fails, and I keep seend the error on the logs: Apr 27 17:36:01 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:06 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:08 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:09 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:16 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:21 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:21 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:26 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:28 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:29 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:36 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Any hints? Is this a bug? -- *Simone Lazzaris* *Qcom S.p.A. a socio unico* simone.lazzaris(a)qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook[4]* [5]