Hi, I am having an unusual slowdown using VMware with ISCSI gws. I have two ISCSI gateways with two RBD images. I have checked the following in the logs: Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock acquisition operation is already in process.2019-12-24 09:00:26.973 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:28.990 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Can anyone help-me please? Gesiel
In addition: I turned off one of the GWs, and with just one it works fine. When the two go up, one of the images is changing the "active / optimized" all time (where generates the logs above) and everything is extremely slow. I'm using: tcmu-runner-1.4 ceph-iscsi-3.3 ceph 13.2.7 Regards, Gesiel Em ter., 24 de dez. de 2019 às 09:09, Gesiel Galvão Bernardes < gesiel.bernardes@gmail.com> escreveu:
Hi,
I am having an unusual slowdown using VMware with ISCSI gws. I have two ISCSI gateways with two RBD images. I have checked the following in the logs:
Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock acquisition operation is already in process.2019-12-24 09:00:26.973 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:28.990 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock.
Can anyone help-me please?
Gesiel
On 12/24/2019 06:40 AM, Gesiel Galvão Bernardes wrote:
In addition: I turned off one of the GWs, and with just one it works fine. When the two go up, one of the images is changing the "active / optimized" all time (where generates the logs above) and everything is extremely slow.
Your multipathing in ESX is probably misconfigured and you have set it up for active active, or one host can't see all the iscsi paths either because it's not logged into all the sessions or because the network is not up on one of the paths.
I'm using: tcmu-runner-1.4 ceph-iscsi-3.3 ceph 13.2.7
Regards, Gesiel
Em ter., 24 de dez. de 2019 às 09:09, Gesiel Galvão Bernardes <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>> escreveu:
Hi,
I am having an unusual slowdown using VMware with ISCSI gws. I have two ISCSI gateways with two RBD images. I have checked the following in the logs:
Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock acquisition operation is already in process.2019-12-24 09:00:26.973 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:28.990 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock.
Can anyone help-me please?
Gesiel
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Only now have I been able to act on this problem. My environment is relatively simple: I have two ESXi 6.7 hosts, connected to two ISCSI gateways, using two RBD images. When this mail started, the workaround was to keep only one ISCSI gateway connected, so it works normally. After the answer that received here that the problem could be in the configuration of VMWare, I reviewed the configuration of both (they are exactly according to Ceph documentation), and rebooted both. It turns out, now the gateway "ceph-iscsi1" is working. When I turn on "ceph-iscsi2" it appears as "Active / Not optimized" for the two RBD images (before was an Active / Optimized for each image), and if I turn off "ceph-iscsi1" it (ceph-iscs2) remains as "Active / No optimized "and images are unavailable. Nothing is recorded in the logs. Can you help me? Em qui., 26 de dez. de 2019 às 17:44, Mike Christie <mchristi@redhat.com> escreveu:
On 12/24/2019 06:40 AM, Gesiel Galvão Bernardes wrote:
In addition: I turned off one of the GWs, and with just one it works fine. When the two go up, one of the images is changing the "active / optimized" all time (where generates the logs above) and everything is extremely slow.
Your multipathing in ESX is probably misconfigured and you have set it up for active active, or one host can't see all the iscsi paths either because it's not logged into all the sessions or because the network is not up on one of the paths.
I'm using: tcmu-runner-1.4 ceph-iscsi-3.3 ceph 13.2.7
Regards, Gesiel
Em ter., 24 de dez. de 2019 às 09:09, Gesiel Galvão Bernardes <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>
escreveu:
Hi,
I am having an unusual slowdown using VMware with ISCSI gws. I have two ISCSI gateways with two RBD images. I have checked the following in the logs:
Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock acquisition operation is already in process.2019-12-24 09:00:26.973 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:28.990 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock.
Can anyone help-me please?
Gesiel
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 01/20/2020 10:29 AM, Gesiel Galvão Bernardes wrote:
Hi,
Only now have I been able to act on this problem. My environment is relatively simple: I have two ESXi 6.7 hosts, connected to two ISCSI gateways, using two RBD images.
When this mail started, the workaround was to keep only one ISCSI gateway connected, so it works normally. After the answer that received here that the problem could be in the configuration of VMWare, I reviewed the configuration of both (they are exactly according to Ceph documentation), and rebooted both.
Can you give me some basic info. The output of: # "gwcli ls" from one of the iscsi nodes, and give me the output of: # targetcli ls from both iscsi nodes. The ceph-iscsi and tcmu-runner versions and did you build them yourself, get them from the ceph repos or get it from a distro repo.
It turns out, now the gateway "ceph-iscsi1" is working. When I turn on "ceph-iscsi2" it appears as "Active / Not optimized" for the two RBD images (before was an Active / Optimized for each image), and if I turn off "ceph-iscsi1" it (ceph-iscs2) remains as "Active / No optimized "and images are unavailable.
On the ESX side can you give me the output of: esxcli storage nmp path list -d disk_id esxcli storage core device list -d disk_id esxcli storage nmp device list -d disk_id esxcli storage nmp satp list and the /var/log/vmkernel.log for when you stop a node and the image goes to the unavailable state.
Nothing is recorded in the logs. Can you help me?
Em qui., 26 de dez. de 2019 às 17:44, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 12/24/2019 06:40 AM, Gesiel Galvão Bernardes wrote: > In addition: I turned off one of the GWs, and with just one it works > fine. When the two go up, one of the images is changing the "active / > optimized" all time (where generates the logs above) and everything is > extremely slow.
Your multipathing in ESX is probably misconfigured and you have set it up for active active, or one host can't see all the iscsi paths either because it's not logged into all the sessions or because the network is not up on one of the paths.
> > I'm using: > tcmu-runner-1.4 > ceph-iscsi-3.3 > ceph 13.2.7 > > Regards, > Gesiel > > Em ter., 24 de dez. de 2019 às 09:09, Gesiel Galvão Bernardes > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>> escreveu: > > Hi, > > I am having an unusual slowdown using VMware with ISCSI gws. I have > two ISCSI gateways with two RBD images. I have checked the following > in the logs: > > Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 > [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: > Starting lock acquisition operation.2019-12-24 09:00:26.040 969 > [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock > acquisition operation is already in process.2019-12-24 09:00:26.973 > 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired > exclusive lock. > Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 > rbd/pool1.vmware_iscsi1: Acquired exclusive lock. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 > [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock > drop. Old state 1 > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 > rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: > alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock > acquisition operation. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 > [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: > Starting lock acquisition operation.2019-12-24 09:00:28.990 969 > [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive > lock. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 > rbd/pool1.vmware_iscsi1: Acquired exclusive lock. > > > Can anyone help-me please? > > Gesiel > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> >
Hi, Just now was possible continue this. Below is the information required. Thanks advance, Gesiel Em seg., 20 de jan. de 2020 às 15:06, Mike Christie <mchristi@redhat.com> escreveu:
On 01/20/2020 10:29 AM, Gesiel Galvão Bernardes wrote:
Hi,
Only now have I been able to act on this problem. My environment is relatively simple: I have two ESXi 6.7 hosts, connected to two ISCSI gateways, using two RBD images.
When this mail started, the workaround was to keep only one ISCSI gateway connected, so it works normally. After the answer that received here that the problem could be in the configuration of VMWare, I reviewed the configuration of both (they are exactly according to Ceph documentation), and rebooted both.
Can you give me some basic info.
The output of: # "gwcli ls"
o- / ......................................................................................................... [...] o- cluster ......................................................................................... [Clusters: 1] | o- ceph ............................................................................................ [HEALTH_OK] | o- pools .......................................................................................... [Pools: 7] | | o- data_ecpool ................................. [(x3), Commit: 0.00Y/25836962M (0%), Used: 25011763551408b] | | o- pool1 ....................................... [(x2), Commit: 5.0T/38755444M (13%), Used: 16162480333070b] | | o- pool2 ....................................... [(x3), Commit: 6.0T/25836962M (24%), Used: 24261305669809b] | | o- pool3 ........................................ [(4+0), Commit: 0.00Y/58133164M (0%), Used: 719866819603b] | | o- pool_cache ...................................... [(x3), Commit: 0.00Y/706164M (0%), Used: 422549509342b] | | o- poolfs ................................................ [(x3), Commit: 0.00Y/25836962M (0%), Used: 0.00Y] | | o- rbd ................................................... [(x3), Commit: 0.00Y/25836962M (0%), Used: 5737b] | o- topology ............................................................................... [OSDs: 61,MONs: 2] o- disks ....................................................................................... [11.0T, Disks: 3] | o- pool1 ........................................................................................ [pool1 (5.0T)] | | o- vmware_iscsi1 ................................................................ [pool1/vmware_iscsi1 (5.0T)] | o- pool2 ........................................................................................ [pool2 (6.0T)] | o- iscsi-test ...................................................................... [pool2/iscsi-test (1.0T)] | o- vmware_iscsi2 ................................................................ [pool2/vmware_iscsi2 (5.0T)] o- iscsi-targets ............................................................... [DiscoveryAuth: None, Targets: 1] o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................... [Auth: None, Gateways: 2] o- disks .......................................................................................... [Disks: 3] | o- pool1/vmware_iscsi1 ................................................................ [Owner: ceph-iscsi2] | o- pool2/iscsi-test ................................................................... [Owner: ceph-iscsi2] | o- pool2/vmware_iscsi2 ................................................................ [Owner: ceph-iscsi1] o- gateways ............................................................................ [Up: 2/2, Portals: 2] | o- ceph-iscsi1 ........................................................................ [192.168.201.1 (UP)] | o- ceph-iscsi2 ........................................................................ [192.168.201.2 (UP)] o- host-groups .................................................................................. [Groups : 0] o- hosts ....................................................................... [Auth: ACL_ENABLED, Hosts: 2] o- iqn.1994-05.com.redhat:rh7-client .............................. [LOGGED-IN, Auth: CHAP, Disks: 2(10.0T)] | o- lun 0 ................................................. [pool1/vmware_iscsi1(5.0T), Owner: ceph-iscsi2] | o- lun 1 ................................................. [pool2/vmware_iscsi2(5.0T), Owner: ceph-iscsi1] o- iqn.1994-05.com.redhat:tcnvh8 .................................. [LOGGED-IN, Auth: CHAP, Disks: 2(10.0T)] o- lun 0 ................................................. [pool1/vmware_iscsi1(5.0T), Owner: ceph-iscsi2] o- lun 1 ................................................. [pool2/vmware_iscsi2(5.0T), Owner: ceph-iscsi1]
from one of the iscsi nodes, and give me the output of:
# targetcli ls
ceph-iscsi1 ~]# targetcli ls Warning: Could not load preferences file /root/.targetcli/prefs.bin. o- / ......................................................................................................... [...] o- backstores .............................................................................................. [...] | o- block .................................................................................. [Storage Objects: 0] | o- fileio ................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................ [Storage Objects: 0] | o- user:glfs .............................................................................. [Storage Objects: 0] | o- user:qcow .............................................................................. [Storage Objects: 0] | o- user:rbd ............................................................................... [Storage Objects: 3] | | o- pool1.vmware_iscsi1 ............................ [pool1/vmware_iscsi1;osd_op_timeout=30 (5.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.iscsi-test .................................. [pool2/iscsi-test;osd_op_timeout=30 (1.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.vmware_iscsi2 ............................ [pool2/vmware_iscsi2;osd_op_timeout=30 (5.0TiB) activated] | | o- alua ................................................................................... [ALUA Groups: 3] | | o- ano2 ............................................................... [ALUA state: Active/non-optimized] | | o- ao ..................................................................... [ALUA state: Active/optimized] | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | o- user:zbc ............................................................................... [Storage Objects: 0] o- iscsi ............................................................................................ [Targets: 1] | o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................................... [TPGs: 2] | o- tpg1 .......................................................................... [no-gen-acls, auth per-acl] | | o- acls .......................................................................................... [ACLs: 2] | | | o- iqn.1994-05.com.redhat:rh7-client ........................................ [1-way auth, Mapped LUNs: 2] | | | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | | | o- iqn.1994-05.com.redhat:tcnvh8 ............................................ [1-way auth, Mapped LUNs: 2] | | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | | o- luns .......................................................................................... [LUNs: 3] | | | o- lun0 ................................................................ [user/pool1.vmware_iscsi1 (ano1)] | | | o- lun1 ................................................................... [user/pool2.iscsi-test (ano1)] | | | o- lun2 .................................................................. [user/pool2.vmware_iscsi2 (ao)] | | o- portals .................................................................................... [Portals: 1] | | o- 192.168.201.1:3260 ............................................................................... [OK] | o- tpg2 ........................................................................................... [disabled] | o- acls .......................................................................................... [ACLs: 0] | o- luns .......................................................................................... [LUNs: 3] | | o- lun0 .................................................................. [user/pool1.vmware_iscsi1 (ao)] | | o- lun1 ..................................................................... [user/pool2.iscsi-test (ao)] | | o- lun2 ................................................................ [user/pool2.vmware_iscsi2 (ano2)] | o- portals .................................................................................... [Portals: 1] | o- 192.168.201.2:3260 ............................................................................... [OK] o- loopback ......................................................................................... [Targets: 0] ceph-iscsi2 ~]# targetcli ls Warning: Could not load preferences file /root/.targetcli/prefs.bin. o- / ......................................................................................................... [...] o- backstores .............................................................................................. [...] | o- block .................................................................................. [Storage Objects: 0] | o- fileio ................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................ [Storage Objects: 0] | o- user:glfs .............................................................................. [Storage Objects: 0] | o- user:qcow .............................................................................. [Storage Objects: 0] | o- user:rbd ............................................................................... [Storage Objects: 3] | | o- pool1.vmware_iscsi1 ............................ [pool1/vmware_iscsi1;osd_op_timeout=30 (5.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.iscsi-test .................................. [pool2/iscsi-test;osd_op_timeout=30 (1.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.vmware_iscsi2 ............................ [pool2/vmware_iscsi2;osd_op_timeout=30 (5.0TiB) activated] | | o- alua ................................................................................... [ALUA Groups: 3] | | o- ano2 ............................................................... [ALUA state: Active/non-optimized] | | o- ao ..................................................................... [ALUA state: Active/optimized] | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | o- user:zbc ............................................................................... [Storage Objects: 0] o- iscsi ............................................................................................ [Targets: 1] | o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................................... [TPGs: 2] | o- tpg1 ........................................................................................... [disabled] | | o- acls .......................................................................................... [ACLs: 0] | | o- luns .......................................................................................... [LUNs: 3] | | | o- lun0 ................................................................ [user/pool1.vmware_iscsi1 (ano1)] | | | o- lun1 ................................................................... [user/pool2.iscsi-test (ano1)] | | | o- lun2 .................................................................. [user/pool2.vmware_iscsi2 (ao)] | | o- portals .................................................................................... [Portals: 1] | | o- 192.168.201.1:3260 ............................................................................... [OK] | o- tpg2 .......................................................................... [no-gen-acls, auth per-acl] | o- acls .......................................................................................... [ACLs: 2] | | o- iqn.1994-05.com.redhat:rh7-client ........................................ [1-way auth, Mapped LUNs: 2] | | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | | o- iqn.1994-05.com.redhat:tcnvh8 ............................................ [1-way auth, Mapped LUNs: 2] | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | o- luns .......................................................................................... [LUNs: 3] | | o- lun0 .................................................................. [user/pool1.vmware_iscsi1 (ao)] | | o- lun1 ..................................................................... [user/pool2.iscsi-test (ao)] | | o- lun2 ................................................................ [user/pool2.vmware_iscsi2 (ano2)] | o- portals .................................................................................... [Portals: 1] | o- 192.168.201.2:3260 ............................................................................... [OK] o- loopback ......................................................................................... [Targets: 0]
from both iscsi nodes.
The ceph-iscsi and tcmu-runner versions and did you build them yourself, get them from the ceph repos or get it from a distro repo.
I use in both gateways: ceph-iscsi-3.3-1.el7.noarch from ceph-iscsi repo ( http://download.ceph.com/ceph-iscsi/3/rpm/el7/noarch) tcmu-runner-1.4.0-0.1.51.geef5115.el7.x86_64 from https://3.chacra.ceph.com/r/tcmu-runner/master/eef511565078fb4e2ed52caaff16e...
It turns out, now the gateway "ceph-iscsi1" is working. When I turn on "ceph-iscsi2" it appears as "Active / Not optimized" for the two RBD images (before was an Active / Optimized for each image), and if I turn off "ceph-iscsi1" it (ceph-iscs2) remains as "Active / No optimized "and images are unavailable.
On the ESX side can you give me the output of:
esxcli storage nmp path list -d disk_id
esxcli storage nmp path list -d naa.6001405ba48e0b99e4c418ca13506c8e iqn.1994-05.com.redhat:rh7-client-00023d000001,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,1-naa.6001405ba48e0b99e4c418ca13506c8e Runtime Name: vmhba68:C0:T0:L0 Device: naa.6001405ba48e0b99e4c418ca13506c8e Device Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Group State: active unoptimized Array Priority: 0 Storage Array Type Path Config: {TPG_id=1,TPG_state=ANO,RTP_id=1,RTP_health=UP} Path Selection Policy Path Config: {current path; rank: 0}
esxcli storage core device list -d disk_id
:~] esxcli storage core device list -d naa.6001405ba48e0b99e4c418ca13506c8e naa.6001405ba48e0b99e4c418ca13506c8e Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Has Settable Display Name: true Size: 5242880 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/naa.6001405ba48e0b99e4c418ca13506c8e Vendor: LIO-ORG Model: TCMU device Revision: 0002 SCSI Level: 5 Is Pseudo: false Status: degraded Is RDM Capable: true Is Local: false Is Removable: false Is SSD: false Is VVOL PE: false Is Offline: false Is Perennially Reserved: false Queue Full Sample Size: 0 Queue Full Threshold: 0 Thin Provisioning Status: yes Attached Filters: VAAI Status: supported Other UIDs: vml.02000000006001405ba48e0b99e4c418ca13506c8e54434d552064 Is Shared Clusterwide: true Is SAS: false Is USB: false Is Boot Device: false Device Max Queue Depth: 128 No of outstanding IOs with competing worlds: 32 Drive Type: unknown RAID Level: unknown Number of Physical Drives: unknown Protection Enabled: false PI Activated: false PI Type: 0 PI Protection Mask: NO PROTECTION Supported Guard Types: NO GUARD SUPPORT DIX Enabled: false DIX Guard Type: NO GUARD SUPPORT Emulated DIX/DIF Enabled: false
esxcli storage nmp device list -d disk_id
esxcli storage nmp device list -d naa.6001405ba48e0b99e4c418ca13506c8e naa.6001405ba48e0b99e4c418ca13506c8e Device Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} Path Selection Policy: VMW_PSP_MRU Path Selection Policy Device Config: Current Path=vmhba68:C0:T0:L0 Path Selection Policy Device Custom Config: Working Paths: vmhba68:C0:T0:L0 Is USB: false
esxcli storage nmp satp list
esxcli storage nmp satp list Name Default PSP Description
-------------------------------------------------------------------------------- VMW_SATP_ALUA VMW_PSP_MRU Supports non-specific arrays that use the ALUA protocol VMW_SATP_MSA VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_DEFAULT_AP VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_SVC VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_EQL VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_INV VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_EVA VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_ALUA_CX VMW_PSP_RR Placeholder (plugin not loaded) VMW_SATP_SYMM VMW_PSP_RR Placeholder (plugin not loaded) VMW_SATP_CX VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_LSI VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_DEFAULT_AA VMW_PSP_FIXED Supports non-specific active/active arrays VMW_SATP_LOCAL VMW_PSP_FIXED Supports direct attached devices
and the /var/log/vmkernel.log for when you stop a node and the image goes to the unavailable state.
2020-02-02T00:14:59.094Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:14:59.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:14:59.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:01.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405cb71b882378c4138826c2da30" - issuing command 0x45a283482940 2020-02-02T00:15:01.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405cb71b882378c4138826c2da30" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:01.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405cb71b882378c4138826c2da30": awaiting fast path state update... 2020-02-02T00:15:06.979Z cpu25:2124996)ScsiDeviceIO: 3449: Cmd(0x459a9abd2380) 0x9e, CmdSN 0x1dc0cf from world 0 to dev "naa.6001405ba48e0b99e4c418ca13506c8e" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x46 0x80 0x41. 2020-02-02T00:15:06.979Z cpu25:2124996)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:07.094Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405ba48e0b99e4c418ca13506c8e" - no more commands to retry 2020-02-02T00:15:07.095Z cpu33:2098078)WARNING: NMP: nmp_IssueCommandToDevice:5726: I/O could not be issued to device "naa.6001405ba48e0b99e4c418ca13506c8e" due to Not found 2020-02-02T00:15:07.095Z cpu33:2098078)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 2020-02-02T00:15:07.095Z cpu33:2098078)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:08.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:08.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:08.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:08.154Z cpu25:2124996)ScsiDeviceIO: 3449: Cmd(0x45a283482940) 0x12, CmdSN 0x1dc10c from world 0 to dev "naa.6001405cb71b882378c4138826c2da30" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x46 0x80 0x41. 2020-02-02T00:15:08.154Z cpu25:2124996)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405cb71b882378c4138826c2da30" is blocked. Not starting I/O from device. 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405cb71b882378c4138826c2da30" - no more commands to retry 2020-02-02T00:15:14.094Z cpu22:2098078)NMP: nmp_ResetDeviceLogThrottling:3575: last error status from device naa.6001405cb71b882378c4138826c2da30 repeated 35 times 2020-02-02T00:15:14.095Z cpu22:2098078)NMP: nmp_ResetDeviceLogThrottling:3575: last error status from device naa.6001405ba48e0b99e4c418ca13506c8e repeated 8 times 2020-02-02T00:15:39.094Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:39.095Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:39.095Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:46.981Z cpu25:2124996)ScsiDeviceIO: 3449: Cmd(0x459a9abd2380) 0x25, CmdSN 0x1dc10d from world 0 to dev "naa.6001405ba48e0b99e4c418ca13506c8e" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. 2020-02-02T00:15:46.981Z cpu25:2124996)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:47.094Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405ba48e0b99e4c418ca13506c8e" - no more commands to retry 2020-02-02T00:15:47.095Z cpu22:2098078)WARNING: NMP: nmp_IssueCommandToDevice:5726: I/O could not be issued to device "naa.6001405ba48e0b99e4c418ca13506c8e" due to Not found 2020-02-02T00:15:47.095Z cpu22:2098078)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 2020-02-02T00:15:47.095Z cpu22:2098078)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:48.094Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9ba62b40 2020-02-02T00:15:48.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:48.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:49.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9ba62b40 2020-02-02T00:15:49.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:49.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:52.967Z cpu0:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430e49d44a90 network resource pool netsched.pools.persist.iscsi associated 2020-02-02T00:15:52.967Z cpu0:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430e49d44a90 network tracker id 193945594 tracker.iSCSI.192.168.201.1 associated 2020-02-02T00:15:54.712Z cpu0:2098255)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba68:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE" 2020-02-02T00:15:54.712Z cpu0:2098255)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw TPGT: 1 TSIH: 0] 2020-02-02T00:15:54.712Z cpu0:2098255)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 192.168.201.107:35264 R: 192.168.201.1:3260] 2020-02-02T00:15:54.712Z cpu30:2097595)ScsiDevice: 4481: Handle REPORTED LUNS CHANGED DATA unit attention 2020-02-02T00:15:54.712Z cpu30:2097595)ScsiDevice: 4512: Handle INQUIRY PARAMETERS CHANGED unit attention 2020-02-02T00:15:54.716Z cpu2:2097312)ScsiDevice: 6001: Setting Device naa.6001405ba48e0b99e4c418ca13506c8e state back to 0x2 2020-02-02T00:15:54.716Z cpu2:2097312)ScsiDevice: 8708: No Handlers registered! (naa.6001405ba48e0b99e4c418ca13506c8e)! 2020-02-02T00:15:54.716Z cpu2:2097312)ScsiDevice: 6022: Device naa.6001405ba48e0b99e4c418ca13506c8e is Out of APD; token num:1 2020-02-02T00:15:54.716Z cpu2:2097312)StorageApdHandler: 1315: APD exit for 0x4305de04a9d0 [naa.6001405ba48e0b99e4c418ca13506c8e] 2020-02-02T00:15:54.716Z cpu0:2097602)StorageApdHandler: 507: APD exit event for 0x4305de04a9d0 [naa.6001405ba48e0b99e4c418ca13506c8e] 2020-02-02T00:15:54.716Z cpu0:2097602)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6001405ba48e0b99e4c418ca13506c8e] has exited the All Paths Down state. 2020-02-02T00:15:54.717Z cpu11:2097618)ScsiDevice: 6001: Setting Device naa.6001405cb71b882378c4138826c2da30 state back to 0x2 2020-02-02T00:15:54.717Z cpu11:2097618)ScsiDevice: 8708: No Handlers registered! (naa.6001405cb71b882378c4138826c2da30)! 2020-02-02T00:15:54.717Z cpu11:2097618)ScsiDevice: 6022: Device naa.6001405cb71b882378c4138826c2da30 is Out of APD; token num:1 2020-02-02T00:15:54.717Z cpu11:2097618)StorageApdHandler: 1315: APD exit for 0x4305de041660 [naa.6001405cb71b882378c4138826c2da30] 2020-02-02T00:15:54.717Z cpu0:2097602)StorageApdHandler: 507: APD exit event for 0x4305de041660 [naa.6001405cb71b882378c4138826c2da30] 2020-02-02T00:15:54.717Z cpu0:2097602)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6001405cb71b882378c4138826c2da30] has exited the All Paths Down state. 2020-02-02T00:15:55.096Z cpu0:2098243)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6001405ba48e0b99e4c418ca13506c8e" 2020-02-02T00:15:55.096Z cpu8:2099418 opID=17fe6cc5)World: 11943: VC opID sps-Main-52618-962-fe-1-62ed maps to vmkernel opID 17fe6cc5 2020-02-02T00:15:55.096Z cpu8:2099418 opID=17fe6cc5)WARNING: ScsiDeviceIO: 10750: READ CAPACITY on device "naa.6001405ba48e0b99e4c418ca13506c8e" from Plugin "NMP" failed. I/O error 2020-02-02T00:15:55.097Z cpu3:2098243)NMP: nmp_ThrottleLogForDevice:3788: Cmd 0x28 (0x459a9abd2380, 0) to dev "naa.6001405ba48e0b99e4c418ca13506c8e" on path "vmhba68:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. Act:FAILOVER 2020-02-02T00:15:55.097Z cpu3:2098243)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 2020-02-02T00:15:56.095Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:56.099Z cpu3:2098243)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6001405ba48e0b99e4c418ca13506c8e" 2020-02-02T00:16:02.095Z cpu23:2097316)qfle3: qfle3_queue_alloc_with_attr:517: [vmnic0] QueueOps.qfle3_queue_alloc_with_attr num_attr :1 attrs: 0x451a4521bcd0 2020-02-02T00:16:02.095Z cpu23:2097316)qfle3: qfle3_queue_alloc_with_attr:545: [vmnic0] Feature LRO requested. 2020-02-02T00:16:02.095Z cpu23:2097316)qfle3: qfle3_rq_alloc:282: [vmnic0] allocating RX queue at 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_alloc:316: [vmnic0] Marking RX queue 1 IN_USE 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_queue_start:1529: [vmnic0] QueueOps.queueStart 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_queue_start:1537: [vmnic0] RxQ, QueueIDVal:1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rx_queue_start:1452: [vmnic0] qfle3_rx_queue_start, QueueIDVal:1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1287: [vmnic0] qfle3_rq_start 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_init_eth_fp:68: [vmnic0] qfle3_init_eth_fp for fp 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1334: [vmnic0] RX fp[1]: wrote prods bd_prod=4078 cqe_prod=4030 sge_prod=1024 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1351: [vmnic0] enabled netpoll for q_index 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1357: [vmnic0] Enabling interrupt on vector # 3 2020-02-02T00:16:02.097Z cpu23:2097316)qfle3: qfle3_rq_start:1377: [vmnic0] RX queue setup_queue successful for 1 2020-02-02T00:16:02.097Z cpu23:2097316)qfle3: qfle3_rq_start:1411: [vmnic0] active Rx queue Count 2 2020-02-02T00:16:02.097Z cpu23:2097316)qfle3: qfle3_rq_start:1412: [vmnic0] RX queue 1 successfully started 2020-02-02T00:16:02.098Z cpu23:2097316)qfle3: qfle3_queue_remove_filter:2063: [vmnic0] QueueOps.queueRemoveFilter 2020-02-02T00:16:02.099Z cpu23:2097316)qfle3: qfle3_remove_queue_filter:2012: [vmnic0] NetQ removed RX filter: queue:0 mac: 00:50:56:6f:59:4f filter id:3 2020-02-02T00:16:02.099Z cpu23:2097316)qfle3: qfle3_queue_apply_filter:1923: [vmnic0] QueueOps.queueApplyFilter 1 2020-02-02T00:16:02.101Z cpu23:2097316)qfle3: qfle3_apply_queue_mac_filter:1798: [vmnic0] NetQ set RX filter: queue:1 mac: 00:50:56:6f:59:4f filter id:0 2020-02-02T00:16:04.924Z cpu0:2099418 opID=cff7ff5)World: 11943: VC opID sps-Main-52618-962-d0-d6-6321 maps to vmkernel opID cff7ff5 2020-02-02T00:16:04.924Z cpu0:2099418 opID=cff7ff5)Unmap6: 7133: [Unmap] 'datastore1':device(0x43078fc9a1f0)does not support unmap 2020-02-02T00:16:07.164Z cpu9:2271849)J6: 2651: 'Storage_Ceph_pool1': Exiting async journal replay manager world 2020-02-02T00:16:09.665Z cpu12:2271850)J6: 2651: 'Storage_Ceph_pool2': Exiting async journal replay manager world 2020-02-02T00:16:10.930Z cpu4:2271852)J6: 2651: 'datastore1': Exiting async journal replay manager world 2020-02-02T00:16:11.094Z cpu22:2271853)J6: 2651: 'SSD1': Exiting async journal replay manager world 2020-02-02T00:16:22.094Z cpu26:2097316)qfle3: qfle3_queue_remove_filter:2063: [vmnic0] QueueOps.queueRemoveFilter 2020-02-02T00:16:22.097Z cpu26:2097316)qfle3: qfle3_remove_queue_filter:2012: [vmnic0] NetQ removed RX filter: queue:1 mac: 00:50:56:6f:59:4f filter id:0 2020-02-02T00:16:22.098Z cpu26:2097316)qfle3: qfle3_queue_apply_filter:1923: [vmnic0] QueueOps.queueApplyFilter 0 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_apply_queue_mac_filter:1798: [vmnic0] NetQ set RX filter: queue:0 mac: 00:50:56:6f:59:4f filter id:3 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_queue_quiesce:1061: [vmnic0] QueueOps.queueQuiesce 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_queue_quiesce:1069: [vmnic0] RxQ, QueueIDVal:1 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_rx_queue_stop:1558: [vmnic0] qfle3_rx_queue_stop, QueueIDVal:1 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_rq_stop:740: [vmnic0] qfle3_rq_stop 1 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_rq_stop:811: [vmnic0] Stopping queue 0 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_stop:831: [vmnic0] disable netpoll for q_index 1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_stop:842: [vmnic0] Disabling interrupt on vector # 3 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_stop:867: [vmnic0] active Rx queue Count 1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_queue_free:690: [vmnic0] QueueOps.queueFree 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_queue_free:697: [vmnic0] RxQ, QueueIDVal:1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_free:618: [vmnic0] Loop through 1 RSS queues 1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_cmd_remove_q:19191: [vmnic0] Releasing Q idx 1
2020-02-02T00:17:05.925Z cpu4:2100126 opID=4a500647)World: 11943: VC opID sps-Main-52618-962-3a-dd-633d maps to vmkernel opID 4a500647 2020-02-02T00:17:05.925Z cpu4:2100126 opID=4a500647)Unmap6: 7133: [Unmap] 'datastore1':device(0x43078fc99b10)does not support unmap 2020-02-02T00:17:11.930Z cpu15:2271855)J6: 2651: 'datastore1': Exiting async journal replay manager world 2020-02-02T00:17:12.100Z cpu34:2271856)J6: 2651: 'SSD1': Exiting async journal replay manager world
Nothing is recorded in the logs. Can you help me?
Em qui., 26 de dez. de 2019 às 17:44, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 12/24/2019 06:40 AM, Gesiel Galvão Bernardes wrote: > In addition: I turned off one of the GWs, and with just one it
works
> fine. When the two go up, one of the images is changing the
"active /
> optimized" all time (where generates the logs above) and
everything is
> extremely slow.
Your multipathing in ESX is probably misconfigured and you have set
it
up for active active, or one host can't see all the iscsi paths
either
because it's not logged into all the sessions or because the network
is
not up on one of the paths.
> > I'm using: > tcmu-runner-1.4 > ceph-iscsi-3.3 > ceph 13.2.7 > > Regards, > Gesiel > > Em ter., 24 de dez. de 2019 às 09:09, Gesiel Galvão Bernardes > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>> escreveu: > > Hi, > > I am having an unusual slowdown using VMware with ISCSI gws. I have > two ISCSI gateways with two RBD images. I have checked the following > in the logs: > > Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 > [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: > Starting lock acquisition operation.2019-12-24 09:00:26.040 969 > [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1:
Lock
> acquisition operation is already in process.2019-12-24 09:00:26.973 > 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired > exclusive lock. > Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 > rbd/pool1.vmware_iscsi1: Acquired exclusive lock. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 > [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock > drop. Old state 1 > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner:
tcmu_notify_lock_lost:201
> rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: > alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock > acquisition operation. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 > [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: > Starting lock acquisition operation.2019-12-24 09:00:28.990 969 > [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive > lock. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 > rbd/pool1.vmware_iscsi1: Acquired exclusive lock. > > > Can anyone help-me please? > > Gesiel > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> >
Hi, Do you have any suggestions on where I can look? Regards, Gesiel Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes < gesiel.bernardes@gmail.com> escreveu:
Hi,
Just now was possible continue this. Below is the information required. Thanks advance,
Gesiel
Em seg., 20 de jan. de 2020 às 15:06, Mike Christie <mchristi@redhat.com> escreveu:
On 01/20/2020 10:29 AM, Gesiel Galvão Bernardes wrote:
Hi,
Only now have I been able to act on this problem. My environment is relatively simple: I have two ESXi 6.7 hosts, connected to two ISCSI gateways, using two RBD images.
When this mail started, the workaround was to keep only one ISCSI gateway connected, so it works normally. After the answer that received here that the problem could be in the configuration of VMWare, I reviewed the configuration of both (they are exactly according to Ceph documentation), and rebooted both.
Can you give me some basic info.
The output of: # "gwcli ls"
o- / ......................................................................................................... [...] o- cluster ......................................................................................... [Clusters: 1] | o- ceph ............................................................................................ [HEALTH_OK] | o- pools .......................................................................................... [Pools: 7] | | o- data_ecpool ................................. [(x3), Commit: 0.00Y/25836962M (0%), Used: 25011763551408b] | | o- pool1 ....................................... [(x2), Commit: 5.0T/38755444M (13%), Used: 16162480333070b] | | o- pool2 ....................................... [(x3), Commit: 6.0T/25836962M (24%), Used: 24261305669809b] | | o- pool3 ........................................ [(4+0), Commit: 0.00Y/58133164M (0%), Used: 719866819603b] | | o- pool_cache ...................................... [(x3), Commit: 0.00Y/706164M (0%), Used: 422549509342b] | | o- poolfs ................................................ [(x3), Commit: 0.00Y/25836962M (0%), Used: 0.00Y] | | o- rbd ................................................... [(x3), Commit: 0.00Y/25836962M (0%), Used: 5737b] | o- topology ............................................................................... [OSDs: 61,MONs: 2] o- disks ....................................................................................... [11.0T, Disks: 3] | o- pool1 ........................................................................................ [pool1 (5.0T)] | | o- vmware_iscsi1 ................................................................ [pool1/vmware_iscsi1 (5.0T)] | o- pool2 ........................................................................................ [pool2 (6.0T)] | o- iscsi-test ...................................................................... [pool2/iscsi-test (1.0T)] | o- vmware_iscsi2 ................................................................ [pool2/vmware_iscsi2 (5.0T)] o- iscsi-targets ............................................................... [DiscoveryAuth: None, Targets: 1] o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................... [Auth: None, Gateways: 2] o- disks .......................................................................................... [Disks: 3] | o- pool1/vmware_iscsi1 ................................................................ [Owner: ceph-iscsi2] | o- pool2/iscsi-test ................................................................... [Owner: ceph-iscsi2] | o- pool2/vmware_iscsi2 ................................................................ [Owner: ceph-iscsi1] o- gateways ............................................................................ [Up: 2/2, Portals: 2] | o- ceph-iscsi1 ........................................................................ [192.168.201.1 (UP)] | o- ceph-iscsi2 ........................................................................ [192.168.201.2 (UP)] o- host-groups .................................................................................. [Groups : 0] o- hosts ....................................................................... [Auth: ACL_ENABLED, Hosts: 2] o- iqn.1994-05.com.redhat:rh7-client .............................. [LOGGED-IN, Auth: CHAP, Disks: 2(10.0T)] | o- lun 0 ................................................. [pool1/vmware_iscsi1(5.0T), Owner: ceph-iscsi2] | o- lun 1 ................................................. [pool2/vmware_iscsi2(5.0T), Owner: ceph-iscsi1] o- iqn.1994-05.com.redhat:tcnvh8 .................................. [LOGGED-IN, Auth: CHAP, Disks: 2(10.0T)] o- lun 0 ................................................. [pool1/vmware_iscsi1(5.0T), Owner: ceph-iscsi2] o- lun 1 ................................................. [pool2/vmware_iscsi2(5.0T), Owner: ceph-iscsi1]
from one of the iscsi nodes, and give me the output of:
# targetcli ls
ceph-iscsi1 ~]# targetcli ls Warning: Could not load preferences file /root/.targetcli/prefs.bin. o- / ......................................................................................................... [...] o- backstores .............................................................................................. [...] | o- block .................................................................................. [Storage Objects: 0] | o- fileio ................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................ [Storage Objects: 0] | o- user:glfs .............................................................................. [Storage Objects: 0] | o- user:qcow .............................................................................. [Storage Objects: 0] | o- user:rbd ............................................................................... [Storage Objects: 3] | | o- pool1.vmware_iscsi1 ............................ [pool1/vmware_iscsi1;osd_op_timeout=30 (5.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.iscsi-test .................................. [pool2/iscsi-test;osd_op_timeout=30 (1.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.vmware_iscsi2 ............................ [pool2/vmware_iscsi2;osd_op_timeout=30 (5.0TiB) activated] | | o- alua ................................................................................... [ALUA Groups: 3] | | o- ano2 ............................................................... [ALUA state: Active/non-optimized] | | o- ao ..................................................................... [ALUA state: Active/optimized] | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | o- user:zbc ............................................................................... [Storage Objects: 0] o- iscsi ............................................................................................ [Targets: 1] | o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................................... [TPGs: 2] | o- tpg1 .......................................................................... [no-gen-acls, auth per-acl] | | o- acls .......................................................................................... [ACLs: 2] | | | o- iqn.1994-05.com.redhat:rh7-client ........................................ [1-way auth, Mapped LUNs: 2] | | | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | | | o- iqn.1994-05.com.redhat:tcnvh8 ............................................ [1-way auth, Mapped LUNs: 2] | | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | | o- luns .......................................................................................... [LUNs: 3] | | | o- lun0 ................................................................ [user/pool1.vmware_iscsi1 (ano1)] | | | o- lun1 ................................................................... [user/pool2.iscsi-test (ano1)] | | | o- lun2 .................................................................. [user/pool2.vmware_iscsi2 (ao)] | | o- portals .................................................................................... [Portals: 1] | | o- 192.168.201.1:3260 ............................................................................... [OK] | o- tpg2 ........................................................................................... [disabled] | o- acls .......................................................................................... [ACLs: 0] | o- luns .......................................................................................... [LUNs: 3] | | o- lun0 .................................................................. [user/pool1.vmware_iscsi1 (ao)] | | o- lun1 ..................................................................... [user/pool2.iscsi-test (ao)] | | o- lun2 ................................................................ [user/pool2.vmware_iscsi2 (ano2)] | o- portals .................................................................................... [Portals: 1] | o- 192.168.201.2:3260 ............................................................................... [OK] o- loopback ......................................................................................... [Targets: 0]
ceph-iscsi2 ~]# targetcli ls Warning: Could not load preferences file /root/.targetcli/prefs.bin. o- / ......................................................................................................... [...] o- backstores .............................................................................................. [...] | o- block .................................................................................. [Storage Objects: 0] | o- fileio ................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................ [Storage Objects: 0] | o- user:glfs .............................................................................. [Storage Objects: 0] | o- user:qcow .............................................................................. [Storage Objects: 0] | o- user:rbd ............................................................................... [Storage Objects: 3] | | o- pool1.vmware_iscsi1 ............................ [pool1/vmware_iscsi1;osd_op_timeout=30 (5.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.iscsi-test .................................. [pool2/iscsi-test;osd_op_timeout=30 (1.0TiB) activated] | | | o- alua ................................................................................... [ALUA Groups: 3] | | | o- ano1 ............................................................... [ALUA state: Active/non-optimized] | | | o- ao ..................................................................... [ALUA state: Active/optimized] | | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | | o- pool2.vmware_iscsi2 ............................ [pool2/vmware_iscsi2;osd_op_timeout=30 (5.0TiB) activated] | | o- alua ................................................................................... [ALUA Groups: 3] | | o- ano2 ............................................................... [ALUA state: Active/non-optimized] | | o- ao ..................................................................... [ALUA state: Active/optimized] | | o- default_tg_pt_gp ....................................................... [ALUA state: Active/optimized] | o- user:zbc ............................................................................... [Storage Objects: 0] o- iscsi ............................................................................................ [Targets: 1] | o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................................... [TPGs: 2] | o- tpg1 ........................................................................................... [disabled] | | o- acls .......................................................................................... [ACLs: 0] | | o- luns .......................................................................................... [LUNs: 3] | | | o- lun0 ................................................................ [user/pool1.vmware_iscsi1 (ano1)] | | | o- lun1 ................................................................... [user/pool2.iscsi-test (ano1)] | | | o- lun2 .................................................................. [user/pool2.vmware_iscsi2 (ao)] | | o- portals .................................................................................... [Portals: 1] | | o- 192.168.201.1:3260 ............................................................................... [OK] | o- tpg2 .......................................................................... [no-gen-acls, auth per-acl] | o- acls .......................................................................................... [ACLs: 2] | | o- iqn.1994-05.com.redhat:rh7-client ........................................ [1-way auth, Mapped LUNs: 2] | | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | | o- iqn.1994-05.com.redhat:tcnvh8 ............................................ [1-way auth, Mapped LUNs: 2] | | o- mapped_lun0 .................................................... [lun0 user/pool1.vmware_iscsi1 (rw)] | | o- mapped_lun1 .................................................... [lun2 user/pool2.vmware_iscsi2 (rw)] | o- luns .......................................................................................... [LUNs: 3] | | o- lun0 .................................................................. [user/pool1.vmware_iscsi1 (ao)] | | o- lun1 ..................................................................... [user/pool2.iscsi-test (ao)] | | o- lun2 ................................................................ [user/pool2.vmware_iscsi2 (ano2)] | o- portals .................................................................................... [Portals: 1] | o- 192.168.201.2:3260 ............................................................................... [OK] o- loopback ......................................................................................... [Targets: 0]
from both iscsi nodes.
The ceph-iscsi and tcmu-runner versions and did you build them yourself, get them from the ceph repos or get it from a distro repo.
I use in both gateways: ceph-iscsi-3.3-1.el7.noarch from ceph-iscsi repo ( http://download.ceph.com/ceph-iscsi/3/rpm/el7/noarch) tcmu-runner-1.4.0-0.1.51.geef5115.el7.x86_64 from https://3.chacra.ceph.com/r/tcmu-runner/master/eef511565078fb4e2ed52caaff16e...
It turns out, now the gateway "ceph-iscsi1" is working. When I turn on "ceph-iscsi2" it appears as "Active / Not optimized" for the two RBD images (before was an Active / Optimized for each image), and if I turn off "ceph-iscsi1" it (ceph-iscs2) remains as "Active / No optimized "and images are unavailable.
On the ESX side can you give me the output of:
esxcli storage nmp path list -d disk_id
esxcli storage nmp path list -d naa.6001405ba48e0b99e4c418ca13506c8e
iqn.1994-05.com.redhat:rh7-client-00023d000001,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,1-naa.6001405ba48e0b99e4c418ca13506c8e Runtime Name: vmhba68:C0:T0:L0 Device: naa.6001405ba48e0b99e4c418ca13506c8e Device Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Group State: active unoptimized Array Priority: 0 Storage Array Type Path Config: {TPG_id=1,TPG_state=ANO,RTP_id=1,RTP_health=UP} Path Selection Policy Path Config: {current path; rank: 0}
esxcli storage core device list -d disk_id
:~] esxcli storage core device list -d naa.6001405ba48e0b99e4c418ca13506c8e naa.6001405ba48e0b99e4c418ca13506c8e Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Has Settable Display Name: true Size: 5242880 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/naa.6001405ba48e0b99e4c418ca13506c8e Vendor: LIO-ORG Model: TCMU device Revision: 0002 SCSI Level: 5 Is Pseudo: false Status: degraded Is RDM Capable: true Is Local: false Is Removable: false Is SSD: false Is VVOL PE: false Is Offline: false Is Perennially Reserved: false Queue Full Sample Size: 0 Queue Full Threshold: 0 Thin Provisioning Status: yes Attached Filters: VAAI Status: supported Other UIDs: vml.02000000006001405ba48e0b99e4c418ca13506c8e54434d552064 Is Shared Clusterwide: true Is SAS: false Is USB: false Is Boot Device: false Device Max Queue Depth: 128 No of outstanding IOs with competing worlds: 32 Drive Type: unknown RAID Level: unknown Number of Physical Drives: unknown Protection Enabled: false PI Activated: false PI Type: 0 PI Protection Mask: NO PROTECTION Supported Guard Types: NO GUARD SUPPORT DIX Enabled: false DIX Guard Type: NO GUARD SUPPORT Emulated DIX/DIF Enabled: false
esxcli storage nmp device list -d disk_id
esxcli storage nmp device list -d naa.6001405ba48e0b99e4c418ca13506c8e naa.6001405ba48e0b99e4c418ca13506c8e Device Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} Path Selection Policy: VMW_PSP_MRU Path Selection Policy Device Config: Current Path=vmhba68:C0:T0:L0 Path Selection Policy Device Custom Config: Working Paths: vmhba68:C0:T0:L0 Is USB: false
esxcli storage nmp satp list
esxcli storage nmp satp list Name Default PSP Description
-------------------------------------------------------------------------------- VMW_SATP_ALUA VMW_PSP_MRU Supports non-specific arrays that use the ALUA protocol VMW_SATP_MSA VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_DEFAULT_AP VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_SVC VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_EQL VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_INV VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_EVA VMW_PSP_FIXED Placeholder (plugin not loaded) VMW_SATP_ALUA_CX VMW_PSP_RR Placeholder (plugin not loaded) VMW_SATP_SYMM VMW_PSP_RR Placeholder (plugin not loaded) VMW_SATP_CX VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_LSI VMW_PSP_MRU Placeholder (plugin not loaded) VMW_SATP_DEFAULT_AA VMW_PSP_FIXED Supports non-specific active/active arrays VMW_SATP_LOCAL VMW_PSP_FIXED Supports direct attached devices
and the /var/log/vmkernel.log for when you stop a node and the image goes to the unavailable state.
2020-02-02T00:14:59.094Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:14:59.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:14:59.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:01.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405cb71b882378c4138826c2da30" - issuing command 0x45a283482940 2020-02-02T00:15:01.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405cb71b882378c4138826c2da30" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:01.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405cb71b882378c4138826c2da30": awaiting fast path state update... 2020-02-02T00:15:06.979Z cpu25:2124996)ScsiDeviceIO: 3449: Cmd(0x459a9abd2380) 0x9e, CmdSN 0x1dc0cf from world 0 to dev "naa.6001405ba48e0b99e4c418ca13506c8e" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x46 0x80 0x41. 2020-02-02T00:15:06.979Z cpu25:2124996)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:07.094Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405ba48e0b99e4c418ca13506c8e" - no more commands to retry 2020-02-02T00:15:07.095Z cpu33:2098078)WARNING: NMP: nmp_IssueCommandToDevice:5726: I/O could not be issued to device "naa.6001405ba48e0b99e4c418ca13506c8e" due to Not found 2020-02-02T00:15:07.095Z cpu33:2098078)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 2020-02-02T00:15:07.095Z cpu33:2098078)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:08.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:08.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:08.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:08.154Z cpu25:2124996)ScsiDeviceIO: 3449: Cmd(0x45a283482940) 0x12, CmdSN 0x1dc10c from world 0 to dev "naa.6001405cb71b882378c4138826c2da30" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x46 0x80 0x41. 2020-02-02T00:15:08.154Z cpu25:2124996)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405cb71b882378c4138826c2da30" is blocked. Not starting I/O from device. 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:09.095Z cpu2:2098080)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405cb71b882378c4138826c2da30" - no more commands to retry 2020-02-02T00:15:14.094Z cpu22:2098078)NMP: nmp_ResetDeviceLogThrottling:3575: last error status from device naa.6001405cb71b882378c4138826c2da30 repeated 35 times 2020-02-02T00:15:14.095Z cpu22:2098078)NMP: nmp_ResetDeviceLogThrottling:3575: last error status from device naa.6001405ba48e0b99e4c418ca13506c8e repeated 8 times 2020-02-02T00:15:39.094Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:39.095Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:39.095Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:46.981Z cpu25:2124996)ScsiDeviceIO: 3449: Cmd(0x459a9abd2380) 0x25, CmdSN 0x1dc10d from world 0 to dev "naa.6001405ba48e0b99e4c418ca13506c8e" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. 2020-02-02T00:15:46.981Z cpu25:2124996)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:47.094Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405ba48e0b99e4c418ca13506c8e" - no more commands to retry 2020-02-02T00:15:47.095Z cpu22:2098078)WARNING: NMP: nmp_IssueCommandToDevice:5726: I/O could not be issued to device "naa.6001405ba48e0b99e4c418ca13506c8e" due to Not found 2020-02-02T00:15:47.095Z cpu22:2098078)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 2020-02-02T00:15:47.095Z cpu22:2098078)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405ba48e0b99e4c418ca13506c8e" is blocked. Not starting I/O from device. 2020-02-02T00:15:48.094Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9ba62b40 2020-02-02T00:15:48.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:48.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:49.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9ba62b40 2020-02-02T00:15:49.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - failed to issue command due to Not found (APD), try again... 2020-02-02T00:15:49.095Z cpu20:2098080)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update... 2020-02-02T00:15:52.967Z cpu0:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430e49d44a90 network resource pool netsched.pools.persist.iscsi associated 2020-02-02T00:15:52.967Z cpu0:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430e49d44a90 network tracker id 193945594 tracker.iSCSI.192.168.201.1 associated 2020-02-02T00:15:54.712Z cpu0:2098255)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba68:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE" 2020-02-02T00:15:54.712Z cpu0:2098255)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw TPGT: 1 TSIH: 0] 2020-02-02T00:15:54.712Z cpu0:2098255)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 192.168.201.107:35264 R: 192.168.201.1:3260] 2020-02-02T00:15:54.712Z cpu30:2097595)ScsiDevice: 4481: Handle REPORTED LUNS CHANGED DATA unit attention 2020-02-02T00:15:54.712Z cpu30:2097595)ScsiDevice: 4512: Handle INQUIRY PARAMETERS CHANGED unit attention 2020-02-02T00:15:54.716Z cpu2:2097312)ScsiDevice: 6001: Setting Device naa.6001405ba48e0b99e4c418ca13506c8e state back to 0x2 2020-02-02T00:15:54.716Z cpu2:2097312)ScsiDevice: 8708: No Handlers registered! (naa.6001405ba48e0b99e4c418ca13506c8e)! 2020-02-02T00:15:54.716Z cpu2:2097312)ScsiDevice: 6022: Device naa.6001405ba48e0b99e4c418ca13506c8e is Out of APD; token num:1 2020-02-02T00:15:54.716Z cpu2:2097312)StorageApdHandler: 1315: APD exit for 0x4305de04a9d0 [naa.6001405ba48e0b99e4c418ca13506c8e] 2020-02-02T00:15:54.716Z cpu0:2097602)StorageApdHandler: 507: APD exit event for 0x4305de04a9d0 [naa.6001405ba48e0b99e4c418ca13506c8e] 2020-02-02T00:15:54.716Z cpu0:2097602)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6001405ba48e0b99e4c418ca13506c8e] has exited the All Paths Down state. 2020-02-02T00:15:54.717Z cpu11:2097618)ScsiDevice: 6001: Setting Device naa.6001405cb71b882378c4138826c2da30 state back to 0x2 2020-02-02T00:15:54.717Z cpu11:2097618)ScsiDevice: 8708: No Handlers registered! (naa.6001405cb71b882378c4138826c2da30)! 2020-02-02T00:15:54.717Z cpu11:2097618)ScsiDevice: 6022: Device naa.6001405cb71b882378c4138826c2da30 is Out of APD; token num:1 2020-02-02T00:15:54.717Z cpu11:2097618)StorageApdHandler: 1315: APD exit for 0x4305de041660 [naa.6001405cb71b882378c4138826c2da30] 2020-02-02T00:15:54.717Z cpu0:2097602)StorageApdHandler: 507: APD exit event for 0x4305de041660 [naa.6001405cb71b882378c4138826c2da30] 2020-02-02T00:15:54.717Z cpu0:2097602)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6001405cb71b882378c4138826c2da30] has exited the All Paths Down state. 2020-02-02T00:15:55.096Z cpu0:2098243)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6001405ba48e0b99e4c418ca13506c8e" 2020-02-02T00:15:55.096Z cpu8:2099418 opID=17fe6cc5)World: 11943: VC opID sps-Main-52618-962-fe-1-62ed maps to vmkernel opID 17fe6cc5 2020-02-02T00:15:55.096Z cpu8:2099418 opID=17fe6cc5)WARNING: ScsiDeviceIO: 10750: READ CAPACITY on device "naa.6001405ba48e0b99e4c418ca13506c8e" from Plugin "NMP" failed. I/O error 2020-02-02T00:15:55.097Z cpu3:2098243)NMP: nmp_ThrottleLogForDevice:3788: Cmd 0x28 (0x459a9abd2380, 0) to dev "naa.6001405ba48e0b99e4c418ca13506c8e" on path "vmhba68:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. Act:FAILOVER 2020-02-02T00:15:55.097Z cpu3:2098243)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405ba48e0b99e4c418ca13506c8e": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. 2020-02-02T00:15:56.095Z cpu27:2098080)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405ba48e0b99e4c418ca13506c8e" - issuing command 0x459a9abd2380 2020-02-02T00:15:56.099Z cpu3:2098243)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6001405ba48e0b99e4c418ca13506c8e" 2020-02-02T00:16:02.095Z cpu23:2097316)qfle3: qfle3_queue_alloc_with_attr:517: [vmnic0] QueueOps.qfle3_queue_alloc_with_attr num_attr :1 attrs: 0x451a4521bcd0 2020-02-02T00:16:02.095Z cpu23:2097316)qfle3: qfle3_queue_alloc_with_attr:545: [vmnic0] Feature LRO requested. 2020-02-02T00:16:02.095Z cpu23:2097316)qfle3: qfle3_rq_alloc:282: [vmnic0] allocating RX queue at 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_alloc:316: [vmnic0] Marking RX queue 1 IN_USE 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_queue_start:1529: [vmnic0] QueueOps.queueStart 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_queue_start:1537: [vmnic0] RxQ, QueueIDVal:1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rx_queue_start:1452: [vmnic0] qfle3_rx_queue_start, QueueIDVal:1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1287: [vmnic0] qfle3_rq_start 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_init_eth_fp:68: [vmnic0] qfle3_init_eth_fp for fp 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1334: [vmnic0] RX fp[1]: wrote prods bd_prod=4078 cqe_prod=4030 sge_prod=1024 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1351: [vmnic0] enabled netpoll for q_index 1 2020-02-02T00:16:02.096Z cpu23:2097316)qfle3: qfle3_rq_start:1357: [vmnic0] Enabling interrupt on vector # 3 2020-02-02T00:16:02.097Z cpu23:2097316)qfle3: qfle3_rq_start:1377: [vmnic0] RX queue setup_queue successful for 1 2020-02-02T00:16:02.097Z cpu23:2097316)qfle3: qfle3_rq_start:1411: [vmnic0] active Rx queue Count 2 2020-02-02T00:16:02.097Z cpu23:2097316)qfle3: qfle3_rq_start:1412: [vmnic0] RX queue 1 successfully started 2020-02-02T00:16:02.098Z cpu23:2097316)qfle3: qfle3_queue_remove_filter:2063: [vmnic0] QueueOps.queueRemoveFilter 2020-02-02T00:16:02.099Z cpu23:2097316)qfle3: qfle3_remove_queue_filter:2012: [vmnic0] NetQ removed RX filter: queue:0 mac: 00:50:56:6f:59:4f filter id:3 2020-02-02T00:16:02.099Z cpu23:2097316)qfle3: qfle3_queue_apply_filter:1923: [vmnic0] QueueOps.queueApplyFilter 1 2020-02-02T00:16:02.101Z cpu23:2097316)qfle3: qfle3_apply_queue_mac_filter:1798: [vmnic0] NetQ set RX filter: queue:1 mac: 00:50:56:6f:59:4f filter id:0 2020-02-02T00:16:04.924Z cpu0:2099418 opID=cff7ff5)World: 11943: VC opID sps-Main-52618-962-d0-d6-6321 maps to vmkernel opID cff7ff5 2020-02-02T00:16:04.924Z cpu0:2099418 opID=cff7ff5)Unmap6: 7133: [Unmap] 'datastore1':device(0x43078fc9a1f0)does not support unmap 2020-02-02T00:16:07.164Z cpu9:2271849)J6: 2651: 'Storage_Ceph_pool1': Exiting async journal replay manager world 2020-02-02T00:16:09.665Z cpu12:2271850)J6: 2651: 'Storage_Ceph_pool2': Exiting async journal replay manager world 2020-02-02T00:16:10.930Z cpu4:2271852)J6: 2651: 'datastore1': Exiting async journal replay manager world 2020-02-02T00:16:11.094Z cpu22:2271853)J6: 2651: 'SSD1': Exiting async journal replay manager world 2020-02-02T00:16:22.094Z cpu26:2097316)qfle3: qfle3_queue_remove_filter:2063: [vmnic0] QueueOps.queueRemoveFilter 2020-02-02T00:16:22.097Z cpu26:2097316)qfle3: qfle3_remove_queue_filter:2012: [vmnic0] NetQ removed RX filter: queue:1 mac: 00:50:56:6f:59:4f filter id:0 2020-02-02T00:16:22.098Z cpu26:2097316)qfle3: qfle3_queue_apply_filter:1923: [vmnic0] QueueOps.queueApplyFilter 0 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_apply_queue_mac_filter:1798: [vmnic0] NetQ set RX filter: queue:0 mac: 00:50:56:6f:59:4f filter id:3 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_queue_quiesce:1061: [vmnic0] QueueOps.queueQuiesce 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_queue_quiesce:1069: [vmnic0] RxQ, QueueIDVal:1 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_rx_queue_stop:1558: [vmnic0] qfle3_rx_queue_stop, QueueIDVal:1 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_rq_stop:740: [vmnic0] qfle3_rq_stop 1 2020-02-02T00:16:22.099Z cpu26:2097316)qfle3: qfle3_rq_stop:811: [vmnic0] Stopping queue 0 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_stop:831: [vmnic0] disable netpoll for q_index 1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_stop:842: [vmnic0] Disabling interrupt on vector # 3 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_stop:867: [vmnic0] active Rx queue Count 1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_queue_free:690: [vmnic0] QueueOps.queueFree 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_queue_free:697: [vmnic0] RxQ, QueueIDVal:1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_rq_free:618: [vmnic0] Loop through 1 RSS queues 1 2020-02-02T00:16:22.102Z cpu26:2097316)qfle3: qfle3_cmd_remove_q:19191: [vmnic0] Releasing Q idx 1
2020-02-02T00:17:05.925Z cpu4:2100126 opID=4a500647)World: 11943: VC opID sps-Main-52618-962-3a-dd-633d maps to vmkernel opID 4a500647 2020-02-02T00:17:05.925Z cpu4:2100126 opID=4a500647)Unmap6: 7133: [Unmap] 'datastore1':device(0x43078fc99b10)does not support unmap 2020-02-02T00:17:11.930Z cpu15:2271855)J6: 2651: 'datastore1': Exiting async journal replay manager world 2020-02-02T00:17:12.100Z cpu34:2271856)J6: 2651: 'SSD1': Exiting async journal replay manager world
Nothing is recorded in the logs. Can you help me?
Em qui., 26 de dez. de 2019 às 17:44, Mike Christie <
mchristi@redhat.com
<mailto:mchristi@redhat.com>> escreveu:
On 12/24/2019 06:40 AM, Gesiel Galvão Bernardes wrote: > In addition: I turned off one of the GWs, and with just one it works > fine. When the two go up, one of the images is changing the "active / > optimized" all time (where generates the logs above) and everything is > extremely slow.
Your multipathing in ESX is probably misconfigured and you have set it up for active active, or one host can't see all the iscsi paths either because it's not logged into all the sessions or because the network is not up on one of the paths.
> > I'm using: > tcmu-runner-1.4 > ceph-iscsi-3.3 > ceph 13.2.7 > > Regards, > Gesiel > > Em ter., 24 de dez. de 2019 às 09:09, Gesiel Galvão Bernardes > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>> escreveu: > > Hi, > > I am having an unusual slowdown using VMware with ISCSI gws. I have > two ISCSI gateways with two RBD images. I have checked the following > in the logs: > > Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 > [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: > Starting lock acquisition operation.2019-12-24 09:00:26.040 969 > [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock > acquisition operation is already in process.2019-12-24 09:00:26.973 > 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired > exclusive lock. > Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 > rbd/pool1.vmware_iscsi1: Acquired exclusive lock. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 > [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock > drop. Old state 1 > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 > rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: > alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock > acquisition operation. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 > [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: > Starting lock acquisition operation.2019-12-24 09:00:28.990 969 > [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive > lock. > Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 > rbd/pool1.vmware_iscsi1: Acquired exclusive lock. > > > Can anyone help-me please? > > Gesiel > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> >
On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote:
Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>> escreveu:
Hi,
Just now was possible continue this. Below is the information required. Thanks advan
Hey, sorry for the late reply. I just back from PTO.
esxcli storage nmp device list -d naa.6001405ba48e0b99e4c418ca13506c8e naa.6001405ba48e0b99e4c418ca13506c8e Device Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} Path Selection Policy: VMW_PSP_MRU Path Selection Policy Device Config: Current Path=vmhba68:C0:T0:L0 Path Selection Policy Device Custom Config: Working Paths: vmhba68:C0:T0:L0 Is USB: false
........
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. Act:FAILOVER
Are you sure you are using tcmu-runner 1.4? Is that the actual daemon reversion running? Did you by any chance install the 1.4 rpm, but you/it did not restart the daemon? The error code above is returned in 1.3 and earlier. You are probably hitting a combo of 2 issues. We had only listed ESX 6.5 in the docs you probably saw, and in 6.7 the value of action_OnRetryErrors defaulted to on instead of off. You should set this back to off. You should also upgrade to the current version of tcmu-runner 1.5.x. It should fix the issue you are hitting, so non IO commands like inquiry, RTPG, etc are executed while failing over/back, so you would not hit the problem where path initialization and path testing IO is failed causing the path to marked as failed.
Hi, Em qui., 6 de fev. de 2020 às 18:56, Mike Christie <mchristi@redhat.com> escreveu:
On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote:
Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>> escreveu:
Hi,
Just now was possible continue this. Below is the information required. Thanks advan
Hey, sorry for the late reply. I just back from PTO.
esxcli storage nmp device list -d
naa.6001405ba48e0b99e4c418ca13506c8e
naa.6001405ba48e0b99e4c418ca13506c8e Device Display Name: LIO-ORG iSCSI Disk (naa.6001405ba48e0b99e4c418ca13506c8e) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} Path Selection Policy: VMW_PSP_MRU Path Selection Policy Device Config: Current Path=vmhba68:C0:T0:L0 Path Selection Policy Device Custom Config: Working Paths: vmhba68:C0:T0:L0 Is USB: false
........
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. Act:FAILOVER
Are you sure you are using tcmu-runner 1.4? Is that the actual daemon reversion running? Did you by any chance install the 1.4 rpm, but you/it did not restart the daemon? The error code above is returned in 1.3 and earlier.
You are probably hitting a combo of 2 issues.
We had only listed ESX 6.5 in the docs you probably saw, and in 6.7 the value of action_OnRetryErrors defaulted to on instead of off. You should set this back to off.
You should also upgrade to the current version of tcmu-runner 1.5.x. It should fix the issue you are hitting, so non IO commands like inquiry, RTPG, etc are executed while failing over/back, so you would not hit the problem where path initialization and path testing IO is failed causing the path to marked as failed.
I updated tcmu-runner to 1.5.2, and change action_OnRetryErrors to off, but the problem continue 😭
Attached is vmkernel.log. Regards Gesiel
On 02/08/2020 11:34 PM, Gesiel Galvão Bernardes wrote:
Hi,
Em qui., 6 de fev. de 2020 às 18:56, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote: > Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>> escreveu: > > Hi, > > Just now was possible continue this. Below is the information > required. Thanks advan
Hey, sorry for the late reply. I just back from PTO.
> > esxcli storage nmp device list -d naa.6001405ba48e0b99e4c418ca13506c8e > naa.6001405ba48e0b99e4c418ca13506c8e > Device Display Name: LIO-ORG iSCSI Disk > (naa.6001405ba48e0b99e4c418ca13506c8e) > Storage Array Type: VMW_SATP_ALUA > Storage Array Type Device Config: {implicit_support=on; > explicit_support=off; explicit_allow=on; alua_followover=on; > action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} > Path Selection Policy: VMW_PSP_MRU > Path Selection Policy Device Config: Current Path=vmhba68:C0:T0:L0 > Path Selection Policy Device Custom Config: > Working Paths: vmhba68:C0:T0:L0 > Is USB: false
........
> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. Act:FAILOVER
Are you sure you are using tcmu-runner 1.4? Is that the actual daemon reversion running? Did you by any chance install the 1.4 rpm, but you/it did not restart the daemon? The error code above is returned in 1.3 and earlier.
You are probably hitting a combo of 2 issues.
We had only listed ESX 6.5 in the docs you probably saw, and in 6.7 the value of action_OnRetryErrors defaulted to on instead of off. You should set this back to off.
You should also upgrade to the current version of tcmu-runner 1.5.x. It should fix the issue you are hitting, so non IO commands like inquiry, RTPG, etc are executed while failing over/back, so you would not hit the problem where path initialization and path testing IO is failed causing the path to marked as failed.
I updated tcmu-runner to 1.5.2, and change action_OnRetryErrors to off, but the problem continue 😭
Attached is vmkernel.log.
When you stopped the iscsi gw at around 2020-02-09T01:51:25.820Z, how many paths did your device have? Did: esxcli storage nmp path list -d your_device report only one path? Did esxcli iscsi session connection list show a iscsi connection to each gw? The logs look like when you brought the gw down, we lost the only path we had. We then went into all paths down, so IO could not execute. It looks like the gw was brought back up at the end of the log and the path seem to have got added back.
Hi Em dom., 9 de fev. de 2020 às 18:27, Mike Christie <mchristi@redhat.com> escreveu:
Hi,
Em qui., 6 de fev. de 2020 às 18:56, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote: > Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>> escreveu: > > Hi, > > Just now was possible continue this. Below is the information > required. Thanks advan
Hey, sorry for the late reply. I just back from PTO.
> > esxcli storage nmp device list -d naa.6001405ba48e0b99e4c418ca13506c8e > naa.6001405ba48e0b99e4c418ca13506c8e > Device Display Name: LIO-ORG iSCSI Disk > (naa.6001405ba48e0b99e4c418ca13506c8e) > Storage Array Type: VMW_SATP_ALUA > Storage Array Type Device Config: {implicit_support=on; > explicit_support=off; explicit_allow=on; alua_followover=on; > action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} > Path Selection Policy: VMW_PSP_MRU > Path Selection Policy Device Config: Current Path=vmhba68:C0:T0:L0 > Path Selection Policy Device Custom Config: > Working Paths: vmhba68:C0:T0:L0 > Is USB: false
........
> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. Act:FAILOVER
Are you sure you are using tcmu-runner 1.4? Is that the actual daemon reversion running? Did you by any chance install the 1.4 rpm, but you/it did not restart the daemon? The error code above is returned in 1.3 and earlier.
You are probably hitting a combo of 2 issues.
We had only listed ESX 6.5 in the docs you probably saw, and in 6.7
On 02/08/2020 11:34 PM, Gesiel Galvão Bernardes wrote: the
value of action_OnRetryErrors defaulted to on instead of off. You
should
set this back to off.
You should also upgrade to the current version of tcmu-runner 1.5.x.
It
should fix the issue you are hitting, so non IO commands like
inquiry,
RTPG, etc are executed while failing over/back, so you would not hit
the
problem where path initialization and path testing IO is failed
causing
the path to marked as failed.
I updated tcmu-runner to 1.5.2, and change action_OnRetryErrors to off, but the problem continue 😭
Attached is vmkernel.log.
When you stopped the iscsi gw at around 2020-02-09T01:51:25.820Z, how many paths did your device have? Did:
esxcli storage nmp path list -d your_device
report only one path? Did
esxcli iscsi session connection list
show a iscsi connection to each gw?
Hmmm, I believe the problem may be here. I verified that I was listing only one GW for each path. So I ran a "rescan HBA" on VMware on both ESX, now one of them lists the 3 (I added one more) gateways, but an ESX host with the same configuration continues to list only one gateway. See the different outputs:
[root@tcnvh7:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.107 SessionCreateTime: 01/19/20 00:11:25 ConnectionCreateTime: 01/19/20 00:11:25 ConnectionStartTime: 02/13/20 23:03:10 State: logged_in vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000002,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000002 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.2 RemoteAddress: 192.168.201.2 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000003,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000003 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.3 RemoteAddress: 192.168.201.3 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in ===== [root@tcnvh8:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.108 SessionCreateTime: 01/12/20 02:53:53 ConnectionCreateTime: 01/12/20 02:53:53 ConnectionStartTime: 02/13/20 23:06:40 State: logged_in Is that the problem? Any ideas on how to proceed from here? The logs look like when you brought the gw down, we lost the only path
we had. We then went into all paths down, so IO could not execute. It looks like the gw was brought back up at the end of the log and the path seem to have got added back.
On 02/13/2020 08:52 PM, Gesiel Galvão Bernardes wrote:
Hi
Em dom., 9 de fev. de 2020 às 18:27, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 02/08/2020 11:34 PM, Gesiel Galvão Bernardes wrote: > Hi, > > Em qui., 6 de fev. de 2020 às 18:56, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com> > <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>> escreveu: > > On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote: > > Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes > > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>>> escreveu: > > > > Hi, > > > > Just now was possible continue this. Below is the information > > required. Thanks advan > > > Hey, sorry for the late reply. I just back from PTO. > > > > > esxcli storage nmp device list -d > naa.6001405ba48e0b99e4c418ca13506c8e > > naa.6001405ba48e0b99e4c418ca13506c8e > > Device Display Name: LIO-ORG iSCSI Disk > > (naa.6001405ba48e0b99e4c418ca13506c8e) > > Storage Array Type: VMW_SATP_ALUA > > Storage Array Type Device Config: {implicit_support=on; > > explicit_support=off; explicit_allow=on; alua_followover=on; > > action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} > > Path Selection Policy: VMW_PSP_MRU > > Path Selection Policy Device Config: Current > Path=vmhba68:C0:T0:L0 > > Path Selection Policy Device Custom Config: > > Working Paths: vmhba68:C0:T0:L0 > > Is USB: false > > ........ > > > Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. > Act:FAILOVER > > > Are you sure you are using tcmu-runner 1.4? Is that the actual daemon > reversion running? Did you by any chance install the 1.4 rpm, but you/it > did not restart the daemon? The error code above is returned in 1.3 and > earlier. > > You are probably hitting a combo of 2 issues. > > We had only listed ESX 6.5 in the docs you probably saw, and in 6.7 the > value of action_OnRetryErrors defaulted to on instead of off. You should > set this back to off. > > You should also upgrade to the current version of tcmu-runner 1.5.x. It > should fix the issue you are hitting, so non IO commands like inquiry, > RTPG, etc are executed while failing over/back, so you would not hit the > problem where path initialization and path testing IO is failed causing > the path to marked as failed. > > > I updated tcmu-runner to 1.5.2, and change action_OnRetryErrors to off, > but the problem continue 😭 > > Attached is vmkernel.log. >
When you stopped the iscsi gw at around 2020-02-09T01:51:25.820Z, how many paths did your device have? Did:
esxcli storage nmp path list -d your_device
report only one path? Did
esxcli iscsi session connection list
show a iscsi connection to each gw?
Hmmm, I believe the problem may be here. I verified that I was listing only one GW for each path. So I ran a "rescan HBA" on VMware on both ESX, now one of them lists the 3 (I added one more) gateways, but an ESX host with the same configuration continues to list only one gateway. See the different outputs:
[root@tcnvh7:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.107 SessionCreateTime: 01/19/20 00:11:25 ConnectionCreateTime: 01/19/20 00:11:25 ConnectionStartTime: 02/13/20 23:03:10 State: logged_in
vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000002,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000002 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.2 RemoteAddress: 192.168.201.2 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in
vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000003,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000003 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.3 RemoteAddress: 192.168.201.3 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in
===== [root@tcnvh8:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.108 SessionCreateTime: 01/12/20 02:53:53 ConnectionCreateTime: 01/12/20 02:53:53 ConnectionStartTime: 02/13/20 23:06:40 State: logged_in
Is that the problem? Any ideas on how to proceed from here?
Yes. Normally, you would have the connection already created, and when one path/gateway goes down, then the multipath layer will switch to another path. When the path/gateway comes back up, the initiator side's iscsi layer will reconnect automatically and the multipath layer will re-setup the path structure, so it can failback if its a higher priority path or failover later if other paths go down. Something happened with the automatic path connection process on that node. We know it works for that one gateway you brought up/down. For the other gateways I would check: 1. Check that all target portals are being discovered. In the GUI screen you entered in the discovery address, you should also see a list of all target portals that were found in the static section. Do you only see 1 portal? See here: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc... 2. If you see all the portals then when you hit the rescan HBA button, do you see any errors on the target side in /var/log/messages? Maybe something about CHAP/login/auth errors? What about in the /var/log/vmkernel.log on the initiator side? Any iscsi errors?
On 02/14/2020 10:25 AM, Mike Christie wrote:
On 02/13/2020 08:52 PM, Gesiel Galvão Bernardes wrote:
Hi
Em dom., 9 de fev. de 2020 às 18:27, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 02/08/2020 11:34 PM, Gesiel Galvão Bernardes wrote: > Hi, > > Em qui., 6 de fev. de 2020 às 18:56, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com> > <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>> escreveu: > > On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote: > > Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes > > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>>> escreveu: > > > > Hi, > > > > Just now was possible continue this. Below is the information > > required. Thanks advan > > > Hey, sorry for the late reply. I just back from PTO. > > > > > esxcli storage nmp device list -d > naa.6001405ba48e0b99e4c418ca13506c8e > > naa.6001405ba48e0b99e4c418ca13506c8e > > Device Display Name: LIO-ORG iSCSI Disk > > (naa.6001405ba48e0b99e4c418ca13506c8e) > > Storage Array Type: VMW_SATP_ALUA > > Storage Array Type Device Config: {implicit_support=on; > > explicit_support=off; explicit_allow=on; alua_followover=on; > > action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} > > Path Selection Policy: VMW_PSP_MRU > > Path Selection Policy Device Config: Current > Path=vmhba68:C0:T0:L0 > > Path Selection Policy Device Custom Config: > > Working Paths: vmhba68:C0:T0:L0 > > Is USB: false > > ........ > > > Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. > Act:FAILOVER > > > Are you sure you are using tcmu-runner 1.4? Is that the actual daemon > reversion running? Did you by any chance install the 1.4 rpm, but you/it > did not restart the daemon? The error code above is returned in 1.3 and > earlier. > > You are probably hitting a combo of 2 issues. > > We had only listed ESX 6.5 in the docs you probably saw, and in 6.7 the > value of action_OnRetryErrors defaulted to on instead of off. You should > set this back to off. > > You should also upgrade to the current version of tcmu-runner 1.5.x. It > should fix the issue you are hitting, so non IO commands like inquiry, > RTPG, etc are executed while failing over/back, so you would not hit the > problem where path initialization and path testing IO is failed causing > the path to marked as failed. > > > I updated tcmu-runner to 1.5.2, and change action_OnRetryErrors to off, > but the problem continue 😭 > > Attached is vmkernel.log. >
When you stopped the iscsi gw at around 2020-02-09T01:51:25.820Z, how many paths did your device have? Did:
esxcli storage nmp path list -d your_device
report only one path? Did
esxcli iscsi session connection list
show a iscsi connection to each gw?
Hmmm, I believe the problem may be here. I verified that I was listing only one GW for each path. So I ran a "rescan HBA" on VMware on both ESX, now one of them lists the 3 (I added one more) gateways, but an ESX host with the same configuration continues to list only one gateway. See the different outputs:
[root@tcnvh7:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.107 SessionCreateTime: 01/19/20 00:11:25 ConnectionCreateTime: 01/19/20 00:11:25 ConnectionStartTime: 02/13/20 23:03:10 State: logged_in
vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000002,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000002 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.2 RemoteAddress: 192.168.201.2 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in
vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000003,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000003 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.3 RemoteAddress: 192.168.201.3 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in
===== [root@tcnvh8:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.108 SessionCreateTime: 01/12/20 02:53:53 ConnectionCreateTime: 01/12/20 02:53:53 ConnectionStartTime: 02/13/20 23:06:40 State: logged_in
Is that the problem? Any ideas on how to proceed from here?
Yes. Normally, you would have the connection already created, and when one path/gateway goes down, then the multipath layer will switch to another path. When the path/gateway comes back up, the initiator side's iscsi layer will reconnect automatically and the multipath layer will re-setup the path structure, so it can failback if its a higher priority path or failover later if other paths go down.
Something happened with the automatic path connection process on that node. We know it works for that one gateway you brought up/down. For the other gateways I would check:
1. Check that all target portals are being discovered. In the GUI screen you entered in the discovery address, you should also see a list of all target portals that were found in the static section. Do you only see 1 portal?
See here:
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc...
Oh yeah, make sure you check the basics. If after a rescan you are seeing only the one portal at 192.168.201.1, then make sure from tcnvh8 you can ping the other addresses 192.168.201.3 and 192.168.201.2.
2. If you see all the portals then when you hit the rescan HBA button, do you see any errors on the target side in /var/log/messages? Maybe something about CHAP/login/auth errors?
What about in the /var/log/vmkernel.log on the initiator side? Any iscsi errors?
On 02/13/2020 08:52 PM, Gesiel Galvão Bernardes wrote:
Hi
Em dom., 9 de fev. de 2020 às 18:27, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 02/08/2020 11:34 PM, Gesiel Galvão Bernardes wrote: > Hi, > > Em qui., 6 de fev. de 2020 às 18:56, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com> > <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>> escreveu: > > On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote: > > Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes > > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto: gesiel.bernardes@gmail.com>> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>>> escreveu: > > > > Hi, > > > > Just now was possible continue this. Below is the information > > required. Thanks advan > > > Hey, sorry for the late reply. I just back from PTO. > > > > > esxcli storage nmp device list -d > naa.6001405ba48e0b99e4c418ca13506c8e > > naa.6001405ba48e0b99e4c418ca13506c8e > > Device Display Name: LIO-ORG iSCSI Disk > > (naa.6001405ba48e0b99e4c418ca13506c8e) > > Storage Array Type: VMW_SATP_ALUA > > Storage Array Type Device Config: {implicit_support=on; > > explicit_support=off; explicit_allow=on; alua_followover=on; > > action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} > > Path Selection Policy: VMW_PSP_MRU > > Path Selection Policy Device Config: Current > Path=vmhba68:C0:T0:L0 > > Path Selection Policy Device Custom Config: > > Working Paths: vmhba68:C0:T0:L0 > > Is USB: false > > ........ > > > Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. > Act:FAILOVER > > > Are you sure you are using tcmu-runner 1.4? Is that the actual daemon > reversion running? Did you by any chance install the 1.4 rpm, but you/it > did not restart the daemon? The error code above is returned in 1.3 and > earlier. > > You are probably hitting a combo of 2 issues. > > We had only listed ESX 6.5 in the docs you probably saw, and in 6.7 the > value of action_OnRetryErrors defaulted to on instead of off. You should > set this back to off. > > You should also upgrade to the current version of tcmu-runner 1.5.x. It > should fix the issue you are hitting, so non IO commands like inquiry, > RTPG, etc are executed while failing over/back, so you would not hit the > problem where path initialization and path testing IO is failed causing > the path to marked as failed. > > > I updated tcmu-runner to 1.5.2, and change action_OnRetryErrors to off, > but the problem continue 😭 > > Attached is vmkernel.log. >
When you stopped the iscsi gw at around 2020-02-09T01:51:25.820Z, how many paths did your device have? Did:
esxcli storage nmp path list -d your_device
report only one path? Did
esxcli iscsi session connection list
show a iscsi connection to each gw?
Hmmm, I believe the problem may be here. I verified that I was listing only one GW for each path. So I ran a "rescan HBA" on VMware on both ESX, now one of them lists the 3 (I added one more) gateways, but an ESX host with the same configuration continues to list only one gateway. See the different outputs:
[root@tcnvh7:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.107 SessionCreateTime: 01/19/20 00:11:25 ConnectionCreateTime: 01/19/20 00:11:25 ConnectionStartTime: 02/13/20 23:03:10 State: logged_in
vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000002,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000002 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.2 RemoteAddress: 192.168.201.2 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in
vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000003,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000003 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.3 RemoteAddress: 192.168.201.3 LocalAddress: 192.168.201.107 SessionCreateTime: 02/13/20 23:09:16 ConnectionCreateTime: 02/13/20 23:09:16 ConnectionStartTime: 02/13/20 23:09:16 State: logged_in
===== [root@tcnvh8:~] esxcli iscsi session connection list vmhba68,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0 Adapter: vmhba68 Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ISID: 00023d000001 CID: 0 DataDigest: NONE HeaderDigest: NONE IFMarker: false IFMarkerInterval: 0 MaxRecvDataSegmentLength: 131072 MaxTransmitDataSegmentLength: 262144 OFMarker: false OFMarkerInterval: 0 ConnectionAddress: 192.168.201.1 RemoteAddress: 192.168.201.1 LocalAddress: 192.168.201.108 SessionCreateTime: 01/12/20 02:53:53 ConnectionCreateTime: 01/12/20 02:53:53 ConnectionStartTime: 02/13/20 23:06:40 State: logged_in
Is that the problem? Any ideas on how to proceed from here?
Yes. Normally, you would have the connection already created, and when one path/gateway goes down, then the multipath layer will switch to another path. When the path/gateway comes back up, the initiator side's iscsi layer will reconnect automatically and the multipath layer will re-setup the path structure, so it can failback if its a higher priority path or failover later if other paths go down.
Something happened with the automatic path connection process on that node. We know it works for that one gateway you brought up/down. For the other gateways I would check:
1. Check that all target portals are being discovered. In the GUI screen you entered in the discovery address, you should also see a list of all target portals that were found in the static section. Do you only see 1 portal?
Yes, all targers are listen in "Targets" section. But in "Paths", just
Em sex., 14 de fev. de 2020 às 13:25, Mike Christie <mchristi@redhat.com> escreveu: paths of host 192.168.201.1 are showing.
See here:
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc...
2. If you see all the portals then when you hit the rescan HBA button, do you see any errors on the target side in /var/log/messages? Maybe something about CHAP/login/auth errors?
When I hit rescan HBA, I received this messages in Logs (below). He seems to complain about the username / password, but if that is the case, the password is the same for all gateways, why does one connect and another does not? ceph-iscsi2 - /var/log/messages Feb 16 07:31:21 ceph-iscsi2 kernel: Security negotiation failed. Feb 16 07:31:21 ceph-iscsi2 kernel: iSCSI Login negotiation failed. tcnvh8 - /var/log/vmkernel.log 2020-02-16T07:25:44.718Z cpu13:2097316)qfle3: qfle3_queue_remove_filter:2063: [vmnic0] QueueOps.queueRemoveFilter 2020-02-16T07:25:44.720Z cpu13:2097316)qfle3: qfle3_remove_queue_filter:2012: [vmnic0] NetQ removed RX filter: queue:1 mac: 00:50:56:6c:24:9d filter id:0 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_queue_apply_filter:1923: [vmnic0] QueueOps.queueApplyFilter 0 2020-02-16T07:25:44.721Z cpu13:2097316)WARNING: qfle3: qfle3_apply_queue_mac_filter:1734: [vmnic0] NetQ could not add RX filter, no filters for queue 0 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_queue_quiesce:1061: [vmnic0] QueueOps.queueQuiesce 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_queue_quiesce:1069: [vmnic0] RxQ, QueueIDVal:1 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_rx_queue_stop:1558: [vmnic0] qfle3_rx_queue_stop, QueueIDVal:1 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_rq_stop:740: [vmnic0] qfle3_rq_stop 1 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_rq_stop:811: [vmnic0] Stopping queue 0 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_stop:831: [vmnic0] disable netpoll for q_index 1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_stop:842: [vmnic0] Disabling interrupt on vector # 3 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_stop:867: [vmnic0] active Rx queue Count 1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_queue_free:690: [vmnic0] QueueOps.queueFree 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_queue_free:697: [vmnic0] RxQ, QueueIDVal:1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_free:618: [vmnic0] Loop through 1 RSS queues 1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_cmd_remove_q:19191: [vmnic0] Releasing Q idx 1 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd072fc00 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd072fc00 network tracker id 974086138 tracker.iSCSI.192.168.201.2 associated 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd097ae20 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd097ae20 network tracker id 974086138 tracker.iSCSI.192.168.201.3 associated 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:1 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:39679 R: 192.168.201.2:3260] 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:1 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:39679 R: 192.168.201.2:3260] 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:2 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:45598 R: 192.168.201.3:3260] 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:2 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:50.342Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.342Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:45598 R: 192.168.201.3:3260] 2020-02-16T07:25:50.595Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:50.595Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.595Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:39679 R: 192.168.201.2:3260] 2020-02-16T07:25:50.600Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:50.600Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.600Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:45598 R: 192.168.201.3:3260] 2020-02-16T07:25:51.204Z cpu0:2837132)J6: 2651: 'Storage_Ceph_pool1': Exiting async journal replay manager world 2020-02-16T07:25:55.360Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd06ef2a0 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:55.360Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd06ef2a0 network tracker id 974086138 tracker.iSCSI.192.168.201.2 associated 2020-02-16T07:25:55.361Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd0938540 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:55.361Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd0938540 network tracker id 974086138 tracker.iSCSI.192.168.201.3 associated 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:1 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:17752 R: 192.168.201.2:3260] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:1 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:17752 R: 192.168.201.2:3260] 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:2 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:15616 R: 192.168.201.3:3260] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:2 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:15616 R: 192.168.201.3:3260] 2020-02-16T07:25:56.121Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:56.121Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:56.121Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:17752 R: 192.168.201.2:3260] 2020-02-16T07:25:56.127Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:56.127Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:56.127Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:15616 R: 192.168.201.3:3260] 2020-02-16T07:30:49.971Z cpu7:2098243)NMP: nmp_ThrottleLogForDevice:3788: Cmd 0x28 (0x45a296a54040, 0) to dev "naa.6001405ba48e0b99e4c418ca13506c8e" on path "vmhba68:C0:T0:L0" Failed: H:0x0 D:0x8 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:NONE
What about in the /var/log/vmkernel.log on the initiator side? Any iscsi errors?
On 02/16/2020 04:51 AM, Gesiel Galvão Bernardes wrote:
Em sex., 14 de fev. de 2020 às 13:25, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com>> escreveu:
On 02/13/2020 08:52 PM, Gesiel Galvão Bernardes wrote: > Hi > > Em dom., 9 de fev. de 2020 às 18:27, Mike Christie <mchristi@redhat.com <mailto:mchristi@redhat.com> > <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>> escreveu: > > On 02/08/2020 11:34 PM, Gesiel Galvão Bernardes wrote: > > Hi, > > > > Em qui., 6 de fev. de 2020 às 18:56, Mike Christie > <mchristi@redhat.com <mailto:mchristi@redhat.com> <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>> > > <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com> <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>>> escreveu: > > > > On 02/05/2020 07:03 AM, Gesiel Galvão Bernardes wrote: > > > Em dom., 2 de fev. de 2020 às 00:37, Gesiel Galvão Bernardes > > > <gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>> > > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>> > > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com> > <mailto:gesiel.bernardes@gmail.com <mailto:gesiel.bernardes@gmail.com>>>>> escreveu: > > > > > > Hi, > > > > > > Just now was possible continue this. Below is the > information > > > required. Thanks advan > > > > > > Hey, sorry for the late reply. I just back from PTO. > > > > > > > > esxcli storage nmp device list -d > > naa.6001405ba48e0b99e4c418ca13506c8e > > > naa.6001405ba48e0b99e4c418ca13506c8e > > > Device Display Name: LIO-ORG iSCSI Disk > > > (naa.6001405ba48e0b99e4c418ca13506c8e) > > > Storage Array Type: VMW_SATP_ALUA > > > Storage Array Type Device Config: {implicit_support=on; > > > explicit_support=off; explicit_allow=on; alua_followover=on; > > > action_OnRetryErrors=on; {TPG_id=1,TPG_state=ANO}} > > > Path Selection Policy: VMW_PSP_MRU > > > Path Selection Policy Device Config: Current > > Path=vmhba68:C0:T0:L0 > > > Path Selection Policy Device Custom Config: > > > Working Paths: vmhba68:C0:T0:L0 > > > Is USB: false > > > > ........ > > > > > Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0xa. > > Act:FAILOVER > > > > > > Are you sure you are using tcmu-runner 1.4? Is that the actual > daemon > > reversion running? Did you by any chance install the 1.4 rpm, > but you/it > > did not restart the daemon? The error code above is returned > in 1.3 and > > earlier. > > > > You are probably hitting a combo of 2 issues. > > > > We had only listed ESX 6.5 in the docs you probably saw, and > in 6.7 the > > value of action_OnRetryErrors defaulted to on instead of off. > You should > > set this back to off. > > > > You should also upgrade to the current version of tcmu-runner > 1.5.x. It > > should fix the issue you are hitting, so non IO commands like > inquiry, > > RTPG, etc are executed while failing over/back, so you would > not hit the > > problem where path initialization and path testing IO is > failed causing > > the path to marked as failed. > > > > > > I updated tcmu-runner to 1.5.2, and change action_OnRetryErrors to > off, > > but the problem continue 😭 > > > > Attached is vmkernel.log. > > > > > When you stopped the iscsi gw at around 2020-02-09T01:51:25.820Z, how > many paths did your device have? Did: > > esxcli storage nmp path list -d your_device > > report only one path? Did > > esxcli iscsi session connection list > > show a iscsi connection to each gw? > > Hmmm, I believe the problem may be here. I verified that I was listing > only one GW for each path. So I ran a "rescan HBA" on VMware on both > ESX, now one of them lists the 3 (I added one more) gateways, but an ESX > host with the same configuration continues to list only one gateway. See > the different outputs: > > [root@tcnvh7:~] esxcli iscsi session connection list > vmhba68,iqn.2003-01.com <http://iqn.2003-01.com>.redhat.iscsi-gw:iscsi-igw,00023d000001,0 > Adapter: vmhba68 > Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw > ISID: 00023d000001 > CID: 0 > DataDigest: NONE > HeaderDigest: NONE > IFMarker: false > IFMarkerInterval: 0 > MaxRecvDataSegmentLength: 131072 > MaxTransmitDataSegmentLength: 262144 > OFMarker: false > OFMarkerInterval: 0 > ConnectionAddress: 192.168.201.1 > RemoteAddress: 192.168.201.1 > LocalAddress: 192.168.201.107 > SessionCreateTime: 01/19/20 00:11:25 > ConnectionCreateTime: 01/19/20 00:11:25 > ConnectionStartTime: 02/13/20 23:03:10 > State: logged_in > > vmhba68,iqn.2003-01.com <http://iqn.2003-01.com>.redhat.iscsi-gw:iscsi-igw,00023d000002,0 > Adapter: vmhba68 > Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw > ISID: 00023d000002 > CID: 0 > DataDigest: NONE > HeaderDigest: NONE > IFMarker: false > IFMarkerInterval: 0 > MaxRecvDataSegmentLength: 131072 > MaxTransmitDataSegmentLength: 262144 > OFMarker: false > OFMarkerInterval: 0 > ConnectionAddress: 192.168.201.2 > RemoteAddress: 192.168.201.2 > LocalAddress: 192.168.201.107 > SessionCreateTime: 02/13/20 23:09:16 > ConnectionCreateTime: 02/13/20 23:09:16 > ConnectionStartTime: 02/13/20 23:09:16 > State: logged_in > > vmhba68,iqn.2003-01.com <http://iqn.2003-01.com>.redhat.iscsi-gw:iscsi-igw,00023d000003,0 > Adapter: vmhba68 > Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw > ISID: 00023d000003 > CID: 0 > DataDigest: NONE > HeaderDigest: NONE > IFMarker: false > IFMarkerInterval: 0 > MaxRecvDataSegmentLength: 131072 > MaxTransmitDataSegmentLength: 262144 > OFMarker: false > OFMarkerInterval: 0 > ConnectionAddress: 192.168.201.3 > RemoteAddress: 192.168.201.3 > LocalAddress: 192.168.201.107 > SessionCreateTime: 02/13/20 23:09:16 > ConnectionCreateTime: 02/13/20 23:09:16 > ConnectionStartTime: 02/13/20 23:09:16 > State: logged_in > > ===== > [root@tcnvh8:~] esxcli iscsi session connection list > vmhba68,iqn.2003-01.com <http://iqn.2003-01.com>.redhat.iscsi-gw:iscsi-igw,00023d000001,0 > Adapter: vmhba68 > Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw > ISID: 00023d000001 > CID: 0 > DataDigest: NONE > HeaderDigest: NONE > IFMarker: false > IFMarkerInterval: 0 > MaxRecvDataSegmentLength: 131072 > MaxTransmitDataSegmentLength: 262144 > OFMarker: false > OFMarkerInterval: 0 > ConnectionAddress: 192.168.201.1 > RemoteAddress: 192.168.201.1 > LocalAddress: 192.168.201.108 > SessionCreateTime: 01/12/20 02:53:53 > ConnectionCreateTime: 01/12/20 02:53:53 > ConnectionStartTime: 02/13/20 23:06:40 > State: logged_in > > Is that the problem? Any ideas on how to proceed from here? >
Yes. Normally, you would have the connection already created, and when one path/gateway goes down, then the multipath layer will switch to another path. When the path/gateway comes back up, the initiator side's iscsi layer will reconnect automatically and the multipath layer will re-setup the path structure, so it can failback if its a higher priority path or failover later if other paths go down.
Something happened with the automatic path connection process on that node. We know it works for that one gateway you brought up/down. For the other gateways I would check:
1. Check that all target portals are being discovered. In the GUI screen you entered in the discovery address, you should also see a list of all target portals that were found in the static section. Do you only see 1 portal?
Yes, all targers are listen in "Targets" section. But in "Paths", just paths of host 192.168.201.1 are showing.
See here:
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc...
2. If you see all the portals then when you hit the rescan HBA button, do you see any errors on the target side in /var/log/messages? Maybe something about CHAP/login/auth errors?
When I hit rescan HBA, I received this messages in Logs (below). He seems to complain about the username / password, but if that is the case, the password is the same for all gateways, why does one connect and another does not?
Did you change your config, but then maybe not re-update everything? It looks like some config data is stale. For example, in your gwcli/targetcli output there are only 2 iscsi gws at .1 and .2, but in the vmware log we also see it trying to log into something at 192.168.201.3. Did you maybe set up a portal/portals as static, then not modify the settings used, or delete the static ones when rediscovering the new target config? I would reset the target, portal and chap settings on the vmware side.
ceph-iscsi2 - /var/log/messages Feb 16 07:31:21 ceph-iscsi2 kernel: Security negotiation failed. Feb 16 07:31:21 ceph-iscsi2 kernel: iSCSI Login negotiation failed.
tcnvh8 - /var/log/vmkernel.log 2020-02-16T07:25:44.718Z cpu13:2097316)qfle3: qfle3_queue_remove_filter:2063: [vmnic0] QueueOps.queueRemoveFilter 2020-02-16T07:25:44.720Z cpu13:2097316)qfle3: qfle3_remove_queue_filter:2012: [vmnic0] NetQ removed RX filter: queue:1 mac: 00:50:56:6c:24:9d filter id:0 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_queue_apply_filter:1923: [vmnic0] QueueOps.queueApplyFilter 0 2020-02-16T07:25:44.721Z cpu13:2097316)WARNING: qfle3: qfle3_apply_queue_mac_filter:1734: [vmnic0] NetQ could not add RX filter, no filters for queue 0 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_queue_quiesce:1061: [vmnic0] QueueOps.queueQuiesce 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_queue_quiesce:1069: [vmnic0] RxQ, QueueIDVal:1 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_rx_queue_stop:1558: [vmnic0] qfle3_rx_queue_stop, QueueIDVal:1 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_rq_stop:740: [vmnic0] qfle3_rq_stop 1 2020-02-16T07:25:44.721Z cpu13:2097316)qfle3: qfle3_rq_stop:811: [vmnic0] Stopping queue 0 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_stop:831: [vmnic0] disable netpoll for q_index 1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_stop:842: [vmnic0] Disabling interrupt on vector # 3 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_stop:867: [vmnic0] active Rx queue Count 1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_queue_free:690: [vmnic0] QueueOps.queueFree 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_queue_free:697: [vmnic0] RxQ, QueueIDVal:1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_rq_free:618: [vmnic0] Loop through 1 RSS queues 1 2020-02-16T07:25:44.724Z cpu13:2097316)qfle3: qfle3_cmd_remove_q:19191: [vmnic0] Releasing Q idx 1 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd072fc00 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd072fc00 network tracker id 974086138 tracker.iSCSI.192.168.201.2 associated 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd097ae20 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:49.833Z cpu37:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd097ae20 network tracker id 974086138 tracker.iSCSI.192.168.201.3 associated 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:1 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:39679 <http://192.168.201.108:39679> R: 192.168.201.2:3260 <http://192.168.201.2:3260>] 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:1 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:39679 <http://192.168.201.108:39679> R: 192.168.201.2:3260 <http://192.168.201.2:3260>] 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:2 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.341Z cpu20:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:45598 <http://192.168.201.108:45598> R: 192.168.201.3:3260 <http://192.168.201.3:3260>] 2020-02-16T07:25:50.341Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:2 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:50.342Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.342Z cpu20:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:45598 <http://192.168.201.108:45598> R: 192.168.201.3:3260 <http://192.168.201.3:3260>] 2020-02-16T07:25:50.595Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:50.595Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.595Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:39679 <http://192.168.201.108:39679> R: 192.168.201.2:3260 <http://192.168.201.2:3260>] 2020-02-16T07:25:50.600Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:50.600Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:50.600Z cpu37:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:45598 <http://192.168.201.108:45598> R: 192.168.201.3:3260 <http://192.168.201.3:3260>] 2020-02-16T07:25:51.204Z cpu0:2837132)J6: 2651: 'Storage_Ceph_pool1': Exiting async journal replay manager world 2020-02-16T07:25:55.360Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd06ef2a0 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:55.360Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd06ef2a0 network tracker id 974086138 tracker.iSCSI.192.168.201.2 associated 2020-02-16T07:25:55.361Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430fd0938540 network resource pool netsched.pools.persist.iscsi associated 2020-02-16T07:25:55.361Z cpu38:2098255)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430fd0938540 network tracker id 974086138 tracker.iSCSI.192.168.201.3 associated 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:1 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:17752 <http://192.168.201.108:17752> R: 192.168.201.2:3260 <http://192.168.201.2:3260>] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:1 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:17752 <http://192.168.201.108:17752> R: 192.168.201.2:3260 <http://192.168.201.2:3260>] 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba68:CH:2 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.108:15616 <http://192.168.201.108:15616> R: 192.168.201.3:3260 <http://192.168.201.3:3260>] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba68:CH:2 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:55.868Z cpu21:2098243)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.108:15616 <http://192.168.201.108:15616> R: 192.168.201.3:3260 <http://192.168.201.3:3260>] 2020-02-16T07:25:56.121Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:56.121Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:56.121Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:17752 <http://192.168.201.108:17752> R: 192.168.201.2:3260 <http://192.168.201.2:3260>] 2020-02-16T07:25:56.127Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba68:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-02-16T07:25:56.127Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-02-16T07:25:56.127Z cpu38:2098255)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.108:15616 <http://192.168.201.108:15616> R: 192.168.201.3:3260 <http://192.168.201.3:3260>] 2020-02-16T07:30:49.971Z cpu7:2098243)NMP: nmp_ThrottleLogForDevice:3788: Cmd 0x28 (0x45a296a54040, 0) to dev "naa.6001405ba48e0b99e4c418ca13506c8e" on path "vmhba68:C0:T0:L0" Failed: H:0x0 D:0x8 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:NONE
What about in the /var/log/vmkernel.log on the initiator side? Any iscsi errors?
participants (2)
-
Gesiel Galvão Bernardes
-
Mike Christie