CEPH-ISCSI fails when restarting rbd-target-api and won't work anymore - ceph-users

28 Nov 2020

I have  an issue on ceph-iscsi ( ubuntu 20 LTS and Ceph 15.2.6)  after I restart 
rbd-target-api, it fails and not starting again:

```

sudo systemctl status rbd-target-api.service
● rbd-target-api.service - Ceph iscsi target configuration API
     Loaded: loaded (/lib/systemd/system/rbd-target-api.service; enabled; vendor preset:
enabled)
     Active: deactivating (stop-sigterm) since Sat 2020-11-28 17:01:40 +0330; 20s ago
   Main PID: 37651 (rbd-target-api)
      Tasks: 55 (limit: 9451)
     Memory: 141.4M
     CGroup: /system.slice/rbd-target-api.service
             ├─15289 /usr/bin/python3 /usr/bin/rbd-target-api
             └─37651 /usr/bin/python3 /usr/bin/rbd-target-api

Nov 28 14:36:53 dev11 systemd[1]: Started Ceph iscsi target configuration API.
Nov 28 14:36:54 dev11 rbd-target-api[37651]: Started the configuration object watcher
Nov 28 14:36:54 dev11 rbd-target-api[37651]: Processing osd blacklist entries for this
node
Nov 28 14:36:54 dev11 rbd-target-api[37651]: Checking for config object changes every 1s
Nov 28 14:36:55 dev11 rbd-target-api[37651]: Reading the configuration object to update
local LIO configuration
Nov 28 14:36:55 dev11 rbd-target-api[37651]: Processing Gateway configuration
Nov 28 14:36:55 dev11 rbd-target-api[37651]: Setting up
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 14:36:55 dev11 rbd-target-api[37651]: (Gateway.load_config) successfully loaded
existing target definition
Nov 28 17:01:40 dev11 systemd[1]: Stopping Ceph iscsi target configuration API...

```
journalctl:
```
Nov 28 17:00:01 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:01 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:04 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:04 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:06 dev11 ceph-mgr[3184]: [172.16.1.3:57002] [GET] [500] [45.074s] [admin]
[513.0B] /api/health/minimal
Nov 28 17:00:06 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal
Server Error", "detail": "The server encountered an unexpected
condition which prevented it from fulfilling the request.", "request_id":
"68eed46b-3ece-4e60-bc17-a172358f2d76"}                                         

']
Nov 28 17:00:06 dev11 ceph-mgr[3184]: [172.16.1.3:60128] [GET] [500] [45.070s] [admin]
[513.0B] /api/health/minimal
Nov 28 17:00:06 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal
Server Error", "detail": "The server encountered an unexpected
condition which prevented it from fulfilling the request.", "request_id":
"5b6fdaa2-dc70-48a7-b01f-ca554ecfec41"}                                         

']
Nov 28 17:00:07 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:07 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:11 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:11 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:11 dev11 ceph-mgr[3184]: ::ffff:127.0.0.1 - - [28/Nov/2020:17:00:11]
"GET /metrics HTTP/1.1" 200 151419 "" "Prometheus/2.7.2"
Nov 28 17:00:14 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:14 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:17 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:17 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:20 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:20 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:22 dev11 ceph-mgr[3184]: [172.16.1.3:59834] [GET] [500] [45.062s] [admin]
[513.0B] /api/health/minimal
Nov 28 17:00:22 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal
Server Error", "detail": "The server encountered an unexpected
condition which prevented it from fulfilling the request.", "request_id":
"1ba61331-1dfd-43e7-8ced-9f28aeb8a39c"}                                         

']
Nov 28 17:00:23 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:23 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:26 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:26 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:26 dev11 ceph-mgr[3184]: ::ffff:127.0.0.1 - - [28/Nov/2020:17:00:26]
"GET /metrics HTTP/1.1" 200 151420 "" "Prometheus/2.7.2"
Nov 28 17:00:27 dev11 ceph-mgr[3184]: [172.16.1.3:60132] [GET] [500] [45.081s] [admin]
[513.0B] /api/health/minimal
Nov 28 17:00:27 dev11 ceph-mgr[3184]: [b'{"status": "500 Internal
Server Error", "detail": "The server encountered an unexpected
condition which prevented it from fulfilling the request.", "request_id":
"9c1dd49b-07fb-4c49-a033-f5d8d82d9cbe"}                                         

']
Nov 28 17:00:29 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:29 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:32 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:32 dev11 kernel: iSCSI Login negotiation failed.
Nov 28 17:00:35 dev11 kernel: Unable to locate Target Portal Group on
iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
Nov 28 17:00:35 dev11 kernel: iSCSI Login negotiation failed.

```
What should I do for this problem?