I’m new to Ceph and have just deployed my first cluster using ceph-ansible and running into some issues I was hoping someone could point me in the right direction. I have 5 servers to start with. 5 OSD, 3 Monitors, 2 Manager, 4 iSCSI gateway. My intention is to use this environment as iSCSI storage for ESXi. When I went to add an iSCSI Target, I got an error that popped up in the browser (went away to fast to read it) but after this, now if I click on the “Targets” tab in Dashboard for iSCSI, I just continuously get errors popping up: 500 - Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. I’m unable to add any targets now. The log files show this: 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 907, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 266, in list IscsiTarget._set_info(target) File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 990, in _set_info raise e File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 980, in _set_info target_iqn) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 531, in func_wrapper **kwargs) File "/usr/share/ceph/mgr/dashboard/services/iscsi_client.py", line 254, in get_targetinfo return request() File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 326, in __call__ data, raw_content, headers) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 449, in do_request resp.content) dashboard.rest_client.RequestException: iscsi REST API failed request with status code 503 (b'{\n "message": "failed, gateway(s) unavailable:cxcto-c240-j27-01(UNKNOWN' b' state)"\n}\n') 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR request] [10.117.244.166:58270] [GET] [500] [0.398s] [admin] [513.0B] /api/iscsi/target 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "7bc691e8-e814-48c1-a69c-57ea1f3f3dbd"} ‘] Clearly it’s showing an error indicating that the gateway is “unavailable”. The output from gwcli shows this: [root@cxcto-c240-j27-01 ~]# ./gwcli 1 gateway is inaccessible - updates will be disabled /> ls o- / ......................................................................................................................... [...] o- cluster ......................................................................................................... [Clusters: 1] | o- ceph ............................................................................................................ [HEALTH_OK] | o- pools .......................................................................................................... [Pools: 4] | | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y] | | o- iscsi ............................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 262511b] | | o- rbd ................................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 1650b] | | o- test .................................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y] | o- topology .............................................................................................. [OSDs: 110,MONs: 3] o- disks ....................................................................................................... [0.00Y, Disks: 0] o- iscsi-targets ............................................................................... [DiscoveryAuth: CHAP, Targets: 1] o- iqn.2001-07.com.ceph:1621437640904 ................................................................ [Auth: None, Gateways: 1] o- disks .......................................................................................................... [Disks: 0] o- gateways ............................................................................................ [Up: 0/1, Portals: 1] | o- cxcto-c240-j27-01 ....................................................................... [10.122.242.196 (UNAUTHORIZED)] o- host-groups .................................................................................................. [Groups : 0] o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0] In this case it says “UNAUTHORIZED” When I created the target, I had selected all 4 iSCSI gateways, but it looks like something happened during the addition process that has left it in a weird state. The dashboard seems to think that all the gateways are up and running : [cid:8CA421EB-DFD1-43A9-87B6-7ADEE5B2DF69] Notice that it only shows the target on the one node which it is reporting an error. I’ve tried deleting the target from gwcli but that fails and I’m not really sure where to look next. The “UNAUTHORIZED” in the gwcli output makes me wonder if there is some kind of authorization issue, but I’m not sure what that would be. This is what I see if I navigate to the Targets tab on the iSCSI page: [cid:828103A0-C321-41F6-B018-AC8A883F6C7E] Any thoughts or guidance are greatly appreciated. -Paul
