Hi, I'm about setting up a Ceph stage cluster using `cephadm` and `podman`. As I'm not allowed to connect to the outside, our Artifactory keeps the needed quay.io packages mirrored. What I did: * put a config `/etc/containers/registries.conf.d` for `prefix=quay.io` on the first node * created an Artifactory user with permissions for the quay.io packages * bootstrapped the cluster using `registry-url`, `registry-username` and `registry-password` An `/etc/ceph/podman-auth.json` got created; the bootstrapped cluster is up & running (reef, 18.2.2). But the "New version" check does not work, and it floods the logs with messages. **Lots** of messages. **Of course**, a direct connection to quay.io will lead to a `connection refused` - that's why it should use the mirror. So: why does the update check ignore my mirror settings? And what can I do to make it work? Thanks a lot for any kind of an idea, Marianne ``` 7/17/24 7:45:47 PM [ERR] Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connection.py", line 162, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request self._validate_conn(conn) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn conn.connect() File "/lib/python3.6/site-packages/urllib3/connection.py", line 315, in connect conn = self._new_conn() File "/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/registry.py", line 43, in get_tags r = requests.get(url, headers=headers) File "/lib/python3.6/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 535, in request resp = self.send(prep, **send_kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 648, in send r = adapter.send(request, **kwargs) File "/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 283, in upgrade_ls ls = reg.get_tags(bare_image) File "/usr/share/ceph/mgr/cephadm/registry.py", line 46, in get_tags raise ValueError(msg) from e ValueError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 137, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in upgrade_ls return self.upgrade.upgrade_ls(image, tags, show_all_versions) File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 285, in upgrade_ls raise OrchestratorError(f'{e}') orchestrator._interface.OrchestratorError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) ```