On Wednesday, July 17, 2024 2:03:58 PM EDT Marianne Spiller wrote:
Hi,
I'm about setting up a Ceph stage cluster using `cephadm` and `podman`. As I'm not allowed to connect to the outside, our Artifactory keeps the needed quay.io packages mirrored. What I did:
* put a config `/etc/containers/registries.conf.d` for `prefix=quay.io` on the first node * created an Artifactory user with permissions for the quay.io packages * bootstrapped the cluster using `registry-url`, `registry-username` and `registry-password`
An `/etc/ceph/podman-auth.json` got created; the bootstrapped cluster is up & running (reef, 18.2.2).
But the "New version" check does not work, and it floods the logs with messages. **Lots** of messages. **Of course**, a direct connection to quay.io will lead to a `connection refused` - that's why it should use the mirror.
So: why does the update check ignore my mirror settings? And what can I do to make it work?
Thanks a lot for any kind of an idea, Marianne
There are a couple of parts of ceph that try to talk to registries directly, not via podman (or docker). Because this code doesn't know anything about libcontainer config files it will ignore your `prefix=` setting. You may be running into this. The typical method to use a disconnected (no internet) cluster is to remap the container images used: see https://docs.ceph.com/en/reef/cephadm/install/ #deployment-in-an-isolated-environment
``` 7/17/24 7:45:47 PM [ERR] Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connection.py", line 162, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request self._validate_conn(conn) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn conn.connect() File "/lib/python3.6/site-packages/urllib3/connection.py", line 315, in connect conn = self._new_conn() File "/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/registry.py", line 43, in get_tags r = requests.get(url, headers=headers) File "/lib/python3.6/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 535, in request resp = self.send(prep, **send_kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 648, in send r = adapter.send(request, **kwargs) File "/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 283, in upgrade_ls ls = reg.get_tags(bare_image) File "/usr/share/ceph/mgr/cephadm/registry.py", line 46, in get_tags raise ValueError(msg) from e ValueError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 137, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in upgrade_ls return self.upgrade.upgrade_ls(image, tags, show_all_versions) File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 285, in upgrade_ls raise OrchestratorError(f'{e}') orchestrator._interface.OrchestratorError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) ```