Can't pull container images from quay-quay-quay.apps.os.sepia.ceph.com
Hello, I am trying to set up a 3-node Ceph cluster using cephadm and bootstrapping fails because podman can't pull ceph container images from quay-quay-quay.apps.os.sepia.ceph.com. Excerpt of the error message I encountered: ... /usr/bin/podman: stderr time="2024-10-09T21:57:22+08:00" level=warning msg="Failed, retrying in 1s ... (3/3). Error: parsing image configuration: Get \"https://quay-quay-quay.apps.os.sepia.ceph.com/_storage_proxy/ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltVkJiMFpMTFVvMFFrNUNZeTFpV0ZSU2IxWkdORXBxVFVsQ2QycExRbmQ1UjJnNU5FOURURUpHUzAwaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVTMXhkV0Y1TFhGMVlYa3VZWEJ3Y3k1dmN5NXpaWEJwWVM1alpYQm9MbU52YlNJc0ltNWlaaUk2TVRjeU9EUTRNakl4TWl3aWFXRjBJam94TnpJNE5EZ3lNakV5TENKbGVIQWlPakUzTWpnME9ESXlORElzSW5OMVlpSTZJbk4wYjNKaFoyVndjbTk0ZVNJc0ltRmpZMlZ6Y3lJNlczc2lkSGx3WlNJNkluTjBiM0poWjJWd2NtOTRlU0lzSW5WeWFTSTZJbkYxWVhrdlpHRjBZWE4wYjNKaFoyVXZjbVZuYVhOMGNua3ZjMmhoTWpVMkwySTNMMkkzTTJJMk1XVTVOR1E0TVRBeFpHWmtZbUk1WXpJME9XRTROREZtTW1VMk1tRmlNV1E0TWpVMll6RTJNalJrWWprMk5XTTBOVGN3TURFME1HVmlOREVfUVZkVFFXTmpaWE56UzJWNVNXUTlUamxEVUVvelZqZzNUekJZV1VKTVR6bEhORXNtVTJsbmJtRjBkWEpsUFU1dlNrZENNblpoTlVGVGIzcFlhalY2ZFdKS2VXa3lkVmwxYXlVelJDWkZlSEJwY21WelBURTNNamcwT0RJNE1USWlMQ0pvYjNOMElqb2ljbVZsYzJrd01ESXVabkp2Ym5RdWMyVndhV0V1WTJWd2FDNWpiMjA2T0RBaUxDSnpZMmhsYldVaU9pSm9kSFJ3SW4xZExDSmpiMjUwWlhoMElqcDdmWDAubGY2SjNZc1ZSTUY4ZTEtb0Nia1ZONFRlcGFBTFBIbmptTUZZUWYtMWUtUUJ4SzAtTGRlclpfdC1FSjZ1b2hkMEprcG15Yld3UEVmYkh4NVlseUNuWGwxbGhFMHhpb1dXdGM4bVA3b1hsRGtjNDhnSUxNaVVucDZuUExiaTU4M3o4c0ZaTXRRUWltWTZfU2RqcFlaU0lIbm1DbnE4Z3p2ZTQtaFprakJpOWF5MjktQzNPT1dfWTZ5X0l6UjI4bmtadTl1LTRIeGY0WjhYeDNDRnpfaGFZSUlWbGFEN3FhU3BWUWF0VFlWM2J0TnJzWml1eXhGbVlhUUdJU3VlLWp0ZDM4Szh1Q3VkbDFUSW9NVzJNLUs1MjU2NzZIdmZ0SzRTRVZ5cGtIVmxST2Y1OVJKbVc0STNiUllFVk11REhBa3ZiczBkdlczSWxoQXFVV1ZFbGpuNjhB/http/reesi002.front.sepia.ceph.com:80/quay/datastorage/registry/sha256/b7/b73b61e94d8101dfdbb9c249a841f2e62ab1d8256c1624db965c45700140eb41?AWSAccessKeyId=N9CPJ3V87O0XYBLO9G4K&Signature=NoJGB2va5ASozXj5zubJyi2uYuk%3D&Expires=1728482812\": dial tcp 172.21.0.91:443: i/o timeout" ... There must be a mistake with quay-quay-quay.apps.os.sepia.ceph.com DNS record because it's resolving to a RFC1918 address (172.16/12 prefix). I tried various online DNS tools (including Google's Admin Toolbox) to help isolate the issue and they all return the same IP address (172.21.0.91). Anyone else having this problem?
On Wednesday, October 9, 2024 10:22:51 AM EDT John Robert Mendoza wrote:
Hello,
I am trying to set up a 3-node Ceph cluster using cephadm and bootstrapping fails because podman can't pull ceph container images from quay-quay-quay.apps.os.sepia.ceph.com. Excerpt of the error message I encountered:
...
/usr/bin/podman: stderr time="2024-10-09T21:57:22+08:00" level=warning msg="Failed, retrying in 1s ... (3/3). Error: parsing image configuration: Get \"https://quay-quay-quay.apps.os.sepia.ceph.com/_storage_proxy/ZXlKaGJHY2lP aUpTVXpJMU5pSXNJbXRwWkNJNkltVkJiMFpMTFVvMFFrNUNZeTFpV0ZSU2IxWkdORXBxVFVsQ2Qy cExRbmQ1UjJnNU5FOURURUpHUzAwaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpwYzNNaU9pSnhkV0Y1 SWl3aVlYVmtJam9pY1hWaGVTMXhkV0Y1TFhGMVlYa3VZWEJ3Y3k1dmN5NXpaWEJwWVM1alpYQm9M bU52YlNJc0ltNWlaaUk2TVRjeU9EUTRNakl4TWl3aWFXRjBJam94TnpJNE5EZ3lNakV5TENKbGVI QWlPakUzTWpnME9ESXlORElzSW5OMVlpSTZJbk4wYjNKaFoyVndjbTk0ZVNJc0ltRmpZMlZ6Y3lJ Nlczc2lkSGx3WlNJNkluTjBiM0poWjJWd2NtOTRlU0lzSW5WeWFTSTZJbkYxWVhrdlpHRjBZWE4w YjNKaFoyVXZjbVZuYVhOMGNua3ZjMmhoTWpVMkwySTNMMkkzTTJJMk1XVTVOR1E0TVRBeFpHWmtZ bUk1WXpJME9XRTROREZtTW1VMk1tRmlNV1E0TWpVMll6RTJNalJrWWprMk5XTTBOVGN3TURFME1H VmlOREVfUVZkVFFXTmpaWE56UzJWNVNXUTlUamxEVUVvelZqZzNUekJZV1VKTVR6bEhORXNtVTJs bmJtRjBkWEpsUFU1dlNrZENNblpoTlVGVGIzcFlhalY2ZFdKS2VXa3lkVmwxYXlVelJDWkZlSEJw Y21WelBURTNNamcwT0RJNE1USWlMQ0pvYjNOMElqb2ljbVZsYzJrd01ESXVabkp2Ym5RdWMyVndh V0V1WTJWd2FDNWpiMjA2T0RBaUxDSnpZMmhsYldVaU9pSm9kSFJ3SW4xZExDSmpiMjUwWlhoMElq cDdmWDAubGY2SjNZc1ZSTUY4ZTEtb0Nia1ZONFRlcGFBTFBIbmptTUZZUWYtMWUtUUJ4SzAtTGRl clpfdC1FSjZ1b2hkMEprcG15Yld3UEVmYkh4NVlseUNuWGwxbGhFMHhpb1dXdGM4bVA3b1hsRGtj NDhnSUxNaVVucDZuUExiaTU4M3o4c0ZaTXRRUWltWTZfU2RqcFlaU0lIbm1DbnE4Z3p2ZTQtaFpr akJpOWF5MjktQzNPT1dfWTZ5X0l6UjI4bmtadTl1LTRIeGY0WjhYeDNDRnpfaGFZSUlWbGFEN3Fh U3BWUWF0VFlWM2J0TnJzWml1eXhGbVlhUUdJU3VlLWp0ZDM4Szh1Q3VkbDFUSW9NVzJNLUs1MjU2 NzZIdmZ0SzRTRVZ5cGtIVmxST2Y1OVJKbVc0STNiUllFVk11REhBa3ZiczBkdlczSWxoQXFVV1ZF bGpuNjhB/http/reesi002.front.sepia.ceph.com:80/quay/datastorage/registry/sha 256/b7/b73b61e94d8101dfdbb9c249a841f2e62ab1d8256c1624db965c45700140eb41?AWSA ccessKeyId=N9CPJ3V87O0XYBLO9G4K&Signature=NoJGB2va5ASozXj5zubJyi2uYuk%3D&Exp ires=1728482812\": dial tcp 172.21.0.91:443: i/o timeout"
...
There must be a mistake with quay-quay-quay.apps.os.sepia.ceph.com DNS record because it's resolving to a RFC1918 address (172.16/12 prefix). I tried various online DNS tools (including Google's Admin Toolbox) to help isolate the issue and they all return the same IP address (172.21.0.91). Anyone else having this problem?
I think everyone who is trying to pull ceph-ci images outside of the sepia lab will be hitting this issue. Others have brought this issue up in the ceph Slack/IRC channels too. FWIW, this issue did not occur prior to the lab outage yesterday. For whatever reason the quay.ceph.io registry now tries to redirect to quay-quay- quay.apps.os.sepia.ceph.com. Prior to this you could access images from either quay.ceph.io outside the lab or quay-quay-quay.apps.os.sepia.ceph.com inside the lab (only inside the lab). One more FWIW: if you are setting up a new cluster and are hitting this error you must be setting up a cluster using dev images, rather than release images. Release images live on `quay.io` (not `quay.ceph.io`). If this is intentional, great. However, you may just want to try and deploy a `squid` cluster for now unless you plan on testing `main` branch changes.
participants (2)
-
John Mulligan
-
John Robert Mendoza