On Wednesday, October 9, 2024 10:22:51 AM EDT John Robert Mendoza wrote:
Hello,
I am trying to set up a 3-node Ceph cluster using cephadm and bootstrapping fails because podman can't pull ceph container images from quay-quay-quay.apps.os.sepia.ceph.com. Excerpt of the error message I encountered:
...
/usr/bin/podman: stderr time="2024-10-09T21:57:22+08:00" level=warning msg="Failed, retrying in 1s ... (3/3). Error: parsing image configuration: Get \"https://quay-quay-quay.apps.os.sepia.ceph.com/_storage_proxy/ZXlKaGJHY2lP aUpTVXpJMU5pSXNJbXRwWkNJNkltVkJiMFpMTFVvMFFrNUNZeTFpV0ZSU2IxWkdORXBxVFVsQ2Qy cExRbmQ1UjJnNU5FOURURUpHUzAwaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpwYzNNaU9pSnhkV0Y1 SWl3aVlYVmtJam9pY1hWaGVTMXhkV0Y1TFhGMVlYa3VZWEJ3Y3k1dmN5NXpaWEJwWVM1alpYQm9M bU52YlNJc0ltNWlaaUk2TVRjeU9EUTRNakl4TWl3aWFXRjBJam94TnpJNE5EZ3lNakV5TENKbGVI QWlPakUzTWpnME9ESXlORElzSW5OMVlpSTZJbk4wYjNKaFoyVndjbTk0ZVNJc0ltRmpZMlZ6Y3lJ Nlczc2lkSGx3WlNJNkluTjBiM0poWjJWd2NtOTRlU0lzSW5WeWFTSTZJbkYxWVhrdlpHRjBZWE4w YjNKaFoyVXZjbVZuYVhOMGNua3ZjMmhoTWpVMkwySTNMMkkzTTJJMk1XVTVOR1E0TVRBeFpHWmtZ bUk1WXpJME9XRTROREZtTW1VMk1tRmlNV1E0TWpVMll6RTJNalJrWWprMk5XTTBOVGN3TURFME1H VmlOREVfUVZkVFFXTmpaWE56UzJWNVNXUTlUamxEVUVvelZqZzNUekJZV1VKTVR6bEhORXNtVTJs bmJtRjBkWEpsUFU1dlNrZENNblpoTlVGVGIzcFlhalY2ZFdKS2VXa3lkVmwxYXlVelJDWkZlSEJw Y21WelBURTNNamcwT0RJNE1USWlMQ0pvYjNOMElqb2ljbVZsYzJrd01ESXVabkp2Ym5RdWMyVndh V0V1WTJWd2FDNWpiMjA2T0RBaUxDSnpZMmhsYldVaU9pSm9kSFJ3SW4xZExDSmpiMjUwWlhoMElq cDdmWDAubGY2SjNZc1ZSTUY4ZTEtb0Nia1ZONFRlcGFBTFBIbmptTUZZUWYtMWUtUUJ4SzAtTGRl clpfdC1FSjZ1b2hkMEprcG15Yld3UEVmYkh4NVlseUNuWGwxbGhFMHhpb1dXdGM4bVA3b1hsRGtj NDhnSUxNaVVucDZuUExiaTU4M3o4c0ZaTXRRUWltWTZfU2RqcFlaU0lIbm1DbnE4Z3p2ZTQtaFpr akJpOWF5MjktQzNPT1dfWTZ5X0l6UjI4bmtadTl1LTRIeGY0WjhYeDNDRnpfaGFZSUlWbGFEN3Fh U3BWUWF0VFlWM2J0TnJzWml1eXhGbVlhUUdJU3VlLWp0ZDM4Szh1Q3VkbDFUSW9NVzJNLUs1MjU2 NzZIdmZ0SzRTRVZ5cGtIVmxST2Y1OVJKbVc0STNiUllFVk11REhBa3ZiczBkdlczSWxoQXFVV1ZF bGpuNjhB/http/reesi002.front.sepia.ceph.com:80/quay/datastorage/registry/sha 256/b7/b73b61e94d8101dfdbb9c249a841f2e62ab1d8256c1624db965c45700140eb41?AWSA ccessKeyId=N9CPJ3V87O0XYBLO9G4K&Signature=NoJGB2va5ASozXj5zubJyi2uYuk%3D&Exp ires=1728482812\": dial tcp 172.21.0.91:443: i/o timeout"
...
There must be a mistake with quay-quay-quay.apps.os.sepia.ceph.com DNS record because it's resolving to a RFC1918 address (172.16/12 prefix). I tried various online DNS tools (including Google's Admin Toolbox) to help isolate the issue and they all return the same IP address (172.21.0.91). Anyone else having this problem?
I think everyone who is trying to pull ceph-ci images outside of the sepia lab will be hitting this issue. Others have brought this issue up in the ceph Slack/IRC channels too. FWIW, this issue did not occur prior to the lab outage yesterday. For whatever reason the quay.ceph.io registry now tries to redirect to quay-quay- quay.apps.os.sepia.ceph.com. Prior to this you could access images from either quay.ceph.io outside the lab or quay-quay-quay.apps.os.sepia.ceph.com inside the lab (only inside the lab). One more FWIW: if you are setting up a new cluster and are hitting this error you must be setting up a cluster using dev images, rather than release images. Release images live on `quay.io` (not `quay.ceph.io`). If this is intentional, great. However, you may just want to try and deploy a `squid` cluster for now unless you plan on testing `main` branch changes.