Thanks a lot, I can of course confirm things are back :-). That structure (symlink to absolute path) likely also explains why mirrors are effectively not mirroring these versions, some official mirrors seem to have these directories and show 404 when trying to access them (i.e. symlink mirrored, e.g. au.ceph.com) and some are missing the directories completely in their listing and show "Forbidden" when trying to access them (e.g. de.ceph.com). Cheers and thanks for the quick fix, Oliver Am 25.09.25 um 20:13 schrieb Casey Bodley:
thanks for the reports. the broken links appear to be working again. quoting Dan Mick from #ceph-devel slack: "ah. there's another volume to hold some older stuff, including octopus. It was mounted at /overflow and symlinks were pointed to it, but I apparently didn't add it to /etc/fstab, so when I updated download.ceph.com yesterday it wasn't remounted. Fixing."
On Thu, Sep 25, 2025 at 12:01 PM Oliver Freyermuth <freyermuth@physik.uni-bonn.de> wrote:
Am 25.09.25 um 17:54 schrieb Matt Vandermeulen:
Interesting, from our mirror it looks like these (and more) are absolute symlinks:
lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-15.2.16 -> /overflow/debian-15.2.16 lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-15.2.17 -> /overflow/debian-15.2.17 lrwxrwxrwx 1 root root 24 Nov 14 2024 debian-octopus -> /overflow/debian-octopus lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-15.2.16 -> /overflow/rpm-15.2.16 lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-15.2.17 -> /overflow/rpm-15.2.17 lrwxrwxrwx 1 root root 21 Nov 14 2024 rpm-octopus -> /overflow/rpm-octopus
Looks like it has been this way for quite some time based on our zfs snapshots:
lrwxrwxrwx 1 1109 1109 11 Aug 9 2022 .zfs/snapshot/2024-11-01-0620/rpm-octopus -> rpm-15.2.17 lrwxrwxrwx 1 root root 21 Nov 14 2024 .zfs/snapshot/2024-12-01-0620/rpm-octopus -> /overflow/rpm-octopus
If you need 15.2.16, a snapshot of it is accessible via: https://mirror.reenigne.net/ceph-snapshots/2024-11-01-0620/rpm-15.2.16/
Many thanks for the helpful mirroring of the snapshots!
The way we noticed is since some of our older clients started to report:
- Status code: 404 for https://download.ceph.com/rpm-octopus/el8/x86_64/repodata/repomd.xml (IP: 131.220.166.104)
today when running DNF, so maybe the absolute symlink target "/overflow/" was only available on the download.ceph.com server and was purged today, or on a separate storage which failed...
Cheers and thanks, Oliver
--- Thanks, Matt Vandermeulen
Software Engineering, and Server Solutions Reenigne
Ph: 716.217.0502
On 2025-09-25 12:41, Oliver Freyermuth wrote:
Dear Cephalopodians,
it seems that recently some directories were emptied / became inaccessible on download.ceph.com, for example (some randomly selected ones):
https://download.ceph.com/rpm-octopus https://download.ceph.com/rpm-15.2.16
As older releases such as:
https://download.ceph.com/rpm-mimic/
are still there, I presume this was not intentional, so I am alerting here in case there might for example be a file system problem on the download mirror.
Cheers, Oliver
-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 -- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.055 Käthe-Kümmel-Straße 1 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --