sharing a retrospective on the 'make check' failures over the past few days: On Fri, Jul 26, 2024 at 3:15 PM Casey Bodley <cbodley@redhat.com> wrote:
On Thu, Jul 18, 2024 at 2:03 PM Casey Bodley <cbodley@redhat.com> wrote:
now that distros are starting to adopt gcc-14, it's time to revisit our supported compiler versions
ceph has been pinned to gcc-11 ever since we enabled c++20 in https://github.com/ceph/ceph/pull/45133
this means that we're now trying to support 4 gcc major versions. an example where this is problematic: gcc-14 complains about the use of deprecated overloads atomic_store_explicit/atomic_load_explicit in ceph_context.h, with a warning that says "use 'std::atomic<std::shared_ptr<T>>' instead". i made this change in https://github.com/ceph/ceph/pull/58176, but it fails to compile under gcc-11 with "std::atomic requires a trivially copyable type". that works fine in gcc-12 and later
crimson builds recently switched to gcc-13 in https://github.com/ceph/ceph/pull/55886, but an lto-related bug in gcc prevented the whole project from switching. in order to make that switch, we'll either need to: a) verify that the compiler bug is resolved in all supported distros, or
for reference, this bug is tracked by: https://tracker.ceph.com/issues/63867 https://bugzilla.redhat.com/show_bug.cgi?id=2241339 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113359
the gcc bug is RESOLVED with 13.3 as the target milestone
in an up-to-date centos stream 9 vm, i see that appstream's gcc-toolset-13-gcc-c++ provides 13.3.1-2.1.el9. so we could switch to gcc-13 for rpm builds
ubuntu jammy doesn't provide gcc-13. its toolchain ppa does, but i only see version 13.1.0-8ubuntu1~22.04 there. so we might stick to gcc-12 on ubuntu until we get 24.04 builders going in the lab
b) measure the performance impact of disabling lto and decide whether it's significant
if we're not able to switch to gcc-13 yet, then we should require at least gcc-12
in https://github.com/ceph/ceph/pull/58384, i tried what i considered the most conservative approach: bumping gcc from 11 to 12 this was successful in ceph-ci builds, but caused an unforeseen interaction with the clang builds for pull request checks. the presence of gcc-12 on these workers caused clang to choose the standard library from gcc-12 where it had previously used 11's. this affected the clang builds of all pull requests and all release branches the first symptom, tracked in https://tracker.ceph.com/issues/67233, was due to the fact that gcc-12 was installed but not libstdc++-12-dev. with help from #sepia, we went through and installed this package on all of the ubuntu builders while this fixed the immediate issue, crimson unit tests built with this toolchain still failed with exceptions from libfmt tracked in https://tracker.ceph.com/issues/66887. a libfmt upgrade was proposed in https://github.com/ceph/ceph/pull/58960 to address this, but could not be reasonably backported all the way to quincy so the only way to restore 'make check' on all releases was to remove all of the gcc-12-related packages from these workers and give up on gcc upgrades for now. this cleanup should be complete soon (if not already), thanks to help from Adam Kraitman. if you see further issues here, please raise them in the #sepia slack channel i apologize for the disruption, especially while the squid release process is still ongoing. i'll wait until after the squid release before trying to make any further progress here after squid is out, i think we need to prioritize the upgrade of these ubuntu 22.04 workers to 24.04 where we have access to gcc-13 (which has been working for crimson builds) and newer clang. this is tracked in https://tracker.ceph.com/issues/66914