We believe the root cause of all failures has been identified and resolved. A recent shellcheck suggestion broke some bash logic so a podman command to prepare the container environment wasn’t getting run. Additionally, a post-job script to chown files was missing a GID. The Wipe Workspace plugin that attempts to clean up the build environment before each job doesn’t use sudo and was unable to delete some files left over. https://github.com/ceph/ceph-build/pull/2428/files Thanks Dan, Zack, and John for the quick identification and resolution. From: David Galloway <David.Galloway@ibm.com> Date: Thursday, August 21, 2025 at 12:00 PM To: dev <dev@ceph.io>, sepia@ceph.com <sepia@ceph.io> Subject: Re: Maintenance Notification: Branch Push Behavior Change - 8/19/25 All, I have reverted this change for the time being. We are seeing a few (what I believe to be) container-related permissions issues that need to be resolved. https://github.com/ceph/ceph-build/pull/2427 Please re-push any branches you need rebuilt. Apologies for the inconvenience. From: David Galloway <David.Galloway@ibm.com> Date: Tuesday, August 19, 2025 at 4:10 PM To: dev <dev@ceph.io>, sepia@ceph.com <sepia@ceph.io> Subject: Re: Maintenance Notification: Branch Push Behavior Change - 8/19/25 This is complete. Any branches pushed to ceph-ci after this message will be using the new ceph-dev-pipeline job by default. Let us know if you encounter any issues. From: David Galloway <David.Galloway@ibm.com> Date: Wednesday, August 13, 2025 at 4:11 PM To: dev <dev@ceph.io>, sepia@ceph.com <sepia@ceph.io> Subject: Maintenance Notification: Branch Push Behavior Change - 8/19/25 Upstream development community, You may recall we announced the availability of a new Jenkins pipeline in June: https://lists.ceph.io/hyperkitty/list/sepia@ceph.io/thread/HBA4CVO2F6VVBZEOL... Since that announcement, this pipeline has been opt-in only. We feel we’ve observed enough builds succeed and ironed out a few bugs that we’re comfortable making this pipeline the default Jenkins job for both ceph.git and ceph-ci.git branches. This switch will be made Tuesday, August 19 at roughly 3PM Eastern. There is no additional action required on your part. Some things to remember: * DWZ is disabled by default * SCCACHE is enabled by default * The above behavior, as well as which DISTROs/ARCHs to build for, can be overridden using git trailers * Builds will be done inside a container For a refresher on how to use the git trailers and their behavior in our environment, see https://github.com/ceph/ceph-build/blob/main/ceph-trigger-build/README.md. -- David Galloway Ceph Engineering Labs – Infrastructure Architect +1 989 295 0091 - Mobile david.galloway@ibm.com IBM