ceph_leadership_team_meeting_s18e06.mkv - Dev

6 Sep 2023

Dear Cephers,

Today brought us an eventful CTL meeting: it looks like Jitsi recently started
requiring user authentication
<https://jitsi.org/blog/authentication-on-meet-jit-si/> (anonymous users
will get a "Waiting for a moderator" modal), but authentication didn't work
against Google or GitHub accounts, so we had to move to the good old Google
Meet.

As a result of this, Neha has kindly set up a new private Slack channel
(#clt) to allow for quicker communication among CLT members (if you usually
attend the CLT meeting and have not been added, please ping any CLT member
to request that).

Now, let's move on the important stuff:

*The latest Pacific Release (v16.2.14)*

*The Bad*
The 14th drop of the Pacific release has landed with a few hiccups:

   - Some .deb packages were made available to downloads.ceph.com before
   the release process completion. Although this is not the first time it
   happens, we want to ensure this is the last one, so we'd like to gather
   ideas to improve the release publishing process. Neha encouraged everyone
   to share ideas here:
      - https://tracker.ceph.com/issues/62671
      - https://tracker.ceph.com/issues/62672
      - v16.2.14 also hit issues during the ceph-container stage. Laura
   wanted to raise awareness of its current setbacks
   <https://pad.ceph.com/p/16.2.14-struggles> and collect ideas to tackle
   them:
      - Enforce reviews and mandatory CI checks
      - Rework the current approach to use simple Dockerfiles
      <https://github.com/ceph/ceph/pull/43292>
      - Call the Ceph community for help: ceph-container is currently
      maintained part-time by a single contributor (Guillaume Abrioux). This
      sub-project would benefit from the sound expertise on containers
among Ceph
      users. If you have ever considered contributing to Ceph, but felt a bit
      intimidated by C++, Paxos and race conditions, ceph-container is a good
      place to shed your fear.

*The Good*
Not everything about v16.2.14 was going to be bleak: David Orman brought us
really good news. They tested v16.2.14 on a large production cluster
(10gbit/s+ RGW and ~13PiB raw) and found that it solved a major issue
affecting RGW in Pacific <https://github.com/ceph/ceph/pull/52552>.

*The Ugly*
During that testing, they noticed that ceph-mgr was occasionally OOM killed
(nothing new to 16.2.14, as it was previously reported). They already tried:

   - Disabling modules (like the restful one, which was a suspect)
   - Enabling debug 20
   - Turning the pg autoscaler off

Debugging will continue to characterize this issue:

   - Enable profiling (Mark Nelson)
   - Try Bloomberg's Python mem profiler
   <https://github.com/bloomberg/memray> (Matthew Leonard)

*Infrastructure*

*Reminder: Infrastructure Meeting Tomorrow. **11:30-12:30 Central Time*

Patrick brought up the following topics:

   - Need to reduce the OVH spending ($72k/year, which is a good cut in the
   Ceph Foundation budget, that's a lot less avocado sandwiches for the next
   Cephalocon):
      - Move services (e.g.: Chacra) to the Sepia lab
      - Re-use CentOS (and any spared/unused) machines for devel purposes
   - Current Ceph sys admins are overloaded, so devel/community involvement
   would be much appreciated.
   - More to be discussed in tomorrow's meeting. Please join if you
   think you can help solve/improve the Ceph infrastrucru!

*BTW*: today's CDM will be canceled, since no topics were proposed.

Kind Regards,

Ernesto