How do you know if its safe to set `require-min-compat-client=ree f` if you have kernel clients?

At the moment it is not enforced when it comes to accepting connections from clients. There is undergoing discussion what was intended and what the contract in librados finally became. Regardless of this, the upmap-primary genuinely requires clients to understand the concept. It shouldn't be turned on if an operator is not sure that older clients can be neglected. What is connected at given moment can be dumped the mon's `sessions` command. I would expect the feature will become "universally useful" in long time from now, together with proliferation of modern-enough kernels. For right now upmap-primary has several defects, also about version checks, and the best is to simply to not enable it (by adding mappings). If mappings are already present, it's recommended to purge them, Another reason for that is incompatible clients may get exposed (by failing connection attempts) when the version / feature bit enforcement gets fixed. Regards, Radek On Wed, May 29, 2024 at 5:36 PM Fox, Kevin M <Kevin.Fox@pnnl.gov> wrote:

How do you know if its safe to set `require-min-compat-client=reef` if you have kernel clients?

Thanks, Kevin

________________________________________ From: Laura Flores <lflores@redhat.com> Sent: Wednesday, May 29, 2024 8:12 AM To: ceph-users; dev; clt Cc: Radoslaw Zarzynski; Yuri Weinstein Subject: [ceph-users] Bug Found in Reef Releases - Action Required for pg-upmap-primary Interface Users

Check twice before you click! This email originated from outside PNNL.

Dear Ceph Users,

We have discovered a bug with the pg-upmap-primary interface (related to the offline read balancer [1]) that affects all Reef releases.

In all Reef versions, users are required to set `require-min-compat-client=reef` in order to use the pg-upmap-primary interface to prevent pre-reef clients from connecting and not understanding the new interface. We found this setting is simply not enforced [2], which leads to miscommunication between older and newer peers or, depending on version, to an assert in the mons and/or osds [3]. However, the fundamental precondition is making use of the new `pg-upmap-primary` feature.

If you have not yet upgraded to v18.2.2, we recommend that you refrain from upgrading to v18.2.2 until a later version is out with a fix. We also recommend removing any existing pg-upmap-primary mappings to prevent hitting the assert [3], as well as to prevent any miscommunication between older and newer peers about pg primaries [2]. Remove mappings by: $ `ceph osd dump` For each pg_upmap_primary entry in the above output: $ `ceph osd rm-pg-upmap-primary <pgid>`

If you have already upgraded to v18.2.2, your cluster is more likely to hit the osd/mon assert [3] when you set a `pg-upmap-primary` mapping (this would involve explicitly setting a mapping via the osdmaptool or the CLI command). As long as you refrain from setting any pg-upmap-primary mappings, your cluster will NOT be affected by [3].

Follow the trackers below for further updates.

1. pg-upmap-primary documentation: https://docs.ceph.com/en/reef/rados/operations/read-balancer/ 2. mon, osd, *: require-min-compat-client is not really honored - https://tracker.ceph.com/issues/66260 3. Failed assert "pg_upmap_primaries.empty()" in the read balancer - https://tracker.ceph.com/issues/61948

Thanks, Laura Flores

--

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage <https://ceph.io/>

Chicago, IL

lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io