Hi Patrick, Thanks for catching this! f1ecf99a86edfe899392b6b734351f1015a93be6 <https://github.com/ceph/ceph/commit/f1ecf99a86edfe899392b6b734351f1015a93be6> claims the 31st bit was retired since Pacific. ``` // this bit is ignored by release *unused* and not advertised by // release *unadvertised* #define DEFINE_CEPH_FEATURE_RETIRED(bit, inc, name, unused, unadvertised) ``` ``` -DEFINE_CEPH_FEATURE_DEPRECATED(31, 1, MON_SINGLE_PAXOS, NAUTILUS) +DEFINE_CEPH_FEATURE_RETIRED(31, 1, MON_SINGLE_PAXOS, NAUTILUS, PACIFIC) +// available for R ``` The problem is that the commit was merged into Quincy; it has *not* been merged into Pacific. ``` $ git show origin/pacific:src/include/ceph_features.h | grep MON_SINGLE_PAXOS, DEFINE_CEPH_FEATURE_DEPRECATED(31, 1, MON_SINGLE_PAXOS, NAUTILUS) $ git show origin/quincy:src/include/ceph_features.h | grep MON_SINGLE_PAXOS, DEFINE_CEPH_FEATURE_RETIRED(31, 1, MON_SINGLE_PAXOS, NAUTILUS, PACIFIC) ``` Pacific entities still advertise bit no. 31 in their `CEPH_FEATURES_ALL`. Reef uses the same bit for SERVER_REEF. Therefore the same bit has two meanings between releases that can overlap in a single cluster because of the N-2 upgradeability rule. As Quincy does not advertise it, upgrading through it should be safe. Confusing SERVER_REEF as reappeared MON_SINGLE_PAXOS is uninteresting. However, I do wonder if there may be other unintended failures
resulting from the Pacific and Reef OSDs believing they are operating with the same version. This problem would have existed since the first reef release (v18.2.0) but to my knowledge there are no documented problems that may have arisen during a rolling upgrade. I request the core team have a second look so we can document the problem properly for users.
I agree, what is interesting is confusing MON_SINGLE_PAXOS as SERVER_REEF. Apart from *prematurely* raising the OSD_UPGRADE_FINISHED warning, the impact might come from the pg-upmap-primary feature. I'm still refreshing my mind on the SERVER_REEF branching but, at the moment, I can think of knocking out the prevention on using these upmaps while there are still some Pacific entities which don't understand them. Also, Reef encoders of OSDMap would append-at-the-end even empty `pg_upmap_primaries`. Pacific decoders should ignore them but, because of CRC mismatches, OSDs would be requesting full maps. Yet, these 2 things wouldn't be new nor specific to v18.2.5 (all Reef minors are affected). IMHO we should update Reef notes to suggest avoiding N-2 upgrades in favor of always going through Quincy. Regards, Radek On Thu, Mar 5, 2026 at 5:23 PM Patrick Donnelly <pdonnell@redhat.com> wrote:
Hi Radoslaw/Laura,
I discovered an issue while fixing the broken upgrade suite for the final reef release, ticket here:
https://tracker.ceph.com/issues/75034
The short of it is that Reef chose a feature bit for SERVER_REEF that was still in use by Pacific:
https://github.com/ceph/ceph/commit/35d483af5425eb08593bc1c3065cfaec99938244
that bit was deprecated in
https://github.com/ceph/ceph/commit/6c097015bbc1bcfa8abe518680a3d3a17ff39884
and retired in
https://github.com/ceph/ceph/commit/f1ecf99a86edfe899392b6b734351f1015a93be6
Thus Pacific OSDs are advertising SERVER_REEF from the Reef Monitor's perspective.
This was only discovered because this backport was merged for v18.2.5:
https://github.com/ceph/ceph/commit/25e8b22c6f29cd3947b501f6aaf7614ba204a2c8 https://tracker.ceph.com/issues/69150
That PR would cause these warnings to be raised during stress splits of reef upgrades (4/8 OSDs upgraded to reef):
https://pulpito.ceph.com/pdonnell-2026-03-04_15:16:52-upgrade:pacific-x-reef...
Unfortunately the bug was shadowed by other failures during QA for v18.2.5:
https://tracker.ceph.com/issues/70563
in
https://pulpito.ceph.com/yuriw-2025-03-20_16:00:43-upgrade:pacific-x-reef-re...
I have worked around the problem by muting the OSD_UPGRADE_FINISHED health warning:
https://github.com/ceph/ceph/pull/67657/changes/5a52bfaf4400a7d7632602f455c6...
However, I do wonder if there may be other unintended failures resulting from the Pacific and Reef OSDs believing they are operating with the same version. This problem would have existed since the first reef release (v18.2.0) but to my knowledge there are no documented problems that may have arisen during a rolling upgrade. I request the core team have a second look so we can document the problem properly for users.
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D