Hi Patrick,
Thanks for catching this!
```
// this bit is ignored by release *unused* and not advertised by
// release *unadvertised*
#define DEFINE_CEPH_FEATURE_RETIRED(bit, inc, name, unused, unadvertised)
```
```
-DEFINE_CEPH_FEATURE_DEPRECATED(31, 1, MON_SINGLE_PAXOS, NAUTILUS)
+DEFINE_CEPH_FEATURE_RETIRED(31, 1, MON_SINGLE_PAXOS, NAUTILUS, PACIFIC)
+// available for R
```
The problem is that the commit was merged into Quincy; it has *not* been merged into Pacific.
```
$ git show origin/pacific:src/include/ceph_features.h | grep MON_SINGLE_PAXOS,
DEFINE_CEPH_FEATURE_DEPRECATED(31, 1, MON_SINGLE_PAXOS, NAUTILUS)
$ git show origin/quincy:src/include/ceph_features.h | grep MON_SINGLE_PAXOS,
DEFINE_CEPH_FEATURE_RETIRED(31, 1, MON_SINGLE_PAXOS, NAUTILUS, PACIFIC)
```
Pacific entities still advertise bit no. 31 in their `CEPH_FEATURES_ALL`.
Reef uses the same bit for SERVER_REEF.
Therefore the same bit has two meanings between releases that can
overlap in a single cluster because of the N-2 upgradeability rule.
As Quincy does not advertise it, upgrading through it should be safe.
Confusing SERVER_REEF as reappeared MON_SINGLE_PAXOS
is uninteresting.
However, I do wonder if there may be other unintended failures
resulting from the Pacific and Reef OSDs believing they are operating
with the same version. This problem would have existed since the first
reef release (v18.2.0) but to my knowledge there are no documented
problems that may have arisen during a rolling upgrade. I request the
core team have a second look so we can document the problem properly
for users.
I agree, what is interesting is confusing MON_SINGLE_PAXOS as SERVER_REEF.
Apart from *prematurely* raising the OSD_UPGRADE_FINISHED warning,
the impact might come from the pg-upmap-primary feature. I'm still refreshing
my mind on the SERVER_REEF branching but, at the moment, I can
think of knocking out the prevention on using these upmaps while there are still
some Pacific entities which don't understand them. Also, Reef encoders of OSDMap
would append-at-the-end even empty `pg_upmap_primaries`. Pacific decoders
should ignore them but, because of CRC mismatches, OSDs would be requesting
full maps. Yet, these 2 things wouldn't be new nor specific to v18.2.5 (all Reef
minors are affected).
IMHO we should update Reef notes to suggest avoiding N-2 upgrades in favor
of always going through Quincy.
Regards,