Den fre 23 apr. 2021 kl 08:45 skrev Eugen Block <eblock@nde.ag>:
you can't really avoid having different versions in a cluster, think about an update/upgrade. Usually it's not an issue, but I would recommend to have that state for as short as possible. Just a recently a customer complained about inconsistent objects on a regular basis although all disks were healthy. They run a multi-site RGW and after updating all nodes to the same version they haven't faced those inconsistencies anymore.
There were some bug(s) relating to rgw's mis-caching negative answers in recent Octopus, so you might have just managed to get past that when you upgraded the rgws. Still, I agree that it seldom poses a problem if you do it in the recommended (and most tested) order. -- May the most significant bit of your life be positive.