Hi,
This problem was reported some time ago [1] though it seemed then the root
cause was not identified.
Now we have a customer that is experiencing the same problem so we did
some investigation. The issue is that in a bluestore/filestore mixed
cluster the deep scrub may incorrectly report missing objects due to
object sorting order is different on bluestore and filestore osds.
Looking at the cases we have it happens for objects that have the same
hash. When building the sorting key the bluestore escapes the object
name string, while the filestore sorts with the raw object name, which
may result in different order.
I pushed a PR for review that changes the sorting order for the
filestore [2]. Though using this solution would fix a cluster of mixed
bluestore and upgraded filestore osds but introduce the same issue for
a cluster of mixed old version and new version filestore osds.
[1]
https://tracker.ceph.com/issues/43174
[2]
https://github.com/ceph/ceph/pull/35938
--
Mykola Golub