filestore and bluestore may list objects in different order
Hi, This problem was reported some time ago [1] though it seemed then the root cause was not identified. Now we have a customer that is experiencing the same problem so we did some investigation. The issue is that in a bluestore/filestore mixed cluster the deep scrub may incorrectly report missing objects due to object sorting order is different on bluestore and filestore osds. Looking at the cases we have it happens for objects that have the same hash. When building the sorting key the bluestore escapes the object name string, while the filestore sorts with the raw object name, which may result in different order. I pushed a PR for review that changes the sorting order for the filestore [2]. Though using this solution would fix a cluster of mixed bluestore and upgraded filestore osds but introduce the same issue for a cluster of mixed old version and new version filestore osds. [1] https://tracker.ceph.com/issues/43174 [2] https://github.com/ceph/ceph/pull/35938 -- Mykola Golub
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola Golub wrote:
Looking at the cases we have it happens for objects that have the same hash. When building the sorting key the bluestore escapes the object name string, while the filestore sorts with the raw object name, which may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the bluestor escapes the string incorrectly, which results in the order of escaped strings is different than the order of not escaped ones. But we can't just fix the bluestore escape function as the current keys in db will become invalid. [1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the filestore [2]. Though using this solution would fix a cluster of mixed bluestore and upgraded filestore osds but introduce the same issue for a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174 [2] https://github.com/ceph/ceph/pull/35938
-- Mykola Golub
And one more question. Is the issue with deep-scrub incorrectly reporting missing shards the only problem that may be caused by different ordering? Can anyone think of any potential problem that might lead to a real data loss, e.g. when recovering an EC pg? The cluster will eventually be migrated to the bluestore only, but the migration process is long. On Mon, Jul 06, 2020 at 01:33:03PM +0300, Mykola Golub wrote:
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola Golub wrote:
Looking at the cases we have it happens for objects that have the same hash. When building the sorting key the bluestore escapes the object name string, while the filestore sorts with the raw object name, which may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the bluestor escapes the string incorrectly, which results in the order of escaped strings is different than the order of not escaped ones.
But we can't just fix the bluestore escape function as the current keys in db will become invalid.
[1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the filestore [2]. Though using this solution would fix a cluster of mixed bluestore and upgraded filestore osds but introduce the same issue for a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174 [2] https://github.com/ceph/ceph/pull/35938
-- Mykola Golub
-- Mykola Golub
There may also be issues with backfill listing objects in different orders across primary and secondary shards. Sam, would this cause correctness issues like potentially missing objects? Josh On 7/6/20 11:14 PM, Mykola Golub wrote:
And one more question. Is the issue with deep-scrub incorrectly reporting missing shards the only problem that may be caused by different ordering? Can anyone think of any potential problem that might lead to a real data loss, e.g. when recovering an EC pg?
The cluster will eventually be migrated to the bluestore only, but the migration process is long.
On Mon, Jul 06, 2020 at 01:33:03PM +0300, Mykola Golub wrote:
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola Golub wrote:
Looking at the cases we have it happens for objects that have the same hash. When building the sorting key the bluestore escapes the object name string, while the filestore sorts with the raw object name, which may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the bluestor escapes the string incorrectly, which results in the order of escaped strings is different than the order of not escaped ones.
But we can't just fix the bluestore escape function as the current keys in db will become invalid.
[1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the filestore [2]. Though using this solution would fix a cluster of mixed bluestore and upgraded filestore osds but introduce the same issue for a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174 [2] https://github.com/ceph/ceph/pull/35938
-- Mykola Golub
Yes, inconsistent listing order would potentially cause backfill to skip objects. -Sam On Thu, Jul 9, 2020 at 9:01 AM Josh Durgin <jdurgin@redhat.com> wrote:
There may also be issues with backfill listing objects in different orders across primary and secondary shards. Sam, would this cause correctness issues like potentially missing objects?
Josh
On 7/6/20 11:14 PM, Mykola Golub wrote:
And one more question. Is the issue with deep-scrub incorrectly reporting missing shards the only problem that may be caused by different ordering? Can anyone think of any potential problem that might lead to a real data loss, e.g. when recovering an EC pg?
The cluster will eventually be migrated to the bluestore only, but the migration process is long.
On Mon, Jul 06, 2020 at 01:33:03PM +0300, Mykola Golub wrote:
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola Golub wrote:
Looking at the cases we have it happens for objects that have the same hash. When building the sorting key the bluestore escapes the object name string, while the filestore sorts with the raw object name, which may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the bluestor escapes the string incorrectly, which results in the order of escaped strings is different than the order of not escaped ones.
But we can't just fix the bluestore escape function as the current keys in db will become invalid.
[1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the filestore [2]. Though using this solution would fix a cluster of mixed bluestore and upgraded filestore osds but introduce the same issue for a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174 [2] https://github.com/ceph/ceph/pull/35938
-- Mykola Golub
participants (3)
-
Josh Durgin
-
Mykola Golub
-
Sam Just