6 Jul
2020
6 Jul
'20
9:13 a.m.
Hi,
This problem was reported some time ago [1] though it seemed then the root
cause was not identified.
Now we have a customer that is experiencing the same problem so we did
some investigation. The issue is that in a bluestore/filestore mixed
cluster the deep scrub may incorrectly report missing objects due to
object sorting order is different on bluestore and filestore osds.
Looking at the cases we have it happens for objects that have the same
hash. When building the sorting key the bluestore escapes the object
name string, while the filestore sorts with the raw object name, which
may result in different order.
I pushed a PR for review that changes the sorting order for the
filestore [2]. Though using this solution would fix a cluster of mixed
bluestore and upgraded filestore osds but introduce the same issue for
a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174
[2] https://github.com/ceph/ceph/pull/35938
--
Mykola Golub
6 Jul
6 Jul
1:33 p.m.
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola Golub wrote:
Looking at the cases we have it happens for objects
that have the same
hash. When building the sorting key the bluestore escapes the object
name string, while the filestore sorts with the raw object name, which
may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the
bluestor escapes the string incorrectly, which results in the order of
escaped strings is different than the order of not escaped ones.
But we can't just fix the bluestore escape function as the current
keys in db will become invalid.
[1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the
filestore [2]. Though using this solution would fix a cluster of mixed
bluestore and upgraded filestore osds but introduce the same issue for
a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174
[2] https://github.com/ceph/ceph/pull/35938
--
Mykola Golub
7 Jul
7 Jul
9:14 a.m.
And one more question. Is the issue with deep-scrub incorrectly
reporting missing shards the only problem that may be caused by
different ordering? Can anyone think of any potential problem that
might lead to a real data loss, e.g. when recovering an EC pg?
The cluster will eventually be migrated to the bluestore only, but the
migration process is long.
On Mon, Jul 06, 2020 at 01:33:03PM +0300, Mykola Golub wrote:
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola Golub
wrote:
--
Mykola Golub
Looking at the cases we have it happens for
objects that have the same
hash. When building the sorting key the bluestore escapes the object
name string, while the filestore sorts with the raw object name, which
may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the
bluestor escapes the string incorrectly, which results in the order of
escaped strings is different than the order of not escaped ones.
But we can't just fix the bluestore escape function as the current
keys in db will become invalid.
[1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the
filestore [2]. Though using this solution would fix a cluster of mixed
bluestore and upgraded filestore osds but introduce the same issue for
a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174
[2] https://github.com/ceph/ceph/pull/35938
--
Mykola Golub 
9 Jul
9 Jul
7:01 p.m.
There may also be issues with backfill listing objects in different
orders across primary and secondary shards. Sam, would this cause
correctness issues like potentially missing objects?
Josh
On 7/6/20 11:14 PM, Mykola Golub wrote:
And one more question. Is the issue with deep-scrub
incorrectly
reporting missing shards the only problem that may be caused by
different ordering? Can anyone think of any potential problem that
might lead to a real data loss, e.g. when recovering an EC pg?
The cluster will eventually be migrated to the bluestore only, but the
migration process is long.
On Mon, Jul 06, 2020 at 01:33:03PM +0300, Mykola Golub wrote:
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola
Golub wrote:
Looking at the cases we have it happens for
objects that have the same
hash. When building the sorting key the bluestore escapes the object
name string, while the filestore sorts with the raw object name, which
may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the
bluestor escapes the string incorrectly, which results in the order of
escaped strings is different than the order of not escaped ones.
But we can't just fix the bluestore escape function as the current
keys in db will become invalid.
[1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the
filestore [2]. Though using this solution would fix a cluster of mixed
bluestore and upgraded filestore osds but introduce the same issue for
a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174
[2] https://github.com/ceph/ceph/pull/35938
--
Mykola Golub 
10 Jul
10 Jul
3:04 a.m.
Yes, inconsistent listing order would potentially cause backfill to
skip objects.
-Sam
On Thu, Jul 9, 2020 at 9:01 AM Josh Durgin <jdurgin(a)redhat.com> wrote:
There may also be issues with backfill listing objects in different
orders across primary and secondary shards. Sam, would this cause
correctness issues like potentially missing objects?
Josh
On 7/6/20 11:14 PM, Mykola Golub wrote:
And one more question. Is the issue with
deep-scrub incorrectly
reporting missing shards the only problem that may be caused by
different ordering? Can anyone think of any potential problem that
might lead to a real data loss, e.g. when recovering an EC pg?
The cluster will eventually be migrated to the bluestore only, but the
migration process is long.
On Mon, Jul 06, 2020 at 01:33:03PM +0300, Mykola Golub wrote:
On Mon, Jul 06, 2020 at 09:13:58AM +0300, Mykola
Golub wrote:
Looking at the cases we have it happens for
objects that have the same
hash. When building the sorting key the bluestore escapes the object
name string, while the filestore sorts with the raw object name, which
may result in different order.
I left a note to the tracker ticket [1]. Actually it seems the
bluestor escapes the string incorrectly, which results in the order of
escaped strings is different than the order of not escaped ones.
But we can't just fix the bluestore escape function as the current
keys in db will become invalid.
[1] https://tracker.ceph.com/issues/43174#note-10
I pushed a PR for review that changes the sorting order for the
filestore [2]. Though using this solution would fix a cluster of mixed
bluestore and upgraded filestore osds but introduce the same issue for
a cluster of mixed old version and new version filestore osds.
[1] https://tracker.ceph.com/issues/43174
[2] https://github.com/ceph/ceph/pull/35938
--
Mykola Golub
1409
days inactive
1413
days old
4 comments
3 participants
participants (3)
-
Josh Durgin -
Mykola Golub -
Sam Just