22 Apr
2021
22 Apr
'21
1:16 p.m.
Hi,
I would like to bring some attention to a problem we have been
observing with nautilus, and which I reported here [1].
If a pg is in backfill_unfound state ("unfound" objects were detected
during backfill), and one of the osds from the active set is restarted
the state changes to clean, losing the information about unfound
objects.
And when I tired to reproduce the issue on the master with the same
scenario, the status did not change, but I was observing the primary
osd crash after a non-primary restart.
I looked through the commit log and did not find a commit explicitely
saying (or giving a hint) this problem was adressing in the master and
I see there was large refactoring in the related code since
nautilus. So probably the issue was "solved" during refactoring?
We would love to see the problem fixed in the nautilus, and I would
like to backport the "fix", but right now I don't have a clear
understanding if there really was a fix in the master and what to do
with that crash that may be related to the "fix".
I might try to find the commit that changed the behaviour by
bisecting, but this looks like a long way, so I want to ask here first
if anybody has a hint.
[1] https://tracker.ceph.com/issues/50351
Thanks,
--
Mykola Golub
26 Apr
26 Apr
9:22 a.m.
Hi,
I think this is a big problem. The reason is as follows.
1: unfound becomes `active + clean`
In fact, it looks normal even though unfound hasn't been resolved.
This eliminates a chance for users to notice data loss.
2: primary osd crash
This is more serious. User can notice the anomaly, but it is unclear if the data can be
recovered successfully.
The following tickets have low priority and severity, but I think they should be raised,
and need to discuss more detail.
https://tracker.ceph.com/issues/50351
1 May
1 May
5:29 p.m.
On Thu, Apr 22, 2021 at 04:16:34PM +0300, Mykola Golub wrote:
I would like to bring some attention to a problem we
have been
observing with nautilus, and which I reported here [1].
If a pg is in backfill_unfound state ("unfound" objects were detected
during backfill), and one of the osds from the active set is restarted
the state changes to clean, losing the information about unfound
objects.
And when I tired to reproduce the issue on the master with the same
scenario, the status did not change, but I was observing the primary
osd crash after a non-primary restart.
Ok. Now I seem to have better understanding what is going on here.
As I wrote in [1], when `PrimaryLogPG::on_failed_pull` is called when
the object is not found on the backfill source osd, the oid is removed
from `backfills_in_flight` only if the backfill source is primary [2].
In our case we are backfilling a non-primary EC shard, so the oid is
not removed from `backfills_in_flight`. And later it causes the
assertion failure in `PrimaryLogPG::_clear_recovery_state`.
The behavior seemed to be changed during post-nautilus refactoring, in
[3]. Previously for the EC backend the oid was removed from
`backfills_in_flight` unconditionally, and now it is removed only if
the source is primary.
In [1] I questioned this change, but after investigating how it works,
now it looks quite reasonable to me.
So, the current behavior is: In `PrimaryLogPG::recover_backfill`, due
to the "unfound" oid is not removed from `backfills_in_flight`,
`next_backfill_to_complete` is always set to the "unfound" oid [4],
and `new_last_backfill` is not updated any more pointing to the object
before the "unfound" oid. The backfill still continues and terminates
only after all objects are pulled/pushed, but "complete" position
remains on the object before "unfound". After the backfill is finished
the pg enters "backfill_unfound" state. When the pg is re-peered
(e.g. after restarting an osd) it enters "backfilling" state starting
the backfill from "unfound" oid position, detects the "unfound"
object
again, scans the remaining objects detecting they are already copied,
and enters "backfill_unfound" state again with the same "complete"
position on the "unfound" object.
This looks like a reasonable behavoir to me, and the only problem is
that reported assertion failure, which probably is just needed to be
removed?
In Nautilus, because the "unfound" oid is removed from
`backfills_in_flight`, the "complete" position is not stopped on this
oid, and when the backfill is finished it also enters
"backfill_unfound" state, but "complete" backfill postion is at the
end now. So when the pg is re-peered, the backfill is not re-started
from "unfound" position, the "unfound" object is not detected and the
pg enters "clean" state.
If my understanding is correct, it looks like we have to:
1) in master, fix the assertion failure, probably by just removing the
assertion, and backport the fix.
2) in nautilus (direct commit), make the EC backend not remove
"unfound" oid from `backfills_in_flight` to have post-nautilus
behavior.
Does it make sense?
[1] https://tracker.ceph.com/issues/50351#note-1
[2]
https://github.com/ceph/ceph/blob/813933f81e3d682a0b1ae6dd906e38e78c4859a4/…
[3] https://github.com/ceph/ceph/commit/8a8947d2a32d6390cb17099398e7f2212660c9a1
[4]
https://github.com/ceph/ceph/blob/813933f81e3d682a0b1ae6dd906e38e78c4859a4/…
--
Mykola Golub
7 May
7 May
6:55 a.m.
Hi,
This problem also happened in my customer's environment, so I want to solve this
problem.
To facilitate the discussion, I restate the problem and the current solution.
(Mykola has already written the solution idea. I am sorry if there is anything different
from Mykola's idea.)
In master:
Problem: A primary OSD crashes in an unnecessary situation. (I think this is a bug.)
Solution: Remove the ceph_assert from the code below.
---------------------------------------------------------------
diff --git a/src/osd/PrimaryLogPG.cc b/src/osd/PrimaryLogPG.cc
index 626e8ccefb..12956424bd 100644
--- a/src/osd/PrimaryLogPG.cc
+++ b/src/osd/PrimaryLogPG.cc
@@ -13079,7 +13079,6 @@ void PrimaryLogPG::_clear_recovery_state()
last_backfill_started = hobject_t();
set<hobject_t>::iterator i = backfills_in_flight.begin();
while (i != backfills_in_flight.end()) {
- ceph_assert(recovering.count(*i));
backfills_in_flight.erase(i++);
}
---------------------------------------------------------------
The reason is as follows.
- The above code assumes that all of the objects contained in backfills_in_flight are
contained in recovering.
- However, the current implementation of on_failed_pull[1], if it is non-primary OSD,
unfound objects will remain only in backfills_in_flight. (but unconditionally removed
from
recovering[2])
Therefore, the above ceph_assert does not match the current implementation of
on_failed_pull.
I thinks this ceph_assert should be removed, but I would like to hear opinion from the
community.
[1]:
https://github.com/ceph/ceph/blob/813933f81e3d682a0b1ae6dd906e38e78c4859a4/…
[2]:
https://github.com/ceph/ceph/blob/813933f81e3d682a0b1ae6dd906e38e78c4859a4/…
In nautilus:
Problem: backfill_unfound state becomes clear when the OSD is restarted. (This is also a
bug.)
This causes a user to mistakenly think the problem has been solved and cause unexpected
trouble.
Solution: Remain unfound objects in backfills_in_flight such as on_failed_pull, if it is
non-primary OSD.
There is the following commit[3], but as the range of correction of this commit is wide,
so I think only the minimum correction necessary for problem solving should be directly
committed to nautilus.
[3]: https://github.com/ceph/ceph/commit/8a8947d2a32d6390cb17099398e7f2212660c9a1
In addition, if this problem is solved, the problem that primary OSD crashes occurs, so
the commit of the master described above needs to be backported.
I am considering sending PRs next week, so please let me know if you have any opinions
from the community before that.
--
Jin
12 May
12 May
7:10 a.m.
On Sat, May 01, 2021 at 08:29:48PM +0300, Mykola Golub wrote:
On Thu, Apr 22, 2021 at 04:16:34PM +0300, Mykola Golub
wrote:
https://github.com/ceph/ceph/pull/41270
I would like to bring some attention to a problem
we have been
observing with nautilus, and which I reported here [1].
If a pg is in backfill_unfound state ("unfound" objects were detected
during backfill), and one of the osds from the active set is restarted
the state changes to clean, losing the information about unfound
objects.
And when I tired to reproduce the issue on the master with the same
scenario, the status did not change, but I was observing the primary
osd crash after a non-primary restart.
Ok. Now I seem to have better understanding what is going on here.
As I wrote in [1], when `PrimaryLogPG::on_failed_pull` is called when
the object is not found on the backfill source osd, the oid is removed
from `backfills_in_flight` only if the backfill source is primary [2].
In our case we are backfilling a non-primary EC shard, so the oid is
not removed from `backfills_in_flight`. And later it causes the
assertion failure in `PrimaryLogPG::_clear_recovery_state`.
The behavior seemed to be changed during post-nautilus refactoring, in
[3]. Previously for the EC backend the oid was removed from
`backfills_in_flight` unconditionally, and now it is removed only if
the source is primary.
In [1] I questioned this change, but after investigating how it works,
now it looks quite reasonable to me.
So, the current behavior is: In `PrimaryLogPG::recover_backfill`, due
to the "unfound" oid is not removed from `backfills_in_flight`,
`next_backfill_to_complete` is always set to the "unfound" oid [4],
and `new_last_backfill` is not updated any more pointing to the object
before the "unfound" oid. The backfill still continues and terminates
only after all objects are pulled/pushed, but "complete" position
remains on the object before "unfound". After the backfill is finished
the pg enters "backfill_unfound" state. When the pg is re-peered
(e.g. after restarting an osd) it enters "backfilling" state starting
the backfill from "unfound" oid position, detects the "unfound"
object
again, scans the remaining objects detecting they are already copied,
and enters "backfill_unfound" state again with the same "complete"
position on the "unfound" object.
This looks like a reasonable behavoir to me, and the only problem is
that reported assertion failure, which probably is just needed to be
removed?
In Nautilus, because the "unfound" oid is removed from
`backfills_in_flight`, the "complete" position is not stopped on this
oid, and when the backfill is finished it also enters
"backfill_unfound" state, but "complete" backfill postion is at the
end now. So when the pg is re-peered, the backfill is not re-started
from "unfound" position, the "unfound" object is not detected and
the
pg enters "clean" state.
If my understanding is correct, it looks like we have to:
1) in master, fix the assertion failure, probably by just removing the
assertion, and backport the fix.
2) in nautilus (direct commit), make the EC backend not remove
"unfound" oid from `backfills_in_flight` to have post-nautilus
behavior.
https://github.com/ceph/ceph/pull/41293
Does it make sense?
[1] https://tracker.ceph.com/issues/50351#note-1
[2]
https://github.com/ceph/ceph/blob/813933f81e3d682a0b1ae6dd906e38e78c4859a4/…
[3] https://github.com/ceph/ceph/commit/8a8947d2a32d6390cb17099398e7f2212660c9a1
[4]
https://github.com/ceph/ceph/blob/813933f81e3d682a0b1ae6dd906e38e78c4859a4/…
--
Mykola Golub
--
Mykola Golub
17 May
17 May
4:37 p.m.
On Wed, May 12, 2021 at 10:10:45AM +0300, Mykola Golub wrote:
This PR is merged. Thanks! And "backfill_unfound" state is not reset
after a non-primary OSD restart. But it is still possible to reset the
state to clean when the primary osd is restarted.
I will re-describe the situation.
Suppose we have a 2+1 EC pool, and an object is missing 2 shards on
both non-primary osds. We initiate backfill by setting a non-primary
osd out. During the backfill the primary osd detects the missing
shards and the pg enters "backfill_unfound" state, the last_backfill
position is properly set to the object before the "unfound" (in
post-nautilus, for nautilus I opened [1] to make it work). If
re-peering occurs due to a non-primary osd is restarted, the backfill
is restarted from the last_backfill position and the "unfound" object
is detected again. But if re-peering occurs due the primary osd is
temporarily stopped (restarted), another non-primary osd becomes
primary and "drives" the backfill from the last_backfill position, and
as the shard is missing here it is just skipped from the backfill, the
missing object is not detected and the pg enters clean state.
Is there something that can/should be improved here? It is rather
unfortunate that the information about missing object is lost on the
restart (until scrub or next backfill). On the other hand the
situation when we have many shards are missing for an object is rather
unlikely. Also, if for example it happened that the shard was missing
on the primary it would not even be detected on backfill.
[1] https://github.com/ceph/ceph/pull/41293
--
Mykola Golub
So, the
current behavior is: In `PrimaryLogPG::recover_backfill`, due
to the "unfound" oid is not removed from `backfills_in_flight`,
`next_backfill_to_complete` is always set to the "unfound" oid [4],
and `new_last_backfill` is not updated any more pointing to the object
before the "unfound" oid. The backfill still continues and terminates
only after all objects are pulled/pushed, but "complete" position
remains on the object before "unfound". After the backfill is finished
the pg enters "backfill_unfound" state. When the pg is re-peered
(e.g. after restarting an osd) it enters "backfilling" state starting
the backfill from "unfound" oid position, detects the "unfound"
object
again, scans the remaining objects detecting they are already copied,
and enters "backfill_unfound" state again with the same "complete"
position on the "unfound" object.
This looks like a reasonable behavoir to me, and the only problem is
that reported assertion failure, which probably is just needed to be
removed?
https://github.com/ceph/ceph/pull/41270
19 May
19 May
9:39 a.m.
Suppose we have a 2+1 EC pool, and an object is
missing 2 shards on
both non-primary osds. We initiate backfill by setting a non-primary
osd out. During the backfill the primary osd detects the missing
shards and the pg enters "backfill_unfound" state, the last_backfill
position is properly set to the object before the "unfound" (in
post-nautilus, for nautilus I opened [1] to make it work). If
re-peering occurs due to a non-primary osd is restarted, the backfill
is restarted from the last_backfill position and the "unfound" object
is detected again. But if re-peering occurs due the primary osd is
temporarily stopped (restarted), another non-primary osd becomes
primary and "drives" the backfill from the last_backfill position, and
as the shard is missing here it is just skipped from the backfill, the
missing object is not detected and the pg enters clean state.
Is there something that can/should be improved here? It is rather
unfortunate that the information about missing object is lost on the
restart (until scrub or next backfill). On the other hand the
situation when we have many shards are missing for an object is rather
unlikely. Also, if for example it happened that the shard was missing
on the primary it would not even be detected on backfill.
[1] https://github.com/ceph/ceph/pull/41293
In the case of primary osd, is there a case where the user wants to reset the state (from
unfound state)?
If we fix this behavior, is there another problem because we can't reset the state?
--
Jin
10:21 a.m.
On Wed, May 19, 2021 at 09:39:08AM -0000, Jin Hase wrote:
I am not sure I quite understand your question. Anyway trying to
answer.
It is not that someone "wants" to reset the "backfill_unfound" state.
The pg state machine can enter "backfill_unfound" state onfly after
"backfilling" state, if "missing" objects are detected during
backfill.
Now, when the pg is peering (e.g. after one of osds is stopped) and is
"finding out" its state, on activating step it calls
`needs_backfill()` function [1] to check if it needs to enter the
backfilling state. And if it needs (when last_backfill position is
not MAX for one of backfill targets) it starts backfilling, detects a
"missing" object (if there is one) and enters "backfill_unfound"
state. And it does it on every pg peering. But if the pg peering is
due to the primary osd change, the new primary osd may not detect the
"missing" object, if its shard happens to be missing on this osd and
the backfill completes with clean state.
This how it works. The current behavior seems "by design" to me, I
don't know if it could (wanted to) be improved. That was actually my
question to the community.
Hope it helps.
[1]
https://github.com/ceph/ceph/blob/5d8d691da2b96981a6d5d11e4c4142a2e08c930b/…
--
Mykola Golub
Suppose we
have a 2+1 EC pool, and an object is missing 2 shards on
both non-primary osds. We initiate backfill by setting a non-primary
osd out. During the backfill the primary osd detects the missing
shards and the pg enters "backfill_unfound" state, the last_backfill
position is properly set to the object before the "unfound" (in
post-nautilus, for nautilus I opened [1] to make it work). If
re-peering occurs due to a non-primary osd is restarted, the backfill
is restarted from the last_backfill position and the "unfound" object
is detected again. But if re-peering occurs due the primary osd is
temporarily stopped (restarted), another non-primary osd becomes
primary and "drives" the backfill from the last_backfill position, and
as the shard is missing here it is just skipped from the backfill, the
missing object is not detected and the pg enters clean state.
Is there something that can/should be improved here? It is rather
unfortunate that the information about missing object is lost on the
restart (until scrub or next backfill). On the other hand the
situation when we have many shards are missing for an object is rather
unlikely. Also, if for example it happened that the shard was missing
on the primary it would not even be detected on backfill.
[1] https://github.com/ceph/ceph/pull/41293
In the case of primary osd, is there a case where the user wants to reset the state (from
unfound state)?
If we fix this behavior, is there another problem because we can't reset the state?
11:12 a.m.
I understand the detail behavior, thanks.
My concerns are:
How to recover backfill_unfound state if this behavior is modified to be detected as
backfill_unfound again even if in the above case . Whether deep scrub can be executed in
the backfill_unfound state and recover this situation. If that doesn't work, I'm
wondering how to address situation.
Anyway, I'd like to know this is expected behavior or not, and how to fix it if this
is not expected behavior.
11:49 a.m.
On Wed, May 19, 2021 at 11:12:06AM -0000, Jin Hase wrote:
I understand the detail behavior, thanks.
My concerns are:
How to recover backfill_unfound state if this behavior
is modified
to be detected as backfill_unfound again even if in the above case.
Whether deep scrub can be executed in the backfill_unfound state
and recover this situation. If that doesn't work, I'm wondering how
to address situation.
I believe the deep scrub will not run when the state is
backfill_unfound. You will need to resolve "unfound" objects first to
change the state. Usually it means just to run
ceph pg $pgid mark_unfound_lost delete
Because you can't do much in this situation -- the object is not
recoverable, just remove the objects remnants. After this the pg will
enter the clean state.
Anyway, I'd like to know this is expected behavior
or not, and how
to fix it if this is not expected behavior.
+1
--
Mykola Golub
1072
days inactive
1099
days old
9 comments
3 participants
participants (3)
-
hase.jin＠fujitsu.com -
Jin Hase -
Mykola Golub