MDS Upgrade from 17.2.5 to 17.2.6 not possible
Hi all, we did a major update from Pacific to Quincy (17.2.5) a month ago without any problems. Now we have tried a minor update from 17.2.5 to 17.2.6 (ceph orch upgrade). It stucks at mds upgrade phase. At this point the cluster tries to scale down mds (ceph fs set max_mds 1). We waited a few hours. We are running two active mds with 1 standby. No subdir pinning configured. CephFS data pool: 575 TB While Upgrading, Rank 1 MDS remains in state stopping. During this state clients are not able to reconnect. So we paused this upgrade and set max_mds to 2 back again and fail rank 1. After that, standby becomes active. In the mds (rank 1 in stopping state) logs we can see: waiting for strays to migrate In our second try, we have evicted all clients first without success. We make daily snapshots on / and rotate them via snapshot scheduler after one week. Is there a way to get rid of stray entries without scale down mds or do we have to wait longer? We had about the same amount of strays before we did the major upgrade. So, it is a bit curious. Current output from ceph perf dump Rank0: "num_strays": 417304, "num_strays_delayed": 3, "num_strays_enqueuing": 0, "strays_created": 567879, "strays_enqueued": 561803, "strays_reintegrated": 13751, "strays_migrated": 4, Rank1: ceph daemon mds.fdi-cephfs.ceph-service-13.rwdkqs perf dump | grep stray "num_strays": 172528, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 418365, "strays_enqueued": 396142, "strays_reintegrated": 67406, "strays_migrated": 4, Any help would be appreciated. best regards Henning
Hi Henning, On Wed, May 17, 2023 at 9:25 PM Henning Achterrath <achhen@uni-bonn.de> wrote:
Hi all,
we did a major update from Pacific to Quincy (17.2.5) a month ago without any problems.
Now we have tried a minor update from 17.2.5 to 17.2.6 (ceph orch upgrade). It stucks at mds upgrade phase. At this point the cluster tries to scale down mds (ceph fs set max_mds 1). We waited a few hours.
We are running two active mds with 1 standby. No subdir pinning configured. CephFS data pool: 575 TB
While Upgrading, Rank 1 MDS remains in state stopping. During this state clients are not able to reconnect. So we paused this upgrade and set max_mds to 2 back again and fail rank 1. After that, standby becomes active.
In the mds (rank 1 in stopping state) logs we can see: waiting for strays to migrate
mds.1 on shutdown will export its strays to mds.0 - this is expected.
In our second try, we have evicted all clients first without success.
We make daily snapshots on / and rotate them via snapshot scheduler after one week.
Is there a way to get rid of stray entries without scale down mds or do we have to wait longer?
Do you see the perf counters related to strays (esp. strays_migrated) increasing for mds.1? If those are not changing, then the stray export has probably hung - which could be due to a bug. If you see this, could you send back the logs for both ranks? (assuming those are debug logs).
We had about the same amount of strays before we did the major upgrade. So, it is a bit curious.
Current output from ceph perf dump
Rank0:
"num_strays": 417304, "num_strays_delayed": 3, "num_strays_enqueuing": 0, "strays_created": 567879, "strays_enqueued": 561803, "strays_reintegrated": 13751, "strays_migrated": 4,
Rank1:
ceph daemon mds.fdi-cephfs.ceph-service-13.rwdkqs perf dump | grep stray
"num_strays": 172528, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 418365, "strays_enqueued": 396142, "strays_reintegrated": 67406, "strays_migrated": 4,
Any help would be appreciated.
best regards Henning _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Cheers, Venky
Hi Venky, thank you for your help. We managed to shut down mds.1: We set "ceph fs set max_mds 1" and waited for about 30 minutes. In the first couple minutes, strays were migrated from mds.1 to mds.0. After this, the stray export hung. The mds.1 remained in the state_stopping. After about 30 minutes, we restarted mds.1. This resulted in one active mds and two standby mds. However, we are not sure, if the remaining strays could be migrated. When we had a closer look at the perf counter of the mds, we realized that the number of strays_enqueued is quite high and constantly increasing. Is this to be expected? What does the counter "strays_enqueued" mean in detail? ceph daemon mds.0 perf dump | grep stray "num_strays": 49846, "num_strays_delayed": 21, "num_strays_enqueuing": 0, "strays_created": 2042124, "strays_enqueued": 2396076, "strays_reintegrated": 44207, "strays_migrated": 38, Would it be safe to perform "ceph orch upgrade resume" at this point? At the moment, the MONs and OSDs are running 17.2.6, while the MDSs and RGWs are running 17.2.5. So we have to upgrade the MDS and RGW eventually. Best, Tobias
In addition, i would like to mention that the number of "strays_created" also increases after this action, but the number of num_strays is lower now. If desired, we can provide debug logs from mds at the time the mds was in stopping state and we did a systemctl restart mds1. The only active mds server has a ram usage of about 50G. The memory limit is 32G, but we get no warnings about that. Maybe the separate purge_queue is consuming a lot of RAM and it does not count for the limit? Usually we get notified when the mds is behind the memory limit. thank you On 22.05.23 15:23, T.Kulschewski@uni-bonn.de wrote:
Hi Venky,
thank you for your help. We managed to shut down mds.1: We set "ceph fs set max_mds 1" and waited for about 30 minutes. In the first couple minutes, strays were migrated from mds.1 to mds.0. After this, the stray export hung. The mds.1 remained in the state_stopping. After about 30 minutes, we restarted mds.1. This resulted in one active mds and two standby mds. However, we are not sure, if the remaining strays could be migrated.
When we had a closer look at the perf counter of the mds, we realized that the number of strays_enqueued is quite high and constantly increasing. Is this to be expected? What does the counter "strays_enqueued" mean in detail?
ceph daemon mds.0 perf dump | grep stray "num_strays": 49846, "num_strays_delayed": 21, "num_strays_enqueuing": 0, "strays_created": 2042124, "strays_enqueued": 2396076, "strays_reintegrated": 44207, "strays_migrated": 38,
Would it be safe to perform "ceph orch upgrade resume" at this point? At the moment, the MONs and OSDs are running 17.2.6, while the MDSs and RGWs are running 17.2.5. So we have to upgrade the MDS and RGW eventually.
Best, Tobias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello again, In two days, the number has increased by about one and a half million and the ram usage of mds remains high by about 50G. We are very unsure if this is a normal behavior. Today: "num_strays": 53695, "num_strays_delayed": 4, "num_strays_enqueuing": 0, "strays_created": 3618390, "strays_enqueued": 3943542, "strays_reintegrated": 144545, "strays_migrated": 38, On 22.05.23 ceph daemon mds.0 perf dump | grep stray "num_strays": 49846, "num_strays_delayed": 21, "num_strays_enqueuing": 0, "strays_created": 2042124, "strays_enqueued": 2396076, "strays_reintegrated": 44207, "strays_migrated": 38, Maybe someone can explain to us what these counters mean in detail. The perf schema is not very revealing. Our idea is to add a standbye-replay (hot-standbye mds) temporary, to ensure the journal is replayable before we resume the upgrade. I would be grateful for any advise. best regards Henning On 23.05.23 17:24, Henning Achterrath wrote:
In addition, i would like to mention that the number of "strays_created" also increases after this action, but the number of num_strays is lower now. If desired, we can provide debug logs from mds at the time the mds was in stopping state and we did a systemctl restart mds1.
The only active mds server has a ram usage of about 50G. The memory limit is 32G, but we get no warnings about that. Maybe the separate purge_queue is consuming a lot of RAM and it does not count for the limit? Usually we get notified when the mds is behind the memory limit.
thank you
On 22.05.23 15:23, T.Kulschewski@uni-bonn.de wrote:
Hi Venky,
thank you for your help. We managed to shut down mds.1: We set "ceph fs set max_mds 1" and waited for about 30 minutes. In the first couple minutes, strays were migrated from mds.1 to mds.0. After this, the stray export hung. The mds.1 remained in the state_stopping. After about 30 minutes, we restarted mds.1. This resulted in one active mds and two standby mds. However, we are not sure, if the remaining strays could be migrated.
When we had a closer look at the perf counter of the mds, we realized that the number of strays_enqueued is quite high and constantly increasing. Is this to be expected? What does the counter "strays_enqueued" mean in detail?
ceph daemon mds.0 perf dump | grep stray "num_strays": 49846, "num_strays_delayed": 21, "num_strays_enqueuing": 0, "strays_created": 2042124, "strays_enqueued": 2396076, "strays_reintegrated": 44207, "strays_migrated": 38,
Would it be safe to perform "ceph orch upgrade resume" at this point? At the moment, the MONs and OSDs are running 17.2.6, while the MDSs and RGWs are running 17.2.5. So we have to upgrade the MDS and RGW eventually.
Best, Tobias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Henning, I think the increasing strays_created is normal. This is a counter that is monotonically increasing when any file is deleted. And is only reset when the MDS is restarted. The num_strays is the actual number of strays in your system, and they are not necessarily reside in memory. Weiwen Hu
在 2023年5月24日，20:22，Henning Achterrath <achhen@uni-bonn.de> 写道：
Hello again,
In two days, the number has increased by about one and a half million and the ram usage of mds remains high by about 50G. We are very unsure if this is a normal behavior.
Today: "num_strays": 53695, "num_strays_delayed": 4, "num_strays_enqueuing": 0, "strays_created": 3618390, "strays_enqueued": 3943542, "strays_reintegrated": 144545, "strays_migrated": 38,
On 22.05.23
ceph daemon mds.0 perf dump | grep stray "num_strays": 49846, "num_strays_delayed": 21, "num_strays_enqueuing": 0, "strays_created": 2042124, "strays_enqueued": 2396076, "strays_reintegrated": 44207, "strays_migrated": 38,
Maybe someone can explain to us what these counters mean in detail. The perf schema is not very revealing.
Our idea is to add a standbye-replay (hot-standbye mds) temporary, to ensure the journal is replayable before we resume the upgrade.
I would be grateful for any advise.
best regards Henning
On 23.05.23 17:24, Henning Achterrath wrote: In addition, i would like to mention that the number of "strays_created" also increases after this action, but the number of num_strays is lower now. If desired, we can provide debug logs from mds at the time the mds was in stopping state and we did a systemctl restart mds1. The only active mds server has a ram usage of about 50G. The memory limit is 32G, but we get no warnings about that. Maybe the separate purge_queue is consuming a lot of RAM and it does not count for the limit? Usually we get notified when the mds is behind the memory limit. thank you
On 22.05.23 15:23, T.Kulschewski@uni-bonn.de wrote: Hi Venky,
thank you for your help. We managed to shut down mds.1: We set "ceph fs set max_mds 1" and waited for about 30 minutes. In the first couple minutes, strays were migrated from mds.1 to mds.0. After this, the stray export hung. The mds.1 remained in the state_stopping. After about 30 minutes, we restarted mds.1. This resulted in one active mds and two standby mds. However, we are not sure, if the remaining strays could be migrated.
When we had a closer look at the perf counter of the mds, we realized that the number of strays_enqueued is quite high and constantly increasing. Is this to be expected? What does the counter "strays_enqueued" mean in detail?
ceph daemon mds.0 perf dump | grep stray "num_strays": 49846, "num_strays_delayed": 21, "num_strays_enqueuing": 0, "strays_created": 2042124, "strays_enqueued": 2396076, "strays_reintegrated": 44207, "strays_migrated": 38,
Would it be safe to perform "ceph orch upgrade resume" at this point? At the moment, the MONs and OSDs are running 17.2.6, while the MDSs and RGWs are running 17.2.5. So we have to upgrade the MDS and RGW eventually.
Best, Tobias _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks. In the meantime we were able to narrow down the cause of the RAM consumption a little. ceph mds cache status shows, that the cache is within the limit (32G): { "pool": { "items": 758820483, "bytes": 32642572344 } } The remaining memory belongs to buffer_anon: ceph daemon mds.fdi-cephfs.ceph-service-10.lajqwl dump_mempools buffer_anon": { "items": 3841090, "bytes": 12331436210 Any idea what is that related to? Maybe the StrayManager. I can not find any information about buffer_anon. Regards Henning
On Wed, May 17, 2023 at 9:26 PM Henning Achterrath <achhen@uni-bonn.de> wrote:
Hi all,
we did a major update from Pacific to Quincy (17.2.5) a month ago without any problems.
Now we have tried a minor update from 17.2.5 to 17.2.6 (ceph orch upgrade). It stucks at mds upgrade phase. At this point the cluster tries to scale down mds (ceph fs set max_mds 1). We waited a few hours.
Just an FYI (if you use cephadm to carry out upgrades), having max_mds 1 can be disastrous (especially for huge CephFS deployments) because cluster cannot quickly reduce active MDSs to 1 and a single active MDS cannot easily handle the load of all clients. To overcome this, you can upgrade MDSs without reducing max_mds, the fail_fs option can to be set to true prior to the upgrade. There's a note in the beginning of the "STARTING THE UPGRADE" section that might be helpful to understand this better. https://docs.ceph.com/en/latest/cephadm/upgrade/#starting-the-upgrade
We are running two active mds with 1 standby. No subdir pinning configured. CephFS data pool: 575 TB
While Upgrading, Rank 1 MDS remains in state stopping. During this state clients are not able to reconnect. So we paused this upgrade and set max_mds to 2 back again and fail rank 1. After that, standby becomes active. In the mds (rank 1 in stopping state) logs we can see: waiting for strays to migrate
In our second try, we have evicted all clients first without success.
We make daily snapshots on / and rotate them via snapshot scheduler after one week.
Is there a way to get rid of stray entries without scale down mds or do we have to wait longer?
We had about the same amount of strays before we did the major upgrade. So, it is a bit curious.
Current output from ceph perf dump
Rank0:
"num_strays": 417304, "num_strays_delayed": 3, "num_strays_enqueuing": 0, "strays_created": 567879, "strays_enqueued": 561803, "strays_reintegrated": 13751, "strays_migrated": 4,
Rank1:
ceph daemon mds.fdi-cephfs.ceph-service-13.rwdkqs perf dump | grep stray
"num_strays": 172528, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 418365, "strays_enqueued": 396142, "strays_reintegrated": 67406, "strays_migrated": 4,
Any help would be appreciated.
best regards Henning _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
achhen＠uni-bonn.de
-
Dhairya Parmar
-
Henning Achterrath
-
T.Kulschewski＠uni-bonn.de
-
Venky Shankar
-
胡 玮文