On Wed, May 17, 2023 at 9:26 PM Henning Achterrath <achhen@uni-bonn.de> wrote:
Hi all,
we did a major update from Pacific to Quincy (17.2.5) a month ago without any problems.
Now we have tried a minor update from 17.2.5 to 17.2.6 (ceph orch upgrade). It stucks at mds upgrade phase. At this point the cluster tries to scale down mds (ceph fs set max_mds 1). We waited a few hours.
Just an FYI (if you use cephadm to carry out upgrades), having max_mds 1 can be disastrous (especially for huge CephFS deployments) because cluster cannot quickly reduce active MDSs to 1 and a single active MDS cannot easily handle the load of all clients. To overcome this, you can upgrade MDSs without reducing max_mds, the fail_fs option can to be set to true prior to the upgrade. There's a note in the beginning of the "STARTING THE UPGRADE" section that might be helpful to understand this better. https://docs.ceph.com/en/latest/cephadm/upgrade/#starting-the-upgrade
We are running two active mds with 1 standby. No subdir pinning configured. CephFS data pool: 575 TB
While Upgrading, Rank 1 MDS remains in state stopping. During this state clients are not able to reconnect. So we paused this upgrade and set max_mds to 2 back again and fail rank 1. After that, standby becomes active. In the mds (rank 1 in stopping state) logs we can see: waiting for strays to migrate
In our second try, we have evicted all clients first without success.
We make daily snapshots on / and rotate them via snapshot scheduler after one week.
Is there a way to get rid of stray entries without scale down mds or do we have to wait longer?
We had about the same amount of strays before we did the major upgrade. So, it is a bit curious.
Current output from ceph perf dump
Rank0:
"num_strays": 417304, "num_strays_delayed": 3, "num_strays_enqueuing": 0, "strays_created": 567879, "strays_enqueued": 561803, "strays_reintegrated": 13751, "strays_migrated": 4,
Rank1:
ceph daemon mds.fdi-cephfs.ceph-service-13.rwdkqs perf dump | grep stray
"num_strays": 172528, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 418365, "strays_enqueued": 396142, "strays_reintegrated": 67406, "strays_migrated": 4,
Any help would be appreciated.
best regards Henning _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io