Hi Eugen.
how much changes do you see? About 1 new map every 5-10 seconds. The time interval varies.
but I would first try to find out what exactly is causing them That's what I'm trying to do. However, I'm out of ideas what to look for. I followed all the cases I could find to no avail. The leader just says that there is a new map, nothing else:
2024-11-18T15:57:37.147396+0100 mon.ceph-01 (mon.0) 3081080 : cluster [DBG] osdmap e3075180: 1317 total, 1308 up, 1308 in The MGRs are equally shy: 2024-11-18T15:51:18.159+0100 7fdfc9437700 0 [progress INFO root] Processing OSDMap change 3075139..3075140 I have no clue what else to look for and where. The cluster is re-balancing at the moment, so there are actual osdmap changes along the way. However, the vast majority of map changes has empty diff and must have a different reason. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Monday, November 18, 2024 4:38 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: constant increase in osdmap epoch Hi Frank, do you use snapshots a lot? Purging snaps would create a new osdmap as well. Have you checked debug logs of the mon leader to see what triggers the osdmap change? I see in our moderately used Pacific cluster "only" around 60 osdmap changes per day (I haven't looked too deep yet), in a much busier customer cluster I see around 200 epoch changes per day, how much changes do you see? I guess you could play with the pruning configs (many of the defaults don't necessarily fit a production load), but I would first try to find out what exactly is causing them. Regards, Eugen Zitat von Frank Schilder <frans@dtu.dk>:
Hi all,
we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html).
We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs:
# diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 ---
epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100
modified 2024-11-18T15:38:47.858092+0100
This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html). Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning.
Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional.
I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 .
I can't find a resolution anywhere. Any help would be very much appreciated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io