Hi Hüseyin, Did you manage to solve the issue? On Sun, 26 Oct 2025 at 12:29, Eugen Block <eblock@nde.ag> wrote:
Good morning,
if the osdmap doesn't exist on that OSD yet (which obviously it doesn't), you can use the --force flag to create it. I played a bit on a test cluster to try and reproduce. I stopped one OSD at epoch 20990, waited until the epoch increased and tried the same:
[ceph: root@nautilus /]# ceph osd getmap > osdmap-20993.bin got osdmap epoch 20993 [ceph: root@nautilus /]# ceph-objectstore-tool --op set-osdmap --data-path /var/lib/ceph/osd/ceph-0/ --file osdmap-20993.bin osdmap (#-1:9885c172:::osdmap.20993:0#) does not exist.
So I repeated it with the force flag:
[ceph: root@nautilus /]# ceph-objectstore-tool --op set-osdmap --data-path /var/lib/ceph/osd/ceph-0/ --file osdmap-20993.bin --force osdmap (#-1:9885c172:::osdmap.20993:0#) does not exist. Creating a new epoch.
The OSD then started successfully. But nothing was broken in this cluster, so even if that worked for me, there's no guarantee that the force flag will work for you, too. There might be a different underlying issue, maybe you're barking up the wrong tree wrt osdmaps. But I guess it's worth a shot on one OSD.
Did I interpret correctly that the most affected pool would be the EC pool with 11 chunks and min_size 9? And CRUSH can't find 4 of those chunks according to one of those logs, which means rebuilding the OSDs wouldn't work and most likely resulted in data loss. So trying to bring back at least enough of those down OSDs would be my approach as well.
And is there a history that lead to more than 20 down OSDs? That could be helpful to undestand what happened. Was it really just the (successful) upgrade?
Regards, Eugen
Zitat von Huseyin Cotuk <hcotuk@gmail.com>:
Hi again,
By the way, I ran into a similar problem a few years ago and I have set the following config parameter to 5000 some time ago.
osd_map_cache_size < https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#confval-os...
The number of OSD maps to keep cached.
type : int default : 50 I could not find any other related config parameters.
BR, Huseyin Cotuk hcotuk@gmail.com
On 24 Oct 2025, at 21:14, Huseyin Cotuk <hcotuk@gmail.com> wrote:
Hi Eugen,
I have already tried as the below method I stated before. I got the current osdmap and tried to set-osdmap via ceph-object-store-tool. But the command failed with error:
ceph osd getmap 72555 > /tmp/osd_map_72555 CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-11/ --op set-osdmap --file /tmp/osd_map_72555
osdmap (#-1:9c8e9ef2:::osdmap.72555:0#) does not exist.
https://www.mail-archive.com/ceph-users@ceph.io/msg11545.html
BR, Huseyin hcotuk@gmail.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io