Question about object maps and index rebuilding.
We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements. One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment? Thankfully the above does not happen regularly, but we would like to make use of the HA features of proxmox to ensure some VMs are always available. Requiring the rebuild step limits what can be done automatically, and how quickly it can be recovered. Any advice on ceph configuration, or how others may adapt to the requirements in HA situations would be appreciated. Cheers, Gary -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam@uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566
Hi, we work with Openstack and Ceph as well, and we also support customers with such deployments, but in 10 years I haven't had to rebuild any object maps yet, ever. So I'm wondering what exactly you're seeing when you do have to rebuild them. One of our customers has a middle sized cloud (around 40 compute nodes), and I'd say once or twice a year one of them dies. But we never had to rebuild any object maps. Our own Ceph cluster also started with Hammer, I believe, and around that time we also started using Openstack (Kilo maybe?). The customer I mentioned has been working with Openstack and Ceph even longer, long before we came into play. I guess my response is not really helpful to you, but I'm curious what could make the difference here, why we are not experiencing object map issues. Regards, Eugen Zitat von Gary Molenkamp <molenkam@uwo.ca>:
We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements.
One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment?
Thankfully the above does not happen regularly, but we would like to make use of the HA features of proxmox to ensure some VMs are always available. Requiring the rebuild step limits what can be done automatically, and how quickly it can be recovered.
Any advice on ceph configuration, or how others may adapt to the requirements in HA situations would be appreciated.
Cheers, Gary
-- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam@uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Jun 24, 2025 at 11:19 PM Gary Molenkamp <molenkam@uwo.ca> wrote:
We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements.
One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment?
Hi Gary, It's definitely not expected. Have you ever run "rbd object-map check" command and captured its output before rebuilding the object map? Some object map inconsistencies following a hard crash are expected, but they shouldn't be leading to the VM not booting/rootfs not mounting. Thanks, Ilya
Thank you Ilya and Anthony. The clients are all running the Quincy client packages. Given that this is not expected behaviour, I will try to replicate the failure conditions on my dev cluster. I can then run the "rbd object-map check" on the volumes if they fail to boot, and proceed from there. Cheers, Gary On 2025-06-26 09:33, Ilya Dryomov wrote:
On Tue, Jun 24, 2025 at 11:19 PM Gary Molenkamp <molenkam@uwo.ca> wrote:
We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements.
One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment? Hi Gary,
It's definitely not expected. Have you ever run "rbd object-map check" command and captured its output before rebuilding the object map? Some object map inconsistencies following a hard crash are expected, but they shouldn't be leading to the VM not booting/rootfs not mounting.
Thanks,
Ilya
-- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam@uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566
I confirmed and can consistently replicate the failure event that forces the object-map rebuild. If the VM is terminated cleanly, such as a hypervisor reboot, then the VMs and their rbd volumes are all well. If the hypervisor goes down hard, such as a hard power cycle, then any VMs on the hypervisor will have sufficient I/O errors to prevent either a boot or the root volume from mounting. This usually manifests as "I/O error, dev sda, sector ......" The object-map check when this happens appears clean: [root@eda84984a767 /]# rbd object-map check proxmox/vm-179-disk-0 Object Map Check: 100% complete...done. And to confirm, rebuilding the above object-map, then allows the VM to function correctly with no apparent I/O error reports from the OS' kernel. Is there something else I should be checking? Could it be related to the rbd_invalidate_object_map_on_timeout setting on the pool? [root@eda84984a767 /]# rbd config pool list proxmox |grep object rbd_cache_max_dirty_object 0 config rbd_invalidate_object_map_on_timeout true config rbd_journal_max_concurrent_object_sets 0 config rbd_journal_object_flush_age 0.000000 config rbd_journal_object_flush_bytes 1048576 config rbd_journal_object_flush_interval 0 config rbd_journal_object_max_in_flight_appends 0 config rbd_journal_object_writethrough_until_flush true config Cheers, Gary On 2025-06-26 09:33, Ilya Dryomov wrote:
On Tue, Jun 24, 2025 at 11:19 PM Gary Molenkamp <molenkam@uwo.ca> wrote:
We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements.
One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment? Hi Gary,
It's definitely not expected. Have you ever run "rbd object-map check" command and captured its output before rebuilding the object map? Some object map inconsistencies following a hard crash are expected, but they shouldn't be leading to the VM not booting/rootfs not mounting.
Thanks,
Ilya
-- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam@uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566
On Wed, Jul 2, 2025 at 1:36 PM Gary Molenkamp <molenkam@uwo.ca> wrote:
I confirmed and can consistently replicate the failure event that forces the object-map rebuild.
If the VM is terminated cleanly, such as a hypervisor reboot, then the VMs and their rbd volumes are all well. If the hypervisor goes down hard, such as a hard power cycle, then any VMs on the hypervisor will have sufficient I/O errors to prevent either a boot or the root volume from mounting. This usually manifests as "I/O error, dev sda, sector ......"
The object-map check when this happens appears clean: [root@eda84984a767 /]# rbd object-map check proxmox/vm-179-disk-0 Object Map Check: 100% complete...done.
Hi Gary, Sorry, I forgot to specify "--debug-rbd 1" option for "rbd object-map check" command in the previous email. Is proxmox/vm-179-disk-0 a cloned image?
And to confirm, rebuilding the above object-map, then allows the VM to function correctly with no apparent I/O error reports from the OS' kernel.
Since this appears to be easily reproducible, can you grab "rbd info" output, the object listing, run "rbd object-map check --debug-rbd 1" and also extract the object map object for one of the images: a) after powering on the hypervisor but before starting the VM b) after the VM is started and I/O errors are observed c) after running "rbd object-map rebuild" The object listing can be obtained with: $ IMAGE_ID=$(rbd info proxmox/vm-179-disk-0 --format json | jq -r '.id') $ rados -p proxmox ls | grep $IMAGE_ID >objs-a.txt The object map can be extracted with: $ rados -p proxmox get rbd_object_map.$IMAGE_ID objmap-a.bin Attach the resulting outputs and files for a, b and c here or file a tracker ticket, whichever you prefer.
Is there something else I should be checking? Could it be related to the rbd_invalidate_object_map_on_timeout setting on the pool?
Is rados_osd_op_timeout option set? If so, what is the value? To get the most out of the reproducing attempt, it would be great to enable verbose logging before step b) and disable it immediately after. I'm not sure how it's set up in Proxmox, but you would need to do the equivalent of adding debug ms = 1 debug rbd = 20 log file = <some path> log to file = true to ceph.conf file that is picked up by the QEMU process. Thanks, Ilya
Thanks to everyone that replied so far. During my debugging, I discovered that either an 'object-map rebuild' or an 'object-map check' is sufficient to clear the conditions that are preventing the volume from being used properly. Is there something in common that the two command both do that could be affecting the volume? ie clearing locks, state, flush cache, etc. I checked for locks, watchers, etc when the volume was not usable, but nothing evident. Cheers, Gary On 2025-06-26 9:33 a.m., Ilya Dryomov wrote:
On Tue, Jun 24, 2025 at 11:19 PM Gary Molenkamp <molenkam@uwo.ca> wrote:
We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements.
One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment? Hi Gary,
It's definitely not expected. Have you ever run "rbd object-map check" command and captured its output before rebuilding the object map? Some object map inconsistencies following a hard crash are expected, but they shouldn't be leading to the VM not booting/rootfs not mounting.
Thanks,
Ilya
-- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam@uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566
On Thu, Jul 3, 2025 at 7:50 PM Gary Molenkamp <molenkam@uwo.ca> wrote:
Thanks to everyone that replied so far.
During my debugging, I discovered that either an 'object-map rebuild' or an 'object-map check' is sufficient to clear the conditions that are preventing the volume from being used properly. Is there something in common that the two command both do that could be affecting the volume? ie clearing locks, state, flush cache, etc.
Breaking leftover locks would be my first guess. When you run any of these commands, do you use the same user entity as Proxmox uses for QEMU or the default (client.admin)? What is the output of "ceph auth get client.<what is used by Proxmox>" (edit out the base64-encoded key)? It could be that this user entity is missing the permission to blocklist the pre-crash lock owner.
I checked for locks, watchers, etc when the volume was not usable, but nothing evident.
Are you saying that "rbd lock ls" on the image immediately after powering on the hypervisor produces no output? Thanks, Ilya
Cheers, Gary
On 2025-06-26 9:33 a.m., Ilya Dryomov wrote:
On Tue, Jun 24, 2025 at 11:19 PM Gary Molenkamp <molenkam@uwo.ca> wrote:
We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements.
One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment? Hi Gary,
It's definitely not expected. Have you ever run "rbd object-map check" command and captured its output before rebuilding the object map? Some object map inconsistencies following a hard crash are expected, but they shouldn't be leading to the VM not booting/rootfs not mounting.
Thanks,
Ilya
-- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam@uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Gary Molenkamp
-
Ilya Dryomov