On Thu, Jul 25, 2024 at 10:10 PM Dan O'Brien <dobrie2@gmu.edu> wrote:
Ilya -
I don't think images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b is the problem; it was just the last RBD image listed in the log before the crash. The commands you suggested work fine when using that image:
[root@os-storage ~]# rbd info images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b rbd image '144ebab3-b2ee-4331-9d41-8505bcc4e19b': size 0 B in 0 objects order 23 (8 MiB objects) snapshot_count: 1 id: f01052f76969e7 block_name_prefix: rbd_data.f01052f76969e7 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Feb 12 17:50:54 2024 access_timestamp: Mon Feb 12 17:50:54 2024 modify_timestamp: Mon Feb 12 17:50:54 2024 [root@os-storage ~]# rbd diff --whole-object images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b
I'm sorry, I meant "rbd du images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b". I suspect that you are hitting [1]. One workaround would be to go through all images in all RBD pools that you have and remove any of them that are 0-sized, meaning that "rbd info" reports "size 0 B in 0 objects".
The other 2 images, related to 2 OpenStack volumes stuck in "error_deleting" state, appear to be the cause of the problem:
[root@os-storage ~]# rbd info volumes-gpu/volume-28bbca8c-fec5-4a33-bbe2-30408f1ea37f rbd: error opening image volume-28bbca8c-fec5-4a33-bbe2-30408f1ea37f: (2) No such file or directory
[root@os-storage ~]# rbd diff --whole-object volumes-gpu/volume-28bbca8c-fec5-4a33-bbe2-30408f1ea37f rbd: error opening image volume-28bbca8c-fec5-4a33-bbe2-30408f1ea37f: (2) No such file or directory
I don't think these ENOENT errors are related -- the image isn't there so I don't see a way for the assert to be reached. [1] https://tracker.ceph.com/issues/66418 Thanks, Ilya