Objects not removed (completely) when removing a rbd image
Hello, My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration? br, Xu Yun
Hi, this might happen if you try to delete images/instances/volumes in openstack that are somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too. Deleting a base image worked but there were existing clones so basically just the openstack database was updated, but the base image still existed within ceph. Try to figure out if that is also the case. If it's something else, check the logs in your openstack environment, maybe they reveal something. Also check the ceph logs. Regards, Eugen Zitat von 徐蕴 <yunxu@me.com>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration?
br, Xu Yun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, Thank you for sharing your experience. I will dig into OpenStack cinder logs to check if something happened. The strange thing is the volume I deleted is not created from a snapshot, or doesn’t have any snapshot. And the rbd_id.xxx, rbd_header.xxx and rbd_object_map.xxx were deleted, just left out a lot of rbd_data objects. I plan to delete those objects manually. br, Xu Yun
2020年1月15日 下午3:50，Eugen Block <eblock@nde.ag> 写道：
Hi,
this might happen if you try to delete images/instances/volumes in openstack that are somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too. Deleting a base image worked but there were existing clones so basically just the openstack database was updated, but the base image still existed within ceph.
Try to figure out if that is also the case. If it's something else, check the logs in your openstack environment, maybe they reveal something. Also check the ceph logs.
Regards, Eugen
Zitat von 徐蕴 <yunxu@me.com>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration?
br, Xu Yun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Then it's probably something different. Does that happen with every volume/image or just this one time? Zitat von 徐蕴 <yunxu@me.com>:
Hi Eugen,
Thank you for sharing your experience. I will dig into OpenStack cinder logs to check if something happened. The strange thing is the volume I deleted is not created from a snapshot, or doesn’t have any snapshot. And the rbd_id.xxx, rbd_header.xxx and rbd_object_map.xxx were deleted, just left out a lot of rbd_data objects. I plan to delete those objects manually.
br, Xu Yun
2020年1月15日 下午3:50，Eugen Block <eblock@nde.ag> 写道：
Hi,
this might happen if you try to delete images/instances/volumes in openstack that are somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too. Deleting a base image worked but there were existing clones so basically just the openstack database was updated, but the base image still existed within ceph.
Try to figure out if that is also the case. If it's something else, check the logs in your openstack environment, maybe they reveal something. Also check the ceph logs.
Regards, Eugen
Zitat von 徐蕴 <yunxu@me.com>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration?
br, Xu Yun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
No every volume. It seems that volumes with high capacity have higher probability to trigger this problem.
2020年1月15日 下午4:28，Eugen Block <eblock@nde.ag> 写道：
Then it's probably something different. Does that happen with every volume/image or just this one time?
Zitat von 徐蕴 <yunxu@me.com>:
Hi Eugen,
Thank you for sharing your experience. I will dig into OpenStack cinder logs to check if something happened. The strange thing is the volume I deleted is not created from a snapshot, or doesn’t have any snapshot. And the rbd_id.xxx, rbd_header.xxx and rbd_object_map.xxx were deleted, just left out a lot of rbd_data objects. I plan to delete those objects manually.
br, Xu Yun
2020年1月15日 下午3:50，Eugen Block <eblock@nde.ag> 写道：
Hi,
this might happen if you try to delete images/instances/volumes in openstack that are somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too. Deleting a base image worked but there were existing clones so basically just the openstack database was updated, but the base image still existed within ceph.
Try to figure out if that is also the case. If it's something else, check the logs in your openstack environment, maybe they reveal something. Also check the ceph logs.
Regards, Eugen
Zitat von 徐蕴 <yunxu@me.com>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration?
br, Xu Yun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, since we upgraded to Luminous we have had an issue with snapshot deletion that could be related: when a largish (a few TB) snapshot gets deleted we see a spike in the load of the OSD daemon followed by a brief flap of the daemons themselves. It seems that while the snapshot would have been deleted according to rbd the trimming process would just stop because of the flapping, leaving the PGs in an inconsistent state. Such inconsistency would be later detected during the scrub process in the case of a snapshot but I wonder if something similar happens during the deletion process it would be undetected by the mons/mgr. We fixed the issue changing "osd_snap_trim_sleep" from "0" to "2.0" but I am still pondering whether "osd_delete_sleep" should be changed as well and the issue you are reporting could be an indication of that. Best Mattia On 1/15/20 10:16 AM, 徐蕴 wrote:
No every volume. It seems that volumes with high capacity have higher probability to trigger this problem.
2020年1月15日 下午4:28，Eugen Block <eblock@nde.ag> 写道：
Then it's probably something different. Does that happen with every volume/image or just this one time?
Zitat von 徐蕴 <yunxu@me.com>:
Hi Eugen,
Thank you for sharing your experience. I will dig into OpenStack cinder logs to check if something happened. The strange thing is the volume I deleted is not created from a snapshot, or doesn’t have any snapshot. And the rbd_id.xxx, rbd_header.xxx and rbd_object_map.xxx were deleted, just left out a lot of rbd_data objects. I plan to delete those objects manually.
br, Xu Yun
2020年1月15日 下午3:50，Eugen Block <eblock@nde.ag> 写道：
Hi,
this might happen if you try to delete images/instances/volumes in openstack that are somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too. Deleting a base image worked but there were existing clones so basically just the openstack database was updated, but the base image still existed within ceph.
Try to figure out if that is also the case. If it's something else, check the logs in your openstack environment, maybe they reveal something. Also check the ceph logs.
Regards, Eugen
Zitat von 徐蕴 <yunxu@me.com>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration?
br, Xu Yun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Mattia Belluco S3IT Services and Support for Science IT Office Y11 F 52 University of Zürich Winterthurerstrasse 190, CH-8057 Zürich (Switzerland) Tel: +41 44 635 42 22
Hi For huge volumes in Openstack and Ceph, setup in your cinder this param: volume_clear_size = 50 That will wipe only the first 50MB of the file and then ask to ceph to fully delete instead wipe all disk with zeros that sometimes in huge volumes cause timeout. In our deploy that was the solution, Openstack Queens here -----Mensaje original----- De: Eugen Block <eblock@nde.ag> Enviado el: miércoles, 15 de enero de 2020 8:51 Para: ceph-users@ceph.io Asunto: [ceph-users] Re: Objects not removed (completely) when removing a rbd image Hi, this might happen if you try to delete images/instances/volumes in openstack that are somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too. Deleting a base image worked but there were existing clones so basically just the openstack database was updated, but the base image still existed within ceph. Try to figure out if that is also the case. If it's something else, check the logs in your openstack environment, maybe they reveal something. Also check the ceph logs. Regards, Eugen Zitat von 徐蕴 <yunxu@me.com>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration?
br, Xu Yun _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
EDH - Manuel Rios
-
Eugen Block
-
Mattia Belluco
-
徐蕴