15 Jan
2020
15 Jan
'20
5:49 a.m.
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the
space was not reclaimed. I checked with rbd ls and confirmed that this image was
disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx
are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I
waited for several hours and there is no further deletion happed. Is it a known issue, or
something wrong with my configuration?
br,
Xu Yun
15 Jan
15 Jan
8:50 a.m.
Hi,
this might happen if you try to delete images/instances/volumes in
openstack that are somehow linked, e.g. if there are snapshots etc. I
have experienced this in Ocata, too. Deleting a base image worked but
there were existing clones so basically just the openstack database
was updated, but the base image still existed within ceph.
Try to figure out if that is also the case. If it's something else,
check the logs in your openstack environment, maybe they reveal
something. Also check the ceph logs.
Regards,
Eugen
Zitat von 徐蕴 <yunxu(a)me.com>om>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an
image, I found that the space was not reclaimed. I checked with rbd
ls and confirmed that this image was disappeared. But when I check
the objects with rados ls, most objects named rbd_data.xxx are still
existed in my cluster. rbd_object_map and rbd_header were already
deleted. I waited for several hours and there is no further deletion
happed. Is it a known issue, or something wrong with my configuration?
br,
Xu Yun
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:17 a.m.
Hi Eugen,
Thank you for sharing your experience. I will dig into OpenStack cinder logs to check if
something happened. The strange thing is the volume I deleted is not created from a
snapshot, or doesn’t have any snapshot. And the rbd_id.xxx, rbd_header.xxx and
rbd_object_map.xxx were deleted, just left out a lot of rbd_data objects. I plan to delete
those objects manually.
br,
Xu Yun
2020年1月15日 下午3:50，Eugen Block <eblock(a)nde.ag>
写道：
Hi,
this might happen if you try to delete images/instances/volumes in openstack that are
somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too.
Deleting a base image worked but there were existing clones so basically just the
openstack database was updated, but the base image still existed within ceph.
Try to figure out if that is also the case. If it's something else, check the logs in
your openstack environment, maybe they reveal something. Also check the ceph logs.
Regards,
Eugen
Zitat von 徐蕴 <yunxu(a)me.com>om>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the
space was not reclaimed. I checked with rbd ls and confirmed that this image was
disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx
are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I
waited for several hours and there is no further deletion happed. Is it a known issue, or
something wrong with my configuration?
br,
Xu Yun
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:28 a.m.
Then it's probably something different. Does that happen with every
volume/image or just this one time?
Zitat von 徐蕴 <yunxu(a)me.com>om>:
Hi Eugen,
Thank you for sharing your experience. I will dig into OpenStack
cinder logs to check if something happened. The strange thing is the
volume I deleted is not created from a snapshot, or doesn’t have any
snapshot. And the rbd_id.xxx, rbd_header.xxx and rbd_object_map.xxx
were deleted, just left out a lot of rbd_data objects. I plan to
delete those objects manually.
br,
Xu Yun
> 2020年1月15日 下午3:50，Eugen Block <eblock(a)nde.ag> 写道：
>
> Hi,
>
> this might happen if you try to delete images/instances/volumes in
> openstack that are somehow linked, e.g. if there are snapshots etc.
> I have experienced this in Ocata, too. Deleting a base image worked
> but there were existing clones so basically just the openstack
> database was updated, but the base image still existed within ceph.
>
> Try to figure out if that is also the case. If it's something else,
> check the logs in your openstack environment, maybe they reveal
> something. Also check the ceph logs.
>
> Regards,
> Eugen
>
>
> Zitat von 徐蕴 <yunxu(a)me.com>om>:
>
>> Hello,
>>
>> My setup is Ceph pike working with OpenStack. When I deleted an
>> image, I found that the space was not reclaimed. I checked with
>> rbd ls and confirmed that this image was disappeared. But when I
>> check the objects with rados ls, most objects named rbd_data.xxx
>> are still existed in my cluster. rbd_object_map and rbd_header
>> were already deleted. I waited for several hours and there is no
>> further deletion happed. Is it a known issue, or something wrong
>> with my configuration?
>>
>> br,
>> Xu Yun
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:16 a.m.
No every volume. It seems that volumes with high capacity have higher probability to
trigger this problem.
2020年1月15日 下午4:28，Eugen Block <eblock(a)nde.ag>
写道：
Then it's probably something different. Does that happen with every volume/image or
just this one time?
Zitat von 徐蕴 <yunxu(a)me.com>om>:
Hi Eugen,
Thank you for sharing your experience. I will dig into OpenStack cinder logs to check if
something happened. The strange thing is the volume I deleted is not created from a
snapshot, or doesn’t have any snapshot. And the rbd_id.xxx, rbd_header.xxx and
rbd_object_map.xxx were deleted, just left out a lot of rbd_data objects. I plan to delete
those objects manually.
br,
Xu Yun
> 2020年1月15日 下午3:50，Eugen Block <eblock(a)nde.ag> 写道：
>
> Hi,
>
> this might happen if you try to delete images/instances/volumes in openstack that are
somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too.
Deleting a base image worked but there were existing clones so basically just the
openstack database was updated, but the base image still existed within ceph.
>
> Try to figure out if that is also the case. If it's something else, check the
logs in your openstack environment, maybe they reveal something. Also check the ceph
logs.
>
> Regards,
> Eugen
>
>
> Zitat von 徐蕴 <yunxu(a)me.com>om>:
>
>> Hello,
>>
>> My setup is Ceph pike working with OpenStack. When I deleted an image, I found
that the space was not reclaimed. I checked with rbd ls and confirmed that this image was
disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx
are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I
waited for several hours and there is no further deletion happed. Is it a known issue, or
something wrong with my configuration?
>>
>> br,
>> Xu Yun
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
11:37 a.m.
Hi,
since we upgraded to Luminous we have had an issue with snapshot
deletion that could be related: when a largish (a few TB) snapshot gets
deleted we see a spike in the load of the OSD daemon followed by a brief
flap of the daemons themselves.
It seems that while the snapshot would have been deleted according to
rbd the trimming process would just stop because of the flapping,
leaving the PGs in an inconsistent state. Such inconsistency would be
later detected during the scrub process in the case of a snapshot but I
wonder if something similar happens during the deletion process it would
be undetected by the mons/mgr.
We fixed the issue changing "osd_snap_trim_sleep" from "0" to
"2.0" but
I am still pondering whether "osd_delete_sleep" should be changed as
well and the issue you are reporting could be an indication of that.
Best
Mattia
On 1/15/20 10:16 AM, 徐蕴 wrote:
No every volume. It seems that volumes with high
capacity have higher probability to trigger this problem.
--
Mattia Belluco
S3IT Services and Support for Science IT
Office Y11 F 52
University of Zürich
Winterthurerstrasse 190, CH-8057 Zürich (Switzerland)
Tel: +41 44 635 42 22
2020年1月15日 下午4:28，Eugen Block
<eblock(a)nde.ag> 写道：
Then it's probably something different. Does that happen with every volume/image or
just this one time?
Zitat von 徐蕴 <yunxu(a)me.com>om>:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi Eugen,
Thank you for sharing your experience. I will dig into OpenStack cinder logs to check if
something happened. The strange thing is the volume I deleted is not created from a
snapshot, or doesn’t have any snapshot. And the rbd_id.xxx, rbd_header.xxx and
rbd_object_map.xxx were deleted, just left out a lot of rbd_data objects. I plan to delete
those objects manually.
br,
Xu Yun
> 2020年1月15日 下午3:50，Eugen Block <eblock(a)nde.ag> 写道：
>
> Hi,
>
> this might happen if you try to delete images/instances/volumes in openstack that are
somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too.
Deleting a base image worked but there were existing clones so basically just the
openstack database was updated, but the base image still existed within ceph.
>
> Try to figure out if that is also the case. If it's something else, check the
logs in your openstack environment, maybe they reveal something. Also check the ceph
logs.
>
> Regards,
> Eugen
>
>
> Zitat von 徐蕴 <yunxu(a)me.com>om>:
>
>> Hello,
>>
>> My setup is Ceph pike working with OpenStack. When I deleted an image, I found
that the space was not reclaimed. I checked with rbd ls and confirmed that this image was
disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx
are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I
waited for several hours and there is no further deletion happed. Is it a known issue, or
something wrong with my configuration?
>>
>> br,
>> Xu Yun
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
12:36 p.m.
Hi
For huge volumes in Openstack and Ceph, setup in your cinder this param:
volume_clear_size = 50
That will wipe only the first 50MB of the file and then ask to ceph to fully delete
instead wipe all disk with zeros that sometimes in huge volumes cause timeout.
In our deploy that was the solution, Openstack Queens here
-----Mensaje original-----
De: Eugen Block <eblock(a)nde.ag>
Enviado el: miércoles, 15 de enero de 2020 8:51
Para: ceph-users(a)ceph.io
Asunto: [ceph-users] Re: Objects not removed (completely) when removing a rbd image
Hi,
this might happen if you try to delete images/instances/volumes in openstack that are
somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too.
Deleting a base image worked but there were existing clones so basically just the
openstack database was updated, but the base image still existed within ceph.
Try to figure out if that is also the case. If it's something else, check the logs in
your openstack environment, maybe they reveal something. Also check the ceph logs.
Regards,
Eugen
Zitat von 徐蕴 <yunxu(a)me.com>om>:
Hello,
My setup is Ceph pike working with OpenStack. When I deleted an image,
I found that the space was not reclaimed. I checked with rbd ls and
confirmed that this image was disappeared. But when I check the
objects with rados ls, most objects named rbd_data.xxx are still
existed in my cluster. rbd_object_map and rbd_header were already
deleted. I waited for several hours and there is no further deletion
happed. Is it a known issue, or something wrong with my configuration?
br,
Xu Yun
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1573
days inactive
1573
days old
6 comments
4 participants
participants (4)
-
EDH - Manuel Rios -
Eugen Block -
Mattia Belluco -
徐蕴