Hi Casey, I did use full `radosgw-admin gc process --include-all` I know about that background gc delay. Is running `radosgw-admin gc process --include-all` from terminal any different than gc process running in the background? I wonder if I should use it while trying to recreate this issue. Best regards Adam Prycki W dniu 2024-06-27 20:58, Casey Bodley napisał(a):
hi Adam,
On Thu, Jun 27, 2024 at 4:41 AM Adam Prycki <aprycki@man.poznan.pl> wrote:
Hello,
I have a question. Do people use rgw lifecycle policies in production? I had big hopes for this technology bug in practice it seems to be very unreliable.
Recently I've been testing different pool layouts and using lifecycle policy to move data between them. Once I've checked orphaned objects I've discovered that my pools were full of orphaned objects. One pool was over 1/3 orphans by volume. Orphan object belonged to data that was moved by lifecycle.
Yesterday I decided to recreate one of the pools with 3TiB of data. All 3TiB was located in a single directory of some buckets. I've created a lifecycle which should move it all to STANDARD pool and run radosgw-admin lc process --bucket. After lifecycle finished executing ceph pool still contained 1TiB of data. Removing objects from rgw-orphan-list output reduced pool size to 65GiB and 17k objects.
The 17k rados __shadow objects seem to belong to s3 objects which were not moved by lifecycle. I tried lifecycle from radosgw-admin but lifecycle seems to be unable to move them. s3cmd info show that they still report old storage class. Filenames don't contain special characters other than spaces. I have directories with sequentially named objects, some of them cannot be moved by lifecycle.
Deleting all the objects form original 3TiB dataset also doesn't help. After running gc and orphan finding tool there are still 1,2k rados objects which should have been deleted but are not considered orphans.
i assume you used `radosgw-admin gc process` here - can you confirm whether you added the --include-all option? without that option, garbage collection won't delete objects newer than rgw_gc_obj_min_wait=2hours in case they're still being read. it sounds like these rados objects may still be in the gc queue, which could explain why they aren't considered orphans
I've been testing on 18.2.2.
Best regards Adam Prycki _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io