Hello, I have a question. Do people use rgw lifecycle policies in production? I had big hopes for this technology bug in practice it seems to be very unreliable. Recently I've been testing different pool layouts and using lifecycle policy to move data between them. Once I've checked orphaned objects I've discovered that my pools were full of orphaned objects. One pool was over 1/3 orphans by volume. Orphan object belonged to data that was moved by lifecycle. Yesterday I decided to recreate one of the pools with 3TiB of data. All 3TiB was located in a single directory of some buckets. I've created a lifecycle which should move it all to STANDARD pool and run radosgw-admin lc process --bucket. After lifecycle finished executing ceph pool still contained 1TiB of data. Removing objects from rgw-orphan-list output reduced pool size to 65GiB and 17k objects. The 17k rados __shadow objects seem to belong to s3 objects which were not moved by lifecycle. I tried lifecycle from radosgw-admin but lifecycle seems to be unable to move them. s3cmd info show that they still report old storage class. Filenames don't contain special characters other than spaces. I have directories with sequentially named objects, some of them cannot be moved by lifecycle. Deleting all the objects form original 3TiB dataset also doesn't help. After running gc and orphan finding tool there are still 1,2k rados objects which should have been deleted but are not considered orphans. I've been testing on 18.2.2. Best regards Adam Prycki