Hi community, I'm running a large s3 running with ingress and backend rgw, total traffic of my cluster ~30Gbps real time traffic. I' am have a problem with lifecycle in rgw, object can't delete with following log Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067328.134916_15b007878cf011ee960ba1c41ff4eaf9_3007777.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 1 lifecycle: ERROR: publishing notification failed, with error: -2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 Nov 27 13:46:29 ceph-osd-211 bash[1836289]: debug 2023-11-27T06:46:29.933+0000 7f0d1ee36700 0 lifecycle: ERROR: remove_expired_obj :ec3204cam05[f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2]):003_8792934df4756401_231127_1701067327.693377_14a3adc68cf011ee960ba1c41ff4eaf9_3007677.test (2) No such file or directory wp_thrd: 4, 2 My cluster setting rgw_lc_max_worker = 6 rgw_lc_debug_interval=60 (setting during testing) I am testing bucket lifecycle expire every 5 minute and 1 minute radosgw-lc list of bucket is processing "bucket": ":ec3204cam05:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.2", "started": "Mon, 27 Nov 2023 07:29:58 GMT", "status": "PROCESSING" s3cmd getlifecycle s3://ec3204cam04 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 5 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>5</Days> </Expiration> </Rule> </LifecycleConfiguration> s3cmd getlifecycle s3://ec3204cam05 <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>Expire after 1 day</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> </Rule> </LifecycleConfiguration> I don't know why the rgw show log can't delete object with no such file, how i can find the problem, I hope someone can help me Thanks