- ceph-users - lists.ceph.io

Newly added node osds getting full before rebalance finished
by Szabo, Istvan (Agoda) 27 May '25

27 May '25

Hi, For some reason this is the 2nd time I got into the same issue, added a new node at the time when in the cluster the most full osd was 73% and all the newly added osds now started to be between 70-80% full half way on the way to finish the rebalance. I don't understand why, could this be because mgr upmap mode with deviation 1? Another question, I have 7 disks in each servers, however in the newly added one I still have 4 as spare which is not added. Just to get out of this space issue, would it cause any issue if I add the remaining 4 spare disk (which will cause unbalanced server size in the cluster, the weight of each server is 100TB, this would cause only this node to be 160TB). Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 11

Multisite bucket removal stuck
by Matthew Vernon 27 May '25

27 May '25

Hi, I'm running reef / cephadm. A user wanted two buckets removed (one of which was empty, and one of which had a large number (~half a million) of small objects in). In the primary zone I ran: radosgw-admin bucket rm --bucket=name --bypass-gc --purge-objects [which I've previously used for removing buckets with very-many-objects in single-site just fine] That returned OK for both buckets (quickly for the empty one, after about 20m for the many-objects one). 24 hours later, the empty bucket is gone in both zones, but the many-object one is still present in the secondary zone: source bucket :gitlab-artifacts[64f0dd71-48bf-45aa-9741-69a51c083556.75705.1]) incremental sync on 11 shards bucket is behind on 11 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10] ...which I think is showing that no shards have caught up, even a day later? Similar, sync status shows being behind on 1 metadata shard, and 11 data shards. There are no reported errors in sync. Is there a way to see if this is actually making headway? And/or to get rid of the bucket in the secondary zone? For future reference, would I have been better off trying to stop sync of the bucket and then run a bucket rm in both zones separately? FWIW, this is a small (3-4 OSD hosts per site) and slow (hdds, with nvme for block.db) cluster. Thanks, Matthew

1 2

Re: About pg incomplete
by Phong Tran Thanh 27 May '25

27 May '25

Yes i was remove osd down and purge it now pg can't get active state Vào Th 2, 26 thg 5, 2025 vào lúc 21:31 Anthony D'Atri < anthony.datri(a)gmail.com> đã viết: > Did an OSD go down that you can bring back? > > > On May 26, 2025, at 5:23 AM, Phong Tran Thanh <tranphong079(a)gmail.com> > wrote: > > > > Hi community > > > > How can I force create a new pg when pg status is incomplete. I accept > data > > lost for creating a new pg. > > > > health: HEALTH_WARN > > Reduced data availability: 1 pg inactive, 1 pg incomplete > > 1 pgs not deep-scrubbed in time > > 1 pgs not scrubbed in time > > 154 slow ops, oldest one blocked for 1355676 sec, osd.101 has > > slow ops > > > > pg 8.87b is incomplete, acting [101,70,4,36,16,18] (reducing pool > > fss.motion.data min_size from 4 may help; search ceph.com/docs for > > 'incomplete') > > > > My pool ec4+2 and i was set min_size of pool 4 and did not resolve the > > problem. > > > > PG incomplete effect to cluster slow ops. > > > > > ---------------------------------------------------------------------------- > > *Tran Thanh Phong* > > > > Email: tranphong079(a)gmail.com > > Skype: tranphong079 > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

1 0

ceph reef on AlmaLinux9 (and the dependency problem on openssl-libs)
by Massimo Sgaravatto 26 May '25

26 May '25

Dear all Some weeks ago we updated our ceph cluster (installed on AlmaLinux9 servers) to Ceph reef v. 18.2.6. The day after the email talking about the serious regression bug in that release was sent to this mailing list Frankly it is not fully clear to me if/how that bug is relevant for our deployment: I can say that the OSD start without problems and I can't see obvious problems 18.2.7 can't be installed because it needed openssl-libs v. 3.4 (as discussed in this mailing list) which is in CentOS9 stream but not yet in AlmaLinux9 I have no idea if/when that openssl-libs 3.4 stuff will be released in AlmaLinux9 (AlmaLinux 9.6, released a couple of days ago, doesn't provide an update for openssl-libs). I have now to install some new servers and I have no clear ideas how to proceed Should I keep version 18.2.6 on the existing servers and install 18.2.5 on the new ones ? Should I try installing version 18.2.6 also on the new servers ? Any suggestions? Thanks, Massimo

6 6

CSC Meeting Notes - 05/19
by Venky Shankar 26 May '25

26 May '25

(sending this rather late) * Are we ever going to make a statement about Reef and later shirts, or do we just quietly hope nobody remembers that they used to be a thing we promised? * RH was paying that and costed 80$ * we are 3 release behind and we need to take a call * [Patrick] open source it and allow folks to print it * [Greg] ask our new community manager * [Anthony Middleton] Board has bought this topics up - need to evaluate and process * [Matan] Undefined cmake build type https://tracker.ceph.com/issues/71360 * RelWithDebInfo resulting in a regression * Need to take a decision for BUILD_TYPE - [Sam] set it to DEBUG by default (for shaman builds) - For crimson: s/asert/ceph_assert and stick with RelWithDebInfo and measure performance - [Matan] summary: fine with falling back to RelWithDebInfo and giving up on builtin asserts (crimson specific things can be dealt separately) - (option #1) [Patrick] We might need a build type that has the asserts but has all the optimization benefits - (option #2) Change RelWithDebInfo to include builtin asserts - Not merge the PR and discuss in the tracker (for Crimson, continue offline) * [Mike] Next release: Umbrella * https://ceph-storage.slack.com/archives/C05R975114L/p1747358988415529 * https://en.m.wikipedia.org/wiki/Umbrella_octopus * [Yuri] 17.2.9 final Q release * currently stuck - get quincy (temp) back into build process * [David] Did the needful to get build done * [Yuri] centos8 build by CERN * Yuri and David to catch up to discuss centos8 build -- Cheers, Venky

1 0

18.2.7: modification of osd_deep_scrub_interval in config ignored by mon
by Michel Jouvin 26 May '25

26 May '25

Hi, Last week we increased osd_deep_scrub_interval from 10 days to 14 days as we tended to have permanently 1 PG with a late deep scrub (the PG changing all the time). We did it with `ceph config set ...`. From what we have seen, the deep scrubs are now spread over 14 days (the oldest are 14 days) meaning that OSDs took this change into account (without being restarted). But the number of late deep scrubs reported by `ceph -s) is ~700 which is unexpected. Does it mean that the mon (who is in charge of the report if I am right) have not seen the changes (they have not been restarted)? Cheers, Michel

4 7

About pg incomplete
by Phong Tran Thanh 26 May '25

26 May '25

Hi community How can I force create a new pg when pg status is incomplete. I accept data lost for creating a new pg. health: HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete 1 pgs not deep-scrubbed in time 1 pgs not scrubbed in time 154 slow ops, oldest one blocked for 1355676 sec, osd.101 has slow ops pg 8.87b is incomplete, acting [101,70,4,36,16,18] (reducing pool fss.motion.data min_size from 4 may help; search ceph.com/docs for 'incomplete') My pool ec4+2 and i was set min_size of pool 4 and did not resolve the problem. PG incomplete effect to cluster slow ops. ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

1 1

Unexpected data sync on deleted bucket instances with sync disabled
by Ramin Najjarbashi 25 May '25

25 May '25

Hi I am running a multi-region Ceph cluster with version 18.2.6. Our system has been operating reliably for a long time and currently holds over 100,000 buckets and approximately 1.5 billion objects. We recently attempted to add a new zone to the primary region. To control the synchronization scope, we globally disabled bucket sync using sync disable on all buckets, intending to enable sync selectively only for specific buckets in a bidirectional manner. A Lua script is in place and functioning correctly, issuing a sync disable request via POST whenever a new bucket is created. Despite this setup, we’ve observed that once the new zone was added, data synchronization began across all buckets, including those where sync was explicitly disabled. Upon investigation, we found that a large number of deleted buckets still have existing bucket instances on the master zone with sync flag set to 0. For example, while we have ~100k active buckets, we have over 150k bucket instances. To mitigate this, we manually updated all stale bucket instances using radosgw-admin metadata put to set the sync flag to 8. The change is reflected correctly on the secondary zone, and the metadata synchronization is working as expected. However, the data is still being fully synced for those deleted buckets. An example scenario: • A bucket x was previously created with bucket ID 123456 and later deleted. • A new bucket x is created by a user with a different ID 654321. • Both bucket instances exist: x:123456 and x:654321. • During data sync, it appears the system tries to replicate bucket x regardless of which bucket ID is associated with it. We also attempted to clean up logs by deleting all relevant data sync and full sync logs from the logging pool, and performed datalog trim, but the issue persists. I have two main questions: 1. Why is full data sync still occurring for bucket instances with sync flag set to 8? 2. What is the recommended and safe approach to remove old/stale bucket instances without risking data loss for users? Currently, the sync configuration is set to replicate data between all zones (sync from all), with "log_meta": false and "log_data": true. Looking forward to your insights. Please let me know if further details or logs are needed. Best regards, Ramin

1 0

SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
by Dmitrijs Demidovs 23 May '25

23 May '25

Hi List! We are in the middle of migration from Ceph v16 to v18. As a first step of migration we decided to change OS version on all OSD hosts from Rocky Linux 8.10 to Rocky Linux 9.5 (by using clean reinstall). We also upgraded OSD Block disks from 4TB to 10TB. Now we have 6 OSD hosts running Rocky 9 with new 10TB disks (8 OSD Block disks per Host). OSD hosts have 64GB of RAM. At this moment Ceph cluster is still on version 16.2.15 "Pacific". Migration to Ceph 18.2.x will be performed later after upgrading to Rocky 9 all remaining parts (MGR+MON, MDS, RGW). Problem: After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%. Docker containers running OSD instances are restarted from time to time (and there is a lot of ugly "Call Trace" messages in dmesg). I was wondering if anyone else has similar issues? Will it be gone after migration to Ceph v18? P.S. Configuration Details: ===== [ceph: root@ceph-mon12 /]# ceph versions { "mon": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "osd": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 48 }, "mds": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "overall": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 58 } } [ceph: root@ceph-mon12 /]# === Old Rocky 8 OSD Host: [root@ceph-osd11 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1a4c959eb813 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-37 e37a366347db quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-1 0fdfb95fd631 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-14 276ad8a11148 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-31 9dbd30d11124 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-19 0ad59f583e6a quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-7 e551a652c3ae quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-42 07ff632ae3e1 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-25 605b7a1c00f7 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd11 fe2a6b8a35e8 quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd11 [root@ceph-osd11 ~]# uptime 10:52:59 up 216 days, 20:12, 1 user, load average: 0.20, 0.41, 0.53 [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi [root@ceph-osd11 ~]# rpm -qa | grep docker docker-buildx-plugin-0.14.0-1.el8.x86_64 docker-ce-cli-26.1.3-1.el8.x86_64 docker-ce-26.1.3-1.el8.x86_64 docker-scan-plugin-0.23.0-3.el8.x86_64 docker-ce-rootless-extras-26.1.3-1.el8.x86_64 docker-compose-plugin-2.27.0-1.el8.x86_64 [root@ceph-osd11 ~]# cat /etc/rocky-release Rocky Linux release 8.10 (Green Obsidian) [root@ceph-osd11 ~]# uname -a Linux ceph-osd11 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 8 17:47:08 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux === New Rocky 9 OSD Host: [root@ceph-osd17 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5cc58e4a77b2 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 days ago Up 4 days ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 3a60fecf648d quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 9c20407e79eb quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 9deadafef9dd quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 fcfe62a25fd9 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 38d2d96cc491 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 e29c6bbc1ae7 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 40346a7a45ea quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 43c3e3a65531 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 d9e436f9788c quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 [root@ceph-osd17 ~]# uptime 10:25:10 up 122 days, 22:37, 1 user, load average: 0.37, 0.26, 0.27 [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi [root@ceph-osd17 ~]# rpm -qa | grep docker docker-compose-plugin-2.31.0-1.el9.x86_64 docker-buildx-plugin-0.19.2-1.el9.x86_64 docker-ce-cli-27.4.0-1.el9.x86_64 docker-ce-rootless-extras-27.4.0-1.el9.x86_64 docker-ce-27.4.0-1.el9.x86_64 [root@ceph-osd17 ~]# cat /etc/rocky-release Rocky Linux release 9.5 (Blue Onyx) [root@ceph-osd17 ~]# uname -a Linux ceph-osd17 5.14.0-503.15.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 26 17:24:29 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux =====

6 6

rgw crash in RGWDeleteMultiObj::handle_individual_object for bucket notification reserve
by xu chenhui 23 May '25

23 May '25

Hi, RGW enable bucket notification for http push_endpoint, it will crash when exec delete multi objects command. ceph version: v16.2.13 Thread 483 "radosgw" received signal SIGABRT, Aborted. [Switching to Thread 0x7f94c0ac1700 (LWP 500)] 0x00007f95b86f2acf in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f95b86f2acf in raise () from /lib64/libc.so.6 #1 0x00007f95b86c5ea5 in abort () from /lib64/libc.so.6 #2 0x00007f95b86c5d79 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x00007f95b86eb426 in __assert_fail () from /lib64/libc.so.6 #4 0x00007f95c7480332 in boost::context::continuation::resume() && (this=0x55a1b3705158) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:263 #5 0x00007f95c74802cd in boost::context::continuation::resume() & (this=0x55a1b3705158) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:259 #6 0x00007f95c74803db in spawn::detail::continuation_context::resume (this=0x55a1b3705158) at /opt/ceph-16.2.13/src/spawn/include/spawn/impl/spawn.hpp:37 #7 0x00007f95c7499de0 in spawn::detail::coro_async_result<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, void>::get (this=0x55a1b4b0b248) at /opt/ceph-16.2.13/src/spawn/include/spawn/impl/spawn.hpp:256 #8 0x00007f95c7d28659 in librados::async_operate<boost::asio::io_context, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > > > (ctx=..., io=..., oid="ifind-project-other-tv", write_op=0x55a1b4b0b460, flags=1024, token=...) at /opt/ceph-16.2.13/src/librados/librados_asio.h:185 #9 0x00007f95c7d24d9d in rgw_rados_operate (dpp=0x55a1b463c000, ioctx=..., oid="ifind-project-other-tv", op=0x55a1b4b0b460, y=..., flags=1024) at /opt/ceph-16.2.13/src/rgw/rgw_tools.cc:269 #10 0x00007f95c7f4caff in rgw::notify::publish_reserve (dpp=0x55a1b463c000, event_type=rgw::notify::ObjectRemovedDelete, res=..., req_tags=0x0) at /opt/ceph-16.2.13/src/rgw/rgw_notify.cc:796 #11 0x00007f95c7b182e0 in RGWDeleteMultiObj::handle_individual_object (this=0x55a1b463c000, o=..., y=..., formatter_flush_cond=0x7f9485248130) at /opt/ceph-16.2.13/src/rgw/rgw_op.cc:7164 #12 0x00007f95c7b18c45 in RGWDeleteMultiObj::<lambda(yield_context)>::operator()(yield_context) const (__closure=0x55a1b0160fa8, yield=...) at /opt/ceph-16.2.13/src/rgw/rgw_op.cc:7267 #13 0x00007f95c7b2f43f in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)>::operator()(boost::context::continuation &&) const ( this=0x7f9485247f60, c=...) at /opt/ceph-16.2.13/src/spawn/include/spawn/impl/spawn.hpp:390 #14 0x00007f95c7b30610 in std::__invoke_impl<boost::context::continuation, spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(std::__invoke_other, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)> &, boost::context::continuation &&) (__f=..., __args#0=...) at /usr/include/c++/8/bits/invoke.h:60 #15 0x00007f95c7b30531 in std::__invoke<spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)> &, boost::context::continuation &&) (__fn=..., __args#0=...) at /usr/include/c++/8/bits/invoke.h:96 #16 0x00007f95c7b304a2 in std::invoke<spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>, boost::context::basic_fixedsize_stack<boost::context::stack_traits> >::<lambda(boost::context::continuation&&)> &, boost::context::continuation &&) (__fn=..., __args#0=...) at /usr/include/c++/8/functional:82 #17 0x00007f95c7b3036f in boost::context::detail::record<boost::context::continuation, boost::context::basic_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)> >::run(boost::context::detail::fcontext_t) ( this=0x55a1b4b0bf00, fctx=0x7f9485247d10) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:146 #18 0x00007f95c7b3015a in boost::context::detail::context_entry<boost::context::detail::record<boost::context::continuation, boost::context::basic_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<Handler, Function, StackAllocator>::operator()() [with Handler = boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >; Function = RGWDeleteMultiObj::execute(optional_yield)::<lambda(yield_context)>; StackAllocator = boost::context::basic_fixedsize_stack<boost::context::stack_traits>]::<lambda(boost::context::continuation&&)> > >(boost::context::detail::transfer_t) (t=...) at /opt/ceph-16.2.13/build/boost/include/boost/context/continuation_fcontext.hpp:80 #19 0x00007f95c809175f in make_fcontext () from /lib64/libradosgw.so.2 #20 0x9435c9c14cf40130 in ?? () #21 0xacf2d40e6f64164e in ?? () #22 0x00000000d9ff03a8 in ?? () #23 0x0000000000000025 in ?? () #24 0x5f7265767265535f in ?? () #25 0x000055a1b4b0bef8 in ?? () #26 0x0000000000000007 in ?? () #27 0x00322d7466697773 in ?? () #28 0x0000000000020000 in ?? () #29 0x000055a1b4b0c000 in ?? () #30 0x0000000000020000 in ?? () #31 0x00007f9485247f60 in ?? () #32 0x3234346330623365 in ?? () Reproduce： 1. create topic rgw_topic='{"push-endpoint": "'$httpserver'","verify-ssl": "false","persistent":"true"}' aws --region=default --endpoint-url $srcEndpoint sns create-topic --name=$userId --attributes="$rgw_topic" 2. create bucket notification bucket_notify='{"TopicConfigurations": [{"Id": "'$bk'", "TopicArn": "'$topic'", "Events": ["s3:ObjectCreated:*", "s3:ObjectRemoved:*"]}]}' aws --region=default --endpoint-url $srcEndpoint s3api put-bucket-notification-configuration --bucket=$bucketName --notification-configuration="$bucket_notify" 3. delete multi objects aws s3api --endpoint-url $srcEndpoint delete-objects --bucket $bucketName --delete '{"Objects":[{"Key":"obj1"}]}' I had report this problem in https://tracker.ceph.com/issues/71427 rgw_multi_object_del_aio set to 1 didn't help, anyone can have a workaround to resolve this problem ? thanks.

1 0