- ceph-users - lists.ceph.io

CephFS MDS - Crash restart loop
by Michael Götting 23 May '25

23 May '25

Hi all, we have the following problem with our CephFS Setup (Ceph version 19.2.2). Today our two active MDS nodes failed and then the nodes that were in „stand-by replay“ took over and failed as well. The CephFS system is equipped as follows: - 3 monitor nodes - 4 MDS nodes (2 active/ 2 stand-by) - 2 active - 2 stand-by replay - CephFS Pool - max_mds = 2 - 1x Meta data pool - 2x data pools (hdd_pool, ssd_pool) << ----------------- Ceph fs status output START----------------- >> cephfs - 0 clients RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 failed << ----------------- Ceph fs status output END----------------- >> To restore the service, we used the documentation https://docs.ceph.com/en/quincy/cephfs/disaster-recovery-experts/?highlight… we carried out the steps up to and including "MDS table wipes". We did not carry out the MDS MAP RESET step as we were not sure that we would then lose all the data from RANK 1. We also carried out the steps under "Avoiding recovery roadblocks" https://docs.ceph.com/en/quincy/cephfs/troubleshooting/#avoiding-recovery-r…. Parameters set of the MDS nodes: mds advanced mds_abort_on_newly_corrupt_dentry false mds advanced mds_bal_interval 0 mds basic mds_cache_memory_limit 274877906944 mds advanced mds_cache_trim_threshold 524288 mds advanced mds_go_bad_corrupt_dentry false mds advanced mds_heartbeat_grace 3600. 000000 mds advanced mds_min_caps_working_set 60000 mds advanced mds_oft_prefetch_dirfrags false * After trying the recovery steps (truncating the journal) the MDS daemons are in a crash -> restart loop behavior. << ----------------- Example log file mds-1: START ----------------- >> -14> 2025-05-21T17:49:27.003+0200 7f36cfc7e640 1 mds.0.42052 active_start -13> 2025-05-21T17:49:27.003+0200 7f36d2c84640 10 monclient: get_auth_request con 0x559a76e8a400 auth_method 0 -12> 2025-05-21T17:49:27.003+0200 7f36d2483640 10 monclient: get_auth_request con 0x559a7e041400 auth_method 0 -11> 2025-05-21T17:49:27.003+0200 7f36d3485640 10 monclient: get_auth_request con 0x559a76e8b000 auth_method 0 -10> 2025-05-21T17:49:27.003+0200 7f36d3485640 10 monclient: get_auth_request con 0x559ab89cb800 auth_method 0 -9> 2025-05-21T17:49:27.003+0200 7f36d2c84640 10 monclient: get_auth_request con 0x559a7bcc6400 auth_method 0 -8> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 1 mds.0.42052 cluster recovered. -7> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 4 mds.0.42052 set_osd_epoch_barrier: epoch=492573 -6> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 5 quiesce.mds.0 <quiesce_cluster_update> epoch:42055 me:7764062 leader:7764062 members:7764062 -5> 2025-05-21T17:49:27.015+0200 7f36cfc7e640 5 quiesce.mgr.0 <update_membership> starting the db mgr thread at epoch: 42055 -4> 2025-05-21T17:49:27.015+0200 7f36c5c6a640 5 quiesce.mgr.0 <quiesce_db_thread_main> Entering the main thread -3> 2025-05-21T17:49:27.015+0200 7f36c5c6a640 5 quiesce.mgr.0 <membership_upkeep> a reset of the db has been requested -2> 2025-05-21T17:49:27.015+0200 7f36c9471640 -1 mds.0.cache.den(0x1 techfak) newly corrupt dentry to be committed: [dentry #0x1/techfak [c,head] auth (dversion lock) pv=0 v=52947746 ino=0x1000a58d072 state=1073741824 | inodepin=1 0x559a755b2c80] -1> 2025-05-21T17:49:27.015+0200 7f36c9471640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/mds/MDCache.cc: In function 'void MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)' thread 7f36c9471640 time 2025-05-21T17:49:27.020101+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/mds/MDCache.cc: 1687: FAILED ceph_assert(follows >= realm->get_newest_seq()) ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f36d5709cf9] 2: /usr/lib64/ceph/libceph-common.so.2(+0x182eb8) [0x7f36d5709eb8] 3: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xac3) [0x559a51ca4583] 4: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xbd) [0x559a51ca50cd] 5: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xe71) [0x559a51cab8f1] 6: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x473) [0x559a51d55a33] 7: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x390) [0x559a51d2e750] 8: (MDSContext::complete(int)+0x5c) [0x559a51e4617c] 9: (MDSIOContextBase::complete(int)+0x34c) [0x559a51e4884c] 10: /usr/bin/ceph-mds(+0x4f5970) [0x559a51eed970] 11: /usr/bin/ceph-mds(+0x160f0d) [0x559a51b58f0d] 12: (Finisher::finisher_thread_entry()+0x17d) [0x7f36d57c885d] 13: /lib64/libc.so.6(+0x8a0ca) [0x7f36d50a30ca] 14: /lib64/libc.so.6(+0x10f150) [0x7f36d5128150] 0> 2025-05-21T17:49:27.019+0200 7f36c9471640 -1 *** Caught signal (Aborted) ** in thread 7f36c9471640 thread_name: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f36d5057bf0] 2: /lib64/libc.so.6(+0x8be0c) [0x7f36d50a4e0c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f36d5709d53] 6: /usr/lib64/ceph/libceph-common.so.2(+0x182eb8) [0x7f36d5709eb8] 7: (MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)+0xac3) [0x559a51ca4583] 8: (MDCache::journal_dirty_inode(MutationImpl*, EMetaBlob*, CInode*, snapid_t)+0xbd) [0x559a51ca50cd] 9: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xe71) [0x559a51cab8f1] 10: (Locker::check_inode_max_size(CInode*, bool, unsigned long, unsigned long, utime_t)+0x473) [0x559a51d55a33] 11: (RecoveryQueue::_recovered(CInode*, int, unsigned long, utime_t)+0x390) [0x559a51d2e750] 12: (MDSContext::complete(int)+0x5c) [0x559a51e4617c] 13: (MDSIOContextBase::complete(int)+0x34c) [0x559a51e4884c] 14: /usr/bin/ceph-mds(+0x4f5970) [0x559a51eed970] 15: /usr/bin/ceph-mds(+0x160f0d) [0x559a51b58f0d] 16: (Finisher::finisher_thread_entry()+0x17d) [0x7f36d57c885d] 17: /lib64/libc.so.6(+0x8a0ca) [0x7f36d50a30ca] 18: /lib64/libc.so.6(+0x10f150) [0x7f36d5128150] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. << ----------------- Example log file mds-1: END ----------------- >> << ----------------- Ceph fs fs dump output START----------------- >> e41371 btime 2025-05-21T16:19:27:085643+0200 enable_multiple, ever_enabled_multiple: 1,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 3 Filesystem 'cephfs' (3) fs_name cephfs epoch 41370 flags 73 allow_snaps allow_multimds_snaps allow_standby_replay refuse_client_session created 2024-03-31T23:36:25.302389+0200 modified 2025-05-21T16:19:09.237977+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 max_xattr_size 65536 required_client_features {} last_failure 0 last_failure_osd_epoch 492429 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} max_mds 2 in 0,1 up {1=7761302} failed 0 damaged stopped data_pools [5,3] metadata_pool 4 inline_data disabled balancer bal_rank_mask -1 standby_count_wanted 2 qdb_cluster leader: 7761302 members: 7761302 [mds.mds-1{1:7761302} state up:active seq 5 addr [v2:[2001:638:504:2011:9:3:1:1]:6800/2463826788,v1:[2001:638:504:2011:9:3:1:1]:6801/2463826788] compat {c=[1],r=[1],i=[1fff]}] Standby daemons: [mds.mds-2{-1:7771506} state up:standby seq 1 addr [v2:[2001:638:504:2011:6:3:2:2]:6800/2809657192,v1:[2001:638:504:2011:6:3:2:2]:6801/2809657192] compat {c=[1],r=[1],i=[1fff]}] dumped fsmap epoch 41371 << ----------------- Ceph fs fs dump output END———————— >> But to be honest, out of all those things we tried, I don't know what to provide exactly. We can provide much more but ... We really need the service back online, so help will be very much appreciated. Regards, Michael

2 2

ceph node upgrades
by Bandelow, Gunnar 23 May '25

23 May '25

Hi everyone out there, i have a hopefully simple to answer question: For example a reef cluster with cephadm and all containers and packages on version 18.2.4 on debian12 nodes. The nodes itself have now ceph debian package upgrades available for the ceph packages with version 18.2.7. Do I have to maintain the same version between the version within the containers and the OS? Can I just upgrade the nodes with a newer version of packages and keep the containers at 18.2.4? I marked ceph-common and cephadm as hold, because I wasn't sure, but now I want to know. Any input is appreciated. Best regards, Gunnar ======================================================= Gunnar Bandelow (dipl. phys.) Universitätsrechenzentrum (URZ) Universität Greifswald Felix-Hausdorff-Straße 18 17489 GreifswaldGermany

3 2

Re: Getting alarm emails every 600s after Ceph Pacific install
by Malte Stroem 22 May '25

22 May '25

Hi Stefan, why did you install Ceph with the almost obsolete ceph-ansible and why did you install the very old Pacific? Since this is a new cluster try the easy way and install Ceph Squid with cephadm. https://docs.ceph.com/en/latest/cephadm/install/ Best, Malte On 11.08.21 21:21, Stefan Schneebeli wrote: > Hi all, > > I just installed a new Ceph Pacific Cluster with ceph-ansible which > worked very well. > > But now one of the mon servers sends every 600s an email with this subject: > > ALERT localhost/trap: trap timeout > > and this body: > > Summary output : trap timeout > > Group : localhost > Service : trap > Time noticed : Wed Aug 11 21:06:16 2021 > Secs until next alert : > Members : localhost > > Detailed text (if any) follows: > ------------------------------- > trap timeout after 600s at Wed Aug 11 21:06:16 2021 > > > I have never seen this before. I also don't see any messages like this > in the mon logfiles. > The two other mon servers don't sent such emails. > > Does anybody have an idea what the problem could be and how to solve it? > > Thank you! > > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

SWAP usage 100% on OSD hosts after migration to Rocky Linux 9 (Ceph 16.2.15)
by Dmitrijs Demidovs 22 May '25

22 May '25

Hi List! We are in the middle of migration from Ceph v16 to v18. As a first step of migration we decided to change OS version on all OSD hosts from Rocky Linux 8.10 to Rocky Linux 9.5 (by using clean reinstall). We also upgraded OSD Block disks from 4TB to 10TB. Now we have 6 OSD hosts running Rocky 9 with new 10TB disks (8 OSD Block disks per Host). OSD hosts have 64GB of RAM. At this moment Ceph cluster is still on version 16.2.15 "Pacific". Migration to Ceph 18.2.x will be performed later after upgrading to Rocky 9 all remaining parts (MGR+MON, MDS, RGW). Problem: After migration to Rocky 9 (and new version of Docker) we see what our OSD hosts consumes 100% of SWAP space! It takes approximately one week to fill SWAP from 0% to 100%. Docker containers running OSD instances are restarted from time to time (and there is a lot of ugly "Call Trace" messages in dmesg). I was wondering if anyone else has similar issues? Will it be gone after migration to Ceph v18? P.S. Configuration Details: ===== [ceph: root@ceph-mon12 /]# ceph versions { "mon": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3 }, "osd": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 48 }, "mds": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2 }, "overall": { "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 58 } } [ceph: root@ceph-mon12 /]# === Old Rocky 8 OSD Host: [root@ceph-osd11 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1a4c959eb813 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-37 e37a366347db quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-1 0fdfb95fd631 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-14 276ad8a11148 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-31 9dbd30d11124 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-19 0ad59f583e6a quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-7 e551a652c3ae quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-42 07ff632ae3e1 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-25 605b7a1c00f7 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd11 fe2a6b8a35e8 quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 7 months ago Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd11 [root@ceph-osd11 ~]# uptime 10:52:59 up 216 days, 20:12, 1 user, load average: 0.20, 0.41, 0.53 [root@ceph-osd11 ~]# free -h total used free shared buff/cache available Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi Swap: 8.0Gi 2.8Gi 5.2Gi [root@ceph-osd11 ~]# rpm -qa | grep docker docker-buildx-plugin-0.14.0-1.el8.x86_64 docker-ce-cli-26.1.3-1.el8.x86_64 docker-ce-26.1.3-1.el8.x86_64 docker-scan-plugin-0.23.0-3.el8.x86_64 docker-ce-rootless-extras-26.1.3-1.el8.x86_64 docker-compose-plugin-2.27.0-1.el8.x86_64 [root@ceph-osd11 ~]# cat /etc/rocky-release Rocky Linux release 8.10 (Green Obsidian) [root@ceph-osd11 ~]# uname -a Linux ceph-osd11 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 8 17:47:08 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux === New Rocky 9 OSD Host: [root@ceph-osd17 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5cc58e4a77b2 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 days ago Up 4 days ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52 3a60fecf648d quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50 9c20407e79eb quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49 9deadafef9dd quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48 fcfe62a25fd9 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55 38d2d96cc491 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51 e29c6bbc1ae7 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54 40346a7a45ea quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53 43c3e3a65531 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17 d9e436f9788c quay.io/prometheus/node-exporter:v1.3.1 "/bin/node_exporter …" 4 months ago Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17 [root@ceph-osd17 ~]# uptime 10:25:10 up 122 days, 22:37, 1 user, load average: 0.37, 0.26, 0.27 [root@ceph-osd17 ~]# free -h total used free shared buff/cache available Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi Swap: 8.0Gi 8.0Gi 7.0Mi [root@ceph-osd17 ~]# rpm -qa | grep docker docker-compose-plugin-2.31.0-1.el9.x86_64 docker-buildx-plugin-0.19.2-1.el9.x86_64 docker-ce-cli-27.4.0-1.el9.x86_64 docker-ce-rootless-extras-27.4.0-1.el9.x86_64 docker-ce-27.4.0-1.el9.x86_64 [root@ceph-osd17 ~]# cat /etc/rocky-release Rocky Linux release 9.5 (Blue Onyx) [root@ceph-osd17 ~]# uname -a Linux ceph-osd17 5.14.0-503.15.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 26 17:24:29 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux =====

1 0

HBA Card vs. HBA Mode (on a RAID card)
by Anthony Fecarotta 21 May '25

21 May '25

Does running a RAID controller in HBA mode (not to be confused with IT mode) impact Ceph performance compared to using a dedicated HBA card? Is there any documentation or benchmarking data showing improved performance with true HBA hardware? For what it's worth my cluster is on Dell PowerEdge R740 machines. Thank you for your insights. Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

3 3

radosgw-admin: period update error after deleting and recreating zones
by Michel Jouvin 21 May '25

21 May '25

Hi, It is not the first time that after doing configuration changes in RADOS for a realm/zonegroup/zone with radosgw-admin, we get errors when trying to do a "period update --commit". We never found a good documentation on how to fix these problems, up to now we always managed at some point to restore a good configuration that can be commited but it is probably time for us to have a more informed approach! Last occurence of the problem happened today with a realm/zonegroup/zone created recently. Trying to fix a problem with the non working haproxy associated with it, one of my colleagues decided to delete and recreate the zone and zonegroup (with the same names). The related commands worked but since it has been done any attempt to do "period update --commit" results in the following error: ------- 2025-05-06T11:56:20.939+0200 7fdc7d41da80 0 failed reading obj info from .rgw.root:zone_info.93af6e0c-4552-4c2e-b167-36114a5a81e4: (2) No such file or directory 2025-05-06T11:56:20.945+0200 7fdc7d41da80 0 failed reading obj info from .rgw.root:zonegroup_info.d7221099-4e7d-43cb-a1e8-28a750de1cd5: (2) No such file or directory 2025-05-06T11:56:21.160+0200 7fdc7d41da80 0 failed reading obj info from .rgw.root:zone_info.93af6e0c-4552-4c2e-b167-36114a5a81e4: (2) No such file or directory 2025-05-06T11:56:21.160+0200 7fdc7d41da80 -1 Cannot find zone id=93af6e0c-4552-4c2e-b167-36114a5a81e4 (name=default) 2025-05-06T11:56:21.160+0200 7fdc7d41da80 0 ERROR: failed to start notify service ((22) Invalid argument 2025-05-06T11:56:21.160+0200 7fdc7d41da80 0 ERROR: failed to init services (ret=(22) Invalid argument) couldn't init storage provider ------- I have the feeling that it is related to the delete objects that are no longer found but it is not completely clear what is the way out of it? Is the problem related to recreating the zone/zonegroup with the same names? There are several realms already in production so we cannot do a .rgw.root reset but this particular realm has never been put in production so we can delete everything related to it. Thanks in advance for any hint or pointer. Best regards, Michel

3 5

rgw crash on thread notif-worker0
by xu chenhui 21 May '25

21 May '25

Hi, all rgw crash on thread notif-worker0 when using rgw persistent notification function in ceph version: v16.2.13. I haven't find root cause. Backtrace: 0> 2025-05-13T13:36:45.435+0000 7f0813c64700 -1 *** Caught signal (Aborted) ** in thread 7f0813c64700 thread_name:notif-worker0 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f084607acf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f084506809b] 5: /lib64/libstdc++.so.6(+0x9654c) [0x7f084506e54c] 6: /lib64/libstdc++.so.6(+0x965a7) [0x7f084506e5a7] 7: /lib64/libstdc++.so.6(+0x96808) [0x7f084506e808] 8: (rgw::notify::Manager::Manager(ceph::common::CephContext*, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, rgw::sal::RGWRadosStore*)::{lambda()#2}::operator()() const+0x240) [0x7f0852c176c0] 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f084509ab23] 10: /lib64/libpthread.so.0(+0x81ca) [0x7f08460701ca] 11: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. =================== Above coredump info had no symbol info , so I compile same version with debuginfo. However, there are differen coredump info: gdb) bt #0 0x00007fd4edc6dacf in raise () from /lib64/libc.so.6 #1 0x00007fd4edc40ea5 in abort () from /lib64/libc.so.6 #2 0x00007fd4edc40d79 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x00007fd4edc66426 in __assert_fail () from /lib64/libc.so.6 #4 0x00007fd4fb81d7cd in boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>::allocate() () from /lib64/libradosgw.so.2 #5 0x00007fd4fbe8491b in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> > boost::context::callcc<boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}>(std::allocator_arg_t, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>&&, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}&&) () from /lib64/libradosgw.so.2 #6 0x00007fd4fbe8554f in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >::operator()() () from /lib64/libradosgw.so.2 #7 0x00007fd4fbe85999 in void boost::asio::detail::strand_executor_service::dispatch<boost::asio::io_context::executor_type const, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >&, std::allocator<void> >(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl> const&, boost::asio::io_context::executor_type const&, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >&, std::allocator<void> const&) () from /lib64/libradosgw.so.2 #8 0x00007fd4fbe85c55 in std::enable_if<spawn::detail::is_stack_allocator<std::decay<boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >::type, void>::value, void>::type spawn::spawn<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}::stack_traits> >(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >, rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#7}&&, std::decay&&) () from /lib64/libradosgw.so.2 #9 0x00007fd4fbe888fe in rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >) () from /lib64/libradosgw.so.2 #10 0x00007fd4fbe8c711 in rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::operator()(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >) const () from /lib64/libradosgw.so.2 #11 0x00007fd4fbe8cab4 in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}, boost::context::basic_protected_fixedsize_stack<rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::stack_traits> >::operator()()::{lambda(rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::continuation&&)#1}::operator()(rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}::continuation) const () from /lib64/libradosgw.so.2 #12 0x00007fd4fbe8cc90 in boost::context::detail::record<boost::context::continuation, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}>::run(void*) () from /lib64/libradosgw.so.2 #13 0x00007fd4fbe8cd58 in void boost::context::detail::context_entry<boost::context::detail::record<boost::context::continuation, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>, spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, rgw::notify::Manager::process_queues(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)::{lambda(spawn::basic_yield_c--Type <RET> for more, q to quit, c to continue without paging-- ontext<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> > >)#6}, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits> >::operator()()::{lambda(boost::context::continuation&&)#1}> >(boost::context::detail::transfer_t) () from /lib64/libradosgw.so.2 #14 0x00007fd4fbf5e94f in make_fcontext () from /lib64/libradosgw.so.2 #15 0x0000000000000000 in ?? () I have found a similar issue in https://github.com/ceph/ceph/pull/39521, but it haven't reslove it.

2 2

RGW Bucket Replication in Admin Ops API
by Gregory O'Neill 20 May '25

20 May '25

Hey folks, I have three rgw clusters setup with multi-site. replication is bi-direcitonal for all zones between each cluster. They were running Luminous 12.2.12 before I upgraded them to Reef 18.2.2 We have an internal app that manages S3 buckets in ceph for users which uses the admin ops api. There was an option in the api to enable or disable replication but it looks like it was removed in Nautilus (`OPT_BUCKET_SYNC_ENABLE` in src/rgw/rgw_admin.cc in luminous or mimic branches). I know that `radosgw-admin` has a command to toggle bucket sync, but that's not scalable for our user count. What is the intended way to S3 manage bucket replication in Ceph? Thanks, Greg -- Gregory O’Neill

2 1

rpm-quincy
by farhad kh 20 May '25

20 May '25

Hi, I need to install ceph-common from the Quincy repository, but I'm getting this error: --- Ceph x86_64 0.0 B/s | 0 B 00:01 Errors during downloading metadata for repository 'Ceph': - Status code: 404 for http://download.ceph.com/rpm-quincy/el8/x86_64/repodata/repomd.xml (IP: 172.16.110.219) Error: Failed to download metadata for repo 'Ceph': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried --- I think the Ceph Quincy repository might have been removed or is no longer functional. Is that the case? And how can I still use the Ceph Quincy repository for EL8 if possible?

3 2

BLUEFS_SPILLOVER after Reef upgrade
by Kasper Rasmussen 20 May '25

20 May '25

I've just upgraded our ceph cluster from pacific 16.2.15 -> Reef 18.2.7 After that I see the warnings: [WRN] BLUEFS_SPILLOVER: 5 OSD(s) experiencing BlueFS spillover osd.110 spilled over 4.5 GiB metadata from 'db' device (8.0 GiB used of 83 GiB) to slow device osd.455 spilled over 1.1 GiB metadata from 'db' device (11 GiB used of 83 GiB) to slow device osd.533 spilled over 426 MiB metadata from 'db' device (10 GiB used of 83 GiB) to slow device osd.560 spilled over 389 MiB metadata from 'db' device (9.8 GiB used of 83 GiB) to slow device osd.597 spilled over 8.6 GiB metadata from 'db' device (7.7 GiB used of 83 GiB) to slow device [WRN] BLUESTORE_SLOW_OP_ALERT: 4 OSD(s) experiencing slow operations in BlueStore osd.410 observed slow operation indications in BlueStore osd.443 observed slow operation indications in BlueStore osd.508 observed slow operation indications in BlueStore osd.593 observed slow operation indications in BlueStore I've tried to run ceph tell osd.XXX compact with no result. Bluefs stats: ceph tell osd.110 bluefs stats 1 : device size 0x14b33fe000 : using 0x202c00000(8.0 GiB) 2 : device size 0x8e8ffc00000 : using 0x5d31d150000(5.8 TiB) RocksDBBlueFSVolumeSelector >>Settings<< extra=0 B, l0_size=1 GiB, l_base=1 GiB, l_multi=8 B DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 16 MiB 0 B 0 B 0 B 15 MiB 1 WAL 0 B 18 MiB 0 B 0 B 0 B 6.3 MiB 1 DB 0 B 8.0 GiB 0 B 0 B 0 B 8.0 GiB 140 SLOW 0 B 0 B 4.5 GiB 0 B 0 B 4.5 GiB 78 TOTAL 0 B 8.0 GiB 4.5 GiB 0 B 0 B 0 B 220 MAXIMUMS: LOG 0 B 25 MiB 0 B 0 B 0 B 21 MiB WAL 0 B 118 MiB 0 B 0 B 0 B 93 MiB DB 0 B 8.2 GiB 0 B 0 B 0 B 8.2 GiB SLOW 0 B 0 B 14 GiB 0 B 0 B 14 GiB TOTAL 0 B 8.2 GiB 14 GiB 0 B 0 B 0 B >> SIZE << 0 B 79 GiB 8.5 TiB Help with what to do next will, be much appreciated

5 15