- ceph-users - lists.ceph.io

Re: More objects misplaced than exist?
by Andre Tann 17 Jan '25

17 Jan '25

Hi Anthony, answering also to the list... Am 16.01.25 um 15:52 schrieb Anthony D'Atri: > When I see anomalous status my first thought is to manually failover the mgr I stopped the active mgr, another took over, but still the status is the same: data: volumes: 1/1 healthy pools: 4 pools, 2081 pgs objects: 3.53M objects, 12 TiB usage: 36 TiB used, 226 TiB / 262 TiB avail pgs: 17669564/10601739 objects misplaced (166.667%) 2081 active+clean+remapped What I still don't understand is: why does Ceph report usage 100% for all pools, which is nonsense? Can that be related to the confusing numbers of misplaces objects? root@pve01:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 262 TiB 226 TiB 36 TiB 36 TiB 13.58 TOTAL 262 TiB 226 TiB 36 TiB 36 TiB 13.58 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 7.6 MiB 3 15 MiB 100.00 0 B ReplicationPool 2 1024 8.0 TiB 2.11M 24 TiB 100.00 0 B cephfs_data 7 1024 3.9 TiB 1.43M 12 TiB 100.00 0 B cephfs_metadata 8 32 268 MiB 143 804 MiB 100.00 0 B -- Andre Tann

1 1

More objects misplaced than exist?
by Andre Tann 16 Jan '25

16 Jan '25

Hi all, # ceph -w ... volumes: 1/1 healthy pools: 4 pools, 2081 pgs objects: 3.72M objects, 12 TiB usage: 36 TiB used, 226 TiB / 262 TiB avail pgs: 18590764/11154459 objects misplaced (166.667%) 2081 active+clean+remapped How can more objects be misplaced than exist in the pools? Also ceph reports for all pools 100% usage. But also it shows that 35 out of 250 TB are used. So there is sth weird. Any ideas what could be wrong here? -- Andre Tann

1 0

issue with new AWS cli when upload: MissingContentLength
by Szabo, Istvan (Agoda) 16 Jan '25

16 Jan '25

Hi, Amazon released a new version of their cli today https://github.com/aws/aws-cli/tags and seems to break our stuffs with the following error when PUT object happens: bash-4.2# /usr/local/bin/aws --endpoint=https://endpoint --no-verify-ssl s3 cp online.txt s3://bucket/ upload failed: ./online.txt to s3://bucket/online.txt An error occurred (MissingContentLength) when calling the PutObject operation: Unknown Wonder is there anything on ceph side that I can do to eliminate this? The amazon workaround response_checksum_validation = when_required did not work. https://github.com/aws/aws-cli/issues/9214 Thank you

2 1

Ceph symbols for v15_2_0 in pacific libceph-common
by Frank Schilder 16 Jan '25

16 Jan '25

Hi all, during debugging of an MDS problem we observed something that looks odd. The output of perf top seems to show symbols from v15 (octopus) on a pacific (v16) installation: 23.56% ceph-mds [.] std::_Rb_tree<inodeno_t, inodeno_t, std::_Identity<inodeno_t> 7.02% libceph-common.so.2 [.] ceph::buffer::v15_2_0::ptr::copy_out 4.99% ceph-mds [.] std::_Hashtable<inodeno_t, std::pair<inodeno_t const, CInode* 4.69% libceph-common.so.2 [.] ceph::buffer::v15_2_0::list::iterator_impl<true>::copy 2.53% ceph-mds [.] std::_Rb_tree<SnapRealm*, SnapRealm*, std::_Identity<SnapReal 2.46% libtcmalloc.so.4.5.3 [.] tcmalloc::CentralFreeList::Populate 2.31% [kernel] [k] _raw_spin_lock 2.11% ceph-mds [.] std::_Rb_tree<dentry_key_t, std::pair<dentry_key_t const, CDe 1.90% libstdc++.so.6.0.25 [.] std::_Rb_tree_insert_and_rebalance 1.86% [kernel] [k] copy_user_enhanced_fast_string 1.83% libtcmalloc.so.4.5.3 [.] operator new[] 1.83% libceph-common.so.2 [.] ceph::buffer::v15_2_0::list::iterator_impl<true>::operator+= 1.68% ceph-mds [.] MDCache::populate_mydir 1.56% ceph-mds [.] std::_Rb_tree<frag_t, std::pair<frag_t const, CDir*>, std::_S 1.38% [kernel] [k] clear_page_erms 1.06% [kernel] [k] native_irq_return_iret The pacific packages are installed from download.ceph.com and all package candidates claim to be v16: # yum provides "/*/libceph-common.so.2" Last metadata expiration check: 0:45:57 ago on Wed 15 Jan 2025 12:14:09 PM CET. librados2-2:16.2.15-0.el8.x86_64 : RADOS distributed object store client library Repo : @System Matched from: Filename : /usr/lib64/ceph/libceph-common.so.2 librados2-2:16.2.15-0.el8.x86_64 : RADOS distributed object store client library Repo : ceph Matched from: Filename : /usr/lib64/ceph/libceph-common.so.2 Why do we see v15 symbols here or am I interpreting the symbol name ceph::buffer::v15_2_0::list::iterator_impl<true>::copy incorrectly? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 1

Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.
by bruno.pessanha＠gmail.com 16 Jan '25

16 Jan '25

Hi everyone. I'm still learning how to run Ceph properly in production. I have a a cluster (Reef 18.2.4) with 10 nodes (8 x 15TB nvme's each). There are prod 2 pools, one for RGW (3 x replica) and one for CephFS (EC 8k2m). It was all fine but one users started store more data I started seeing: 1. Very high number of misplaced PG's. 2. OSD's very unbalanced and getting 90% full ``` ceph -s cluster: id: 7805xxxe-6ba7-11ef-9cda-0xxxcxxx0 health: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 195 pgs backfill_toofull 150 pgs not deep-scrubbed in time 150 pgs not scrubbed in time services: mon: 5 daemons, quorum host01,host02,host03,host04,host05 (age 7w) mgr: host01.bwqkna(active, since 7w), standbys: host02.dycdqe mds: 5/5 daemons up, 6 standby osd: 80 osds: 80 up (since 7w), 80 in (since 4M); 323 remapped pgs rgw: 30 daemons active (10 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 1394 pgs objects: 159.65M objects, 279 TiB usage: 696 TiB used, 421 TiB / 1.1 PiB avail pgs: 230137879/647342099 objects misplaced (35.551%) 1033 active+clean 180 active+remapped+backfill_toofull 123 active+remapped+backfill_wait 28 active+clean+scrubbing 15 active+remapped+backfill_wait+backfill_toofull 10 active+clean+scrubbing+deep 5 active+remapped+backfilling io: client: 668 MiB/s rd, 11 MiB/s wr, 1.22k op/s rd, 1.15k op/s wr recovery: 479 MiB/s, 283 objects/s progress: Global Recovery Event (5w) [=====================.......] (remaining: 11d) ``` I've been trying to rebalance the OSD's manually since the balancer does not work due to: ``` "optimize_result": "Too many objects (0.355160 > 0.050000) are misplaced; try again later", ``` I manually re-weighted the top 10 most used OSD's and the number of misplaced objects are going down very slowly. I think it could take many weeks at that ratio. There's almost 40% of total free space but the RGW pool is almost full at ~94% I think because of OSD's unbalancing. ``` ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 TOTAL 1.1 PiB 421 TiB 697 TiB 697 TiB 62.34 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 69 MiB 15 207 MiB 0 13 TiB .nfs 2 32 172 KiB 43 574 KiB 0 13 TiB .rgw.root 3 32 2.7 KiB 6 88 KiB 0 13 TiB default.rgw.log 4 32 2.1 MiB 209 7.0 MiB 0 13 TiB default.rgw.control 5 32 0 B 8 0 B 0 13 TiB default.rgw.meta 6 32 97 KiB 280 3.5 MiB 0 13 TiB default.rgw.buckets.index 7 32 16 GiB 2.41k 47 GiB 0.11 13 TiB default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 13 TiB default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB 0.04 13 TiB cephfs.cephfs01.data 12 144 83 TiB 23.99M 103 TiB 72.18 32 TiB cephfs.cephfs01.metadata 13 1 952 MiB 483.14k 3.7 GiB 0 10 TiB ``` I also tried changing the following but it does not seem to persist: ``` # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 # ceph config set osd osd_max_backfills 10 # ceph-conf --show-config | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" osd_max_backfills = 1 osd_recovery_max_active = 0 osd_recovery_max_active_hdd = 3 osd_recovery_max_active_ssd = 10 osd_recovery_op_priority = 3 ``` 1. Why I ended up with so many misplaced PG's since there were no changes on the cluster: number of osd's, hosts, etc. 2. Is it ok to change the target_max_misplaced_ratio to something higher than .05 so the autobalancer would work and I wouldn't have to constantly rebalance the osd's manually? 3. Is there a way to speed up the rebalance? 4. Any other recommendation that could help to make my cluster healthy again? Thank you! Bruno

6 16

Ceph User + Dev Meeting Information
by Laura Flores 16 Jan '25

16 Jan '25

Hi all, It was brought to my attention that some users were unaware that the old User + Dev meetup day changed from Thursdays to Wednesdays. To stay informed about future User + Dev meetings, you can: 1. Join the Meetup group here: https://www.meetup.com/ceph-user-group/, which will allow you to RSVP to upcoming events. 2. Download the Ceph Community Calendar, which is accessible here: https://ceph.io/en/community/meetups/ 3. Follow updates about upcoming topics, etc. for these meetings in the #ceph-user-dev-meetup Slack channel at this link: https://ceph-storage.slack.com/archives/C072CRU63H9 Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

MDS crashing on startup
by Frank Schilder 15 Jan '25

15 Jan '25

Hi Dan, hi all, this is related to the thread "Help needed, ceph fs down due to large stray dir". We deployed a bare metal host for debugging ceph daemon issues, here, to run "perf top" to find out where our MDS becomes unresponsive. Unfortunately, we encounter a strange issue: The bare-metal MDS crashes very quickly during the initial reconnect phase: -61> 2025-01-14T08:59:47.202-0500 7f676e519700 3 mds.2.server not active yet, waiting -60> 2025-01-14T08:59:47.202-0500 7f676e519700 5 mds.2.server dispatch request in up:reconnect: client_request(client.425250501:594886 lookup #0x3001059de1e/02JanParetoR esults_n5_a7_m3 2025-01-13T15:25:32.427929-0500 RETRY=6 caller_uid=315104, caller_gid=315104{}) v2 -59> 2025-01-14T08:59:47.202-0500 7f676e519700 3 mds.2.server not active yet, waiting -58> 2025-01-14T08:59:47.202-0500 7f6771d20700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MAC HINE_SIZE/gigantic/release/16.2.15/rpm/el8/BUILD/ceph-16.2.15/src/msg/async/AsyncMessenger.cc: In function 'void Processor::accept()' thread 7f6771d20700 time 2025-01-14T08: 59:47.200795-0500 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.15/rpm/el8/BUILD/ceph -16.2.15/src/msg/async/AsyncMessenger.cc: 214: ceph_abort_msg("abort() called") ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f6776f6e904] 2: (Processor::accept()+0x862) [0x7f6777261502] 3: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x7f67772b6b87] 4: /usr/lib64/ceph/libceph-common.so.2(+0x5c90bc) [0x7f67772bd0bc] 5: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6775380b23] 6: /lib64/libpthread.so.0(+0x81ca) [0x7f6775f4b1ca] 7: clone() -57> 2025-01-14T08:59:47.202-0500 7f676e519700 0 log_channel(cluster) log [DBG] : reconnect by client.425250501 v1:192.168.57.60:0/1003106369 after 1.01409 ... -3> 2025-01-14T08:59:47.202-0500 7f676e519700 5 mds.2.server dispatch request in up:reconnect: client_request(client.42561 2912:251430 lookup #0x10000f5568d/util-linux 2025-01-11T17:43:34.212128-0500 RETRY=8 caller_uid=298337, caller_gid=298337{}) v2 -2> 2025-01-14T08:59:47.202-0500 7f676e519700 3 mds.2.server not active yet, waiting -1> 2025-01-14T08:59:47.202-0500 7f676e519700 0 log_channel(cluster) log [DBG] : reconnect by client.425612912 v1:192.168. 58.11:0/4294630612 after 1.01409 0> 2025-01-14T08:59:47.203-0500 7f6771d20700 -1 *** Caught signal (Aborted) ** in thread 7f6771d20700 thread_name:msgr-worker-0 ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) 1: /lib64/libpthread.so.0(+0x12d10) [0x7f6775f55d10] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator< char> > const&)+0x1b6) [0x7f6776f6e9d5] 5: (Processor::accept()+0x862) [0x7f6777261502] 6: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x7 f67772b6b87] 7: /usr/lib64/ceph/libceph-common.so.2(+0x5c90bc) [0x7f67772bd0bc] 8: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6775380b23] 9: /lib64/libpthread.so.0(+0x81ca) [0x7f6775f4b1ca] 10: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. All other log messages look normal. The strange thing is that the bare metal and the containerized MDSes are the same version, yet the containerized daemon does *not* crash. Versions are: bare-metal# ceph-mds --version ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) container# ceph-mds --version ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable) Both binaries have the same md5 sum as well. The only possibly relevant difference might be the kernel version: bare-metal: 4.18.0-553.34.1.el8_10.x86_64 containerized: 5.14.13-1.el7.elrepo.x86_64 I also installed all sorts of debuginfo packages. Still, this symbol is not resolved: 7: /usr/lib64/ceph/libceph-common.so.2(+0x5c90bc) [0x7f67772bd0bc] Which package is it in? I did install ceph-common-debuginfo-16.2.15-0.el8.x86_64.rpm . For installing the MDS we followed the instructions for manual start here: https://docs.ceph.com/en/pacific/install/manual-deployment/#adding-mds . Thanks for any pointers! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

Multi Active MDS on old kernel client (<4.14)
by Jesse Galley 15 Jan '25

15 Jan '25

Hello! The documentation on kernel client features states: "The feature has been supported since the Luminous release. It is recommended to use Linux kernel clients >= 4.14 when there are multiple active MDS." My question is: what is the actual impact of using an old (3.10) client with a multi-mds FS? I used a 3.10 client to do some reads and writes of metadata pinned to different MDSs, and it all ostensibly worked fine. Is it purely a performance problem? Or is there a more serious issue lurking that my simple tests would not reveal. Thanks! *Jesse Galley* *Director, Mail Platform & Migrations, Hostopia* *C: 416.816.0866* *jesse.galley(a)hostpapa.com <jesse.galley(a)hostpapa.com>*

2 2

Re: [ceph-users] Installing Ceph on ARM fails
by filip Mutterer 15 Jan '25

15 Jan '25

Its Debian 12 On Tuesday, January 14, 2025 19:34 CET, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Which OS? > > > On Jan 14, 2025, at 1:29 PM, filip Mutterer <filip(a)3dds.io> wrote: > > > > hi, > > > > trying to install Ceph 19.2 on ARM. > > > > My commands are: > > > > for i in 192.168.1.{11,12,13};do ssh $i "apt install -y chrony curl";done > > for i in 192.168.1.{11,12,13};do ssh $i \ > > "curl --silent --remote-name --locationhttps://download.ceph.com/rpm-19.2.0/el9/noarch/cephadm && chmod +x cephadm";done > > for i in 192.168.1.{11,12,13};do ssh $i "./cephadm add-repo --release squid && ./cephadm install";done > > > > So far no problem, but if I want to check the version "/usr/sbin/cephadm version" i am getting no result, it just hangs. > > > > Ignoring this and continue with the installation: > > > > apt-get install -y ceph-common > > > > Warning: Permanently added '192.168.1.13' (ED25519) to the list of known hosts. > > Installing packages ['ceph-common']... > > Non-zero exit code 100 from apt-get install -y ceph-common > > apt-get: stdout Reading package lists... > > apt-get: stdout Building dependency tree... > > apt-get: stdout Reading state information... > > apt-get: stdout Some packages could not be installed. This may mean that you have > > apt-get: stdout requested an impossible situation or if you are using the unstable > > apt-get: stdout distribution that some required packages have not yet been created > > apt-get: stdout or been moved out of Incoming. > > apt-get: stdout The following information may help to resolve the situation: > > apt-get: stdout > > apt-get: stdout The following packages have unmet dependencies: > > apt-get: stdout ceph-common : Depends: python3-ceph-argparse (= 16.2.15+ds-0+deb12u1) but 19.2.0-1~bpo12+1 is to be installed > > apt-get: stdout Depends: python3-ceph-common (< 16.2.15+ds-0+deb12u1.1~) but 19.2.0-1~bpo12+1 is to be installed > > apt-get: stdout Depends: python3-cephfs (= 16.2.15+ds-0+deb12u1) but it is not going to be installed > > apt-get: stderr E: Unable to correct problems, you have held broken packages. > > Traceback (most recent call last): > > File "<frozen runpy>", line 198, in _run_module_as_main > > File "<frozen runpy>", line 88, in _run_code > > File "/root/./cephadm/__main__.py", line 5579, in <module> > > File "/root/./cephadm/__main__.py", line 5567, in main > > File "/root/./cephadm/__main__.py", line 4586, in command_install > > File "/root/./cephadm/cephadmlib/packagers.py", line 209, in install > > File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws > > RuntimeError: Failed command: apt-get install -y ceph-common: E: Unable to correct problems, you have held broken packages. > > > > On my x86 system this step was different, it just installed version 19.2.0 > > > > How can I get the version wanted? > > > > > > Greetings filip > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Installing Ceph on ARM fails
by filip Mutterer 14 Jan '25

14 Jan '25

hi, trying to install Ceph 19.2 on ARM. My commands are: for i in 192.168.1.{11,12,13};do ssh $i "apt install -y chrony curl";done for i in 192.168.1.{11,12,13};do ssh $i \ "curl --silent --remote-name --locationhttps://download.ceph.com/rpm-19.2.0/el9/noarch/cephadm && chmod +x cephadm";done for i in 192.168.1.{11,12,13};do ssh $i "./cephadm add-repo --release squid && ./cephadm install";done So far no problem, but if I want to check the version "/usr/sbin/cephadm version" i am getting no result, it just hangs. Ignoring this and continue with the installation: apt-get install -y ceph-common Warning: Permanently added '192.168.1.13' (ED25519) to the list of known hosts. Installing packages ['ceph-common']... Non-zero exit code 100 from apt-get install -y ceph-common apt-get: stdout Reading package lists... apt-get: stdout Building dependency tree... apt-get: stdout Reading state information... apt-get: stdout Some packages could not be installed. This may mean that you have apt-get: stdout requested an impossible situation or if you are using the unstable apt-get: stdout distribution that some required packages have not yet been created apt-get: stdout or been moved out of Incoming. apt-get: stdout The following information may help to resolve the situation: apt-get: stdout apt-get: stdout The following packages have unmet dependencies: apt-get: stdout ceph-common : Depends: python3-ceph-argparse (= 16.2.15+ds-0+deb12u1) but 19.2.0-1~bpo12+1 is to be installed apt-get: stdout Depends: python3-ceph-common (< 16.2.15+ds-0+deb12u1.1~) but 19.2.0-1~bpo12+1 is to be installed apt-get: stdout Depends: python3-cephfs (= 16.2.15+ds-0+deb12u1) but it is not going to be installed apt-get: stderr E: Unable to correct problems, you have held broken packages. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/./cephadm/__main__.py", line 5579, in <module> File "/root/./cephadm/__main__.py", line 5567, in main File "/root/./cephadm/__main__.py", line 4586, in command_install File "/root/./cephadm/cephadmlib/packagers.py", line 209, in install File "/root/./cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: apt-get install -y ceph-common: E: Unable to correct problems, you have held broken packages. On my x86 system this step was different, it just installed version 19.2.0 How can I get the version wanted? Greetings filip

1 0