- ceph-users - lists.ceph.io

unknown PGs after adding hosts in different subtree
by Eugen Block 31 May '24

31 May '24

Hi, I got into a weird and unexpected situation today. I added 6 hosts to an existing Pacific cluster (16.2.13, 20 existing OSD hosts across 2 DCs). The hosts were added to the root=default subtree, their designated location is one of two datacenters underneath the default root. Nothing unusual, I believe many people use different subtrees to organize their cluster, as do we in our own (and haven't seen above issue yet). The main application is RGW, the main pool is erasure-coded (k=7, m=11). The crush rule looks like this: rule rule-ec-k7m11 { id 1 type erasure min_size 3 max_size 18 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 2 type datacenter step chooseleaf indep 9 type host step emit } After almost all peering had finished the status showed 6 inactive + peering PGs for a while. I had to fail the mgr because it didn't report correct stats anymore, then it showed 16 unknown PGs. Their application noticed the (unexpected) disruption, after putting the hosts into their designated crush bucket (datacenter) the situation resolved. But I can't make any sense of it, I tried to reproduce it in my lab environment (Quincy), but to no avail. In my tests it behaves as expected, after new OSDs become active there are remapped PGs, but nothing happens until I add them to their designated location. I know I could have prevented that with either osd_crush_initial_weight = 0, then move the crush buckets, then reweight, or by adding the crush buckets first, but usually I don't need to bother about these things. Does anyone have an explanation? I'd appreciate any comments. Thanks! Eugen

3 16

Ceph Reef v18.2.3 - release date?
by Peter Razumovsky 30 May '24

30 May '24

Hello! We're waiting brand new minor 18.2.3 due to https://github.com/ceph/ceph/pull/56004. Why? Timing in our work is a tough thing. Could you kindly share an estimation of 18.2.3 release timeframe? It is 16 days passed from original tag creation so I want to understand when it will be released for upgrade time planning.

2 1

CephFS HA: mgr finish mon failed to return metadata for mds
by kvesligaj＠cs.hr 30 May '24

30 May '24

Hi, we have a stretched cluster (Reef 18.2.1) with 5 nodes (2 nodes on each side + witness). You can se our daemon placement below. [admin] ceph-admin01 labels="['_admin', 'mon', 'mgr']" [nodes] [DC1] ceph-node01 labels="['mon', 'mgr', 'mds', 'osd']" ceph-node02 labels="['mon', 'rgw', 'mds', 'osd']" [DC2] ceph-node03 labels="['mon', 'mgr', 'mds', 'osd']" ceph-node04 labels="['mon', 'rgw', 'mds', 'osd']" We have been testing CephFS HA and noticed when we have active MDS (we have two active MDS daemons at all times) and active MGR (MGR is either on admin node or in one of the DC's) in one DC and when we shutdown that site (DC) we have a problem when one of the MDS metadata can't be retrieved thus showing in logs as: "mgr finish mon failed to return metadata for mds" After we turn that site back on the problem persists and metadata of MDS in question can't be retrieved with "ceph mds metadata" After I manually fail MDS daemon in question with "ceph mds fail" the problem is solved and I can retrieve MDS metadata. My question is, would this be related to the following bug (https://tracker.ceph.com/issues/63166) - I can see that it is showed as backported to 18.2.1 but I can't find it in release notes for Reef. Second question is should this work in current configuration at all as MDS and MGR are both at the same moment disconnected from the rest of the cluster? And final question would be what would be the solution here and is there any loss of data when this happens? Any help is appreciated.

2 1

Help needed! First MDs crashing, then MONs. How to recover ?
by Noe P. 30 May '24

30 May '24

Hi, we ran into a bigger problem today with our ceph cluster (Quincy, Alma8.9). We have 4 filesystems and a total of 6 MDs, the largest fs having two ranks assigned (i.e. one standby). Since we often have the problem of MDs lagging behind, we restart the MDs occasionally. Helps ususally, the standby taking over. Today however, the restart didn't work and the rank 1 MDs started to crash for unclear reasons. Rank 0 seemed ok. We decided at some point to go back to one rank by settings max_mds to 1. Due to the permanent crashing, the rank1 didn't stop however, and at some point we set it to failed and the fs not joinable. At this point it looked like this: fs_cluster - 716 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephmd6a Reqs: 0 /s 13.1M 13.1M 1419k 79.2k 1 failed POOL TYPE USED AVAIL fs_cluster_meta metadata 1791G 54.2T fs_cluster_data data 421T 54.2T with rank1 still being listed. The next attempt was to remove that failed ceph mds rmfailed fs_cluster:1 --yes-i-really-mean-it which, after a short while brought down 3 out of five MONs. They keep crashing shortly after restart with stack traces like this: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff8813adcf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff8809bf09b] 5: /lib64/libstdc++.so.6(+0x9654c) [0x7ff8809c554c] 6: /lib64/libstdc++.so.6(+0x965a7) [0x7ff8809c55a7] 7: /lib64/libstdc++.so.6(+0x96808) [0x7ff8809c5808] 8: /lib64/libstdc++.so.6(+0x92045) [0x7ff8809c1045] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xa9e) [0x55f05d9a5e8e] 10: (MDSMonitor::tick()+0x18a) [0x55f05d9b18da] 11: (MDSMonitor::on_active()+0x2c) [0x55f05d99a17c] 12: (Context::complete(int)+0xd) [0x55f05d76c56d] 13: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x9d) [0x55f05 d799d7d] 14: (Paxos::finish_round()+0x74) [0x55f05d8c5c24] 15: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x41b) [0x55f05d8c7e5b] 16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x123e) [0x55f05d76a2ae] 17: (Monitor::_ms_dispatch(Message*)+0x406) [0x55f05d76a976] 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x55f05d79b3ed] 19: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7ff88367fed8] 20: (DispatchQueue::entry()+0x50f) [0x7ff88367d31f] 21: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff883747381] 22: /lib64/libpthread.so.0(+0x81ca) [0x7ff8813a31ca] 23: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. The MDSMonitor::maybe_resize_cluster somehow suggests a connection to the above MDs operation. Does anyone have an idea how to get this cluster back together again ? Like manually fixing the MD ranks ? 'fs get' can still be called in short moments where enough MONs are reachable: fs_name fs_cluster epoch 3065486 flags 13 allow_snaps allow_multimds_snaps created 2022-08-26T15:55:07.186477+0200 modified 2024-05-28T12:21:59.294364+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 4398046511104 required_client_features {} last_failure 0 last_failure_osd_epoch 1777109 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0,1 up {0=911794623} failed damaged stopped 2,3 data_pools [32] metadata_pool 33 inline_data disabled balancer standby_count_wanted 1 [mds.cephmd6a{0:911794623} state up:active seq 22777 addr [v2:10.13.5.6:6800/189084355,v1:10.13.5.6:6801/189084355] compat {c=[1],r=[1],i=[7ff]}] Regards, Noe

2 1

MDS Abort druing FS scrub
by Malcolm Haak 30 May '24

30 May '24

When running a cephfs scrub the MDS will crash with the following backtrace -1> 2024-05-25T09:00:23.028+1000 7ef2958006c0 -1 /usr/src/debug/ceph/ceph-18.2.2/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7ef2958006c0 time 2024-05-25T09:00:23.031373+1000 /usr/src/debug/ceph/ceph-18.2.2/src/mds/MDSRank.cc: 938: ceph_abort_msg("abort() called") ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd8) [0x7ef2a3a1ea65] 2: (MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x89) [0x5b97f9c811b9] 3: (CDentry::check_corruption(bool)+0x4e3) [0x5b97f9ef2583] 4: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x59) [0x5b97f9d554a9] 5: (Locker::scatter_writebehind(ScatterLock*)+0x2e9) [0x5b97f9e775c9] 6: (Locker::simple_sync(SimpleLock*, bool*)+0x16a) [0x5b97f9e7db4a] 7: (Locker::file_eval(ScatterLock*, bool*)+0x139) [0x5b97f9e84299] 8: (Locker::_drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*, bool)+0x1b1) [0x5b97f9e96221] 9: (Locker::drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*)+0x7f) [0x5b97f9e9726f] 10: (MDCache::repair_inode_stats_work(boost::intrusive_ptr<MDRequestImpl>&)+0x8f7) [0x5b97f9dd2dd7] 11: (MDCache::repair_inode_stats(CInode*)+0x90) [0x5b97f9df61e0] 12: /usr/bin/ceph-mds(+0x460cb8) [0x5b97f9f58cb8] 13: (Continuation::_continue_function(int, int)+0x1c3) [0x5b97f9f6a903] 14: /usr/bin/ceph-mds(+0x46efa1) [0x5b97f9f66fa1] 15: (Continuation::_continue_function(int, int)+0x1c3) [0x5b97f9f6a903] 16: (Continuation::Callback::finish(int)+0x14) [0x5b97f9f6aa14] 17: (Context::complete(int)+0xd) [0x5b97f9c75bfd] 18: (MDSContext::complete(int)+0x5f) [0x5b97f9fd2b0f] 19: (MDSIOContextBase::complete(int)+0x38c) [0x5b97f9fd2ffc] 20: (Finisher::finisher_thread_entry()+0x175) [0x7ef2a3adac95] 21: /usr/lib/libc.so.6(+0x8b55a) [0x7ef2a32a955a] 22: /usr/lib/libc.so.6(+0x108a3c) [0x7ef2a3326a3c] This is after running through the "Advanced recovery procedure" I assume I need a flag or something on the MDS to allow it to keep running during said scan?

2 1

How to setup NVMeoF?
by Robert Sander 30 May '24

30 May '24

Hi, I am trying to follow the documentation at https://docs.ceph.com/en/reef/rbd/nvmeof-target-configure/ to deploy an NVMe over Fabric service. Step 2b of the configuration section is currently the showstopper. First the command says: error: the following arguments are required: --host-name/-t Then it tells me (after adding --host-name): error: unrecognized arguments: --gateway-name XXX and when I remove --gateway-name the error is: both gateway_name and traddr or neither must be specified So I am stuck in a kind of a loop here. Is there a working description for NVMe over TCP available? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

5 6

RBD-Images are not shown in the Dashbord: Failed to execute RBD [errno 19] error generating diff from snapshot None
by Maximilian Dauer 30 May '24

30 May '24

Dear Community, I hope, you can guide me to solve this error or I can assist in solving a bug: RBD-Images are not shown in my Dashbord. - When accessing the dashboard page (block -> images) no images were listed and the error "Failed to execute RBD [errno 19] error generating diff from snapshot None" is shown. - When asking for the list via cli by "rbd ls" the test-image is shown. - There are no snapshots listed when asking for "rbd snap ls --all test". - Back to the ui: I can create and use a new image via UI, this will be shown via cli, but not in the ui. - The cluster manages several RBD, RGW and Cephfs pools successfully with ~13TiB used capacity out of ~18TiB. **Environment details** - Ceph-Cluster with 7 Debian 12 nodes and 3 Ubuntu 22.04 nodes at latest patch level. - Cluster-Version: 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable). - Error persists since Quincy. - Orchestrator: cephadm. - Dashboard is accessed via haproxy, but it's the same behavior when accessing it directly via ip. - All nodes are updated and rebooted. - Dashboard was removed and reactivated. **Steps to reproduce ** I need your help to find the root cause. **Logs** Health status is ok. I can't find useful logs except the attached error message shown in the Dashboard ("ailed to execute RBD [errno 19] error generating diff from snapshot None"). Do you have an idea, where to search for further useful logs? Last but not least: Thanks for this great piece of software! All the best Max

1 0

Missing ceph data
by Prabu GJ 30 May '24

30 May '24

Hi Team, We are using Ceph Octopus version with a total disk size of 136 TB, configured with two replicas. Currently, our usage is 57 TB, and the available size is 5.3 TB. An incident occurred yesterday where around 3 TB of data was deleted automatically. Upon analysis, we couldn't find the reason for the deletion. All OSDs are functioning properly and actively running. We have 3 MDS , we try to restarted all MDS services. Is there any solution to recover those data. Can anyone please help us find the issue? cluster: id: 0d605d58-5caf-4f76-b6bd-e12402a22296 health: HEALTH_WARN insufficient standby MDS daemons available 5 nearfull osd(s) 3 pool(s) nearfull 1 pool(s) have non-power-of-two pg_num services: mon: 4 daemons, quorum download-mon3,download-mon4,download-mon1,download-mon2 (age 14h) mgr: download-mon2(active, since 14h), standbys: download-mon1, download-mon3 mds: integdownload:2 {0=download-mds3=up:active,1=download-mds1=up:active} osd: 39 osds: 39 up (since 16h), 39 in (since 4d) data: pools: 3 pools, 1087 pgs objects: 71.76M objects, 51 TiB usage: 105 TiB used, 31 TiB / 136 TiB avail pgs: 1087 active+clean io: client: 414 MiB/s rd, 219 MiB/s wr, 513 op/s rd, 1.22k op/s wr ================================================================ ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 download-osd1 2995G 581G 14 4785k 6 6626k exists,up 1 download-osd2 2578G 998G 84 3644k 18 10.1M exists,up 2 download-osd3 3093G 483G 17 5114k 5 4152k exists,nearfull,up 3 download-osd4 2757G 819G 12 996k 2 4107k exists,up 4 download-osd5 2889G 687G 28 3355k 20 8660k exists,up 5 download-osd6 2448G 1128G 183 3312k 10 9435k exists,up 6 download-osd7 2814G 762G 7 1667k 4 6354k exists,up 7 download-osd8 2872G 703G 14 1672k 15 10.5M exists,up 8 download-osd9 2577G 999G 10 6615k 3 6960k exists,up 9 download-osd10 2651G 924G 16 4736k 3 7378k exists,up 10 download-osd11 2889G 687G 15 4810k 6 8980k exists,up 11 download-osd12 2912G 664G 11 2516k 2 4106k exists,up 12 download-osd13 2785G 791G 74 4643k 11 3717k exists,up 13 download-osd14 3150G 426G 214 6133k 4 7389k exists,nearfull,up 14 download-osd15 2728G 848G 11 4959k 4 6603k exists,up 15 download-osd16 2682G 894G 13 3170k 3 2503k exists,up 16 download-osd17 2555G 1021G 53 2183k 7 5058k exists,up 17 download-osd18 3013G 563G 18 3497k 3 4427k exists,up 18 download-osd19 2924G 651G 24 3534k 12 10.4M exists,up 19 download-osd20 3003G 573G 19 5149k 3 2531k exists,up 20 download-osd21 2757G 819G 16 3707k 9 9816k exists,up 21 download-osd22 2576G 999G 15 2526k 8 7739k exists,up 22 download-osd23 2758G 818G 13 4412k 16 7125k exists,up 23 download-osd24 2862G 714G 18 4424k 6 5787k exists,up 24 download-osd25 2792G 783G 16 1972k 9 9749k exists,up 25 download-osd26 2397G 1179G 14 4296k 9 12.0M exists,up 26 download-osd27 2308G 1267G 8 3149k 22 6280k exists,up 27 download-osd29 2732G 844G 12 3357k 3 7372k exists,up 28 download-osd28 2814G 761G 11 476k 5 3316k exists,up 29 download-osd30 3069G 507G 15 9043k 17 5628k exists,nearfull,up 30 download-osd31 2660G 916G 15 841k 14 7798k exists,up 31 download-osd32 2037G 1539G 10 1153k 15 3719k exists,up 32 download-osd33 3116G 460G 20 7704k 12 9041k exists,nearfull,up 33 download-osd34 2847G 728G 19 5788k 4 9014k exists,up 34 download-osd35 3088G 488G 17 7178k 7 5730k exists,nearfull,up 35 download-osd36 2414G 1161G 27 2017k 14 7612k exists,up 36 download-osd37 2760G 815G 17 4292k 5 10.6M exists,up 37 download-osd38 2679G 897G 12 2610k 5 10.0M exists,up 38 download-osd39 3013G 563G 18 1804k 7 9235k exists,up Regards Prabu GJ

2 1

How to recover from an MDs rank in state 'failed'
by Noe P. 30 May '24

30 May '24

Hi, after our desaster yesterday, it seems that we got our MONs back. One of the filesystems, however, seems in a strange state: % ceph fs status .... fs_cluster - 782 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephmd6a Reqs: 5 /s 13.2M 13.2M 1425k 51.4k 1 failed POOL TYPE USED AVAIL fs_cluster_meta metadata 3594G 53.5T fs_cluster_data data 421T 53.5T .... STANDBY MDS cephmd6b cephmd4b MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) % ceph fs dump .... Filesystem 'fs_cluster' (3) fs_name fs_cluster epoch 3068261 flags 12 joinable allow_snaps allow_multimds_snaps created 2022-08-26T15:55:07.186477+0200 modified 2024-05-29T12:43:30.606431+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 4398046511104 required_client_features {} last_failure 0 last_failure_osd_epoch 1777109 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 2 in 0,1 up {0=911794623} failed damaged stopped 2,3 data_pools [32] metadata_pool 33 inline_data disabled balancer standby_count_wanted 1 [mds.cephmd6a{0:911794623} state up:active seq 44701 addr [v2:10.13.5.6:6800/189084355,v1:10.13.5.6:6801/189084355] compat {c=[1],r=[1],i=[7ff]}] We would like to get rid of the failed rank 1 (without crashing the MONs) and have a 2nd MD from the standbys step in . Anyone have an idea how to do this ? I'm a bit reluctant to try 'ceph mds rmfailed', as this seems to have triggered the MONs to crash. Regards, Noe

3 3

We are using ceph octopus environment. For client can we use ceph quincy?
by s.dhivagar.cse＠gmail.com 30 May '24

30 May '24

We are using ceph octopus environment. For client can we use ceph quincy?

2 1