There a couple of ways, find your MDS daemon with: ceph fs status -> should show you the to-be-active MDS On that host run: cephadm logs --name mds.{MDS} or alternatively: cephadm ls --no-detail | grep mds journalctl -u ceph-{FSID}@mds.{MDS} --no-pager > {MDS}.log Zitat von nguyenvandiep@baoviet.com.vn:

How can we get log of MDS, pls guide me T_T _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

If it crashes after two minutes you have your time window to look for. Restart the mds daemon and capture everything after that until the crash. Zitat von nguyenvandiep@baoviet.com.vn:

it suck too long log, could you pls guide me how to grep/filter important things in logs ? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

You still haven't provided any details (logs) of what happened. The short excerpt from yesterday isn't useful as it only shows the startup of the daemon. Zitat von nguyenvandiep@baoviet.com.vn:

Could you pls help me explain the status of volume: recovering ? what is it ? and do we need to wait for volume recovery progress finished ?? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

This seems to be the relevant stack trace: ---snip--- Feb 23 15:18:39 cephgw02 conmon[2158052]: debug -1> 2024-02-23T08:18:39.609+0000 7fccc03c0700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: In function 'std::ostream& operator<<(std::ostream&, const ClientMetricType&)' thread 7fccc03c0700 time 2024-02-23T08:18:39.609581+0000 Feb 23 15:18:39 cephgw02 conmon[2158052]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: 56: ceph_abort_msg("abort() called") Feb 23 15:18:39 cephgw02 conmon[2158052]: Feb 23 15:18:39 cephgw02 conmon[2158052]: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) Feb 23 15:18:39 cephgw02 conmon[2158052]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7fccc9021cdc] Feb 23 15:18:39 cephgw02 conmon[2158052]: 2: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fccc92a642e] Feb 23 15:18:39 cephgw02 conmon[2158052]: 3: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fccc92a6601] Feb 23 15:18:39 cephgw02 conmon[2158052]: 4: (DispatchQueue::pre_dispatch(boost::intrusive_ptr<Message> const&)+0x710) [0x7fccc9259c30] Feb 23 15:18:39 cephgw02 conmon[2158052]: 5: (DispatchQueue::entry()+0xdeb) [0x7fccc925b69b] Feb 23 15:18:39 cephgw02 conmon[2158052]: 6: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fccc930bb71] Feb 23 15:18:39 cephgw02 conmon[2158052]: 7: /lib64/libpthread.so.0(+0x814a) [0x7fccc7dc314a] Feb 23 15:18:39 cephgw02 conmon[2158052]: 8: clone() Feb 23 15:18:39 cephgw02 conmon[2158052]: Feb 23 15:18:39 cephgw02 conmon[2158052]: debug 0> 2024-02-23T08:18:39.610+0000 7fccc03c0700 -1 *** Caught signal (Aborted) ** Feb 23 15:18:39 cephgw02 conmon[2158052]: in thread 7fccc03c0700 thread_name:ms_dispatch Feb 23 15:18:39 cephgw02 conmon[2158052]: Feb 23 15:18:39 cephgw02 conmon[2158052]: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) Feb 23 15:18:39 cephgw02 conmon[2158052]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fccc7dcdb20] Feb 23 15:18:39 cephgw02 conmon[2158052]: 2: gsignal() Feb 23 15:18:39 cephgw02 conmon[2158052]: 3: abort() Feb 23 15:18:39 cephgw02 conmon[2158052]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7fccc9021dad] Feb 23 15:18:39 cephgw02 conmon[2158052]: 5: (opera Feb 23 15:18:39 cephgw02 conmon[2158052]: tor<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fccc92a642e] Feb 23 15:18:39 cephgw02 conmon[2158052]: 6: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fccc92a6601] Feb 23 15:18:39 cephgw02 conmon[2158052]: 7: (DispatchQueue::pre_dispatch(boost::intrusive_ptr<Message> const&)+0x710) [0x7fccc9259c30] Feb 23 15:18:39 cephgw02 conmon[2158052]: 8: (DispatchQueue::entry()+0xdeb) [0x7fccc925b69b] Feb 23 15:18:39 cephgw02 conmon[2158052]: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fccc930bb71] Feb 23 15:18:39 cephgw02 conmon[2158052]: 10: /lib64/libpthread.so.0(+0x814a) [0x7fccc7dc314a] Feb 23 15:18:39 cephgw02 conmon[2158052]: 11: clone() ---snip--- But I can't really help here, hopefully someone else can chime in and interpret it. Zitat von nguyenvandiep@baoviet.com.vn:

https://drive.google.com/file/d/1OIN5O2Vj0iWfEMJ2fyHN_xV6fpknBmym/view?usp=s...

Pls check my mds log which generate by command

cephadm logs --name mds.cephfs.cephgw02.qqsavr --fsid 258af72a-cff3-11eb-a261-d4f5ef25154c _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi David, Could you pls helo me understand, Does it affect to RGW service ? And if something go bad, how can i rollback ?

look at ALL cephfs kernel clients (no effect on RGW) Le ven. 23 févr. 2024 à 16:38, <nguyenvandiep@baoviet.com.vn> a écrit :

And we dont have parameter folder

cd /sys/module/ceph/ [root@cephgw01 ceph]# ls coresize holders initsize initstate notes refcnt rhelversion sections srcversion taint uevent

My Ceph is 16.2.4 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Can you send sudo ceph -s and sudo ceph health detail Sent from Bloomberg Professional for iPhone ----- Original Message ----- From: nguyenvandiep@baoviet.com.vn To: ceph-users@ceph.io At: 02/23/24 20:27:53 UTC-05:00 Could you pls guide me more detail :( im very newbie in Ceph :( _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Do you have the possibility to stop/unmount cephfs clients ? If so, do that and restart the MDS. It should restart. Have the clients restart one by one and check that the MDS does not crash (by monitoring the logs) ________________________________________________________ Cordialement, *David CASIER* *Ligne directe: +33(0) 9 72 61 98 29* ________________________________________________________ Le sam. 24 févr. 2024 à 10:01, <nguyenvandiep@baoviet.com.vn> a écrit :

Hi Mathew

Pls chekc my ceph -s

ceph -s cluster: id: 258af72a-cff3-11eb-a261-d4f5ef25154c health: HEALTH_WARN 3 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available 1 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 21 pgs backfill_toofull 15 pool(s) nearfull 11 daemons have recently crashed

services: mon: 6 daemons, quorum cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h) mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz, cephgw03.aipvii mds: 1/1 daemons up osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 18 daemons active (2 hosts)

data: volumes: 0/1 healthy, 1 recovering pools: 15 pools, 1457 pgs objects: 36.87M objects, 25 TiB usage: 75 TiB used, 41 TiB / 116 TiB avail pgs: 17759672/110607480 objects misplaced (16.056%) 1055 active+clean 363 active+remapped+backfill_wait 18 active+remapped+backfilling 14 active+remapped+backfill_toofull 7 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

if rebalancing tasks have been launched it's not a big deal, but I don't think it's the priority. The priority being to get the MDS back on its feet. I haven't seen an answer to this question: can you stop/unmount cephfs clients or not ? There are other solutions but as you are not comfortable I am making the simplest one on the ceph side but not the most comfortable on the business side. There is no one (in Vietnam?) who could help you more seriously (and in a lasting way) ? Le sam. 24 févr. 2024 à 15:55, <nguyenvandiep@baoviet.com.vn> a écrit :

Hi David,

I ll follow your suggestion. Do you have Telegram ? If yes, could you pls add my Telegram, +84989177619. Thank you so much _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

It looks like you have quite a few problems I’ll try and address them one by one. 1) Looks like you had a bunch of crashes, from the ceph -s it looks like you don’t have enough MDS daemons running for a quorum. So you’ll need to restart the crashed containers. 2) It looks like you might have an interesting crush map. Allegedly you have 41TiB of space but you can’t finish rococering you have lots of PGs stuck as their destination is too full. Are you running homogenous hardware or do you have different drive sizes? Are all the weights set correctly? One you correct item 1 you’ll need to correct item 2 to get back to a healthy spot. Sent from Bloomberg Professional for iPhone ----- Original Message ----- From: nguyenvandiep@baoviet.com.vn To: ceph-users@ceph.io At: 02/24/24 09:01:22 UTC Hi Mathew Pls chekc my ceph -s ceph -s cluster: id: 258af72a-cff3-11eb-a261-d4f5ef25154c health: HEALTH_WARN 3 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available 1 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 21 pgs backfill_toofull 15 pool(s) nearfull 11 daemons have recently crashed services: mon: 6 daemons, quorum cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h) mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz, cephgw03.aipvii mds: 1/1 daemons up osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 18 daemons active (2 hosts) data: volumes: 0/1 healthy, 1 recovering pools: 15 pools, 1457 pgs objects: 36.87M objects, 25 TiB usage: 75 TiB used, 41 TiB / 116 TiB avail pgs: 17759672/110607480 objects misplaced (16.056%) 1055 active+clean 363 active+remapped+backfill_wait 18 active+remapped+backfilling 14 active+remapped+backfill_toofull 7 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Mr Anthony, pls check the output https://anotepad.com/notes/s7nykdmc

I will correct some small things: we have 6 nodes, 3 osd node and 3 gaeway node ( which run RGW, mds and nfs service) you r corrct, 2/3 osd node have ONE-NEW 10tib disk About your suggestion, add another osd host, we will. But we need to end this nightmare, my NFS folder which have 10tib data is down :( My ratio ceph osd dump | grep ratio full_ratio 0.95 backfillfull_ratio 0.92 nearfull_ratio 0.85

Thank you, Sir. But i think i ll wait for PG BACKFILLFULL finish, my boss is very angry now and will not allow me to add one more disk( this action make him think that ceph would take more time for recovering and rebalancing ). We want to wait volume recovering progress finish

HolySh*** First, we change the mon_max_pg_per_osd to 1000 About adding disk for cephosd02, for more detail , what is TO, sir ? I ll make conversation with my boss. To be honest, im thinking that the volume recovery progress will get problem...

Anthony is correct, this is what I was getting at as well when seeing your ceph -s output. More details in the Ceph docs here if you want to understand the details of why you need to balance your nodes. https://docs.ceph.com/en/quincy/rados/operations/monitoring-osd-pg/ But you need to get your core rados cluster healthy, then you can pivot to trying to bringing back your gateway services. Sent from Bloomberg Professional for iPhone ----- Original Message ----- From: Anthony D'Atri <anthony.datri@gmail.com> To: nguyenvandiep@baoviet.com.vn CC: ceph-users@ceph.io At: 02/24/24 15:46:24 UTC Your recovery is stuck because there are no OSDs that have enough space to accept data. Your second OSD host appears to only have 9 OSDs currently, so you should be able to add a 10TB OSD there without removing anything. That will enable data to move to all three of your 10TB OSDs.

On Feb 24, 2024, at 10:41 AM, nguyenvandiep@baoviet.com.vn wrote:

HolySh***

First, we change the mon_max_pg_per_osd to 1000

About adding disk for cephosd02, for more detail , what is TO, sir ? I ll make conversation with my boss. To be honest, im thinking that the volume recovery progress will get problem... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Sudo watch ceph -s You should see stats on the recovery and see PGs transition from all the backfill* states to active+clean Once you get everything active clean then we can focus on your rgws and MDSs Sent from Bloomberg Professional for iPhone ----- Original Message ----- From: nguyenvandiep@baoviet.com.vn To: ceph-users@ceph.io At: 02/24/24 16:14:12 UTC Thank you Matthew. Im following guidance from Mr Anthony and now my recovery progress speed is much faster. I will update my case day by day. Thank you so much _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Yes, Sir. We added 10TIb to cephosd02 node. Now the disk is IN, but DOWN state. What should we do now :( For additional, the recovery speed is x10 times :)

Low space hindering backfill (add storage if this doesn't resolve itself): 21 pgs backfill_toofull

^^^ Ceph even told you what you need to do ;) If your have recovery taking place and the numbers of misplaced objects and *full PGs/pools keeps decreasing, then yes wait. As for getting your MDS / CephFS going after that, I’ll defer to others.

On Feb 24, 2024, at 11:07 AM, nguyenvandiep@baoviet.com.vn wrote:

Hi Mr Anthony,

Forget it, the osd is UP and recovery speed is x10times

Amazing

And now we just wait, right ? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

You also might want to increase mon_max_pg_per_osd since you have a wide spread of OSD sizes. Default is 250. Set it to 1000.

On Feb 24, 2024, at 10:30 AM, Anthony D'Atri <anthony.datri@gmail.com> wrote:

Add a 10tb HDD to the third node as I suggested, that will help your cluster.

...

On Feb 24, 2024, at 10:29 AM, nguyenvandiep@baoviet.com.vn wrote:

I will correct some small things:

we have 6 nodes, 3 osd node and 3 gaeway node ( which run RGW, mds and nfs service) you r corrct, 2/3 osd node have ONE-NEW 10tib disk

About your suggestion, add another osd host, we will. But we need to end this nightmare, my NFS folder which have 10tib data is down :(

My ratio ceph osd dump | grep ratio full_ratio 0.95 backfillfull_ratio 0.92 nearfull_ratio 0.85 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

  Hello Van Diep,   I read this after you got out of trouble.   According to your ceph osd tree, it looks like your problems started when the ceph orchestrator created osd.29 on node 'cephgw03' because it looks very unlikely that you created a 100MB OSD on a node that's named after "GW".   You may have added the 'osds' label to the 'cephgw03' node and/or played with the service_type:osd and/or added the cephgw03 node to the crushmap, which triggered the creation of osd.29 by the orchestrator. cephgw03 node being part of the 'default' root bucket, other OSDs legitimately started to send objects to osd.29, way to small to accommodate them, PGs then becoming 'backfill_toofull'.   To get out of this situation, you could have:   $ ceph osd crush add-bucket closet root $ ceph osd crush move cephgw03 root=closet   This would have moved 'cephgw03' node out of the 'default' root and probably fixed your problem instantly. Regards, Frédéric. -----Message original----- De: Anthony <anthony.datri@gmail.com> à: nguyenvandiep <nguyenvandiep@baoviet.com.vn> Cc: ceph-users <ceph-users@ceph.io> Envoyé: samedi 24 février 2024 16:24 CET Sujet : [ceph-users] Re: [Urgent] Ceph system Down, Ceph FS volume in recovering There ya go. You have 4 hosts, one of which appears to be down and have a single OSD that is so small as to not be useful. Whatever cephgw03 is, it looks like a mistake. OSDs much smaller than, say, 1TB often aren’t very useful. Your pools appear to be replicated, size=3. So each of your cephosd* hosts stores one replica of each RADOS object. You added the 10TB spinners to only two of your hosts, which means that they’re only being used as though they were 4TB OSDs. That’s part of what’s going on. You want to add a 10TB spinner to cephosd02. That will help your situation significantly. After that, consider adding a cephosd04 host. Having at least one more failure domain than replicas lets you better use uneven host capacities.

On Feb 24, 2024, at 10:06 AM, nguyenvandiep@baoviet.com.vn wrote:

Hi Mr Anthony,

pls check the output

https://anotepad.com/notes/s7nykdmc _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Mr Anthony, Could you tell me more details about raising the full and backfullfull threshold is it ceph tell 'osd.*' injectargs --osd-max-backfills=2 --osd-recovery-max-active=6 ??

Hi Mathew, 1) We have 2 MDS service running before this nightmare. Now we trying to apply mds on 3 nodes, but all of them will stop within 2 minutes. 2) You are correct. We just add two 10TIB disk to cluster ( which currently have 27 x 4TIB disk), all of them have weight 1.0 About volume recovery and pg backfill full, will volume recovery process finish after pg backfill full process completed ? And do you have Telegram ? If yes, could you pls add my Telegram, +84989177619. Thank you so much

Feb 22 13:39:43 cephgw02 conmon[1340927]: log_file /var/lib/ceph/crash/2024-02-22T06:39:43.618845Z_78ee38bc-9115-4bc6-8c3a-4bf42284c970/log Feb 22 13:39:43 cephgw02 conmon[1340927]: --- end dump of recent events --- Feb 22 13:39:45 cephgw02 systemd[1]: ceph-258af72a-cff3-11eb-a261-d4f5ef25154c@mds.cephfs.cephgw02.qqsavr.service: Main process exited, code=exited, status=134/n/a Feb 22 13:39:45 cephgw02 systemd[1]: ceph-258af72a-cff3-11eb-a261-d4f5ef25154c@mds.cephfs.cephgw02.qqsavr.service: Failed with result 'exit-code'. Feb 22 13:39:55 cephgw02 systemd[1]: ceph-258af72a-cff3-11eb-a261-d4f5ef25154c@mds.cephfs.cephgw02.qqsavr.service: Service RestartSec=10s expired, scheduling restart. Feb 22 13:39:55 cephgw02 systemd[1]: ceph-258af72a-cff3-11eb-a261-d4f5ef25154c@mds.cephfs.cephgw02.qqsavr.service: Scheduled restart job, restart counter is at 4. Feb 22 13:39:55 cephgw02 systemd[1]: Stopped Ceph mds.cephfs.cephgw02.qqsavr for 258af72a-cff3-11eb-a261-d4f5ef25154c. Feb 22 13:39:55 cephgw02 systemd[1]: Starting Ceph mds.cephfs.cephgw02.qqsavr for 258af72a-cff3-11eb-a261-d4f5ef25154c... Feb 22 13:39:56 cephgw02 bash[1341570]: 636aedce612582726e7eb229eb63d0f641491994fc76f1de1828b61d07d6e8ac Feb 22 13:39:56 cephgw02 systemd[1]: Started Ceph mds.cephfs.cephgw02.qqsavr for 258af72a-cff3-11eb-a261-d4f5ef25154c. Feb 22 13:39:56 cephgw02 conmon[1341683]: debug 2024-02-22T06:39:56.151+0000 7fc4e3309780 0 set uid:gid to 167:167 (ceph:ceph) Feb 22 13:39:56 cephgw02 conmon[1341683]: debug 2024-02-22T06:39:56.151+0000 7fc4e3309780 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mds, pid 7

Glad to hear it all worked out for you! From: nguyenvandiep@baoviet.com.vn At: 02/26/24 05:32:32 UTC-5:00To: ceph-users@ceph.io Subject: [ceph-users] Re: [Urgent] Ceph system Down, Ceph FS volume in recovering Dear Mr Eugen, Mr Matthew, Mr David, Mr Anthony My System is UP. Thank you so much. We get many support from all of you., mazing, kindly support from Top professional in Ceph. Hope we have a chance to cooperate in the future. And If you travel to VietNam in future, let me know. I ll be your local tour guide and we can get some beers. Once again, Thank you ====the solution is===== Try : ceph config set mds mds_deny_all_reconnect true and restart MDS. After that (mds active) : ceph config rm mds mds_deny_all_reconnect ========From Mr David==== _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io