- ceph-users - lists.ceph.io

Mon crash when client mounts CephFS
by Phil Merricks 16 Jun '21

16 Jun '21

Hey folks, I have deployed a 3 node dev cluster using cephadm. Deployment went smoothly and all seems well. If I try to mount a CephFS from a client node, 2/3 mons crash however. I've begun picking through the logs to see what I can see, but so far other than seeing the crash in the log itself, it's unclear what the cause of the crash is. Here's a log. <https://termbin.com/isaz>. You can see where the crash is occurring around the line that begins with "Jun 08 18:56:04 okcomputer podman[790987]:" I would welcome any advice on either what the cause may be, or how I can advance the analysis of what's wrong. Best regards Phil

3 3

JSON output schema
by Vladimir Prokofev 15 Jun '21

15 Jun '21

Good day. I'm writing some code for parsing output data for monitoring purposes. The data is that of "ceph status -f json", "ceph df -f json", "ceph osd perf -f json" and "ceph osd pool stats -f json". I also need support for all major CEPH releases, starting with Jewel till Pacific. What I've stumbled upon is that: - keys in JSON output are not present if there's no appropriate data. For example the key ['pgmap', 'read_bytes_sec'] will not be present in "ceph status" output if there's no read activity in the cluster; - some keys changed between versions. For example ['health']['status'] key is not present in Jewel, but is available in all the following versions; vice-versa, key ['osdmap', 'osdmap'] is not present in Pacific, but is in all the previous versions. So I need to get a list of all possible keys for all CEPH releases. Any ideas how this can be achieved? My only thought atm is to build a "failing" cluster with all the possible states and get a reference data out of it. Not only this is tedious work since it requires each possible cluster version, but it is also prone for error. Is there any publicly available JSON schema for output?

1 0

Re: Issues with Ceph network redundancy using L2 MC-LAG
by Serkan Çoban 15 Jun '21

15 Jun '21

Do you observe the same behaviour when you pull a cable? Maybe a flapping port might cause this kind of behaviour, other than that you should't see any network disconnects. Are you sure about LACP configuration, what is the output of 'cat /proc/net/bonding/bond0' On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Cephers, > > I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP > mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. > > Did i miss something important here? and how to really achieve networking HA in Ceph cluster? > > best regards, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 4

How to orch apply single site rgw with custom front-end
by Vladimir Brik 15 Jun '21

15 Jun '21

Hello How can I use ceph orch apply to deploy single site rgw daemons with custom frontend configuration? Basically, I have three servers in a DNS round-robin, each running a 15.2.12 rgw daemon with this configuration: rgw_frontends = civetweb num_threads=5000 port=443s ssl_certificate=/etc/ceph/rgw.crt I would like to deploy 16.2.4 rgw daemons, but I don't know how to configure them. When I used "ceph orch apply rgw <name> <placement>", it created a new entry in the monitor configuration database instead of using existing rgw_frontends entry. I am guessing that I need to name the config db entry correctly, but I don't know what name to use. Currently I have $ ceph config get client rgw_frontends civetweb num_threads=5000 port=443s ssl_certificate=/etc/ceph/rgw.crt Can anybody help? Thanks, Vlad

1 0

problem using gwcli; package dependancy lockout
by Philip Brown 15 Jun '21

15 Jun '21

I'm trying to update a ceph octopus install, to add an iscsi gateway, using ceph-ansible, and gwcli wont run for me. The ansible run went well.. but when I try to actually use gwcli, I get (blahblah) ImportError: No module named rados which isnt too surprising, since "python-rados" is not installed. HOWEVER. The ceph repos installed by ceph-ansible (5.0, the octopus release) are http://download.ceph.com/ceph-iscsi/3/rpm/el7/noarch which provides python3-rados This supposedly "obsoletes" python-rados. Except it doesnt, because python-rados is for python2. But even if it didnt.. there is no provided python v2 rados module in the ceph-iscsi repo, or the main ceph repo. and gwcli is still python 2. So, what am I supposed to do now? It seems like I need a python2 version of python rados, but the ceph repos dont provide one, so they basically provide a broken gwcli? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

Failover with 2 nodes
by nORKy 15 Jun '21

15 Jun '21

Hi, I'm building a lab with virtual machines. I build a set up with only 2 nodes, 2 osd per nodes and I have a host that use mount.cephfs Each 2 ceph nodes runs services mon + mgr + mds and has cephadm command. If I stop a node, all commands hang. Can't use dashboard, can't use ceph -s or any ceph command, and my cephfs on the third host stop to respond too (ex: with ls command) All come back, when I power on the stopped node Why is there no failover ?? Thanks you 'Jof

6 6

Re: Issues with Ceph network redundancy using L2 MC-LAG
by huxiaoyu＠horebdata.cn 15 Jun '21

15 Jun '21

My big worry is about, when a single link under a bond breaks, it breaks hardly such that the whole bond does not work. How to make it "failover" in such cases? best regards, samuel huxiaoyu(a)horebdata.cn From: Anthony D'Atri Date: 2021-06-15 18:22 To: huxiaoyu(a)horebdata.cn Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Which hash mode are you using on the hosts? layer 3+4 ? Are they set up active/active, or active/passive? I often see suboptimal bonding configurations that result in most or all traffic going over only one ilnk. > On Jun 15, 2021, at 9:19 AM, huxiaoyu(a)horebdata.cn wrote: > > Dear Cephers, > > I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP > mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. > > Did i miss something important here? and how to really achieve networking HA in Ceph cluster? > > best regards, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph PGs issues
by Aly, Adel 15 Jun '21

15 Jun '21

Dears, We have a ceph cluster with 4096 PGs out of with +100 PGs are not active+clean. On top of the ceph cluster, we have a ceph FS, with 3 active MDS servers. It seems that we can’t get all the files out of it because of the affected PGs. The object store has more than 400 million objects. When we do “rados -p cephfs-data ls”, the listing stops (hangs) after listing +11 million objects. When we try to access an object which we can’t copy, the rados command hangs forever: ls -I <filename> 2199140525188 printf "%x\n" 2199140525188 20006fd6484 rados -p cephfs-data stat 20006fd6484.00000000 (hangs here) This is the current status of the ceph cluster: health: HEALTH_WARN 1 MDSs report slow metadata IOs 1 MDSs report slow requests 1 MDSs behind on trimming 1 osds exist in the crush map but not in the osdmap *Reduced data availability: 22 pgs inactive, 22 pgs incomplete* 240324 slow ops, oldest one blocked for 391503 sec, daemons [osd.144,osd.159,osd.180,osd.184,osd.242,osd.271,osd.275,osd.278,osd.280,osd.332]... h ave slow ops. mons xyz01,xyz02 are low on available space services: mon: 4 daemons, quorum abc001,abc002,xyz02,xyz01 mgr: abc002(active), standbys: xyz01, xyz02, abc001 mds: cephfs-3/3/3 up {0=xyz02=up:active,1=abc001=up:active,2=abc002=up:active}, 1 up:standby osd: 421 osds: 421 up, 421 in; 7 remapped pgs data: pools: 2 pools, 4096 pgs objects: 403.4 M objects, 846 TiB usage: 1.2 PiB used, 1.4 PiB / 2.6 PiB avail pgs: 0.537% pgs not active 3968 active+clean 96 active+clean+scrubbing+deep+repair 15 incomplete 10 active+clean+scrubbing 7 remapped+incomplete io: client: 89 KiB/s rd, 13 KiB/s wr, 34 op/s rd, 1 op/s wr The 100+ PGs have been in this state for a long time already. Sometimes when we try to copy some files the rsync process hangs and we can’t kill it and from the process stack, it seems to be hanging on ceph i/o operation. # cat /proc/51795/stack [<ffffffffc184206d>] ceph_mdsc_do_request+0xfd/0x280 [ceph] [<ffffffffc181e92e>] __ceph_do_getattr+0x9e/0x200 [ceph] [<ffffffffc181eb08>] ceph_getattr+0x28/0x100 [ceph] [<ffffffffab853689>] vfs_getattr+0x49/0x80 [<ffffffffab8537b5>] vfs_fstatat+0x75/0xc0 [<ffffffffab853bc1>] SYSC_newlstat+0x31/0x60 [<ffffffffab85402e>] SyS_newlstat+0xe/0x10 [<ffffffffabd93f92>] system_call_fastpath+0x25/0x2a [<ffffffffffffffff>] 0xffffffffffffffff # cat /proc/51795/mem cat: /proc/51795/mem: Input/output error Any idea on how to move forward with debugging and fixing this issue so we can get the data out of the ceph FS? Thank you in advance. Kind regards, adel This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request.

2 3

Issues with Ceph network redundancy using L2 MC-LAG
by huxiaoyu＠horebdata.cn 15 Jun '21

15 Jun '21

Dear Cephers, I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. Did i miss something important here? and how to really achieve networking HA in Ceph cluster? best regards, Samuel huxiaoyu(a)horebdata.cn

1 0

CephFS mount fails after Centos 8.4 Upgrade
by Ackermann, Christoph 15 Jun '21

15 Jun '21

Hello all, after upgrading Centos clients to version 8.4 CephFS ( Kernel 4.18.0-305.3.1.el8 ) mount did fail. Message: *mount error 110 = Connection timed out* ..unfortunately the kernel log was flooded with zeros... :-( The monitor connection seems to be ok, but libceph said: kernel: libceph: corrupt full osdmap (-2) epoch off xxxxxxxxxxxxx of yyyyyyyyyyyyyyy After client VM restore with Kernel 4.18.0-240.22.1.el8_3.x86_64 everything runs well. Did someone recently upgrade clients to Centos 8.4? Best regards, Christoph

2 1