June 2021 - ceph-users - lists.ceph.io

by nORKy

Hi, I'm building a lab with virtual machines. I build a set up with only 2 nodes, 2 osd per nodes and I have a host that use mount.cephfs Each 2 ceph nodes runs services mon + mgr + mds and has cephadm command. If I stop a node, all commands hang. Can't use dashboard, can't use ceph -s or any ceph command, and my cephfs on the third host stop to respond too (ex: with ls command) All come back, when I power on the stopped node Why is there no failover ?? Thanks you 'Jof

2 years, 11 months

6
6
0 0

Re: Issues with Ceph network redundancy using L2 MC-LAG

by huxiaoyu＠horebdata.cn

My big worry is about, when a single link under a bond breaks, it breaks hardly such that the whole bond does not work. How to make it "failover" in such cases? best regards, samuel huxiaoyu(a)horebdata.cn From: Anthony D'Atri Date: 2021-06-15 18:22 To: huxiaoyu(a)horebdata.cn Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Which hash mode are you using on the hosts? layer 3+4 ? Are they set up active/active, or active/passive? I often see suboptimal bonding configurations that result in most or all traffic going over only one ilnk. > On Jun 15, 2021, at 9:19 AM, huxiaoyu(a)horebdata.cn wrote: > > Dear Cephers, > > I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP > mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. > > Did i miss something important here? and how to really achieve networking HA in Ceph cluster? > > best regards, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 years, 11 months

1
0
0 0

ceph PGs issues

by Aly, Adel

Dears, We have a ceph cluster with 4096 PGs out of with +100 PGs are not active+clean. On top of the ceph cluster, we have a ceph FS, with 3 active MDS servers. It seems that we can’t get all the files out of it because of the affected PGs. The object store has more than 400 million objects. When we do “rados -p cephfs-data ls”, the listing stops (hangs) after listing +11 million objects. When we try to access an object which we can’t copy, the rados command hangs forever: ls -I <filename> 2199140525188 printf "%x\n" 2199140525188 20006fd6484 rados -p cephfs-data stat 20006fd6484.00000000 (hangs here) This is the current status of the ceph cluster: health: HEALTH_WARN 1 MDSs report slow metadata IOs 1 MDSs report slow requests 1 MDSs behind on trimming 1 osds exist in the crush map but not in the osdmap *Reduced data availability: 22 pgs inactive, 22 pgs incomplete* 240324 slow ops, oldest one blocked for 391503 sec, daemons [osd.144,osd.159,osd.180,osd.184,osd.242,osd.271,osd.275,osd.278,osd.280,osd.332]... h ave slow ops. mons xyz01,xyz02 are low on available space services: mon: 4 daemons, quorum abc001,abc002,xyz02,xyz01 mgr: abc002(active), standbys: xyz01, xyz02, abc001 mds: cephfs-3/3/3 up {0=xyz02=up:active,1=abc001=up:active,2=abc002=up:active}, 1 up:standby osd: 421 osds: 421 up, 421 in; 7 remapped pgs data: pools: 2 pools, 4096 pgs objects: 403.4 M objects, 846 TiB usage: 1.2 PiB used, 1.4 PiB / 2.6 PiB avail pgs: 0.537% pgs not active 3968 active+clean 96 active+clean+scrubbing+deep+repair 15 incomplete 10 active+clean+scrubbing 7 remapped+incomplete io: client: 89 KiB/s rd, 13 KiB/s wr, 34 op/s rd, 1 op/s wr The 100+ PGs have been in this state for a long time already. Sometimes when we try to copy some files the rsync process hangs and we can’t kill it and from the process stack, it seems to be hanging on ceph i/o operation. # cat /proc/51795/stack [<ffffffffc184206d>] ceph_mdsc_do_request+0xfd/0x280 [ceph] [<ffffffffc181e92e>] __ceph_do_getattr+0x9e/0x200 [ceph] [<ffffffffc181eb08>] ceph_getattr+0x28/0x100 [ceph] [<ffffffffab853689>] vfs_getattr+0x49/0x80 [<ffffffffab8537b5>] vfs_fstatat+0x75/0xc0 [<ffffffffab853bc1>] SYSC_newlstat+0x31/0x60 [<ffffffffab85402e>] SyS_newlstat+0xe/0x10 [<ffffffffabd93f92>] system_call_fastpath+0x25/0x2a [<ffffffffffffffff>] 0xffffffffffffffff # cat /proc/51795/mem cat: /proc/51795/mem: Input/output error Any idea on how to move forward with debugging and fixing this issue so we can get the data out of the ceph FS? Thank you in advance. Kind regards, adel This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request.

2 years, 11 months

2
3
0 0

Issues with Ceph network redundancy using L2 MC-LAG

by huxiaoyu＠horebdata.cn

Dear Cephers, I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. Did i miss something important here? and how to really achieve networking HA in Ceph cluster? best regards, Samuel huxiaoyu(a)horebdata.cn

2 years, 11 months

1
0
0 0

CephFS mount fails after Centos 8.4 Upgrade

by Ackermann, Christoph

Hello all, after upgrading Centos clients to version 8.4 CephFS ( Kernel 4.18.0-305.3.1.el8 ) mount did fail. Message: *mount error 110 = Connection timed out* ..unfortunately the kernel log was flooded with zeros... :-( The monitor connection seems to be ok, but libceph said: kernel: libceph: corrupt full osdmap (-2) epoch off xxxxxxxxxxxxx of yyyyyyyyyyyyyyy After client VM restore with Kernel 4.18.0-240.22.1.el8_3.x86_64 everything runs well. Did someone recently upgrade clients to Centos 8.4? Best regards, Christoph

2 years, 11 months

2
1
0 0

slow ops at restarting OSDs (octopus)

by Manuel Lausch

Hello, I implemented a new cluster with 48 Nodes á 24 OSDs. I have a replicated pool with 4 replica. The crushrule distributes the replicas to different racks. With this cluster I tested a upgrade from Nautilis (14.2.20) to Octopus (15.2.13). The update itself worked well until I began the restarts of the OSDs in the 4th rack. Since then I get slow ops while stopping OSDs. I think something happend here, after all replica partners are running on the new version. This issue remains after completing the upgrade. With Nautilus I had similar issues with slow ops when stopping OSDs. I could resolve this with the option „osd_fast_shutdown → false“. I let this option set to false while upgrading. For testing/debugging, I set this to true (default value) and got better results when stopping OSDs, but the problem is not completely vanished. Had someone else this problem and could fix it? What can I do to get rid of slow ops when resarting OSDs? All Servers are connected with 2x10G network links Manuel

2 years, 11 months

3
11
0 0

Re: ceph df: pool stored vs bytes_used -- raw or not?

by Konstantin Shalygin

> On 20 May 2021, at 19:47, Igor Fedotov <ifedotov(a)suse.de> wrote: > > which PR/ticket are you referring to? This thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW… <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW…> And this ticket https://tracker.ceph.com/issues/48385 <https://tracker.ceph.com/issues/48385> All pools show me stored == used, even if all osd in/up/weighted on 14.2.21 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 595 TiB 334 TiB 261 TiB 261 TiB 43.87 nvme 52 TiB 34 TiB 18 TiB 18 TiB 34.68 TOTAL 647 TiB 368 TiB 279 TiB 279 TiB 43.13 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL replicated_rbd 1 1024 29 TiB 7.74M 29 TiB 10.93 80 TiB erasure_rbd_meta 6 16 2.9 MiB 154 2.9 MiB 0 7.3 TiB erasure_rbd_data 7 512 44 TiB 11.64M 44 TiB 15.61 144 TiB .rgw.root 12 16 11 KiB 22 11 KiB 0 80 TiB default.rgw.control 13 16 0 B 8 0 B 0 80 TiB default.rgw.meta 14 16 45 KiB 211 45 KiB 0 7.3 TiB default.rgw.log 15 16 88 MiB 345 88 MiB 0 80 TiB default.rgw.buckets.index 16 16 6.3 GiB 717 6.3 GiB 0.03 7.3 TiB default.rgw.buckets.non-ec 17 16 810 KiB 5.74k 810 KiB 0 80 TiB default.rgw.buckets.data 18 512 15 TiB 33.99M 15 TiB 5.70 80 TiB replicated_rbd_nvme 19 256 3.3 TiB 880.95k 3.3 TiB 13.25 7.3 TiB unredundant_rbd_nvme 20 128 3.9 TiB 1.03M 3.9 TiB 15.27 11 TiB fs_data 21 256 11 TiB 12.32M 11 TiB 4.38 80 TiB fs_meta 22 16 4.8 GiB 636.79k 4.8 GiB 0.02 7.3 TiB fs_data_nvme 23 64 20 GiB 5.97k 20 GiB 0.09 7.3 TiB kubernetes_rbd 24 32 142 GiB 36.71k 142 GiB 0.06 80 TiB Thanks, k

2 years, 11 months

3
4
0 0

Module 'devicehealth' has failed:

by Torkil Svensgaard

Hi Looking at this error in v15.2.13: " [ERR] MGR_MODULE_ERROR: Module 'devicehealth' has failed: Module 'devicehealth' has failed: " It used to work. Since the module is always on I can't seem to restart it and I've found no clue as to why it failed. I've tried rebooting all hosts to no avail. Suggestions? Thanks, Torkil

2 years, 11 months

2
2
0 0

Upgrading ceph to latest version, skipping minor versions?

by Matt Larson

Looking at the documentation for ( https://docs.ceph.com/en/latest/cephadm/upgrade/) - I have a question on whether you need to sequentially upgrade for each minor versions, 15.2.1 -> 15.2.3 -> ... -> 15.2.XX? Can you safely upgrade by directly specifying the latest version from several minor versions behind? ceph orch upgrade start --ceph-version 15.2.13 -Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

2 years, 11 months

2
1
0 0

Updated ceph-osd package, now get -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]

by Matt Larson

I was working through updates of CentOS and packages on each of the Ceph storage nodes of a cluster, and I hit an issue after I updated a package `ceph-osd` on the original Ceph node of the cluster. 1. Before the updates, the Ceph cluster was running 15.2.3 version 2. After the update of the package with `dnf install ceph-osd`, I was unable to check the status any more with `ceph -s`: 2021-06-14T14:01:53.216-0500 7efdef7fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] [errno 13] RADOS permission denied (error connecting to the cluster) 3. The `ceph -v` command now shows as version 15.2.13. Based on other reported issues this could be due recent changes with security fixes ( https://docs.ceph.com/en/latest/security/CVE-2021-20288/ ) that were introduced in 15.2.11. This cluster had originally been built with `cephadm` tool and containerized daemons. How can I restore the ability to connect with the command-line `ceph` client to check the status and all other interactions? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

2 years, 11 months

1
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users June 2021