- ceph-users - lists.ceph.io

Issues with Ceph network redundancy using L2 MC-LAG
by huxiaoyu＠horebdata.cn 15 Jun '21

15 Jun '21

Dear Cephers, I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. Did i miss something important here? and how to really achieve networking HA in Ceph cluster? best regards, Samuel huxiaoyu(a)horebdata.cn

1 0

CephFS mount fails after Centos 8.4 Upgrade
by Ackermann, Christoph 15 Jun '21

15 Jun '21

Hello all, after upgrading Centos clients to version 8.4 CephFS ( Kernel 4.18.0-305.3.1.el8 ) mount did fail. Message: *mount error 110 = Connection timed out* ..unfortunately the kernel log was flooded with zeros... :-( The monitor connection seems to be ok, but libceph said: kernel: libceph: corrupt full osdmap (-2) epoch off xxxxxxxxxxxxx of yyyyyyyyyyyyyyy After client VM restore with Kernel 4.18.0-240.22.1.el8_3.x86_64 everything runs well. Did someone recently upgrade clients to Centos 8.4? Best regards, Christoph

2 1

slow ops at restarting OSDs (octopus)
by Manuel Lausch 15 Jun '21

15 Jun '21

Hello, I implemented a new cluster with 48 Nodes á 24 OSDs. I have a replicated pool with 4 replica. The crushrule distributes the replicas to different racks. With this cluster I tested a upgrade from Nautilis (14.2.20) to Octopus (15.2.13). The update itself worked well until I began the restarts of the OSDs in the 4th rack. Since then I get slow ops while stopping OSDs. I think something happend here, after all replica partners are running on the new version. This issue remains after completing the upgrade. With Nautilus I had similar issues with slow ops when stopping OSDs. I could resolve this with the option „osd_fast_shutdown → false“. I let this option set to false while upgrading. For testing/debugging, I set this to true (default value) and got better results when stopping OSDs, but the problem is not completely vanished. Had someone else this problem and could fix it? What can I do to get rid of slow ops when resarting OSDs? All Servers are connected with 2x10G network links Manuel

3 11

Re: ceph df: pool stored vs bytes_used -- raw or not?
by Konstantin Shalygin 15 Jun '21

15 Jun '21

> On 20 May 2021, at 19:47, Igor Fedotov <ifedotov(a)suse.de> wrote: > > which PR/ticket are you referring to? This thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW… <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW…> And this ticket https://tracker.ceph.com/issues/48385 <https://tracker.ceph.com/issues/48385> All pools show me stored == used, even if all osd in/up/weighted on 14.2.21 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 595 TiB 334 TiB 261 TiB 261 TiB 43.87 nvme 52 TiB 34 TiB 18 TiB 18 TiB 34.68 TOTAL 647 TiB 368 TiB 279 TiB 279 TiB 43.13 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL replicated_rbd 1 1024 29 TiB 7.74M 29 TiB 10.93 80 TiB erasure_rbd_meta 6 16 2.9 MiB 154 2.9 MiB 0 7.3 TiB erasure_rbd_data 7 512 44 TiB 11.64M 44 TiB 15.61 144 TiB .rgw.root 12 16 11 KiB 22 11 KiB 0 80 TiB default.rgw.control 13 16 0 B 8 0 B 0 80 TiB default.rgw.meta 14 16 45 KiB 211 45 KiB 0 7.3 TiB default.rgw.log 15 16 88 MiB 345 88 MiB 0 80 TiB default.rgw.buckets.index 16 16 6.3 GiB 717 6.3 GiB 0.03 7.3 TiB default.rgw.buckets.non-ec 17 16 810 KiB 5.74k 810 KiB 0 80 TiB default.rgw.buckets.data 18 512 15 TiB 33.99M 15 TiB 5.70 80 TiB replicated_rbd_nvme 19 256 3.3 TiB 880.95k 3.3 TiB 13.25 7.3 TiB unredundant_rbd_nvme 20 128 3.9 TiB 1.03M 3.9 TiB 15.27 11 TiB fs_data 21 256 11 TiB 12.32M 11 TiB 4.38 80 TiB fs_meta 22 16 4.8 GiB 636.79k 4.8 GiB 0.02 7.3 TiB fs_data_nvme 23 64 20 GiB 5.97k 20 GiB 0.09 7.3 TiB kubernetes_rbd 24 32 142 GiB 36.71k 142 GiB 0.06 80 TiB Thanks, k

3 4

Module 'devicehealth' has failed:
by Torkil Svensgaard 15 Jun '21

15 Jun '21

Hi Looking at this error in v15.2.13: " [ERR] MGR_MODULE_ERROR: Module 'devicehealth' has failed: Module 'devicehealth' has failed: " It used to work. Since the module is always on I can't seem to restart it and I've found no clue as to why it failed. I've tried rebooting all hosts to no avail. Suggestions? Thanks, Torkil

2 2

Upgrading ceph to latest version, skipping minor versions?
by Matt Larson 15 Jun '21

15 Jun '21

Looking at the documentation for ( https://docs.ceph.com/en/latest/cephadm/upgrade/) - I have a question on whether you need to sequentially upgrade for each minor versions, 15.2.1 -> 15.2.3 -> ... -> 15.2.XX? Can you safely upgrade by directly specifying the latest version from several minor versions behind? ceph orch upgrade start --ceph-version 15.2.13 -Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

2 1

Updated ceph-osd package, now get -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
by Matt Larson 14 Jun '21

14 Jun '21

I was working through updates of CentOS and packages on each of the Ceph storage nodes of a cluster, and I hit an issue after I updated a package `ceph-osd` on the original Ceph node of the cluster. 1. Before the updates, the Ceph cluster was running 15.2.3 version 2. After the update of the package with `dnf install ceph-osd`, I was unable to check the status any more with `ceph -s`: 2021-06-14T14:01:53.216-0500 7efdef7fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] [errno 13] RADOS permission denied (error connecting to the cluster) 3. The `ceph -v` command now shows as version 15.2.13. Based on other reported issues this could be due recent changes with security fixes ( https://docs.ceph.com/en/latest/security/CVE-2021-20288/ ) that were introduced in 15.2.11. This cluster had originally been built with `cephadm` tool and containerized daemons. How can I restore the ability to connect with the command-line `ceph` client to check the status and all other interactions? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

1 1

Cephfs mount not recovering after icmp-not-reachable
by Simon Sutter 14 Jun '21

14 Jun '21

Hello everyone! We had a switch outage and the ceph kernel mount did not work anymore. This is the fstab entry: 10.99.10.1:/somefolder /cephfs ceph _netdev,nofail,name=cephcluster,secret=IsSecret 0 0 I reproduced it with disabling the vlan on the switch on which the ceph is reachable, which gives a icmp-not-reachable. I did this for five minutes, after that, "ls /cephfs" just gives a "permission denied" in dmesg i can see this: [ 1412.994921] libceph: mon1 10.99.10.4:6789 session lost, hunting for new mon [ 1413.009325] libceph: mon0 10.99.10.1:6789 session established [ 1452.998646] libceph: mon2 10.99.15.3:6789 session lost, hunting for new mon [ 1452.998679] libceph: mon0 10.99.10.1:6789 session lost, hunting for new mon [ 1461.989549] libceph: mon4 10.99.15.5:6789 socket closed (con state CONNECTING) --- [ 1787.045148] libceph: mon3 10.99.15.4:6789 socket closed (con state CONNECTING) [ 1787.062587] libceph: mon0 10.99.10.1:6789 session established [ 1787.086103] libceph: mon4 10.99.15.5:6789 session established [ 1814.028761] libceph: mds0 10.99.10.4:6801 socket closed (con state OPEN) [ 1815.029811] libceph: mds0 10.99.10.4:6801 connection reset [ 1815.029829] libceph: reset on mds0 [ 1815.029831] ceph: mds0 closed our session [ 1815.029833] ceph: mds0 reconnect start [ 1815.052219] ceph: mds0 reconnect denied [ 1815.052229] ceph: dropping dirty Fw state for ffff9d9085da1340 1099512175611 [ 1815.052231] ceph: dropping dirty+flushing Fw state for ffff9d9085da1340 1099512175611 [ 1815.273008] libceph: mds0 10.99.10.4:6801 socket closed (con state NEGOTIATING) [ 1816.033241] ceph: mds0 rejected session [ 1829.018643] ceph: mds0 hung [ 1880.088504] ceph: mds0 came back [ 1880.088662] ceph: mds0 caps renewed [ 1880.094018] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null i_snap_realm [ 1881.100367] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null i_snap_realm [ 2046.768969] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module. [ 2061.731126] ceph: get_quota_realm: ino (10000000afe.fffffffffffffffe) null i_snap_realm Is this a bug to report or wrong configuration? Did someone else had this before? To solve the problem, a simple remount does the trick. Thanks in advance Simon

3 2

CephFS design
by Szabo, Istvan (Agoda) 14 Jun '21

14 Jun '21

Hi, Can you suggest me what is a good cephfs design? I've never used it, only rgw and rbd we have, but want to give a try. Howvere in the mail list I saw a huge amount of issues with cephfs so would like to go with some let's say bulletproof best practices. Like separate the mds from mon and mgr? Need a lot of memory? Should be on ssd or nvme? How many cpu/disk ... Very appreciate it. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

7 11

cephadm failed in Pacific release: Unable to set up "admin" label
by Ralph Soika 14 Jun '21

14 Jun '21

hi, if you follow the latest install guide here <https://docs.ceph.com/en/latest/cephadm/install/> to install pacific release, the bootstrap command will print the following error message: .... ...... mgr epoch 13 is available Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... Ceph Dashboard is now available at: URL: https://ceph-1:8443/ User: admin Password: xxx Enabling client.admin keyring and conf on hosts with "admin" label Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16 -e NODE_NAME=tikal-ceph-1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d6c1ba28-cd0d-11eb-8b39-960000bd038e:/var/log/ceph:z -v /tmp/ceph-tmp9t3y8y39:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaepq1rte:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16 orch client-keyring set client.admin label:_admin /usr/bin/ceph: stderr Invalid command: client-keyring not in start|stop|restart|redeploy|reconfig /usr/bin/ceph: stderr orch start|stop|restart|redeploy|reconfig <service_name> : Start, stop, restart, redeploy, or reconfig an entire service (i.e. all daemons) /usr/bin/ceph: stderr Error EINVAL: invalid command Unable to set up "admin" label; assuming older version of Ceph ...... .... As a result the first node is not working correctly and you can not add mom hosts. I tried to set the label manually afterwards with : # ceph orch host label add ceph-1 _admin But this seems to have no effect. Is this problem known? So only solution seems to stay with the octopus release :-/ === Ralph

2 3