- ceph-users - lists.ceph.io

msgr-v2 log flooding on OSD proceses
by Void Star Nill 06 Nov '20

06 Nov '20

Hello, I am running 14.2.13-1xenial version and I am seeing lot of logs from msgv2 layer on the OSDs. Attached are some of the logs. It looks like these logs are not controlled by the standard log level configuration, so I couldn't find a way to disable these logs. I am concerned that these logs may be hurting the system performance. Any inputs on how I can disable the level of these logs? Regards, Shridhar

1 0

Problem with checking mon for new map after upgrade
by Ingo Ebel 06 Nov '20

06 Nov '20

Hi, we upgraded our ceph cluster from 14.2.9 to 15.2.5 but osds with 15.2.5 are not joining the cluster after restart. They hang with "1234 tick checking mon for new map" The Systems are Centos 7.8 I tried everything i could think of. But nothing helped. The mons and mgrs are 15.2.5. Anyone an Idea want could cause this? 2020-11-05T15:18:13.142+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[5.c9s0( v 839'39171 (839'37583,839'39171] local-lis/les=1129/1130 n=495 ec=813/813 lis/c=1129/1081 les/c/f=1130/1082/0 sis=1210) [NONE,NONE,NONE,NONE,124,NONE,157,NONE,NONE,149,NONE]p124(4) r=-1 lpr=1212 pi=[1081,1210)/1 crt=839'39171 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[1~3]] state<Start>: transitioning to Stray 2020-11-05T15:18:13.143+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[1.736( v 1105'9685 (815'8100,1105'9685] local-lis/les=1131/1132 n=6 ec=719/719 lis/c=1131/1019 les/c/f=1132/1020/0 sis=1210 pruub=11.135073235s) [] r=-1 lpr=1212 pi=[1019,1210)/1 crt=1105'9685 lcod 0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Stray 2020-11-05T15:18:13.143+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[4.3fd( v 1109'105689 (1109'102600,1109'105689] local-lis/les=1131/1132 n=32 ec=740/740 lis/c=1131/1089 les/c/f=1132/1090/0 sis=1210 pruub=11.134586488s) [] r=-1 lpr=1212 pi=[1089,1210)/1 crt=1109'105689 lcod 0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Stray 2020-11-05T15:18:13.144+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[5.123s10( v 839'38976 (839'37410,839'38976] local-lis/les=1119/1120 n=498 ec=813/813 lis/c=1119/1064 les/c/f=1120/1065/0 sis=1210) [NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE]p? r=-1 lpr=1212 pi=[1064,1210)/1 crt=839'38976 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[1~3]] state<Start>: transitioning to Stray 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity public network bond0 numa node 2 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity cluster network bond1 numa node 0 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity public and cluster network numa nodes do not match 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity not setting numa affinity 2020-11-05T15:18:14.072+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:18:44.397+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:19:15.370+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:19:46.206+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:20:16.400+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:20:46.483+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map Regards -- Ingo Ebel Human knowledge belongs to the world. RadioTux.de - Internet-Radio rund um Linux und Open Source ## https://twitter.com/ingoebel ## https://keybase.io/savar ## Jabber: ingo.ebel(a)ingoebel.de

2 2

high latency after maintenance
by Marcel Kuiper 05 Nov '20

05 Nov '20

Hi We had a rack down for 2hours for maintenance. 5 storage nodes were involved. We had noout en norebalance flags set before the start of the maintenance When the systems were brought back online we noticed a lot of osds with high latency (in 20 seconds range) . Mostly osds that are not on the storage nodes that were down. It took about 20 minutes for things to settle down. We're running nautilus 14.2.11. The storage nodes run bluestore and have 9 x 8T HDD's and 3 x SSD for rocksdb. Each with 3 x 123G LV - Can anyone give a reason for these high latencies? - Is there a way to avoid or lower these latencies when bringing systems back into operation? Best Regards Marcel

1 0

Re: cephadm POC deployment with two networks, can't mount cephfs
by Juan Miguel Olmo Martinez 05 Nov '20

05 Nov '20

Hi Oliver Review this "step by step" guide to see if you forgot something: BR NFS: 1. chmod +x cephadm 2. ./cephadm bootstrap - Record dashboard user & password printed out at the end 3. ADD OTHER HOSTS (assuming 3+ total after adding) 4. ./cephadm shell 5. ceph orch apply osd --all-available-devices 6. ceph fs volume create test 1 7. ceph orch apply mds test 3 8. ceph nfs cluster create cephfs testnfs 9. ceph nfs cluster info testnfs - (verify hostname, ip and port are listed) - Record ip and port for later 10. ceph nfs export create cephfs test testnfs /cephfs 11. ceph auth ls - (check “client.testnfs1” keyring is present) 12. ceph nfs export get testnfs /cephfs - (should have output) 13. rados -p nfs-ganesha -N testnfs get export-1 - testnfs/cephnfs - (check that export was successfully created) 14. ceph nfs export ls testnfs - (should show pseudo path “/cephfs”) 15. Verify nfs export exists on dashboard - Login to dashboard with credentials from bootstrap - URL will be https://{host-ip}:8443/ - Navigate to NFS page - Table should contain the export you just created 16. Exit shell - Command should just be “exit” 17. systemctl status nfs-server - If service is listed as inactive, run “systemctl start nfs-server” - Run “systemctl status nfs-server”. Should now be active 18. sudo mount -t nfs -o port={nfs-port} {nfs-ip}:/cephfs /mnt - Port and ip should be from “ceph nfs cluster info testnfs” command ran earlier Ex: mount -t nfs -o port=2049 10.8.128.94:/cephfs /mnt/cephfs/ Then give mount command -> to check if its mounted #mount Output: 10.8.128.94:/cephfs on /mnt/cephfs type nfs4 (rw,relatime,seclabel,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.8.128.94,local_lock=none,addr=10.8.128.94) -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

1 0

RGW pubsub deprecation
by Yuval Lifshitz 05 Nov '20

05 Nov '20

Dear Community, Since Nautilus, we have 2 mechanisms for notifying 3rd parties on changes in buckets and objects: "bucket notifications" [1] and "pubsub" [2]. In "bucket notifications" (="push mode") the events are sent from the RGW to an external entity (kafka, rabbitmq etc.), while in "pubsub" (="pull mode") the events are synched with a special zone, where they are stored and could be later fetched by an external app. From communications that I've seen so far, users preferred to use "bucket notifications" over "pubsub". Since supporting both modes has maintenance overhead, I was considering deprecating "pubsub". However, before doing that I would like to see what the community has to say! So, if you are currently using pubsub, or plan to use it, as "pull mode" fits your usecase better than "push mode" please chime in. Yuval [1] https://docs.ceph.com/en/latest/radosgw/notifications/ [2] https://docs.ceph.com/en/latest/radosgw/pubsub-module/

1 0

cephadm POC deployment with two networks, can't mount cephfs
by Oliver Weinmann 05 Nov '20

05 Nov '20

Hi, I setup a small 3 node cluster as a POC. I bootstrapped the cluster with separate networks for frontend (public network 192.168.30.0/24) and backend (cluster network 192.168.41.0/24). 1st small question: After the bootstrap, I recognized that I had mixed up cluster and public network. :( Is there a way to fix this on a running cluster? Last resort I would rebuild the cluster. Never the less I can't mount cephfs on a Linux client using any of the two networks. My linux client is CentOS 7 (latest updates) and has 3 nics, two of them in one of the public and cluster networks. I bootstrapped the cluster using the following conf file to have two networks: /root/ceph.conf: [global] public network = 192.168.41.0/24 cluster network = 192.168.30.0/24 cephadm bootstrap -c /root/ceph.conf --mon-ip 192.168.30.11 I have 2 mons, one running on the bootstrap host (192.168.30.11 / 192.168.41.11) and one (gedaopl01 192.168.30.12/ 192.168.41.12) running on one of the 3 osds: [root@gedasvl02 ~]# ceph -s cluster: id: dad3c9fa-1ec7-11eb-94d6-005056b703af health: HEALTH_OK services: mon: 2 daemons, quorum gedasvl02,gedaopl01 (age 5h) mgr: gedasvl02.cspuee(active, since 12h), standbys: gedaopl01.llogef mds: cephfs:1 {0=cephfs.gedaopl03.prrkll=up:active} 1 up:standby osd: 3 osds: 3 up (since 11h), 3 in (since 11h) task status: scrub status: mds.cephfs.gedaopl03.prrkll: idle data: pools: 3 pools, 81 pgs objects: 29 objects, 2.2 KiB usage: 450 GiB used, 407 GiB / 857 GiB avail pgs: 81 active+clean [root@gedasvl02 ~]# ceph osd metadata 2 | grep addr "back_addr": "[v2:192.168.30.12:6800/3112350288,v1:192.168.30.12:6801/3112350288]", "front_addr": "[v2:192.168.41.12:6800/3112350288,v1:192.168.41.12:6801/3112350288]", "hb_back_addr": "[v2:192.168.30.12:6802/3112350288,v1:192.168.30.12:6803/3112350288]", "hb_front_addr": "[v2:192.168.41.12:6802/3112350288,v1:192.168.41.12:6803/3112350288]", Now when I try to mount cephfs from the linux client, the mount command is just stuck and runs into a timeout. I can ping the mon from the client on both IPs, public (192.168.41.12) and cluster (192.168.30.12) and I can also see packets coming in on the mon using tcpdump. What could be wrong here? I'm using fuse-cephfs. One more question regarding rebuilding the cluster using cephadm, is there a simple tear-down command? My bootstrap host is a VM so I can use snapshots, but the nodes I have to clean manually, by removing all pods and ceph directories. Best Regards, Oliver

1 0

File read are not completing and IO shows in bytes able to not reading from cephfs
by Amudhan P 05 Nov '20

05 Nov '20

Hi, In my test ceph octopus cluster I was trying to simulate a failure case of when client mounted cephfs thru kernel client and doing read and write process, shutting down entire cluster with OSD flags like no down, no out, no backfiling and no recovery. Cluster is 4 node composed of 3 mons, 2 mgr, 2 mds, 48 OSD's. Public IP range : 10.0.103.0 and Cluster IP range : 10.0.104.0 Write and Read got stalled after some time cluster was brought live and healthy. But when reading file thru kernel mount read start at above 100MB/s and suddenly drops to byte and continues for long. only error msg I could see in the client machine. [ 167.591095] ceph: loaded (mds proto 32) [ 167.600010] libceph: mon0 10.0.103.1:6789 session established [ 167.601167] libceph: client144519 fsid f8bc7682-0d11-11eb-a332-0cc47a5ec98a [ 272.132787] libceph: osd1 10.0.104.1:6891 socket closed (con state CONNECTING) What went wrong why is this issue.? regards Amudhan P

1 1

Ceph flash deployment
by Seena Fallah 05 Nov '20

05 Nov '20

Hi all, Does this guid is still valid for a bluestore deployment with nautilus or octopus? https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployments Thanks.

3 5

RBD image stuck and no erros on logs
by Salsa 04 Nov '20

04 Nov '20

Hi, This same error keeps happening to me: after writing some amount of data to an RBD image it gets stuck and no read or write operation on it works. Every operation hangs. I cannot resize, alter features, read or write data. I can mount it, but using parted or fdisk hangs indefinitely. In the end all I can do is remove the image. Again, I see no errors on the logs and Ceph's status is OK. I tried to alter some log levels, but still no helpful info. Is there anything I should check? Rados? -- Salsa Sent with ProtonMail Secure Email.

1 0

Re: Cephadm: module not found
by Nadiia Kotelnikova 04 Nov '20

04 Nov '20

Hi, i am experience the same problem. Could you please advise something how to resolve this issue? The fix should be shipped with 15.2.6 version of "ceph-common" or ceph version? I have my cluster in docker containers and systemd services. How can I upgrade cluster to 15.2.6 if the command for upgrading fails? sudo ceph orch upgrade start --ceph-version 15.2.5 Error ENOENT: Module not found

2 5