- ceph-users - lists.ceph.io

Multiple OSDs down, and won't come up (possibly related to other Nautilus issues)
by aoanla＠gmail.com 08 Apr '20

08 Apr '20

So, this is following on from a discussion in the #ceph IRC channel, where we seem to have reached the limit of what we can do. I have a ~15 node, 311 OSD cluster. (20 OSDs per node). The cluster is Nautilus - the 3 MONs + the first 8 OSD hosts were installed as Mimic and upgraded to Nautilus with ceph-ansible ; the remaining OSD hosts were added directly with Nautilus as they were only added in a few weeks ago. Yesterday, suddenly, about half of the OSDs (~140) were marked Down, and a number of slow operations were detected. Initially, examining the logs (and with a bit of help from IRC), I noticed that the ansible roles used to build the newer OSDs had configured chrony incorrectly, and their clocks were drifting. (There were BADAUTHORIZER errors in OSD logs, too.) I fixed the chrony configuration... and we (including people in IRC) expected everything to just... stabilise. Things have not stabilised, which leads me to suspect that there are other issues at play. After noticing a number of issues with mgrs deadlocking in Nautilus - eg https://tracker.ceph.com/issues/17170 https://tracker.ceph.com/issues/43048 - I tried stopping all mgrs and mons, and then slowly bringing them up. This has not helped. Interestingly, the OSDs with slow ops (some of which are marked down) report ops_in_flight which are "wait for new map", whilst the lead mon believes those same ops are timed out. (I can of course, telnet to every OSD, even the down ones, from other OSDs, including ones which report issues talking to them on the same port; and from the lead mon.) I am wondering if this is an example of: https://tracker.ceph.com/issues/44184 as we did create a new pool shortly after adding the new OSD host nodes... but it isn't clear from that ticket [or the discussion on this list] how to fix this, other than removing the pool - which I can't do, as we need this pool to exist, and the pool is replaces needs to be decomissioned. Can anyone advise what I should do next? At present, obviously, the cluster is unusable.

3 5

Maximum CephFS Filesystem Size
by DHilsbos＠performair.com 07 Apr '20

07 Apr '20

All; We set up a CephFS on a Nautilus (14.2.8) cluster in February, to hold backups. We finally have all the backups running, and are just waiting for the system reach steady-state. I'm concerned about usage numbers, in the Dashboard Capacity it shows the cluster as 37% used, while under Filesystems --> <FSName> --> Pools -_> <data> --> Usage, it shows 71% used. Does CephFS place a limit on the size of a CephFS? Is there a limit to how large a pool can be in Ceph? Where is the sizing discrepancy coming from, and do I need to address it? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 2

Re: ceph mds can't recall client caps anymore
by Patrick Donnelly 07 Apr '20

07 Apr '20

On Tue, Apr 7, 2020 at 3:36 AM alean Huang <woalean(a)gmail.com> wrote: > > hi, > i am appreciate you work very much, there is a problem about ceph mds i meet. > ceph version: luminous > the mds cache grow to 20G and limit is 4G after 'stat *' from client which mount by kernel(centos7 , 4.14). > there are 1000w files in this dir. mds log show recall_client_state does not work after recall some caps. > > log is like this: > 2020-04-07 17:08:36.841304 7f23ddd15700 10 mds.2.server recall_client_state: session client.2366013 172.16.200.20:0/4162573294 caps 6801282, leases 0 > 2020-04-07 17:08:36.841323 7f23ddd15700 15 mds.2.server session recall threshold (16384) hit at 0; skipping! > 2020-04-07 17:08:36.841326 7f23ddd15700 7 mds.2.server recalled (throttled) 0 client caps. > > mds_recall_max_decay_rate = 2.5 It looks like your setting for mds_recall_max_caps is larger than mds_recall_max_decay_threshold. Are you changing these configurations? If so, why? -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

Load on drives of different sizes in ceph
by Andras Pataki 07 Apr '20

07 Apr '20

Hi cephers, I'm looking for some advice on what to do about drives of different sizes in the same cluster. We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this. I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? Andras

4 4

virtual machines crashes after upgrade to octopus
by Lomayani S. Laizer 06 Apr '20

06 Apr '20

Hello, After upgrade our ceph cluster to octopus few days ago we are seeing vms crashes with below error. We are using ceph with openstack(rocky). Everything running ubuntu 18.04 with kernel 5.3. We seeing this crashes in busy vms. this is cluster was upgraded from nautilus. kernel: [430751.176904] fn-radosclient[3905]: segfault at da0801 ip 00007fe78e076686 sp 00007fe7697f9470 erro r 4 in librbd.so.1.12.0[7fe78de73000+5cb000] Apr 6 03:26:00 compute6 kernel: [430751.176922] Code: 00 64 48 8b 04 25 28 00 00 00 48 89 44 24 18 31 c0 48 85 db 0f 84 fa 00 00 00 8 0 bf 38 01 00 00 00 48 89 fd 0f 84 ea 00 00 00 <83> bb 20 3f 00 00 ff 0f 84 dd 00 00 00 48 8b 83 18 3f 00 00 48 8d Apr 6 03:26:11 compute6 libvirtd[1671]: 2020-04-06 03:26:11.955+0000: 1671: error : qemuMonitorIO:719 : internal error: End of file f rom qemu monitor

3 4

rgw multisite with https endpoints
by Richard Kearsley 06 Apr '20

06 Apr '20

Hi there I have a fairly simple ceph multisite configuration with 2 ceph clusters in 2 different datacenters in the same city The rgws have this config for ssl: rgw_frontends = civetweb port=7480+443s ssl_certificate=/opt/ssl/ceph-bundle.pem The certificate is a real issued certificate, not self signed I configured the multisite with the guide from https://docs.ceph.com/docs/nautilus/radosgw/multisite/ More or less ok so far, some learning curve but that's ok I can access and upload to buckets at both endpoints with s3 client using https - https://ceph01cs1.domain.com and https://ceph01cs2.domain.com - all good Now the problem seems to be when my zones in the zonegroup use https endpoints, e.g. { "id": "4c6774fb-01eb-41fe-a74a-c2693f8e69fc", "name": "eu", "api_name": "eu", "is_master": "true", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "zones": [ { "id": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "name": "ceph01cs1", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "fec1fec8-a3c1-454d-8ed2-2c1da45f9c33", "name": "ceph01cs2", "endpoints": [ "https://ceph01cs2.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "08921dd5-1523-41b6-908f-2f58aa38c969" } Meta syncs ok - buckets and users get created, but data doesn't, and period can be commited and appears on both clusters I can also curl between the two clusters over 443 However, data sync gets stuck on 'init': realm 08921dd5-1523-41b6-908f-2f58aa38c969 (world) zonegroup 4c6774fb-01eb-41fe-a74a-c2693f8e69fc (eu) zone 0c203df2-6f31-4ad1-a899-91f85bf34c4e (ceph01cs2) metadata sync no sync (zone is master) data sync source: fec1fec8-a3c1-454d-8ed2-2c1da45f9c33 (ceph01cs1) init full sync: 128/128 shards full sync: 0 buckets to sync incremental sync: 0/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] I find errors like: 2020-03-31 20:27:11.372 7f60c84e1700 0 RGW-SYNC:data:sync: ERROR: failed to init sync, retcode=-16 2020-03-31 20:27:29.548 7f60c84e1700 0 RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote data log shards 2020-03-31 20:29:48.499 7f60c94e3700 0 RGW-SYNC:meta: ERROR: failed to fetch all metadata keys If I change the endpoints in the zonegroup to plain http, e.g. http://ceph01cs1.domain.com:7480 and http://ceph01cs2.domain.com:7480 then sync starts! So my question, and I couldn't find any examples of people using https to sync.. are https endpoints supported with multisite? and why would meta work over https but not data? Many thanks Richard

3 3

can not visit dashboard of ceph mgr
by sz_cuitao＠163.com 06 Apr '20

06 Apr '20

hi: I enabled the dashboard module of ceph mgr and configed it,but it does not works. why it is? [cephuser@cadmin ~]$ sudo ceph mgr module enable dashboard module 'dashboard' is already enabled [cephuser@cadmin ~]$ sudo ceph config set mgr mgr/dashboard/server_addr 192.168.137.90 [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ sudo ceph config set mgr mgr/dashboard/server_port 6000 [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ sudo ceph dashboard ac-user-create admin ceph123 administrator [root@cadmin ~]# systemctl restart ceph-mgr@cadmin [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# netstat -an | grep 6000 [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# ceph mgr services {} [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# systemctl status ceph-mgr@cadmin ● ceph-mgr(a)cadmin.service - Ceph cluster manager daemon Loaded: loaded (/usr/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2020-04-05 18:57:44 CST; 5min ago Main PID: 86161 (ceph-mgr) CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr(a)cadmin.service └─86161 /usr/bin/ceph-mgr -f --cluster ceph --id cadmin --setuser ceph --setgroup ceph Apr 05 18:57:44 cadmin systemd[1]: Stopping Ceph cluster manager daemon... Apr 05 18:57:44 cadmin systemd[1]: Stopped Ceph cluster manager daemon. Apr 05 18:57:44 cadmin systemd[1]: Started Ceph cluster manager daemon. [root@cadmin ~]# sz_cuitao(a)163.com

2 1

osd with specifiying directories
by Micha 06 Apr '20

06 Apr '20

Hi, I want to try using object storage with java. Is it possible to set up osds with "only" directories as data destination (using cephadmin) , instead of whole disks? I have read through much of the docu but didn't found how to do it (if it's possible). Thanks Michael

3 5

Commands on cephfs mounts getting stuck in uninterruptible sleep
by david.piper＠metaswitch.com 06 Apr '20

06 Apr '20

Hello, I am seeing some commands running on CephFS mounts getting stuck in an uninterruptible sleep, at which point I can only terminate them by rebooting the client. Has anyone experienced anything similar and found a way to safe-guard against this? My mount is using the ceph kernel driver, with the following config in fstab: 10.225.44.236,10.225.44.237,10.225.44.238:6789:/albacore/system/deploy on /opt/dcl/deploy type ceph (rw,noatime,name=albacore,secret=<hidden>,acl,wsize=32768,rsize=32768,_netdev) The vast majority of commands complete successfully on the mounted filesystem but on one occasion a "chmod -R +r *" command hung indefinitely (despite having run successfully numerous times before). Attempts to terminate the process using `kill` fail. Repeated attempts to run the same command also get blocked in the same state. A `ps` command shows the processes are stuck in uninterruptable sleep: [root@svr01 albacore] ~> ps -Al | grep chmod 4 D 0 18657 18656 0 80 0 - 26998 rwsem_ pts/2 00:00:00 chmod 4 D 0 21835 1 0 80 0 - 26998 rwsem_ ? 00:00:00 chmod Ceph seems to be unaware of the hung process. There are no slow ops / ops in flight in either of the dump_ops_in_flight output on the server, or under sys/kernel/debug/ceph/ on the client. Similarly there are no logs in dmesg for the command / process. Ceph health reports no MDS issues, and there's nothing in the logs from my MDS from when the processes hung. The only method I've found of clearing the processes is to reboot my client. Has anyone got experience with this? Are there ceph mount options that would guard against this? Some details of the current setup: • ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) • We're using the ceph kernel driver, kernel: 5.5.7-1.el7.elrepo.x86_64 • The client server has 38 separate directories mounted, all from the same CephFS filesystem. • All 38 directories are mounted with the same config by three separate clients. • Mount config (in fstab): 10.225.44.236,10.225.44.237,10.225.44.238:6789:/albacore/system/deploy on /opt/dcl/deploy type ceph (rw,noatime,name=albacore,secret=<hidden>,acl,wsize=32768,rsize=32768,_netdev) Kind regards, Dave

1 0