- ceph-users - lists.ceph.io

Can we deprecate FileStore in Quincy?
by Neha Ojha 29 Jun '21

29 Jun '21

Hello everyone, Given that BlueStore has been the default and more widely used objectstore since quite some time, we would like to understand whether we can consider deprecating FileStore in our next release, Quincy and remove it in the R release. There is also a proposal [0] to add a health warning to report FileStore OSDs. We discussed this topic in the Ceph Month session today [1] and there were no objections from anybody on the call. I wanted to reach out to the list to check if there are any concerns about this or any users who will be impacted by this decision. Thanks, Neha [0] https://github.com/ceph/ceph/pull/39440 [1] https://pad.ceph.com/p/ceph-month-june-2021

10 15

Where did links to official MLs are moved?
by Satoru Takeuchi 28 Jun '21

28 Jun '21

Today I visited Ceph's official site and found that links to `resources` page seemed to be missing. https://ceph.io/en/ In addition, this page is no longer exists. https://ceph.io/resources/ Could you tell me where did they are moved? Thanks, Satoru

1 0

Nic bonding (lacp) settings for ceph
by mhnx 28 Jun '21

28 Jun '21

Hello. Im looking for proper way for the nic bonding for ceph. I was using the bonding driver with default settings ad_select=stable(default) and hash algorithm layer2. With this settings osds are using only one port and lacp only usefull for different nodes because layer2 hash algorithm using Mac. I've changed ad_select to bandwitdh and both nic is in use now but layer2 hash prevents dual nic usage for between two nodes (because layer2 using only Mac ). People advice using layer2+3 for best performance but it has no effect on osds because mac and ip is the same. I've tried layer3+4 to split by ports instead mac and it works. But i dont know what will the effect and also my switch is layer2. With "iperf -Parallel 2" now i can reach 19gbit on 2x 10gbit nics. I think there is no way to use both nic without parallel Usage with different ports. If I use same port and too many process I can only use one port at a time. What settings are you using? What is the best for ceph?

4 6

radosgw user "check_on_raw" setting
by Jared Jacob 28 Jun '21

28 Jun '21

Hello, I am setting up user quotas and I would like to enable the check on raw setting for my user's quota. I can't find any documentation on how to change this setting in any of the ceph documents. Do any of you know how to change this setting? Possibly using radosgw-admin? Thanks in advance! Jared Jacob

2 1

NFS Ganesha ingress parameter not valid?
by Oliver Weinmann 26 Jun '21

26 Jun '21

Dear All, I have deployed the latest CEPH Pacific release in my lab and started to check out the new ?stable? NFS Ganesha features. First of all I'm a bit confused which method to actually use to deploy the NFS cluster: cephadm or ceph nfs cluster create? I used "nfs cluster create" for now and noticed a minor problem in the docs. https://docs.ceph.com/en/latest/cephfs/fs-nfs-exports/#cephfs-nfs The command is stated as: $ ceph nfs cluster create <clusterid> [<placement>] [--ingress --virtual-ip <ip>] while as it needs a type (cephfs) to be specified nfs cluster create <type> <clusterid> [<placement>] : Create an NFS Cluster Also I can't manage to use the --ingress --virtual-ip parameter. Every time I try to use it I get this: [root@cephboot~]# ceph nfs cluster create cephfs ec9e031a-cd10-11eb-a3c3-005056b7db1f --ingress --virtual-ip 192.168.9.199 Invalid command: Unexpected argument '--ingress' nfs cluster create <type> <clusterid> [<placement>] : Create an NFS Cluster Error EINVAL: invalid command So i just deployed a NFS cluster without a VIP. Maybe I'm missing something? What about this note in the docs: >> From Pacific, the nfs mgr module must be enabled prior to use. << I can't find any info on how to enable it. Maybe this is already the case? ceph nfs cluster create cephfs ec9e031a-cd10-11eb-a3c3-005056b7db1f "cephnode01" This seems to be working fine. I managed to connect a CentOS 7 VM and I can access the NFS export just fine. Great stuff. For testing I tried to attach the same NFS export to a standalone ESXi 6.5 Server. This also works, but its diskspace is shown as 0 bytes: I'm not sure if this supported or I'm missing something. I could not find any clear info in the docs only some reddit posts where users mentioned that they were able to use it with VMware. Thanks and Best Regards, Oliver

1 1

Ceph Disk Prediction module issues
by Justas Balcas 25 Jun '21

25 Jun '21

Hello Folks, We are running Ceph Octopus 15.2.13 release and would like to use the disk prediction module. So far issues we faced are: 1. Ceph documentation does not mention to install `ceph-mgr-diskprediction-local.noarch` 2. Even if I install the needed package, after mgr restart, it does not appear on Ceph cluster. Detailed log is here: gist:b687798ea97ef13e36d466f2d7b1470a <https://gist.github.com/juztas/b687798ea97ef13e36d466f2d7b1470a> . Ceph -s shows [1]. Are you aware of this issue and are there any workarounds? Thanks! [1] # ceph -s cluster: id: 12d9d70a-e993-464c-a6f8-4f674db35136 health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum ceph-mon-cms-1,ceph-mon-cms-2,ceph-mon-cms-3 (age 2d) mgr: no daemons active (since 11m) mds: cephfs:1 {0=ceph-mds-cms-1=up:active} 1 up:standby ceph health detail HEALTH_WARN no active mgr [WRN] MGR_DOWN: no active mgr

1 0

Re: Why you might want packages not containers for Ceph deployments
by Marc 25 Jun '21

25 Jun '21

> but our experience so > far has been a big improvement over the complexity of managing package > dependencies across even just a handful of distros Do you have some charts or docs that show this complexity problem, because I have problems understanding it. This is very likely due to that my understanding of ceph internals is limited. For instance my view of the osd daemon. Now working with logical volumes for writing/reading data and then you have osd<->osd,mon,mgr communication. What dependency hell is there to be expected? > (Podman has been > the only real culprit here, tbh, but I give them a partial pass as the > tool is relatively new.) Is it not better for the sake of stability, security and future support to choose something with a proven record?

1 0

speeding up EC recovery
by Andrej Filipcic 25 Jun '21

25 Jun '21

Hi, on a large cluster with ~1600 OSDs, 60 servers and using 16+3 erasure coded pools, the recovery after OSD failure (HDD) is quite slow. Typical values are at 4GB/s with 125 ops/s and 32MB object sizes, which then takes 6-8 hours, during that time the pgs are degraded. I tried to speed it up with osd advanced osd_max_backfills 32 osd advanced osd_recovery_max_active 10 osd advanced osd_recovery_op_priority 63 osd advanced osd_recovery_sleep_hdd 0.000000 which at least kept the iops/s at a constant level. The recovery does not seem to be cpu or memory bound. Is there any way to speed it up? While testing the recovery on replicated pools, it reached 50GB/s. In contrast, replacing the failed drive with a new one and re-adding the OSD is quite fast, with 1GB/s recovery rate of misplaced pgs, or ~120MB/s average HDD write speed, which is not very far from HDD throughput. Regards, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

2 1

Can not mount rbd device anymore
by Ml Ml 25 Jun '21

25 Jun '21

Hello List, oversudden i can not mount a specific rbd device anymore: root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0 root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out) Any idea how to debug that mount? Tcpdump does show some active traffic. Cheers, Michael

4 9

Missing objects in pg
by Vadim Bulst 25 Jun '21

25 Jun '21

Dear List, since my update yesterday from 14.2.18 to 14.2.20 i got an unhealthy cluster. As I remember right, it appeared after rebooting the second server. They are 7 missing objects from pgs of a cache pool (pool 3). This pool is now changed writeback to proxy and i'm not able to flush all objects. root@scvirt06:/home/urzadmin/ceph_issue# ceph -s cluster: id: 5349724e-fa96-4fd6-8e44-8da2a39253f7 health: HEALTH_ERR 7/15893342 objects unfound (0.000%) Possible data damage: 7 pgs recovery_unfound Degraded data redundancy: 21/47680026 objects degraded (0.000%), 7 pgs degraded, 7 pgs undersized client is using insecure global_id reclaim mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum scvirt03,scvirt06,scvirt01 (age 19h) mgr: scvirt04(active, since 21m), standbys: scvirt03, scvirt02 mds: scfs:1 {0=scvirt04=up:active} 1 up:standby-replay 1 up:standby osd: 54 osds: 54 up (since 17m), 54 in (since 10w); 7 remapped pgs task status: scrub status: mds.scvirt03: idle data: pools: 5 pools, 704 pgs objects: 15.89M objects, 49 TiB usage: 139 TiB used, 145 TiB / 285 TiB avail pgs: 21/47680026 objects degraded (0.000%) 7/15893342 objects unfound (0.000%) 694 active+clean 7 active+recovery_unfound+undersized+degraded+remapped 3 active+clean+scrubbing+deep io: client: 3.7 MiB/s rd, 6.6 MiB/s wr, 40 op/s rd, 31 op/s wr my cluster: scvirt01 - mon,osds scvirt02 - mgr,osds scvirt03 - mon,mgr,mds,osds scvirt04 - mgr,mds,osds scvirt05 - osds scvirt06 - mon,mds,osds log of osd.49: root@scvirt03:/home/urzadmin# tail -f /var/log/ceph/ceph-osd.49.log AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.64 GB write, 0.01 MB/s write, 0.54 GB read, 0.01 MB/s read, 6.5 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2021-06-24 08:53:08.865 7f88ab86c700 -1 log_channel(cluster) log [ERR] : 3.9 has 1 objects unfound and apparently lost 2021-06-24 08:53:08.865 7f88a505f700 -1 log_channel(cluster) log [ERR] : 3.1e has 1 objects unfound and apparently lost 2021-06-24 08:53:40.570 7f88ab86c700 -1 log_channel(cluster) log [ERR] : 3.9 has 1 objects unfound and apparently lost 2021-06-24 08:53:40.570 7f88a9067700 -1 log_channel(cluster) log [ERR] : 3.1e has 1 objects unfound and apparently lost 2021-06-24 08:54:45.042 7f88b487e700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2021-06-24 08:54:45.042 7f88b487e700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 85202.3 total, 600.0 interval Cumulative writes: 1148K writes, 8640K keys, 1148K commit groups, 1.0 writes per commit group, ingest: 1.24 GB, 0.01 MB/s Cumulative WAL: 1148K writes, 546K syncs, 2.10 writes per sync, written: 1.24 GB, 0.01 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 369 writes, 1758 keys, 369 commit groups, 1.0 writes per commit group, ingest: 0.41 MB, 0.00 MB/s Interval WAL: 369 writes, 155 syncs, 2.37 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 104.40 MB 0.8 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 67.8 2.89 2.70 6 0.482 0 0 L1 2/0 131.98 MB 0.5 0.2 0.1 0.1 0.2 0.1 0.0 1.8 149.9 120.9 1.53 1.41 1 1.527 2293K 140K L2 16/0 871.57 MB 0.3 0.3 0.1 0.3 0.3 -0.0 0.0 5.2 158.1 132.3 2.05 1.93 1 2.052 3997K 1089K Sum 21/0 1.08 GB 0.0 0.5 0.2 0.4 0.6 0.2 0.0 3.3 85.5 100.8 6.47 6.03 8 0.809 6290K 1229K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 If I run ceph pg repair 3.1e it doesn't change anything and i do not understand why these pgs are undersized. All OSDs are up. ceph.conf: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.10.144.0/24 filestore_xattr_use_omap = true fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7 mon_allow_pool_delete = true mon_cluster_log_file_level = info mon_host = 172.26.8.151,172.26.8.153,172.26.8.156 osd_journal_size = 5120 osd_pool_default_min_size = 1 public_network = 172.26.8.128/26 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.scvirt03] host = scvirt03 mds_standby_for_rank = 0 mds_standby_replay = true [mds.scvirt04] host = scvirt04 mds standby for name = pve [mds.scvirt06] host = scvirt06 mds_standby_for_rank = 0 mds_standby_replay = true [mon.scvirt01] public_addr = 172.26.8.151 [mon.scvirt03] public_addr = 172.26.8.153 [mon.scvirt06] public_addr = 172.26.8.156 ceph health detail: HEALTH_ERR 7/15893333 objects unfound (0.000%); Possible data damage: 7 pgs recovery_unfound; Degraded data redundancy: 21/47679999 objects degraded (0.000%), 7 pgs degraded, 7 pgs undersized; client is using insecure global_id reclaim; mons are allowing insecure global_id reclaim OBJECT_UNFOUND 7/15893333 objects unfound (0.000%) pg 3.1e has 1 unfound objects pg 3.1f has 1 unfound objects pg 3.1b has 1 unfound objects pg 3.15 has 1 unfound objects pg 3.16 has 1 unfound objects pg 3.b has 1 unfound objects pg 3.9 has 1 unfound objects PG_DAMAGED Possible data damage: 7 pgs recovery_unfound pg 3.9 is active+recovery_unfound+undersized+degraded+remapped, acting [49,52], 1 unfound pg 3.b is active+recovery_unfound+undersized+degraded+remapped, acting [43,52], 1 unfound pg 3.15 is active+recovery_unfound+undersized+degraded+remapped, acting [44,52], 1 unfound pg 3.16 is active+recovery_unfound+undersized+degraded+remapped, acting [43,51], 1 unfound pg 3.1b is active+recovery_unfound+undersized+degraded+remapped, acting [43,52], 1 unfound pg 3.1e is active+recovery_unfound+undersized+degraded+remapped, acting [49,51], 1 unfound pg 3.1f is active+recovery_unfound+undersized+degraded+remapped, acting [48,51], 1 unfound PG_DEGRADED Degraded data redundancy: 21/47679999 objects degraded (0.000%), 7 pgs degraded, 7 pgs undersized pg 3.9 is stuck undersized for 64516.343966, current state active+recovery_unfound+undersized+degraded+remapped, last acting [49,52] pg 3.b is stuck undersized for 64516.351507, current state active+recovery_unfound+undersized+degraded+remapped, last acting [43,52] pg 3.15 is stuck undersized for 64521.368841, current state active+recovery_unfound+undersized+degraded+remapped, last acting [44,52] pg 3.16 is stuck undersized for 64516.351599, current state active+recovery_unfound+undersized+degraded+remapped, last acting [43,51] pg 3.1b is stuck undersized for 64517.427120, current state active+recovery_unfound+undersized+degraded+remapped, last acting [43,52] pg 3.1e is stuck undersized for 64521.369635, current state active+recovery_unfound+undersized+degraded+remapped, last acting [49,51] pg 3.1f is stuck undersized for 64517.426392, current state active+recovery_unfound+undersized+degraded+remapped, last acting [48,51] AUTH_INSECURE_GLOBAL_ID_RECLAIM client is using insecure global_id reclaim client.admin at 172.26.8.154:0/3925203408 is using insecure global_id reclaim mds.scvirt04 at [v2:172.26.8.154:6836/3778505565,v1:172.26.8.154:6837/3778505565] is using insecure global_id reclaim AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED mons are allowing insecure global_id reclaim mon.scvirt03 has auth_allow_insecure_global_id_reclaim set to true mon.scvirt06 has auth_allow_insecure_global_id_reclaim set to true mon.scvirt01 has auth_allow_insecure_global_id_reclaim set to true ceph osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 284.51312 root default -2 48.75215 host scvirt01 0 hdd 9.09560 osd.0 up 1.00000 1.00000 3 hdd 9.09560 osd.3 up 1.00000 1.00000 6 hdd 9.09560 osd.6 up 1.00000 1.00000 9 hdd 9.09560 osd.9 up 1.00000 1.00000 12 hdd 9.09560 osd.12 up 1.00000 1.00000 42 nvme 0.97029 osd.42 up 1.00000 1.00000 43 nvme 0.97029 osd.43 up 1.00000 1.00000 44 nvme 0.97029 osd.44 up 1.00000 1.00000 37 ssd 0.36330 osd.37 up 1.00000 1.00000 -3 48.75215 host scvirt02 1 hdd 9.09560 osd.1 up 1.00000 1.00000 4 hdd 9.09560 osd.4 up 1.00000 1.00000 7 hdd 9.09560 osd.7 up 1.00000 1.00000 10 hdd 9.09560 osd.10 up 1.00000 1.00000 13 hdd 9.09560 osd.13 up 1.00000 1.00000 45 nvme 0.97029 osd.45 up 1.00000 1.00000 46 nvme 0.97029 osd.46 up 1.00000 1.00000 47 nvme 0.97029 osd.47 up 1.00000 1.00000 38 ssd 0.36330 osd.38 up 1.00000 1.00000 -4 48.75224 host scvirt03 2 hdd 9.09569 osd.2 up 1.00000 1.00000 5 hdd 9.09560 osd.5 up 1.00000 1.00000 8 hdd 9.09560 osd.8 up 1.00000 1.00000 11 hdd 9.09560 osd.11 up 1.00000 1.00000 14 hdd 9.09560 osd.14 up 1.00000 1.00000 48 nvme 0.97029 osd.48 up 1.00000 1.00000 49 nvme 0.97029 osd.49 up 1.00000 1.00000 50 nvme 0.97029 osd.50 up 1.00000 1.00000 39 ssd 0.36330 osd.39 up 1.00000 1.00000 -9 56.75706 host scvirt04 15 hdd 9.09560 osd.15 up 1.00000 1.00000 17 hdd 9.09560 osd.17 up 1.00000 1.00000 20 hdd 9.09560 osd.20 up 1.00000 1.00000 22 hdd 9.09560 osd.22 up 1.00000 1.00000 23 hdd 9.09560 osd.23 up 1.00000 1.00000 25 hdd 3.63860 osd.25 up 1.00000 1.00000 26 hdd 3.63860 osd.26 up 1.00000 1.00000 27 hdd 3.63860 osd.27 up 1.00000 1.00000 40 ssd 0.36330 osd.40 up 1.00000 1.00000 -11 56.75706 host scvirt05 16 hdd 9.09560 osd.16 up 1.00000 1.00000 18 hdd 9.09560 osd.18 up 1.00000 1.00000 19 hdd 9.09560 osd.19 up 1.00000 1.00000 21 hdd 9.09560 osd.21 up 1.00000 1.00000 24 hdd 9.09560 osd.24 up 1.00000 1.00000 28 hdd 3.63860 osd.28 up 1.00000 1.00000 29 hdd 3.63860 osd.29 up 1.00000 1.00000 30 hdd 3.63860 osd.30 up 1.00000 1.00000 41 ssd 0.36330 osd.41 up 1.00000 1.00000 -13 24.74245 host scvirt06 31 hdd 3.63860 osd.31 up 1.00000 1.00000 32 hdd 3.63860 osd.32 up 1.00000 1.00000 33 hdd 3.63860 osd.33 up 1.00000 1.00000 34 hdd 3.63860 osd.34 up 1.00000 1.00000 35 hdd 3.63860 osd.35 up 1.00000 1.00000 36 hdd 3.63860 osd.36 up 1.00000 1.00000 51 nvme 0.97029 osd.51 up 1.00000 1.00000 52 nvme 0.97029 osd.52 up 1.00000 1.00000 53 nvme 0.97029 osd.53 up 1.00000 1.00000 Regards, Vadim -- Vadim Bulst Universität Leipzig / URZ 04109 Leipzig, Augustusplatz 10 phone: +49-341-97-33380 mail: vadim.bulst(a)uni-leipzig.de

1 1