- ceph-users - lists.ceph.io

el6 / centos6 rpm's for luminous?
by Marc Roos 08 Oct '20

08 Oct '20

Nobody ever used luminous on el6?

1 0

What are mon.<hostname>-safe containers?
by Sebastian Luna Valero 08 Oct '20

08 Oct '20

Hi, When I run `ceph orch ps` I see a couple of containers running on our MON nodes whose names end with the `-safe` suffix, and I was wondering what they are? I couldn't find information about it in https://docs.ceph.com This cluster is running Ceph 15.2.5, recently upgraded from 15.2.4 Many thanks, Sebastian

1 0

Quick/easy access to rbd on el6
by Marc Roos 07 Oct '20

07 Oct '20

Normally I would install ceph-common.rpm and access some rbd image via rbdmap. What would be the best way to do this on an old el6? There is not even a luminous el6 on download.ceph.com.

1 0

Kubernetes Luminous client acting on Nautilus pool: protocol feature mismatch: missing 200000 (CEPH_FEATURE_MON_GV ?)
by Fulvio Galeazzi 07 Oct '20

07 Oct '20

Hallo all! I am configuring a new storage class on my Kubernetes cluster, pointing to a pool on a Ceph cluster which was recently upgraded to Nautilus (was Luminous). The old storage class points to a Luminous pool in a separate cluster and works fine. On the new one, I think I did the configuration properly, yet when creating a volume I get this: 2020-10-07 10:00:53.849128 7f2f8c6f9700 0 -- 10.2.3.13:0/3520982056 >> 10.2.3.23:6789/0 pipe(0x7f2f780008c0 sd=3 :60192 s=1 pgs=0 cs=0 l=1 c=0x7f2f780068e0).connect protocol feature mismatch, my 27ffffffefdfbfff < peer 27fddff8efacbfff missing 200000 Looking at https://ceph.io/geen-categorie/feature-set-mismatch-error-on-ceph-kernel-cl… looks like the 200000 corresponds to CEPH_FEATURE_MON_GV, which by the way is listed here https://github.com/ceph/ceph/pull/8214 as a feature which could/should be removed. Things being as I described, I guess it would be safe to change the value of the tunable, correct? Unfortunately, I was unable to find any way to achieve this... the trivial "ceph osd crush set-tunable mon_gv 0" does not work. Any idea, please, how to fix my error? Upgrading Ceph packages on Kubernetes workers (now at Luminous) would help, may be? Thanks! Fulvio -- Fulvio Galeazzi GARR-CSD Department skype: fgaleazzi70 tel.: +39-334-6533-250

1 0

Pool quotas vs. rados df vs. RBD images (phantom objects in pool?)
by Miroslav Kalina 07 Oct '20

07 Oct '20

Hello everyone, I have single Ceph cluster with multiple pools for purposes of Kubernetes PVs and Virtualization (biggest pool, using snapshots here). Every client cluster has its own pool with max_bytes and max_objects quotas in place. When quotas are reached, Ceph halts all write I/O (which I suppose is OK). I am trying to setup best possible monitoring to prevent reaching pool quotas, but I am really struggling to find correct metrics Ceph is driven by, because almost every command I try gives me different results (rados df, rbd du, ceph health detail) and I feel like quotas are not usable for this purpose. After some fiddling with test pool I have somehow got it into state, when there are some phantom objects in stats (maybe also written to OSDs) which cannot be listed, accessed or deleted, but are counted into pool quotas. -> ceph version ceph version 14.2.10 (9f0d3f5a3ce352651da4c2437689144fcbec0131) nautilus (stable) I have my test pool here (quotas 4k objects, 15 GiB data): -> ceph osd pool ls detail | grep mirektest pool 2 'mirektest' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 21360 lfor 0/10929/10927 flags hashpspool,selfmanaged_snaps max_bytes 16106127360 max_objects 4000 stripe_width 0 target_size_bytes 10737418240 application rbd Cluster health says pool it getting full: -> ceph health detail HEALTH_WARN 1 pools nearfull POOL_NEAR_FULL 1 pools nearfull pool 'mirektest' has 3209 objects (max 4000) pool 'mirektest' has 13 GiB (max 15 GiB) But pool looks completely empty (no output, so I tried JSON output also): -> rbd -p mirektest --format json ls [] -> rbd -p mirektest --format json du {"images":[],"total_provisioned_size":0,"total_used_size":0} I am not using pool snapshots: -> rados -p mirektest lssnap 0 snaps But there are some RADOS object in it: -> rados -p mirektest df POOL_NAME *USED**OBJECTS*CLONES *COPIES*MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR mirektest *192 KiB* *3209* 0 *9627* 0 0 0 23998515 192 GiB 1221030 297 GiB 0 B 0 B Okay, lets try to list them: -> rados -p mirektest ls --all rbd_directory rbd_info rbd_trash Thats weird, expected more than 3 objetcs. Let's check it's sizes and content: -> rados -p mirektest stat rbd_directory mirektest/rbd_directory mtime 2020-10-07 11:55:16.000000, size 0 -> rados -p mirektest stat rbd_info mirektest/rbd_info mtime 2019-12-05 12:11:39.000000, size 19 -> rados -p mirektest get rbd_info - overwrite validated -> rados -p mirektest stat rbd_trash mirektest/rbd_trash mtime 2020-10-07 11:55:17.000000, size 0 Do any of you have some idea what's happening here? I am trying to find way how to cleanup pool without interrupting existing content in general. Also I have probably no idea how to replicate this and it's not the story of every pool in cluster. There is e.g. pool which was used for Kubernetes but it has no issues: -> rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR pool12 192 KiB 3 0 9 0 0 0 8710688 807 GiB 5490880 576 GiB 0 B 0 B I don't know how to check other pools, because there is data in them . How can I rely on quotas when in empty cluster there is already 13GiB data and 3.2k objects (which btw corresponds because 3200 * 4MiB = 12.8 GiB). Also is there any way how to calculate same number of objects/bytes Ceph is using to enforce quotas? Thanks for you time if you read it until the end, really appreciated that! :-) Any idea or hint is welcome. -- Miroslav Kalina Systems development specialist miroslav.kalina(a)livesport.eu +420 773 071 848 Livesport s.r.o. Aspira Business Centre Bucharova 2928/14a, 158 00 Praha 5 www.livesport.eu

1 0

Slow ops on OSDs
by Kristof Coucke 07 Oct '20

07 Oct '20

Hi all, We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large. The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion. We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again. All seemed to go fine, till yesterday morning... IOs towards the system were slowing down. Diving onto the nodes we could see that the OSD daemons are consuming the CPU power, resulting in average CPU loads going near 10 (!). The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either. Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized). Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs) As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight... We are running Ceph 14.2.11 Who can give me some thoughts on how I can better pinpoint the bottle neck? Thanks Kristof

4 16

Ceph iSCSI Performance
by DHilsbos＠performair.com 06 Oct '20

06 Oct '20

All; I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible. We're talking 1/4 to 1/3 of our current solution. I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client). What can I look at to tune this, preferably on the iSCSI gateways? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

5 8

pool pgp_num not updated
by norman 06 Oct '20

06 Oct '20

Hi guys, When I update the pg_num of a pool, I found it not worked(no rebalanced), anyone know the reason? Pool's info: pool 21 'openstack-volumes-rs' replicated size 3 min_size 2 crush_rule 21 object_hash rjenkins pg_num 1024 pgp_num 512 pgp_num_target 1024 autoscale_mode warn last_change 85103 lfor 82044/82044/82044 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~1e6,1e8~300,4e9~18,502~3f,542~11,554~1a,56f~1d7] pool 22 'openstack-vms-rs' replicated size 3 min_size 2 crush_rule 22 object_hash rjenkins pg_num 512 pgp_num 512 pg_num_target 256 pgp_num_target 256 autoscale_mode warn last_change 84769 lfor 0/0/55294 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd The pgp_num_target is set, but pgp_num not set. I have scale out new OSDs and is backfilling before setting the value, is it the reason?

3 2

Re: Slow ops on OSDs
by Kristof Coucke 06 Oct '20

06 Oct '20

Thanks to @Anthony: Diving further I see that I probably was blinded by the CPU load... I see that some disks are very slow (so my first observations were incorrect), and the latency seen using iostat seems more or less the same as what we see in the dump_historic_ops. (+ 3s for r_await) So, it looks like a few OSDs are causing a bottleneck in the whole system. I'm now wondering what my options are to improve the performance... The main goal is to use the system again, and make sure write operations are not affected. - Putting weight on 0 for the slow OSDs (temporary)? This way they recovery can go on but new files are not written to that disk? - .... Still investigating...

2 1

CephFS user mapping
by René Bartsch 06 Oct '20

06 Oct '20

Hi, is there any documentation about mapping usernames, user-ids, groupnames and group-ids between hosts sharing the same CephFS storage? Thanx for any hint, Renne

4 3