September 2020 - ceph-users

by Ricky Paul

Assignments help provide professional assignment writers’ assistance to peers so that they can complete their assignments before due dates. Assignments, indeed, are a necessary part of students’ education. For more info : https://www.greatassignmenthelp.com

3 years

1
0
0 0

Spirit Airlines Phone Number

by emmashiralee＠gmail.com

You can talk to our travel specialist to find out who will help you with all your queries to dial the <b><a href="https://spiritphonenumber.com/spirit-airlines-phone-number">Spirit Airlines Phone Number</a></b>. We also offer deals such as flight food, medical guidelines, flight timing, ticket cancellation, your flight status, travel packages, affordable tickets, refunds, problems with your seat, ticket booking, flight checks, and many more.

3 years

1
0
1 0

Ceph RGW Performance

by Dylan Griff

Hey folks! Just shooting this out there in case someone has some advice. We're just setting up RGW object storage for one of our new Ceph clusters (3 mons, 1072 OSDs, 34 nodes) and doing some benchmarking before letting users on it. We have 10Gb network to our two RGW nodes behind a single ip on haproxy, and some iperf testing shows I can push that much; latencies look okay. However, when using a small cosbench cluster I am unable to get more than ~250Mb of read speed total. If I add more nodes to the cosbench cluster it just spreads out the load evenly with the same cap Same results when running two cosbench clusters from different locations. I don't see any obvious bottlenecks in terms of the RGW server hardware limitations, but here I am asking for assistance so I don't put it past me missing something. I have attached one of my cosbench load files with keys removed, but I get similar results with different numbers of workers, objects, buckets, object sizes, and cosbench drivers. Does anyone have any pointers on what I could find to nail this bottleneck down? Am I wrong in expecting more throughput? Let me know if I can get any other info for you. Cheers, Dylan -- Dylan Griff Senior System Administrator CLE D063 RCS - Systems - University of Victoria

3 years

4
4
0 0

Doing minor version update of Ceph cluster with ceph-ansible and rolling-update playbook

by andreas.elvers+lists.ceph.io＠lfda.de

I want to update my mimic cluster to the latest minor version using the rolling-update script of ceph-ansible. The cluster was rolled out with that setup. So as long as ceph_stable_release stays on the current installed version (mimic) the rolling update script will do only a minor update. Is this assumption correct? The documentation (https://docs.ceph.com/projects/ceph-ansible/en/latest/day-2/upgrade.html) is short on this. Thanks! - Andreas

3 years

2
1
0 0

How OSD encryption affects latency/iops on NVMe, SSD and HDD

by tri＠postix.net

Hi all, For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't much detail available. From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys. Best, Tri Hoang

3 years

3
3
0 0

Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early

by Wido den Hollander

Hi, On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing OSDs randomly drop out of the cluster. Usually it's 2 to 4 OSDs spread out over different nodes. Each node has 16 OSDs and not all the failing OSDs are on the same node. The OSDs are marked as down and all they keep print in their logs: monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-06-04T07:57:17.706529-0400) Looking at their status through the admin socket: { "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", "whoami": 206, "state": "active", "oldest_map": 73697, "newest_map": 75795, "num_pgs": 19 } The message brought me to my own ticket I created 2 years ago: https://tracker.ceph.com/issues/23460 The first thing I've checked is NTP/time. Double, triple check this. All the times are in sync on the cluster. Nothing wrong there. Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. Restarting them brings them back right away and then within 24h some other OSDs will drop out. Has anybody seen this behavior with Octopus as well? Wido

3 years

2
1
0 0

online Assignment Help

by tillockwatson＠gmail.com

There are many, but we are the only one who has the largest amount of satisfied clients. Catering higher services to students is our solely main motivation and accomplishment. We have been successful in doing this for the past twelve years and our online assignment help services have gotten approval from the students. We perceive the necessity of the students and committed to essentially fulfil it anyhow. https://www.assignmentachievers.com/service-detail/assignment+help+online

3 years

1
0
0 0

Assignment Help Experts

by tillockwatson＠gmail.com

I'm good with writing; love to read, write, edit. I'm a versatile professional researcher, assignment help and academic writer with more than six years of writing experience.

3 years

1
0
0 0

[orch] redeploying OSDs

by mnaser＠vexxhost.com

Hi folks: I've been playing around with the new Ceph orchestrator and I've been running into an interesting limitation. I have not found any documented process or combination of things to do which can allow me to 're-adopt' cephadm deployed OSDs. Think use case of reinstalling operating systems, in the old world, we'd just run: `ceph-volume lvm activate` --all and we'll be good to go. Is there any equivalent right now? Thanks Mohammed

3 years

1
0
0 0

Fwd: [ceph-mgr - 15.2.4 octopus] the cpeh-mgr failed to get the correct status of all PGs

by HAO Xiong

Hi guys, We recently upgrade the ceph-mgr to 15.2.4, Octopus in our production clusters. The status of the cluster now is as follow: *# ceph versions* *{* * "mon": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 5* * },* * "mgr": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 3* * },* * "osd": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 1933* * },* * "mds": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 14* * },* * "overall": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 1955* * }* *}* Now we suffered some problems in this cluster: 1. it always took a significant longer time to get the result of `ceph pg dump`. 2. the ceph-exportor might failed to get cluster metrics. 3. sometimes the cluster showed a few inactive/down pgs but recovered very soon. We did a investigation on the ceph-mgr, didn't get the root cause yet. But there are some dispersed clews (I am not sure if they ca help): 1. the ms_dispatch thread is always busy with one core. 2. the msg size is significant larger than 40K. *2020-09-24T14:47:50.216+0000 7f8f811f6700 1 -- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] <== osd.3038 v2:{osd_ip}:6800/384927 431 ==== pg_stats(17 pgs tid 0 v 0) v2 ==== 42153+0+0 (secure 0 0 0) 0x55dae07c1800 con 0x55daf6dde400* 3. get some errors of "Fail to parse JSON result". *2020-09-24T15:47:42.739+0000 7f8f8da0f700 0 [devicehealth ERROR root] Fail to parse JSON result from daemon osd.1292 ()* 4. in the sending channel, we could see lots of faults. *2020-09-24T14:53:17.725+0000 7f8fa866e700 1 -- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] >> v1:{osd_ip}:0/1442957044 conn(0x55db38757400 legacy=0x55db03d8e800 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (909347879) for more than 900000000 us, fault.* *2020-09-24T14:53:17.725+0000 7f8fa866e700 1 --1- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] >> v1:{osd_ip}:0/1442957044 conn(0x55db38757400 0x55db03d8e800 :6801 s=OPENED pgs=1572189 cs=1 l=1).fault on lossy channel, failing* 5. or the mgr-fin thread would be busy with one core. [image: image.png] and from the perf dump we got: * "finisher-Mgr": { "queue_len": 1359862, "complete_latency": { "avgcount": 14, "sum": 40300.307764855, "avgtime": 2878.593411775 } },* Sorry about these clews are a little messy. Could you have any comments on this? Thanks. Regards, Hao

3 years

1
0
0 0

2023

2022

2021

2020

2019

ceph-users September 2020