- ceph-users - lists.ceph.io

Doing minor version update of Ceph cluster with ceph-ansible and rolling-update playbook
by andreas.elvers+lists.ceph.io＠lfda.de 28 Sep '20

28 Sep '20

I want to update my mimic cluster to the latest minor version using the rolling-update script of ceph-ansible. The cluster was rolled out with that setup. So as long as ceph_stable_release stays on the current installed version (mimic) the rolling update script will do only a minor update. Is this assumption correct? The documentation (https://docs.ceph.com/projects/ceph-ansible/en/latest/day-2/upgrade.html) is short on this. Thanks! - Andreas

2 1

How OSD encryption affects latency/iops on NVMe, SSD and HDD
by tri＠postix.net 28 Sep '20

28 Sep '20

Hi all, For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't much detail available. From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys. Best, Tri Hoang

3 3

Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early
by Wido den Hollander 28 Sep '20

28 Sep '20

Hi, On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing OSDs randomly drop out of the cluster. Usually it's 2 to 4 OSDs spread out over different nodes. Each node has 16 OSDs and not all the failing OSDs are on the same node. The OSDs are marked as down and all they keep print in their logs: monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-06-04T07:57:17.706529-0400) Looking at their status through the admin socket: { "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", "whoami": 206, "state": "active", "oldest_map": 73697, "newest_map": 75795, "num_pgs": 19 } The message brought me to my own ticket I created 2 years ago: https://tracker.ceph.com/issues/23460 The first thing I've checked is NTP/time. Double, triple check this. All the times are in sync on the cluster. Nothing wrong there. Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. Restarting them brings them back right away and then within 24h some other OSDs will drop out. Has anybody seen this behavior with Octopus as well? Wido

2 1

online Assignment Help
by tillockwatson＠gmail.com 28 Sep '20

28 Sep '20

There are many, but we are the only one who has the largest amount of satisfied clients. Catering higher services to students is our solely main motivation and accomplishment. We have been successful in doing this for the past twelve years and our online assignment help services have gotten approval from the students. We perceive the necessity of the students and committed to essentially fulfil it anyhow. https://www.assignmentachievers.com/service-detail/assignment+help+online

1 0

Assignment Help Experts
by tillockwatson＠gmail.com 28 Sep '20

28 Sep '20

I'm good with writing; love to read, write, edit. I'm a versatile professional researcher, assignment help and academic writer with more than six years of writing experience.

1 0

[orch] redeploying OSDs
by mnaser＠vexxhost.com 27 Sep '20

27 Sep '20

Hi folks: I've been playing around with the new Ceph orchestrator and I've been running into an interesting limitation. I have not found any documented process or combination of things to do which can allow me to 're-adopt' cephadm deployed OSDs. Think use case of reinstalling operating systems, in the old world, we'd just run: `ceph-volume lvm activate` --all and we'll be good to go. Is there any equivalent right now? Thanks Mohammed

1 0

Fwd: [ceph-mgr - 15.2.4 octopus] the cpeh-mgr failed to get the correct status of all PGs
by HAO Xiong 26 Sep '20

26 Sep '20

Hi guys, We recently upgrade the ceph-mgr to 15.2.4, Octopus in our production clusters. The status of the cluster now is as follow: *# ceph versions* *{* * "mon": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 5* * },* * "mgr": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 3* * },* * "osd": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 1933* * },* * "mds": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 14* * },* * "overall": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 1955* * }* *}* Now we suffered some problems in this cluster: 1. it always took a significant longer time to get the result of `ceph pg dump`. 2. the ceph-exportor might failed to get cluster metrics. 3. sometimes the cluster showed a few inactive/down pgs but recovered very soon. We did a investigation on the ceph-mgr, didn't get the root cause yet. But there are some dispersed clews (I am not sure if they ca help): 1. the ms_dispatch thread is always busy with one core. 2. the msg size is significant larger than 40K. *2020-09-24T14:47:50.216+0000 7f8f811f6700 1 -- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] <== osd.3038 v2:{osd_ip}:6800/384927 431 ==== pg_stats(17 pgs tid 0 v 0) v2 ==== 42153+0+0 (secure 0 0 0) 0x55dae07c1800 con 0x55daf6dde400* 3. get some errors of "Fail to parse JSON result". *2020-09-24T15:47:42.739+0000 7f8f8da0f700 0 [devicehealth ERROR root] Fail to parse JSON result from daemon osd.1292 ()* 4. in the sending channel, we could see lots of faults. *2020-09-24T14:53:17.725+0000 7f8fa866e700 1 -- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] >> v1:{osd_ip}:0/1442957044 conn(0x55db38757400 legacy=0x55db03d8e800 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (909347879) for more than 900000000 us, fault.* *2020-09-24T14:53:17.725+0000 7f8fa866e700 1 --1- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] >> v1:{osd_ip}:0/1442957044 conn(0x55db38757400 0x55db03d8e800 :6801 s=OPENED pgs=1572189 cs=1 l=1).fault on lossy channel, failing* 5. or the mgr-fin thread would be busy with one core. [image: image.png] and from the perf dump we got: * "finisher-Mgr": { "queue_len": 1359862, "complete_latency": { "avgcount": 14, "sum": 40300.307764855, "avgtime": 2878.593411775 } },* Sorry about these clews are a little messy. Could you have any comments on this? Thanks. Regards, Hao

1 0

Re: rebalancing adapted during rebalancing with new updates?
by Martin Verges 26 Sep '20

26 Sep '20

Without knowing the source code and just from my observations, I would say everytime the osd map changes, the crush/pgmap tries to fix that. However a running backfill is not stopped and only backfill_wait would be reconsidered. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 26. Sept. 2020 um 13:33 Uhr schrieb Marc Roos < M.Roos(a)f1-outsourcing.eu>: > > When I add an osd rebalancing is taking place, lets say ceph relocates > 40 pg's. > > When I add another osd during rebalancing, when ceph has only relocated > 10 pgs and has to do still 30 pgs. > > What happens then: > > 1. Is ceph just finishing the relocation of these 30 pgs and then > calculates how the new environment with the newly added osd should be > relocated and starts relocating that. > > 2. or is ceph is finishing only the relocation of the pg it is currently > doing, and then recalculates immediately how pg's should be distributed > without finishing these 30 pg's it was planning to do. > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

rebalancing adapted during rebalancing with new updates?
by Marc Roos 26 Sep '20

26 Sep '20

When I add an osd rebalancing is taking place, lets say ceph relocates 40 pg's. When I add another osd during rebalancing, when ceph has only relocated 10 pgs and has to do still 30 pgs. What happens then: 1. Is ceph just finishing the relocation of these 30 pgs and then calculates how the new environment with the newly added osd should be relocated and starts relocating that. 2. or is ceph is finishing only the relocation of the pg it is currently doing, and then recalculates immediately how pg's should be distributed without finishing these 30 pg's it was planning to do.

1 0

Technical Assistance
by lirkasilmi＠nedoz.com 26 Sep '20

26 Sep '20

http://garmin-com-express.support/ http://webrootloginn.com/ http://www.123hpcom.de/ https://123hpcomsetup.us/ http://nortonlogins.com/ http://getgarminexpress.com/ https://canoncomijsetup.me/

1 0