- ceph-users - lists.ceph.io

orch upgrade mgr starts too slow and is terminated?
by Kai Börnert 07 May '21

07 May '21

Hi all, upon updating to 16.2.2 via cephadm the upgrade is being stuck on the first mgr Looking into this via docker logs I see that it is still loading modules when it is apparently terminated and restarted in a loop When pausing the update, the mgr succeeds to start with the new version, however when resuming the update, it seems to try to update it again even tho it already has the new version, leading to the exact same loop. Is there some setting or workaround to increase the time before it is attempted to be redeployed, or can this behavior be caused by something else? Greetings, Kai

3 5

Re: fixing future rctimes
by David Rivera 07 May '21

07 May '21

Hi guys, Did anyone ever figure out how to fix rctime? I had a directory that was robocopied from a windows host that contained files with modified times in the future. Now the directory tree up to the root will not update rctime. Thanks, David

1 0

Ceph cluster not recover after OSD down
by Andres Rojas Guerrero 07 May '21

07 May '21

Hi, I have a Nautilus cluster version 14.2.6 , and I have noted that when some OSD go down the cluster doesn't start recover. I have checked that the option noout is unset. What could be the reason for this behavior? -- ******************************************************* Andrés Rojas Guerrero Unidad Sistemas Linux Area Arquitectura Tecnológica Secretaría General Adjunta de Informática Consejo Superior de Investigaciones Científicas (CSIC) Pinar 19 28006 - Madrid Tel: +34 915680059 -- Ext. 990059 email: a.rojas(a)csic.es ID comunicate.csic.es: @50852720l:matrix.csic.es *******************************************************

7 27

Slow performance and many slow ops
by codignotto 06 May '21

06 May '21

Hello, I have 6 hosts with 12 SSD disks on each host for a total of 72 OSD, I am using CEPH Octopos in its latest version, the deployment was done using ceph admin and containers according to the dosing, we are having some problems with performance of the cluster, I mount it on a proxmox cluster and on windows VMs I have the problem of the disks being 100% occupied with a simple browser opening, when I switch to another NFS storage for example everything goes back to normal, I have the CEPH cluster now mounted and with only 1 VM inside it, and we have the problem of slowness and slow ops, the network speed between the hosts in the cluster is 25Gb tested with iperf, between ceph and proxmox is 25Gb per host, someone already passed that? Many Tks

2 2

v16.2.3 Pacific released
by David Galloway 06 May '21

06 May '21

This is the third backport release in the Pacific series. We recommend all users update to this release. Notable Changes --------------- * This release fixes a cephadm upgrade bug that caused some systems to get stuck in a loop restarting the first mgr daemon. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-16.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 381b476cb3900f9a92eb95d03b4850b953cfd79a

1 0

How to find out why osd crashed with cephadm/podman containers?
by mabi 06 May '21

06 May '21

Hello, I have a small 6 nodes Octopus 15.2.11 cluster installed on bare metal with cephadm and I added a second OSD to one of my 3 OSD nodes. I started then copying data to my ceph fs mounted with kernel mount but then both OSDs on that specific nodes crashed. To this topic I have the following questions: 1) How can I find out why the two OSD crashed? because everything is in podman containers I don't know where are the logs to find out the reason why this happened. From the OS itself everything looks ok, there was no out of memory error. 2) I would assume the two OSD container would restart on their own but this is not the case it looks like. How can I restart manually these 2 OSD containers on that node? I believe this should be a "cephadm orch" command? The health of the cluster right now is: CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) PG_DEGRADED: Degraded data redundancy: 132518/397554 objects degraded (33.333%), 65 pgs degraded, 65 pgs undersized Thank your for your hints. Best regards, Mabi

2 2

Upgrade problem with cephadm
by fcid 06 May '21

06 May '21

Hello ceph community, I'm trying to upgrade a pacific (v16.2.0) cluster to the last version, but the upgrading process seems to be stuck. The mgr log (debug level) does not show any significant message regarding the upgrade, other than when it is started/paused/resumed/stopped. 2021-05-06T14:29:59.294725+0000 mgr.hostc.riclju (mgr.3935983) 35645 : cephadm [INF] Upgrade: Started with target docker.io/ceph/ceph:v16.2.2 2021-05-06T14:49:55.710023+0000 mgr.hostc.riclju (mgr.3935983) 36285 : cephadm [INF] Paused 2021-05-06T14:50:24.444742+0000 mgr.hostc.riclju (mgr.3935983) 36302 : cephadm [INF] Resumed 2021-05-06T14:51:36.888269+0000 mgr.hostc.riclju (mgr.3935983) 36349 : cephadm [INF] Upgrade: Paused upgrade to docker.io/ceph/ceph:v16.2.2 2021-05-06T14:51:50.411779+0000 mgr.hostc.riclju (mgr.3935983) 36357 : cephadm [INF] Upgrade: Resumed upgrade to docker.io/ceph/ceph:v16.2.2 2021-05-06T14:52:01.660682+0000 mgr.hostc.riclju (mgr.3935983) 36365 : cephadm [INF] Upgrade: Stopped It may be worth mentioning that last week I had trouble trying to deploy RGWs. It was not possible to deploy de RGWs using this command ceph orch apply rgw orbyta --realm=realma --zone=zonea --placement="2" So the following were used ceph orch daemon add rgw zonea --placement hostb ceph orch daemon add rgw zonea --placement hosta After those commands were issued the orchestrator would still not deploy the RGWs, unless the current MGR failed over to another standby MGR. After that, the RGWs where depoyed. Another problem I have is the refresh parameter of the orchestrator. The last time the daemons listed in ceph orch ps where refreshed is the last time a MGR was set to failed, and issuing ceph orch ps --refresh does not seem to update It looks like all those symptoms are related somehow, but I don't know how to dig further into the internals of the orchestrator to get more information. I greatly appreciate if you can point me in the right direction. Thank you, kind regards. -- AltaVoz <https://www.altavoz.net/> Fernando Cid Ingeniero de Operaciones www.altavoz.net <https://www.altavoz.net/> Ubicación AltaVoz Viña del Mar: 2 Poniente 355 of 53 <https://www.altavoz.net/altavoz/contacto> | +56 32 276 8060 <tel:+56322768060> Santiago: Antonio Bellet 292 of 701 <https://www.altavoz.net/altavoz/contacto> | +56 2 2585 4264 <tel:+562225854264>

1 0

Out of Memory after Upgrading to Nautilus
by Christoph Adomeit 06 May '21

06 May '21

I manage a historical cluster of severak ceph nodes with each 128 GB Ram and 36 OSD each 8 TB size. The cluster ist just for archive purpose and performance is not so important. The cluster was running fine for long time using ceph luminous. Last week I updated it to Debian 10 and Ceph Nautilus. Now I can see that the memory usage of each osd grows slowly to 4 GB each and once the system has no memory left it will oom-kill processes I have already configured osd_memory_target = 1073741824 . This helps for some hours but then memory usage will grow from 1 GB to 4 GB per OSD. Any ideas what I can do to further limit osd memory usage ? It would be good to keep the hardware running some more time without upgrading RAM on all OSD machines. Any Ideas ? Thanks Christoph

5 6

RGW Beast SSL version
by Glen Baars 06 May '21

06 May '21

Hello Ceph, Can you set the SSL min version? Such as TLS1.2? Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

1 0

Re: How to set bluestore_rocksdb_options_annex
by ceph＠elchaka.de 05 May '21

05 May '21

Hello Anthony, it was introduced in octopus 15.2.10 See: https://docs.ceph.com/en/latest/releases/octopus/ Do you know how you would set it in pacific? :) Guess, there shouldnt be much difference... Thank you Mehmet Am 28. April 2021 19:21:19 MESZ schrieb Anthony D'Atri <anthony.datri(a)gmail.com>: >I think that’s new with Pacific. > >> On Apr 28, 2021, at 1:26 AM, ceph(a)elchaka.de wrote: >> >> >> >> Hello, >> >> I have an octopus cluster and want to change some values - but i >cannot find any documentation on how to set values(multiple) with >> >> bluestore_rocksdb_options_annex >> >> Could someone give me some examples. >> I would like to do this like ceph config set ... >> >> Thanks in advice >> Mehmet >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 5