- ceph-users - lists.ceph.io

Re: Certificate for Dashboard / Grafana
by E Taka 25 Nov '20

25 Nov '20

FYI: I've found a solution for the Grafana Certificate. Just run the following commands: 1. ceph config-key set mgr/cephadm/grafana_crt -i <cert> ceph config-key set mgr/cephadm/grafana_key -i <key> 2. ceph orch redeploy grafana 3. ceph config set mgr mgr/dashboard/GRAFANA_API_URL https://ceph01.domain.tld:3000

1 0

osd crash: Caught signal (Aborted) thread_name:tp_osd_tp
by Milan Kupcevic 25 Nov '20

25 Nov '20

Hello, Three OSD daemons crash at the same time while processing the same object located in an rbd ec4+2 pool leaving a placement group in inactive down state. Soon after I start the osd daemons back up they crash again choking on the same object. ----------------------------8<------------------------------------ _dump_onode 0x5605a27ca000 4#7:8565da11:::rbd_data.6.a8a8356fd674f.00000000003dce34:head# nid 1889617 size 0x100000 (1048576) expected_object_size 0 expected_write_size 0 in 8 shards, 32768 spanning blobs ----------------------------8<------------------------------------ Please take a look at the attached log file. Ceph status reports: Reduced data availability: 1 pg inactive, 1 pg down Any hints on how to get this placement group back online would be greatly appreciated. Milan -- Milan Kupcevic Senior Cyberinfrastructure Engineer at Project NESE Harvard University FAS Research Computing

2 4

smartctl UNRECOGNIZED OPTION: json=o
by Tony Liu 24 Nov '20

24 Nov '20

Hi, With Ceph Octopus 15.2.5, here is the output of command "ceph device get-health-metrics SEAGATE_DL2400MM0159_WBM2WP2S". =============================== "20201123-000939": { "dev": "/dev/sde", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 231", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "seagate", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\nsmartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-147.el8.x86_64] (local build)\nCopyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org\n\n=======> UNRECOGNIZED OPTION: json=o\n\nUse smartctl -h to get a usage summary\n\n" }, =============================== Is this error expected? The disk is SAS HDD, why those keywords with "nvme_" prefix? Is the same keywords used for all types of disk? Where can I see the command line options for running smartctl? Thanks! Tony

3 2

Ceph on ARM ?
by Adrian Nicolae 24 Nov '20

24 Nov '20

Hi guys, I was looking at some Huawei ARM-based servers and the datasheets are very interesting. The high CPU core numbers and the SoC architecture should be ideal for a distributed storage like Ceph, at least in theory. I'm planning to build a new Ceph cluster in the future and my best case scenario right now is to buy 7 servers with 2 x Intel Silver 12-core or 2 x Gold 20-core CPUs, 32 SATA drives each. And of course - SSDs/NVMEs for wal/db and the metadata pools , 256GB RAM and so on. I'm curious however if the ARM servers are better or not for this use case (object-storage only). For example, instead of using 2xSilver/Gold server, I can use a Taishan 5280 server with 2x Kungpen 920 ARM CPUs with up to 128 cores in total . So I can have twice as many CPU cores (or even more) per server comparing with x86. Probably the price is lower for the ARM servers as well. Has anyone tested Ceph in such scenario ? Is the Ceph software really optimised for the ARM architecture ? What do you think about this ? Thanks !

11 11

Certificate for Dashboard / Grafana
by E Taka 24 Nov '20

24 Nov '20

Hello, I'm having difficulties with setting up the web certificates for the Dashboard on hostnames ceph*01..n*.domain.tld. I set the keys and crt with ceph-config-key. ceph-config-key get mgr/dashbord/crt shows the correct certificate, the same applies to mgr/dashbord/key, mgr/cephadm/grafana_key and mgr/cephadm/grafana_crt. The hosts were rebooted since then, I used different browsers etc. It is no cache or proxy problem. But: 1. The browser complains Grafana web page via https://ceph01.domain.tld:3000 about the certificate and uses the self-signed certificate from cephadm. 2. The Dashboard forwards always from https://ceph01.domain.tld:8443 to https://ceph01:8443, which means that the browser complains again, since it needs the FQDN. How do you handle this? Thanks, Erich

1 0

Unable to find further optimization, or distribution is already perfect
by Toby Darling 24 Nov '20

24 Nov '20

Hi We're having problems getting our erasure coded ec82pool to upmap balance. "ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)": 554 The pool consists of 20 nodes in 10 racks, each rack containing a pair of nodes 1@45*8TB drives and 1@10*16TB. https://pastebin.com/YLwu8VVi The problem is the 8TB drives are roughly 62-74% full, while the 16TB drives are 84-87% full. https://pastebin.com/j7Dx883i Neither osdmaptool or reweight-by-utilization are able to improve the distribution . There's an osdmap ftp://ftp.mrc-lmb.cam.ac.uk/pub/toby/osdmap.2135441. Any thoughts/pointers much appreciated. Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology

2 2

Documentation of older Ceph version not accessible anymore on docs.ceph.com
by Martin Palma 24 Nov '20

24 Nov '20

Hello, maybe I missed the announcement but why is the documentation of the older ceph version not accessible anymore on docs.ceph.com? Best, Martin

6 7

Cephfs snapshots and previous version
by Oliver Weinmann 24 Nov '20

24 Nov '20

Today I played with a samba gateway and cephfs. I couldn’t get previous versions displayed on a windows client and found very little info on the net how to accomplish this. It seems that I need a vfs module called ceph_snapshots. It’s not included in the latest samba version on Centos 8. by this I also noticed that there is no vfs ceph module. Are these modules not stable and therefore not included in centos8? I can compile them but I would like to know why they are not included. And one more question. Are there any plans to add samba gateway support to cephadm? Best regards, Oliver Von meinem iPhone gesendet

4 3

Re: 14. 2.15: Question to collection_list_legacy osd bug fixed in 14.2.15
by Konstantin Shalygin 24 Nov '20

24 Nov '20

No, you are not affected. Affected only clusters with mixed versions. k Sent from my iPhone > On 24 Nov 2020, at 18:25, Rainer Krienke <krienke(a)uni-koblenz.de> wrote: > > Hello, > > thanks for your answer. If I understand you correctly then only if I > upgrade from 14.2.11 to 14.2.(12|14) this could lead to problems. Does > this mean that running 14.2.13 what I have currently installed is not > affected? > > The release note says: > > * BlueStore: Fixes a bug in collection_list_legacy which makes pgs > inconsistent during scrub when running mixed versions of osds, prior to > 14.2.12 with newer. > > This sounds to me like if I create a new OSD having ceph vers < 14.2.15 > (like 14.2.13 in my case) installed and have OSDs created before 14.2.12 > and then create a new one I could be affected by inconsitent pgs during > scrubbing. > > Have a nice day > Rainer > >> Am 24.11.20 um 11:18 schrieb Konstantin Shalygin: >> This bug may be affect when you upgrade from 14.2.11 to 14.2.(12|14) with low speed (e.g. one by one node). If you already upgrade from 14.2.11 you just jump over this bug. >> >> >> k >> >> Sent from my iPhone >> >>>> On 24 Nov 2020, at 10:43, Rainer Krienke <krienke(a)uni-koblenz.de> wrote: >>> >>> Hello, >>> >>> I am running a productive ceph cluster with Nautilus 14.2.13. All OSDs >>> are bluestore and were created with a ceph version prior to 14.2.12. >>> >>> What I would like to know is how urgent I should consider the >>> collection_list_legacy bug since at the moment I am not going to add a >>> brand new OSD to the system. However any time a disk could fail and so I >>> would have to destroy the OSD with the failed disk and them with a new >>> disk run ceph-volume to create a new bluestore OSD. >>> >>> Would this scenario also lead to inconsistent pgs? >>> >>> Thanks >>> Rainer >>> >>>> Am 24.11.20 um 02:35 schrieb David Galloway: >>>> This is the 15th backport release in the Nautilus series. This release >>>> fixes a ceph-volume regression introduced in v14.2.13 and includes few >>>> other fixes. We recommend users to update to this release. >>>> >>>> For a detailed release notes with links & changelog please refer to the >>>> official blog entry at https://ceph.io/releases/v14-2-15-nautilus-released >>>> >>>> >>>> Notable Changes >>>> --------------- >>>> * ceph-volume: Fixes lvm batch --auto, which breaks backward >>>> compatibility when using non rotational devices only (SSD and/or NVMe). >>>> * BlueStore: Fixes a bug in collection_list_legacy which makes pgs >>>> inconsistent during scrub when running mixed versions of osds, prior to >>>> 14.2.12 with newer. >>>> * MGR: progress module can now be turned on/off, using the commands: >>>> `ceph progress on` and `ceph progress off`. >>>> >>>> >>>> Getting Ceph >>>> ------------ >>>> * Git at git://github.com/ceph/ceph.git >>>> * Tarball at http://download.ceph.com/tarballs/ceph-14.2.15.tar.gz >>>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ >>>> * Release git sha1: afdd217ae5fb1ed3f60e16bd62357ca58cc650e5 >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> >>> -- >>> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 >>> 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 >>> Web: http://userpages.uni-koblenz.de/~krienke >>> PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > > > -- > Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 > 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 > Web: http://userpages.uni-koblenz.de/~krienke > PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html

1 0

Prometheus monitoring
by Michael Thomas 24 Nov '20

24 Nov '20

I am gathering prometheus metrics from my (unhealthy) Octopus (15.2.4) cluster and notice a discrepency (or misunderstanding) with the ceph dashboard. In the dashboard, and with ceph -s, it reports 807 million objects objects: pgs: 169747/807333195 objects degraded (0.021%) 78570293/807333195 objects misplaced (9.732%) 24/101158245 objects unfound (0.000%) But in the prometheus metrics (and in ceph df), it reports almost a factor of 10 fewer objects (dominated by pool 7): # HELP ceph_pool_objects DF pool objects # TYPE ceph_pool_objects gauge ceph_pool_objects{pool_id="4"} 3920.0 ceph_pool_objects{pool_id="5"} 372743.0 ceph_pool_objects{pool_id="7"} 86972464.0 ceph_pool_objects{pool_id="8"} 9287431.0 ceph_pool_objects{pool_id="13"} 8961.0 ceph_pool_objects{pool_id="15"} 0.0 ceph_pool_objects{pool_id="17"} 4.0 ceph_pool_objects{pool_id="18"} 206.0 ceph_pool_objects{pool_id="19"} 8.0 ceph_pool_objects{pool_id="20"} 7.0 ceph_pool_objects{pool_id="21"} 22.0 ceph_pool_objects{pool_id="22"} 203.0 ceph_pool_objects{pool_id="23"} 4415522.0 Why are these two values different? How can I get the total number of objects (807 million) from the prometheus metrics? --Mike

1 0