- ceph-users - lists.ceph.io

fault tolerant about erasure code pool
by Zhenshi Zhou 28 Jun '20

28 Jun '20

Hi all, I'm going to deploy a cluster with erasure code pool for cold storage. There are 3 servers for me to set up the cluster, 12 OSDs on each server. Does that mean the data is secure while 1/3 OSDs of the cluster is down, or only 2 of the OSDs is down , if I set the ec profile with k=4 and m=2. Thanks

7 12

High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs
by Benoît Knecht 27 Jun '20

27 Jun '20

Hi, We have a Nautilus (14.2.9) Ceph cluster with two types of HDDs: - TOSHIBA MG07ACA14TE [1] - HGST HUH721212ALE604 [2] They're all bluestore OSDs with no separate DB+WAL and part of the same pool. We noticed that while the HGST OSDs have a commit latency of about 15ms, the Toshiba OSDs hover around 150ms (these values come from the `ceph_osd_commit_latency_ms` metric in Prometheus). On paper, it seems like those drives have very similar specs, so it's not clear to me why we're seeing such a large difference when it comes to commit latency. Has anyone had any experience with those Toshiba drives? Or looking at the specs, do you spot anything suspicious? And if you're running a Ceph cluster with various disk brands/models, have you ever noticed some of them standing out when looking at `ceph_osd_commit_latency_ms`? Thanks in advance for your feedback. Cheers, -- Ben [1]: https://toshiba.semicon-storage.com/content/dam/toshiba-ss/asia-pacific/doc… [2]: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/p…

15 27

bluestore_throttle_bytes
by Seena Fallah 27 Jun '20

27 Jun '20

Hi all. Can someone explain or reference a doc about the bluestore_throttle_bytes option? I don't find any docs for this config. Thanks.

1 0

CephFS: What is the maximum number of files per directory
by Martin Palma 27 Jun '20

27 Jun '20

Hi, What is the maximum number of files per directory? I could find the answer in the docs. Best, Martin

4 3

re Centos8 / octopus installation question
by Mazzystr 26 Jun '20

26 Jun '20

Can anyone explain why ktdryer's dev repo is still landing on production installs? [root@centos8 ~]# ./cephadm add-repo --release octopus INFO:root:Writing repo to /etc/yum.repos.d/ceph.repo... INFO:cephadm:Enabling EPEL... INFO:cephadm:Enabling supplementary copr repo ktdreyer/ceph-el8... [root@centos8 ~]# ls /etc/yum.repos.d/ CentOS-AppStream.repo CentOS-Debuginfo.repo CentOS-Media.repo ceph.repo epel.repo CentOS-Base.repo CentOS-Extras.repo CentOS-PowerTools.repo _copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8.repo epel-testing-modular.repo CentOS-centosplus.repo CentOS-fasttrack.repo CentOS-Sources.repo epel-modular.repo epel-testing.repo CentOS-CR.repo CentOS-HA.repo CentOS-Vault.repo epel-playground.repo [root@centos8 ~]# cat /etc/yum.repos.d/_copr\:copr.fedorainfracloud.org \:ktdreyer\:ceph-el8.repo [copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8] name=Copr repo for ceph-el8 owned by ktdreyer baseurl= https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/epel-8… type=rpm-md skip_if_unavailable=True gpgcheck=1 gpgkey= https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey… repo_gpgcheck=0 enabled=1 enabled_metadata=1 Thanks, /Chris C

1 0

Pointers in __crush_do_rule__ function of CRUSH mapper file
by Bobby 26 Jun '20

26 Jun '20

Hi all, I have a question regarding pointer variables used in the __crush_do_rule__ function of CRUSH __mapper.c__. Can someone please help me understand the purpose of following four pointer variables inside __crush_do_rule__: int *b = a + result_max; int *c = b + result_max; int *w = a; int *o = b; The function __crush_do_rule__ is below: /** * crush_do_rule - calculate a mapping with the given input and rule * @map: the crush_map * @ruleno: the rule id * @x: hash input * @result: pointer to result vector * @result_max: maximum result size * @weight: weight vector (for map leaves) * @weight_max: size of weight vector * @cwin: Pointer to at least map->working_size bytes of memory or NULL. */ int crush_do_rule(const struct crush_map *map, int ruleno, int x, int *result, int result_max, const __u32 *weight, int weight_max, void *cwin, const struct crush_choose_arg *choose_args) { int result_len; struct crush_work *cw = cwin; int *a = (int *)((char *)cw + map->working_size); int *b = a + result_max; int *c = b + result_max; int *w = a; int *o = b; int recurse_to_leaf; int wsize = 0; int osize; int *tmp; const struct crush_rule *rule; __u32 step; int i, j; int numrep; int out_size; Thanks Bobby !

1 0

v14.2.10 Nautilus released
by Abhishek Lekshmanan 26 Jun '20

26 Jun '20

We're happy to announce the tenth release in the Nautilus series. In addition to fixing a security-related bug in RGW, this release brings a number of bugfixes across all major components of Ceph. We recommend that all Nautilus users upgrade to this release. For a detailed changelog please refer to the ceph release blog at: https://ceph.io/releases/v14-2-10-nautilus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * RGW: Bucket notifications now support Kafka endpoints. This requires librdkafka of version 0.9.2 and up. Note that Ubuntu 16.04.6 LTS (Xenial Xerus) has an older version of librdkafka, and would require an update to the library. * The pool parameter `target_size_ratio`, used by the pg autoscaler, has changed meaning. It is now normalized across pools, rather than specifying an absolute ratio. For details, see :ref:`pg-autoscaler`. If you have set target size ratios on any pools, you may want to set these pools to autoscale `warn` mode to avoid data movement during the upgrade:: ceph osd pool set <pool-name> pg_autoscale_mode warn * The behaviour of the `-o` argument to the rados tool has been reverted to its orignal behaviour of indicating an output file. This reverts it to a more consistent behaviour when compared to other tools. Specifying object size is now accomplished by using an upper case O `-O`. * The format of MDSs in `ceph fs dump` has changed. * Ceph will issue a health warning if a RADOS pool's `size` is set to 1 or in other words the pool is configured with no redundancy. This can be fixed by setting the pool size to the minimum recommended value with:: ceph osd pool set <pool-name> size <num-replicas> The warning can be silenced with:: ceph config set global mon_warn_on_pool_no_redundancy false * RGW: bucket listing performance on sharded bucket indexes has been notably improved by heuristically -- and significantly, in many cases -- reducing the number of entries requested from each bucket index shard. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: b340acf629a010a74d90da5782a2c5fe0b54ac20 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

osd init authentication failed: (1) Operation not permitted
by Naumann, Thomas 26 Jun '20

26 Jun '20

Hi, in our production cluster (proxmox 5.4, ceph 12.2) there is an issue since yesterday. after an increase of a pool 5 OSDs do not start, status is "down/in", ceph health: HEALTH_WARN nodown,noout flag(s) set, 5 osds down, 128 osds: 123 up, 128 in. last lines of OSD-logfile: 2020-06-26 08:40:26.240005 7f6d245fff80 1 freelist init 2020-06-26 08:40:26.243779 7f6d245fff80 1 bluestore(/var/lib/ceph/osd/ceph-45) _open_alloc opening allocation metadata 2020-06-26 08:40:26.251501 7f6d245fff80 1 bluestore(/var/lib/ceph/osd/ceph-45) _open_alloc loaded 3.47TiB in 1 extents 2020-06-26 08:40:26.253058 7f6d245fff80 0 <cls> /mnt/big/pve/ceph/ceph-12.2.13/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2020-06-26 08:40:26.253309 7f6d245fff80 0 _get_class not permitted to load sdk 2020-06-26 08:40:26.256486 7f6d245fff80 0 _get_class not permitted to load kvs 2020-06-26 08:40:26.256611 7f6d245fff80 0 <cls> /mnt/big/pve/ceph/ceph-12.2.13/src/cls/hello/cls_hello.cc:296: loading cls_hello 2020-06-26 08:40:26.258362 7f6d245fff80 0 _get_class not permitted to load lua 2020-06-26 08:40:26.259850 7f6d245fff80 0 osd.45 46770 crush map has features 288514051259236352, adjusting msgr requires for clients 2020-06-26 08:40:26.259859 7f6d245fff80 0 osd.45 46770 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons 2020-06-26 08:40:26.259863 7f6d245fff80 0 osd.45 46770 crush map has features 1009089991638532096, adjusting msgr requires for osds 2020-06-26 08:40:26.305880 7f6d245fff80 0 osd.45 46770 load_pgs 2020-06-26 08:40:28.024638 7f6d245fff80 0 osd.45 46770 load_pgs opened 129 pgs 2020-06-26 08:40:28.024803 7f6d245fff80 0 osd.45 46770 using weightedpriority op queue with priority op cut off at 64. 2020-06-26 08:40:28.025741 7f6d245fff80 -1 osd.45 46770 log_to_monitors {default=true} 2020-06-26 08:40:28.028397 7f6d245fff80 -1 osd.45 46770 init authentication failed: (1) Operation not permitted Does anyone know how to fix this? -- Thomas Naumann Abteilung Netze und Kommunikation Otto-von-Guericke Universität Magdeburg Universitätsrechenzentrum Universitätsplatz 2 39106 Magdeburg fon: +49 391 67-58563 email: thomas.naumann(a)ovgu.de

2 1

Removing pool in nautilus is incredibly slow
by Francois Legrand 26 Jun '20

26 Jun '20

Hello, I am running ceph nautilus 14.2.8 I had to remove 2 pools (old cephfs data and metadata pool with 1024 pgs). The removal of the pools seems to take a incredible time to free the space (the data pool I deleted was more than 100 TB and in 36h I got back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~1h30 mn for a 32 GB image and writing 10MB in cephfs takes half a minute !!) which makes the cluster almost unusable. It seems that the removal of deleted pg is done by deep-scrubs according tohttps://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301 Also it has been reported that this could be a regression in nautilushttps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXFGJGDYZALG6TQ4QBVGGAJ/#W4M5XQRDBLXFGJGDYZALG6TQ4QBVGGAJ But I couldn't find a fix or a way to speedup (or slow down) the process and get back the cluster to a decent reactivity. Is there a way ? Thanks F.

7 21

ceph qos
by 展荣臻（信泰） 26 Jun '20

26 Jun '20

Hi everyone: There are two types of qos in ceph(one based on tokenbucket algorithm,another based on mclock ). Which one I can use in nautilus production environment ？Thank you

2 1