- ceph-users - lists.ceph.io

Ceph User + Dev Monthly June Meetup
by Neha Ojha 13 Jun '23

13 Jun '23

Hi everyone, This month's Ceph User + Dev Monthly meetup is on June 15, 14:00-15:00 UTC. We'd love to share details about the first Reef release candidate. Please feel free to add more topics to https://pad.ceph.com/p/ceph-user-dev-monthly-minutes. Hope to see you there! Thanks, Neha

1 0

First Reef release candidate - v18.1.0
by Neha Ojha 13 Jun '23

13 Jun '23

Hi everyone, This is the first release candidate for Reef. The Reef release comes with a new RockDB version (7.9.2) [0], which incorporates several performance improvements and features. Our internal testing doesn't show any side effects from the new version, but we are very eager to hear community feedback on it. This is the first release to have the ability to tune RockDB settings per column family [1], which allows for more granular tunings to be applied to different kinds of data stored in RocksDB. A new set of settings has been used in Reef to optimize performance for most kinds of workloads with a slight penalty in some cases, outweighed by large improvements in use cases such as RGW, in terms of compactions and write amplification. We would highly encourage community members to give these a try against their performance benchmarks and use cases. The detailed list of changes in terms of RockDB and BlueStore can be found in https://pad.ceph.com/p/reef-rc-relnotes. If any of our community members would like to help us with performance investigations or regression testing of the Reef release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/reef_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. Overall things are looking pretty good based on our testing. Please try it out and report any issues you encounter. Happy testing! Thanks, Neha Get the release from * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: c2214eb5df9fa034cc571d81a32a5414d60f0405 [0] https://github.com/ceph/ceph/pull/49006 [1] https://github.com/ceph/ceph/pull/51821

1 0

RGW striping configuration.
by Teja A 13 Jun '23

13 Jun '23

Hello I am working on an application that uses rados gateway to store objects onto a ceph cluster. I am currently working on optimizing the latency for storing/retrieving objects from the cluster. My goal to improve read/write latencies is to have RGW write/read multiple rados objects in parallel as described here <https://docs.ceph.com/en/latest/architecture/#data-striping> - "Significant write performance occurs when the client writes the stripe units to their corresponding objects in parallel.". Just like with RAID0, by having a large number of rados objects that each radosgw object gets mapped to, we can achieve lower latencies as we are not bound by the throughput of a single disk. That documentation suggests that we can configure the stripe count as well as stripe width which would let us indirectly control how many rados objects each radosgw object gets mapped to. I want to be able to change these parameters and run benchmarks against my pools. The key parameter I am therefore interested in controlling is the stripe count (i.e. the number of distinct objects each radosgw object is mapped to). More specifically, in the diagram <https://docs.ceph.com/en/latest/_images/ditaa-96a6fc80dad17fb53f161987ed64f…> attached to those docs, I see that the stripe_count is 4 (4 rados objects being written to for a single RGW object). I want to be able to experiment with varying numbers for that stripe_count. I am having trouble figuring out what configuration parameters exist in radosgw that lets me control this. I see that there is a stripe_width <https://github.com/ceph/ceph/blob/714cdc4e8767a153f825e857efdc28bb481528a1/…> and a rgw_max_chunk_size <https://docs.ceph.com/en/latest/radosgw/config-ref/#confval-rgw_max_chunk_s…> but did not find anything for stripe_count. Configuring the stripe_width alone is not sufficient as I would need to set the stripe_unit size as well to get the desired number for stripe_count, but I did not find either. Am I understanding that correctly? If so, can someone please point me to where and how this configuration should be set? I appreciate your help!

2 2

RGW: exposing multi-tenant
by Rok Jaklič 13 Jun '23

13 Jun '23

Hi, are there any drawbacks of exposing multi-tenant deployment of RGWs directly to users so they can use any S3 client to connect to service or should we put something in front of RGWs? How many users in multi-tenant deployment can CEPH handle? Kind regards, Rok

1 0

Announcing go-ceph v0.22.0
by Anoop C S 13 Jun '23

13 Jun '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.22.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S

1 0

ceph fs perf stats output is empty
by Denis Polom 13 Jun '23

13 Jun '23

Hi I'm running latest Ceph Pacific 16.2.13 with Cephfs. I need to collect performance stats per client, but getting empty list without any numbers I even run dd on client against mounted ceph fs, but output is only like this: #> ceph fs perf stats 0 4638 192.168.121.1 {"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", " opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metada ta_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} Do I need to set some extra options? Does it work for some of you guys? Thank you dp

3 4

Monitor mkfs with keyring failed
by Michel Niyoyita 12 Jun '23

12 Jun '23

Hello Team , I am trying to make a ceph cluster with 3 nodes running ubuntu 20.04 and is configured using ceph-ansible , because it is a testing cluster OSD servers are same which will run as monitors . while installation I am facing following error : TASK [ceph-mon : ceph monitor mkfs with keyring] ************************************************************************************************* Monday 12 June 2023 15:48:18 +0000 (0:00:00.089) 0:01:20.254 *********** fatal: [ceph-osd1]: FAILED! => changed=true cmd: - ceph-mon - --cluster - ceph - --setuser - ceph - --setgroup - ceph - --mkfs - -i - ceph-osd1 - --fsid - d80c5efc-1d3e-4054-a6a4-3855dddd3543 - --keyring - /var/lib/ceph/tmp/ceph.mon..keyring delta: '0:00:00.282151' end: '2023-06-12 15:48:19.062350' msg: non-zero return code rc: -6 start: '2023-06-12 15:48:18.780199' stderr: |- /build/ceph-16.2.13/src/mon/MonMap.h: In function 'void MonMap::add(const mon_info_t&)' thread 7ffae5cb8700 time 2023-06-12T15:48:18.810345+0000 if you face this before Kindly help to solve . Regards Michel

1 0

bucket notification retries
by Yuval Lifshitz 12 Jun '23

12 Jun '23

Dear Community, I would like to collect your feedback on this issue. This is a followup from a discussion that started in the RGW refactoring meeting on 31-May-23 (thanks @Krunal Chheda <kchheda3(a)bloomberg.net> for bringing up this topic!). Currently persistent notifications are retried indefinitely. The only limiting mechanism that exists is that all notifications to a specific topic are stored in one RADOS object (of size 128MB). Assuming notifications are ~1KB at most, this would give us at least 128K notifications that can wait in the queue. When the queue fills up (e.g. kafka broker is down for 20 minutes, we are sending ~100 notifications per second) we start sending "slow down" replies to the client, and in this case the S3 operation will not be performed. This means that, for example, an outage of the kafka system would eventually cause an outage of our service. Note that this may also be a result of a misconfiguration of the kafka broker, or decommissioning of a broker. To avoid that, we propose several options: * use a fifo instead of a queue. This would allow us to hold more than 128K messages - survive longer broker outages, and at a higher message rate. there should still probably be a limit set on the size of the fifo * define maximum number of retries allowed for a notification * define maximum time the notification may stay in the queue before it is removed We should probably start with these definitions done as topic attributes, reflecting our delivery guarantees for this specific destination. Will try to capture the results of the discussion in this tracker: https://tracker.ceph.com/issues/61532 Thanks, Yuval

2 3

stray daemons not managed by cephadm
by farhad kh 12 Jun '23

12 Jun '23

i deployed the ceph cluster with 8 node (v17.2.6) and after add all of hosts, ceph create 5 mon daemon instances i try decrease that to 3 instance with ` ceph orch apply mon --placement=label:mon,count:3 it worked, but after that i get error "2 stray daemons not managed by cephadm" . But every time I tried to deploy and delete other instances, this number increased Now I have 7 daemon that are not managed by cephadm How to deal with this issue? ------------------------ [root@opcsdfpsbpp0201 ~]# ceph -s cluster: id: 79a2627c-0821-11ee-a494-00505695c58c health: HEALTH_WARN 16 stray daemon(s) not managed by cephadm services: mon: 3 daemons, quorum opcsdfpsbpp0201,opcsdfpsbpp0205,opcsdfpsbpp0203 (age 2m) mgr: opcsdfpsbpp0201.vttwxa(active, since 27h), standbys: opcsdfpsbpp0207.kzxepm mds: 1/1 daemons up, 2 standby osd: 74 osds: 74 up (since 26h), 74 in (since 26h) data: volumes: 1/1 healthy pools: 6 pools, 6 pgs objects: 2.10k objects, 8.1 GiB usage: 28 GiB used, 148 TiB / 148 TiB avail pgs: 6 active+clean io: client: 426 B/s rd, 0 op/s rd, 0 op/s wr [root@opcsdfpsbpp0201 ~]# ceph health detail HEALTH_WARN 16 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 16 stray daemon(s) not managed by cephadm stray daemon mon.opcsdfpsbpp0207 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0211 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0207 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0211 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0207 not managed by cephadm stray daemon mon.opcsdfpsbpp0207 on host opcsdfpsbpp0209 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0209 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0211 not managed by cephadm stray daemon mon.opcsdfpsbpp0215 on host opcsdfpsbpp0211 not managed by cephadm stray daemon mon.opcsdfpsbpp0211 on host opcsdfpsbpp0213 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0215 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0215 not managed by cephadm

3 2

Keepalived configuration with cephadm
by Luis Domingues 12 Jun '23

12 Jun '23

Hi all, We are running 1 test cluster ceph with cephadm. Currently last pacific (16.2.13). We use cephadm to deploy keepalived:2.1.5 and HAProxy:2.3. We have 3 VIPs, 1 for each instance of HAProxy. But, we do not use the same network for managing the cluster and for the public traffic. We have a management network to connect to the machines, and for cephadm to do the deployments, and a prod network where the connections to HAproxy will be done. Our spec file looks like: --- service_type: ingress service_id: rgw.rgw placement: label: rgws spec: backend_service: rgw.rgw virtual_ips_list: - 10.X.X.10/24 - 10.X.X.2/24 - 10.X.X.3/24 frontend_port: 443 monitor_port: 1967 Our issue is that cephadm will populate `unicast_src_ip` and `unicast_peer` using the IPs from mgmt network and not the ones from prod network. A quick look into the code and it seems to be design that way. Our issue is that doing so, Keepalived instances will not talk to each other because the VRRP traffic is only allowed on our prod network. I quicky tested removing `unicast_src_ip` and `unicast_peer` and keepalived instances where able to talk to each other. My question, did I missed something on the configuration? Or should we add some kind of option to generate keepalived's config without `unicast_src_ip` and `unicast_peer`? Thanks, Luis Domingues Proton AG

1 0