- ceph-users - lists.ceph.io

How to create buckets in secondary zonegroup?
by Adam Prycki 29 Jul '25

29 Jul '25

Hello, I have a question about ceph multisite theory. How to create buckets in non-master zonegroup? I'm trying to create crate a test enviroment with 2 regions. I've crated deployment which looks like this REALM region-1 (master zonegroup) region-1-zone-a (master zone) region-1-zone-b region-2 region-2-zone-a (master zone) The goal would be to have 2 regions, each with separate data and 2 active-active zones in each. Just after creating region-2-zone-a I've decided to create a bucket there in region-2 and upload something. But I cannot create a bucket in new region. Rgw allows you to create buckets only on master zone. If I try to use s3cmd to create bucket on region-1-zone-b endpoint I get error 503. (as expected) Trying to list region-1 bucket on region-2 zone gives me 301 (moved permanently) so region-2 has metadata and working authentication. But I cannot figure out how to make a bucket in region-2. s3cmd .s3cfg-region-2 mb s3://bucket gives me error 503 like if I were trying to create a bucket on non-master zone. Creating bucket on region-1 endpoint with s3cmd flags specifying LocationConstraint region-2 creates bucket in region-1 I see in region-1-zone-a logs that rgw sees region-2 in LocationConstraint argument but creates bucket in region-1 I'm working on ceph 19.2.2 Best regards Adam Prycki

2 5

Warning about ceph discard on kioxia CD6 KCD61LUL7T68 NVMes
by Adam Prycki 29 Jul '25

29 Jul '25

Hi, I'm just throwing it here as a warning to other users. Kioxia KCD61LUL7T68 NVMe device have terrible discard performance. (All devices report firmware 8001) I have multiple of these kioxia SSDs in a ceph cluster. I normally configured bdev_async_discard_threads and bdev_enable_discard on all SSD. I have a script which makes new rbd snapshots and clears old ones. It has plenty of sleep commands to make whole process easier on the cluster, but these SSDs just cannot handle it. Example iostat: avg-cpu: %user %nice %system %iowait %steal %idle 45.6% 0.0% 27.0% 10.4% 0.0% 17.0% r/s rkB/s rrqm/s %rrqm r_await rareq-sz Device 1482.00 55.9M 208.00 12.3% 0.22 38.6k nvme1n1 13.00 220.0k 0.00 0.0% 0.15 16.9k nvme2n1 w/s wkB/s wrqm/s %wrqm w_await wareq-sz Device 7333.00 92.6M 15303.00 67.6% 0.15 12.9k nvme1n1 1470.00 13.2M 294.00 16.7% 0.10 9.2k nvme2n1 d/s dkB/s drqm/s %drqm d_await dareq-sz Device 807.00 3.8M 0.00 0.0% 2.27 4.8k nvme1n1 924.00 7.5M 0.00 0.0% 2.08 8.4k nvme2n1 NVMes start to choke around 700-800 discard IOPS. These discard ops sizes are probably terrible for the SSDs but afaik there is no way to fix this without just disabling discard on bluestore. In this case SSDs were about half full. I've used multiple brands of SSDs on ceph with discard enabled and I've never seen results this bad. Best regards Adam Prycki

1 0

squid 19.2.2 - RGW performance tuning
by Steven Vacaroaia 29 Jul '25

29 Jul '25

Hi, Are there any performance tuning settings for RGW that you can share please ? With the hardware I have, I believe I should be able to "squeeze" more 7 Supermicro dual CPU, 128 cores ( Intel Gold 6530), 1 TB RAM, 2 x 100GB bonded cluster network, 2 x 25 GB bonded private network cephadm, 5 RGW daemons (one per server) with haproxy in front of them Uploading a 5.5 GB file using s3cmd takes 2m21 ( the client has a 10GB connection) hardware/other tweaks HDD +DB/WAL on NVME pool EC 4+2 with meta on NVME vm.swapineess = 0 CPU intel_idle.max_cstate=0 processor.max_cstate=0 The servers have a less than 2 % CPU utilization and 30% memory usage Cluster is not under heavy usage - I only have one client using the bucket which has 7001 shards I have been trying these ones with very little success (taken from here https://links.imagerelay.com/cdn/3404/ql/3cd809aeba7c42f395ca8a7256ba488a/B… ) rgw thread pool size = 512 #default 100, increase for heavier load rgw max chunk size = 4194304# default is 128k, increase to 4MB helps with larger objects rgw_obj_stripe_size = 4194304 # (default 4M for luminous) rgw_list_bucket_min_readahead = 4000 #(default 1000) rgw_max_listing_results = 4000 rgw_cache_expiry_interval = 1800 #(default 900s) rgw_enable_usage_log = false rgw_enable_ops_log = false rgw dynamic resharding = false rgw override bucket index max shards = 50 # rgw bucket index max aio = 16 # default 8 rgw cache lru size = 50000 #number of entries in the cache rgw_gc_obj_min_wait = 21600 #in seconds(default 2_hr) rgw gc processor period = 7200 #(default 1hr) rgw objexp gc interval = 3600 # default 10_min, objecter inflight op bytes = 1073741824 # default 100_M, this is 1GB objecter inflight ops = 24576 Many thanks Steven

1 0

Re: SMB service configuration
by Robert Sander 29 Jul '25

29 Jul '25

Hi, Am 7/29/25 um 09:18 schrieb Ryan Sleeth: > Have you seen this? https://ceph.io/en/news/blog/2025/smb-manager- > module/ <https://ceph.io/en/news/blog/2025/smb-manager-module/> When I try to enable the smb mgr module on a 19.2.3 cluster, it tells me that it is not available: # ceph mgr module enable smb Error ENOENT: all mgr daemons do not support module 'smb', pass --force to force enablement Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

3 4

Getting SSL Certificate Verify failed Error while installation
by gagan tiwari 29 Jul '25

29 Jul '25

Hi Guys, I am trying to install via cephadm but getting the following error. There is no firewall running on the server. Please advise [root@ceph-mon1 ~]# curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm [root@ceph-mon1 ~]# [root@ceph-mon1 ~]# chmod +x cephadm [root@ceph-mon1 ~]# [root@ceph-mon1 ~]# getenforce ^C [root@ceph-mon1 ~]# ./cephadm install Installing packages ['cephadm']... [root@ceph-mon1 ~]# [root@ceph-mon1 ~]# ./cephadm add-repo --release squid Traceback (most recent call last): File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib64/python3.9/http/client.py", line 1285, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/lib64/python3.9/http/client.py", line 980, in send self.connect() File "/usr/lib64/python3.9/http/client.py", line 1454, in connect self.sock = self._context.wrap_socket(self.sock, File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/usr/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/usr/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/root/./cephadm/__main__.py", line 5581, in <module> File "/root/./cephadm/__main__.py", line 5569, in main File "/root/./cephadm/__main__.py", line 4576, in command_add_repo File "/root/./cephadm/cephadmlib/packagers.py", line 412, in validate File "/usr/lib64/python3.9/urllib/request.py", line 214, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.9/urllib/request.py", line 517, in open response = self._open(req, data) File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain result = func(*args) File "/usr/lib64/python3.9/urllib/request.py", line 1389, in https_open return self.do_open(http.client.HTTPSConnection, req, File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)>

2 2

Odd RBD stats metrics
by Christopher James 29 Jul '25

29 Jul '25

Hi, We have encountered some issues regarding the RBD stats metrics. We have some Grafana panels that show the per-pool rate of changes in RBD IOPS and Throughput. We use these queries for IOPS round(sum(irate(ceph_rbd_write_ops[1m])) by (pool)) round(sum(irate(ceph_rbd_read_ops[1m])) by (pool)) and these queries for throughput round(sum(irate(ceph_rbd_write_bytes[1m])) by (pool)) round(sum(irate(ceph_rbd_read_bytes[1m])) by (pool)) Now, the problem we encounter here is that at some points in time, there are some odd outputs. In a way that the output has a lot of spikes, and at many points in time, there seems to be no change in the mentioned values, hence the irate()'s output would become zero, and the next data point would show that the data has somehow doubled from the last time. I investigated the mgr/prometheus module to see if it is related to the cache mechanism, and it doesn't seem to be about that because the metrics collection method is done in less than 10 seconds most of the time and there is no "collecting data took more than ..." log. I also investigated the raw time series data and saw that the two datapoints are exactly the same at the times that the irate() returns zero. I have put some screenshots of the panels in this read-only Google Doc: https://docs.google.com/document/d/1Rf0dl4qAWnOtG80BsxVY9cjljQqliG2JHxgzVxD… It is somewhat odd, to be honest, that every image in every pool does not change in IOPS or throughput. Is this the natural way the metrics are exposed? Should I not use irate()? Has anybody else encountered this issue as well? Cheers, Chris

2 2

Re: Odd RBD stats metrics
by Christopher James 29 Jul '25

29 Jul '25

The scrape_interval is set to 15 seconds on prom. The mgr configuration is also set to 15 seconds. I'm using irate() to try to observe sudden increases in the mentioned metrics. Most RBD pools are on SSD OSDs, and the HDD OSDs use NVMe drives for DB. On Mon, Jul 28, 2025 at 4:51 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > What is your scrape period? The templated default 10s? > > My thoughts: > > * Why are you using irate? Try rate() instead, and try a longer period > than one minute, especially if that isn’t more than a few scrape intervals > * Is your DB on SSDs? > > > On Jul 28, 2025, at 7:39 AM, Christopher James < > christopher.jamesjr2000(a)gmail.com> wrote: > > Is this the natural way the metrics are exposed? Should I not use irate()? > > >

1 0

SMB service configuration
by Robert Sander 29 Jul '25

29 Jul '25

Hi, where do I find more information about the config objects needed to deploy an smb service? https://docs.ceph.com/en/latest/cephadm/services/smb/ mentions "config_uri" but does not tell me what to put there. It also mentions "cluster_meta_uri" for a clustered setup but does not tell me what to put there. I am familiar with setting up a Samba CTDB cluster manually but wanted to try the new smb service of the orchestrator. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

CephFS limp mode when fullest OSD is between nearfull & backfillfull value
by Jeremi-Ernst Avenant 28 Jul '25

28 Jul '25

Good day We've been struggling with this issue since we've upgraded post 16.2.11 to 16.2.15 and now up to Reef 18.2.7. We didn't have this issue prior to upgrading from 16.2.11 to 16.2.15. The moment the fullest OSD % is between the nearful and backfillful value, CephFS goes into a limp mode. Meaning our Client IO goes from ~30 *GiB*/s to 100 *MiB*/s. Doesn't matter what the threshold values are set. If my nearful% is 75% and backfillfull 80%, with the highest OSD 75.01% it will limp. If my nearful% is 85% and backfillfull 90%, with the highest OSD 85.01% it will go into limp mode. The only way to get CephFS operational again, is to set both ratios to either far below or above the highest OSD value. e.g. if the fullest is 72% , I need to set it to ceph osd set-nearfull-ratio 0.76 & ceph osd set-backfillfull-ratio 0.81 When I deploy a brand new cluster from scratch on our testbed I get the same issue, Cephadm / Reef 18.2.4. Back in the days I was able to manipulate these settings with the following as well, but I think since Pacific they are no longer tuneable: ceph tell osd.1231 config show | grep -E 'osd_nearfull_ratio|osd_backfillfull_ratio|osd_full_ratio' "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", "mon_osd_nearfull_ratio": "0.850000", ceph tell osd.$osd injectargs '--osd_nearfull_ratio=0.85' ceph tell osd.$osd injectargs '--osd_backfillfull_ratio=0.90' ceph tell osd.$osd injectargs '--osd_full_ratio=0.95' URL to the issue: https://tracker.ceph.com/issues/70129 Any ideas would be greatly appreciated. -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

2 2

Squid: successfully drained host can't be removed
by Eugen Block 26 Jul '25

26 Jul '25

Hi *, an unexpected issue occurred today, at least twice, so it seems kind of reproducable. I've been preparing a demo in a (virtual) lab cluster (19.2.2) and wanted to drain multiple hosts. The first time I didn't pay much attention, but the draining seemed stuck (kind of a common issue these days), so I intervened and cleaned up until I got into a healthy state, all good. Then I did my thing, changed the crush tree, added the removed hosts again, cephadm created the OSDs, backfill finished successfully. Now I wanted to reset the cluster again to my starting point, so I issued the drain command again for multiple hosts (each host has 2 OSDs): # for i in {5..8}; do ceph orch host drain host$i; done This time all OSDs were drained successfully (I watched 'ceph orch osd rm status'), so I wanted to remove the hosts, but it failed: # for i in {5..8}; do ceph orch host rm host$i --rm-crush-entry; done Removed host 'host5' Removed host 'host6' Removed host 'host7' Error EINVAL: Not allowed to remove host8 from cluster. The following daemons are running in the host: type id -------------------- --------------- osd 6 Please run 'ceph orch host drain host8' to remove daemons from host But there was nothing to drain anymore, osd.6 was already successfully removed from the crush tree. But on host8 there was still a daemon I had to clean up manually: host8:~ # cephadm rm-daemon --name osd.6 --fsid 543967bc-e586-32b8-bd2c-2d8b8b168f02 --force I compared the cephadm.log files (3 out of 4 to-be-drained hosts were successfully drained) and on host8 the command rm-daemon was never executed (until I did manually). Is this a known issue? It doesn't seem to happen with only on host, at least I didn't notice in the past. Should I create a tracker for this? Thanks, Eugen

2 3