- ceph-users - lists.ceph.io

Number of pgs
by Nikolaos Dandoulakis 05 Mar '24

05 Mar '24

Hi all, Pretty sure not the first time you see a thread like this. Our cluster consists of 12 nodes/153 OSDs/1.2 PiB used, 708 TiB /1.9 PiB avail The data pool is 2048 pgs big exactly the same number as when the cluster started. We have no issues with the cluster, everything runs as expected and very efficiently. We support about 1000 clients. The question is should we increase the number of pgs? If you think so, what is the sensible number to go to? 4096? More? I will eagerly await for your response. Best, Nick P.S. Yes, autoscaler is off :) The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

3 5

Number of pgs
by ndandoul＠ed.ac.uk 05 Mar '24

05 Mar '24

Hi all, Pretty sure not the very first time you see a thread like this. Our cluster consists of 12 nodes/153 OSDs/1.2 PiB used, 708 TiB /1.9 PiB avail The data pool is 2048 pgs big exactly the number when the cluster started. We have no issues with the cluster, everything runs as expected and very efficiently. We support about 1000 clients. The question is should we increase the number of pgs? If you think so, what is the sensible number to go to? 4096? More? I will eagerly await for your response. P.S. Yes, autoscaler is off :)

1 0

Upgraded 16.2.14 to 16.2.15
by Zakhar Kirpichenko 05 Mar '24

05 Mar '24

Hi, I have upgraded my test and production cephadm-managed clusters from 16.2.14 to 16.2.15. The upgrade was smooth and completed without issues. There were a few things which I noticed after each upgrade: 1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back. 2. Monitor debug_rocksdb option got silently reset back to the default 4/5, I had to set it back to 1/5. 3. For roughly 2 hours after the upgrade, despite the clusters being healthy and operating normally, all monitors would run manual compactions very often and write to disks at very high rates. For example, production monitors had their rocksdb:low0 thread write to store.db: monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour; monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour. After roughly 2 hours with no changes to the cluster the write rates dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason for frequent manual compactions and high write rates wasn't immediately apparent. 4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually after the upgrade to 16.2.14 which had broken it as well. 5. Mgr RAM usage appears to be increasing at a slower rate than it did with 16.2.14, although it's too early to tell whether the issue with mgrs randomly consuming all RAM and getting OOM-killed has been fixed - with 16.2.14 this would normally take several days. Overall, things look good. Thanks to the Ceph team for this release! Zakhar

4 9

PGs with status active+clean+laggy
by mori.ricardo＠gmail.com 05 Mar '24

05 Mar '24

Dear community, I have a ceph quincy cluster with 5 nodes currently. But only 3 with SSDs and others with nvme. On separate pools. I have had many alerts from PGs with active-clean-laggy status. This has caused problems with slow writing. I wanted to know how to troubleshoot properly. I checked several things related to the network, I have 10 GB cards on all nodes and everything seems to be correct. Many thanks

1 0

Uninstall ceph rgw
by Albert Shih 05 Mar '24

05 Mar '24

Hi everyone, I'm currently trying to understand how to deploy rgw, so I test few things but now I'm not sure what's are installed what not. First I try to install according to https://docs.ceph.com/en/quincy/cephadm/services/rgw/ then I see in that page they are https://docs.ceph.com/en/quincy/mgr/rgw/#mgr-rgw-module so now I got some rgw daemon running. But I like to clean up and «erase» everything about rgw ? not only to try to understand but also because I think I mixted up between realm and zonegroup... Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 05 mars 2024 11:01:30 CET

2 2

PGs with status active+clean+laggy
by ricardomori＠soujmv.com 05 Mar '24

05 Mar '24

Dear community, I have a ceph quincy cluster with 5 nodes currently. But only 3 with SSDs. I have had many alerts from PGs with active-clean-laggy status. This has caused problems with slow writing. I wanted to know how to troubleshoot properly. I checked several things related to the network, I have 10 GB cards on all nodes and everything seems to be correct. Many thanks

3 2

[RGW] Restrict a subuser to access only one specific bucket
by Huy Nguyen 05 Mar '24

05 Mar '24

Hi community, I have a user that owns some buckets. I want to create a subuser that has permission to access only one bucket. What can I do to archive this? Thanks

2 1

Slow RGW multisite sync due to "304 Not Modified" responses on primary zone
by Alam Mohammad 05 Mar '24

05 Mar '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our grafana instance and out of 1000 requests / second, 200 are "200 OK" and 800 are "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks, Saif

2 2

OSDs not balanced
by Ml Ml 04 Mar '24

04 Mar '24

Hello, i wonder why my autobalancer is not working here: root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_ERR 1 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) 4 pool(s) full => osd.17 was too full (92% or something like that) root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME -25 209.50084 - 213 TiB ... 69.56 ... - datacenter xxx-dc-root -19 84.59369 - 86 TiB ... 56.97 ... - rack RZ1.Reihe4.R10 -3 35.49313 - 37 TiB ... 57.88 ... - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 -9 22.88817 - 23 TiB ... 56.96 ... - host ceph06 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 -17 26.21239 - 26 TiB ... 55.67 ... - host ceph08 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 -20 59.11053 - 60 TiB ... 82.47 ... - rack RZ1.Reihe4.R9 -4 23.09996 - 24 TiB ... 79.92 ... - host ceph03 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 -13 36.01057 - 36 TiB ... 84.12 ... - host ceph05 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 -21 65.79662 - 66 TiB ... 74.29 ... - rack RZ3.Reihe3.R10 -2 28.49664 - 29 TiB ... 74.79 ... - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 -11 12.39999 - 12 TiB ... 73.40 ... - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 -15 24.89998 - 25 TiB ... 74.15 ... - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 -1 0 - 0 B ... 0 ... - root default TOTAL 213 TiB ... 69.56 root@ceph01:~# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.256761", "last_optimize_started": "Mon Mar 4 10:25:10 2024", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } Where as reweight-by-utilization would change quite a bit!: root@ceph01:~# ceph osd test-reweight-by-utilization 110 .5 10 moved 260 / 6627 (3.92334%) avg 127.442 stddev 81.345 -> 78.169 (expected baseline 11.18) min osd.17 with 24 -> 16 pgs (0.188321 -> 0.125547 * mean) max osd.42 with 401 -> 320 pgs (3.14652 -> 2.51094 * mean) oload 110 max_change 0.5 max_change_osds 10 average_utilization 0.6956 overload_utilization 0.7652 osd.42 weight 1.0000 -> 0.7561 osd.6 weight 0.4500 -> 0.6616 osd.17 weight 0.8500 -> 0.6466 osd.20 weight 0.8500 -> 0.6688 osd.5 weight 0.7501 -> 0.6004 osd.11 weight 0.9000 -> 0.7326 osd.43 weight 1.0000 -> 0.8199 osd.27 weight 0.9000 -> 0.7430 osd.63 weight 0.9000 -> 0.7440 osd.10 weight 0.8500 -> 0.7104 no change root@ceph01:~# ceph versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 52 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 61 } } Cheers, Michael

5 4

v16.2.15 Pacific released
by Yuri Weinstein 04 Mar '24

04 Mar '24

We're happy to announce the 15th, and expected to be the last, backport release in the Pacific series. https://ceph.io/en/news/blog/2024/v16-2-15-pacific-released/ Notable Changes --------------- * `ceph config dump --format <json|xml>` output will display the localized option names instead of their normalized version. For example, "mgr/prometheus/x/server_port" will be displayed instead of "mgr/prometheus/server_port". This matches the output of the non pretty-print formatted version of the command. * CephFS: MDS evicts clients who are not advancing their request tids, which causes a large buildup of session metadata, resulting in the MDS going read-only due to the RADOS operation exceeding the size threshold. The `mds_session_metadata_threshold` config controls the maximum size that an (encoded) session metadata can grow. * RADOS: The `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to its susceptibility to false negative results. Its safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.15.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 618f440892089921c3e944a991122ddc44e60516

2 1