September 2024 - ceph-users

PGs not deep-scrubbed in time
by Eugen Block 07 Sep '24

07 Sep '24

Hi, I finally managed to take the time to get to the bottom of this infamous health warning. I decided to write it up in a blog post [0] and also contacted Zac to improve the documentation. The short version is: If you want to change the config setting for osd_deep_scrub_interval in general, change it either globally or for both MGR *and* OSD services. Or configure individual intervals per pool. If you don't change it for the MGR, you'll still get the warning because the MGR will still compare the last deep-scrub timestamp with the default deep-scrub interval (1 week). Comments are welcome, of course! Regards, Eugen [0] https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-ti…

1 0

Setting up Ceph RGW with SSE-S3 - Any examples?
by Michael Worsham 06 Sep '24

06 Sep '24

Has anyone been successful at standing up a Ceph RGW S3 bucket with Hashicorp Vault for S3 bucket encryption? The documentation for doing it with Ceph jumps all over the page between token and agent, so it's nearly impossible to know which variables and parameters are required for each. I was able to successfully setup a Hashicorp Vault in a high-availability cluster via an Ansible playbook I wrote. It's the configuration of what needs to be configured for the transit engine for vault and then what changes I need to do to Ceph that has me stumped. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 1

Discovery (port 8765) service not starting
by Matthew Vernon 06 Sep '24

06 Sep '24

Hi, I'm running reef, with locally-built containers based on upstream .debs. I've now enabled prometheus metrics thus: ceph mgr module enable prometheus And that seems to have worked (the active mgr is listening on port 9283); but per the docs[0] there should also be a service discovery endpoint (on port 8765) that I can point Prometheus' http_sd_config at. But there is in fact no listener on that port, nor anything obvious in logs if I failover the mgr - see a log extract here: https://phabricator.wikimedia.org/P68521 Is there something else I need to be doing to get the service discovery endpoint working? Thanks, Matthew [0] https://docs.ceph.com/en/reef/cephadm/services/monitoring/#deploying-monito…

4 21

v19.1.1 Squid RC1 released
by Yuri Weinstein 06 Sep '24

06 Sep '24

This is the second release candidate for Squid. Feature highlights: RGW Fixed a regression in bucket ownership for Keystone users and implicit tenants. The User Accounts feature unlocks several new AWS-compatible IAM APIs for the self-service management of users, keys, groups, roles, policy and more. RADOS BlueStore has been optimized for better performance in snapshot-intensive workloads. BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. Other improvements include more flexible EC configurations, an OpTracker to help debug mgr module issues, and better scrub scheduling. Dashboard * Rearranged Navigation Layout: The navigation layout has been reorganized for improved usability and easier access to key features. CephFS Improvements * Support for managing CephFS snapshots and clones, as well as snapshot schedule management * Manage authorization capabilities for CephFS resources * Helpers on mounting a CephFS volume RGW Improvements * Support for managing bucket policies * Add/Remove bucket tags * ACL Management * Several UI/UX Improvements to the bucket form Monitoring: Grafana dashboards are now loaded into the container at runtime rather than building a grafana image with the grafana dashboards. Official Ceph grafana images can be found in quay.io/ceph/grafana * Monitoring: RGW S3 Analytics: A new Grafana dashboard is now available, enabling you to visualize per bucket and user analytics data, including total GETs, PUTs, Deletes, Copies, and list metrics. Crimson/Seastore Crimson's first tech preview release! Supporting RBD workloads on Replicated pools. For more information please visit: https://ceph.io/en/news/crimson If any of our community members would like to help us with performance investigations or regression testing of the Squid release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/squid_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.1.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 1d9f35852eef16b81614e38a05cf88b505cc142b

2 2

Prefered distro for Ceph
by Denis Polom 06 Sep '24

06 Sep '24

Hi guys, what distro would you prefer and why for the production Ceph? We use Ubuntu on most of our Ceph clusters and some are Debian. Now we are thinking about unifying it by using only Debian or Ubuntu. I personally prefer Debian mainly for its stability and easy upgrade-in-place. What are yours preferences? Thank you

9 10

Stretch cluster data unavailable
by przemek.kuczynski＠gmail.com 06 Sep '24

06 Sep '24

Ceph reef 18.2.4 We have pool with size 3 (2 copies in first dc , 1 copy in second ) replicated between datacenter. When we put host in maintanance in different datacenter some data is unavailable - why ? How to prevent it or fix ? 2 nodes in each dc + witness pool 13 'VolumesStandardW2' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 6257 lfor 0/2232/2230 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.30 Policy take W2 chooseleaf firstn 2 type host emit take W1 chooseleaf firstn -1 type host emit HEALTH_WARN 1 host is in maintenance mode; 1/5 mons down, quorum xxx xxx xxx xxx xxx; 3 osds down; 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; 1 host (3 osds) down; Reduced data availability: 137 pgs inactive; Degraded data redundancy: 203797/779132 objects degraded (26.157%), 522 pgs degraded, 554 pgs undersized [WRN] HOST_IN_MAINTENANCE: 1 host is in maintenance mode [WRN] PG_AVAILABILITY: Reduced data availability: 137 pgs inactive pg 12.5c is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,9] pg 12.5d is stuck undersized for 2m, current state active+undersized+degraded, last acting [0,6] pg 12.5e is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,11] pg 12.5f is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,9] pg 13.0 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7,11] pg 13.1 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8,9] pg 13.2 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11,6] pg 13.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [9,6] ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000198", "last_optimize_started": "Wed Sep 4 13:03:53 2024", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Some objects (0.261574) are degraded; try again later", "plans": []

2 1

[S3] Scale/tuning strategy for RGW in high load
by Huy Nguyen 06 Sep '24

06 Sep '24

Hi, I'm using Ceph v18.2.4, non-cephadm. Over time, my RGW cluster receives more requests, and when it reaches a certain threshold, I will manually scale more Radosgw instances to handle the increased traffic. - Is this a normal practice? - Are there any tunings I can do with RGW to make it handle more requests per instance? - Also, I'd like to know if there is any way to auto-scale those RGW instances (in a non-cephadm cluster)? Thanks

1 0

[RGW] Radosgw instances hang for a long time while doing realm update
by Huy Nguyen 05 Sep '24

05 Sep '24

Hi community, I'm using ceph v18.2.4. Each time I commit a period, all of my radosgw instances paused for about 30s which is kinda long, and make my object service down during that time. how to avoid the down time? Is there better/safer practice? Thanks Here is radosgw logs each time I did a period commit. You can see it was paused for a while since the last request 2024-08-23T22:43:32.152+0700 7f4594d2f700 1 ====== starting new request req=0x7f42dcfbf730 ===== 2024-08-23T22:43:32.152+0700 7f438b11c700 1 ====== req done req=0x7f42dcfbf730 op status=0 http_status=200 latency=0.000000000s ====== 2024-08-23T22:43:32.152+0700 7f438b11c700 1 beast: 0x7f42dcfbf730: 172.24.9.101 - anonymous [23/Aug/2024:22:43:32.152 +0700] "GET / HTTP/1.0" 200 231 - "check_http/v2.2 (monitoring-plugins 2.2)" - latency=0.000000000s 2024-08-23T22:44:02.778+0700 7f42e1fca700 1 rgw realm reloader: Pausing frontends for realm update... 2024-08-23T22:44:02.778+0700 7f42e1fca700 1 rgw realm reloader: Frontends paused 2024-08-23T22:44:03.358+0700 7f42e1fca700 1 rgw realm reloader: driver closed 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Creating new driver 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 mgrc service_daemon_register rgw.5055090799 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4.... 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Finishing initialization of new driver 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: - REST subsystem init 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: - usage subsystem init 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Resuming frontends with new realm configuration. 2024-08-23T22:44:06.135+0700 7f4637674700 1 ====== starting new request req=0x7f42dcfbf730 ===== 2024-08-23T22:44:06.136+0700 7f4637674700 -1 res_query() failed 2024-08-23T22:44:06.136+0700 7f4637674700 1 ====== req done req=0x7f42dcfbf730 op status=0 http_status=405 latency=0.000999984s ====== 2024-08-23T22:44:06.136+0700 7f4637674700 1 beast: 0x7f42dcfbf730: 172.24.17.21 - - [23/Aug/2024:22:44:06.135 +0700] "GET /admin HTTP/1.1" 405 133 - "curl/7.61.1" - latency=0.000999984s

2 1

Re: The snaptrim queue of PGs has not decreased for several days.
by Eugen Block 05 Sep '24

05 Sep '24

Hi (please don't drop the ML from your responses), > All PGs of pool cephfs are affected and they are in all OSDs then just pick a random one and check if anything stands out. I'm not sure if you mentioned it already, did you also try restarting OSDs? > Oh, not yesterday. I do it now, then I compat all osds with nostrim. > Do I add OSDs? Let's wait for the other results first (compaction, reducing defaults, OSD restart). If that doesn't change anything, I would probably try to add three more OSDs. I assume you have three hosts? Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: > Hello Eugen, > > Am 20.08.2024 um 09:44 schrieb Eugen Block: >> You could also look into the historic_ops of the primary OSD for >> one affected PG: > > All PGs of pool cephfs are affected and they are in all OSDs :-( > >> Did you reduce the default values I mentioned? > > Oh, not yesterday. I do it now, then I compat all osds with nostrim. > > Do I add OSDs? > > Regars, > > Gio > > >> >> ceph tell osd.<OSD_ID> dump_historic_ops_by_duration >> >> But I'm not sure if that can actually help here. There are plenty >> of places to look at, you could turn on debug logs on one primary >> OSD and inspect the output. >> >> I just get the feeling that this is one of the corner cases with >> too few OSDs, although the cluster load seems to be low. >> >> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >> >>> Hello Eugen, >>> >>> yesterday after stop and go of snaptrim the queue decrease a >>> little and then remain blocked. >>> They didn't grow and didn't decrease. >>> >>> Is that good or bad? >>> >>> >>> Am 19.08.2024 um 15:43 schrieb Eugen Block: >>>> There's a lengthy thread [0] where several approaches are >>>> proposed. The worst is a OSD recreation, but that's the last >>>> resort, of course. >>>> >>>> What's are the current values for these configs? >>>> >>>> ceph config get osd osd_pg_max_concurrent_snap_trims >>>> ceph config get osd osd_max_trimming_pgs >>>> >>>> Maybe decrease them to 1 each while the nosnaptrim flag is set, >>>> then unset it. You could also try online (and/or offline osd >>>> compaction) before unsetting the flag. Are the OSD processes >>>> utilizing an entire CPU? >>>> >>>> [0] https://www.spinics.net/lists/ceph-users/msg75626.html >>>> >>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>> >>>>> Hallo Eugen, >>>>> >>>>> yes, the load is for now not too much. >>>>> >>>>> I stop the snap and now this is the output. No changes in the queue. >>>>> >>>>> root@kube-master02:~# k ceph -s >>>>> Info: running 'ceph' command with args: [-s] >>>>> cluster: >>>>> id: 3a35629a-6129-4daf-9db6-36e0eda637c7 >>>>> health: HEALTH_WARN >>>>> nosnaptrim flag(s) set >>>>> 32 pgs not deep-scrubbed in time >>>>> 32 pgs not scrubbed in time >>>>> >>>>> services: >>>>> mon: 3 daemons, quorum bx,bz,ca (age 30h) >>>>> mgr: a(active, since 29h), standbys: b >>>>> mds: 1/1 daemons up, 1 hot standby >>>>> osd: 6 osds: 6 up (since 21h), 6 in (since 6d) >>>>> flags nosnaptrim >>>>> >>>>> data: >>>>> volumes: 1/1 healthy >>>>> pools: 4 pools, 97 pgs >>>>> objects: 4.21M objects, 2.5 TiB >>>>> usage: 7.7 TiB used, 76 TiB / 84 TiB avail >>>>> pgs: 65 active+clean >>>>> 32 active+clean+snaptrim_wait >>>>> >>>>> io: >>>>> client: 7.4 MiB/s rd, 7.9 MiB/s wr, 11 op/s rd, 35 op/s wr >>>>> >>>>> Am 19.08.2024 um 14:54 schrieb Eugen Block: >>>>>> What happens when you disable snaptrimming entirely? >>>>>> >>>>>> ceph osd set nosnaptrim >>>>>> >>>>>> So the load on your cluster seems low, but are the OSDs heavily >>>>>> utilized? Have you checked iostat? >>>>>> >>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>> >>>>>>> Hello Eugen, >>>>>>> >>>>>>> *root@kube-master02:~# k ceph -s* >>>>>>> >>>>>>> Info: running 'ceph' command with args: [-s] >>>>>>> cluster: >>>>>>> id: 3a35629a-6129-4daf-9db6-36e0eda637c7 >>>>>>> health: HEALTH_WARN >>>>>>> 32 pgs not deep-scrubbed in time >>>>>>> 32 pgs not scrubbed in time >>>>>>> >>>>>>> services: >>>>>>> mon: 3 daemons, quorum bx,bz,ca (age 13h) >>>>>>> mgr: a(active, since 13h), standbys: b >>>>>>> mds: 1/1 daemons up, 1 hot standby >>>>>>> osd: 6 osds: 6 up (since 5h), 6 in (since 5d) >>>>>>> >>>>>>> data: >>>>>>> volumes: 1/1 healthy >>>>>>> pools: 4 pools, 97 pgs >>>>>>> objects: 4.20M objects, 2.5 TiB >>>>>>> usage: 7.7 TiB used, 76 TiB / 84 TiB avail >>>>>>> pgs: 65 active+clean >>>>>>> 20 active+clean+snaptrim_wait >>>>>>> 12 active+clean+snaptrim >>>>>>> >>>>>>> io: >>>>>>> client: 3.5 MiB/s rd, 3.6 MiB/s wr, 6 op/s rd, 12 op/s wr >>>>>>> >>>>>>> If I understand the documentation correctly, I will never have >>>>>>> a scrub unless the PGs (Placement Groups) are active and clean. >>>>>>> >>>>>>> All 32 PGs of the CephFS pool have been in this status for >>>>>>> several days: >>>>>>> >>>>>>> * 20 active+clean+snaptrim_wait >>>>>>> * 12 active+clean+snaptrim" >>>>>>> >>>>>>> Today, I restarted the MON, MGR, and MDS, but no changes in >>>>>>> the growing. >>>>>>> >>>>>>> Am 18.08.2024 um 18:39 schrieb Eugen Block: >>>>>>>> Can you share the current ceph status? Are the OSDs reporting >>>>>>>> anything suspicious? How is the disk utilization? >>>>>>>> >>>>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>>>> >>>>>>>>> More information: >>>>>>>>> >>>>>>>>> The snaptrim take a lot of time but the he objects_trimmed are "0" >>>>>>>>> >>>>>>>>> "objects_trimmed": 0, >>>>>>>>> "snaptrim_duration": 500.58076017500002, >>>>>>>>> >>>>>>>>> It could explain, why the queue are growing up.. >>>>>>>>> >>>>>>>>> >>>>>>>>> Am 17.08.2024 um 14:37 schrieb Giovanna Ratini: >>>>>>>>>> Hello again, >>>>>>>>>> >>>>>>>>>> I checked the pgs dump. Snapshot grow up >>>>>>>>>> >>>>>>>>>> Query für PG: 3.12 >>>>>>>>>> { >>>>>>>>>> "snap_trimq": >>>>>>>>>> "[5b974~3b,5cc3a~1,5cc3c~1,5cc3e~1,5cc40~1,5cd83~1,5cd85~1,5cd87~1,5cd89~1,5cecc~1,5cece~4,5ced3~2,5cf72~1,5cf74~4,5cf79~a2,5d0b8~1,5d0bb~1,5d0bd~a5,5d1f9~2,5d204~a5,5d349~a7,5d48e~3,5d493~a4,5d5d7~a7,5d71e~a3,5d7c2~3,5d860~1,5d865~4,5d86a~a2,5d9aa~1,5d9ac~1,5d9ae~a5,5daf3~a5,5db9a~2,5dc3a~a5,5dce1~1,5dce3~1,5dd81~a7,5dec8~a7,5e00f~a7,5e156~a8,5e29d~1,5e29f~a7,5e3e6~a8,5e52e~a6,5e5d6~2,5e676~a6,5e71e~2,5e7be~a9,5e907~a5,5e9ad~3,5ea50~a7,5eaf9~1,5eafb~1,5eb99~a7,5ec42~2,5ece2~a7,5ed8a~2,5ee2b~a9,5ef74~a7,5f01c~1,5f0bd~a1,5f15f~1,5f161~1,5f163~1,5f167~1,5f206~a1,5f2a8~1,5f2aa~1,5f2ac~1,5f2ae~1,5f34f~a1,5f3f1~1,5f3f3~1,5f3f5~1,5f3f7~1,5f499~a1,5f53b~1,5f53d~1,5f53f~1,5f541~1,5f5e3~a1,5f685~1,5f687~1,5f689~1,5f68d~1,5f72d~a1,5f7cf~1,5f7d1~1,5f7d3~1]", >>>>>>>>>> * "snap_trimq_len": 5421,* >>>>>>>>>> "state": "active+clean+snaptrim", >>>>>>>>>> "epoch": 734130, >>>>>>>>>> >>>>>>>>>> Query für PG: 3.12 >>>>>>>>>> { >>>>>>>>>> "snap_trimq": >>>>>>>>>> "[5b976~39,5ba53~1,5ba56~a0,5cc3a~1,5cc3c~1,5cc3e~1,5cc40~1,5cd83~1,5cd85~1,5cd87~1,5cd89~1,5cecc~1,5cece~4,5ced3~2,5cf72~1,5cf74~4,5cf79~a2,5d0b8~1,5d0bb~1,5d0bd~a5,5d1f9~2,5d204~a5,5d349~a7,5d48e~3,5d493~a4,5d5d7~a7,5d71e~a3,5d7c2~3,5d860~1,5d865~4,5d86a~a2,5d9aa~1,5d9ac~1,5d9ae~a5,5daf3~a5,5db9a~2,5dc3a~a5,5dce1~1,5dce3~1,5dd81~a7,5dec8~a7,5e00f~a7,5e156~a8,5e29d~1,5e29f~a7,5e3e6~a8,5e52e~a6,5e5d6~2,5e676~a6,5e71e~2,5e7be~a9,5e907~a5,5e9ad~3,5ea50~a7,5eaf9~1,5eafb~1,5eb99~a7,5ec42~2,5ece2~a7,5ed8a~2,5ee2b~a9,5ef74~a7,5f01c~1,5f0bd~a1,5f15f~1,5f161~1,5f163~1,5f167~1,5f206~a1,5f2a8~1,5f2aa~1,5f2ac~1,5f2ae~1,5f34f~a1,5f3f1~1,5f3f3~1,5f3f5~1,5f3f7~1,5f499~a1,5f53b~1,5f53d~1,5f53f~1,5f541~1,5f5e3~a1,5f685~1,5f687~1,5f689~1,5f68d~1,5f72d~a1,5f7cf~1,5f7d1~1,5f7d3~1,5f875~a1]", >>>>>>>>>> * "snap_trimq_len": 5741,* >>>>>>>>>> "state": "active+clean+snaptrim", >>>>>>>>>> "epoch": 734240, >>>>>>>>>> "up": [ >>>>>>>>>> >>>>>>>>>> Do you know the way to see if the snaptim "process" works? >>>>>>>>>> >>>>>>>>>> Best Regard >>>>>>>>>> >>>>>>>>>> Gio >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Am 17.08.2024 um 12:59 schrieb Giovanna Ratini: >>>>>>>>>>> Hello Eugen, >>>>>>>>>>> >>>>>>>>>>> thank you for your answer. >>>>>>>>>>> >>>>>>>>>>> I restarted all the kube-ceph nodes one after the other. >>>>>>>>>>> Nothing has changed. >>>>>>>>>>> >>>>>>>>>>> ok, I deactivate the snap ... : ceph fs snap-schedule deactivate / >>>>>>>>>>> >>>>>>>>>>> Is there a way to see how many snapshots will be deleted per hour? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Gio >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Am 17.08.2024 um 10:12 schrieb Eugen Block: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> have you tried to fail the mgr? Sometimes the PG stats >>>>>>>>>>>> are not correct. You could also temporarily disable >>>>>>>>>>>> snapshots to see if things settle down. >>>>>>>>>>>> >>>>>>>>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>>>>>>>> >>>>>>>>>>>>> Hello all, >>>>>>>>>>>>> >>>>>>>>>>>>> We use Ceph (v18.2.2) and Rook (1.14.3) as the CSI for a >>>>>>>>>>>>> Kubernetes environment. Last week, we had a problem with >>>>>>>>>>>>> the MDS falling behind on trimming every 4-5 days >>>>>>>>>>>>> (GitHub issue link). We resolved the issue using the >>>>>>>>>>>>> steps outlined in the GitHub issue. >>>>>>>>>>>>> >>>>>>>>>>>>> We have 3 hosts (I know, I need to increase this as soon >>>>>>>>>>>>> as possible, and I will!) and 6 OSDs. After running the >>>>>>>>>>>>> commands: >>>>>>>>>>>>> >>>>>>>>>>>>> ceph config set mds mds_dir_max_commit_size 80, >>>>>>>>>>>>> >>>>>>>>>>>>> ceph fs fail <fs_name>, and >>>>>>>>>>>>> >>>>>>>>>>>>> ceph fs set <fs_name> joinable true, >>>>>>>>>>>>> >>>>>>>>>>>>> After that, the snaptrim queue for our PGs has stopped >>>>>>>>>>>>> decreasing. All PGs of our CephFS are in either >>>>>>>>>>>>> active+clean+snaptrim_wait or active+clean+snaptrim >>>>>>>>>>>>> states. For example, the PG 3.12 is in the >>>>>>>>>>>>> active+clean+snaptrim state, and its snap_trimq_len was >>>>>>>>>>>>> 4077 yesterday but has increased to 4538 today. >>>>>>>>>>>>> >>>>>>>>>>>>> I increased the osd_snap_trim_priority to 10 (ceph >>>>>>>>>>>>> config set osd osd_snap_trim_priority 10), but it didn't >>>>>>>>>>>>> help. Only the PGs of our CephFS have this problem. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have any ideas on how we can resolve this issue? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>> Giovanna >>>>>>>>>>>>> p.s. I'm not a ceph expert :-). >>>>>>>>>>>>> Faulkener asked me for more information, so here it is: >>>>>>>>>>>>> MDS Memory: 11GB >>>>>>>>>>>>> mds_cache_memory_limit: 11,811,160,064 bytes >>>>>>>>>>>>> >>>>>>>>>>>>> root@kube-master02:~# ceph fs snap-schedule status / >>>>>>>>>>>>> { >>>>>>>>>>>>> "fs": "rook-cephfs", >>>>>>>>>>>>> "subvol": null, >>>>>>>>>>>>> "path": "/", >>>>>>>>>>>>> "rel_path": "/", >>>>>>>>>>>>> "schedule": "3h", >>>>>>>>>>>>> "retention": {"h": 24, "w": 4}, >>>>>>>>>>>>> "start": "2024-05-05T00:00:00", >>>>>>>>>>>>> "created": "2024-05-05T17:28:18", >>>>>>>>>>>>> "first": "2024-05-05T18:00:00", >>>>>>>>>>>>> "last": "2024-08-15T18:00:00", >>>>>>>>>>>>> "last_pruned": "2024-08-15T18:00:00", >>>>>>>>>>>>> "created_count": 817, >>>>>>>>>>>>> "pruned_count": 817, >>>>>>>>>>>>> "active": true >>>>>>>>>>>>> } >>>>>>>>>>>>> I do not understand if the snapshots in the PGs are >>>>>>>>>>>>> correlated with the snapshots on CephFS. Until we >>>>>>>>>>>>> encountered the issue with the "MDS falling behind on >>>>>>>>>>>>> trimming every 4-5 days," we didn't have any problems >>>>>>>>>>>>> with snapshots. >>>>>>>>>>>>> >>>>>>>>>>>>> Could someone explain me this or send me to the documentation? >>>>>>>>>>>>> Thank you >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>> >>>>>>> -- >>>>>>> Giovanna Ratini >>>>>>> Mail:ratini@dbvis.inf.uni-konstanz.de >>>>>>> Phone: +49 (0) 7531 88 - 4550 >>>>>>> >>>>>>> Technical Support >>>>>>> Data Analysis and Visualization Group >>>>>>> Department of Computer and Information Science >>>>>>> University of Konstanz (Box 78) >>>>>>> Universitätsstr. 10 >>>>>>> 78457 Konstanz, Germany >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> -- >>>>> Giovanna Ratini >>>>> Mail:ratini@dbvis.inf.uni-konstanz.de >>>>> Phone: +49 (0) 7531 88 - 4550 >>>>> >>>>> Technical Support >>>>> Data Analysis and Visualization Group >>>>> Department of Computer and Information Science >>>>> University of Konstanz (Box 78) >>>>> Universitätsstr. 10 >>>>> 78457 Konstanz, Germany >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> -- >>> Giovanna Ratini >>> Mail:ratini@dbvis.inf.uni-konstanz.de >>> Phone: +49 (0) 7531 88 - 4550 >>> >>> Technical Support >>> Data Analysis and Visualization Group >>> Department of Computer and Information Science >>> University of Konstanz (Box 78) >>> Universitätsstr. 10 >>> 78457 Konstanz, Germany >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Giovanna Ratini > Mail:ratini@dbvis.inf.uni-konstanz.de > Phone: +49 (0) 7531 88 - 4550 > > Technical Support > Data Analysis and Visualization Group > Department of Computer and Information Science > University of Konstanz (Box 78) > Universitätsstr. 10 > 78457 Konstanz, Germany

4 11

Issue Replacing OSD with cephadm: Partition Path Not Accepted
by Herbert Faleiros 04 Sep '24

04 Sep '24

I am on a journey, so far successful, to update our clusters to supported versions. I started with Luminous and Ubuntu 16.04, and now we are on Reef with Ubuntu 20.04. We still have more updates to do, but at the moment, I encountered an issue with an OSD, and it was necessary to replace a disk. Since the cluster was adopted, I'm not entirely sure what the best way to replace this OSD is, as with cephadm, it doesn't like when the path for the device is a partition. I could recreate the OSD using traditional methods and then adopt the OSD, but that doesn't seem like the best approach. Does anyone know how I should proceed to recreate this OSD? I had the same problem in my lab, where I am already on Quincy. What I am trying to do is: # ceph orch osd rm 6 --replace --zap # ceph orch daemon add osd osd-nodeXXX:data_devices=/dev/sdb1,db_devices=/dev/sda3 The error it gives is: /usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices

5 6

CephFS troubleshooting
by Eugenio Tampieri 04 Sep '24

04 Sep '24

Hello, I'm writing to troubleshoot an otherwise functional Ceph quincy cluster that has issues with cephfs. I cannot mount it with ceph-fuse (it gets stuck), and if I mount it with NFS I can list the directories but I cannot read or write anything. Here's the output of ceph -s cluster: id: 3b92e270-1dd6-11ee-a738-000c2937f0ec health: HEALTH_WARN mon ceph-storage-a is low on available space 1 daemons have recently crashed too many PGs per OSD (328 > max 250) services: mon: 5 daemons, quorum ceph-mon-a,ceph-storage-a,ceph-mon-b,ceph-storage-c,ceph-storage-d (age 105m) mgr: ceph-storage-a.ioenwq(active, since 106m), standbys: ceph-mon-a.tiosea mds: 1/1 daemons up, 2 standby osd: 4 osds: 4 up (since 104m), 4 in (since 24h) rbd-mirror: 2 daemons active (2 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 481 pgs objects: 231.83k objects, 648 GiB usage: 1.3 TiB used, 1.8 TiB / 3.1 TiB avail pgs: 481 active+clean io: client: 1.5 KiB/s rd, 8.6 KiB/s wr, 1 op/s rd, 0 op/s wr Best regards, Eugenio Tampieri

3 5

quincy radosgw-admin log list show entries with only the date
by Boris 04 Sep '24

04 Sep '24

I am not sure how I should describe this. We enable ops log and have these entries here together with the normal ops logs. # radosgw-admin log list | grep ^.*-- "2024-09-03-12--", I did a head on the file and got this: B��f�� anonymous list_bucketsHEAD / HTTP/1.0200 ;tx00000e66dccefcb9f8a39-0066d6fa42-171a... Anything I can do about this? I can not remove them via radosgw-admin (only the rados objects directly) and I can not read it via radosgw-admin command. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 2

SMB Service in Squid
by Robert W. Eckert 04 Sep '24

04 Sep '24

I have upgraded my home cluster to 19.1.0 and wanted to try out the SMB orchestration features to improve my hacked SMB shared using CTDB and SMB services on each host. My smb.yaml file looks like service_type: smb service_id: home placement: hosts: - HOST1 - HOST2 - HOST3 - HOST4 spec: cluster_id: home features: - domain #clustered: true config_uri: rados://.smb/home/scc.toml custom_dns: - "<DNS SERVERS>" join_sources: - "rados:mon-config-key:smb/config/home/join1.json" # cluster_meta_uri: rados://.smb/home/meta #cluster_lock_uri: rados://.smb/home/lock include_ceph_users: - client.smb.fs.cluster.home #cluster_public_addrs: #address: "192.168.2.175" #destination: "192.168.2.0/24" When I first ran ceph orch apply -i smb.yaml, it didn't like the sections I commented out related to clusters- this may be that I formatted them wrong? I would get errors like: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'cluster_meta_uri' After commenting out the clustering (for now), I successfully applied this YAML, however the .smb pool was never created so I cannot go on to the next task of fiddling around with the config files and config json. Is there a way to create the .smb pool manually? Also is there any good basic examples of a config json? I am not connecting to active directory (On Windows 365 accounts so no local AD). I will eventually write a script to pull the user details and map them to the local hosts, but want to get basic services up first. Thanks, Rob Eckert

3 6

MDS cache always increasing
by Sake Ceph 04 Sep '24

04 Sep '24

I hope someone can help us with a MDS caching problem. Ceph version 18.2.4 with cephadm container deployment. Question 1: For me it's not clear how much cache/memory you should allocate for the MDS. Is this based on the number of open files, caps or something else? Question 2/Problem: At the moment we have MDS nodes with 32 GB of memory and a configured cache limit of 20 GB. There are 4 MDS nodes: 2 active and 2 in standby-replay mode (with max_mds set at 2 of course). We pinned top directories to specific ranks, so the balancer isn't used. The memory usage is for the most part increasing, sometimes a little dip with couple hundred MB's freed. After all the memory is consumed, SWAP gets used. This results in a couple of hundred MB's of freed memory, but not much. When eventually the SWAP runs out and the memory is full, the MDS service stops and the cluster logs show: 1. no beacon from mds 2. marking mds up:active laggy 3. replacing mds 4. MDS daemon <daemon> is removed because it is dead or otherwise unavailable For example: we have the top folder app2 and app4 which is pinnend to rank 1. Folder app2 is always accessed by 4 clients (application servers), the same happens with folder app4. Folder app2 is 3 times larger than folder app4 (last time I checked, don't wanna do a du at the moment). After a couple of hours the memory usage of the MDS server stays around 18% (Grafana shows a flatline for 7 hours). At night the 9the client connects and makes first a backup with rsync of the latest snapshot folder of app2 and afterwards the same happens for folder app4 with a pause for 5 minutes. When the backup starts, the memory increases to 70% and stays at 70% after the backup of app2 is completed. 5 minutes later the memory starts increases again with the start of the backup of folder app4. When the backup is done, it's at 78% and stays there for the rest of the day. Why isn't the memory usage decreasing after the rsync is completed? Is there a memory leak with the MDS service? Ps. I have some small log files/Grafana screenshots, not sure how to share. Kind regards, Sake

3 13

The journey to CephFS metadata pool’s recovery
by m＠marcofaggian.com 03 Sep '24

03 Sep '24

> FYI: Also posted in L1Techs forum: https://forum.level1techs.com/t/recover-bluestore-osd-in-ceph-cluster/215715. ## The epic intro Through self-inflicted pain, I’m writing here to ask for volunteers in the journey of recovering the lost partitions housing CephFS metadata pool. ## The setup 1 proxmox host (I know) 1 replication rule only for NVMes (2x OSD) 1 replication rule only for HDDs (8x OSD) Each with failure domain to osd. Each OSD configured to use bluestore backend in an LVM. No backup (I know, I know). ## The cause (me) Long story short: I needed the PCIe lanes and decided to remove the two NVMEs that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to remove the two OSDs (out and destroy). This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one, so the cluster never moved the PGs stored in the NVMes, to the HDDs. ## What I’ve done untill now 1. Re-seated the NVMes to their original place. 2. Found out that the LVM didn’t have the OSD’s labels applied 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity that thought that archiving LVM config was a good thing, it payed back) 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode label at offset 102 in the LVM for that ODS. ## Wishes 1. Does anyone know a way to recover what I feel is a lost partition, given that the “file system” is ceph’s bluestore? 2. Is there a way to know, if it is, how the partition has been nuked? And possibly find a way to reverse that process. ## Closing statement Eternal reminder: If you don’t want to lose it, back it up. Thanks for your time, to the kind souls that are willing to die on this hill with me, or come up victorious!

3 3

SOLVED: How to Limit S3 Access to One Subuser
by Ansgar Jazdzewski 03 Sep '24

03 Sep '24

Hi folks, I found countless questions but no real solution on how to have multiple subusers and buckets in one account while limiting access to a bucket to just one specific subuser. Here’s how I managed to make it work: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "DenyAllUsersButOne", "Effect": "Deny", "Action": "s3:*", "Resource": [ "arn:aws:s3:::test-a", "arn:aws:s3:::test-a/*" ], "NotPrincipal": { "AWS": "arn:aws:iam:::user/<UID>:<ACCESSKEY>" } } ] } ``` I hope this might be useful for others as well. Best regards, Ansgar

2 1

rgw: swift tempurl in 18.2.1 cannot handle list_bucket correctly
by Henry Zhang （张搏航） 03 Sep '24

03 Sep '24

Dear team, hi,I met a swift tempurl problem in my prod ceph cluster, I found rgw in 18.2.1 cannot handle list_bucket correctly and led to rgw crash I created an issue: https://tracker.ceph.com/issues/67825 Can someone help to solve the issue? Best regards Henry

1 0

ceph-ansible installation error
by Michel Niyoyita 02 Sep '24

02 Sep '24

Dear team , I configuring ceph cluster using ceph-ansible , ubuntu OS 20.04 . my previous production cluster have been configured using the same configuration and it is working perfectly , now I am trying to build the new cluster using the same configurations , but I am facing the following errors: ansible-playbook site.yml Traceback (most recent call last): File "/usr/bin/ansible-playbook", line 66, in <module> from ansible.utils.display import Display, initialize_locale ImportError: cannot import name 'initialize_locale' from 'ansible.utils.display' (/usr/local/lib/python3.8/dist-packages/ansible/utils/display.py) Can someone help to solve the issue? Best regards Michel

6 11

Large volume of rgw requests on idle multisite.
by Adam Prycki 01 Sep '24

01 Sep '24

Hello, I've been testing rgw sync policies with bucket sync policies on reef 18.2.2 and discovered something that looks like a bug. My setup is really simple. I have 2 ceph clusters with RGW. Main cluster A and backup cluster B. Sync policy allows user to enable replication from A to B. I've enabled replication on few buckets via `aws s3api put-bucket-replication`. Data seems to replicate between cluster without major issues. But after sync completes and there is no data waiting for replication cluster's B rgw starts to spam cluster's A RWG with thousands of requests per second generating unnecessary load on cluster's A rgw.log pool and huge ammounts of system logs. It looks like RGW on cluster B seems to constantly check metadata shard updates. Is there a way to throttle this behavior? Log example from RGW B: 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]canonical request = GET /admin/log id=60&max-entries=100&period=579115d4-b9df-435a-99f1-40c7eb03da28&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55&type=metadata date:Sun, 01 Sep 2024 13:45:00 +0000 host:10.0.6.10:80 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240901T134500Z date;host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]canonical request hash = a89e3cdb353e45bbaa5356d1b35cd1bbfd5cb4118da8a42677b8456931b16d13 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]string to sign = AWS4-HMAC-SHA256 20240901T134500Z 20240901/home/s3/aws4_request a89e3cdb353e45bbaa5356d1b35cd1bbfd5cb4118da8a42677b8456931b16d13 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]date_k = 665f3152284a19f7eb31f7feb891577b045da2ebe65a12675d7b9b549bd7aa55 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]region_k = 81abe23f881a8ff25c0352f5f61a873bcdac805fba60c0001dedba940aac60a7 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]service_k = 49e4c3a8bce4feed131eccf6df5280dacd75d0a7fec6405a1d41e31804c4d742 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]signing_k = 2578ef828af150f3b303e7f02b87663ed9b6f783bec697048a760cc589491d86 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]generated signature = 1291a73a183e4fabac1494e1124bde56647e5f63618d1cd91c1ab894757e8da1 2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10 http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=47&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]canonical headers format = date:Sun, 01 Sep 2024 13:45:00 +0000 host:10.0.6.10:80 x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240901T134500Z Best regards Adam Prycki

1 0

How many MDS & MON servers are required
by s.dhivagar.cse＠gmail.com 01 Sep '24

01 Sep '24

Hi, We are using raw cephfs data 182 TB replica 2 and single MDS seemed to regularly run around 4002 req/s, So how many MDS & MON servers are required? Also mentioned current ceph cluster servers Client : 60 MDS: 3 (2 Active + 1 Standby) MON: 4 MGR: 3 (1 Active + 2 Standby) OSD: 52 PG : auto scale

2 1