- ceph-users - lists.ceph.io

Stretch cluster data unavailable
by przemek.kuczynski＠gmail.com 06 Sep '24

06 Sep '24

Ceph reef 18.2.4 We have pool with size 3 (2 copies in first dc , 1 copy in second ) replicated between datacenter. When we put host in maintanance in different datacenter some data is unavailable - why ? How to prevent it or fix ? 2 nodes in each dc + witness pool 13 'VolumesStandardW2' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 6257 lfor 0/2232/2230 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.30 Policy take W2 chooseleaf firstn 2 type host emit take W1 chooseleaf firstn -1 type host emit HEALTH_WARN 1 host is in maintenance mode; 1/5 mons down, quorum xxx xxx xxx xxx xxx; 3 osds down; 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; 1 host (3 osds) down; Reduced data availability: 137 pgs inactive; Degraded data redundancy: 203797/779132 objects degraded (26.157%), 522 pgs degraded, 554 pgs undersized [WRN] HOST_IN_MAINTENANCE: 1 host is in maintenance mode [WRN] PG_AVAILABILITY: Reduced data availability: 137 pgs inactive pg 12.5c is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,9] pg 12.5d is stuck undersized for 2m, current state active+undersized+degraded, last acting [0,6] pg 12.5e is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,11] pg 12.5f is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,9] pg 13.0 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7,11] pg 13.1 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8,9] pg 13.2 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11,6] pg 13.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [9,6] ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000198", "last_optimize_started": "Wed Sep 4 13:03:53 2024", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Some objects (0.261574) are degraded; try again later", "plans": []

2 1

[S3] Scale/tuning strategy for RGW in high load
by Huy Nguyen 06 Sep '24

06 Sep '24

Hi, I'm using Ceph v18.2.4, non-cephadm. Over time, my RGW cluster receives more requests, and when it reaches a certain threshold, I will manually scale more Radosgw instances to handle the increased traffic. - Is this a normal practice? - Are there any tunings I can do with RGW to make it handle more requests per instance? - Also, I'd like to know if there is any way to auto-scale those RGW instances (in a non-cephadm cluster)? Thanks

1 0

[RGW] Radosgw instances hang for a long time while doing realm update
by Huy Nguyen 05 Sep '24

05 Sep '24

Hi community, I'm using ceph v18.2.4. Each time I commit a period, all of my radosgw instances paused for about 30s which is kinda long, and make my object service down during that time. how to avoid the down time? Is there better/safer practice? Thanks Here is radosgw logs each time I did a period commit. You can see it was paused for a while since the last request 2024-08-23T22:43:32.152+0700 7f4594d2f700 1 ====== starting new request req=0x7f42dcfbf730 ===== 2024-08-23T22:43:32.152+0700 7f438b11c700 1 ====== req done req=0x7f42dcfbf730 op status=0 http_status=200 latency=0.000000000s ====== 2024-08-23T22:43:32.152+0700 7f438b11c700 1 beast: 0x7f42dcfbf730: 172.24.9.101 - anonymous [23/Aug/2024:22:43:32.152 +0700] "GET / HTTP/1.0" 200 231 - "check_http/v2.2 (monitoring-plugins 2.2)" - latency=0.000000000s 2024-08-23T22:44:02.778+0700 7f42e1fca700 1 rgw realm reloader: Pausing frontends for realm update... 2024-08-23T22:44:02.778+0700 7f42e1fca700 1 rgw realm reloader: Frontends paused 2024-08-23T22:44:03.358+0700 7f42e1fca700 1 rgw realm reloader: driver closed 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Creating new driver 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 mgrc service_daemon_register rgw.5055090799 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4.... 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Finishing initialization of new driver 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: - REST subsystem init 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: - usage subsystem init 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Resuming frontends with new realm configuration. 2024-08-23T22:44:06.135+0700 7f4637674700 1 ====== starting new request req=0x7f42dcfbf730 ===== 2024-08-23T22:44:06.136+0700 7f4637674700 -1 res_query() failed 2024-08-23T22:44:06.136+0700 7f4637674700 1 ====== req done req=0x7f42dcfbf730 op status=0 http_status=405 latency=0.000999984s ====== 2024-08-23T22:44:06.136+0700 7f4637674700 1 beast: 0x7f42dcfbf730: 172.24.17.21 - - [23/Aug/2024:22:44:06.135 +0700] "GET /admin HTTP/1.1" 405 133 - "curl/7.61.1" - latency=0.000999984s

2 1

Re: The snaptrim queue of PGs has not decreased for several days.
by Eugen Block 05 Sep '24

05 Sep '24

Hi (please don't drop the ML from your responses), > All PGs of pool cephfs are affected and they are in all OSDs then just pick a random one and check if anything stands out. I'm not sure if you mentioned it already, did you also try restarting OSDs? > Oh, not yesterday. I do it now, then I compat all osds with nostrim. > Do I add OSDs? Let's wait for the other results first (compaction, reducing defaults, OSD restart). If that doesn't change anything, I would probably try to add three more OSDs. I assume you have three hosts? Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: > Hello Eugen, > > Am 20.08.2024 um 09:44 schrieb Eugen Block: >> You could also look into the historic_ops of the primary OSD for >> one affected PG: > > All PGs of pool cephfs are affected and they are in all OSDs :-( > >> Did you reduce the default values I mentioned? > > Oh, not yesterday. I do it now, then I compat all osds with nostrim. > > Do I add OSDs? > > Regars, > > Gio > > >> >> ceph tell osd.<OSD_ID> dump_historic_ops_by_duration >> >> But I'm not sure if that can actually help here. There are plenty >> of places to look at, you could turn on debug logs on one primary >> OSD and inspect the output. >> >> I just get the feeling that this is one of the corner cases with >> too few OSDs, although the cluster load seems to be low. >> >> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >> >>> Hello Eugen, >>> >>> yesterday after stop and go of snaptrim the queue decrease a >>> little and then remain blocked. >>> They didn't grow and didn't decrease. >>> >>> Is that good or bad? >>> >>> >>> Am 19.08.2024 um 15:43 schrieb Eugen Block: >>>> There's a lengthy thread [0] where several approaches are >>>> proposed. The worst is a OSD recreation, but that's the last >>>> resort, of course. >>>> >>>> What's are the current values for these configs? >>>> >>>> ceph config get osd osd_pg_max_concurrent_snap_trims >>>> ceph config get osd osd_max_trimming_pgs >>>> >>>> Maybe decrease them to 1 each while the nosnaptrim flag is set, >>>> then unset it. You could also try online (and/or offline osd >>>> compaction) before unsetting the flag. Are the OSD processes >>>> utilizing an entire CPU? >>>> >>>> [0] https://www.spinics.net/lists/ceph-users/msg75626.html >>>> >>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>> >>>>> Hallo Eugen, >>>>> >>>>> yes, the load is for now not too much. >>>>> >>>>> I stop the snap and now this is the output. No changes in the queue. >>>>> >>>>> root@kube-master02:~# k ceph -s >>>>> Info: running 'ceph' command with args: [-s] >>>>> cluster: >>>>> id: 3a35629a-6129-4daf-9db6-36e0eda637c7 >>>>> health: HEALTH_WARN >>>>> nosnaptrim flag(s) set >>>>> 32 pgs not deep-scrubbed in time >>>>> 32 pgs not scrubbed in time >>>>> >>>>> services: >>>>> mon: 3 daemons, quorum bx,bz,ca (age 30h) >>>>> mgr: a(active, since 29h), standbys: b >>>>> mds: 1/1 daemons up, 1 hot standby >>>>> osd: 6 osds: 6 up (since 21h), 6 in (since 6d) >>>>> flags nosnaptrim >>>>> >>>>> data: >>>>> volumes: 1/1 healthy >>>>> pools: 4 pools, 97 pgs >>>>> objects: 4.21M objects, 2.5 TiB >>>>> usage: 7.7 TiB used, 76 TiB / 84 TiB avail >>>>> pgs: 65 active+clean >>>>> 32 active+clean+snaptrim_wait >>>>> >>>>> io: >>>>> client: 7.4 MiB/s rd, 7.9 MiB/s wr, 11 op/s rd, 35 op/s wr >>>>> >>>>> Am 19.08.2024 um 14:54 schrieb Eugen Block: >>>>>> What happens when you disable snaptrimming entirely? >>>>>> >>>>>> ceph osd set nosnaptrim >>>>>> >>>>>> So the load on your cluster seems low, but are the OSDs heavily >>>>>> utilized? Have you checked iostat? >>>>>> >>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>> >>>>>>> Hello Eugen, >>>>>>> >>>>>>> *root@kube-master02:~# k ceph -s* >>>>>>> >>>>>>> Info: running 'ceph' command with args: [-s] >>>>>>> cluster: >>>>>>> id: 3a35629a-6129-4daf-9db6-36e0eda637c7 >>>>>>> health: HEALTH_WARN >>>>>>> 32 pgs not deep-scrubbed in time >>>>>>> 32 pgs not scrubbed in time >>>>>>> >>>>>>> services: >>>>>>> mon: 3 daemons, quorum bx,bz,ca (age 13h) >>>>>>> mgr: a(active, since 13h), standbys: b >>>>>>> mds: 1/1 daemons up, 1 hot standby >>>>>>> osd: 6 osds: 6 up (since 5h), 6 in (since 5d) >>>>>>> >>>>>>> data: >>>>>>> volumes: 1/1 healthy >>>>>>> pools: 4 pools, 97 pgs >>>>>>> objects: 4.20M objects, 2.5 TiB >>>>>>> usage: 7.7 TiB used, 76 TiB / 84 TiB avail >>>>>>> pgs: 65 active+clean >>>>>>> 20 active+clean+snaptrim_wait >>>>>>> 12 active+clean+snaptrim >>>>>>> >>>>>>> io: >>>>>>> client: 3.5 MiB/s rd, 3.6 MiB/s wr, 6 op/s rd, 12 op/s wr >>>>>>> >>>>>>> If I understand the documentation correctly, I will never have >>>>>>> a scrub unless the PGs (Placement Groups) are active and clean. >>>>>>> >>>>>>> All 32 PGs of the CephFS pool have been in this status for >>>>>>> several days: >>>>>>> >>>>>>> * 20 active+clean+snaptrim_wait >>>>>>> * 12 active+clean+snaptrim" >>>>>>> >>>>>>> Today, I restarted the MON, MGR, and MDS, but no changes in >>>>>>> the growing. >>>>>>> >>>>>>> Am 18.08.2024 um 18:39 schrieb Eugen Block: >>>>>>>> Can you share the current ceph status? Are the OSDs reporting >>>>>>>> anything suspicious? How is the disk utilization? >>>>>>>> >>>>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>>>> >>>>>>>>> More information: >>>>>>>>> >>>>>>>>> The snaptrim take a lot of time but the he objects_trimmed are "0" >>>>>>>>> >>>>>>>>> "objects_trimmed": 0, >>>>>>>>> "snaptrim_duration": 500.58076017500002, >>>>>>>>> >>>>>>>>> It could explain, why the queue are growing up.. >>>>>>>>> >>>>>>>>> >>>>>>>>> Am 17.08.2024 um 14:37 schrieb Giovanna Ratini: >>>>>>>>>> Hello again, >>>>>>>>>> >>>>>>>>>> I checked the pgs dump. Snapshot grow up >>>>>>>>>> >>>>>>>>>> Query für PG: 3.12 >>>>>>>>>> { >>>>>>>>>> "snap_trimq": >>>>>>>>>> "[5b974~3b,5cc3a~1,5cc3c~1,5cc3e~1,5cc40~1,5cd83~1,5cd85~1,5cd87~1,5cd89~1,5cecc~1,5cece~4,5ced3~2,5cf72~1,5cf74~4,5cf79~a2,5d0b8~1,5d0bb~1,5d0bd~a5,5d1f9~2,5d204~a5,5d349~a7,5d48e~3,5d493~a4,5d5d7~a7,5d71e~a3,5d7c2~3,5d860~1,5d865~4,5d86a~a2,5d9aa~1,5d9ac~1,5d9ae~a5,5daf3~a5,5db9a~2,5dc3a~a5,5dce1~1,5dce3~1,5dd81~a7,5dec8~a7,5e00f~a7,5e156~a8,5e29d~1,5e29f~a7,5e3e6~a8,5e52e~a6,5e5d6~2,5e676~a6,5e71e~2,5e7be~a9,5e907~a5,5e9ad~3,5ea50~a7,5eaf9~1,5eafb~1,5eb99~a7,5ec42~2,5ece2~a7,5ed8a~2,5ee2b~a9,5ef74~a7,5f01c~1,5f0bd~a1,5f15f~1,5f161~1,5f163~1,5f167~1,5f206~a1,5f2a8~1,5f2aa~1,5f2ac~1,5f2ae~1,5f34f~a1,5f3f1~1,5f3f3~1,5f3f5~1,5f3f7~1,5f499~a1,5f53b~1,5f53d~1,5f53f~1,5f541~1,5f5e3~a1,5f685~1,5f687~1,5f689~1,5f68d~1,5f72d~a1,5f7cf~1,5f7d1~1,5f7d3~1]", >>>>>>>>>> * "snap_trimq_len": 5421,* >>>>>>>>>> "state": "active+clean+snaptrim", >>>>>>>>>> "epoch": 734130, >>>>>>>>>> >>>>>>>>>> Query für PG: 3.12 >>>>>>>>>> { >>>>>>>>>> "snap_trimq": >>>>>>>>>> "[5b976~39,5ba53~1,5ba56~a0,5cc3a~1,5cc3c~1,5cc3e~1,5cc40~1,5cd83~1,5cd85~1,5cd87~1,5cd89~1,5cecc~1,5cece~4,5ced3~2,5cf72~1,5cf74~4,5cf79~a2,5d0b8~1,5d0bb~1,5d0bd~a5,5d1f9~2,5d204~a5,5d349~a7,5d48e~3,5d493~a4,5d5d7~a7,5d71e~a3,5d7c2~3,5d860~1,5d865~4,5d86a~a2,5d9aa~1,5d9ac~1,5d9ae~a5,5daf3~a5,5db9a~2,5dc3a~a5,5dce1~1,5dce3~1,5dd81~a7,5dec8~a7,5e00f~a7,5e156~a8,5e29d~1,5e29f~a7,5e3e6~a8,5e52e~a6,5e5d6~2,5e676~a6,5e71e~2,5e7be~a9,5e907~a5,5e9ad~3,5ea50~a7,5eaf9~1,5eafb~1,5eb99~a7,5ec42~2,5ece2~a7,5ed8a~2,5ee2b~a9,5ef74~a7,5f01c~1,5f0bd~a1,5f15f~1,5f161~1,5f163~1,5f167~1,5f206~a1,5f2a8~1,5f2aa~1,5f2ac~1,5f2ae~1,5f34f~a1,5f3f1~1,5f3f3~1,5f3f5~1,5f3f7~1,5f499~a1,5f53b~1,5f53d~1,5f53f~1,5f541~1,5f5e3~a1,5f685~1,5f687~1,5f689~1,5f68d~1,5f72d~a1,5f7cf~1,5f7d1~1,5f7d3~1,5f875~a1]", >>>>>>>>>> * "snap_trimq_len": 5741,* >>>>>>>>>> "state": "active+clean+snaptrim", >>>>>>>>>> "epoch": 734240, >>>>>>>>>> "up": [ >>>>>>>>>> >>>>>>>>>> Do you know the way to see if the snaptim "process" works? >>>>>>>>>> >>>>>>>>>> Best Regard >>>>>>>>>> >>>>>>>>>> Gio >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Am 17.08.2024 um 12:59 schrieb Giovanna Ratini: >>>>>>>>>>> Hello Eugen, >>>>>>>>>>> >>>>>>>>>>> thank you for your answer. >>>>>>>>>>> >>>>>>>>>>> I restarted all the kube-ceph nodes one after the other. >>>>>>>>>>> Nothing has changed. >>>>>>>>>>> >>>>>>>>>>> ok, I deactivate the snap ... : ceph fs snap-schedule deactivate / >>>>>>>>>>> >>>>>>>>>>> Is there a way to see how many snapshots will be deleted per hour? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Gio >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Am 17.08.2024 um 10:12 schrieb Eugen Block: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> have you tried to fail the mgr? Sometimes the PG stats >>>>>>>>>>>> are not correct. You could also temporarily disable >>>>>>>>>>>> snapshots to see if things settle down. >>>>>>>>>>>> >>>>>>>>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>>>>>>>> >>>>>>>>>>>>> Hello all, >>>>>>>>>>>>> >>>>>>>>>>>>> We use Ceph (v18.2.2) and Rook (1.14.3) as the CSI for a >>>>>>>>>>>>> Kubernetes environment. Last week, we had a problem with >>>>>>>>>>>>> the MDS falling behind on trimming every 4-5 days >>>>>>>>>>>>> (GitHub issue link). We resolved the issue using the >>>>>>>>>>>>> steps outlined in the GitHub issue. >>>>>>>>>>>>> >>>>>>>>>>>>> We have 3 hosts (I know, I need to increase this as soon >>>>>>>>>>>>> as possible, and I will!) and 6 OSDs. After running the >>>>>>>>>>>>> commands: >>>>>>>>>>>>> >>>>>>>>>>>>> ceph config set mds mds_dir_max_commit_size 80, >>>>>>>>>>>>> >>>>>>>>>>>>> ceph fs fail <fs_name>, and >>>>>>>>>>>>> >>>>>>>>>>>>> ceph fs set <fs_name> joinable true, >>>>>>>>>>>>> >>>>>>>>>>>>> After that, the snaptrim queue for our PGs has stopped >>>>>>>>>>>>> decreasing. All PGs of our CephFS are in either >>>>>>>>>>>>> active+clean+snaptrim_wait or active+clean+snaptrim >>>>>>>>>>>>> states. For example, the PG 3.12 is in the >>>>>>>>>>>>> active+clean+snaptrim state, and its snap_trimq_len was >>>>>>>>>>>>> 4077 yesterday but has increased to 4538 today. >>>>>>>>>>>>> >>>>>>>>>>>>> I increased the osd_snap_trim_priority to 10 (ceph >>>>>>>>>>>>> config set osd osd_snap_trim_priority 10), but it didn't >>>>>>>>>>>>> help. Only the PGs of our CephFS have this problem. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have any ideas on how we can resolve this issue? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>> Giovanna >>>>>>>>>>>>> p.s. I'm not a ceph expert :-). >>>>>>>>>>>>> Faulkener asked me for more information, so here it is: >>>>>>>>>>>>> MDS Memory: 11GB >>>>>>>>>>>>> mds_cache_memory_limit: 11,811,160,064 bytes >>>>>>>>>>>>> >>>>>>>>>>>>> root@kube-master02:~# ceph fs snap-schedule status / >>>>>>>>>>>>> { >>>>>>>>>>>>> "fs": "rook-cephfs", >>>>>>>>>>>>> "subvol": null, >>>>>>>>>>>>> "path": "/", >>>>>>>>>>>>> "rel_path": "/", >>>>>>>>>>>>> "schedule": "3h", >>>>>>>>>>>>> "retention": {"h": 24, "w": 4}, >>>>>>>>>>>>> "start": "2024-05-05T00:00:00", >>>>>>>>>>>>> "created": "2024-05-05T17:28:18", >>>>>>>>>>>>> "first": "2024-05-05T18:00:00", >>>>>>>>>>>>> "last": "2024-08-15T18:00:00", >>>>>>>>>>>>> "last_pruned": "2024-08-15T18:00:00", >>>>>>>>>>>>> "created_count": 817, >>>>>>>>>>>>> "pruned_count": 817, >>>>>>>>>>>>> "active": true >>>>>>>>>>>>> } >>>>>>>>>>>>> I do not understand if the snapshots in the PGs are >>>>>>>>>>>>> correlated with the snapshots on CephFS. Until we >>>>>>>>>>>>> encountered the issue with the "MDS falling behind on >>>>>>>>>>>>> trimming every 4-5 days," we didn't have any problems >>>>>>>>>>>>> with snapshots. >>>>>>>>>>>>> >>>>>>>>>>>>> Could someone explain me this or send me to the documentation? >>>>>>>>>>>>> Thank you >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>> >>>>>>> -- >>>>>>> Giovanna Ratini >>>>>>> Mail:ratini@dbvis.inf.uni-konstanz.de >>>>>>> Phone: +49 (0) 7531 88 - 4550 >>>>>>> >>>>>>> Technical Support >>>>>>> Data Analysis and Visualization Group >>>>>>> Department of Computer and Information Science >>>>>>> University of Konstanz (Box 78) >>>>>>> Universitätsstr. 10 >>>>>>> 78457 Konstanz, Germany >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> -- >>>>> Giovanna Ratini >>>>> Mail:ratini@dbvis.inf.uni-konstanz.de >>>>> Phone: +49 (0) 7531 88 - 4550 >>>>> >>>>> Technical Support >>>>> Data Analysis and Visualization Group >>>>> Department of Computer and Information Science >>>>> University of Konstanz (Box 78) >>>>> Universitätsstr. 10 >>>>> 78457 Konstanz, Germany >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> -- >>> Giovanna Ratini >>> Mail:ratini@dbvis.inf.uni-konstanz.de >>> Phone: +49 (0) 7531 88 - 4550 >>> >>> Technical Support >>> Data Analysis and Visualization Group >>> Department of Computer and Information Science >>> University of Konstanz (Box 78) >>> Universitätsstr. 10 >>> 78457 Konstanz, Germany >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Giovanna Ratini > Mail:ratini@dbvis.inf.uni-konstanz.de > Phone: +49 (0) 7531 88 - 4550 > > Technical Support > Data Analysis and Visualization Group > Department of Computer and Information Science > University of Konstanz (Box 78) > Universitätsstr. 10 > 78457 Konstanz, Germany

4 11

Issue Replacing OSD with cephadm: Partition Path Not Accepted
by Herbert Faleiros 04 Sep '24

04 Sep '24

I am on a journey, so far successful, to update our clusters to supported versions. I started with Luminous and Ubuntu 16.04, and now we are on Reef with Ubuntu 20.04. We still have more updates to do, but at the moment, I encountered an issue with an OSD, and it was necessary to replace a disk. Since the cluster was adopted, I'm not entirely sure what the best way to replace this OSD is, as with cephadm, it doesn't like when the path for the device is a partition. I could recreate the OSD using traditional methods and then adopt the OSD, but that doesn't seem like the best approach. Does anyone know how I should proceed to recreate this OSD? I had the same problem in my lab, where I am already on Quincy. What I am trying to do is: # ceph orch osd rm 6 --replace --zap # ceph orch daemon add osd osd-nodeXXX:data_devices=/dev/sdb1,db_devices=/dev/sda3 The error it gives is: /usr/bin/docker: stderr ceph-volume lvm batch: error: /dev/sdb1 is a partition, please pass LVs or raw block devices

5 6

CephFS troubleshooting
by Eugenio Tampieri 04 Sep '24

04 Sep '24

Hello, I'm writing to troubleshoot an otherwise functional Ceph quincy cluster that has issues with cephfs. I cannot mount it with ceph-fuse (it gets stuck), and if I mount it with NFS I can list the directories but I cannot read or write anything. Here's the output of ceph -s cluster: id: 3b92e270-1dd6-11ee-a738-000c2937f0ec health: HEALTH_WARN mon ceph-storage-a is low on available space 1 daemons have recently crashed too many PGs per OSD (328 > max 250) services: mon: 5 daemons, quorum ceph-mon-a,ceph-storage-a,ceph-mon-b,ceph-storage-c,ceph-storage-d (age 105m) mgr: ceph-storage-a.ioenwq(active, since 106m), standbys: ceph-mon-a.tiosea mds: 1/1 daemons up, 2 standby osd: 4 osds: 4 up (since 104m), 4 in (since 24h) rbd-mirror: 2 daemons active (2 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 481 pgs objects: 231.83k objects, 648 GiB usage: 1.3 TiB used, 1.8 TiB / 3.1 TiB avail pgs: 481 active+clean io: client: 1.5 KiB/s rd, 8.6 KiB/s wr, 1 op/s rd, 0 op/s wr Best regards, Eugenio Tampieri

3 5

quincy radosgw-admin log list show entries with only the date
by Boris 04 Sep '24

04 Sep '24

I am not sure how I should describe this. We enable ops log and have these entries here together with the normal ops logs. # radosgw-admin log list | grep ^.*-- "2024-09-03-12--", I did a head on the file and got this: B��f�� anonymous list_bucketsHEAD / HTTP/1.0200 ;tx00000e66dccefcb9f8a39-0066d6fa42-171a... Anything I can do about this? I can not remove them via radosgw-admin (only the rados objects directly) and I can not read it via radosgw-admin command. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 2

SMB Service in Squid
by Robert W. Eckert 04 Sep '24

04 Sep '24

I have upgraded my home cluster to 19.1.0 and wanted to try out the SMB orchestration features to improve my hacked SMB shared using CTDB and SMB services on each host. My smb.yaml file looks like service_type: smb service_id: home placement: hosts: - HOST1 - HOST2 - HOST3 - HOST4 spec: cluster_id: home features: - domain #clustered: true config_uri: rados://.smb/home/scc.toml custom_dns: - "<DNS SERVERS>" join_sources: - "rados:mon-config-key:smb/config/home/join1.json" # cluster_meta_uri: rados://.smb/home/meta #cluster_lock_uri: rados://.smb/home/lock include_ceph_users: - client.smb.fs.cluster.home #cluster_public_addrs: #address: "192.168.2.175" #destination: "192.168.2.0/24" When I first ran ceph orch apply -i smb.yaml, it didn't like the sections I commented out related to clusters- this may be that I formatted them wrong? I would get errors like: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'cluster_meta_uri' After commenting out the clustering (for now), I successfully applied this YAML, however the .smb pool was never created so I cannot go on to the next task of fiddling around with the config files and config json. Is there a way to create the .smb pool manually? Also is there any good basic examples of a config json? I am not connecting to active directory (On Windows 365 accounts so no local AD). I will eventually write a script to pull the user details and map them to the local hosts, but want to get basic services up first. Thanks, Rob Eckert

3 6

MDS cache always increasing
by Sake Ceph 04 Sep '24

04 Sep '24

I hope someone can help us with a MDS caching problem. Ceph version 18.2.4 with cephadm container deployment. Question 1: For me it's not clear how much cache/memory you should allocate for the MDS. Is this based on the number of open files, caps or something else? Question 2/Problem: At the moment we have MDS nodes with 32 GB of memory and a configured cache limit of 20 GB. There are 4 MDS nodes: 2 active and 2 in standby-replay mode (with max_mds set at 2 of course). We pinned top directories to specific ranks, so the balancer isn't used. The memory usage is for the most part increasing, sometimes a little dip with couple hundred MB's freed. After all the memory is consumed, SWAP gets used. This results in a couple of hundred MB's of freed memory, but not much. When eventually the SWAP runs out and the memory is full, the MDS service stops and the cluster logs show: 1. no beacon from mds 2. marking mds up:active laggy 3. replacing mds 4. MDS daemon <daemon> is removed because it is dead or otherwise unavailable For example: we have the top folder app2 and app4 which is pinnend to rank 1. Folder app2 is always accessed by 4 clients (application servers), the same happens with folder app4. Folder app2 is 3 times larger than folder app4 (last time I checked, don't wanna do a du at the moment). After a couple of hours the memory usage of the MDS server stays around 18% (Grafana shows a flatline for 7 hours). At night the 9the client connects and makes first a backup with rsync of the latest snapshot folder of app2 and afterwards the same happens for folder app4 with a pause for 5 minutes. When the backup starts, the memory increases to 70% and stays at 70% after the backup of app2 is completed. 5 minutes later the memory starts increases again with the start of the backup of folder app4. When the backup is done, it's at 78% and stays there for the rest of the day. Why isn't the memory usage decreasing after the rsync is completed? Is there a memory leak with the MDS service? Ps. I have some small log files/Grafana screenshots, not sure how to share. Kind regards, Sake

3 13

The journey to CephFS metadata pool’s recovery
by m＠marcofaggian.com 03 Sep '24

03 Sep '24

> FYI: Also posted in L1Techs forum: https://forum.level1techs.com/t/recover-bluestore-osd-in-ceph-cluster/215715. ## The epic intro Through self-inflicted pain, I’m writing here to ask for volunteers in the journey of recovering the lost partitions housing CephFS metadata pool. ## The setup 1 proxmox host (I know) 1 replication rule only for NVMes (2x OSD) 1 replication rule only for HDDs (8x OSD) Each with failure domain to osd. Each OSD configured to use bluestore backend in an LVM. No backup (I know, I know). ## The cause (me) Long story short: I needed the PCIe lanes and decided to remove the two NVMEs that were hosting the metadata pool for CephFS and .mgr pool. I proceeded to remove the two OSDs (out and destroy). This is where I’ve done goof: I didn’t change the replication rule to HDDs’ one, so the cluster never moved the PGs stored in the NVMes, to the HDDs. ## What I’ve done untill now 1. Re-seated the NVMes to their original place. 2. Found out that the LVM didn’t have the OSD’s labels applied 3. Forced the backed-up LVM config to the two NVMes (thanks to the holy entity that thought that archiving LVM config was a good thing, it payed back) 4. Trying ceph-volume lvm activate 8 <id> to find out that it’s unable to decode label at offset 102 in the LVM for that ODS. ## Wishes 1. Does anyone know a way to recover what I feel is a lost partition, given that the “file system” is ceph’s bluestore? 2. Is there a way to know, if it is, how the partition has been nuked? And possibly find a way to reverse that process. ## Closing statement Eternal reminder: If you don’t want to lose it, back it up. Thanks for your time, to the kind souls that are willing to die on this hill with me, or come up victorious!

3 3