- ceph-users - lists.ceph.io

osd memory target not work
by farhad kh 20 Jun '23

20 Jun '23

when set osd_memory_target for limitation usage memory for osd disk ,This value is expected to be set for the OSD container .But with the docker stats command, this value is not seen Is my perception of this process wrong? ----------- [root@opcsdfpsbpp0201 ~]# ceph orch ps | grep osd.12 osd.12 opcsdfpsbpp0201 running (9d) 5m ago 9d 1205M 1953M 17.2.6 c9a1062f7289 bf27cfe16046 [root@opcsdfpsbpp0201 ~]# docker stats | grep osd 1253766d6a78 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-48 0.05% 2.237GiB / 7.732GiB 28.93% 0B / 0B 86.8GB / 562GB 63 2bc012e5c604 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-67 0.20% 727MiB / 7.732GiB 9.18% 0B / 0B 37.5GB / 1.29TB 63 dc0bf068050b ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-62 0.11% 360.5MiB / 7.732GiB 4.55% 0B / 0B 125MB / 1.85GB 63 c5f119a37652 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-55 0.12% 312.5MiB / 7.732GiB 3.95% 0B / 0B 86.6MB / 1.66GB 63 7f0b7b61807d ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-5 0.11% 299.4MiB / 7.732GiB 3.78% 0B / 0B 119MB / 1.6GB 63 dadffc77f7b6 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-40 0.11% 274MiB / 7.732GiB 3.46% 0B / 0B 110MB / 1.5GB 63 e439e58d907e ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-34 0.12% 355.9MiB / 7.732GiB 4.49% 0B / 0B 125MB / 1.78GB 63 5e500e2197d6 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-25 0.11% 273.3MiB / 7.732GiB 3.45% 0B / 0B 128MB / 1.55GB 63 a63709567669 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-19 0.11% 714.6MiB / 7.732GiB 9.03% 0B / 0B 89.8MB / 167GB 63 bf27cfe16046 ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-12 0.16% 1.177GiB / 7.732GiB 15.23% 0B / 0B 40.8GB / 644GB 63 ----------- # ceph orch ps | grep osd | grep opcsdfpsbpp0201 osd.5 opcsdfpsbpp0201 running (9d) 6m ago 9d 298M 1953M 17.2.6 c9a1062f7289 7f0b7b61807d osd.12 opcsdfpsbpp0201 running (9d) 6m ago 9d 1205M 1953M 17.2.6 c9a1062f7289 bf27cfe16046 osd.19 opcsdfpsbpp0201 running (9d) 6m ago 9d 704M 1953M 17.2.6 c9a1062f7289 a63709567669 osd.25 opcsdfpsbpp0201 running (9d) 6m ago 9d 273M 1953M 17.2.6 c9a1062f7289 5e500e2197d6 osd.34 opcsdfpsbpp0201 running (9d) 6m ago 9d 355M 1953M 17.2.6 c9a1062f7289 e439e58d907e osd.40 opcsdfpsbpp0201 running (9d) 6m ago 9d 273M 1953M 17.2.6 c9a1062f7289 dadffc77f7b6 osd.48 opcsdfpsbpp0201 running (4h) 6m ago 9d 2290M 1953M 17.2.6 c9a1062f7289 1253766d6a78 osd.55 opcsdfpsbpp0201 running (9d) 6m ago 9d 312M 1953M 17.2.6 c9a1062f7289 c5f119a37652 osd.62 opcsdfpsbpp0201 running (9d) 6m ago 9d 359M 1953M 17.2.6 c9a1062f7289 dc0bf068050b osd.67 opcsdfpsbpp0201 running (9d) 6m ago 9d 727M 1953M 17.2.6 c9a1062f7289 2bc012e5c604 ---------------------------------- #ceph config get mgr osd_memory_target 2048000000

2 1

X large objects found in pool 'XXX.rgw.buckets.index'
by Gilles Mocellin 20 Jun '23

20 Jun '23

Hello, I still have large OMAP objects since a year. These objects are probably from an ancient bucket that has been removed. So I cannot use bilog trim. Depp-scrub dos nothing. Also, even if I don't have a huge cluster (my Object Storage pools is only arounde 10TB), the rgw-orphan-list is too long to run. So, As I have only 6 Large OMAP objects (just above 200000 keys), I would like to find and remove the orphan rados objects. If someone can tell me if I'm right in my assumptions ? Let's take that log : 2023-06-09T00:51:10.222449+0200 osd.66 (osd.66) 12 : cluster [WRN] Large omap object found. Object: 3:56f3a469:::.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.9:head PG: 3.9625cf6a (3.2a) Key count: 200304 Size (bytes): 58606170 I deduce that the bucket-id is : aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2 I cannot find a bucket name with that ID in the metadata list : # radosgw-admin metadata list --metadata-key bucket.instance | grep -i aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2 returns nothing. If I list all objects begining with that bucket ID, I find : gmo_admin@fidcl-mrs4-sto-sds-01:~$ sudo rados -p MRS4.rgw.buckets.index ls | grep aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2 | cat .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.25 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.0 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.15 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.17 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.9 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.26 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.7 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.23 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.2 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.3 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.16 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.13 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.22 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.21 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.4 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.20 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.19 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.11 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.27 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.24 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.28 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.10 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.14 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.18 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.1 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.6 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.8 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.12 .dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.5 Do you think I'm safe to delete them ? If they are all about a non-existent bucket... How can I be sure that an RGW index object and the omap keys it has, are not used ?

1 0

OpenStack (cinder) volumes retyping on Ceph back-end
by andrea.martra＠oscct.it 20 Jun '23

20 Jun '23

Hello, I configured different back-end storage on OpenStack (Yoga release) and using Ceph (ceph version 17.2.4) with different pools (volumes, cloud-basic, shared-hosting-os, shared-hosting-homes,...) for RBD application. I created different volume types towards each of the backends and everything works perfectly. If I perform a "change volume type" from OpenStack on volumes attached to the VMs the system successfully migrates the volume from the source pool to the destination pool and at the end of the process the volume is visible in the new pool and is removed from the old pool. The problem encountered is that when reconfiguring the VM, to specify the new pool associated with the volumes (performed through a resize of the VM, I haven't found any other method to change the information on the nova/cinder db automatically. I also did some tests shut-off of the VM and modification of the xml through virsh edit and startup the VM) the volume presented to the VM is exactly the version and content on the retype date of the volume itself. All data written and modified after the retype is lost. The VM after the retype continues to work perfectly in RW but the "new" volume created in the new pool is not used to write data and consequently when the VM is shut down all the changes are lost. Do you have any idea how to carry out a check and possibly how to proceed in order not to lose the data of the vm of which I have retyped the volume? The data is written somewhere because the VMs work perfectly. Thank you

3 3

Transmit rate metric based per bucket
by Szabo, Istvan (Agoda) 20 Jun '23

20 Jun '23

Hello, I'd like to know is there a way to query some metrics/logs in octopus (or if has newer version I'm interested for the future too) about the bandwidth used in the bucket for put/get operations? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots
by Janek Bevendorff 19 Jun '23

19 Jun '23

Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of around 15TB which hosts our CephFS metadata pool. Usually, that's more than enough. The normal size of the pool ranges between 200 and 800GiB (which is quite a lot of fluctuation already). Yesterday, we had suddenly had the pool fill up entirely and they only way to fix it was to add more capacity. I increased the pool size to 18TB by adding more SSDs and could resolve the problem. After a couple of hours of reshuffling, the pool size finally went back to 230GiB. But then we had another fill-up tonight to 7.6TiB. Luckily, I had adjusted the weights so that not all disks could fill up entirely like last time, so it ended there. I wasn't really able to identify the problem yesterday, but under the more controllable scenario today, I could check the MDS logs at debug_mds=10 and to me it seems like the problem is caused by snapshot trimming. The logs contain a lot of snapshot-related messages for paths that haven't been touched in a long time. The messages all look something like this: May 31 09:16:48 XXX ceph-mds[2947525]: 2023-05-31T09:16:48.292+0200 7f7ce1bd9700 10 mds.1.cache.ino(0x1000b3c3670) add_client_cap first cap, joining realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1 b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 ... May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.396+0200 7f0e6a6ca700 10 mds.0.cache | |______ 3 rep [dir 0x100000218fe.101111101* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 replicated=0 dirty=0 waiter=0 authpin=0 tempexporting=0 0x5607759d9600] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.452+0200 7f0e6a6ca700 10 mds.0.cache | | |____ 4 rep [dir 0x100000ff904.100111101010* /storage/REDACTED/| ptrwaiter=0 request=0 child=0 frozen=0 subtree=1 importing=0 replicated=0 waiter=0 authpin=0 tempexporting=0 0x56034ed25200] May 31 09:25:03 XXX ceph-mds[3268481]: 2023-05-31T09:25:03.716+0200 7f0e6becd700 10 mds.0.server set_trace_dist snaprealm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' 2023-01-01T00:00:04.657252+0100),1941=snap(1941 0x10000000000 'monthly_20230201' 2023-02-01T00:00:01.854059+0100),19a6=snap(19a6 0x10000000000 'monthly_20230301' 2023-03-01T00:00:01.215197+0100),1a24=snap(1a24 0x10000000000 'monthly_20230401' ...) len=384 May 31 09:25:36 deltaweb055 ceph-mds[3268481]: 2023-05-31T09:25:36.076+0200 7f0e6becd700 10 mds.0.cache.ino(0x10004d74911) remove_client_cap last cap, leaving realm snaprealm(0x10000000000 seq 1b1c lc 1b1b cr 1b1b cps 2 snaps={185f=snap(185f 0x10000000000 'monthly_20221201' 2022-12-01T00:00:01.530830+0100),18de=snap(18de 0x10000000000 'monthly_20230101' ...) The daily_*, montly_* etc. names are the names of our regular snapshots. I posted a larger log file snippet using ceph-post-file with the ID: da0eb93d-f340-4457-8a3f-434e8ef37d36 Is it possible that the MDS are trimming old snapshots without taking care not to fill up the entire metadata pool? Cheers Janek

3 16

Starting v17.2.5 RGW SSE with default key (likely others) no longer works
by Jayanth Reddy 19 Jun '23

19 Jun '23

Hello Folks, I've been experimenting with RGW encryption and found this out. Focusing on Quincy and Reef dev, for the SSE (any methods) to work, transit has to be end to end encrypted, however if there is a proxy, then [1] can be made use to tell RGW that SSL is being terminated. As per docs, RGW can still continue to accept SSE if rgw_crypt_require_ssl is set to false as an overriding item for the requirement of encryption in transit. Below are my observations. Until v17.2.3 ( quay.io/ceph/ceph@sha256:43f6e905f3e34abe4adbc9042b9d6f6b625dee8fa8d93c2bae53fa9b61c3df1a), setting the same key as in [2], would show the object unreadable when copied using # rados -p default.rgw.buckets.data get 03c2ef32-b7c8-4e18-8e0c-ebac10a42f10.17254.1_file.plain file.enc The object would be unreadable. The original object is in plain text. Ofcourse, with rgw_crypt_require_ssl to false or [1] However, starting with v17.2.4 onwards and even until my recent testing with reef-dev (18.0.0-4353-g1e3835ab 1e3835abb2d19ce6ac4149c260ef804f1041d751) When I try getting the same object onto the disk using rados command, the object (contains plain text) would still be readable. Has something changed since v17.2.4? I'll also test with Pacific and let you know. Not sure if it affects other SSE mechanisms as well. [1] https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_trust_forwa… [2] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo… Thanks, Jayanth Reddy

2 2

header_limit in AsioFrontend class
by Vahideh Alinouri 19 Jun '23

19 Jun '23

Dear Ceph Users, I am writing to request the backporting changes related to the AsioFrontend class and specifically regarding the header_limit value. In the Pacific release of Ceph, the header_limit value in the AsioFrontend class was set to 4096. From Quincy release, there has been a configurable option introduced to set the header_limit value and the default value is 16384. I would greatly appreciate it if someone from the Ceph development team backport this change to the older version. Best regards, Vahideh Alinouri

2 1

Re: same OSD in multiple CRUSH hierarchies
by Budai Laszlo 19 Jun '23

19 Jun '23

Hi, Actually I've learned that it's not needed for a rule to start with a root bucket, so I can heve rules that will only consider a subtree of my total resources, and achieve what I was trying to do with the different disjunct hierarchies. BTW: it is possible to have different trees with different roots, with some OSDs being part of multiple such trees, and create different rules that will start with one root or the other. But I was told that this could mess up the calculation for pg autoscaler and other housekeeping functions. So it seems a better option to have each OSD in one single tree, and use rules that will only consider subtrees ... Regards, Laszlo > Date: Mon, 19 Jun 2023 07:41:35 +0000 > From: Eugen Block<eblock(a)nde.ag> > Subject: [ceph-users] Re: same OSD in multiple CRUSH hierarchies > To:ceph-users@ceph.io > Message-ID: > <20230619074135.Horde.gS8nAKQgZhlbV0HpymJ-lqf(a)webmail.nde.ag> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > Hi, > I don't think this is going to work. Each OSD belongs to a specific > host and you can't have multiple buckets (e.g. bucket type "host") > with the same name in the crush tree. But if I understand your > requirement correctly, there should be no need to do it this way. If > you structure your crush tree according to your separation > requirements and the critical pools use designated rules, you can > still have a rule that doesn't care about the data separation but > distributes the replicas across the available hosts (given your > failure domain would be "host"), which is already the default for the > replicated_rule. Did I misunderstand something? > > Regards, > Eugen > > > Zitat von Budai Laszlo<laszlo.budai(a)gmail.com>: > >> Hi there, >> >> I'm curious if there is anything against configuring an ODS to be >> part in multiple CRUSH hierarchies. I'm thinking of the following >> scenario: >> >> I want to create pools that are using distinct sets of OSDs. I want >> to make sure that a piece data which replicated at application level >> will not end up on the same OSD. So I would creat multiple CRUSH >> hierarchies (root - host - osd) but using different OSDs in each, >> and different rules that are using those hierarchies. Then I would >> create pools with the different rules, and use those different pools >> for storing the data for the different application instances. But I >> would also like to use the OSDs in the "default hierarchy" set up by >> ceph where all the hosts are in the same root bucket, and have the >> default replicated rule, so my generic data volumes would be able to >> spread across all the OSDs available. >> >> Is there something against this setup? >> >> Thank you for any advice! >> Laszlo >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

EC 8+3 Pool PGs stuck in remapped+incomplete
by Jayanth Reddy 19 Jun '23

19 Jun '23

Hello Users, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)* 5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and incomplete+remapped states. Below are the PGs, # ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] 151 [151,464,146,503,166,41,555,542,9,565,268] 151 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] 584 [584,281,672,699,199,224,239,430,355,504,196] 584 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] 396 [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] 214 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] 404 [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] 223 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] 297 [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] 548 Some of the OSDs had gone down on the cluster. Below is the # ceph status # ceph -s cluster: id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 1 pools have many more objects per pg than average Reduced data availability: 5 pgs inactive, 5 pgs incomplete Degraded data redundancy: 44798/8718528059 objects degraded (0.001%), 1 pg degraded, 1 pg undersized 22726 pgs not deep-scrubbed in time 23552 pgs not scrubbed in time 77 slow ops, oldest one blocked for 56400 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops. too many PGs per OSD (330 > max 250) services: mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs flags noscrub,nodeep-scrub rgw: 2 daemons active (brc1rgw1, brc1rgw2) data: pools: 17 pools, 23552 pgs objects: 863.74M objects, 1.2 PiB usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail pgs: 0.021% pgs not active 44798/8718528059 objects degraded (0.001%) 23546 active+clean 3 remapped+incomplete 2 incomplete 1 active+undersized+degraded io: client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr And the health detail shows as # ceph health detail HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more objects per pg than average; Reduced data availability: 5 pgs inactive, 5 pgs incomplete; Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too many PGs per OSD (330 > max 250) OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool iscsi-images objects per pg (540004) is more than 14.7248 times cluster average (36673) PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs incomplete pg 15.3f3 is incomplete, acting [584,281,672,699,199,224,239,430,355,504,196] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.985 is remapped+incomplete, acting [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.d46 is remapped+incomplete, acting [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.251e is incomplete, acting [151,464,146,503,166,41,555,542,9,565,268] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') pg 15.39d3 is remapped+incomplete, acting [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] (reducing pool default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs for 'incomplete') PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded (0.001%), 1 pg degraded, 1 pg undersized pg 15.28f0 is stuck undersized for 67359238.592403, current state active+undersized+degraded, last acting [2147483647,343,355,415,426,640,302,392,78,202,607] PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time We've the pools as below # ceph osd lspools 1 iscsi-images 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.control 6 default.rgw.meta 7 default.rgw.log 8 default.rgw.buckets.index 13 rbd 15 default.rgw.buckets.data 16 default.rgw.buckets.non-ec 19 cephfs_data-ec 22 rbd-ec 23 iscsi-images-ec 24 hpecpool 25 hpec.rgw.buckets.index 26 hpec.rgw.buckets.non-ec We've been struggling for a long time to fix this but out of luck! Our RGW daemons hosted on dedicated machines are continuously failing to respond, being behind a load balancer, LB throws 504 Gateway Timeout as the daemons are failing to respond in the expected time. We perform active health checks from the LB on '/' by HTTP HEAD but these are failing as well, very frequently. Currently we're surviving by writing a script that restarts RGW daemons whenever the LB responds with HTTP status code 504. Any help is highly appreciated! Regards, Jayanth Reddy

3 7

same OSD in multiple CRUSH hierarchies
by Budai Laszlo 19 Jun '23

19 Jun '23

Hi there, I'm curious if there is anything against configuring an ODS to be part in multiple CRUSH hierarchies. I'm thinking of the following scenario: I want to create pools that are using distinct sets of OSDs. I want to make sure that a piece data which replicated at application level will not end up on the same OSD. So I would creat multiple CRUSH hierarchies (root - host - osd) but using different OSDs in each, and different rules that are using those hierarchies. Then I would create pools with the different rules, and use those different pools for storing the data for the different application instances. But I would also like to use the OSDs in the "default hierarchy" set up by ceph where all the hosts are in the same root bucket, and have the default replicated rule, so my generic data volumes would be able to spread across all the OSDs available. Is there something against this setup? Thank you for any advice! Laszlo

2 1