- ceph-users - lists.ceph.io

autocaling not work and active+remapped+backfilling
by farhad kh 19 Jun '23

19 Jun '23

hi i have a problem with ceph 17.2.6 , cephfs with mds daemons but see an unusual behavior. create a data pool with default crush rule but data just store in 3 specific osd and other osd is clean PG auto-scaling is also active but its size does not change when the pool is biger I did this manually but the problem was not solved and I got the error pg are not balanced across osds How do I solve this problem? Is this a bug? I did not have this problem in previous versions I solved this problem. There are several identical crash rules in the folder step chooseleaf firstn 0 type host I think this confuses the balancer and autoscale and output for ceph osd pool autoscale-status is empty after remove other crush rules autoscale runing but move data from osd full to clear osd is slow trying with Reducing the weight of filled OSDs, I tried to prioritize the use of other OSDs ceph osd reweight-by-utilization I hope this works Is there a solution that makes the process of autoscaling and cleaning placement groups faster? ------------------------------- [root@opcsdfpsbpp0201 ~]# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "r3-host", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "r3", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ------------------------------------ # ceph osd status | grep back 23 opcsdfpsbpp0211 1900G 147G 0 0 0 0 backfillfull,exists,up 48 opcsdfpsbpp0201 1900G 147G 0 0 0 0 backfillfull,exists,up 61 opcsdfpsbpp0205 1900G 147G 0 0 0 0 backfillfull,exists,up ---------------------------------------------- Every 2.0s: ceph -s opcsdfpsbpp0201: Sun Jun 18 11:44:29 2023 cluster: id: 79a2627c-0821-11ee-a494-00505695c58c health: HEALTH_WARN 3 backfillfull osd(s) 6 pool(s) backfillfull services: mon: 3 daemons, quorum opcsdfpsbpp0201,opcsdfpsbpp0205,opcsdfpsbpp0203 (age 6d) mgr: opcsdfpsbpp0201.vttwxa(active, since 5d), standbys: opcsdfpsbpp0205.tpodbs, opcsdfpsbpp0203.jwjkcl mds: 1/1 daemons up, 2 standby osd: 74 osds: 74 up (since 7d), 74 in (since 7d); 107 remapped pgs data: volumes: 1/1 healthy pools: 6 pools, 359 pgs objects: 599.64k objects, 2.2 TiB usage: 8.1 TiB used, 140 TiB / 148 TiB avail pgs: 923085/1798926 objects misplaced (51.313%) 252 active+clean 87 active+remapped+backfill_wait 20 active+remapped+backfilling io: client: 255 B/s rd, 0 op/s rd, 0 op/s wr recovery: 33 MiB/s, 8 objects/s progress: Global Recovery Event (5h) [===================.........] (remaining: 2h)

1 0

cephfs mount with kernel driver
by farhad kh 19 Jun '23

19 Jun '23

I noticed that in my scenario, when I mount cephfs via the kernel module, it directly copies to one or three of the OSDs. And the writing speed of the client is higher than the speed of replication and auto scaling This causes the writing operation to stop as soon as those OSDs are filled, and the error of free space is not available What should be done to solve this problem? Is there a way to increase the speed of scaling or moving objects in OSD? Or a way to mount cephfs that does not have these problems?

1 0

Re: EC 8+3 Pool PGs stuck in remapped+incomplete
by Jayanth Reddy 17 Jun '23

17 Jun '23

Hello Anthony / Users, After some initial analysis, I had increased max_pg_per_osd to 480, but we're out of luck. Also tried force-backfill and force-repair as well. On querying PG using *# ceph pg **<pg.ID> query* the output says blocked_by 3 to 4 OSDs which are out of the cluster already. Guessing if these have to do something with the recovery. Thanks, Jayanth Reddy On Sat, Jun 17, 2023 at 4:17 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Your cluster’s configuration is preventing CRUSH from calculating full > placements > > set max_pg_per_osd = 1000, either in central config (or ceph.conf if you > have it set there now). > > If you have it set in ceph.conf, you may need to serially restart the mons. > > ceph osd down 214 > sleep 60 > ceph osd down 223 > sleep 60 > ceph osd down 548 > sleep 60 > ceph osd down 584 > > > > > > > > On Jun 17, 2023, at 2:22 AM, Jayanth Reddy <jayanthreddy5666(a)gmail.com> > wrote: > > > > Hello Users, > > Greetings. We've a Ceph Cluster with the version > > *ceph version 14.2.5-382-g8881d33957 > > (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable)* > > > > 5 PGs belonging to our RGW 8+3 EC Pool are stuck in incomplete and > > incomplete+remapped states. Below are the PGs, > > > > # ceph pg dump_stuck inactive > > ok > > PG_STAT STATE UP > > UP_PRIMARY ACTING > > ACTING_PRIMARY > > 15.251e incomplete [151,464,146,503,166,41,555,542,9,565,268] > > 151 > > [151,464,146,503,166,41,555,542,9,565,268] 151 > > 15.3f3 incomplete [584,281,672,699,199,224,239,430,355,504,196] > > 584 > > [584,281,672,699,199,224,239,430,355,504,196] 584 > > 15.985 remapped+incomplete [396,690,493,214,319,209,546,91,599,237,352] > > 396 > > > [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] > > 214 > > 15.39d3 remapped+incomplete [404,221,223,585,38,102,533,471,568,451,195] > > 404 > > [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] > > 223 > > 15.d46 remapped+incomplete [297,646,212,254,110,169,500,372,623,470,678] > > 297 > > [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] > > 548 > > > > Some of the OSDs had gone down on the cluster. Below is the # ceph status > > > > # ceph -s > > cluster: > > id: 30d6f7ee-fa02-4ab3-8a09-9321c8002794 > > health: HEALTH_WARN > > noscrub,nodeep-scrub flag(s) set > > 1 pools have many more objects per pg than average > > Reduced data availability: 5 pgs inactive, 5 pgs incomplete > > Degraded data redundancy: 44798/8718528059 objects degraded > > (0.001%), 1 pg degraded, 1 pg undersized > > 22726 pgs not deep-scrubbed in time > > 23552 pgs not scrubbed in time > > 77 slow ops, oldest one blocked for 56400 sec, daemons > > [osd.214,osd.223,osd.548,osd.584] have slow ops. > > too many PGs per OSD (330 > max 250) > > > > services: > > mon: 3 daemons, quorum brc1mon2,brc1mon3,brc1mon1 (age 2y) > > mgr: brc1mon2(active, since 8d), standbys: brc1mon1, brc1mon3 > > mds: cephfs:1 {0=brc1mds2=up:active} 1 up:standby > > osd: 1012 osds: 698 up (since 14h), 698 in (since 2d); 3 remapped pgs > > flags noscrub,nodeep-scrub > > rgw: 2 daemons active (brc1rgw1, brc1rgw2) > > > > data: > > pools: 17 pools, 23552 pgs > > objects: 863.74M objects, 1.2 PiB > > usage: 2.4 PiB used, 6.2 PiB / 8.6 PiB avail > > pgs: 0.021% pgs not active > > 44798/8718528059 objects degraded (0.001%) > > 23546 active+clean > > 3 remapped+incomplete > > 2 incomplete > > 1 active+undersized+degraded > > > > io: > > client: 24 MiB/s rd, 3.2 KiB/s wr, 56 op/s rd, 4 op/s wr > > > > And the health detail shows as > > > > # ceph health detail > > HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pools have many more > > objects per pg than average; Reduced data availability: 5 pgs inactive, 5 > > pgs incomplete; Degraded data redundancy: 44798/8718528081 objects > degraded > > (0.001%), 1 pg degraded, 1 pg undersized; 22726 pgs not deep-scrubbed in > > time; 23552 pgs not scrubbed in time; 77 slow ops, oldest one blocked for > > 56440 sec, daemons [osd.214,osd.223,osd.548,osd.584] have slow ops.; too > > many PGs per OSD (330 > max 250) > > OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set > > MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average > > pool iscsi-images objects per pg (540004) is more than 14.7248 times > > cluster average (36673) > > PG_AVAILABILITY Reduced data availability: 5 pgs inactive, 5 pgs > incomplete > > pg 15.3f3 is incomplete, acting > > [584,281,672,699,199,224,239,430,355,504,196] (reducing pool > > default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs > for > > 'incomplete') > > pg 15.985 is remapped+incomplete, acting > > > [2147483647,2147483647,2147483647,214,319,2147483647,546,91,599,2147483647,352] > > (reducing pool default.rgw.buckets.data min_size from 9 may help; search > > ceph.com/docs for 'incomplete') > > pg 15.d46 is remapped+incomplete, acting > > [2147483647,548,2147483647,2147483647,110,169,500,372,2147483647,470,678] > > (reducing pool default.rgw.buckets.data min_size from 9 may help; search > > ceph.com/docs for 'incomplete') > > pg 15.251e is incomplete, acting > > [151,464,146,503,166,41,555,542,9,565,268] (reducing pool > > default.rgw.buckets.data min_size from 9 may help; search ceph.com/docs > for > > 'incomplete') > > pg 15.39d3 is remapped+incomplete, acting > > [2147483647,2147483647,223,585,38,102,533,2147483647,231,451,2147483647] > > (reducing pool default.rgw.buckets.data min_size from 9 may help; search > > ceph.com/docs for 'incomplete') > > PG_DEGRADED Degraded data redundancy: 44798/8718528081 objects degraded > > (0.001%), 1 pg degraded, 1 pg undersized > > pg 15.28f0 is stuck undersized for 67359238.592403, current state > > active+undersized+degraded, last acting > > [2147483647,343,355,415,426,640,302,392,78,202,607] > > PG_NOT_DEEP_SCRUBBED 22726 pgs not deep-scrubbed in time > > > > We've the pools as below > > > > # ceph osd lspools > > 1 iscsi-images > > 2 cephfs_data > > 3 cephfs_metadata > > 4 .rgw.root > > 5 default.rgw.control > > 6 default.rgw.meta > > 7 default.rgw.log > > 8 default.rgw.buckets.index > > 13 rbd > > 15 default.rgw.buckets.data > > 16 default.rgw.buckets.non-ec > > 19 cephfs_data-ec > > 22 rbd-ec > > 23 iscsi-images-ec > > 24 hpecpool > > 25 hpec.rgw.buckets.index > > 26 hpec.rgw.buckets.non-ec > > > > > > We've been struggling for a long time to fix this but out of luck! Our > RGW > > daemons hosted on dedicated machines are continuously failing to respond, > > being behind a load balancer, LB throws 504 Gateway Timeout as the > daemons > > are failing to respond in the expected time. We perform active health > > checks from the LB on '/' by HTTP HEAD but these are failing as well, > very > > frequently. Currently we're surviving by writing a script that restarts > RGW > > daemons whenever the LB responds with HTTP status code 504. Any help is > > highly appreciated! > > > > Regards, > > Jayanth Reddy > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

OSD stuck down
by Nicola Mori 16 Jun '23

16 Jun '23

Dear Ceph users, after a host reboot one of the OSDs is now stuck down (and out). I tried several times to restart it and even to reboot the host, but it still remains down. # ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 4 OSD(s) have spurious read errors (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 16h) mgr: bofur.tklnrn(active, since 16h), standbys: aka.wzystq, balin.hvunfe mds: 2/2 daemons up, 1 standby osd: 104 osds: 103 up (since 16h), 103 in (since 13h); 4 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.85M objects, 41 TiB usage: 56 TiB used, 139 TiB / 195 TiB avail pgs: 68130/150150628 objects misplaced (0.045%) 522 active+clean 4 active+remapped+backfilling 3 active+clean+scrubbing+deep io: recovery: 46 MiB/s, 21 objects/s The host is reachable (its other OSDs are in) and from the systemd logs of the OSD I don't see anything wrong: $ sudo systemctl status ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.34 ● ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.34.service - Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 Loaded: loaded (/etc/systemd/system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45@.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2023-06-12 17:00:25 CEST; 15h ago Main PID: 36286 (bash) Tasks: 11 (limit: 152154) Memory: 20.0M CGroup: /system.slice/system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.34.service ├─36286 /bin/bash /var/lib/ceph/b1029256-7bb3-11ec-a8ce-ac1f6b627b45/osd.34/unit.run └─36657 /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --init --name ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45-osd-34 --pids-limit=0 -e CONTAINER_IMAGE=snack14/ceph-wizard@sha> Jun 12 17:00:25 balin systemd[1]: Started Ceph osd.34 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45. Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-34 --no-mon-config --dev /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/ln -s /dev/mapper/ceph--9a4c3927--d3da--4b49--80fe--6cdc00c7897c-osd--block--36d2f793--e5c7--4247--a314--bcc40389d50d /var/lib/ceph/osd/ceph-34/block Jun 12 17:00:27 balin bash[36306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-34 Jun 12 17:00:27 balin bash[36306]: --> ceph-volume raw activate successful for osd ID: 34 Jun 12 17:00:29 balin bash[36657]: debug 2023-06-12T15:00:29.066+0000 7f818e356540 -1 Falling back to public interface I'd need some help to understand how to fix this. Thank you, Nicola

5 9

Improving write performance on ceph 17.6.2 HDDs + DB/WAL storage on nvme
by alexey.blinkov＠gmail.com 16 Jun '23

16 Jun '23

Greetings. My cluster consists of 3 nodes. Each node has 4 OSD HDDs with a capacity of 6 TB each and 1 nvme for db/wal storage. 2 * 10Gbps network assembled in bond and some parameters changed in rc.local to improve performance. They are below: #Set network interface buffer size ethtool -G eno1 rx 4096 tx 4096; ethtool -G eno2 rx 4096 tx 4096 #Set txqueuelen eno1, eno2, bond0, vmbr0 ip link set eno1 txqueuelen 10000 ip link set eno2 txqueuelen 10000 ip link set bond0 txqueuelen 20000 ip link set vmbr0 txqueuelen 20000 ip link set vmbr0.4040 txqueuelen 10000 ip link set vmbr0.4043 txqueuelen 10000 ip link set vmbr0.4045 txqueuelen 10000 ip link set vmbr0.4053 txqueuelen 10000 #Disable Offload ethtool -K eno1 gso off gro off lro off tso off ethtool -K eno2 gso off gro off lro off tso off ethtool -K eno1 rx on tx on sg on ethtool -K eno2 rx on tx on sg on ethtool -L eno1 combined 8 ethtool -L eno2 combined 8 ethtool -K eno1 rxhash on ethtool -K eno2 rxhash on rss-ladder eno1 0 rss-ladder eno2 1 With the default settings of ceph.conf, when testing a 1 GB file with the SEQ1MQ8T1 profile inside the kvm virtual machine, I get the following write throughput: 152 MB/s The bandwidth of each of the disks, if you do not use CEPH, is at least 200 MB/s, and I have 4 of them per node. In total, this is 800 MB/s. If we discard the overhead and abstraction levels, a loss of 25% of performance is possible, this should be +- 600 MB/s I may be wrong, but I'm purely theoretically guessing. When testing and monitoring disk load with the nmon utility, I see write rates of 30 to 50 MB/s per node disk. The network cannot be a bottleneck, since the nodes are empty and there is only 1 virtual machine on them that generates the load. iperf3 from one node to another passes at least 8 Gbit/s, which in the test will completely allow to drive 4.8 Gbit/s equal to 600 MB/s Write caching in the virtual machine is disabled. I started changing the [osd] parameters in the ceph.conf file with these changes, I managed to increase the read speed by 2 times from 195 MB/s to 550 MB/s, but the write speed did not budge. The conversation now is not about IOPS, but about the write speed. What needs to be changed? Below is my ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.50.251.0/24 fsid = 7d00b675-2f1e-47ff-a71c-b95d1745bc39 mon_allow_pool_delete = true mon_host = 10.50.250.1 10.50.250.2 10.50.250.3 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.50.250.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring #[osd] # bluestore_compression_mode = none # bluestore_cache_autotune = true # bluestore_cache_size = 3221225472 # bluestore_cache_kv_ratio = 0.2 # bluestore_cache_kv_max = 1610612736 # bluestore_min_alloc_size_hdd = 65536 # bluestore_max_alloc_size_hdd = 67108864 # bluestore_min_alloc_size_ssd = 16384 # bluestore_max_alloc_size_ssd = 33554432 # bluestore_throttle_bytes = 268435456 # bluestore_throttle_deferred_bytes = 1073741824 # bluestore_rocksdb_options = write_buffer_size=268435456;max_write_buffer_number=4 # osd_op_num_threads_per_shard = 4 # osd_client_message_cap = 1024 # osd_client_message_size_cap = 536870912 [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.nd01] host = nd01 mds_standby_for_name = pve [mds.nd02] host = nd02 mds_standby_for_name = pve [mds.nd03] host = nd03 mds_standby_for_name = pve [mon.nd01] public_addr = 10.50.250.1 [mon.nd02] public_addr = 10.50.250.2 [mon.nd03] public_addr = 10.50.250.3 Perhaps I do not quite understand the philosophy, logic and functionality of CEPH? I am aware that there may be competition for resources and so on. But I see a clear 50-50 write throughput split when another VM runs a similar test. And at a speed of 150 MB / s, it is very scary that at some point there will be competition between virtual machines and the write speed may drop to very low

1 0

ceph blocklist
by Budai Laszlo 16 Jun '23

16 Jun '23

Hello everyone, can someone explain, or direct me to some documentation that explains the role of the blocklists (former blacklist). What do they useful for? how do they work? Thank you, Laszlo

1 0

How to change RGW certificate in Cephadm?
by Kai Stian Olstad 16 Jun '23

16 Jun '23

When I enabled RGW in cephadm I used this spec file rgw.yml service_type: rgw service_id: pech placement: label: cog spec: ssl: true rgw_frontend_ssl_certificate: | -----BEGIN CERTIFICATE----- <snip /> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <snip /> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <snip /> -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- <snip /> -----END RSA PRIVATE KEY----- And enabled it with ceph orch apply -i /etc/ceph/rgw.yml The certificate is about to expire so I would like to update it. I updated rgw.yml spec with the new certificate and run ceph orch apply -i /etc/ceph/rgw.yml But nothing happened, so I tried to redeploy one of them with ceph orch daemon redeploy rgw.pech.pech-mon-3.upnvrd It redeployed the RGW, but still uses the old certificate. ceph config-key list | grep rgw gives me two keys of interest mgr/cephadm/spec.rgw.pech and rgw/cert/rgw.pech The content of mgr/cephadm/spec.rgw.pech is the new spec file with the updated certificates, but the rgw/cert/rgw.pech only contains certificate and private key, but the certificate is the old ones about to expire. I have looked in the documentation and can't find how to update the certificate for RGW. Can anyone shed some light on how to replace the certificate? -- Kai Stian Olstad

2 5

RGW versioned bucket index issues
by Cory Snyder 16 Jun '23

16 Jun '23

Hi all, I wanted to call attention to some RGW issues that we've observed on a Pacific cluster over the past several weeks. The problems relate to versioned buckets and index entries that can be left behind after transactions complete abnormally. The scenario is multi-faceted and we're still investigating some of the details, but I wanted to provide a big-picture summary of what we've found so far. It looks like most of these issues should be reproducible on versions before and after Pacific as well. I'll enumerate the individual issues below: 1. PUT requests during reshard of versioned bucket fail with 404 and leave behind dark data Tracker: https://tracker.ceph.com/issues/61359 2. When bucket index ops are cancelled it can leave behind zombie index entries This one was merged a few months ago and did make the v16.2.13 release, but in our case we had billions of extra index entries by the time that we had upgraded to the patched version. Tracker: https://tracker.ceph.com/issues/58673 3. Issuing a delete for a key that already has a delete marker as the current version leaves behind index entries and OLH objects Note that the tracker's original description describes the problem a bit differently, but I've clarified the nature of the issue in a comment. Tracker: https://tracker.ceph.com/issues/59663 The extra index entries and OLH objects that are left behind due to these sorts of issues are obviously annoying in regards to the fact that they unnecessarily consume space, but we've found that they can also cause severe performance degradation for bucket listings, lifecycle processing, and other ops indirectly due to higher osd latencies. The reason for the performance impact is that bucket listing calls must repeatedly perform additional OSD ops until they find the requisite number of entries to return. The OSD cls method for bucket listing also does its own internal iteration for the same purpose. Since these entries are invalid, they are skipped. In the case that we observed, where some of our bucket indexes were filled with a sea of contiguous leftover entries, the process of continually iterating over and skipping invalid entries caused enormous read amplification. I believe that the following tracker is describing symptoms that are related to the same issue: https://tracker.ceph.com/issues/59164. Note that this can also cause LC processing to repeatedly fail in cases where there are enough contiguous invalid entries, since the OSD cls code eventually gives up and returns an error that isn't handled. The severity of these issues likely varies greatly based upon client behavior. If anyone has experienced similar problems, we'd love to hear about the nature of how they've manifested for you so that we can be more confident that we've plugged all of the holes. Thanks, Cory Snyder 11:11 Systems

3 5

degraded objects increasing
by Angelo Höngens 16 Jun '23

16 Jun '23

Hey guys, I'm trying to understand what is happening in my cluster, I see the number of degraded objects increasing, while all OSD's are still up and running. Can someone explain what's happening? I would expect the number of misplaced objects to increase when ceph's balancing algorithm decides blocks should be on different OSDs, but I would only expect the degraded objects to increase when OSD's die?? 6/15/23 11:31:25 AM[WRN]Health check update: Degraded data redundancy: 809035/219783315 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:20 AM[WRN]Health check update: Degraded data redundancy: 809044/219788544 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:15 AM[WRN]Health check update: Degraded data redundancy: 809044/219788616 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:10 AM[WRN]Health check update: Degraded data redundancy: 808944/219777540 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:05 AM[WRN]Health check update: Degraded data redundancy: 808944/219776271 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:31:00 AM[WRN]Health check update: Degraded data redundancy: 808821/219683475 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:30:55 AM[WRN]Health check update: Degraded data redundancy: 808740/219672240 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 6/15/23 11:30:50 AM[WRN]Health check update: Degraded data redundancy: 808667/219645417 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) health: health: HEALTH_WARN Degraded data redundancy: 810779/220602543 objects degraded (0.368%), 6 pgs degraded, 6 pgs undersized services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 12h) mgr: ceph-mon01.vzbglj(active, since 12h), standbys: ceph-mon02.qtuntk mds: 1/1 daemons up, 2 standby osd: 118 osds: 118 up (since 12h), 118 in (since 23h); 510 remapped pgs data: volumes: 1/1 healthy pools: 5 pools, 1249 pgs objects: 44.55M objects, 60 TiB usage: 118 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 810779/220602543 objects degraded (0.368%) 39586388/220602543 objects misplaced (17.945%) 739 active+clean 406 active+remapped+backfill_wait 98 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfilling io: client: 585 KiB/s rd, 22 MiB/s wr, 357 op/s rd, 1.41k op/s wr recovery: 150 MiB/s, 82 objects/s Kind regards, Angelo Hongens

3 2

Restful API and Cephfs quota usage
by Sake 16 Jun '23

16 Jun '23

1 1