- ceph-users - lists.ceph.io

radosgw-admin sync error trim seems to do nothing
by Richard Bade 03 Oct '23

03 Oct '23

Hi Matthew, At least for nautilus (14.2.22) i have discovered through trial and error that you need to specify a beginning or end date. Something like this: radosgw-admin sync error trim --end-date="2023-08-20 23:00:00" --rgw-zone={your_zone_name} I specify the zone as there's a error list for each zone. Hopefully that helps. Rich ------------------------------ Date: Sat, 19 Aug 2023 12:48:55 -0400 From: Matthew Darwin <bugs(a)mdarwin.ca> Subject: [ceph-users] radosgw-admin sync error trim seems to do nothing To: Ceph Users <ceph-users(a)ceph.io> Message-ID: <95e7edfd-ca29-fc0e-a30a-987f1c43e2d4(a)mdarwin.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Hello all, "radosgw-admin sync error list" returns errors from 2022. I want to clear those out. I tried "radosgw-admin sync error trim" but it seems to do nothing. The man page seems to offer no suggestions https://protect-au.mimecast.com/s/26o0CzvkGRhLoOXfXjZR3?domain=docs.ceph.com Any ideas what I need to do to remove old errors? (or at least I want to see more recent errors) ceph version 17.2.6 (quincy) Thanks.

2 2

ceph osd down doesn't seem to work
by Simon Oosthoek 03 Oct '23

03 Oct '23

Hi I'm trying to mark one OSD as down, so we can clean it out and replace it. It keeps getting medium read errors, so it's bound to fail sooner rather than later. When I command ceph from the mon to mark the osd down, it doesn't actually do it. When the service on the osd stops, it is also marked out and I'm thinking (but perhaps incorrectly?) that it would be good to keep the OSD down+in, to try to read from it as long as possible. Why doesn't it get marked down and stay that way when I command it? Context: Our cluster is in a bit of a less optimal state (see below), this is after one of OSD nodes had failed and took a week to get back up (long story). Due to a seriously unbalanced filling of our OSDs we kept having to reweight OSDs to keep below the 85% threshold. Several disks are starting to fail now (they're 4+ years old and failures are expected to occur more frequently). I'm open to suggestions to help get us back to health_ok more quickly, but I think we'll get there eventually anyway... Cheers /Simon ---- # ceph -s cluster: health: HEALTH_ERR 1 clients failing to respond to cache pressure 1/843763422 objects unfound (0.000%) noout flag(s) set 14 scrub errors Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent Degraded data redundancy: 13795525/7095598195 objects degraded (0.194%), 13 pgs degraded, 12 pgs undersized 70 pgs not deep-scrubbed in time 65 pgs not scrubbed in time services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 11h) mgr: cephmon3(active, since 35h), standbys: cephmon1 mds: 1/1 daemons up, 1 standby osd: 264 osds: 264 up (since 2m), 264 in (since 75m); 227 remapped pgs flags noout rgw: 8 daemons active (4 hosts, 1 zones) data: volumes: 1/1 healthy pools: 15 pools, 3681 pgs objects: 843.76M objects, 1.2 PiB usage: 2.0 PiB used, 847 TiB / 2.8 PiB avail pgs: 13795525/7095598195 objects degraded (0.194%) 54839263/7095598195 objects misplaced (0.773%) 1/843763422 objects unfound (0.000%) 3374 active+clean 195 active+remapped+backfill_wait 65 active+clean+scrubbing+deep 20 active+remapped+backfilling 11 active+clean+snaptrim 10 active+undersized+degraded+remapped+backfill_wait 2 active+undersized+degraded+remapped+backfilling 2 active+clean+scrubbing 1 active+recovery_unfound+degraded 1 active+clean+inconsistent progress: Global Recovery Event (8h) [==========================..] (remaining: 2h)

3 3

Impacts on doubling the size of pgs in a rbd pool?
by Hervé Ballans 03 Oct '23

03 Oct '23

Hi all, I have a Ceph cluster on Quincy (17.2.6), with 3 pools (1 rbd and 1 CephFS volume), each configured with 3 replicas. $ sudo ceph osd pool ls detail pool 7 'cephfs_data_home' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 6287147 lfor 0/5364613/5364611 flags hashpspool stripe_width 0 application cephfs pool 8 'cephfs_metadata_home' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 6333341 lfor 0/6333341/6333339 flags hashpspool stripe_width 0 application cephfs pool 9 'rbd_backup_vms' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 6365131 lfor 0/211948/249421 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 10 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode warn last_change 6365131 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth $ sudo ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 306 TiB 186 TiB 119 TiB 119 TiB 39.00 nvme 4.4 TiB 4.3 TiB 118 GiB 118 GiB 2.63 TOTAL 310 TiB 191 TiB 119 TiB 119 TiB 38.49 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data_home 7 512 12 TiB 28.86M 12 TiB 12.85 27 TiB cephfs_metadata_home 8 32 33 GiB 3.63M 33 GiB 0.79 1.3 TiB rbd_backup_vms 9 1024 24 TiB 6.42M 24 TiB 58.65 5.6 TiB .mgr 10 1 35 MiB 9 35 MiB 0 12 TiB I am going to extend the rbd pool (rbd_backup_vms), currently used at 60%. This pool contains 60 disks, i.e. 20 disks by rack in the crushmap. This pool is used for storing VM disk images (available to a separate ProxmoxVE cluster) For this purpose, I am going to add 42 disks of the same size as those currently in the pool, i.e. 14 additional disks on each rack. Currently, this pool is configured with 1024 pgs. Before this operation, I would like to extend the number of pgs, let's say 2048 (i.e. double). I wonder about the overall impact of this change on the cluster. I guess that the heavy moves in the pgs will have a strong impact regarding the iops? I have two questions: 1) Is it useful to make this modification before adding the new OSDs? (I'm afraid of warnings about full or nearfull pgs if not) 2) are there any configuration recommendations in order to minimize these anticipated impacts? Thank you! Cheers, Hervé

3 3

S3 user with more than 1000 buckets
by Thomas Bennett 03 Oct '23

03 Oct '23

Hi, I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than 1000 buckets. When the client tries to list all their buckets using s3cmd, rclone and python boto3, they all three only ever return the first 1000 bucket names. I can confirm the buckets are all there (and more than 1000) by checking with the radosgw-admin command. Have I missed a pagination limit for listing user buckets in the rados gateway? Thanks, Tom

5 6

ingress of haproxy is down after I specify the haproxy.cfg in quincy
by wjsherry075＠outlook.com 03 Oct '23

03 Oct '23

Hello, I have a haproxy problem in ceph quincy 17.2.6. Ununtu 22.04 The haproxy image can't be up after I specify the haproxy.cfg, and there is no error in the logs. I set the haproxy.cfg: ceph config-key set mgr/cephadm/services/ingress/haproxy.cfg -i haproxy.cfg If I remove the haproxy, and let cephadm to generate automatically, it works. I tried to create a file same as the cephadm generated, but haproxy were still down. NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID haproxy.rgw.foo.test01.pfiiix test01 *:80,9101 error 2m ago 6h - - <unknown> <unknown> <unknown> haproxy.rgw.foo.test02.fsnhnb test02 *:80,9101 error 2m ago 6h - - <unknown> <unknown> <unknown> Any advice on what to do? Thanks, Jie

1 0

Ceph Quarterly (CQ) - Issue #2
by Zac Dover 03 Oct '23

03 Oct '23

The second issue of "Ceph Quarterly" is attached to this email. Ceph Quarterly (or "CQ") is an overview of the past three months of upstream Ceph development. We provide CQ in three formats: A4, letter, and plain text wrapped at 80 columns. Two news items arrived after the deadline for typesetting this issue. They are included here: Grace Hopper Open Source Day 2023: - On 22 Sep 2023, Ceph participated in Grace Hopper Open Source Day, an all-day hackathon for women and nonbinary developers. Laura Flores led the Ceph division, and Yaarit Hatuka, Shreyansh Sancheti,and Aishwarya Mathuria participated as mentors. From 12pm EST to 7:30pm EST, Laura showed more than 40 attendees how to run a Ceph vstart cluster in an Ubuntu Docker container. Yaarit, Shreyansh,and Aishwarya spent the day working one-on-one with attendees, helping them troubleshoot and work through a curated list of low-hanging-fruit issues. By the day's end, Grace Hopper attendees submitted eight pull requests. As of the publication of this sentence, two have been merged and the others are expected to be merged soon. - For more information about GHC Open Source Day, see https://ghc.anitab.org/awards-programs/open-source-day/ Ceph partners with RCOS: - Ceph has partnered for the first time with the Rensselaer Center for Open Source (RCOS), an organization at Rensselaer Polytechnic Institute that helps students jumpstart their careers in software by giving them the opportunity to work on various open source projects for class credit. - Laura Flores, representing Ceph, is mentoring three RPI students on a project to improve the output of the `ceph balancer status` command. - For more information about RCOS, see https://rcos.io/ Zac Dover Upstream DocumentationCeph Foundation

1 0

Performance drop and retransmits with CephFS
by Tom Wezepoel 03 Oct '23

03 Oct '23

Hi all, Have a question regarding CephFS and write performance. Possibly I am overlooking a setting. We recently started using Ceph, where we want to use CephFS as a shared storage system for a Sync-and-Share solution. Now we are still in a testing phase, where we are also mainly looking at the performance of the system, where we are seeing some strange issues. We are using Ceph Quincy release 17.2.6, with a replica 3 data policy across 21 hosts spread across 3 locations. When I write multiple files of 1G, the writing performance drops from 400MiB/s to 18 MiB/s with also multiple retries. However, when I empty the page caches every minute on the client, the performance remains good. But that's not really a solution of course. Have already played a lot with the sysctl settings, like vm.dirty etc, but it makes no difference at all. When I enable the fuse_disable_pagecache, the write performance does stay reasonable at 70MiB/s, but the read performance completely collapses from 600 MiB/s to 40 MiB/s There is no difference in behavior between the kernel or fuse client. Have already played around with client_oc_max_dirty, client_oc_max_objects, client_oc_size , etc. But haven't found the right setting. Anyone familiar with this who can give me some hints? Thanks for your help! :-) Kind regards, Tom

1 0

MDS failing to respond to capability release while `ls -lR`
by E Taka 01 Oct '23

01 Oct '23

Dockerized Ceph 17.2.6 on Ununtu 22.04 The Cephfs filesystem has a size of 180TB, used are only 66TB. When running a `ls -lR` the output stops and all accesses to the directory stall. ceph health says: # ceph health detail HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.vol.ppc721.mvxstq(mds.0): Client dessert failing to respond to capability release client_id: 6899709 [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.vol.ppc721.mvxstq(mds.0): 1 slow requests are blocked > 30 secs ceph -w shows: [WRN] slow request 31.421408 seconds old, received at 2023-10-01T09:53:44.634849+0000: client _request(client.7360117:2224947 getattr AsLsFs #0x4000012503f 2023-10-01T09:53:44.631148+0000 caller_uid=0, caller_gid=0{0,}) currently failed to rdlock, waiting [WRN] client.6899709 isn't responding to mclientcaps(revoke), ino 0x4000012503f pending pAsLs XsFsc issued pAsLsXsFscb, sent 61.422148 seconds ago The full output of ceph daemon [mds] dump inode 0x4000012503f, config show, dump_ops_in_flight and "ceph -w" with timestamps can be found on https://gist.github.com/test-erik/5de4a7bd632f62ab58c3115cfb876ae0 Do you have an idea what we can do about this?

1 0

SSD SATA performance
by Murilo Morais 30 Sep '23

30 Sep '23

Good morning everybody! Guys, I have 9x Kingston DC600M/1920 SSDs (SATA) in 3x DL380e, using the P420 (I still don't have an HBA to perform the exchange) in RAID 0. The device's specifications indicate that it achieves 94k/78k RAND-RW IOPS at 4K. I'm using it exclusively for VMs with RBD (I'm using OpenStack). (pool size 3) Performing sequential testing directly on the device I can easily beat these rates, but with random tests I reach fixed rates of 15k, and I think the device just doesn't deliver that. On VMs I get rates of 20k randwrite. What I've already tried: Disable Controller Cache Enable HP Smart Path (it ended up worsening SSD performance) Change the scheduler to "none" I haven't tried putting the controller in HBA mode yet, as the boot disks are on the P420, I still have to align this issue to perform the test. I would like to know if I can improve these rates or if it is just that. Thanks in advance!

1 0

Re: VM hangs when overwriting a file on erasure coded RBD
by peter.linder＠fiberdirekt.se 29 Sep '23

29 Sep '23

(sorry for duplicate emails) This turns out to be a good question actually. The cluster is running Quincy, 17.2.6. The compute node that is running the VM is proxmox, version 7.4-3. Supposedly this is fairly new, but the version of librbd1 claims to be 14.2.21 when I check with "apt list". We are not using proxmox's own ceph cluster release. However we haven't had any issues with this setup before, but we haven't been using neither erasure coded pools nor had the node-half-dead problem for such a long time. The VM is configured using proxmox which is not libvirt but similar, and krbd is not enabled. I don't know for sure if proxmox has its own librbd linked in qemu/kvm. "ceph features" looks like this: { "mon": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 5 } ], "osd": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 24 } ], "client": [ { "features": "0x3f01cfb87fecffff", "release": "luminous", "num": 4 }, { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 12 } ], "mgr": [ { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 2 } ] } Regards, Peter Den 2023-09-29 kl. 17:55, skrev Anthony D'Atri: > Which Ceph releases are installed on the VM and the back end? Is the VM using librbd through libvirt, or krbd? > >> On Sep 29, 2023, at 09:09, Peter Linder <peter.linder(a)fiberdirekt.se> wrote: >> >> Dear all, >> >> I have a problem that after an OSD host lost connection to the sync/cluster rear network for many hours (the public network was online), a test VM using RBD cant overwrite its files. I can create a new file inside it just fine, but not overwrite it, the process just hangs. >> >> The VM's disk is on an erasure coded data pool and a replicated pool in front of it. EC overwrites is on for the pool. >> >> The custer consists of 5 hosts and 4 OSDs on each, and separate hosts for compute. There is a public and separate cluster network, separated. In this case, the AOC cable to the cluster network went link down on a host and it had to be replaced and the host was rebooted. Recovery took about a week to complete. The host was half-down for about 12 hours like this. >> >> I have some other VMs as well with images in the same pool (totally 4), and they seem to work fine, it is just this one that cant overwrite. >> >> I'm thinking there is somehow something wrong with just this image? >> >> Regards, >> >> Peter >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0