- ceph-users - lists.ceph.io

Very slow backfilling/remapping of EC pool PGs
by Gauvain Pocentek 21 Mar '23

21 Mar '23

Hello all, We have an EC (4+2) pool for RGW data, with HDDs + SSDs for WAL/DB. This pool has 9 servers with each 12 disks of 16TBs. About 10 days ago we lost a server and we've removed its OSDs from the cluster. Ceph has started to remap and backfill as expected, but the process has been getting slower and slower. Today the recovery rate is around 12 MiB/s and 10 objects/s. All the remaining unclean PGs are backfilling: data: volumes: 1/1 healthy pools: 14 pools, 14497 pgs objects: 192.38M objects, 380 TiB usage: 764 TiB used, 1.3 PiB / 2.1 PiB avail pgs: 771559/1065561630 objects degraded (0.072%) 1215899/1065561630 objects misplaced (0.114%) 14428 active+clean 50 active+undersized+degraded+remapped+backfilling 18 active+remapped+backfilling 1 active+clean+scrubbing+deep We've checked the health of the remaining servers, and everything looks like (CPU/RAM/network/disks). Any hints on what could be happening? Thank you, Gauvain

2 1

Upgrade 16.2.10 --> 16.2.11 OSD "UPGRADE_REDEPLOY_DAEMON" failed
by Marco Pizzolo 20 Mar '23

20 Mar '23

Hello Everyone, We made the mistake of trying to patch to 16.2.11 from 16.2.10 which has been stable as we felt that 16.2.11 had been out for a while already. As luck would have it, we are having failure after failure with OSDs not upgrading successfully, and have 355 more OSDs to go. I'm pretty sure we're not alone on this and am wondering what others have done to address. Appreciate all suggestions. Thanks, Marco

1 1

Multiple instance_id and services for rbd-mirror daemon
by Aielli, Elia 20 Mar '23

20 Mar '23

Hi, I'm facing a strange issue and google doesn't seem to help me. I've a couple of clusters with Octopus v15.2.17, recently upgraded from 15.2.13 I had a rbd mirror service correctly working between the two clusters, i then updated, and after some days where all was ok, I've come to the situation where I have a single DAEMON, but multiple services (on different version tho), each one with its own instance_id, giving you some insight: rbd mirror pool status mypool --verbose health: WARNING daemon health: OK image health: WARNING images: 37 total 2 starting_replay 35 replaying DAEMONS service 123673101: instance_id: 127483700 client_id: admin hostname: DR-Host1 version: *15.2.13* leader: false health: OK service 123674106: instance_id: 127208040 client_id: admin hostname: DR-Host1 version: *15.2.13* leader: false health: OK service 123675375: instance_id: 124630539 client_id: admin hostname: DR-Host1 version: *15.2.13* leader: true health: OK service 124670331: instance_id: 127208013 client_id: backup hostname: DR-Host1 version: *15.2.17* leader: false health: OK As you can see one has the "leader" section as true, and the other are false, I'd like to delete the first 2 false ones, then delete the true one with 15.2.13 and hope the last one becomes true. Do any of you faced any similar issue or can help me in killing the wrong services? Let me know, thanks in advance! Elia

1 0

The release time of v16.2.12 is?
by Louis Koo 20 Mar '23

20 Mar '23

https://github.com/ceph/ceph/pull/47702?notification_referrer_id=NT_kwDOANW… This issue had been backported to pacific, and the release time of v16.2.12 is?

1 0

RBD latency
by Murilo Morais 17 Mar '23

17 Mar '23

Good evening everyone! Guys, what to expect latency for RBD images in a cluster with only HDD (36 HDDs)? Sometimes I see that the write latency is around 2-5 ms in some images even with very low IOPS and bandwidth while the read latency is around 0.2-0.7 ms. For a cluster with only HDD is this latency expected? Is there any parameter I can study to improve? What do you recommend for tuning in the OS? The latency between machines is always around 0.1ms, all connected via fiber optics. Thanks in advance!

3 2

radosgw SSE-C is not working (InvalidRequest)
by Boris Behrens 17 Mar '23

17 Mar '23

Hi, I try to evaluate SSE-C (so customer provides keys) for our object storages. We do not provide a KMS server. I've added "Access-Control-Allow-Headers" to the haproxy frontend. rspadd Access-Control-Allow-Headers... x-amz-server-side-encryption-customer-algorithm,\ x-amz-server-side-encryption-customer-key,\ x-amz-server-side-encryption-customer-key-MD5 I've also enabled "rgw_trust_forwarded_https = true" in the client section in the ceph.conf and restarted the RGW daemons. I now try to get it working, but I am not sure if I am doing it correctly. $ encKey=$(openssl rand -base64 32) $ md5Key=$(echo $encKey | md5sum | awk '{print $1}' | base64) $ aws s3api --endpoint=https://radosgw put-object \ --body ~/Downloads/TESTFILE \ --bucket test-bb-encryption \ --key TESTFILE \ --sse-customer-algorithm AES256 \ --sse-customer-key $encKey \ --sse-customer-key-md5 $md5Key This is what the RGW log gives me: 2023-03-17T10:55:55.465+0000 7f42bbe5f700 1 ====== starting new request req=0x7f448c185700 ===== 2023-03-17T10:55:55.469+0000 7f434df83700 1 ====== req done req=0x7f448c185700 op status=-2021 http_status=400 latency=3999985ns ====== 2023-03-17T10:55:55.469+0000 7f434df83700 1 beast: 0x7f448c185700: IPV6 - - [2023-03-17T10:55:55.469539+0000] "PUT /test-bb-encryption/TESTFILE HTTP/1.1" 400 221 - "aws-cli/2.4.18 Python/3.9.10 Darwin/22.3.0 source/x86_64 prompt/off command/s3api.put-object" - Maybe someone got a wroking example and is willing to share it with me, or did also encounter this problem and knows what to do? It's and octopus cluster. Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 1

tracker.ceph.com is slow
by Satoru Takeuchi 17 Mar '23

17 Mar '23

Hi, tracker.ceph.com seems to be quite slow recently. Since my colleagues feel so as well, this problem wouldn't be specific to me. Could you tel me if there is a plan to fix this problem near future? Thanks, Satoru

1 0

How to submit a bug report ?
by Patrick Vranckx 16 Mar '23

16 Mar '23

Hi, I suspect a bug in cephadm to configure ingress service for rgw. Our production server was upgraded from continuously from Luminous to Pacific. When configuring ingress service for rgw, the haproxy.cfg is incomplete. The same yaml file applied on our test cluster does the job. Regards, Patrick

2 1

Concerns about swap in ceph nodes
by sbryan Song 16 Mar '23

16 Mar '23

Hello, We have a 6-node ceph cluster, all of them have osd running and 3 of them (ceph-1 to ceph-3 )also has the ceph-mgr and ceph-mon. Here is the detailed configuration of each node (swap on ceph-1 to ceph-3 has been disabled after the alarm): # ceph-1 free -h total used free shared buff/cache available Mem: 187Gi 38Gi 5.4Gi 4.1Gi 143Gi 142Gi Swap: 0B 0B 0B # ceph-2 free -h total used free shared buff/cache available Mem: 187Gi 49Gi 2.6Gi 4.0Gi 135Gi 132Gi Swap: 0B 0B 0B # ceph-3 free -h total used free shared buff/cache available Mem: 187Gi 37Gi 4.6Gi 4.0Gi 145Gi 144Gi Swap: 0B 0B 0B # ceph-4 free -h total used free shared buff/cache available Mem: 251Gi 31Gi 8.3Gi 231Mi 211Gi 217Gi Swap: 124Gi 3.8Gi 121Gi # ceph-5 free -h total used free shared buff/cache available Mem: 251Gi 32Gi 14Gi 135Mi 204Gi 216Gi Swap: 124Gi 4.0Gi 121Gi # ceph-6 free -h total used free shared buff/cache available Mem: 251Gi 30Gi 16Gi 145Mi 204Gi 218Gi Swap: 124Gi 4.0Gi 121Gi We have configured swap space on all of them, for ceph-mgr nodes, we have 8G swap space and 128G swap configured for osd nodes, and our zabbix has monitored a swap over 50% usage for ceph-1 to ceph-3, but our available space are still around 140G against the total 187G. Just wondering whether the swap space is necessary when we have lots of memory available? Thanks very much for your answering.

3 3

Ceph NFS data - cannot read files, getattr returns NFS4ERR_PERM
by Wyll Ingersoll 16 Mar '23

16 Mar '23

ceph pacific 16.2.11 (cephadm managed) I have configured some NFS mounts from the ceph GUI from cephfs. We can mount the filesystems and view file/directory listings, but cannot read any file data. The permissions on the shares are RW. We mount from the client using "vers=4.1". Looking at debug logs from the container running nfs-ganesha, I see the following errors when trying to read a file's content: 15/03/2023 15:27:13 : epoch 6411e209 : gw01 : ganesha.nfsd-7[svc_8] complete_op :NFS4 :DEBUG :Status of OP_READ in position 2 = NFS4ERR_PERM, op response size is 7480 total response size is 7568 15/03/2023 15:27:13 : epoch 6411e209 : gw01 : ganesha.nfsd-7[svc_8] complete_nfs4_compound :NFS4 :DEBUG :End status = NFS4ERR_PERM lastindex = 3 Also, watching the TCP traffic, I see errors in the NFS protocol corresponding to these messages: 11:44:43.745570 IP xxx.747 > gw01.nfs: Flags [P.], seq 24184536:24184748, ack 11409577, win 602, options [nop,nop,TS val 342245425 ecr 2683489461], length 212: NFS request xid 156024373 208 getattr fh 0,1/53 11:44:43.745683 IP gw01.nfs > xxx.747: Flags [P.], seq 11409577:11409677, ack 24184748, win 3081, options [nop,nop,TS val 2683489461 ecr 342245425], length 100: NFS reply xid 156024373 reply ok 96 getattr ERROR: Operation not permitted So there appears to be a permissions problem where nfs-ganesha is not able to "getattr" on cephfs data. The export looks like this (read from rados): EXPORT { FSAL { name = "CEPH"; user_id = "nfs.cephfs.7"; filesystem = "cephfs"; secret_access_key = "xxx"; } export_id = 7; path = "/exports/nfs/foobar"; pseudo = "/foobar"; access_type = "RW"; squash = "no_root_squash"; attr_expiration_time = 0; security_label = false; protocols = 4; transports = "TCP"; } ceph auth permissions for the nfs.cephfs.7 client: [client.nfs.cephfs.7] key = xxx caps mds = "allow rw path=/exports/nfs/foobar" caps mon = "allow r" caps osd = "allow rw pool=.nfs namespace=cephfs, allow rw tag cephfs data=cephfs" Any suggestions?

2 8