February 2023 - ceph-users

Does cephfs subvolume have commands similar to `rbd perf` to query iops, bandwidth, and latency of rbd image?

by 郑亮

Hi all, Does cephfs subvolume have commands similar to rbd perf to query iops, bandwidth, and latency of rbd image? `ceph fs perf stats` shows metrics of the client side, not the metrics of the cephfs subvolume. What I want to get is the metrics at the subvolume level like below. [root@smd-exporter-5f87dcb946-wt7tl /]# rbd perf image iostat rbd: waiting for initial image stats NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT pool/csi-vol-5c2d22e3-9195-11ed-aab9-a6732a88c7dd 8/s 0/s 215 KiB/s 0 B/s 14.62 ms 0.00 ns pool/csi-vol-10e31971-9196-11ed-aab9-a6732a88c7dd 5/s 0/s 79 KiB/s 0 B/s 13.18 ms 0.00 ns pool/csi-vol-0f6e0dbe-9196-11ed-aab9-a6732a88c7dd 1/s 0/s 23 KiB/s 0 B/s 20.60 ms 0.00 ns pool/csi-vol-88704e7a-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 12 KiB/s 0 B/s 12.38 ms 0.00 ns pool/csi-vol-a87c21e9-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 14 KiB/s 0 B/s 13.21 ms 0.00 ns pool/csi-vol-b6d040a2-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 8 KiB/s 0 B/s 9.88 ms 0.00 ns pool/csi-vol-efb7c236-6fc9-11ed-aab9-a6732a88c7dd 0/s 0/s 3.2 KiB/s 0 B/s 2.96 ms 0.00 ns NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT pool/csi-vol-5c2d22e3-9195-11ed-aab9-a6732a88c7dd 12/s 0/s 165 KiB/s 0 B/s 26.07 ms 0.00 ns pool/csi-vol-10e31971-9196-11ed-aab9-a6732a88c7dd 4/s 0/s 91 KiB/s 0 B/s 9.83 ms 0.00 ns pool/csi-vol-b6d040a2-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 22 KiB/s 0 B/s 15.25 ms 0.00 ns pool/csi-vol-efb7c236-6fc9-11ed-aab9-a6732a88c7dd 1/s 0/s 38 KiB/s 0 B/s 20.40 ms 0.00 ns pool/csi-vol-88704e7a-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 9.6 KiB/s 0 B/s 12.96 ms 0.00 ns pool/csi-vol-a87c21e9-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 10 KiB/s 0 B/s 3.99 ms 0.00 ns pool/csi-vol-0f6e0dbe-9196-11ed-aab9-a6732a88c7dd 0/s 0/s 2.4 KiB/s 0 B/s 7.33 ms 0.00 ns Best Regards, Liang Zheng

1 year, 3 months

2
1
0 0

Any issues with podman 4.2 and Quincy?

by Vladimir Brik

Has anybody run into issues with Quincy and podman 4.2? 4x podman series are not mentioned in https://docs.ceph.com/en/quincy/cephadm/compatibility/ but podman 3x is no longer available in Alma Linux Vlad

1 year, 3 months

2
1
0 0

Health warning - POOL_TARGET_SIZE_BYTES_OVERCOMMITED

by Sake Paulusma

Hello, I configured a stretched cluster on two datacenters. It's working fine, except this weekend the Raw Capicity exceeded 50% and the error POOL_TARGET_SIZE_BYTES_OVERCOMMITED showed up. The command "ceph df" is showing the correct cluster size, but "ceph osd pool autoscale-status" is showing half of the total Raw Capacity. What could be wrong? ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph status cluster: id: adbe7bb6-5h6d-11ed-8511-004449ede0c health: HEALTH_WARN 1 MDSs report oversized cache 1 subtrees have overcommitted pool target_size_bytes services: mon: 5 daemons, quorum host1,host2,host3,host4,host5 (age 4w) mgr: aqsoel11445.nqamuz(active, since 5w), standbys: host1.wujgas mds: 2/2 daemons up, 2 standby osd: 12 osds: 12 up (since 5w), 12 in (since 9w) data: volumes: 2/2 healthy pools: 5 pools, 193 pgs objects: 17.31M objects, 1.2 TiB usage: 5.0 TiB used, 3.8 TiB / 8.8 TiB avail pgs: 192 active+clean 1 active+clean+scrubbing ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 8.8 TiB 3.8 TiB 5.0 TiB 5.0 TiB 56.83 TOTAL 8.8 TiB 3.8 TiB 5.0 TiB 5.0 TiB 56.83 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 449 KiB 2 1.8 MiB 0 320 GiB cephfs.application-tst.meta 2 16 540 MiB 18.79k 2.1 GiB 0.16 320 GiB cephfs.application-tst.data 3 32 4.4 GiB 8.01k 17 GiB 1.33 320 GiB cephfs.application-acc.meta 4 16 11 GiB 3.54M 45 GiB 3.37 320 GiB cephfs.application-acc.data 5 128 1.2 TiB 13.74M 4.8 TiB 79.46 320 GiB ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 448.5k 4.0 4499G 0.0000 1.0 1 on False cephfs.application-tst.meta 539.8M 4.0 4499G 0.0005 4.0 16 on False cephfs.application-tst.data 4488M 51200M 4.0 4499G 0.0444 1.0 32 on False cephfs.application-acc.meta 11430M 4.0 4499G 0.0099 4.0 16 on False cephfs.application-acc.data 1244G 4.0 4499G 1.1062 1.0000 0.9556 1.0 128 on False ----------------------------------------------------------------------------

1 year, 3 months

2
3
0 0

mds damage cannot repair

by Andrej Filipcic

Hi, there is mds damage on our cluster, version 17.2.5, [ { "damage_type": "backtrace", "id": 2287166658, "ino": 3298564401782, "path": "/hpc/home/euliz/.Xauthority" } ] The recursive repair does not fix it, ...ceph tell mds.0 scrub start /hpc/home/euliz force,repair,recursive mds log: 2023-02-10T07:01:34.012+0100 7f46df3ea700 0 mds.0.cache failed to open ino 0x30001c26a76 err -116/0 2023-02-10T07:01:34.012+0100 7f46df3ea700 0 mds.0.cache open_remote_dentry_finish bad remote dentry [dentry #0x1/hpc/home/euliz/.Xauthority [568,head] auth REMOTE(reg) (dversion lock) pv=0 v=4425667830 ino=(nil) state=1073741824 | ptrwaiter=1 0x5560eb33a780] Any clue how to fix this? or remove the file from namespace? it is not important... Thanks, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail:Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 year, 3 months

2
2
0 0

Migrate a bucket from replicated pool to ec pool

by Boris Behrens

Hi, we use rgw as our backup storage, and it basically holds only compressed rbd snapshots. I would love to move these out of the replicated into a ec pool. I've read that I can set a default placement target for a user ( https://docs.ceph.com/en/octopus/radosgw/placement/). What does happen to the existing user data? How do I move the existing data to the new pool? Does it somehow interfere with ongoing data upload (it is one internal user, with 800 buckets which constantly get new data and old data removed)? Cheers Boris ps: Can't wait to see some of you at the cephalocon :) -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 year, 3 months

3
4
0 0

[RGW - octopus] too many omapkeys on versioned bucket

by Boris Behrens

Hi, I have one bucket that showed up with a large omap warning, but the amount of objects in the bucket, does not align with the amount of omap keys. The bucket is sharded to get rid of the "large omapkeys" warning. I've counted all the omapkeys of one bucket and it came up with 33.383.622 (rados -p INDEXPOOL listomapkeys INDEXOBJECT | wc -l) I've checked the amount of actual rados objects and it came up with 17.095.877 (rados -p DATAPOOL ls | grep BUCKETMARKER | wc -l) I've checked the bucket index and it came up with 16.738.482 (radosgw-admin bi list --bucket BUCKET | grep -F '"idx":' | wc -l) I have tried to fix it with radosgw-admin bucket check --check-objects --fix --bucket BUCKET but this did not change anything. Is this a known bug or might there be something else going on. How can I investigate further? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 year, 3 months

2
3
0 0

Is ceph bootstrap keyrings in use after bootstrap?

by Zhongzhou Cai

Hi, I'm on Ceph version 16.2.10, and I found there are a bunch of bootstrap keyrings (i.e., client.bootstrap-<mds|mgr|osd|rbd|rbd-mirror|rgw>) located at /var/lib/ceph/bootstrap-<mds|mgr|osd|rbd|rbd-mirror|rgw>/ceph.keyring after bootstrap. Are they still in use after bootstrap? Is it safe to remove them from host and even from ceph monitor? Thanks, Zhongzhou Cai

1 year, 3 months

2
1
0 0

Health warning - POOL_TARGET_SIZE_BYTES_OVERCOMMITED

by Sake Paulusma

The RATIO for cephfs.application-acc.data shouldn't be over 1.0, I believe this triggered the error. All weekend I was thinking about this issue, but couldn't find an option to correct this. But minutes after posting I found a blog about the autoscaler (https://ceph.io/en/news/blog/2022/autoscaler_tuning) and it speaks about the option to set the Rate. Shouldn't this option be set to 2 when using a stretched cluster and not 4?

1 year, 3 months

1
0
0 0

Limited set of permissions for an RGW user (S3)

by Gauvain Pocentek

Hi list, A little bit of background: we provide S3 buckets using RGW (running quincy), but users are not allowed to manage their buckets, just read and write objects in them. Buckets are created by an admin user, and read/write permissions are given to end users using S3 bucket policies. We set the users quota to 0 for everything to forbid them to create buckets. This is not really scalable and a bit annoying for the users. So we are trying to find a solution to allow users to create their own buckets but with a limited set of APIs available (no policy change for example). The ceph doc says that policies cannot be applied on users, groups or roles yet. Is there any other way to achieve this? Any feedback will be appreciated. Thanks! Gauvain

1 year, 3 months

1
0
0 0

Re%3A%20%5Bceph-users%5D%20Re%3A%20Exit%20yolo%20mode%20by%20increasing%20size/min_size%20does%20not%20%28really%29%20work

by Stefan Pinter

Hi! Thank you 😊 Your message was very helpful! The main reason why “ceph df“ went to “100% USAGE” was because of the crush rule said this: "min_size": 2 "max_size": 2 And the new “size” was 3, so the rule did not want to work with the pool. After creating a new rule and setting the pools to this new rule, it seems to work (in our test cluster) rule newrule { ruleset 4 type replicated min_size 3 max_size 3 step take default step choose firstn 2 type room step chooseleaf firstn 2 type host step emit } Thanks again! Further testing now 😊 ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 year, 3 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users February 2023