- ceph-users - lists.ceph.io

Advices : RGW and Citrix ADC
by BEAUDICHON Hubert (Acoss) 22 May '23

22 May '23

Hi everyone, Is there any advices, best practices for a Citrix ADC VIP configuration in front of a dual RadosGW (same zone) ? I find many documentations for HAProxy but nothing for this hardware. For information, the CEPH cluster version is Pacific 16.2.10. Thanks for your help. Regards, Hubert [cid:image001.png@01D98CA0.D075D720] Hubert Beaudichon Secteur Infrastructures / Gamme CET-Infrastructures SDAT UT AUTOMATION ET CLOUD Hubert.beaudichon(a)acoss.fr<mailto:Hubert.beaudichon@acoss.fr> 01 82 30 77 03 * 06 00 00 00 00 36 rue de Valmy * 93108 Montreuil CEDEX www.urssaf.fr<http://www.urssaf.fr> [cid:image002.png@01D98CA1.F1ADE8F0]<https://www.youtube.com/channel/UCSjS8hOXCs5FlAnTi74rPrA>[cid:image003.png@01D98CA1.F1ADE8F0]<https://www.linkedin.com/company/acoss/>[cid:image002.png@01D98CA1.F1ADE8F0]<https://twitter.com/urssaf>[cid:image005.png@01D98CA0.D075D720]

1 0

RGW: are there any ways to limit access from user to a storage class?
by huy nguyen 22 May '23

22 May '23

There are tags for placement, so any user that has the tag is allowed to access the placement. But I can't find how to do it for storage class? Anyone have any ideal? Thanks.

1 0

Colocated RGW using same port, but different IPs
by Ondřej Kukla 22 May '23

22 May '23

Hello, As a part of performance/HW utilization testing I would like to compare how a VM with single RGW instance compares to a VM with 2 or more colocated RGWs. I’m deploying Quincy using Ansible but I’m struggling with setting the config to use the same port across all RGW instances, but to have a different IP as a beast endpoint. I would like to ask if anyone has any suggestion on this topic. Is this even supported scenario or it is better to have the instances using different ports? Any help is much appreciated. Kind regards, Ondrej

1 0

MDS migrations
by Odair M. 21 May '23

21 May '23

Hi, I'm doing a master's degree in distributed file systems, specifically CephFS in HPC, and I have some questions about the metadata migration process of MDSes that balances the MDS cluster. * Sage Weil Ph.D. thesis' and the documentation in the Ceph website say that before migrating to balance the MDS, the first step is to freeze the subtree and when the migration is complete, to unfreeze it. Doesn't it affect the performance since the subtree stays unavailable until migration is over? * Could you explain why the freezes are required at all? * Is It possible to change the migration process to migrate a copy of the inodes and then migrate inodes that have modifications after the migration? Like taking a snapshot of the subtree and migrate that, and then migrate any changes that happened after the snapshot migration is done, reducing the time spent with the subtree frozen. -- Odair M. Ditkun Jr Master's networks and distributed systems — University of State Parana — Brazil

1 0

RGW: request fields in Lua script return wrong data?
by viplanghe6＠gmail.com 21 May '23

21 May '23

Thanks for your answer. I was able to get the Lua debug log. But I think some request fields don't work: I have this lua script, for example: if Request.HTTP.StorageClass == 'COLD' then RGWDebugLog(Request.RGWOp .. " request with StorageClass: " .. Request.HTTP.StorageClass .. " Obj name: " .. Request.Object.Name .. " Obj ID: " .. Request.Object.Id .. " Size: " .. Request.Object.Size .. " MTime: " .. Request.Object.MTime ) end I put an 1kb object using boto3: s3.Object(bucket, 'test-object6').put(Body="0"*1000, ACL='public-read', StorageClass='COLD') But the size and mtime field provided wrong data? Lua INFO: put_obj request with StorageClass: COLD Obj name: test-object6 Obj ID: test-object6 Size: 0 MTime: 1970-01-01 08:00:00

3 3

Dedicated radosgw gateways
by Michal Strnad 19 May '23

19 May '23

Hi all, at Cephalocon 2023, it was mentioned several times that for service tasks such as data deletion via garbage collection or data replication in S3 via zoning, it is good to do them on dedicated radosgw gateways and not mix them with gateways used by users. How can this be achieved? How can we isolate these tasks? Will using dedicated keyrings instead of admin keys be sufficient? How do you operate this in your environment? Thx Michal

6 10

Ceph OSDs suddenly use public network for heardbeat_check
by Lee, H. (Hurng-Chun) 19 May '23

19 May '23

Hi all, We are running the Nautilus cluster. Today due to UPS work, we shut down the whole cluster. After we start the cluster, many OSDs go down and they seem to start doing the heardbeat_check using the public network. For example, we see the following logs: --- 2023-05-16 19:35:29.254 7efcd4ce7700 -1 osd.101 42916 heartbeat_check: no reply from 131.174.45.223:6825 osd.185 ever on either front or back, first ping sent 2023-05-16 19:34:48.593701 (oldest deadline 2023-05-16 19:35:08.593701) --- While I was expect the heardbeat to go through the cluster network, e.g. instead of 131.174.45.223, it should use 172.20.128.223. In fact, when we start up the cluster, we don't have DNS available to resolve the IP addresses, and for a short while, all OSDs are located in a new host called "localhost.localdomain". At that point, I fixed it by setting the static hostname using `hostnamectl set -hostname xxx`. Now we cannot bring the cluster back to healthy state. We are stuck at: --- cluster: id: 86c9bc85-b7f3-49a1-9e1f-8c9f2b31fca8 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available pauserd,pausewr,noout,nobackfill,norebalance,norecover flag(s) set 88 osds down Reduced data availability: 2544 pgs inactive, 2369 pgs down, 159 pgs peering, 294 pgs stale Degraded data redundancy: 870424/2714593746 objects degraded (0.032%), 30 pgs degraded, 9 pgs undersized 8631 slow ops, oldest one blocked for 803 sec, mon.ceph- mon01 has slow ops services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 2h) mgr: ceph-mon02(active, since 2h), standbys: ceph-mon01, ceph-mon03 mds: cephfs:0/1, 1 failed osd: 191 osds: 103 up (since 1.46827s), 191 in (since 4w) flags pauserd,pausewr,noout,nobackfill,norebalance,norecover data: pools: 2 pools, 2560 pgs objects: 456.13M objects, 657 TiB usage: 1.1 PiB used, 638 TiB / 1.7 PiB avail pgs: 100.000% pgs not active 870424/2714593746 objects degraded (0.032%) 2087 down 282 stale+down 129 peering 32 stale+peering 30 undersized+degraded+peered --- Any idea we could fix it and get the OSDs to use cluster network to do heartbeat checks. Any help would be highly appreciated. Thank you very much. Cheers, Hong -- Hurng-Chun (Hong) Lee, PhD ICT manager Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging Radboud University Nijmegen e-mail: h.lee(a)donders.ru.nl tel: +31(0)631132518 web: http://www.ru.nl/donders/ pgp: 3AC505B2B787A8ABE2C551B1362976D838ABF09E * Mon, Tue and Thu at Trigon; Wed and Fri working from home

3 3

ls: cannot access '/cephfs': Stale file handle
by Harry G Coin 18 May '23

18 May '23

I have two autofs entries that mount the same cephfs file system to two different mountpoints. Accessing the first of the two fails with 'stale file handle'. The second works normally. Other than the name of the mount point, the lines in autofs are identical. No amount of 'umount -f' or restarting autofs resolves it. Any ideas?

2 1

User + Dev Monthly Meeting Minutes 2023-05-18
by Laura Flores 18 May '23

18 May '23

Hey all, Here are the minutes from today's meeting: Reef RC0 status? - ETA is the last week of May - Several users have volunteered to help with Reef scale testing; efforts are documented in this etherpad: https://pad.ceph.com/p/reef_scale_testing [Ben Gao] feature request: for non root user with password to install cluster with cephadm - There was work done on this previously, but it was closed in this PR: https://github.com/ceph/ceph/pull/44085 - Before, Sebastian questioned the use case of this feature and overhead - Ben's use case: Customer network security policy; currently ceph admin only supports root install - Adam King: We would be open to reviewing any patches submitted, but ultimately ssh keys are a better option. [Andras Pataki] Crush upmap fix: https://tracker.ceph.com/issues/51729 - There is a fix, but it needs to be well tested, as this is a high-traffic area of code - Andras is interested in testing the fix in a test environment; would need to upgrade the OSDs to include the crushwrapper code fix - Laura recommends against testing the fix on production clusters - Update the Tracker if you need a test build See you all next month! - Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

CLT Meeting minutes 17/05/23
by Nizamudeen A 18 May '23

18 May '23

Hey all, Ceph Quarterly announcement [Josh and Zac] One page digest that may published quarterly Planning for 1st of June, September and December Reef RC https://pad.ceph.com/p/reef_scale_testing https://pad.ceph.com/p/ceph-user-dev-monthly-minutes#L17 ETA last week of May Missing CentOS 9 Python deps Ken Dreyer has volunteered to help get the packages in epel Lab update There have been reimaging issues and kernel timeouts seen in the lab; infra team is working through fixing it Please raise infrastructure trackers for any bugs you see, as we have started to do monthly infrastructure bug scrubs Regards, Nizam

1 0