- ceph-users - lists.ceph.io

Seeking a (paid) consultant - Ceph expert.
by ceph＠denovogroup.org 12 Sep '24

12 Sep '24

We are a small non-profit company that develops technologies for affordable delivery of broadband Internet access to rural communities in developing regions. We are experimenting with a small Ceph cluster that is housed in a street-side cabinet. Our goals are to maintain availability, and avoid data-loss, even under extreme and hardware failures. We are NOT sensitive to performance, scalability or optimization of storage size --- rather high degree of replication and focus on redundancy. Our current experiment consists of 3 rack-mount chassis by Dell - PowerEdge C6220. Each chassis holds 4 server blades. Total of 12 server blades. Each blade has two 10Gig network cards, used in an MLAG (Multi-chassis Link Aggregation Group) configuration to two redundant network switches. Each blade has 4 SAS drives (two SSDs and two HDDs). Each blade has 192Gb of RAM and 24 cores of Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (2 Sockets). We are using Proxmox (we're no experts, but got it working pretty nicely, migration of workloads and all). We are using Ceph version 16.2.9 and manage its basics via the Proxmox integration/GUI with nearly default settings, and without understanding much of what we're doing. (We made minor edits to the Crash-maps - but I'll admit that I don't really understand Ceph's building blocks and nuances). We need hand-holding in re-designing the system with our specific (extreme) availability goals -- we don't need step-by-step spoon-feeding, but rather big-picture design guidelines. Someone who understands deeply the specifics and nuances of Ceph, but also can maintain a big-picture operational view of our needs. Ideally reply directly to me via email - although a discussion on the list could probably be useful as well. (I don't even know what questions to ask). Thanks! Yahel. Yahel Ben-David, Ph.D. De Novo Group - Executive Director Bridging the gap between research and impact.

1 0

Re: [RGW][CEPHADM] Multisite configuration and Ingress
by Gilles Mocellin 12 Sep '24

12 Sep '24

Yes, I've read some stories about that in this mailing list, but, my question was not clear enough, I want to know what's possible to do with cephadm : - Is it possible to create several Ingress with cephadm ? (I can't reach my test env to test until tomorrow !) - If we do, how can we configure differently thoses groups of RGWs daemons ? (related to my previous question on ceph config / ceph.conf) Le mercredi 11 septembre 2024, 22:23:14 CEST Anthony D'Atri a écrit : > I've seen deployments with more than one cohort of RGWs. E.g. > > * one group of RGWs that serve internal requests with an LB > * one group of RGWs that serve external requests with a different LB > * one group of RGWs that only process LC / GC events > > > On Sep 11, 2024, at 4:01 PM, Gilles Mocellin > > <gilles.mocellin(a)nuagelibre.org> wrote: > > > > Hi again, > > > > I wonder if there is a better way of configuring multisite configuration, > > especially zone/zonegroup endpoints : > > > > - many endpoints, pointing directly at RGW daemons > > - one endpoint, pointing at the load balancer VIP of Ingress > > > > If we want to separate client access from replication, is it possible to > > have two Ingress ? It can be great to have them in two different > > networks. > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

[RGW][CEPHADM] Multisite configuration and Ingress
by Gilles Mocellin 12 Sep '24

12 Sep '24

Hi again, I wonder if there is a better way of configuring multisite configuration, especially zone/zonegroup endpoints : - many endpoints, pointing directly at RGW daemons - one endpoint, pointing at the load balancer VIP of Ingress If we want to separate client access from replication, is it possible to have two Ingress ? It can be great to have them in two different networks.

1 0

Re: bilog trim fails with "No such file or directory"
by Florian Schwab 11 Sep '24

11 Sep '24

Hi, sorry forgot to add this. Ceph release: 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) The cluster consists out of 360 OSDs and the index pool has 128 PGs and auto-resharding is enabled. Cheers, Florian > On 11. Sep 2024, at 18:18, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > > Which Ceph release? How many OSDs and how many PBs in the index pool? Auto-resharding enabled? > >> On Sep 11, 2024, at 10:24 AM, Florian Schwab <fschwab(a)impossiblecloud.com> wrote: >> >> Hi everyone, >> >> I hope maybe someone here has an idea what is happening here or can give some pointers how to debug it further. >> >> We currently have a bucket which has large omap objects. Following this guide (https://access.redhat.com/solutions/6450561) we are able to identify the bucket etc. >> >> $ ceph health detail >> HEALTH_WARN 25 large omap objects >> [WRN] LARGE_OMAP_OBJECTS: 25 large omap objects >> 25 large objects found in pool 'ceph-objectstore.rgw.buckets.index' >> Search the cluster log for 'Large omap object found' for more details. >> >> $ radosgw-admin metadata list --metadata-key bucket.instance | grep -i XXX >> “YYY:XXX”, >> >> $ radosgw-admin bilog list --bucket-id=“XXX" --bucket=“YYY" --max-entries=600000 | grep -c op_id >> 600000 >> >> So far so good! But when try to trim the bilog we get the following error: >> >> $ radosgw-admin bilog trim --bucket-id=“XXX" --bucket=“YYY" >> ERROR: trim_bi_log_entries(): (2) No such file or directory >> >> >> The bucket itself doesn’t show any issues - all S3 operations are working. >> >> >> Thanks for any input! >> >> >> Cheers, >> Florian >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

bilog trim fails with "No such file or directory"
by Florian Schwab 11 Sep '24

11 Sep '24

Hi everyone, I hope maybe someone here has an idea what is happening here or can give some pointers how to debug it further. We currently have a bucket which has large omap objects. Following this guide (https://access.redhat.com/solutions/6450561) we are able to identify the bucket etc. $ ceph health detail HEALTH_WARN 25 large omap objects [WRN] LARGE_OMAP_OBJECTS: 25 large omap objects 25 large objects found in pool 'ceph-objectstore.rgw.buckets.index' Search the cluster log for 'Large omap object found' for more details. $ radosgw-admin metadata list --metadata-key bucket.instance | grep -i XXX “YYY:XXX”, $ radosgw-admin bilog list --bucket-id=“XXX" --bucket=“YYY" --max-entries=600000 | grep -c op_id 600000 So far so good! But when try to trim the bilog we get the following error: $ radosgw-admin bilog trim --bucket-id=“XXX" --bucket=“YYY" ERROR: trim_bi_log_entries(): (2) No such file or directory The bucket itself doesn’t show any issues - all S3 operations are working. Thanks for any input! Cheers, Florian

1 0

Blocking/Stuck/Corrupted files
by dominik.baack 11 Sep '24

11 Sep '24

Hi, we started upgrading our Ceph Cluster consisting of 7 Nodes from quincy to reef two days ago. This included the upgrade of the underlying OS and several other small changes. After hitting the osd_remove_queue bug, we could recover mostly, but are still in a non-healthy state because of changes on our network. (See attached image) But overall we can mount the filesystem on all nodes, read and write. The Problem now is that at least one file created by slurmctld exist which seems to be somehow compromised during this. In addition there a several more files which are currently unidentified but noticable through stuck container executions. Each file cannot be read or removed on any of the storage or working nodes. All operations (cat, rm, less, ...) are stuck until they are forcefully terminated. I checked all nodes manually and could not identify any process with an open handle to the file. A manual deep scrub of all pgs and normal scrub does not show any further problems If possible we would like to identify and unblock or remove the compromised files. Cheers Dominik

1 0

Re: Somehow throotle recovery even further than basic options?
by Szabo, Istvan (Agoda) 11 Sep '24

11 Sep '24

Hi, I've played around a bit with these scripts, finally I went with the pgremapper one because it's documented better and no need multiple run but I'd like to get some confirmation about the functionality from someone who used it because it was too fast. What I've tested is the cancel-backfill (pgremapper cancel-backfill --yes) option as written in the documentation: https://github.com/digitalocean/pgremapper which is based on the upmap-remapped.py It made me 2 questions, hopefully someone can answer: Q1 regarding osd removal: What I've done in test env: 1. Set the flags: ceph osd set nobackfill;ceph osd set norebalance 2. Reweight all the osds that I want to remove: for i in {36..43};do ceph osd reweight $i 0.0;done 3. Run the tool and wait couple of minutes to get back the prompt: pgremapper cancel-backfill --yes 4. Unset flags: ceph osd unset norebalance;ceph osd unset nobackfill 5. When cluster health ok, I removed the osds: for num in {36..43}; do ceph osd out osd.$num;systemctl disable ceph-osd@$num;systemctl purge $num --yes-i-really-mean-it;umount /var/lib/ceph/osd/ceph-$num;done 1. In theory here I'm missing a lot of chunks so how I know that all the data actually recovered/regenerated somewhere else in the cluster? Because it was too fast. If the ceph balancer status says no more optimization that is considered to be the done signal? Q2 regarding the process: Could this tool be used for removal actually or only extension? Reweight can be up and down however they say augment which for me extension. "Example - Cancel all backfill in the system as a part of an augment This is useful during augment scenarios, if you want to control PG movement to the new nodes via the upmap balancer (a technique based on this CERN talk<https://www.slideshare.net/Inktank_Ceph/ceph-day-berlin-mastering-ceph-oper…>." Thank you ________________________________ From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Sent: Monday, September 9, 2024 2:07 PM To: Janne Johansson <icepic.dz(a)gmail.com> Subject: Re: [ceph-users] Re: Somehow throotle recovery even further than basic options? Let me try to understand, so this actually move pg by pg to the new osds which means will not push the osds on the same hosts too hard due to 1 pg at a time. Am I correct? Also if something goes wrong, is it possible to stop this script and let it go as is or if I want it faster, can I? As I see this can be used during rebalance also, just need to stop rebalance, run the script and unset rebalance. ________________________________ From: Janne Johansson <icepic.dz(a)gmail.com> Sent: Saturday, September 7, 2024 2:00 PM To: Eugen Block <eblock(a)nde.ag> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Somehow throotle recovery even further than basic options? Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ The pgremapper (and the python one) will allow you to mark all the PGs that a new disk gets as an empty-misplaced PG to be correct where they currently are. This means that after you run one of the remappers, the upmap will tell the cluster to stay as it is even though new empty OSDs have arrived with correct crush weights and all. So you set norebalance, add N+1 new OSDs, the cluster shudders for a short while when the new-empty PGs are created on the new drives, then you have lots and lots of misplaced PGs which norebalance prevents from starting backfills on. Then you run the remapper and "fix" the upmap so all the current placements are considered "correct", and hence the PGs that were supposed to move stop being misplaced. After this, you remove "norebalance" to allow moves to start happening. By now, no movement (or at least very few PGs) should occur. What happens next is that the balancer notices there actually is more space, and figures out the optimal result is more or less the same as above where lots of PGs should go to the new OSDs, but it does this with the max-misplaced-ratio in mind, and it will "move" the PGs by just unsetting their upmap entry that forced them to want to stay in place, so as time passes, it moves a few PGs at a time, and moves them by removing the upmaps from them so that most of the time your other OSDs will look perfectly healthy and will continue to do all the scrubs and things a healthy OSD should do and which it will not do if it has a long queue of backfills waiting to eat up all the slots for non-client IO. While you can do soft additions with increasing crush weights, this potentially causes lots of more movements since an OSD host with OSDs going from 0.1 to 0.2 weight might not place all PGs in the same spot in those two cases, so you could have movement within the host and so on from the recalculated pseudorandom placements on every increase. Den lör 7 sep. 2024 kl 00:15 skrev Eugen Block <eblock(a)nde.ag>: > > I can’t say anything about the pgremapper, but have you tried > increasing the crush weight gradually? Add new OSDs with crush initial > weight 0 and then increase it in small steps. I haven’t used that > approach for years, but maybe that can help here. Or are all OSDs > already up and in? Or you could reduce the max misplaced ratio to 1% > or even lower (default is 5%)? > > Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo(a)agoda.com>: > > > Forgot to paste, somehow I want to reduce this recovery operation: > > recovery: 0 B/s, 941.90k keys/s, 188 objects/s > > To 2-300Keys/sec > > > > > > > > ________________________________ > > From: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> > > Sent: Friday, September 6, 2024 11:18 PM > > To: Ceph Users <ceph-users(a)ceph.io> > > Subject: [ceph-users] Somehow throotle recovery even further than > > basic options? > > > > Hi, > > > > 4 years ago we've created our cluster with all disks 4osds (ssds and > > nvme disks) on octopus. > > The 15TB SSDs still working properly with 4 osds but the small 1.8T > > nvmes with the index pool not. > > Each new nvme osd adding to the existing nodes generates slow ops > > with scrub off, recovery_op_priority 1, backfill and recovery 1-1. > > I even turned off all index pool heavy sync mechanism but the read > > latency still high which means recovery op pushes it even higher. > > > > I'm trying to somehow add resource to the cluster to spread the 2048 > > index pool pg (in replica 3 means 6144pg index pool) but can't make > > it more gentle. > > > > The balancer is working in upmap with max deviation 1. > > > > Have this script from digitalocean > > https://github.com/digitalocean/pgremapper, is there anybody tried > > it before how is it or could this help actually? > > > > Thank you the ideas. > > > > ________________________________ > > This message is confidential and is for the sole use of the intended > > recipient(s). It may also be privileged or otherwise protected by > > copyright or other legal rules. If you have received it by mistake > > please let us know by reply email and delete it from your system. It > > is prohibited to copy this message or disclose its content to > > anyone. Any confidentiality or privilege is not waived or lost by > > any mistaken delivery or unauthorized disclosure of the message. All > > messages sent to and from Agoda may be monitored to ensure > > compliance with company policies, to protect the company's interests > > and to remove potential malware. Electronic messages may be > > intercepted, amended, lost or deleted, or contain viruses. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

RFC: cephfs fallocate
by Milind Changire 11 Sep '24

11 Sep '24

Problem: CephFS fallocate implementation does not actually reserve data blocks when mode is 0. It only truncates the file to the given size by setting the file size in the inode. So, there is no guarantee that writes to the file will succeed Solution: Since an immediate remediation of this problem is not possible, I request users to vote on the most suitable approach to avoid breaking dependent software: 1. report EOPNOTSUPP (operation not supported) when mode is 0 2. continue with existing implementation of file size truncation when mode is 0 -- Milind

2 2

User + Dev Monthly Meetup coming up on Sept. 25th!
by Laura Flores 11 Sep '24

11 Sep '24

Hi all, The User + Dev Monthly Meetup is coming up on Sept. 25th! This month, we will have a discussion on incorporating the upmap-remapped [1] & pgremapper [2] tools with the balancer mgr module. There has been interest in the community for a while about incorporating the logic from these external tools with the existing Ceph code. There will be a presentation on this high-level design doc: https://tracker.ceph.com/issues/67418 See you all there! RSVP here: https://www.meetup.com/ceph-user-group/events/303358179/?utm_medium=referra… 1. https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-rema… 2. https://github.com/digitalocean/pgremapper -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

CLT meeting notes: Sep 09, 2024
by David Orman 10 Sep '24

10 Sep '24

CLT discussion on Sep 09, 2024 19.2.0 release: * Cherry picked patch: https://github.com/ceph/ceph/pull/59492 * Approvals requested for re-runs CentOS Stream/distribution discussions ongoing * Significant implications in infrastructure for building/testing requiring ongoing discussions/work to determine best course of action/path forward Vstart documentation cleanup * Zac Dover working on this documentation, open to any suggestions/ideas (commands that people new to Ceph would find useful — idea like "test-driving a vehicle"

1 0