Hoping to learn from this myself, why will the current setup never work? On 11/11/20 10:29 AM, Robert Sander wrote: > Am 07.11.20 um 01:14 schrieb seffyroff(a)gmail.com: >> I've inherited a Ceph Octopus cluster that seems like it needs urgent maintenance before data loss begins to happen. I'm the guy with the most Ceph experience on hand and that's not saying much. I'm experiencing most of the ops and repair tasks for the first time here. > > My condolences. Get the data from that cluster and put the cluster down. > > In the current setup it will never work. > > Regards > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

That was a bit harsh to have said. Without seeing your EC profile and the topology, it’s hard to say for sure, but I suspect that adding another node with at least one larger OSD might help. The info that Hans asked for would help us say for sure. Agreed on both points, neither is ideal, but it’s not clear that even the OP thinks it is. Agreed, but again we don’t *know* that they expect it to. Is this just a PoC / demo cluster cobbled together out of whatever was laying around? The OSDs, by chance, aren’t running on top of RAID volumes are they? How many nodes? Is it possible that this cluster previously had more nodes/OSDs that were removed? I’ll speculate that the 4 OSDs are spread across a total of 2 nodes? — aad

Am 12.11.20 um 23:18 schrieb Phil Merricks: The cluster is just too small to do anything useful. It can be used to learn Ceph but really not for much more. Erasure Coding needs a lot of CPU, this is usually achieved with a large number of nodes (more than 10) and a proportional number of OSDs. Mixed HDDs and SSDs in one pool is not good practice as a pool should have OSDs of the same speed. Kindest Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

And also the erasure coded profile, so an example on my cluster would be: $ ceph osd pool get objects.rgw.buckets.data erasure_code_profile erasure_code_profile: objects_ecprofile $ ceph osd erasure-code-profile get objects_ecprofile crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 On 11/11/20 12:46 PM, Hans van den Bogert wrote: > Can you show a `ceph osd tree` ? > > On 11/7/20 1:14 AM, seffyroff(a)gmail.com wrote: >> I've inherited a Ceph Octopus cluster that seems like it needs urgent >> maintenance before data loss begins to happen. I'm the guy with the >> most Ceph experience on hand and that's not saying much. I'm >> experiencing most of the ops and repair tasks for the first time here. >> >> Ceph health output looks like this: >> >> HEALTH_WARN Degraded data redundancy: 3640401/8801868 objects degraded >> (41.359%), >>   128 pgs degraded, 128 pgs undersized; 128 pgs not deep-scrubbed in >> time; >>   128 pgs not scrubbed in time >> >> Ceph -s output: https://termbin.com/i06u >> >> The crush rule 'cephfs.media' is here: https://termbin.com/2klmq >> >> So, it seems like all PGs are in a 'warning' state for the main pool, >> which is erasure coded and 11TiB across 4 OSDs, of which around 6.4TiB >> is used. The Ceph services themselves seem happy, they're stable and >> have Quorum. I'm able to access the web panel fine also.  The block >> devices are of different sizes and types (2 large, different sized >> spinners, and 2 identical SSDs) >> >> I would welcome any pointers on what my steps to bring this up to full >> health may be.  If it's undersized, can I simply add another block >> device/OSD? Or perhaps adjusting config somewhere will get it to >> rebalance successfully? (the rebalance jobs have been stuck at 0% for >> weeks) >> >> Thank you for your time reading this message. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

That's rather situation dependent. I don't have really large disks, so the repair time isn't that large. Further, my SLO isn't that high that I need 99.xxx% uptime, if 2 disks break in the same repair window, that would be unfortunate, but I'd just grab a backup from a mirroring cluster. Looking at it from another perspective, I came from a single host RAID5 scenario, I'd argue this is better since I can survive a host failure. Also this is a sliding problem right? Someone with K+3 could argue K+2 is not enough as well. Hans

I think we're deviating from the original thread quite a bit and I would never argue that in a production environment with plenty OSDs you should go for R=2 or K+1, so my example cluster which happens to be 2+1 is a bit unlucky. However I'm interested in the following On 11/16/20 11:31 AM, Janne Johansson wrote: I'm not sure I follow, from a logical perspective they *are* down at the same time right? In your scenario 1 up-to-date replica was left, but even that had a surprise. Okay well that's the risk you take with R=2, but it's not intrinsically different than R=3.

I was trying to describe something like this http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-September/013237.h… There are more posts from ceph consultants that get called in after someone with "only" R=2/EC K+1 seeing data loss, but I didn't dig them all up. Ie, a kind of split-brain scenario where a small fault/outage on one of the drives and later a bigger fault on another will hurt you in R=2 or K+1 scenarios, even if you don't have two full faults, only one that is temporarily out and the second being the one "real" fault which could be "disk died" or something which we usually imagine as the scenario to handle for raids or replication sizes. Not trying to say you don't understand this, but rather that people who run small ceph clusters tend to start out with R=2 or K+1 EC because the larger faults are easier to imagine. When you have R=3 and you move one of the 3 PG copies for disk resize or something, then you are temporarily reduced to two copies (at least two up-to-date copies if writes are happening during the move), so you can still bear one surprise until this is completed without losing data. With R=2/EC K+1 not so much. Also, me calling it small and large faults mean that there is a huge difference in "a few PGs with issues" and "disk completely broken", but if the PGs are for a pool with disk images, then all images on the pool are prone to have errors and not just "we lost 1% of the files, we can get only those back", but rather the disk images are all having random holes in them. -- May the most significant bit of your life be positive.

Thanks for all the replies folks. I think it's testament to the versatility of Ceph that there are some differences of opinion and experience here. With regards to the purpose of this cluster, it is providing distributed storage for stateful workloads of containers. The data produced is somewhat immutable, it can be regenerated over time, however that does cause some slowdown for the teams that use the data as part of their development pipeline. To the best of my understanding the goals here were to provide a data loss safety net but still make efficient use of the block devices assigned to the cluster, which is I imagine where the EC direction came from. The cluster is 3 nodes with the OSDs themselves mainly housed in two of those. Additionally there was an initiative to 'use what we have' (or as I like to put it, 'cobble it together') with commodity hardware that was immediately available to hand. The departure of my predecessor has left some unanswered questions so I am not going to bother second guessing beyond what I already know. As I understand it my steps are: 1: Move off the data and scrap the cluster as it stands currently. (already under way) 2: Group the block devices into pools of the same geometry and type (and maybe do some tiering?) 3. Spread the OSDs across all 3 nodes so recovery scope isn't so easily compromised by a loss at the bare metal level 4. Add more hosts/OSDs if EC is the right solution (this may be outside of the scope of this implementation, but I'll keep a-cobblin'!) The additional ceph outputs follow: ceph osd tree <https://termbin.com/vq63> ceph osd erasure-code-profile get cephfs-media-ec <https://termbin.com/h33h> I am fully prepared to do away with EC to keep things simple and efficient in terms of CPU occupancy. On Mon, 16 Nov 2020 at 02:32, Janne Johansson &lt;icepic.dz(a)gmail.com&gt; wrote:

Phil; I'm probably going to get crucified for this, but I put a year of testing into this before determining it was sufficient to the needs of my organization... If the primary concerns are capability and cost (not top of the line performance), then I can tell you that we have had great success utilizing Intel Atom C3000 series CPUs. We have built 2 clusters with capacities on the order of 130TiB, for less than $30,000 each. The initial clusters cost $20,000 each, for half the capacity. Our testing cluster cost $8,000 to build, and most of that hardware could have been wrapped into the first production cluster build. For those keeping track, no that is not the lowest cost / unit space. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com -----Original Message----- From: Phil Merricks [mailto:seffyroff@gmail.com] Sent: Monday, November 16, 2020 5:52 PM To: Janne Johansson Cc: Hans van den Bogert; ceph-users Subject: [ceph-users] Re: (Ceph Octopus) Repairing a neglected Ceph cluster - Degraded Data Reduncancy, all PGs degraded, undersized, not scrubbed in time Thanks for all the replies folks. I think it's testament to the versatility of Ceph that there are some differences of opinion and experience here. With regards to the purpose of this cluster, it is providing distributed storage for stateful workloads of containers. The data produced is somewhat immutable, it can be regenerated over time, however that does cause some slowdown for the teams that use the data as part of their development pipeline. To the best of my understanding the goals here were to provide a data loss safety net but still make efficient use of the block devices assigned to the cluster, which is I imagine where the EC direction came from. The cluster is 3 nodes with the OSDs themselves mainly housed in two of those. Additionally there was an initiative to 'use what we have' (or as I like to put it, 'cobble it together') with commodity hardware that was immediately available to hand. The departure of my predecessor has left some unanswered questions so I am not going to bother second guessing beyond what I already know. As I understand it my steps are: 1: Move off the data and scrap the cluster as it stands currently. (already under way) 2: Group the block devices into pools of the same geometry and type (and maybe do some tiering?) 3. Spread the OSDs across all 3 nodes so recovery scope isn't so easily compromised by a loss at the bare metal level 4. Add more hosts/OSDs if EC is the right solution (this may be outside of the scope of this implementation, but I'll keep a-cobblin'!) The additional ceph outputs follow: ceph osd tree <https://termbin.com/vq63> ceph osd erasure-code-profile get cephfs-media-ec <https://termbin.com/h33h> I am fully prepared to do away with EC to keep things simple and efficient in terms of CPU occupancy. On Mon, 16 Nov 2020 at 02:32, Janne Johansson &lt;icepic.dz(a)gmail.com&gt; wrote: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io