- ceph-users - lists.ceph.io

CephFS exposing public storage network
by Jaan Vaks 08 Oct '19

08 Oct '19

Hi all, I'm evaluation cephfs to serve our business as a file share that span across our 3 datacenters. One concern that I have is that when using cephfs and OpenStack Manila is that all guest vms needs access to the public storage net. This to me feels like a security concern. I've seen one suggestion is to put NFS gateways in between to prevent this, I would prefer not having to use NFS. Is there another way to solve this or is this a no concern to others, both the network and NFS? We are a small cloud provider and having different customers exposed to each other on the same storage net seems risky to me. Regards Jaan

4 4

Re: PG is stuck in repmapped and degraded
by 展荣臻（信泰） 08 Oct '19

08 Oct '19

>If the journal is no longer readable: the safe variant is to >completely re-create the OSDs after replacing the journal disk. (The >unsafe way to go is to just skip the --flush-journal part, not >recommended) Hello paul, Thank for your reply.we has replaced the journal disk. Last week we were on vacation,so this email is delayed. My confusion is that why pg was stuck? PG should to be repaired automatically,when osd is down isn't it?

2 1

Nautilus: BlueFS spillover
by Eugen Block 07 Oct '19

07 Oct '19

Hi, I'm following the discussion for a tracker issue [1] about spillover warnings that affects our upgraded Nautilus cluster. Just to clarify, would a resize of the rocksDB volume (and expanding with 'ceph-bluestore-tool bluefs-bdev-expand...') resolve that or do we have to recreate every OSD? Regards, Eugen [1] https://tracker.ceph.com/issues/38745

4 8

Re: cephx user performance impact
by iwesley＠mail.de 07 Oct '19

07 Oct '19

For performance stuff you’d better setup a running environment to benchmark by yourself. Regards, Wesley Peng > Am Oct 7, 2019 - 1:17 AM schrieb frankaritchie(a)gmail.com: > > > Would RBD performance be hurt by having thousands of cephx users defined? >

1 0

cephx user performance impact
by Frank R 06 Oct '19

06 Oct '19

Would RBD performance be hurt by having thousands of cephx users defined?

1 0

Comercial support - Brazil
by Gesiel Galvão Bernardes 04 Oct '19

04 Oct '19

Hi everyone, I searching to consulting and support of Ceph in Brazil. Does anyone on the list provide consulting in Brazil? Regards, Gesiel

2 1

Re: RAM recommendation with large OSDs?
by Anthony D'Atri 04 Oct '19

04 Oct '19

It’s not that the limit is *ignored*; sometimes the failure of the subtree isn’t *detected*. Eg., I’ve seen this happen when a node experienced kernel weirdness or OOM conditions such that the OSDs didn’t all get marked down at the same time, so the PGs all started recovering. Admitedly it’s been a while since I’ve seen this, my sense is that with Luminous the detection became a *lot* better. > On Oct 3, 2019, at 9:55 AM, Darrell Enns <darrelle(a)knowledge.ca> wrote: > > Thanks for the reply Anthony. > > Those are all considerations I am very much aware of. I'm very curious about this though: > >> mon_osd_down_out_subtree_limit. There are cases where it doesn’t kick in and a whole node will attempt to rebalance > > In what cases is the limit ignored? Do these exceptions also apply to mon_osd_min_in_ratio? Is this in the docs somewhere? > [ good Cephers trim their quoted text ]

1 0

Re: RAM recommendation with large OSDs?
by Anthony D'Atri 03 Oct '19

03 Oct '19

This is in part a question of *how many* of those dense OSD nodes you have. If you have a hundred of them, then most likely they’re spread across a decent number of racks and the loss of one or two is a tolerable *fraction* of the whole cluster. If you have a cluster of just, say, 3-4 of these dense nodes, component failure, network glitches, and even maintenance become problematic. You can *mostly* forestall whole-node rebalancing by careful alignment of fault domains with the value of mon_osd_down_out_subtree_limit. There are cases where it doesn’t kick in and a whole node will attempt to rebalance, which — assuming the CRUSH rules and topology are fault-tolerant — may cause surviving OSDs to reach full or backfillfull states, potentially resulting in an outage. If the limit does kick in, you’ll have reduced or no redundancy until you either bring the host/OSDs back up, or manually cause the recovery to proceed. As was already mentioned as well, having a small number of fault domains also limits the EC strategies you can safely use. > Thanks Paul. I was speaking more about total OSDs and RAM, rather than a single node. However, I am considering building a cluster with a large OSD/node count. This would be for archival use, with reduced performance and availability requirements. What issues would you anticipate with a large OSD/node count? Is the concern just the large rebalance if a node fails and takes out a large portion of the OSDs at once?

2 1

Objects degraded after adding disks
by Frank Schilder 03 Oct '19

03 Oct '19

I'm running a cepf fs with an 8+2 EC data pool. Disks are on 10 hosts and failure domain is host. Version is mimic 13.2.2. Today I added a few OSDs to one of the hosts and observed that a lot of PGs became inactive even though 9 out of 10 hosts were up all the time. After getting the 10th host and all disks up, I still ended up with a large amount of undersized PGs and degraded objects, which I don't understand as no OSD was removed. Here some details about the steps taken on the host with new disks, main questions at the end: - shut down OSDs (systemctl stop docker) - reboot host (this is necessary due to OS deployment via warewulf) Devices got renamed and not all disks came back up (4 OSDs remained down). This is expected, I need to re-deploy the containers to adjust for device name changes. Around this point PGs started peering and some failed waiting for 1 of the down OSDs. I don't understand why they didn't just remain active with 9 out of 10 disks. Until this moment of some OSDs coming up, all PGs were active. With min_size=9 I would expect all PGs to remain active with no changes to 9 out of the 10 hosts. - redeploy docker containers - all disks/OSDs come up, including the 4 OSDs from above - inactive PGs complete peering and become active - now I have a los of degraded Objects and undersized PGs even though not a single OSD was removed I don't understand why I have degraded objects. I should just have misplaced objects: HEALTH_ERR 22995992/145698909 objects misplaced (15.783%) Degraded data redundancy: 5213734/145698909 objects degraded (3.578%), 208 pgs degraded, 208 pgs undersized Degraded data redundancy (low space): 169 pgs backfill_toofull Note: The backfill_toofull with low utilization (usage: 38 TiB used, 1.5 PiB / 1.5 PiB avail) is a known issue in ceph (https://tracker.ceph.com/issues/39555) Also, I should be able to do whatever with 1 out of 10 hosts without loosing data access. What could be the problem here? Questions summary: Why does peering not succeed to keep all PGs active with 9 out of 10 OSDs up and in? Why do undersized PGs arise even though all OSDs are up? Why do degraded objects arise even though no OSD was removed? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

OSD down when deleting CephFS files/leveldb compaction
by Robert LeBlanc 03 Oct '19

03 Oct '19

I have one or two more stability issues I'm trying to solve in a cluster that I inherited that I just can't seem to figure out. One issue may be the cause for the other. This is a Jewel 10.2.11 cluster with ~760 - 10TB HDDs and 5GB journals on SSD. When a large number of files are deleted from CephFS (and possibly when leveldb compacts), the OSD will stop responding to heartbeats and get marked down, then come back and start recovery and then other OSDs will have the same issue until client load on the cluster eases up then it settles down. Is there a way to have leveldb compact more frequently or cause it to come up for air more frequently and respond to heartbeats and process some IO? I thought splitting PGs would help, but we are still seeing the problem (previously ~20 PGs per OSD to now ~150). I still have some space on the SSDs that I can double, almost triple the journal, but not sure if that will help in this situation. The other issue I'm seeing is that some IO just gets stuck when the OSDs are getting marked down and coming back through the cluster. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 2