- ceph-users - lists.ceph.io

Long heartbeat ping times
by Frank Schilder 12 Oct '20

12 Oct '20

Dear all, occasionally, I find messages like Health check update: Long heartbeat ping times on front interface seen, longest is 1043.153 msec (OSD_SLOW_PING_TIME_FRONT) in the cluster log. Unfortunately, I seem to be unable to find out which OSDs were affected (a-posteriori). I cannot find related messages in any OSD log and the messages I find in /var/log/messages do not contain IP addresses or OSD IDs. Is there a way to find out which OSDs/hosts were the problem after health status is back to healthy? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Re: Cluster under stress - flapping OSDs?
by Kristof Coucke 12 Oct '20

12 Oct '20

Diving into the different logging and searching for answers, I came across the following: PG_DEGRADED Degraded data redundancy: 2101057/10339536570 objects degraded (0.020%), 3 pgs degraded, 3 pgs undersized pg 1.4b is stuck undersized for 63114.227655, current state active+undersized+degraded+remapped+backfilling, last acting [62,20,33,25,97,2,159,2147483647,88] pg 1.115 is stuck undersized for 67017.759147, current state active+undersized+degraded+remapped+backfilling, last acting [2147483647,6,28,48,171,160,51,7,84] pg 1.1ec is stuck undersized for 67017.772311, current state active+undersized+degraded+remapped+backfilling, last acting [65,82,2147483647,161,6,36,105,106,48] Note the PG# 2147483647... That doesn't seem correct. Any ideas?

2 1

Re: Cluster under stress - flapping OSDs?
by Kristof Coucke 12 Oct '20

12 Oct '20

I'll answer it myself: When CRUSH fails to find enough OSDs to map to a PG, it will show as a 2147483647 which is ITEM_NONE or no OSD found.

1 0

Re: Ubuntu 20 with octopus
by Seena Fallah 12 Oct '20

12 Oct '20

I've seen this PR that reverts the latest ubuntu version from 20.04 to 18.04 because of some failures! Are there any updates on this? https://github.com/ceph/ceph/pull/35110 On Mon, Oct 12, 2020 at 4:11 AM Robert Ruge <robert.ruge(a)deakin.edu.au> wrote: > I am using Ubuntu 20.04 LTS for a five node 1PB cephfs setup with no > problems that I can attribute to using Ubuntu 20. > > Regards > Robert Ruge > > > > -----Original Message----- > From: Seena Fallah <seenafallah(a)gmail.com> > Sent: Monday, 12 October 2020 11:35 AM > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Ubuntu 20 with octopus > > The main reason I asked is because I don’t see any Ubuntu 20 in this doc > https://docs.ceph.com/en/latest/start/os-recommendations/ > > On Mon, Oct 12, 2020 at 4:01 AM Seena Fallah <seenafallah(a)gmail.com> > wrote: > > > Hi all, > > > > Does anyone has any production cluster with ubuntu 20 (focal) or any > > suggestion or any bugs that prevents to deploy Ceph octopus on Ubuntu 20? > > > > Thanks. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > Important Notice: The contents of this email are intended solely for the > named addressee and are confidential; any unauthorised use, reproduction or > storage of the contents is expressly prohibited. If you have received this > email in error, please delete it and any attachments immediately and advise > the sender by return email or telephone. > > Deakin University does not warrant that this email and any attachments are > error or virus free. >

2 1

Cluster under stress - flapping OSDs?
by Kristof Coucke 12 Oct '20

12 Oct '20

Hi all, We're now having trouble over a week with our Ceph cluster. Short info regarding our situation: - Original cluster had 10 OSD nodes, each having 16 OSDs - Expansion was necessary, so another 6 nodes have been added - Version: 14.2.11 Last week we saw heavily loaded OSD servers, after help here we identified the disk load being too high due to compaction of the Rocks.db. Taking the disk offline and running ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/xxxx compact does take the load away temporarily as mentioned. Most of the new disks still have a weight of 0 as we want to get the system stable first, but there is something I simply don't understand. When setting the following flags: noout, norecover, nobackfill and norebalance prior taking the disk offline for compaction, we still get a raise in PGs degraded after the OSD is back marked as "up". This night, a flapping OSD was also temporarily marked as "offline", I assume because it was heavily loaded, causing again a rise in degraded PGs. I know that there is a flag "nodown", but I've never used it. Reading the docs, it states these flags are "temporary" and the blocking action will be performed anyhow afterwards... So I have a few questions: 1. Why is the cluster marking PGs as "degraded" and indicating degraded data redundancy, while this was not the case before? It rises (2196398/10339524249 objects degraded (0.021%)) and I simply cannot understand why it keeps going up... 2. The flag "nodown". Can I use this to prevent the flapping... I don't want to get a deeper mess. As far as I understood, it would help in our case as the OSDs are heavily used. 3. Is it a good idea to start adding the other disks as well (slowly increasing their weight)? Thansk, Kristof

1 0

Cephdeploy support
by Amudhan P 11 Oct '20

11 Oct '20

Hi, Future releases of Ceph support cephdeploy or only Cephadm will be the choice. Thanks, Amudhan

1 0

Is cephfs multi-volume support stable?
by Alexander E. Patrakov 11 Oct '20

11 Oct '20

Hello, I found that documentation on the Internet on the question whether I can safely have two instances of cephfs in my cluster is inconsistent. For the record, I don't use snapshots. FOSDEM 19 presentation by Sage Weil: https://archive.fosdem.org/2019/schedule/event/ceph_project_status_update/a… Slide 25 is specifically devoted to this topic, and declares multi-volume support as stable. But, https://docs.ceph.com/en/nautilus/cephfs/experimental-features/ declares that multiple filesystems in the same cluster are an experimental feature, and the "latest" version of the same doc makes the same claim. What should I believe - the presentation or the official docs? -- Alexander E. Patrakov CV: http://pc.cd/PLz7

2 1

Q on enabling application on the pool
by Void Star Nill 11 Oct '20

11 Oct '20

Hello, What is the necessity for enabling the application on the pool? As per the documentation, we need to enable application before using the pool. However, in my case, I have a single pool running on the cluster used for RBD. I am able to run all RBD operations on the pool even if I dont enable the application. So what is the downside of not enabling the application on a pool? Regards, Shridhar

1 0

Monitor recovery
by Brian Topping 10 Oct '20

10 Oct '20

Hello experts, I have accidentally created a situation where the only monitor in a cluster has been moved to a new node without it’s /var/lib/ceph contents. Not realizing what I had done, I decommissioned the original node, but still have the contents of it’s /var/lib/ceph. Can I shut down the monitor running on the new node, copy monitor data from the original node to the new node and restart the monitor? Or is there information in the monitor database that is tied to the original node? If that’s the case, I suspect I need to somehow recommission the original node. Thanks for any feedback on this situation! Brian

2 2

How to clear Health Warning status?
by Tecnología CHARNE.NET 10 Oct '20

10 Oct '20

Hello! Today, I started the morning with a WARNING STATUS on our Ceph cluster. # ceph health detail HEALTH_WARN Too many repaired reads on 1 OSDs [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs osd.67 had 399911 reads repaired I made "ceph osd out 67" and PGs where migrated to another OSDs. I stopped the osd.67 daemon, inspected the logs, etc... Then I started the daemon and made "# ceph osd in 67". OSD started backfilling with some PGs and no other error appeared in the rest of the day, but Warning status still remains. Can I clear it? Shoud I remove the osd and start with a new one? Thanks in advance for your time! Javier.-

2 2