- ceph-users - lists.ceph.io

rgw luminous 12.2.12
by Marc Roos 15 Aug '19

15 Aug '19

FYI I just had an issue with radosgw / civetweb. Wanted to upload 40 MB file, it started with poor transfer speed, which was decreasing over time to 20KB/s when I stopped the transfer. I had to kill radosgw and start it to get 'normal' operation back.

1 0

"Signature check failed" from certain clients
by Peter Sarossy 15 Aug '19

15 Aug '19

hey folks, I spent the past 2 hours digging through the forums and similar sources with no luck.. I use ceph storage for docker stacks, and this issue has taken the whole thing down as I cannot mount their volumes back... Starting yesterday, some of my nodes cannot mount the filesystem and it just hangs on the mount command, while the logs are full of the messages below. It doesn't matter which MDS node is active for some clients it just works, and for others it doesn't. Any hints? 2019-08-15 02:20:55.230 7f6e80e47700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13406 cs=3203 l=0).handle_message_footer Signature check failed 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 Message signature does not match contents. 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1Signature on message: 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 sig: 1045888080092928376 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1Locally calculated signature: 2019-08-15 02:20:56.254 7f6e7fe45700 0 SIGN: MSG 1 sig_check:13982737427498638198 2019-08-15 02:20:56.254 7f6e7fe45700 0 Signature failed. 2019-08-15 02:20:56.254 7f6e7fe45700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13410 cs=3205 l=0).handle_message_footer Signature check failed 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 Message signature does not match contents. 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1Signature on message: 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 sig: 1045888080092928376 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1Locally calculated signature: 2019-08-15 02:20:57.246 7f6e80646700 0 SIGN: MSG 1 sig_check:13982737427498638198 2019-08-15 02:20:57.246 7f6e80646700 0 Signature failed. 2019-08-15 02:20:57.246 7f6e80646700 0 --1- [v2:10.0.0.8:6800/859279545,v1:10.0.0.8:6801/859279545] >> v1:10.0.0.1:0/844039356 conn(0x564928535000 0x56492851b800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=13414 cs=3207 l=0).handle_message_footer Signature check failed Cheers, Peter

2 1

Failing heartbeats when no backfill is running
by Lorenz Kiefner 14 Aug '19

14 Aug '19

Dear ceph-users, I'm having trouble with heartbeats, there are a lot of "heartbeat_check: no reply from..."-messages in my logs when there is no backfilling or repairing running (yes, it's failing when all PGs are active+clean). Only a few OSDs are failing, even when there are several OSDs on the same host. Doesn't look like a network issue to me. When I set the flags "nobackfill" and "norecover" there are no heartbeat issues. My cluster is kind of heterogenous: it's ARMv7 and x86_64, connected mostly via VPN. Some hosts are Debian Stretch, so I'm still using Ceph Luminous (12.2.12). Is there someone having the same issue? What could be the next steps to debug? Any ideas? Thanks for any help! Lorenz

3 4

Re: Fw: Ceph-Deploy
by Paul Emmerich 14 Aug '19

14 Aug '19

You cannot add OSDs to a cluster that is offline. -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Aug 14, 2019 at 5:43 PM Cory Mueller <corymueller(a)hotmail.com> wrote: > > Good Morning, > > > I was wondering if you were aware of a way to add more OSD nodes to an existing OFFLINE cluster. > > Ive tried to use ceph-deploy new > ceph-deploy admin > ceph-deploy install > > and none of them allow me to join the cluster. I tried install the ceph packages beforehand and that still didnt work. The one part that im noticing causing the most issues is the ASC key. > I installed the asc key the same way i did on the previous 5 nodes before, yet when I take it offline to connect it to cluster, it still tries to install one from online which wont work in a offline environment. > > I tried copying the asc key from the other nodes in the cluster that are working, and that didnt change anything either. > > > I was hoping you had a few steps that I should be following to bypass this. > > Thank you in advance > Cory Mueller > > > ________________________________________ > From: Sage Weil <sweil(a)redhat.com> > Sent: August 14, 2019 2:20 PM > To: Cory Mueller > Subject: Re: Ceph-Deploy > > Try emailing ceph-users(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

rbd image usage per osd
by Frank R 14 Aug '19

14 Aug '19

I have an all RBD pool/cluster. I am interested in tracking how much disk space is being used by each RBD image on every OSD drive. The OSDs are Filestore. Does anyone know of any existing scripts that accomplish this task? If not, what commands can be used to generate this info?

2 1

Fw: Ceph-Deploy
by Cory Mueller 14 Aug '19

14 Aug '19

Good Morning, I was wondering if you were aware of a way to add more OSD nodes to an existing OFFLINE cluster. Ive tried to use ceph-deploy new ceph-deploy admin ceph-deploy install and none of them allow me to join the cluster. I tried install the ceph packages beforehand and that still didnt work. The one part that im noticing causing the most issues is the ASC key. I installed the asc key the same way i did on the previous 5 nodes before, yet when I take it offline to connect it to cluster, it still tries to install one from online which wont work in a offline environment. I tried copying the asc key from the other nodes in the cluster that are working, and that didnt change anything either. I was hoping you had a few steps that I should be following to bypass this. Thank you in advance Cory Mueller ________________________________________ From: Sage Weil <sweil(a)redhat.com> Sent: August 14, 2019 2:20 PM To: Cory Mueller Subject: Re: Ceph-Deploy Try emailing ceph-users(a)ceph.io

1 0

BlueStore _txc_add_transaction errors (possibly related to bug #38724)
by Florian Haas 14 Aug '19

14 Aug '19

Hi everyone, it seems there have been several reports in the past related to BlueStore OSDs crashing from unhandled errors in _txc_add_transaction: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-April/034444.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032172.html http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031960.ht… http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-December/031964.ht… Bug #38724 tracks this, has been fixed in master with https://github.com/ceph/ceph/pull/27929, and is pending backports (and, I dare say, is *probably* misclassified as being only minor, as this does cause potential data loss as soon as it affects enough OSDs simultaneously): https://tracker.ceph.com/issues/38724 We just ran into a similar issue with a couple of BlueStore OSDs that we recently added to a Luminous (12.2.12) cluster that was upgraded from Jewel, and hence, still largely runs on FileStore. I say similar because evidently other people reporting this problem have been running into ENOENT (No such file or directory) or ENOTEMPTY (Directory not empty); for us it's interestingly E2BIG (Argument list too long): https://tracker.ceph.com/issues/38724#note-26 So I'm wondering if someone could shed light on these questions: * Is this the same issue as that which https://github.com/ceph/ceph/pull/27929 fixes? * Thus, since https://github.com/ceph/ceph/pull/29115 (the Nautilus backport for that fix) has been merged, but is not yet included in a release, do *Nautilus* users get a fix in the upcoming 14.2.3 release, and once they update, would this bug go away with no further intervention required? * For users on *Luminous*, since https://tracker.ceph.com/issues/39694 (the Luminous version of 38724) says "non-trivial backport", is it fair to say that a fix might still take a while for that release? * Finally, are Luminous users safe from this bug if they keep using, or revert to, FileStore? Thanks in advance for your thoughts! Please keep Erik CC'd on your reply. Cheers, Florian

4 6

Ceph Tech Talk for August 22nd
by Mike Perez 13 Aug '19

13 Aug '19

Hi everyone, We had a change with our original planned tech talk for August 22nd. This is short notice, but if you have a presentation topic you want to give and discuss with the community, now is a good opportunity. https://ceph.com/ceph-tech-talks/ -- Mike Perez (thingee)

1 0

Planning Ceph User Survey for 2019
by Mike Perez 12 Aug '19

12 Aug '19

Hi everyone, It's that time of the year again for us to form this year's user survey. The user survey gives the Ceph community insight with how people are using Ceph and where we should be spending our efforts. You can see last year's survey in this blog post: https://ceph.com/ceph-blog/ceph-user-survey-2018-results To start, let's use the Ceph pad to look at the previous set of questions and the answer options that will be available. https://pad.ceph.com/p/user-survey-2019 Let the discussing begin! -- Mike Perez (thingee)

1 0

Re: Built-in HA?
by Volodymyr Litovka 09 Aug '19

09 Aug '19

Hi Robert, > Another option is if both RDMA ports are on the same card, then you can do RDMA with a bond. This does not work if you have two separate cards. Yes, we recently talked to Mellanox and their engineers also recommend this way. > As far as your questions go, my guess would be that you would want to have the different NICs in different broadcast domains yes, the idea was to use two public networks on two different NICs with addresses from different subnets. It is possible to set 2+ networks in Ceph configuration, but it’s unclear how Ceph is going to use this configuration. > or set up Source Based Routing and bind the source port on the connection (not the easiest, but allows you to have multiple NICs in the same broadcast domain). I don't have experience with Ceph in this type of configuration. it’s too complicated and, frankly, when you’re trying to reach max performance, Source Based Routing is a bit from another area :-) At the end of the all, we’re going to test bonding of two ports on same NIC. Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison > On Fri, Aug 2, 2019 at 9:41 AM Volodymyr Litovka <doka.ua(a)gmx.com <mailto:doka.ua@gmx.com>> wrote: > Dear colleagues, > > at the moment, we use Ceph in routed environment (OSPF, ECMP) and everything is ok, reliability is high and there is nothing to complain about. But for hardware reasons (to be more precise - RDMA offload), we are faced with the need to operate Ceph directly on physical interfaces. > > According to documentation, "We generally recommend that dual-NIC systems either be configured with two IPs on the same network, or bonded." > > Q1: Did anybody test and can explain, how Ceph will behave in first scenario (two IPs on the same network)? I think this configuration require just one statement in 'public network' (where both interfaces reside)? How it will distribute traffic between links, how it will detect link failures and how it will switchover? > > Q2: Did anybody test a bit another scenario - both NICs have addresses in different networks and Ceph configuration contain two 'public networks'? Questions are same - how Ceph distributes traffic between links and how it recovers from link failures? > > Thank you. > > -- > Volodymyr Litovka > "Vision without Execution is Hallucination." -- Thomas Edison > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>

1 0