- ceph-users - lists.ceph.io

Re: PG inconsistent with empty inconsistent objects
by Richard Bade 27 Jan '21

27 Jan '21

Hi Everyone, I also have seen this inconsistent with empty when you do list-inconsistent-obj $ sudo ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent; 1 pgs not deep-scrubbed in time OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 17.7ff is active+clean+inconsistent, acting [232,242,34,280,266,21] PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time pg 17.1c2 not deep-scrubbed since 2021-01-15 02:46:16.271811 $ sudo rados list-inconsistent-obj 17.7ff --format=json-pretty { "epoch": 183807, "inconsistents": [] } Usually these are caused by read errors on the disks, but I've checked all osd hosts that are part of this osd and there's no smart or dmesg errors. Rich ------------------------------ > > Date: Sun, 17 Jan 2021 14:00:01 +0330 > From: Seena Fallah <seenafallah(a)gmail.com> > Subject: [ceph-users] Re: PG inconsistent with empty inconsistent > objects > To: "Alexander E. Patrakov" <patrakov(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > <CAK3+OmXvdC_x2R-Kox-ui4K3oSDvXh4o8ZeqYbztBUmqMYEAZw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > It's for a long time ago and I don't have the `ceph health detail` output! > > On Sat, Jan 16, 2021 at 9:42 PM Alexander E. Patrakov <patrakov(a)gmail.com> > wrote: > > > For a start, please post the "ceph health detail" output. > > > > сб, 19 дек. 2020 г. в 23:48, Seena Fallah <seenafallah(a)gmail.com>: > > > > > > Hi, > > > > > > I'm facing something strange! One of the PGs in my pool got inconsistent > > > and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty` > > > the `inconsistents` key was empty! What is this? Is it a bug in Ceph > > or..? > > > > > > Thanks. > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > -- > > Alexander E. Patrakov > > CV: http://u.pc.cd/wT8otalK > > >

4 5

radosgw not working - upgraded from mimic to octopus
by Youzhong Yang 27 Jan '21

27 Jan '21

I upgraded our ceph cluster (6 bare metal nodes, 3 rgw VMs) from v13.2.4 to v15.2.8. The mon, mgr, mds and osd daemons were all upgraded successfully, everything looked good. After the radosgw was upgraded, they refused to work, the log messages are at the end of this e-mail. Here are the things I tried: 1. I moved aside the pools for the rgw service, started from scratch (creating realm, zonegroup, zone, users), but when I tried to run 'radosgw-admin user create ...', it appeared to be stuck and never returned, other command like 'radosgw-admin period update --commit' also got stuck. 2. I rolled back radosgw to the old version v13.2.4, then everything works great again. What am I missing here? Is there anything extra that needs to be done for rgw after upgrading from mimic to octopus? Please kindly help. Thanks. --------------------- 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 898 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework: civetweb 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: port, val: 80 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:24:10.192-0500 7f638f79f9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:24:10.192-0500 7f638f79f9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:29:10.195-0500 7f638cbcd700 -1 Initialization timeout, failed to initialize 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 1541 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework: civetweb 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: port, val: 80 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:29:10.367-0500 7f4c213ba9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:29:25.883-0500 7f4c213ba9c0 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-01-24T09:29:25.883-0500 7f4c213ba9c0 0 ERROR: failed to distribute cache for coredumps.rgw.log:meta.history 2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 deferred set uid:gid to 64045:64045 (ceph:ceph) 2021-01-24T09:32:27.754-0500 7fcdac2bf9c0 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw, pid 978 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework: civetweb 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: port, val: 80 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: num_threads, val: 1024 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 0 framework conf key: request_timeout_ms, val: 50000 2021-01-24T09:32:27.758-0500 7fcdac2bf9c0 1 radosgw_Main not setting numa affinity 2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 1 robust_notify: If at first you don't succeed: (110) Connection timed out 2021-01-24T09:32:44.719-0500 7fcdac2bf9c0 0 ERROR: failed to distribute cache for coredumps.rgw.log:meta.history

1 1

Re: Balancing with upmap
by Francois Legrand 27 Jan '21

27 Jan '21

Nope ! Le 27/01/2021 à 17:40, Anthony D'Atri a écrit : > Do you have any override reweights set to values less than 1.0? > > The REWEIGHT column when you run `ceph osd df` > >> On Jan 27, 2021, at 8:15 AM, Francois Legrand <fleg(a)lpnhe.in2p3.fr> wrote: >> >> Hi all, >> I have a cluster with 116 disks (24 new disks of 16TB added in december and the rest of 8TB) running nautilus 14.2.16. >> I moved (8 month ago) from crush_compat to upmap balancing. >> But the cluster seems not well balanced, with a number of pgs on the 8TB disks varying from 26 to 52 ! And an occupation from 35 to 69%. >> The recent 16 TB disks are more homogeneous with 48 to 61 pgs and space between 30 and 43%. >> Last week, I realized that some osd were maybe not using upmap because I did a ceph osd crush weight-set ls and got (compat) as result. >> Thus I ran a ceph osd crush weight-set rm-compat which triggered some rebalancing. Now there is no more recovery for 2 days, but the cluster is still unbalanced. >> As far as I understand, upmap is supposed to reach an equal number of pgs on all the disks (I guess weighted by their capacity). >> Thus I would expect more or less 30 pgs on the 8TB disks and 60 on the 16TB and around 50% usage on all. Which is not the case (by far). >> The problem is that it impact the free available space in the pools (264Ti while there is more than 578Ti free in the cluster) because free space seems to be based on space available before the first osd will be full ! >> Is it normal ? Did I missed something ? What could I do ? >> >> F. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 27 Jan '21

27 Jan '21

On Wed, Jan 27, 2021 at 5:34 PM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Looks like there’s already a ticket open for AMQP SSL support: > https://tracker.ceph.com/issues/42902 (you opened it ;-)) > > I will give a try myself if I have some time, but don’t hold your breath > with lockdown and home schooling. Also I am not much of a C++ coder. > > I need to go over the logs with Tom Byrne to see why it is not working > properly. And perhaps I will be able to come up with a fix then. > > However this is what I have run into so far today: > > 1. After configuring a bucket with a topic using the non-SSL port, I tried > a couple of uploads to this bucket. They all hanged, which seemed like > something was very wrong, so I Ctrl-C’ed every time. After some time I > figured out from the RabbitMQ admin UI that Ceph was indeed connecting to > it, and the connections remained so I killed them from the UI. > sending the notification to the rabbitmq server is synchronous with the upload to the bucket. so, if the server is slow or not acking the notification, the upload request would hang. not that the upload itself is done first, but the reply to the client does not happen until rabbitmq server acks. would be great if you can share the radosgw logs. maybe the issue is related to the user/password method we use? we use: AMQP_SASL_METHOD_PLAIN one possible workaround would be to set "amqp-ack-level" to "none". in this case the radosgw does not wait for an ack in "pacific" you could use "persistent topics" where the notifications are sent asynchronously to the endpoint. 2. I then wrote a python script with Pika to consume the events, hoping > that would stop the blocking. I had some minor success with this. Usually > the first three or four uploaded files would generate events that I could > consume with my script. > the radosgw is waiting for an ack from the broker, not the end consumer, so this should not have mattered... did you actually see any notifications delivered to the consumer? > However, the rest would block for ever. I repeated this a couple of times > but always the same result. I noticed that after I stopped uploading, > removed the bucket and the topic, the connection from Ceph in the RabbitMQ > UI remained. I killed it but it came back seconds later from another port > on the Ceph cluster. I ended up playing whack-a-mole with this until no > more connections would be established from Ceph to RabbitMQ. I probably > killed a 100 or so of them. > once you remove the bucket there cannot be new notification sent. if you create the bucket again you may see notifications again (this is fixed in "pacific"). either way, even if the connection to the rabbitmq server would still be open, but no new notifications should be sent there. just having the connection should not be an issue but would be nice to fix that as well: https://tracker.ceph.com/issues/49033 3. After this I couldn’t get any events sent anymore. There is no more > blocking when uploading, files get written but nothing else happens. No > connections are made anymore from Ceph to RabbitMQ. > > Hope this helps… > yes, this is very helpful! > Best, > > Tom > > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 27 Jan 2021, at 13:04, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> Switching to non-SSL connections to RabbitMQ allowed us to get things >> working, although currently it’s not very reliable. >> > > can you please add more about that? what reliability issues did you see? > > >> I will open a new ticket over this if we can’t fix things ourselves. >> >> > this would be great. we have ssl support for kafka and http endpoint, so, > if you decide to give it a try you can look at them as examples. > and let me know if you have questions or need help. > > > >> I will open an issue on the tracker as soon as my account request has >> been approved :-) >> >> Best, >> >> Tom >> >> >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> >> >> On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < >> Tom.Schoonjans(a)rfi.ac.uk> wrote: >> >>> Hi Yuval, >>> >>> >>> I worked on this earlier today with Tom Byrne and I think I may be able >>> to provide some more information. >>> >>> I set up the RabbitMQ server myself, and created the exchange with type >>> ’topic’ before configuring the bucket. >>> >>> Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, >>> using certificates generated with Letsencrypt. >>> >>> >> it actually does. we don't support amqp over ssl. >> feel free to open a tracker for that - as we should probably support that! >> but note that it would probably be backported only to later versions than >> nautilus. >> >> >> >>> Many thanks, >>> >>> Tom >>> >>> >>> >>> Dr Tom Schoonjans >>> >>> Research Software Engineer - HPC and Cloud >>> >>> Rosalind Franklin Institute >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0FA >>> United Kingdom >>> >>> https://www.rfi.ac.uk >>> >>> The Rosalind Franklin Institute is a registered charity in England and >>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>> and Wales, No.11266143. Funded by UK Research and Innovation through >>> the Engineering and Physical Sciences Research Council. >>> >>> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>> >>> Hi Tom, >>> Did you create the exchange in rabbitmq? The RGW does not create it and >>> assume it is already created? >>> Could you increase the log level in RGW and see if there are more log >>> messages that have "AMQP" in them? >>> >>> Thanks, >>> >>> Yuval >>> >>> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >>> tom.byrne(a)stfc.ac.uk> wrote: >>> >>>> Hi all, >>>> >>>> We've been trying to get RGW Bucket notifications working with a >>>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>>> a message to the endpoint, the message never appears in the queue, and we >>>> get this error from in the RGW logs: >>>> >>>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>>> Endpoint >>>> URI: amqp://user:pass@host:5671 >>>> Topic: ceph-topic-test >>>> Exchange: ceph-test >>>> Ack Level: broker failed, with error: -4098 >>>> >>>> We've confirmed the URI is correct, and that the gateway host can send >>>> messages to the RabbitMQ via a standalone script (using the same >>>> information as in the URI). Does anyone have any hints about how to dig >>>> into this? >>>> >>>> Cheers, >>>> Tom >>>> >>>> This email and any attachments are intended solely for the use of the >>>> named recipients. If you are not the intended recipient you must not use, >>>> disclose, copy or distribute this email or any of its attachments and >>>> should notify the sender immediately and delete this email from your >>>> system. UK Research and Innovation (UKRI) has taken every reasonable >>>> precaution to minimise risk of this email or any attachments containing >>>> viruses or malware but the recipient should carry out its own virus and >>>> malware checks before opening the attachments. UKRI does not accept any >>>> liability for any losses or damages which the recipient may sustain due to >>>> presence of any viruses. Opinions, conclusions or other information in this >>>> message and attachments that are not related directly to UKRI business are >>>> solely those of the author and do not represent the views of UKRI. >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>> >> >

1 0

"ceph orch restart mgr" command creates mgr restart loop
by Chris Read 27 Jan '21

27 Jan '21

Greetings all... I'm busy testing out Ceph and have hit this troublesome bug while following the steps outlined here: https://docs.ceph.com/en/octopus/cephadm/monitoring/#configuring-ssl-tls-fo… When I issue the "ceph orch restart mgr" command, it appears the command is not cleared from a message queue somewhere (I'm still very unclear on many ceph specifics), and so each time the mgr process returns from restart it picks up the message again and keeps restarting itself forever (so far it's been stuck in this state for 45 minutes). Watching the logs we see this going on: $ ceph log last cephadm -w root@ceph-poc-000:~# ceph log last cephadm -w cluster: id: d23bc326-543a-11eb-bfe0-b324db228b6c health: HEALTH_OK services: mon: 5 daemons, quorum ceph-poc-000,ceph-poc-003,ceph-poc-004,ceph-poc-002,ceph-poc-001 (age 2h) mgr: ceph-poc-000.himivo(active, since 4s), standbys: ceph-poc-001.unjulx osd: 10 osds: 10 up (since 2h), 10 in (since 2h) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 10 GiB used, 5.4 TiB / 5.5 TiB avail pgs: 1 active+clean 2021-01-11T20:46:32.976606+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:32.980749+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:33.061519+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:39.156420+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:39.160618+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:39.242603+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:45.299953+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:45.304006+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:45.733495+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:51.871903+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:51.877107+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:51.976190+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:58.000720+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:58.006843+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:58.097163+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:47:04.188630+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:47:04.193501+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:47:04.285509+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:47:10.348099+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:47:10.352340+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:47:10.752243+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available And in the logs for the mgr instance itself we see it keep replaying the message over and over: $ docker logs -f ceph-d23bc326-543a-11eb-bfe0-b324db228b6c-mgr.ceph-poc-000.himivo debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 set uid:gid to 167:167 (ceph:ceph) debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process ceph-mgr, pid 1 debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 pidfile_write: ignore empty --pid-file debug 2021-01-11T20:47:31.414+0000 7f48b0d0d200 1 mgr[py] Loading python module 'alerts' debug 2021-01-11T20:47:31.486+0000 7f48b0d0d200 1 mgr[py] Loading python module 'balancer' debug 2021-01-11T20:47:31.542+0000 7f48b0d0d200 1 mgr[py] Loading python module 'cephadm' debug 2021-01-11T20:47:31.742+0000 7f48b0d0d200 1 mgr[py] Loading python module 'crash' debug 2021-01-11T20:47:31.798+0000 7f48b0d0d200 1 mgr[py] Loading python module 'dashboard' debug 2021-01-11T20:47:32.258+0000 7f48b0d0d200 1 mgr[py] Loading python module 'devicehealth' debug 2021-01-11T20:47:32.306+0000 7f48b0d0d200 1 mgr[py] Loading python module 'diskprediction_local' debug 2021-01-11T20:47:32.498+0000 7f48b0d0d200 1 mgr[py] Loading python module 'influx' debug 2021-01-11T20:47:32.550+0000 7f48b0d0d200 1 mgr[py] Loading python module 'insights' debug 2021-01-11T20:47:32.598+0000 7f48b0d0d200 1 mgr[py] Loading python module 'iostat' debug 2021-01-11T20:47:32.642+0000 7f48b0d0d200 1 mgr[py] Loading python module 'k8sevents' debug 2021-01-11T20:47:33.034+0000 7f48b0d0d200 1 mgr[py] Loading python module 'localpool' debug 2021-01-11T20:47:33.082+0000 7f48b0d0d200 1 mgr[py] Loading python module 'orchestrator' debug 2021-01-11T20:47:33.266+0000 7f48b0d0d200 1 mgr[py] Loading python module 'osd_support' debug 2021-01-11T20:47:33.310+0000 7f48b0d0d200 1 mgr[py] Loading python module 'pg_autoscaler' debug 2021-01-11T20:47:33.382+0000 7f48b0d0d200 1 mgr[py] Loading python module 'progress' debug 2021-01-11T20:47:33.442+0000 7f48b0d0d200 1 mgr[py] Loading python module 'prometheus' debug 2021-01-11T20:47:33.794+0000 7f48b0d0d200 1 mgr[py] Loading python module 'rbd_support' debug 2021-01-11T20:47:33.870+0000 7f48b0d0d200 1 mgr[py] Loading python module 'restful' debug 2021-01-11T20:47:34.086+0000 7f48b0d0d200 1 mgr[py] Loading python module 'rook' debug 2021-01-11T20:47:34.606+0000 7f48b0d0d200 1 mgr[py] Loading python module 'selftest' debug 2021-01-11T20:47:34.654+0000 7f48b0d0d200 1 mgr[py] Loading python module 'status' debug 2021-01-11T20:47:34.710+0000 7f48b0d0d200 1 mgr[py] Loading python module 'telegraf' debug 2021-01-11T20:47:34.762+0000 7f48b0d0d200 1 mgr[py] Loading python module 'telemetry' debug 2021-01-11T20:47:35.034+0000 7f48b0d0d200 1 mgr[py] Loading python module 'test_orchestrator' debug 2021-01-11T20:47:35.146+0000 7f48b0d0d200 1 mgr[py] Loading python module 'volumes' debug 2021-01-11T20:47:35.294+0000 7f48b0d0d200 1 mgr[py] Loading python module 'zabbix' debug 2021-01-11T20:47:35.366+0000 7f489df86700 0 ms_deliver_dispatch: unhandled message 0x55d57b71a420 mon_map magic: 0 v1 from mon.0 v2: 10.128.18.16:3300/0 debug 2021-01-11T20:47:35.758+0000 7f489df86700 1 mgr handle_mgr_map Activating! debug 2021-01-11T20:47:35.758+0000 7f489df86700 1 mgr handle_mgr_map I am now activating debug 2021-01-11T20:47:35.770+0000 7f484ca14700 0 [balancer DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.770+0000 7f484ca14700 1 mgr load Constructed class from module: balancer debug 2021-01-11T20:47:35.774+0000 7f484ca14700 0 [cephadm DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.874+0000 7f484ca14700 1 mgr load Constructed class from module: cephadm debug 2021-01-11T20:47:35.874+0000 7f484ca14700 0 [crash DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.874+0000 7f484ca14700 1 mgr load Constructed class from module: crash debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [dashboard DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: dashboard debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: devicehealth debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [iostat DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: iostat debug 2021-01-11T20:47:35.886+0000 7f484ca14700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.886+0000 7f484ca14700 1 mgr load Constructed class from module: orchestrator debug 2021-01-11T20:47:35.890+0000 7f484ca14700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.890+0000 7f484ca14700 1 mgr load Constructed class from module: pg_autoscaler debug 2021-01-11T20:47:35.890+0000 7f484ca14700 0 [progress DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.890+0000 7f484ca14700 1 mgr load Constructed class from module: progress debug 2021-01-11T20:47:35.898+0000 7f484ca14700 0 [prometheus DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.910+0000 7f484ca14700 1 mgr load Constructed class from module: prometheus debug 2021-01-11T20:47:35.914+0000 7f484ca14700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5) [11/Jan/2021:20:47:35] ENGINE Bus STARTING Warning: Permanently added 'ceph-poc-000,10.128.18.16' (ECDSA) to the list of known hosts. Warning: Permanently added 'ceph-poc-002,10.128.19.248' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.974+0000 7f484ca14700 1 mgr load Constructed class from module: rbd_support Warning: Permanently added 'ceph-poc-001,10.128.28.223' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [restful DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: restful debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [status DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: status debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [telemetry DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: telemetry debug 2021-01-11T20:47:35.982+0000 7f486f3cc700 0 [restful WARNING root] server not running: no certificate configured debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [volumes DEBUG root] setting log level based on debug_mgr: WARNING (1/5) Warning: Permanently added 'ceph-poc-004,10.128.31.60' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.994+0000 7f484ca14700 1 mgr load Constructed class from module: volumes Warning: Permanently added 'ceph-poc-003,10.128.30.127' (ECDSA) to the list of known hosts. CherryPy Checker: The Application mounted at '' has an empty config. [11/Jan/2021:20:47:36] ENGINE Serving on http://:::9283 [11/Jan/2021:20:47:36] ENGINE Bus STARTED debug 2021-01-11T20:47:36.778+0000 7f484d215700 0 log_channel(audit) log [DBG] : from='client.19140 -' entity='client.admin' cmd=[{"prefix": "orch", "action": "restart", "service_name": "mgr", "target": ["mon-mgr", ""]}]: dispatch debug 2021-01-11T20:47:36.778+0000 7f484ca14700 0 log_channel(cephadm) log [INF] : Restart service mgr debug 2021-01-11T20:47:36.806+0000 7f484d215700 0 log_channel(cluster) log [DBG] : pgmap v3: 1 pgs: 1 active+clean; 0 B data, 208 MiB used, 5.4 TiB / 5.5 TiB avail debug 2021-01-11T20:47:37.406+0000 7f489ef88700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 debug 2021-01-11T20:47:37.406+0000 7f489ef88700 -1 mgr handle_mgr_signal *** Got signal Terminated *** Is there anyway to manually clear this stuck message? Will it eventually timeout by itself? Either way it's a pretty unpleasant surprise to find when following the docs so early in the process. Thanks, Chris

2 1

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 27 Jan '21

27 Jan '21

On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > Switching to non-SSL connections to RabbitMQ allowed us to get things > working, although currently it’s not very reliable. > can you please add more about that? what reliability issues did you see? > I will open a new ticket over this if we can’t fix things ourselves. > > this would be great. we have ssl support for kafka and http endpoint, so, if you decide to give it a try you can look at them as examples. and let me know if you have questions or need help. > I will open an issue on the tracker as soon as my account request has been > approved :-) > > Best, > > Tom > > > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> I worked on this earlier today with Tom Byrne and I think I may be able >> to provide some more information. >> >> I set up the RabbitMQ server myself, and created the exchange with type >> ’topic’ before configuring the bucket. >> >> Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, >> using certificates generated with Letsencrypt. >> >> > it actually does. we don't support amqp over ssl. > feel free to open a tracker for that - as we should probably support that! > but note that it would probably be backported only to later versions than > nautilus. > > > >> Many thanks, >> >> Tom >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> Hi Tom, >> Did you create the exchange in rabbitmq? The RGW does not create it and >> assume it is already created? >> Could you increase the log level in RGW and see if there are more log >> messages that have "AMQP" in them? >> >> Thanks, >> >> Yuval >> >> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >> tom.byrne(a)stfc.ac.uk> wrote: >> >>> Hi all, >>> >>> We've been trying to get RGW Bucket notifications working with a >>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>> a message to the endpoint, the message never appears in the queue, and we >>> get this error from in the RGW logs: >>> >>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>> Endpoint >>> URI: amqp://user:pass@host:5671 >>> Topic: ceph-topic-test >>> Exchange: ceph-test >>> Ack Level: broker failed, with error: -4098 >>> >>> We've confirmed the URI is correct, and that the gateway host can send >>> messages to the RabbitMQ via a standalone script (using the same >>> information as in the URI). Does anyone have any hints about how to dig >>> into this? >>> >>> Cheers, >>> Tom >>> >>> This email and any attachments are intended solely for the use of the >>> named recipients. If you are not the intended recipient you must not use, >>> disclose, copy or distribute this email or any of its attachments and >>> should notify the sender immediately and delete this email from your >>> system. UK Research and Innovation (UKRI) has taken every reasonable >>> precaution to minimise risk of this email or any attachments containing >>> viruses or malware but the recipient should carry out its own virus and >>> malware checks before opening the attachments. UKRI does not accept any >>> liability for any losses or damages which the recipient may sustain due to >>> presence of any viruses. Opinions, conclusions or other information in this >>> message and attachments that are not related directly to UKRI business are >>> solely those of the author and do not represent the views of UKRI. >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >> >

1 0

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 26 Jan '21

26 Jan '21

On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > I worked on this earlier today with Tom Byrne and I think I may be able to > provide some more information. > > I set up the RabbitMQ server myself, and created the exchange with type > ’topic’ before configuring the bucket. > > Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, > using certificates generated with Letsencrypt. > > it actually does. we don't support amqp over ssl. feel free to open a tracker for that - as we should probably support that! but note that it would probably be backported only to later versions than nautilus. > Many thanks, > > Tom > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > Hi Tom, > Did you create the exchange in rabbitmq? The RGW does not create it and > assume it is already created? > Could you increase the log level in RGW and see if there are more log > messages that have "AMQP" in them? > > Thanks, > > Yuval > > On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < > tom.byrne(a)stfc.ac.uk> wrote: > >> Hi all, >> >> We've been trying to get RGW Bucket notifications working with a RabbitMQ >> endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate >> with the rabbitMQ server just fine, but when RGW tries to send a message to >> the endpoint, the message never appears in the queue, and we get this error >> from in the RGW logs: >> >> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >> Endpoint >> URI: amqp://user:pass@host:5671 >> Topic: ceph-topic-test >> Exchange: ceph-test >> Ack Level: broker failed, with error: -4098 >> >> We've confirmed the URI is correct, and that the gateway host can send >> messages to the RabbitMQ via a standalone script (using the same >> information as in the URI). Does anyone have any hints about how to dig >> into this? >> >> Cheers, >> Tom >> >> This email and any attachments are intended solely for the use of the >> named recipients. If you are not the intended recipient you must not use, >> disclose, copy or distribute this email or any of its attachments and >> should notify the sender immediately and delete this email from your >> system. UK Research and Innovation (UKRI) has taken every reasonable >> precaution to minimise risk of this email or any attachments containing >> viruses or malware but the recipient should carry out its own virus and >> malware checks before opening the attachments. UKRI does not accept any >> liability for any losses or damages which the recipient may sustain due to >> presence of any viruses. Opinions, conclusions or other information in this >> message and attachments that are not related directly to UKRI business are >> solely those of the author and do not represent the views of UKRI. >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

1 0

RGW Bucket notification troubleshooting
by Byrne, Thomas (STFC,RAL,SC) 26 Jan '21

26 Jan '21

Hi all, We've been trying to get RGW Bucket notifications working with a RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate with the rabbitMQ server just fine, but when RGW tries to send a message to the endpoint, the message never appears in the queue, and we get this error from in the RGW logs: 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) Endpoint URI: amqp://user:pass@host:5671 Topic: ceph-topic-test Exchange: ceph-test Ack Level: broker failed, with error: -4098 We've confirmed the URI is correct, and that the gateway host can send messages to the RabbitMQ via a standalone script (using the same information as in the URI). Does anyone have any hints about how to dig into this? Cheers, Tom This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

2 1

Re: Where has my capacity gone?
by George Yil 26 Jan '21

26 Jan '21

Sorry for replying late :(. And thanks for the tips. This is a fresh cluster. And I didn’t think data distribution would be a problem. Is this normal? Below is the ceph osd df output. The related pool is hdd only (prod.rgw.buckets.data). I guess there is variance but I couldn’t get the reason. Is it because of PG number which I get help from pg-calculator <https://ceph.io/pgcalc/>? Or is this expected ceph behaviour? #ceph osd df https://pastebin.ubuntu.com/p/ZmQZsGYpr7/ <https://pastebin.ubuntu.com/p/7C9zpXYntR/> I am also sharing related cluster information. Any suggestion would be appreciated. #ceph df https://pastebin.ubuntu.com/p/sXpf99zhnV/ <https://pastebin.ubuntu.com/p/sXpf99zhnV/> #ceph detail df https://pastebin.ubuntu.com/p/dwvwBnnBmv/ <https://pastebin.ubuntu.com/p/dwvwBnnBmv/> #ceph osd pool ls detail https://pastebin.ubuntu.com/p/c2KQD5CGMV/ <https://pastebin.ubuntu.com/p/c2KQD5CGMV/> #crush rules https://pastebin.ubuntu.com/p/X6WsZhV3Zz/ <https://pastebin.ubuntu.com/p/X6WsZhV3Zz/> Thanks. > On 26 Jan 2021, at 11:18, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > ceph osd df | sort -nk8 > >> On Jan 25, 2021, at 11:22 PM, George Yil <georgeyil75(a)gmail.com> wrote: >> >> Hi, >> >> I have a ceph nautilus (14.2.9) cluster with 10 nodes. Each node has >> 19x16TB disks attached. >> >> I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured >> as EC 8+2 (jerasure). >> ceph df shows 2.1PiB MAX AVAIL space. >> >> Then I configured radosgw as a secondary zone and 100TiB of S3 data is >> replicated. >> >> But weirdly enough ceph df shows 1.8PiB MAX AVAIL for the same pool. But >> there is only 100TiB of written data. ceph df also confirms it. I can not >> figure out where 200TiB capacity is gone. >> >> Would someone please tell me what I am missing? >> >> Thanks. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Ceph-mds using a lot of buffer_anon memory
by David Turner 26 Jan '21

26 Jan '21

We just upgraded a cephfs cluster from 12.2.12 to 14.2.11. Our next step is to upgrade to 14.2.16 to troubleshoot this issue, but I thought I'd reach out here first if anyone had any ideas. The clients are still running an older version of ceph-fuse 12.2.4 and it's very difficult to remount all of them. Would probably take a team of us a couple days to restart all of them. I've looked around online and release notes and all of the known memory leaks I've been able to find have been fixed prior to version 14.2.11 so this would be an unknown memory leak. All of the memory is in use in [1] buffer_anon. If left unchecked it will use up over 700GB of memory within 24 hours. On an identical cluster with an equivalent workload still running 12.2.12 [2] buffer_anon information is much healthier. Without any other options or ideas our plan is to upgrade the cluster to 14.2.16 first and then upgrade the clients. Has anyone else come across high buffer_anon usage? [1] "buffer_anon": { "items": 33756758, "bytes": 135025912897 }, [2] "buffer_anon": { "items": 636, "bytes": 273118 },

1 0