January 2021 - ceph-users

Re: RGW Bucket notification troubleshooting

by Yuval Lifshitz

On Wed, Jan 27, 2021 at 5:34 PM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Looks like there’s already a ticket open for AMQP SSL support: > https://tracker.ceph.com/issues/42902 (you opened it ;-)) > > I will give a try myself if I have some time, but don’t hold your breath > with lockdown and home schooling. Also I am not much of a C++ coder. > > I need to go over the logs with Tom Byrne to see why it is not working > properly. And perhaps I will be able to come up with a fix then. > > However this is what I have run into so far today: > > 1. After configuring a bucket with a topic using the non-SSL port, I tried > a couple of uploads to this bucket. They all hanged, which seemed like > something was very wrong, so I Ctrl-C’ed every time. After some time I > figured out from the RabbitMQ admin UI that Ceph was indeed connecting to > it, and the connections remained so I killed them from the UI. > sending the notification to the rabbitmq server is synchronous with the upload to the bucket. so, if the server is slow or not acking the notification, the upload request would hang. not that the upload itself is done first, but the reply to the client does not happen until rabbitmq server acks. would be great if you can share the radosgw logs. maybe the issue is related to the user/password method we use? we use: AMQP_SASL_METHOD_PLAIN one possible workaround would be to set "amqp-ack-level" to "none". in this case the radosgw does not wait for an ack in "pacific" you could use "persistent topics" where the notifications are sent asynchronously to the endpoint. 2. I then wrote a python script with Pika to consume the events, hoping > that would stop the blocking. I had some minor success with this. Usually > the first three or four uploaded files would generate events that I could > consume with my script. > the radosgw is waiting for an ack from the broker, not the end consumer, so this should not have mattered... did you actually see any notifications delivered to the consumer? > However, the rest would block for ever. I repeated this a couple of times > but always the same result. I noticed that after I stopped uploading, > removed the bucket and the topic, the connection from Ceph in the RabbitMQ > UI remained. I killed it but it came back seconds later from another port > on the Ceph cluster. I ended up playing whack-a-mole with this until no > more connections would be established from Ceph to RabbitMQ. I probably > killed a 100 or so of them. > once you remove the bucket there cannot be new notification sent. if you create the bucket again you may see notifications again (this is fixed in "pacific"). either way, even if the connection to the rabbitmq server would still be open, but no new notifications should be sent there. just having the connection should not be an issue but would be nice to fix that as well: https://tracker.ceph.com/issues/49033 3. After this I couldn’t get any events sent anymore. There is no more > blocking when uploading, files get written but nothing else happens. No > connections are made anymore from Ceph to RabbitMQ. > > Hope this helps… > yes, this is very helpful! > Best, > > Tom > > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 27 Jan 2021, at 13:04, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> Switching to non-SSL connections to RabbitMQ allowed us to get things >> working, although currently it’s not very reliable. >> > > can you please add more about that? what reliability issues did you see? > > >> I will open a new ticket over this if we can’t fix things ourselves. >> >> > this would be great. we have ssl support for kafka and http endpoint, so, > if you decide to give it a try you can look at them as examples. > and let me know if you have questions or need help. > > > >> I will open an issue on the tracker as soon as my account request has >> been approved :-) >> >> Best, >> >> Tom >> >> >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> >> >> On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < >> Tom.Schoonjans(a)rfi.ac.uk> wrote: >> >>> Hi Yuval, >>> >>> >>> I worked on this earlier today with Tom Byrne and I think I may be able >>> to provide some more information. >>> >>> I set up the RabbitMQ server myself, and created the exchange with type >>> ’topic’ before configuring the bucket. >>> >>> Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, >>> using certificates generated with Letsencrypt. >>> >>> >> it actually does. we don't support amqp over ssl. >> feel free to open a tracker for that - as we should probably support that! >> but note that it would probably be backported only to later versions than >> nautilus. >> >> >> >>> Many thanks, >>> >>> Tom >>> >>> >>> >>> Dr Tom Schoonjans >>> >>> Research Software Engineer - HPC and Cloud >>> >>> Rosalind Franklin Institute >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0FA >>> United Kingdom >>> >>> https://www.rfi.ac.uk >>> >>> The Rosalind Franklin Institute is a registered charity in England and >>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>> and Wales, No.11266143. Funded by UK Research and Innovation through >>> the Engineering and Physical Sciences Research Council. >>> >>> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>> >>> Hi Tom, >>> Did you create the exchange in rabbitmq? The RGW does not create it and >>> assume it is already created? >>> Could you increase the log level in RGW and see if there are more log >>> messages that have "AMQP" in them? >>> >>> Thanks, >>> >>> Yuval >>> >>> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >>> tom.byrne(a)stfc.ac.uk> wrote: >>> >>>> Hi all, >>>> >>>> We've been trying to get RGW Bucket notifications working with a >>>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>>> a message to the endpoint, the message never appears in the queue, and we >>>> get this error from in the RGW logs: >>>> >>>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>>> Endpoint >>>> URI: amqp://user:pass@host:5671 >>>> Topic: ceph-topic-test >>>> Exchange: ceph-test >>>> Ack Level: broker failed, with error: -4098 >>>> >>>> We've confirmed the URI is correct, and that the gateway host can send >>>> messages to the RabbitMQ via a standalone script (using the same >>>> information as in the URI). Does anyone have any hints about how to dig >>>> into this? >>>> >>>> Cheers, >>>> Tom >>>> >>>> This email and any attachments are intended solely for the use of the >>>> named recipients. If you are not the intended recipient you must not use, >>>> disclose, copy or distribute this email or any of its attachments and >>>> should notify the sender immediately and delete this email from your >>>> system. UK Research and Innovation (UKRI) has taken every reasonable >>>> precaution to minimise risk of this email or any attachments containing >>>> viruses or malware but the recipient should carry out its own virus and >>>> malware checks before opening the attachments. UKRI does not accept any >>>> liability for any losses or damages which the recipient may sustain due to >>>> presence of any viruses. Opinions, conclusions or other information in this >>>> message and attachments that are not related directly to UKRI business are >>>> solely those of the author and do not represent the views of UKRI. >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>> >> >

3 years, 2 months

1
0
0 0

"ceph orch restart mgr" command creates mgr restart loop

by Chris Read

Greetings all... I'm busy testing out Ceph and have hit this troublesome bug while following the steps outlined here: https://docs.ceph.com/en/octopus/cephadm/monitoring/#configuring-ssl-tls-fo… When I issue the "ceph orch restart mgr" command, it appears the command is not cleared from a message queue somewhere (I'm still very unclear on many ceph specifics), and so each time the mgr process returns from restart it picks up the message again and keeps restarting itself forever (so far it's been stuck in this state for 45 minutes). Watching the logs we see this going on: $ ceph log last cephadm -w root@ceph-poc-000:~# ceph log last cephadm -w cluster: id: d23bc326-543a-11eb-bfe0-b324db228b6c health: HEALTH_OK services: mon: 5 daemons, quorum ceph-poc-000,ceph-poc-003,ceph-poc-004,ceph-poc-002,ceph-poc-001 (age 2h) mgr: ceph-poc-000.himivo(active, since 4s), standbys: ceph-poc-001.unjulx osd: 10 osds: 10 up (since 2h), 10 in (since 2h) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 10 GiB used, 5.4 TiB / 5.5 TiB avail pgs: 1 active+clean 2021-01-11T20:46:32.976606+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:32.980749+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:33.061519+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:39.156420+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:39.160618+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:39.242603+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:45.299953+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:45.304006+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:45.733495+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:51.871903+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:51.877107+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:51.976190+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:46:58.000720+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:46:58.006843+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:46:58.097163+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:47:04.188630+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:47:04.193501+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:47:04.285509+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available 2021-01-11T20:47:10.348099+0000 mon.ceph-poc-000 [INF] Active manager daemon ceph-poc-000.himivo restarted 2021-01-11T20:47:10.352340+0000 mon.ceph-poc-000 [INF] Activating manager daemon ceph-poc-000.himivo 2021-01-11T20:47:10.752243+0000 mon.ceph-poc-000 [INF] Manager daemon ceph-poc-000.himivo is now available And in the logs for the mgr instance itself we see it keep replaying the message over and over: $ docker logs -f ceph-d23bc326-543a-11eb-bfe0-b324db228b6c-mgr.ceph-poc-000.himivo debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 set uid:gid to 167:167 (ceph:ceph) debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process ceph-mgr, pid 1 debug 2021-01-11T20:47:31.390+0000 7f48b0d0d200 0 pidfile_write: ignore empty --pid-file debug 2021-01-11T20:47:31.414+0000 7f48b0d0d200 1 mgr[py] Loading python module 'alerts' debug 2021-01-11T20:47:31.486+0000 7f48b0d0d200 1 mgr[py] Loading python module 'balancer' debug 2021-01-11T20:47:31.542+0000 7f48b0d0d200 1 mgr[py] Loading python module 'cephadm' debug 2021-01-11T20:47:31.742+0000 7f48b0d0d200 1 mgr[py] Loading python module 'crash' debug 2021-01-11T20:47:31.798+0000 7f48b0d0d200 1 mgr[py] Loading python module 'dashboard' debug 2021-01-11T20:47:32.258+0000 7f48b0d0d200 1 mgr[py] Loading python module 'devicehealth' debug 2021-01-11T20:47:32.306+0000 7f48b0d0d200 1 mgr[py] Loading python module 'diskprediction_local' debug 2021-01-11T20:47:32.498+0000 7f48b0d0d200 1 mgr[py] Loading python module 'influx' debug 2021-01-11T20:47:32.550+0000 7f48b0d0d200 1 mgr[py] Loading python module 'insights' debug 2021-01-11T20:47:32.598+0000 7f48b0d0d200 1 mgr[py] Loading python module 'iostat' debug 2021-01-11T20:47:32.642+0000 7f48b0d0d200 1 mgr[py] Loading python module 'k8sevents' debug 2021-01-11T20:47:33.034+0000 7f48b0d0d200 1 mgr[py] Loading python module 'localpool' debug 2021-01-11T20:47:33.082+0000 7f48b0d0d200 1 mgr[py] Loading python module 'orchestrator' debug 2021-01-11T20:47:33.266+0000 7f48b0d0d200 1 mgr[py] Loading python module 'osd_support' debug 2021-01-11T20:47:33.310+0000 7f48b0d0d200 1 mgr[py] Loading python module 'pg_autoscaler' debug 2021-01-11T20:47:33.382+0000 7f48b0d0d200 1 mgr[py] Loading python module 'progress' debug 2021-01-11T20:47:33.442+0000 7f48b0d0d200 1 mgr[py] Loading python module 'prometheus' debug 2021-01-11T20:47:33.794+0000 7f48b0d0d200 1 mgr[py] Loading python module 'rbd_support' debug 2021-01-11T20:47:33.870+0000 7f48b0d0d200 1 mgr[py] Loading python module 'restful' debug 2021-01-11T20:47:34.086+0000 7f48b0d0d200 1 mgr[py] Loading python module 'rook' debug 2021-01-11T20:47:34.606+0000 7f48b0d0d200 1 mgr[py] Loading python module 'selftest' debug 2021-01-11T20:47:34.654+0000 7f48b0d0d200 1 mgr[py] Loading python module 'status' debug 2021-01-11T20:47:34.710+0000 7f48b0d0d200 1 mgr[py] Loading python module 'telegraf' debug 2021-01-11T20:47:34.762+0000 7f48b0d0d200 1 mgr[py] Loading python module 'telemetry' debug 2021-01-11T20:47:35.034+0000 7f48b0d0d200 1 mgr[py] Loading python module 'test_orchestrator' debug 2021-01-11T20:47:35.146+0000 7f48b0d0d200 1 mgr[py] Loading python module 'volumes' debug 2021-01-11T20:47:35.294+0000 7f48b0d0d200 1 mgr[py] Loading python module 'zabbix' debug 2021-01-11T20:47:35.366+0000 7f489df86700 0 ms_deliver_dispatch: unhandled message 0x55d57b71a420 mon_map magic: 0 v1 from mon.0 v2: 10.128.18.16:3300/0 debug 2021-01-11T20:47:35.758+0000 7f489df86700 1 mgr handle_mgr_map Activating! debug 2021-01-11T20:47:35.758+0000 7f489df86700 1 mgr handle_mgr_map I am now activating debug 2021-01-11T20:47:35.770+0000 7f484ca14700 0 [balancer DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.770+0000 7f484ca14700 1 mgr load Constructed class from module: balancer debug 2021-01-11T20:47:35.774+0000 7f484ca14700 0 [cephadm DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.874+0000 7f484ca14700 1 mgr load Constructed class from module: cephadm debug 2021-01-11T20:47:35.874+0000 7f484ca14700 0 [crash DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.874+0000 7f484ca14700 1 mgr load Constructed class from module: crash debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [dashboard DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: dashboard debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: devicehealth debug 2021-01-11T20:47:35.878+0000 7f484ca14700 0 [iostat DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.878+0000 7f484ca14700 1 mgr load Constructed class from module: iostat debug 2021-01-11T20:47:35.886+0000 7f484ca14700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.886+0000 7f484ca14700 1 mgr load Constructed class from module: orchestrator debug 2021-01-11T20:47:35.890+0000 7f484ca14700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.890+0000 7f484ca14700 1 mgr load Constructed class from module: pg_autoscaler debug 2021-01-11T20:47:35.890+0000 7f484ca14700 0 [progress DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.890+0000 7f484ca14700 1 mgr load Constructed class from module: progress debug 2021-01-11T20:47:35.898+0000 7f484ca14700 0 [prometheus DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.910+0000 7f484ca14700 1 mgr load Constructed class from module: prometheus debug 2021-01-11T20:47:35.914+0000 7f484ca14700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5) [11/Jan/2021:20:47:35] ENGINE Bus STARTING Warning: Permanently added 'ceph-poc-000,10.128.18.16' (ECDSA) to the list of known hosts. Warning: Permanently added 'ceph-poc-002,10.128.19.248' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.974+0000 7f484ca14700 1 mgr load Constructed class from module: rbd_support Warning: Permanently added 'ceph-poc-001,10.128.28.223' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [restful DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: restful debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [status DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: status debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [telemetry DEBUG root] setting log level based on debug_mgr: WARNING (1/5) debug 2021-01-11T20:47:35.982+0000 7f484ca14700 1 mgr load Constructed class from module: telemetry debug 2021-01-11T20:47:35.982+0000 7f486f3cc700 0 [restful WARNING root] server not running: no certificate configured debug 2021-01-11T20:47:35.982+0000 7f484ca14700 0 [volumes DEBUG root] setting log level based on debug_mgr: WARNING (1/5) Warning: Permanently added 'ceph-poc-004,10.128.31.60' (ECDSA) to the list of known hosts. debug 2021-01-11T20:47:35.994+0000 7f484ca14700 1 mgr load Constructed class from module: volumes Warning: Permanently added 'ceph-poc-003,10.128.30.127' (ECDSA) to the list of known hosts. CherryPy Checker: The Application mounted at '' has an empty config. [11/Jan/2021:20:47:36] ENGINE Serving on http://:::9283 [11/Jan/2021:20:47:36] ENGINE Bus STARTED debug 2021-01-11T20:47:36.778+0000 7f484d215700 0 log_channel(audit) log [DBG] : from='client.19140 -' entity='client.admin' cmd=[{"prefix": "orch", "action": "restart", "service_name": "mgr", "target": ["mon-mgr", ""]}]: dispatch debug 2021-01-11T20:47:36.778+0000 7f484ca14700 0 log_channel(cephadm) log [INF] : Restart service mgr debug 2021-01-11T20:47:36.806+0000 7f484d215700 0 log_channel(cluster) log [DBG] : pgmap v3: 1 pgs: 1 active+clean; 0 B data, 208 MiB used, 5.4 TiB / 5.5 TiB avail debug 2021-01-11T20:47:37.406+0000 7f489ef88700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 debug 2021-01-11T20:47:37.406+0000 7f489ef88700 -1 mgr handle_mgr_signal *** Got signal Terminated *** Is there anyway to manually clear this stuck message? Will it eventually timeout by itself? Either way it's a pretty unpleasant surprise to find when following the docs so early in the process. Thanks, Chris

3 years, 2 months

2
1
0 0

Re: RGW Bucket notification troubleshooting

by Yuval Lifshitz

On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > Switching to non-SSL connections to RabbitMQ allowed us to get things > working, although currently it’s not very reliable. > can you please add more about that? what reliability issues did you see? > I will open a new ticket over this if we can’t fix things ourselves. > > this would be great. we have ssl support for kafka and http endpoint, so, if you decide to give it a try you can look at them as examples. and let me know if you have questions or need help. > I will open an issue on the tracker as soon as my account request has been > approved :-) > > Best, > > Tom > > > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> I worked on this earlier today with Tom Byrne and I think I may be able >> to provide some more information. >> >> I set up the RabbitMQ server myself, and created the exchange with type >> ’topic’ before configuring the bucket. >> >> Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, >> using certificates generated with Letsencrypt. >> >> > it actually does. we don't support amqp over ssl. > feel free to open a tracker for that - as we should probably support that! > but note that it would probably be backported only to later versions than > nautilus. > > > >> Many thanks, >> >> Tom >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> Hi Tom, >> Did you create the exchange in rabbitmq? The RGW does not create it and >> assume it is already created? >> Could you increase the log level in RGW and see if there are more log >> messages that have "AMQP" in them? >> >> Thanks, >> >> Yuval >> >> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >> tom.byrne(a)stfc.ac.uk> wrote: >> >>> Hi all, >>> >>> We've been trying to get RGW Bucket notifications working with a >>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>> a message to the endpoint, the message never appears in the queue, and we >>> get this error from in the RGW logs: >>> >>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>> Endpoint >>> URI: amqp://user:pass@host:5671 >>> Topic: ceph-topic-test >>> Exchange: ceph-test >>> Ack Level: broker failed, with error: -4098 >>> >>> We've confirmed the URI is correct, and that the gateway host can send >>> messages to the RabbitMQ via a standalone script (using the same >>> information as in the URI). Does anyone have any hints about how to dig >>> into this? >>> >>> Cheers, >>> Tom >>> >>> This email and any attachments are intended solely for the use of the >>> named recipients. If you are not the intended recipient you must not use, >>> disclose, copy or distribute this email or any of its attachments and >>> should notify the sender immediately and delete this email from your >>> system. UK Research and Innovation (UKRI) has taken every reasonable >>> precaution to minimise risk of this email or any attachments containing >>> viruses or malware but the recipient should carry out its own virus and >>> malware checks before opening the attachments. UKRI does not accept any >>> liability for any losses or damages which the recipient may sustain due to >>> presence of any viruses. Opinions, conclusions or other information in this >>> message and attachments that are not related directly to UKRI business are >>> solely those of the author and do not represent the views of UKRI. >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >> >

3 years, 2 months

1
0
0 0

Re: RGW Bucket notification troubleshooting

by Yuval Lifshitz

On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > I worked on this earlier today with Tom Byrne and I think I may be able to > provide some more information. > > I set up the RabbitMQ server myself, and created the exchange with type > ’topic’ before configuring the bucket. > > Not sure if this matters, but the RabbitMQ endpoint is reached over SSL, > using certificates generated with Letsencrypt. > > it actually does. we don't support amqp over ssl. feel free to open a tracker for that - as we should probably support that! but note that it would probably be backported only to later versions than nautilus. > Many thanks, > > Tom > > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > Hi Tom, > Did you create the exchange in rabbitmq? The RGW does not create it and > assume it is already created? > Could you increase the log level in RGW and see if there are more log > messages that have "AMQP" in them? > > Thanks, > > Yuval > > On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < > tom.byrne(a)stfc.ac.uk> wrote: > >> Hi all, >> >> We've been trying to get RGW Bucket notifications working with a RabbitMQ >> endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate >> with the rabbitMQ server just fine, but when RGW tries to send a message to >> the endpoint, the message never appears in the queue, and we get this error >> from in the RGW logs: >> >> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >> Endpoint >> URI: amqp://user:pass@host:5671 >> Topic: ceph-topic-test >> Exchange: ceph-test >> Ack Level: broker failed, with error: -4098 >> >> We've confirmed the URI is correct, and that the gateway host can send >> messages to the RabbitMQ via a standalone script (using the same >> information as in the URI). Does anyone have any hints about how to dig >> into this? >> >> Cheers, >> Tom >> >> This email and any attachments are intended solely for the use of the >> named recipients. If you are not the intended recipient you must not use, >> disclose, copy or distribute this email or any of its attachments and >> should notify the sender immediately and delete this email from your >> system. UK Research and Innovation (UKRI) has taken every reasonable >> precaution to minimise risk of this email or any attachments containing >> viruses or malware but the recipient should carry out its own virus and >> malware checks before opening the attachments. UKRI does not accept any >> liability for any losses or damages which the recipient may sustain due to >> presence of any viruses. Opinions, conclusions or other information in this >> message and attachments that are not related directly to UKRI business are >> solely those of the author and do not represent the views of UKRI. >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

3 years, 3 months

1
0
0 0

RGW Bucket notification troubleshooting

by Byrne, Thomas (STFC,RAL,SC)

Hi all, We've been trying to get RGW Bucket notifications working with a RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate with the rabbitMQ server just fine, but when RGW tries to send a message to the endpoint, the message never appears in the queue, and we get this error from in the RGW logs: 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) Endpoint URI: amqp://user:pass@host:5671 Topic: ceph-topic-test Exchange: ceph-test Ack Level: broker failed, with error: -4098 We've confirmed the URI is correct, and that the gateway host can send messages to the RabbitMQ via a standalone script (using the same information as in the URI). Does anyone have any hints about how to dig into this? Cheers, Tom This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

3 years, 3 months

2
1
0 0

Re: Where has my capacity gone?

by George Yil

Sorry for replying late :(. And thanks for the tips. This is a fresh cluster. And I didn’t think data distribution would be a problem. Is this normal? Below is the ceph osd df output. The related pool is hdd only (prod.rgw.buckets.data). I guess there is variance but I couldn’t get the reason. Is it because of PG number which I get help from pg-calculator <https://ceph.io/pgcalc/>? Or is this expected ceph behaviour? #ceph osd df https://pastebin.ubuntu.com/p/ZmQZsGYpr7/ <https://pastebin.ubuntu.com/p/7C9zpXYntR/> I am also sharing related cluster information. Any suggestion would be appreciated. #ceph df https://pastebin.ubuntu.com/p/sXpf99zhnV/ <https://pastebin.ubuntu.com/p/sXpf99zhnV/> #ceph detail df https://pastebin.ubuntu.com/p/dwvwBnnBmv/ <https://pastebin.ubuntu.com/p/dwvwBnnBmv/> #ceph osd pool ls detail https://pastebin.ubuntu.com/p/c2KQD5CGMV/ <https://pastebin.ubuntu.com/p/c2KQD5CGMV/> #crush rules https://pastebin.ubuntu.com/p/X6WsZhV3Zz/ <https://pastebin.ubuntu.com/p/X6WsZhV3Zz/> Thanks. > On 26 Jan 2021, at 11:18, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > ceph osd df | sort -nk8 > >> On Jan 25, 2021, at 11:22 PM, George Yil <georgeyil75(a)gmail.com> wrote: >> >> Hi, >> >> I have a ceph nautilus (14.2.9) cluster with 10 nodes. Each node has >> 19x16TB disks attached. >> >> I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured >> as EC 8+2 (jerasure). >> ceph df shows 2.1PiB MAX AVAIL space. >> >> Then I configured radosgw as a secondary zone and 100TiB of S3 data is >> replicated. >> >> But weirdly enough ceph df shows 1.8PiB MAX AVAIL for the same pool. But >> there is only 100TiB of written data. ceph df also confirms it. I can not >> figure out where 200TiB capacity is gone. >> >> Would someone please tell me what I am missing? >> >> Thanks. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 years, 3 months

1
0
0 0

Ceph-mds using a lot of buffer_anon memory

by David Turner

We just upgraded a cephfs cluster from 12.2.12 to 14.2.11. Our next step is to upgrade to 14.2.16 to troubleshoot this issue, but I thought I'd reach out here first if anyone had any ideas. The clients are still running an older version of ceph-fuse 12.2.4 and it's very difficult to remount all of them. Would probably take a team of us a couple days to restart all of them. I've looked around online and release notes and all of the known memory leaks I've been able to find have been fixed prior to version 14.2.11 so this would be an unknown memory leak. All of the memory is in use in [1] buffer_anon. If left unchecked it will use up over 700GB of memory within 24 hours. On an identical cluster with an equivalent workload still running 12.2.12 [2] buffer_anon information is much healthier. Without any other options or ideas our plan is to upgrade the cluster to 14.2.16 first and then upgrade the clients. Has anyone else come across high buffer_anon usage? [1] "buffer_anon": { "items": 33756758, "bytes": 135025912897 }, [2] "buffer_anon": { "items": 636, "bytes": 273118 },

3 years, 3 months

1
0
0 0

libRADOS semantics

by Cary FitzHugh

are there any good documents on the implicit requirements for librados calls? I can write a file, if it doesn't exist successfully. I can write_full a file, and it works (but many of my files will need to be chunked off disk, not enough RAM) I can append to a nonexistent file and it works. I *cannot* write to a part of an existing file (overwrite a section of it or write past it's current size) --> Errno 95 I *cannot* append to an existing file --> Errno 95 I *cannot* trunc an existing file (nor a non existent file) --> Errno 95 (what is the point of this then??) any ideas / where can one see more detailed semantics of what operations are allowed and when? The librados C API does not really go into any detail about errors and why things happen as they do. This behavior seems similar whether I am admin or a separate user. Thanks for any pointers. Cary

3 years, 3 months

1
1
0 0

Auth Questions w/ librados

by Cary FitzHugh

Hello! I've got a new cluster set up. I created a new user and pool. Creating the user like so: ceph auth add client.myclient mon 'allow r' osd 'allow rw pool=testpool' I have set up librados and python bindings, with a ceph.conf and a keyring file. My trouble is that when I connect with client.admin, things seem to work correctly. but my client.myuser cannot connect.. When I try to connect with my client.myclient I get a RADOS permission denied. cluster = rados.Rados(conffile=conffile.name) My keyring only has one user in it. The conffile only has a [global] section with a mon_host string and the keyring path. I am also unable to connect to the cluster using that user with the ceph CLI client. I have had limited success in debugging any of these situations - all the debugging I might put into the ceph.conf file seems to only work after the client is connected. In testing, I also made the client.myclient user have allow * on all capabilities, and that did nothing that I could see. If anyone has thoughts on where I am stumbling, or how to debug things, that would be much appreciated. It feels like it's a combination of a few small mistakes, but I can't get a handle on the issues. Thank you. Cary

3 years, 3 months

1
1
0 0

rados bench error after running vstart script- HELP PLEASE

by Bobby

Hello, I am having an error while trying to run rados benchmark after running vstart script I run : ../src/vstart.sh -d -n -l and then when I try to run: bin/rados -p rbd bench 30 write it gives me error saying: error opening pool rbd: (2) No such file or directory Can someone please help me in this. I will be grateful Thanks

3 years, 3 months

3
4
0 0

2024

2023

2022

2021

2020

2019

ceph-users January 2021