- ceph-users - lists.ceph.io

rbd resize progress report?
by Jorge Garcia 01 Feb '21

01 Feb '21

I'm trying to resize a block device using "rbd --resize". The block device is pretty huge (100+ TB). The resize has been running for over a week, and I have no idea if it's actually doing anything, or if it's just hanging or in some infinite loop. Is there any way of getting a progress report from the resize to get an idea if this is ever going to finish? Thanks!

1 0

Unable to enable RBD-Mirror Snapshot on image when VM is using RBD
by Adam Boyhan 01 Feb '21

01 Feb '21

This is a odd one. I don't hit it all the time so I don't think its expected behavior. Sometimes I have no issues enabling rbd-mirror snapshot mode on a rbd when its in use by a KVM VM. Other times I hit the following error, the only way I can get around it is to power down the KVM VM. root@Ccscephtest1:~# rbd mirror image enable CephTestPool1/vm-101-disk-0 snapshot 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f1e7c012440 handle_create_snapshot: failed to create mirror snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1e99ffb700 -1 librbd::mirror::EnableRequest: 0x5597667fd200 handle_create_primary_snapshot: failed to create initial primary snapshot: (22) Invalid argument 2021-01-29T09:29:07.875-0500 7f1ea559f3c0 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: (22) Invalid argument

2 4

Balancing with upmap
by Francois Legrand 01 Feb '21

01 Feb '21

Hi all, I have a cluster with 116 disks (24 new disks of 16TB added in december and the rest of 8TB) running nautilus 14.2.16. I moved (8 month ago) from crush_compat to upmap balancing. But the cluster seems not well balanced, with a number of pgs on the 8TB disks varying from 26 to 52 ! And an occupation from 35 to 69%. The recent 16 TB disks are more homogeneous with 48 to 61 pgs and space between 30 and 43%. Last week, I realized that some osd were maybe not using upmap because I did a ceph osd crush weight-set ls and got (compat) as result. Thus I ran a ceph osd crush weight-set rm-compat which triggered some rebalancing. Now there is no more recovery for 2 days, but the cluster is still unbalanced. As far as I understand, upmap is supposed to reach an equal number of pgs on all the disks (I guess weighted by their capacity). Thus I would expect more or less 30 pgs on the 8TB disks and 60 on the 16TB and around 50% usage on all. Which is not the case (by far). The problem is that it impact the free available space in the pools (264Ti while there is more than 578Ti free in the cluster) because free space seems to be based on space available before the first osd will be full ! Is it normal ? Did I missed something ? What could I do ? F.

3 11

Multisite recovering shards
by Szabo, Istvan (Agoda) 01 Feb '21

01 Feb '21

Hi, I’ve never seen in our multisite sync status healthy output, almost all the sync shards are recovering. What can I do with recovering shards? We have 1 realm, 1 zonegroup and inside the zonegroup we have 3 zones in 3 different geo location. We are using octopus 15.2.7 for bucket sync with symmetrical replication. The user is at the moment migrating their data and the sites are always behind which is replicated from the place where it was uploaded. I’ve restarted all rgw and disable / enable bucket sync, it started to work, but I think when it comes to close sync it will stop again due to the recovering shards. Any idea? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4

data on sda with metadata on lvm partition?
by Matt Piermarini 30 Jan '21

30 Jan '21

Hello all - Hi. After running this dev cluster with a single osd (/dev/sda) hdd in each node (6), I want to now put the metadata on the nvme disk which is also used as boot. There is plenty of space left on the nvme, so I re-did the logical volumes to make a 50gb LV for the metadata, thinking I'd put the metdata on thenvme/LV and use the entire /dev/sda as data. Before I really go down this rabbit hole, just want opinions if this is something that should work? I've tried both Cepth 15.2.7 & Ceph 15.2.8, each with different errors. This particular trace is ceph15.2.8. This is under Rook, so rook is doing: <snip> exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 /dev/sda --db-devices /dev/cephDB/database --report provision 2021-01-28 01:46:56.186043 D | exec: --> passed data devices: 1 physical, 0 LVM provision 2021-01-28 01:46:56.186074 D | exec: --> relative data size: 1.0 provision 2021-01-28 01:46:56.186079 D | exec: --> passed block_db devices: 0 physical, 1 LVM provision 2021-01-28 01:46:56.186092 D | exec: provision 2021-01-28 01:46:56.186104 D | exec: Total OSDs: 1 provision 2021-01-28 01:46:56.186107 D | exec: provision 2021-01-28 01:46:56.186111 D | exec: Type Path LV Size % of device provision 2021-01-28 01:46:56.186114 D | exec: ---------------------------------------------------------------------------------------------------- provision 2021-01-28 01:46:56.186117 D | exec: data /dev/sda 3.64 TB 100.00% provision 2021-01-28 01:46:56.186121 D | exec: block_db /dev/cephDB/database 51.65 GB 10000.00%<snip> It fails with the stack trace below, essentially complaining it can't PARTUUID for the LV. exec: Running command: /usr/sbin/lvcreate --yes -l 953861 -n osd-block-2acd94f6-0fed-423b-8540-ae93c0621c2e ceph-b6da5679-543b-4a79-9cc2-e4e308ba61a4 exec: stderr: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical volume symlinks in device directory. exec: stderr: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list by scanning device directory. exec: stderr: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, device-mapper library will manage device nodes in device directory. exec: stdout: Logical volume "osd-block-2acd94f6-0fed-423b-8540-ae93c0621c2e" created. exec: --> blkid could not detect a PARTUUID for device: /dev/cephDB/database exec: --> Was unable to complete a new OSD, will rollback changes exec: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it exec: stderr: purged osd.0 exec: Traceback (most recent call last): exec: File "/usr/sbin/ceph-volume", line 11, in <module> exec: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__ exec: self.main(self.argv) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc exec: return f(*a, **kw) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main exec: terminal.dispatch(self.mapper, subcommand_args) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch exec: instance.main() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main exec: terminal.dispatch(self.mapper, self.argv) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch exec: instance.main() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root exec: return func(*a, **kw) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 415, in main exec: self._execute(plan) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 431, in _execute exec: p.safe_prepare(argparse.Namespace(**args)) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare exec: self.prepare() exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root exec: return func(*a, **kw) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 382, in prepare exec: self.args.block_db_slots) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 189, in setup_device exec: name_uuid = self.get_ptuuid(device_name) exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 135, in get_ptuuid exec: raise RuntimeError('unable to use device') exec: RuntimeError: unable to use device provision failed to configure devices: failed to initialize devices: failed ceph-volume: exit status 1 Thanks for any ideas/help.

1 0

RBD-Mirror Snapshot Backup Image Uses
by Adam Boyhan 29 Jan '21

29 Jan '21

I have been doing some testing with RBD-Mirror Snapshots to a remote Ceph cluster. Does anyone know if the images on the remote cluster can be utilized in anyway? Would love the ability to clone them, or even readonly would be nice.

5 26

Rotating Service Keys
by William Law 29 Jan '21

29 Jan '21

Hi - We keep on getting errors like these on specific OSDs with Nautilus (14.2.16): 2021-01-29 06:14:19.174 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:14:49.173 7fbeaab92c00 0 monclient: wait_auth_rotating timed out after 30 2021-01-29 06:14:49.173 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:15:19.173 7fbeaab92c00 0 monclient: wait_auth_rotating timed out after 30 2021-01-29 06:15:19.173 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:15:49.174 7fbeaab92c00 0 monclient: wait_auth_rotating timed out after 30 2021-01-29 06:15:49.174 7fbeaab92c00 -1 osd.8 12568359 unable to obtain rotating service keys; retrying 2021-01-29 06:15:49.174 7fbeaab92c00 -1 osd.8 12568359 init wait_auth_rotating timed out From googling it seems like it could be a variety of things. We do think time is in sync. It is particularly perplexing as we'll have a single OSD get this error while all other OSDs on the same node are fine. It seems exactly like this: https://tracker.ceph.com/issues/17170 Stopping the managers and restarting the mons fixes it temporarily. From this old thread we do have msgr2 enabled: https://www.spinics.net/lists/ceph-users/msg60631.html This blog seems to point to storage slowness being the root cause in there env: http://www.florentflament.com/blog/ceph-monitor-status-switching-due-to-slo… Any advice for sorting out what is causing this? Thanks, Will

1 0

Unable to use ceph command
by Jens Hyllegaard (Soft Design A/S) 29 Jan '21

29 Jan '21

Hi. I am not sure why this is not working, but I am now unable to use the ceph command on any of my hosts. When I try to launch ceph, I get the following response: [errno 13] RADOS permission denied (error connecting to the cluster) The web management interface is working fine. I have a suspicion that this started after trying to recreate an nfs cluster I first removed the existing one with: ceph nfs cluster delete <id> And then tried to create it again with: ceph nfs cluster create cephfs <id> The command seemed to hang, and after several hours I ended the command with ctrl-c. Since then I have been unable to use the ceph command. This is fortunately a test environment, and it is running Octopus 15.2.8 Does anyone have an idea on how I can get access again? Regards Jens Hyllegaard

3 5

Cannot access "Object Gateway" in dashboard after setting rgw api keys
by Troels Hansen 29 Jan '21

29 Jan '21

Hi, I have tried to enable RGW management in the dashboard. The dashboard works fine, and I tried to add a new system user: radosgw-admin user create --uid=some-user --display-name="User for dashboard" --system and set the accesskey and secret: ceph dashboard set-rgw-api-access-key access-key ceph dashboard set-rgw-api-secret-key secret-key However, after adding these I get this in the dashboard, when I try to access the Object Gateway: "key data not in dict [u'user1', u'user2', u'some-user']" If I try to verify that the credentials are correct with: ceph dashboard get-rgw-api-access-key ceph dashboard get-rgw-api-secret-key secret-key and compare then with the key and secret from: radosgw-admin user info --uid=some-user they match. Running Ceph 14.2.16 To me it looks like it cannot find the user, or match the credentials, but I cannot see what I have done wrong? -- Med venlig hilsen *Troels Hansen* Senior Linux Konsulent Tlf.: 22 43 71 57 tha(a)miracle.dk www.miracle.dk

1 0

Re: RGW Bucket notification troubleshooting
by Yuval Lifshitz 29 Jan '21

29 Jan '21

On Fri, Jan 29, 2021 at 9:18 AM Schoonjans, Tom (RFI,RAL,-) < Tom.Schoonjans(a)rfi.ac.uk> wrote: > Hi Yuval, > > > What do I need to do if I want to switch to using a different exchange on > the RabbitMQ endpoint? Or change the amqp-ack-level option that was used? > Would you expect the same problem again? Will the existing connections to > the RabbitMQ server be cleanly terminated? > > I think that changing the ack level would take effect on the next publish (as this is not a feature of the connection, but the calling code), but to change the exchange (or any other parameter of the connection itself), or even creating a new topic to the same endpoint with a different exchange, you would need a restart :-( (tracking this here: https://tracker.ceph.com/issues/46127) > I tried the topic example > <https://www.rabbitmq.com/tutorials/tutorial-five-python.html> from the > RabbitMQ tutorial and I actually got the same behaviour as with Ceph: > messages sent before the consumer queue is attached are lost. From what I > understand this is a *feature* of this type of exchange. See also this > <https://stackoverflow.com/questions/6148381/rabbitmq-persistent-message-wit…> stackoverflow > post. > > this is interesting. and looks like a better model than what we currently have. we should declare our own fanout "gateway" exchange, connected to an "eat all" queue. the users may then connect the consumers directly to it, or via a topic exchange they declare. that would actually fix 2 issues: - durability of messages before clients are connected - exchange name configuration issues > Best, > > Tom > > > Dr Tom Schoonjans > > Research Software Engineer - HPC and Cloud > > Rosalind Franklin Institute > Harwell Science & Innovation Campus > Didcot > Oxfordshire > OX11 0FA > United Kingdom > > https://www.rfi.ac.uk > > The Rosalind Franklin Institute is a registered charity in England and > Wales, No. 1179810 Company Limited by Guarantee Registered in England > and Wales, No.11266143. Funded by UK Research and Innovation through > the Engineering and Physical Sciences Research Council. > > On 28 Jan 2021, at 18:16, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: > > > > On Thu, Jan 28, 2021 at 7:34 PM Schoonjans, Tom (RFI,RAL,-) < > Tom.Schoonjans(a)rfi.ac.uk> wrote: > >> Hi Yuval, >> >> >> Together with Tom Byrne I ran some more tests today while keeping an eye >> on the logs as well. >> >> We immediately noticed that the nodes were logging errors when uploading >> files like: >> >> 2021-01-28 16:10:45.825 7f56ff5cf700 1 ====== starting new request req=0x7f56ff5c87f0 ===== >> 2021-01-28 16:10:45.828 7f5721e14700 1 AMQP connect: exchange mismatch >> 2021-01-28 16:10:45.828 7f5721e14700 1 ERROR: failed to create push endpoint: amqp://<username>:<password>@<my.rabbitmq.server>:5672 due to: pubsub endpoint configuration error: AMQP: failed to create connection to: amqp://<username>:<password>@<my.rabbitmq.server>:5672 >> 2021-01-28 16:10:45.828 7f571ee0e700 1 ====== req done req=0x7f571ee077f0 op status=0 http_status=200 latency=0.0569997s ====== >> >> >> Which resulted in no connections being established to the RabbitMQ server. >> >> Tom restarted then the Ceph services on one gateway node, which led to >> events being sent to RabbitMQ without blocking, but only if this particular >> node was picked up by the boto3 upload request in the round-robin DNS. >> >> Restarting the Ceph service on all nodes fixed the problem and I got a >> nice steady stream of events to my consumer Python script! >> >> > we should fix it. no restart should be needed if one of the connection > parameters was wrong > > > >> I did notice that any events that were sent while my consumer script was >> not running are lost, as they are not picked up after I restart the script. >> Any thoughts on this? >> >> > this is strange. in our code [1] we don't require immediate transfer of > messages. > how is the exchange declared? > can you check if this is happening when you send messages from a python > producer as well? > > [1] https://github.com/ceph/ceph/blob/master/src/rgw/rgw_amqp.cc#L575 > > > >> Many thanks!! >> >> Best, >> >> Tom >> >> >> >> Dr Tom Schoonjans >> >> Research Software Engineer - HPC and Cloud >> >> Rosalind Franklin Institute >> Harwell Science & Innovation Campus >> Didcot >> Oxfordshire >> OX11 0FA >> United Kingdom >> >> https://www.rfi.ac.uk >> >> The Rosalind Franklin Institute is a registered charity in England and >> Wales, No. 1179810 Company Limited by Guarantee Registered in England >> and Wales, No.11266143. Funded by UK Research and Innovation through >> the Engineering and Physical Sciences Research Council. >> >> On 27 Jan 2021, at 16:21, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >> >> >> On Wed, Jan 27, 2021 at 5:34 PM Schoonjans, Tom (RFI,RAL,-) < >> Tom.Schoonjans(a)rfi.ac.uk> wrote: >> >>> Looks like there’s already a ticket open for AMQP SSL support: >>> https://tracker.ceph.com/issues/42902 (you opened it ;-)) >>> >>> I will give a try myself if I have some time, but don’t hold your breath >>> with lockdown and home schooling. Also I am not much of a C++ coder. >>> >>> I need to go over the logs with Tom Byrne to see why it is not working >>> properly. And perhaps I will be able to come up with a fix then. >>> >>> However this is what I have run into so far today: >>> >>> 1. After configuring a bucket with a topic using the non-SSL port, I >>> tried a couple of uploads to this bucket. They all hanged, which seemed >>> like something was very wrong, so I Ctrl-C’ed every time. After some time I >>> figured out from the RabbitMQ admin UI that Ceph was indeed connecting to >>> it, and the connections remained so I killed them from the UI. >>> >> >> sending the notification to the rabbitmq server is synchronous with the >> upload to the bucket. so, if the server is slow or not acking the >> notification, the upload request would hang. not that the upload itself is >> done first, but the reply to the client does not happen until rabbitmq >> server acks. >> >> would be great if you can share the radosgw logs. >> maybe the issue is related to the user/password method we use? we use: >> AMQP_SASL_METHOD_PLAIN >> >> one possible workaround would be to set "amqp-ack-level" to "none". in >> this case the radosgw does not wait for an ack >> >> in "pacific" you could use "persistent topics" where the notifications >> are sent asynchronously to the endpoint. >> >> 2. I then wrote a python script with Pika to consume the events, hoping >>> that would stop the blocking. I had some minor success with this. Usually >>> the first three or four uploaded files would generate events that I could >>> consume with my script. >>> >> >> the radosgw is waiting for an ack from the broker, not the end consumer, >> so this should not have mattered... >> did you actually see any notifications delivered to the consumer? >> >> >>> However, the rest would block for ever. I repeated this a couple of >>> times but always the same result. I noticed that after I stopped uploading, >>> removed the bucket and the topic, the connection from Ceph in the RabbitMQ >>> UI remained. I killed it but it came back seconds later from another port >>> on the Ceph cluster. I ended up playing whack-a-mole with this until no >>> more connections would be established from Ceph to RabbitMQ. I probably >>> killed a 100 or so of them. >>> >> >> once you remove the bucket there cannot be new notification sent. if you >> create the bucket again you may see notifications again (this is fixed in >> "pacific"). >> either way, even if the connection to the rabbitmq server would still be >> open, but no new notifications should be sent there. just having the >> connection should not be an issue but would be nice to fix that as well: >> https://tracker.ceph.com/issues/49033 >> >> 3. After this I couldn’t get any events sent anymore. There is no more >>> blocking when uploading, files get written but nothing else happens. No >>> connections are made anymore from Ceph to RabbitMQ. >>> >>> Hope this helps… >>> >> >> yes, this is very helpful! >> >> >>> Best, >>> >>> Tom >>> >>> >>> >>> >>> Dr Tom Schoonjans >>> >>> Research Software Engineer - HPC and Cloud >>> >>> Rosalind Franklin Institute >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0FA >>> United Kingdom >>> >>> https://www.rfi.ac.uk >>> >>> The Rosalind Franklin Institute is a registered charity in England and >>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>> and Wales, No.11266143. Funded by UK Research and Innovation through >>> the Engineering and Physical Sciences Research Council. >>> >>> On 27 Jan 2021, at 13:04, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>> >>> >>> >>> On Wed, Jan 27, 2021 at 11:33 AM Schoonjans, Tom (RFI,RAL,-) < >>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>> >>>> Hi Yuval, >>>> >>>> >>>> Switching to non-SSL connections to RabbitMQ allowed us to get things >>>> working, although currently it’s not very reliable. >>>> >>> >>> can you please add more about that? what reliability issues did you see? >>> >>> >>>> I will open a new ticket over this if we can’t fix things ourselves. >>>> >>>> >>> this would be great. we have ssl support for kafka and http endpoint, >>> so, if you decide to give it a try you can look at them as examples. >>> and let me know if you have questions or need help. >>> >>> >>> >>>> I will open an issue on the tracker as soon as my account request has >>>> been approved :-) >>>> >>>> Best, >>>> >>>> Tom >>>> >>>> >>>> >>>> >>>> >>>> Dr Tom Schoonjans >>>> >>>> Research Software Engineer - HPC and Cloud >>>> >>>> Rosalind Franklin Institute >>>> Harwell Science & Innovation Campus >>>> Didcot >>>> Oxfordshire >>>> OX11 0FA >>>> United Kingdom >>>> >>>> https://www.rfi.ac.uk >>>> >>>> The Rosalind Franklin Institute is a registered charity in England and >>>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>>> and Wales, No.11266143. Funded by UK Research and Innovation through >>>> the Engineering and Physical Sciences Research Council. >>>> >>>> On 26 Jan 2021, at 20:02, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>> >>>> >>>> >>>> On Tue, Jan 26, 2021 at 9:48 PM Schoonjans, Tom (RFI,RAL,-) < >>>> Tom.Schoonjans(a)rfi.ac.uk> wrote: >>>> >>>>> Hi Yuval, >>>>> >>>>> >>>>> I worked on this earlier today with Tom Byrne and I think I may be >>>>> able to provide some more information. >>>>> >>>>> I set up the RabbitMQ server myself, and created the exchange with >>>>> type ’topic’ before configuring the bucket. >>>>> >>>>> Not sure if this matters, but the RabbitMQ endpoint is reached over >>>>> SSL, using certificates generated with Letsencrypt. >>>>> >>>>> >>>> it actually does. we don't support amqp over ssl. >>>> feel free to open a tracker for that - as we should probably support >>>> that! >>>> but note that it would probably be backported only to later versions >>>> than nautilus. >>>> >>>> >>>> >>>>> Many thanks, >>>>> >>>>> Tom >>>>> >>>>> >>>>> >>>>> Dr Tom Schoonjans >>>>> >>>>> Research Software Engineer - HPC and Cloud >>>>> >>>>> Rosalind Franklin Institute >>>>> Harwell Science & Innovation Campus >>>>> Didcot >>>>> Oxfordshire >>>>> OX11 0FA >>>>> United Kingdom >>>>> >>>>> https://www.rfi.ac.uk >>>>> >>>>> The Rosalind Franklin Institute is a registered charity in England and >>>>> Wales, No. 1179810 Company Limited by Guarantee Registered in England >>>>> and Wales, No.11266143. Funded by UK Research and Innovation through >>>>> the Engineering and Physical Sciences Research Council. >>>>> >>>>> On 26 Jan 2021, at 19:37, Yuval Lifshitz <ylifshit(a)redhat.com> wrote: >>>>> >>>>> Hi Tom, >>>>> Did you create the exchange in rabbitmq? The RGW does not create it >>>>> and assume it is already created? >>>>> Could you increase the log level in RGW and see if there are more log >>>>> messages that have "AMQP" in them? >>>>> >>>>> Thanks, >>>>> >>>>> Yuval >>>>> >>>>> On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) < >>>>> tom.byrne(a)stfc.ac.uk> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We've been trying to get RGW Bucket notifications working with a >>>>>> RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can >>>>>> communicate with the rabbitMQ server just fine, but when RGW tries to send >>>>>> a message to the endpoint, the message never appears in the queue, and we >>>>>> get this error from in the RGW logs: >>>>>> >>>>>> 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) >>>>>> Endpoint >>>>>> URI: amqp://user:pass@host:5671 >>>>>> Topic: ceph-topic-test >>>>>> Exchange: ceph-test >>>>>> Ack Level: broker failed, with error: -4098 >>>>>> >>>>>> We've confirmed the URI is correct, and that the gateway host can >>>>>> send messages to the RabbitMQ via a standalone script (using the same >>>>>> information as in the URI). Does anyone have any hints about how to dig >>>>>> into this? >>>>>> >>>>>> Cheers, >>>>>> Tom >>>>>> >>>>>> This email and any attachments are intended solely for the use of the >>>>>> named recipients. If you are not the intended recipient you must not use, >>>>>> disclose, copy or distribute this email or any of its attachments and >>>>>> should notify the sender immediately and delete this email from your >>>>>> system. UK Research and Innovation (UKRI) has taken every reasonable >>>>>> precaution to minimise risk of this email or any attachments containing >>>>>> viruses or malware but the recipient should carry out its own virus and >>>>>> malware checks before opening the attachments. UKRI does not accept any >>>>>> liability for any losses or damages which the recipient may sustain due to >>>>>> presence of any viruses. Opinions, conclusions or other information in this >>>>>> message and attachments that are not related directly to UKRI business are >>>>>> solely those of the author and do not represent the views of UKRI. >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> >

1 0