- ceph-users - lists.ceph.io

by Byrne, Thomas (STFC,RAL,SC)

Hi all, We've been trying to get RGW Bucket notifications working with a RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate with the rabbitMQ server just fine, but when RGW tries to send a message to the endpoint, the message never appears in the queue, and we get this error from in the RGW logs: 2021-01-26 16:28:17.271 7f0468b1f700 1 push to endpoint AMQP(0.9.1) Endpoint URI: amqp://user:pass@host:5671 Topic: ceph-topic-test Exchange: ceph-test Ack Level: broker failed, with error: -4098 We've confirmed the URI is correct, and that the gateway host can send messages to the RabbitMQ via a standalone script (using the same information as in the URI). Does anyone have any hints about how to dig into this? Cheers, Tom This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

1 hour, 8 minutes

1
0
0 0

libRADOS semantics

by Cary FitzHugh

are there any good documents on the implicit requirements for librados calls? I can write a file, if it doesn't exist successfully. I can write_full a file, and it works (but many of my files will need to be chunked off disk, not enough RAM) I can append to a nonexistent file and it works. I *cannot* write to a part of an existing file (overwrite a section of it or write past it's current size) --> Errno 95 I *cannot* append to an existing file --> Errno 95 I *cannot* trunc an existing file (nor a non existent file) --> Errno 95 (what is the point of this then??) any ideas / where can one see more detailed semantics of what operations are allowed and when? The librados C API does not really go into any detail about errors and why things happen as they do. This behavior seems similar whether I am admin or a separate user. Thanks for any pointers. Cary

1 hour, 28 minutes

1
1
0 0

Unable to use ceph command

by Jens Hyllegaard (Soft Design A/S)

Hi. I am not sure why this is not working, but I am now unable to use the ceph command on any of my hosts. When I try to launch ceph, I get the following response: [errno 13] RADOS permission denied (error connecting to the cluster) The web management interface is working fine. I have a suspicion that this started after trying to recreate an nfs cluster I first removed the existing one with: ceph nfs cluster delete <id> And then tried to create it again with: ceph nfs cluster create cephfs <id> The command seemed to hang, and after several hours I ended the command with ctrl-c. Since then I have been unable to use the ceph command. This is fortunately a test environment, and it is running Octopus 15.2.8 Does anyone have an idea on how I can get access again? Regards Jens Hyllegaard

2 hours, 30 minutes

2
2
0 0

Auth Questions w/ librados

by Cary FitzHugh

Hello! I've got a new cluster set up. I created a new user and pool. Creating the user like so: ceph auth add client.myclient mon 'allow r' osd 'allow rw pool=testpool' I have set up librados and python bindings, with a ceph.conf and a keyring file. My trouble is that when I connect with client.admin, things seem to work correctly. but my client.myuser cannot connect.. When I try to connect with my client.myclient I get a RADOS permission denied. cluster = rados.Rados(conffile=conffile.name) My keyring only has one user in it. The conffile only has a [global] section with a mon_host string and the keyring path. I am also unable to connect to the cluster using that user with the ceph CLI client. I have had limited success in debugging any of these situations - all the debugging I might put into the ceph.conf file seems to only work after the client is connected. In testing, I also made the client.myclient user have allow * on all capabilities, and that did nothing that I could see. If anyone has thoughts on where I am stumbling, or how to debug things, that would be much appreciated. It feels like it's a combination of a few small mistakes, but I can't get a handle on the issues. Thank you. Cary

2 hours, 33 minutes

1
1
0 0

rados bench error after running vstart script- HELP PLEASE

by Bobby

Hello, I am having an error while trying to run rados benchmark after running vstart script I run : ../src/vstart.sh -d -n -l and then when I try to run: bin/rados -p rbd bench 30 write it gives me error saying: error opening pool rbd: (2) No such file or directory Can someone please help me in this. I will be grateful Thanks

5 hours, 6 minutes

3
4
0 0

Running ceph cluster on different os

by Szabo, Istvan (Agoda)

Hi, Is there anybody running a cluster with different os? Due to the centos 8 change I might try to add ubuntu osd nodes to centos cluster and decommission the centos slowly but I'm not sure this is possible or not. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

10 hours, 24 minutes

4
3
0 0

Where has my capacity gone?

by George Yil

Hi, I have a ceph nautilus (14.2.9) cluster with 10 nodes. Each node has 19x16TB disks attached. I created radosgw pools. secondaryzone.rgw.buckets.data pool is configured as EC 8+2 (jerasure). ceph df shows 2.1PiB MAX AVAIL space. Then I configured radosgw as a secondary zone and 100TiB of S3 data is replicated. But weirdly enough ceph df shows 1.8PiB MAX AVAIL for the same pool. But there is only 100TiB of written data. ceph df also confirms it. I can not figure out where 200TiB capacity is gone. Would someone please tell me what I am missing? Thanks.

11 hours, 19 minutes

1
0
0 0

14.2.16 Low space hindering backfill after reboot

by Marco Pizzolo

Hello Everyone, We seem to be having a problem on one of our ceph clusters post the OS patch and reboot of one of the nodes. The three other nodes are showing OSD fill rates of 77%-81%, but the 60 OSDs contained in the host that was just rebooted are varying between 64% and 90% since the reboot occurred. The three other nodes have not yet been patched or rebooted. The result is: health: HEALTH_WARN 15 nearfull osd(s) 7 pool(s) nearfull Low space hindering backfill (add storage if this doesn't resolve itself): 15 pgs backfill_toofull Degraded data redundancy: 170940/1437684990 objects degraded (0.012%), 4 pgs degraded, 4 pgs undersized services: mon: 3 daemons, quorum prdceph01,prdceph02,prdceph03 (age 6h) mgr: prdceph01(active, since 5w), standbys: prdceph02, prdceph03, prdceph04 mds: ArchiveRepository:1 {0=prdceph01=up:active} 3 up:standby osd: 240 osds: 240 up (since 6h), 240 in (since 27h); 16 remapped pgs task status: scrub status: mds.prdceph01: idle data: pools: 7 pools, 8384 pgs objects: 479.23M objects, 557 TiB usage: 1.7 PiB used, 454 TiB / 2.1 PiB avail pgs: 170940/1437684990 objects degraded (0.012%) 4155186/1437684990 objects misplaced (0.289%) 8332 active+clean 36 active+clean+scrubbing+deep 11 active+remapped+backfill_toofull 2 active+undersized+degraded+remapped+backfill_toofull 2 active+forced_recovery+undersized+degraded+remapped+forced_backfill+backfill_toofull 1 active+remapped+backfilling io: client: 9.6 MiB/s rd, 820 KiB/s wr, 1.02k op/s rd, 189 op/s wr recovery: 0 B/s, 25 keys/s, 10 objects/s Any suggestions would be greatly appreciated, as currently it is not able to complete the repair, nor will it backfill, even when attempting to force. Many thanks in advance. Marco

21 hours, 7 minutes

1
0
0 0

Permissions for OSD

by George Shuklin

Docs for permissions are super vague. What each flag does? What is 'x' permitting? What's the difference between class-write and write? And the last question: can we limit user to reading/writing only to existing objects in the pool? Thanks!

1 day, 1 hour

1
0
0 0

January Ceph Science Virtual User Group Meeting

by Kevin Hrpcek

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday January 27th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20210127 <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: January 27th, 2020 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 day, 2 hours

1
0
0 0

2021

2020

2019