- ceph-users - lists.ceph.io

Re: Debian 12 (bookworm) / Reef 18.2.1 problems
by Chris Palmer 21 Feb '24

21 Feb '24

I have logged this as https://tracker.ceph.com/issues/64213 On 16/01/2024 14:18, DERUMIER, Alexandre wrote: > Hi, > >>> ImportError: PyO3 modules may only be initialized once per >>> interpreter >>> process >>> >>> and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 >>> modules may only be initialized once per interpreter process > We have the same problem on proxmox8 (based on debian12) with ceph > quincy or reef. > > It seem to be related to python version on debian12 > > (we have no fix for this currently) > > >

6 7

User + Dev Meetup February 22 - CephFS Snapshots story!
by Neha Ojha 21 Feb '24

21 Feb '24

Hi everyone, You are invited to join us at the User + Dev meeting this week Thursday, February 22 at 10:00 AM Eastern Time! Focus Topic: CephFS Snapshots Evaluation Presented by: Enrico Bocchi and Abhishek Lekshmanan, Ceph operators from CERN From the presenters: Ceph at CERN provides block, object, and file storage backing the IT infrastructure of the Organization. CephFS, in particular, is largely used through the integration with OpenStack Manila by container-based workloads (Kubernetes, OpenShift), HPC MPI clusters, and as a general-purpose networked file system for enterprise groupware and open infrastructure technologies (code/software repositories, monitoring, analytics, etc.). Our presentation focuses on CephFS snapshots and their implications on performance and stability. Snapshots would be a valuable addition to our existing CephFS service, as they allow for storage rollback and disaster recovery through mirroring. According to our observations, however, they introduce a non-negligible performance penalty and may jeopardize the stability of the file system. In particular, we would like to discuss: 1. Experiences with CephFS snapshots from other operators in the Ceph community. 2. Tools and strategies one can deploy to pre-empty or mitigate issues. 3. How to effectively contribute with upstream developers and interested community users to address the identified limitations. Feel free to add questions or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Neha

1 0

RoCE?
by Jan Marek 20 Feb '24

20 Feb '24

Hello, we are configure new ceph cluster with Mellanox 2x100Gbps cards. We bond this two ports to MLAG bond0 interface. In the async+posix mode everythink is OK, cluster is in the HELTH_OK state. CEPH version is 18.2.1. Then we tried to configure RoCE for cluster part of network, but without success. Our ceph config dump (only relevant config): global advanced ms_async_rdma_device_name mlx5_bond_0 * global advanced ms_async_rdma_gid_idx 3 global host:ceph1-nvme advanced ms_async_rdma_local_gid 0000:0000:0000:0000:0000:ffff:a0d9:05d8 * global host:ceph2-nvme advanced ms_async_rdma_local_gid 0000:0000:0000:0000:0000:ffff:a0d9:05d7 * global host:ceph3-nvme advanced ms_async_rdma_local_gid 0000:0000:0000:0000:0000:ffff:a0d9:05d6 * global advanced ms_async_rdma_roce_ver 2 global advanced ms_async_rdma_type rdma * global advanced ms_cluster_type async+rdma * global advanced ms_public_type async+posix * On the ceph1-nvme there is this show_gids.sh list: # ./show_gids.sh DEV PORT INDEX GID IPv4 VER DEV --- ---- ----- --- ------------ --- --- mlx5_bond_0 1 0 fe80:0000:0000:0000:0e42:a1ff:fe93:b004 v1 bond0 mlx5_bond_0 1 1 fe80:0000:0000:0000:0e42:a1ff:fe93:b004 v2 bond0 mlx5_bond_0 1 2 0000:0000:0000:0000:0000:ffff:a0d9:05d8 160.217.5.216 v1 bond0 mlx5_bond_0 1 3 0000:0000:0000:0000:0000:ffff:a0d9:05d8 160.217.5.216 v2 bond0 n_gids_found=4 I have set this line in /etc/security/limits.conf: * hard memlock unlimited But when I tried to restart ceph.target, OSD nodes didn't start with this errors, see attachment. Mellanox drivers are from Debian bookworm kernel. Is there somethink missing in config, or some errors? When I change ms_cluster_type to async+posix and restart ceph.target, cluster converged to HEALTH_OK state... Thanks for advices... Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

1 1

What is the proper way to setup Rados Gateway (RGW) under Ceph?
by Michael Worsham 19 Feb '24

19 Feb '24

I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-…. The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up? Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ``` What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

3 6

cephadm Failed to apply 1 service(s)
by Zakhar Kirpichenko 16 Feb '24

16 Feb '24

Hi, We had a physical drive malfunction in one of our Ceph OSD hosts managed by cephadm (Ceph 16.2.14). I have removed the drive from the system, and the kernel no longer sees it: ceph03 ~]# ls -al /dev/sde ls: cannot access '/dev/sde': No such file or directory I have removed the corresponding OSD from cephadm, crush map, etc. For all intents and purposes that OSD and its block device no longer exist: root@ceph01:/# ceph orch ps | grep osd.26 root@ceph01:/# ceph osd tree| grep 26 root@ceph01:/# ceph orch device ls | grep -E "ceph03.*sde" None of the above commands return anything. Cephadm correctly sees 8 remaining OSDs on the host: root@ceph01:/# ceph orch ls | grep ceph03_c osd.ceph03_combined_osd 8 33s ago 2y ceph03 Unfortunately, cephadm appears to be trying to apply a spec to host ceph03 including the disk that is now missing: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE= quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b -e NODE_NAME=ceph03 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=ceph03_combined_osd -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z -v /var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpc7b33pf0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpq45nkmd6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi --db-devices /dev/nvme0n1 /dev/nvme1n1 --yes --no-systemd Note that `lvm batch` includes the missing drive, /dev/sde. This fails because the drive no longer exists. Other than this cephadm ceph-volume thingy, the cluster is healthy.How can I tell cephadm that it should stop trying to use /dev/sde, which no longer exists, without affecting other OSDs on the host? I would very much appreciate any advice or pointers. Best regards, Zakhar

2 6

concept of ceph and 2 datacenters
by ronny.lippold＠spark5.de 16 Feb '24

16 Feb '24

hi there, i have a design/concept question, to see, whats outside and which kind of redundancy you use. actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby clone. but, rbd mirror is not application consistend. so we cannot be sure, that all vms (kvm/proxy) are running. we also waste a lot of hardware. so now, we think about one big cluster over the two datacenters (two buildings). my queston is, do you care about ceph redundancy or is one ceph with backups enough for you? i know, with ceph, we are aware of hdd or server failure. but, are software failures a real scenario? would be great, to get some ideas from you. also about the bandwidth between the 2 datacenters. we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd). thanks for help, my minds are rotating. kind regards, ronny -- Ronny Lippold System Administrator -- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold(a)spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 --

4 4

HA service for RGW and dnsmasq
by Jean-Marc FONTANA 16 Feb '24

16 Feb '24

Hello everyone, We operate 2 clusters as S3 external storage for owncloud. Both were installed with ceph-deploy (Nautilus), then converted to cephadm and updated till Reef. Each cluster has one RGW host, managing a dnsmasq pack for DNS delegation. We feel know we should create a HA endpoint for RGW as explained in Reef doc. We have a few questions that we didn't find answers in the doc before proceeding : - is dnsmasq still needed on the existing ans new RGW hosts ? - if answer is yes, how should it be configured ? - if answer is no, how does DNS delegation work ? Any help will be useful. Thanks and best regards JM

1 0

RBD Mirroring
by Michel Niyoyita 15 Feb '24

15 Feb '24

Hello team, I have two clusters in testing environment deployed using ceph-ansible running on ubuntu 20.04 with ceph Pacific version . I am testing mirroring between two clusters , in pool mode . Our production Cluster is for backend storage for openstack. This is how I configured the rbdmirros.yml: ceph_rbd_mirror_configure: true ceph_rbd_mirror_mode: "pool" ceph_rbd_mirror_remote_cluster: "prod" ceph_rbd_mirror_remote_user: "admin" This is my primary cluster "/etc/ceph/ "directory: root@ceph-osd1:/etc/ceph# ls -l total 52 -r-------- 1 root root 151 Feb 13 08:47 bup.client.admin.keyring -rw-r--r-- 1 root root 866 Feb 13 08:45 bup.conf -r-------- 1 ceph ceph 151 Feb 12 13:35 ceph.client.admin.keyring -rw------- 1 ceph ceph 131 Feb 12 13:41 ceph.client.crash.keyring -rw-r--r-- 1 ceph ceph 863 Feb 12 13:35 ceph.conf -rw-rw-r-- 1 ceph ceph 1294 Feb 12 13:40 ceph-dashboard.crt -rw------- 1 ceph ceph 1704 Feb 12 13:40 ceph-dashboard.key -r-------- 1 ceph ceph 140 Feb 12 13:36 ceph.mgr.ceph-osd1.keyring -r-------- 1 ceph ceph 140 Feb 12 13:36 ceph.mgr.ceph-osd2.keyring -r-------- 1 ceph ceph 140 Feb 12 13:36 ceph.mgr.ceph-osd3.keyring -r-------- 1 root root 151 Feb 13 08:38 prod.client.admin.keyring -rw-r--r-- 1 root root 863 Feb 13 08:37 prod.conf -rw-r--r-- 1 root root 92 Aug 29 16:38 rbdmap and this is my secondary "/etc/ceph" directory: root@ceph-osdb1:/etc/ceph# ls -l total 60 -rw------- 1 root root 0 Feb 13 09:26 ansible.1e1q9lzv_ceph-ansible -rw------- 1 root root 0 Feb 13 09:32 ansible.dk1h4kzp_ceph-ansible -r-------- 1 root root 151 Feb 13 09:02 bup.client.admin.keyring -rw-r--r-- 1 root root 866 Feb 13 09:02 bup.conf -r-------- 1 ceph ceph 151 Feb 13 08:23 ceph.client.admin.keyring -rw------- 1 ceph ceph 131 Feb 13 08:29 ceph.client.crash.keyring -rw------- 1 ceph ceph 138 Feb 13 09:19 ceph.client.rbd-mirror.ceph-osdb1.keyring -rw------- 1 ceph ceph 132 Feb 13 09:19 ceph.client.rbd-mirror-peer.keyring -rw-r--r-- 1 ceph ceph 866 Feb 13 08:23 ceph.conf -rw-rw-r-- 1 ceph ceph 1302 Feb 13 08:28 ceph-dashboard.crt -rw------- 1 ceph ceph 1708 Feb 13 08:28 ceph-dashboard.key -r-------- 1 ceph ceph 141 Feb 13 08:24 ceph.mgr.ceph-osdb1.keyring -r-------- 1 ceph ceph 141 Feb 13 08:24 ceph.mgr.ceph-osdb2.keyring -r-------- 1 ceph ceph 141 Feb 13 08:24 ceph.mgr.ceph-osdb3.keyring -r-------- 1 root root 151 Feb 13 09:01 prod.client.admin.keyring -rw-r--r-- 1 root root 863 Feb 13 09:01 prod.conf -rw-r--r-- 1 root root 92 Aug 29 16:38 rbdmap Kindly I need your help if I missing something because while running I am facing the following error: TASK [ceph-rbd-mirror : create a temporary file] ************************************************************************************** Tuesday 13 February 2024 09:58:59 +0000 (0:00:00.349) 0:04:00.698 ****** changed: [ceph-osdb1 -> ceph-osdb1] TASK [ceph-rbd-mirror : write secret to temporary file] ******************************************************************************* Tuesday 13 February 2024 09:58:59 +0000 (0:00:00.533) 0:04:01.232 ****** fatal: [ceph-osdb1 -> {{ groups[mon_group_name][0] }}]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ceph_rbd_mirror_remote_key' is undefined The error appears to be in '/opt/ceph-ansible/roles/ceph-rbd-mirror/tasks/configure_mirroring.yml': line 150, column 7, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: write secret to temporary file ^ here Kindly help

2 8

CONFIGURE THE CEPH OBJECT GATEWAY
by ashar.khan＠rapidcompute.com 15 Feb '24

15 Feb '24

I have configured Ceph S3 encryption successfully, and the configuration is created successfully. However, when I try to upload a file to the bucket, a failed request occurs. Could you please guide me on how to properly configure it? I follow this link: https://docs.ceph.com/en/quincy/radosgw/vault/#:~:text=aes256%2Dgcm96-,CONF…

1 0

Setting Alerts/Notifications for Full Buckets in Ceph Object Storage
by asad.siddiqui＠rapidcompute.com 15 Feb '24

15 Feb '24

Hi I'm currently working with Ceph object storage (version Reef), and I'd like to know how we can set up alerts/notifications for buckets when they become full.

1 0