- ceph-users - lists.ceph.io

Issue with using the block device inside a pod.
by Kushagr Gupta 17 Nov '23

17 Nov '23

Hi Team, Components: Kubernetes, Ceph Problem statement: We are trying to integrate Ceph with kubernetes. We are unable to utilize the block volume mode in the pod. Description: OS: Almalinux 8.8 Ceph version: 18.2 Kubernetes version: 1.28.2 We have deployed a single node kubernetes cluster and a 3 node ceph cluster. We are trying to integrate kubernetes with ceph using csi-rbd plugin. We have used the following link to do the same: https://docs.ceph.com/en/reef/rbd/rbd-kubernetes/ We are strugling with the usage of the pvc we created in the block mode. Kindly refer the "raw-block-pvc-4.yaml". We created a pod using the file "raw-block-pod-4.yaml". The pod was created successfully. The device path we used was /dev/xvda. We are unable to use this path inside the pod. Its not gettting shown in the lsblk of the pod and its not even shown in the df -kh of the pod. When we try to mount /dev/xvda on /tmp but we get permission denied error. We get a similar error when we try to make partiotion of the disk. The partition gets created but partiotion table is not updated. At the same time we are able to format the disk using the command mkfs.ext4 /dev/xvda. Kindly refer the output: " [root@kube-cluster-1 ceph_provisioner]# k exec -it pod-with-raw-block-volume-2 sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. sh-4.4# hostname pod-with-raw-block-volume-2 sh-4.4# df -kh Filesystem Size Used Avail Use% Mounted on overlay 70G 14G 57G 19% / tmpfs 64M 0 64M 0% /dev tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/mapper/almalinux-root 70G 14G 57G 19% /etc/hosts shm 64M 0 64M 0% /dev/shm tmpfs 63G 12K 63G 1% /run/secrets/ kubernetes.io/serviceaccount tmpfs 32G 0 32G 0% /proc/acpi tmpfs 32G 0 32G 0% /proc/scsi tmpfs 32G 0 32G 0% /sys/firmware sh-4.4# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 1G 0 loop loop1 7:1 0 500G 0 loop sda 8:0 0 446.6G 0 disk |-sda1 8:1 200M 0 part |-sda2 8:2 1M 0 part |-sda3 8:3 1G 0 part `-sda4 8:4 0 445.4G 0 part sdb 8:16 0 893.3G 0 disk rbd0 252:0 0 1G 0 disk rbd1 252:16 0 500G 0 disk rbd2 252:32 0 100G 0 disk sh-4.4# ls /dev core fd full mqueue null ptmx pts random shm stderr stdin stdout termination-log tty urandom xvda zero sh-4.4# " Altough we are able to see a volume corresponding to the PV. But we are not able to use the block device. Could anyone help me out. Thanks and Regards, Kushagra Gupta

2 1

Re: Large size differences between pgs
by Miroslav Svoboda 17 Nov '23

17 Nov '23

Namely, the problem what I am trying to solve is that with such a large cluster I will lose a lot of capacity like unused. I have deviation set to value 1 at the balancer, that is, if I'm not mistaken +-1pg per OSD and then due to the size dispersion between the largest and smallest PGs on the one OSDs host.Svoboda Miroslav -------- Původní zpráva --------Od: Anthony D'Atri <aad(a)dreamsnake.net> Datum: 15.11.23 21:54 (GMT+01:00) Komu: Miroslav Svoboda <miroslav.svoboda(a)metode.cz> Předmět: Re: [ceph-users] Large size differences between pgs How are you determining PG size?> On Nov 15, 2023, at 15:46, Miroslav Svoboda <miroslav.svoboda(a)metode.cz> wrote:> > Hi,is it possible decrease large size differences between pgs? I have 5PB cluster and differences between smalest and bigest pgs are somewhere about 25GB.thanks,Svoboda Miroslav> _______________________________________________> ceph-users mailing list -- ceph-users(a)ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Different behaviors for ceph kernel client in limiting IOPS when data pool enters `nearfull`?
by Matt Larson 17 Nov '23

17 Nov '23

On CentOS 7 systems with the CephFS kernel client, if the data pool has a `nearfull` status there is a slight reduction in write speeds (possibly 20-50% fewer IOPS). On a similar Rocky 8 system with the CephFS kernel client, if the data pool has `nearfull` status, a similar test shows write speeds at different block sizes shows the IOPS < 150 bottlenecked vs the typical write performance that might be with 20000-30000 IOPS at a particular block size. Is there any way to avoid the extremely bottlenecked IOPS seen on the Rocky 8 system CephFS kernel clients during the `nearfull` condition or to have behavior more similar to the CentOS 7 CephFS clients? Do different OS or Linux kernels have greatly different ways they respond or limit on the IOPS? Are there any options to adjust how they limit on IOPS? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

3 6

CFP closing soon: Everything Open 2024 (Gladstone, Queensland, Australia, April 16-18)
by Tim Serong 16 Nov '23

16 Nov '23

Everything Open (auspiced by Linux Australia) is happening again in 2024. The CFP closes at the end of this weekend (November 19): https://2024.everythingopen.au/programme/proposals/ More details below. -------- Forwarded Message -------- Date: Sun, 15 Oct 2023 09:16:31 +1000 From: Everything Open <contact(a)everythingopen.au> To: eo-announce(a)lists.linux.org.au, announce(a)lists.linux.org.au Subject: [Announce] Everything Open 2024: Call for Sessions Now Open User-Agent: Roundcube Webmail/1.1.1 Submit your session proposals today - the Everything Open 2024 Call for Sessions is now open. ## Call for Sessions We invite you to submit a session proposal on a topic you are familiar with via our proposals portal at https://2024.everythingopen.au/programme/proposals/. The Call for Sessions will remain open until 11:59pm on Sunday 19 November 2023 anywhere on earth (AoE). There will be multiple streams catering for a wide range of interest areas across the many facets of open technology, including Linux, open source software, open hardware, standards, formats and documentation, and our communities. In keeping with the conference’s aim to be inclusive to all community members, presentations can be aimed at any level, ranging from technical deep-dives through to beginner and intermediate level presentations for those who are newer to the subject. There will be two types of sessions at Everything Open: talks and tutorials. Talks will nominally be 45 minutes long on a single topic presented in lecture format. We will also have a few short talk slots of 25 minutes available, which are perfect for people new to presenting at a conference. Tutorials are interactive and hands-on in nature, presented in classroom format. Each accepted session will receive one Professional level ticket to attend the conference. The Session Selection Committee is looking forward to reading your submissions. We would also like to thank them for coming together and volunteering their time to help put this conference together. ## Sponsor Early As usual, we have a range of sponsorship opportunities available, for the conference overall as well as the ability to contribute towards specific parts of the event. We encourage you to sponsor the conference early, to get the maximum promotion during the lead up to the event. If you or your organisation is interested in sponsoring Everything Open, please get in touch via https://2024.everythingopen.au/sponsors/prospectus/. ---- Read this online at https://2024.everythingopen.au/news/call-for-sessions-open/ _______________________________________________ announce mailing list announce(a)lists.linux.org.au http://lists.linux.org.au/mailman/listinfo/announce ----- End forwarded message -----

1 0

Upgrading From RHCS v4 to OSS Ceph
by jarulsam＠uwyo.edu 16 Nov '23

16 Nov '23

Hi everyone, I have a storage cluster running RHCS v4 (old, I know) and am looking to upgrade it soon. I would also like to migrate from RHCS to the open source version of Ceph at some point, as our support contract with RedHat for Ceph is likely going to not be renewed going forward. I was wondering if anyone has any advice as to how to upgrade our cluster with minimal production impact. I have the following server configuration: + 3x monitors + 3x metadata servers + 2x RadosGWs with 2x servers running HAproxy and keepalived for HA RadosGWs. + 19x OSDs - 110TB HDD and 1TB NVMe each. (Total ~2.1PB raw) Currently, I have RHCS v4 installed baremetal on RHEL 7. I see that newer versions of Ceph require containerized deployments so I am thinking it is best to first migrate to a containerized installation then try and upgrade everything else. My first inclination is to do the upgrade like this: 1. Move existing installation to containerized, maintain all the same versions and OS installations. 2. Pull one monitor, fresh reinstall RHEL 9, reinstall RHCS v4, readd to cluster. Repeat for all the monitors. 3. Pull one MDS, do the same as step 2 but for MDS. 4. Pull one RadosGW, do the same as step 2 but for RadosGW. 5. Pull one OSD, rebalance, fresh reinstall RHEL 9, reinstall RHCS v4, readd to cluster, rebalance. Repeat for all OSDs. 6. Upgrade RHCS to OSS Ceph Octopus -> Pacific -> Quincy -> Reef. Does such a plan seem reasonable? Are there any major pitfalls of an approach like this? Ideally, I would just rebuild an entire new cluster on Ceph Reef, however there are obvious budgetary issues with such a plan. My biggest concerns are with moving to a containerized installation, then migrating from RHCS to OSS Ceph. Any advice or feedback is much appreciated. Best, Josh

2 1

Debian 12 support
by nessero karuzo 16 Nov '23

16 Nov '23

Hi to all ceph community. I have a question about Debian 12 support for ceph 17. I didn’t find repo for that release at https://download.ceph.com/, does it supports and how to install ceph Quincy on Debian 12 ? Thank you to those who will answer.

9 12

ceph-volume lvm new-db throws errors
by Giuliano Maggi 16 Nov '23

16 Nov '23

Hi, ceph-volume lvm new-db does not work as expected due to missing /var/lib/ceph/osd/ceph-OSDID. However, the database device seems to be added to the OSD. Until now, it is unclear to me if the DB device was actually successfully added to the OSD or this is a bug in ceph version 17.2.7 The detailed explanation is in a github ticket: https://github.com/ceph/ceph/pull/52941#issuecomment-1814251890 does anyone else have this issue ? Any feedback is welcome of course. Thanks, Giuliano.

1 0

[CEPH] OSD Memory Usage
by Nguyễn Hữu Khôi 16 Nov '23

16 Nov '23

Hello, I am using a CEPH cluster. After monitoring it, I set: ceph config set osd osd_memory_target_autotune false ceph config set osd osd_memory_target 1G Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case? Ceph version: Quincy OSD: 3 nodes with 11 nvme each and 512GB ram per node. CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket. Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000 Thank you very much. Nguyen Huu Khoi

3 8

remove spurious data
by Giuliano Maggi 16 Nov '23

16 Nov '23

Hi, I’d like to remove some “spurious" data: root@nerffs03:/# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.0 PiB 1.0 PiB 47 GiB 47 GiB 0 TOTAL 1.0 PiB 1.0 PiB 47 GiB 47 GiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 70 MiB 19 209 MiB 0 327 TiB .nfs 3 32 572 B 4 36 KiB 0 327 TiB root@nerffs03:/# as you can see, there are no data pools. The 47GiB could be from previous pools/filesystems that I used for testing. how can I remove this “spurious” data without affecting the OSDs? I was checking "ceph-volume lvm zap”, but I am not sure it this is the right one. (I am running ceph version quincy 17.2.7) Thanks, Giuliano.

2 1

planning upgrade from pacific to quincy
by Simon Oosthoek 16 Nov '23

16 Nov '23

Hi All (apologies if you get again, I suspect mails from my @science.ru.nl account get dropped by most receiving mail servers, due to the strict DMARC policy (p=reject) in place) after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation is via packages from ceph.com, unless it's quincy from ubuntu. ceph versions: "mon": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "mgr": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3}, "osd": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 252, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12 }, "mds": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 2 }, "rgw": {"ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 }, "overall": {"ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 260, "ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)": 12, "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 8 } The OS on the mons and mdss are still ubuntu 18.04, the osds are a mix of ubuntu 18 and ubuntu 20. The gateways are ubuntu 22.04, which is why these are already on quincy. The plan is to move to quincy and eventually cephadm/containered ceph, since that is apparently "the way to go", though I have my doubts. The steps we think are the right order are: - reinstall the mons with ubuntu 22.04 + quincy - reinstall the osds (same) - reinstall the mdss (same) Once this is up and running, we want to investigate and migrate to cephadm orchestration. Alternative appear to be: move to orchestration first and then upgrade ceph to quincy (possibly skipping the ubuntu upgrade?) Another alternative could be to upgrade to quincy on ubuntu 18.04 using packages, but I haven't investigated the availability of quincy packages for ubuntu 18.04 (which is out of free (LTS) support by canonical) Cheers /Simon -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

1 0