- ceph-users - lists.ceph.io

CLT weekly notes October 11th 2023
by Adam King 11 Oct '23

11 Oct '23

Here are the notes from this week's CLT call. The call focused heavily on release process, specifically around figuring out which patches are required for a release. - 17.2.7 status - A few more FS PRs and one core PR then we can start release process - Trying to finalize list of PRs needed for 18.2.1 - General discussion about the process for getting the list of required PRs for a given release - Using per-release github milestones. E.g. a milestone specifically for 18.2.1 rather than just reef - Would require fixing some scripts that refer to the milestone - https://github.com/ceph/ceph/blob/main/src/script/backport-resolve-issue - For now, continue using etherpad until something more automated exists - Create pads a lot earlier - Could use existing clt call to try to finalize required PRs for releases - should be on agenda for every clt call - couple of build related PRs that were stalled. - for a while, it's not possible to build w/FIO - PR https://github.com/ceph/ceph/pull/53346 - for a while, it's not possible to "make (or ninja) install" with dashboard disabled - PR https://github.com/ceph/ceph/pull/52313 - Some more general discussion of how to get more attention for build PRs - Laura will start grouping some build PRs with RADOS PRs for build/testing in the ci - Can make CI builds with CMAKE_BUILD_TYPE=Debug - https://github.com/ceph/ceph-build/pull/2167 - https://github.com/ceph/ceph/pull/53855#issuecomment-1751367302 - https://shaman.ceph.com/builds/ceph/wip-batrick-testing-20231006.014828-deb… - relies on us removing centos 8 from all testing suites and dropping that as a build target - Last Pacific? - Yes, 17.2.7, then 18.2.1, then 16.2.15 (final) - PTLs will need to go through and find what backports still need to get into pacific - A lot of open pacific backports right now Thanks, - Adam King

1 0

Ceph 18: Unable to delete image after imcomplete migration "image being migrated"
by Rhys Goodwin 11 Oct '23

11 Oct '23

Hi Folks, I'm running Ceph 18 with OpenStack for my lab (and home services) in a 3 node cluster on Ubuntu 22.04. I'm quite new to these platforms. Just learning. This is my build, for what it's worth: https://blog.rhysgoodwin.com/it/openstack-ceph-hyperconverged/ I got myself into some trouble as follows. This is the sequence of events: I don't recall when but at some stage I must have tried an image migration from one pool to another. The source pool/image is infra-pool/sophosbuild I don't know what the target would have been. In any case on my travels, I found the infra-pool/sophosbuild image in the trash: rhys@hcn03:/imagework# rbd trash ls --all infra-pool 65a87bb2472fe sophosbuild I tried to delete it but got the following: rhys@hcn03:/imagework# rbd trash rm infra-pool/65a87bb2472fe 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28bbfff640 -1 librbd::ImageState: 0x7f28a804b600 failed to open image: (30) Read-only file system 2023-10-06T04:23:13.775+0000 7f28a2ffd640 -1 librbd::image::RemoveRequest: 0x7f28a8000b90 handle_open_image: error opening image: (30) Read-only file system rbd: remove error: (30) Read-only file systemRemoving image: 0% complete...failed. Next, I tried to restore the image, and this also failed: rhys@hcn03:/imagework:# rbd trash restore infra-pool/65a87bb2472fe librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4) Probably stupidly, I followed the steps in this post: https://www.spinics.net/lists/ceph-users/msg72786.html to change offset 07 from 02 (TRASH_IMAGE_SOURCE_MIGRATION) in omap value to 00(TRASH_IMAGE_SOURCE_USER) After this I was able to restore the image successfully. However, I still could not delete it: rhys@hcn03:/imagework:# rbd rm infra-pool/sophosbuild 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::RefreshRequest: image being migrated 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-10-06T05:52:30.708+0000 7ff5937fe640 -1 librbd::ImageState: 0x564d3f83d680 failed to open image: (30) Read-only file system Removing image: 0% complete...failed.rbd: delete error: (30) Read-only file system I tried to abort the migration with: root@hcn03:/imagework# rbd migration abort infra-pool/sophosbuild This took a few mins but failed at 99% (sorry, terminal scroll back lost) So now I'm stuck, I don't know how to get rid of this image and while everything is otherwise healthy in the cluster, the dashboard is throwing errors when it tries to enumerate the images in that pool. I'm considering migrating the good images off this pool and deleing the pool. But I don't even know if I'll be allowed to delete the pool while this issue is present. Any advice would be much appreciated. Kind regards, Rhys

3 8

cephadm configuration in git
by Kamil Madac 11 Oct '23

11 Oct '23

Hello ceph community, Currently we have deployed ceph clusters with ceph-ansible and whole configuration (number od daemons, osd configurations, rgw configurations, crush configuration, ...) of each cluster is stored in git and ansible variables and we can recreate clusters with ceph-ansible in case we need it. To change the configuration of a cluster we change appropriate Ansible variable, we test it on testing cluster and if new configuration works correctly we apply it on prod cluster. Is it possible to do it with cephadm? Is it possible to have some config files in git and then apply same cluster configuration on multiple clusters? Or is this approach not aligned with cephadm and we should do it different way? Kamil Madac

3 2

What's the best practices of accessing ceph over flaky network connection?
by nanericwang＠gmail.com 11 Oct '23

11 Oct '23

(transferred from https://github.com/ceph/ceph-csi/discussions/4181) > What's the best practices of accessing ceph over flaky network connection? For example, can I setup a local dm-cache binding ceph with a local SSD to buffer the I/O? Thanks. A flaky network will usually be quite problematic. There is no guarantee that data has not been modified on an other system once the network comes back after a temporary interruption. In case the data was changed remotely, and something wrote to the local dm-cache device, a split-brain can happen. Someone needs to decide which side of the data is the right one. Maybe it is possible to tune RBD for working with dm-cache and network interruptions. Or, for CephFS, you may want to look into FS-Cache (I don't know if CephFS supports that though). The best venue to ask about guidance and experience from others is at https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/ . The discussions in this GitHub project are for Ceph-CSI, which is _only_ a driver to provision/mount Ceph based storage.

1 0

ceph-dashboard python warning with new pyo3 0.17 lib (debian12)
by DERUMIER, Alexandre 11 Oct '23

11 Oct '23

Hi, on debian12, ceph-dashboard is throwing a warning "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process" Seem to be related to pyo3 0.17 change https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/… " Each #[pymodule] can now only be initialized once per process To make PyO3 modules sound in the presence of Python sub-interpreters, for now it has been necessary to explicitly disable the ability to initialize a #[pymodule] more than once in the same process. Attempting to do this will now raise an ImportError. "

3 3

Copying big objects (>5GB) doesn't work after upgrade to Quincy on S3
by Arvydas Opulskis 11 Oct '23

11 Oct '23

Hi all, after upgrading our cluster from Nautilus -> Pacific -> Quincy we noticed we can't copy bigger objects anymore via S3. An error we get: "Aws::S3::Errors::EntityTooLarge (Aws::S3::Errors::EntityTooLarge)" After some tests we have following findings: * Problems starts for objects bigger than 5 GB (multipart limit) * Issue starts after upgrading to Quincy (17.2.6). In latest Pacific (16.2.13) it works fine. * For Quincy it works ok with AWS S3 CLI "cp" command, but doesn't work using AWS Ruby3 SDK client with copy_object command. * For Pacific setup both clients work ok * From RGW logs seems like AWS S3 CLI client handles multipart copying "under the hood", so it is succesful. It is stated in AWS documentation, that for uploads (and copying) bigger than 5GB files we should use multi part API for AWS S3. For some reason it worked for years in Ceph and stopped working after Quincy release, even I couldn't find something in release notes addressing this change. So, is this change permanent and should be considered as bug fix? Both Pacific and Quincy clusters were running on Rocky 8.6 OS, using Beast frontend. Arvydas

2 2

Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.
by Konstantin Shalygin 10 Oct '23

10 Oct '23

Hi, You need revert your packages from Quincy to Pacific. `dnf downgrade ceph-mon` command should help with this k Sent from my iPhone > On Oct 11, 2023, at 03:22, Waywatcher <sconnary32(a)gmail.com> wrote: > > I am unable to get any of the current monitors to run. They all fail to start

1 0

Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.
by Waywatcher 10 Oct '23

10 Oct '23

I upgraded my CEPH cluster without properly following the mon upgrade so they were no longer on leveldb. Proxmox and CEPH were updated to the latest for the current release. https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy 1. The upgrade to Quincy states a recommendation that Mons are using RocksDB. 2. Leveldb support has been removed from quincy. The monitors were still running as leveldb. 1. Does this mean the mons cannot work at all since they are levelDB? I upgraded all nodes to the quincy release 17.2.6 and restarted the mons. At this point the cluster stopped responding. `ceph` commands do not work since the service fails to start. Are there steps for recovery? 1) Roll back to Pacific without being able to use CEPH commands (ceph orch upgrade start --ceph-version <version>). 2) Rebuild the monitors using data from the OSDs while maintaining Quincy release. 3) Is this actually related to the bug about 17.2.6 (which is what Proxmox/CEPH upgrades to) https://tracker.ceph.com/issues/58156 ? I ran the upgrade on another cluster prior to this without issue. The Mons were set with RocksDB and running on Quincy 17.2.6. I appreciate any suggestions.

3 2

Hardware recommendations for a Ceph cluster
by Gustavo Fahnle 10 Oct '23

10 Oct '23

Hi, Currently, I have an OpenStack installation with a Ceph cluster consisting of 4 servers for OSD, each with 16TB SATA HDDs. My intention is to add a second, independent Ceph cluster to provide faster disks for OpenStack VMs. The idea for this second cluster is to exclusively provide RBD services to OpenStack. I plan to start with a cluster composed of 3 mon/mgr nodes similar to what we currently have (3 virtualized servers with VMware) with 4 cores, 8GB of memory, 80GB disk and 10GB network each server. In the current cluster, these nodes have low resource consumption, less than 10% CPU usage, 40% memory usage, and less than 100Mb/s of network usage. For the OSDs, I'm thinking of starting with 3 or 4 servers, specifically Supermicro AS-1114S-WN10RT, each with: 1 AMD EPYC 7713P Gen 3 processor (64 Core, 128 Threads, 2.0GHz) 256GB of RAM 2 x NVME 1TB for the operating system 10 x NVME Kingston DC1500M U.2 7.68TB for the OSDs Two Intel NIC E810-XXVDA2 25GbE Dual Port (2 x SFP28) PCIe 4.0 x8 cards Connected to 2 MikroTik CRS518-16XS-2XQ-RM switches at 100GbE per server Connection to OpenStack would be via 4 x 10GB to our core switch. I would like to hear opinions about this configuration, recommendations, criticisms, etc. If any of you have references or experience with any of the components in this initial configuration, they would be very welcome. Thank you very much in advance. Gustavo Fahnle

3 5

Announcing go-ceph v0.24.0
by John Mulligan 10 Oct '23

10 Oct '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.24.0 Changes include fixes to the rgw admin and rbd packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0