- ceph-users - lists.ceph.io

Mounting A RBD Image via Kernal Modules
by matthew＠peregrineit.net 21 Mar '24

21 Mar '24

Hi All, I'm looking for a bit of advice on the subject of this post. I've been "staring at the trees so long I can't see the forest any more". :-) Rocky Linux Client latest version. Ceph Reef latest version. I have read *all* the doco on the Ceph website. I have created a pool (my_pool) and an image (my_image). I had activated the pool for RBD. I can run the `rbdmap map` command on the client and the image shows up as /dev/rbd0 (and also /dev/rbd/my_pool/my_image). But here's where I'm running into issues - and I'm pretty sure it's a 'Level 8' issue, so it'll be something simple that I'm just not "getting": Do I need to run `mkfs` on /dev/rbd0 before I try `mount /dev/rbd/my_pool/my_image /mnt/rbd_image`? The reason I ask is that I've tried to mount the image before I run 'mkfs' and I get back `mount: /mnt/rbd_image: wrong fs type, bad option, bad superblock on /dev/rbdo, missing codepage or helper program, or other error`. I've also tried to mount the image after I run 'mkfs' and I get back `mount: /mnt/rbd_image: can't read superblock on /dev/rbd0`. Basically, as I've said, I'm missing or don't understand *something* about this process - which is why I'm now seeking the collective wisdom of the Community. All help and advice greatly appreciated - thanks in advance Cheers Dulux-Oz

1 0

Need easy way to calculate Ceph cluster space for SolarWinds
by Michael Worsham 20 Mar '24

20 Mar '24

Is there an easy way to poll a Ceph cluster to see how much space is available and how much space is available per bucket? Looking for a way to use SolarWinds to monitor the entire Ceph cluster space utilization and then also be able to break down each RGW bucket to see how much space it was provisioned for and how much is available. -- Michael Get Outlook for Android<https://aka.ms/AAb9ysg> This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

3 5

Why a lot of pgs are degraded after host(+osd) restarted?
by Jaemin Joo 20 Mar '24

20 Mar '24

Hi all, While I am testing host failover, there are a lot of degraded pg after host(+osd) is up. In spite that it takes a short time to restart, I don't understand why pg should check all objects related to the failed host(+osd). I'd like to know how to prevent to become degraded pg when osd restart. FYI. degraded pg means "active+undersized+degraded+remapped+backfilling" pg state

3 2

#1359 (update) Ceph filesystem failure | Ceph filesystem probleem
by Postmaster C&CZ (Simon) 20 Mar '24

20 Mar '24

English follows Dutch ## Update 2024-03-19 Positief nieuws; we zijn nu bezig met het kopiëren van data uit CephFS. We konden het filesystem weer mounten met hulp van 42on, onze support club. We kopiëren nu de data en dat lijkt goed te gaan. Op elk moment kunnen we tegen problematische metadata aanlopen, dus afwachten en duimen. Als we de data uit CephFS hebben gekopieerd op tijdelijke storage, kunnen we verdere oplossingen voor de toekomst verzinnen en implementeren. NB: neem contact op met postmaster (mailto:postmaster@science.ru.nl) als je een urgent verzoek hebt voor een *kleine* set van files in een specifieke locatie, dan kunnen we daar prioriteit aan geven. Een Petabyte aan data kopiëren duurt weken/maanden, kleine datasets (< 1TB) kan wel relatief snel. bron: https://cncz.science.ru.nl/nl/cpk/1359 ===================== ## Update 2024-03-19 Somewhat good news; we are now copying the RDR data out of CephFS. We have been able to mount CephFS again with help from 42on (our support party). Copying to temporary storage is going OK, but we can run into issues at any time (or not). We're hopeful that this process will let us recover the data stored in CephFs and then we can look for future solutions. NB: contact postmaster (mailto:postmaster@science.ru.nl) if you have urgent requests for *small* sets of files in a particular location, so we may restore this with priority. A Petabyte of data takes weeks/months to copy, but a small amount (< 1TB) can be retrieved relatively fast. source: https://cncz.science.ru.nl/en/cpk/1359 -- Postmaster: Simon Oosthoek Postmaster Phone: +31 24 365 3535 Personal Phone: +31 24 365 2097

1 0

Leaked clone objects
by Frédéric Nass 19 Mar '24

19 Mar '24

Hello, Over the last few weeks, we have observed a abnormal increase of a pool's data usage (by a factor of 2). It turns out that we are hit by this bug [1]. In short, if you happened to take pool snapshots and removed them by using the following command 'ceph osd pool rmsnap {pool-name} {snap-name}' instead of using this command 'rados -p {pool-name} rmsnap {snap-name}' then you may have leaked clone objects (not trimmed) in your cluster, occupying space that you can't reclaim. You may have such leaked objects if (not exclusively): - 'rados df' reports CLONES for pools with no snapshots - 'rgw-orphan-list' (if RWG pools) reports objects that you can't 'stat' but for which 'listsnaps' shows a cloneid. 'ceph osd pool force-remove-snap {pool-name}' should have the OSDs re-trim these leaked clone objects when [2] makes it to quincy, reef, squid (and hopefully pacific). Hope this helps, Regards, Frédéric. [1] https://tracker.ceph.com/issues/64646 [2] https://github.com/ceph/ceph/pull/53545

1 0

MDS_CLIENT_LATE_RELEASE, MDS_SLOW_METADATA_IO, and MDS_SLOW_REQUEST errors and slow osd_ops despite hardware being fine
by Ivan Clayson 19 Mar '24

19 Mar '24

Hello everyone, We've been experiencing on our quincy CephFS clusters (one 17.2.6 and another 17.2.7) repeated slow ops with our client kernel mounts (Ceph 17.2.7 and version 4 Linux kernels on all clients) that seem to originate from slow ops on osds despite the underlying hardware being fine. Our 2 clusters are similar and are both Alma8 systems where more specifically: * Cluster (1) is Alma8.8 running Ceph version 17.2.7 with 7 NVMe SSD OSDs storing the metadata and 432 spinning SATA disks storing the bulk data in an EC pool (8 data shards and 2 parity blocks) across 40 nodes. The whole cluster is used to support a single file system with 1 active MDS and 2 standby ones. * Cluster (2) is Alma8.7 running Ceph version 17.2.6 with 4 NVMe SSD OSDs storing the metadata and 348 spinning SAS disks storing the bulk data in EC pools (8 data shards and 2 parity blocks). This cluster houses multiple filesystems each with their own dedicated MDS along with 3 communal standby ones. Nearly daily we often find that we're get the following error messages: MDS_CLIENT_LATE_RELEASE, MDS_SLOW_METADATA_IO, and MDS_SLOW_REQUEST. Along with these messages, certain files and directory cannot be stat-ed and any processes involving these files hang indefinitely. We have been fixing this by: 1. First, finding the oldest blocked MDS op and the inode listed there: ~$ ceph tell mds.${my_mds} dump_blocked_ops 2>> /dev/null | grep -c description "description": "client_request(client.251247219:662 getattr AsLsXsFs #0x100922d1102 2024-03-13T12:51:57.988115+0000 caller_uid=26983, caller_gid=26983)", # inode/ object of interest: 100922d1102 2. Second, finding all the current clients that have a cap for this blocked inode from the faulty MDS' session list (i.e. ceph tell mds.${my_mds} session ls --cap-dump) and then examining the client whose had the cap the longest: ~$ ceph tell mds.${my_mds} session ls --cap-dump ... 2024-03-13T13:01:36: client.251247219 2024-03-13T12:50:28: client.245466949 3. Then on the aforementioned oldest client, get the current ops in flight to the OSDs (via the "/sys/kernel/debug/ceph/*/osdc" files) and get the op corresponding to the blocked inode along with the OSD the I/O is going to: root@client245466949 $ grep 100922d1102 /sys/kernel/debug/ceph/*/osdc 48366 osd79 2.249f8a51 2.a51s0 [79,351,232,179,107,195,323,14,128,167]/79 [79,351,232,179,107,195,323,14,128,167]/79 e374191 100922d1102.000000f5 0x400024 1 write # osd causing errors is osd.79 4. Finally, we restart this "hanging" OSD where this results in ls and I/O on the previously "stuck" files no longer "hanging" . Once we get this OSD for which the blocked inode is waiting for, we can see in the system logs that the OSD has slow ops: ~$ systemctl --no-pager --full status ceph-osd@79 ... 2024-03-13T12:49:37 -1 osd.79 374175 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.245466949.0:41350 2.ca4s0 2.ce648ca4 (undecoded) ondisk+write+known_if_redirected e374173) ... Files that these "hanging" inodes correspond to as well as the directories housing these files can't be opened or stat-ed (causing directories to hang) where we've found restarting this OSD with slow ops to be the least disruptive way of resolving this (compared with a forced umount and then re-mount on the client). There are no issues with the underlying hardware for either the osd reporting these slow ops or any other drive within the acting PG and there seems to be no correlation between what processes are involved or what type of files these are. What could be causing these slow ops and certain files and directories to "hang"? There aren't workflows being performed that generate a large number of small files nor are there directories with a large number of files within them. This seems to happen with a wide range of hard-drives and we see this on SATA and SAS type drives where our nodes are interconnected with 25 Gb/s NICs so we can't see how the underlying hardware would be causing any I/O bottlenecks. Has anyone else seen this type of behaviour before and have any ideas? Is there a way to stop these from happening as we are having to solve these nearly daily now and we can't seem to find a way to reduce them. We do use snapshots to backup our cluster where we've been doing this for ~6 months and these issues have only been occurring on and off for a couple of months but much more frequently now. Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

4 4

Adding new OSD's - slow_ops and other issues.
by Jesper Agerbo Krogh [JSKR] 19 Mar '24

19 Mar '24

Hi All We have a nicely functional NVMe cluster running – but in the process of expanding it we have encountered slow_ops The drill was: 1. Maintaince mode 2. Ceph orch osd add 3. Change weights to 1 4. Disable maintaince mode. Somethings goes wrong around 3 – where slowops begins to kick in – but restarting the OSD’s underneath can make it go away – tested twice. In one situation the 2) step failed to bring up the OSD – causing the entire process to work 100% correctly. Thus changing weight from 7.68 to 1 In above process on live OSD seem to be the issue. It is actually a flaw that we managed to get the entire cluster created with weight 1 instead of 7.68 in the first place thus: 1. Can we bulk change weights on all existing OSD’s without huge data movement? If so – how? 2. Can we ceph orch add – but with a specific initial weight? Thanks. Best regards, Jesper Agerbo Krogh Director Digitalization Digitalization [cid:image001.png@01DA794D.5131C5B0]<http://www.linkedin.com/company/haldor-topsoe> Topsoe A/S Haldor Topsøes Allé 1 2800 Kgs. Lyngby Denmark Phone (direct): 27773240 [cid:image002.jpg@01DA794D.5131C5B0]<http://www.linkedin.com/company/haldor-topsoe> [cid:image003.jpg@01DA794D.5131C5B0] <https://twitter.com/topsoe_official> [cid:image004.jpg@01DA794D.5131C5B0] <https://www.facebook.com/TopsoeOfficial> [cid:image005.jpg@01DA794D.5131C5B0] <https://www.topsoe.com/our-resources/knowledge/videos/> Read more at topsoe.com<http://www.topsoe.com> Topsoe A/S and/or its affiliates. This e-mail message (including attachments, if any) is confidential and may be privileged. It is intended only for the addressee. Any unauthorised distribution or disclosure is prohibited. Disclosure to anyone other than the intended recipient does not constitute waiver of privilege. If you have received this email in error, please notify the sender by email and delete it and any attachments from your computer system and records.

2 1

Call for interest: VMWare Photon OS support in Cephadm
by Ernesto Puerta 18 Mar '24

18 Mar '24

Hi Cephers, This is to sound out potential interest in supporting VMWare's Photon OS distribution with Ceph/Cephadm. Photon OS is a lightweight distribution that is optimized for deploying containers within virtual machines. It is based on systemd, includes Docker, and uses the tdnf (Tiny DNF) package manager. Cephadm already supports Azure Linux/CBL-Mariner <https://github.com/microsoft/azurelinux>, which is a Microsoft derivative from the Photon OS project for the Azure ecosystem. Thus, no major changes would be required. If you're interested and willing to take part in the deployment beta-test, please drop us a line! Thank you! Kind Regards, Ernesto

2 2

Re: Fwd: Ceph fs snapshot problem
by Marcus 18 Mar '24

18 Mar '24

Hi, There is no such attribute. /mnt: ceph.dir.subvolume: No such attribute I did not have getfattr installed so needed to install attr package. Can it be that this package was not installed when fs was created so ceph.dir.subvolume could not be set at creation? Did not get any warnings at creation though. Thanks for you help!! On lör, mar 16 2024 at 00:53:22 +0530, Neeraj Pratap Singh <neesingh(a)redhat.com> wrote: > Can u pls do getfattr on root directory and tell whats the output? > Run this command: getfattr -n ceph.dir.subvolume /mnt > > On Thu, Mar 14, 2024 at 4:38 PM Marcus <marcus(a)marcux.org > <mailto:marcus@marcux.org>> wrote: >> >> Hi all, >> I have just setup a small ceph cluster with ceph fs. >> The setup is reef 18.2.1 on Debian bookworm. >> The system is up and running the way it should, >> though I have a problem with ceph fs snapshots. >> >> When I read the doc I should be able to make a >> snapshot in any directory in the filesystem. >> I can do a snapshot in the root of the filesystem >> but if I try somewhere else I get: >> Operation not permitted >> This is the same if I do it with mkdir or >> with ceph fs subvolume snapshot create ... >> >> I have created an auth client with rws: >> [client.snap-mount] >> key = **** >> caps mds = "allow rws fsname=gds-common" >> caps mon = "allow r fsname=gds-common" >> caps osd = "allow rw tag cephfs data=gds-common" >> >> Where the filsystem is called gds-common, >> saved in a file on the client: >> /etc/ceph/ceph.client.snap-mount.keyring >> >> I mount ceph fs with: >> mount -t ceph :/ -o name=snap-mount /mnt >> >> If I create a snapshot in root, it works fine, as in: >> mkdir /mnt/.snap/mysnap >> I also notice that in every subdir there is a "snapshot dir" as well >> with the name _mysnap_1, as in: >> /mnt/dir/.snap/_mysnap_1 >> >> My guess that is is a part of the snapshot system, this "snapshot" >> dissapear when the snapshot is removed with: >> rmdir /mnt/.snap/mysnap >> >> If I try to make a snapshot in another directory this does not work: >> mkdir /mnt/dir/.snap/othersnap >> Get the error: >> cannot create directory ‘/mnt/dir/.snap/othersnap’: Operation >> not >> permitted >> >> It is the same thing on the commandline, root works: >> ceph fs subvolume snapshot create gds-common / fromcmd >> >> But not in a subdir: >> ceph fs subvolume snapshot create gds-common /dir dirsnap >> Error EINVAL: invalid value specified for ceph.dir.subvolume >> >> I also notice that when you use any command of type: >> ceph fs subvolume snapshot ... >> You get a new directory (volumes) in the root: >> /mnt/volumes/_legacy/6666cd76f96956469e7be39d750cc7d9.meta >> >> I do not know if I am missing something, some lacking of >> config or so. >> >> Thanks for your help!! >> >> Best regards >> Marcus >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io>

2 2

activating+undersized+degraded+remapped
by Deep Dish 17 Mar '24

17 Mar '24

Hello I found myself in the following situation: [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive pg 4.3d is stuck inactive for 8d, current state activating+undersized+degraded+remapped, last acting [4,NONE,46,NONE,10,13,NONE,74] pg 4.6e is stuck inactive for 9d, current state activating+undersized+degraded+remapped, last acting [NONE,27,77,79,55,48,50,NONE] pg 4.cb is stuck inactive for 8d, current state activating+undersized+degraded+remapped, last acting [6,NONE,42,8,60,22,35,45] I have one cephfs with two backing pools -- one for replicated data, the other for erasure data. Each pool is mapped to REPLICATED/ vs. ERASURE/ directories on the filesystem. The above pgs. are affecting the ERASURE pool (5+3) backing the FS. How can I get ceph to recover these three PGs? Thank you.

4 4