- ceph-users - lists.ceph.io

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

FS S5850 - 7 nodes lots and continuous "slow OSD heartbeats"
by Steven Vacaroaia 25 Apr '25

25 Apr '25

Hi, We are in the process of using for production a 7 host reef ( 18.2.2) cluster and we noticed lots and continuous "slow OSD heartbeats" although there is no clients, no data and the nodes are using 2 x 25GB for Frontend and 2 x 100GB for backend The logical culprits is the network we have 2 x S5850 with MLAG and LACP I was wondering if any of you are familiar with FS and can provide some guidance as to how to properly configured ( or troubleshoot) Note We are running in a air gapped environment so cannot provide outputs/config but here are some pictures Many thanks Steven [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png]

2 2

Increase amount of OSDs on nodes
by Florian Schwab 24 Apr '25

24 Apr '25

Hi everyone, I’m currently facing the following challenge were I’d like to hear what the community thinks on how to solve it. We have a ceph cluster (only using RGW) with which we had some performance issues when initially setting it up and instead of using all 60 HDDs installed in each server we only use half (30) of the HDDs - they exist but are set to “out”. Using only half of the OSDs per server solved our performance issues - not the part of my question just a bit of backstory for this weird setup. We have now optimised things and are looking into getting those OSDs back “in”. However we have quite some client load on the system and we want to keep the any necessary downtime (for users) low. The HDDs are only used for the buckets.data pool (using erasure coding) as the other pools necessary for RGW are on separate NVMe drives. To achieve this I was looking into using pgremapper and upmap-remapped. My expectation was to be able to remap all PGs to stay on the “old” OSDs (the ones that are currently “in”) and have zero PGs on the “new” OSDs (the ones that are currently “out”). If that part is done I thought somehow I can double the amount of PGs to cover the now doubled capacity for the pool. What I tried so far (with some variations): - Set norecover and nobackfill flags - Set “new” OSDs to “in” - Use pgremapper and/or upmap-remapped to create upmap entries so the acting OSDs of a PGs are unchanged - Increase the amount of PGs - Map new PGs to “new” OSDs - (Maybe remove the upmap entries for the PGs one at a time over an extend period to allow for automatic balancing again) I’m trying this on a smaller test setup with 6 servers and 8 OSDs per server (4 in / 4 out) so 48 OSDs in total. For testing I filled the “in” OSDs to around 50% so the overall used capacity is also 50% - when bringing the other 24 OSDs "in" this is of course means only 25% of the overall capacity is used. The following information is based on this test setup. The pgremapper/upmap-remapped part is where I’m currently stuck as I always end up with around 2% of objects being misplaced. What I was also able to manage is that if I after setting all OSDs to “in”, remap the PGs, increase the PG count and remap again I end up around 3% of misplaced objects however the additional/new OSDs then will have zero PGs again. Is it somehow possible to add the OSDs and double the PG count without causing any (or only minimal) backfill? If possible I also want to bring in the “new” OSDs in batches so we can check the performance after each batch to avoid having issues in the final state with all OSDs set to “in”. Thanks in advance for any input. Cheers, Florian

2 1

Inquiry About IO Ordering in Ceph Octopus Version
by 段世博 24 Apr '25

24 Apr '25

We are currently using the Ceph Octopus version and have some questions regarding a specific commit in librbd ( https://github.com/ceph/ceph/commit/081d28ae7ca46fd1f40034cc558def77a95a9294 ). Could you please clarify why Ceph implements ordering for overlapping IO? Our understanding is that overlapping IO in block storage can typically be parallelized. We are curious about which components of Ceph depend on this feature. Additionally, we noticed that this feature was removed in the subsequent Pacific version. Could you provide some insight into the reasons behind this decision? Thank you for your assistance. Best regards, shibo

2 1

Removing a zonegroup from multi-site cluster
by Boris 24 Apr '25

24 Apr '25

Hi, I wanted to remove a zonegroup from a multisite cluster. I deleted it from the master with radosgw-admin zonegroup delete --rgw-zonegroup=zonegroup radosgw-admin period update --commit After that, I had to restart all RGW daemons, because the metadata sync stopped. The zonegroup is gone on the master, but it is still listed in all other sites. I checked the current period on all sites and they are all the same, but the deleted zonegroup is still in the radosgw-admin period get on all zonegroups that are not the master. How do I proceed? Best wishes Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

ceph deployment best practice
by gagan tiwari 23 Apr '25

23 Apr '25

Hi Guys, We have an HPC environment which currently has a single master host that stores the entire data around 100T and exports that data via NFS to the clients. We are using OpenZFS on the single master host. But now we need to store much more data around 500T and we are facing scalability issues with ZFS as it can't be scaled out. So, we are considering replacing OpenZFS with Ceph. I have gone the online docs on https://docs.ceph.com and I have come up with the following ceph deployment plan :- We would start deploying Ceph with 4 hosts ( HP Proliant servers ) each running RockyLinux 9. One of the hosts called ceph-adm will be smaller one and will have following hardware :- 2x4T SSD with raid 1 to install OS on. 8 Core with 3600MHz freq. 64G RAM We are planning to run all Ceph daemons except OSD daemon like monitor , metadata ,etc on this host. We will have 3 hosts to run OSD which will store actual data. Each OSD host will have following hardware 2x4T SSD with raid 1 to install OS on. 22X8T SSG to store data ( OSDs ) ( without partition ). We will use entire disk without partitions Each OSD host will have 128G RAM ( No swap space ) Each OSD host will have 16 cores. All 4 hosts will connect to each via 10G nic. The 500T data will be accessed by the clients. We need to have read performance as fast as possible. We can't afford data loss and downtime. So, we want to have a Ceph deployment which serves our purpose. So, please advise me if the plan that I have designed will serve our purpose. Or is there a better way , please advise that. Thanks, Gagan We have a HP storage server with 12 SDD of 5T each and have set-up hardware RAID6 on these disks. HP storage server has 64G RAM and 18 cores. So, please advise how I should go about setting up Ceph on it to have best read performance. We need fastest read performance. Thanks, Gagan

7 22

Cluster recovery: DC power failure killed OSD node BlueStore block.DB devices
by Paul Browne 23 Apr '25

23 Apr '25

Hi ceph-users, We recently suffered a total power failure at our main DC; fortunately, our production Ceph cluster emerged unscathed but a smaller Ceph cluster came back with the majority of its dedicated SSD BlueStore block.DB devices dead (but its HDD OSD devices unharmed). This cluster underpinned a small OpenStack cloud, so it would be preferable to recover it rather than writing it off. In terms of deployment tooling, this ailing Ceph cluster is a fairly standard Red Hat Ceph Storage 7 (so Quincy) cephadm deployed cluster, with the main wrinkle about it being that both the DB and HDD OSD devices make use of the cephadm supported LVM->LUKS layering above the BlueStore devices. The dead BlueStore block.DB devices are of course blocking the surviving HDD OSD daemons (in cephadm deployed containers) from coming up cleanly and so the Ceph cluster status is currently very degraded (attached status for the ugly picture) I've kicked around some ideas of recovering the dead DB devices and restarting down+out OSDs by; * Manually partitioning replacement SSDs into new DB device partitions+LVs * Installing the same LUKS keys on them retrieved from the Ceph config DB, matching up against which OSD is on which OSD host. * Manually changing over device links for OSDs to their DB device with "ceph-bluestore-tool bluefs-bdev-new-db" or similar * Try restarting OSDs with updated links to new LVM->LUKS->block.DB devices This approach seems highly messy and subject to needing to extract a lot of information error-free from dumps of ceph volume lvm, list in order to exactly match extant OSD UUIDs to newly created DB devicemapper devices. Is there going to be some smarter/better/faster way to non-destructively recover these intact HDD OSDs which have links to dead block.DB devices, using native cephadm tooling rather than getting so low-level as all the above? Many thanks for any advice, ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

5 9

Help with HA NFS
by Devin A. Bougie 23 Apr '25

23 Apr '25

Hello, We’ve found that if we lose one of the nfs.cephfs service daemons in our cephadm 19.2.2 cluster, all NFS traffic is blocked until either: - the down nfs.cephfs daemon is restarted - or we reconfigure the placement of the nfs.cephs service to not use the affected host. After this, the ingress.nfs.cephfs service is automatically reconfigured and everything resumes Our current setup follows the "HIGH-AVAILABILITY NFS” documentation, which gives us an ingress.nfs.cephfs service with the haproxy and keepalived daemons and an nfs.cephfs service for the actual nfs daemons. This service was deployed using: ceph nfs cluster create cephfs "label:_admin" --ingress --virtual_ip virtual_ip And then we updated the ingress.nfs.cephfs service to only deploy a single service (which in this case, results in two daemons on a single host). This gives us the following: ——— [root@cephman1 ~]# ceph orch ls --service_name=ingress.nfs.cephfs --export service_type: ingress service_id: nfs.cephfs service_name: ingress.nfs.cephfs placement: count: 1 label: _admin spec: backend_service: nfs.cephfs first_virtual_router_id: 50 frontend_port: 2049 monitor_port: 9049 virtual_ip: 128.84.45.48/22 [root@cephman1 ~]# ceph orch ls --service_name=nfs.cephfs --export service_type: nfs service_id: cephfs service_name: nfs.cephfs placement: label: _admin spec: port: 12049 ——— Can anyone show us the config for a true “HA” nfs service where they can lose any single host without impacting access to the NFS export from clients? I would expect to be able to lose the host running the ingress.nfs.cephfs service, and have it automatically restarted on a different host. Likewise, I would expect to be able to lose an nfs.cephs daemon without impacting access to the export. Or should we be taking a completely different approach and move our NFS service out of Ceph and into our pacemaker / corosync cluster? Sorry if this sounds redundant to questions I’ve previously asked, but we’ve reconfigured things a little and it feels like we’re getting closer with each attempt? Many thanks, Devin

5 5

NVMeOF GW with Squid
by Michel Raabe 23 Apr '25

23 Apr '25

Hi all, does anyone have a successfully - with connected clients/traffic - running nvmeof gw with squid? https://docs.ceph.com/en/squid/rbd/nvmeof-overview/ Thanks! Michel -- Michel Raabe Cloud Solution Architect Mail: raabe(a)b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / https://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

1 0

Re: Cephadm upgrade from 16.2.15 -> 17.2.0
by Jeremy Hansen 22 Apr '25

22 Apr '25

Snipped some of the irrelevant logs to keep message size down. ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v17.2.0", "progress_id": "e7e1a809-558d-43a7-842a-c6229fdc57af", "target_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "target_digests": ["quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667"], "target_version": "17.2.0", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": null, "paused": false, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} What should I do next? Thank you! -jeremy > On Sunday, Apr 06, 2025 at 1:38 AM, Eugen Block <eblock(a)nde.ag (mailto:eblock@nde.ag)> wrote: > Can you check if you have this config-key? > > ceph config-key get mgr/cephadm/upgrade_state > > If you reset the MGRs, it might be necessary to clear this key, > otherwise you might end up in some inconsistency. Just to be sure. > > Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > > > Thanks. I’m trying to be extra careful since this cluster is > > actually in use. I’ll wait for your feedback. > > > > -jeremy > > > > > On Saturday, Apr 05, 2025 at 3:39 PM, Eugen Block <eblock(a)nde.ag > > > (mailto:eblock@nde.ag)> wrote: > > > No, that's not necessary, just edit the unit.run file for the MGRs to > > > use a different image. See Frédéric's instructions: > > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRA… > > > > > > But I'm not entirely sure if you need to clear some config-keys first > > > in order to reset the upgrade state. If I have time, I'll try to check > > > tomorrow, or on Monday. > > > > > > Zitat von Jeremy Hansen <jeremy(a)skidrow.la>: > > > > > > > Would I follow this process to downgrade? > > > > > > > > > > > https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying… > > > > > > > > Thank you > > > > > > > > > On Saturday, Apr 05, 2025 at 2:04 PM, Jeremy Hansen > > > > > <jeremy(a)skidrow.la (mailto:jeremy@skidrow.la)> wrote: > > > > > ceph -s claims things are healthy: > > > > > > > > > > ceph -s > > > > > cluster: > > > > > id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1 > > > > > health: HEALTH_OK > > > > > > > > > > services: > > > > > mon: 3 daemons, quorum cn01,cn03,cn02 (age 20h) > > > > > mgr: cn03.negzvb(active, since 26m), standbys: cn01.tjmtph, > > > > > cn02.ceph.xyz.corp.ggixgj > > > > > mds: 1/1 daemons up, 2 standby > > > > > osd: 15 osds: 15 up (since 19h), 15 in (since 14M) > > > > > > > > > > data: > > > > > volumes: 1/1 healthy > > > > > pools: 6 pools, 610 pgs > > > > > objects: 284.59k objects, 1.1 TiB > > > > > usage: 3.3 TiB used, 106 TiB / 109 TiB avail > > > > > pgs: 610 active+clean > > > > > > > > > > io: > > > > > client: 255 B/s rd, 1.2 MiB/s wr, 10 op/s rd, 16 op/s wr > > > > > > > > > > > > > > > > > > > > — > > > > > How do I downgrade if the orch is down? > > > > > > > > > > Thank you > > > > > -jeremy > > > > > > > > > > > > > > > > > > > > > On Saturday, Apr 05, 2025 at 1:56 PM, Eugen Block <eblock(a)nde.ag > > > > > (mailto:eblock@nde.ag)> wrote: > > > > > > It would help if you only pasted the relevant parts. Anyway, these two > > > > > > sections stand out: > > > > > > > > > > > > ---snip--- > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 0 [balancer INFO root] > > > > > > Some PGs (1.000000) are unknown; try again later > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Failed to > > > > > > construct class in 'cephadm' > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.917+0000 7f2663400700 -1 mgr load Traceback > > > > > > (most recent call last): > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/cephadm/module.py", line 470, in __init__ > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > self.upgrade = CephadmUpgrade(self) > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112, in __init__ > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > self.upgrade_state: Optional[UpgradeState] = > > > > > > UpgradeState.from_json(json.loads(t)) > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93, in from_json > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > return cls(**c) > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > TypeError: __init__() got an unexpected keyword argument > > > 'daemon_types' > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:48.918+0000 7f2663400700 -1 mgr operator() > > > > > > Failed to run module in active mode ('cephadm') > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Failed to > > > > > > construct class in 'snap_schedule' > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:49.273+0000 7f2663400700 -1 mgr load Traceback > > > > > > (most recent call last): > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/snap_schedule/module.py", line 38, > > > in __init__ > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > self.client = SnapSchedClient(self) > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > > > > > > 158, in __init__ > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > with self.get_schedule_db(fs_name) as conn_mgr: > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > File "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line > > > > > > 192, in get_schedule_db > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > db.executescript(dump) > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > sqlite3.OperationalError: table schedules already exists > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp > > > > > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]: > > > > > > debug 2025-04-05T20:33:49.274+0000 7f2663400700 -1 mgr operator() > > > > > > Failed to run module in active mode ('snap_schedule') > > > > > > ---snip--- > > > > > > > > > > > > Your cluster seems to be in an error state (ceph -s) because of an > > > > > > unknown PG. It's recommended to have a healthy cluster before > > > > > > attemping an upgrade. It's possible that these errors come from the > > > > > > not upgraded MGR, I'm not sure. > > > > > > > > > > > > Since the upgrade was only successful for two MGRs, I am thinking > > > > > > about downgrading both MGRs back to 16.2.15, then retry an upgrade to > > > > > > a newer version, either 17.2.8 or 18.2.4. I haven't checked the > > > > > > snap_schedule error yet, though. Maybe someone else knows > > > that already. > > > > > >

2 15