- ceph-users - lists.ceph.io

Best Practices and real world experiences for Big Scale S3 Clusters
by Şenol Çolak 28 Apr '25

28 Apr '25

Hello everyone, We’re running a 30+ node cluster (+plan to add more) to provide S3 services to our consumers. Each node has: - 64 Cores, 300+ GB RAM (Monitors have less memory, we have 5 monitors separated from OSD nodes) - 14 NVMe (14TB) or 14 HDD (10TB) disks - Each OSD node also hosts RGW services(pods). - HP/Dell/Lenovo hardware - Around 6 PB Raw capacity. - roughly 3 CPU cores and 8GB RAM are allocated per OSD sizing. - We use Rook to deploy our cluster I know that some of the answers might be long, I also have my own experiences on some of them(which I included), but I would highly appreciate to see the state for Ceph S3 on 04/2025. and here we have several questions we’d love your input on: * Performance Counters - We want to enable performance counters to identify noisy buckets and diagnose issues when they occur. - Does enabling performance counters cause noticeable performance degradation? If so, by how much? - For now, we plan to keep them disabled.(Because we are not sure) * OS Tuning for Ceph Nodes - WriteCache is disabled for NVMe disks. - No multi-socket nodes (no NUMA configuration). - No other workloads running on Ceph nodes. - Plan is to keep the default installation unless strongly advised otherwise. - Is there anything else critical we should tune (kernel parameters, etc.)? * Tracking Over-utilization - How can we detect and identify users overloading the cluster (e.g., excessive object creation/deletion, heavy throughput)? * NVMe Configuration for Ceph - Beyond disabling cache, what are the best practices around discard/trimming operations for NVMe drives? * Buying New NVMe Disks - When we order disks, we request 2 batches per disk group (e.g., 100 disks from Batch1, 100 from Batch2). We are looking for more professional recommendations. Is it a good idea to have different batches or shall we combine two different vendor disk groups? - We always check the rebranding values for NVMe disks and try to group them with rebranding informations. * Tuning Ceph for Large Scale S3 - What default parameters should be adjusted for a large-scale S3 deployment? * Customer Limitations - In theory we don't need any limitations, but we don't want to see non-sharding objects, etc. , If we allow everything our customers will find ways to abuse this. What parameters will protect us from operational difficulties. - Are there standard limitations we should enforce, like maximum object size, maximum number of objects per bucket, etc. * Inventory Management and Netbox Integration - Is it a good practice to integrate Netbox into Ceph operations? - Like, should we register all disks in Netbox for easier daily maintenance, adding automation to the netbox and replacing the disks via inventory changes on netbox ? * Cluster Deployment Best Practices - Ansible, Rook, custom deployments, or manual host installations — what’s best for large clusters, might depend on the team and company structure but would be nice to see opinions? * Why Some Large Deployments Avoid Rook - I mostly see Rook deployments for Openstack HCI deployments only (maybe I am wrong on this) - Is it about operational complexity, or is Rook seen as unsuitable for very large clusters? * Monitoring Metrics - What are the most critical metrics for diagnosing S3 issues? * RocksDB Tuning and Archiving - How should we manage RocksDB in large-scale setups? - Should we periodically split/archive the DB? - How important is RocksDB latency monitoring? * OS-Level Monitoring - Besides standard exporter logs, what else should we monitor (e.g., network latency, OSD-to-OSD throughput)? - we are planning to create a separate exporter than the ceph_exporter to export some specific metrics (mostly disk and NW related) * Design Mistakes to Avoid - What are the common mistakes (e.g., settings that block easy upgrades) that we should be aware of when planning a large-scale S3 service? * Vendor Relations - Is there a resource to check disk types, endurance, wearout timelines, etc. When we order new HW the vendor doesn't provide us any data? - Maybe there is a group of Ceph users holding some internal information about different vendors and their HW, if not I would be happy to start this.. - What's the best approach when dealing with hardware vendors, getting the cheapest or getting the reliable is the right approach? * Scheduled Operations - Are there standard Ansible playbooks or procedures for recurring maintenance tasks? * Unsupported Ceph Versions - Some large deployments stick with Ceph 16.x (unsupported). - Why do they prefer being alone on the old version and backporting features themselves? this seems very dangerous (there has to be a valid reason why?) * Cluster Health Monitoring - Do large deployments rely only on HEALTH_OK? - Or do they combine metrics for a custom health status definition? * Security Best Practices - What's the best approach for securing the cluster? - How should we handle disk encryption, key rotation (frequency/method), and secure disposal? * Diversity of Pools - If our service is focused purely on S3, should we still consider using other pool types (RBD, CephFS)? * Disk Size Diversity - Is it problematic to have OSDs with different disk sizes? * Multisite Clusters - In a multisite setup, if replication is enabled for some pools, is the performance impact localized to those pools, or does it affect the entire cluster? * WAF or Smart Proxy for RGW nodes - Our idea is to have RGW on each of our OSD node, (Monitor and Mgr would be excluded on any operation), what would be the best idea besides having a LDB in front of these RGW services. How are the others designing RGW, is it safe enough to just use it directly? I would highly appreciate your answers/opinions on these topics. I also assume that most of these comments would come from real use cases and experiences. Thanks in advance, _ Senol Colak

2 1

April 28th CSC Meeting Notes
by Laura Flores 28 Apr '25

28 Apr '25

- [Adam King] PyO3 issue (https://tracker.ceph.com/issues/64213) - pecastro/John Mulligan/Ernesto have been working on this - https://github.com/ceph/ceph/pull/61737 - Can be fixed for cephadm and dashboard through that work - still an issue with rook/k8sevents modules - https://pad.ceph.com/p/pyo3-cryptography-imports-k8s - Is anyone actually using these? - Ongoing issue on newer versions that affects the mgr (patch ready to work around this) - Rook issue still WIP; seeking advice on discontinuing some of the affected modules - Yaarit will check on Telemetry and follow up next week - Suggestion to reach out to the user list - [cbodley] follow up on 'ceph api tests' with Ernesto (bump to next week) - Tentacle branch cut last week on Fri. - Can we build packages now? - https://jenkins.ceph.com/job/ceph-dev/58072/ - David can check that the branch is protected - Done - Tentacle milestone created: https://github.com/ceph/ceph/milestone/31 - Patrick needs reviews on https://github.com/ceph/ceph/pull/62089 - Need to check how upgrade tests are doing - v20.3.0 needs tagged on main to allow upgrades from tentacle -> main - Adam Kraitman: upgraded redmine and improved database - A bunch of people no longer get emails from Redmine, the last email is from Apr 24th -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: Squid Manager Daemon: balancer crashing orchestrator and dashboard
by Laimis Juzeliūnas 28 Apr '25

28 Apr '25

Hi all, Since there were a few people asking around and I have not seen any users confirming I’ve just wanted to put this fact in writing for anyone in the future: 19.2.2 version fixes all 19.2.0 balancer issues. Everything started working immediately once the mgr modules were upgraded. I’m sure 19.2.1 already does that, but it’s recommended to go to 19.2.2 directly. Huge thanks to everyone involved, thread closed. Best, Laimis J.

1 0

Stretched pool or not ?
by Soeren Malchow 28 Apr '25

28 Apr '25

Hi, i am working on the plan for a 3 datacenter setup using ceph (in proxmox nodes). Each datacenter has 3 physical nodes to start with and 100Gbit switches. I will also have 2 x 100 Gbit/s connectivity between the datacenters (each datacenter to each other). the physical nodes have 2 x 100Gbit/s for the public network and 2 x 100Gbit/s for the cluster network. About this setup i have 2 questions. 1. is it even necessary to evaluate a stretched cluster since the WAN connections are as fast as the local ones (including the latency, since it is only 25km) ? 2. If using a stretched pool across all 3 datacenters, what happens if one datacenter fails ? I did read the documentation and the question came up, because it do not understand the sentence "Individual Stretch Pools do not support I/O operations during a netsplit scenario between two or more zones" completely, does it mean there is no IO already if one datacenter fails ? If i am on the wrong path, maybe someone has a link for me, where is can find information on this setup ? Cheers Soeren

3 7

Re: HEALTH_ERR: 1 MDSs report damaged metadata - damage_type=dentry
by Christophe DIARRA 28 Apr '25

28 Apr '25

Hello David, Thank you for the tip about the scrubbing. I have tried the commands found in the documentation but it seems to have no effect: [root@mon-01 ~]#*ceph tell mds.cfs_irods_test:0 scrub start / recursive,repair,force* 2025-04-17T12:07:20.958+0200 7fd4157fa640 0 client.86301 ms_handle_reset on v2:130.84.80.10:6800/3218663047 2025-04-17T12:07:20.979+0200 7fd4157fa640 0 client.86307 ms_handle_reset on v2:130.84.80.10:6800/3218663047 { "return_code": 0, "scrub_tag": "733b1c6d-a418-4c83-bc8e-b28b556e970c", "mode": "asynchronous" } [root@mon-01 ~]#*ceph tell mds.cfs_irods_test:0 scrub status* 2025-04-17T12:07:30.734+0200 7f26cdffb640 0 client.86319 ms_handle_reset on v2:130.84.80.10:6800/3218663047 2025-04-17T12:07:30.753+0200 7f26cdffb640 0 client.86325 ms_handle_reset on v2:130.84.80.10:6800/3218663047 { "status": "no active scrubs running", "scrubs": {} } [root@mon-01 ~]# ceph -s cluster: id: b87276e0-1d92-11ef-a9d6-507c6f66ae2e *health: HEALTH_ERR 1 MDSs report damaged metadata* services: mon: 3 daemons, quorum mon-01,mon-03,mon-02 (age 19h) mgr: mon-02.mqaubn(active, since 19h), standbys: mon-03.gvywio, mon-01.xhxqdi mds: 1/1 daemons up, 2 standby osd: 368 osds: 368 up (since 18h), 368 in (since 3w) data: volumes: 1/1 healthy pools: 10 pools, 4353 pgs objects: 1.25M objects, 3.9 TiB usage: 417 TiB used, 6.4 PiB / 6.8 PiB avail pgs: 4353 active+clean Did I miss something ? The server didn't crash. I don't understand what you are meaning by "there may be a design flaw in the infrastructure (insecure cache, for example)". How to know if we have a design problem ? What should we check ? Best regards, Christophe On 17/04/2025 11:07, David C. wrote: > Hello Christophe, > > Check the file system scrubbing procedure => > https://docs.ceph.com/en/latest/cephfs/scrub/ But this doesn't > guarantee data recovery. > > Was the cluster crashed? > Ceph should be able to handle it; there may be a design flaw in the > infrastructure (insecure cache, for example). > > David > > Le jeu. 17 avr. 2025 à 10:44, Christophe DIARRA > <christophe.diarra(a)idris.fr> a écrit : > > Hello, > > After an electrical maintenance I restarted our ceph cluster but it > remains in an unhealthy state: HEALTH_ERR 1 MDSs report damaged > metadata. > > How to repair this damaged metadata ? > > To bring down the cephfs cluster I unmounted the fs from the client > first and then did: ceph fs set cfs_irods_test down true > > To bring up the cephfs cluster I did: ceph fs set cfs_irods_test > down false > > Fortunately the cfs_irods_test fs is almost empty and is a fs for > tests.The ceph cluster is not in production yet. > > Following is the current status: > > [root@mon-01 ~]# ceph health detail > HEALTH_ERR 1 MDSs report damaged metadata > *[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata > mds.cfs_irods_test.mon-03.vlmeuz(mds.0): Metadata damage > detected* > > [root@mon-01 ~]# ceph -s > cluster: > id: b87276e0-1d92-11ef-a9d6-507c6f66ae2e > health: HEALTH_ERR > 1 MDSs report damaged metadata > > services: > mon: 3 daemons, quorum mon-01,mon-03,mon-02 (age 17h) > mgr: mon-02.mqaubn(active, since 17h), standbys: mon-03.gvywio, > mon-01.xhxqdi > mds: 1/1 daemons up, 2 standby > osd: 368 osds: 368 up (since 17h), 368 in (since 3w) > > data: > volumes: 1/1 healthy > pools: 10 pools, 4353 pgs > objects: 1.25M objects, 3.9 TiB > usage: 417 TiB used, 6.4 PiB / 6.8 PiB avail > pgs: 4353 active+clean > > > [root@mon-01 ~]# ceph fs ls > name: cfs_irods_test, metadata pool: cfs_irods_md_test, data pools: > [cfs_irods_def_test cfs_irods_data_test ] > > [root@mon-01 ~]# ceph mds stat > cfs_irods_test:1 {0=cfs_irods_test.mon-03.vlmeuz=up:active} 2 > up:standby > > [root@mon-01 ~]# ceph fs status > cfs_irods_test - 0 clients > ============== > RANK STATE MDS ACTIVITY DNS > INOS DIRS CAPS > 0 active cfs_irods_test.mon-03.vlmeuz Reqs: 0 /s 12 15 > 14 0 > POOL TYPE USED AVAIL > cfs_irods_md_test metadata 11.4M 34.4T > cfs_irods_def_test data 0 34.4T > cfs_irods_data_test data 0 4542T > STANDBY MDS > cfs_irods_test.mon-01.hitdem > cfs_irods_test.mon-02.awuygq > MDS version: ceph version 18.2.2 > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) > [root@mon-01 ~]# > > [root@mon-01 ~]# ceph tell mds.cfs_irods_test:0 damage ls > 2025-04-17T10:23:31.849+0200 7f4b87fff640 0 client.86181 > ms_handle_reset on v2:130.84.80.10:6800/3218663047 > <http://130.84.80.10:6800/3218663047> > 2025-04-17T10:23:31.866+0200 7f4b87fff640 0 client.86187 > ms_handle_reset on v2:130.84.80.10:6800/3218663047 > <http://130.84.80.10:6800/3218663047> > [ > { > *"damage_type": "dentry",* > "id": 241447932, > "ino": 1, > "frag": "*", > "dname": "testdir2", > "snap_id": "head", > "path": "/testdir2" > }, > { > *"damage_type": "dentry"*, > "id": 2273238993, > "ino": 1, > "frag": "*", > "dname": "testdir1", > "snap_id": "head", > "path": "/testdir1" > } > ] > [root@mon-01 ~]# > > Any help will be appreciated, > > Thanks, > > Christophe > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

5 25

Missing Grafana dash boards
by Nicola Mori 28 Apr '25

28 Apr '25

Dear Ceph users, my Cephadm cluster (19.2.1) is currenlty lacking all the Grafana dashboards so I cannot see plots of any metrics. For example, trying to access the "Performance Details" tab of my filesystem on the web dashboard I get the following message: Grafana Dashboard doesn't exist. Please refer to documentation on how to add dashboards to Grafana. The linked instructions reports how to enable the monitoring stack, which was already deployed on my system. So I first removed it: ceph orch rm prometheus ceph orch rm node-exporter ceph orch rm ceph-exporter ceph mgr module disable prometheus and the re-deployed it: ceph orch apply node-exporter ceph orch apply alertmanager ceph orch apply prometheus ceph orch apply grafana ceph mgr module enable prometheus but still I get no Grafana dashboards. How can I fix this? Thank you, Nicola

2 1

Putting an OSD host with NFS daemons in maintenance
by Devin A. Bougie 28 Apr '25

28 Apr '25

Hello, When trying to put one of our OSD hosts in maintenance, "ceph orch host maintenance enter” displays: WARNING: Removing NFS daemons can cause clients to lose connectivity. We have a single Cephfs filesystem in our cluster. Five hosts are “admin” hosts that run cephadm, mds, mgr, mon, etc. The rests of our hosts are OSDs with the spinning disks that make up our Cephfs data pool. Our current setup follows the "HIGH-AVAILABILITY NFS” documentation, which gives us an Ingress.nfs.cephfs service with the haproxy and keepalived daemons and a nfs.cephfs service for the actual nfs daemons. This service was deployed using: ceph nfs cluster create cephfs "label:_admin" --ingress --virtual_ip virtual_ip And then we manually updated the nfs.cephfs service this created to place the nfs daemons on our OSD nodes. This gives us the following: ——— service_type: ingress service_id: nfs.cephfs service_name: ingress.nfs.cephfs placement: label: _admin spec: backend_service: nfs.cephfs first_virtual_router_id: 50 frontend_port: 2049 monitor_port: 9049 virtual_ip: virtual_ip/prefix service_type: nfs service_id: cephfs service_name: nfs.cephfs placement: label: osd spec: port: 12049 ——— How can we safely put one of our osd nodes in maintenance? Or is there a better way to organize our daemons or setup and manage our NFS service that would avoid this issue? Many thanks, Devin

2 1

rbd commands don't return to prompt
by Dominique Ramaekers 25 Apr '25

25 Apr '25

Hi, Weird thing happened with my ceph. I've got nice nightly scripts (bash scripts) for making backups, snapshots, cleaning up etc... Starting from my last upgrade to ceph v19.2.2 my scripts hang during execution. The rbd map and rbd unmap commands doesn't return to the prompt. So my script invoces a command like "cephadm shell -- rbd --pool libvirt-pool map --read-only --image CmsrvXCH2-SWAP@snap_4" but my script doesn't continue because the rbd map command never quits. I've also tried it myself. After killing my script a few times, I saw several images being mapped. So let's unmap them. This happened... root@hvs001:/# rbd unmap /dev/rbd0 ^C root@hvs001:/# rbd unmap /dev/rbd1 ^C The devices aren't mounted so the unmapping isn't blocked. Strangely enough after activating the unmap command, the device is removed from the /dev but I still have to do a ^C to return to the shell... Do I have an issue with my ceph cluster? Has anybody experienced something similar? Should I report a bug? Greetings, Dominique.

2 3

Pull from local registry fails
by Sake Ceph 25 Apr '25

25 Apr '25

We're used a local registry (docker registry), but needed to switch to Harbor. The old registry needed credentials and was working fine. I used "sudo cephadm -v registry-login --registry-json login" to login to the new registry. login is the json with the credentials. When starting the upgrade, I get the following error: "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image". But using "sudo cephadm --image <url> pull" works fine. $ sudo cephadm --image harbor.url/project/ceph/ceph:v18.2.6-20250417 pull Pulling container image harbor.url/project/ceph/ceph:v18.2.6-20250417... { "ceph_version": "ceph version 18.2.6 (ff498e17d264a1a4d588c361cbce9cc65afa2327) reef (stable)", "image_id": "d4baa6a728fa46bda2d58db9f458847bde23ca99135c6c9bb025a9d50fb9f011", "repo_digests": [ "harbor.url/project/ceph/ceph@sha256:848ef15fd4bb29a42d1063aba7b2543432ecec78afe35e7b64cc91f9748de551" ] } I also tried updating the config, didn't help either. ceph config-key set mgr/cephadm/registry_url ceph config-key set mgr/cephadm/registry_username ceph config-key set mgr/cephadm/registry_password So what's wrong? Currently using 18.2.4.

3 7

help：unsubscribe
by 黄浩然 25 Apr '25

25 Apr '25

| | 黄浩然 | | huanghaoran_1989(a)163.com | ---- Replied Message ---- | From | <ceph-users-request(a)ceph.io> | | Date | 04/25/2025 17:02 | | To | <ceph-users(a)ceph.io> | | Subject | ceph-users Digest, Vol 130, Issue 155 | Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Re: Patching Ceph cluster (Sake Ceph) 2. Re: Patching Ceph cluster (Sake Ceph) ---------------------------------------------------------------------- Date: Fri, 25 Apr 2025 10:59:52 +0200 (CEST) From: Sake Ceph <ceph(a)paulusma.eu> Subject: [ceph-users] Re: Patching Ceph cluster To: Michael Worsham <mworsham(a)datadimensions.com>, "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> Message-ID: <683056728.264684.1745571592421(a)webmail.strato.com> Content-Type: text/plain; charset=UTF-8 The tiebreaker is for a stretch cluster, which we deployed. It's only used to assign the host to a group. The Playbook is indeed written for RHEL, because that's the OS we use. It can be improved a lot, but it's a start for someone else. I know I still need to share this on GitHub, but to busy at the moment at work and at home. Op 25-04-2025 06:05 CEST schreef Michael Worsham <mworsham(a)datadimensions.com>: I've been reading over the playbook code, and it's nicely written. I know it's primarily RHEL focused, but I think it could be modified for Ubuntu/Debian platforms as well. A couple of questions though... In the test example hosts file, what is the tiebreaker? I know there isn't a role in the roles folder, but do you have an example of one, just so we know what it does? Thanks. -- Michael Get Outlook for Android (https://aka.ms/AAb9ysg) ------------------------------ From: Sake Ceph <ceph(a)paulusma.eu> Sent: Friday, June 14, 2024 4:28:34 AM To: Michael Worsham <mworsham(a)datadimensions.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Re: Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. I needed to do some cleaning before I could share this :) Maybe you or someone else can use it. Kind regards, Sake Op 14-06-2024 03:53 CEST schreef Michael Worsham <mworsham(a)datadimensions.com>: I'd love to see what your playbook(s) looks like for doing this. -- Michael ________________________________ From: Sake Ceph <ceph(a)paulusma.eu> Sent: Thursday, June 13, 2024 4:05 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. Yeah we fully automated this with Ansible. In short we do the following. 1. Check if cluster is healthy before continuing (via REST-API) only health_ok is good 2. Disable scrub and deep-scrub 3. Update all applications on all the hosts in the cluster 4. For every host, one by one, do the following: 4a. Check if applications got updated 4b. Check via reboot-hint if a reboot is necessary 4c. If applications got updated or reboot is necessary, do the following : 4c1. Put host in maintenance 4c2. Reboot host if necessary 4c3. Check and wait via 'ceph orch host ls' if status of the host is maintance and nothing else 4c4. Get host out of maintenance 4d. Check if cluster is healthy before continuing (via Rest-API) only warning about scrub and deep-scrub is allowed, but no pg's should be degraded 5. Enable scrub and deep-scrub when all hosts are done 6. Check if cluster is healthy (via Rest-API) only health_ok is good 7. Done For upgrade the OS we have something similar, but exiting maintenance mode is broken (with 17.2.7) :( I need to check the tracker for similar issues and if I can't find anything, I will create a ticket. Kind regards, Sake Op 12-06-2024 19:02 CEST schreef Daniel Brown <daniel.h.brown(a)thermify.cloud>: I have two ansible roles, one for enter, one for exit. There’s likely better ways to do this — and I’ll not be surprised if someone here lets me know. They’re using orch commands via the cephadm shell. I’m using Ansible for other configuration management in my environment, as well, including setting up clients of the ceph cluster. Below excerpts from main.yml in the “tasks” for the enter/exit roles. The host I’m running ansible from is one of my CEPH servers - I’ve limited which process run there though so it’s in the cluster but not equal to the others. ————— Enter ————— - name: Ceph Maintenance Mode Enter shell: cmd: ' cephadm shell ceph orch host maintenance enter {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} --force --yes-i-really-mean-it ‘ become: True ————— Exit ————— - name: Ceph Maintenance Mode Exit shell: cmd: 'cephadm shell ceph orch host maintenance exit {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} ‘ become: True connection: local - name: Wait for Ceph to be available ansible.builtin.wait_for: delay: 60 host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}’ port: 9100 connection: local On Jun 12, 2024, at 11:28 AM, Michael Worsham <mworsham(a)datadimensions.com> wrote: Interesting. How do you set this "maintenance mode"? If you have a series of documented steps that you have to do and could provide as an example, that would be beneficial for my efforts. We are in the process of standing up both a dev-test environment consisting of 3 Ceph servers (strictly for testing purposes) and a new production environment consisting of 20+ Ceph servers. We are using Ubuntu 22.04. -- Michael From: Daniel Brown <daniel.h.brown(a)thermify.cloud> Sent: Wednesday, June 12, 2024 9:18 AM To: Anthony D'Atri <anthony.datri(a)gmail.com> Cc: Michael Worsham <mworsham(a)datadimensions.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. There’s also a Maintenance mode that you can set for each server, as you’re doing updates, so that the cluster doesn’t try to move data from affected OSD’s, while the server being updated is offline or down. I’ve worked some on automating this with Ansible, but have found my process (and/or my cluster) still requires some manual intervention while it’s running to get things done cleanly. On Jun 12, 2024, at 8:49 AM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: Do you mean patching the OS? If so, easy -- one node at a time, then after it comes back up, wait until all PGs are active+clean and the mon quorum is complete before proceeding. On Jun 12, 2024, at 07:56, Michael Worsham <mworsham(a)datadimensions.com> wrote: What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. ------------------------------ Date: Fri, 25 Apr 2025 11:02:28 +0200 (CEST) From: Sake Ceph <ceph(a)paulusma.eu> Subject: [ceph-users] Re: Patching Ceph cluster To: Lukasz Borek <lukasz(a)borek.org.pl> Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> Message-ID: <796303375.265011.1745571748485(a)webmail.strato.com> Content-Type: text/plain; charset=UTF-8 Nope, it was really broken in 17.2.7. When RHEL 10 comes available, I will look into this part again :) Op 25-04-2025 07:22 CEST schreef Lukasz Borek <lukasz(a)borek.org.pl>: For upgrade the OS we have something similar, but exiting maintenance mode is broken (with 17.2.7) :( I need to check the tracker for similar issues and if I can't find anything, I will create a ticket For 18.2.2 first maint exit command threw an exception for some reason. In my patching script I execute commands in a loop and the 2nd shoot usually works. exit maint 1/3 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1809, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 183, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 119, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 108, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 778, in _host_maintenance_exit raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 237, in raise_if_exception e = pickle.loads(c.serialized_exception) TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' exit maint 2/3 Ceph cluster f3e63d9e-2f4c-11ef-87a2-0f1170f55ed5 on cephbackup-osd1 has exited maintenance mode exit maint 3/3 Error EINVAL: Host cephbackup-osd1 is not in maintenance mode Fri Apr 25 07:17:58 CEST 2025 cluster state is HEALTH_WARN Fri Apr 25 07:18:02 CEST 2025 cluster state is HEALTH_WARN [...] On Thu, 13 Jun 2024 at 22:07, Sake Ceph <ceph(a)paulusma.eu> wrote: For upgrade the OS we have something similar, but exiting maintenance mode is broken (with 17.2.7) :( I need to check the tracker for similar issues and if I can't find anything, I will create a ticket. Kind regards, Sake Op 12-06-2024 19:02 CEST schreef Daniel Brown <daniel.h.brown(a)thermify.cloud>: I have two ansible roles, one for enter, one for exit. There’s likely better ways to do this — and I’ll not be surprised if someone here lets me know. They’re using orch commands via the cephadm shell. I’m using Ansible for other configuration management in my environment, as well, including setting up clients of the ceph cluster. Below excerpts from main.yml in the “tasks” for the enter/exit roles. The host I’m running ansible from is one of my CEPH servers - I’ve limited which process run there though so it’s in the cluster but not equal to the others. ————— Enter ————— - name: Ceph Maintenance Mode Enter shell: cmd: ' cephadm shell ceph orch host maintenance enter {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} --force --yes-i-really-mean-it ‘ become: True ————— Exit ————— - name: Ceph Maintenance Mode Exit shell: cmd: 'cephadm shell ceph orch host maintenance exit {{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }} ‘ become: True connection: local - name: Wait for Ceph to be available ansible.builtin.wait_for: delay: 60 host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}’ port: 9100 connection: local On Jun 12, 2024, at 11:28 AM, Michael Worsham <mworsham(a)datadimensions.com> wrote: Interesting. How do you set this "maintenance mode"? If you have a series of documented steps that you have to do and could provide as an example, that would be beneficial for my efforts. We are in the process of standing up both a dev-test environment consisting of 3 Ceph servers (strictly for testing purposes) and a new production environment consisting of 20+ Ceph servers. We are using Ubuntu 22.04. -- Michael From: Daniel Brown <daniel.h.brown(a)thermify.cloud> Sent: Wednesday, June 12, 2024 9:18 AM To: Anthony D'Atri <anthony.datri(a)gmail.com> Cc: Michael Worsham <mworsham(a)datadimensions.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Patching Ceph cluster This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. There’s also a Maintenance mode that you can set for each server, as you’re doing updates, so that the cluster doesn’t try to move data from affected OSD’s, while the server being updated is offline or down. I’ve worked some on automating this with Ansible, but have found my process (and/or my cluster) still requires some manual intervention while it’s running to get things done cleanly. On Jun 12, 2024, at 8:49 AM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: Do you mean patching the OS? If so, easy -- one node at a time, then after it comes back up, wait until all PGs are active+clean and the mon quorum is complete before proceeding. On Jun 12, 2024, at 07:56, Michael Worsham <mworsham(a)datadimensions.com> wrote: What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Łukasz Borek lukasz(a)borek.org.pl ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 130, Issue 155 ********************************************

1 0