- ceph-users - lists.ceph.io

Multisite replication design
by Nathan MALO 10 Sep '24

10 Sep '24

Hi, I am currently setting up a multi-site replication with rook-ceph. My setup is the following: - clusterA in regionA containing local buckets (let's call them bucketsA) - clusterB in regionB containing local buckets (let's call them bucketsB) - clusterC in regionC containing local buckets (let's call them bucketsC) I want to replicate bucketsA on clusterB, bucketsB on clusterC and bucketsC on clusterA. I am designing my setup and wonder what to do, realms wise: - have one global realm initially created on clusterA and pulled on clusterB and C, one zonegroup named "country" and 3 zones in it, one for each region. And then setup sync policies to replicate buckets according to my needs - have a realm per cluster, that is pulled by the cluster in charge to replicate the data (e.g: clusterB pulls the realm of clusterA), one zonegroup named "country" and two zones in it: one with the local data and the other one "remote" replicating the data, and using sync policies to replicate buckets according to my needs If you have any advice on which setup is best that would be great. I wonder, for the first option, what does happen if clusterA (who originally created the realm) happened to die. Can another cluster becomes the "realm master" (since they previously pulled the realm), or does the realm is completely lost ? If the realm is recoverable, does rook handles this procedure ? Best regards, Nate

1 0

Grafana dashboards is missing data
by Sake Ceph 10 Sep '24

10 Sep '24

After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example the Osd latency under OSD device details or the Osd Overview has a lot of No data messages. I deployed ceph-exporter on all hosts, am I missing something? Did even a redeploy of prometheus. Kind regards, Sake

6 16

CEPH monitor slow ops
by Jan Marek 10 Sep '24

10 Sep '24

Hello, we have CEPH cluster with 144 NVMe "disks", "background" network is RoCE. CEPH cluster is version 18.2.2 and is installed via CEPH orchestrator cephadm, container daemon is podman. OS is Debian bookworm, podman is in the version 4.3.1+ds1-8+b1, now we are installed version 4.3.1+ds1-8+deb12u1 (standard Debian bookworm package) We have sometimes problem with the laggy OSD. This Sunday we had problem with osd.25. It was laggy, some other OSDs reported it to MON, MON dismissed it from crush map, but osd.25 complaint about it, MON tried to add it again to the crush map - but this behavior caused, that whole CEPH cluster was laggy and had slow ops, which was a "kiss of death" to some ProxMox VMs :-(. When I restarted osd.25 daemon, after a couple minutes cluster goes back to HEALTHY state. I have prepared logs from 3 machines: c2-dc1-elected-mon-osd.25.txt - machine c2-dc1 had elected MON and 12 OSDs, same of them complait's about osd.25. c1-dc1-mon-osd.25.txt - machine with another MON and 12 OSDs c4-dc2-osd.25.txt - machine with osd.25 daemon (and other 11 OSDs) Problem was, that the first laggy daemon occured around 6 a.m., but problem insisted up to cca 4:20 p.m., when I've restarted osd.25. There is a question, if will be better to restart laggy OSD daemon by ceph orchestrator? For example: If OSD daemons from 3 host (now it is from 2 hosts by default) and after 20s reported some OSD as a dead, MON give a command to mgr to restart this OSD? Somethink like STONITH in corosync/pcs cluster ( https://en.wikipedia.org/wiki/STONITH )? Logs are very big (around 23MB tar.bz2), I've prepared it here: https://hazard.jcu.cz/osd.25.tar.bz2 We thinking about some problem with network, but on the same machine, there is another 11 OSDs and they parformed well. This is not the first occurence of this problem, when it happened the first time, we tried to reload whole server, but then we found, that reload of OSD container is enough... Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

1 0

Mon is unable to build mgr service
by Jorge Ventura 09 Sep '24

09 Sep '24

I am trying to configure ceph for the first time manually. I followed the instructions and at this point I have only ceph-mon and ceph-mgr installed. Here is my ceph status: root@ceph-n1:~# ceph status > cluster: > id: a93114e4-b0af-4b56-b019-0900310a14f8 > health: HEALTH_WARN > OSD count 0 < osd_pool_default_size 3 > > services: > mon: 1 daemons, quorum mon-n1 (age 10m) > > * mgr: mgr-n1(active, since 9m)* osd: 0 osds: 0 up, 0 in > > data: > pools: 0 pools, 0 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: For a reason that I do not understand I am getting this message in syslog every second: *Sep 05 21:05:18 ceph-n1 ceph-mon[1089]: 2024-09-05T21:05:18.636+0000 > 7ffae4fa6640 -1 mon.mon-n1@0(leader) e2 get_authorizer failed to build mgr > service session_auth_info (22) Invalid argument* I don't know if this is something about caps and at this point I have no idea what this problem is. -- Ventura

2 1

Somehow throotle recovery even further than basic options?
by Szabo, Istvan (Agoda) 09 Sep '24

09 Sep '24

Hi, 4 years ago we've created our cluster with all disks 4osds (ssds and nvme disks) on octopus. The 15TB SSDs still working properly with 4 osds but the small 1.8T nvmes with the index pool not. Each new nvme osd adding to the existing nodes generates slow ops with scrub off, recovery_op_priority 1, backfill and recovery 1-1. I even turned off all index pool heavy sync mechanism but the read latency still high which means recovery op pushes it even higher. I'm trying to somehow add resource to the cluster to spread the 2048 index pool pg (in replica 3 means 6144pg index pool) but can't make it more gentle. The balancer is working in upmap with max deviation 1. Have this script from digitalocean https://github.com/digitalocean/pgremapper, is there anybody tried it before how is it or could this help actually? Thank you the ideas. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 7

PGs not deep-scrubbed in time
by Eugen Block 08 Sep '24

08 Sep '24

Hi, I finally managed to take the time to get to the bottom of this infamous health warning. I decided to write it up in a blog post [0] and also contacted Zac to improve the documentation. The short version is: If you want to change the config setting for osd_deep_scrub_interval in general, change it either globally or for both MGR *and* OSD services. Or configure individual intervals per pool. If you don't change it for the MGR, you'll still get the warning because the MGR will still compare the last deep-scrub timestamp with the default deep-scrub interval (1 week). Comments are welcome, of course! Regards, Eugen [0] https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-ti…

1 0

Setting up Ceph RGW with SSE-S3 - Any examples?
by Michael Worsham 06 Sep '24

06 Sep '24

Has anyone been successful at standing up a Ceph RGW S3 bucket with Hashicorp Vault for S3 bucket encryption? The documentation for doing it with Ceph jumps all over the page between token and agent, so it's nearly impossible to know which variables and parameters are required for each. I was able to successfully setup a Hashicorp Vault in a high-availability cluster via an Ansible playbook I wrote. It's the configuration of what needs to be configured for the transit engine for vault and then what changes I need to do to Ceph that has me stumped. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 1

Discovery (port 8765) service not starting
by Matthew Vernon 06 Sep '24

06 Sep '24

Hi, I'm running reef, with locally-built containers based on upstream .debs. I've now enabled prometheus metrics thus: ceph mgr module enable prometheus And that seems to have worked (the active mgr is listening on port 9283); but per the docs[0] there should also be a service discovery endpoint (on port 8765) that I can point Prometheus' http_sd_config at. But there is in fact no listener on that port, nor anything obvious in logs if I failover the mgr - see a log extract here: https://phabricator.wikimedia.org/P68521 Is there something else I need to be doing to get the service discovery endpoint working? Thanks, Matthew [0] https://docs.ceph.com/en/reef/cephadm/services/monitoring/#deploying-monito…

4 21

v19.1.1 Squid RC1 released
by Yuri Weinstein 06 Sep '24

06 Sep '24

This is the second release candidate for Squid. Feature highlights: RGW Fixed a regression in bucket ownership for Keystone users and implicit tenants. The User Accounts feature unlocks several new AWS-compatible IAM APIs for the self-service management of users, keys, groups, roles, policy and more. RADOS BlueStore has been optimized for better performance in snapshot-intensive workloads. BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. Other improvements include more flexible EC configurations, an OpTracker to help debug mgr module issues, and better scrub scheduling. Dashboard * Rearranged Navigation Layout: The navigation layout has been reorganized for improved usability and easier access to key features. CephFS Improvements * Support for managing CephFS snapshots and clones, as well as snapshot schedule management * Manage authorization capabilities for CephFS resources * Helpers on mounting a CephFS volume RGW Improvements * Support for managing bucket policies * Add/Remove bucket tags * ACL Management * Several UI/UX Improvements to the bucket form Monitoring: Grafana dashboards are now loaded into the container at runtime rather than building a grafana image with the grafana dashboards. Official Ceph grafana images can be found in quay.io/ceph/grafana * Monitoring: RGW S3 Analytics: A new Grafana dashboard is now available, enabling you to visualize per bucket and user analytics data, including total GETs, PUTs, Deletes, Copies, and list metrics. Crimson/Seastore Crimson's first tech preview release! Supporting RBD workloads on Replicated pools. For more information please visit: https://ceph.io/en/news/crimson If any of our community members would like to help us with performance investigations or regression testing of the Squid release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/squid_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.1.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 1d9f35852eef16b81614e38a05cf88b505cc142b

2 2

Prefered distro for Ceph
by Denis Polom 06 Sep '24

06 Sep '24

Hi guys, what distro would you prefer and why for the production Ceph? We use Ubuntu on most of our Ceph clusters and some are Debian. Now we are thinking about unifying it by using only Debian or Ubuntu. I personally prefer Debian mainly for its stability and easy upgrade-in-place. What are yours preferences? Thank you

9 10