- ceph-users - lists.ceph.io

NFS recommendations
by Devin A. Bougie 13 Feb '25

13 Feb '25

Hi, All. We are new to Ceph, and looking for any general best practices WRT exporting a Cephfs file system over NFS. I see several options in the documentation and have tested several different configurations, but haven’t yet seen much difference in our testing and aren’t sure exactly which configuration is generally recommended to start with. We have a single Cephfs filesystem in our cluster of 10 hosts. Five of our hosts are OSDs with the spinning disks that make up our Cephfs data pool, and only run the osd services (osd, crash, ceph-exporter, node-exporter, and promtail). The other five hosts are “admin” hosts that run everything else (mds, mgr, mon, etc.). Our current setup follows the "HIGH-AVAILABILITY NFS” documentation, which gives us an Ingress.nfs.cephfs service with the haproxy and keepalived daemons and a nfs.cephfs service for the actual nfs daemons. If there are no downsides to this approach, are there any recommendations on placement for these two services? Given our cluster, would it be best to run both on the admin nodes? Or would it be better to have the ingress.nfs.cephfs service on the admin nodes, and the backend nfs.cephfs services on the osd nodes? Alternatively, are there advantages in using the “keepalive only” mode (only keepalived, no haproxy)? Or does anyone recommend doing something completely different, like using Pacemaker and Corosync to manage our NFS services? Any recommendations one way or another would be greatly appreciated. Many thanks, Devin

3 3

cephadm orchestrator feature request: scheduled rebooting of cluster nodes
by Robert Sander 13 Feb '25

13 Feb '25

Hi, it just came across my mind that it would be nice if the orchestrator would be able to reboot all nodes of a cluster in an un-interrupting way. This could be done similar to the upgrade where the orchestrator checks if daemons on a host can be restarted. Including "ceph osd add-noout $(hostname)" before the reboot and removing it afterwards. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

Upgrade quincy to reef
by cristian.tavarez＠hostpapa.com 13 Feb '25

13 Feb '25

Hello I'm trying to update my ceph cluster from the latest release of quincy 17.2.8 to reef 18.2.4. After starting the upgrade with 'ceph orch upgrade start --ceph-version 18.2.4' it stopped having upgraded 3/4 of my MGR daemons and everything else stayed on quincy. Any ceph command is not working saying orchestrator module is not found and Error ENOENT: Module not found. I have already tried looking at the osd issue hinted at https://tracker.ceph.com/issues/67329 but this happened on a test cluster without any OSDs on it as well. The current status given in the upgrade is the following ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v18.2.4", "progress_id": "a9e4d538-4c98-4307-8170-a82ca8ff300c", "target_id": "2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a", "target_digests": ["quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906", "quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77"], "target_version": "18.2.4", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": "UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host hostname-redacted at addr (IPv4-address)", "paused": true, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} How can I make the orchestrator work again to either rollback or upgrade to a newer version? the goal is to update this all the way to squid. Thank you

3 3

ceph rdb + libvirt
by Albert Shih 12 Feb '25

12 Feb '25

Hi, I need to create manually few VM with KVM. I would like to know if they are any difference between using a libvirt module and kernel module to access a ceph cluster. Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mer. 12 févr. 2025 16:14:15 CET

6 6

512e -> 4Kn hdd
by Marc 12 Feb '25

12 Feb '25

I used to have a tool from hgst/wd to 'convert' drives to 4Kn, but after a mentally challenging interaction with some wd support, I decided to try and use other solutions mentioned on the internet this seems to produce output similar to the previously used tool sg_format --format --size=4096 /dev/sdd Anyone have experience with this? This is first time the drives are already in the nodes, so I don't have a power cycle on them.

3 3

Grafana certificate issue
by Devender Singh 12 Feb '25

12 Feb '25

Hello all Recently upgraded to 19.2.1 and started facing .. Invalid grafana certificate on hostnode1: Invalid certificate key: [('PEM routines', '', 'no start line’)] Tried suggest steps, even re-deployed grafana but no solution? 1. [WRN] CEPHADM_CERT_ERROR: Invalid grafana certificate on host node1: Invalid certificate key: [('PEM routines', '', 'no start line')] Detected invalid grafana certificates. Please, use the following commands: > ceph config-key set mgr/cephadm/node1/grafana_crt -i <path-to-ctr-file> > ceph config-key set mgr/cephadm/node1/grafana_key -i <path-to-key-file> to set valid key and certificate or reset their value to an empty string in case you want cephadm to generate self-signed Grafana certificates. Once done, run the following command to reconfig the daemon: > ceph orch daemon reconfig grafana.ld6-converged03n.ld6.netskrt.org Also, when I exit my host from maintenance, I get below… but when I rerun same command, it works fine, why? 2. # ceph orch host maintenance exit node2 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 805, in _host_maintenance_exit raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 238, in raise_if_exception e = pickle.loads(c.serialized_exception) TypeError: __init__() missing 2 required positional arguments: 'hostname' and ‘addr' Regards Dev

2 1

RGW - S3 bucket browser and/or S3 explorer
by notna＠ist-einmalig.de 12 Feb '25

12 Feb '25

Hi all. Is anyone aware of a ("S3 bucket browser" | "S3 explorer"), which works with RadosGW static websites? I've played around with those below, but none of them works (in my environment): - https://github.com/qoomon/aws-s3-bucket-browser - https://github.com/guimou/ceph-js-s3-explorer - https://github.com/awslabs/aws-js-s3-explorer - https://github.com/awslabs/aws-js-s3-explorer/tree/v2-alpha Any hints are highly appreciated!

4 9

Ceph RGW Cloud-Sync Issue
by Mark Selby 12 Feb '25

12 Feb '25

This is a 18.2.4 reef cluster Looking for help with cloud sync module. Here is my setup 1. Two Ceph clusters with RGW setup in a MultiSite configuration. There is no data synchronization between the clusters. No data sync groups, flows, or pipes. 2. We use a multi tenant setup with a small number of tenants (10). We have one tenant that is writing to the same bucket in different rgw zones. The names of files are unique in each zone. 3. We are using the cloud-sync module to copy data from our ceph clusters to aws. I have pasted the commands used to create the sync zone The cloud-sync module works but for one issue that I can not figure out. For some reason another bucket in another tenant is showing up as target to sync but was never specified in the original config. I can never get a clean radosgw-admin sync status because there are recovering shards from a bucket in another tenant that I never intended to sync to the cloud. The data sync status of the recovering shard is pointing to a bucket that never had any cloud sync specified. tenant1/bucket1 was never part of any tier config. radosgw-admin sync error list --rgw-zone cloud-sync shows a never ending list of objects that can not be synced from tenant1/bucket1 there are 4 different types of error messages. "message": "failed to sync bucket instance: (22) Invalid argument" "message": "failed to sync bucket instance: (39) Directory not empty" "message": "failed to sync object(16) Device or resource busy" "message": "failed to sync object(39) Directory not empty" One other point is that radosgw-admin sync info --rgw-zone cloud-sync --bucket tenant1/bucket1 seems to indicate that that is part of the cloud-sync config. What I just want to do is remove tenant1/bucket1 from the sync config and stop the data sync such that the shards recover and the error log stop accumulating messages. I would really like a clean sync status for my cloud tier zone. It would also be nice to help understand what is happening here. All and any help is greatly appreciated. # # sync and data sync status # radosgw-admin sync status --rgw-zone=cloud-sync realm 02b82a81-7b16-4760-ab94-08b79176397e (realm) zonegroup 0e44f33d-4e70-4ba6-8c75-794d3d0b69f8 (zonegroup) zone 34e5298b-4537-4ada-b0d9-75579dc9640a (cloud-sync) current time 2025-02-11T23:56:52Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 792edbc3-8516-4014-aad4-8e7b6ebe2757 (zone-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards 1 shards are recovering recovering shards: [72] source: f3e72377-2fb1-4ab6-aa0a-76b91d157fb3 (zone-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [72,75,77,78] oldest incremental change not applied: 2025-02-11T15:56:34.654956-0800 [75] 10 shards are recovering recovering shards: [68,69,70,71,72,73,74,76,77,78] radosgw-admin data sync status --rgw-zone cloud-sync --source-zone zone-2 --shard-id 68 { "shard_id": 68, "marker": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000357645", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2025-02-11T23:42:34.266680Z" }, "pending_buckets": [ "tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0" ], "recovering_buckets": [ "tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0[0]" ], "current_time": "2025-02-11T23:58:08Z" } # # cloud sync zone creation # radosgw-admin zone create \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --endpoints=https://ceph-server:4430 \ --tier-type=cloud radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=s-cloud-sync \ --access-key=<ACCESS> \ --secret=<SECRET> radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connections[].id=aws-cloud-sync radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connections[-1].endpoint=https://s3.amazonaws.com,connections[-1].access_key=<ACCESS>,connections[-1].secret=<SECRET> radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=profiles[].connection_id=aws-cloud-sync radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=profiles[-1].connection_id=aws-cloud-sync,profiles[-1].source_bucket='tenant0:bucket0',profiles[-1].target_path=aws-bucket radosgw-admin zone modify \ --rgw-zonegroup=us \ --rgw-zone=cloud-sync \ --tier-config=connection_id=aws-cloud-sync,target_path=aws-bucket

1 0

Cephadm cluster setup with unit-dir and data-dir
by Rajmohan Ramamoorthy 12 Feb '25

12 Feb '25

Hi, I wanted to understand the behaviour of using --unit-dir and --data-dir while setting up the ceph cluster using cephadm. Can I pass on a reference to a mount points on a different disk (other than the root disk) for these options? The deployment I am trying to achieve is to persist the Ceph containers systemd unit files and their persistent cluster data on a different disk than the root disk. This way, the nodes retains its identity even if the Operating System disk is redeployed on the root disk and the persistent data (systemd unit files and container data) is retained on the other disk. Is this possible with cephadm? I tried pointing unit-dir and data-dir to a different disk mount point but it failed. The unit files are always created in the /etc/systemd/system directory in the root disk. Unit directory: https:// docs.ceph.com/en/latest/man/8/cephadm/?highlight=orch%20device%20zap#cmdopt… Data directory: https:// docs.ceph.com/en/latest/man/8/cephadm/?highlight=orch%20device%20zap#cmdopt… Regards, Rajmohan R

1 0

Ceph Steering Committee Notes 2025-02-10
by Ernesto Puerta 11 Feb '25

11 Feb '25

Hi Cephers, These are the topics covered in today's meeting: - *[Patrick] Any objections to running clean-ci now?* Do you have any important branches in ceph-ci? - https://github.com/ceph/ceph/pull/61709 - Joseph asked if it could delete branches similarly to GitHub UI/API so that they can be restored if needed? - *[Radek] Switching default build type for non-.git builds** to RelWithDebInfo.* This affects downstreams (if no build type is specified)! - Matan's PR: https://github.com/ceph/ceph/pull/61637/ - [Sam] Can it be done with CFLAGS instead? To be announced in ceph-dev. Set default/fall back to Release, which seems the safest. - [Greg] Not all releases are of "Release" type, some include debug info. - [Matan] 1. Release: `-O3 -DNDEBUG` 2. Debug: `-O0 -g` 3. RelWithDebInfo: `-O2 -g -DNDEBUG` 4. MinSizeRel: `-Os -DNDEBUG` - [JosephM] this needs to be backported. - *[Dan-probably absent today**] Upgrade 17.2.8 to 18.2.4 is currently broken*: https://tracker.ceph.com/issues/67660 (Cephadm feature backported to quincy before reef). - 18.2.5 is next, right? Can it be quick pls? - QA question: do we test upgrade from the candidate to the current maj_ver+1? This kind of thing should have been caught. - [Neha]To be followed up offline (QA is currently blocked by the ceph-ci too-many-branches issue). - *[Joseph Mundackal] - containerized builds* - https://github.com/ceph/ceph/pull/59841 - initial version of this merged. Looking for feedback and missing features. - Open trackers to build on ubuntu 24 - https://tracker.ceph.com/issues/62804 - https://tracker.ceph.com/issues/69240 (open PR: https://github.com/ceph/ceph/pull/61326) - https://tracker.ceph.com/issues/69525 - [Matan] There might be issues with compiler versions (e.g.: Ubuntu 22) - *[cbodley] stalebot on backport branches was enabled recently* in https://github.com/ceph/ceph/pull/60736 - can we revert? backport prs aren't tested promptly so someone would have to micromanage them to prevent closing - Revert PR: https://github.com/ceph/ceph/pull/61743 - *[Adam Kraitman] Virtualization solution for hosting the lab services* - VMWare (IBM paid/agreement with Broadcom) vs. Proxmox - [DavidG] temporary (~1 year) solution while infra is containerized. - [Greg] precedent of iSCSI. - NVMe-oF support in Proxmox? Not needed, librbd. Discussion on Tentacle schedule postponed till next week. Kind Regards, Ernesto

1 0