- ceph-users - lists.ceph.io

ATTN: DOCS
by Lander Duncan 01 Jul '24

01 Jul '24

Good morning, Here are a few issues I noticed with the documentation: 1: Page : Hardware Recommendation ( https://docs.ceph.com/en/latest/start/hardware-recommendations/) Issue: Spelling on word "overprovsioning" 2: Page : Hardware Recommendation ( https://docs.ceph.com/en/latest/start/hardware-recommendations/) Issue: Missing word in sentence "You should also each host’s percentage of the cluster’s overall capacity." Potential fix: Add the word check between "also" and "each" 3: Page: https://docs.ceph.com/en/reef/cephfs/mount-prerequisites/ Issue: Broken Link: "You can use CephFS by mounting it to your local filesystem or by using [cephfs-shell]( https://docs.ceph.com/en/reef/cephfs/cephfs-shell)" 4: Page: https://docs.ceph.com/en/reef/rados/configuration/mon-lookup-dns/#looking-u… Issue: lacking information: As you can see in the code here: https://github.com/ceph/ceph/blob/cac7dcd63481ddbc4fe26f4b8d5918510e8be457/… in lines 790-793 and through experimentation you can use the "mon_dns_srv_name" directive to pass either the "ceph-mon" and use DNS search domains, as mentioned on this page or you can pass the name followed by an underscore to manually pass the search domain. For example: ceph-mon_example.com with records placed in _ceph-mon._example.com Have a lovely day

2 4

rbd migration / cp maintaining modified/create timestamps
by Marc 30 Jun '24

30 Jun '24

How to move an image to a different pool maintaining create/modified dates etc.

1 0

cannot delete service by ceph orchestrator
by Alex from North 29 Jun '24

29 Jun '24

Hi everybody! never seen this before and google stay silent. Just found the same question in 2021 but no answer there ((( so, by ceph orch ls I see: root@ceph1:~/ceph-rollout# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 9m ago 6d count:1 ceph-exporter 10/10 9m ago 6d * crash 10/10 9m ago 6d * grafana ?:3000 1/1 9m ago 6d count:1 mgr 3/3 9m ago 6d ceph1;ceph6;ceph10;count:3 mon 3/3 9m ago 6d ceph1;ceph6;ceph10;count:3 node-exporter ?:9100 10/10 9m ago 6d * osd.osd_using_paths 108 9m ago - <unmanaged> prometheus ?:9095 3/3 9m ago 6d ceph1;ceph6;ceph10;count:3 And then ceph orch rm <service_name> root@ceph1:~/ceph-rollout# ceph orch rm osd.osd_using_paths Invalid service 'osd.osd_using_paths'. Use 'ceph orch ls' to list available services. cannot find it for some reason. In the same time root@ceph1:~/ceph-rollout# ceph orch ls --service_name osd.osd_using_paths --export service_type: osd service_id: osd_using_paths service_name: osd.osd_using_paths unmanaged: true spec: filter_logic: AND objectstore: bluestore Why does it happen? And how to delete this not needed service? Thanks in advance.

2 2

OSD service specs in mixed environment
by Torkil Svensgaard 28 Jun '24

28 Jun '24

Hi We have a bunch of HDD OSD hosts with DB/WAL on PCI NVMe, either 2 x 3.2TB or 1 x 6.4TB. We used to have 4 SSDs pr node for journals before bluestore and those have been repurposed for an SSD pool (wear level is fine). We've been using the following service specs to avoid the PCI NVMe devices for bluestore being provisioned as OSDs: --- service_type: osd service_id: fast service_name: osd.fast placement: host_pattern: '*' spec: data_devices: rotational: 0 size: :1000G <-- only use devices smaller than 1TB = not PCI NVMe filter_logic: AND objectstore: bluestore --- service_type: osd service_id: slow service_name: osd.slow placement: host_pattern: '*' spec: block_db_size: 290966113186 data_devices: rotational: 1 db_devices: rotational: 0 size: '1000G:' <-- only use devices larger than 1TB for DB/WAL filter_logic: AND objectstore: bluestore --- We just bought a few 7.68 TB SATA SSDs to add to the SSD pool which aren't being picked up by the osd.fast spec because they are too large and they could also be picked up as DB/WAL with the current specs. As far as I can determine there is no way to achieve what I want with the existing specs, as I can't filter on PCI vs SATA, only rotational or not, I can't use size, as it only can define an in between range, not an outside range, and I can't use filter_logic OR for the sizes because I need the rotational qualifier to be AND. I can do a osd.fast2 spec with size: 7000G: and change the db_devices size for osd.slow to something like 1000G:7000G but curious to see if anyone would have a different suggestion? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 7

NFSv3 on Reef
by Ramon Orrù 28 Jun '24

28 Jun '24

Hello, I’m using NFSGW on a Reef release, I managed to mount cephfs volumes over NFSv4 with no problems. Now I need to export a CephFS volume using NFSv3 (I need to mount it from a client that only supports v3), and I tried everything I can imagine to make it work. I unchecked the flag created an export: but I got the error: that sounds like a wrong configuration. I created the export using a json file like: { "access_type": "RW", "clients": [ { "access_type": "RW", "addresses": [ “XX.XX.XX.XX" ], "squash": "no_root_squash" }, { "access_type": "RW", "addresses": [ “XX.XX.XX.XX" ], "squash": "no_root_squash" } ], "cluster_id": “nfsgw", "export_id": 1, "fsal": { "fs_name": “XXX", "name": "CEPH", "user_id": "nfs.nfsgw.1" }, "path": "/", "protocols": [ 3, 4 ], "pseudo": "***", "security_label": false, "squash": "no_root_squash", "transports": [ "TCP", "UDP" ] } and applied the conf with ceph nfs export apply nfsgw -i export.yaml But /etc/ganesha/ganesha.conf still contains NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 4; NFS_Port = 12049; } even after pushing a custom config: # ceph nfs cluster config get nfsgw NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 3, 4; NFS_Port = 12049; } and redeploying everything with ceph orch redeploy. Any help would be very appreciated, thanks in advance… Ramon -- Ramon Orrù Servizio di Calcolo Laboratori Nazionali di Frascati Istituto Nazionale di Fisica Nucleare Via E. Fermi, 54 - 00044 Frascati (RM) Italy Tel. +39 06 9403 2345

1 0

Multisite RGW with Self-signed CA & Disconnected Upgrade
by Alex Hussein-Kershaw (HE/HIM) 28 Jun '24

28 Jun '24

Hi ceph-users! I'm going through the process of migrating to use cephadm for my clusters. Previously I used ceph-ansible. I have a few questions related to this. 1. How can I configure RGW multisite with self-signed certificates? I have prototyped the migration and redeployed RGWs. Everything is running the latest version of Reef (18.2.2). My RGWs on both sites are up and I can make requests to them, but they are failing to sync, and "radosgw-admin sync status" shows a generic input/output error. Taking some network capture I can see that the TLS handshake is failing with "Unknown CA", so it looks like the RGWs don't trust my self-signed certificate, I suppose that's not a surprise. However, I can't work out how to establish the trust. I'm running in docker containers on RHEL 9, I've tried mounting in the /etc/pki directory from the machine it's running on, which does contain the self-signed CA (I can curl from the machine), but I still see errors in my multi-site sync. 2. How can I use the "ceph orch upgrade" command in a disconnected environment? After adopting my cluster, I want to upgrade it. It works great, I run something like "ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.2" and it all just works. Awesome. My problem is that I can't reach out to the quay.io registry on my production systems to do this upgrade, they are all locked down and don't have internet access. I can install the docker image directly by copying up the file, but I don't see a way to invoke the "ceph orch upgrade" command with a locally available docker image. Am I missing the obvious? Many thanks, Alex

1 0

Huge amounts of objects orphaned by lifecycle policy.
by Adam Prycki 27 Jun '24

27 Jun '24

Hello, I have a question. Do people use rgw lifecycle policies in production? I had big hopes for this technology bug in practice it seems to be very unreliable. Recently I've been testing different pool layouts and using lifecycle policy to move data between them. Once I've checked orphaned objects I've discovered that my pools were full of orphaned objects. One pool was over 1/3 orphans by volume. Orphan object belonged to data that was moved by lifecycle. Yesterday I decided to recreate one of the pools with 3TiB of data. All 3TiB was located in a single directory of some buckets. I've created a lifecycle which should move it all to STANDARD pool and run radosgw-admin lc process --bucket. After lifecycle finished executing ceph pool still contained 1TiB of data. Removing objects from rgw-orphan-list output reduced pool size to 65GiB and 17k objects. The 17k rados __shadow objects seem to belong to s3 objects which were not moved by lifecycle. I tried lifecycle from radosgw-admin but lifecycle seems to be unable to move them. s3cmd info show that they still report old storage class. Filenames don't contain special characters other than spaces. I have directories with sequentially named objects, some of them cannot be moved by lifecycle. Deleting all the objects form original 3TiB dataset also doesn't help. After running gc and orphan finding tool there are still 1,2k rados objects which should have been deleted but are not considered orphans. I've been testing on 18.2.2. Best regards Adam Prycki

2 2

Ceph Days London CFP Deadline
by Noah Lehman 27 Jun '24

27 Jun '24

Hi everyone, This is a reminder that the deadline to submit a speaking proposal for Ceph Days London is this Sunday, June 30th. Be sure to submit a proposal before then! Details here <https://ceph.io/en/community/events/2024/ceph-days-london/>. Best, Noah

1 0

ceph rgw zone create fails EINVAL
by Matthew Vernon 27 Jun '24

27 Jun '24

Hi, I'm running cephadm/reef 18.2.2. I'm trying to set up multisite. I created realm/zonegroup/master zone OK (I think!), edited the zonegroup json to include hostnames. I have this spec file for the secondary zone: rgw_zone: codfw rgw_realm_token: "SECRET" placement: label: "rgw" [I get "SECRET" by doing ceph rgw realm tokens on the master, and C&Ping the field labelled "token"] If I then try and apply this with: ceph rgw zone create -i /root/rgw_secondary.yaml It doesn't work, and I get an unhelpful backtrace: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 96, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 304, in _cmd_rgw_zone_create return HandleCommandResult(retval=0, stdout=f"Zones {', '.join(created_zones)} created successfully") ^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: sequence item 0: expected str instance, int found I assume I've messed up the spec file, but it looks like the one in the docs[0]. Can anyone point me in the right direction, please? [if the underlying command emits anything useful, I can't find it in the logs] Thanks, Matthew [0] https://docs.ceph.com/en/reef/mgr/rgw/#realm-credentials-token

3 7

pg deep-scrub control scheme
by David Yang 27 Jun '24

27 Jun '24

Hello everyone. I have a cluster with 8321 pgs and recently I started to get pg not deep-scrub warnings. The reason is that I reduced max_scrub to avoid the impact of scrub on IO. Here is my current scrub configuration: ~]# ceph tell osd.1 config show|grep scrub "mds_max_scrub_ops_in_progress": "5", "mon_scrub_inject_crc_mismatch": "0.000000", "mon_scrub_inject_missing_keys": "0.000000", "mon_scrub_interval": "86400", "mon_scrub_max_keys": "100", "mon_scrub_timeout": "300", "mon_warn_pg_not_deep_scrubbed_ratio": "0.800000", "mon_warn_pg_not_scrubbed_ratio": "0.500000", "osd_debug_deep_scrub_sleep": "0.000000", "osd_deep_scrub_interval": "1296000.000000", "osd_deep_scrub_keys": "1024", "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", "osd_deep_scrub_randomize_ratio": "0.080000", "osd_deep_scrub_stride": "131072", "osd_deep_scrub_update_digest_min_age": "7200", "osd_max_scrubs": "1", "osd_requested_scrub_priority": "120", "osd_scrub_auto_repair": "false", "osd_scrub_auto_repair_num_errors": "5", "osd_scrub_backoff_ratio": "0.660000", "osd_scrub_begin_hour": "0", "osd_scrub_begin_week_day": "0", "osd_scrub_chunk_max": "25", "osd_scrub_chunk_min": "5", "osd_scrub_cost": "52428800", "osd_scrub_during_recovery": "false", "osd_scrub_end_hour": "0", "osd_scrub_end_week_day": "0", "osd_scrub_extended_sleep": "0.000000", "osd_scrub_interval_randomize_ratio": "0.500000", "osd_scrub_invalid_stats": "true", "osd_scrub_load_threshold": "0.500000", "osd_scrub_max_interval": "1296000.000000", "osd_scrub_max_preemptions": "5", "osd_scrub_min_interval": "259200.000000", "osd_scrub_priority": "5", "osd_scrub_sleep": "0.000000", I am currently trying to adjust the interval of scrub. Is there a calculation formula that can be used to easily configure the scrub/deepscrub strategy? At present, there is only the adjustment of individual values, and then it is a long wait, and there may be no progress in the end.

2 2