- ceph-users - lists.ceph.io

Limits on the number of cephx keys used
by James Tocknell 11 Jun '25

11 Jun '25

Hi All As far as I can see, there is no guidance on the number of cephx keys that can be in use at one time. Is there a number at which ceph becomes much slower e.g. 100, 10000, 1000000? I'm wondering how best to manage keys across many clients (let's say 1000s for now), most of which won't actually be connected at the same time. Regards James

4 4

rgw + LDAP
by Albert Shih 11 Jun '25

11 Jun '25

Hi, I'm currently running ceph 18.2.7 and I try to connect my RGW to my LDAP After many hours to battle with that I end up to turn on every debug flag I can find. It seem the RGW try to bind anonymously to my ldap server, here the log on my ldap server (openldap) Jun 10 16:32:02 ldaps2-m2 slapd[453]: conn=836633 op=1 SRCH base="dc=obspm,dc=fr" scope=2 deref=0 filter="(&(&(objectClass=inetOrgPerson)(memberOf=cn=s3storage,ou=groups,ou=services_centraux,dc=obspm,dc=fr))(uid=jas))" Jun 10 16:32:02 ldaps2-m2 slapd[453]: conn=836633 op=1 SRCH attr=uid Jun 10 16:32:02 ldaps2-m2 slapd[453]: ==> limits_get: conn=836633 op=1 self="[anonymous]" this="dc=obspm,dc=fr" We don't want to allow the anonymous bind here. I set ceph config set client.rgw rgw_ldap_binddn "cn=s3storage,ou=dsa,ou=services_centraux,dc=obspm,dc=fr" ceph config set client.rgw rgw_ldap_secret "/etc/ceph/ldappw.txt" ceph config set client.rgw rgw_ldap_searchdn "dc=obspm,dc=fr" I check the file /etc/ceph/ldappw.txt, give it to root:root, to ceph:ceph, change the right 600/644 but whatever I does it'still trying to bind anonymously. I can even find the string s3storage,ou=dsa inside the debug log of my ldap server. It's like rgw don't even try ton bind with the binddn I give it to him I try to run a shell inside the podman container but was unable to find where the rgw config is «instanciated» inside the container (to see what it get). Any clue ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mar. 10 juin 2025 16:36:11 CEST

3 6

bucket notification for kafka with different sasl-mechanism
by farhad kh 11 Jun '25

11 Jun '25

hi , for bucket notification anybody have experience use kafka with encryption? i try config topic with this configuration { "topics": [ { "owner": "farhad", "name": "sre-s3", "dest": { "push_endpoint": "kafka://sre_fkhedrian:d2Q9BAYN9RVj@10.248.17.6:9092", "push_endpoint_args": "Version=2010-03-31&kafka-ack-level=broker&sasl_mechanism=SCRAM-SHA-256&security_protocol=SASL_PLAINTEXT&verify-ssl=False", "push_endpoint_topic": "sre-s3", "stored_secret": true, "persistent": true, "persistent_queue": ":sre-s3", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "None" }, "arn": "arn:aws:sns:podspace::sre-s3", "opaqueData": "", "policy": "", "subscribed_buckets": [] } ] } but get this error in kafka [2025-06-09 12:58:12,087] INFO [SocketServer listenerType=BROKER, nodeId=1] Failed authentication with /10.248.35.142 (channelId=10.248.17.6:9092-10.248.35.142:40518-1-1055) (Unsupported SASL mechanism PLAIN) (org.apache.kafka.common.network.Selector), of course i have use sasl_mechanism=SCRAM-SHA-256 in kafka for other program and test with a python script i sure this working. where is problem ?how can i fix this ?

1 0

Change netmask of CIDR
by Mahdi Noorbala 10 Jun '25

10 Jun '25

Hello everybody We have a ceph cluster in production. It's used for openstack as rbd image. Now we want to change netmask from /22 to /21. What do you think? Is there any problem in this change? Is there any concern about this change? Thanks.

2 5

Announcing go-ceph v0.34.0
by Anoop C S 10 Jun '25

10 Jun '25

Hi, I'm happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.34.0 Highlights for this release includes the introduction of new smb admin API sub-package for handling SMB shares. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S.

1 0

Configure RGW without ceph.conf
by Albert Shih 10 Jun '25

10 Jun '25

Hi, I'm trying to configure my rgw with keycloak, according to the documentation I need to modify /etc/ceph/ceph.conf Is they are anyway to configure the RGW with keycloak WITHOUT modifying this file ? Like using ceph config set or event in /etc/ceph/rgw.conf (or whatever) I ask that because my ceph.conf are manage with some deployment tools (puppet in my case). Regards JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 09 juin 2025 22:37:23 CEST

3 5

mds daemon damaged
by bbk＠nocloud.ch 10 Jun '25

10 Jun '25

Hi, after running the cluster for years, it is the first time i have a problem that seems to require some expert knowlege ;-) I had some network problems, which i suspect that lead to the damaged mds service. Just as a side note: i upgraded to 17.2.2 but it had been running for about a hour till the network outage, so i don't think this was related. However, now i wonder how to go from here. ``` # ceph -s cluster: health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged services: mon: 5 daemons, quorum ceph3,ceph4,ceph1,ceph5,ceph2 (age 46m) mgr: ceph5.zmvagf(active, since 5h), standbys: ceph2.defhpj mds: 0/1 daemons up, 2 standby osd: 10 osds: 10 up (since 46m), 10 in (since 2h) data: volumes: 0/1 healthy, 1 recovering; 1 damaged pools: 4 pools, 193 pgs objects: 5.42M objects, 13 TiB usage: 26 TiB used, 11 TiB / 37 TiB avail pgs: 193 active+clean io: client: 200 KiB/s rd, 84 KiB/s wr, 199 op/s rd, 169 op/s wr ``` ``` # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [ERR] MDS_ALL_DOWN: 1 filesystem is offline fs cephfs is offline because no MDS is active for it. [ERR] MDS_DAMAGE: 1 mds daemon damaged fs cephfs mds.0 is damaged ``` ``` # ceph fs status cephfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs.cephfs.meta metadata 1293M 1954G cephfs.cephfs.data data 2797G 1954G ecpool data 23.2T 2931G STANDBY MDS cephfs.ceph1.yzqmuo cephfs.ceph3.vmieie MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) ``` According to * https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ i came up with the following procedure. Deny all reconnect to clients. ```sh ceph config set mds mds_deny_all_reconnect true ``` Deny new clients. ```sh ceph fs set cephfs refuse_client_session true ``` Backup the current journal: ```sh mkdir /root/mds-damaged cephfs-journal-tool --rank=cephfs:0 journal export /root/mds-damaged/backup-rank0.bin ``` Recover file metadata and discard what is damaged: ```sh cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary ``` Truncate any journal that is corrupt or that an MDS cannot replay: ```sh cephfs-journal-tool --rank=cephfs:0 journal reset --yes-i-really-really-mean-it ``` Reset the SessionMap: ```sh cephfs-table-tool all reset session ``` Does this make sense? Yours, bbk

2 3

Ceph Orchestrator error
by Iztok Gregori 10 Jun '25

10 Jun '25

Hi to all! I'm trying to setup a RGW service on my cluster and I want that the SSL certificates are create automatically, so I wrote the following spec file: # rgw.yaml service_type: rgw service_id: rgw.foo placement: label: rgw count_per_host: 1 networks: - 172.16.10.0/24 spec: rgw_realm: 'foo' rgw_zone: 'fool' rgw_zonegroup: 'fool_group' ssl: true generate_cert: true rgw_frontend_port: 4443 wildcard_enabled: true zonegroup_hostnames: - "fool.foo.com" extra_container_args: - "--stop-timeout=120" config: rgw_exit_timeout_secs: "120" rgw_graceful_stop: true When I apply it gives me the following error: # ceph orch apply -i rgw.yaml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'generate_cert' Am I missing something obvious regarding the "generate_cert" part? The cluster is a fresh install of Squid (19.2.2). Any help is appreciated! Regards Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. Telephone: +39 040 3758948 http://www.elettra.eu

2 3

Confuse by rgw and certificate
by Albert Shih 10 Jun '25

10 Jun '25

Hi everyone. I'm trying to configure a RGW for S3. I'm currently running reef 18.2.7 I was able to make the rgw working (= the service is up and listen on the correct port) without ssl. I'm trying to configure the ssl port and I didn't find the correct syntaxe for that. I create a yaml file with something like spec: rgw_frontend_port: 8080 zone_endpoints: https://host1:8080, https://host2:8080,etc. ssl: true rgw_frontend_ssl_certificate: | -----BEGIN RSA PRIVATE KEY----- .... -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- or with the all chain of CA spec: rgw_frontend_port: 8080 zone_endpoints: https://host1:8080, https://host2:8080,etc. ssl: true rgw_frontend_ssl_certificate: | -----BEGIN RSA PRIVATE KEY----- .... -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- ..... -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- both not working the journalctl say ssl_private_key was not found: rgw/cert/obspm/meudon.key I find somewhere on the net this syntaxe spec: rgw_frontend_port: 8080 zone_endpoints: https://host1:8080, https://host2:8080,etc. ssl: true ssl_private_key: | -----BEGIN RSA PRIVATE KEY----- .... -----END RSA PRIVATE KEY----- ssl_certificate: | -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- and I got Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'ssl_private_key' Any clue ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 06 juin 2025 10:47:08 CEST

3 5

Discrepancy between users in Ceph dashboard Bucket users and radosgw-admin user list
by Al 05 Jun '25

05 Jun '25

I've been playing with installing rook-ceph in a Rancher managed k8s cluster for the past week or so and while it does work, some things aren't working as expected. In particular I'm trying to use the CLI to add users and buckets during cluster creation, to integrate with my IaC/IaaS scripts, and am running into a confusing issue. I can use the Ceph dashboard to create users and buckets, and then use the aws s3api commands to manipulate the buckets and all works well; however the documentation says that I can add bucket users with radosgw-admin, but when I do this, the users I create do not show up anywhere I can find in the Ceph dashboard, and they cannot manipulate buckets via the aws s3api commands. I'm sitting on a fresh install of rook-orchestrator and ceph-rook-cluster now via helm and the dashboard shows a single user in the Buckets->Users area named 'dashboard-admin' while 'radosgw-admin user list' returns an empty array. radosgw-admin zone list and zonegroups list show two zones and two zonegroups; one named 'ceph-objectstore' and one named 'default'. I'm not specifically trying to use zones or zonegroups but I thought I would mention this since it is relevant. Both show in the Object-Multi site page in the dash as well, though there's an extra "something" above the ceph-objectstore tree that has a balance scale icon; i don't know what these represent as there is no legend on the page. I haven't tried to add a user in this current config since I'd like to wait for some ideas from the group, but the commands I've been trying are: TOOLS_POD=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[*].metadata.name}') kubectl -n rook-ceph exec -it $TOOLS_POD -- radosgw-admin user create --uid=testuser --display-name="Test User" --access=full The documentation indicates that this is how you add bucket users to the cluster. TIA for any assistance.

1 0