Amazing. How did I miss that. Dropping "ssl: true" and adding "ssl_port=1234" to the rgw_frontend_extra_args values has me sorted. Many thanks! ________________________________ From: Redouane Kachach Sent: Thursday, January 16, 2025 4:39 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: [EXTERNAL] Re: [ceph-users] Cephadm: Specifying RGW Certs & Keys By Filepath You are getting the double option because "ssl: true" ... try to disable ssl since you are passing the arguments and certificates by hand! Another option is to have cephadm generate the certificates for you by setting the `generate_cert` field in the spec to true. But I'm not sure if that works for your environment or not ... Best, Redo. On Thu, Jan 16, 2025 at 4:37 PM Alex Hussein-Kershaw (HE/HIM) <alexhus@microsoft.com<mailto:alexhus@microsoft.com>> wrote: Hi Folks, Looking for some advice on RGW service specs and Cephadm. I've read the docs here: RGW Service — Ceph Documentation<https://docs.ceph.com/en/reef/cephadm/services/rgw/> Using a service spec I can deploy a RGW: service_type: rgw service_id: '25069123' service_name: rgw.25069123 placement: hosts: - raynor-sc-1 extra_container_args: - -v - /etc/pki:/etc/pki:ro - -v - /etc/ssl/:/etc/ssl:ro spec: rgw_frontend_port: 7480 rgw_realm: geored_realm rgw_zone: siteA rgw_zonegroup: geored_zg ssl: true rgw_frontend_extra_args: - "ssl_certificate=/etc/ssl/certs/server.crt" - "ssl_private_key=/etc/ssl/private/server.key" But it fails to start because my "rgw_frontends" config ends up as this. Notice the two entries of "ssl_certificate". beast ssl_port=7480 ssl_certificate=config://rgw/cert/rgw.25069123 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key That causes it to fail to start: debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:26:54.520+0000 7f65bcfdc840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:26:54.597+0000 7f65bcfdc840 0 starting handler: beast debug 2025-01-16T15:26:54.601+0000 7f65bcfdc840 -1 ssl_certificate was not found: rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 no ssl_certificate configured for ssl_port debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: failed initializing frontend debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: initialize frontend fail, r = 22 Correcting that config manually: $ ceph config set client.rgw.25069123.raynor-sc-1.qiatgr rgw_frontends "beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key" Then allows the RGW to start: debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:28:17.767+0000 7fba6e85a840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:28:17.860+0000 7fba6e85a840 0 starting handler: beast debug 2025-01-16T15:28:17.908+0000 7fba6e85a840 0 set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.912+0000 7fba6e85a840 1 mgrc service_daemon_register rgw.643295 metadata {arch=x86_64,ceph_release=squid,ceph_version=ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable),ceph_version_short=19.2.0,container_hostname=raynor-sc-1,container_image=ceph/squid:v19.2.0,cpu=Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz,distro=centos,distro_description=CentOS Stream 9,distro_version=9,frontend_config#0=beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key,frontend_type#0=beast,hostname=raynor-sc-1,id=25069123.raynor-sc-1.qiatgr,kernel_description=#1 SMP Thu Dec 19 20:58:14 EST 2024,kernel_version=5.4.288-1.el8.elrepo.x86_64,mem_swap_kb=0,mem_total_kb=12251748,num_handles=1,os=Linux,pid=7,realm_id=c99973d7-27cb-4abd-8391-bfbe4fa928be,realm_name=geored_realm,zone_id=30816d1d-e315-471b-bc84-6cdc4f6f1408,zone_name=siteA,zonegroup_id=c109a213-6638-4028-b21b-8bc9fcb49cd3,zonegroup_name=geored_zg} Specifying the certificate by file path I think is the right choice for me as: * I don't have a domain, I have a per host certificate covering the IP of the host. Each is different, signed by a self-signed trusted CA. * I don't want to put the hardcoded key into the service spec, as suggested in the docs, as it appears in the mgr logs. My org policy (sensibly) doesn't allow for secrets to appear in logs. * It's just generally more hassle to manage keys on a per VM basis, rather than a static file path. * It will also allow me to not manage RGW services on a per host basis, but use the same spec for all and a count=1 placement, which would be lovely. So my question - is there a way around this such that I can specify the key and certificate by file path and not have to manually change the config to make it work? Thanks, Alex _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>