tls certs per manager - does it work?
Hi guys. I thought I had a problem with restarting dashboard - following in docs on ssl certs. But ! I might have other issue - restarting dashboard does what is expected of - but, with these: -> $ ceph dashboard set-ssl-certificate-key podster2.mine.priv -i /root/podster2.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster2.mine.priv -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key podster1.mine.priv -i /root/podster1.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster1.mine.priv -i /root/podster1.mine.priv.crt If I go to podster2 I get podster1's cert, perhaps these do not do anything, If I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster2.mine.priv.key then podster1 has podster2's cert - naturally - but (weirdly?) if go to podster2 (by FQDN) then I get redirected to IP (as in URL) of when I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster1.mine.priv.key and I go to podster2 then, no redirection, "only" warning about cert being of podster1's, podster1 has a good cert! If I do (again) - having podster1 have "correct" cert, with no per-node/manager cert - per-node/mgr cert, as shown earlier, then.. again, podster1 has its correct cert, podster2 has podster1's cert (no ! redirection to IP though). How much of an "issue" this might be - given the fact that I deployed anew cluster twice (lab) - and this reproduces each time. Or, I'm missing some obvious, big picture - and it's simply bad luck, deploying cluster twice. ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) all thoughts are much appreciated. many thanks, L.
Hi, if I'm not mistaken, setting a cert/key combination with ceph dashboard set-ssl-certificate[-key] -i cert[key] only populates this config-keys: mgr/dashboard/crt mgr/dashboard/key This cert/key pair should then contain either a wildcard to be applicable to all mgr daemons. If you need per daemon cert/key pairs, you need to add more cert/key pairs like this: ceph config-key set mgr/dashboard/{MGR1}/crt -i cert.pem ceph config-key set mgr/dashboard/{MGR1}/key -i key.pem ceph config-key set mgr/dashboard/{MGR2}/crt -i cert.pem ceph config-key set mgr/dashboard/{MGR2}/key -i key.pem We use the latter approach (not wildcard certs), but some of our customers use wildcards and it works as well. Hope this helps! Eugen Zitat von lejeczek <peljasz@yahoo.co.uk>:
Hi guys.
I thought I had a problem with restarting dashboard - following in docs on ssl certs. But ! I might have other issue - restarting dashboard does what is expected of - but, with these:
-> $ ceph dashboard set-ssl-certificate-key podster2.mine.priv -i /root/podster2.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster2.mine.priv -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key podster1.mine.priv -i /root/podster1.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster1.mine.priv -i /root/podster1.mine.priv.crt
If I go to podster2 I get podster1's cert, perhaps these do not do anything,
If I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster2.mine.priv.key then podster1 has podster2's cert - naturally - but (weirdly?) if go to podster2 (by FQDN) then I get redirected to IP (as in URL) of when I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster1.mine.priv.key and I go to podster2 then, no redirection, "only" warning about cert being of podster1's, podster1 has a good cert!
If I do (again) - having podster1 have "correct" cert, with no per-node/manager cert - per-node/mgr cert, as shown earlier, then.. again, podster1 has its correct cert, podster2 has podster1's cert (no ! redirection to IP though).
How much of an "issue" this might be - given the fact that I deployed anew cluster twice (lab) - and this reproduces each time. Or, I'm missing some obvious, big picture - and it's simply bad luck, deploying cluster twice.
ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) all thoughts are much appreciated. many thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I just noticed that you're already using the per-mgr config (I missed the hostnames after the set-ssl-certificate-key command), sorry about that, I haven't used that dashboard command like that (yet). So in theory you should aready see per mgr entries in the config-key list. Can you confirm? Zitat von Eugen Block <eblock@nde.ag>:
Hi,
if I'm not mistaken, setting a cert/key combination with
ceph dashboard set-ssl-certificate[-key] -i cert[key]
only populates this config-keys:
mgr/dashboard/crt mgr/dashboard/key
This cert/key pair should then contain either a wildcard to be applicable to all mgr daemons. If you need per daemon cert/key pairs, you need to add more cert/key pairs like this:
ceph config-key set mgr/dashboard/{MGR1}/crt -i cert.pem ceph config-key set mgr/dashboard/{MGR1}/key -i key.pem
ceph config-key set mgr/dashboard/{MGR2}/crt -i cert.pem ceph config-key set mgr/dashboard/{MGR2}/key -i key.pem
We use the latter approach (not wildcard certs), but some of our customers use wildcards and it works as well.
Hope this helps! Eugen
Zitat von lejeczek <peljasz@yahoo.co.uk>:
Hi guys.
I thought I had a problem with restarting dashboard - following in docs on ssl certs. But ! I might have other issue - restarting dashboard does what is expected of - but, with these:
-> $ ceph dashboard set-ssl-certificate-key podster2.mine.priv -i /root/podster2.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster2.mine.priv -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key podster1.mine.priv -i /root/podster1.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster1.mine.priv -i /root/podster1.mine.priv.crt
If I go to podster2 I get podster1's cert, perhaps these do not do anything,
If I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster2.mine.priv.key then podster1 has podster2's cert - naturally - but (weirdly?) if go to podster2 (by FQDN) then I get redirected to IP (as in URL) of when I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster1.mine.priv.key and I go to podster2 then, no redirection, "only" warning about cert being of podster1's, podster1 has a good cert!
If I do (again) - having podster1 have "correct" cert, with no per-node/manager cert - per-node/mgr cert, as shown earlier, then.. again, podster1 has its correct cert, podster2 has podster1's cert (no ! redirection to IP though).
How much of an "issue" this might be - given the fact that I deployed anew cluster twice (lab) - and this reproduces each time. Or, I'm missing some obvious, big picture - and it's simply bad luck, deploying cluster twice.
ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) all thoughts are much appreciated. many thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
According to ceph own docs - https://docs.ceph.com/en/quincy/mgr/dashboard/?highlight=certificate#ssl-tls... 'dashboard' commands do take per-name/mgr certs, case into account, there should be no need to go to 'config' directly. Those cmds seem to work - meaning they create config paths and values - I see: -> $ ceph config-key get mgr/dashboard/podster2.mine.priv/crt -> $ ceph config-key get mgr/dashboard/podster2.mine.priv/key But it seems that whatever is "global" - is under mgr/dashboard/key|cert - takes precedence and per-node/mgr certs are ignored by ceph, namely, if I do: -> $ ceph dashboard create-self-signed-cert then, I do per-node cert (with FQDN or with short hostname), self-signed dashboard cert will be still in effect, on all nodes/mgrs. Would this be a bug - could anybody confirm/reproduce? many thanks, L
Yes, that's what I was referring to. If you have one "global" cert for all mgrs it will override the per-mgr config. I would recommend to remove the global entry and then see if your per-mgr certs work as expected. During a fresh cluster bootstrap (if you don't skip the dashboard), it will generate a global cert for you. So I wouldn't consider this a bug, but a note in the docs might help. Zitat von lejeczek <peljasz@yahoo.co.uk>:
According to ceph own docs - https://docs.ceph.com/en/quincy/mgr/dashboard/?highlight=certificate#ssl-tls... 'dashboard' commands do take per-name/mgr certs, case into account, there should be no need to go to 'config' directly. Those cmds seem to work - meaning they create config paths and values - I see: -> $ ceph config-key get mgr/dashboard/podster2.mine.priv/crt -> $ ceph config-key get mgr/dashboard/podster2.mine.priv/key But it seems that whatever is "global" - is under mgr/dashboard/key|cert - takes precedence and per-node/mgr certs are ignored by ceph, namely, if I do: -> $ ceph dashboard create-self-signed-cert then, I do per-node cert (with FQDN or with short hostname), self-signed dashboard cert will be still in effect, on all nodes/mgrs. Would this be a bug - could anybody confirm/reproduce? many thanks, L _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If I do: -> $ ceph config-key rm mgr/dashboard/key -> $ ceph config-key rm mgr/dashboard/crt having per-mgr certs, then there is no communication to mgrs' https at all. and -> $ ceph health detail HEALTH_OK
Did you fail the mgr after removing the keys? Or at least disable and enable the dashboard? I just tested this again on a two-node (virtual lab) Quincy cluster, removed the global crt/key and injected per mgr cert and keys, and it works: "mgr/dashboard/quincy-1.pcasdd/crt", "mgr/dashboard/quincy-1.pcasdd/key", "mgr/dashboard/quincy-2.mwrlmj/crt", "mgr/dashboard/quincy-2.mwrlmj/key", quincy-1:~ # ceph mgr services { "dashboard": "https://172.17.2.252:8443/", "prometheus": "http://172.17.2.252:9283/" } And I have access to the dashboard. Zitat von lejeczek <peljasz@yahoo.co.uk>:
If I do: -> $ ceph config-key rm mgr/dashboard/key -> $ ceph config-key rm mgr/dashboard/crt having per-mgr certs, then there is no communication to mgrs' https at all. and -> $ ceph health detail HEALTH_OK _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
nope, not in my env - and I believe I do a plain-vanilla deployment with 'cephadm' I might have: -> $ ceph config-key get mgr/dashboard/key Error ENOENT: -> $ ceph config-key get mgr/dashboard/crt Error ENOENT: -> $ ceph config-key get mgr/dashboard/podster3.mine.priv/key -----BEGIN PRIVATE KEY----- MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCw7whSWhNzKln/ ... -> $ ceph config-key get mgr/dashboard/podster3.mine.priv/crt -----BEGIN CERTIFICATE----- MIIFIjCCA4qgAwIBAgIED/sABDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlN ... -> $ ceph config-key get mgr/dashboard/podster1.mine.priv/key -----BEGIN PRIVATE KEY----- MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDbKrY2d/Tsg3xG ... -> $ ceph config-key get mgr/dashboard/podster1.mine.priv/crt -----BEGIN CERTIFICATE----- MIIFIjCCA4qgAwIBAgIED/sAAjANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlN ... -> $ ceph mgr services { "prometheus": "http://10.1.1.63:9283/" } The moment I add/recreate "global" cert: -> $ ceph dashboard set-ssl-certificate -i /root/bin/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/bin/podster1.mine.priv.key dashboard starts and ! without fail/disable/enable: -> $ ceph mgr services { "dashboard": "https://10.1.1.63:8443/", "prometheus": "http://10.1.1.63:9283/" } But that, as show above, as per my original, first message, makes... In this case: 10.1.1.63 is poster3 - makes poster3 report "invalid" cert for it's poster1's cert. I believe this is a bug and is easily reproducible - unless it's intended behavior? Since I reported, I moved to 'reef' from 'squid' - the same (mis)behavior - I'm on Centos 9. I merely deploy a cluster with: -> $ cephadm bootstrap --mon-ip 10.1.1.61 --mon-id podster1.mine.priv --allow-fqdn-hostname which does its things. I add host, monitors, etc. Everything else seems to be a ok, it reproduces every time.
"mgr/dashboard/quincy-1.pcasdd/crt", "mgr/dashboard/quincy-1.pcasdd/key", "mgr/dashboard/quincy-2.mwrlmj/crt", "mgr/dashboard/quincy-2.mwrlmj/key", out of curiosity - what are you using here for $name - these are hostnames/FQDNs? According do CEPH's doc: "..(where |$name| is the name of the |ceph-mgr| instance, usually the hostname):" And for certs with hostnames there, dns resolutions must work - does that work for you or you connect to mgrs' IPs only? thanks, L.
Those aren't FQDNs but the MGR daemons: quincy-1:~ # ceph orch ps --daemon-type mgr -f json | jq -r '.[].daemon_name' mgr.quincy-1.pcasdd mgr.quincy-2.mwrlmj So the hostname (short) with a random suffix created by cephadm. DNS resolution works, although in this specific (test) environment I do that with /etc/hosts. Zitat von lejeczek <peljasz@yahoo.co.uk>:
"mgr/dashboard/quincy-1.pcasdd/crt", "mgr/dashboard/quincy-1.pcasdd/key", "mgr/dashboard/quincy-2.mwrlmj/crt", "mgr/dashboard/quincy-2.mwrlmj/key",
out of curiosity - what are you using here for $name - these are hostnames/FQDNs? According do CEPH's doc: "..(where |$name| is the name of the |ceph-mgr| instance, usually the hostname):" And for certs with hostnames there, dns resolutions must work - does that work for you or you connect to mgrs' IPs only?
thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
lejeczek