- ceph-users - lists.ceph.io

Temporay credentials in STS
by Rok Jaklič 15 Dec '25

15 Dec '25

Hi, I try to follow https://docs.ceph.com/en/reef/radosgw/STS/ If I print role_response, there are no credentials in the response object? Where can I get credentials for TESTER1 after creating a role and setting policy for it? Kind regards, Rok

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by David Walter 15 Dec '25

15 Dec '25

Hi Jeff, What migration did you do that you say the mismatches went away? Did the crashing of OSDs also stop? Migration to a new pool on the same or different OSDs? With or without EC optimization enabled? Or re-writing the data in the same pool by purging one OSD at a time? Best, David On 12/5/25 11:02, Jeff Bailey via ceph-users wrote: > I've seen OSD's crash when running an fstrim on an RBD with an EC data > pool with the EC optimizations enabled. Always two OSD's. It's a 8+2 > pool but I don't know if the "2"s are a coincidence or not. Other than > that, I haven't seen any OSD crashes with EC optimizations. We had > the same size mismatches that went away after migrating the data. > > > On 12/5/2025 10:34 AM, Reto Gysi via ceph-users wrote: >> Hi Bill >> >> Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < >> bill_scales(a)uk.ibm.com>: >> >>> Hi Reto, >>> >>> Sorry to hear about your problems with turning on ec optimizations. >>> I've >>> led the team that developed this feature, so we are keen to help >>> understand >>> what happened to your system. >>> >>> Your configuration looks fine as far as being supported with ec >>> optimizations. The daemons (mons, osds and mgr) need to be running >>> tentacle >>> code to use this feature, there is no requirement to update any of your >>> clients. >>> >>> Do you have any more examples of the inconsistent objects logs that you >>> can share with me? >>> >>> There was a bug that we fixed late in the development cycle which was >>> scrubbing incorrectly reporting a size mismatch for objects written >>> prior >>> to turning on optimizations. This is because prior to turning on >>> optimizations objects are padded to a multiple of the stripe_width, >>> afterwards objects are no longer padded. The scrub code was >>> occasionally >>> getting confused and incorrectly reporting an inconsistency. In this >>> case >>> the scrub was reporting false positive errors - there was nothing wrong >>> with the data and no problems accessing the data. I notice the log you >>> shared in the email was for a size mismatch, I'm interested whether >>> all the >>> logs were for mismatched sizes. We will do some further work to confirm >>> that the fix is in the 20.2.0 tentacle release and that there are no >>> problems with the back port. >>> >> As far I could see the errors where all about the mismatched sizes. >> >> >>> You also mention that you saw some OSD crashes. Do you have any further >>> information about these? >>> >> I just could cause another crash with starting up a windows VM, and >> doing a >> WIndows file history backup to a drive that is on an RBD Image on pool >> rbd_ecpool where I had the allow_ec_optimization flag enabled. >> >> <disk type="network" device="disk"> >> <driver name="qemu" type="raw" cache="writethrough" io="threads" >> discard="unmap" detect_zeroes="unmap"/> >> <auth username="admin"> >> <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> >> </auth> >> <source protocol="rbd" name="rbd/game_windows_backup_drive" >> index="3"> >> <host name="zephir" port="6789"/> >> </source> >> <target dev="vdd" bus="virtio"/> >> <alias name="virtio-disk3"/> >> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" >> function="0x0"/> >> </disk> >> >> root@zephir:~# ceph -s >> cluster: >> id: 27923302-87a5-11ec-ac5b-976d21a49941 >> health: HEALTH_WARN >> 2 osds down >> Reduced data availability: 49 pgs inactive >> Degraded data redundancy: 15722192/79848340 objects degraded >> (19.690%), 246 pgs degraded, 247 pgs undersized >> >> services: >> mon: 3 daemons, quorum zephir,debian,raspi (age 20h) >> [leader: >> zephir] >> mgr: zephir.enywvy(active, since 21h), standbys: >> debian.nusuye >> mds: 3/3 daemons up, 3 standby >> osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 >> remapped >> pgs >> cephfs-mirror: 2 daemons active (2 hosts) >> rbd-mirror: 2 daemons active (2 hosts) >> rgw: 1 daemon active (1 hosts, 1 zones) >> tcmu-runner: 5 portals active (2 hosts) >> >> data: >> volumes: 3/3 healthy >> pools: 25 pools, 450 pgs >> objects: 17.52M objects, 62 TiB >> usage: 116 TiB used, 63 TiB / 179 TiB avail >> pgs: 10.889% pgs not active >> 15722192/79848340 objects degraded (19.690%) >> 197 active+undersized+degraded >> 183 active+clean >> 49 undersized+degraded+peered >> 15 active+clean+scrubbing >> 5 active+clean+scrubbing+deep >> 1 active+undersized >> >> io: >> client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr >> >> root@zephir:~# ceph osd status >> ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE >> 0 debian 11.2T 5289G 0 0 0 0 exists,up >> 1 zephir 11.6T 4846G 0 0 0 0 exists >> 2 zephir 10.7T 5827G 0 0 2 4 exists,up >> 3 zephir 11.0T 5497G 0 0 0 0 exists >> 4 zephir 159G 1376G 0 0 2 0 exists,up >> 5 zephir 120G 1415G 0 0 0 0 exists,up >> 6 debian 11.3T 5200G 0 0 1 0 exists,up >> 7 zephir 305G 1230G 0 0 0 0 exists,up >> 8 zephir 10.9T 5594G 0 0 0 0 exists,up >> 9 zephir 11.9T 4616G 0 0 0 0 exists,up >> 10 zephir 235G 1300G 0 0 0 0 exists,up >> 11 zephir 223G 1312G 0 0 0 0 exists,up >> 12 zephir 11.0T 5479G 0 0 0 0 exists,up >> 13 debian 118G 631G 0 0 3 8 exists,up >> 14 debian 345G 1190G 0 0 0 0 exists,up >> 15 debian 11.4T 5079G 0 0 0 0 exists,up >> 16 debian 554G 3029G 0 0 3 8 exists,up >> 17 debian 12.5T 5874G 0 0 1 0 exists,up >> root@zephir:~# >> >> I attached the log of osd.1 to >> https://filebin.net/jwvs6kuqrc7hx8id >> >> Best Regards, >> >> Reto >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: scrape-health-metrics fails with sudo PAM account management error
by Flemming Frandsen 15 Dec '25

15 Dec '25

I think I tracked down the problem to lacking permissions on the /etc/shadow file in the squid container image: docker exec -it ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-9 ls -l /etc/shadow ---------- 1 root root 583 Nov 4 16:31 /etc/shadow I have verified that the same situation exists in the tentacle images. if I make the file readable by root: docker exec -it ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-9 chmod u+r /etc/shadow docker exec -it ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-9 ls -l /etc/shadow -r-------- 1 root root 583 Nov 4 16:31 /etc/shadow Then the sudo command needed by osd works perfectly. Looking at the hosts dmesg I finally noticed that the problem is with the default apparmor profile called unix_chkpwd: [Mon Dec 15 09:52:25 2025] audit: type=1400 audit(1765792348.851:248): apparmor="DENIED" operation="open" class="file" profile="unix-chkpwd" name="/dev/console" pid=1028560 comm="unix_chkpwd" requested_mask="w" denied_mask="w" fsuid= 0 ouid=0 I disabled the profile by running these commands on the host: mkdir /etc/apparmor.d/disable/ ln -s /etc/apparmor.d/unix-chkpwd /etc/apparmor.d/disable/ apparmor_parser -R /etc/apparmor.d/unix-chkpwd After getting rid of the app armor profile all the osd containers started working perfectly. It's quite possible that there is a more elegant solution, but I can't be bothered to find it at the moment. On Fri, 12 Dec 2025 at 11:58, Flemming Frandsen via ceph-users < ceph-users(a)ceph.io> wrote: > I've been building a new cluster with cephadm, the OS is Ubuntu 24.04 and > I'm using the ubuntu provided host packages, docker is 29.1.2 and > containerd is 2.2.0 and the ceph release is squid 19.2.3. > > Everything seems to work just perfectly, except for scrape-health-metrics, > which records this result (for all of the osds in the 4 hosts in the > cluster): > "20251212-074916": { > "dev": "/dev/nvme5n1", > "error": "smartctl failed", > "nvme_smart_health_information_add_log_error": "nvme returned an > error: sudo: exit status: 1", > "nvme_smart_health_information_add_log_error_code": -22, > "nvme_vendor": "samsung", > "smartctl_error_code": -22, > "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: > exit status: 1\nstdout:\n" > }, > > Digging though the source I found that it's the OSD container that runs the > command: sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 > > I can run the command in the container using: > docker exec -it -u ceph ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-3 > sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 > > The result of the manual command is: > sudo: PAM account management error: Authentication service cannot retrieve > authentication info > sudo: a password is required > > I have enabled debugging for sudo, in the container and verified that the > command that the osd is running, is same one and that the error is the same > as well, running ceph device scrape-health-metrics causes this sudo debug > output: > > Dec 12 10:30:35 sudo[845] user command "/usr/sbin/smartctl -x --json=o > /dev/nvme5n1" matches sudoers command "/usr/sbin/smartctl -x --json=o > /dev/*": true @ command_matches() ./match_command.c:667 > Dec 12 10:30:35 sudo[845] userspec matched @ > /etc/sudoers.d/ceph-smartctl:4:57: allowed @ sudoers_lookup_check() > ./parse.c:167 > Dec 12 10:30:35 sudo[845] sudo_putenv: SUDO_COMMAND=/usr/sbin/smartctl -x > --json=o /dev/nvme5n1 > Dec 12 10:30:35 sudo[845] <- new_logline @ ./eventlog.c:218 := PAM account > management error: Authentication service cannot retrieve authentication > info ; TTY=pts/8 ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x > --json=o /dev/nvme5n1 > > > My only guess is that I have chosen a too-new Ubuntu version for Ceph and I > should just bite the bullet and re-install with Ubuntu 22.04, but if anyone > has a better idea, please let me know. > > -- > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 13 Dec '25

13 Dec '25

On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > This test branch was built loner then 2.5 weeks ago > I have to rebuild it first and then will schedule krbd krbd approved based on https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… Thanks, Ilya > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users <ceph-users(a)ceph.io> wrote: >> >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: >> > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >> > <ceph-users(a)ceph.io> wrote: >> > > >> > > Hi Yuri, >> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> > > > >> > > > Note: this is an attempt to test and publish post EOL reef point release >> > > > >> > > > Details of this release are summarized here: >> > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >> > > > Release Notes - TBD >> > > > LRC upgrade - N/A >> > > > >> > > > distro => jammy focal centos9 windows bookworm >> > > > >> > > > Seeking approvals/reviews for: >> > > > >> > > > smoke - Laura approved? >> > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? >> > > > (rados is also running on build 2 == build 1 + >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) >> > > > >> > > > rgw - Adam E approved? >> > > > >> > > > fs - Venky approved? >> > > >> > > There are lot many failures in the fs suite resembling >> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs >> > > type, bad option ... >> > > >> > > which is an issue with the testing kernel. Adding Alex who was looking >> > > into this issue. >> > > >> > > As far as the run validation is concerned, it's risky to ignore the >> > > large number of failures. We have two options: >> > > >> > > - fix the testing kernel issue and rerun >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >> > > teuthology-suite schedule command) >> > > >> > > The latter would be using an older kernel branch, but I think it >> > > should suffice for this case. >> > >> > Hi Venky, >> > >> > With your report of the fs suite being affected as well, I carved out >> > some time to take a look. >> > >> > I think it's happening because XFS maintainers disabled support for V4 >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's >> > no longer compiled in by default. The proper fix would be to create an >> > image with/move to V5 on-disk format; I have opened [1] to try to >> > achieve that. We can also re-enable V4 on-disk format in our kernel >> > builds as a workaround. >> > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >> >> This should be fixed now. Yuri, please schedule a rerun for krbd. >> >> Thanks, >> >> Ilya >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Re: v20.2.0 Tentacle intentional released but no build for el10?
by Gregory Farnum 12 Dec '25

12 Dec '25

el10 upgrades the python release and ceph-mgr DOES NOT WORK past I believe Python 3.12 due to ABI changes. (ceph-mgr makes extensive use of submodules, which has always been a bit iffy, and cpython upstream did a big rework of them which should be more supportable long-term but changed the implementation surface in the meantime.) Work is in progress to resolve and we will release el10 packages once that happens. -Greg On Fri, Dec 12, 2025 at 3:19 AM Moritz Baumann via ceph-users < ceph-users(a)ceph.io> wrote: > Hi > > is there a reason that there is no el10 under: > > https://download.ceph.com/rpm-20.2.0/ > https://download.ceph.com/rpm-tentacle/ > > > building with mock for Alma10.1/Rocky10.1 just works fine: > > curl -O -L > https://download.ceph.com/rpm-tentacle/el9/SRPMS/ceph-20.2.0-0.el9.src.rpm > mkdir rpm > mock -r /etc/mock/alma+epel-10-x86_64.cfg rebuild > ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm > # or > mock -r /etc/mock/rocky+epel-10-x86_64.cfg rebuild > ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm > > both work fine. > > Best > Moritz > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

v20.2.0 Tentacle intentional released but no build for el10?
by Moritz Baumann 12 Dec '25

12 Dec '25

Hi is there a reason that there is no el10 under: https://download.ceph.com/rpm-20.2.0/ https://download.ceph.com/rpm-tentacle/ building with mock for Alma10.1/Rocky10.1 just works fine: curl -O -L https://download.ceph.com/rpm-tentacle/el9/SRPMS/ceph-20.2.0-0.el9.src.rpm mkdir rpm mock -r /etc/mock/alma+epel-10-x86_64.cfg rebuild ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm # or mock -r /etc/mock/rocky+epel-10-x86_64.cfg rebuild ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm both work fine. Best Moritz

1 0

scrape-health-metrics fails with sudo PAM account management error
by Flemming Frandsen 12 Dec '25

12 Dec '25

I've been building a new cluster with cephadm, the OS is Ubuntu 24.04 and I'm using the ubuntu provided host packages, docker is 29.1.2 and containerd is 2.2.0 and the ceph release is squid 19.2.3. Everything seems to work just perfectly, except for scrape-health-metrics, which records this result (for all of the osds in the 4 hosts in the cluster): "20251212-074916": { "dev": "/dev/nvme5n1", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "samsung", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" }, Digging though the source I found that it's the OSD container that runs the command: sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 I can run the command in the container using: docker exec -it -u ceph ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-3 sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 The result of the manual command is: sudo: PAM account management error: Authentication service cannot retrieve authentication info sudo: a password is required I have enabled debugging for sudo, in the container and verified that the command that the osd is running, is same one and that the error is the same as well, running ceph device scrape-health-metrics causes this sudo debug output: Dec 12 10:30:35 sudo[845] user command "/usr/sbin/smartctl -x --json=o /dev/nvme5n1" matches sudoers command "/usr/sbin/smartctl -x --json=o /dev/*": true @ command_matches() ./match_command.c:667 Dec 12 10:30:35 sudo[845] userspec matched @ /etc/sudoers.d/ceph-smartctl:4:57: allowed @ sudoers_lookup_check() ./parse.c:167 Dec 12 10:30:35 sudo[845] sudo_putenv: SUDO_COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme5n1 Dec 12 10:30:35 sudo[845] <- new_logline @ ./eventlog.c:218 := PAM account management error: Authentication service cannot retrieve authentication info ; TTY=pts/8 ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme5n1 My only guess is that I have chosen a too-new Ubuntu version for Ceph and I should just bite the bullet and re-install with Ubuntu 22.04, but if anyone has a better idea, please let me know. -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

1 0

Mechanism behind the fix for EC Data Corruption (min_size=k) in Tentacle v20.2.0
by 유찬우 12 Dec '25

12 Dec '25

Hi Ceph Community, I am writing to inquire about a specific behavior change regarding data consistency in Erasure Coded pools between Ceph Tentacle v20.1.0 and v20.2.0. I previously encountered a deterministic data corruption issue during recovery in v20.1.0. After upgrading to v20.2.0 and repeating the same test, the issue appears to be resolved. I am interested in understanding the technical logic behind this fix. Here are the details of my test scenario: Cluster Configuration Version: Tested on Tentacle v20.1.0 (Issue observed) vs. v20.2.0 (Issue resolved) Layout: 8 OSD hosts EC Profile: k=6, m=2 (8 total shards) min_size: Manually lowered from 7 to 6 (k) Test Scenario (Reproduction Steps) 1. Client Setup: Created RBD images, formatted with ext4, and mounted them on 3 separate clients (1 TB image per client). 2. I/O Start: Started writing data patterns to the mount points until the mounted filesystems were filled to capacity. 3. Failure Injection: While writing, I forcibly brought down 2 OSD hosts. 4. Write Completion: Since min_size=6, writes continued and succeeded in a degraded state (Acting set size = 6). 5. Pre-Recovery Verification: Before bringing the nodes back up, I verified the data (MD5 checksum). - Result: Data was intact/readable (likely reading the 6 available systematic chunks). 6. Recovery: I brought the 2 down OSD hosts back up. The cluster performed recovery/peering and reached Active+Clean. 7. Post-Recovery Verification: After the cluster became Clean, I performed the integrity check again   The Observation On v20.1.0 MD5 Mismatch. The data became corrupted after recovery. This happened with 100/100 files tested. On v20.2.0 No data loss or corruption. MD5 checksums matched perfectly after recovery. It is great to see that v20.2.0 has resolved this critical issue. I am curious to learn what caused the corruption in the earlier version and how the recovery logic has been improved to handle this scenario correctly. Could anyone share insights into the root cause or the nature of the fix? If there are specific Pull Requests or Issue tracker links available for reference, I would greatly appreciate it. Thank you for your time and assistance. Best regards, Chanu Yu

1 0

Re: rgw index auto resharding not acting
by Anthony D'Atri 11 Dec '25

11 Dec '25

I don't recall where it is in the pipeline, but AIUI this has been recognized and there has been recent or incipient code work for resharding to account for versioned objects (and to reshard without blocking the bucket). > On Dec 11, 2025, at 5:12 AM, Johan Thomsen via ceph-users <ceph-users(a)ceph.io> wrote: > > I looks like "limit check" counts _all_ objects, but the resharder only > uses stats from rgw.main. Is that correct? > > If so, why? Shouldn't the auto-resharding act on the total number of > objects in the bucket? > > I have around 5 million objects in rgw.none and 3 mil in rgw.main... limit > check reports 8-9 million objects. > > (ceph version 19.2.3)

2 2

Re: [mgmt-gateway] monitoring problems, origin not allowed in grafana
by Redouane Kachach 11 Dec '25

11 Dec '25

Hi Gilles, I was able to reproduce a similar problem locally (the error related to "origin not allowed). As of my local testing (and observations) Grafana was failing with “origin not allowed” only when user login was enabled (user/password), but everything worked fine when using anonymous access. I think this difference is due to how Grafana’s CSRF protection works: With anonymous access: - You are effectively a non-authenticated user. - Most requests are simple GETs and there is no session cookie to protect. - Grafana’s CSRF checks are much less strict, so it doesn’t really care about the Origin header. With user login enabled: - After logging in, the browser sends requests with a session cookie and an Origin header like: Origin: https://<your-fqdn-domain> - Because this is now an authenticated session, Grafana enforces strict CSRF checks. - If the Origin does not exactly match what Grafana considers its own URL or trusted origins (Grafana is configured with an internal hostname, while you access it via the mgmt-gateway FQDN), Grafana rejects the request with: “origin not allowed”. I think that explains why the problem only appears once user authentication is enabled. The problem is that Grafana is deployed in this case behind mgmt-gateway (which acts as reverse-proxy). So Grafana has no idea what's the fqdn in use as it comes from the mgmt-gateway. When oauth2-proxy is also used so authentication goes through SSO this issue doesn't happen. To solve the problem, grafana provides a config field csrf_trusted_origins which can be used to indicate manually your trusted origins. In this case, you would need to add something like: [security] csrf_trusted_origins = <your-fqdn-domain> For example: [security] csrf_trusted_origins = my-cloud.ceph-node-0.com In your grafana.ini (located at /var/lib/ceph/<fsid>/grafana.<your-grafana-node>/etc/grafana/grafana.ini To test this quickly you can edit .ini manually and then run "ceph orch restart grafana" from the cephadm shell. This change is unfortunately not permanent, if you redeploy your grafana or move it to another node, you will lose the change. To make it permanent you have to adjust the grafana Jinja template, used by cephadm to generate the ini file. For that you have to modify the template to add the above entry and store it for example in a grafana.ini.j2 file then run the following commands: ceph config-key set mgr/cephadm/services/grafana/grafana.ini -i grafana.ini.j2 ceph orch reconfig grafana To get the template you can either find it in github, for tentacle (plz adjust the path to your exact release): https://github.com/ceph/ceph/blob/tentacle/src/pybind/mgr/cephadm/templates… or directly from the ceph-mgr container: /usr/share/ceph/mgr/cephadm/templates/services/grafana/grafana.ini.j2 Best, Redouane. On Thu, Dec 11, 2025 at 9:27 AM Redouane Kachach via ceph-users < ceph-users(a)ceph.io> wrote: > Thank you Gilles for updating the ticket with the info, > > One last thing that could shed some light is to see the cert used by > Prometheus (then let's continue the discussion on the tracker), > > openssl x509 -in > /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text > > (Plz omit any sensitive information for your lab from the output) > > BTW: there's a Slack channel dedicated for cephadm that can be used for a > more interactive dicussion > https://ceph-storage.slack.com/archives/C04SNUBD2M6 > > Best, > Redouane. > > > > On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < > gilles.mocellin(a)nuagelibre.org> wrote: > > > Thank you Redouane, > > > > I have updated the ticket with all what you made me do and what you > > asked. > > Except that "--include-cephadm-signed" does not exist on my version for > > `ceph orch certmgr cert ls`. > > > > https://tracker.ceph.com/issues/74164 > > > > Le 2025-12-10 15:36, Redouane Kachach a écrit : > > > Thanks for the feedback Gilles. > > > > > > Using your own cert for the mgmt-gateway is Okay and shouldn't be an > > > issue. > > > Errors arise when you use custom certs for the Dashboard itself as > > > bcz when the mgmt-gateway is enabled mTLS is enforced for internal > > > communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord > > > i.e) > > > > > > From what you are commenting it seems like Grafana is not able to get > > > metrics from Prometheus. > > > > > > Can you plz update the ticket with the output of the command (obviously > > > omitting any private/sensitive data): > > > > > >> ceph orch certmgr cert ls --include-cephadm-signed > > > > > > This would show all the certificates in use for the different services. > > > > > > It may also help see the config of Grafana: > > > > > > cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini > > > cat > > > > > > /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml > > > > > > > > > Thanks, > > > Redouane. > > > > > > > > > > > > On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < > > > gilles.mocellin(a)nuagelibre.org> wrote: > > > > > >> Better but not completely. > > >> > > >> I don't have errors anymore concerning alertmanager, and I can see > > >> alerts, thereis on concerning mgr/prometheus module not responding. > > >> Even if I disable/enable it. > > >> > > >> In the MGR logs, I see certificate problem : > > >> > > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 > > >> > > > ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: > > >> > > >> [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, > > >> 0, > > >> 0) lost — peer dropped the TLS connection suddenly, during handshake: > > >> (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > > >> (_ssl.c:1147)') > > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus > > >> INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client > > >> ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS > > >> connection suddenly, during handshake: (1, '[SSL: > > >> SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > > >> (_ssl.c:1147)') > > >> > > >> Grafana Dashboard still don't work. > > >> In embeded mode, I still have Unauthorized messages. > > >> If I open directly Grafana, with the DNS name, I can list the > > >> Dashboard, > > >> but in them, there is still "origin not allowed" messages. > > >> I cannot log in, with also the message "origin not allowed". > > >> > > >> > > >> Le 2025-12-10 14:15, Gilles Mocellin a écrit : > > >> > Hi Redouane, > > >> > > > >> > Yes, I'm using my own certificates, signed by our internal PKI. > > >> > > > >> > I will try, but I still want a valid certificate for my DNS entry, > > >> > certificate guven to the mgmt-gateway service. > > >> > > > >> > Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach > > >> > <rkachach(a)redhat.com> a écrit : > > >> >> Hi Gilles, > > >> >> > > >> >> What certificate are you using on your Dashboard? > > >> >> > > >> >> Just to rule out any certificate issues: when using mgmt-gateway, > the > > >> >> dashboard must be configured to use cephadm-signed certificates. > > >> >> This is normally done automatically, but if you were using custom > > >> >> certificates for your dashboard, those will not be overwritten by > > >> >> cephadm. > > >> >> If this is the case then plz just follow this procedure to fix > them: > > >> >> > > >> >> 1) generate cephadm-signed certs for the dashboard (from cephadm > > >> >> shell): > > >> >> > > >> >> ceph orch certmgr generate-certificates dashboard \ > > >> >> | tee \ > > >> >> >(jq -r '.cert' > dashboard.cert.pem) \ > > >> >> >(jq -r '.key' > dashboard.key.pem) \ > > >> >> > /dev/null > > >> >> > > >> >> 2) The above will generate two files: dashboard.cert.pem and > > >> >> dashboard.key.pem, now we have to use them to set the dashboard > > >> >> cert/key: > > >> >> > > >> >> ceph dashboard set-ssl-certificate -i dashboard.cert.pem > > >> >> ceph dashboard set-ssl-certificate-key -i dashboard.key.pem > > >> >> > > >> >> 3) Reload the dashboard, so it starts using the new populated > > >> >> certificates: > > >> >> > > >> >> ceph mgr module disable dashboard; ceph mgr module enable > dashboard > > >> >> --force > > >> >> > > >> >> Best, > > >> >> Redouane. > > >> >> > > >> >> > > >> >> On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < > > >> >> ceph-users(a)ceph.io> wrote: > > >> >> > > >> >>> Hello ! > > >> >>> > > >> >>> Does someone manage to use mgmt-gateway, with a FQDN, and the > > >> >>> monitoring > > >> >>> stack working ? > > >> >>> > > >> >>> In my LAB, after upgrading from 10.2.3 to 20.2.0, > > >> >>> My dashboard was working fine, with grafana dashboards. > > >> >>> > > >> >>> I decided to test mgmt-gateway. > > >> >>> I use a VIP with a specific DNS name pointing at that VIP. > > >> >>> I have a certificate generated with our internal PKI, valid for > the > > >> >>> DNS > > >> >>> name and the VIP. > > >> >>> > > >> >>> Dashboard works, but no monitoring anymore. > > >> >>> - Popups concerning alermanager : > > >> >>> > > >> >>> Failed to execute prometheus > > >> >>> Error parsing Prometheus Alertmanager response: Expecting value > > >> >>> 9/12/25 04:07 PM > > >> >>> > > >> >>> - No graphs in dashboard, with errors "origin not allowed" in each > > >> >>> panel > > >> >>> > > >> >>> If I open Grafana directly, I see : > > >> >>> - that the VIP is used, not my DNS name, but none works. > > >> >>> - Grafana works but I can't see dashboards, and if I put a URL > that > > >> >>> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I > > have a > > >> >>> message : > > >> >>> > > >> >>> Failed to load dashboard > > >> >>> Unauthorized > > >> >>> > > >> >>> Some more infos : > > >> >>> > > >> >>> https://$DNS/grafana/dashboards => show the list of Dashboards > > >> >>> Open a dashboard show errors 'origin not allowed" on panels > > >> >>> > > >> >>> https://$VIP/grafana/dashboards => cannot list the Dashboards, > > error > > >> >>> Unauthorized > > >> >>> > > >> >>> Concerning the Altermanger errors, I see thoses XHR calls in my > > >> >>> browser, > > >> >>> ending in HTTP error 400 Bad Request : > > >> >>> https://$DNS/api/prometheus?cluster_filter=true > > >> >>> > > >> >>> I opened an issue here : https://tracker.ceph.com/issues/74164 > > >> >>> > > >> >>> -- > > >> >>> Gilles > > >> >>> _______________________________________________ > > >> >>> ceph-users mailing list -- ceph-users(a)ceph.io > > >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >>> > > >> >>> > > >> > > >> > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1