- ceph-users - lists.ceph.io

Re: Version compatibility with QEMU/KVM
by Anthony D'Atri 11 Dec '25

11 Dec '25

The clients being post-Luminous is the biggest factor here, so they can handle pg-umap. > On Dec 10, 2025, at 6:39 AM, Kees Meijs | Nefos via ceph-users <ceph-users(a)ceph.io> wrote: > >> And as far as ceph stuff goes, RBD is (when used with mostly defaults) >> is quite simple and the protocol or stored data isn't changing much >> between releases. It does develop new features like being able to be >> stored on EC pools, fast diffs and things like that, but the basic >> read and write operations are quite the same.

2 1

Re: rgw index auto resharding not acting
by Janne Johansson 11 Dec '25

11 Dec '25

> I looks like "limit check" counts _all_ objects, but the resharder only > uses stats from rgw.main. Is that correct? > > If so, why? Shouldn't the auto-resharding act on the total number of > objects in the bucket? > > I have around 5 million objects in rgw.none and 3 mil in rgw.main... limit > check reports 8-9 million objects. In the older days, ceph sometimes got confused an thought there were 18 billion million objects belonging to the category rgw.none, and based its reshard decisions on that, which also wasn't so good. -- May the most significant bit of your life be positive.

2 1

Re: Changing crush map and editing crush rules to match
by Torkil Svensgaard 11 Dec '25

11 Dec '25

On 11/12/2025 12:27, Torkil Svensgaard via ceph-users wrote: > Hi > > Ceph squid. We currently have all our ceph servers at one physical > location but in different rooms. We are looking at creating a new pool > with some servers placed at a different physical location. > > Current crush map is this: > > root default > datacenter 714 > hosts > datacenter HX1 > hosts > datacenter UX1 > hosts > > Sample rule: > > { > "rule_id": 0, > "rule_name": "rbd_ec_data", > "type": 3, > "steps": [ > { > "op": "set_chooseleaf_tries", > "num": 5 > }, > { > "op": "set_choose_tries", > "num": 100 > }, > { > "op": "take", > "item": -24, > "item_name": "default~hdd" > }, > { > "op": "choose_indep", > "num": 0, > "type": "datacenter" > }, > { > "op": "chooseleaf_indep", > "num": 2, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > > New crush map: > > root default > datacenter HVH > room 714 > hosts > room HX1 > hosts > room UX1 > hosts > datacenter TBA > hosts > > Edited rule, changed root from default to HVH and failure domain from > datacenter to room: Not failure domain, just changed datacenter to room for the relevant step. Mvh. Torkil > > { > "rule_id": 0, > "rule_name": "rbd_ec_data", > "type": 3, > "steps": [ > { > "op": "set_chooseleaf_tries", > "num": 5 > }, > { > "op": "set_choose_tries", > "num": 100 > }, > { > "op": "take", > "item": -24, > "item_name": "HVH~hdd" > }, > { > "op": "choose_indep", > "num": 0, > "type": "room" > }, > { > "op": "chooseleaf_indep", > "num": 2, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > > Does that look right? Can we just set norebalance, move hosts around in > the crush map, edit our crush rules, run pgremapper and unset norebalance? > > Mvh. > > Torkil > -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 0

Changing crush map and editing crush rules to match
by Torkil Svensgaard 11 Dec '25

11 Dec '25

Hi Ceph squid. We currently have all our ceph servers at one physical location but in different rooms. We are looking at creating a new pool with some servers placed at a different physical location. Current crush map is this: root default datacenter 714 hosts datacenter HX1 hosts datacenter UX1 hosts Sample rule: { "rule_id": 0, "rule_name": "rbd_ec_data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -24, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 0, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 2, "type": "host" }, { "op": "emit" } ] }, New crush map: root default datacenter HVH room 714 hosts room HX1 hosts room UX1 hosts datacenter TBA hosts Edited rule, changed root from default to HVH and failure domain from datacenter to room: { "rule_id": 0, "rule_name": "rbd_ec_data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -24, "item_name": "HVH~hdd" }, { "op": "choose_indep", "num": 0, "type": "room" }, { "op": "chooseleaf_indep", "num": 2, "type": "host" }, { "op": "emit" } ] }, Does that look right? Can we just set norebalance, move hosts around in the crush map, edit our crush rules, run pgremapper and unset norebalance? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 0

Re: Tentacle 20.2.0 NFS ganesh and mgmt-gateway - The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
by Salatiel Filho 11 Dec '25

11 Dec '25

Does anyone know where I can get the Dockerfile ceph uses to create those containers for amd64 so I can try to build them for arm ? Atenciosamente/Kind regards, Salatiel On Fri, Dec 5, 2025 at 8:07 PM Salatiel Filho via ceph-users <ceph-users(a)ceph.io> wrote: > > Are there any plans to make a linux/arm64/v8 container image for > quay.io/ceph/keepalived:2.2.4, quay.io/ceph/haproxy:2.3 and > quay.io/ceph/nginx:sclorg-nginx-126 ? > Currently it is not possible to start a HA NFS Ganesh in ceph or even > the management gateway on ARM devices. > They all complain about: > WARNING: The requested image's platform (linux/amd64) does not match > the detected host platform (linux/arm64/v8) and no specific platform > was requested > > Atenciosamente/Kind regards, > Salatiel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw index auto resharding not acting
by Johan Thomsen 11 Dec '25

11 Dec '25

I looks like "limit check" counts _all_ objects, but the resharder only uses stats from rgw.main. Is that correct? If so, why? Shouldn't the auto-resharding act on the total number of objects in the bucket? I have around 5 million objects in rgw.none and 3 mil in rgw.main... limit check reports 8-9 million objects. (ceph version 19.2.3) Den man. 8. dec. 2025 kl. 10.10 skrev Johan Thomsen via ceph-users < ceph-users(a)ceph.io>: > Hi > > For some reason, the auto-resharding is not doing anything, even though it > reports fill status over 100%. See below: > > $ radosgw-admin bucket limit check --rgw-zonegroup=k1-objectstore > --rgw-zone=k1-objectstore > > > 2025-12-08T08:45:57.143+0000 7f8921528900 0 ERROR: current period > 43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not contain zone id > 1da0f602-9397-4997-8e76-3e3425468746 > > 2025-12-08T08:45:57.166+0000 7f8921528900 0 period > (43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not have zone > 1da0f602-9397-4997-8e76-3e3425468746 configured > > [ > > { > > "user_id": "k1-ceph-user", > > "buckets": [ > > { > > "bucket": "k1-snapshots", > > "tenant": "", > > "num_objects": 5766564, > > "num_shards": 97, > > "objects_per_shard": 59449, > > "fill_status": "OVER 114%" > > } > > ] > > } > > > Here's a dump of related config: > > > rgw_dynamic_resharding true > > rgw_max_dynamic_shards 1999 > > rgw_max_objs_per_shard 50000 > > rgw_md_log_max_shards 64 > > rgw_objexp_hints_num_shards 127 > > rgw_override_bucket_index_max_shards 1031 > > rgw_reshard_batch_size 64 > > rgw_reshard_bucket_lock_duration 360 > > rgw_reshard_max_aio 128 > > rgw_reshard_num_logs 16 > > rgw_reshard_thread_interval 600 > > rgw_safe_max_objects_per_shard 52000 > > rgw_shard_warning_threshold 90.000000 > > rgw_usage_max_shards 32 > > > Does anyone have an idea why it is not auto-resharding to a value below at > least 52k objs per shard? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: [mgmt-gateway] monitoring problems, origin not allowed in grafana
by Redouane Kachach 11 Dec '25

11 Dec '25

Thank you Gilles for updating the ticket with the info, One last thing that could shed some light is to see the cert used by Prometheus (then let's continue the discussion on the tracker), openssl x509 -in /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text (Plz omit any sensitive information for your lab from the output) BTW: there's a Slack channel dedicated for cephadm that can be used for a more interactive dicussion https://ceph-storage.slack.com/archives/C04SNUBD2M6 Best, Redouane. On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < gilles.mocellin(a)nuagelibre.org> wrote: > Thank you Redouane, > > I have updated the ticket with all what you made me do and what you > asked. > Except that "--include-cephadm-signed" does not exist on my version for > `ceph orch certmgr cert ls`. > > https://tracker.ceph.com/issues/74164 > > Le 2025-12-10 15:36, Redouane Kachach a écrit : > > Thanks for the feedback Gilles. > > > > Using your own cert for the mgmt-gateway is Okay and shouldn't be an > > issue. > > Errors arise when you use custom certs for the Dashboard itself as > > bcz when the mgmt-gateway is enabled mTLS is enforced for internal > > communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord > > i.e) > > > > From what you are commenting it seems like Grafana is not able to get > > metrics from Prometheus. > > > > Can you plz update the ticket with the output of the command (obviously > > omitting any private/sensitive data): > > > >> ceph orch certmgr cert ls --include-cephadm-signed > > > > This would show all the certificates in use for the different services. > > > > It may also help see the config of Grafana: > > > > cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini > > cat > > > /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml > > > > > > Thanks, > > Redouane. > > > > > > > > On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < > > gilles.mocellin(a)nuagelibre.org> wrote: > > > >> Better but not completely. > >> > >> I don't have errors anymore concerning alertmanager, and I can see > >> alerts, thereis on concerning mgr/prometheus module not responding. > >> Even if I disable/enable it. > >> > >> In the MGR logs, I see certificate problem : > >> > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 > >> > ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: > >> > >> [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, > >> 0, > >> 0) lost — peer dropped the TLS connection suddenly, during handshake: > >> (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > >> (_ssl.c:1147)') > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus > >> INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client > >> ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS > >> connection suddenly, during handshake: (1, '[SSL: > >> SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > >> (_ssl.c:1147)') > >> > >> Grafana Dashboard still don't work. > >> In embeded mode, I still have Unauthorized messages. > >> If I open directly Grafana, with the DNS name, I can list the > >> Dashboard, > >> but in them, there is still "origin not allowed" messages. > >> I cannot log in, with also the message "origin not allowed". > >> > >> > >> Le 2025-12-10 14:15, Gilles Mocellin a écrit : > >> > Hi Redouane, > >> > > >> > Yes, I'm using my own certificates, signed by our internal PKI. > >> > > >> > I will try, but I still want a valid certificate for my DNS entry, > >> > certificate guven to the mgmt-gateway service. > >> > > >> > Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach > >> > <rkachach(a)redhat.com> a écrit : > >> >> Hi Gilles, > >> >> > >> >> What certificate are you using on your Dashboard? > >> >> > >> >> Just to rule out any certificate issues: when using mgmt-gateway, the > >> >> dashboard must be configured to use cephadm-signed certificates. > >> >> This is normally done automatically, but if you were using custom > >> >> certificates for your dashboard, those will not be overwritten by > >> >> cephadm. > >> >> If this is the case then plz just follow this procedure to fix them: > >> >> > >> >> 1) generate cephadm-signed certs for the dashboard (from cephadm > >> >> shell): > >> >> > >> >> ceph orch certmgr generate-certificates dashboard \ > >> >> | tee \ > >> >> >(jq -r '.cert' > dashboard.cert.pem) \ > >> >> >(jq -r '.key' > dashboard.key.pem) \ > >> >> > /dev/null > >> >> > >> >> 2) The above will generate two files: dashboard.cert.pem and > >> >> dashboard.key.pem, now we have to use them to set the dashboard > >> >> cert/key: > >> >> > >> >> ceph dashboard set-ssl-certificate -i dashboard.cert.pem > >> >> ceph dashboard set-ssl-certificate-key -i dashboard.key.pem > >> >> > >> >> 3) Reload the dashboard, so it starts using the new populated > >> >> certificates: > >> >> > >> >> ceph mgr module disable dashboard; ceph mgr module enable dashboard > >> >> --force > >> >> > >> >> Best, > >> >> Redouane. > >> >> > >> >> > >> >> On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < > >> >> ceph-users(a)ceph.io> wrote: > >> >> > >> >>> Hello ! > >> >>> > >> >>> Does someone manage to use mgmt-gateway, with a FQDN, and the > >> >>> monitoring > >> >>> stack working ? > >> >>> > >> >>> In my LAB, after upgrading from 10.2.3 to 20.2.0, > >> >>> My dashboard was working fine, with grafana dashboards. > >> >>> > >> >>> I decided to test mgmt-gateway. > >> >>> I use a VIP with a specific DNS name pointing at that VIP. > >> >>> I have a certificate generated with our internal PKI, valid for the > >> >>> DNS > >> >>> name and the VIP. > >> >>> > >> >>> Dashboard works, but no monitoring anymore. > >> >>> - Popups concerning alermanager : > >> >>> > >> >>> Failed to execute prometheus > >> >>> Error parsing Prometheus Alertmanager response: Expecting value > >> >>> 9/12/25 04:07 PM > >> >>> > >> >>> - No graphs in dashboard, with errors "origin not allowed" in each > >> >>> panel > >> >>> > >> >>> If I open Grafana directly, I see : > >> >>> - that the VIP is used, not my DNS name, but none works. > >> >>> - Grafana works but I can't see dashboards, and if I put a URL that > >> >>> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I > have a > >> >>> message : > >> >>> > >> >>> Failed to load dashboard > >> >>> Unauthorized > >> >>> > >> >>> Some more infos : > >> >>> > >> >>> https://$DNS/grafana/dashboards => show the list of Dashboards > >> >>> Open a dashboard show errors 'origin not allowed" on panels > >> >>> > >> >>> https://$VIP/grafana/dashboards => cannot list the Dashboards, > error > >> >>> Unauthorized > >> >>> > >> >>> Concerning the Altermanger errors, I see thoses XHR calls in my > >> >>> browser, > >> >>> ending in HTTP error 400 Bad Request : > >> >>> https://$DNS/api/prometheus?cluster_filter=true > >> >>> > >> >>> I opened an issue here : https://tracker.ceph.com/issues/74164 > >> >>> > >> >>> -- > >> >>> Gilles > >> >>> _______________________________________________ > >> >>> ceph-users mailing list -- ceph-users(a)ceph.io > >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> >>> > >> >>> > >> > >> > >

1 0

Re: Tentacle 20.2.0 NFS ganesh and mgmt-gateway - The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
by Salatiel Filho 10 Dec '25

10 Dec '25

Not sure if anyone is interested, but I created this issue https://tracker.ceph.com/issues/74180 Atenciosamente/Kind regards, Salatiel On Fri, Dec 5, 2025 at 8:07 PM Salatiel Filho via ceph-users <ceph-users(a)ceph.io> wrote: > > Are there any plans to make a linux/arm64/v8 container image for > quay.io/ceph/keepalived:2.2.4, quay.io/ceph/haproxy:2.3 and > quay.io/ceph/nginx:sclorg-nginx-126 ? > Currently it is not possible to start a HA NFS Ganesh in ceph or even > the management gateway on ARM devices. > They all complain about: > WARNING: The requested image's platform (linux/amd64) does not match > the detected host platform (linux/arm64/v8) and no specific platform > was requested > > Atenciosamente/Kind regards, > Salatiel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Tentacle and Ubuntu 24
by Mark Lehrer 10 Dec '25

10 Dec '25

>> Ubuntu 24 might become fully supported and tested by Tentacle (or Squid)? The current version in the Ubuntu 24.04 repos is Squid (19.2). On Wed, Dec 10, 2025 at 12:15 PM Ryan Sleeth via ceph-users <ceph-users(a)ceph.io> wrote: > > Does the community know when Ubuntu 24 might become fully supported and > tested by Tentacle (or Squid)? I am referring to the web documentation > here: https://docs.ceph.com/en/tentacle/start/os-recommendations/ > > -- > Ryan Sleeth > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle and Ubuntu 24
by Ryan Sleeth 10 Dec '25

10 Dec '25

Does the community know when Ubuntu 24 might become fully supported and tested by Tentacle (or Squid)? I am referring to the web documentation here: https://docs.ceph.com/en/tentacle/start/os-recommendations/ -- Ryan Sleeth

1 0