- ceph-users - lists.ceph.io

Ceph Day Seattle 2026: Still time to register and attend!
by Anthony D'Atri 26 May '26

26 May '26

Ceph Day Seattle 2026 happens this Thursday May 28. https://ceph.io/en/community/events/2026/ceph-days-seattle/ Register and attend for all the latest squiggly goodness. Proxmox-fu! CephFS-fu! RGW-fu! Anthony-Bob sez check it out. -- Anthony D'Atri, Ceph Ambassador and Master of Tentacles

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 26 May '26

26 May '26

Hi Eugen! On 23/05/26 23:54, Eugen Block via ceph-users wrote: > Hi, > > I might be wrong, but I don't think it's a cert issue. If you look in > the mgr log, do you see more information that just the > SignatureDoesNotMatch error? On further investigations I'm also convinced that is nothing to do with the certificates but more about the permissions of the 'dashboard' user. This is the exception that is thrown from the mgr logs: > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-01.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-03.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-02.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-04.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-05.domain.com, port=4443, ssl=True > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.005s] [admin] [308.0B] /api/health/snapshot > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.007s] [admin] [22.0B] /api/prometheus/alertgroup > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.012s] [admin] [376.0B] /api/multi-cluster/get_config > [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] [0.028s] [admin] [1.2K] /api/summary > [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 > [dashboard ERROR rgw_client] RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') > Traceback (most recent call last): > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 413, in __init__ > self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \ > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in func_wrapper > return func( > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 448, in _get_user_id > response = request() > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in __call__ > resp = self.rest_client.do_request(method, self._gen_path(), params, > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in do_request > raise RequestException( > dashboard.rest_client.RequestException: RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') > > Could it be mismatching dashboard-rgw-api settings? Have you checked > these settings? > > ceph dashboard get-rgw-api-admin-resource It's the default, 'admin'. I also reset the value with "reset-rgw-api-admin-resource", so technically now is 'unset'. Question what should be the value if not 'admin'? > > ceph dashboard get-rgw-api-access-key > > ceph dashboard get-rgw-api-secret-key > > They should match with: > > radosgw-admin user info --uid dashboard | jq -r '.keys' The values are matching, I'see in both get-rgw-api-secret-key and get-rgw-api-access-key something like this: {'<REALM>': '<(ACCESS}SECRET)_KEY>'} > > But as I wrote, it might something else, I would expect the mgr log to > contain more details. > Indeed, I'll tried to backtrace the error in the code and I see that is failing in the rgw_client.py when is trying to get the user_id "of the user that is used to communicate with the RGW Admin Ops API". I suppose that is the 'admin' user from rgw-api-admin-resource, am I right? I didn't had time till now to investigate further. Thanks! Iztok

1 2

Re: Looking for recommendations on Ceph node specifications
by Tony Liu 26 May '26

26 May '26

It depends on how RGW pools are configured. https://docs.ceph.com/en/squid/radosgw/pools/ There are two layers, OSD and pool. OSD can be 1) pure HDD, 2) pure SSD, 3) HDD with db on SSD. By setting crush rule, you may choose which OSD is for which pool, like #1 for rgw.data and #2 for rgw.index. Tony ________________________________________ From: Adam Prycki via ceph-users <ceph-users(a)ceph.io> Sent: May 26, 2026 08:57 AM To: Tony Liu; Anthony D'Atri Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Looking for recommendations on Ceph node specifications In the past we have deployed HDD OSDs (without SSD block.db) and SSD OSDs for S3 cluster. It worked fine. From our understanding, on a ceph consisting of HDDs with SSD block.db most of RGW meta (index shards) will end up on SSD block.db. So, if we can afford it, HDD+block.db setup for S3 would be better for general performance and recovery speeds. Is this assumption correct? Best regards Adam Prycki On 19/05/2026 18:44, Tony Liu wrote: > For RBD on HDD, with DB on SSD (typically shared by multiple HDDs), getting better write performance, > but SSD will be worn out fast. For S3, index on SSD, data and DB on HDD, it will work fine. For density, > it will be fine normally, but in case of failure or maintenance, recovery on dense HDD will be very slow. > For EC, 8+3 is fine on HDD without performance requirement. We tried 6+3 on NVMe, CPU usage is very > high which affects performance. For networking, 2x50G or 2x100G is too much for HDD. 2x25G is sufficient. > > > Tony > ________________________________________ > From: Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> > Sent: May 19, 2026 08:45 AM > To: Adam Prycki > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Looking for recommendations on Ceph node specifications > >>> . >> >> At least 16 or 32 nodes >> in 16 racks. >> Erasure coding 8+3 with failure domain on rack level. >> > Ack. > > >> We initially selected 8+3 over 4+2 because we expect rebuilds to take very long with nodes this big and we don't want to loose redundancy > > Fair enough. You get more nines with m=3 for sure, though the wider profile itself will mean slower scrubs and recovery. I suspect you set mon_osd_down_out_subtree_limit? > >> >>>> >>>> >>>> Splitting JBOD logically into 2 servers isn't an issue for use because we will replicate data on rack level and not host level. >>>> >>>> >>>> Common specifications for all variants >>>> >>>> 5-6GB of RAM per 1 HDD >>> Plus more for mons and other daemons? Especially MDS? >> >> Other daemons will be on some dedicated non-storage servers. > > Ack. Had to ask. > >> We aim for low RAM/HDD on storage nodes. Other daemons won't fit there. >> >>>> 2% of HDD capacity in NVMe devices for block.db (or none) >>>> 2x 50Gb or 2x 100Gb Ethernet per server (active-backup bonded interfaces) >>>> (CPU per OSD to be determined) >>>> >>>> >>>> Variant A1 is very unlikely to happen but we are curious what network interface speeds would you suggest for so many HDDs in one node. >>> 100GE bonded at the least. Depends on your workload. >>>> >>>> Variant A2 is the most likely the one we will choose for large deployment. >>>> >>>> Variant B1/B2 for smaller deployments. >>>> >>>> Does anyone of you run ceph on similar setups? Did you find any pitfall with it? >>>> >>>> What are your minimal recommendations for network speed per HDD, cpu per HDD, etc? >>>> >>>> In our experience most of our servers, even in large clusters, never max out the network interfaces or CPUs. We almost never rebuild or rebalance whole servers. 27 HDD nodes of our biggest CephFS cluster with EC usually have only 2-3Gbps of network traffic. >>> Your workload is archival? >> >> Yes, mostly archival. >> We have big demand for S3 and CephFS. >> But we may move to pure s3 cluster in the future. >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle 20.2.2 QE validation status
by Afreen 26 May '26

26 May '26

Dashboard approved On Tue, May 26, 2026 at 10:00 PM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Resending because of emails problems > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/75878#note-6 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > nvmeof: Aviv, Vallari > > > > upgrade: Patrick > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Dashboard 'SignatureDoesNotMatch' error when connecting to RGW
by Iztok Gregori 26 May '26

26 May '26

Hi Eugen! On 23/05/26 23:54, Eugen Block via ceph-users wrote: > Hi, > > I might be wrong, but I don't think it's a cert issue. If you look in > the mgr log, do you see more information that just the > SignatureDoesNotMatch error? On further investigations I'm also convinced that is nothing to do with the certificates but more about the permissions of the 'dashboard' user. This is the exception that is thrown from the mgr logs: > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-01.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-03.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-02.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-04.domain.com, port=4443, ssl=True > [dashboard INFO rgw_client] Found RGW daemon with configuration: host=node-05.domain.com, port=4443, ssl=True > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.005s] [admin] [308.0B] /api/health/snapshot > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.007s] [admin] [22.0B] /api/prometheus/alertgroup > [dashboard INFO request] [::ffff:140.105.2.28:41732] [GET] [200] [0.012s] [admin] [376.0B] /api/multi-cluster/get_config > [dashboard INFO request] [::ffff:140.105.2.28:41760] [GET] [200] [0.028s] [admin] [1.2K] /api/summary > [dashboard ERROR rest_client] RGW REST API failed GET req status: 403 > [dashboard ERROR rgw_client] RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-eros","HostId":"308299-zone-zonegroup"}') > Traceback (most recent call last): > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 413, in __init__ > self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \ > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in func_wrapper > return func( > File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 448, in _get_user_id > response = request() > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in __call__ > resp = self.rest_client.do_request(method, self._gen_path(), params, > File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in do_request > raise RequestException( > dashboard.rest_client.RequestException: RGW REST API failed request with status code 403 > (b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000629cfb33d5e' > b'da4c7-006a141bf6-308299-zone","HostId":"308299-zone-zonegroup"}') > > Could it be mismatching dashboard-rgw-api settings? Have you checked > these settings? > > ceph dashboard get-rgw-api-admin-resource It's the default, 'admin'. I also reset the value with "reset-rgw-api-admin-resource", so technically now is 'unset'. Question what should be the value if not 'admin'? > > ceph dashboard get-rgw-api-access-key > > ceph dashboard get-rgw-api-secret-key > > They should match with: > > radosgw-admin user info --uid dashboard | jq -r '.keys' The values are matching, I'see in both get-rgw-api-secret-key and get-rgw-api-access-key something like this: {'<REALM>': '<(ACCESS}SECRET)_KEY>'} > > But as I wrote, it might something else, I would expect the mgr log to > contain more details. > Indeed, I'll tried to backtrace the error in the code and I see that is failing in the rgw_client.py when is trying to get the user_id "of the user that is used to communicate with the RGW Admin Ops API". I suppose that is the 'admin' user from rgw-api-admin-resource, am I right? I didn't had time till now to investigate further. Thanks! Iztok

1 0

otel, rgw -> bluestore
by C U 26 May '26

26 May '26

did some stuff with claude, maybe it's interesting if your thing is otel tracing rgw -> bluestore. https://github.com/tu503/warp-otel/blob/main/README.md regards,

1 0

CSC minutes 2026-05-25
by Venky Shankar 26 May '26

26 May '26

- Low turnout due to the US holiday; the agenda moved to the CSC meeting on 8th June. Radek discussed only one item (see below) - [Radek] Umbrella release update Blocker: The fix is in progress and currently being retested. See https://github.com/ceph/ceph/pull/66459 for details. -- Cheers, Venky

1 0

Re: Looking for recommendations on Ceph node specifications
by Tony Liu 25 May '26

25 May '26

For RBD on HDD, with DB on SSD (typically shared by multiple HDDs), getting better write performance, but SSD will be worn out fast. For S3, index on SSD, data and DB on HDD, it will work fine. For density, it will be fine normally, but in case of failure or maintenance, recovery on dense HDD will be very slow. For EC, 8+3 is fine on HDD without performance requirement. We tried 6+3 on NVMe, CPU usage is very high which affects performance. For networking, 2x50G or 2x100G is too much for HDD. 2x25G is sufficient. Tony ________________________________________ From: Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> Sent: May 19, 2026 08:45 AM To: Adam Prycki Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Looking for recommendations on Ceph node specifications >> . > > At least 16 or 32 nodes > in 16 racks. > Erasure coding 8+3 with failure domain on rack level. > Ack. > We initially selected 8+3 over 4+2 because we expect rebuilds to take very long with nodes this big and we don't want to loose redundancy Fair enough. You get more nines with m=3 for sure, though the wider profile itself will mean slower scrubs and recovery. I suspect you set mon_osd_down_out_subtree_limit? > >>> >>> >>> Splitting JBOD logically into 2 servers isn't an issue for use because we will replicate data on rack level and not host level. >>> >>> >>> Common specifications for all variants >>> >>> 5-6GB of RAM per 1 HDD >> Plus more for mons and other daemons? Especially MDS? > > Other daemons will be on some dedicated non-storage servers. Ack. Had to ask. > We aim for low RAM/HDD on storage nodes. Other daemons won't fit there. > >>> 2% of HDD capacity in NVMe devices for block.db (or none) >>> 2x 50Gb or 2x 100Gb Ethernet per server (active-backup bonded interfaces) >>> (CPU per OSD to be determined) >>> >>> >>> Variant A1 is very unlikely to happen but we are curious what network interface speeds would you suggest for so many HDDs in one node. >> 100GE bonded at the least. Depends on your workload. >>> >>> Variant A2 is the most likely the one we will choose for large deployment. >>> >>> Variant B1/B2 for smaller deployments. >>> >>> Does anyone of you run ceph on similar setups? Did you find any pitfall with it? >>> >>> What are your minimal recommendations for network speed per HDD, cpu per HDD, etc? >>> >>> In our experience most of our servers, even in large clusters, never max out the network interfaces or CPUs. We almost never rebuild or rebalance whole servers. 27 HDD nodes of our biggest CephFS cluster with EC usually have only 2-3Gbps of network traffic. >> Your workload is archival? > > Yes, mostly archival. > We have big demand for S3 and CephFS. > But we may move to pure s3 cluster in the future. > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Re: Looking for recommendations on Ceph node specifications
by Anthony D'Atri 25 May '26

25 May '26

> We have 4 potential variants for OSD nodes. > > Variant A1 > ~100 HDD JOBD + x86_64 server with block.db NVMe > > > Variant A2 > ~100 HDD JOBD + 2x x86_64 server with block.db NVMe > JBOD split in half, each node gets 50 HDDs > > > Variant B1 > ~60 HDD JOBD + x86_64 server with block.db NVMe > > > Variant B2 > ~60 HDD JOBD + 2x x86_64 server with block.db NVMe > JBOD split in half, each node gets 30 HDDs How big of a cluster are you planning? Nodes that dense can present various problems, such that of your variants I would reluctantly pick B2. > > > Splitting JBOD logically into 2 servers isn't an issue for use because we will replicate data on rack level and not host level. > > > Common specifications for all variants > > 5-6GB of RAM per 1 HDD Plus more for mons and other daemons? Especially MDS? > 2% of HDD capacity in NVMe devices for block.db (or none) > 2x 50Gb or 2x 100Gb Ethernet per server (active-backup bonded interfaces) > (CPU per OSD to be determined) > > > Variant A1 is very unlikely to happen but we are curious what network interface speeds would you suggest for so many HDDs in one node. 100GE bonded at the least. Depends on your workload. > > Variant A2 is the most likely the one we will choose for large deployment. > > Variant B1/B2 for smaller deployments. > > Does anyone of you run ceph on similar setups? Did you find any pitfall with it? > > What are your minimal recommendations for network speed per HDD, cpu per HDD, etc? > > In our experience most of our servers, even in large clusters, never max out the network interfaces or CPUs. We almost never rebuild or rebalance whole servers. 27 HDD nodes of our biggest CephFS cluster with EC usually have only 2-3Gbps of network traffic. Your workload is archival? > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 4

Ceph Day Seattle 2026: Still time to register and attend!
by Anthony D'Atri 25 May '26

25 May '26

Ceph Day Seattle 2026 happens this Thursday May 28. https://ceph.io/en/community/events/2026/ceph-days-seattle/ Register and attend for all the latest squiggly goodness. Proxmox-fu! CephFS-fu! RGW-fu! Anthony-Bob sez check it out. -- Anthony D'Atri, Ceph Ambassador and Master of Tentacles

1 0