Hello,
I'm attempting to setup an OpenIDConnect provider with RGW. I'm doing this using the boto3 API & Python. However it seems that the APIs are failing in some unexpected ways because radosgw was not setup correctly. There is sample code below, and yes, I know there are "secrets" in it - but this is an offline test lab so I am fine with this.
The first error shows this in the logs.
2023-02-16T00:45:26.860-0500 7fe19fef7700 1 ====== starting new request req=0x7fe2ccb54680 =====
2023-02-16T00:45:26.904-0500 7fe19def3700 0 req 17562030806519127926 0.044000439s ERROR: listing filtered objects failed: OIDC pool: default.rgw.meta: oidc_url.: (2) No such file or directory
2023-02-16T00:45:26.904-0500 7fe19aeed700 1 ====== req done req=0x7fe2ccb54680 op status=-2 http_status=404 latency=0.044000439s ======
2023-02-16T00:45:26.904-0500 7fe19aeed700 1 beast: 0x7fe2ccb54680: 10.20.104.178 - authentik [16/Feb/2023:00:45:26.860 -0500] "POST / HTTP/1.1" 404 189 - "Boto3/1.26.71 Python/3.11.1 Linux/6.0.6-76060006-generic Botocore/1.29.72" - latency=0.044000439s
So the object "oidc_url" is missing from the "default.rgw.meta" pool?
rados --pool default.rgw.meta ls --all
users.uid root.buckets
users.uid authentik.buckets
root test4
root .bucket.meta.test2:3866fac0-854b-48b5-b3b7-bf84a166a404.1165645.1
users.keys ZVBTLTYRRPY7JU39WOR9
users.uid authentik
users.uid cephadmin
users.keys NIVIV0JSKD9D2LDC3IH4
users.uid root
users.email tester(a)lab.dev
users.keys L70QT3LN71SQXWHS97Y4
root .bucket.meta.test:3866fac0-854b-48b5-b3b7-bf84a166a404.1204730.1
root .bucket.meta.test4:3866fac0-854b-48b5-b3b7-bf84a166a404.1204730.2
root test
root test2
Well the object is clearly not there and I do not know how to fix this.
The second error produces this error in the log:
2023-02-16T01:11:29.304-0500 7fe1976e6700 1 ====== starting new request req=0x7fe2ccb54680 =====
2023-02-16T01:11:29.312-0500 7fe18c6d0700 1 ====== req done req=0x7fe2ccb54680 op status=-22 http_status=400 latency=0.008000083s ======
2023-02-16T01:11:29.312-0500 7fe18c6d0700 1 beast: 0x7fe2ccb54680: 10.20.104.178 - authentik [16/Feb/2023:01:11:29.304 -0500] "POST / HTTP/1.1" 400 189 - "Boto3/1.26.71 Python/3.11.1 Linux/6.0.6-76060006-generic Botocore/1.29.72" - latency=0.008000083s
Its much less clear what is going on here, it just returns 400. Boto raises this exception, "botocore.exceptions.ClientError: An error occurred (Unknown) when calling the CreateOpenIDConnectProvider operation: Unknown".
Has anyone seen this before and know how to setup the correct objects for OpenidConnect?
Version info
==============================================
ceph version 17.2.5 (e04241aa9b639588fa6c864845287d2824cb6b55) quincy (stable)
Examples below
==============================================
# creating the client works fine - I can see my user authenticate in the radosgw logs
access_key_id = 'L70QT3LN71SQXWHS97Y4'
secret_access_key = 'QEXLa5V0Zm38068n3goDtm8V6WlaDwxVmAq9W2XV'
iam = boto3.client('iam',
aws_access_key_id=access_key_id,
aws_secret_access_key=secret_access_key,
region_name="default",
endpoint_url="https://s3.lab")
# First error
providers_response = iam.list_open_id_connect_providers()
# Second Error
oidc_response = iam.create_open_id_connect_provider(
# Issuer URL
Url="https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/",
ClientIDList=['authentik'],
ThumbprintList=['BDCC44F40254E7E1258DA4698833FFE2E8AECA3D3799044D8A1F97F7DFF20511'])
Hi,
We have a discussion going on about which is the correct flag to use for some maintenance on an OSD, should it be "noout" or "norebalance"? This was sparked because we need to take an OSD out of service for a short while to upgrade the firmware.
One school of thought is:
- "ceph norebalance" prevents automatic rebalancing of data between OSDs, which Ceph does to ensure all OSDs have roughly the same amount of data.
- "ceph noout" on the other hand prevents OSDs from being marked as out-of-service during maintenance, which helps maintain cluster performance and availability.
- Additionally, if another OSD fails while the "norebalance" flag is set, the data redundancy and fault tolerance of the Ceph cluster may be compromised.
- So if we're going to maintain the performance and reliability we need to set the "ceph noout" flag to prevent the OSD from being marked as OOS during maintenance and allow the automatic data redistribution feature of Ceph to work as intended.
The other opinion is:
- With the noout flag set, Ceph clients are forced to think that OSD exists and is accessible - so they continue sending requests to such OSD. The OSD also remains in the crush map without any signs that it is actually out. If an additional OSD fails in the cluster with the noout flag set, Ceph is forced to continue thinking that this new failed OSD is OK. It leads to stalled or delayed response from the OSD side to clients.
- Norebalance instead takes into account the in/out OSD status, but prevents data rebalance. Clients are also aware of the real OSD status, so no requests go to the OSD which is actually out. If an additional OSD fails - only the required temporary PG are created to maintain at least 2 existing copies of the same data (well, generally it is set by the pool min size).
The upstream docs seem pretty clear that noout should be used for maintenance (https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-osd/), but the second opinion strongly suggests that norebalance is actually better and the Ceph docs are out of date.
So what is the feedback from the wider community?
Thanks,
Will
Hey guys,
most of my osds have HDD for block and SSD for db. But according to "ceph osd metadata" bluefs_db_type = hdd and bluefs_db_rotational = 1.
lsblk -o name, rota reveals the following (sdb is db device for 3 hdds):
sdb 0
├─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--b164ba4c--48c9--41a0--8b5e--ae3a6a23a22c 1
├─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--1c7aa9a1--791d--4ed6--8049--9fba8d5ac828 1
└─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--ec92f9c6--d651--46ed--b6cd--4cf37c8ce284 1
I am pretty sure sdb had rota 0 during osd deployment (still has, but lvm volumes don’t).
Question 1:
Is the output of osd metadata for bluefs_db_type and bluefs_db_rotational relevant for how the osd process is treating the disks? (or does it just reflect the value of /sys/block/<device>/queue/rotational?)
Question 2:
How can I verify that the osd process is treating the db device as an ssd? If i remember correctly, the osd process is using different parameters if db is on SSD instead of HDD.
Best regards
Felix
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
Hi
I am trying to setup the “High availability service for RGW” using SSL
both to the HAProxy and from the HAProxy to the RGW backend.
The SSL certificate gets applied to both HAProxy and the RGW. If I use
the RGW instances directly they work as expected.
The RGW config is as follows:
service_type: rgw
service_id: rgw
service_name: rgw.rgw
placement:
label: rgw
count_per_host: 2
spec:
ssl: true
rgw_frontend_port: 6443
rgw_frontend_ssl_certificate: |
-----BEGIN CERTIFICATE----
-----END PRIVATE KEY-----
Ingress as follows:
service_type: ingress
service_id: rgw.rgw
placement:
hosts:
- cephrgw01
- cephrgw02
- cephrgw03
spec:
backend_service: rgw.rgw
virtual_ip: 172.16.1.130/16
frontend_port: 443
monitor_port: 1967
ssl_cert: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
The issue is that the haproxy.cfg gets generated like this, without SSL
enabled on the backends:
# This file is generated by cephadm.
global
log127.0.0.1 local2
chroot/var/lib/haproxy
pidfile/var/lib/haproxy/haproxy.pid
maxconn8000
daemon
stats socket /var/lib/haproxy/stats
defaults
modehttp
logglobal
optionhttplog
optiondontlognull
option http-server-close
option forwardforexcept 127.0.0.0/8
optionredispatch
retries3
timeout queue20s
timeout connect5s
timeout http-request1s
timeout http-keep-alive 5s
timeout client1s
timeout server1s
timeout check5s
maxconn8000
frontend stats
mode http
bind 172.16.1.130:1967
bind localhost:1967
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:abcdefg
http-request use-service prometheus-exporter if { path /metrics }
monitor-uri /health
frontend frontend
bind 172.16.1.130:443 ssl crt /var/lib/haproxy/haproxy.pem
default_backend backend
backend backend
option forwardfor
balance static-rr
option httpchk HEAD / HTTP/1.0
server rgw.rgw.cephrgw01.euvqmd 172.16.1.131:6443 check weight 100
server rgw.rgw.cephrgw01.aphsnx 172.16.1.131:6444 check weight 100
server rgw.rgw.cephrgw02.ovckaw 172.16.1.132:6443 check weight 100
server rgw.rgw.cephrgw02.jevtrb 172.16.1.132:6444 check weight 100
server rgw.rgw.cephrgw03.gzdame 172.16.1.133:6443 check weight 100
server rgw.rgw.cephrgw03.bchspq 172.16.1.133:6444 check weight 100
This of course does not work as the backend use SSL.
Is there some configuration that I have missed or should I file a bug
report?
/Jimmy
Hi,
today our entire cluster froze. or anything that uses librbd to be specific.
ceph version 16.2.10
The message that saved me was "256 slow ops, oldest one blocked for
2893 sec, osd.7 has slow ops" , because it makes it immediately clear
that this osd is the issue.
I stopped the osd, which made the cluster available again. Restarting
the osd makes it stuck again, although that osd has nothing in the
error log, and the underlying ssd is healthy. It's just that one out
of 27. There's nothing unique about it. We use the same disk product
in other osds, and the host is also running other osds just fine.
How does this happen, and why can the cluster not recover from this
automatically? For example by stopping the affected osd or at least
having a timeout for ops.
Thanks
--
+4916093821054
Hi all,
today's topics were:
- Labs:
- Keeping a catalog
- Have a dedicated group to debug/work through the issues.
- Looking for interested parties that would like to contribute in the
lab maintenance tasks
- Poll for meeting time, looking for a central person to follow up /
organize
- No one's been actively coordinating on the lab issues apart from
Laura. David Orman volunteered if we need help coordinating the lab issues
- Reef release
- [casey] things aren't looking good for end-of-february freeze
- Since the whole thing depends on test-infra, can't really estimate
the time frame.
- The freeze maybe delayed
- Dev Summit in Amsterdam: estimate how many would attend in person,
remote
- 50/50 of those present would attend (as per the voting)
- Ad hoc virtual could work
- Need to update the component leads page:
https://ceph.io/en/community/team/
- Vikhyath volunteered before, so Josh will check with him.
Regards,
--
Nizamudeen A
Software Engineer
Red Hat <https://www.redhat.com/>
<https://www.redhat.com/>
Hi,
I have two Ceph clusters in a multi-zone setup. The first one (master zone) would be accessible to users for their interaction using RGW.
The second one is set to sync from the master zone with the tier type of the zone set as an archive (to version all files).
My question here is. Is there an option to set a lifecycle for the version files saved on the archive zone? For example, keep only 5 versions per file or delete version files older than one year?
Thanks a lot.
Hi, Experts,
we already have a CephFS cluster, called A, and now we want to setup another CephFS cluster(called B) in other site.
And we need to synchronize data with each other for some directory(if all directory can synchronize , then very very good), Means when we write a file in A cluster, this file can auto sync to B cluster, and when we create a file or directory on B Cluster, this file or directory can auto sync to A Cluster.
our question is does there any best practices to do that on CephFS?
Thanks in advance!
Thanks,
zx