quincy rgw with ceph orch and two realms only get answers from first realm
Hi, we are currently in the process of adopting the main s3 cluster to orchestrator. We have two realms (one for us and one for the customer). The old config worked fine and depending on the port I requested, I got different x-amz-request-id header back: x-amz-request-id: tx00000307170ac0d734ab4-0066508120-94aa0f66-eu-central-1 x-amz-request-id: tx00000a9d8fc1821bbe258-00665081d1-949b2447-eu-customer-1 I then deployed the orchestrator config to other systems and checked, but now I get always an answer from the eu-central-1 zone and never from the customer zone. The test is very simple and works fine for the old rgw config, but fails for the new orchestrator config: curl [IPv6]:7481/somebucket -v All rgw instances are running on ceph version 17.2.7. Configs attached below. The old ceph.conf looked like this: [client.eu-central-1-s3db1-old] rgw_frontends = beast endpoint=[::]:7480 rgw_region = eu rgw_zone = eu-central-1 rgw_dns_name = example.com rgw_dns_s3website_name = eu-central-1.example.com [client.eu-customer-1-s3db1] rgw_frontends = beast endpoint=[::]:7481 rgw_region = eu-customer rgw_zone = eu-customer-1 rgw_dns_name = s3.customer.domain rgw_dns_s3website_name = eu-central-1.s3.customer.domain And this is the new service.yaml service_type: rgw service_id: ab12 placement: label: rgw config: debug_rgw: 0 rgw_thread_pool_size: 2048 rgw_dns_name: example.com rgw_dns_s3website_name: eu-central-1.example.com rgw_enable_gc_threads: false spec: rgw_frontend_port: 7480 rgw_frontend_type: beast rgw_realm: company rgw_zone: eu-central-1 rgw_zonegroup: eu --- service_type: rgw service_id: customer placement: label: rgw config: debug_rgw: 0 rgw_dns_name: s3.customer.domain rgw_dns_s3website_name: eu-central-1.s3.customer.domain rgw_enable_gc_threads: false spec: rgw_frontend_port: 7481 rgw_frontend_type: beast rgw_realm: customer rgw_zone: eu-customer-1 rgw_zonegroup: eu-customer -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Thanks for being my rubber ducky. Turns out I didn't had the rgw_zonegroup configured in the first apply. Then adding it to the config and applying it, does not restart or reconfigure the containers. After doing a ceph orch restart rgw.customer it seems to work now. Happy weekend everybody. Am Fr., 24. Mai 2024 um 14:19 Uhr schrieb Boris <bb@kervyn.de>:
Hi,
we are currently in the process of adopting the main s3 cluster to orchestrator. We have two realms (one for us and one for the customer).
The old config worked fine and depending on the port I requested, I got different x-amz-request-id header back: x-amz-request-id: tx00000307170ac0d734ab4-0066508120-94aa0f66-eu-central-1 x-amz-request-id: tx00000a9d8fc1821bbe258-00665081d1-949b2447-eu-customer-1
I then deployed the orchestrator config to other systems and checked, but now I get always an answer from the eu-central-1 zone and never from the customer zone.
The test is very simple and works fine for the old rgw config, but fails for the new orchestrator config: curl [IPv6]:7481/somebucket -v
All rgw instances are running on ceph version 17.2.7. Configs attached below.
The old ceph.conf looked like this: [client.eu-central-1-s3db1-old] rgw_frontends = beast endpoint=[::]:7480 rgw_region = eu rgw_zone = eu-central-1 rgw_dns_name = example.com rgw_dns_s3website_name = eu-central-1.example.com [client.eu-customer-1-s3db1] rgw_frontends = beast endpoint=[::]:7481 rgw_region = eu-customer rgw_zone = eu-customer-1 rgw_dns_name = s3.customer.domain rgw_dns_s3website_name = eu-central-1.s3.customer.domain
And this is the new service.yaml service_type: rgw service_id: ab12 placement: label: rgw config: debug_rgw: 0 rgw_thread_pool_size: 2048 rgw_dns_name: example.com rgw_dns_s3website_name: eu-central-1.example.com rgw_enable_gc_threads: false spec: rgw_frontend_port: 7480 rgw_frontend_type: beast rgw_realm: company rgw_zone: eu-central-1 rgw_zonegroup: eu --- service_type: rgw service_id: customer placement: label: rgw config: debug_rgw: 0 rgw_dns_name: s3.customer.domain rgw_dns_s3website_name: eu-central-1.s3.customer.domain rgw_enable_gc_threads: false spec: rgw_frontend_port: 7481 rgw_frontend_type: beast rgw_realm: customer rgw_zone: eu-customer-1 rgw_zonegroup: eu-customer
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
participants (1)
-
Boris