On Nov 6, 2023, at 16:52, Anthony D'Atri &lt;aad(a)dreamsnake.net&gt; wrote: I'm having difficulty adding and using a non-default placement target & storage class and would appreciate insights. Am I going about this incorrectly? Rook does not yet have the ability to do this, so I'm adding it by hand. Following instructions on the net I added a second bucket pool, placement target, and storageclass, and created a user defaulting to the new pg/sc, but I get an error when trying to create a bucket: [rook@rook-ceph-tools-5ff8d58445-gkl5w .aws]$ s5cmd --endpoint-url http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc mb s3://foofoobars ERROR "mb s3://foofoobars": InvalidLocationConstraint: status code: 400, request id: tx0000057b71002881d48ca-0065495d54-1abe555-ceph-objectstore, host id: I found an article suggesting that the placement target and/or storage class should have the api_name prepended, so I tried setting either or both to "ceph-objectstore:HDD-EC" / "ceph-objectstore:GLACIER" with no success. I suspect that I'm missing something subtle -- or that Rook has provisioned these bits in an atypical fashion. Log entry: /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.543+0000 7f6573a9f700 1 ====== starting new request req=0x7f64818ba730 ===== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 0 req 6320538205097380042 0.003000009s s3:create_bucket could not find user default placement id HDD-EC/GLACIER within zonegroup /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log-2023-11-06T21:40:36.546+0000 7f6570a99700 1 ====== req done req=0x7f64818ba730 op status=-2208 http_status=400 latency=0.003000009s ====== /var/log/ceph/ceph-client.rgw.ceph.objectstore.a.log:2023-11-06T21:40:36.546+0000 7f6570a99700 1 beast: 0x7f64818ba730: 10.233.90.156 - aad [06/Nov/2023:21:40:36.543 +0000] "PUT /foofoobars HTTP/1.1" 400 266 - "aws-sdk-go/1.40.25 (go1.18.3; linux; amd64)" - latency=0.003000009s [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ ceph -v ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Here's the second buckets pool, constrained to HDDs. AFAICT it can share the index and data_extra_pool created for the default / STANDARD pt/sc by Rook. I initially omitted ec_overwrites but enabled it after creation. pool 19 'ceph-objectstore.rgw.buckets.data' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile size 6 min_size 5 crush_rule 10 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 165350 lfor 0/156300/165341 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw pool 21 'ceph-objectstore.rgw.buckets.data.hdd' erasure profile ceph-objectstore.rgw.buckets.data_ecprofile_hdd size 6 min_size 5 crush_rule 11 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 167193 lfor 0/0/164453 flags hashpspool,ec_overwrites stripe_width 16384 application rook-ceph-rgw [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zonegroup get { "id": "d994155c-2a9c-4e37-ae30-64fd2934ff99", "name": "ceph-objectstore", "api_name": "ceph-objectstore", "is_master": "true", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "72035401-a6d9-426b-8c89-9a17e268825f", "zones": [ { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "endpoints": [ "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "HDD-EC", "tags": [], "storage_classes": [ "GLACIER" ] }, { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "51fb8875-31ac-40ef-ab21-0ffd4e229f15", "sync_policy": { "groups": [] } } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin zone get { "id": "72035401-a6d9-426b-8c89-9a17e268825f", "name": "ceph-objectstore", "domain_root": "ceph-objectstore.rgw.meta:root", "control_pool": "ceph-objectstore.rgw.control", "gc_pool": "ceph-objectstore.rgw.log:gc", "lc_pool": "ceph-objectstore.rgw.log:lc", "log_pool": "ceph-objectstore.rgw.log", "intent_log_pool": "ceph-objectstore.rgw.log:intent", "usage_log_pool": "ceph-objectstore.rgw.log:usage", "roles_pool": "ceph-objectstore.rgw.meta:roles", "reshard_pool": "ceph-objectstore.rgw.log:reshard", "user_keys_pool": "ceph-objectstore.rgw.meta:users.keys", "user_email_pool": "ceph-objectstore.rgw.meta:users.email", "user_swift_pool": "ceph-objectstore.rgw.meta:users.swift", "user_uid_pool": "ceph-objectstore.rgw.meta:users.uid", "otp_pool": "ceph-objectstore.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "HDD-EC", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "GLACIER": { "data_pool": "ceph-objectstore.rgw.buckets.data.hdd" }, "STANDARD": {} # <------------- seems like this shouldn't be here? }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } }, { "key": "default-placement", "val": { "index_pool": "ceph-objectstore.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "ceph-objectstore.rgw.buckets.data" } }, "data_extra_pool": "ceph-objectstore.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "", "notif_pool": "ceph-objectstore.rgw.log:notif" } [rook@rook-ceph-tools-5ff8d58445-gkl5w ~]$ radosgw-admin user info --uid=aad { "user_id": "aad", "display_name": "Anthony", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "aad", "access_key": "xxxx", "secret_key": "yyyyyy" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "HDD-EC", "default_storage_class": "GLACIER", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }