[ceph-users] Strange behavior when using storage classes

22 Feb 2023

      Hi all,

we encountered some strange behavior when using storage classes for S3 
protocol. Some objects end up in a different pool than we would expect.

Below is a list of commands used for create an account with replicated 
storage class, upload some files to the bucket and checked that they 
were uploaded to the correct location. Most of the files were uploaded 
to the correct pool, but some were uploaded to the erasure-code pool. 
Specifically, certain multipart objects (but they have zero size) and 
stub. The multipart objects probably wouldn't mind so much, but stub is 
important because we tried to delete it and without it the operations 
will not go through.

Are there any errors in our setting (see zonegroup and zone setttings 
below)?

Can someone please send us a setup that is functional so we can build it 
accordingly?

We have a Pacific version.

=== Create the account ===

radosgw-admin user create --tenant vo_du_test --uid 
s3_test_tomash_ec_replicated --display-name s3_test_tomash_ec_replicated 
--storage-class "" --placement-id "EC_replicated" --tags ""

radosgw-admin user info --tenant vo_du_test --uid 
s3_test_tomash_ec_replicated | jq 
'.default_placement,.default_storage_class,.placement_tags'
"EC_replicated"
""
[]

=== Create the bucket with replicated storage class  ===

s3cmd  mb s3://b7-user-ec-stcl-replicated --storage-class=replicated
Bucket 's3://b7-user-ec-stcl-replicated/' created

s3cmd -c ~/.s3cfg/.s3cfg_clx_vo_du_test_s3_test_tomash_ec_replicated put 
20MiB.dat s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat 
--storage-class=replicated
upload: '20MiB.dat' -> 
's3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat'  [part 1 of 2, 15MB] 
[1 of 1]
  15728640 of 15728640   100% in    0s    20.86 MB/s  done
upload: '20MiB.dat' -> 
's3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat'  [part 2 of 2, 5MB] 
[1 of 1]
  5242880 of 5242880   100% in    0s    18.34 MB/s  done

s3cmd -c ~/.s3cfg/.s3cfg_clx_vo_du_test_s3_test_tomash_ec_replicated 
info s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat
s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat (object):
    File size: 20971520
    Storage:   replicated
    ...
    ACL:       s3_test_tomash_ec_replicated: FULL_CONTROL
    ...

=== Checking where objects are stored ===

rados -p storage-clx.rgw.replicated.data ls | grep _b7_repl
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1

ceph_mon:# rados -p storage-clx.rgw.EC.data ls | grep _b7_repl
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat

ceph_mon:# rados stat -p storage-clx.rgw.EC.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat
storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat 
mtime 2023-01-24T12:46:48.000000+0100, size 0

ceph_mon:# rados stat -p storage-clx.rgw.EC.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 
mtime 2023-01-24T12:46:48.000000+0100, size 0

ceph_mon:# rados stat -p storage-clx.rgw.EC.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 
mtime 2023-01-24T12:46:48.000000+0100, size 0

ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 
mtime 2023-01-24T12:46:47.000000+0100, size 4194304
rados stat -p storage-clx.rgw.replicated.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1 
mtime 2023-01-24T12:46:48.000000+0100, size 4194304
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2 
mtime 2023-01-24T12:46:48.000000+0100, size 4194304
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3 
mtime 2023-01-24T12:46:48.000000+0100, size 3145728

ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 
mtime 2023-01-24T12:46:48.000000+0100, size 4194304
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1 
mtime 2023-01-24T12:46:48.000000+0100, size 1048576

=== Zonegroup setting ===

radosgw-admin zonegroup get
{
     "id": "kda7dadb-7bbc-4197-b070-147f76e74717",
     "name": "storage",
     "api_name": "storage",
     "is_master": "true",
     "endpoints": [
         "https://s3.clx.domain.cz"
     ],
     "hostnames": [],
     "hostnames_s3website": [],
     "master_zone": "69922cb9-d006-4b86-b8f0-3722d952fd33",
     "zones": [
         {
             "id": "24654cb9-d006-4b86-b8f0-3722d952fd33",
             "name": "storage-clx",
             "endpoints": [
                 "https://s3.clx.domain.cz"
             ],
             "log_meta": "false",
             "log_data": "false",
             "bucket_index_max_shards": 11,
             "read_only": "false",
             "tier_type": "",
             "sync_from_all": "true",
             "sync_from": [],
             "redirect_zone": ""
         }
     ],
     "placement_targets": [
         {
             "name": "EC_replicated",
             "tags": [],
             "storage_classes": [
                 "EC",
                 "STANDARD",
                 "replicated"
             ]
         },
         {
             "name": "default-placement",
             "tags": [],
             "storage_classes": [
                 "STANDARD"
             ]
         },
         {
             "name": "replicated",
             "tags": [],
             "storage_classes": [
                 "STANDARD"
             ]
         }
     ],
     "default_placement": "default-placement",
     "realm_id": "87a9217-7e8d-4a1b-b508-eb634cc55647",
     "sync_policy": {
         "groups": []
     }
}

=== Zone setting ===

radosgw-admin zone get
{
     "id": "24654cb9-d006-4b86-b8f0-3722d952fd33",
     "name": "storage-clx",
     "domain_root": "storage-clx.rgw.meta:root",
     "control_pool": "storage-clx.rgw.control",
     "gc_pool": "storage-clx.rgw.log:gc",
     "lc_pool": "storage-clx.rgw.log:lc",
     "log_pool": "storage-clx.rgw.log",
     "intent_log_pool": "storage-clx.rgw.log:intent",
     "usage_log_pool": "storage-clx.rgw.log:usage",
     "roles_pool": "storage-clx.rgw.meta:roles",
     "reshard_pool": "storage-clx.rgw.log:reshard",
     "user_keys_pool": "storage-clx.rgw.meta:users.keys",
     "user_email_pool": "storage-clx.rgw.meta:users.email",
     "user_swift_pool": "storage-clx.rgw.meta:users.swift",
     "user_uid_pool": "storage-clx.rgw.meta:users.uid",
     "otp_pool": "storage-clx.rgw.otp",
     "system_key": {
         "access_key": "",
         "secret_key": ""
     },
     "placement_pools": [
         {
             "key": "EC_replicated",
             "val": {
                 "index_pool": "storage-clx.rgw.EC.index",
                 "storage_classes": {
                     "STANDARD": {
                         "data_pool": "storage-clx.rgw.EC.data"
                     },
                     "replicated": {
                         "data_pool": "storage-clx.rgw.replicated.data"
                     }
                 },
                 "data_extra_pool": "storage-clx.rgw.EC.non-ec",
                 "index_type": 0
             }
         },
         {
             "key": "default-placement",
             "val": {
                 "index_pool": "storage-clx.rgw.EC.index",
                 "storage_classes": {
                     "STANDARD": {
                         "data_pool": "storage-clx.rgw.EC.data"
                     }
                 },
                 "data_extra_pool": "storage-clx.rgw.EC.non-ec",
                 "index_type": 0
             }
         },
         {
             "key": "replicated",
             "val": {
                 "index_pool": "storage-clx.rgw.replicated.index",
                 "storage_classes": {
                     "STANDARD": {
                         "data_pool": "storage-clx.rgw.replicated.data"
                     }
                 },
                 "data_extra_pool": "",
                 "index_type": 0
             }
         }
     ],
     "realm_id": "87a9217-7e8d-4a1b-b508-eb634cc55647",
     "notif_pool": "storage-clx.rgw.log:notif"
}

Thank you

Regards,
Michal Strnad