Hi Matt, On 12/10/19 1:52 PM, Matt Benjamin wrote: That would probably be an acceptable alternative. I did have such a policy in place to automatically change the storage class (and I verified this works from the perspective of s3cmd). But when I looked at `rados -p tier1-ssd ls`, the object that I presumed was moved was still there. Another example I just executed: directly setting the storage class gives me the following output: <<<<<<< client $ s3cmd put git-tree.png s3://bucket/ --storage-class=SPINNING_RUST WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: 'git-tree.png' -> 's3://bucket/git-tree.png' [1 of 1] 16490 of 16490 100% in 0s 532.37 kB/s done client $ s3cmd info s3://bucket/git-tree.png Last mod: Tue, 10 Dec 2019 12:58:52 GMT Storage: SPINNING_RUST ======= On the cluster: <<<<<<< root@node1:~# rados -p tier2-hdd ls ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1__shadow_.aQHkk2RcTCHN64E_XJjA0wGTnUtWSN2_0 ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1__shadow_.FEruUOZaVJXJcOG-e2tO1xcInNzoEvN_0 root@node1:~# rados -p tier1-ssd ls ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_git-tree.png ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_darthvader.png ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_2019-10-15-090436_1254x522_scrubbed.png ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_kanariepiet.jpg ======= whereas I would expect "git-tree.png" to only reside in the pool tier2-hdd. This gives me the suggestion that I made an error in configuring the storage_class->pool association. My hunch is that the zone(group) placement is incorrect, but I can't really find clear documentation on that subject. Any thoughts on that? Best regards, Gerdriaan Mulder > On Tue, Dec 10, 2019 at 7:44 AM Gerdriaan Mulder &lt;gerdriaan(a)hexon.nl&gt; wrote: >> >> Hi, >> >> If I change the storage class of an object via s3cmd, the object's >> storage class is reported as being changed. However, when inspecting >> where the objects are placed (via `rados -p <pool> ls`, see further on), >> the object seems to be retained in the original pool. >> >> The idea behind this test setup is to simulate two storage locations, >> one based on SSDs or similar flash storage, the other on slow HDDs. We >> want to be able to alter the storage location of objects on the fly, >> typically only from fast to slow storage. The object should then only >> reside on slow storage. >> >> The setup is as follows on Nautilus (Ubuntu 16.04, see >> <https://gist.github.com/mrngm/bba6ffdc545bfa52ebf79d6d2c002a6d> for the >> full dump): >> >> <<<<<<<< >> root@node1:~# ceph -s >> health: HEALTH_OK >> >> mon: 3 daemons, quorum node1,node3,node5 (age 12d) >> mgr: node2(active, since 6d), standbys: node4 >> osd: 4 osds: 4 up (since 12d), 4 in (since 12d) >> rgw: 1 daemon active (node1) >> >> pools: 7 pools, 296 pgs >> objects: 229 objects, 192 KiB >> usage: 3.2 GiB used, 6.8 GiB / 10 GiB avail >> pgs: 296 active+clean >> >> root@node1:~# ceph osd tree >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 0.00970 root default >> -16 0.00970 datacenter nijmegen >> -3 0.00388 host node2 >> 0 hdd 0.00388 osd.0 up 1.00000 1.00000 >> -5 0.00388 host node3 >> 1 hdd 0.00388 osd.1 up 1.00000 1.00000 >> -7 0.00098 host node4 >> 2 ssd 0.00098 osd.2 up 1.00000 1.00000 >> -9 0.00098 host node5 >> 3 ssd 0.00098 osd.3 up 1.00000 1.00000 >> >> root@node1:~# ceph osd pool ls detail >> pool 1 'tier1-ssd' replicated size 2 min_size 1 crush_rule 1 object_hash >> rjenkins pg_num 128 pgp_num 128 [snip] application rgw >> pool 2 'tier2-hdd' replicated size 1 min_size 1 crush_rule 2 object_hash >> rjenkins pg_num 128 pgp_num 128 [snip] application rgw >> pool 3 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash >> rjenkins pg_num 8 pgp_num 8 [snip] application rgw >> pool 4 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 >> [snip] application rgw >> pool 5 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 >> [snip] application rgw >> pool 6 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 >> [snip] application rgw >> pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 >> crush_rule 0 [snip] application rgw >> >> root@node1:~# ceph osd pool application get # compacted >> tier1-ssd => rgw {} >> tier2-hdd => rgw {} >> .rgw.root => rgw {} >> default.rgw.control => rgw {} >> default.rgw.meta => rgw {} >> default.rgw.log => rgw {} >> default.rgw.buckets.index => rgw {} >> >> root@node1:~# radosgw-admin zonegroup placement list >> [ >> { >> "key": "default-placement", >> "val": { >> "name": "default-placement", >> "tags": [], >> "storage_classes": [ >> "SPINNING_RUST", >> "STANDARD" >> ] >> } >> } >> ] >> >> root@node1:~# radosgw-admin zone placement list >> [ >> { >> "key": "default-placement", >> "val": { >> "index_pool": "default.rgw.buckets.index", >> "storage_classes": { >> "SPINNING_RUST": { >> "data_pool": "tier2-hdd" >> }, >> "STANDARD": { >> "data_pool": "tier1-ssd" >> } >> }, >> "data_extra_pool": "default.rgw.buckets.non-ec", >> "index_type": 0 >> } >> } >> ] >> ======== >> >> I can also post the relevant s3cmd commands for putting objects and >> setting the storage class, but perhaps this is already enough >> information. Please let me know. >> >> <<<<<<<< >> root@node1:~# rados -p tier1-ssd ls >> ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_darthvader.png >> ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_2019-10-15-090436_1254x522_scrubbed.png >> ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_kanariepiet.jpg >> >> root@node1:~# rados -p tier2-hdd ls >> ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1__shadow_.FEruUOZaVJXJcOG-e2tO1xcInNzoEvN_0 >> >> $ s3cmd info s3://bucket/kanariepiet.jpg >> [snip] >> Last mod: Tue, 10 Dec 2019 08:09:58 GMT >> Storage: STANDARD >> [snip] >> >> $ s3cmd info s3://bucket/darthvader.png >> [snip] >> Last mod: Wed, 04 Dec 2019 10:35:14 GMT >> Storage: SPINNING_RUST >> [snip] >> >> $ s3cmd info s3://bucket/2019-10-15-090436_1254x522_scrubbed.png >> [snip] >> Last mod: Tue, 10 Dec 2019 10:33:24 GMT >> Storage: STANDARD >> [snip] >> ========== >> >> Any thoughts on what might occur here? >> >> Best regards, >> Gerdriaan Mulder >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > >

Hi Casey, On 12/10/19 3:00 PM, Casey Bodley wrote: Thanks for the explanation. It seems the documentation on Nautilus is somewhat lacking on these particular intricacies. Best regards, Gerdriaan Mulder