9 Dec
2019
9 Dec
'19
2:31 p.m.
Hi Jacek,
thanks! I wanted to do follow exactly that plan with the metadata!
For moving the index objects to the proper pool - did you lock the buckets somehow?I
wanted to avoid moving the indeces in the first step, when the placement targets allow
leaving everything as it is.
We have a lot of traffic on the buckets, so i thought about modifying the reshard command
and store the "resharded" index on the new place. Just now, i am not sure, if i
need that and if i am able to do the right modifications on the code..
Kind regards,
Ingo
Von: "Jacek Suchenia" <jacek.suchenia(a)gmail.com>
An: "Ingo Reimann" <ireimann(a)dunkel.de>de>, "ceph-users"
<ceph-users(a)ceph.io>
Gesendet: Montag, 9. Dezember 2019 13:34:26
Betreff: Re: [ceph-users] Re: nautilus radosgw fails with pre jewel buckets - index
objects not at right place
Ingo
Yes, we had to use multicore machine to do this efficiently ;-)
Update procedure is very similar to commands described here: [
https://docs.ceph.com/docs/master/radosgw/layout/ |
https://docs.ceph.com/docs/master/radosgw/layout/ ]
so radosgw-admin metadata get bucket.instance:<bucket>:<instance>
then fix JSON
then radoshw-admin metadata set bucket.instance:<bucket>:<instnace>
Good luck!
ps: using this tweak we were able also to move some index objects from storage pool to
index pool, but you have to migrate objects along with OMAP keys
Jacek
pon., 9 gru 2019 o 08:39 Ingo Reimann < [ mailto:ireimann@dunkel.de |
ireimann(a)dunkel.de ] > napisał(a):
Hi Jacek,
thanks for your help. I will try to inject my new placement_target "pre-jewel"
as placement_rule for that buckets with the modified radosgw-admin. Just now struggeling
with the compile prozess.
kind regards,
Ingo
Von: "Jacek Suchenia" < [ mailto:jacek.suchenia@gmail.com |
jacek.suchenia(a)gmail.com ] >
An: "ceph-users" < [ mailto:ceph-users@ceph.io | ceph-users(a)ceph.io ] >
Gesendet: Sonntag, 8. Dezember 2019 16:47:16
Betreff: [ceph-users] Re: nautilus radosgw fails with pre jewel buckets - index objects
not at right place
Hell Ingo
We had the same issue, the fix is to update bucket instance metadata information.
Unfortunately in a source code there is a special code to avoid this kind of change.
[ https://github.com/ceph/ceph/blob/master/src/rgw/rgw_bucket.cc#L3005 |
https://github.com/ceph/ceph/blob/master/src/rgw/rgw_bucket.cc#L3005 ]
So to fix a bucket you have to comment those two lines, compile radosgw-admin and update
bucket index metadata with proper location (In my example they were wrong or empty)
Jacek
pon., 2 gru 2019 o 12:28 Ingo Reimann < [ mailto:ireimann@dunkel.de |
ireimann(a)dunkel.de ] > napisał(a):
BQ_BEGIN
Hi,
2 years after my issue [ [ https://tracker.ceph.com/issues/22928 |
https://tracker.ceph.com/issues/22928 ] | [ https://tracker.ceph.com/issues/22928 |
https://tracker.ceph.com/issues/22928 ] ] the next one fires back.
The Problem:
Old Buckets have their index and data in rgw.buckets:
root@cephrgw01:~# radosgw-admin metadata get bucket:testtesttesty
{
"key": "bucket:testtesttesty",
"ver": {
"tag": "_E_OHNhD28Zu1DeuvyGq8Q8b",
"ver": 1
},
"mtime": "2013-11-11 09:25:56.000000Z",
"data": {
"bucket": {
"name": "testtesttesty",
"marker": "default.2542971.19",
"bucket_id": "default.2542971.19",
"tenant": "",
"explicit_placement": {
"data_pool": "rgw.buckets",
"data_extra_pool": "",
"index_pool": "rgw.buckets"
}
},
"owner": "123",
"creation_time": "2013-11-11 09:25:56.000000Z",
"linked": "true",
"has_bucket_info": "false"
}
}
After upgrade from luminous to nautilus i get 400(InvalidArgument) and "NOTICE:
invalid dest placement" in the radosgw-log on access to the buckets
My zone defines:
root@cephrgw01:~# radosgw-admin zone get
{
"id": "default",
"name": "default",
"domain_root": ".rgw",
"control_pool": ".rgw.control",
"gc_pool": ".rgw.gc",
"lc_pool": ".log:lc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"reshard_pool": ".log:reshard",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "rgw.buckets"
}
},
"data_extra_pool": "rgw.buckets.non-ec",
"index_type": 0
}
}
],
"metadata_heap": ".rgw.meta",
"realm_id": "*********************c"
}
Now i am a little bit lost. I added a new placement to my zone and zonegroup
radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id pre-jewel
radosgw-admin zone placement add --rgw-zonegroup default --placement-id pre-jewel
--data-pool rgw.buckets --index-pool rgw.buckets data-extra-pool ""
radosgw-admin period update --commit
root@cephrgw01:~# radosgw-admin zone get
{
"id": "default",
"name": "default",
"domain_root": ".rgw",
"control_pool": ".rgw.control",
"gc_pool": ".rgw.gc",
"lc_pool": ".log:lc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"reshard_pool": ".log:reshard",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "rgw.buckets"
}
},
"data_extra_pool": "rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "pre-jewel",
"val": {
"index_pool": "rgw.buckets",
"storage_classes": {
"STANDARD": {
"data_pool": "rgw.buckets"
}
},
"data_extra_pool": "",
"index_type": 0
}
}
],
"metadata_heap": ".rgw.meta",
"realm_id": "****************c"
}
Nevertheless, only the luminous gateways may list my old buckets. As far as I can see, I
may only change the placement_rule for new buckets. Is there any chance to make radosgw
find the old indices and complete the upgrade to nautilus?
Many thanks,
Ingo
--
Ingo Reimann
[ [ https://www.dunkel.de/ | https://www.dunkel.de/ ] ]
Dunkel GmbH
Philipp-Reis-Straße 2
65795 Hattersheim
Fon: +49 6190 889-100
Fax: +49 6190 889-399
eMail: [ mailto:support@dunkel.de | support(a)dunkel.de ]
[ http://www.dunkel.de/ | http://www.Dunkel.de/ ] Amtsgericht Frankfurt/Main
HRB: 37971
Geschäftsführer: Axel Dunkel
Ust-ID: DE 811622001
_______________________________________________
ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users(a)ceph.io ]
To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io |
ceph-users-leave(a)ceph.io ]
--
Jacek Suchenia
[ mailto:jacek.suchenia@gmail.com | jacek.suchenia(a)gmail.com ]
_______________________________________________
ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users(a)ceph.io ]
To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io |
ceph-users-leave(a)ceph.io ]
_______________________________________________
ceph-users mailing list -- [ mailto:ceph-users@ceph.io | ceph-users(a)ceph.io ]
To unsubscribe send an email to [ mailto:ceph-users-leave@ceph.io |
ceph-users-leave(a)ceph.io ]
BQ_END
--
Jacek Suchenia
[ mailto:jacek.suchenia@gmail.com | jacek.suchenia(a)gmail.com ]