Metadata for LibRADOS
Hello - We're trying to use native libRADOS and the only challenge we're running into is searching metadata. Using the rgw metadata sync seems to require all data to be pushed through the rgw, which is not something we're interested in setting up at the moment. Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)? Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things? Are there any features like this in libRADOS? Thank you
Hi Cary, As you've said, these are well-developed features of RGW, I think that would be the way to go, in the Ceph ecosystem. Matt On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Hello -
We're trying to use native libRADOS and the only challenge we're running into is searching metadata.
Using the rgw metadata sync seems to require all data to be pushed through the rgw, which is not something we're interested in setting up at the moment.
Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)?
Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things?
Are there any features like this in libRADOS?
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Understood. With the RGW architecture comes more load balancing concerns, more moving parts, more tedious (to me) ACLs, less features (append and some other things not supported in S3). Was hoping for a solution which didn't require us to be hamstrung and only read / write to a pool via the gateway. If the RGW Metadata search was able to "source" it's data from the OSDs and sync that way, then I'd be up for setting up a skeleton implementation, but it sounds like RGW Metadata is only going to record things which are flowing through the gateway. (Is that correct?) On Tue, Mar 2, 2021 at 3:46 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Hi Cary,
As you've said, these are well-developed features of RGW, I think that would be the way to go, in the Ceph ecosystem.
Matt
On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Hello -
We're trying to use native libRADOS and the only challenge we're running into is searching metadata.
Using the rgw metadata sync seems to require all data to be pushed
through
the rgw, which is not something we're interested in setting up at the moment.
Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)?
Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things?
Are there any features like this in libRADOS?
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Right. The elastic search integration--or something custom you could base on s3 bucket notifications--would both be working with events generated in RGW. Matt On Tue, Mar 2, 2021 at 3:55 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Understood.
With the RGW architecture comes more load balancing concerns, more moving parts, more tedious (to me) ACLs, less features (append and some other things not supported in S3). Was hoping for a solution which didn't require us to be hamstrung and only read / write to a pool via the gateway.
If the RGW Metadata search was able to "source" it's data from the OSDs and sync that way, then I'd be up for setting up a skeleton implementation, but it sounds like RGW Metadata is only going to record things which are flowing through the gateway. (Is that correct?)
On Tue, Mar 2, 2021 at 3:46 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Hi Cary,
As you've said, these are well-developed features of RGW, I think that would be the way to go, in the Ceph ecosystem.
Matt
On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Hello -
We're trying to use native libRADOS and the only challenge we're running into is searching metadata.
Using the rgw metadata sync seems to require all data to be pushed through the rgw, which is not something we're interested in setting up at the moment.
Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)?
Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things?
Are there any features like this in libRADOS?
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Phooey. :) Do you know of any notification subsystems in libRADOS that might be useful? Will have to think on this... Thanks On Tue, Mar 2, 2021 at 4:05 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Right. The elastic search integration--or something custom you could base on s3 bucket notifications--would both be working with events generated in RGW.
Matt
On Tue, Mar 2, 2021 at 3:55 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Understood.
With the RGW architecture comes more load balancing concerns, more
moving parts, more tedious (to me) ACLs, less features (append and some other things not supported in S3). Was hoping for a solution which didn't require us to be hamstrung and only read / write to a pool via the gateway.
If the RGW Metadata search was able to "source" it's data from the OSDs
and sync that way, then I'd be up for setting up a skeleton implementation, but it sounds like RGW Metadata is only going to record things which are flowing through the gateway. (Is that correct?)
On Tue, Mar 2, 2021 at 3:46 PM Matt Benjamin <mbenjami@redhat.com>
Hi Cary,
As you've said, these are well-developed features of RGW, I think that would be the way to go, in the Ceph ecosystem.
Matt
On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@gmail.com>
wrote:
Hello -
We're trying to use native libRADOS and the only challenge we're
running
into is searching metadata.
Using the rgw metadata sync seems to require all data to be pushed
wrote: through
the rgw, which is not something we're interested in setting up at the moment.
Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)?
Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things?
Are there any features like this in libRADOS?
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Unfortunately, nothing like this exists in RADOS. It can't really -- scaling is inimical to the sort of data collation you seem to be looking for. If you use librados, you need to maintain all your own metadata. RGW has done a lot of work to support these features; depending on what you need you may be able to make do by implementing your own centralized metadata system in other objects (omap might be useful); if not you'll need to set up a more extensive system. -Greg On Tue, Mar 2, 2021 at 1:12 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Phooey. :)
Do you know of any notification subsystems in libRADOS that might be useful?
Will have to think on this...
Thanks
On Tue, Mar 2, 2021 at 4:05 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Right. The elastic search integration--or something custom you could base on s3 bucket notifications--would both be working with events generated in RGW.
Matt
On Tue, Mar 2, 2021 at 3:55 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Understood.
With the RGW architecture comes more load balancing concerns, more
moving parts, more tedious (to me) ACLs, less features (append and some other things not supported in S3). Was hoping for a solution which didn't require us to be hamstrung and only read / write to a pool via the gateway.
If the RGW Metadata search was able to "source" it's data from the OSDs
and sync that way, then I'd be up for setting up a skeleton implementation, but it sounds like RGW Metadata is only going to record things which are flowing through the gateway. (Is that correct?)
On Tue, Mar 2, 2021 at 3:46 PM Matt Benjamin <mbenjami@redhat.com>
Hi Cary,
As you've said, these are well-developed features of RGW, I think that would be the way to go, in the Ceph ecosystem.
Matt
On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@gmail.com>
wrote:
Hello -
We're trying to use native libRADOS and the only challenge we're
running
into is searching metadata.
Using the rgw metadata sync seems to require all data to be pushed
wrote: through
the rgw, which is not something we're interested in setting up at the moment.
Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)?
Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things?
Are there any features like this in libRADOS?
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Cary, take a look at the Deutsche Telekom project. It is also based on librados and they also needed a solution for managing the metadata of over 30 million email accounts. https://github.com/ceph-dovecot/dovecot-ceph-plugin but also think about how such a system will behave during recovery. I also listened to an interesting lecture on this subject by nasa at cern. https://indico.cern.ch/event/765214/timetable/?print=1&view=standard#23-ceph-at-nasa-atmosphere-sip Hope this helps you, Joachim ___________________________________ Clyso GmbH - Ceph Foundation Member support@clyso.com https://www.clyso.com Am 03.03.2021 um 21:20 schrieb Gregory Farnum:
Unfortunately, nothing like this exists in RADOS. It can't really -- scaling is inimical to the sort of data collation you seem to be looking for. If you use librados, you need to maintain all your own metadata. RGW has done a lot of work to support these features; depending on what you need you may be able to make do by implementing your own centralized metadata system in other objects (omap might be useful); if not you'll need to set up a more extensive system. -Greg
On Tue, Mar 2, 2021 at 1:12 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Phooey. :)
Do you know of any notification subsystems in libRADOS that might be useful?
Will have to think on this...
Thanks
On Tue, Mar 2, 2021 at 4:05 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Right. The elastic search integration--or something custom you could base on s3 bucket notifications--would both be working with events generated in RGW.
Matt
Understood.
With the RGW architecture comes more load balancing concerns, more moving parts, more tedious (to me) ACLs, less features (append and some other things not supported in S3). Was hoping for a solution which didn't require us to be hamstrung and only read / write to a pool via the gateway. If the RGW Metadata search was able to "source" it's data from the OSDs and sync that way, then I'd be up for setting up a skeleton implementation, but it sounds like RGW Metadata is only going to record
On Tue, Mar 2, 2021 at 3:55 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote: things which are flowing through the gateway. (Is that correct?)
On Tue, Mar 2, 2021 at 3:46 PM Matt Benjamin <mbenjami@redhat.com>
Hi Cary,
As you've said, these are well-developed features of RGW, I think that would be the way to go, in the Ceph ecosystem.
Matt
On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Hello -
We're trying to use native libRADOS and the only challenge we're running into is searching metadata.
Using the rgw metadata sync seems to require all data to be pushed
wrote: through
the rgw, which is not something we're interested in setting up at the moment.
Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)? Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things?
Are there any features like this in libRADOS?
Thank you _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 03/04 09:46, Clyso GmbH - Ceph Foundation Member wrote:
Hello Cary,
take a look at the Deutsche Telekom project. It is also based on librados and they also needed a solution for managing the metadata of over 30 million email accounts.
https://github.com/ceph-dovecot/dovecot-ceph-plugin
but also think about how such a system will behave during recovery.
I also listened to an interesting lecture on this subject by nasa at cern.
https://indico.cern.ch/event/765214/timetable/?print=1&view=standard#23-ceph-at-nasa-atmosphere-sip
watching O.O, thanks for sharing!
Hope this helps you, Joachim
___________________________________ Clyso GmbH - Ceph Foundation Member support@clyso.com https://www.clyso.com
Am 03.03.2021 um 21:20 schrieb Gregory Farnum:
Unfortunately, nothing like this exists in RADOS. It can't really -- scaling is inimical to the sort of data collation you seem to be looking for. If you use librados, you need to maintain all your own metadata. RGW has done a lot of work to support these features; depending on what you need you may be able to make do by implementing your own centralized metadata system in other objects (omap might be useful); if not you'll need to set up a more extensive system. -Greg
On Tue, Mar 2, 2021 at 1:12 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote:
Phooey. :)
Do you know of any notification subsystems in libRADOS that might be useful?
Will have to think on this...
Thanks
On Tue, Mar 2, 2021 at 4:05 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Right. The elastic search integration--or something custom you could base on s3 bucket notifications--would both be working with events generated in RGW.
Matt
Understood.
With the RGW architecture comes more load balancing concerns, more moving parts, more tedious (to me) ACLs, less features (append and some other things not supported in S3). Was hoping for a solution which didn't require us to be hamstrung and only read / write to a pool via the gateway. If the RGW Metadata search was able to "source" it's data from the OSDs and sync that way, then I'd be up for setting up a skeleton implementation, but it sounds like RGW Metadata is only going to record
On Tue, Mar 2, 2021 at 3:55 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote: things which are flowing through the gateway. (Is that correct?)
On Tue, Mar 2, 2021 at 3:46 PM Matt Benjamin <mbenjami@redhat.com>
Hi Cary,
As you've said, these are well-developed features of RGW, I think that would be the way to go, in the Ceph ecosystem.
Matt
On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@gmail.com> wrote: > Hello - > > We're trying to use native libRADOS and the only challenge we're running > into is searching metadata. > > Using the rgw metadata sync seems to require all data to be pushed
wrote: through
> the rgw, which is not something we're interested in setting up at the > moment. > > Are there hooks or features of libRADOS which could be leveraged to enable > syncing of metadata to an external system (elastic-search / postgres / etc)? > Is there a way to listen to a stream of updates to a pool in real-time, > with some guarantees I wouldn't miss things? > > Are there any features like this in libRADOS? > > Thank you > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
participants (5)
-
Cary FitzHugh
-
Clyso GmbH - Ceph Foundation Member
-
David Caro
-
Gregory Farnum
-
Matt Benjamin