Re: [ceph-users] details about cloning objects using librados

Hi Greg, We were trying to implement this however having issues in assigning the destination object name with this api. There is a rados command "rados -p <poolname> cp <src obj> <dst obj>" , is there any librados api equivalent to this ?

Thanks, Muthu On Fri, Jul 5, 2019 at 4:00 PM nokia ceph &lt;nokiacephusers(a)gmail.com&gt; wrote: > Thank you Greg, we will try this out . > > Thanks, > Muthu > > On Wed, Jul 3, 2019 at 11:12 PM Gregory Farnum &lt;gfarnum(a)redhat.com&gt; > wrote: > >> Well, the RADOS interface doesn't have a great deal of documentation >> so I don't know if I can point you at much. >> >> But if you look at Objecter.h, you see that the ObjectOperation has >> this function: >> void copy_from(object_t src, snapid_t snapid, object_locator_t >> src_oloc, version_t src_version, unsigned flags, unsigned >> src_fadvise_flags) >> >> src: the object to copy from >> snapid: if you want to copy a specific snap instead of HEAD >> src_oloc: the object locator for the object >> src_version: the version of the object to copy from (helps identify if >> it was updated in the meantime) >> flags: probably don't want to set these, but see >> PrimaryLogPG::_copy_some for the choices >> src_fadvise_flags: these are the fadvise flags we have in various >> places that let you specify things like not to cache the data. >> Probably leave them unset. >> >> -Greg >> >> >> >> On Wed, Jul 3, 2019 at 2:47 AM nokia ceph &lt;nokiacephusers(a)gmail.com&gt; >> wrote: >> > >> > Hi Greg, >> > >> > Can you please share the api details for COPY_FROM or any reference >> document? >> > >> > Thanks , >> > Muthu >> > >> > On Wed, Jul 3, 2019 at 4:12 AM Brad Hubbard &lt;bhubbard(a)redhat.com&gt; >> wrote: >> >> >> >> On Wed, Jul 3, 2019 at 4:25 AM Gregory Farnum &lt;gfarnum(a)redhat.com&gt; >> wrote: >> >> > >> >> > I'm not sure how or why you'd get an object class involved in doing >> >> > this in the normal course of affairs. >> >> > >> >> > There's a copy_from op that a client can send and which copies an >> >> > object from another OSD into the target object. That's probably the >> >> > primitive you want to build on. Note that the OSD doesn't do much >> >> >> >> Argh! yes, good idea. We really should document that! >> >> >> >> > consistency checking (it validates that the object version matches >> an >> >> > input, but if they don't it just returns an error) so the client >> >> > application is responsible for any locking needed. >> >> > -Greg >> >> > >> >> > On Tue, Jul 2, 2019 at 3:49 AM Brad Hubbard &lt;bhubbard(a)redhat.com&gt; >> wrote: >> >> > > >> >> > > Yes, this should be possible using an object class which is also a >> >> > > RADOS client (via the RADOS API). You'll still have some client >> >> > > traffic as the machine running the object class will still need to >> >> > > connect to the relevant primary osd and send the write >> (presumably in >> >> > > some situations though this will be the same machine). >> >> > > >> >> > > On Tue, Jul 2, 2019 at 4:08 PM nokia ceph < >> nokiacephusers(a)gmail.com&gt; wrote: >> >> > > > >> >> > > > Hi Brett, >> >> > > > >> >> > > > I think I was wrong here in the requirement description. It is >> not about data replication , we need same content stored in different >> object/name. >> >> > > > We store video contents inside the ceph cluster. And our new >> requirement is we need to store same content for different users , hence >> need same content in different object name . if client sends write request >> for object x and sets number of copies as 100, then cluster has to clone >> 100 copies of object x and store it as object x1, objectx2,etc. Currently >> this is done in the client side where objectx1, object x2...objectx100 are >> cloned inside the client and write request sent for all 100 objects which >> we want to avoid to reduce network consumption. >> >> > > > >> >> > > > Similar usecases are rbd snapshot , radosgw copy . >> >> > > > >> >> > > > Is this possible in object class ? >> >> > > > >> >> > > > thanks, >> >> > > > Muthu >> >> > > > >> >> > > > >> >> > > > On Mon, Jul 1, 2019 at 7:58 PM Brett Chancellor < >> bchancellor(a)salesforce.com&gt; wrote: >> >> > > >> >> >> > > >> Ceph already does this by default. For each replicated pool, >> you can set the 'size' which is the number of copies you want Ceph to >> maintain. The accepted norm for replicas is 3, but you can set it higher if >> you want to incur the performance penalty. >> >> > > >> >> >> > > >> On Mon, Jul 1, 2019, 6:01 AM nokia ceph < >> nokiacephusers(a)gmail.com&gt; wrote: >> >> > > >>> >> >> > > >>> Hi Brad, >> >> > > >>> >> >> > > >>> Thank you for your response , and we will check this video as >> well. >> >> > > >>> Our requirement is while writing an object into the cluster , >> if we can provide number of copies to be made , the network consumption >> between client and cluster will be only for one object write. However , the >> cluster will clone/copy multiple objects and stores inside the cluster. >> >> > > >>> >> >> > > >>> Thanks, >> >> > > >>> Muthu >> >> > > >>> >> >> > > >>> On Fri, Jun 28, 2019 at 9:23 AM Brad Hubbard < >> bhubbard(a)redhat.com&gt; wrote: >> >> > > >>>> >> >> > > >>>> On Thu, Jun 27, 2019 at 8:58 PM nokia ceph < >> nokiacephusers(a)gmail.com&gt; wrote: >> >> > > >>>> > >> >> > > >>>> > Hi Team, >> >> > > >>>> > >> >> > > >>>> > We have a requirement to create multiple copies of an >> object and currently we are handling it in client side to write as separate >> objects and this causes huge network traffic between client and cluster. >> >> > > >>>> > Is there possibility of cloning an object to multiple >> copies using librados api? >> >> > > >>>> > Please share the document details if it is feasible. >> >> > > >>>> >> >> > > >>>> It may be possible to use an object class to accomplish what >> you want >> >> > > >>>> to achieve but the more we understand what you are trying to >> do, the >> >> > > >>>> better the advice we can offer (at the moment your >> description sounds >> >> > > >>>> like replication which is already part of RADOS as you know). >> >> > > >>>> >> >> > > >>>> More on object classes from Cephalocon Barcelona in May this >> year: >> >> > > >>>> https://www.youtube.com/watch?v=EVrP9MXiiuU >> >> > > >>>> >> >> > > >>>> > >> >> > > >>>> > Thanks, >> >> > > >>>> > Muthu >> >> > > >>>> > _______________________________________________ >> >> > > >>>> > ceph-users mailing list >> >> > > >>>> > ceph-users(a)lists.ceph.com >> >> > > >>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > > >>>> -- >> >> > > >>>> Cheers, >> >> > > >>>> Brad >> >> > > >>> >> >> > > >>> _______________________________________________ >> >> > > >>> ceph-users mailing list >> >> > > >>> ceph-users(a)lists.ceph.com >> >> > > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Cheers, >> >> > > Brad >> >> > > _______________________________________________ >> >> > > ceph-users mailing list >> >> > > ceph-users(a)lists.ceph.com >> >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> >> >> >> -- >> >> Cheers, >> >> Brad >> >