Hey, I'm about to add rbd-mirror support to teh ssh orch and have a few
questions after skimming doc/rbd/rbd-mirroring.rst
The orchestrator thinks of service in terms of sets. Rook, for example,
can scale the number of daemons up and down. I assume I want one set of
daemons per peer cluster? Does the rbd-mirror daemon then service any
rbd-mirroring-enabled pool that works with taht peer cluster? Or is there
anything I can/should do to tell a particular rbd-mirror daemon to work on
one pool or peer vs another?
FWIW on the RGW side, the 'group' is a zone, and we're making the client
ids hierarchical, like client.rgw.$zone.$id. And the config subsystem has
been updated so that we pull optoins from global, client, client.rgw,
client.rgw.$zone, and client.rgw.$zone.$id. In the rgw case, we 'config
set client.rgw.$zone rgw_zone $zone', which ensures all those daemons are
working on the right thing. We might be able/want to do something similar
here?
Thanks!
sage
Hi Folks,
Perf meeting is on in ~40 minutes! Discussion topics for today include
Deep Scrubbing strategies by Robert Bruno and Luke Higgins at Morgan
Stanley and some critical performance issues we discovered with
DeleteRange in RocksDB 6.1.2. See you there!
Etherpad:
https://pad.ceph.com/p/performance_weekly
Bluejeans:
https://bluejeans.com/908675367
Thanks,
Mark
Apple Certified Technical Coordinators are Mac OS X technical coordinators with considerable knowledge of Mac OS X and the fundamentals of Mac OS X Server core functionality. They configure crucial services and execute basic troubleshooting, network deployment, policy management, etc. Coordinators handle the most common network job tasks of Mac OS.
They also handle tools for implementing Mac OS and software updates and after that efficiently manage them. Coordinators provide the first-level of support by acting to enable stable and efficient IT infrastructure. They respond to tickets that clients raise through email, phone or in person by resolving or by escalating them to their senior team members.
Read More: https://www.fieldengineer.com/skills/apple-certified-technical-coordinator
Hi all, there seems to be some confusion concerning the backport tracker.
The purpose of the backport tracker is to track backports of fixes that need to
go into multiple stable branches, so they don't "fall through the cracks".
If you need to backport something from master *only* to nautilus *and no
farther*, there is no need to create tracker issues for backporting purposes.
Just open your nautilus PR with the cherry-pick [1].
(If there is already a master tracker issue, you can just mention the URL of the
nautilus backport in a comment on the tracker. You don't need to change the
status to Pending Backport or fill in the Backport field.)
Hope this helps,
Nathan
[1] of course, follow the cherry-picking rules
https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst#ch…
--
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037
Hi,
Is anyone using librados AIO APIs? I seem to have a problem with that where
the rados_aio_wait_for_complete() call just waits for a long period of time
before it finishes without error.
More info on my setup:
I am using Ceph 14.2.4 and write 8MB objects.
I run my AIO program on 24 nodes at the same time each writing a different
data (splits into 8MB objects and ), each data is about 2G.
Normally, it takes about 10 mins for all of them to complete. But often one
or more nodes takes considerably longer to finish. When looking at the one
of those, I mostly see that the IO requests have been submitted and waits
at:
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00002aaaaad0c8fa in rados_aio_wait_for_complete () from
/cgv/geovation/2/test/ceph/lib/librados.so.2
Then it eventually completes with no errors from
rados_aio_wait_for_complete() call.
The (pseudo) code looks like:
while (data remains to be written) {
size_t aio_ops_count = 0;
rados_completion_t aio_comp[12];
for (size_t j = 0; j < 12; ++j) {
int err = rados_aio_create_completion(NULL, NULL, NULL,
&aio_comp[j]);
if (err < 0) {
cerr << "rados_aio_create_completion: " <<
strerror(-err) << endl;
return 1;
}
string obj_ = getobjectid();
err = rados_aio_write_full(io, obj_.c_str(), aio_comp[j],
read_buf[j], bytes);
if (err < 0) {
cerr << "rados_write_full: " << strerror(-err) << endl;
return 1;
}
++aio_ops_count;
}
for (size_t j = 0; j < aio_ops_count; ++j) {
rados_aio_wait_for_complete(aio_comp[j]);
int err = rados_aio_get_return_value(aio_comp[j]); //
Considerably longer delay here ??
if (err < 0) {
cerr << "rados_aio_get_return_value: " <<
strerror(-err) << endl;
return 1;
}
rados_aio_release(aio_comp[j]);
}
}
I ran under Valgrind and see no issues and also read the data back and
checksum it to verify no corruption issues. So everything appears to "work"
as expected except for longer delays at times.
Wondering if anyone is using the AIO APIs to write objects and had
experienced any similar problems.
--
Regards,
Ponnuvel P
On 07:31 Mon 28 Oct, Mason-Williams, Gabryel (DLSLtd,RAL,LSCI) wrote:
> I am using ceph version 12.2.8
> (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable).
>
> I have not checked the master branch do you think this is an issue in
> luminous that has been removed in later versions?
I haven't hit problem on master branch. Ceph/RDMA changed a lot
from luminous to master branch.
Is below configuration really needed in luminous/ceph.conf?
> ms_async_rdma_local_gid = xxxx
On master branch, this parameter is not needed at all.
B.R.
Changcheng
> __________________________________________________________________
>
> From: Liu, Changcheng <changcheng.liu(a)intel.com>
> Sent: 25 October 2019 18:04
> To: Mason-Williams, Gabryel (DLSLtd,RAL,LSCI)
> <gabryel.mason-williams(a)diamond.ac.uk>
> Cc: ceph-users(a)ceph.com <ceph-users(a)ceph.com>; dev(a)ceph.io
> <dev(a)ceph.io>
> Subject: Re: RMDA Bug?
>
> What's your ceph version? Have you verified whether the problem could
> be
> reproduced on master branch?
> On 08:33 Fri 25 Oct, Mason-Williams, Gabryel (DLSLtd,RAL,LSCI) wrote:
> > I am currently trying to run Ceph on RDMA, either RoCE 1 or 2.
> However,
> > I am experiencing issues with this.
> >
> > When using Ceph on RDMA I experience issues where OSD’s will
> randomly
> > become unreachable even if the cluster is left alone alone, it
> also is
> > not properly talking over RDMA and using Ethernet when the config
> > states it should as shown by the same results in the bench marking
> of
> > the two setups.
> >
> > After reloading the cluster
> > [cid:36020940-0085-40fc-bb5b-d91de6ace453]
> >
> > After 5m 9s the cluster went from being healthy to down.
> >
> > [cid:ed084bcc-0b97-44bd-9648-ce2e06859cd5]
> >
> > This problem even happens when running a bench mark test on the
> > cluster, OSD’s will just fall over. Another curious issue is that
> it is
> > not properly talking over RDMA as well and instead using the
> Ethernet.
> >
> > [cid:05e9dc68-075e-425d-b76b-ce7fa1d2f7a8]
> >
> > Next test:
> >
> > [cid:4183557e-b1da-41f3-afc3-f081b9fb4034]
> >
> > The config used for the RDMA is a so:
> >
> > [global]
> >
> > fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
> >
> > mon_initial_members = node1, node2, node3
> >
> > mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx
> >
> > auth_cluster_required = cephx
> >
> > auth_service_required =cephx
> >
> > auth_client_required = cephx
> >
> > public_network = xxx.xxx.xxx.xxx/24
> >
> > cluster_network = yyy.yyy.yyy.yyy/16
> >
> > ms_cluster_type =async+rdma
> >
> > ms_public_type = async+posix
> >
> > ms_async_rdma_device_name = mlx4_0
> >
> > [osd.0]
> >
> > ms_async_rdma_local_gid = xxxx
> >
> > [osd.1]
> >
> > ms_async_rdma_local_gid = xxxx
> >
> > [osd.2]
> >
> > ms_async_rdma_local_gid =xxxx
> >
> > Tests to check the system is using RDMA
> >
> > sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show
> |
> > grep ms_cluster
> >
> > OUTPUT
> >
> > "ms_cluster_type": "async+rdma",
> >
> > sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1
> >
> > OUTPUT
> >
> > {
> >
> > "AsyncMessenger::RDMAWorker-1": {
> >
> > "tx_no_mem": 0,
> >
> > "tx_parital_mem": 0,
> >
> > "tx_failed_post": 0,
> >
> > "rx_no_registered_mem": 0,
> >
> > "tx_chunks": 9,
> >
> > "tx_bytes": 2529,
> >
> > "rx_chunks": 0,
> >
> > "rx_bytes": 0,
> >
> > "pending_sent_conns": 0
> >
> > }
> >
> > }
> >
> > When running over Ethernet I have a completely stable system with
> the
> > current benchmarks as so
> >
> > [cid:544ecbbc-10d9-43e6-ab2f-aa7c2bcd88c0]
> >
> > Config setup when using Ethernet is
> >
> > The Config setup when using Ethernet is
> >
> > [global]
> >
> > fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
> >
> > mon_initial_members = node1, node2, node3
> >
> > mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx
> >
> > auth_cluster_required = cephx
> >
> > auth_service_required =cephx
> >
> > auth_client_required = cephx
> >
> > public_network = xxx.xxx.xxx.xxx/24
> >
> > cluster_network = yyy.yyy.yyy.yyy/16
> >
> > ms_cluster_type =async+posix
> >
> > ms_public_type = async+posix
> >
> > ms_async_rdma_device_name = mlx4_0
> >
> > [osd.0]
> >
> > ms_async_rdma_local_gid = xxxx
> >
> > [osd.1]
> >
> > ms_async_rdma_local_gid = xxxx
> >
> > [osd.2]
> >
> > ms_async_rdma_local_gid =xxxx
> > Tests to check the system is using async+posix
> >
> > sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show
> |
> > grep ms_cluster
> >
> > OUTPUT
> >
> > "ms_cluster_type": "async+posix"
> >
> > sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1
> >
> > OUTPUT
> >
> > {}
> >
> > This clearly a issue with RDMA and not with the OSD's shown by the
> fact
> > the system is completely fine over Ethernet and not with RDMA.
> >
> > Any guidance or ideas on how to approach this problem to make Ceph
> work
> > with RDMA would be greatly appreciated.
> >
> > Regards
> >
> > Gabryel Mason-Williams, Placement Student
> >
> > Address: Diamond Light Source Ltd., Diamond House, Harwell Science
> &
> > Innovation Campus, Didcot, Oxfordshire OX11 0DE
> >
> > Email: gabryel.mason-williams(a)diamond.ac.uk
> >
> >
> > --
> >
> > This e-mail and any attachments may contain confidential,
> copyright and
> > or privileged material, and are for the use of the intended
> addressee
> > only. If you are not the intended addressee or an authorised
> recipient
> > of the addressee please notify us of receipt by returning the
> e-mail
> > and do not use, copy, retain, distribute or disclose the
> information in
> > or attached to the e-mail.
> > Any opinions expressed within this e-mail are those of the
> individual
> > and not necessarily of Diamond Light Source Ltd.
> > Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> > attachments are free from viruses and we cannot accept liability
> for
> > any damage which you may sustain as a result of software viruses
> which
> > may be transmitted in or with the message.
> > Diamond Light Source Limited (company no. 4375679). Registered in
> > England and Wales with its registered office at Diamond House,
> Harwell
> > Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE,
> United
> > Kingdom
> > _______________________________________________
> > Dev mailing list -- dev(a)ceph.io
> > To unsubscribe send an email to dev-leave(a)ceph.io
>
>
> --
>
> This e-mail and any attachments may contain confidential, copyright and
> or privileged material, and are for the use of the intended addressee
> only. If you are not the intended addressee or an authorised recipient
> of the addressee please notify us of receipt by returning the e-mail
> and do not use, copy, retain, distribute or disclose the information in
> or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual
> and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for
> any damage which you may sustain as a result of software viruses which
> may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in
> England and Wales with its registered office at Diamond House, Harwell
> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
> Kingdom