3 Dec
2019
3 Dec
'19
6:58 p.m.
We utilize 4 iSCSI gateways in a cluster and have noticed the following
during patching cycles when we sequentially reboot single iSCSI-gateways:
"gwcli" often hangs on the still-up iSCSI GWs but sometimes still functions
and gives the message:
"1 gateway is inaccessible - updates will be disabled"
This got me thinking about what the course of action would be should an
iSCSI gateway fail permanently or semi-permanently, say a hardware issue.
What would be the best course of action to instruct the remaining iSCSI
gateways that one of them is no longer available and that they should allow
updates again and take ownership of the now-defunct-node's LUNS?
I'm guessing pulling down the RADOS config object and rewriting it and
re-put'ing it followed by a rbd-target-api restart might do the trick but
am hoping there is a more "in-band" and less potentially devastating way to
do this.
Thanks for any insights.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
3 Dec
3 Dec
7:19 p.m.
If I recall correctly, the recent ceph-iscsi release supports the
removal of a gateway via the "gwcli". I think the Ceph dashboard can
do that as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham <wes(a)wesdillingham.com> wrote:
We utilize 4 iSCSI gateways in a cluster and have noticed the following during patching
cycles when we sequentially reboot single iSCSI-gateways:
"gwcli" often hangs on the still-up iSCSI GWs but sometimes still functions and
gives the message:
"1 gateway is inaccessible - updates will be disabled"
This got me thinking about what the course of action would be should an iSCSI gateway
fail permanently or semi-permanently, say a hardware issue. What would be the best course
of action to instruct the remaining iSCSI gateways that one of them is no longer available
and that they should allow updates again and take ownership of the now-defunct-node's
LUNS?
I'm guessing pulling down the RADOS config object and rewriting it and re-put'ing
it followed by a rbd-target-api restart might do the trick but am hoping there is a more
"in-band" and less potentially devastating way to do this.
Thanks for any insights.
Respectfully,
Wes Dillingham
wes(a)wesdillingham.com
LinkedIn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
7:23 p.m.
Gateway removal is indeed supported since ceph-iscsi 3.0 (or was it
2.7?) and it works while it is offline :)
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Tue, Dec 3, 2019 at 8:21 PM Jason Dillaman <jdillama(a)redhat.com> wrote:
>
> If I recall correctly, the recent ceph-iscsi release supports the
> removal of a gateway via the "gwcli". I think the Ceph dashboard can
> do that as well.
>
> On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
> >
> > We utilize 4 iSCSI gateways in a cluster and have noticed the following during
patching cycles when we sequentially reboot single iSCSI-gateways:
> >
> > "gwcli" often hangs on the still-up iSCSI GWs but sometimes still
functions and gives the message:
> >
> > "1 gateway is inaccessible - updates will be disabled"
> >
> > This got me thinking about what the course of action would be should an iSCSI
gateway fail permanently or semi-permanently, say a hardware issue. What would be the best
course of action to instruct the remaining iSCSI gateways that one of them is no longer
available and that they should allow updates again and take ownership of the
now-defunct-node's LUNS?
> >
> > I'm guessing pulling down the RADOS config object and rewriting it and
re-put'ing it followed by a rbd-target-api restart might do the trick but am hoping
there is a more "in-band" and less potentially devastating way to do this.
> >
> > Thanks for any insights.
> >
> > Respectfully,
> >
> > Wes Dillingham
> > wes(a)wesdillingham.com
> > LinkedIn
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
>
> --
> Jason
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:45 p.m.
I do not think it's going to do what you want when the node you want to
delete is down.
It looks like we only temporarily stop the gw from being exported. It
does not update the gateway.cfg, because we do the config removal call
on the node we want to delete.
So gwcli would report success and the ls command will show it as no
longer running/exported, but if you restart the rbd-target-api service
then it will show up again.
There is an internal command to do what you want. I will post a PR for
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
If I recall correctly, the recent ceph-iscsi release
supports the
removal of a gateway via the "gwcli". I think the Ceph dashboard can
do that as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham <wes(a)wesdillingham.com> wrote:
We utilize 4 iSCSI gateways in a cluster and have noticed the following during patching
cycles when we sequentially reboot single iSCSI-gateways:
"gwcli" often hangs on the still-up iSCSI GWs but sometimes still functions and
gives the message:
"1 gateway is inaccessible - updates will be disabled"
This got me thinking about what the course of action would be should an iSCSI gateway
fail permanently or semi-permanently, say a hardware issue. What would be the best course
of action to instruct the remaining iSCSI gateways that one of them is no longer available
and that they should allow updates again and take ownership of the now-defunct-node's
LUNS?
I'm guessing pulling down the RADOS config object and rewriting it and re-put'ing
it followed by a rbd-target-api restart might do the trick but am hoping there is a more
"in-band" and less potentially devastating way to do this.
Thanks for any insights.
Respectfully,
Wes Dillingham
wes(a)wesdillingham.com
LinkedIn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:19 p.m.
Thanks. If I am reading this correctly the ability to remove an iSCSI
gateway would allow the remaining iSCSI gateways to take over for the
removed gateway's LUN's as of > 3.0. Thats good, we run 3.2. However,
because the actual update of the central config object happens from the
to-be-deleted iSCSI gateway, despite where the gwcli command is issued, it
will fail to actually remove said gateway from the object if that gateway
is not functioning.
I guess this leaves the question still of how to proceed when one of the
iSCSI gateways fails permanently? Is that possible, or is it potentially
possible other than manually intervening on the config object? If its not
possible would the best course of action be to have standby hardware and
quickly recreate the node or perhaps run the gateways more ephemerally,
from a VM or container?
Thanks again.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Dec 3, 2019 at 2:45 PM Mike Christie <mchristi(a)redhat.com> wrote:
I do not think it's going to do what you want when
the node you want to
delete is down.
It looks like we only temporarily stop the gw from being exported. It
does not update the gateway.cfg, because we do the config removal call
on the node we want to delete.
So gwcli would report success and the ls command will show it as no
longer running/exported, but if you restart the rbd-target-api service
then it will show up again.
There is an internal command to do what you want. I will post a PR for
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
If I recall correctly, the recent ceph-iscsi
release supports the
removal of a gateway via the "gwcli". I think the Ceph dashboard can
do that as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham <wes(a)wesdillingham.com>
wrote:
>
> We utilize 4 iSCSI gateways in a cluster and have noticed the following
during
patching cycles when we sequentially reboot single iSCSI-gateways:
>
> "gwcli" often hangs on the still-up iSCSI GWs but sometimes still
functions and gives the message:
>
> "1 gateway is inaccessible - updates will be disabled"
>
> This got me thinking about what the course of action would be should an
iSCSI
gateway fail permanently or semi-permanently, say a hardware issue.
What would be the best course of action to instruct the remaining iSCSI
gateways that one of them is no longer available and that they should allow
updates again and take ownership of the now-defunct-node's LUNS?
>
> I'm guessing pulling down the RADOS config object and rewriting it and
re-put'ing it followed by a rbd-target-api restart might do the trick but
am hoping there is a more "in-band" and less potentially devastating way to
do this.
Thanks for any insights.
Respectfully,
Wes Dillingham
wes(a)wesdillingham.com
LinkedIn
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4 Dec
4 Dec
3:31 a.m.
On 12/03/2019 04:19 PM, Wesley Dillingham wrote:
Thanks. If I am reading this correctly the ability to
remove an iSCSI
gateway would allow the remaining iSCSI gateways to take over for the
removed gateway's LUN's as of > 3.0. Thats good, we run 3.2. However,
because the actual update of the central config object happens from the
to-be-deleted iSCSI gateway, despite where the gwcli command is issued,
it will fail to actually remove said gateway from the object if that
gateway is not functioning.
Yes.
I guess this leaves the question still of how to proceed when one of the
iSCSI gateways fails permanently? Is that possible, or is it
potentially possible other than manually intervening on the config
You could edit the gateway.cfg manually, but I would not do it, because
it's error prone.
It's probably safest to run in degraded mode and wait for an updated
ceph-iscsi package with a fix. If you are running into the problem right
now, I can bump the priority.
object? If its not possible would the best course of
action be to have
standby hardware and quickly recreate the node or perhaps run the
gateways more ephemerally, from a VM or container?
Thanks again.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Dec 3, 2019 at 2:45 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
I do not think it's going to do what you want when the node you want to
delete is down.
It looks like we only temporarily stop the gw from being exported. It
does not update the gateway.cfg, because we do the config removal call
on the node we want to delete.
So gwcli would report success and the ls command will show it as no
longer running/exported, but if you restart the rbd-target-api service
then it will show up again.
There is an internal command to do what you want. I will post a PR for
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
If I recall correctly, the recent ceph-iscsi
release supports the
removal of a gateway via the "gwcli". I think the Ceph dashboard can
do that as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham
<wes(a)wesdillingham.com
<mailto:wes@wesdillingham.com>> wrote:
>
> We utilize 4 iSCSI gateways in a cluster and have noticed the
following
during patching cycles when we sequentially reboot single
iSCSI-gateways:
>
> "gwcli" often hangs on the still-up iSCSI GWs but sometimes still
functions and gives the message:
>
> "1 gateway is inaccessible - updates will be disabled"
>
> This got me thinking about what the course of action would be
should an
iSCSI gateway fail permanently or semi-permanently, say a
hardware issue. What would be the best course of action to instruct
the remaining iSCSI gateways that one of them is no longer available
and that they should allow updates again and take ownership of the
now-defunct-node's LUNS?
>
> I'm guessing pulling down the RADOS config object and rewriting
it and
re-put'ing it followed by a rbd-target-api restart might do
the trick but am hoping there is a more "in-band" and less
potentially devastating way to do this.
>
> Thanks for any insights.
>
> Respectfully,
>
> Wes Dillingham
> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
> LinkedIn
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
2:26 p.m.
Hi,
Em qua., 4 de dez. de 2019 às 00:31, Mike Christie <mchristi(a)redhat.com>
escreveu:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
On 12/03/2019 04:19 PM, Wesley Dillingham wrote:
need to add another redundancy gateway, and it does not allow with the
gateway "offline".
In this case, what can I do? If I create a new gateway with the same name
and IP as the lost one, and then try to use "delete" in gwcli, will it work?
Thanks. If I am reading this correctly the
ability to remove an iSCSI
gateway would allow the remaining iSCSI gateways to take over for the
removed gateway's LUN's as of > 3.0. Thats good, we run 3.2. However,
because the actual update of the central config object happens from the
to-be-deleted iSCSI gateway, despite where the gwcli command is issued,
it will fail to actually remove said gateway from the object if that
gateway is not functioning.
Yes.
I guess this leaves the question still of how to proceed when one of the
iSCSI gateways fails permanently? Is that possible, or is it
potentially possible other than manually intervening on the config
You could edit the gateway.cfg manually, but I would not do it, because
it's error prone.
It's probably safest to run in degraded mode and wait for an updated
ceph-iscsi package with a fix. If you are running into the problem right
now, I can bump the priority.
I permanently lost a gateway. I can not leave running "degraded" because I
object? If its
not possible would the best course of action be to have
standby hardware and quickly recreate the node or perhaps run the
gateways more ephemerally, from a VM or container?
Thanks again.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Dec 3, 2019 at 2:45 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
I do not think it's going to do what you want when the node you want
to
delete is down.
It looks like we only temporarily stop the gw from being exported. It
does not update the gateway.cfg, because we do the config removal
call
on the node we want to delete.
So gwcli would report success and the ls command will show it as no
longer running/exported, but if you restart the rbd-target-api
service
then it will show up again.
There is an internal command to do what you want. I will post a PR
for
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
> If I recall correctly, the recent ceph-iscsi release supports the
> removal of a gateway via the "gwcli". I think the Ceph dashboard
can
do that
as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham
<wes(a)wesdillingham.com
<mailto:wes@wesdillingham.com>> wrote:
>
> We utilize 4 iSCSI gateways in a cluster and have noticed the
following
during patching cycles when we sequentially reboot single
iSCSI-gateways:
>
> "gwcli" often hangs on the still-up iSCSI GWs but sometimes still
functions and gives the message:
>
> "1 gateway is inaccessible - updates will be disabled"
>
> This got me thinking about what the course of action would be
should an
iSCSI gateway fail permanently or semi-permanently, say a
hardware issue. What would be the best course of action to instruct
the remaining iSCSI gateways that one of them is no longer available
and that they should allow updates again and take ownership of the
now-defunct-node's LUNS?
>
> I'm guessing pulling down the RADOS config object and rewriting
it and
re-put'ing it followed by a rbd-target-api restart might do
the trick but am hoping there is a more "in-band" and less
potentially devastating way to do this.
>
> Thanks for any insights.
>
> Respectfully,
>
> Wes Dillingham
> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
> LinkedIn
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
2:29 p.m.
On Wed, Dec 4, 2019 at 9:27 AM Gesiel Galvão Bernardes
<gesiel.bernardes(a)gmail.com> wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
Hi,
Em qua., 4 de dez. de 2019 às 00:31, Mike Christie <mchristi(a)redhat.com> escreveu:
Yes, it will work if they are configured w/ the same name + IPs and
you re-install the ceph-iscsi SW.
On 12/03/2019 04:19 PM, Wesley Dillingham wrote:
I permanently lost a gateway. I can not leave running "degraded" because
I need to add another redundancy gateway, and it does not allow with the gateway
"offline".
In this case, what can I do? If I create a new gateway with the same name and IP as the
lost one, and then try to use "delete" in gwcli, will it work? Thanks. If I am reading this correctly the
ability to remove an iSCSI
gateway would allow the remaining iSCSI gateways to take over for the
removed gateway's LUN's as of > 3.0. Thats good, we run 3.2. However,
because the actual update of the central config object happens from the
to-be-deleted iSCSI gateway, despite where the gwcli command is issued,
it will fail to actually remove said gateway from the object if that
gateway is not functioning.
Yes.
I guess this leaves the question still of how to proceed when one of the
iSCSI gateways fails permanently? Is that possible, or is it
potentially possible other than manually intervening on the config
You could edit the gateway.cfg manually, but I would not do it, because
it's error prone.
It's probably safest to run in degraded mode and wait for an updated
ceph-iscsi package with a fix. If you are running into the problem right
now, I can bump the priority.
object? If its not possible would the best course
of action be to have
standby hardware and quickly recreate the node or perhaps run the
gateways more ephemerally, from a VM or container?
Thanks again.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Tue, Dec 3, 2019 at 2:45 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
I do not think it's going to do what you want when the node you want to
delete is down.
It looks like we only temporarily stop the gw from being exported. It
does not update the gateway.cfg, because we do the config removal call
on the node we want to delete.
So gwcli would report success and the ls command will show it as no
longer running/exported, but if you restart the rbd-target-api service
then it will show up again.
There is an internal command to do what you want. I will post a PR for
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io If I recall correctly, the recent ceph-iscsi
release supports the
removal of a gateway via the "gwcli". I think the Ceph dashboard can
do that as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham
<wes(a)wesdillingham.com
<mailto:wes@wesdillingham.com>> wrote:
>
> We utilize 4 iSCSI gateways in a cluster and have noticed the
following
during patching cycles when we sequentially reboot single
iSCSI-gateways:
>
> "gwcli" often hangs on the still-up iSCSI GWs but sometimes still
functions and gives the message:
>
> "1 gateway is inaccessible - updates will be disabled"
>
> This got me thinking about what the course of action would be
should an
iSCSI gateway fail permanently or semi-permanently, say a
hardware issue. What would be the best course of action to instruct
the remaining iSCSI gateways that one of them is no longer available
and that they should allow updates again and take ownership of the
now-defunct-node's LUNS?
>
> I'm guessing pulling down the RADOS config object and rewriting
it and
re-put'ing it followed by a rbd-target-api restart might do
the trick but am hoping there is a more "in-band" and less
potentially devastating way to do this.
>
> Thanks for any insights.
>
> Respectfully,
>
> Wes Dillingham
> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
> LinkedIn
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
> To unsubscribe send an email to
ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
4:18 p.m.
On 12/04/2019 08:26 AM, Gesiel Galvão Bernardes wrote:
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>> wrote:
single
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
Hi,
Em qua., 4 de dez. de 2019 às 00:31, Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> escreveu:
On 12/03/2019 04:19 PM, Wesley Dillingham wrote:
Yes.
If you can have a temp stop in services you can also do the following as
a workaround:
0. Stop applications accessing iscsi luns, and have the initiator log
out of the iscsi target.
1. Stop ceph iscsi service. On all iscsi gw nodes do:
systemctl stop rbd-target-api
2. Delete gateway.cfg. This will delete the configuration info like the
target and its ACL and LUN mappings. It does not delete the actual
images or pools that you have data on.
rados -p rbd rm gateway.cfg
3. Start ceph iscsi services again. On all iscsi gw nodes do:
systemctl start rbd-target-api
4. Resetup target with gwcli. For the image/disk setup stage, instead of
doing the "create" command do the "attach"command:
attach pool=your_pool image=image_name
Then just re-add your target, ACLs and LUN mappings.
5. On the initiator side relogin to the iscsi target.
Thanks. If I am reading this correctly the
ability to remove an iSCSI
gateway would allow the remaining iSCSI gateways to take over for the
removed gateway's LUN's as of > 3.0. Thats good, we run 3.2. However,
because the actual update of the central config object happens
from the
to-be-deleted iSCSI gateway, despite where the
gwcli command is
issued,
it will fail to actually remove said gateway from
the object if that
gateway is not functioning.
Yes.
I guess this leaves the question still of how to proceed when one
of the
iSCSI gateways fails permanently? Is that
possible, or is it
potentially possible other than manually intervening on the config
You could edit the gateway.cfg manually, but I would not do it, because
it's error prone.
It's probably safest to run in degraded mode and wait for an updated
ceph-iscsi package with a fix. If you are running into the problem right
now, I can bump the priority.
I permanently lost a gateway. I can not leave running "degraded" because
I need to add another redundancy gateway, and it does not allow with the
gateway "offline".
In this case, what can I do? If I create a new gateway with the same
name and IP as the lost one, and then try to use "delete" in gwcli, will
it work? object? If its not possible would the best course
of action be to have
standby hardware and quickly recreate the node or perhaps run the
gateways more ephemerally, from a VM or container?
Thanks again.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
On Tue, Dec 3, 2019 at 2:45 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com
<mailto:mchristi@redhat.com>>> wrote:
I do not think it's going to do what you want when the node
you want
to
delete is down.
It looks like we only temporarily stop the gw from being
exported. It
does not update the gateway.cfg, because we
do the config
removal call
on the node we want to delete.
So gwcli would report success and the ls command will show it
as no
longer running/exported, but if you restart
the rbd-target-api
service
then it will show up again.
There is an internal command to do what you want. I will post
a PR for
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
> If I recall correctly, the recent ceph-iscsi release
supports the
> removal of a gateway via the
"gwcli". I think the Ceph
dashboard can
do that
as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham
<wes(a)wesdillingham.com
<mailto:wes@wesdillingham.com> >
> We utilize 4 iSCSI gateways in a cluster and have noticed the
following
during patching cycles when we sequentially reboot iSCSI-gateways:
>>
>> "gwcli" often hangs on the still-up iSCSI GWs but sometimes
still
functions and gives the message:
say a
>
> "1 gateway is inaccessible - updates will be disabled"
>
> This got me thinking about what the course of action would be
should an
iSCSI gateway fail permanently or semi-permanently, hardware issue. What would be the best course
of action to
instruct
the remaining iSCSI gateways that one of them
is no longer
available
and that they should allow updates again and
take ownership of the
now-defunct-node's LUNS?
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
>
> I'm guessing pulling down the RADOS config object and rewriting
it and
re-put'ing it followed by a rbd-target-api restart might do
the trick but am hoping there is a more "in-band" and less
potentially devastating way to do this.
>>
>> Thanks for any insights.
>>
>> Respectfully,
>>
>> Wes Dillingham
>> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com> >> LinkedIn
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
8:34 p.m.
I have never had a permanent loss of a gateway but I'm a believer in
Murphy's law and want to have a plan. Glad to hear that there is a solution
in-the-works, curious when might that be available in a release? If sooner
than later I'll plan to upgrade then immediately, otherwise, if far down
the queue I would like to know if I should ready a standby server.
Thanks so much for all your great work on this product.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Dec 4, 2019 at 11:18 AM Mike Christie <mchristi(a)redhat.com> wrote:
On 12/04/2019 08:26 AM, Gesiel Galvão Bernardes
wrote:
that
because
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>> wrote:
be
the
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi,
Em qua., 4 de dez. de 2019 às 00:31, Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> escreveu:
On 12/03/2019 04:19 PM, Wesley Dillingham wrote:
> Thanks. If I am reading this correctly the ability to remove an
iSCSI
> gateway would allow the remaining iSCSI
gateways to take over for
the
> removed gateway's LUN's as of
> 3.0. Thats good, we run 3.2.
However,
because
the actual update of the central config object happens
from the
to-be-deleted iSCSI gateway, despite where the
gwcli command is
issued,
> it will fail to actually remove said gateway from the object if gateway
is not functioning.
Yes.
I guess this leaves the question still of how to proceed when one
of the
iSCSI gateways fails permanently? Is that
possible, or is it
potentially possible other than manually intervening on the config
You could edit the gateway.cfg manually, but I would not do it, it's error prone.
It's probably safest to run in degraded mode and wait for an updated
ceph-iscsi package with a fix. If you are running into the problem
right
now, I can bump the priority.
I permanently lost a gateway. I can not leave running "degraded" because
I need to add another redundancy gateway, and it does not allow with the
gateway "offline".
In this case, what can I do? If I create a new gateway with the same
name and IP as the lost one, and then try to use "delete" in gwcli, will
it work?
Yes.
If you can have a temp stop in services you can also do the following as
a workaround:
0. Stop applications accessing iscsi luns, and have the initiator log
out of the iscsi target.
1. Stop ceph iscsi service. On all iscsi gw nodes do:
systemctl stop rbd-target-api
2. Delete gateway.cfg. This will delete the configuration info like the
target and its ACL and LUN mappings. It does not delete the actual
images or pools that you have data on.
rados -p rbd rm gateway.cfg
3. Start ceph iscsi services again. On all iscsi gw nodes do:
systemctl start rbd-target-api
4. Resetup target with gwcli. For the image/disk setup stage, instead of
doing the "create" command do the "attach"command:
attach pool=your_pool image=image_name
Then just re-add your target, ACLs and LUN mappings.
5. On the initiator side relogin to the iscsi target.
> object? If its not possible would the best course of action be to
have
standby
hardware and quickly recreate the node or perhaps run the
gateways more ephemerally, from a VM or container?
Thanks again.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
On Tue, Dec 3, 2019 at 2:45 PM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com
<mailto:mchristi@redhat.com>>> wrote:
I do not think it's going to do what you want when the node
you want
to
delete is down.
It looks like we only temporarily stop the gw from being
exported. It
does not update the gateway.cfg, because we
do the config
removal call
on the node we want to delete.
So gwcli would report success and the ls command will show it
as no
longer running/exported, but if you restart
the rbd-target-api
service
then it will show up again.
There is an internal command to do what you want. I will post
a PR for
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
> If I recall correctly, the recent ceph-iscsi release
supports the
> removal of a gateway via the
"gwcli". I think the Ceph
dashboard can
do that
as well.
On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham
<wes(a)wesdillingham.com
<mailto:wes@wesdillingham.com> > >>
> >> We utilize 4 iSCSI gateways in a cluster and have noticed
the
following during patching cycles when we sequentially reboot
single
iSCSI-gateways:
>>
>> "gwcli" often hangs on the still-up iSCSI GWs but sometimes
still
> functions and gives the message:
> >>
> >> "1 gateway is inaccessible - updates will be disabled"
> >>
> >> This got me thinking about what the course of action would
should an iSCSI gateway fail permanently or semi-permanently,
say a
hardware issue. What would be the best course
of action to
instruct
the remaining iSCSI gateways that one of them
is no longer
available
> and that they should allow updates again and take ownership of > now-defunct-node's LUNS?
> >>
> >> I'm guessing pulling down the RADOS config object and
rewriting
> it and re-put'ing it followed by
a rbd-target-api restart
might do
the
trick but am hoping there is a more "in-band" and less
potentially devastating way to do this.
>>
>> Thanks for any insights.
>>
>> Respectfully,
>>
>> Wes Dillingham
>> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
>> LinkedIn
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
5 Dec
5 Dec
9:14 p.m.
On 12/04/2019 02:34 PM, Wesley Dillingham wrote:
I have never had a permanent loss of a gateway but
I'm a believer in
Murphy's law and want to have a plan. Glad to hear that there is a
solution in-the-works, curious when might that be available in a
release? If sooner than later I'll plan to upgrade then immediately,
It should be in the next release which I think we would just make when
the patch gets merged since we have a good number of fixes sitting in
the repo.
Patch/PR is here
https://github.com/ceph/ceph-iscsi/pull/156
if you have a non production setup and are used to applying patches and
testing upstream.
otherwise, if far down the queue I would like to know
if I should ready
a standby server.
Thanks so much for all your great work on this product.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Dec 4, 2019 at 11:18 AM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 12/04/2019 08:26 AM, Gesiel Galvão Bernardes wrote:
if that
> I guess this leaves the question
still of how to proceed when one
> So gwcli would report success
and the ls command will show it
sometimes
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
Hi,
Em qua., 4 de dez. de 2019 às 00:31, Mike Christie
<mchristi(a)redhat.com
<mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com
<mailto:mchristi@redhat.com>>> escreveu:
On 12/03/2019 04:19 PM, Wesley Dillingham wrote:
> Thanks. If I am reading this correctly the ability to remove
an
iSCSI
> gateway would allow the remaining iSCSI
gateways to take
over for the
> removed gateway's LUN's as of
> 3.0. Thats good, we run 3.2.
However,
because
the actual update of the central config object happens
from the
to-be-deleted iSCSI gateway, despite where the
gwcli command is
issued,
> it will fail to actually remove said gateway from the object gateway
is not functioning.
Yes.
of the
> iSCSI gateways fails permanently? Is that possible, or is it
> potentially possible other than manually intervening on the
config
You could edit the gateway.cfg manually, but I would not do
it, because
it's error prone.
It's probably safest to run in degraded mode and wait for an
updated
ceph-iscsi package with a fix. If you are
running into the
problem right
now, I can bump the priority.
I permanently lost a gateway. I can not leave running "degraded"
because
I need to add another redundancy gateway, and it
does not allow
with the
gateway "offline".
In this case, what can I do? If I create a new gateway with the same
name and IP as the lost one, and then try to use "delete" in
gwcli,
will
it work?
Yes.
If you can have a temp stop in services you can also do the following as
a workaround:
0. Stop applications accessing iscsi luns, and have the initiator log
out of the iscsi target.
1. Stop ceph iscsi service. On all iscsi gw nodes do:
systemctl stop rbd-target-api
2. Delete gateway.cfg. This will delete the configuration info like the
target and its ACL and LUN mappings. It does not delete the actual
images or pools that you have data on.
rados -p rbd rm gateway.cfg
3. Start ceph iscsi services again. On all iscsi gw nodes do:
systemctl start rbd-target-api
4. Resetup target with gwcli. For the image/disk setup stage, instead of
doing the "create" command do the "attach"command:
attach pool=your_pool image=image_name
Then just re-add your target, ACLs and LUN mappings.
5. On the initiator side relogin to the iscsi target.
> object? If its not possible would the best course of action
be to
have
> standby hardware and quickly recreate
the node or perhaps
run the
> gateways more ephemerally, from a VM or
container?
> Thanks again.
> Respectfully,
> *Wes Dillingham*
> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>>
> LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
> On Tue, Dec 3, 2019 at 2:45 PM Mike Christie
<mchristi(a)redhat.com <mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com
<mailto:mchristi@redhat.com>>
> <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>>> wrote:
I do not think it's going to do what you want when the node
you want
to
delete is down.
It looks like we only temporarily stop the gw from being
exported. It
does not update the gateway.cfg, because we
do the config
removal call
> on the node we want to delete.
as no
> longer running/exported, but if you restart the
rbd-target-api
service
> then it will show up again.
> There is an internal command
to do what you want. I will
post
a PR for
> > On Tue, Dec 3,
2019 at 1:59 PM Wesley Dillingham
> <wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
> If I recall correctly, the recent ceph-iscsi release
supports the
> removal of a gateway via the
"gwcli". I think the Ceph
dashboard can
> > do that as well.
> <mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>>> wrote:
> >>
> >> We utilize 4 iSCSI gateways in a cluster and have
noticed the
following during patching cycles when we sequentially reboot
single
> iSCSI-gateways:
> >>
> >> "gwcli" often hangs on the still-up iSCSI GWs but still
> functions and gives the message:
> >>
> >> "1 gateway is inaccessible - updates will be disabled"
> >>
> >> This got me thinking about what the course of action
would be
> should an iSCSI gateway fail
permanently or
semi-permanently,
say a
ownership of
the
hardware issue. What would be the best course
of action to
instruct
the remaining iSCSI gateways that one of them
is no longer
available
> and that they should allow updates again and take > now-defunct-node's LUNS?
> >>
> >> I'm guessing pulling down the RADOS config object and
rewriting
> it and re-put'ing it followed by
a rbd-target-api
restart might do
> the trick but am hoping there is a
more "in-band" and less
> potentially devastating way to do this.
> >>
> >> Thanks for any insights.
> >>
> >> Respectfully,
> >>
> >> Wes Dillingham
> >> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>>
> >> LinkedIn
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>>>
> >> To unsubscribe send an
email to
ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
> <mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
9:23 p.m.
Thats great thank you so much. I will try and get this patch in my test env
asap but will likely wait for official release cut for prod. I really
appreciate you adding this in to the product.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Thu, Dec 5, 2019 at 4:14 PM Mike Christie <mchristi(a)redhat.com> wrote:
On 12/04/2019 02:34 PM, Wesley Dillingham wrote:
happens
when one
same
following as
wrote:
reboot
to
less
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
I have never had a permanent loss of a gateway
but I'm a believer in
Murphy's law and want to have a plan. Glad to hear that there is a
solution in-the-works, curious when might that be available in a
release? If sooner than later I'll plan to upgrade then immediately,
It should be in the next release which I think we would just make when
the patch gets merged since we have a good number of fixes sitting in
the repo.
Patch/PR is here
https://github.com/ceph/ceph-iscsi/pull/156
if you have a non production setup and are used to applying patches and
testing upstream.
otherwise, if far down the queue I would like to
know if I should ready
a standby server.
Thanks so much for all your great work on this product.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Wed, Dec 4, 2019 at 11:18 AM Mike Christie <mchristi(a)redhat.com
<mailto:mchristi@redhat.com>> wrote:
On 12/04/2019 08:26 AM, Gesiel Galvão Bernardes wrote:
escreveu:
Hi,
Em qua., 4 de dez. de 2019 às 00:31, Mike Christie
<mchristi(a)redhat.com
<mailto:mchristi@redhat.com>
> <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>>
On 12/03/2019 04:19 PM, Wesley Dillingham wrote:
> Thanks. If I am reading this correctly the ability to remove
an
iSCSI
> gateway would allow the remaining iSCSI
gateways to take
over for the
> removed gateway's LUN's as of
> 3.0. Thats good, we run 3.2.
However,
> > because the actual update of the central config object > from the
> > to-be-deleted iSCSI gateway, despite where the gwcli command
is
issued,
> it will fail to actually remove said gateway from the object
if that
gateway
is not functioning.
Yes.
>
> I guess this leaves the question still of how to proceed of the
> iSCSI gateways fails permanently? Is that possible, or is it
> potentially possible other than manually intervening on the
config
You could edit the gateway.cfg manually, but I would not do
it, because
it's error prone.
It's probably safest to run in degraded mode and wait for an
updated
ceph-iscsi package with a fix. If you are
running into the
problem right
now, I can bump the priority.
I permanently lost a gateway. I can not leave running "degraded"
because
I need to add another redundancy gateway, and it
does not allow
with the
> gateway "offline".
>
> In this case, what can I do? If I create a new gateway with the name and
IP as the lost one, and then try to use "delete" in
gwcli, will
it work?
Yes.
If you can have a temp stop in services you can also do the a workaround:
0. Stop applications accessing iscsi luns, and have the initiator log
out of the iscsi target.
1. Stop ceph iscsi service. On all iscsi gw nodes do:
systemctl stop rbd-target-api
2. Delete gateway.cfg. This will delete the configuration info like
the
target and its ACL and LUN mappings. It does
not delete the actual
images or pools that you have data on.
rados -p rbd rm gateway.cfg
3. Start ceph iscsi services again. On all iscsi gw nodes do:
systemctl start rbd-target-api
4. Resetup target with gwcli. For the image/disk setup stage,
instead of
doing the "create" command do the
"attach"command:
attach pool=your_pool image=image_name
Then just re-add your target, ACLs and LUN mappings.
5. On the initiator side relogin to the iscsi target.
node
> object? If its not possible would the best course of action
be to
have
> standby hardware and quickly recreate
the node or perhaps
run the
> gateways more ephemerally, from a VM or
container?
>
> Thanks again.
>
> Respectfully,
>
> *Wes Dillingham*
> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>>
> LinkedIn
<http://www.linkedin.com/in/wesleydillingham>
>
>
> On Tue, Dec 3, 2019 at 2:45 PM Mike Christie
<mchristi(a)redhat.com
<mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com
<mailto:mchristi@redhat.com>>
> <mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>
<mailto:mchristi@redhat.com <mailto:mchristi@redhat.com>>>> wrote:
> >
> > I do not think it's going to do what you want when the you
want to
show it
delete is down.
It looks like we only temporarily stop the gw from being
exported. It
does not update the gateway.cfg, because we
do the config
removal call
> on the node we want to delete.
>
> So gwcli would report success and the ls command will as no
> longer running/exported, but if you restart the
rbd-target-api
service
> then it will show up again.
>
> There is an internal command to do what you want. I will
post
a PR for
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
gwlci and so it can be used by dashboard.
On 12/03/2019 01:19 PM, Jason Dillaman wrote:
> If I recall correctly, the recent ceph-iscsi release
supports the
> removal of a gateway via the
"gwcli". I think the Ceph
dashboard can
> > do that as well.
> >
> > On Tue, Dec 3, 2019 at 1:59 PM Wesley Dillingham
> <wes(a)wesdillingham.com <mailto:wes@wesdillingham.com> <mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>>> >
>>
> >> We utilize 4 iSCSI gateways in a cluster and have
noticed the
> > following during patching cycles when we sequentially
single
> iSCSI-gateways:
> >>
> >> "gwcli" often hangs on the still-up iSCSI GWs but
sometimes
still
> functions and gives the message:
> >>
> >> "1 gateway is inaccessible - updates will be disabled"
> >>
> >> This got me thinking about what the course of action
would be
> should an iSCSI gateway fail
permanently or
semi-permanently,
> say a
> > hardware issue. What would be the best course of action > instruct
> > the remaining iSCSI gateways that one of them is no
longer
available
> and that they should allow updates again and take
ownership of
the
> now-defunct-node's LUNS?
> >>
> >> I'm guessing pulling down the RADOS config object and
rewriting
> it and re-put'ing it followed by
a rbd-target-api
restart might do
> > the trick but am hoping there is a more "in-band" and
>
potentially devastating way to do this.
> >>
> >> Thanks for any insights.
> >>
> >> Respectfully,
> >>
> >> Wes Dillingham
> >> wes(a)wesdillingham.com <mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com <mailto:wes@wesdillingham.com>>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>
<mailto:wes@wesdillingham.com
<mailto:wes@wesdillingham.com>>>
> >> LinkedIn
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
<mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>
<mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>>>
> >> To unsubscribe send an
email to
ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
> <mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>>
<mailto:ceph-users@ceph.io
<mailto:ceph-users@ceph.io>>
To unsubscribe send an email to ceph-users-leave(a)ceph.io
<mailto:ceph-users-leave@ceph.io>
<mailto:ceph-users-leave@ceph.io
<mailto:ceph-users-leave@ceph.io>>
1603
days inactive
1605
days old
11 comments
5 participants
participants (5)
-
Gesiel Galvão Bernardes -
Jason Dillaman -
Mike Christie -
Paul Emmerich -
Wesley Dillingham