4 Sep
2019
4 Sep
'19
11:01 p.m.
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr is
not a terminal.
Reproducible on different servers, so there's nothing weird about a
particular disk.
Any idea where/how this is happening?
This makes 14.2.3 unusable for us as we need to re-activate all OSDs
after reboots because we don't have a persistent system disk.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
4 Sep
4 Sep
11:37 p.m.
On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich(a)croit.io> wrote:
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr is
not a terminal.
Reproducible on different servers, so there's nothing weird about a
particular disk.
Any idea where/how this is happening?
That looks very odd, haven't seen it other than a unit test we have
that fails in some machines. I was just investigating that today.
Is it possible that the locale is set to something that is not
en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
Another thing I would try is to enable debug (or show/paste the
traceback) so that tracebacks are immediately available in the output:
CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
I'll follow up in the tracker ticket
>
> This makes 14.2.3 unusable for us as we need to re-activate all OSDs
> after reboots because we don't have a persistent system disk.
>
>
> Paul
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
5 Sep
5 Sep
1:35 a.m.
How do you fix it? Or you wait till 14.2.4?
On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza(a)redhat.com> wrote:
On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich
<paul.emmerich(a)croit.io>
wrote:
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr is
not a terminal.
Reproducible on different servers, so there's nothing weird about a
particular disk.
Any idea where/how this is happening?
That looks very odd, haven't seen it other than a unit test we have
that fails in some machines. I was just investigating that today.
Is it possible that the locale is set to something that is not
en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
Another thing I would try is to enable debug (or show/paste the
traceback) so that tracebacks are immediately available in the output:
CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
I'll follow up in the tracker ticket
This makes 14.2.3 unusable for us as we need to re-activate all OSDs
after reboots because we don't have a persistent system disk.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
2:54 a.m.
On Wed, Sep 4, 2019 at 6:35 PM Sasha Litvak
<alexander.v.litvak(a)gmail.com> wrote:
How do you fix it? Or you wait till 14.2.4?
This is a high priority for me, I will provide a fix as soon as
possible and hopefully a workaround.
>
> On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza(a)redhat.com> wrote:
>>
>> On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich(a)croit.io>
wrote:
>> >
>> > Hi,
>> >
>> > see https://tracker.ceph.com/issues/41660
>> >
>> > ceph-volume lvm activate --all fails on the second OSD when stderr is
>> > not a terminal.
>> > Reproducible on different servers, so there's nothing weird about a
>> > particular disk.
>> >
>> > Any idea where/how this is happening?
>>
>> That looks very odd, haven't seen it other than a unit test we have
>> that fails in some machines. I was just investigating that today.
>>
>> Is it possible that the locale is set to something that is not
>> en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
>>
>> Another thing I would try is to enable debug (or show/paste the
>> traceback) so that tracebacks are immediately available in the output:
>>
>> CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
>>
>> I'll follow up in the tracker ticket
>> >
>> > This makes 14.2.3 unusable for us as we need to re-activate all OSDs
>> > after reboots because we don't have a persistent system disk.
>> >
>> >
>> > Paul
>> >
>> > --
>> > Paul Emmerich
>> >
>> > Looking for help with your Ceph cluster? Contact us at https://croit.io
>> >
>> > croit GmbH
>> > Freseniusstr. 31h
>> > 81247 München
>> > www.croit.io
>> > Tel: +49 89 1896585 90
>> _______________________________________________
>> Dev mailing list -- dev(a)ceph.io
>> To unsubscribe send an email to dev-leave(a)ceph.io
2:56 p.m.
While we try to fix this, in the meantime the only workaround is not
to redirect stderr. This is far from ideal if you require redirection,
but so far is the only workaround to avoid this problem.
On Wed, Sep 4, 2019 at 7:54 PM Alfredo Deza <adeza(a)redhat.com> wrote:
>
> On Wed, Sep 4, 2019 at 6:35 PM Sasha Litvak
> <alexander.v.litvak(a)gmail.com> wrote:
> >
> > How do you fix it? Or you wait till 14.2.4?
>
> This is a high priority for me, I will provide a fix as soon as
> possible and hopefully a workaround.
>
> >
> > On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza(a)redhat.com> wrote:
> >>
> >> On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich(a)croit.io>
wrote:
> >> >
> >> > Hi,
> >> >
> >> > see https://tracker.ceph.com/issues/41660
> >> >
> >> > ceph-volume lvm activate --all fails on the second OSD when stderr is
> >> > not a terminal.
> >> > Reproducible on different servers, so there's nothing weird about
a
> >> > particular disk.
> >> >
> >> > Any idea where/how this is happening?
> >>
> >> That looks very odd, haven't seen it other than a unit test we have
> >> that fails in some machines. I was just investigating that today.
> >>
> >> Is it possible that the locale is set to something that is not
> >> en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
> >>
> >> Another thing I would try is to enable debug (or show/paste the
> >> traceback) so that tracebacks are immediately available in the output:
> >>
> >> CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
> >>
> >> I'll follow up in the tracker ticket
> >> >
> >> > This makes 14.2.3 unusable for us as we need to re-activate all OSDs
> >> > after reboots because we don't have a persistent system disk.
> >> >
> >> >
> >> > Paul
> >> >
> >> > --
> >> > Paul Emmerich
> >> >
> >> > Looking for help with your Ceph cluster? Contact us at
https://croit.io
> >> >
> >> > croit GmbH
> >> > Freseniusstr. 31h
> >> > 81247 München
> >> > www.croit.io
> >> > Tel: +49 89 1896585 90
> >> _______________________________________________
> >> Dev mailing list -- dev(a)ceph.io
> >> To unsubscribe send an email to dev-leave(a)ceph.io
6 Sep
6 Sep
8:24 a.m.
Alfredo,
I have seen that you posted a fix. Is this becomes a part of the standard
package update or I need to custom build it? I am running clusters with
podman and docker.
On Thu, Sep 5, 2019, 6:56 AM Alfredo Deza <adeza(a)redhat.com> wrote:
While we try to fix this, in the meantime the only
workaround is not
to redirect stderr. This is far from ideal if you require redirection,
but so far is the only workaround to avoid this problem.
On Wed, Sep 4, 2019 at 7:54 PM Alfredo Deza <adeza(a)redhat.com> wrote:
On Wed, Sep 4, 2019 at 6:35 PM Sasha Litvak
<alexander.v.litvak(a)gmail.com> wrote:
wrote:
How do you fix it? Or you wait till 14.2.4?
This is a high priority for me, I will provide a fix as soon as
possible and hopefully a workaround.
>
> On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza(a)redhat.com> wrote:
>>
>> On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich(a)croit.io>
>> >
>> > Hi,
>> >
>> > see https://tracker.ceph.com/issues/41660
>> >
>> > ceph-volume lvm activate --all fails on the second OSD when stderr
is
>> > not a terminal.
>> > Reproducible on different servers, so there's nothing weird about a
>> > particular disk.
>> >
>> > Any idea where/how this is happening?
>>
>> That looks very odd, haven't seen it other than a unit test we have
>> that fails in some machines. I was just investigating that today.
>>
>> Is it possible that the locale is set to something that is not
>> en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
>>
>> Another thing I would try is to enable debug (or show/paste the
>> traceback) so that tracebacks are immediately available in the output:
>>
>> CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
>>
>> I'll follow up in the tracker ticket
>> >
>> > This makes 14.2.3 unusable for us as we need to re-activate all OSDs
>> > after reboots because we don't have a persistent system disk.
>> >
>> >
>> > Paul
>> >
>> > --
>> > Paul Emmerich
>> >
>> > Looking for help with your Ceph cluster? Contact us at
https://croit.io
>> >
>> > croit GmbH
>> > Freseniusstr. 31h
>> > 81247 München
>> > www.croit.io
>> > Tel: +49 89 1896585 90
>> _______________________________________________
>> Dev mailing list -- dev(a)ceph.io
>> To unsubscribe send an email to dev-leave(a)ceph.io
11:22 a.m.
Am 05.09.19 um 13:56 schrieb Alfredo Deza:
While we try to fix this, in the meantime the only
workaround is not
to redirect stderr. This is far from ideal if you require redirection,
but so far is the only workaround to avoid this problem.
This bug also broke our ceph-deploy based installation scripts, even
with a single OSD. I can confirm that the fix you posted makes
ceph-deploy work for us again.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Werner-Voß-Damm 62 Fax: +49 30 99296856
12101 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
3:08 p.m.
Thank you all for confirming the fix, I'll follow up today with a
proper pull request, but I can't tell when we can get a release once
it lands in the various release branches.
On Fri, Sep 6, 2019 at 4:23 AM Amon Ott <a.ott(a)m-privacy.de> wrote:
>
> Am 05.09.19 um 13:56 schrieb Alfredo Deza:
> > While we try to fix this, in the meantime the only workaround is not
> > to redirect stderr. This is far from ideal if you require redirection,
> > but so far is the only workaround to avoid this problem.
>
> This bug also broke our ceph-deploy based installation scripts, even
> with a single OSD. I can confirm that the fix you posted makes
> ceph-deploy work for us again.
>
> Amon Ott
> --
> Dr. Amon Ott
> m-privacy GmbH Tel: +49 30 24342334
> Werner-Voß-Damm 62 Fax: +49 30 99296856
> 12101 Berlin http://www.m-privacy.de
>
> Amtsgericht Charlottenburg, HRB 84946
>
> Geschäftsführer:
> Dipl.-Kfm. Holger Maczkowsky,
> Roman Maczkowsky
>
> GnuPG-Key-ID: 0x2DD3A649
>
7 Sep
7 Sep
12:39 a.m.
our debian repository now contains packages with the fix:
https://croit.io/2019/07/07/2019-07-07-debian-mirror
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 2:08 PM Alfredo Deza <adeza(a)redhat.com> wrote:
>
> Thank you all for confirming the fix, I'll follow up today with a
> proper pull request, but I can't tell when we can get a release once
> it lands in the various release branches.
>
> On Fri, Sep 6, 2019 at 4:23 AM Amon Ott <a.ott(a)m-privacy.de> wrote:
> >
> > Am 05.09.19 um 13:56 schrieb Alfredo Deza:
> > > While we try to fix this, in the meantime the only workaround is not
> > > to redirect stderr. This is far from ideal if you require redirection,
> > > but so far is the only workaround to avoid this problem.
> >
> > This bug also broke our ceph-deploy based installation scripts, even
> > with a single OSD. I can confirm that the fix you posted makes
> > ceph-deploy work for us again.
> >
> > Amon Ott
> > --
> > Dr. Amon Ott
> > m-privacy GmbH Tel: +49 30 24342334
> > Werner-Voß-Damm 62 Fax: +49 30 99296856
> > 12101 Berlin http://www.m-privacy.de
> >
> > Amtsgericht Charlottenburg, HRB 84946
> >
> > Geschäftsführer:
> > Dipl.-Kfm. Holger Maczkowsky,
> > Roman Maczkowsky
> >
> > GnuPG-Key-ID: 0x2DD3A649
> >
5:25 a.m.
Now if you guys would only build docker images :-)
On Fri, Sep 6, 2019, 4:39 PM Paul Emmerich <paul.emmerich(a)croit.io> wrote:
our debian repository now contains packages with the
fix:
https://croit.io/2019/07/07/2019-07-07-debian-mirror
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 2:08 PM Alfredo Deza <adeza(a)redhat.com> wrote:
Thank you all for confirming the fix, I'll follow up today with a
proper pull request, but I can't tell when we can get a release once
it lands in the various release branches.
On Fri, Sep 6, 2019 at 4:23 AM Amon Ott <a.ott(a)m-privacy.de> wrote:
>
> Am 05.09.19 um 13:56 schrieb Alfredo Deza:
> > While we try to fix this, in the meantime the only workaround is not
> > to redirect stderr. This is far from ideal if you require
redirection,
> > but so far is the only workaround to
avoid this problem.
>
> This bug also broke our ceph-deploy based installation scripts, even
> with a single OSD. I can confirm that the fix you posted makes
> ceph-deploy work for us again.
>
> Amon Ott
> --
> Dr. Amon Ott
> m-privacy GmbH Tel: +49 30 24342334
> Werner-Voß-Damm 62 Fax: +49 30 99296856
> 12101 Berlin http://www.m-privacy.de
>
> Amtsgericht Charlottenburg, HRB 84946
>
> Geschäftsführer:
> Dipl.-Kfm. Holger Maczkowsky,
> Roman Maczkowsky
>
> GnuPG-Key-ID: 0x2DD3A649
>
1704
days inactive
1707
days old
9 comments
4 participants
participants (4)
-
Alfredo Deza -
Amon Ott -
Paul Emmerich -
Sasha Litvak