ceph-volume lvm activate --all broken in 14.2.3
Hi, see https://tracker.ceph.com/issues/41660 ceph-volume lvm activate --all fails on the second OSD when stderr is not a terminal. Reproducible on different servers, so there's nothing weird about a particular disk. Any idea where/how this is happening? This makes 14.2.3 unusable for us as we need to re-activate all OSDs after reboots because we don't have a persistent system disk. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr is not a terminal. Reproducible on different servers, so there's nothing weird about a particular disk.
Any idea where/how this is happening?
That looks very odd, haven't seen it other than a unit test we have that fails in some machines. I was just investigating that today. Is it possible that the locale is set to something that is not en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C Another thing I would try is to enable debug (or show/paste the traceback) so that tracebacks are immediately available in the output: CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all I'll follow up in the tracker ticket
This makes 14.2.3 unusable for us as we need to re-activate all OSDs after reboots because we don't have a persistent system disk.
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
How do you fix it? Or you wait till 14.2.4? On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza@redhat.com> wrote:
On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr is not a terminal. Reproducible on different servers, so there's nothing weird about a particular disk.
Any idea where/how this is happening?
That looks very odd, haven't seen it other than a unit test we have that fails in some machines. I was just investigating that today.
Is it possible that the locale is set to something that is not en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
Another thing I would try is to enable debug (or show/paste the traceback) so that tracebacks are immediately available in the output:
CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
I'll follow up in the tracker ticket
This makes 14.2.3 unusable for us as we need to re-activate all OSDs after reboots because we don't have a persistent system disk.
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wed, Sep 4, 2019 at 6:35 PM Sasha Litvak <alexander.v.litvak@gmail.com> wrote:
How do you fix it? Or you wait till 14.2.4?
This is a high priority for me, I will provide a fix as soon as possible and hopefully a workaround.
On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza@redhat.com> wrote:
On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr is not a terminal. Reproducible on different servers, so there's nothing weird about a particular disk.
Any idea where/how this is happening?
That looks very odd, haven't seen it other than a unit test we have that fails in some machines. I was just investigating that today.
Is it possible that the locale is set to something that is not en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
Another thing I would try is to enable debug (or show/paste the traceback) so that tracebacks are immediately available in the output:
CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
I'll follow up in the tracker ticket
This makes 14.2.3 unusable for us as we need to re-activate all OSDs after reboots because we don't have a persistent system disk.
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
While we try to fix this, in the meantime the only workaround is not to redirect stderr. This is far from ideal if you require redirection, but so far is the only workaround to avoid this problem. On Wed, Sep 4, 2019 at 7:54 PM Alfredo Deza <adeza@redhat.com> wrote:
On Wed, Sep 4, 2019 at 6:35 PM Sasha Litvak <alexander.v.litvak@gmail.com> wrote:
How do you fix it? Or you wait till 14.2.4?
This is a high priority for me, I will provide a fix as soon as possible and hopefully a workaround.
On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza@redhat.com> wrote:
On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr is not a terminal. Reproducible on different servers, so there's nothing weird about a particular disk.
Any idea where/how this is happening?
That looks very odd, haven't seen it other than a unit test we have that fails in some machines. I was just investigating that today.
Is it possible that the locale is set to something that is not en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
Another thing I would try is to enable debug (or show/paste the traceback) so that tracebacks are immediately available in the output:
CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
I'll follow up in the tracker ticket
This makes 14.2.3 unusable for us as we need to re-activate all OSDs after reboots because we don't have a persistent system disk.
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Alfredo, I have seen that you posted a fix. Is this becomes a part of the standard package update or I need to custom build it? I am running clusters with podman and docker. On Thu, Sep 5, 2019, 6:56 AM Alfredo Deza <adeza@redhat.com> wrote:
While we try to fix this, in the meantime the only workaround is not to redirect stderr. This is far from ideal if you require redirection, but so far is the only workaround to avoid this problem.
On Wed, Sep 4, 2019 at 7:54 PM Alfredo Deza <adeza@redhat.com> wrote:
On Wed, Sep 4, 2019 at 6:35 PM Sasha Litvak <alexander.v.litvak@gmail.com> wrote:
How do you fix it? Or you wait till 14.2.4?
This is a high priority for me, I will provide a fix as soon as possible and hopefully a workaround.
On Wed, Sep 4, 2019, 3:38 PM Alfredo Deza <adeza@redhat.com> wrote:
On Wed, Sep 4, 2019 at 4:01 PM Paul Emmerich <paul.emmerich@croit.io>
wrote:
Hi,
see https://tracker.ceph.com/issues/41660
ceph-volume lvm activate --all fails on the second OSD when stderr
is
not a terminal. Reproducible on different servers, so there's nothing weird about a particular disk.
Any idea where/how this is happening?
That looks very odd, haven't seen it other than a unit test we have that fails in some machines. I was just investigating that today.
Is it possible that the locale is set to something that is not en_US.UTF-8 ? I was able to replicate some failures with LC_ALL=C
Another thing I would try is to enable debug (or show/paste the traceback) so that tracebacks are immediately available in the output:
CEPH_VOLUME_DEBUG=1 ceph-volume lvm activate --all
I'll follow up in the tracker ticket
This makes 14.2.3 unusable for us as we need to re-activate all OSDs after reboots because we don't have a persistent system disk.
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Am 05.09.19 um 13:56 schrieb Alfredo Deza:
While we try to fix this, in the meantime the only workaround is not to redirect stderr. This is far from ideal if you require redirection, but so far is the only workaround to avoid this problem.
This bug also broke our ceph-deploy based installation scripts, even with a single OSD. I can confirm that the fix you posted makes ceph-deploy work for us again. Amon Ott -- Dr. Amon Ott m-privacy GmbH Tel: +49 30 24342334 Werner-Voß-Damm 62 Fax: +49 30 99296856 12101 Berlin http://www.m-privacy.de Amtsgericht Charlottenburg, HRB 84946 Geschäftsführer: Dipl.-Kfm. Holger Maczkowsky, Roman Maczkowsky GnuPG-Key-ID: 0x2DD3A649
Thank you all for confirming the fix, I'll follow up today with a proper pull request, but I can't tell when we can get a release once it lands in the various release branches. On Fri, Sep 6, 2019 at 4:23 AM Amon Ott <a.ott@m-privacy.de> wrote:
Am 05.09.19 um 13:56 schrieb Alfredo Deza:
While we try to fix this, in the meantime the only workaround is not to redirect stderr. This is far from ideal if you require redirection, but so far is the only workaround to avoid this problem.
This bug also broke our ceph-deploy based installation scripts, even with a single OSD. I can confirm that the fix you posted makes ceph-deploy work for us again.
Amon Ott -- Dr. Amon Ott m-privacy GmbH Tel: +49 30 24342334 Werner-Voß-Damm 62 Fax: +49 30 99296856 12101 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer: Dipl.-Kfm. Holger Maczkowsky, Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
our debian repository now contains packages with the fix: https://croit.io/2019/07/07/2019-07-07-debian-mirror -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Sep 6, 2019 at 2:08 PM Alfredo Deza <adeza@redhat.com> wrote:
Thank you all for confirming the fix, I'll follow up today with a proper pull request, but I can't tell when we can get a release once it lands in the various release branches.
On Fri, Sep 6, 2019 at 4:23 AM Amon Ott <a.ott@m-privacy.de> wrote:
Am 05.09.19 um 13:56 schrieb Alfredo Deza:
While we try to fix this, in the meantime the only workaround is not to redirect stderr. This is far from ideal if you require redirection, but so far is the only workaround to avoid this problem.
This bug also broke our ceph-deploy based installation scripts, even with a single OSD. I can confirm that the fix you posted makes ceph-deploy work for us again.
Amon Ott -- Dr. Amon Ott m-privacy GmbH Tel: +49 30 24342334 Werner-Voß-Damm 62 Fax: +49 30 99296856 12101 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer: Dipl.-Kfm. Holger Maczkowsky, Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
Now if you guys would only build docker images :-) On Fri, Sep 6, 2019, 4:39 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
our debian repository now contains packages with the fix: https://croit.io/2019/07/07/2019-07-07-debian-mirror
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 2:08 PM Alfredo Deza <adeza@redhat.com> wrote:
Thank you all for confirming the fix, I'll follow up today with a proper pull request, but I can't tell when we can get a release once it lands in the various release branches.
On Fri, Sep 6, 2019 at 4:23 AM Amon Ott <a.ott@m-privacy.de> wrote:
Am 05.09.19 um 13:56 schrieb Alfredo Deza:
While we try to fix this, in the meantime the only workaround is not to redirect stderr. This is far from ideal if you require
redirection,
but so far is the only workaround to avoid this problem.
This bug also broke our ceph-deploy based installation scripts, even with a single OSD. I can confirm that the fix you posted makes ceph-deploy work for us again.
Amon Ott -- Dr. Amon Ott m-privacy GmbH Tel: +49 30 24342334 Werner-Voß-Damm 62 Fax: +49 30 99296856 12101 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer: Dipl.-Kfm. Holger Maczkowsky, Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
participants (4)
-
Alfredo Deza
-
Amon Ott
-
Paul Emmerich
-
Sasha Litvak