tcmalloc v2.8 and bad_alloc exceptions
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.) Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up. It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987...
On 25-2-2021 20:59, Casey Bodley wrote:
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.)
Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up.
It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987... Ah, great, thanx for reporting that back.
A while back I noticed that all my tests where crashing in malloc code. And the date coincided with a full package upgrade I did, and it did include new google-perf-tools. So I reverted that change and went on with my life, thinking that it was a sort of cosmic incompatibility hickup. Never gave it much tought, let alone to report it here. I'll see if there is a 2.8.1 port available, and try with that --WjW
Thanks Casey. I'm tracking this for EPEL 8 at https://bugzilla.redhat.com/show_bug.cgi?id=1933792 Kaleb's also filed https://bugzilla.redhat.com/show_bug.cgi?id=1933756 for Fedora 33. - Ken On Thu, Feb 25, 2021 at 1:01 PM Casey Bodley <cbodley@redhat.com> wrote:
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.)
Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up.
It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987... _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
thanks, ken Matt On Mon, Mar 1, 2021 at 1:14 PM Ken Dreyer <kdreyer@redhat.com> wrote:
Thanks Casey.
I'm tracking this for EPEL 8 at https://bugzilla.redhat.com/show_bug.cgi?id=1933792
Kaleb's also filed https://bugzilla.redhat.com/show_bug.cgi?id=1933756 for Fedora 33.
- Ken
On Thu, Feb 25, 2021 at 1:01 PM Casey Bodley <cbodley@redhat.com> wrote:
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.)
Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up.
It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987... _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
I've submitted gperftools-2.8.1 for Fedora 33 at https://bodhi.fedoraproject.org/updates/FEDORA-2021-67257daa36 I've submitted gperftools-2.7-8 for EPEL 8, where Yaakov has backported the s390x and ppc64le fixes specifically for 2.7. Thanks Yaakov. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-0eda4297eb Please test these versions and provide feedback in Bodhi so the updates can go to the stable repos. - Ken On Mon, Mar 1, 2021 at 11:13 AM Ken Dreyer <kdreyer@redhat.com> wrote:
Thanks Casey.
I'm tracking this for EPEL 8 at https://bugzilla.redhat.com/show_bug.cgi?id=1933792
Kaleb's also filed https://bugzilla.redhat.com/show_bug.cgi?id=1933756 for Fedora 33.
- Ken
On Thu, Feb 25, 2021 at 1:01 PM Casey Bodley <cbodley@redhat.com> wrote:
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.)
Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up.
It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987... _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
are these not resolved in teuthology? i'm still seeing them. for example, on smiths: On Tue, Mar 2, 2021 at 3:57 PM Ken Dreyer <kdreyer@redhat.com> wrote:
I've submitted gperftools-2.8.1 for Fedora 33 at https://bodhi.fedoraproject.org/updates/FEDORA-2021-67257daa36
I've submitted gperftools-2.7-8 for EPEL 8, where Yaakov has backported the s390x and ppc64le fixes specifically for 2.7. Thanks Yaakov. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-0eda4297eb
Please test these versions and provide feedback in Bodhi so the updates can go to the stable repos.
- Ken
On Mon, Mar 1, 2021 at 11:13 AM Ken Dreyer <kdreyer@redhat.com> wrote:
Thanks Casey.
I'm tracking this for EPEL 8 at https://bugzilla.redhat.com/show_bug.cgi?id=1933792
Kaleb's also filed https://bugzilla.redhat.com/show_bug.cgi?id=1933756 for Fedora 33.
- Ken
On Thu, Feb 25, 2021 at 1:01 PM Casey Bodley <cbodley@redhat.com> wrote:
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.)
Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up.
It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987... _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
for example, on smithis: http://qa-proxy.ceph.com/teuthology/cbodley-2021-03-16_17:32:34-rgw-wip-cbod... On Tue, Mar 16, 2021 at 3:16 PM Casey Bodley <cbodley@redhat.com> wrote:
are these not resolved in teuthology? i'm still seeing them. for example, on smiths:
On Tue, Mar 2, 2021 at 3:57 PM Ken Dreyer <kdreyer@redhat.com> wrote:
I've submitted gperftools-2.8.1 for Fedora 33 at https://bodhi.fedoraproject.org/updates/FEDORA-2021-67257daa36
I've submitted gperftools-2.7-8 for EPEL 8, where Yaakov has backported the s390x and ppc64le fixes specifically for 2.7. Thanks Yaakov. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-0eda4297eb
Please test these versions and provide feedback in Bodhi so the updates can go to the stable repos.
- Ken
On Mon, Mar 1, 2021 at 11:13 AM Ken Dreyer <kdreyer@redhat.com> wrote:
Thanks Casey.
I'm tracking this for EPEL 8 at https://bugzilla.redhat.com/show_bug.cgi?id=1933792
Kaleb's also filed https://bugzilla.redhat.com/show_bug.cgi?id=1933756 for Fedora 33.
- Ken
On Thu, Feb 25, 2021 at 1:01 PM Casey Bodley <cbodley@redhat.com> wrote:
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.)
Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up.
It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987... _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Since we have tester feedback at https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-0eda4297eb I pushed that to the stable EPEL repo now. On Tue, Mar 16, 2021 at 1:17 PM Casey Bodley <cbodley@redhat.com> wrote:
for example, on smithis: http://qa-proxy.ceph.com/teuthology/cbodley-2021-03-16_17:32:34-rgw-wip-cbod...
On Tue, Mar 16, 2021 at 3:16 PM Casey Bodley <cbodley@redhat.com> wrote:
are these not resolved in teuthology? i'm still seeing them. for example, on smiths:
On Tue, Mar 2, 2021 at 3:57 PM Ken Dreyer <kdreyer@redhat.com> wrote:
I've submitted gperftools-2.8.1 for Fedora 33 at https://bodhi.fedoraproject.org/updates/FEDORA-2021-67257daa36
I've submitted gperftools-2.7-8 for EPEL 8, where Yaakov has backported the s390x and ppc64le fixes specifically for 2.7. Thanks Yaakov. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-0eda4297eb
Please test these versions and provide feedback in Bodhi so the updates can go to the stable repos.
- Ken
On Mon, Mar 1, 2021 at 11:13 AM Ken Dreyer <kdreyer@redhat.com> wrote:
Thanks Casey.
I'm tracking this for EPEL 8 at https://bugzilla.redhat.com/show_bug.cgi?id=1933792
Kaleb's also filed https://bugzilla.redhat.com/show_bug.cgi?id=1933756 for Fedora 33.
- Ken
On Thu, Feb 25, 2021 at 1:01 PM Casey Bodley <cbodley@redhat.com> wrote:
re: https://tracker.ceph.com/issues/49387 (rgw) and https://tracker.ceph.com/issues/49240 (mgr etc.)
Mark and Yuval have shown that these crashes reproduce on fedora 33 only after tcmalloc is upgraded from 2.7 to 2.8. I'm guessing that upgrade was installed on the smithis and gibbas somewhere around the beginning of February when these crashes started showing up.
It looks like there was a regression in 2.8 that was reverted in 2.8.1: https://github.com/gperftools/gperftools/commit/02d5264018cc76a36713f9732987... _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (4)
-
Casey Bodley
-
Ken Dreyer
-
Matt Benjamin
-
Willem Jan Withagen