Debian packages for Reef - any chance of reviews / builds?
Hi, When Reef was released, the announcement said that Debian packages would be built once the blocking bug in Bookworm was fixed. As I noted on the tracker item https://tracker.ceph.com/issues/61845 a couple of weeks ago, that is now the case after the most recent Bookworm point release. I also opened a PR to make the minimal change that would build Reef packages on Bookworm[0]. I subsequently opened another PR to fix some low-hanging fruit in terms of packaging errors - missing #! in maintscripts, syntax errors in debian/control, erroneous dependencies on Essential packages[1]. Neither PR has had any feedback/review as far as I can see. Those packages (and the previous state of the debian/ tree) had some significant problems - no copyright file, and some of them contain python scripts without declaring a python dependency, so I've today submitted a slightly larger PR that brings the dh compatibility level up to what I think the latest lowest-common-denominator level is, as well as fixing these errors[2]. I believe these changes all ought to go into the reef branch, but obviously you might prefer to just make the bare-minimum-to-build change in the first PR. Is there any chance of having some reef packages for Bookworm, please? Relatedly, is there interest in further packaging fixes for future branches? lintian still has quite a lot to say about the .debs for Ceph, and while you might reasonably not want to care about crossing every t of Debian policy, I think there are still changes that would be worth doing... I should declare a bit of an interest here - I'd like to evaluate cephadm for work use, which would require us to be able to build our own packages per local policy[3], which in turn would mean I'd want to get Debian-based images going again. But that requires Reef .debs being available to install onto said images :) Thanks, Matthew [0] https://github.com/ceph/ceph/pull/53342 [1] https://github.com/ceph/ceph/pull/53397 [2] https://github.com/ceph/ceph/pull/53546 [3] https://wikitech.wikimedia.org/wiki/Kubernetes/Images#Production_images
Belated reply — we actually discussed this at last week’s CLT, but neglected to respond directly. Thanks for the patches, Matthew! One of the challenges with building Debian regularly is that nobody involved in Ceph’s technical leadership is also involved enough with Debian to track these issues, so our support is always on a best-effort basis, and we’ve leaned on others to do packaging reviews and keep us aligned. I was glad to see Kefu reviewed and merged your patches over the weekend, as it meant I didn’t need to do so blindly. :) We’ll discuss building releases at the next meeting. Also, I’d like to connect you and Thomas, who has been working on Debian builds in irc again and maintains Ceph packages in Debian proper. -Greg On Wed, Sep 20, 2023 at 6:14 AM Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi,
When Reef was released, the announcement said that Debian packages would be built once the blocking bug in Bookworm was fixed. As I noted on the tracker item https://tracker.ceph.com/issues/61845 a couple of weeks ago, that is now the case after the most recent Bookworm point release.
I also opened a PR to make the minimal change that would build Reef packages on Bookworm[0]. I subsequently opened another PR to fix some low-hanging fruit in terms of packaging errors - missing #! in maintscripts, syntax errors in debian/control, erroneous dependencies on Essential packages[1]. Neither PR has had any feedback/review as far as I can see.
Those packages (and the previous state of the debian/ tree) had some significant problems - no copyright file, and some of them contain python scripts without declaring a python dependency, so I've today submitted a slightly larger PR that brings the dh compatibility level up to what I think the latest lowest-common-denominator level is, as well as fixing these errors[2].
I believe these changes all ought to go into the reef branch, but obviously you might prefer to just make the bare-minimum-to-build change in the first PR.
Is there any chance of having some reef packages for Bookworm, please?
Relatedly, is there interest in further packaging fixes for future branches? lintian still has quite a lot to say about the .debs for Ceph, and while you might reasonably not want to care about crossing every t of Debian policy, I think there are still changes that would be worth doing...
I should declare a bit of an interest here - I'd like to evaluate cephadm for work use, which would require us to be able to build our own packages per local policy[3], which in turn would mean I'd want to get Debian-based images going again. But that requires Reef .debs being available to install onto said images :)
Thanks,
Matthew
[0] https://github.com/ceph/ceph/pull/53342 [1] https://github.com/ceph/ceph/pull/53397 [2] https://github.com/ceph/ceph/pull/53546 [3] https://wikitech.wikimedia.org/wiki/Kubernetes/Images#Production_images _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 9/27/23 17:05, Gregory Farnum wrote:
Also, I’d like to connect you and Thomas, who has been working on Debian builds in irc again and maintains Ceph packages in Debian proper. -Greg
Hi there! Thanks Gregory for getting in touch. I also CC-ed Bzed, who's co-maintaining Ceph with me (even though I was alone for the last 2 years or so, I still have hope that Bzed helps). My mission is probably a bit different from what Matthew is doing. Trying to maintain Ceph in Debian has to be done in Unstable/Testing, which is a way more challenging than building for one's own unofficial backport repository for amd64 alone. For example, I'm currently fighting to get Ceph Pacific fixed for SID and RiscV (we believe we have a solution for it, ie: using -latomic). See here, it still doesn't build: https://buildd.debian.org/status/package.php?p=ceph&suite=sid At the same time, I'm trying to get Reef to build in Experimental: https://buildd.debian.org/status/package.php?p=ceph&suite=experimental As you may see, it currently fails under all 32 bits arch. The issue is, we need Ceph for these arch too, so that Qemu can be built against librbd, otherwise all 32 bits arch wont get RBD support. Lucky, Mipsel was removed, so at least, we have 3GB of RAM to build, not 2. I'm not so much interested in packaging in non-official Debian, or even upstream. In fact, I'd be for removing the Debian folder from the Ceph upstream master branch, as this is causing issues on the downstream distributions (ie: I have to remove the debian folder from the orig tarball). Best would be if we could all (wikimedia, Ubuntu, Debian) share the same packaging branch on top of master or stable branches of Ceph, and somehow build a CI to automatically build packages. I have enough experience to write such a CI (which I do already for OpenStack). Though it'd probably be nicer to have this done in the upstream infrastructure, so that we could build Debian and Ubuntu packages for stable AND unstable (Ubuntu/Debian, but also Ceph), aso we see things break. My Debian packaging is also a way more Debian policy compliant than what I saw from upstream. For example, I noticed that during the build, Ceph build Arrow, which in its turn, download xsimd (and of course, downloading at build time is strictly forbidden in Debian), so I had to add "extraopts += -DWITH_RADOSGW_SELECT_PARQUET=OFF" until we get arrow properly packaged in Debian. I also fixed many of the things you fixed in the merge request from this week-end. I very much would love to get more help on the packaging, and do it with you Matthew. Note that I already work with Arturo Borrero Gonzalez from Wikimedia as well, as I am also the package maintainer for all things OpenStack (which is why I am interested in Ceph). My current plan is to finish the packaging of Ceph Reef in Experimental, which means, make it build for all arch in the Debian buildd, before opening a transition bug to the release team and upload to Unstable (at which point, the advise is to attempt rebuilding all reverse depends and file bugs as necessary). Once the package reaches Unstable, and then Testing/Trixie, I want to also upload Reef to official bookworm-backports. Our current production plan is to run Ceph on bookworm+arm64 for a new region of our public cloud. Hopefully, this can be done on Reef, using the official bookworm backports I described above. I can forecast a lot of fun testing all that in a real production environment... :) FYI, we want to use HPE's RL300 servers (though if anyone has other suggestions for ARM hardware and Ceph, please let me know). The Ceph packaging is maintained in Salsa, like almost all of Debian: https://salsa.debian.org/ceph-team/ceph/ There's also a lot of work that should be done in order to de-embed many of the Ceph included libraries: - jerasure + isal (I already packaged this in Debian) - arrow (there's a start of a package from the Debian science team [1]) - uring (Ceph Reef currently fails with the Debian version) - boost (same thing... we need to transition to 1.8.2 in Unstable) This is a lot of work, and really, too much for only myself, who's also in charge of OpenStack, OpenVSwitch/OVN, RabbitMQ, and many more in Debian. So, Matthew, would you like to do packaging with me directly in Debian? If so, I warmly welcome you in the #debian-openstack IRC channel in OFTC if you would like to kickstart this (or we continue by email if you prefer). Cheers, Thomas Goirand (zigo) [1] See https://bugs.debian.org/970021 and https://salsa.debian.org/science-team/arrow This start of packaging would need to be upgraded to version 9.0.0 *at least* (we got to figure out if Ceph can cope with an even newer version), and must make sure it doesn't download xsimd at build time. Probably, some of the arrow (build-)depends should also be packaged separately. P.S: I'm not registered to this list (I'm registered to a way too many lists...), so please do CC me.
Hi Thomas, I don't really have the time to help maintaining it, sorry. But regarding ceph for 32bit: please drop it. It might compile, but it's not supported by upstream and as long as you can't run the full test suite it doesn't make sense to risk it. Last time I've checked it there were lots assumptions on 64bit in the code that were not even detected by the compiler on 32bit. It will also save you a lot of work... Bernd 28.09.2023 09:11:32 Thomas Goirand <zigo@debian.org>:
On 9/27/23 17:05, Gregory Farnum wrote:
Also, I’d like to connect you and Thomas, who has been working on Debian builds in irc again and maintains Ceph packages in Debian proper. -Greg
Hi there!
Thanks Gregory for getting in touch.
I also CC-ed Bzed, who's co-maintaining Ceph with me (even though I was alone for the last 2 years or so, I still have hope that Bzed helps).
My mission is probably a bit different from what Matthew is doing. Trying to maintain Ceph in Debian has to be done in Unstable/Testing, which is a way more challenging than building for one's own unofficial backport repository for amd64 alone. For example, I'm currently fighting to get Ceph Pacific fixed for SID and RiscV (we believe we have a solution for it, ie: using -latomic). See here, it still doesn't build: https://buildd.debian.org/status/package.php?p=ceph&suite=sid
At the same time, I'm trying to get Reef to build in Experimental: https://buildd.debian.org/status/package.php?p=ceph&suite=experimental
As you may see, it currently fails under all 32 bits arch. The issue is, we need Ceph for these arch too, so that Qemu can be built against librbd, otherwise all 32 bits arch wont get RBD support. Lucky, Mipsel was removed, so at least, we have 3GB of RAM to build, not 2.
I'm not so much interested in packaging in non-official Debian, or even upstream. In fact, I'd be for removing the Debian folder from the Ceph upstream master branch, as this is causing issues on the downstream distributions (ie: I have to remove the debian folder from the orig tarball). Best would be if we could all (wikimedia, Ubuntu, Debian) share the same packaging branch on top of master or stable branches of Ceph, and somehow build a CI to automatically build packages. I have enough experience to write such a CI (which I do already for OpenStack). Though it'd probably be nicer to have this done in the upstream infrastructure, so that we could build Debian and Ubuntu packages for stable AND unstable (Ubuntu/Debian, but also Ceph), aso we see things break.
My Debian packaging is also a way more Debian policy compliant than what I saw from upstream. For example, I noticed that during the build, Ceph build Arrow, which in its turn, download xsimd (and of course, downloading at build time is strictly forbidden in Debian), so I had to add "extraopts += -DWITH_RADOSGW_SELECT_PARQUET=OFF" until we get arrow properly packaged in Debian. I also fixed many of the things you fixed in the merge request from this week-end.
I very much would love to get more help on the packaging, and do it with you Matthew. Note that I already work with Arturo Borrero Gonzalez from Wikimedia as well, as I am also the package maintainer for all things OpenStack (which is why I am interested in Ceph).
My current plan is to finish the packaging of Ceph Reef in Experimental, which means, make it build for all arch in the Debian buildd, before opening a transition bug to the release team and upload to Unstable (at which point, the advise is to attempt rebuilding all reverse depends and file bugs as necessary). Once the package reaches Unstable, and then Testing/Trixie, I want to also upload Reef to official bookworm-backports.
Our current production plan is to run Ceph on bookworm+arm64 for a new region of our public cloud. Hopefully, this can be done on Reef, using the official bookworm backports I described above. I can forecast a lot of fun testing all that in a real production environment... :) FYI, we want to use HPE's RL300 servers (though if anyone has other suggestions for ARM hardware and Ceph, please let me know).
The Ceph packaging is maintained in Salsa, like almost all of Debian:
https://salsa.debian.org/ceph-team/ceph/
There's also a lot of work that should be done in order to de-embed many of the Ceph included libraries: - jerasure + isal (I already packaged this in Debian) - arrow (there's a start of a package from the Debian science team [1]) - uring (Ceph Reef currently fails with the Debian version) - boost (same thing... we need to transition to 1.8.2 in Unstable)
This is a lot of work, and really, too much for only myself, who's also in charge of OpenStack, OpenVSwitch/OVN, RabbitMQ, and many more in Debian.
So, Matthew, would you like to do packaging with me directly in Debian? If so, I warmly welcome you in the #debian-openstack IRC channel in OFTC if you would like to kickstart this (or we continue by email if you prefer).
Cheers,
Thomas Goirand (zigo)
[1] See https://bugs.debian.org/970021 and https://salsa.debian.org/science-team/arrow This start of packaging would need to be upgraded to version 9.0.0 *at least* (we got to figure out if Ceph can cope with an even newer version), and must make sure it doesn't download xsimd at build time. Probably, some of the arrow (build-)depends should also be packaged separately.
P.S: I'm not registered to this list (I'm registered to a way too many lists...), so please do CC me.
Hi, On 28/09/2023 08:11, Thomas Goirand wrote:
On 9/27/23 17:05, Gregory Farnum wrote:
Also, I’d like to connect you and Thomas, who has been working on Debian builds in irc again and maintains Ceph packages in Debian proper.
My mission is probably a bit different from what Matthew is doing.
Mmm. My Wikimedia hat wants to look at cephadm for our next Ceph cluster (rather than puppet&packages); that requires (because of local policy) images for Ceph that are built on Debian; so that really requires Ceph.io to provide at least .debs (and ideally the images too, but I need to get to that) to build them from. Also, I think having Ceph publish .debs is useful more generally.
Trying to maintain Ceph in Debian has to be done in Unstable/Testing, which is a way more challenging than building for one's own unofficial
My Debian developer hat is sympathetic to this problem, too (but doesn't have a lot of free time!). I think I'm inclined to agree with Bernd that it may not be a valuable use of time trying to get packages built for architectures that Ceph upstream don't support, particularly given how resource-hungry the build process is. And, obviously, if we can get debian/ under the upstream main branch into a better state, that makes everyone's lives easier - upstream packages build better, and distros don't have to re-do the packaging work.
My Debian packaging is also a way more Debian policy compliant than what I saw from upstream. For example, I noticed that during the build, Ceph
Yes; I was trying to do a minimum-viable-PR sort of job to try and get reef packages built again - so I fixed the most serious errors (like the lack of copyright file) where they could be done with minimal changes. I wasn't intending that to be the final state of packaging for main.
So, Matthew, would you like to do packaging with me directly in Debian? If so, I warmly welcome you in the #debian-openstack IRC channel in OFTC if you would like to kickstart this (or we continue by email if you prefer).
Particularly if we can arrange for packaging improvements to end up in Ceph main (and thus subsequent releases, and thus hopefully images), I may well be able to carve out some time to do so. [once there are Ceph Reef .debs available, I was going to look at the lintian output by way of next steps] Regards, Matthew
On 9/29/23 14:53, Matthew Vernon wrote:
Also, I think having Ceph publish .debs is useful more generally.
Sure! But I've been disappointed by it multiple times. Like when the Ceph upstream started to use feature of GCC that were not available at the time in Stable, and then Debian packages were given-up. It gave me the feeling that it couldn't be trusted. This time, it happened again for the Reef packages, that were not really usable in Debian. I'm really motivated to fix both the packages in Debian itself, and upstream too. I'm convince that we should work on both.
Trying to maintain Ceph in Debian has to be done in Unstable/Testing, which is a way more challenging than building for one's own unofficial
My Debian developer hat is sympathetic to this problem, too (but doesn't have a lot of free time!). I think I'm inclined to agree with Bernd that it may not be a valuable use of time trying to get packages built for architectures that Ceph upstream don't support, particularly given how resource-hungry the build process is.
I don't agree with this, especially considering we need librbd for many packages in Debian. I counted 10 packages in Debian that have build-depends on librbd-dev or librados-dev. If Ceph gets removed from Debian, then the support for Ceph by those packages is gone too...
And, obviously, if we can get debian/ under the upstream main branch into a better state, that makes everyone's lives easier - upstream packages build better, and distros don't have to re-do the packaging work.
Definitively, I'm all for getting all of the work I did merged at some point, and lower the differences as much as possible. I also think it's a waste of time if we work on things twice (upstream, like you did, and downstream, like I did).
My Debian packaging is also a way more Debian policy compliant than what I saw from upstream. For example, I noticed that during the build, Ceph
Yes; I was trying to do a minimum-viable-PR sort of job to try and get reef packages built again - so I fixed the most serious errors (like the lack of copyright file) where they could be done with minimal changes. I wasn't intending that to be the final state of packaging for main.
Hopefully, we can get to a state where the diff between the Debian package and the upstream Ceph one is very small.
So, Matthew, would you like to do packaging with me directly in Debian? If so, I warmly welcome you in the #debian-openstack IRC channel in OFTC if you would like to kickstart this (or we continue by email if you prefer).
Particularly if we can arrange for packaging improvements to end up in Ceph main (and thus subsequent releases, and thus hopefully images), I may well be able to carve out some time to do so.
Great! The question is: where to start. What I envisioned was fixing all I could on the Debian side of things first, as this wouldn't depend on having patches merged upstream, like you experimented (as I saw it, it looked kind of frustrating to see nobody seemed to care at first). I don't think I should change my workflow for the moment, and probably polishing things in Debian will get faster to the end result. But I'm opened to do differently if you think I should. Cheers, Thomas Goirand (zigo)
On Fri, 2023-09-29 at 16:48 +0200, Thomas Goirand wrote:
On 9/29/23 14:53, Matthew Vernon wrote:
My Debian developer hat is sympathetic to this problem, too (but doesn't have a lot of free time!). I think I'm inclined to agree with Bernd that it may not be a valuable use of time trying to get packages built for architectures that Ceph upstream don't support, particularly given how resource-hungry the build process is.
I don't agree with this, especially considering we need librbd for many packages in Debian. I counted 10 packages in Debian that have build-depends on librbd-dev or librados-dev. If Ceph gets removed from Debian, then the support for Ceph by those packages is gone too...
How many of these packages are really needed on a 32bit arch? Even if you don't like it, the reality is that - at least the last time I checked it - not even reading the default config values is possible with the current code base, and the Debian patch that fixed the compilation NEEDS to be removed as it will absolutely lead to memory errors on runtime. What you can try is to build librbd and librados on 32bit arches and ignore everything else. I'm not sure how much they need from the rest of the code base, but its much more reasonable to ship only them on 32bit instead of a definitely broken ceph.
Hopefully, we can get to a state where the diff between the Debian package and the upstream Ceph one is very small.
When I started looking into ceph I've removed lots of stuff from the debian folder that the ubuntu people added for unknown reasons, its much closer to upstream these days then it was before. Actually close enough that it is no problem to switch to the upstream packages at all. Cheers, Bernd -- Bernd Zeimetz Debian GNU/Linux Developer http://bzed.de http://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Sent from Workspace ONE Boxer On Sep 29, 2023 6:07 PM, Bernd Zeimetz <bzed@debian.org> wrote:
On Fri, 2023-09-29 at 16:48 +0200, Thomas Goirand wrote:
On 9/29/23 14:53, Matthew Vernon wrote:
My Debian developer hat is sympathetic to this problem, too (but
doesn't
have a lot of free time!). I think I'm inclined to agree with Bernd
that
it may not be a valuable use of time trying to get packages built
for
architectures that Ceph upstream don't support, particularly given
how
resource-hungry the build process is.
I don't agree with this, especially considering we need librbd for
many
packages in Debian. I counted 10 packages in Debian that have
build-depends on librbd-dev or librados-dev. If Ceph gets removed
from
Debian, then the support for Ceph by those packages is gone too...
How many of these packages are really needed on a 32bit arch?
Even if you don't like it, the reality is that - at least the last time
I checked it - not even reading the default config values is possible
with the current code base, and the Debian patch that fixed the
compilation NEEDS to be removed as it will absolutely lead to memory
errors on runtime.
What you can try is to build librbd and librados on 32bit arches and
ignore everything else. I'm not sure how much they need from the rest
of the code base, but its much more reasonable to ship only them on
32bit instead of a definitely broken ceph.
I don't have much time to reply right now, but today, I wrote bug reports to ask for 32 bits removal of Ceph support on the 10 packages that had librbd or librados build-depends. So I agree... I also don't have skills and time to reasonably build only libs for 32 bits. Contribs welcome...
When I started looking into ceph I've removed lots of stuff from the
debian folder that the ubuntu people added for unknown reasons, its
much closer to upstream these days then it was before. Actually close
enough that it is no problem to switch to the upstream packages at all.
Not entirely, though I did some work, diffing the 2, so that Debian and upstream packaging converges even more. At this point, IMO, I must invest some time on upstream packaging. Cheers, Thomas Goirand
hi, On Fri, 2023-09-29 at 18:28 +0200, thomas@goirand.fr wrote:
I don't have much time to reply right now, but today, I wrote bug reports to ask for 32 bits removal of Ceph support on the 10 packages that had librbd or librados build-depends. So I agree...
nice, thank you. I'm actually wondering if krbd should work just fine on 32bit instead of librbd? qemu at least supports it, so I think the loss should not too bad. Bernd -- Bernd Zeimetz Debian GNU/Linux Developer http://bzed.de http://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
On Fri, Sep 29, 2023 at 8:04 PM Bernd Zeimetz <bzed@debian.org> wrote:
hi,
On Fri, 2023-09-29 at 18:28 +0200, thomas@goirand.fr wrote:
I don't have much time to reply right now, but today, I wrote bug reports to ask for 32 bits removal of Ceph support on the 10 packages that had librbd or librados build-depends. So I agree...
nice, thank you.
I'm actually wondering if krbd should work just fine on 32bit instead of librbd? qemu at least supports it, so I think the loss should not too bad.
If krbd doesn't work, let me know! Like the rest of Ceph, it's not tested in 32-bit environments, but, given that krbd is self-contained, bugs (if any) should be trivial to fix. Owning to being part of the kernel, I assume that it at least builds cleanly. If someone wanted to contribute a test to the krbd test suite (e.g. involving a 32-bit VM image, based on Debian or otherwise), they would be very much welcome ;) Thanks, Ilya
On 10/2/23 13:40, Ilya Dryomov wrote:
On Fri, Sep 29, 2023 at 8:04 PM Bernd Zeimetz <bzed@debian.org> wrote:
hi,
On Fri, 2023-09-29 at 18:28 +0200, thomas@goirand.fr wrote:
I don't have much time to reply right now, but today, I wrote bug reports to ask for 32 bits removal of Ceph support on the 10 packages that had librbd or librados build-depends. So I agree...
nice, thank you.
I'm actually wondering if krbd should work just fine on 32bit instead of librbd? qemu at least supports it, so I think the loss should not too bad.
If krbd doesn't work, let me know! Like the rest of Ceph, it's not tested in 32-bit environments, but, given that krbd is self-contained, bugs (if any) should be trivial to fix. Owning to being part of the kernel, I assume that it at least builds cleanly.
If someone wanted to contribute a test to the krbd test suite (e.g. involving a 32-bit VM image, based on Debian or otherwise), they would be very much welcome ;)
Thanks,
Ilya
Hi Ilya, Do you have any idea how a project like Qemu can be built using krbd? Cheers, Thomas Goirand (zigo)
On Mon, Oct 2, 2023 at 3:17 PM Thomas Goirand <thomas@goirand.fr> wrote:
Hi Ilya,
Do you have any idea how a project like Qemu can be built using krbd?
QEMU doesn't need to be built specially. krbd gives you a /dev/rbd* block device which can be fed to QEMU just like e.g. /dev/sd*. Thanks, Ilya
On 9/29/23 11:01, Bernd Zeimetz wrote:
When I started looking into ceph I've removed lots of stuff from the debian folder that the ubuntu people added for unknown reasons, its much closer to upstream these days then it was before. Actually close enough that it is no problem to switch to the upstream packages at all.
Very much a side note: At once point the Ubuntu (and I think Debian) packages weren't being built with tcmalloc support. I talked to Chris Macnaughton about it back at Cephalocon and he verified it was the case for the Ubuntu packages, but he's moved on and is no longer at Canonical. It would be nice to verify if this is still (or for Debian ever was) the case and get it resolved. Mark -- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
Very much a side note: At once point the Ubuntu (and I think Debian) packages weren't being built with tcmalloc support. I talked to Chris Macnaughton about it back at Cephalocon and he verified it was the case for the Ubuntu packages, but he's moved on and is no longer at Canonical. It would be nice to verify if this is still (or for Debian ever was) the case and get it resolved.
Thanks again for bringing this to our attention. We resolved the build issue with tcmalloc in Ubuntu packages. The regression was introduced when Ubuntu enabled the crimson backend for testing [0]. This regression only impacted Ubuntu, not Debian. [0] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2016845/comments/4 On Mon, Oct 2, 2023 at 7:24 PM Mark Nelson <mark.nelson@clyso.com> wrote:
On 9/29/23 11:01, Bernd Zeimetz wrote:
When I started looking into ceph I've removed lots of stuff from the debian folder that the ubuntu people added for unknown reasons, its much closer to upstream these days then it was before. Actually close enough that it is no problem to switch to the upstream packages at all.
Very much a side note: At once point the Ubuntu (and I think Debian) packages weren't being built with tcmalloc support. I talked to Chris Macnaughton about it back at Cephalocon and he verified it was the case for the Ubuntu packages, but he's moved on and is no longer at Canonical. It would be nice to verify if this is still (or for Debian ever was) the case and get it resolved.
Mark
-- Best Regards, Mark Nelson Head of Research and Development
Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com
We are hiring: https://www.clyso.com/jobs/ _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Glad to hear it! FWIW, I noticed a pretty big improvement in VM performance with lbrbd when qemu was set to use tcmalloc vs LD_PRELOAD. RHEL dropped support for tcmalloc entirely but if you guys have the ability to compile qemu with tcmalloc support in Debian/Ubuntu I suspect you'll see a nice client side win. Mark On 11/6/23 07:35, Dan Hill wrote:
Very much a side note: At once point the Ubuntu (and I think Debian) packages weren't being built with tcmalloc support. I talked to Chris Macnaughton about it back at Cephalocon and he verified it was the case for the Ubuntu packages, but he's moved on and is no longer at Canonical. It would be nice to verify if this is still (or for Debian ever was) the case and get it resolved.
Thanks again for bringing this to our attention. We resolved the build issue with tcmalloc in Ubuntu packages.
The regression was introduced when Ubuntu enabled the crimson backend for testing [0]. This regression only impacted Ubuntu, not Debian.
[0] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2016845/comments/4 <https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2016845/comments/4>
On Mon, Oct 2, 2023 at 7:24 PM Mark Nelson <mark.nelson@clyso.com <mailto:mark.nelson@clyso.com>> wrote:
On 9/29/23 11:01, Bernd Zeimetz wrote: > When I started looking into ceph I've removed lots of stuff from the > debian folder that the ubuntu people added for unknown reasons, its > much closer to upstream these days then it was before. Actually close > enough that it is no problem to switch to the upstream packages at all.
Very much a side note: At once point the Ubuntu (and I think Debian) packages weren't being built with tcmalloc support. I talked to Chris Macnaughton about it back at Cephalocon and he verified it was the case for the Ubuntu packages, but he's moved on and is no longer at Canonical. It would be nice to verify if this is still (or for Debian ever was) the case and get it resolved.
Mark
-- Best Regards, Mark Nelson Head of Research and Development
Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com <https://clyso.com> | e: mark.nelson@clyso.com <mailto:mark.nelson@clyso.com>
We are hiring: https://www.clyso.com/jobs/ <https://www.clyso.com/jobs/> _______________________________________________ Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io>
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Just to be clear: Ceph still uses tcmalloc on RHEL/CentOS, but we can no longer rely on built-in tcmalloc packages and have to do it ourselves. That's why qemu no longer uses tcmalloc (but Ceph does). See: https://bugzilla.redhat.com/show_bug.cgi?id=1717414 Mark On 11/6/23 11:20, Mark Nelson wrote:
Glad to hear it! FWIW, I noticed a pretty big improvement in VM performance with lbrbd when qemu was set to use tcmalloc vs LD_PRELOAD. RHEL dropped support for tcmalloc entirely but if you guys have the ability to compile qemu with tcmalloc support in Debian/Ubuntu I suspect you'll see a nice client side win.
Mark
On 11/6/23 07:35, Dan Hill wrote:
Very much a side note: At once point the Ubuntu (and I think Debian) packages weren't being built with tcmalloc support. I talked to Chris Macnaughton about it back at Cephalocon and he verified it was the case for the Ubuntu packages, but he's moved on and is no longer at Canonical. It would be nice to verify if this is still (or for Debian ever was) the case and get it resolved.
Thanks again for bringing this to our attention. We resolved the build issue with tcmalloc in Ubuntu packages.
The regression was introduced when Ubuntu enabled the crimson backend for testing [0]. This regression only impacted Ubuntu, not Debian.
[0] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2016845/comments/4 <https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2016845/comments/4>
On Mon, Oct 2, 2023 at 7:24 PM Mark Nelson <mark.nelson@clyso.com <mailto:mark.nelson@clyso.com>> wrote:
On 9/29/23 11:01, Bernd Zeimetz wrote: > When I started looking into ceph I've removed lots of stuff from the > debian folder that the ubuntu people added for unknown reasons, its > much closer to upstream these days then it was before. Actually close > enough that it is no problem to switch to the upstream packages at all.
Very much a side note: At once point the Ubuntu (and I think Debian) packages weren't being built with tcmalloc support. I talked to Chris Macnaughton about it back at Cephalocon and he verified it was the case for the Ubuntu packages, but he's moved on and is no longer at Canonical. It would be nice to verify if this is still (or for Debian ever was) the case and get it resolved.
Mark
-- Best Regards, Mark Nelson Head of Research and Development
Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com <https://clyso.com> | e: mark.nelson@clyso.com <mailto:mark.nelson@clyso.com>
We are hiring: https://www.clyso.com/jobs/ <https://www.clyso.com/jobs/> _______________________________________________ Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io>
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
Hi Greg, On 27/09/2023 16:05, Gregory Farnum wrote:
Belated reply — we actually discussed this at last week’s CLT, but neglected to respond directly.
Has the CLT made any headway since, please? Unless I mis-read the note from the 11th of October it didn't look like packages for Reef were discussed. Thanks, Matthew
We currently anticipate shipping them with the 18.2.1 point release. Yuri tried building Debian packages on 18.2.0 but as that doesn’t include your packaging fixes it didn’t go well. ;) -Greg On Fri, Oct 13, 2023 at 5:50 AM Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi Greg,
On 27/09/2023 16:05, Gregory Farnum wrote:
Belated reply — we actually discussed this at last week’s CLT, but neglected to respond directly.
Has the CLT made any headway since, please? Unless I mis-read the note from the 11th of October it didn't look like packages for Reef were discussed.
Thanks,
Matthew
On Wed, 2023-10-18 at 07:41 -0700, Gregory Farnum wrote:
We currently anticipate shipping them with the 18.2.1 point release. Yuri tried building Debian packages on 18.2.0 but as that doesn’t include your packaging fixes it didn’t go well. ;)
If they don't build for some reason I would be happy to help fixing the build issues, just ping me please. Bernd
-Greg
On Fri, Oct 13, 2023 at 5:50 AM Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi Greg,
On 27/09/2023 16:05, Gregory Farnum wrote:
Belated reply — we actually discussed this at last week’s CLT, but neglected to respond directly.
Has the CLT made any headway since, please? Unless I mis-read the note from the 11th of October it didn't look like packages for Reef were discussed.
Thanks,
Matthew
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Bernd Zeimetz Debian GNU/Linux Developer http://bzed.de http://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
participants (11)
-
Bernd Zeimetz
-
Bernd Zeimetz
-
Dan Hill
-
Gregory Farnum
-
Ilya Dryomov
-
Mark Nelson
-
Mark Nelson
-
Matthew Vernon
-
Thomas Goirand
-
Thomas Goirand
-
thomas＠goirand.fr