Hi, Is it possible to only build "cephadm", so not the other ceph packages / daemons? Or can we think about a way to have cephadm packages build for all supported mainstream linux releases during the supported lifetime of a Ceph release: i.e. debian, Ubuntu LTS, CentOS Stream? I went ahead and upgraded one of our (pre-prod) ceph nodes to Ubuntu 24.04 LTS. As it is managed by cephadm and running containers this should be no problem, right? Right. Except that there are is no "cephadm" package (or any other ceph package) for Ubuntu 24.04 from Ceph repository. So, then just build those packages yourself, right? Right. After some python pyyaml 6.0.0 / cython issues in monitoring/ceph-mixin/requirements-alerts.txt and requirements-lint.txt, fixed by bumping pyyaml to 6.0.1) and a hard python 3.10 requirement (I bumped that to 3.12) I ran into an issue with "arrow_ext" breaking the build process which I'm currently trying to figure out ... I know supporting Ceph packages for a bunch of distros is a lot of work, but having _just_ cephadm available on a wider range of platforms, would really help here. It helps avoid upgrading both Ceph and the OS at the same time. This allows the use of the latest (kernel) OS improvements, all while not touching Ceph. Thanks, Gr. Stefan
Hello Stefan, have a look: https://docs.ceph.com/en/latest/cephadm/install/#curl-based-installation Just download cephadm. It will work on any distro. You do not need any ceph package like ceph-common for example to run cephadm. Best, Malte On 11.07.24 08:17, Stefan Kooman wrote:
Hi,
Is it possible to only build "cephadm", so not the other ceph packages / daemons? Or can we think about a way to have cephadm packages build for all supported mainstream linux releases during the supported lifetime of a Ceph release: i.e. debian, Ubuntu LTS, CentOS Stream?
I went ahead and upgraded one of our (pre-prod) ceph nodes to Ubuntu 24.04 LTS. As it is managed by cephadm and running containers this should be no problem, right? Right. Except that there are is no "cephadm" package (or any other ceph package) for Ubuntu 24.04 from Ceph repository. So, then just build those packages yourself, right? Right. After some python pyyaml 6.0.0 / cython issues in monitoring/ceph-mixin/requirements-alerts.txt and requirements-lint.txt, fixed by bumping pyyaml to 6.0.1) and a hard python 3.10 requirement (I bumped that to 3.12) I ran into an issue with "arrow_ext" breaking the build process which I'm currently trying to figure out ...
I know supporting Ceph packages for a bunch of distros is a lot of work, but having _just_ cephadm available on a wider range of platforms, would really help here. It helps avoid upgrading both Ceph and the OS at the same time. This allows the use of the latest (kernel) OS improvements, all while not touching Ceph.
Thanks,
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 11-07-2024 09:55, Malte Stroem wrote:
Hello Stefan,
have a look:
https://docs.ceph.com/en/latest/cephadm/install/#curl-based-installation
Yeah, I have read that part.
Just download cephadm. It will work on any distro.
curl --silent --remote-name --location https://download.ceph.com/rpm-18.2.1/el9/noarch/cephadm ./cephadm gather-facts Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/./cephadm/__main__.py", line 10700, in <module> File "/root/./cephadm/__main__.py", line 10688, in main File "/root/./cephadm/__main__.py", line 9772, in command_gather_facts File "/root/./cephadm/__main__.py", line 9762, in dump File "/root/./cephadm/__main__.py", line 9677, in kernel_security File "/root/./cephadm/__main__.py", line 9658, in _fetch_apparmor ValueError: too many values to unpack (expected 2) The version that works for 22.04 also doesn't work on 24.04 and fails in a similar way. It just isn't that simple I'm afraid. But it _should_ be that simple, agreed.
You do not need any ceph package like ceph-common for example to run cephadm.
Indeed, which is pretty neat. If cephadm works you can basically do all ceph related stuff in a container. Getting cephadm to run on more platforms is the last mile. Gr. Stefan
On Thursday, July 11, 2024 4:22:28 AM EDT Stefan Kooman wrote:
On 11-07-2024 09:55, Malte Stroem wrote:
Hello Stefan,
have a look:
https://docs.ceph.com/en/latest/cephadm/install/#curl-based-installation
Yeah, I have read that part.
Just download cephadm. It will work on any distro.
curl --silent --remote-name --location https://download.ceph.com/rpm-18.2.1/el9/noarch/cephadm
./cephadm gather-facts Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/./cephadm/__main__.py", line 10700, in <module> File "/root/./cephadm/__main__.py", line 10688, in main File "/root/./cephadm/__main__.py", line 9772, in command_gather_facts File "/root/./cephadm/__main__.py", line 9762, in dump File "/root/./cephadm/__main__.py", line 9677, in kernel_security File "/root/./cephadm/__main__.py", line 9658, in _fetch_apparmor ValueError: too many values to unpack (expected 2)
The version that works for 22.04 also doesn't work on 24.04 and fails in a similar way.
It just isn't that simple I'm afraid. But it _should_ be that simple, agreed.
You do not need any ceph package like ceph-common for example to run cephadm.
Indeed, which is pretty neat. If cephadm works you can basically do all ceph related stuff in a container. Getting cephadm to run on more platforms is the last mile.
Gr. Stefan
Hi there, The traceback you are hitting is a bug - there's a fix already applied to main: https://github.com/ceph/ceph/pull/57955 I'll ask to have backport PRs get generated. I'm personally pretty clueless as to how to process backports. The bug is independent of how cephadm is packaged FWIW. Even if you had a package of just cephadm built for ubuntu 24.04 it would have still hit the problem. The code simply didn't understand all the possible syntax that can appear in the apparmor profiles and newer versions of ubuntu appear to use apparmor profiles with spaces in the name more commonly than older versions. The current cephadm build process creates a "zipapp" out of a few select python packages and the cephadm source code. If you really want to you could wrap that, and just that, in a system package what would not need many dependencies. However, this would need to be a bespoke package as the packages created by the ceph project include "everything" ceph builds. But the build script for cephadm (./src/cephadm/build.py) doesn't need any of those other binaries to be built to work. - In case you were still curious about that and want to tinker.
On 11-07-2024 14:20, John Mulligan wrote:
On Thursday, July 11, 2024 4:22:28 AM EDT Stefan Kooman wrote:
On 11-07-2024 09:55, Malte Stroem wrote:
Hello Stefan,
have a look:
https://docs.ceph.com/en/latest/cephadm/install/#curl-based-installation
Yeah, I have read that part.
Just download cephadm. It will work on any distro.
curl --silent --remote-name --location https://download.ceph.com/rpm-18.2.1/el9/noarch/cephadm
./cephadm gather-facts Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/./cephadm/__main__.py", line 10700, in <module> File "/root/./cephadm/__main__.py", line 10688, in main File "/root/./cephadm/__main__.py", line 9772, in command_gather_facts File "/root/./cephadm/__main__.py", line 9762, in dump File "/root/./cephadm/__main__.py", line 9677, in kernel_security File "/root/./cephadm/__main__.py", line 9658, in _fetch_apparmor ValueError: too many values to unpack (expected 2)
The version that works for 22.04 also doesn't work on 24.04 and fails in a similar way.
It just isn't that simple I'm afraid. But it _should_ be that simple, agreed.
You do not need any ceph package like ceph-common for example to run cephadm.
Indeed, which is pretty neat. If cephadm works you can basically do all ceph related stuff in a container. Getting cephadm to run on more platforms is the last mile.
Gr. Stefan
Hi there, The traceback you are hitting is a bug - there's a fix already applied to main: https://github.com/ceph/ceph/pull/57955
Thanks, I hadn't found that one.
I'll ask to have backport PRs get generated. I'm personally pretty clueless as to how to process backports.
Ah, yeah, that's on my todo list as well (work on backports). AFAIK you need to cherry pick from main branch when processing PRs.
The bug is independent of how cephadm is packaged FWIW. Even if you had a package of just cephadm built for ubuntu 24.04 it would have still hit the problem. The code simply didn't understand all the possible syntax that can appear in the apparmor profiles and newer versions of ubuntu appear to use apparmor profiles with spaces in the name more commonly than older versions.
Ah, that explains ...
The current cephadm build process creates a "zipapp" out of a few select python packages and the cephadm source code. If you really want to you could wrap that, and just that, in a system package what would not need many dependencies. However, this would need to be a bespoke package as the packages created by the ceph project include "everything" ceph builds. But the build script for cephadm (./src/cephadm/build.py) doesn't need any of those other binaries to be built to work. - In case you were still curious about that and want to tinker.
You betcha. Okay, that worked perfectly. I backported the fixes from #57955 in cephadm.py (apparently there was a code refactor and cephadmlib does not exist in 18.2.1). This worked for me: ./src/cephadm/build.py --python /usr/bin/python3 /tmp/cephadm -SCEPH_GIT_VER=$(git rev-parse HEAD) -SCEPH_GIT_NICE_VER=$(git describe) -SCEPH_RELEASE_NAME=reef -SCEPH_RELEASE_TYPE=stable This works fine on 24.04. Getting this in a CI/CD pipeline whenever we need a new cephadm version and built a proper cephadm package we can install (instead of downloading a binary) would certainly be an option for us. Not sure if we are the only ones that want cephadm installed through packages though. Otherwise it would warrant an upstream cephadm package IMHO. Thanks a lot! Gr. Stefan
Just my €.02. There is, in fact a cephadm package for the Raspberry Pi OS. If I read the synopis correctly, it's for ceph 16.2.11, which I think is the same release of Ceph Pacific that I'm presently running my own farm on. It appears to derive off Debian Bookworm. Since cephadm is mainly a program to control other programs, I'd not be surprised if it can at least partly manage newer ceph systems. Then again, I don't know what it uses for interfaces beyond its ability to pull down and setup ceph containers. For my newer systems, cephadm is the only package I install and I use cephadm shell. For general management, I do like to install the ceph- common to get the "ceph" command down to the bash shell level without invoking cephadm. Everything else, I haven't found a use for. I do like managed packages because it's useful to have them show up in a software inventory. However, once I have the cephadm package installed, I prefer to let it manage the rest of the ceph infrastructure. Best Regards, Tim
On 11 Jul 2024, at 15:20, John Mulligan <phlogistonjohn@asynchrono.us> wrote:
I'll ask to have backport PRs get generated. I'm personally pretty clueless as to how to process backports.
The how-to described in this doc [1]
Thanks, I hadn't found that one. Added backport for squid release [2], as far as I understand Ceph in containers - you need "latest" `adm`
Thanks, k [1] https://github.com/ceph/ceph/blob/main/SubmittingPatches-backports.rst#openi... [2] https://github.com/ceph/ceph/pull/58542
Hi, Am 11.07.2024 um 14:20 schrieb John Mulligan:
...
as far as I know, we still have an issue https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2063456 with ceph on 24.04. I tried the offered fix, but was still unable to establish a running cluster (may be my fault, I'm still a newbie to this). Kind regards Thomas
On 12-07-2024 09:33, tpDev Tester wrote:
Hi,
Am 11.07.2024 um 14:20 schrieb John Mulligan:
...
as far as I know, we still have an issue
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2063456
with ceph on 24.04. I tried the offered fix, but was still unable to establish a running cluster (may be my fault, I'm still a newbie to this).
You can download this package and install it manually: (first remove the old package: apt purge cephadm) wget http://launchpadlibrarian.net/735185237/cephadm_19.2.0~git20240301.4c76c50-0... dpkg -i cephadm_19.2.0~git20240301.4c76c50-0ubuntu6.1_amd64.deb This version worked for a Reef cluster (18.2.1) on 24.04. But you can also download a 18.2.1 version with the apparmor fix backported from here: https://kooman.org/cephadm sha256sum cephadm 0067d9951cdf31b26788bfd4775913992672c8cf652215650997d4da2dc8d5a3 cephadm Then what? I copied the "cephadm" file to "/usr/sbin/cephadm" and "/var/lib/ceph/{fsid-of-your-cluster/cephadm.verylongstringhere" Or let me know what version you need, building cephadm is really trivial. Note: just to be sure, you do _NOT_ want to use Ceph from Ubuntu 24.04 repositories. The 19.2.x release is not out yet (still RC) and this is a Ubuntu released version (why they ship this in a Ubuntu LTS version instead of a stable Ceph release is beyond me, but they might have their reasons ... @Canonical folks please chime in here). Gr. Stefan
Hi, thanks for your response. Am 12.07.2024 um 10:24 schrieb Stefan Kooman:
... Note: just to be sure, you do _NOT_ want to use Ceph from Ubuntu 24.04 repositories. The 19.2.x release is not out yet (still RC) and this is a Ubuntu released version (why they ship this in a Ubuntu LTS version instead of a stable Ceph release is beyond me, but they might have their reasons ... @Canonical folks please chime in here).
Finally, I'm looking for a solution for production use and it would be great if I don't have to leave the usual Ubuntu procedures, especially when it comes to updates. We are also confused about the "RC vs. LTS"-thing. But I will try you suggestion, I can learn ceph things this way and when the Ubuntu people fixed the issue, I can go ahead with that solution. Thanks so far. Thomas
Hi, On 7/12/24 10:47, tpDev Tester wrote:
Finally, I'm looking for a solution for production use and it would be great if I don't have to leave the usual Ubuntu procedures, especially when it comes to updates. We are also confused about the "RC vs. LTS"-thing.
I would suggest to use Ubuntu 22.04 LTS as the base operating system. You can use cephadm on top of that without issues. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Am 12.07.2024 um 10:57 schrieb Robert Sander:
... I would suggest to use Ubuntu 22.04 LTS as the base operating system. You can use cephadm on top of that without issues.
yes, that's right. But I already upgraded my systems to 24.04, may be to early, my fault. Currently, it's all testing and learning and I will be well prepared, when 22.04 is outdated. Kind regards Thomas
participants (7)
-
John Mulligan
-
Konstantin Shalygin
-
Malte Stroem
-
Robert Sander
-
Stefan Kooman
-
Tim Holloway
-
tpDev Tester