Hi all, I just upgraded the first node of our cluster to Ubuntu 24.04 (Noble) from 23.10 (Mantic). Unfortunately Ceph doesnt work anymore on the new node: === ceph version 19.2.0~git20240301.4c76c50 (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x45320) [0x7c7da3c45320] 2: pthread_kill() 3: gsignal() 4: abort() 5: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5ffe) [0x7c7da40a5ffe] 6: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xbae9c) [0x7c7da40bae9c] 7: (std::unexpected()+0) [0x7c7da40a5a49] 8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xbb128) [0x7c7da40bb128] 9: (ceph::buffer::v15_2_0::list::iterator_impl<true>::copy(unsigned int, char*)+0xc4) [0x7c7da4a6e414] 10: (MDSMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0xc74) [0x7c7da4c98fc4] 11: (MDSDaemon::handle_mds_map(boost::intrusive_ptr<MMDSMap const> const&)+0x2c6) [0x59124a7cd766] 12: (MDSDaemon::handle_core_message(boost::intrusive_ptr<Message const> const&)+0x2c9) [0x59124a7d0d79] 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0xe1) [0x59124a7d1341] 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x3b8) [0x7c7da4930dd8] 15: (DispatchQueue::entry()+0x771) [0x7c7da492ecb1] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7c7da49f3f81] 17: /lib/x86_64-linux-gnu/libc.so.6(+0x9ca94) [0x7c7da3c9ca94] 18: /lib/x86_64-linux-gnu/libc.so.6(+0x129c3c) [0x7c7da3d29c3c] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. === What's more APT says I now got a Ceph Version (19.2.0~git20240301.4c76c50-0ubuntu6) which doesn't even have any official release notes: https://docs.ceph.com/en/latest/releases/ How did I get here? What can I do to get back a working Ceph version? Until this is resolved, I obviously don't want to upgrade the other nodes. Thank you / BR Ranjan
On 13.06.24 18:18, Ranjan Ghosh wrote:
What's more APT says I now got a Ceph Version (19.2.0~git20240301.4c76c50-0ubuntu6) which doesn't even have any official release notes:
Ubuntu 24.04 ships with that version from a git snapshot. You have to ask Canonical why they did this. I would not use Ceph packages shipped from a distribution but always the ones from download.ceph.com or even better the container images that come with the orchestrator. Why version do your other Ceph nodes run on? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
In addition to Robert's recommendations, Remember to respect the update order (mgr->mon->(crash->)osd->mds->...) Before everything was containerized, it was not recommended to have different services on the same machine. Le jeu. 13 juin 2024 à 19:37, Robert Sander <r.sander@heinlein-support.de> a écrit :
On 13.06.24 18:18, Ranjan Ghosh wrote:
What's more APT says I now got a Ceph Version (19.2.0~git20240301.4c76c50-0ubuntu6) which doesn't even have any official release notes:
Ubuntu 24.04 ships with that version from a git snapshot.
You have to ask Canonical why they did this.
I would not use Ceph packages shipped from a distribution but always the ones from download.ceph.com or even better the container images that come with the orchestrator.
Why version do your other Ceph nodes run on?
Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de
Tel: 030-405051-43 Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If this is one node from many it’s not a problem because you can reinstall system and ceph and rebalance cluster. BTW. Read release notes before :) I’m also not reading it in case of my personal desktop, but on servers where I keep data I’m doing it. but what canonical did in this case is… this is LTS version :/ BR, Sebastian
On 13 Jun 2024, at 19:47, David C. <david.casier@aevoo.fr> wrote:
In addition to Robert's recommendations,
Remember to respect the update order (mgr->mon->(crash->)osd->mds->...)
Before everything was containerized, it was not recommended to have different services on the same machine.
Le jeu. 13 juin 2024 à 19:37, Robert Sander <r.sander@heinlein-support.de> a écrit :
On 13.06.24 18:18, Ranjan Ghosh wrote:
What's more APT says I now got a Ceph Version (19.2.0~git20240301.4c76c50-0ubuntu6) which doesn't even have any official release notes:
Ubuntu 24.04 ships with that version from a git snapshot.
You have to ask Canonical why they did this.
I would not use Ceph packages shipped from a distribution but always the ones from download.ceph.com or even better the container images that come with the orchestrator.
Why version do your other Ceph nodes run on?
Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de
Tel: 030-405051-43 Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm still in doubt whether any reinstall will fix this issue, because the packages seem to be buggy and there a no better packages right now for Ubuntu 24.04 it seems. Canonical is really crazy if you ask me. Even for a non-LTS version but especially for an LTS version. What were they thinking? Just get a preliminary obviously buggy GIT version and shove it out with a release to unsuspecting users. Just imagine they did sth. like this with Apache etc. Thanks for all tips y'all. Still need to figure out what the *best* option right now would be to fix this. Sigh. Am 13.06.24 um 20:00 schrieb Sebastian:
If this is one node from many it’s not a problem because you can reinstall system and ceph and rebalance cluster. BTW. Read release notes before :) I’m also not reading it in case of my personal desktop, but on servers where I keep data I’m doing it. but what canonical did in this case is… this is LTS version :/
BR, Sebastian
On 13 Jun 2024, at 19:47, David C. <david.casier@aevoo.fr> wrote:
In addition to Robert's recommendations,
Remember to respect the update order (mgr->mon->(crash->)osd->mds->...)
Before everything was containerized, it was not recommended to have different services on the same machine.
Le jeu. 13 juin 2024 à 19:37, Robert Sander <r.sander@heinlein-support.de> a écrit :
On 13.06.24 18:18, Ranjan Ghosh wrote:
What's more APT says I now got a Ceph Version (19.2.0~git20240301.4c76c50-0ubuntu6) which doesn't even have any official release notes: Ubuntu 24.04 ships with that version from a git snapshot.
You have to ask Canonical why they did this.
I would not use Ceph packages shipped from a distribution but always the ones from download.ceph.com or even better the container images that come with the orchestrator.
Why version do your other Ceph nodes run on?
Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de
Tel: 030-405051-43 Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
<troll>Debian unstable</troll> The situation is absolutely not dramatic, but if it is a large production, you should benefit from product support. Based on the geographical area of your email domain, perhaps ask Robert for a local service ? Le jeu. 13 juin 2024 à 20:35, Ranjan Ghosh <ghosh@pw6.de> a écrit :
I'm still in doubt whether any reinstall will fix this issue, because the packages seem to be buggy and there a no better packages right now for Ubuntu 24.04 it seems.
Canonical is really crazy if you ask me. Even for a non-LTS version but especially for an LTS version. What were they thinking? Just get a preliminary obviously buggy GIT version and shove it out with a release to unsuspecting users. Just imagine they did sth. like this with Apache etc.
Thanks for all tips y'all. Still need to figure out what the *best* option right now would be to fix this. Sigh.
If this is one node from many it’s not a problem because you can reinstall system and ceph and rebalance cluster. BTW. Read release notes before :) I’m also not reading it in case of my
but what canonical did in this case is… this is LTS version :/
BR, Sebastian
On 13 Jun 2024, at 19:47, David C. <david.casier@aevoo.fr> wrote:
In addition to Robert's recommendations,
Remember to respect the update order (mgr->mon->(crash->)osd->mds->...)
Before everything was containerized, it was not recommended to have different services on the same machine.
Le jeu. 13 juin 2024 à 19:37, Robert Sander < r.sander@heinlein-support.de> a écrit :
On 13.06.24 18:18, Ranjan Ghosh wrote:
What's more APT says I now got a Ceph Version (19.2.0~git20240301.4c76c50-0ubuntu6) which doesn't even have any official release notes: Ubuntu 24.04 ships with that version from a git snapshot.
You have to ask Canonical why they did this.
I would not use Ceph packages shipped from a distribution but always
Am 13.06.24 um 20:00 schrieb Sebastian: personal desktop, but on servers where I keep data I’m doing it. the
ones from download.ceph.com or even better the container images that come with the orchestrator.
Why version do your other Ceph nodes run on?
Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de
Tel: 030-405051-43 Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
David C.
-
Ranjan Ghosh
-
Robert Sander
-
Sebastian