- ceph-users - lists.ceph.io

Re: Changing EC profile
by Daniel Parkes 16 Apr '26

16 Apr '26

Hi, Yes, we have it in the docs, https://docs.ceph.com/en/latest/radosgw/placement/#placement-targets : [image: image.png] On Wed, Apr 15, 2026 at 2:06 PM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > It didn't work in my tests without a realm, so I don't believe grok > :-D But I'd love to be proven wrong. > > Zitat von Rok Jaklič <rjaklic(a)gmail.com>: > > > There is: > > > > [root@ctplmon1 ~]# radosgw-admin realm list > > { > > "default_info": "", > > "realms": [] > > } > > [root@ctplmon1 ~]# radosgw-admin realm get-default > > No default realm is set > > [root@ctplmon1 ~]# radosgw-admin period list > > { > > "periods": [] > > } > > [root@ctplmon1 ~]# radosgw-admin zonegroup list > > { > > "default_info": "b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578", > > "zonegroups": [ > > "default" > > ] > > } > > [root@ctplmon1 ~]# radosgw-admin zone list > > { > > "default_info": "c2c70444-7a41-4acd-a0d0-9f87d324ec72", > > "zones": [ > > "default" > > ] > > } > > > > Since there is no realm set, I can skip radosgw-admin period update > > --commit entirely if grok is to be believed? > > > > On Wed, Apr 15, 2026 at 11:54 AM Eugen Block via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> From my understanding the new placement will not be available if the > >> period can't be updated. Your zonegroup/zone configuration seems off > >> compared to the test cluster. Is there a default zonegroup/zone? > >> > >> Zitat von Rok Jaklič via ceph-users <ceph-users(a)ceph.io>: > >> > >> > So I've created a little plan on how to create a new profile and > assign > >> new > >> > S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new > >> > pool, and old ones pool with EC 3+2 and data is intact. > >> > > >> > --- > >> > ceph osd erasure-code-profile set ec-83-profile k=8 m=3 > >> > crush-failure-domain=host > >> > ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile > >> > ceph osd pool application enable rgw.buckets.ec-83.data rgw > >> > > >> > radosgw-admin zonegroup placement add --placement-id=ec-83-placement > >> > > >> > radosgw-admin zone placement add \ > >> > --placement-id ec-83-placement \ > >> > --data-pool rgw.buckets.ec-83.data \ > >> > --index-pool default.rgw.buckets.index \ > >> > --data-extra-pool default.rgw.buckets.non-ec \ > >> > --placement-index-type 0 > >> > > >> > radosgw-admin period update --commit > >> > > >> > # restart rgw > >> > systemctl restart ceph-radosgw.target > >> > # verify the new placement > >> > radosgw-admin zone get | grep -A 20 "ec-83-placement" > >> > > >> > # for new s3 users, set the default placement > >> > radosgw-admin user create --uid=<user> --tenant=<tenant> > >> > --placement-id=ec-83-placement > >> > --- > >> > > >> > It seems to work on "test cluster", however I get an error on: > >> > > >> > [root@a8-mon1 ~]# radosgw-admin period update --commit > >> > cannot commit period: period does not have a master zone of a master > >> > zonegroup > >> > failed to commit period: (22) Invalid argument > >> > > >> > Do I actually need to run this command or is "rgw restart" enough? > >> > > >> > Any comments appriciated. > >> > > >> > Rok > >> > > >> > > >> > On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < > >> > ceph-users(a)ceph.io> wrote: > >> > > >> >> Hi, > >> >> > >> >> we expanded the cluster to 12 nodes with 28 osds, but we are still > >> running > >> >> on EC 3+2, since we started cluster with 6 nodes. We would like to > >> change > >> >> it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too > >> much > >> >> CPU/network capacities? Should I use isa/isa-l instead of jerasure? > >> >> > >> >> Can anyone share experience with 8+3? > >> >> > >> >> Kind regards, > >> >> Rok > >> >> _______________________________________________ > >> >> ceph-users mailing list -- ceph-users(a)ceph.io > >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> >> > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users(a)ceph.io > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

3 5

Re: Upgrade path from Pacific to Quincy (and to Reef)
by Eugen Block 15 Apr '26

15 Apr '26

Thanks for sharing, I was not aware of that. I must admit, it’s been a while since we upgraded our own or our customer’s clusters to Reef, we usually try to stay near supported releases. Anyway, I don’t have a clear recommendation which Quincy version to upgrade to. We skipped it as did almost all of our customers (several of them came from SUSE Enterprise Storage which ended with version Pacific, so they upgraded to Reef). One other customer was on Quincy (came from Octopus) until recently, they didn’t have any issues with Q, although I don’t recall the exact versions of Q they had in use. I can try to look it up tomorrow if that helps. Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: > From the "Known issues" section of the Reef 18.2.8 release notes > (https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/) : > > "During QA for v18.2.8, it was found that there was a bug for > upgrades from Pacific to Reef. Pacific OSDs (and other Ceph daemons) > were still using a deprecated connection feature bit that was > adopted to indicate a Reef OSD. This can cause a > OSD_UPGRADE_FINISHED warning before all OSDs are actually upgraded > to Reef. There are no known issues associated with Pacific and Reef > OSDs interoperating where Pacific OSDs are "advertising" Reef > compatibility; however, out of an abundance of caution, we no longer > recommend upgrading from Pacific to Reef directly." > > On 4/15/26 16:02, Eugen Block via ceph-users wrote: >> Why is it not recommended to upgrade from Pacific to Reef? We’ve >> done that multiple times, I’m quite surprised by that statement. >> >> Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: >> >>> Hi, >>> >>> We have a fairly old Ceph Pacific cephadm cluster that we would >>> like to upgrade to an active release. Since upgrading from Pacific >>> to Reef is not recommended, we will need to upgrade to Quincy >>> first. However, there seems to be bugs with Pacific to Quincy >>> upgrades for several Quincy releases. Would it be best to upgrade >>> Quincy 17.2.8, which has a bluestore regression bug, or to 17.2.7 >>> which seems to also have caused issues to other users? I >>> understand that 17.2.9 is not obtainable, as there was no >>> container image built for it. We like to run our ceph versions for >>> a few weeks inbetween upgrades, so the cluster needs to remain >>> stable. >>> >>> >>> -- >>> Mes salutations distinguées | Best regards >>> Jean-Philippe MÉTHOT (Il, Lui | He/Him) >>> Administrateur Infra sénior | Senior Infra administrator >>> PlanetHoster >>> >>> AVIS: Ce courriel est confidentiel et peut être l'objet de secret >>> professionnel. Tout usage ou diffusion de celui-ci par une >>> personne autre que son destinataire est illégal. Si vous recevez >>> ce courriel par erreur, veuillez nous en aviser sans délai et le >>> supprimer de votre système. >>> >>> This message is confidential. It may also be privileged or >>> otherwise protected by work product immunity or other legal rules. >>> If you have received it by mistake, please let us know by e-mail >>> reply and delete it from your system. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Mes salutations distinguées | Best regards > Jean-Philippe MÉTHOT (Il, Lui | He/Him) > Administrateur Infra sénior | Senior Infra administrator > PlanetHoster > > AVIS: Ce courriel est confidentiel et peut être l'objet de secret > professionnel. Tout usage ou diffusion de celui-ci par une personne > autre que son destinataire est illégal. Si vous recevez ce courriel > par erreur, veuillez nous en aviser sans délai et le supprimer de > votre système. > > This message is confidential. It may also be privileged or otherwise > protected by work product immunity or other legal rules. If you have > received it by mistake, please let us know by e-mail reply and > delete it from your system. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Upgrade path from Pacific to Quincy (and to Reef)
by J-P Methot 15 Apr '26

15 Apr '26

From the "Known issues" section of the Reef 18.2.8 release notes (https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/) : "During QA for v18.2.8, it was found that there was a bug for upgrades from Pacific to Reef. Pacific OSDs (and other Ceph daemons) were still using a deprecated connection feature bit that was adopted to indicate a Reef OSD. This can cause a OSD_UPGRADE_FINISHED warning before all OSDs are actually upgraded to Reef. There are no known issues associated with Pacific and Reef OSDs interoperating where Pacific OSDs are "advertising" Reef compatibility; however, out of an abundance of caution, we no longer recommend upgrading from Pacific to Reef directly." On 4/15/26 16:02, Eugen Block via ceph-users wrote: > Why is it not recommended to upgrade from Pacific to Reef? We’ve done > that multiple times, I’m quite surprised by that statement. > > Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: > >> Hi, >> >> We have a fairly old Ceph Pacific cephadm cluster that we would like >> to upgrade to an active release. Since upgrading from Pacific to Reef >> is not recommended, we will need to upgrade to Quincy first. However, >> there seems to be bugs with Pacific to Quincy upgrades for several >> Quincy releases. Would it be best to upgrade Quincy 17.2.8, which has >> a bluestore regression bug, or to 17.2.7 which seems to also have >> caused issues to other users? I understand that 17.2.9 is not >> obtainable, as there was no container image built for it. We like to >> run our ceph versions for a few weeks inbetween upgrades, so the >> cluster needs to remain stable. >> >> >> -- >> Mes salutations distinguées | Best regards >> Jean-Philippe MÉTHOT (Il, Lui | He/Him) >> Administrateur Infra sénior | Senior Infra administrator >> PlanetHoster >> >> AVIS: Ce courriel est confidentiel et peut être l'objet de secret >> professionnel. Tout usage ou diffusion de celui-ci par une personne >> autre que son destinataire est illégal. Si vous recevez ce courriel >> par erreur, veuillez nous en aviser sans délai et le supprimer de >> votre système. >> >> This message is confidential. It may also be privileged or otherwise >> protected by work product immunity or other legal rules. If you have >> received it by mistake, please let us know by e-mail reply and delete >> it from your system. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Mes salutations distinguées | Best regards Jean-Philippe MÉTHOT (Il, Lui | He/Him) Administrateur Infra sénior | Senior Infra administrator PlanetHoster AVIS: Ce courriel est confidentiel et peut être l'objet de secret professionnel. Tout usage ou diffusion de celui-ci par une personne autre que son destinataire est illégal. Si vous recevez ce courriel par erreur, veuillez nous en aviser sans délai et le supprimer de votre système. This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system.

1 0

Re: Upgrade path from Pacific to Quincy (and to Reef)
by Eugen Block 15 Apr '26

15 Apr '26

Why is it not recommended to upgrade from Pacific to Reef? We’ve done that multiple times, I’m quite surprised by that statement. Zitat von J-P Methot via ceph-users <ceph-users(a)ceph.io>: > Hi, > > We have a fairly old Ceph Pacific cephadm cluster that we would like > to upgrade to an active release. Since upgrading from Pacific to > Reef is not recommended, we will need to upgrade to Quincy first. > However, there seems to be bugs with Pacific to Quincy upgrades for > several Quincy releases. Would it be best to upgrade Quincy 17.2.8, > which has a bluestore regression bug, or to 17.2.7 which seems to > also have caused issues to other users? I understand that 17.2.9 is > not obtainable, as there was no container image built for it. We > like to run our ceph versions for a few weeks inbetween upgrades, so > the cluster needs to remain stable. > > > -- > Mes salutations distinguées | Best regards > Jean-Philippe MÉTHOT (Il, Lui | He/Him) > Administrateur Infra sénior | Senior Infra administrator > PlanetHoster > > AVIS: Ce courriel est confidentiel et peut être l'objet de secret > professionnel. Tout usage ou diffusion de celui-ci par une personne > autre que son destinataire est illégal. Si vous recevez ce courriel > par erreur, veuillez nous en aviser sans délai et le supprimer de > votre système. > > This message is confidential. It may also be privileged or otherwise > protected by work product immunity or other legal rules. If you have > received it by mistake, please let us know by e-mail reply and > delete it from your system. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Upgrade path from Pacific to Quincy (and to Reef)
by J-P Methot 15 Apr '26

15 Apr '26

Hi, We have a fairly old Ceph Pacific cephadm cluster that we would like to upgrade to an active release. Since upgrading from Pacific to Reef is not recommended, we will need to upgrade to Quincy first. However, there seems to be bugs with Pacific to Quincy upgrades for several Quincy releases. Would it be best to upgrade Quincy 17.2.8, which has a bluestore regression bug, or to 17.2.7 which seems to also have caused issues to other users? I understand that 17.2.9 is not obtainable, as there was no container image built for it. We like to run our ceph versions for a few weeks inbetween upgrades, so the cluster needs to remain stable. -- Mes salutations distinguées | Best regards Jean-Philippe MÉTHOT (Il, Lui | He/Him) Administrateur Infra sénior | Senior Infra administrator PlanetHoster AVIS: Ce courriel est confidentiel et peut être l'objet de secret professionnel. Tout usage ou diffusion de celui-ci par une personne autre que son destinataire est illégal. Si vous recevez ce courriel par erreur, veuillez nous en aviser sans délai et le supprimer de votre système. This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system.

1 0

Announcing go-ceph v0.39.0
by John Mulligan 15 Apr '26

15 Apr '26

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.39.0 Enhancements include additions to the smb admin package, and a new command builder package. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

Re: Upgrade to tentacle failed! (Now recovered, upgrade paused)
by Johan 15 Apr '26

15 Apr '26

I realized I can't manually upgrade specific daemon-types when upgrade already in progress (paused in this case). I therefore decided to resume the upgrade from the dashboard and all has gone well, the cluster is now fully upgraded. /Johan Den 2026-04-15 kl. 13:47, skrev Johan via ceph-users: > Im running two private ceph clusters. Both are cephadm/docker on debian. > > The first cluster was upgraded without problem to latest tentacle > (19.2.3 -> 20.2.1) using > ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.1 > > Today I tried to upgrade the second cluster. During the upgrade, when a > couple of managers had been upgraded, the upgrade process stopped with > the error message > Orchestrator is unavailable: Module not found > and the suggestion I should enable it (ceph mgr module enable > orchestrator). From here Im not exactly certain about the errors/ > messages I received, but I could not enable the module (it already was) > and several pages on the dashboard didn't give any information as it > apparently dependent on the orchestrator. > > As I neither were near the servers nor had the time to deal with the > failure I chose to shutdown the servers. I have now restarted the > servers and after a while the cluster got up again. > > The dashboard (Administration/Upgrade) now shows version 20.2.1 and > "Upgrade is paused". Two managers (incl the active) are upgraded and the > third is old version(19.2.3). Everything else seems old version > (logical, mgr is upgraded first and the error occured during mgr upgrade). > > When I got the cluster running again > ceph orch uprade status > showed > > { > "in_progress": true, > "target_image": "quay.io/ceph/ > ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337", > "services_complete": [], > "which": "Upgrading all daemon types on all hosts", > "progress": "2/52 daemons upgraded", > "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an > unexpected exception", > "is_paused": true > } > > > Any suggestions regarding the continued upgrade? Resume the upgrade from > the dashboard? First manually upgrade the last manager and then resume > the normal upgrade process? or... ? > > Anyone seen anything like this before or understand the reason for the > failure? > > /Johan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Bluestore deferred writes (Squid)
by Igor Fedotov 15 Apr '26

15 Apr '26

Hi Magnus, Here is the brief overview for the relevant perf counters: - 'issued deferred writes' denotes amount of user write ops which have got deferred processing and have been persisted at Key-Value DB (The first stage of deferred write procedure). - 'submitted deferred writes' denotes amount of writes which have been submitted to main device to reach their final disk location. (the second stage of deferred write procedure). Additionally there is the last stage which is staged deferred writes cleanup from KV. But there is no relevant counter to track this state though at the moment. Frankly speaking I recall no modifications to bluestore_prefer_deferred_size_hdd happened to Squid. The only not-that-recent PR which might look relevant is https://github.com/ceph/ceph/pull/48490 but it's been backported back to Pacific and Quincy. May be you had had another upgrade(s) prior to R->S one recently which triggered the regression? It would be great to get more information on the issue including perf counter dumps and more details on that performance degradation and write patters which are affected. Thanks, Igor On 4/14/2026 5:00 PM, Magnus Grönlund via ceph-users wrote: > Hi All, > > We are running a 5 year old 15 node Ceph cluster with spinners for data and > NVMe for DB/Wal. > After the recent upgrade to Squid (from Reef) some users reported issues > with fluctuating (often bad) disk performance especially affecting k8s etcd. > They point to the change to bluestore_prefer_deferred_size_hdd claiming > that it's detrimental to the "synchronous write patter" of escd. > They also point to submitted_deferred_writes and issued_deferred_writes > claiming that there is a massive and growing backlog of deferred writes. > > I'm not 100% correct that this is the issue but I'm having a hard time > finding any relevant information (bad google-fu?) especially about what > submitted and issued deferred writes actually represents and why the > difference between issued - submitted keeps growing. > Anyone got some insights to share? > > Thanks in advance > Magnus > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Changing EC profile
by Rok Jaklič 15 Apr '26

15 Apr '26

There is: [root@ctplmon1 ~]# radosgw-admin realm list { "default_info": "", "realms": [] } [root@ctplmon1 ~]# radosgw-admin realm get-default No default realm is set [root@ctplmon1 ~]# radosgw-admin period list { "periods": [] } [root@ctplmon1 ~]# radosgw-admin zonegroup list { "default_info": "b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578", "zonegroups": [ "default" ] } [root@ctplmon1 ~]# radosgw-admin zone list { "default_info": "c2c70444-7a41-4acd-a0d0-9f87d324ec72", "zones": [ "default" ] } Since there is no realm set, I can skip radosgw-admin period update --commit entirely if grok is to be believed? On Wed, Apr 15, 2026 at 11:54 AM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > From my understanding the new placement will not be available if the > period can't be updated. Your zonegroup/zone configuration seems off > compared to the test cluster. Is there a default zonegroup/zone? > > Zitat von Rok Jaklič via ceph-users <ceph-users(a)ceph.io>: > > > So I've created a little plan on how to create a new profile and assign > new > > S3 users to EC 8+3 pool. Idea is that all new s3 users are using a new > > pool, and old ones pool with EC 3+2 and data is intact. > > > > --- > > ceph osd erasure-code-profile set ec-83-profile k=8 m=3 > > crush-failure-domain=host > > ceph osd pool create rgw.buckets.ec-83.data 256 erasure ec-83-profile > > ceph osd pool application enable rgw.buckets.ec-83.data rgw > > > > radosgw-admin zonegroup placement add --placement-id=ec-83-placement > > > > radosgw-admin zone placement add \ > > --placement-id ec-83-placement \ > > --data-pool rgw.buckets.ec-83.data \ > > --index-pool default.rgw.buckets.index \ > > --data-extra-pool default.rgw.buckets.non-ec \ > > --placement-index-type 0 > > > > radosgw-admin period update --commit > > > > # restart rgw > > systemctl restart ceph-radosgw.target > > # verify the new placement > > radosgw-admin zone get | grep -A 20 "ec-83-placement" > > > > # for new s3 users, set the default placement > > radosgw-admin user create --uid=<user> --tenant=<tenant> > > --placement-id=ec-83-placement > > --- > > > > It seems to work on "test cluster", however I get an error on: > > > > [root@a8-mon1 ~]# radosgw-admin period update --commit > > cannot commit period: period does not have a master zone of a master > > zonegroup > > failed to commit period: (22) Invalid argument > > > > Do I actually need to run this command or is "rgw restart" enough? > > > > Any comments appriciated. > > > > Rok > > > > > > On Fri, Apr 3, 2026 at 12:42 PM Rok Jaklič via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Hi, > >> > >> we expanded the cluster to 12 nodes with 28 osds, but we are still > running > >> on EC 3+2, since we started cluster with 6 nodes. We would like to > change > >> it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too > much > >> CPU/network capacities? Should I use isa/isa-l instead of jerasure? > >> > >> Can anyone share experience with 8+3? > >> > >> Kind regards, > >> Rok > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Upgrade to tentacle failed! (Now recovered, upgrade paused)
by Johan 15 Apr '26

15 Apr '26

Im running two private ceph clusters. Both are cephadm/docker on debian. The first cluster was upgraded without problem to latest tentacle (19.2.3 -> 20.2.1) using ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.1 Today I tried to upgrade the second cluster. During the upgrade, when a couple of managers had been upgraded, the upgrade process stopped with the error message Orchestrator is unavailable: Module not found and the suggestion I should enable it (ceph mgr module enable orchestrator). From here Im not exactly certain about the errors/messages I received, but I could not enable the module (it already was) and several pages on the dashboard didn't give any information as it apparently dependent on the orchestrator. As I neither were near the servers nor had the time to deal with the failure I chose to shutdown the servers. I have now restarted the servers and after a while the cluster got up again. The dashboard (Administration/Upgrade) now shows version 20.2.1 and "Upgrade is paused". Two managers (incl the active) are upgraded and the third is old version(19.2.3). Everything else seems old version (logical, mgr is upgraded first and the error occured during mgr upgrade). When I got the cluster running again ceph orch uprade status showed { "in_progress": true, "target_image": "quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337", "services_complete": [], "which": "Upgrading all daemon types on all hosts", "progress": "2/52 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } Any suggestions regarding the continued upgrade? Resume the upgrade from the dashboard? First manually upgrade the last manager and then resume the normal upgrade process? or... ? Anyone seen anything like this before or understand the reason for the failure? /Johan

1 0