- ceph-users - lists.ceph.io

User Experience upgrading Hyperconverged CEPH Quincy on Proxmox 8.4 to Proxmox 9.x and CEPH tentacle.
by Tino Todino 15 Sep '26

15 Sep '26

Hi folks, We have a 7 node Hyperconverged CEPH Quincy cluster with 3 OSD's per host. So small fry really 🙂 We are upgrading our backbone from 10 to 25Gbe and as part of the process I'd like to upgrade the nodes from Promxox 8.4 to the latest version 9.2 (iirc). As part of this, CEPH needs to be upgraded. I was wondering if anyone has done this before and what the experience was like please? So, gotcha's, learning points, must do's, never do's etc. If no-one has tried this, I guess I'll be the one updating the mailing list with the findings, but that won't be until November most likely. 🙂 Thanks Tino Coastsense Ltd This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.

3 2

Experience with SEDs in Ceph clusters
by Serhat Rıfat Demircan 15 Sep '26

15 Sep '26

Hi all, Is anyone using Self-Encrypting Drives (SEDs) in production Ceph clusters? I’d be interested to hear about your experience, especially any operational issues, compatibility problems, performance impact, or unexpected behavior you have encountered. Also, if there are any Ceph-specific considerations or recommendations when using SEDs, I’d appreciate your feedback. Thanks, Serhat

2 2

Activity Sent From Your Own Account
by ceph-users-owner＠ceph.io 14 Sep '26

14 Sep '26

Your message to the ceph-users mailing-list was rejected for the following reasons: The message is not from a list member The original message as received by Mailman is attached.

1 0

Re: Tentacle 20.2.4: OSDs crash in PGLog::merge_log after monitor was rebuilt with --mkfs
by Eugen Block 14 Sep '26

14 Sep '26

Please don't drop the list off your responses. I'm not entirely sure yet what to make of the epochs. I'm thinking about mon_min_osdmap_epochs (default 500) and similar parameters. But I'm heading out on vacation now, hopefully someone else will chime in. Good luck! Eugen Zitat von Martin Konold <martin.konold(a)konsec.com>: > Hi Eugen, > > thanks for the quick reply -- one clarification: the monitor was > already rebuilt with --mkfs yesterday evening, before I posted today. > ceph-pve-07.bak is the leftover original store, not something I > am proposing to move. So the question is whether restoring it is > the right way back, not whether to abandon the current one. > >> Did you verify that all service keyrings were successfully >> rotated after the upgrade? > > No, and that turned out to be the core of the auth problem. The > rotation touched the mon DB but not the bluestore osd_key labels: > > osd.2 mon DB AgBf7qJq... label/tmpfs AgA8cKJq... > osd.5 mon DB AgBg7qJq... label/tmpfs AgA8cKJq... > > osd.0, osd.3 and osd.9 are still in that state -- they run fine > on existing sessions but hold pre-rotation keys in their labels, > so they would not come back after a restart. set-label-key > refuses to write while the OSD is up, so I cannot fix that > pre-emptively. > > Caps were inconsistent too: osd.0 and osd.3 have > "caps mon = allow rwx" (the pre-profile form), osd.1 and osd.9 > were missing "caps mgr" entirely. > >> You could have allowed the insecure cipher again > > I did, that was the first thing after finding the monmap policy: > > ceph mon set auth_allowed_ciphers aes,aes256k > > That fixed authentication -- osd.2 and osd.5 now get through and > reach peering. client.admin was fine throughout, so the command > went through without trouble. > > What remains is the PGLog assert, and it is not an auth problem. > Checking the timestamps: osd.5 first hit the assert at 21:04:21, > while its monclient log at 21:04:15 shows a healthy session > (wait_auth_rotating done, uptodate secrets). The auth failures > only start at 21:24. So the crashes came first; the cipher > lockout landed on top and obscured them. > > The assert shows an empty authoritative log against a populated > local one: > > merge_log log((573038'28581091,573038'28581091], > crt=573038'28581091) from osd.3 > into log((573040'28581121,573042'28581681], > crt=573042'28581681) > > Also worth noting: at 21:04 osd.2 had to re-add itself to the > CRUSH map ("osd crush create-or-move id 2 weight 7.2774 > host=pve-08"), which suggests the rebuilt store did not know it. > My working assumption is that the incomplete CRUSH map during > that window let writes be acked against acting sets that later > could not reconcile. > > So the open question stands: with the original store sitting at > osdmap 573036 and the divergence starting at 573038, is a > restore the right approach, and how should the ~500 epoch gap to > the OSDs be handled? > > root@pve-07:~# ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 43.66434 root default > -61 14.55478 host pve-07 > 0 nvme 7.27739 osd.0 up 1.00000 1.00000 > 3 nvme 7.27739 osd.3 up 1.00000 1.00000 > -66 14.55478 host pve-08 > 2 nvme 7.27739 osd.2 down 1.00000 1.00000 > 5 nvme 7.27739 osd.5 down 1.00000 1.00000 > -71 14.55478 host pve-09 > 1 nvme 7.27739 osd.1 down 1.00000 1.00000 > 9 nvme 7.27739 osd.9 up 1.00000 1.00000 > > > Kind Regards, > --martin > > Am Freitag, September 11, 2026 10:56 CEST, schrieb Eugen Block > <ceph-users(a)ceph.io>: > >> Hi, >> >> that seems a little drastic to move the mon store. Did you verify that >> all service keyrings (mgr,mon,osd,mds) were successfully rotated after >> the upgrade? >> >> You could have allowed the insecure cipher again to see if that >> resolves the issue and finish the upgrade (if steps were missing). >> >> ceph mon set auth_allowed_ciphers aes,aes256k >> >> I assume you rotated the admin keyring as well so changing >> auth_allowed_ciphers would be possible? >> >> Zitat von Martin Konold <ceph-users(a)ceph.io>: >> >> > Hi, >> > >> > Yesterday we got an update via Proxmox. The initial update went well >> > but hell broke loose when I tried to dissallow the old aes >> > authentification in order top get rid of a warning. >> > >> > I'm looking for guidance before taking any further destructive >> > action on a 3-node Proxmox cluster (6 OSDs, one replicated pool >> > 3/2, ~2.5 TiB). No users on it, no time pressure. >> > >> > Timeline >> > -------- >> > 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: >> > auth_allowed_ciphers aes256k >> > auth_preferred_cipher aes >> > New keys are created as aes but aes is not allowed >> > to authenticate. Any OSD restarted after this point >> > was locked out. >> > 10.09 15:20 last good osdmap in the original mon store: epoch >> > 573036, all 6 OSDs up/in. >> > 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, >> > 72 MB, 26k keys, intact). >> > 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has >> > 56 auth keys vs 128 in the original, 28 mon_config_key >> > vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- >> > clearly a reconstruction. >> > 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was >> > still working at this point (wait_auth_rotating done, >> > and osd.2 had to re-add itself via "osd crush >> > create-or-move"). >> > 10.09 21:24 auth failures start on top of the crashes (handle_auth >> > _bad_method). Resolved since: mon key was rotated >> > without updating the bluestore osd_key label. >> > >> > Current state >> > ------------- >> > 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx >> > sessions. osd.1, osd.2, osd.5 crash during peering: >> > >> > ./src/osd/PGLog.cc: 401: FAILED ceph_assert( >> > log.head >= olog.tail && olog.head >= log.tail) >> > PeeringState::Stray::react(MLogRec const&) -> merge_log >> > >> > The merge_log lines show an *empty* authoritative log being sent >> > against a populated local one: >> > >> > merge_log log((573038'28581091,573038'28581091], >> > crt=573038'28581091) from osd.3 >> > into log((573040'28581121,573042'28581681], >> > crt=573042'28581681) >> > >> > So the surviving OSDs are behind, and the crashing ones hold log >> > entries from epochs 573038-573042 that the primary has no record >> > of. That range starts two epochs after the last osdmap in the >> > original store. >> > >> > original mon store max osdmap 573036 >> > divergent pg log entries 573038-573042 >> > current osdmap (after restarts) 573538 >> > >> > ceph -s (before I stopped the crashing OSDs): 278/708728 objects >> > unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down. >> > >> > Questions >> > --------- >> > 1. Is restoring the original mon store the right approach here? >> > It is only 2 epochs behind where the divergence starts, which >> > seems like the best possible starting point. >> > 2. How should the 502-epoch gap be handled -- the OSDs have >> > osdmaps far newer than the restored monitor would have? >> > 3. Is the assert itself worth a tracker issue? An empty >> > authoritative log against a populated local one looks like a >> > case that should fall back to backfill rather than assert. >> > >> > What I have NOT done: no mark_unfound_lost, no min_size 1, no >> > ceph-objectstore-tool remove. All three mon stores >> > are backed up. The original store is untouched. >> > >> > One more constraint: the bluestore osd_key labels on the three >> > running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. >> > They run fine now but would not come back after a restart, and >> > set-label-key refuses to write while the OSD is up. So any plan >> > involving an OSD restart has to fix the label in the same stop/ >> > start window. >> > >> > Happy to provide full logs with debug_osd=20, ceph pg query >> > output, or monstore dumps. >> > >> > Thanks, >> > --martin >> > >> > -- >> > martin.konold(a)konsec.com >> > Phone: +49 711 21 72 74-90 >> > KONSEC GmbH -⁠ make things real >> > Amtsgericht Stuttgart, HRB 23690 >> > Geschäftsführer: Andreas Mack >> > Im Köller 3, 70794 Filderstadt, Germany >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > martin.konold(a)konsec.com > Phone: +49 711 21 72 74-90 > KONSEC GmbH -⁠ make things real > Amtsgericht Stuttgart, HRB 23690 > Geschäftsführer: Andreas Mack > Im Köller 3, 70794 Filderstadt, Germany

2 1

Tentacle: Emergency Allowed Ciphers
by Eugen Block 11 Sep '26

11 Sep '26

Hi, I was trying to use the mon_auth_emergency_allowed_ciphers setting to rescue a test cluster as described in the docs [0]. It's a cephadm managed cluster where I set auth_allowed_ciphers to aes256k before rotating the admin keyring. Then I added mon_auth_emergency_allowed_ciphers = true to all mon's config files: reef1:~ # vi /var/lib/ceph/{FSID}/mon.reef1/config and restarted the mon services. According to the config dump the setting is active (for all 3 daemons): [ceph: root@reef2 /]# ceph daemon mon.reef2 config show | grep cipher "mon_auth_emergency_allowed_ciphers": "true", The daemon logs this during startup but I'm not sure if it's relevant right now: Sep 11 10:25:08 reef2 ceph-{FSID}-mon-reef2[88175]: 2026-09-11T08:25:08.611+0000 7f0a8765bd80 -1 mon.reef2@-1(???) e6 init: invalid cipher: true I can't use the unrotated admin keyring although the docs state that I should be able to: > This will allow your existing client.admin or other administrative > key to authenticate as normal. What am I missing? Thanks. Eugen [0] https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h…

3 4

CFPs Reminder
by Anthony Middleton 11 Sep '26

11 Sep '26

Happy Friday, Ceph friends! Just a reminder that the CFPs for Ceph Days Zürich <https://docs.google.com/forms/d/e/1FAIpQLSfUAulFnfdrHesAi92Y2Dkkm7whEXQBel1…> and Ceph Meetup Melbourne <https://docs.google.com/forms/d/e/1FAIpQLSdDgJGT1AyhJFaXPrjvx0X9lYSasU1Iju8…> are still open. Please let me know if you have any questions, and I can connect you with the event organizers. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Tentacle 20.2.4: OSDs crash in PGLog::merge_log after monitor was rebuilt with --mkfs
by Martin Konold 11 Sep '26

11 Sep '26

Hi, Yesterday we got an update via Proxmox. The initial update went well but hell broke loose when I tried to dissallow the old aes authentification in order top get rid of a warning. I'm looking for guidance before taking any further destructive action on a 3-node Proxmox cluster (6 OSDs, one replicated pool 3/2, ~2.5 TiB). No users on it, no time pressure. Timeline -------- 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: auth_allowed_ciphers aes256k auth_preferred_cipher aes New keys are created as aes but aes is not allowed to authenticate. Any OSD restarted after this point was locked out. 10.09 15:20 last good osdmap in the original mon store: epoch 573036, all 6 OSDs up/in. 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, 72 MB, 26k keys, intact). 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has 56 auth keys vs 128 in the original, 28 mon_config_key vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- clearly a reconstruction. 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was still working at this point (wait_auth_rotating done, and osd.2 had to re-add itself via "osd crush create-or-move"). 10.09 21:24 auth failures start on top of the crashes (handle_auth _bad_method). Resolved since: mon key was rotated without updating the bluestore osd_key label. Current state ------------- 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx sessions. osd.1, osd.2, osd.5 crash during peering: ./src/osd/PGLog.cc: 401: FAILED ceph_assert( log.head >= olog.tail && olog.head >= log.tail) PeeringState::Stray::react(MLogRec const&) -> merge_log The merge_log lines show an *empty* authoritative log being sent against a populated local one: merge_log log((573038'28581091,573038'28581091], crt=573038'28581091) from osd.3 into log((573040'28581121,573042'28581681], crt=573042'28581681) So the surviving OSDs are behind, and the crashing ones hold log entries from epochs 573038-573042 that the primary has no record of. That range starts two epochs after the last osdmap in the original store. original mon store max osdmap 573036 divergent pg log entries 573038-573042 current osdmap (after restarts) 573538 ceph -s (before I stopped the crashing OSDs): 278/708728 objects unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down. Questions --------- 1. Is restoring the original mon store the right approach here? It is only 2 epochs behind where the divergence starts, which seems like the best possible starting point. 2. How should the 502-epoch gap be handled -- the OSDs have osdmaps far newer than the restored monitor would have? 3. Is the assert itself worth a tracker issue? An empty authoritative log against a populated local one looks like a case that should fall back to backfill rather than assert. What I have NOT done: no mark_unfound_lost, no min_size 1, no ceph-objectstore-tool remove. All three mon stores are backed up. The original store is untouched. One more constraint: the bluestore osd_key labels on the three running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. They run fine now but would not come back after a restart, and set-label-key refuses to write while the OSD is up. So any plan involving an OSD restart has to fix the label in the same stop/ start window. Happy to provide full logs with debug_osd=20, ceph pg query output, or monstore dumps. Thanks, --martin -- martin.konold(a)konsec.com Phone: +49 711 21 72 74-90 KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

2 1

ceph squid nfs export create uses old keyring format
by Meekhof, Benjeman 10 Sep '26

10 Sep '26

We recently upgraded from 19.2.3 to 19.2.6. Cluster is deployed with cephadm and I used ceph orch to do the upgrade. I've encountered a small issue with new aes256 keyrings and the 'ceph nfs export create' command. When I disable insecure keyring creation ('ceph config set mon mon_auth_allow_insecure_key false') the nfs export create command fails: ceph nfs export create cephfs --cluster-id clusterid --pseudo-path /test --fsname fsname --path=/test Error EINVAL: Expecting value: line 1 column 1 (char 0) When I set allow insecure key to true and try the command again it creates the export but with an insecure key that generates health warnings on the cluster. The export works fine as far as mounting it and seeing the data I expect. thanks, Ben

2 1

whoX
by ivax.ai 09 Sep '26

09 Sep '26

whoX por ivaX.ai whoX ( https://ivax.ai ) ***************************************************************** Logo incluido. Plazo de entrega 25 días después de aprobación diseño. ***************************************************************** ****************************** +34 938 424 303 x(a)ivax.ai ****************************** AVISO LEGAL COMPLETO DE PRIVACIDAD Y PROTECCIÓN DE DATOS PERSONALES (LGPD) 1. Identificación del Controlador de Datos: De conformidad con las disposiciones de la Ley General de Protección de Datos Personales de Brasil (Ley Nº 13.709/2018 - LGPD), se informa al titular de los datos que la entidad responsable del tratamiento de su información personal es Worxfly SL, con domicilio enc/ Velázquez, 27, 28001, Madrid, en adelante denominada el "Controlador". 2. Datos Recopilados y Base Legal del Tratamiento: Para el envío de este boletín informativo (newsletter), comunicaciones operativas, novedades, contenidos educativos y campañas de marketing directo, el Controlador trata datos personales limitados a: su nombre completo, dirección de correo electrónico. La base legal que legitima el tratamiento de estos datos es el Consentimiento libre, informado e inequívoco del titular (Art. 7, inciso I de la LGPD), otorgado de forma voluntaria al suscribirse a nuestros canales de comunicación, o bien el Interés Legítimo del Controlador (Art. 7, inciso IX de la LGPD) para el desarrollo y promoción de nuestras actividades, siempre respetando las expectativas legítimas del usuario. 3. Finalidad del Tratamiento y Principios de la LGPD: En estricto cumplimiento del principio de finalidad y necesidad (Art. 6, I y III de la LGPD), sus datos serán utilizados única y exclusivamente para: (i) Gestionar su suscripción y remitir el newsletter periódico; (ii) Personalizar el contenido enviado en función de sus interacciones legítimas; (iii) Analizar métricas de apertura y clics para optimizar la calidad de los envíos. Nos comprometemos a no utilizar sus datos para finalidades distintas a las aquí informadas sin su autorización previa. 4. Compartición y Transferencia Internacional de Datos: El Controlador no comercializa, cede ni comparte sus datos personales con terceros con fines lucrativos. No obstante, para la ejecución técnica de los envíos, sus datos son alojados e integrados en plataformas de automatización de email marketing subcontratadas (Operadores de Datos). Dichos proveedores pueden almacenar la información en servidores ubicados fuera del territorio nacional (como en los EE.UU. o la Unión Europea), configurando una Transferencia Internacional de Datos (Art. 33 de la LGPD). El Controlador garantiza que utiliza proveedores que cumplen con los más estrictos estándares de seguridad y que están alineados con cláusulas contractuales contractuales tipo para la protección de datos. 5. Período de Retención de la Información: Sus datos personales se conservarán en nuestra base de datos de forma indefinida mientras se mantenga activa su suscripción y persista el interés mutuo. Se procederá al bloqueo o eliminación definitiva de sus datos cuando: (a) El titular revoque el consentimiento manifestado; (b) Los datos dejen de ser necesarios para las finalidades que motivaron su recogida; o (c) Por mandato legal o requerimiento de la Autoridad Nacional de Protección de Datos (ANPD). 6. Derechos del Titular (Art. 18 de la LGPD): Le recordamos que usted, como titular de los datos, tiene derecho a ejercer en cualquier momento y de forma gratuita ante el Controlador los siguientes derechos: * Confirmación de la existencia del tratamiento de sus datos. * Acceso completo a los datos personales almacenados. * Corrección de datos incompletos, inexactos o desactualizados. * Anonimización, bloqueo o eliminación de datos innecesarios o tratados en desconformidad con la ley. * Portabilidad de sus datos a otro proveedor de servicios. * Eliminación de los datos personales tratados bajo el amparo de su consentimiento. * Información sobre las entidades públicas y privadas con las que el controlador realizó un uso compartido de datos. * Información sobre la posibilidad de no proporcionar el consentimiento y las consecuencias de dicha negativa. * Revocación del consentimiento otorgado previamente. En cumplimiento de lo dispuesto en la Protección de Datos, para el ejercicio de sus derechos de acceso, rectificación, cancelación y oposición al tratamiento de sus datos personales, contenidos en nuestras condiciones de protección de datos, solamente tiene que responder a este e-mail indicando su email en el asunto, o bien a través del envío de un correo ordinario a la dirección: INCLUIR DIRECCIÓN. Si desea darse de baja también puede hacer clic aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) . FULL LEGAL NOTICE ON PRIVACY AND PERSONAL DATA PROTECTION (LGPD) 1. Identification of the Data Controller: In accordance with the provisions of the Brazilian General Data Protection Law (Law No. 13.709/2018 - LGPD), the data subject is informed that the entity responsible for processing their personal information is Worxfly SL, with registered office at Calle Velázquez, 27, 28001, Madrid, hereinafter referred to as the "Controller". 2. Data Collected and Legal Basis for Processing: For the purpose of sending this newsletter, operational communications, news, educational content, and direct marketing campaigns, the Controller processes personal data limited to: your full name and email address. The legal basis for processing this data is the free, informed, and unambiguous consent of the data subject (Article 7, paragraph I of the LGPD), given voluntarily when subscribing to our communication channels, or the legitimate interest of the data controller (Article 7, paragraph IX of the LGPD) for the development and promotion of our activities, always respecting the user's legitimate expectations. 3. Purpose of Processing and LGPD Principles: In strict compliance with the principle of purpose limitation and necessity (Article 6, paragraphs I and III of the LGPD), your data will be used solely and exclusively for: (i) Managing your subscription and sending the periodic newsletter; (ii) Personalizing the content sent based on your legitimate interactions; (iii) Analyzing open and click-through rates to optimize the quality of our mailings. We commit to not using your data for purposes other than those stated here without your prior authorization. 4. Sharing and International Transfer of Data: The Controller does not sell, transfer, or share your personal data with third parties for profit. However, for the technical execution of mailings, your data is hosted and integrated into subcontracted email marketing automation platforms (Data Operators). These providers may store the information on servers located outside of Spain (such as in the USA or the European Union), constituting an International Data Transfer (Art. 33 of the LGPD). The Controller guarantees that it uses providers that comply with the strictest security standards and are aligned with standard contractual clauses for data protection. 5. Data Retention Period: Your personal data will be kept in our database indefinitely as long as your subscription remains active and mutual interest persists. Your data will be blocked or permanently deleted when: (a) You revoke your consent; (b) The data is no longer necessary for the purposes for which it was collected; or (c) By legal mandate or request of the National Data Protection Authority (ANPD). 6. Data Subject Rights (Art. 18 of the LGPD): We remind you that you, as the data subject, have the right to exercise the following rights at any time and free of charge with the Controller: Confirmation of the existence of the processing of your data. Full access to the personal data stored. Correction of incomplete, inaccurate, or outdated data. Anonymization, blocking, or deletion of unnecessary data or data processed unlawfully. Portability of your data to another service provider. Deletion of personal data processed based on your consent. Information about the public and private entities with which the controller has shared data. Information about the possibility of not providing consent and the consequences of such refusal. Revocation of previously granted consent. In compliance with data protection regulations, to exercise your rights of access, rectification, erasure, and objection to the processing of your personal data, as outlined in our data protection policy, simply reply to this email with your email address in the subject line, or send a letter to the following address: [INSERT ADDRESS]. You can also unsubscribe by clicking aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) . MENTIONS LÉGALES COMPLÈTES RELATIVES À LA PROTECTION DES DONNÉES PERSONNELLES (LGPD) 1. Identification du responsable du traitement : Conformément aux dispositions de la Loi générale brésilienne sur la protection des données (Loi n° 13.709/2018 – LGPD), la personne concernée est informée que l’entité responsable du traitement de ses données personnelles est Worxfly SL, dont le siège social est situé Calle Velázquez, 27, 28001 Madrid, ci-après dénommée le « Responsable du traitement ». 2. Données collectées et base juridique du traitement : Aux fins de l’envoi de cette newsletter, de communications opérationnelles, d’actualités, de contenus pédagogiques et de campagnes de marketing direct, le Responsable du traitement traite les données personnelles suivantes : vos nom et prénom ainsi que votre adresse électronique. Le traitement de ces données est fondé sur le consentement libre, éclairé et univoque de la personne concernée (article 7, paragraphe 1 de la LGPD), donné volontairement lors de l'inscription à nos communications, ou sur l'intérêt légitime du responsable du traitement (article 7, paragraphe IX de la LGPD) pour le développement et la promotion de ses activités, dans le respect des attentes légitimes de l'utilisateur. 3. Finalité du traitement et principes de la LGPD : Conformément au principe de limitation des finalités et de nécessité (article 6, paragraphes 1 et 3 de la LGPD), vos données seront utilisées exclusivement pour : (i) gérer votre abonnement et vous envoyer la newsletter ; (ii) personnaliser le contenu envoyé en fonction de vos interactions légitimes ; (iii) analyser les taux d'ouverture et de clics afin d'optimiser la qualité de nos communications. Nous nous engageons à ne pas utiliser vos données à d'autres fins que celles mentionnées ici sans votre autorisation préalable. 4. Partage et transfert international des données : Le responsable du traitement ne vend, ne transfère ni ne partage vos données personnelles avec des tiers à des fins lucratives. Toutefois, pour l'exécution technique des envois, vos données sont hébergées et intégrées à des plateformes d'automatisation marketing par courriel sous-traitées (Opérateurs de données). Ces prestataires peuvent stocker les informations sur des serveurs situés hors d'Espagne (notamment aux États-Unis ou dans l'Union européenne), ce qui constitue un transfert international de données (article 33 de la LGPD). Le Responsable du traitement garantit qu'il fait appel à des prestataires respectant les normes de sécurité les plus strictes et utilisant des clauses contractuelles types en matière de protection des données. 5. Durée de conservation des données : Vos données personnelles seront conservées dans notre base de données indéfiniment, tant que votre abonnement reste actif et que l'intérêt mutuel persiste. Vos données seront bloquées ou supprimées définitivement dans les cas suivants : (a) vous retirez votre consentement ; (b) les données ne sont plus nécessaires aux fins pour lesquelles elles ont été collectées ; ou (c) sur instruction légale ou à la demande de l'Autorité nationale de protection des données (ANPD). 6. Droits de la personne concernée (Art. 18 de la LGPD) : Nous vous rappelons que vous disposez, en tant que personne concernée, du droit d’exercer à tout moment et gratuitement les droits suivants auprès du responsable du traitement : Confirmation de l’existence du traitement de vos données. Accès complet aux données personnelles conservées. Rectification des données incomplètes, inexactes ou obsolètes. Anonymisation, blocage ou suppression des données inutiles ou traitées illégalement. Portabilité de vos données vers un autre prestataire de services. Suppression des données personnelles traitées sur la base de votre consentement. Informations sur les organismes publics et privés avec lesquels le responsable du traitement a partagé des données. Informations sur la possibilité de ne pas donner son consentement et les conséquences d’un tel refus. Retrait d’un consentement précédemment accordé. Conformément à la réglementation sur la protection des données, pour exercer vos droits d'accès, de rectification, d'effacement et d'opposition au traitement de vos données personnelles, tels que décrits dans notre politique de protection des données, il vous suffit de répondre à ce courriel en indiquant votre adresse électronique dans l'objet, ou d'envoyer un courrier à l'adresse suivante : [INSÉRER L'ADRESSE]. Vous pouvez également vous désabonner en cliquant aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) .

1 0

Re: Cephadm staggered upgrade --limit parameter not always applied
by Andreas Eriksson 09 Sep '26

09 Sep '26

I'm doing further tests in a small cluster. Wondering a bit about the rotation of keys, since when I later upgrade the RGW service the mon,mgr and osds are restarted due to key rotation as well. I'm wondering if this is intentional? 3 nodes running mon/mgr/monitoring, 3 OSD nodes with additional RGW daemons. Starting at v17.2.7, upgrading all but the RGWs to v19.2.6 # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon_types mgr,mon,osd,crash Upgrade RGWs # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --services rgw.vps-lab1 On 8/26/26 5:37 PM, Eugen Block wrote: > I started to conduct some tests and document in which version the > limits worked. I’m not done yet, I’ll update the respective tracker > issue when I get there. > To perform the staggered update you could just redeploy rgw daemons > with a new image after all preceding upgrades have finished (mgr, mon, > crash, osd, mds) with ceph orch daemon redeploy rgw.<some-rgw> <image>. > > I don’t have the exact syntax at hand right now, but it should be > quite similar. > > > Zitat von Eugen Block <eblock(a)nde.ag>: > >> Hi, >> >> I haven't found anything explaining this yet, but I also haven't >> looked for a while. I just retried limiting RGW upgrade (from Reef to >> Tentacle): >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 >> --daemon-types rgw --limit 1 >> >> But the limit is still ignored: >> >> 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) >> 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) >> 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) >> 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) >> 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) >> 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1) >> >> All three RGW daemons have been upgraded automatically. I'm pretty >> sure that it worked at some point with RGWs, in my notes from one of >> our customer upgrades where this worked: >> >> ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 >> --daemon-types rgw --limit 1 >> >> I can't tell when it stopped working properly. >> >> Regards, >> Eugen >> >> >> Zitat von Andreas Eriksson <norrland(a)nullbyte.se>: >> >>> Hi, >>> >>> Were you able to find anything more about this? I'm looking at >>> upgrading a bunch of clusters from previous major versions, but do >>> not want to upgrade all RGW services. And also limit the number of >>> osds at the time that are being upgraded, to keep the recovery >>> traffic lower for example. >>> >>> Br, >>> >>> On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote: >>>> Adding the dev list >>>> >>>> Zitat von Eugen Block <eblock(a)nde.ag>: >>>> >>>>> Hi, >>>>> >>>>> I did a quick search on tracker but couldn't find anything >>>>> related. A customer reported this, and I can confirm the behaviour >>>>> on a lab cluster. I usually perform a staggered upgrade with >>>>> --daemon-types and --limit, but not for all daemon types. So I >>>>> haven't stumbled across this yet myself, but our customer did (I >>>>> can reproduce with --daemon-types as well). They upgraded from >>>>> latest Reef to latest Squid and reported that despite providing >>>>> the --limit parameter, the mons were all upgraded. So I tried to >>>>> reproduce, and the behaviour is not really clear to me, I'll try >>>>> to clarify. >>>>> >>>>> # Start with MGRs >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 >>>>> >>>>> Upgrading MGRs with limit works, but it doesn't reflect in the MGR >>>>> log. Usually, I expect a line like this: >>>>> >>>>> ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade >>>>> >>>>> But there is no such line in the logs. Then I upgrade the rest of >>>>> the MGRs. >>>>> >>>>> # Continue with MONs >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 >>>>> >>>>> This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. >>>>> And again, no such line in the log (Hit upgrade limit). What I >>>>> noticed was a MGR respawn after the first MON had been upgraded >>>>> successfully. Maybe some state of the upgrade progress gets lost >>>>> during the respawn? I then upgraded the remaining MON. Then >>>>> ceph-crash is upgraded successfully. >>>>> >>>>> # Upgrade OSD >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1 >>>>> >>>>> And this is the first service that actually reports the upgrade >>>>> limit: >>>>> >>>>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Updating osd.0 (1/1) >>>>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Updating osd.0 (1/1) >>>>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] >>>>> Hit upgrade limit of 1. Stopping upgrade >>>>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Hit upgrade limit of 1. Stopping upgrade >>>>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Setting container_image for all nvmeof >>>>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Setting container_image for all nvmeof >>>>> 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Finalizing container_image settings >>>>> 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Finalizing container_image settings >>>>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Complete! >>>>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Complete! >>>>> >>>>> >>>>> This is really irritating and inconsistent: If the orchestrator >>>>> does honor --limit with other services than OSD, why isn't that >>>>> visible in the logs? And what's with the MONs? Why 2 out of 3? >>>>> >>>>> Any pointers appreciated! Not sure which Ceph versions might be >>>>> affected by this, I'll try out a couple more upgrade paths. >>>>> >>>>> Thanks, >>>>> Eugen >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2