- ceph-users - lists.ceph.io

Tentacle 20.2.4: OSDs crash in PGLog::merge_log after monitor was rebuilt with --mkfs
by Martin Konold 11 Sep '26

11 Sep '26

Hi, Yesterday we got an update via Proxmox. The initial update went well but hell broke loose when I tried to dissallow the old aes authentification in order top get rid of a warning. I'm looking for guidance before taking any further destructive action on a 3-node Proxmox cluster (6 OSDs, one replicated pool 3/2, ~2.5 TiB). No users on it, no time pressure. Timeline -------- 10.09 10:59 monmap epoch 81 shows an inconsistent cipher policy: auth_allowed_ciphers aes256k auth_preferred_cipher aes New keys are created as aes but aes is not allowed to authenticate. Any OSD restarted after this point was locked out. 10.09 15:20 last good osdmap in the original mon store: epoch 573036, all 6 OSDs up/in. 10.09 ~16:32 original mon store moved aside (now ceph-pve-07.bak, 72 MB, 26k keys, intact). 10.09 20:38 monitor rebuilt with monmaptool/--mkfs. New store has 56 auth keys vs 128 in the original, 28 mon_config_key vs 1162, and caps like "mon = allow rwx" on osd.0/3 -- clearly a reconstruction. 10.09 21:04 osd.2 and osd.5 crash in PGLog::merge_log. Auth was still working at this point (wait_auth_rotating done, and osd.2 had to re-add itself via "osd crush create-or-move"). 10.09 21:24 auth failures start on top of the crashes (handle_auth _bad_method). Resolved since: mon key was rotated without updating the bluestore osd_key label. Current state ------------- 3 of 6 OSDs (0, 3, 9) are up and running on pre-existing cephx sessions. osd.1, osd.2, osd.5 crash during peering: ./src/osd/PGLog.cc: 401: FAILED ceph_assert( log.head >= olog.tail && olog.head >= log.tail) PeeringState::Stray::react(MLogRec const&) -> merge_log The merge_log lines show an *empty* authoritative log being sent against a populated local one: merge_log log((573038'28581091,573038'28581091], crt=573038'28581091) from osd.3 into log((573040'28581121,573042'28581681], crt=573042'28581681) So the surviving OSDs are behind, and the crashing ones hold log entries from epochs 573038-573042 that the primary has no record of. That range starts two epochs after the last osdmap in the original store. original mon store max osdmap 573036 divergent pg log entries 573038-573042 current osdmap (after restarts) 573538 ceph -s (before I stopped the crashing OSDs): 278/708728 objects unfound, 40 pgs recovery_unfound, 11 incomplete, 11 down. Questions --------- 1. Is restoring the original mon store the right approach here? It is only 2 epochs behind where the divergence starts, which seems like the best possible starting point. 2. How should the 502-epoch gap be handled -- the OSDs have osdmaps far newer than the restored monitor would have? 3. Is the assert itself worth a tracker issue? An empty authoritative log against a populated local one looks like a case that should fall back to backfill rather than assert. What I have NOT done: no mark_unfound_lost, no min_size 1, no ceph-objectstore-tool remove. All three mon stores are backed up. The original store is untouched. One more constraint: the bluestore osd_key labels on the three running OSDs (0, 3, 9) are stale -- they hold pre-rotation keys. They run fine now but would not come back after a restart, and set-label-key refuses to write while the OSD is up. So any plan involving an OSD restart has to fix the label in the same stop/ start window. Happy to provide full logs with debug_osd=20, ceph pg query output, or monstore dumps. Thanks, --martin -- martin.konold(a)konsec.com Phone: +49 711 21 72 74-90 KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

2 1

ceph squid nfs export create uses old keyring format
by Meekhof, Benjeman 11 Sep '26

11 Sep '26

We recently upgraded from 19.2.3 to 19.2.6. Cluster is deployed with cephadm and I used ceph orch to do the upgrade. I've encountered a small issue with new aes256 keyrings and the 'ceph nfs export create' command. When I disable insecure keyring creation ('ceph config set mon mon_auth_allow_insecure_key false') the nfs export create command fails: ceph nfs export create cephfs --cluster-id clusterid --pseudo-path /test --fsname fsname --path=/test Error EINVAL: Expecting value: line 1 column 1 (char 0) When I set allow insecure key to true and try the command again it creates the export but with an insecure key that generates health warnings on the cluster. The export works fine as far as mounting it and seeing the data I expect. thanks, Ben

2 1

whoX
by ivax.ai 09 Sep '26

09 Sep '26

whoX por ivaX.ai whoX ( https://ivax.ai ) ***************************************************************** Logo incluido. Plazo de entrega 25 días después de aprobación diseño. ***************************************************************** ****************************** +34 938 424 303 x(a)ivax.ai ****************************** AVISO LEGAL COMPLETO DE PRIVACIDAD Y PROTECCIÓN DE DATOS PERSONALES (LGPD) 1. Identificación del Controlador de Datos: De conformidad con las disposiciones de la Ley General de Protección de Datos Personales de Brasil (Ley Nº 13.709/2018 - LGPD), se informa al titular de los datos que la entidad responsable del tratamiento de su información personal es Worxfly SL, con domicilio enc/ Velázquez, 27, 28001, Madrid, en adelante denominada el "Controlador". 2. Datos Recopilados y Base Legal del Tratamiento: Para el envío de este boletín informativo (newsletter), comunicaciones operativas, novedades, contenidos educativos y campañas de marketing directo, el Controlador trata datos personales limitados a: su nombre completo, dirección de correo electrónico. La base legal que legitima el tratamiento de estos datos es el Consentimiento libre, informado e inequívoco del titular (Art. 7, inciso I de la LGPD), otorgado de forma voluntaria al suscribirse a nuestros canales de comunicación, o bien el Interés Legítimo del Controlador (Art. 7, inciso IX de la LGPD) para el desarrollo y promoción de nuestras actividades, siempre respetando las expectativas legítimas del usuario. 3. Finalidad del Tratamiento y Principios de la LGPD: En estricto cumplimiento del principio de finalidad y necesidad (Art. 6, I y III de la LGPD), sus datos serán utilizados única y exclusivamente para: (i) Gestionar su suscripción y remitir el newsletter periódico; (ii) Personalizar el contenido enviado en función de sus interacciones legítimas; (iii) Analizar métricas de apertura y clics para optimizar la calidad de los envíos. Nos comprometemos a no utilizar sus datos para finalidades distintas a las aquí informadas sin su autorización previa. 4. Compartición y Transferencia Internacional de Datos: El Controlador no comercializa, cede ni comparte sus datos personales con terceros con fines lucrativos. No obstante, para la ejecución técnica de los envíos, sus datos son alojados e integrados en plataformas de automatización de email marketing subcontratadas (Operadores de Datos). Dichos proveedores pueden almacenar la información en servidores ubicados fuera del territorio nacional (como en los EE.UU. o la Unión Europea), configurando una Transferencia Internacional de Datos (Art. 33 de la LGPD). El Controlador garantiza que utiliza proveedores que cumplen con los más estrictos estándares de seguridad y que están alineados con cláusulas contractuales contractuales tipo para la protección de datos. 5. Período de Retención de la Información: Sus datos personales se conservarán en nuestra base de datos de forma indefinida mientras se mantenga activa su suscripción y persista el interés mutuo. Se procederá al bloqueo o eliminación definitiva de sus datos cuando: (a) El titular revoque el consentimiento manifestado; (b) Los datos dejen de ser necesarios para las finalidades que motivaron su recogida; o (c) Por mandato legal o requerimiento de la Autoridad Nacional de Protección de Datos (ANPD). 6. Derechos del Titular (Art. 18 de la LGPD): Le recordamos que usted, como titular de los datos, tiene derecho a ejercer en cualquier momento y de forma gratuita ante el Controlador los siguientes derechos: * Confirmación de la existencia del tratamiento de sus datos. * Acceso completo a los datos personales almacenados. * Corrección de datos incompletos, inexactos o desactualizados. * Anonimización, bloqueo o eliminación de datos innecesarios o tratados en desconformidad con la ley. * Portabilidad de sus datos a otro proveedor de servicios. * Eliminación de los datos personales tratados bajo el amparo de su consentimiento. * Información sobre las entidades públicas y privadas con las que el controlador realizó un uso compartido de datos. * Información sobre la posibilidad de no proporcionar el consentimiento y las consecuencias de dicha negativa. * Revocación del consentimiento otorgado previamente. En cumplimiento de lo dispuesto en la Protección de Datos, para el ejercicio de sus derechos de acceso, rectificación, cancelación y oposición al tratamiento de sus datos personales, contenidos en nuestras condiciones de protección de datos, solamente tiene que responder a este e-mail indicando su email en el asunto, o bien a través del envío de un correo ordinario a la dirección: INCLUIR DIRECCIÓN. Si desea darse de baja también puede hacer clic aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) . FULL LEGAL NOTICE ON PRIVACY AND PERSONAL DATA PROTECTION (LGPD) 1. Identification of the Data Controller: In accordance with the provisions of the Brazilian General Data Protection Law (Law No. 13.709/2018 - LGPD), the data subject is informed that the entity responsible for processing their personal information is Worxfly SL, with registered office at Calle Velázquez, 27, 28001, Madrid, hereinafter referred to as the "Controller". 2. Data Collected and Legal Basis for Processing: For the purpose of sending this newsletter, operational communications, news, educational content, and direct marketing campaigns, the Controller processes personal data limited to: your full name and email address. The legal basis for processing this data is the free, informed, and unambiguous consent of the data subject (Article 7, paragraph I of the LGPD), given voluntarily when subscribing to our communication channels, or the legitimate interest of the data controller (Article 7, paragraph IX of the LGPD) for the development and promotion of our activities, always respecting the user's legitimate expectations. 3. Purpose of Processing and LGPD Principles: In strict compliance with the principle of purpose limitation and necessity (Article 6, paragraphs I and III of the LGPD), your data will be used solely and exclusively for: (i) Managing your subscription and sending the periodic newsletter; (ii) Personalizing the content sent based on your legitimate interactions; (iii) Analyzing open and click-through rates to optimize the quality of our mailings. We commit to not using your data for purposes other than those stated here without your prior authorization. 4. Sharing and International Transfer of Data: The Controller does not sell, transfer, or share your personal data with third parties for profit. However, for the technical execution of mailings, your data is hosted and integrated into subcontracted email marketing automation platforms (Data Operators). These providers may store the information on servers located outside of Spain (such as in the USA or the European Union), constituting an International Data Transfer (Art. 33 of the LGPD). The Controller guarantees that it uses providers that comply with the strictest security standards and are aligned with standard contractual clauses for data protection. 5. Data Retention Period: Your personal data will be kept in our database indefinitely as long as your subscription remains active and mutual interest persists. Your data will be blocked or permanently deleted when: (a) You revoke your consent; (b) The data is no longer necessary for the purposes for which it was collected; or (c) By legal mandate or request of the National Data Protection Authority (ANPD). 6. Data Subject Rights (Art. 18 of the LGPD): We remind you that you, as the data subject, have the right to exercise the following rights at any time and free of charge with the Controller: Confirmation of the existence of the processing of your data. Full access to the personal data stored. Correction of incomplete, inaccurate, or outdated data. Anonymization, blocking, or deletion of unnecessary data or data processed unlawfully. Portability of your data to another service provider. Deletion of personal data processed based on your consent. Information about the public and private entities with which the controller has shared data. Information about the possibility of not providing consent and the consequences of such refusal. Revocation of previously granted consent. In compliance with data protection regulations, to exercise your rights of access, rectification, erasure, and objection to the processing of your personal data, as outlined in our data protection policy, simply reply to this email with your email address in the subject line, or send a letter to the following address: [INSERT ADDRESS]. You can also unsubscribe by clicking aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) . MENTIONS LÉGALES COMPLÈTES RELATIVES À LA PROTECTION DES DONNÉES PERSONNELLES (LGPD) 1. Identification du responsable du traitement : Conformément aux dispositions de la Loi générale brésilienne sur la protection des données (Loi n° 13.709/2018 – LGPD), la personne concernée est informée que l’entité responsable du traitement de ses données personnelles est Worxfly SL, dont le siège social est situé Calle Velázquez, 27, 28001 Madrid, ci-après dénommée le « Responsable du traitement ». 2. Données collectées et base juridique du traitement : Aux fins de l’envoi de cette newsletter, de communications opérationnelles, d’actualités, de contenus pédagogiques et de campagnes de marketing direct, le Responsable du traitement traite les données personnelles suivantes : vos nom et prénom ainsi que votre adresse électronique. Le traitement de ces données est fondé sur le consentement libre, éclairé et univoque de la personne concernée (article 7, paragraphe 1 de la LGPD), donné volontairement lors de l'inscription à nos communications, ou sur l'intérêt légitime du responsable du traitement (article 7, paragraphe IX de la LGPD) pour le développement et la promotion de ses activités, dans le respect des attentes légitimes de l'utilisateur. 3. Finalité du traitement et principes de la LGPD : Conformément au principe de limitation des finalités et de nécessité (article 6, paragraphes 1 et 3 de la LGPD), vos données seront utilisées exclusivement pour : (i) gérer votre abonnement et vous envoyer la newsletter ; (ii) personnaliser le contenu envoyé en fonction de vos interactions légitimes ; (iii) analyser les taux d'ouverture et de clics afin d'optimiser la qualité de nos communications. Nous nous engageons à ne pas utiliser vos données à d'autres fins que celles mentionnées ici sans votre autorisation préalable. 4. Partage et transfert international des données : Le responsable du traitement ne vend, ne transfère ni ne partage vos données personnelles avec des tiers à des fins lucratives. Toutefois, pour l'exécution technique des envois, vos données sont hébergées et intégrées à des plateformes d'automatisation marketing par courriel sous-traitées (Opérateurs de données). Ces prestataires peuvent stocker les informations sur des serveurs situés hors d'Espagne (notamment aux États-Unis ou dans l'Union européenne), ce qui constitue un transfert international de données (article 33 de la LGPD). Le Responsable du traitement garantit qu'il fait appel à des prestataires respectant les normes de sécurité les plus strictes et utilisant des clauses contractuelles types en matière de protection des données. 5. Durée de conservation des données : Vos données personnelles seront conservées dans notre base de données indéfiniment, tant que votre abonnement reste actif et que l'intérêt mutuel persiste. Vos données seront bloquées ou supprimées définitivement dans les cas suivants : (a) vous retirez votre consentement ; (b) les données ne sont plus nécessaires aux fins pour lesquelles elles ont été collectées ; ou (c) sur instruction légale ou à la demande de l'Autorité nationale de protection des données (ANPD). 6. Droits de la personne concernée (Art. 18 de la LGPD) : Nous vous rappelons que vous disposez, en tant que personne concernée, du droit d’exercer à tout moment et gratuitement les droits suivants auprès du responsable du traitement : Confirmation de l’existence du traitement de vos données. Accès complet aux données personnelles conservées. Rectification des données incomplètes, inexactes ou obsolètes. Anonymisation, blocage ou suppression des données inutiles ou traitées illégalement. Portabilité de vos données vers un autre prestataire de services. Suppression des données personnelles traitées sur la base de votre consentement. Informations sur les organismes publics et privés avec lesquels le responsable du traitement a partagé des données. Informations sur la possibilité de ne pas donner son consentement et les conséquences d’un tel refus. Retrait d’un consentement précédemment accordé. Conformément à la réglementation sur la protection des données, pour exercer vos droits d'accès, de rectification, d'effacement et d'opposition au traitement de vos données personnelles, tels que décrits dans notre politique de protection des données, il vous suffit de répondre à ce courriel en indiquant votre adresse électronique dans l'objet, ou d'envoyer un courrier à l'adresse suivante : [INSÉRER L'ADRESSE]. Vous pouvez également vous désabonner en cliquant aquí ( https://boletines.ivax.ai/unsubscribe/53pdzm2/click?token=q0xuqxy4 ) .

1 0

Re: Cephadm staggered upgrade --limit parameter not always applied
by Andreas Eriksson 09 Sep '26

09 Sep '26

I'm doing further tests in a small cluster. Wondering a bit about the rotation of keys, since when I later upgrade the RGW service the mon,mgr and osds are restarted due to key rotation as well. I'm wondering if this is intentional? 3 nodes running mon/mgr/monitoring, 3 OSD nodes with additional RGW daemons. Starting at v17.2.7, upgrading all but the RGWs to v19.2.6 # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon_types mgr,mon,osd,crash Upgrade RGWs # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --services rgw.vps-lab1 On 8/26/26 5:37 PM, Eugen Block wrote: > I started to conduct some tests and document in which version the > limits worked. I’m not done yet, I’ll update the respective tracker > issue when I get there. > To perform the staggered update you could just redeploy rgw daemons > with a new image after all preceding upgrades have finished (mgr, mon, > crash, osd, mds) with ceph orch daemon redeploy rgw.<some-rgw> <image>. > > I don’t have the exact syntax at hand right now, but it should be > quite similar. > > > Zitat von Eugen Block <eblock(a)nde.ag>: > >> Hi, >> >> I haven't found anything explaining this yet, but I also haven't >> looked for a while. I just retried limiting RGW upgrade (from Reef to >> Tentacle): >> >> reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v20.2.4 >> --daemon-types rgw --limit 1 >> >> But the limit is still ignored: >> >> 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) >> 2026-08-26T10:15:23.709+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef1.pmauyp (1/1) >> 2026-08-26T10:15:26.554+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) >> 2026-08-26T10:15:26.555+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef2.oqrpma (1/1) >> 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 [cephadm INFO >> cephadm.upgrade] Upgrade: Updating rgw.nde.reef3.boujxu (1/1) >> 2026-08-26T10:15:29.133+0000 7f8c09d01640 0 log_channel(cephadm) log >> [INF] : Upgrade: Updating rgw.nde.reef3.boujxu (1/1) >> >> All three RGW daemons have been upgraded automatically. I'm pretty >> sure that it worked at some point with RGWs, in my notes from one of >> our customer upgrades where this worked: >> >> ceph orch upgrade start --image registry:5000/ceph/ceph:v18.2.5 >> --daemon-types rgw --limit 1 >> >> I can't tell when it stopped working properly. >> >> Regards, >> Eugen >> >> >> Zitat von Andreas Eriksson <norrland(a)nullbyte.se>: >> >>> Hi, >>> >>> Were you able to find anything more about this? I'm looking at >>> upgrading a bunch of clusters from previous major versions, but do >>> not want to upgrade all RGW services. And also limit the number of >>> osds at the time that are being upgraded, to keep the recovery >>> traffic lower for example. >>> >>> Br, >>> >>> On 3/17/26 3:35 PM, Eugen Block via ceph-users wrote: >>>> Adding the dev list >>>> >>>> Zitat von Eugen Block <eblock(a)nde.ag>: >>>> >>>>> Hi, >>>>> >>>>> I did a quick search on tracker but couldn't find anything >>>>> related. A customer reported this, and I can confirm the behaviour >>>>> on a lab cluster. I usually perform a staggered upgrade with >>>>> --daemon-types and --limit, but not for all daemon types. So I >>>>> haven't stumbled across this yet myself, but our customer did (I >>>>> can reproduce with --daemon-types as well). They upgraded from >>>>> latest Reef to latest Squid and reported that despite providing >>>>> the --limit parameter, the mons were all upgraded. So I tried to >>>>> reproduce, and the behaviour is not really clear to me, I'll try >>>>> to clarify. >>>>> >>>>> # Start with MGRs >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 >>>>> >>>>> Upgrading MGRs with limit works, but it doesn't reflect in the MGR >>>>> log. Usually, I expect a line like this: >>>>> >>>>> ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade >>>>> >>>>> But there is no such line in the logs. Then I upgrade the rest of >>>>> the MGRs. >>>>> >>>>> # Continue with MONs >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 >>>>> >>>>> This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. >>>>> And again, no such line in the log (Hit upgrade limit). What I >>>>> noticed was a MGR respawn after the first MON had been upgraded >>>>> successfully. Maybe some state of the upgrade progress gets lost >>>>> during the respawn? I then upgraded the remaining MON. Then >>>>> ceph-crash is upgraded successfully. >>>>> >>>>> # Upgrade OSD >>>>> >>>>> reef1:~ # ceph orch upgrade start --image >>>>> quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1 >>>>> >>>>> And this is the first service that actually reports the upgrade >>>>> limit: >>>>> >>>>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Updating osd.0 (1/1) >>>>> 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Updating osd.0 (1/1) >>>>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] >>>>> Hit upgrade limit of 1. Stopping upgrade >>>>> 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Hit upgrade limit of 1. Stopping upgrade >>>>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Setting container_image for all nvmeof >>>>> 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Setting container_image for all nvmeof >>>>> 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Finalizing container_image settings >>>>> 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Finalizing container_image settings >>>>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO >>>>> cephadm.upgrade] Upgrade: Complete! >>>>> 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) >>>>> log [INF] : Upgrade: Complete! >>>>> >>>>> >>>>> This is really irritating and inconsistent: If the orchestrator >>>>> does honor --limit with other services than OSD, why isn't that >>>>> visible in the logs? And what's with the MONs? Why 2 out of 3? >>>>> >>>>> Any pointers appreciated! Not sure which Ceph versions might be >>>>> affected by this, I'll try out a couple more upgrade paths. >>>>> >>>>> Thanks, >>>>> Eugen >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Replacing an OSD in Squid
by Patrick Begou 08 Sep '26

08 Sep '26

Hi I'm replacing the HDD of osd.2 in my squid cluster following the provided documentation: https://docs.ceph.com/en/squid/rados/operations/add-or-rm-osds/ The OSD was yet out and the daemon down. The osd.2 is destroyed: cephosddestroy2--yes-i-really-mean-it I've stopped the node and replaced the HDD. Ceph (graphic board) see the HDD as available. But I'm unable to go further with the ceph-volume commands: # ceph-volume lvm zap /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found. # ceph-volume lvm prepare --osd-id 2 --data /dev/sda --> RuntimeError: /dev/mapper/osvg-rootvol not found. Something wrong in my cluster or a step is missing in the documentation ? Thank Patrick

2 10

Trouble With Attaching OSDs From A Previous Cluster To A New One
by Jacek Rużyczka 07 Sep '26

07 Sep '26

Hi folks, I've now been struggling for five weeks with my broken Ceph cluster (4 nodes on Ubuntu 24, containerised with Docker) and can't find a solution, not even with help from Google's AI. Here is what happened: 1. The cluster FS was no longer accessible, and even ceph -s no longer worked (error 110). The MON count had gone down to 2, and an hour later, all 4 OSD daemons were also gone. The corresponding containers had also disappeared (docker ps -a no longer displayed them) together with their logs. 2. A week later, one of the nodes got frozen. Accessing its serial console returned a bunch of low-level errors you can read in this forum thread if you wish: https://community.mixtile.com/t/node-3-unreachable-100-cpu-load/941/6 3. It turned out that some essential files on the nodes (like ceph.conf or keyrings) had simply disappeared. Further research revealed trouble with the internal flash storage (eMMC) of the nodes. So, the data on them was apparently corrupt, what forced me to ressort to "safer" eMMC settings, re-flash them, and re-install Ceph (with the FSID of the old cluster, of course). After that, 4 MONs and 2 MGRs worked, but: 4. Re-integrating the existing OSDs (on the NVMe SSDs, not on the internal eMMCs) did not work: Eugen's script https://heiterbiswolkig.blogs.nde.ag/2026/06/28/cephadm-recover-mon-store-u… ended me up with a "No such file or directory" error. Google's AI made a bunch of suggestions, which did not help either (mostly due to "Permission denied" errors). It also told me to shut down one of the MONs, extract a monmap, and inject it into the new cluster. But the "permission denied" errors persisted for at least two weeks until we could get rid of them. 5. Two OSDs could get revivif <https://dict.leo.org/englisch-deutsch/revivify>ied, but ceph -s marked them as "in", but "down" whatever we did. Extracting the keys from the OSDs and adding them to the config of the "new" cluster didn't help. So I got stuck in a dead end. 6. Now, Google's AI wants me to extract the raw data (which is hopefully still complete) from the drives for whatever purpose and store it on another computer, but I'm afraid that doing this won't help me either. So I need some reliable expert advice in order to bring my cluster back to life. What shall I do here? Are the files in my cluster maybe also corrupted by the abovementioned incident? Thank you. Regards Jacek

1 0

Mount Client/Superblock Sharing Broken Bugfix
by Benjamin Huth 07 Sep '26

07 Sep '26

Hey folks, I opened a bug report on the ceph issue tracker a couple months ago detailing how the mount sharing for the cephfs kernel client was broken and how it was impacting me as a talos user with lots of pods sharing the same, readonly cephfs mounts. The bug report is here: https://tracker.ceph.com/issues/78440 I noticed some work being done in approximately the same area by Xiubo Li from Clyso, and was hoping someone could piggyback my fix into what he's doing: https://github.com/ceph/ceph-client/commit/b9d944f1c8c2044d08609fdc238d27df… My patch looks like this and is compatible with his new SB_ACTIVE flag: fs/ceph/super.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/ceph/super.c b/fs/ceph/super.c index 72935f665f11..13a8f246f1a0 100644 --- a/fs/ceph/super.c +++ b/fs/ceph/super.c @@ -1265,7 +1265,8 @@ static int ceph_compare_super(struct super_block *sb, struct fs_context *fc) doutc(cl, "fsid doesn't match\n"); return 0; } - if (fc->sb_flags != (sb->s_flags & ~SB_BORN)) { + if (fc->sb_flags != (sb->s_flags & + ~(SB_BORN | SB_NODIRATIME | SB_NOATIME | SB_ACTIVE))) { doutc(cl, "flags differ\n"); return 0; } base-commit: dc173b37415e8f738fc4de477490056b479ddc9f I've been running this patch on my talos cluster and it does correctly resolve pods not sharing the same mount and inode cache. Thanks!

1 0

Authentication looks broken
by Sake 06 Sep '26

06 Sep '26

Hi all! We upgraded a staging environment from 19.2.4 to 19.2.6 and rotated all of the keys without issues for the cluster. But for the Cephfs clients we noticed something strange. We created new keys for the Cephfs (kernel) clients so they could switch-over with just a remount. All went fine and thus we removed the old keys. Still everything kept working. Only then we rebooted one of the nodes to apply a new kernel. But it wasn't accessible after a couple of minutes, so we checked the console output and there where authentication errors for mds and mon. With a reset the node got online again, but without it's cephfs mount. When investigating the issue, we forgot to adjust the client name in the fstab. It was using the old name instead of the new name. So it looks like when mounting a cephfs filesystem, the backend isn't checking if the combination of client name and key is correct. Just if either is correct, you can mount the filesystem. Also, why isn't the umount failing after some time when authentication fails and then reboots the machine? Are others hitting this issue also? I'm going to check this on another system later on, but I thought I share this observation. System info Ceph is deployed via Cephadm Clients are running on RHEL 9 Using kernel mounts Installed ceph-common 19.2.6 on clients Kind regards, Sake

1 0

Unable to connect to filesystem Tentacle 20.2.4
by Johan 06 Sep '26

06 Sep '26

I have a cephadm managed cluster, Tentacle 20.2.4 and Debian 12 & 13. On the Debian 12 Bookworm servers I can connect to the cluster. id: <ceph-id> health: HEALTH_OK (muted: AUTH_INSECURE_CLIENT_KEY_TYPE(13d AUTH_INSECURE_KEYS_ALLOWED(13d)) services: mon: 3 daemons, quorum lagenhet4,lagenhet3,lagenhet5 (age 117m) [leader: lagenhet4] mgr: lagenhet3.rbkplm(active, since 2h), standbys: lagenhet4.xontzc mds: 1/1 daemons up, 1 standby osd: 8 osds: 8 up (since 2h), 8 in (since 12d) ... Worth noting is that my osd-disks are encrypted and those keys are not rotated. Keys for the crash services are not rotated either (a number of bootstrap-users are left too, but read in a earlier conversation that those should be safe to delete). One client that I previously have been using to connect to a cephfs have had the key rotated, but I can't connect to the filesystem. If the keyfile is written as before [client.fsuser] key = <key> or [client.fsuser] key = <key> caps mds = "allow rw fsname=filesystem" caps mon = "allow r fsname=filesystem" caps osd = "allow rw tag cephfs data=filesystem" and only replace the key I get the error adding ceph secret key to kernel failed: Unknown error 524 couldn't append secret option: -524 when I try to mount the filesystem. If I only have the key itself in the keyfile I get mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorised To test if I have proper authorisation I created a new user sudo ceph fs authorize newuser client.newuser / rw | sudo tee /etc/ceph/ceph.client.newuser.keyring and I still get the same errors. Looking at the keys in the dashboard it shows the same keys as I'm using. I tested to create a new filesystem (information regarding previous filesystems are removed) # ceph osd pool create dummy_data pool 'dummy_data' created # ceph osd pool create dummy_meta pool 'dummy_meta' created # ceph fs new dummyfs dummy_meta dummy_data Pool 'dummy_data' (id '21') has pg autoscale mode 'on' but is not marked as bulk. Consider setting the flag by running # ceph osd pool set dummy_data bulk true new fs with metadata pool 22 and data pool 21 # ceph fs ls name: dummyfs, metadata pool: dummy_meta, data pools: [dummy_data ] # sudo ceph orch apply mds dummyfs Scheduled mds.dummyfs update... # ceph mds stat <deleted>:1 dummyfs:1 {<deleted>:0=<deleted>.lagenhet4.kvscuw=up:active,dummyfs:0=dummyfs.lagenhet5.jpqcdl=up:active} 2 up:standby # ceph fs authorize dummyfs client.dummyuser / rw | tee /etc/ceph/ceph.client.dummyuser.keyring [client.dummyuser] key = AgBvWpxqEJPpOiAAjr6ziBqhdz23pCcU/2pCeY2ZGHNVKV4pAt13jCeKxr0= caps mds = "allow rw fsname=dummyfs" caps mon = "allow r fsname=dummyfs" caps osd = "allow rw tag cephfs data=dummyfs" # mount -t ceph dummyuser@.dummyfs=/ /datastorage/dummyfs adding ceph secret key to kernel failed: Unknown error 524 couldn't append secret option: -524 I manually change keyring-file to include only the key. # mount -t ceph dummyuser@.dummyfs=/ /datastorage/dummyfs mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized This procedure to create a filesystem has worked before. I also tested this procedure on my main cluster, Tentacle 20.2.1, and there was no problem and didn't have to edit keyring-file. Anyone have any idea what could be wrong? /Johan

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Johan 05 Sep '26

05 Sep '26

Sorry for the delayed answer, I have been setting up a new server to try handle the situation. Im using cephadm. The conainers are running fine as far as I can see. Checking both my two clusters I realise ceph-common was not updated since Reef. It hasn't been any problem until now, easy to forget when you have to manually update the release in apt source-file. I now have a server running Debian 12 Bookworm in the cluster (and updated ceph-common) and I can without problem connect to the cluster. id: <ceph-id> health: HEALTH_OK (muted: AUTH_INSECURE_CLIENT_KEY_TYPE(13d AUTH_INSECURE_KEYS_ALLOWED(13d)) services: mon: 3 daemons, quorum lagenhet4,lagenhet3,lagenhet5 (age 117m) [leader: lagenhet4] mgr: lagenhet3.rbkplm(active, since 2h), standbys: lagenhet4.xontzc mds: 1/1 daemons up, 1 standby osd: 8 osds: 8 up (since 2h), 8 in (since 12d) ... I have observed some other problems that I will address in another mail. /Johan Den 2026-08-26 kl. 08:44, skrev Eugen Block: > Hi, > > according to the "OS Recommendations" [0] Ceph Tentacle is not available > for Debian 13. Umbrella will be built for Trixie. > Are you using cephadm managed clusters or package based? With cephadm > you're not that strictly tied to the installed package versions. > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/start/os-recommendations/ > > Zitat von Johan <ceph-users(a)ceph.io>: > >> Ok, >> >> on Debian 12 it was not a problem upgrading to ceph 20.2.4 using apt >> and https://download.ceph.com/debian-tentacle/ >> >> With Debian 13 which I have at my troublesome cluster I get several >> dependency-issues. >> ceph-common is dependent on libicu72 which doesn't have any candidates >> python3-cephfs (and others) is dependent on python3 < 3.12 but >> 3.13.5-1 will be installed >> etc >> >> Is this because I perhaps hadn't upgraded ceph-common and others >> properly before upgrading Debian from 12 to 13? >> >> /Johan >> >> Den 2026-08-24 kl. 11:06, skrev Johan: >>> Im not a professional computer tech but have two ceph clusters >>> (cephadm, Trixie, Debian 12/13). >>> >>> I upgraded the least sensitive one using the dashboard. When I >>> upgrade the admin client I now get >>> auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error >>> setting modifier for [client.admin] type=key val=<key>: Malformed >>> input [buffer:3] >>> auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) >>> Input/ output error >>> >>> I get the same respons when using a ordinary user to connect the >>> filesystem. >>> >>> I found this issue describing the same error >>> https://github.com/rook/rook/issues/18240 >>> >>> As they, in above issue, discussed which version/image of the toolbox >>> was used I checked my version of ceph at the command line. I get >>> ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef >>> (stable) >>> I would guess this is the problem. >>> >>> How do I get to run up-to-date version on the command line? >>> >>> /Johan >>> >>> >>> Den 2026-08-19 kl. 18:09, skrev Patrick Donnelly: >>>> Today, August 19th, 2026, the Ceph project has released Tentacle >>>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>>> components. >>>> >>>> This is the sixth backport release in the Squid series. >>>> >>>> This is the fourth backport release in the Tentacle series. >>>> >>>> Additional details will be posted on the Ceph website/blog: >>>> https://ceph.io/en/news/blog/ >>>> >>>> We strongly recommend that all Ceph operators upgrade to one of these >>>> releases as soon as possible. >>>> >>>> >>>> ## Critical Upgrade Steps >>>> >>>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>>> This is the first time Ceph has introduced a new key type for CephX >>>> credentials. Therefore, a new procedure exists for upgrading and >>>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>>> cluster, the operator should familiarize themselves with the procedure >>>> documented at: >>>> >>>> https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ >>>> index.html#upgrading-and-rotating-cephx-keys >>>> >>>> These steps are for package-based deployments of Ceph. In other >>>> sections, the document also includes new information about key >>>> rotation procedures, monitor authentication controls, and emergency >>>> recovery procedures. >>>> >>>> Deployments using cephadm will automate the process except for client >>>> keys. You may notice that Cephadm spends more time than normal on the >>>> upgrade after updating all daemon images. This is due to a new process >>>> rotating the OSD and MDS keys. Please be patient. There are plans to >>>> improve this in the future. >>>> >>>> Deployments using Rook will also automate rotation of some client keys >>>> with some exclusions. Look to the Rook project’s announcement for more >>>> details. >>>> >>>> Client and kernel upgrades are recommended to support aes256k. Note >>>> that upstream Linux kernel client support began in kernel 7.0 and has >>>> been backported to CentOS Stream 9 and 10. Check with your >>>> distribution vendor for backported key support before rotating client >>>> keys used by the kernel. >>>> >>>> As part of upgrading a Ceph cluster, six new health warnings and >>>> errors will be generated. This is normal. As part of the process to >>>> upgrade/rotate entity keys, you will address the warnings and errors >>>> as you go. These are documented beginning here: >>>> >>>> https://docs.ceph.com/en/latest/rados/operations/health-checks/ >>>> index.html#auth-insecure-keys-creatable >>>> >>>> The CephX documentation has been brought up to date and vetted for >>>> accuracy. Those changes are live here: >>>> >>>> https://docs.ceph.com/en/latest/architecture/index.html#high- >>>> availability-authentication >>>> >>>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>>> requests with `host` and `x-amz-` headers not included in the signed >>>> subset. Unfortunately, the REST client used in multisite was >>>> generating such improperly signed requests. >>>> >>>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>>> option to true before you begin to upgrade. After all clusters are >>>> upgraded, set the option to ``false`` again. >>>> >>>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>>> secrets stored in the Monitor config-key store will be forthcoming. In >>>> the meantime, operators of cephadm-managed clusters are strongly >>>> encouraged to rotate the cephadm SSH key, as an established process >>>> exists: >>>> >>>> See: https://docs.ceph.com/en/latest/cephadm/host-management/#ssh- >>>> configuration >>>> >>>> Rotating other secrets is advised but the process is not yet >>>> established. Operators should assess their cluster's potential >>>> exposure to determine if immediate Monitor config-key secret rotation >>>> is required for their environment. >>>> >>>> >>>> ## CVE Fixes >>>> >>>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>>> of AES-CBC. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2025-30156 >>>> >>>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>>> in improper verification of a cryptographic signature. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-39944 >>>> >>>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>>> subscription handler. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-50152 >>>> >>>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>>> cryptographic signatures in RGW's SigV4 verifier. >>>> >>>> See: https://docs.ceph.com/en/latest/security/CVE-2026-54330 >>>> >>>> ## Other Media >>>> >>>> A Cephalocon 2025 talk covers some of the details surrounding the new >>>> CephX key upgrade process. >>>> >>>> https://cephalocon2025.sched.com/event/27f2O/rotating-heads-and- >>>> cephx- keys-patrick-donnelly-ibm-yehuda-sadeh-weinraub-ubiquiti >>>> >>>> ## Getting Ceph >>>> ------------ >>>> * Source at https://github.com/ceph/ceph >>>> * v19.2.6 Tarball at https://download.ceph.com/tarballs/ >>>> ceph-19.2.6.tar.gz >>>> * v20.2.4 Tarball at https://download.ceph.com/tarballs/ >>>> ceph-20.2.4.tar.gz >>>> * Containers at https://quay.io/repository/ceph/ceph >>>> * For packages, see https://docs.ceph.com/en/latest/install/get- >>>> packages/ >>>> * For upgrade with cephadm, see: >>>> https://docs.ceph.com/en/latest/cephadm/upgrade/ >>>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0